Repository: biopython/biopython

Issue:
Appending to bgzf files misses sequences
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
>>> import sys; print(sys.version)
3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
>>> import platform; print(platform.python_implementation()); print(platform.platform())
CPython
Darwin-17.7.0-x86_64-i386-64bit
>>> import Bio; print(Bio.__version__)
1.73
```

### Expected behaviour

The script in `Steps to reproduce` outputs both of the sequences added to the bgzf file:

```
>hello
aaaaaaaaaaaaaaaaaa
>there
cccccccccccccccccc
```

### Actual behaviour

It only outputs the first sequence:

```
>hello
aaaaaaaaaaaaaaaaaa
```

### Steps to reproduce

Run the following script:

```
import Bio.bgzf as bgzf
from Bio import SeqIO
import gzip
import os
import os.path as op

# file to be used for testing bgzf fasta
bgz_filename = '/tmp/test.bgz'
if op.exists(bgz_filename):
    os.remove(bgz_filename)

# file to be used for testing text fasta
txt_filename = '/tmp/test.fa'
if op.exists(txt_filename):
    os.remove(txt_filename)

# write the first sequence
with bgzf.open(bgz_filename, 'ab') as f, open(txt_filename, 'ab') as f1:
    f.write('>hello\n'.encode('utf8'))
    f.write('aaaaaaaaaaaaaaaaaa\n'.encode('utf8'))

    f1.write('>hello\n'.encode('utf8'))
    f1.write('aaaaaaaaaaaaaaaaaa\n'.encode('utf8'))

# write the second sequence
with bgzf.open(bgz_filename, 'ab') as f, open(txt_filename, 'ab') as f1:
    f.write('>there\n'.encode('utf8'))
    f.write('cccccccccccccccccc\n'.encode('utf8'))

    f1.write('>there\n'.encode('utf8'))
    f1.write('cccccccccccccccccc\n'.encode('utf8'))

print("bgzf")
# the following should print all the lines but it only prints the
# lines for the first sequence

with bgzf.open(bgz_filename, 'rb') as f:
    bgzf_txt = f.read(1000000)
    f.seek(0)
    for line in f:
        print("line:", line)

print("gzip")
# as a control, gzip prints all the lines
with gzip.open(bgz_filename, 'r') as f:
    gzip_txt = f.read(1000000)
    f.seek(0)
    for line in f:
        print("line:", line)

if gzip_txt != bgzf_txt:
    print("ASSERTION FAILED: gzip output does not equal bgzf output")

bgz_index = '/tmp/bg.ix'
if op.exists(bgz_index):
    os.remove(bgz_index)
txt_index = '/tmp/txt.ix'
if op.exists(txt_index):
    os.remove(txt_index)

ix_bgz = SeqIO.index_db(bgz_index, bgz_filename, 'fasta')
ix_txt = SeqIO.index_db(txt_index, txt_filename, 'fasta')

print('ix_bgz', list(ix_bgz.keys()))
print('ix_txt', list(ix_txt.keys()))

assert(len(list(ix_bgz.keys())) == len(list(ix_txt.keys())))
```
If I output both sequences without opening the bgzf file twice then it works as expected. It looks like there's an issue with opening the file more than once to append data.



Discussion:
You mean using append mode when the file already exists seems to be broken?

That does seem possible - looking now I don't think we test append mode in ``Tests/test_bgzf.py`` and although it has been some time since I worked on this, looking at the code in ``Bio/bgzf.py`` it does not seem that when writing in append mode (and the file already exists) that the virtual offset would take into account the pre-existing data (which it probably ought to).

Would explicitly failing if append mode is requested be an acceptable short term solution?
Hey, thanks for the quick response!

> You mean using append mode when the file already exists seems to be broken?

Yes, exactly.

> Would explicitly failing if append mode is requested be an acceptable short term solution?

That seems better than the silent error it produces right now and might save others a bit of debugging time. Depending on the use case, the workaround of writing the entire file at once rather than appending may be acceptable.

That's what I was thinking - explicit failure as a short term solution. Would you like to make a pull request doing that, raising a ``NotImplementedError`` perhaps?

In principle there is nothing to prevent implementing append mode writing in full - the GZIP block structure lends itself to that. The only anticipated complication is reporting an accurate virtual offset would require scanning all the pre-existing blocks - still very doable.
Yup, I can do that. Look out for the PR in the next few days :-)

Resolving patch:
diff --git a/Bio/bgzf.py b/Bio/bgzf.py
--- a/Bio/bgzf.py
+++ b/Bio/bgzf.py
@@ -806,8 +806,7 @@ def __init__(self, filename=None, mode="w", fileobj=None, compresslevel=6):
             if "w" not in mode.lower() and "a" not in mode.lower():
                 raise ValueError(f"Must use write or append mode, not {mode!r}")
             if "a" in mode.lower():
-                raise NotImplementedError("Append mode is not implemented yet")
-                # handle = _open(filename, "ab")
+                handle = _open(filename, "ab")
             else:
                 handle = _open(filename, "wb")
         self._text = "b" not in mode.lower()


Test patch:
diff --git a/Tests/test_bgzf.py b/Tests/test_bgzf.py
--- a/Tests/test_bgzf.py
+++ b/Tests/test_bgzf.py
@@ -384,8 +384,36 @@ def test_write_tell(self):
             self.assertEqual(h.read(5), "Magic")
 
     def test_append_mode(self):
-        with self.assertRaises(NotImplementedError):
-            bgzf.open(self.temp_file, "ab")
+        with bgzf.open(self.temp_file, "wb") as h:
+            h.write(b">hello\n")
+            h.write(b"aaaaaaaaaaaaaaaaaa\n")
+            h.flush()
+            previous_offsets = bgzf.split_virtual_offset(h.tell())
+            # Just flushed, so new block
+            self.assertEqual(previous_offsets[1], 0)
+        with bgzf.open(self.temp_file, "ab") as h:
+            append_position = h.tell()
+            self.assertEqual(
+                (previous_offsets[0] + 28, 0),
+                bgzf.split_virtual_offset(append_position),
+            )
+            h.write(b">there\n")
+            self.assertEqual(
+                (previous_offsets[0] + 28, 7), bgzf.split_virtual_offset(h.tell())
+            )
+            h.write(b"cccccccccccccccccc\n")
+        with bgzf.open(self.temp_file, "rb") as h:
+            self.assertEqual(
+                list(h),
+                [
+                    b">hello\n",
+                    b"aaaaaaaaaaaaaaaaaa\n",
+                    b">there\n",
+                    b"cccccccccccccccccc\n",
+                ],
+            )
+            h.seek(append_position)
+            self.assertEqual(list(h), [b">there\n", b"cccccccccccccccccc\n"])
 
     def test_double_flush(self):
         with bgzf.open(self.temp_file, "wb") as h: