Repository: biopython/biopython

Issue:
Allow modifications of sequence in SFF objects
Migrated from https://redmine.open-bio.org/issues/3438 (_Note - don't write # number as GitHub will auto-link that as if it was a GitHub issue; Peter_)

Original discussion from @mmokrejs:

> I find it a bit awkward but namely, with unnecessary overhead, to edit a sequence in a SFF object. I am fine with the requirement that the length of the sequence must stay same is the length of qualities and other annotation lists. However, to edit a sequence I have to do now:

```
    if _was_modified:
        _letter_annotations = _record.letter_annotations
        _annotations = _record.annotations
        _record.letter_annotations = {}
        _record.annotations = {}
        _record.seq = Seq(_sequence, generic_dna)
        _record.letter_annotations = _letter_annotations
        _record.annotations = _annotations

        _new_record = SeqRecord(Seq(_sequence, generic_dna), id=_record.id, name=_record.name, description=_record.description, annotations=_record.annotations, letter_annotations=_record.letter_annotations)

        _wrote = SeqIO.write(_new_record, _fh, 'sff')
    else:
        _wrote = SeqIO.write(_record, _fh, 'sff')
```

> The whole work in backup&restore of annotation lists is not necessary in my eyes. I think providing _record.rewrite_sequence('tcagnnnnnnnn') would be quite helpful here.



Discussion:
Doesn't seem to be resolved:

```
$ python
Python 2.7.12 (default, Jun 29 2016, 14:05:02) 
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from Bio import SeqIO
>>> from Bio.Seq import Seq
>>> record = SeqIO.read('SRR088776_short_F95S1KP01DA6OU.sff', 'sff')
/usr/local/lib/python2.7/site-packages/Bio/SeqIO/SffIO.py:671: BiopythonParserWarning: Overlapping clip values in SFF record
  warnings.warn("Overlapping clip values in SFF record", BiopythonParserWarning)
/usr/local/lib/python2.7/site-packages/Bio/SeqIO/SffIO.py:988: BiopythonParserWarning: Your SFF file is technically invalid as it is missing a terminal 4 byte null padding region.
  BiopythonParserWarning)
>>> dir(record)
['__add__', '__bool__', '__class__', '__contains__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le___', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__nonzero__', '__radd__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_per_letter_annotations', '_seq', '_set_per_letter_annotations', '_set_seq', 'annotations', 'dbxrefs', 'description', 'features', 'format', 'id', 'letter_annotations', 'lower', 'name', 'reverse_complement', 'seq', 'upper']
>>> new_seq = str(record.seq).replace('n', 'c')
>>> record.seq = Seq(new_seq)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/Bio/SeqRecord.py", line 301, in _set_seq
    raise ValueError("You must empty the letter annotations first!")
ValueError: You must empty the letter annotations first!
>>> 
```

Yes, this is a usability design issue - we enforce the per-letter-annotations dictionary entries being the same length as the sequence, which makes changing them both together tricky.

Currently you are expected to do clear this before you can change the sequence:

```
record.letter_annotations = {}
record.seq = Seq(new_seq)
```

I'm not convinced that adding a new method would actually help here. One alternative would be to allow the new string assignment, and if the length differs, drop the per-letter-annotations with a warning.

Well, as I show in my code I of course do not want to discard any of the annotations, and in the code I copy them all back. The whole workaround just obeys biopython's roadblocks so I could edit the sequence, and in my eyes it would be nice to do just:

_record.rewrite_sequence('tcagnnnnnnnn')

Fine with me, the code should require length of the sequence to be same as of the list in annotation dictionaries. I can live without this, but provided one cannot slice or edit SFF objects at all, this would be a nice although small step forward.

> and if the length differs, drop the per-letter-annotations with a warning.

Better not, maybe an optional argument to override the blocker, but no discards of any data please. I know we could easily run into a problem with too many Warnings, so moving to WarnOnce, etc. I think it is better to avoid this and eventually provide a swithc to override the default deny if lengths differ. But this is certainly a different thing than I asked for, so do not worry about this. ;)

Here's a concrete example:

```
>>> from Bio.Alphabet import generic_dna
>>> from Bio.Seq import Seq
>>> from Bio import SeqIO
>>> record_dict = SeqIO.index("Tests/Roche/E3MFGYR02_random_10_reads.sff", "sff")
>>> record = record_dict["E3MFGYR02GFKUC"]
>>> print(len(record))
299
```

If replacing the sequence, this typically would invalidate the old per-letter-annotations. So, to avoid accidental data loss by design I force you to do this explicitly:

```
>>> record.letter_annotations = {}
>>> record.seq = Seq("ACGT", generic_dna)
>>> record.letter_annotations["phred_quality"] = [40, 40, 40, 40]
```

There is a special case where perhaps the `ValueError` is not needed, which is when the new sequence is the same length (e.g. you are editing the case of the letters, or masking some of them), e.g.

```
>>> record.seq = Seq("N" * len(record), generic_dna)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "Bio/SeqRecord.py", line 301, in _set_seq
    raise ValueError("You must empty the letter annotations first!")
ValueError: You must empty the letter annotations first!
```

@mmokrejs Are you replacing the sequence with one of the same length or not?

> Are you replacing the sequence with one of the same length or not?

Sorry Peter for being unclear here, yes, I just modify the sequence while the length is unchanged. The code I posted initially does not change length of the sequence either. Therefore

`record.letter_annotations = {}`

can be avoided and the ValueError is really more a nuisance than a necessary check.


Resolving patch:
diff --git a/Bio/SeqRecord.py b/Bio/SeqRecord.py
--- a/Bio/SeqRecord.py
+++ b/Bio/SeqRecord.py
@@ -292,19 +292,28 @@ def _set_per_letter_annotations(self, value):
         >>> sub_record.letter_annotations = {}
         >>> sub_record.letter_annotations
         {}
+
+        Note that if replacing the record's sequence with a sequence of a
+        different length you must first clear the letter_annotations dict.
         """)
 
     def _set_seq(self, value):
         # TODO - Add a deprecation warning that the seq should be write only?
         if self._per_letter_annotations:
-            # TODO - Make this a warning? Silently empty the dictionary?
-            raise ValueError("You must empty the letter annotations first!")
-        self._seq = value
-        try:
-            self._per_letter_annotations = _RestrictedDict(length=len(self.seq))
-        except AttributeError:
-            # e.g. seq is None
-            self._per_letter_annotations = _RestrictedDict(length=0)
+            if len(self) != len(value):
+                # TODO - Make this a warning? Silently empty the dictionary?
+                raise ValueError("You must empty the letter annotations first!")
+            else:
+                # Leave the existing per letter annotations unchanged:
+                self._seq = value
+        else:
+            self._seq = value
+            # Reset the (empty) letter annotations dict with new length:
+            try:
+                self._per_letter_annotations = _RestrictedDict(length=len(self.seq))
+            except AttributeError:
+                # e.g. seq is None
+                self._per_letter_annotations = _RestrictedDict(length=0)
 
     seq = property(fget=lambda self: self._seq,
                    fset=_set_seq,


Test patch:
diff --git a/Tests/test_SeqRecord.py b/Tests/test_SeqRecord.py
--- a/Tests/test_SeqRecord.py
+++ b/Tests/test_SeqRecord.py
@@ -1,4 +1,4 @@
-# Copyright 2009 by Peter Cock.  All rights reserved.
+# Copyright 2009-2016 by Peter Cock.  All rights reserved.
 # This code is part of the Biopython distribution and governed by its
 # license.  Please see the LICENSE file that should have been included
 # as part of this package.
@@ -72,6 +72,23 @@ def test_letter_annotations(self):
         except (TypeError, ValueError) as e:
             pass
 
+    def test_replacing_seq(self):
+        """Replacing .seq if .letter_annotation present."""
+        rec = SeqRecord(Seq("ACGT", generic_dna),
+                        id="Test", name="Test", description="Test",
+                        letter_annotations={"example": [1, 2, 3, 4]})
+        try:
+            rec.seq = Seq("ACGTACGT", generic_dna)
+            self.fail("Changing .seq length with letter_annotations present should fail!")
+        except ValueError as e:
+            self.assertEqual(str(e), "You must empty the letter annotations first!")
+        # Check we can replace IF the length is the same
+        self.assertEqual(str(rec.seq), "ACGT")
+        self.assertEqual(rec.letter_annotations, {"example": [1, 2, 3, 4]})
+        rec.seq = Seq("NNNN", generic_dna)
+        self.assertEqual(str(rec.seq), "NNNN")
+        self.assertEqual(rec.letter_annotations, {"example": [1, 2, 3, 4]})
+
     def test_valid_id(self):
         with self.assertRaises(TypeError):
             SeqRecord(Seq("ACGT", generic_dna), id=dict())