Repository: biopython/biopython

Issue:
Provide a function call to check if `Seq` object has undefined sequence data
### Setup

This is more of a feature request, but just to play nice:

```
platform3.8.5 (default, May 27 2021, 13:30:53)
[GCC 9.3.0]
CPython
Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.29
1.79
```

### Expected behaviour

There should be a function or attribute on the new 1.79 `Seq`-with-undefined-sequence that you can check for in the normal flow of the program, without having to try to convert the `Seq` object to `bytes` and catching the exception, like so:

```python
from Bio.Seq import Seq
s = Seq(None, length=6)
if s.is_undefined():  # or s.undefined for the attribute version
    do_stuff()
```

This would provide a similar user experience to using `isinstance(s, UnknownSeq)` in pre-1.79 code.

### Actual behaviour

You need to cast the `Seq` to `bytes` and catch an exception, like so:

```python
from Bio.Seq import Seq, UndefinedSequenceError
s = Seq(None, length=6)

is_undefined = False
try:
    bytes(s)
except UndefinedSequenceError:
    is_undefined = True

if is_undefined:
    do_stuff()
```

Alternatively, I guess you could futz with underscore imports and members, like so:

```python
from Bio.Seq import Seq, _UndefinedSequenceData
s = Seq(None, length=6)

if isinstance(s._data, _UndefinedSequenceData):
    do_stuff()
```

But that relies on implementation details in Biopython that aren't supposed to be exported, hence the underscores...
### Steps to reproduce

Open `Bio/Seq.py` and observe the lack of a dedicated function call or attribute to check for the sequence data being undefined.



Discussion:
@kblin  How exactly would you use this (i.e. what does `do_stuff` do)?

I am not opposed to adding an attribute `defined` to `Seq`. However, there is a gray zone if the sequence contents is defined partially (e.g., the sequence has length 30, but only nucleotides 5-20 are defined, and the other ones are undefined). Such partially defined sequences may occur if the sequence is read from an alignment file, where only the aligned part of the sequence is shown explicitly.
I'm using this to detect WGS master records that don't already have the sequence included. These records can look like this:
```
LOCUS       NZ_BASQ01000000      3396165 bp    DNA     linear   BCT 24-APR-2017
DEFINITION  Microbacterium sp. TS-1, whole genome shotgun sequencing project.
ACCESSION   NZ_BASQ00000000
VERSION     NZ_BASQ00000000.1
FEATURES             Location/Qualifiers
     source          1..3396165
                     /organism="Microbacterium sp. TS-1"
                     /mol_type="genomic DNA"
                     /strain="TS-1"
                     /db_xref="taxon:1344956"
WGS         BASQ01000001-BASQ01000003
//
```
Note how the `LOCUS` line gives a lenght, but only after you fetch the accessions specified in `WGS` will you get the actual sequence.

But because you can also get WGS master records like this:

```
LOCUS       NZ_BASQ01000000      3396165 bp    DNA     linear   BCT 24-APR-2017
DEFINITION  Microbacterium sp. TS-1, whole genome shotgun sequencing project.
ACCESSION   NZ_BASQ00000000
VERSION     NZ_BASQ00000000.1
FEATURES             Location/Qualifiers
     source          1..3396165
                     /organism="Microbacterium sp. TS-1"
                     /mol_type="genomic DNA"
                     /strain="TS-1"
                     /db_xref="taxon:1344956"
WGS         BASQ01000001-BASQ01000003
ORIGIN
        1 aagagggcca cccaatgttg ggtggccctt tttcgtgccc gtgtgaccgc agagttgact
...
//
```
You can't just rely on "there's a `WGS` line, so you need to go fetch the other records to get the sequence", because potentially you already have the sequence available. When downloading the WGS entry from the NCBI website, which version you get depends on some of the view options you set on the page.

Note that in newer WGS records, the counter in the `LOCUS` line more correctly counts the records, like 
```
LOCUS       AAHF01000000              19 rc    DNA     linear   PLN 04-AUG-2014
```
but the same logic applies, really. Especially as the GenBank parser doesn't expose the fact that the count here is records, not base pairs.

In my [`ncbi-genome-download`](https://github.com/kblin/ncbi-acc-download) script, I have a function call to download the WGS parts if needed, like this:
```python
def download_wgs_parts(handle, config):
    """Download all parts of all WGS records in a file handle."""

    if not HAVE_BIOPYTHON:
        return handle

    updated_records = []

    handle.seek(0)
    records = list(SeqIO.parse(handle, config.format))
    for record in records:
        run_download = isinstance(record.seq, UnknownSeq)
        if run_download and ('wgs_scafld' in record.annotations or
                             'wgs' in record.annotations or
                             'tsa' in record.annotations):
            updated_records.extend(download_wgs_for_record(record, config))
        elif run_download and 'contig' in record.annotations:
            updated_records.extend(fix_supercontigs(record, config))
        else:
            updated_records.append(record)

    outhandle = StringIO()
    SeqIO.write(updated_records, outhandle, config.format)
    outhandle.seek(0)
    return outhandle
```

I was using the combination of the sequence being an `UnknownSeq` and one of the many ways of specifying that the sequences are available elsewhere to trigger the download logic. As of 1.79, that check doesn't work anymore because the GenBank parser no longer returns the `UnknownSeq`. 



Hm, actually you still get WGS master records with `bp` counts in them on current records, apparently depending on if you're grabbing the RefSeq or GenBank version of the record.

RefSeq FTP server https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/509/385/GCF_000509385.1_ASM50938v1/GCF_000509385.1_ASM50938v1_wgsmaster.gbff.gz or website https://www.ncbi.nlm.nih.gov/nuccore/NZ_BASQ00000000.1/
```
LOCUS       NZ_BASQ01000000      3396165 bp    DNA     linear   BCT 05-JAN-2021
```

GenBank FTP server https://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/509/385/GCA_000509385.1_ASM50938v1/GCA_000509385.1_ASM50938v1_wgsmaster.gbff.gz or website https://www.ncbi.nlm.nih.gov/nuccore/BASQ00000000.1/
```
LOCUS       BASQ01000000               3 rc    DNA     linear   BCT 22-APR-2017
```
> You need to cast the `Seq` to `bytes` and catch an exception, like so:

It's not really a cast; calling `bytes` on a `Seq` object will call `bytes(self._data)`, and if `self._data` is already a `bytes` object, it will simply return the existing `bytes` object:
```python
>>> from Bio.Seq import Seq
>>> s = Seq("ABCD")
>>> s._data
b'ABCD'
>>> id(s._data)
4496510096
>>> bytes(s)
b'ABCD'
>>> id(bytes(s))
4496510096
```

This snippet will do the same as your original code:

```python
def download_wgs_parts(handle, config):
    """Download all parts of all WGS records in a file handle."""

    if not HAVE_BIOPYTHON:
        return handle

    updated_records = []

    handle.seek(0)
    records = list(SeqIO.parse(handle, config.format))
    for record in records:
        try:
            bytes(record.seq)
        except UndefinedSequenceError:
            if ('wgs_scafld' in record.annotations or
                'wgs' in record.annotations or
                'tsa' in record.annotations):
                updated_records.extend(download_wgs_for_record(record, config))
            elif 'contig' in record.annotations:
                updated_records.extend(fix_supercontigs(record, config))
        else:
            updated_records.append(record)

    outhandle = StringIO()
    SeqIO.write(updated_records, outhandle, config.format)
    outhandle.seek(0)
    return outhandle
```
Yeah, but I really hate having to use an exception as part of my regular control flow. I mean this is kind of similar to what I had above, but I'd really prefer to just have something I can put into a regular `if` statement.
You could create a separate function `is_undefined` and put the `try:` block there.
Any `is_undefined` method or attribute on the `Seq` class in Biopython would use the exact same `try:` block; there is no difference.
with the difference is that I have that kind of check in multiple projects that all need to detect WGS master records and deal with them somehow, so having this upstream would be nice.
I'm happy to do a PR for this if that's something you'd consider adding.
My problem with this is that it is ambiguous. What does `is_undefined` mean if the sequence is partially defined? I am looking for a general solution that will help everybody.
How does the current code decide when to raise that exception?
Alternatively, I'd also be fine if the GenBank parser had a way to tell me that I'm dealing with a WGS master record at `SeqRecord` level, then that'd get me out of that implementation detail level. But at the moment the parser can't even _not_ mangle the `rc` unit on master records for the simple parse and write test case.
> How does the current code decide when to raise that exception?

It happens when Python cannot do what the user is asking for:
- when calling `bytes(seq)`; there is no reasonable `bytes` object we could return here;
- when trying to see one letter from the sequence, so `seq[i]` where `i` is an integer. Again this is because we cannot return any reasonable string here.

But `seq[i:j]` for example will not raise an `UndefinedSequenceError`:
```python
>>> from Bio.Seq import Seq
>>> s = Seq(None, length=10)
>>> s[3:8]
Seq(None, length=5)
```

So we only raise an `UndefinedSequenceError` as a last resort.
So how does that work for your partially defined sequences?
Because in your example, clearly both `s` and `s[3:8]` are undefined.
Can you point me at any code that handles partially defined sequences? Because `Seq.__init___()` doesn't seem to handle these.
Same; we raise the exception if Python cannot do what the user is asking for:
- `bytes(seq)` will raise the exception;
- `seq[i]` will return a letter if the sequence at `i` is known, and raise the exception otherwise;
- `seq[i:j]` will return a fully defined sequence, a partially defined sequence, or an undefined sequence, depending on whether the sequence contents at `[i:j]` is defined.

> Can you point me at any code that handles partially defined sequences? Because Seq.__init___() doesn't seem to handle these.

It's not yet implemented, but this will happen before release 1.80.

Ok, how about this implementation:

```python
    def is_undefined(self):
        return isinstance(self._data, _UndefinedSequenceData)
```

That works with the current code, and adjustments can happen before release 1.80
In your script, if for some reason you get a partially defined sequence, you would want to download it, right? Then this function does not do what you want. Works today, but may break in a couple of weeks.
You mean like the code broke from 1.78 to 1.79 because the meaning of `UnknownSeq` changed?
Ok, again, because I need this in multiple projects, and I'd prefer to have this upstream rather than having to write it on every one of these projects:

```python
    def has_any_undefined(self):
        try:
            bytes(self._data)
            return False
        except UndefinedSequeceError:
            return True
```
This would be nice, especially as a bit of future proofing. It's easier for code relying on biopython to avoid surprises if things like this can be abstracted into a function instead of relying on specific errors being thrown. This change in 1.78 to 1.79 broke some things that didn't expect errors to be thrown, if in the future it's changed to not throw an error (like how the genbank parsers don't throw errors, but instead guess at interpretations) it'd be nice if all the code relying on biopython doesn't have byzantine errors creep in just because 1.80 or later decided that it wasn't worth throwing that error specifically, but still had missing sequence content.
> You mean like the code broke from 1.78 to 1.79 because the meaning of `UnknownSeq` changed?

Exactly. `UnknownSeq` was deprecated in 1.79 because its purpose and usage were not clearly defined, leading to ambiguous behavior. See the code change history and discussion on github for details. If we had thought things through a bit more initially, `UnknownSeq` would not have been introduced into Biopython in the form that it did.

Also, it is easy to add new code to Biopython; perhaps too easy. It is much more painful to remove code from Biopython, as it requires multiple releases with deprecation warnings, and a one year waiting time, and even then some user code may break, as you are now experiencing.

As we don't want to repeat our past mistakes, we now apply a bit more scrutiny and ask more questions when new features are being suggested. 
@peterjc  How do you think (as the `Bio/Seq.py` code owner)?
I am OK with adding a `defined` attribute that
- returns `True` if `bytes(seq)` would be successful, and `False` it if would raise an `UndefinedSequenceError`;
- works on `Seq` and `MultipleSeq`;
- is documented in the Biopython tutorial;
- has a few tests.

It may be better to implement this by looking at the `_data` attribute instead of simply trying to call `bytes`, as `bytes` can be an expensive operation for lazy parsers such as in `BioSQL` and for `twoBit` files.

I think we were too hasty in removing the ``UnknownSeq``, but it is done now. Cross reference #3301 and linked issues/PR.

Given all the recent ``Bio/Seq.py`` work, you (@mdehoon) should be listed as another code owner of this file too:
https://github.com/biopython/biopython/blob/master/.github/CODEOWNERS

I can see the argument for an (all) ``.defined`` attribute here (or something similar more along the lines Kai is suggesting), and having it this way round should work if you want to support "masked" sequences which are only partly known (and which would return ``False``).
@peterjc  Thank you. I have added myself as code owner in #3668 .

>or something similar more along the lines Kai is suggesting

Specifically what are you referring to?
A positive or a negative test (and your example of a partly defined sequence is relevant here), but also method vs attribute vs function. If the check is quick (not computationally expensive), I think the attribute (i.e. read only property) is neatest.
> If the check is quick (not computationally expensive), 

It is always quick. Only `_UndefinedSequenceData` can raise `UndefinedSequenceError`, and it will always do so except when the sequence length is 0. Same is true for a future partial sequence object.
Hi everyone - I came to this thread because I was wondering the same thing as @kblin - I have some code that uses `UnknownSeq` and I want to replace this and keep a simple way to check that the sequence is defined (or not). I'm not super clear from the thread what the correct code to solve this is, but I'd like to +1 on not having to use try: .. except: to test if a sequence is undefined.

Resolving patch:
diff --git a/Bio/Seq.py b/Bio/Seq.py
--- a/Bio/Seq.py
+++ b/Bio/Seq.py
@@ -284,14 +284,14 @@ def lower(self):
     def isupper(self):
         """Return True if all ASCII characters in data are uppercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         return bytes(self).isupper()
 
     def islower(self):
         """Return True if all ASCII characters in data are lowercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         return bytes(self).islower()
 
@@ -1330,14 +1330,14 @@ def lower(self, inplace=False):
     def isupper(self):
         """Return True if all ASCII characters in data are uppercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         return self._data.isupper()
 
     def islower(self):
         """Return True if all ASCII characters in data are lowercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         return self._data.islower()
 
@@ -1981,7 +1981,12 @@ def __init__(self, data, length=None):
         if data is None:
             if length is None:
                 raise ValueError("length must not be None if data is None")
-            self._data = _UndefinedSequenceData(length)
+            elif length == 0:
+                self._data = b""
+            elif length < 0:
+                raise ValueError("length must not be negative.")
+            else:
+                self._data = _UndefinedSequenceData(length)
         elif isinstance(data, (bytes, SequenceDataAbstractBaseClass)):
             self._data = data
         elif isinstance(data, (bytearray, _SeqAbstractBaseClass)):
@@ -1989,7 +1994,42 @@ def __init__(self, data, length=None):
         elif isinstance(data, str):
             self._data = bytes(data, encoding="ASCII")
         elif isinstance(data, dict):
-            self._data = _PartiallyDefinedSequenceData(length, data)
+            if length is None:
+                raise ValueError("length must not be None if data is a dictionary")
+            elif length == 0:
+                self._data = b""
+            elif length < 0:
+                raise ValueError("length must not be negative.")
+            else:
+                end = -1
+                starts = sorted(data.keys())
+                _data = {}
+                for start in starts:
+                    seq = data[start]
+                    if isinstance(seq, str):
+                        seq = bytes(seq, encoding="ASCII")
+                    else:
+                        try:
+                            seq = bytes(seq)
+                        except Exception:
+                            raise ValueError("Expected bytes-like objects or strings")
+                    if start < end:
+                        raise ValueError("Sequence data are overlapping.")
+                    elif start == end:
+                        _data[current] += seq  # noqa: F821
+                    else:
+                        _data[start] = seq
+                        current = start
+                    end = start + len(seq)
+                if end > length:
+                    raise ValueError(
+                        "Provided sequence data extend beyond sequence length."
+                    )
+                elif end == length and current == 0:
+                    # sequence is fully defined
+                    self._data = _data[current]
+                else:
+                    self._data = _PartiallyDefinedSequenceData(length, _data)
         else:
             raise TypeError(
                 "data should be a string, bytes, bytearray, Seq, or MutableSeq object"
@@ -2066,13 +2106,10 @@ def defined(self):
 
         Zero-length sequences are always considered to be defined.
         """
-        if len(self) == 0:
+        if isinstance(self._data, (bytes, bytearray)):
             return True
-        elif isinstance(self, UnknownSeq):
-            return False
-        elif isinstance(self._data, SequenceDataAbstractBaseClass):
+        else:
             return self._data.defined
-        return True
 
 
 class UnknownSeq(Seq):
@@ -2643,6 +2680,16 @@ def join(self, other):
             return self.__class__(len(temp_data), character=self._character)
         return Seq(temp_data)
 
+    @property
+    def defined(self):
+        """Return True if the sequence is defined, False if undefined or partially defined.
+
+        Zero-length sequences are always considered to be defined.
+        """
+        if self._length == 0:
+            return True
+        return False
+
 
 class MutableSeq(_SeqAbstractBaseClass):
     """An editable sequence object.
@@ -2886,9 +2933,11 @@ class _UndefinedSequenceData(SequenceDataAbstractBaseClass):
     __slots__ = ("_length",)
 
     def __init__(self, length):
-        """Initialize the object with the sequence length."""
-        if length < 0:
-            raise ValueError("Length must not be negative.")
+        """Initialize the object with the sequence length.
+
+        The calling function is responsible for ensuring that the length is
+        greater than zero.
+        """
         self._length = length
         super().__init__()
 
@@ -2906,8 +2955,6 @@ def __len__(self):
         return self._length
 
     def __bytes__(self):
-        if self._length == 0:
-            return b""
         raise UndefinedSequenceError("Sequence content is undefined")
 
     def __add__(self, other):
@@ -2944,7 +2991,7 @@ def lower(self):
     def isupper(self):
         """Return True if all ASCII characters in data are uppercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         # Character case is irrelevant for an undefined sequence
         raise UndefinedSequenceError("Sequence content is undefined")
@@ -2952,7 +2999,7 @@ def isupper(self):
     def islower(self):
         """Return True if all ASCII characters in data are lowercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         # Character case is irrelevant for an undefined sequence
         raise UndefinedSequenceError("Sequence content is undefined")
@@ -2968,12 +3015,7 @@ def replace(self, old, new):
 
     @property
     def defined(self):
-        """Return True if the sequence is defined, False if undefined or partially defined.
-
-        Zero-length sequences are always considered to be defined.
-        """
-        if len(self) == 0:
-            return True
+        """Return False, as the sequence is not defined and has a non-zero length."""
         return False
 
 
@@ -2988,35 +3030,16 @@ class _PartiallyDefinedSequenceData(SequenceDataAbstractBaseClass):
     raised.
     """
 
-    __slots__ = ("_length", "_data", "_data_length")
+    __slots__ = ("_length", "_data")
 
     def __init__(self, length, data):
-        """Initialize with the sequence length and defined sequence segments."""
-        if length < 0:
-            raise ValueError("Length must not be negative.")
-        position = 0
-        starts = sorted(data.keys())
-        _data = {}
-        _data_length = 0
-        for start in starts:
-            seq = data[start]
-            if isinstance(seq, str):
-                seq = bytes(seq, encoding="ASCII")
-            else:
-                try:
-                    seq = bytes(seq)
-                except Exception:
-                    raise ValueError("Expected bytes-like objects or strings")
-            if start < position:
-                raise ValueError("Sequence data are overlapping.")
-            _data[start] = seq
-            _data_length += len(seq)
-            position = start + len(seq)
-        if position > length:
-            raise ValueError("Provided sequence data extend beyond sequence length.")
+        """Initialize with the sequence length and defined sequence segments.
+
+        The calling function is responsible for ensuring that the length is
+        greater than zero.
+        """
         self._length = length
-        self._data = _data
-        self._data_length = _data_length
+        self._data = data
         super().__init__()
 
     def __getitem__(self, key):
@@ -3086,8 +3109,6 @@ def __len__(self):
         return self._length
 
     def __bytes__(self):
-        if self._length == 0:
-            return b""
         raise UndefinedSequenceError("Sequence content is only partially defined")
 
     def __add__(self, other):
@@ -3166,7 +3187,7 @@ def lower(self):
     def isupper(self):
         """Return True if all ASCII characters in data are uppercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         # Character case is irrelevant for an undefined sequence
         raise UndefinedSequenceError("Sequence content is only partially defined")
@@ -3174,7 +3195,7 @@ def isupper(self):
     def islower(self):
         """Return True if all ASCII characters in data are lowercase.
 
-        If there are no cased characters, method returns False.
+        If there are no cased characters, the method returns False.
         """
         # Character case is irrelevant for an undefined sequence
         raise UndefinedSequenceError("Sequence content is only partially defined")
@@ -3209,12 +3230,7 @@ def replace(self, old, new):
 
     @property
     def defined(self):
-        """Return True if the sequence is defined, False if undefined or partially defined.
-
-        Zero-length sequences are always considered to be defined.
-        """
-        if self._length == self._data_length:
-            return True
+        """Return False, as the sequence is not fully defined and has a non-zero length."""
         return False
 
 


Test patch:
diff --git a/Tests/test_Seq_objs.py b/Tests/test_Seq_objs.py
--- a/Tests/test_Seq_objs.py
+++ b/Tests/test_Seq_objs.py
@@ -1611,7 +1611,7 @@ def test_init(self):
         self.assertEqual(repr(seq), "Seq({5: 'ACGT'}, length=20)")
         with self.assertRaises(ValueError) as cm:
             Seq({5: "ACGT"}, length=-10)
-        self.assertEqual(str(cm.exception), "Length must not be negative.")
+        self.assertEqual(str(cm.exception), "length must not be negative.")
         with self.assertRaises(ValueError) as cm:
             Seq({5: 1.5}, length=10)
         self.assertEqual(str(cm.exception), "Expected bytes-like objects or strings")