Repository: biopython/biopython

Issue:
CompoundLocation surprises for features that wrap the origin
On a circular genome, features that start upstream of the 0 coordinate are annotated as compound locations in GenBank.
According to the [CompoundLocation docs](https://github.com/biopython/biopython/blob/master/Bio/SeqFeature.py#L981), 

> More generally, you can use the compound location's start and end which give the full range covered

For a wraparound feature, this doesn't work:

``` python
from Bio.SeqFeature import FeatureLocation, CompoundLocation
f1 = FeatureLocation(450, 500, strand=1)
f2 = FeatureLocation(0, 50, strand=1)
cl = CompoundLocation([f1, f2])

print(cl.start, cl.end)
# prints 0 500

print(list(cl)[0], list(cl)[-1])
# prints 450 49
```

This contradicts the docstring. Arguably, all the examples and doctests also give examples for the behaviour that contradicts that docstring statement.

I'm not exactly sure how to fix this for more complex CompoundLocations, but certainly in the example given, 0-500 isn't the range covered by the CompoundLocation, but my complete "genome". For a less artificial example, you can check out the first "gene" or "CDS" feature of RefSeq record NZ_CP014634.1.



Discussion:
I thought this was clear from the full text you quoted from:

```
    More generally, you can use the compound location's start and end which
    give the full range covered, 0 <= start <= end <= full sequence length.
    >>> f.start == min(f)
    True
    >>> f.end == max(f) + 1
    True
    This is consistent with the behaviour of the simple FeatureLocation for
    a single region, where again the 'start' and 'end' do not necessarily
    give the biological start and end, but rather the 'minimal' and 'maximal'
    coordinate boundaries.
```

In your example I do consider the range to be 0 to 500. This is by design, for compound locations you get the min and max numerical values (subject to Python end point counting), i.e. the left-most and right-most coordinates, which is the range as needed for drawing a figure etc.

Some questions:

(a) How can we make this clearer? What do you view as a contradiction in the current description. Do you think an origin wrapping example needs to be included here? Is span a better word than range (both are problematic for discontinuous regions).

(b) What numbers do you want?

(c) Did your usage pre-date the `CompoundLocation`, and if so what did your old code do?

For background see also: http://lists.open-bio.org/pipermail/biopython-dev/2012-July/018979.html ... http://lists.open-bio.org/pipermail/biopython-dev/2012-September/019101.html

Basically, if I have a gene from location 450..50 (because it wraps the origin), I expect the range covered to be 450..50, not "smallest coordinate to largest coordinate". For every feature that wraps around, we're currently always giving 0..( len(genome)-1 ), no matter the length of the feature.

At the very least, for me this violates the principle of least surprise. To me, the range covered are the coordinates that are in `list(compound_loc)`. I fail to see how "oh, it covers the whole genome" is either correct or useful for a feature that wraps over the origin (which is pretty arbitrary on a circular genome anyway).

To put it another way, if I have a compund location from 450..50, and I have an integer location on the genome, say 60, then  I would expect that `60 in compound_loc` returns `False`. Which it does. but if I checked the "range covered", by the current logic 60 would be in that range.

But even in your reply, I don't understand how the current behaviour of the compound location helps in any way.

> In your example I do consider the range to be 0 to 500. This is by design, for compound locations you get the min and max numerical values (subject to Python end point counting), i.e. the left-most and right-most coordinates, which is the range as needed for drawing a figure etc.

If you're drawing a figure, the correct box for the feature would be from 450 to 50, not from 0 to 500. This is how I found this bug in the first place. I was using the `start` and `end` coordinates to draw my feature and got a big fat arrow covering my whole genome. 

I see the use case for how the current code works for spliced features (as provided in your example links), but for a circular genome, that just makes no sense for me whatsoever.

Oh, I realized I missed responding to (c). I don't remember, to be honest. We initially used our own parser code and only migrated to BioPython at a later point in time. Unfortunately, the initial version that uses SeqIO doesn't come with a version specified in the `requirements.txt` file, so I can't tell what the initial version we used was.

It sounds like you want the biological start and end, basically `list(compound_loc)[0]` and `list(compound_loc)[-1]` (subject to the usual end point translation for Python notation). However, the historical usage of start and end in the `SeqFeature`'s simple location has always ignored the strand and had `start <= end`, so this was not sensible for the `CompoundLocation`.

I'm sure we've debated this on the mailing list before, I've certainly considered additional methods/attributes for this. Presumably for strandedness features you'd default to treating them like forward strand for the purposes of assigning a notional start and end?

Regarding drawing, this example on a circular figure would be from 450 to 50 _via the origin_, or on a linear figure two parts from 450 to 500 and 0 to 50. In general for drawing you cannot simplify the GenBank join locations to a single pair of coordinates (and a strand), trans-splicing is the perfect counter example. Biopython's GenomeDiagram also has to deal with this too - the best approach is often to use `compound_loc.parts` to draw a sigil for the pieces (here 450 to 500, and 0 to 50).

When I say range, I mean two coordinates on a _linear_ number line (not modulo circular genome length), with 0 <= a <= b < genome-length. If instead of `.start` and `.end`, we had always used `.left` and `.right` would everything make sense?

Not really, because I guess you'd argue that `.left` is 0, but for me `.left` would be 450. :)

I think we're using different mental models here. I have the impression the code looks at a feature on a circular chromosome the same way as it would look at a feature that was spliced at both ends of a linear chromosome. But I still fail to see how 0..500 is a useful value, even in that mental model.

We are using different mental models, yes. It may to be possible to add attributes for the numbers you want in parallel with the existing system.

Our/my approach does perhaps prioritise working with the more common annotation of spliced genes, but in any system annotations wrapping the origin of a circular genome are going to be special cases. And yes, just like the GenBank/EMBL notation, simple origin wrapping annotations are treated like a splice at the origin.

Here I agree that range `[0, 500)` isn't always the most useful, but it is the only self consistent range we can use with `[a, b)` where 0 <= a <= b < genome-length which captures the full range of positions covered by the feature.

One example usage: If you wanted to zoom a linear genome browser to fully show this feature, you would need 0..500.

Maybe `.min` and `.max` or `.left_most` and `.right_most` is better for the current `.start` and `.end`?

Thinking about a bit more, the actual thing I'm worried about here is that we're currently working around `FeatureLocation.__contains__()` not handling other feature locations by using a [helper function](https://bitbucket.org/antismash/antismash/src/4e14398bebb901a582c5fec2893f1d6326fb072a/antismash/utils.py?at=master&fileviewer=file-view-default#utils.py-217):

``` python
def features_overlap(a, b):
    "Check if two features have overlapping locations"
    astart = min([a.location.start, a.location.end])
    aend = max([a.location.start, a.location.end])
    bstart = min([b.location.start, b.location.end])
    bend = max([b.location.start, b.location.end])
    return (astart >= bstart and astart <= bend) or \
           (aend >= bstart and aend <= bend) or \
           (bstart >= astart and bstart <= aend) or \
           (bend >= astart and bend <= aend)
```

This function falls on its face nicely for wrapped features, because every wrapped feature "contains" the whole genome.

Now, naively I could imagine to fix this by using `list(a.location)[0]` instead of `a.location.start`, but that only works for locations with strand = 1. `FeatueLocation` nicely inverts the `list()` representations for strand = -1 features. So if I take the initial example I gave above, and just set strand to -1 for both `f1` and `f2`, `list(cl)` now goes from 499..450 and then from 49..0.

If I also invert the order I build the `CompoundLocation` in, then the list works again, but I'm not willing to bet that all the GenBank generators out there will get the annotation right. (Actually, if I had to bet, I'd bet that there's plenty that get it wrong.)

So I guess my question is how to implement the "features overlap" check in a way that also works for CompoundLocations that span the origin, if the current behaviour of `.start` and `.stop` is as designed?

> One example usage: If you wanted to zoom a linear genome browser to fully show this feature, you would need 0..500

Ok, that's true, but also rather useless in the real world, as you wouldn't be able to actually see anything at that zoom level. But that just proves that using a linear genome browser for a circular genome is a  bad idea, and a better solution would be to show the (linearized) view of the genome that also spans the origin. But ranting about geometricism in genome browsers probably should go to a blog post, not here. :)

Seeing if one feature contains another seems fraught with corner cases, but an overlap check looks more doable - new feature request please?

Brute force solution which does not consider annoying zero-length features, but otherwise ought to be robust:

```
def features_overlap(a, b):
    # This does not consider zero length features like 123^124 in GenBank
    # Works using the base-wise iteration method of the location objects
    return bool(set(a).intersection(b))
```

(_Update_ example corrected to `123^124`)

Possible reference implementation to optimise against?

Hm, I need to give this a try (and then find all the places in our code that doesn't use the helper function). I'll create a feature request for `FeatureLocation.overlaps()` asap.

I've tried to make the `.start` and `.end` docstrings more explicit with 5ee642d9beea494aae362d78586c587650667226

I think we need to add an origin wrapping example to the `SeqFeature.py` docstrings (perhaps also a trans-splicing example too) to resolve this issue (separate from the proposed feature overlap function/method which will be a separate issue)

Yes, thanks. That makes things more explicit at least. I guess it's a good idea to add examples for the corner cases, especially as the example code also is run as doctest and testing corner cases is always a good idea. :+1: 

I found one of the past threads I was thinking of where I suggested adding `.bio_start` and `.bio_end`

http://mailman.open-bio.org/pipermail/biopython/2014-May/015300.html
...
http://mailman.open-bio.org/pipermail/biopython/2014-May/015308.html

This closed with this example:

> The proposed bio_start and bio_end would also be different for
> simple reverse strand features, e.g.
> 
> complement(1001...2000)
> 
> start = 1000 (Python counting)
> end = 2000
> strand = -1
> len = 1000
> 
> This would get:
> 
> bio_start = 2000
> bio_end = 1000
> 
> Notice start < end (which we might have better called left/right),
> while bio_start > bio_end.

I guess that makes sense. How would you handle mixed-strand CompundLocations, though?

CompoundLocation's `.bio_start` would be `self.parts[0].bio_start`

CompoundLocation's `.bio_end` would be `self.parts[-1].bio_end`

(If the parts had a mixed strand this doesn't matter, only the first and last parts are relevant)

FeatureLocation's `.bio_start` would be `self.end if self.strand == -1 else self.start`

FeatureLocation's `.bio_end` would be `self.start if self.strand == -1 else self.end`

(This means any strandless or dual-strand region would be treated as positive strand)

I guess that would work. :+1: 

Filed #897 for potentially adding `.bio_start` and `.bio_end` - your thoughts on their usefulness (use cases) and exact end point behaviour would be welcome.

I think with Biopython 1.71 giving an error on ``FeatureLocation(start, end)`` with ``start > end`` (see #1497, #1499 and #1608) a large part of the potential for confusion should now be avoided:

```python
>>> from Bio.SeqFeature import FeatureLocation
>>> FeatureLocation(500, 50)
Traceback (most recent call last):
  ...
ValueError: End location (50) must be greater than or equal to start location (500)
```

I think we should update this wording as I suggested earlier https://github.com/biopython/biopython/issues/895#issuecomment-234888919 to change "range" to "span":

```
        More generally, you can use the compound location's start and end which
        give the full range covered, 0 <= start <= end <= full sequence length.
```

Giving:

```
        More generally, you can use the compound location's start and end which
        give the full span covered, 0 <= start <= end <= full sequence length.
```

With that change can we close this issue, or retitle it to focus on the feature location overlap detection?
Wow, this one is ancient. I think we reworked our helper logic to cope with compound locations in the new version of the software. But yes, I guess the new wording would be less confusing.

Resolving patch:
diff --git a/Bio/SeqFeature.py b/Bio/SeqFeature.py
--- a/Bio/SeqFeature.py
+++ b/Bio/SeqFeature.py
@@ -1113,7 +1113,7 @@ def __init__(self, parts, operator="join"):
         12
 
         More generally, you can use the compound location's start and end which
-        give the full range covered, 0 <= start <= end <= full sequence length.
+        give the full span covered, 0 <= start <= end <= full sequence length.
 
         >>> f.start == min(f)
         True


Test patch: