Repository: biopython/biopython

Issue:
Inconsistent enzyme.search behaviour when cutting sequence falls outside sequence
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
import sys; print(sys.version)
import platform; print(platform.python_implementation()); print(platform.platform())
import Bio; print(Bio.__version__)
```

```
3.11.6 (main, Oct  6 2023, 11:55:02) [Clang 13.0.0 (clang-1300.0.29.30)]
CPython
macOS-12.7-x86_64-i386-64bit
1.83
```

### Steps to reproduce

cc: @BjornFJohansson

Let's start with the digestion of the following sequence `TAAAAAAAAAAAAGCCGGCAAAAAAATAAAAA` with the restriction enzyme that makes two cuts `NmeDI`. I am using this example to show what happens at both ends, but the same applies if you use one-cutters only. After digestion we would get 3 fragments:

```
T      AAAAAAAAAAAAGCCGGCAAAAAAA       TAAAAA
ATTTTT      TTTTTTTCGGCCGTTTTTTTATTTT       T
```

If we use the enzyme search, we get the positions (one-based) where these cuts will occur:

```python
from Bio.Seq import Seq
from Bio.Restriction import NmeDI

seq = Seq('TAAAAAAAAAAAAGCCGGCAAAAAAATAAAAA')
print(NmeDI.search(seq))

# Prints `[2, 27]`
```

If we start trimming the molecule from the sides, the sequence where the enzyme would make the cut and leave and overhang will disappear, and the behaviour is different when we trim the left and right side:

```python

for i in range(1,7):
    print(i, NmeDI.search(seq[i:]), NmeDI.search(seq[:-i]))

# Prints:
# 1 [26] [2, 27] < As soon as the initial T is removed, the left site is not returned anymore
# 2 [25] [2, 27]
# 3 [24] [2, 27]
# 4 [23] [2, 27]
# 5 [22] [2, 27]
# 6 [21] [2]     < Until the final T is removed, the right site is still returned
```

I don't know what the right thing to do is. My feeling is that probably the behaviour of the first trimming on either side should work, and the rest not? This would mean accepting a cut like this, where the cutting sites lie exactly at the end, and single-strand DNA is generated (see below). I am not sure if biologically a cut would happen with further trimming.

```
      AAAAAAAAAAAAGCCGGCAAAAAAA       TAAAA
TTTTT      TTTTTTTCGGCCGTTTTTTTATTTT
```



Inconsistent enzyme.search behaviour when cutting sequence falls outside sequence
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
import sys; print(sys.version)
import platform; print(platform.python_implementation()); print(platform.platform())
import Bio; print(Bio.__version__)
```

```
3.11.6 (main, Oct  6 2023, 11:55:02) [Clang 13.0.0 (clang-1300.0.29.30)]
CPython
macOS-12.7-x86_64-i386-64bit
1.83
```

### Steps to reproduce

cc: @BjornFJohansson

Let's start with the digestion of the following sequence `TAAAAAAAAAAAAGCCGGCAAAAAAATAAAAA` with the restriction enzyme that makes two cuts `NmeDI`. I am using this example to show what happens at both ends, but the same applies if you use one-cutters only. After digestion we would get 3 fragments:

```
T      AAAAAAAAAAAAGCCGGCAAAAAAA       TAAAAA
ATTTTT      TTTTTTTCGGCCGTTTTTTTATTTT       T
```

If we use the enzyme search, we get the positions (one-based) where these cuts will occur:

```python
from Bio.Seq import Seq
from Bio.Restriction import NmeDI

seq = Seq('TAAAAAAAAAAAAGCCGGCAAAAAAATAAAAA')
print(NmeDI.search(seq))

# Prints `[2, 27]`
```

If we start trimming the molecule from the sides, the sequence where the enzyme would make the cut and leave and overhang will disappear, and the behaviour is different when we trim the left and right side:

```python

for i in range(1,7):
    print(i, NmeDI.search(seq[i:]), NmeDI.search(seq[:-i]))

# Prints:
# 1 [26] [2, 27] < As soon as the initial T is removed, the left site is not returned anymore
# 2 [25] [2, 27]
# 3 [24] [2, 27]
# 4 [23] [2, 27]
# 5 [22] [2, 27]
# 6 [21] [2]     < Until the final T is removed, the right site is still returned
```

I don't know what the right thing to do is. My feeling is that probably the behaviour of the first trimming on either side should work, and the rest not? This would mean accepting a cut like this, where the cutting sites lie exactly at the end, and single-strand DNA is generated (see below). I am not sure if biologically a cut would happen with further trimming.

```
      AAAAAAAAAAAAGCCGGCAAAAAAA       TAAAA
TTTTT      TTTTTTTCGGCCGTTTTTTTATTTT
```





Discussion:
Do you have a one-cutter example to hand?

```
>>> NmeDI.elucidate()
'cut twice, not yet implemented sorry.'
```

It should be easier for me to understand what is puzzling you with a single cutter ;) 
Without having looked at the code, it seems quite obvious what's happening here. Only the "upper" strand is analyzed and to have a cut before the 2nd position (1-based), you need of course a first position. To have a cut before position 27, you need to have position 27 (but not 28). The "lower" strand is obviously not taken into account.

While this in not a problem for type IIp REs, which cut _inside_ their recognition sequence, it's a problem for type IIs REs, which cut outside. A possible solution would be to check for these enzymes if the reverse complement sequence gives the same result (if the recognition sequence is palindromic), or to check if the sequence is long enough to contain the cuttings for both strands. This could possibly be incorporated into the RE module.

From a biological view, it's most likely unknown, if such enzymes will cut the DNA if one strand cannot be cut at all (because the DNA just ends). (Even some typ IIp REs don't cut (effectively) when their sites are directly at the end of a DNA molecule) So I wouldn't recommend to allow "air cuts", e.g. in your example, I wouldn't return a site if one strand is missing the neighboring base. I think the actual behavior for the "left site" is correct and for the "right site" is incorrect.
As @MarkusPiotrowski points out, this can only happen with enzymes that cut outside their recognition site. It seems that only the positions on the watson/upper strand are returned. It's not so much whether the cut is on the left or on the right, instead what matters is whether the cut occurs  on the top or bottom strand. Let me show you with three examples with BsaI, BsaXI, BspCNI. Recognition sites are shown in blue, and the cuts are marked in black.

<img width="97" alt="Screenshot 2024-01-25 at 11 15 17" src="https://github.com/biopython/biopython/assets/22526102/1fee6894-8a2c-477f-a59b-83d3f8c859eb">

<img width="180" alt="Screenshot 2024-01-25 at 10 35 21" src="https://github.com/biopython/biopython/assets/22526102/5caf28d9-3f01-4e85-86af-09fc940cfa1d">

<img width="126" alt="Screenshot 2024-01-25 at 10 38 06" src="https://github.com/biopython/biopython/assets/22526102/386086d3-c7e1-4bd2-8cf1-659000240ef5">

Note that there is a T among the As where the cut will occur. When that T is removed, the cut is no longer returned, as you can see from the code below:

```python
from Bio.Seq import Seq
from Bio.Restriction import BsaI, BsaXI, BspCNI

print(BsaI.search(Seq('GGTCTCATAAAA')))
print(BsaI.search(Seq('GGTCTCATAAA')))
print(BsaI.search(Seq('GGTCTCATAA')))
print(BsaI.search(Seq('GGTCTCATA')))
print(BsaI.search(Seq('GGTCTCAT')))
print(BsaI.search(Seq('GGTCTCA')))
print()

# Prints
# [8]
# [8]
# [8]
# [8]
# [8]
# []

print(BsaXI.search(Seq('AAATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('AATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('ATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('TAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('AAAAAAAAAACAAAAACTCC')))
print()

# prints
# [5]
# [4]
# [3]
# [2]
# []

print(BspCNI.search(Seq('CTCAGAAAAAAAAAT')))
print(BspCNI.search(Seq('CTCAGAAAAAAAAA')))

# prints
# [15]
# []
``` 

@MarkusPiotrowski proposes that the default behaviour should be like in the BspCNI case (cutsite is returned only if the resulting cut will leave at least one base on each strand). Conventional wisdom for primer design is to add some extra bases after a cutsite, so this is probably correct.

If you agree, I can implement that change and write the tests. If you know where to look immediately, I would appreciate it, but I can find my way.
Hi @MarkusPiotrowski  @peterjc do you agree with the proposed change? Should I go ahead?
Hello, just following up on this in case you missed the last comment.
I'm deferring to @MarkusPiotrowski on this one.
Pinging @MarkusPiotrowski 
@manulera The situation is not clear-cut.
1. Of course, recognition site prediction should work on both strands the same way. So it shouldn't matter which strand is given, because always a dsDNA is assumed as bait. Thus: `BsaI.search(Seq('GGTCTCGT')` should give the same result as `BsaI.search(Seq('GGTCTCGT').reverse_complement())`. This is actually not the case.
2. Now to the question of what limits the detection of a RE site regarding overhangs. I'm pretty convinced that **at least** the length of the longest overhang must be present in the target sequence. Otherwise, the resulting overhangs would not be defined regarding their length, which would make a tool like this unsuitable for cloning predictions. So, `BsaI.search(Seq('GGTCTCNNNN'))` should _never_ give a positive result, since the lower strand would be cut after base 11, but the sequence only has a length of 10.
What is finally debatable is whether it is necessary that (in this example) the sequence must have a length of 12. From a bioinformatician's view, 11 should be OK: The recognition site is present, and the overhangs are well-defined regarding their length. As a wet-bench biologist, I would argue that we don't know if the enzyme really works if only one strand can actually be _cut_. However, also some type IIP enzymes are known to work poorly or not at all, when their recognition sequence is directly at the end of the DNA molecule, but we don't take this into account when doing RE site predictions. Contrary, NEB's online tool NEBcutter and the popular sequence editor SnapGene only recognize BsaI sites if at least one base is cleaved off from the bottom strand. So I tend to be consistent with these.

So please, go on to implement this change. Maybe you come across some references which would strengthen one or the other alternative. In general, it's always difficult for users, if Biopython acts different to some popular tool without a good reason. :-)


Do you have a one-cutter example to hand?

```
>>> NmeDI.elucidate()
'cut twice, not yet implemented sorry.'
```

It should be easier for me to understand what is puzzling you with a single cutter ;) 
Without having looked at the code, it seems quite obvious what's happening here. Only the "upper" strand is analyzed and to have a cut before the 2nd position (1-based), you need of course a first position. To have a cut before position 27, you need to have position 27 (but not 28). The "lower" strand is obviously not taken into account.

While this in not a problem for type IIp REs, which cut _inside_ their recognition sequence, it's a problem for type IIs REs, which cut outside. A possible solution would be to check for these enzymes if the reverse complement sequence gives the same result (if the recognition sequence is palindromic), or to check if the sequence is long enough to contain the cuttings for both strands. This could possibly be incorporated into the RE module.

From a biological view, it's most likely unknown, if such enzymes will cut the DNA if one strand cannot be cut at all (because the DNA just ends). (Even some typ IIp REs don't cut (effectively) when their sites are directly at the end of a DNA molecule) So I wouldn't recommend to allow "air cuts", e.g. in your example, I wouldn't return a site if one strand is missing the neighboring base. I think the actual behavior for the "left site" is correct and for the "right site" is incorrect.
As @MarkusPiotrowski points out, this can only happen with enzymes that cut outside their recognition site. It seems that only the positions on the watson/upper strand are returned. It's not so much whether the cut is on the left or on the right, instead what matters is whether the cut occurs  on the top or bottom strand. Let me show you with three examples with BsaI, BsaXI, BspCNI. Recognition sites are shown in blue, and the cuts are marked in black.

<img width="97" alt="Screenshot 2024-01-25 at 11 15 17" src="https://github.com/biopython/biopython/assets/22526102/1fee6894-8a2c-477f-a59b-83d3f8c859eb">

<img width="180" alt="Screenshot 2024-01-25 at 10 35 21" src="https://github.com/biopython/biopython/assets/22526102/5caf28d9-3f01-4e85-86af-09fc940cfa1d">

<img width="126" alt="Screenshot 2024-01-25 at 10 38 06" src="https://github.com/biopython/biopython/assets/22526102/386086d3-c7e1-4bd2-8cf1-659000240ef5">

Note that there is a T among the As where the cut will occur. When that T is removed, the cut is no longer returned, as you can see from the code below:

```python
from Bio.Seq import Seq
from Bio.Restriction import BsaI, BsaXI, BspCNI

print(BsaI.search(Seq('GGTCTCATAAAA')))
print(BsaI.search(Seq('GGTCTCATAAA')))
print(BsaI.search(Seq('GGTCTCATAA')))
print(BsaI.search(Seq('GGTCTCATA')))
print(BsaI.search(Seq('GGTCTCAT')))
print(BsaI.search(Seq('GGTCTCA')))
print()

# Prints
# [8]
# [8]
# [8]
# [8]
# [8]
# []

print(BsaXI.search(Seq('AAATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('AATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('ATAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('TAAAAAAAAAACAAAAACTCC')))
print(BsaXI.search(Seq('AAAAAAAAAACAAAAACTCC')))
print()

# prints
# [5]
# [4]
# [3]
# [2]
# []

print(BspCNI.search(Seq('CTCAGAAAAAAAAAT')))
print(BspCNI.search(Seq('CTCAGAAAAAAAAA')))

# prints
# [15]
# []
``` 

@MarkusPiotrowski proposes that the default behaviour should be like in the BspCNI case (cutsite is returned only if the resulting cut will leave at least one base on each strand). Conventional wisdom for primer design is to add some extra bases after a cutsite, so this is probably correct.

If you agree, I can implement that change and write the tests. If you know where to look immediately, I would appreciate it, but I can find my way.
Hi @MarkusPiotrowski  @peterjc do you agree with the proposed change? Should I go ahead?
Hello, just following up on this in case you missed the last comment.
I'm deferring to @MarkusPiotrowski on this one.
Pinging @MarkusPiotrowski 
@manulera The situation is not clear-cut.
1. Of course, recognition site prediction should work on both strands the same way. So it shouldn't matter which strand is given, because always a dsDNA is assumed as bait. Thus: `BsaI.search(Seq('GGTCTCGT')` should give the same result as `BsaI.search(Seq('GGTCTCGT').reverse_complement())`. This is actually not the case.
2. Now to the question of what limits the detection of a RE site regarding overhangs. I'm pretty convinced that **at least** the length of the longest overhang must be present in the target sequence. Otherwise, the resulting overhangs would not be defined regarding their length, which would make a tool like this unsuitable for cloning predictions. So, `BsaI.search(Seq('GGTCTCNNNN'))` should _never_ give a positive result, since the lower strand would be cut after base 11, but the sequence only has a length of 10.
What is finally debatable is whether it is necessary that (in this example) the sequence must have a length of 12. From a bioinformatician's view, 11 should be OK: The recognition site is present, and the overhangs are well-defined regarding their length. As a wet-bench biologist, I would argue that we don't know if the enzyme really works if only one strand can actually be _cut_. However, also some type IIP enzymes are known to work poorly or not at all, when their recognition sequence is directly at the end of the DNA molecule, but we don't take this into account when doing RE site predictions. Contrary, NEB's online tool NEBcutter and the popular sequence editor SnapGene only recognize BsaI sites if at least one base is cleaved off from the bottom strand. So I tend to be consistent with these.

So please, go on to implement this change. Maybe you come across some references which would strengthen one or the other alternative. In general, it's always difficult for users, if Biopython acts different to some popular tool without a good reason. :-)



Resolving patch:
diff --git a/Bio/Restriction/Restriction.py b/Bio/Restriction/Restriction.py
--- a/Bio/Restriction/Restriction.py
+++ b/Bio/Restriction/Restriction.py
@@ -536,6 +536,35 @@ def search(cls, dna, linear=True):
             cls.dna = FormattedSeq(dna, linear)
             return cls._search()
 
+    @classmethod
+    def _drop(cls):
+        """Remove cuts that are outsite of the sequence (PRIVATE).
+
+        For internal use only.
+
+        Drop the site that are situated outside the sequence in linear
+        sequence. Modify the index for site in circular sequences.
+        """
+        length = len(cls.dna)
+        if cls.dna.is_linear():
+
+            def filtering_function(cut_on_watson):
+                cut_on_crick = cut_on_watson - cls.ovhg
+                return (1 < cut_on_watson <= length) and (1 < cut_on_crick <= length)
+
+            cls.results = [cut for cut in cls.results if filtering_function(cut)]
+        else:
+            for index, location in enumerate(cls.results):
+                if location < 1:
+                    cls.results[index] += length
+                else:
+                    break
+            for index, location in enumerate(cls.results[::-1]):
+                if location > length:
+                    cls.results[-(index + 1)] -= length
+                else:
+                    break
+
     @classmethod
     def all_suppliers(cls):
         """Print all the suppliers of restriction enzyme."""
@@ -1499,41 +1528,6 @@ class Defined(AbstractCut):
     Internal use only. Not meant to be instantiated.
     """
 
-    @classmethod
-    def _drop(cls):
-        """Remove cuts that are outsite of the sequence (PRIVATE).
-
-        For internal use only.
-
-        Drop the site that are situated outside the sequence in linear
-        sequence. Modify the index for site in circular sequences.
-        """
-        #
-        #   remove or modify the results that are outside the sequence.
-        #   This is necessary since after finding the site we add the distance
-        #   from the site to the cut with the _modify and _rev_modify methods.
-        #   For linear we will remove these sites altogether.
-        #   For circular sequence, we modify the result rather than _drop it
-        #   since the site is in the sequence.
-        #
-        length = len(cls.dna)
-        drop = itertools.dropwhile
-        take = itertools.takewhile
-        if cls.dna.is_linear():
-            cls.results = list(drop(lambda x: x <= 1, cls.results))
-            cls.results = list(take(lambda x: x <= length, cls.results))
-        else:
-            for index, location in enumerate(cls.results):
-                if location < 1:
-                    cls.results[index] += length
-                else:
-                    break
-            for index, location in enumerate(cls.results[::-1]):
-                if location > length:
-                    cls.results[-(index + 1)] -= length
-                else:
-                    break
-
     @classmethod
     def is_defined(cls):
         """Return if recognition sequence and cut are defined.
@@ -1655,33 +1649,6 @@ class Ambiguous(AbstractCut):
 
     """
 
-    @classmethod
-    def _drop(cls):
-        """Remove cuts that are outsite of the sequence (PRIVATE).
-
-        For internal use only.
-
-        Drop the site that are situated outside the sequence in linear
-        sequence. Modify the index for site in circular sequences.
-        """
-        length = len(cls.dna)
-        drop = itertools.dropwhile
-        take = itertools.takewhile
-        if cls.dna.is_linear():
-            cls.results = list(drop(lambda x: x <= 1, cls.results))
-            cls.results = list(take(lambda x: x <= length, cls.results))
-        else:
-            for index, location in enumerate(cls.results):
-                if location < 1:
-                    cls.results[index] += length
-                else:
-                    break
-            for index, location in enumerate(cls.results[::-1]):
-                if location > length:
-                    cls.results[-(index + 1)] -= length
-                else:
-                    break
-
     @classmethod
     def is_defined(cls):
         """Return if recognition sequence and cut are defined.
@@ -1846,17 +1813,7 @@ def _drop(cls):
         if cls.dna.is_linear():
             return
         else:
-            length = len(cls.dna)
-            for index, location in enumerate(cls.results):
-                if location < 1:
-                    cls.results[index] += length
-                else:
-                    break
-            for index, location in enumerate(cls.results[:-1]):
-                if location > length:
-                    cls.results[-(index + 1)] -= length
-                else:
-                    break
+            super()._drop()
 
     @classmethod
     def is_defined(cls):
diff --git a/Bio/Restriction/__init__.py b/Bio/Restriction/__init__.py
--- a/Bio/Restriction/__init__.py
+++ b/Bio/Restriction/__init__.py
@@ -19,7 +19,7 @@
     >>> a = Analysis(AllEnzymes, seq)
     >>> a.print_that()              # no argument -> print all the results
     AbaSI      :  10, 12, 13, 16, 17, 18, 19, 20, 22, 23, 24, 25, 25, 26, 27...
-    BmeDI      :  2, 7, 8, 8, 9, 9, 13, 14, 15, 16, 17, 18, 19, 19, 21, 21...
+    BmeDI      :  7, 8, 8, 9, 9, 13, 14, 15, 16, 17, 18, 19, 19, 21, 21...
     YkrI       :  10, 12, 13, 16, 16, 17, 19, 20, 21, 22, 23, 24, 25, 25, 26...
 
     BmeDI      :  1, 2, 7, 8, 8, 9, 9, 13, 14, 15, 16, 17, 18, 19...


Test patch:
diff --git a/Tests/test_Restriction.py b/Tests/test_Restriction.py
--- a/Tests/test_Restriction.py
+++ b/Tests/test_Restriction.py
@@ -30,6 +30,9 @@
 from Bio.Restriction import SmaI
 from Bio.Restriction import SnaI
 from Bio.Restriction import SphI
+from Bio.Restriction import BsaI
+from Bio.Restriction import BsaXI
+from Bio.Restriction import BspCNI
 from Bio.Seq import MutableSeq
 from Bio.Seq import Seq
 
@@ -191,8 +194,12 @@ def test_shortcuts(self):
     def test_cutting_border_positions(self):
         """Check if cutting after first and penultimate position works."""
         # Use EarI, cuts as follows: CTCTTCN^NNN_N
+        # Only when the cut produces double stranded DNA on both outputs
+        # it is returned.
         seq = Seq("CTCTTCA")
         self.assertEqual(EarI.search(seq), [])
+        seq += "AAA"
+        self.assertEqual(EarI.search(seq), [])
         seq += "A"
         self.assertEqual(EarI.search(seq), [8])
         # Recognition site on reverse-complement strand
@@ -201,6 +208,18 @@ def test_cutting_border_positions(self):
         seq = "A" + seq
         self.assertEqual(EarI.search(seq), [2])
 
+        # Examples from https://github.com/biopython/biopython/issues/4604
+        self.assertEqual(BsaI.search(Seq("GGTCTCATAAAA")), [8])
+        self.assertEqual(BsaI.search(Seq("GGTCTCATAAA")), [])
+        self.assertEqual(BsaI.search(Seq("GGTCTCGT")), [])
+        self.assertEqual(BsaI.search(Seq("GGTCTCGT").reverse_complement()), [])
+
+        self.assertEqual(BsaXI.search(Seq("AAATAAAAAAAAAACAAAAACTCC")), [5])
+        self.assertEqual(BsaXI.search(Seq("AATAAAAAAAAAACAAAAACTCC")), [])
+
+        self.assertEqual(BspCNI.search(Seq("CTCAGAAAAAAAAAT")), [15])
+        self.assertEqual(BspCNI.search(Seq("CTCAGAAAAAAAAA")), [])
+
     def test_recognition_site_on_both_strands(self):
         """Check if recognition sites on both strands are properly handled."""
         seq = Seq("CTCTTCGAAGAG")