Repository: biopython/biopython

Issue:
Parsing Exonerate text output with SearchIO can truncate codons at exon-intron boundaries
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
Python 3.8.3 (default, May 19 2020, 13:54:14)
CPython
macOS-10.15.7-x86_64-i386-64bit
BioPython 1.78
```

### Expected behaviour

When parsing the output of an Exonerate search performed using the model protein2genome, SearchIO.parse(<exonerate_output_file>, 'exonerate-text') should capture complete nucleotide codon triplets in the hsp fragment `'hit_annotation'` values (where a given fragment contains complete, non-split, codon triplets). 

### Actual behaviour

The hsp `'hit_annotation'` values can contain incorrectly truncated nucleotide codons immediately before and after an intron. For example, for the hit alignment:

```
   49 : MetValGlnThrProLeuHisValSerAlaGlyAsnAsnArgAlaAspIleValLysPheLe :   69
        |||..!||||||||||||||||||||||||||| !!|||||||||!!:||||||||||||||
        MetThrGlnThrProLeuHisValSerAlaGlyTyrAsnArgAlaGluIleValLysPheLe
 2392 : ATGACACAAACCCCCCTCCACGTGTCTGCTGGTTACAACAGGGCGGAGATAGTTAAATTTCT : 2332

   70 : uLeuGluPheProGlyProGluLysValGluLeuGluAlaLysAsnMet  >>>> Target :   86
        |||||||||||||||||||||||||||||||||||||||||||||||||            1
        uLeuGluPheProGlyProGluLysValGluLeuGluAlaLysAsnMet++           
 2331 : TCTTGAGTTTCCAGGGCCAGAGAAGGTTGAGTTGGAAGCCAAGAACATGgt........... : 2279
```


... the corresponding hsp fragment `'hit_annotation'` value is:

```
'hit_annotation': ['ATG', 'ACA', 'CAA', 'ACC', 'CCC', 'CTC', 'CAC', 'GTG', 'TCT', 'GCT', 'GGT', 'TAC', 'AAC', 'AGG', 'GCG', 'GAG', 'ATA', 'GTT', 'AAA', 'TTT', 'CTT', 'CTT', 'GAG', 'TTT', 'CCA', 'GGG', 'CCA', 'GAG', 'AAG', 'GTT', 'GAG', 'TTG', 'GAA', 'GCC', 'AAG', 'AAC', 'AT']
```
That is, the final codon triplet is missing the final 'G'.


### Steps to reproduce

Using the attached Exonerate output file  [exonerate_results.fasta.zip](https://github.com/biopython/biopython/files/7361358/exonerate_results.fasta.zip) the SearchIO command is:

`exonerate_hits_from_alignment = list(SearchIO.parse('exonerate_results.fasta', 'exonerate-text'))`

The first hsp fragment contains complete codon triplets. However:

```
for qresult in exonerate_hits_from_alignment:
    for hsp in qresult.hsps:
        for fragment in hsp:
            print(fragment.aln_annotation)
            break
```

Output:

```
{'similarity': ['|||', '..!', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', ' !!', '|||', '|||', '|||', '!!:', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '|||', '||'], 'hit_annotation': ['ATG', 'ACA', 'CAA', 'ACC', 'CCC', 'CTC', 'CAC', 'GTG', 'TCT', 'GCT', 'GGT', 'TAC', 'AAC', 'AGG', 'GCG', 'GAG', 'ATA', 'GTT', 'AAA', 'TTT', 'CTT', 'CTT', 'GAG', 'TTT', 'CCA', 'GGG', 'CCA', 'GAG', 'AAG', 'GTT', 'GAG', 'TTG', 'GAA', 'GCC', 'AAG', 'AAC', 'AT']}
```

This issue appears to be caused by the code [here](https://github.com/biopython/biopython/blob/69a220ba4db4ebe998d10f15583ab0bacee1aa69/Bio/SearchIO/ExonerateIO/exonerate_text.py#L29). In cases where the three-letter code of the final amino-acid in the query fragment end with e.g. a `'t'`, as above for `Met`, the regex picks up the `'t  '` (i.e. a `'t'` followed by two spaces) and interprets it as part of the intron. This results in `_get_block_coords()` recovering incorrect blocks [here](https://github.com/biopython/biopython/blob/69a220ba4db4ebe998d10f15583ab0bacee1aa69/Bio/SearchIO/ExonerateIO/exonerate_text.py#L72), meaning that the coordinates returned are incorrect, and so on.

Naively, could this be solved by simply restricting the repeat range for pattern `[atgc ]` in the regex to `{2}`, rather than `{2,}`, i.e. change:

```
_RE_EXON = re.compile(
    r"[atgc ]{2,}?(?:(?:[<>]+ \w+ Intron \d+ [<>]+)|(?:\.+))[atgc ]{2,}?"
)
```

to:

```
_RE_EXON = re.compile(
    r"[atgc ]{2}?(?:(?:[<>]+ \w+ Intron \d+ [<>]+)|(?:\.+))[atgc ]{2}?"
)
````
?




Cheers,

Chris



Discussion:

Resolving patch:
diff --git a/Bio/SearchIO/ExonerateIO/exonerate_text.py b/Bio/SearchIO/ExonerateIO/exonerate_text.py
--- a/Bio/SearchIO/ExonerateIO/exonerate_text.py
+++ b/Bio/SearchIO/ExonerateIO/exonerate_text.py
@@ -27,7 +27,7 @@
 # for splitting the line based on intron annotations
 # e.g. '  >>>> Target Intron 1 >>>>  ' or 'gt.........................ag'
 _RE_EXON = re.compile(
-    r"[atgc ]{2,}?(?:(?:[<>]+ \w+ Intron \d+ [<>]+)|(?:\.+))[atgc ]{2,}?"
+    r"[atgc ]{2}?(?:(?:[<>]+ \w+ Intron \d+ [<>]+)|(?:\.+))[atgc ]{2}?"
 )
 # captures the intron length
 # from e.g. '61 bp // 154295 bp' (joint intron lengths) or '177446 bp'


Test patch:
diff --git a/Tests/Exonerate/README.txt b/Tests/Exonerate/README.txt
--- a/Tests/Exonerate/README.txt
+++ b/Tests/Exonerate/README.txt
@@ -8,6 +8,8 @@ Exonerate release version, from the most recent first.
 Exonerate 2.4
 -------------
 exn_24_m_protein2genome_revcomp_fshifts.exn
+exn_24_m_protein2genome_met_intron.exn
+
 
 Exonerate 2.2
 -------------
diff --git a/Tests/Exonerate/exn_24_m_protein2genome_met_intron.exn b/Tests/Exonerate/exn_24_m_protein2genome_met_intron.exn
new file mode 100644
--- /dev/null
+++ b/Tests/Exonerate/exn_24_m_protein2genome_met_intron.exn
@@ -0,0 +1,133 @@
+Command line: [exonerate -m protein2genome gene001_baits.fasta gene001_contigs.fasta --showcigar no --showvulgar no --bestn 1 --refine full]
+Hostname: [RBGs-MacBook-Air.local]
+
+C4 Alignment:
+------------
+         Query: Morus-gene001
+        Target: NODE_1_length_2817_cov_100.387732 SPAdes contig NODE_1:[revcomp]
+         Model: protein2genome:local
+     Raw score: 1958
+   Query range: 48 -> 482
+  Target range: 2392 -> 388
+
+   49 : MetValGlnThrProLeuHisValSerAlaGlyAsnAsnArgAlaAspIleValLysPheLe :   69
+        |||..!||||||||||||||||||||||||||| !!|||||||||!!:||||||||||||||
+        MetThrGlnThrProLeuHisValSerAlaGlyTyrAsnArgAlaGluIleValLysPheLe
+ 2392 : ATGACACAAACCCCCCTCCACGTGTCTGCTGGTTACAACAGGGCGGAGATAGTTAAATTTCT : 2332
+
+   70 : uLeuGluPheProGlyProGluLysValGluLeuGluAlaLysAsnMet  >>>> Target :   86
+        |||||||||||||||||||||||||||||||||||||||||||||||||            1
+        uLeuGluPheProGlyProGluLysValGluLeuGluAlaLysAsnMet++           
+ 2331 : TCTTGAGTTTCCAGGGCCAGAGAAGGTTGAGTTGGAAGCCAAGAACATGgt........... : 2279
+
+   87 :  Intron 1 >>>>  TyrGlyGluThrProLeuHisMetAlaAlaLysAsnGlyCysAsnA :  101
+        52 bp           ||||||||||||||||||||||||||||||||||||||||||||||
+                      ++TyrGlyGluThrProLeuHisMetAlaAlaLysAsnGlyCysAsnA
+ 2278 : ..............agTATGGGGAAACTCCTTTGCACATGGCAGCAAAGAATGGATGCAATG : 2084
+
+  102 : spAlaAlaArgLeuLeuLeuAlaHisGlyAlaPheIleGluAlaLysAlaAsn  >>>> Ta :  119
+        |||||||||||:!!|||||||||||||||||||||||||||||||||||||||         
+        spAlaAlaArgValLeuLeuAlaHisGlyAlaPheIleGluAlaLysAlaAsn++       
+ 2083 : ATGCTGCTCGGGTGCTTCTTGCTCATGGTGCTTTTATTGAAGCCAAAGCAAATgt....... : 2028
+
+  120 : rget Intron 2 >>>>  AsnGlyMetThrProLeuHisLeuAlaValTrpTyrSerLeu :  132
+           109 bp           ||||||||||||||||||||||||||||||||||||||||||
+                          ++AsnGlyMetThrProLeuHisLeuAlaValTrpTyrSerLeu
+ 2027 : ..................agAATGGGATGACGCCCCTACACCTTGCTGTCTGGTATTCACTC : 1882
+
+  133 : ArgSerGluAspPheSerThrValLysThrLeuLeuGluTyrAsnAlaAspCysCysAlaGl :  153
+         !!|||! !|||! !||||||||||||||||||||||||||||||||||||||| !!|||||
+        CysSerAlaAspCysSerThrValLysThrLeuLeuGluTyrAsnAlaAspCysSerAlaGl
+ 1881 : TGTTCGGCAGATTGCTCAACTGTAAAGACATTGCTCGAGTACAATGCTGATTGCAGTGCAGA : 1819
+
+  154 : uAspAsn  >>>> Target Intron 3 >>>>  GluGlyLysThrProIleAsnHisLe :  164
+        |||||||            86 bp            ||| !!! !||||||:!!||||||||
+        uAspAsn++                         ++GluArgMetThrProLeuAsnHisLe
+ 1818 : GGACAATgt.........................agGAGAGAATGACTCCTCTAAACCATCT : 1700
+
+  165 : uSerLysGlyProGlySerGluLysLeuArgGluLeuLeuValCysHisLeuGluGluGlnA :  185
+        |||||||||||||||||||||||||||||||||||||||| !!!  ||||||||||||||||
+        uSerLysGlyProGlySerGluLysLeuArgGluLeuLeuPheLeuHisLeuGluGluGlnA
+ 1699 : GTCAAAAGGTCCTGGTAGTGAAAAGTTGAGGGAACTATTATTCTTGCACCTTGAAGAGCAGC : 1637
+
+  186 : rgLysArgArgAlaIleGluAlaCysSerGluThrGlnAlaLysMetAspGluLeuGluAsn :  205
+        |||||||||||||||||||||||||||||||||||:!!||||||||||||||||||||||||
+        rgLysArgArgAlaIleGluAlaCysSerGluThrLysAlaLysMetAspGluLeuGluAsn
+ 1636 : GAAAGAGAAGAGCTATTGAAGCATGCAGTGAAACTAAAGCTAAGATGGATGAACTTGAAAAT : 1577
+
+  206 : GluLeuSerAsnIleValGlyLeuAsnAspLeuLysIleGlnLeuArgArgTrpAlaArgGl :  226
+        |||||||||||||||||||||||||||!!:||||||||||||||||||!:!|||||||||||
+        GluLeuSerAsnIleValGlyLeuAsnGluLeuLysIleGlnLeuArgLysTrpAlaArgGl
+ 1576 : GAATTATCAAACATAGTGGGCTTGAACGAGCTCAAAATACAATTACGGAAATGGGCAAGGGG : 1514
+
+  227 : yMetLeuLeuAspGluArgArgArgAlaLeuGlyLeuLysValGlyThrArgArgProProH :  247
+        ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
+        yMetLeuLeuAspGluArgArgArgAlaLeuGlyLeuLysValGlyThrArgArgProProH
+ 1513 : AATGCTTTTGGATGAAAGGCGTAGGGCCCTCGGTCTAAAAGTTGGTACCCGAAGACCGCCTC : 1451
+
+  248 : isMetAlaPheLeuGlyAsnProGlyThr{G}  >>>> Target Intron 4 >>>>  { :  257
+        |||||||||||||||||||||||||||||{|}            222 bp           {
+        isMetAlaPheLeuGlyAsnProGlyThr{G}++                         ++{
+ 1450 : ATATGGCCTTCCTTGGCAACCCTGGAACA{G}gt.........................ag{ : 1198
+
+  258 : ly}LysThrMetValAlaArgValLeuGlyLysLeuLeuHisMetValGlyIleLeuProTh :  277
+        ||}|||||||||:!!|||||||||||||||||||||||||||||||||||||||||||||||
+        ly}LysThrMetIleAlaArgValLeuGlyLysLeuLeuHisMetValGlyIleLeuProTh
+ 1197 : GC}AAAACCATGATAGCCCGTGTTCTTGGAAAATTACTCCATATGGTGGGAATTCTACCTAC : 1139
+
+  278 : rAspLysValThrGluValGlnArgThrAspLeuValGlyGluPheValGlyHisThrGlyP :  298
+        ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
+        rAspLysValThrGluValGlnArgThrAspLeuValGlyGluPheValGlyHisThrGlyP
+ 1138 : AGACAAGGTAACAGAAGTACAGCGGACTGATTTGGTTGGGGAATTTGTTGGTCATACTGGAC : 1076
+
+  299 : roLysThrArgArgLys  >>>> Target Intron 5 >>>>  IleLysAspAlaGluG :  309
+        |||||||||||||||||            133 bp           ||||||||||||||||
+        roLysThrArgArgLys++                         ++IleLysAspAlaGluG
+ 1075 : CAAAAACTAGGAGGAAGgt.........................agATTAAAGATGCAGAAG :  910
+
+  310 : lyGlyIleLeuPheValAspGluAlaTyrArgLeuIleProMetGlnLysAlaAspAspLys :  329
+        ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
+        lyGlyIleLeuPheValAspGluAlaTyrArgLeuIleProMetGlnLysAlaAspAspLys
+  909 : GAGGAATTTTATTTGTGGATGAAGCATACCGATTGATACCCATGCAGAAGGCAGATGATAAG :  850
+
+  330 : AspTyrGlyLeuGluAlaLeuGluGluIleMetSerValMetAspSerArgLysIleValVa :  350
+        |||||||||||||||||||||||||||||||||||||||||||||||| !!|||||||||||
+        AspTyrGlyLeuGluAlaLeuGluGluIleMetSerValMetAspSerGlyLysIleValVa
+  849 : GACTACGGTTTGGAAGCATTAGAAGAGATCATGTCTGTTATGGACAGTGGAAAAATAGTGGT :  787
+
+  351 : lIlePheAlaGlyTyrSerGluProMetLysArgValIleAlaSerAsnGluGlyPheCysA :  371
+        ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
+        lIlePheAlaGlyTyrSerGluProMetLysArgValIleAlaSerAsnGluGlyPheCysA
+  786 : CATATTCGCTGGTTATAGCGAGCCAATGAAACGTGTAATTGCTTCCAATGAAGGTTTCTGCC :  724
+
+  372 : rgArgValThrLysPhePheHisPheAsnAspPheAsnSerGluGluLeuAlaThrIleLeu :  391
+        ||||||||||||||||||||||||||:!!||||||||||||||||||||||||!.!||||||
+        rgArgValThrLysPhePheHisPheAspAspPheAsnSerGluGluLeuAlaAsnIleLeu
+  723 : GAAGGGTTACCAAGTTCTTCCACTTTGATGACTTCAATTCGGAAGAATTAGCGAATATTCTC :  664
+
+  392 : HisLeuLysMetAsnAsnGlnAlaGluAspSerLeuLeuTyrGlyPheLysLeuHisAlaSe :  412
+        |||:!!||||||!:!||||||.!!|||!!:||||||||||||||||||||||||||||||||
+        HisIleLysMetSerAsnGlnThrGluGluSerLeuLeuTyrGlyPheLysLeuHisAlaSe
+  663 : CATATTAAGATGAGTAACCAAACAGAGGAAAGCTTATTGTATGGGTTTAAGTTGCATGCTTC :  601
+
+  413 : rCysThrValGluAlaValAlaGlyLeuIleGlnArgGluThrThrGluLysGlnArgMetG :  433
+        |||||||:!!||||||:!!|||||||||||||||||||||||||||||||||||||||! !|
+        rCysThrIleGluAlaIleAlaGlyLeuIleGlnArgGluThrThrGluLysGlnArgArgG
+  600 : CTGCACCATAGAAGCCATTGCAGGACTGATACAGAGAGAAACAACTGAAAAGCAGCGTAGGG :  538
+
+  434 : luMetAsnGlyGlyLeuValAspProIleLeuValAsnAlaArgGluAsnLeuAspLeuArg :  453
+        ||||||||||||||||||||||||||!!:|||||||||||||||||||||||||||||||||
+        luMetAsnGlyGlyLeuValAspProMetLeuValAsnAlaArgGluAsnLeuAspLeuArg
+  537 : AGATGAATGGAGGCTTAGTAGACCCAATGCTAGTAAACGCCCGAGAAAACTTGGACCTCAGG :  478
+
+  454 : LeuAsnPheAspCysIleAspThrGluGluLeuArgThrIleThrLeuGluAspLeuGluAl :  474
+        |||!:!||||||||||||||||||||||||||| !!||||||||||||||||||||||||||
+        LeuSerPheAspCysIleAspThrGluGluLeuCysThrIleThrLeuGluAspLeuGluAl
+  477 : CTCAGCTTTGACTGCATCGACACCGAAGAACTATGTACAATTACCCTAGAGGATTTAGAAGC :  415
+
+  475 : aGlyIleGlnLeuLeuSerGln*** :  482
+        ||||||||||||||||!!!||||||
+        aGlyIleGlnLeuLeuThrGln***
+  414 : AGGAATTCAGCTTTTGACGCAATGA :  389
+
+-- completed exonerate analysis
diff --git a/Tests/test_SearchIO_exonerate.py b/Tests/test_SearchIO_exonerate.py
--- a/Tests/test_SearchIO_exonerate.py
+++ b/Tests/test_SearchIO_exonerate.py
@@ -2323,6 +2323,71 @@ def test_exn_24_m_protein2genome_revcomp_fshifts(self):
         self.assertEqual("CLAACLEVEKVVMMGFLVYSASLDQTFKVWRVKVLPDEE", hsp[-1].query.seq)
         self.assertEqual("CLAAC*QVEKMVMMGFLIYSVSLDQTLKVWRVKILPDQE", hsp[-1].hit.seq)
 
+    def test_exn_24_protein2genome_met_intron(self):
+        """Test parsing exonerate output (exn_24_m_protein2genome_met_intron.exn)."""
+        exn_file = get_file("exn_24_m_protein2genome_met_intron.exn")
+        qresult = read(exn_file, self.fmt)
+
+        # check common attributes
+        for hit in qresult:
+            self.assertEqual(qresult.id, hit.query_id)
+            for hsp in hit:
+                self.assertEqual(hit.id, hsp.hit_id)
+                self.assertEqual(qresult.id, hsp.query_id)
+
+        self.assertEqual("Morus-gene001", qresult.id)
+        self.assertEqual("", qresult.description)
+        self.assertEqual("exonerate", qresult.program)
+        self.assertEqual("protein2genome:local", qresult.model)
+        self.assertEqual(1, len(qresult))
+        # first hit
+        hit = qresult[0]
+        self.assertEqual("NODE_1_length_2817_cov_100.387732", hit.id)
+        self.assertEqual("SPAdes contig NODE_1", hit.description)
+        self.assertEqual(1, len(hit))
+        # first hit, first hsp
+        self.assertEqual(1958, hsp.score)
+        self.assertEqual([0, 0, 0, 0, 0, 0], hsp.query_strand_all)
+        self.assertEqual([-1, -1, -1, -1, -1, -1], hsp.hit_strand_all)
+        self.assertEqual(48, hsp.query_start)
+        self.assertEqual(388, hsp.hit_start)
+        self.assertEqual(482, hsp.query_end)
+        self.assertEqual(2392, hsp.hit_end)
+        self.assertEqual(
+            [(48, 85), (85, 118), (118, 155), (155, 256), (257, 303), (303, 482)],
+            hsp.query_range_all,
+        )
+        self.assertEqual(
+            [
+                (2281, 2392),
+                (2030, 2129),
+                (1810, 1921),
+                (1420, 1724),
+                (1058, 1198),
+                (388, 925),
+            ],
+            hsp.hit_range_all,
+        )
+        self.assertEqual(
+            [(2129, 2281), (1921, 2030), (1724, 1810), (1198, 1420), (925, 1058)],
+            hsp.hit_inter_ranges,
+        )
+        self.assertEqual(
+            [(85, 85), (118, 118), (155, 155), (256, 257), (303, 303)],
+            hsp.query_inter_ranges,
+        )
+        self.assertEqual("MVQTPLHVSAGNNRADIVKF", hsp[0].query.seq[:20])
+        self.assertEqual("VKFLLEFPGPEKVELEAKNM", hsp[0].query.seq[-20:])
+        self.assertEqual("|||", hsp[0].aln_annotation["similarity"][0])
+        self.assertEqual("|||", hsp[0].aln_annotation["similarity"][-1])
+        self.assertEqual("ATG", hsp[0].aln_annotation["hit_annotation"][0])
+        self.assertEqual("ATG", hsp[0].aln_annotation["hit_annotation"][-1])
+        self.assertEqual([0, 0, 0, 0, 0, 0], hsp.query_frame_all)
+        self.assertEqual([-2, -3, -2, -2, -3, -2], hsp.hit_frame_all)
+        self.assertEqual(6, len(hsp.query_all))
+        self.assertEqual(6, len(hsp.hit_all))
+        self.assertEqual(6, len(hsp.aln_annotation_all))
+
     def test_exn_22_q_none(self):
         """Test parsing exonerate output (exn_22_q_none.exn)."""
         exn_file = get_file("exn_22_q_none.exn")