Repository: biopython/biopython

Issue:
pairwise2 deprecation: how to align lists of sequences with the new PairwiseAligner?
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
import sys; print(sys.version)
import platform; print(platform.python_implementation()); print(platform.platform())
import Bio; print(Bio.__version__)
```

3.10.8 (main, Oct 13 2022, 10:17:43) [Clang 14.0.0 (clang-1400.0.29.102)]
CPython
macOS-12.6.1-x86_64-i386-64bit
1.80

### Expected behaviour

As the new PairwiseAligner is supposed to cover the functionality of pairwise2, I would expect that it can also align lists of sequences. It would also be nice to customize the gap character like in pairwise2.

### Actual behaviour

When passing two lists to the PairwiseAligner, I get

```python
ValueError: alphabet is None; cannot interpret sequence
```

### Steps to reproduce

```python
from Bio import Align
list1 = ['i', 'love', 'bananas', 'and', 'cheese']
list2 = ['bandanas', 'blind', 'bees']
aligner = Align.PairwiseAligner()
alignments = aligner.align(list1, list2)
```

Would appreciate any direction on this matter. Thanks a lot :)


Discussion:
See section 6.6.11 in the documentation.

You need to tell the aligner the alphabet, i.e. which items are valid, as in this example:
```python
>>> from Bio import Align
>>> list1 = ['i', 'love', 'bananas', 'and', 'cheese']
>>> list2 = ['bandanas', 'blind', 'bees']
>>> aligner = Align.PairwiseAligner()
>>> 
>>> alphabet = sorted(set(list1+list2))
>>> alphabet
['and', 'bananas', 'bandanas', 'bees', 'blind', 'cheese', 'i', 'love']
>>> aligner.alphabet = alphabet
>>> alignments = aligner.align(list1, list2)
>>> print(alignments[0])
i love bananas and cheese -------- ----- ----
- ---- ------- --- ------ -------- ----- ----
- ---- ------- --- ------ bandanas blind bees

>>> list2 = ['bananas', 'blind', 'bees']
>>> alignments = aligner.align(list1, list2)
>>> print(alignments[0])
i love bananas and cheese ----- ----
- ---- ||||||| --- ------ ----- ----
- ---- bananas --- ------ blind bees

```
You can replace the gap character in the string before printing it:
```python
>>> print(str(alignments[0]).replace("-", "♣"))
i love bananas and cheese ♣♣♣♣♣ ♣♣♣♣
♣ ♣♣♣♣ ||||||| ♣♣♣ ♣♣♣♣♣♣ ♣♣♣♣♣ ♣♣♣♣
♣ ♣♣♣♣ bananas ♣♣♣ ♣♣♣♣♣♣ blind bees

```

@mdehoon thanks so much for your quick response! We're starting to head in the right direction :)

Is there any reason you sorted the combination of the lists? Or was it just for readability? 

If I tune the penalty parameters, it's possible to get similar words to align, such as banana and bandana. In the old pairwise aligner, I could get two equal-length lists including the gaps from the output of the align function. E.g.:

```python
In [49]: old_alignments = pairwise2.align.globalms(list1, list2, ms, mms, ogs, egs, gap_char=['♣'])
In [50]: old_alignments[0][:2]
Out[50]: 
(['i', 'love', 'bananas', 'and', 'cheese'],
 ['♣', '♣', 'bandanas', 'blind', 'bees'])
```

How can I do that with the new pairwise aligner? I tried poking around the new alignments object and I didn't see anything properties or methods that could help me out.

Thanks for the help so far!


> Is there any reason you sorted the combination of the lists? Or was it just for readability?

Just for readability.

> How can I do that with the new pairwise aligner? I tried poking around the new alignments object and I didn't see anything properties or methods that could help me out.

What does `pairwise2.align.globalms(list1, list2, ms, mms, ogs, egs, gap_char=['♣'])`  do?
Then do the same with the aligner object, as described in the documentation.
@mdehoon with the pairwise2 function I was just trying to illustrate that I can tune the penalty parameters (match score, open gap score, etc.) to align alike words. The pairwise2 function outputs a nice alignment representation which is very useful for my work.

If I do the same thing with the new aligner function like you proposed, I get an Alignment object rather than the tuple representation.
```python
In [100]: alignments = aligner.align(list1, list2)
In [101]: alignments[0][:2]
Out[101]: <Alignment object (2 rows x 5 columns) at 0x11a8c5210>
```

After looking through the documentation, I don't see how I can get the old pairwise2 tuple representation with equal-length lists out of the Alignment object. the closest I find is 
```python
In [108]: alignments[0].sequences
Out[108]: [['i', 'love', 'bananas', 'and', 'cheese'], ['bandanas', 'blind', 'bees']]
```
but then I lose the information about the gaps that were opened.
> with the pairwise2 function I was just trying to illustrate that I can tune the penalty parameters (match score, open gap score, etc.) to align alike words.

I don't think it is aligning alike words; the pairwise2 aligner has no way to know what "alike" means:

```python
>>> old_alignments = pairwise2.align.globalms(['i', 'love', 'bananas', 'and', 'cheese'], ['bandanas', 'blind', 'bees'], -2, -1, -2, -1, gap_char=['♣'])
>>> old_alignments[0][:2]
(['i', 'love', 'bananas', 'and', 'cheese'], ['♣', '♣', 'bandanas', 'blind', 'bees'])
>>> old_alignments = pairwise2.align.globalms(['i', 'love', 'bananas', 'and', 'cheese'], ['anything', 'blind', 'bees'], -2, -1, -2, -1, gap_char=['♣'])
>>> old_alignments[0][:2]
>>> (['i', 'love', 'bananas', 'and', 'cheese'], ['♣', '♣', 'anything', 'blind', 'bees'])
```

In any case, to get the same alignment, set the gap scoring paramaters on the `aligner` object:
```
>>> aligner.open_gap_score = -2
>>> aligner.extend_gap_score = -1
>>> alignments = aligner.align(['i', 'love', 'bananas', 'and', 'cheese'], ['bandanas', 'blind', 'bees'])
>>> print(alignments[0])
i love bananas  and   cheese
- ---- .......  ...   ....  
- ---- bandanas blind bees  
```
For your last question:
> The pairwise2 function outputs a nice alignment representation which is very useful for my work.

What do you do with the alignment representation you got from `pairwise2`?
@mdehoon I try to find common typos for given words. Instead of trying to interpret which words the user meant to type by hand, the alignment function is super useful for doing this automatically. After using it for a few years, I do get the feeling it has an idea of which words are somewhat alike, even if it's not exactly what it was designed to do. 

I haven't gotten a chance to explore it yet, but I would also like to play around with the new substitution matrices to better represent a keyboard so that e.g. substitutions between 't' and 'y' are penalized less than 't' and 'm'.
> After using it for a few years, I do get the feeling it has an idea of which words are somewhat alike, even if it's not exactly what it was designed to do.

The aligner just checks whether two words are the same or different, but it does not align the two words and does not evaluate their similarity. If one word in a sentence has a spelling error, it may align to the correct word just because the other words in the sentence are aligned to each other.

> I haven't gotten a chance to explore it yet, but I would also like to play around with the new substitution matrices to better represent a keyboard so that e.g. substitutions between 't' and 'y' are penalized less than 't' and 'm'.

Here also, the aligner does not align the words, so the substitution matrices won't make any difference. As an alternative, you could design some function that evaluates how close two words are, and use that function instead of the substitution matrix.

We could modify `Bio.Align` such that `alignment[i]` returns something useful when the aligned sequences are lists. Currently it does
```python
>>> alignment[0]
"['i', 'love', 'bananas', 'and', 'cheese']---"
>>> alignment[1]
"-----['bandanas', 'blind', 'bees']"
```
which is just useless. This could be
```python
>>> alignment[0]
['i', 'love', 'bananas', 'and', 'cheese']
>>> alignment[1]
[None, None, 'bandanas', 'blind', 'bees']
```

@mdehoon 

> We could modify `Bio.Align` such that `alignment[i]` returns something useful when the aligned sequences are lists.

Yes! That would be extremely useful :) 

In the old pairwise2 function, I could also pass a "gap_char" parameter, which would allow me to customize what the Nones would be (I think it defaulted to "-"), which would also be helpful. One not immediately clear thing about that parameter was that it has to be the same type as the sequences to be aligned. If you do consider the gap_char parameter, it would be nice to have appropriate documentation/type-hinting for it.
In the `Alignment` class, we are separating the alignment information (i.e. where are the gaps) from the alignment representation (e.g., gap characters). In the previous `MultipleSeqAlignment` class those two things were mixed, which limited its usability. I am very hesitant to add representation stuff to the output of `alignment[i]`. For example, what to do if one of the objects you are aligning happens to be `"-"`? Or if you are aligning objects of different classes? In any case, you can always replace `None` with the gap character of your choice afterwards.
@mdehoon ah, that's a very good point. Adding in the gap characters after the fact is definitely more robust :)

Resolving patch:
diff --git a/Bio/Align/__init__.py b/Bio/Align/__init__.py
--- a/Bio/Align/__init__.py
+++ b/Bio/Align/__init__.py
@@ -1296,7 +1296,9 @@ def _get_row(self, index):
 
         self[row]
 
-        where row is an integer. Return value is a string.
+        where row is an integer.
+        Return value is a string if the aligned sequences are string, Seq,
+        or SeqRecord objects, otherwise the return value is a list.
         """
         steps = abs(numpy.diff(self.coordinates, 1))
         gaps = steps.max(0)
@@ -1313,14 +1315,24 @@ def _get_row(self, index):
             sequence = reverse_complement(sequence, inplace=False)
             i = len(sequence) - i
         steps = steps[index]
-        line = ""
-        for step, gap in zip(steps, gaps):
-            if step:
-                j = i + step
-                line += str(sequence[i:j])
-                i = j
-            else:
-                line += "-" * gap
+        if isinstance(sequence, (str, Seq)):
+            line = ""
+            for step, gap in zip(steps, gaps):
+                if step:
+                    j = i + step
+                    line += str(sequence[i:j])
+                    i = j
+                else:
+                    line += "-" * gap
+        else:
+            line = []
+            for step, gap in zip(steps, gaps):
+                if step:
+                    j = i + step
+                    line.extend(sequence[i:j])
+                    i = j
+                else:
+                    line.extend([None] * gap)
         return line
 
     def _get_rows(self, key):
@@ -1375,7 +1387,8 @@ def _get_row_cols_slice(
         self[row, cols]
 
         where row is an integer and cols is a slice object with step 1.
-        Return value is a string.
+        Return value is a string if the aligned sequences are string, Seq,
+        or SeqRecord objects, otherwise the return value is a list.
         """
         sequence_indices = coordinate + steps.cumsum()
         indices = gaps.cumsum()
@@ -1385,41 +1398,78 @@ def _get_row_cols_slice(
             sequence = sequence.seq  # stupid SeqRecord
         except AttributeError:
             pass
-        if i == j:
-            length = stop_index - start_index
-            if steps[i] == 0:
-                line = "-" * length
-            else:
-                offset = start_index - indices[i]
-                start = sequence_indices[i] + offset
-                stop = start + length
-                line = str(sequence[start:stop])
-        else:
-            length = indices[i] - start_index
-            stop = sequence_indices[i]
-            if steps[i] == 0:
-                line = "-" * length
+        if isinstance(sequence, (str, Seq)):
+            if i == j:
+                length = stop_index - start_index
+                if steps[i] == 0:
+                    line = "-" * length
+                else:
+                    offset = start_index - indices[i]
+                    start = sequence_indices[i] + offset
+                    stop = start + length
+                    line = str(sequence[start:stop])
             else:
-                start = stop - length
-                line = str(sequence[start:stop])
-            i += 1
-            while i < j:
-                step = gaps[i]
+                length = indices[i] - start_index
+                stop = sequence_indices[i]
                 if steps[i] == 0:
-                    line += "-" * step
+                    line = "-" * length
                 else:
-                    start = stop
-                    stop = start + step
-                    line += str(sequence[start:stop])
+                    start = stop - length
+                    line = str(sequence[start:stop])
                 i += 1
-            length = stop_index - indices[j - 1]
-            if length > 0:
-                if steps[j] == 0:
-                    line += "-" * length
+                while i < j:
+                    step = gaps[i]
+                    if steps[i] == 0:
+                        line += "-" * step
+                    else:
+                        start = stop
+                        stop = start + step
+                        line += str(sequence[start:stop])
+                    i += 1
+                length = stop_index - indices[j - 1]
+                if length > 0:
+                    if steps[j] == 0:
+                        line += "-" * length
+                    else:
+                        start = stop
+                        stop = start + length
+                        line += str(sequence[start:stop])
+        else:
+            if i == j:
+                length = stop_index - start_index
+                if steps[i] == 0:
+                    line = [None] * length
                 else:
-                    start = stop
+                    offset = start_index - indices[i]
+                    start = sequence_indices[i] + offset
                     stop = start + length
-                    line += str(sequence[start:stop])
+                    line = sequence[start:stop]
+            else:
+                length = indices[i] - start_index
+                stop = sequence_indices[i]
+                if steps[i] == 0:
+                    line = [None] * length
+                else:
+                    start = stop - length
+                    line = sequence[start:stop]
+                i += 1
+                while i < j:
+                    step = gaps[i]
+                    if steps[i] == 0:
+                        line.extend([None] * step)
+                    else:
+                        start = stop
+                        stop = start + step
+                        line.extend(sequence[start:stop])
+                    i += 1
+                length = stop_index - indices[j - 1]
+                if length > 0:
+                    if steps[j] == 0:
+                        line.extend([None] * length)
+                    else:
+                        start = stop
+                        stop = start + length
+                        line.extend(sequence[start:stop])
         return line
 
     def _get_row_cols_iterable(self, i, cols, steps, gaps, sequence):
@@ -1430,28 +1480,47 @@ def _get_row_cols_iterable(self, i, cols, steps, gaps, sequence):
         self[row, cols]
 
         where row is an integer and cols is an iterable of integers.
-        Return value is a string.
+        Return value is a string if the aligned sequences are string, Seq,
+        or SeqRecord objects, otherwise the return value is a list.
         """
         try:
             sequence = sequence.seq  # stupid SeqRecord
         except AttributeError:
             pass
-        line = ""
-        for step, gap in zip(steps, gaps):
-            if step:
-                j = i + step
-                line += str(sequence[i:j])
-                i = j
-            else:
-                line += "-" * gap
-        try:
-            line = "".join(line[col] for col in cols)
-        except IndexError:
-            raise
-        except Exception:
-            raise TypeError(
-                "second index must be an integer, slice, or iterable of integers"
-            ) from None
+        if isinstance(sequence, (str, Seq)):
+            line = ""
+            for step, gap in zip(steps, gaps):
+                if step:
+                    j = i + step
+                    line += str(sequence[i:j])
+                    i = j
+                else:
+                    line += "-" * gap
+            try:
+                line = "".join(line[col] for col in cols)
+            except IndexError:
+                raise
+            except Exception:
+                raise TypeError(
+                    "second index must be an integer, slice, or iterable of integers"
+                ) from None
+        else:
+            line = []
+            for step, gap in zip(steps, gaps):
+                if step:
+                    j = i + step
+                    line.extend(sequence[i:j])
+                    i = j
+                else:
+                    line.extend([None] * gap)
+            try:
+                line = [line[col] for col in cols]
+            except IndexError:
+                raise
+            except Exception:
+                raise TypeError(
+                    "second index must be an integer, slice, or iterable of integers"
+                ) from None
         return line
 
     def _get_rows_col(self, coordinates, col, steps, gaps, sequences):


Test patch:
diff --git a/Tests/test_pairwise_aligner.py b/Tests/test_pairwise_aligner.py
--- a/Tests/test_pairwise_aligner.py
+++ b/Tests/test_pairwise_aligner.py
@@ -3349,24 +3349,68 @@ def test_three_letter_amino_acids_global(self):
         self.assertAlmostEqual(score, 3.0)
         alignments = aligner.align(seq1, seq2)
         self.assertEqual(len(alignments), 2)
+        alignment = alignments[0]
         self.assertEqual(
-            str(alignments[0]),
+            str(alignment),
             """\
 Pro Pro Gly Ala --- --- Thr --- ---
 --- --- ||| ||| --- --- ||| --- ---
 --- --- Gly Ala Ala Cys Thr Asn Asn
 """,
         )
+        self.assertAlmostEqual(alignment.score, 3.0)
         self.assertEqual(
-            str(alignments[1]),
+            alignment[0], ["Pro", "Pro", "Gly", "Ala", None, None, "Thr", None, None]
+        )
+        self.assertEqual(
+            alignment[0, :], ["Pro", "Pro", "Gly", "Ala", None, None, "Thr", None, None]
+        )
+        self.assertEqual(
+            alignment[0, 1:], ["Pro", "Gly", "Ala", None, None, "Thr", None, None]
+        )
+        self.assertEqual(alignment[0, ::2], ["Pro", "Gly", None, "Thr", None])
+        self.assertEqual(
+            alignment[1], [None, None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn", "Asn"]
+        )
+        self.assertEqual(
+            alignment[1, :],
+            [None, None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn", "Asn"],
+        )
+        self.assertEqual(
+            alignment[1, 1:], [None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn", "Asn"]
+        )
+        self.assertEqual(alignment[1, ::2], [None, "Gly", "Ala", "Thr", "Asn"])
+        alignment = alignments[1]
+        self.assertEqual(
+            str(alignment),
             """\
 Pro Pro Gly --- Ala --- Thr --- ---
 --- --- ||| --- ||| --- ||| --- ---
 --- --- Gly Ala Ala Cys Thr Asn Asn
 """,
         )
-        self.assertAlmostEqual(alignments[0].score, 3.0)
-        self.assertAlmostEqual(alignments[1].score, 3.0)
+        self.assertAlmostEqual(alignment.score, 3.0)
+        self.assertEqual(
+            alignment[0], ["Pro", "Pro", "Gly", None, "Ala", None, "Thr", None, None]
+        )
+        self.assertEqual(
+            alignment[0, :], ["Pro", "Pro", "Gly", None, "Ala", None, "Thr", None, None]
+        )
+        self.assertEqual(
+            alignment[0, 1:-1], ["Pro", "Gly", None, "Ala", None, "Thr", None]
+        )
+        self.assertEqual(alignment[0, 1::2], ["Pro", None, None, None])
+        self.assertEqual(
+            alignment[1], [None, None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn", "Asn"]
+        )
+        self.assertEqual(
+            alignment[1, :],
+            [None, None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn", "Asn"],
+        )
+        self.assertEqual(
+            alignment[1, 1:-1], [None, "Gly", "Ala", "Ala", "Cys", "Thr", "Asn"]
+        )
+        self.assertEqual(alignment[1, 1::2], [None, "Ala", "Cys", "Asn"])
 
     def test_three_letter_amino_acids_local(self):
         seq1 = ["Asn", "Asn", "Gly", "Ala", "Thr", "Glu", "Glu"]
@@ -3383,24 +3427,46 @@ def test_three_letter_amino_acids_local(self):
         self.assertAlmostEqual(score, 3.0)
         alignments = aligner.align(seq1, seq2)
         self.assertEqual(len(alignments), 2)
+        alignment = alignments[0]
         self.assertEqual(
-            str(alignments[0]),
+            str(alignment),
             """\
 Gly Ala --- --- Thr
 ||| ||| --- --- |||
 Gly Ala Ala Cys Thr
 """,
         )
+        self.assertAlmostEqual(alignment.score, 3.0)
+        self.assertEqual(alignment[0], ["Gly", "Ala", None, None, "Thr"])
+        self.assertEqual(alignment[0, :], ["Gly", "Ala", None, None, "Thr"])
+        self.assertEqual(alignment[0, 1:], ["Ala", None, None, "Thr"])
+        self.assertEqual(alignment[0, :-1], ["Gly", "Ala", None, None])
+        self.assertEqual(alignment[0, ::2], ["Gly", None, "Thr"])
+        self.assertEqual(alignment[1], ["Gly", "Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, :], ["Gly", "Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, 1:], ["Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, :-1], ["Gly", "Ala", "Ala", "Cys"])
+        self.assertEqual(alignment[1, ::2], ["Gly", "Ala", "Thr"])
+        alignment = alignments[1]
         self.assertEqual(
-            str(alignments[1]),
+            str(alignment),
             """\
 Gly --- Ala --- Thr
 ||| --- ||| --- |||
 Gly Ala Ala Cys Thr
 """,
         )
-        self.assertAlmostEqual(alignments[0].score, 3.0)
-        self.assertAlmostEqual(alignments[1].score, 3.0)
+        self.assertAlmostEqual(alignment.score, 3.0)
+        self.assertEqual(alignment[0], ["Gly", None, "Ala", None, "Thr"])
+        self.assertEqual(alignment[0, :], ["Gly", None, "Ala", None, "Thr"])
+        self.assertEqual(alignment[0, 1:], [None, "Ala", None, "Thr"])
+        self.assertEqual(alignment[0, :-1], ["Gly", None, "Ala", None])
+        self.assertEqual(alignment[0, ::2], ["Gly", "Ala", "Thr"])
+        self.assertEqual(alignment[1], ["Gly", "Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, :], ["Gly", "Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, 1:], ["Ala", "Ala", "Cys", "Thr"])
+        self.assertEqual(alignment[1, :-1], ["Gly", "Ala", "Ala", "Cys"])
+        self.assertEqual(alignment[1, ::2], ["Gly", "Ala", "Thr"])
 
 
 class TestArgumentErrors(unittest.TestCase):