Repository: biopython/biopython

Issue:
Bio.Phylo.PAML.codeml's results parser quietly fails to read all the data
For a prettier version of the following go here: http://stackoverflow.com/posts/20872441/edit, or here: https://www.biostars.org/p/89848/.

**Biopython** comes with methods to interface with the **PAML** package for phylogenetic analysis.

In particular I am using [**Bio.Phylo.PAML**](http://biopython.org/wiki/PAML) to run analyses using PAML's codeml.exe program which in my case does Ka/Ks (dN/dS) ratio analysis on pairs of orthologous gene sequences.

After running the analysis using `results = cml.run()` I can see it has successfully generated `result.out` files that look about right. Most importantly the final line of the file is what I need to parse into Python:

```
t= 0.2173  S=   703.9  N=  1489.1  dN/dS=  0.2247  dN = 0.0344  dS = 0.1529
```

What I am most interested in is `dN/dS = 0.2247`

According to Biopython's [**PAML wiki**](http://biopython.org/wiki/PAML) this value can be obtained from Python by doing `results = cml.run()` which generates a dictionary with a set of values I am interested in after running the analysis. The wiki claims I can find the values I need in a key called `'parameters'`. But this only returns one of the values I need `t= 0.2173`, look: 

```
>>> results.values()
['Fcodon', 'One dN/dS ratio for branches, ', '4.7b', {0: {'description': 'one-ratio', 'parameters': {'t': 0.1982}}}, {'htlv': {}, 'stlv': {}}]
```

Notice, that my parameters key only contains the `t= 0.2173` and has omitted `S=   703.9  N=  1489.1  dN/dS=  0.2247  dN = 0.0344  dS = 0.1529`

Could anybody with `codeml` experience explain to me why the parser fails to yield most of the parameters (values) I am interested in?

**Extra details**
- Using Python2.7, PAML4.7a
- Running on Windows 7
- I will readily edit in any data or info you require to help me fix this



Discussion:
I thought this looked familiar - your BioStar question was originally over a year ago https://www.biostars.org/p/89848/

Which version of Biopython are you using here? CC @brandoninvergo

Could you please provide the results file?  Even better would be an alignment, tree and parameters that cause the problem.

The huge combination of possible parameters, the changing output under different parameter combos, and the occasionally changing output from one PAML version to the next all mean that it is very likely that the parser doesn't know how to handle your particular situation.

Also, as a sanity check, have you run the Bio.Phylo.PAML unit tests?


Resolving patch:
diff --git a/Bio/Phylo/PAML/_parse_codeml.py b/Bio/Phylo/PAML/_parse_codeml.py
--- a/Bio/Phylo/PAML/_parse_codeml.py
+++ b/Bio/Phylo/PAML/_parse_codeml.py
@@ -5,7 +5,6 @@
 
 import re
 
-
 line_floats_re = re.compile("-*\d+\.\d+")
 
 try:
@@ -412,6 +411,8 @@ def parse_pairwise(lines, results):
     # t= 0.0126  S=    81.4  N=   140.6  dN/dS= 0.0010  dN= 0.0000  dS= 0.0115
     pair_re = re.compile("\d+ \((.+)\) ... \d+ \((.+)\)")
     pairwise = {}
+    seq1 = None
+    seq2 = None
     for line in lines:
         # Find all floating point numbers in this line
         line_floats_res = line_floats_re.findall(line)
@@ -420,21 +421,21 @@ def parse_pairwise(lines, results):
         if pair_res:
             seq1 = pair_res.group(1)
             seq2 = pair_res.group(2)
-            if pairwise.get(seq1) is None:
+            if seq1 not in pairwise:
                 pairwise[seq1] = {}
-            if pairwise.get(seq2) is None:
+            if seq2 not in pairwise:
                 pairwise[seq2] = {}
-            if len(line_floats) == 1:
-                pairwise[seq1][seq2] = {"lnL": line_floats[0]}
-                pairwise[seq2][seq1] = pairwise[seq1][seq2]
-            elif len(line_floats) == 6:
-                pairwise[seq1][seq2] = {"t": line_floats[0],
-                                        "S": line_floats[1],
-                                        "N": line_floats[2],
-                                        "omega": line_floats[3],
-                                        "dN": line_floats[4],
-                                        "dS": line_floats[5]}
-                pairwise[seq2][seq1] = pairwise[seq1][seq2]
+        if len(line_floats) == 1 and seq1 is not None and seq2 is not None:
+            pairwise[seq1][seq2] = {"lnL": line_floats[0]}
+            pairwise[seq2][seq1] = pairwise[seq1][seq2]
+        elif len(line_floats) == 6 and seq1 is not None and seq2 is not None:
+            pairwise[seq1][seq2].update({"t": line_floats[0],
+                                         "S": line_floats[1],
+                                         "N": line_floats[2],
+                                         "omega": line_floats[3],
+                                         "dN": line_floats[4],
+                                         "dS": line_floats[5]})
+            pairwise[seq2][seq1] = pairwise[seq1][seq2]
     if pairwise:
         results["pairwise"] = pairwise
     return results


Test patch:
diff --git a/Tests/test_PAML_codeml.py b/Tests/test_PAML_codeml.py
--- a/Tests/test_PAML_codeml.py
+++ b/Tests/test_PAML_codeml.py
@@ -6,6 +6,8 @@
 import unittest
 import os
 import os.path
+import itertools
+
 from Bio.Phylo.PAML import codeml
 from Bio.Phylo.PAML._paml import PamlError
 
@@ -470,7 +472,39 @@ def testParsePairwise(self):
             self.assertEqual(len(results), 5, version_msg)
             self.assertTrue("pairwise" in results, version_msg)
             pairwise = results["pairwise"]
-            self.assertEqual(len(pairwise), 5, version_msg)
+            self.assertGreaterEqual(len(pairwise), 2, version_msg +
+                                    ": should have at least two sequences")
+            for seq1, seq2 in itertools.combinations(pairwise.keys(), 2):
+                self.assertEqual(len(pairwise[seq1][seq2]), 7, version_msg +
+                                 ": wrong number of parameters parsed")
+                self.assertEqual(len(pairwise[seq2][seq1]), 7, version_msg +
+                                 ": wrong number of parameters parsed")
+
+    def testParseSitesParamsForPairwise(self):
+        """Verify that pairwise site estimates are indeed parsed. Fixes #483"""
+        res_dir = os.path.join(self.results_dir, "codeml", "pairwise")
+        for results_file in os.listdir(res_dir):
+            version = results_file.split('-')[1].split('.')[0]
+            version_msg = "Improper parsing for version %s" \
+                        % version.replace('_', '.')
+            results_path = os.path.join(res_dir, results_file)
+            results = codeml.read(results_path)
+            self.assertTrue("pairwise" in results)
+            seqs = list(results["pairwise"].keys())
+            self.assertGreaterEqual(len(seqs), 2, version_msg +
+                                    ": should have at least two sequences")
+            for seq1, seq2 in itertools.combinations(seqs, 2):
+                params = results["pairwise"][seq1][seq2]
+                self.assertEqual(len(params), 7,
+                                 version_msg + ": wrong number of parsed parameters" +
+                                 " for %s-%s" % (seq1, seq2))
+                for param in ("t", "S", "N", "omega", "dN", "dS", "lnL"):
+                    self.assertTrue(param in params, version_msg +
+                                    ": '%s' not in parsed parameters"
+                                    % (param))
+                    self.assertTrue(isinstance(params[param], float))
+                    if param != "lnL":
+                        self.assertTrue(params[param] >= 0)
 
     def testParseAA(self):
         res_dir = os.path.join(self.results_dir, "codeml", "aa_model0")