Repository: biopython/biopython

Issue:
Bio.motifs broken on C Python without NumPy
We have a C module `Bio.motifs._pwm` which used the NumPy C API, and is therefore only compiled and installed if NumPy is available at install time. If the user overrides the warning and installs anyway, later importing this will fail - and anything importing `Bio.motifs.matrix` will indirectly try to import this and fail.

See https://github.com/biopython/biopython/blob/master/Bio/motifs/matrix.py#L324

```
class PositionSpecificScoringMatrix(GenericPositionMatrix):

    # Make sure that we use C-accelerated PWM calculations if running under CPython.
    # Fall back to the slower Python implementation if Jython or IronPython.
    try:
        from . import _pwm

        def _calculate(self, sequence, m, n):
            logodds = [[self[letter][i] for letter in "ACGT"] for i in range(m)]
            return self._pwm.calculate(sequence, logodds)
    except ImportError:
        if platform.python_implementation() == 'CPython':
            raise
        else:
            def _calculate(self, sequence, m, n):
                # The C code handles mixed case so Python version must too:
                sequence = sequence.upper()
                scores = []
                for i in range(n - m + 1):
                    score = 0.0
                    for position in range(m):
                        letter = sequence[i + position]
                        try:
                            score += self[letter][position]
                        except KeyError:
                            score = float("nan")
                            break
                    scores.append(score)
                return scores
```

Notice there is a pure Python fall back intended for PyPy / IronPython and probably Jython.

Suggested resolutions:

(A) Make this into a lazy import so that is only fails when used, not at import.

and/or:

(B) Replace the `raise` with `raise MissingPythonDependencyError` saying you need to install numpy and then re-install Biopython.

or:

(C) Use the pure Python fall back on C Python with a warning?

What seems more appropriate @mdehoon?

Either way, the unit tests need updating in some way for use in the special case when NumPy was not installed.

Unfortunately it looks like all our C Python buildbot and TravisCI configurations must have NumPy installed, otherwise we'd have caught this earlier. I found this while trying tox for testing which uses virtual environments which (by default) did not include NumPy. e.g. https://travis-ci.org/peterjc/biopython/builds/137202798 (C Python failures) vs https://travis-ci.org/peterjc/biopython/builds/137210191 where I explicitly install numpy

Bio.motifs broken on C Python without NumPy
We have a C module `Bio.motifs._pwm` which used the NumPy C API, and is therefore only compiled and installed if NumPy is available at install time. If the user overrides the warning and installs anyway, later importing this will fail - and anything importing `Bio.motifs.matrix` will indirectly try to import this and fail.

See https://github.com/biopython/biopython/blob/master/Bio/motifs/matrix.py#L324

```
class PositionSpecificScoringMatrix(GenericPositionMatrix):

    # Make sure that we use C-accelerated PWM calculations if running under CPython.
    # Fall back to the slower Python implementation if Jython or IronPython.
    try:
        from . import _pwm

        def _calculate(self, sequence, m, n):
            logodds = [[self[letter][i] for letter in "ACGT"] for i in range(m)]
            return self._pwm.calculate(sequence, logodds)
    except ImportError:
        if platform.python_implementation() == 'CPython':
            raise
        else:
            def _calculate(self, sequence, m, n):
                # The C code handles mixed case so Python version must too:
                sequence = sequence.upper()
                scores = []
                for i in range(n - m + 1):
                    score = 0.0
                    for position in range(m):
                        letter = sequence[i + position]
                        try:
                            score += self[letter][position]
                        except KeyError:
                            score = float("nan")
                            break
                    scores.append(score)
                return scores
```

Notice there is a pure Python fall back intended for PyPy / IronPython and probably Jython.

Suggested resolutions:

(A) Make this into a lazy import so that is only fails when used, not at import.

and/or:

(B) Replace the `raise` with `raise MissingPythonDependencyError` saying you need to install numpy and then re-install Biopython.

or:

(C) Use the pure Python fall back on C Python with a warning?

What seems more appropriate @mdehoon?

Either way, the unit tests need updating in some way for use in the special case when NumPy was not installed.

Unfortunately it looks like all our C Python buildbot and TravisCI configurations must have NumPy installed, otherwise we'd have caught this earlier. I found this while trying tox for testing which uses virtual environments which (by default) did not include NumPy. e.g. https://travis-ci.org/peterjc/biopython/builds/137202798 (C Python failures) vs https://travis-ci.org/peterjc/biopython/builds/137210191 where I explicitly install numpy



Discussion:
Another option:

(D) Update `setup.py` to make numpy a hard compile-time dependency for Biopython on C Python (continue not to require it for PyPy, Jython, etc). Installing NumPy via a Windows installer was always quite quick, but it was a pain if it needed compiling locally. Now that you can do `pip install numpy --only-binary=numpy` and get a precompiled wheel for numpy, installing NumPy should be very quick.

This will also let us simplify the test environments like TravisCI to avoid piping a yes into the install step in case the prompt to force install without NumPy comes up.

(E) Tweak the test suite to pass anyway, e.g. adding this to ``test_motifs.py`` works:

``` python
# We require NumPy as a build dependency and runtime dependency,
# so check this first:
try:
    import numpy
except ImportError:
    from Bio import MissingPythonDependencyError
    raise MissingPythonDependencyError(
        "Install NumPy if you want to use PWM in Bio.motifs")

# Given NumPy is installed, if we can't import _pwm this is
# most likely due to Biopython being installed without NumPy
try:
    from Bio.motifs import _pwm
except ImportError:
    from Bio import MissingPythonDependencyError
    raise MissingPythonDependencyError("If you want to use PWM in Bio.motifs, "
                                       "install NumPy first and then "
                                       "reinstall Biopython")
```

Similarly the doctests would need to be explicitly tagged as NumPy dependent.
Another option:

(D) Update `setup.py` to make numpy a hard compile-time dependency for Biopython on C Python (continue not to require it for PyPy, Jython, etc). Installing NumPy via a Windows installer was always quite quick, but it was a pain if it needed compiling locally. Now that you can do `pip install numpy --only-binary=numpy` and get a precompiled wheel for numpy, installing NumPy should be very quick.

This will also let us simplify the test environments like TravisCI to avoid piping a yes into the install step in case the prompt to force install without NumPy comes up.

(E) Tweak the test suite to pass anyway, e.g. adding this to ``test_motifs.py`` works:

``` python
# We require NumPy as a build dependency and runtime dependency,
# so check this first:
try:
    import numpy
except ImportError:
    from Bio import MissingPythonDependencyError
    raise MissingPythonDependencyError(
        "Install NumPy if you want to use PWM in Bio.motifs")

# Given NumPy is installed, if we can't import _pwm this is
# most likely due to Biopython being installed without NumPy
try:
    from Bio.motifs import _pwm
except ImportError:
    from Bio import MissingPythonDependencyError
    raise MissingPythonDependencyError("If you want to use PWM in Bio.motifs, "
                                       "install NumPy first and then "
                                       "reinstall Biopython")
```

Similarly the doctests would need to be explicitly tagged as NumPy dependent.

Resolving patch:
diff --git a/Bio/motifs/matrix.py b/Bio/motifs/matrix.py
--- a/Bio/motifs/matrix.py
+++ b/Bio/motifs/matrix.py
@@ -16,6 +16,44 @@
 from Bio import Alphabet
 
 
+# Make sure that we use C-accelerated PWM calculations if running under CPython.
+# Fall back to the slower Python implementation if Jython or IronPython.
+try:
+    from . import _pwm
+
+    def _calculate(score_dict, sequence, m, n):
+        """Calculate scores using C code (PRIVATE)."""
+        logodds = [[score_dict[letter][i] for letter in "ACGT"] for i in range(m)]
+        return _pwm.calculate(sequence, logodds)
+
+except ImportError:
+    if platform.python_implementation() == 'CPython':
+        import warnings
+        from Bio import BiopythonWarning
+        warnings.warn("Using pure-Python as missing Biopython's C code for PWM. "
+                      "This can happen if Biopython was installed without NumPy. "
+                      "Try re-installing NumPy and then Biopython.")
+
+    def _calculate(score_dict, sequence, m, n):
+        """Calculate scores using Python code (PRIVATE).
+
+        The C code handles mixed case so Python version must too.
+        """
+        sequence = sequence.upper()
+        scores = []
+        for i in range(n - m + 1):
+            score = 0.0
+            for position in range(m):
+                letter = sequence[i + position]
+                try:
+                    score += score_dict[letter][position]
+                except KeyError:
+                    score = float("nan")
+                    break
+            scores.append(score)
+        return scores
+
+
 class GenericPositionMatrix(dict):
 
     def __init__(self, alphabet, values):
@@ -323,34 +361,6 @@ def log_odds(self, background=None):
 
 class PositionSpecificScoringMatrix(GenericPositionMatrix):
 
-    # Make sure that we use C-accelerated PWM calculations if running under CPython.
-    # Fall back to the slower Python implementation if Jython or IronPython.
-    try:
-        from . import _pwm
-
-        def _calculate(self, sequence, m, n):
-            logodds = [[self[letter][i] for letter in "ACGT"] for i in range(m)]
-            return self._pwm.calculate(sequence, logodds)
-    except ImportError:
-        if platform.python_implementation() == 'CPython':
-            raise
-        else:
-            def _calculate(self, sequence, m, n):
-                # The C code handles mixed case so Python version must too:
-                sequence = sequence.upper()
-                scores = []
-                for i in range(n - m + 1):
-                    score = 0.0
-                    for position in range(m):
-                        letter = sequence[i + position]
-                        try:
-                            score += self[letter][position]
-                        except KeyError:
-                            score = float("nan")
-                            break
-                    scores.append(score)
-                return scores
-
     def calculate(self, sequence):
         """Returns the PWM score for a given sequence for all positions.
 
@@ -377,7 +387,7 @@ def calculate(self, sequence):
         m = self.length
         n = len(sequence)
 
-        scores = self._calculate(sequence, m, n)
+        scores = _calculate(self, sequence, m, n)
 
         if len(scores) == 1:
             return scores[0]


Test patch: