Repository: biopython/biopython

Issue:
editable/develop install warning: You may be importing Biopython from inside the source tree. 
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```pycon
$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.version)
3.7.4 (default, Aug 13 2019, 15:17:50) 
[Clang 4.0.1 (tags/RELEASE_401/final)]
>>> import platform; print(platform.python_implementation()); print(platform.platform())
CPython
Darwin-18.7.0-x86_64-i386-64bit
>>> import Bio; print(Bio.__version__)
/Users/xxx/repositories/biopython/Bio/__init__.py:128: BiopythonWarning: You may be importing Biopython from inside the source tree. This is bad practice and might lead to downstream issues. In particular, you might encounter ImportErrors due to missing compiled C extensions. We recommend that you try running your code from outside the source tree. If you are outside the source tree then you have a setup.py file in an unexpected directory: /Users/xxx/repositories/biopython.
  format(_parent_dir), BiopythonWarning)
1.75.dev0
```

(*Please copy and run the above in your Python, and copy-and-paste the output*)

### Expected behaviour

No warning ``BiopythonWarning: You may be importing Biopython from inside the source tree. ...``

### Actual behaviour

Noisy warning as above.

### Steps to reproduce

Using pip to install in editable (develop) mode:

```
$ pip install -h
...
  -e, --editable <path/url>   Install a project in editable mode (i.e. setuptools "develop mode") from a local
                              project path or a VCS url.
...
```

```
$ git clone git@github.com:biopython/biopython.git
$ cd biopython
$ pip install -e .
```

This is an unfortunate side effect of the changes in #2007, intended to help with confusing messages when C code was not compiled.


Discussion:
We discussed quite a few alternatives in #2007, none of which were perfect. Do you have I suggested fix? 
I've not looked into how exactly editable mode works - it seems to cope with C extensions with them too in the source tree - this is on macOS:

```
$ cd
$ python
...
>>> import Bio
/Users/xxx/repositories/biopython/Bio/__init__.py:128: BiopythonWarning: You may be importing Biopython from inside the source tree. This is bad practice and might lead to downstream issues. In particular, you might encounter ImportErrors due to missing compiled C extensions. We recommend that you try running your code from outside the source tree. If you are outside the source tree then you have a setup.py file in an unexpected directory: /Users/xxx/repositories/biopython.
  format(_parent_dir), BiopythonWarning)
>>> Bio.__file__
'/Users/xxx/repositories/biopython/Bio/__init__.py'
>>> from Bio.Nexus import cnexus
>>> cnexus.__file__
'/Users/xxx/repositories/biopython/Bio/Nexus/cnexus.cpython-37m-darwin.so'
```

Perhaps the original try/excepts on all the C code imports was better, despite being more work?

CC @mdehoon
On reflection, I'm in favour of removing the warning message. I now test custom biopython branches using`editable` via conda with an `environment.yml` that looks like this:
```
name: biopython_dev
dependencies:
  - python
  - pip
  - pip:
    - "--editable=git+https://github.com/chris-rands/biopython@str_translate#egg=str_translate"
```

I'm not sure if this is a good setup for testing branches(?), but anyway, this means I always get the warning traceback, which is inappropriate. I do not have a better solution than the original custom `ImportError` messages for the C extensions or leaving the user to debug any installation issues.
Yeah, I think so too - as implemented our `BiopythonWarning: You may be importing Biopython from inside the source tree. ...` is breaking a very real use case. e.g. It deters me from using developer mode (which is otherwise especially useful if you tend only to modify the Python code).

I've not thought of any way to fix the warning; reverting back to the old try/except on each major C module may be best.
So you'd only issue the warning if you imported a C extension or a module that might call it? That seems very reasonable!
So we are agreeing towards (re)instating a custom traceback via try-except clauses (like in `Bio/Align/__init__.py ` when `_aligners` is imported). Which imports exactly would need this form of custom traceback?
We need @mdehoon's input before making any change here.

There are also parts of the C code with a Python fall back (Nexus and pairwise2?) which used to be important for Jython (deprecated) and PyPy (C code should generally work now).
These are the extensions that are build when doing `python setup.py build`:

- Bio.Align._aligners
- Bio.cpairwise2
- Bio.Nexus.cnexus
- Bio.PDB.QCPSuperimposer.qcprotmodule
- Bio.motifs._pwm
- Bio.Cluster._cluster
- Bio.PDB.kdtrees

As @peterjc said, `pairwise2` has a pure Python fallback, it prints a warning and continues. The same seems to be true for `Nexus.Nexus` (but without giving a warning).

I thought we got the idea for this type of warning mechanism from another project? Was it `numpy`? Do they have similar reported issues?

Another option is just to remove the warning, without taking care about the imports. And we could move the text of the warning message to our `Readme.rst`. Possibly in a FAQ paragraph?  







Removing the warning and adding an FAQ might be enough given the dominance of wheels nowadays.

Nowadays most people will install from wheels, which will have the compiled files. The only people who will hit the C import errors are self-builders (who are likely to be more knowledgable?) and people running in the source folder (likely to be novices not using a wheel?). It was the later group our warning aimed to help.
> I thought we got the idea for this type of warning mechanism from another project? Was it numpy? Do they have similar reported issues?

We discussed what `numpy` do https://github.com/biopython/biopython/pull/2007#issuecomment-489569364, but never implemented it. I think we dismissed it as too complex, but I guess it has the advantage that what they do is more robust and will not give these false positive warnings for editable/develop installations
+1 on following NumPy with their `__config__` import check approach. It is more complicated, but I accept now that the simple option we took has severe downsides.
Numpy's solution sounds good to me.
`numpy`'s solution is quite involved and difficult to emulate in full- it generates a lot of config info inside the file. Instead, i'm trying a light-weight version where  an empty `Bio.__config__.py` is added during installation but remains absent from the source tree. In my case I copy the file from `Scripts/__config__.py`. Then in `Bio/__init__.py` a try-except clause wraps `import Bio.__config__`.

However, I'm struggling, I tried adding to `setup.py` either `data_files=[("Bio", ["Scripts/__config__.py"]),]` or `package_data={"Bio": ["Scripts/__config__.py"]}` but without success. I'm not sure if the `MANIFEST.in` can be edited to copy files in this way where the structure of the source tree and install dir is not identical. Perhaps someone with more experience in packaging can help?
You can probably do ``touch Bio/__config__.py`` during the build rather than a copy... if in Python land, maybe something like:

```
with open("Bio/__config__.py", "w") as handle:
    handle.write('"""This is a dummy file created during installation to control a local import warning."""\n')
```

I would guess adding ``Bio/__config__.py`` explicitly to ``MANIFEST.in`` will work - it would be there during build time?
Thanks @peterjc that would work too, but the part I'm not sure on is how to add `__config__.py` to the `/path/lib/python3.7/site-packages/biopython-*.egg/Bio/` dir and not the source `Bio/` dir (unless editable mode is used)?
Ah - I begin to see the conundrum. (Possibly relevant question: Does NumPy's solution actually work for develop/editable mode?)

Maybe we remove the clever warning, and go back to catching a few more likely ``ImportError`` exceptions with a little more guidance?
> (Possibly relevant question: Does NumPy's solution actually work for develop/editable mode?)

Yes it does, I tested running `pip install -e .` and it adds the `__config__.py` correctly. The issue is that numpy's install setup is really involved so it's hard to understand/emulate.

I was thinking we could maybe use numpy's own [config class](https://github.com/numpy/numpy/blob/a7401aee4d6d465977760e9e4b16f793d60f4ee4/numpy/distutils/misc_util.py#L726 ) for this since numpy is a biopython dependancy. This would also have the advantage of logging all the system config information at the time of building the package. However, this would mean that numpy would need installing *before* we call `setup.py`
Thanks for all the detective work @chris-rands !

I'm definitely not a packaging expert but let me see if I get this straight. Is the idea to have a `__config__.py` file that _only_ gets created/copied when you do `setup.py install/develop`? And that way, we could control in `Bio/__init__.py` if we are running from source or not?
Yes @JoaoRodrigues that's a good summary thanks

I guess another idea would be to create a dummy C extension for this after all the regular ones are built
One extremely naive idea is to check if we can find any egg file for Biopython. Both `setup.py install` and `setup.py develop` create egg files, `egg-link` and `.egg` respectively, in a `site-packages` folder, which can be global or on the user's home. We can use the `site` module to find the paths to these folders and just check for any file with 'egg' in its extension and 'biopython' in the name? We can add checking for `dist-info` to check if Biopython was installed via `pip install biopython` or `conda install biopython` (wheels).

I wrote this silly function to do this:

```python
def used_setuptools():
    """Returns True if Biopython was setup using setuptools."""

    import pathlib
    import site

    pkgdirs = site.getsitepackages()
    pkgdirs.append(site.getusersitepackages())  # user

    for p in pkgdirs:
        p = pathlib.Path(p)
        if not p.exists():
            continue
        for child in p.iterdir():
            if 'egg' in child.suffix:
                if 'biopython' in child.name:
                    return True  # install from source
            elif child.suffix == 'dist-info':
                if biopython in child.name:
                    return True  # install from wheel

    return False

is_imported = used_setuptools()
print(is_imported)
```

I tested on Windows/Linux with:
 - `python setup.py install` = True
 - `python setup.py develop` = True
 - `pip install -e .` = True
 - `pip install --user -e .` = True
 - `pip install biopython` = True
 - `conda install biopython` = True
 - manually add source folder to PYTHONPATH (only tested on Linux) = False

It's a very simple approach with maybe several caveats that I'm not seeing/ignoring because I don't know enough about the problem. But it would be easy to add to `Bio/__init__.py`, run it on import and if False, issue the warning.
Thanks @JoaoRodrigues ! That looks like a neat approach although I'm unfamiliar with it, so maybe someone else can comment. How much overhead does it add? We want `import Bio` to be fast of course so I wonder if your logic can ran at the end of the build to generate the dummy flag/file (e.g. `__config__.py`) that we then simply need to check for in `Bio/__init__.py`?
I wouldn't say it adds much, only if the user has thousands of packages installed. Also, since `biopython` starts with a 'b' it should find it pretty quickly. The worst case scenario here is that `return False` where you have to go through all the files in those `pkgdirs` if biopython is not installed.

Adding my function to `Bio/__init__.py` and using it instead of the `os.path.exists` test gives this:

`python setup.py develop`
```
(biopy-test) [11:42] joaor@Tests $ time python -X importtime test_imports.py
import time: self [us] | cumulative | imported package
...
import time:       467 |       1792 |   os
...
import time:       853 |       4721 | site
...
import time:       850 |      10974 |   pathlib
import time:      5945 |      17279 | Bio

real    0m0.033s
user    0m0.020s
sys     0m0.000s
```

Added source folder to PYTHONPATH (worse case scenario)
```
(biopy-test) [11:46] joaor@Tests $ time python -X importtime test_imports.py
import time: self [us] | cumulative | imported package
...
import time:     12297 |      29430 | Bio

real    0m0.050s
user    0m0.008s
sys     0m0.016s
```

Left out a bunch of output from other modules being imported, kept the two I added (`site` and `pathlib`). Seems like peanuts, a few milliseconds on my machine.

Resolving patch:
diff --git a/Bio/__init__.py b/Bio/__init__.py
--- a/Bio/__init__.py
+++ b/Bio/__init__.py
@@ -115,15 +115,33 @@ class BiopythonExperimentalWarning(BiopythonWarning):
 
 _parent_dir = os.path.dirname(os.path.dirname(__file__))
 if os.path.exists(os.path.join(_parent_dir, "setup.py")):
-    warnings.warn(
-        "You may be importing Biopython from inside the source tree."
-        " This is bad practice and might lead to downstream issues."
-        " In particular, you might encounter ImportErrors due to"
-        " missing compiled C extensions. We recommend that you"
-        " try running your code from outside the source tree."
-        " If you are outside the source tree then you have a"
-        " setup.py file in an unexpected directory: " + _parent_dir,
-        BiopythonWarning,
-    )
-# See #PR 2007 and issue #1991 for discussion on this warning:
-# https://github.com/biopython/biopython/pull/2007
+    # Looks like we are running from our source directory,
+    # a bad idea except if installed in development mode.
+    #
+    # See https://setuptools.readthedocs.io/en/latest/userguide/development_mode.html
+    # Do we have .../site-packages/biopython.egg-link present?
+    #
+    # Note "pip install -e ." currently calls setuptools internally
+    import site
+
+    _dev_mode = False
+    for _p in site.getsitepackages():
+        if os.path.isfile(os.path.join(_p, "biopython.egg-link")):
+            _dev_mode = True
+            break
+    # Also check the user specific site packages
+    if not _dev_mode and os.path.isfile(
+        os.path.join(site.getusersitepackages(), "biopython.egg-link")
+    ):
+        _dev_mode = True
+    if not _dev_mode:
+        warnings.warn(
+            "You may be importing Biopython from inside the source tree."
+            " This is bad practice and might lead to downstream issues."
+            " In particular, you might encounter ImportErrors due to"
+            " missing compiled C extensions. We recommend that you"
+            " try running your code from outside the source tree."
+            " If you are outside the source tree then you have a"
+            " setup.py file in an unexpected directory: " + _parent_dir,
+            BiopythonWarning,
+        )


Test patch: