Repository: biopython/biopython

Issue:
Remove use of Bio._py3k (Python 2 / 3 compatibility)
As of Biopython 1.76 (released December 2019), we are dropping Python 2 support and focusing on Python 3.6 or later. This means we no longer need our (internal) Python 2 vs 3 compatibility library ``Bio._py3k`` (which is a bit like the third party library ``six``).

This issue is suitable and specifically targeting first time contributors.

There are lots of cases:

```
$ grep _py3k Bio*/*.py Bio/*/*.py Bio/*/*/*.py
Bio/File.py:from Bio._py3k import basestring
Bio/MarkovModel.py:        from Bio._py3k import StringIO
Bio/Seq.py:from Bio._py3k import range
Bio/Seq.py:from Bio._py3k import basestring
...
```

Example One
------------

Taking the first example, ``from Bio._py3k import basestring`` we see that this is defined under Python 3 as an alias of ``str``:

https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L56

```python
    # Lots of our Python 2 code uses isinstance(x, basestring)
    # which after 2to3 becomes isinstance(x, str)
    basestring = str
    unicode = str
```

Therefore the fix for ``Bio/File.py`` is to remove the ``from Bio._py3k import basestring`` line, and update where ``basestring`` was used to instead use ``str``, which in this case means editing one line:

```python
if isinstance(handleish, basestring):
```

with:

```python
if isinstance(handleish, str):
```

Example Two
------------

Taking the second example, ``Bio/MarkovModel.py`` has ``from Bio._py3k import StringIO`` which we find on Python 3 can just be replaced with ``from io import StringIO``

https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L130

Contributing
-----------

Could any newcomer wanting to work on this first comment on this issue saying which file(s) they are going to start with (e.g. ``Bio/File.py``, or ``BioSQL/``) to avoid duplication of effort.

(*Update: The consensus was to switch to be function or constant instead, since they generally require the same technique/fix each time*)

Then read https://github.com/biopython/biopython/blob/master/CONTRIBUTING.rst and setup ``flake8`` on your machine.

Then make a pull request making the necessary changes so that those files no longer import from ``Bio._py3k``. Once that's done, you could pick some more to work on.

Eventually there will be nothing using ``Bio._py3k`` and that code itself can be removed, and this issue closed.


Discussion:
I would be starting in `BioSQL/`
@peterjc What to do incase of methods? 
Example:
https://github.com/biopython/biopython/blob/biopython-176/BioSQL/Loader.py#L564

    from Bio._py3k import _is_int_or_long
    ...
    L564 -> assert _is_int_or_long(parent_taxon_id), repr(parent_taxon_id)
    
uses the `_is_init_or_long` method declared in `Bio._py3k` for assertion

https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L91
https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L176

Python 3 doesn't have longs, so should we drop the assertion and also in other similar cases where methods are used fro `_py3k `what should be done?
Python 3 doesn't separate int and long, just use ``isinstance(parent_taxon_id, int)`` instead.
@peterjc I am new to this code base so I am little bit confused. 
In _py3k there are many Private functions for encoding/decoding. Like: 

https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L64
https://github.com/biopython/biopython/blob/biopython-176/Bio/_py3k/__init__.py#L158

And used like this:

```
from Bio._py3k import _bytes_bytearray_to_str as bytearray_to_str

def execute_and_fetch_col0(self, sql, args=None):
        """Return a list of values from the first column in the row."""
        out = super(MysqlConnectorAdaptor, self).execute_and_fetch_col0(sql, args)
        return [bytearray_to_str(column) for column in out]
```

What should be done in these cases?
I think ``_bytes_bytearray_to_str`` is only used here in the ``MysqlConnectorAdaptor `` class in BioSQL. It is defined as follows for Python 3:

```python
    def _bytes_bytearray_to_str(s):
        """If s is bytes or bytearray, convert to a unicode string (PRIVATE)."""
        if isinstance(s, (bytes, bytearray)):
            return s.decode()
        return s
```

So, moving this definition into ``BioSQL/BioSeqDatabase.py`` might be the best short term solution.
@peterjc  Please review #2440 
Now I would be working in `Tests/`
Started working on functions in `Bio/`
(🍄 = working on it, ✔️ = done) 

| Function 	|  Status	| 
|---	|---	|
|  raise_from	|  ✔️	|
|  StringIO	|  ✔️	|
|  urlretrieve	|  ✔️ |
| urlopen |  ✔️ | 
|  Request |  ✔️ |
| urlparse |  ✔️ |
| urlcleanup | ✔️ |
|  urlencode |  ✔️ |
| quoteURLError |  ✔️ |
| HTTPError |  ✔️ |

That seems like a few, and covers quite a range. I wonder in hindsight if I should have suggested people work on a functions/constants basis rather than a module by module basis? Oh well, this way you'll all learn more about Python 2 vs Python 3.
@peterjc After I started doing it I was thinking of the same, I guess I should do it on a function basis rather than a module. Also, can you please check the code below and tell me if it is the right way of doing it.

![image](https://user-images.githubusercontent.com/20015794/71485251-36be2f00-2836-11ea-8e61-1a3ae31c48b5.png)

@deepaklorkhatri007  For this line:

https://github.com/biopython/biopython/blob/biopython-176/Bio/Align/__init__.py#L1359

In Python 3 we'd just do this:

```python
raise IndexError("index out of range") from None
```

This explicitly discards the traceback context (this was inside a try/except with a ``StopIteration`` error).
@peterjc  thanks for clarifying the change needed, I have one more question, to check if the code is valid I should be running the test file` $ python  test_align.py `and for formatting and code style, I should be using `$ flake8` right?
@deepaklorkhatri007 Depending which code you are editing, there may be more than one test covering it - ``run_tests.py`` would take longer but cover everything (see also the testing chapter in the Tutorial).

And yes, please do run ``flake8`` locally with all the plugins listed on https://github.com/biopython/biopython/blob/master/CONTRIBUTING.rst - the git pre-commit hook helps here.

Together that will catch most of the possible failures the automated testing with TravisCI and AppVeyor would pick up (and more since they don't run the online tests needing network access, like some of the BLAST tests you ought to run if you're editing ``Bio.Blast`` files).
@peterjc thanks for the information working on `raise_from()` function now
@peterjc sorry for bothering you constantly, can you please look at the code below

![image](https://user-images.githubusercontent.com/20015794/71486366-f95ca000-283b-11ea-9a5a-aa65326b1f3f.png)

When I run `$ flake8` on this file I am getting this output

![image](https://user-images.githubusercontent.com/20015794/71486404-2dd05c00-283c-11ea-9605-5374e3a04bcd.png)

all the mentioned lines in this output are for similar code fragment, what should I do here combine the lines or leave it as it is because apart from the formating warning the code is similar to the original code.

Rather than screenshots, links or code snippets wrapped in triple-back-ticks are easier to read. Anyway,

https://github.com/biopython/biopython/blob/biopython-176/Bio/Entrez/Parser.py#L363

```python
                _raise_from(
                    RuntimeError(
                        "Failed to parse the XML file correctly, possibly due to a bug "
                        "in Bio.Entrez. Please contact the Biopython developers via "
                        "the mailing list or GitHub for assistance."
                    ),
                    None,
                )
```

would probably best be replaced with:

```python
                raise RuntimeError(
                    "Failed to parse the XML file correctly, possibly due to a bug "
                    "in Bio.Entrez. Please contact the Biopython developers via "
                    "the mailing list or GitHub for assistance."
                ) from None
```

Notice the closing bracket is indented the same amount as the raise line - and the message lines are all indented by four more spaces. In your work in progress you need to un-indent the message and closing bracket line.

In this case ``Bio/Entrez/Parser.py`` has recently been formatted using black (see #2008), so you would be able to run ``black Bio/Entrez/Parser.py`` to fix up the formatting for you automatically.

[Note many of the Biopython files have not yet been updated to use the black style, and I'd like you to focus on just the python 2 work at the moment - pull requests doing too many things are much harder to review.]
Note in future pull requests please also remove the definition in ``Bio/_py3k/__init__.py`` as well, that way we can easily see what it left, until finally the entire file is no longer needed. I think #2463 brings us up to date. 

P.S. This would be an easy one for a newcomer: Remove all the ``from Bio._py3k import _universal_read_mode`` lines, and change ``open(filename, _universal_read_mode)`` to just ``open(filename)`` instead.
@peterjc  Finally, I was working on `Tests/` all ready to push but found out #2468 has already modified `Tests/test_bgzf.py` by removing `_as_bytes `and `_as_string` and also #2469 modifies `Tests/`. Are we now working function/constraint based? Seems like most of the work is done.  If those PRs were merged It would be easy for future work otherwise we have to deal with a lot of merge conflicts.
It seemed to be the group consensus, yes. There will probably still be some merge conflicts anyway since the import lines are side by side.

@Dsujan how about doing ``_universal_read_mode`` as outlined above?
@peterjc  Okay I would be working on `_universal_read_mode `
@Dsujan did you start working on the remaining ``_is_int_or_long`` yet, or just the one in BioSQL? Just six files to edit as discussed earlier to use ``isinstance(xxx, int)`` instead, and remove the function definition in ``Bio/_py3k/__init__.py`` to finish.
@peterjc  Had worked just the one in `BioSQL`.  Now I would be working in other remaining files. 
@dsujan - thank you! That and the remaining open pull requests should I think cover everything 👍 
2020 Official Death of Python 2 and seems like we have cleaned its remnants. 
Also relevant here, we can remove/refactor `sys.version_info` checks, an approximate list:

```
$ grep "version_info" Bio/*.py Bio/*/*.py Bio/*/*/*.py Tests/*py setup.py
Bio/File.py:if (sys.version_info >= (3, 7)  # Keep this one or assume dict ordering from 3.6?
Bio/File.py:        or (sys.version_info == (3, 6)
Bio/File.py:        if sys.version_info[0] >= 3 and "U" in mode:
Bio/Seq.py:    if sys.version_info[0] == 3:
Bio/Seq.py:        if sys.version_info[0] == 3:
Bio/SeqRecord.py:            if sys.version_info[0] < 3:
Bio/bgzf.py:    if sys.version_info[0] >= 3:
Bio/Phylo/BaseTree.py:if sys.version_info[0] < 3:
Bio/SearchIO/__init__.py:if sys.version_info < (3, 6):
Bio/SearchIO/__init__.py:    if format == "blast-xml" and sys.version_info[0] > 2:
Bio/SeqIO/__init__.py:if sys.version_info < (3, 6):
Tests/common_BioSQL.py:        if sys.version_info[0] == 2:
Tests/test_Phylo.py:        if sys.version_info[0] < 3:
Tests/test_Phylo.py:            if sys.version_info[0] < 3:
Tests/test_SeqIO_SeqXML.py:    if not (sys.platform == "win32" and sys.version_info[0] >= 3):
Tests/test_SeqIO_index.py:    if sys.version_info[0] < 3 or format in SeqIO._BinaryFormats:
Tests/test_SeqRecord.py:    if sys.version_info[0] >= 3:
Tests/test_Seq_objs.py:if sys.version_info[0] < 3:
Tests/test_SffIO.py:        if sys.platform != "win32" and sys.version_info[0] < 3:
Tests/test_geo.py:if sys.version_info[0] >= 3:
Tests/test_pairwise_aligner.py:        if sys.version_info[0] >= 3:
setup.py:    if sys.version_info[0] >= 3:
setup.py:if sys.version_info[:2] < (3, 6):   # Keep this one
setup.py:        "Python %d.%d detected.\n" % sys.version_info[:2] . # Keep this one
``` 
@chris-rands can you create a new issue for ``sys.version_info`` please (aside from ``setup.py`` which is a special case)? I would expect the usage under ``Tests/`` to all be beginner friendly, and some but not all of the rest will be too.
@peterjc  see new issue #2490; for `setup.py`, I'm not experienced with packaging, but think the correct way to specify the dependancy might be via `python-requires` https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires

Resolving patch:
diff --git a/Bio/_py3k/__init__.py b/Bio/_py3k/__init__.py
--- a/Bio/_py3k/__init__.py
+++ b/Bio/_py3k/__init__.py
@@ -14,20 +14,7 @@
 # this file. This line tells flake8 to ignore it for quality assurance:
 # flake8: noqa
 
-import sys
-
-import codecs
-
-
-def _bytes_bytearray_to_str(s):
-    """If s is bytes or bytearray, convert to a unicode string (PRIVATE)."""
-    if isinstance(s, (bytes, bytearray)):
-        return s.decode()
-    return s
-
-
 import io
-import locale
 
 # Python 3.4 onwards, the standard library wrappers should work:
 def _binary_to_string_handle(handle):
@@ -54,9 +41,3 @@ def _binary_to_string_handle(handle):
     except AttributeError:
         pass
     return wrapped
-
-
-# On Python 3 urllib, urllib2, and urlparse were merged:
-from urllib.request import urlopen, Request, urlparse, urlcleanup
-from urllib.parse import urlencode, quote
-from urllib.error import URLError, HTTPError


Test patch: