Repository: biopython/biopython

Issue:
Error 404 in Bio.ExPASy 
Good morning,

I get error 404 when i use Bio.ExPASy.get_prosite_raw(), Bio.ExPASy.get_prosite_entry() and Bio.ExPASY.get_prodoc_entry().
Thanks
Format change in prosite.dat
Hi and thanks for your work on this nice library!

In Biopython 1.68, the new date format of prosite.dat is not supported. Here is the error message:
  
```
/usr/lib64/python3.5/site-packages/Bio/ExPASy/Prosite.py in parse(handle)
     34     handle   - handle to the file."""
     35     while True:
---> 36         record = __read(handle)
     37         if not record:
     38             break

/usr/lib64/python3.5/site-packages/Bio/ExPASy/Prosite.py in __read(handle)
    172                (not dates[1].endswith('(DATA UPDATE)')) or \
    173                (not dates[2].endswith('(INFO UPDATE)')):
--> 174                 raise ValueError("I don't understand date line\n%s" % line)
    175             record.created = dates[0].rstrip(' (CREATED)')
    176             record.data_update = dates[1].rstrip(' (DATA UPDATE)')

ValueError: I don't understand date line
DT   01-APR-1990 CREATED; 01-APR-1990 DATA UPDATE; 01-APR-1990 INFO UPDATE.
```

The format of DT field changed indeed recently according to ftp://ftp.expasy.org/databases/prosite/NEWS to add a day field (and remove parenthesis!) :

New (current) format:
```
DT DD-MMM-YYYY CREATED; DD-MMM-YYYY DATA UPDATE; DD-MMM-YYYY INFO UPDATE.
```
Old format:
```
DT MMM-YYYY (CREATED); MMM-YYYY (DATA UPDATE); MMM-YYYY (INFO UPDATE).
```

Example:
New (current) format:
```
DT 01-APR-1990 CREATED; 01-MAY-2005 DATA UPDATE; 01-OCT-2016 INFO UPDATE.
```
Old format:
```
DT APR-1990 (CREATED); MAY-2005 (DATA UPDATE); OCT-2016 (INFO UPDATE).
```

The release numbers seems to have changed also in January 2017....
Best regard,
François

[Peter: Edited to include triple-back-tick markup of the examples]



Discussion:
Which version of Biopython do you have? You can check with:

```python
>>> import Bio
>>> print(Bio.__version)
...
```
It looks like only ``Bio.ExPASy.get_sprot_raw(...)`` has unit test coverage, which explains why we didn't notice till your report:

https://codecov.io/gh/biopython/biopython/src/master/Bio/ExPASy/__init__.py

Could you give a specific example in each case which ought to work? i.e. Sensible ID for a small record which we could use for a new test.
My version is 1.67

I really do not know what the output of these methods would be like, I'm just reading the documentation for this site http://biopython.org/DIST/docs/tutorial/Tutorial.html#htoc144. In session 10.5.3 you have the use of these methods. I believe the method "handle = ExPASy.get_prosite_raw ('PDOC00001')" returns a  Bio.Prosite.Record object 
Ah - that helps, I've never used these functions myself but should have checked there. e.g.

```
>>> from Bio import ExPASy
>>> from Bio.ExPASy import Prodoc
>>> handle = ExPASy.get_prosite_raw('PDOC00001')
>>> record = Prodoc.read(handle)
```

I'm pretty sure the ExPASy site will explain the new URL structure somewhere - or anyone familiar with the site can probably figure it out...

Cross referencing #1072 which explores including the online doctests into ``test_Tutorial.py``.
This looks like the raw data URL we need: http://prosite.expasy.org/PDOC00001.txt
It looks like for Python 3 we might want to return a text mode handle (as expected in our parser), as done in e.g. ``Bio.TogoWS`` and ``Bio.Entrez``...
Thank you for your well researched report - would you like to try preparing a fix for this (pull request with one new test file) so that it can parse both old and new style ``DT`` lines?
Hi again,
I am not familiar with github and biopython, but the fix can be very simple...
I have made a pull request "small fix to deal with new date format in prosite" #1132
I hope it can help...
Best,
François

Resolving patch:
diff --git a/Bio/ExPASy/Prosite.py b/Bio/ExPASy/Prosite.py
--- a/Bio/ExPASy/Prosite.py
+++ b/Bio/ExPASy/Prosite.py
@@ -9,7 +9,7 @@
 
 Tested with:
 Release 20.43, 10-Feb-2009
-
+Release 2017_03 of 15-Mar-2017.
 
 Functions:
 
@@ -61,7 +61,8 @@ class Record(object):
         - name           ID of the record.  e.g. ADH_ZINC
         - type           Type of entry.  e.g. PATTERN, MATRIX, or RULE
         - accession      e.g. PS00387
-        - created        Date the entry was created.  (MMM-YYYY)
+        - created        Date the entry was created.  (MMM-YYYY for releases
+                         before January 2017, DD-MMM-YYYY since January 2017)
         - data_update    Date the 'primary' data was last updated.
         - info_update    Date data other than 'primary' data was last updated.
         - pdoc           ID of the PROSITE DOCumentation.
@@ -165,14 +166,25 @@ def __read(handle):
         elif keyword == 'AC':
             record.accession = value.rstrip(';')
         elif keyword == 'DT':
+            # e.g. from January 2017,
+            # DT   01-APR-1990 CREATED; 01-APR-1990 DATA UPDATE; 01-APR-1990 INFO UPDATE.
+            # Older files had brackets round the date descriptions and used MMM-YYYY
             dates = value.rstrip('.').split("; ")
-            if (not dates[0].endswith('(CREATED)')) or \
-               (not dates[1].endswith('(DATA UPDATE)')) or \
-               (not dates[2].endswith('(INFO UPDATE)')):
+            if dates[0].endswith((' (CREATED)', ' CREATED')):
+                # Remove last word
+                record.created = dates[0].rsplit(" ", 1)[0]
+            else:
+                raise ValueError("I don't understand date line\n%s" % line)
+            if dates[1].endswith((' (DATA UPDATE)', ' DATA UPDATE')):
+                # Remove last two words
+                record.data_update = dates[1].rsplit(" ", 2)[0]
+            else:
+                raise ValueError("I don't understand date line\n%s" % line)
+            if dates[2].endswith((' (INFO UPDATE)', ' INFO UPDATE')):
+                # Remove last two words
+                record.info_update = dates[2].rsplit(" ", 2)[0]
+            else:
                 raise ValueError("I don't understand date line\n%s" % line)
-            record.created = dates[0].rstrip(' (CREATED)')
-            record.data_update = dates[1].rstrip(' (DATA UPDATE)')
-            record.info_update = dates[2].rstrip(' (INFO UPDATE)')
         elif keyword == 'DE':
             record.description = value
         elif keyword == 'PA':
diff --git a/Bio/ExPASy/__init__.py b/Bio/ExPASy/__init__.py
--- a/Bio/ExPASy/__init__.py
+++ b/Bio/ExPASy/__init__.py
@@ -5,7 +5,7 @@
 
 """Code to access resources at ExPASy over the WWW.
 
-See http://www.expasy.ch/
+See http://www.expasy.org/
 
 
 Functions:
@@ -20,65 +20,103 @@
 # Importing these functions with leading underscore as not intended for reuse
 from Bio._py3k import urlopen as _urlopen
 from Bio._py3k import urlencode as _urlencode
+from Bio._py3k import _binary_to_string_handle
 
 
-def get_prodoc_entry(id, cgi='http://www.expasy.ch/cgi-bin/get-prodoc-entry'):
-    """get_prodoc_entry(id,
-    cgi='http://www.expasy.ch/cgi-bin/get-prodoc-entry') -> handle
+def get_prodoc_entry(id,
+                     cgi='http://prosite.expasy.org/cgi-bin/prosite/get-prodoc-entry'):
+    """Get a text handle to a PRODOC entry at ExPASy in HTML format.
 
-    Get a handle to a PRODOC entry at ExPASy in HTML format.
+    >>> from Bio import ExPASy
+    >>> with ExPASy.get_prodoc_entry('PDOC00001') as in_handle:
+    ...     html = in_handle.read()
+    ...
+    >>> with open("myprodocrecord.html", "w") as out_handle:
+    ...     out_handle.write(html)
+    ...
 
     For a non-existing key XXX, ExPASy returns an HTML-formatted page
-    containing this line:
-    'There is no PROSITE documentation entry XXX. Please try again.'
+    containing this text: 'There is currently no PROSITE entry for'
     """
-    # Open a handle to ExPASy.
-    return _urlopen("%s?%s" % (cgi, id))
+    return _binary_to_string_handle(_urlopen("%s?%s" % (cgi, id)))
 
 
 def get_prosite_entry(id,
-                      cgi='http://www.expasy.ch/cgi-bin/get-prosite-entry'):
-    """get_prosite_entry(id,
-    cgi='http://www.expasy.ch/cgi-bin/get-prosite-entry') -> handle
+                      cgi='http://prosite.expasy.org/cgi-bin/prosite/get-prosite-entry'):
+    """Get a text handle to a PROSITE entry at ExPASy in HTML format.
 
-    Get a handle to a PROSITE entry at ExPASy in HTML format.
+    >>> from Bio import ExPASy
+    >>> with ExPASy.get_prosite_entry('PS00001') as in_handle:
+    ...     html = in_handle.read()
+    ...
+    >>> with open("myprositerecord.html", "w") as out_handle:
+    ...     out_handle.write(html)
+    ...
 
     For a non-existing key XXX, ExPASy returns an HTML-formatted page
-    containing this line:
-    'There is currently no PROSITE entry for XXX. Please try again.'
+    containing this text: 'There is currently no PROSITE entry for'
     """
-    return _urlopen("%s?%s" % (cgi, id))
+    return _binary_to_string_handle(_urlopen("%s?%s" % (cgi, id)))
 
 
-def get_prosite_raw(id, cgi='http://www.expasy.ch/cgi-bin/get-prosite-raw.pl'):
-    """get_prosite_raw(id,
-                       cgi='http://www.expasy.ch/cgi-bin/get-prosite-raw.pl')
-    -> handle
+def get_prosite_raw(id, cgi=None):
+    """Get a text handle to a raw PROSITE or PRODOC record at ExPASy.
 
-    Get a handle to a raw PROSITE or PRODOC entry at ExPASy.
+    The cgi argument is deprecated due to changes in the ExPASy
+    website.
 
     For a non-existing key, ExPASy returns nothing.
+
+    >>> from Bio import ExPASy
+    >>> from Bio.ExPASy import Prosite
+    >>> with ExPASy.get_prosite_raw('PS00001') as handle:
+    ...    record = Prosite.read(handle)
+    ...
+    >>> print(record.accession)
+    PS00001
+
+
+    For a non-existing key, ExPASy returns an error:
+
+    >>> handle = get_prosite_raw("does_not_exist")
+    Traceback (most recent call last):
+    ...
+    urllib.error.HTTPError: HTTP Error 404: Not Found
+
     """
-    return _urlopen("%s?%s" % (cgi, id))
+    url = "http://prosite.expasy.org/%s.txt" % id
+    return _binary_to_string_handle(_urlopen(url))
 
 
 def get_sprot_raw(id):
-    """Get a handle to a raw SwissProt entry at ExPASy.
+    """Get a text handle to a raw SwissProt entry at ExPASy.
 
     For an ID of XXX, fetches http://www.uniprot.org/uniprot/XXX.txt
     (as per the http://www.expasy.ch/expasy_urls.html documentation).
-    """
-    return _urlopen("http://www.uniprot.org/uniprot/%s.txt" % id)
 
+    >>> from Bio import ExPASy
+    >>> from Bio import SwissProt
+    >>> with ExPASy.get_sprot_raw("O23729") as handle:
+    ...     record = SwissProt.read(handle)
+    ...
+    >>> print(record.entry_name)
+    CHS3_BROFI
 
-def sprot_search_ful(text, make_wild=None, swissprot=1, trembl=None,
-                     cgi='http://www.expasy.ch/cgi-bin/sprot-search-ful'):
-    """sprot_search_ful(text, make_wild=None, swissprot=1, trembl=None,
-    cgi='http://www.expasy.ch/cgi-bin/sprot-search-ful') -> handle
+    For a non-existing identifier, UniProt returns an error:
 
-    Search SwissProt by full text.
+    >>> ExPASy.get_sprot_raw("DOES_NOT_EXIST")
+    Traceback (most recent call last):
+    ...
+    urllib.error.HTTPError: HTTP Error 404: Not Found
 
     """
+    url = "http://www.uniprot.org/uniprot/%s.txt" % id
+    return _binary_to_string_handle(_urlopen(url))
+
+
+def sprot_search_ful(text, make_wild=None, swissprot=1, trembl=None,
+                     cgi='http://www.expasy.ch/cgi-bin/sprot-search-ful'):
+    """Search SwissProt by full text (BROKEN)."""
     variables = {'SEARCH': text}
     if make_wild:
         variables['makeWild'] = 'on'
@@ -88,18 +126,15 @@ def sprot_search_ful(text, make_wild=None, swissprot=1, trembl=None,
         variables['T'] = 'on'
     options = _urlencode(variables)
     fullcgi = "%s?%s" % (cgi, options)
-    handle = _urlopen(fullcgi)
+    handle = _binary_to_string_handle(_urlopen(fullcgi))
     return handle
 
 
 def sprot_search_de(text, swissprot=1, trembl=None,
                     cgi='http://www.expasy.ch/cgi-bin/sprot-search-de'):
-    """sprot_search_de(text, swissprot=1, trembl=None,
-    cgi='http://www.expasy.ch/cgi-bin/sprot-search-de') -> handle
-
-    Search SwissProt by name, description, gene name, species, or
-    organelle.
+    """Search SwissProt (BROKEN).
 
+    Search by name, description, gene name, species, or organelle.
     """
     variables = {'SEARCH': text}
     if swissprot:
@@ -108,5 +143,4 @@ def sprot_search_de(text, swissprot=1, trembl=None,
         variables['T'] = 'on'
     options = _urlencode(variables)
     fullcgi = "%s?%s" % (cgi, options)
-    handle = _urlopen(fullcgi)
-    return handle
+    return _binary_to_string_handle(_urlopen(fullcgi))


Test patch:
diff --git a/Tests/test_ExPASy.py b/Tests/test_ExPASy.py
new file mode 100644
--- /dev/null
+++ b/Tests/test_ExPASy.py
@@ -0,0 +1,58 @@
+# Copyright 2017 by Peter Cock.  All rights reserved.
+# This code is part of the Biopython distribution and governed by its
+# license.  Please see the LICENSE file that should have been included
+# as part of this package.
+
+"""Testing Bio.ExPASy online code."""
+
+import unittest
+
+import requires_internet
+requires_internet.check()
+
+# We want to test these:
+from Bio import ExPASy
+
+# In order to check any records returned
+from Bio.ExPASy import Prodoc
+from Bio.ExPASy import Prosite
+
+
+# TODO - Use with statement when drop Python 2
+
+class ExPASyOnlineTests(unittest.TestCase):
+    """Test ExPASy online resources."""
+
+    def test_prosite_raw(self):
+        handle = ExPASy.get_prosite_raw('PS00001')
+        record = Prosite.read(handle)
+        handle.close()
+        self.assertEqual(record.accession, 'PS00001')
+        self.assertEqual(record.name, 'ASN_GLYCOSYLATION')
+
+    def test_prodoc_raw(self):
+        handle = ExPASy.get_prosite_raw('PDOC00001')
+        record = Prodoc.read(handle)
+        handle.close()
+        self.assertEqual(record.accession, 'PDOC00001')
+
+    def test_prosite_html(self):
+        handle = ExPASy.get_prosite_entry('PS00001')
+        html = handle.read()
+        self.assertEqual(handle.url,
+                         'http://prosite.expasy.org/cgi-bin/prosite/get-prosite-entry?PS00001')
+        handle.close()
+        self.assertTrue('<title>PROSITE: PS00001</title>' in html)
+
+    def test_prodoc_html(self):
+        handle = ExPASy.get_prodoc_entry('PDOC00001')
+        html = handle.read()
+        self.assertEqual(handle.url,
+                         'http://prosite.expasy.org/cgi-bin/prosite/get-prodoc-entry?PDOC00001')
+        handle.close()
+        self.assertTrue('{PS00001; ASN_GLYCOSYLATION}' in html)
+
+
+if __name__ == "__main__":
+    runner = unittest.TextTestRunner(verbosity=2)
+    unittest.main(testRunner=runner)