Repository: biopython/biopython

Issue:
SCOP search URL 404
Noted in passing on #2507,

```
$ python test_SCOP_online.py
test_search (__main__.ScopSearch)
Test search. ... ERROR

======================================================================
ERROR: test_search (__main__.ScopSearch)
Test search.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_SCOP_online.py", line 21, in test_search
    handle = SCOP.search("1JOY")
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/Bio/SCOP/__init__.py", line 938, in search
    return _open(cgi, variables)
  File "/Users/xxx/miniconda3/lib/python3.7/site-packages/Bio/SCOP/__init__.py", line 956, in _open
    handle = urlopen(cgi)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/Users/xxx/miniconda3/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

----------------------------------------------------------------------
Ran 1 test in 0.078s

FAILED (errors=1)
```

The original search CGI URL does not work anymore, here:

http://scop.mrc-lmb.cam.ac.uk/scop/search.cgi?pdb=1JOY

Possible replacement:

http://scop.mrc-lmb.cam.ac.uk/search?t=txt;q=1JOY

However, it looks like the API we expose may also need to change:

https://github.com/biopython/biopython/blob/biopython-176/Bio/SCOP/__init__.py#L911





Discussion:
@peterjc, it looks like the legacy interface is still available under a new URL:
scop.mrc-lmb.cam.ac.uk/legacy/search.cgi?pdb=1JOY

In the short term, I will make a PR to update this URL, but it may still be good to move to the new one.
What are common usecases of this function, since it returns handle with html content? I've scoured github, but couldn't find anything except the usage inside this test.
Probably overdue for deprecation rather than updating if there is not good current use age:
https://biopython.org/wiki/Deprecation_policy
There is now a REST API that uses JSON. Perhaps we should use that now instead. As peterjc noted above, though, it might require changing the exposed API.
http://scop.mrc-lmb.cam.ac.uk/api/
Adding a new wrapper for the SCOP JSON based REST API sounds good in principle.

We could go ahead and deprecate the HTML returning legacy API wrapper too.
@peterjc Is there any value in scraping content from the HTML and returning it in a more usable data structure? The REST API does not provide as many features.
I don't know - I've never used SCOP.

This module has been without a direct owner for many years - the vast majority of changes have been code style/maintenance, deprecation cleanup, with very little in the way of bug fixes or extension:

https://github.com/biopython/biopython/commits/master/Bio/SCOP
@peterjc It looks like SCOP has mostly been replaced by SCOPe, which does not seem to have an API of any kind at the moment.
https://scop.berkeley.edu/
I have not used SCOP either, so I cannot speak to any possible use cases. 
Noting the online SCOP test are currently failing with a timeout, currently the entire scop.mrc-lmb.cam.ac.uk domain is unavailable. The site was up and running at of 16 April according to archive.org so I'll assume for now this is a temporary failure.


The old site now redirects to https://www.ebi.ac.uk/pdbe/scop/ and there doesn't seem to be an equivalent to the legacy search. There is a free text search, and the REST API though.

I will remove the now failing test and deprecate the search function etc.

Resolving patch:
diff --git a/Bio/SCOP/__init__.py b/Bio/SCOP/__init__.py
--- a/Bio/SCOP/__init__.py
+++ b/Bio/SCOP/__init__.py
@@ -41,13 +41,14 @@
 This module also provides code to access SCOP over the WWW.
 
 Functions:
- - search        -- Access the main CGI script.
+ - search        -- Access the main CGI script (DEPRECATED, no longer available)..
  - _open         -- Internally used function.
 
 """
 
 import os
 import re
+import warnings
 
 from urllib.parse import urlencode
 from urllib.request import urlopen
@@ -56,6 +57,7 @@
 from . import Cla
 from . import Hie
 from . import Residues
+from Bio import BiopythonDeprecationWarning
 from Bio import SeqIO
 from Bio.Seq import Seq
 
@@ -926,7 +928,7 @@ def search(
     cgi="http://scop.mrc-lmb.cam.ac.uk/legacy/search.cgi",
     **keywds,
 ):
-    """Access SCOP search and return a handle to the results.
+    """Access SCOP search and return a handle to the results (DEPRECATED).
 
     Access search.cgi and return a handle to the results.  See the
     online help file for an explanation of the parameters:
@@ -934,7 +936,15 @@ def search(
 
     Raises an IOError if there's a network error.
 
+    This function is now DEPRECATED and will be removed in a future
+    release of Biopython because this search.cgi API is no longer
+    available with SCOP now hosted the the EBI.
     """
+    warnings.warn(
+        "The SCOP search function is deprecated and will be removed"
+        " as the CGI based SCOP search is no longer availble online.",
+        BiopythonDeprecationWarning,
+    )
     params = {"pdb": pdb, "key": key, "sid": sid, "disp": disp, "dir": dir, "loc": loc}
     variables = {}
     for k, v in params.items():


Test patch:
diff --git a/Tests/test_SCOP_online.py b/Tests/test_SCOP_online.py
deleted file mode 100644
--- a/Tests/test_SCOP_online.py
+++ /dev/null
@@ -1,27 +0,0 @@
-# Copyright 2010 by Peter Cock.  All rights reserved.
-# This code is part of the Biopython distribution and governed by its
-# license.  Please see the LICENSE file that should have been included
-# as part of this package.
-
-"""Testing online code in Bio.SCOP module."""
-
-import unittest
-
-from Bio import SCOP
-
-import requires_internet
-
-requires_internet.check()
-
-
-class ScopSearch(unittest.TestCase):
-    """SCOP search tests."""
-
-    def test_search(self):
-        """Test search."""
-        handle = SCOP.search("1JOY")
-
-
-if __name__ == "__main__":
-    runner = unittest.TextTestRunner(verbosity=2)
-    unittest.main(testRunner=runner)