Repository: AlexsLemonade/refinebio

Issue:
Deal with Ensembl's latest API change
### Context

Ensembl's API has always been weirdly divided into divisions, most of which were at one API: https://rest.ensemblgenomes.org

However they also had a different URL with what I've called their "main division" (since it's not otherwise labeled): https://rest.ensembl.org

### Problem or idea

Ensembl changed their API and broke our transcriptome index code with this release: http://www.ensembl.info/2019/03/08/joint-rest-server-for-ensembl-and-ensembl-genomes-in-ensembl-96/

They completely broke https://rest.ensemblgenomes.org... even when requesting JSON the response is just plain text: "This REST server has been retired, please use rest.ensembl.org instead. For further information please see our blog post http://www.ensembl.info/2019/03/08/joint-rest-server-for-ensembl-and-ensembl-genomes-in-ensembl-96/" (WHICH BTW IS NOT THE WAY TO DEPRECATE AN API!!!!!!!!) It looks like basically we're supposed to get all the data from the other URL. I haven't read all the whole blog post through, but it looks like they at least give a very detailed description of what changed and how.

### Solution or next step

We have a bunch of logic to use the right endpoint based on the division but now it looks like we don't need to worry about that. We should just use the same URL for all organisms and parse out the metadata fields which are equivalent to the ones we're already storing.


Discussion:

Resolving patch:
diff --git a/foreman/data_refinery_foreman/surveyor/transcriptome_index.py b/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
--- a/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
+++ b/foreman/data_refinery_foreman/surveyor/transcriptome_index.py
@@ -17,13 +17,14 @@
 logger = get_and_configure_logger(__name__)
 
 
-DIVISION_URL_TEMPLATE = ("https://rest.ensemblgenomes.org/info/genomes/division/{division}"
+MAIN_DIVISION_URL_TEMPLATE = "https://rest.ensembl.org/info/species?content-type=application/json"
+DIVISION_URL_TEMPLATE = ("https://rest.ensembl.org/info/genomes/division/{division}"
                          "?content-type=application/json")
+
 TRANSCRIPTOME_URL_TEMPLATE = ("ftp://ftp.{url_root}/fasta/{species_sub_dir}/dna/"
                               "{filename_species}.{assembly}.dna.{schema_type}.fa.gz")
 GTF_URL_TEMPLATE = ("ftp://ftp.{url_root}/gtf/{species_sub_dir}/"
                     "{filename_species}.{assembly}.{assembly_version}.gtf.gz")
-MAIN_DIVISION_URL_TEMPLATE = "https://rest.ensembl.org/info/species?content-type=application/json"
 
 
 # For whatever reason the division in the download URL is shortened in
@@ -41,7 +42,7 @@
 # release versions. These urls will return what the most recent
 # release version is.
 MAIN_RELEASE_URL = "https://rest.ensembl.org/info/software?content-type=application/json"
-DIVISION_RELEASE_URL = "https://rest.ensemblgenomes.org/info/eg_version?content-type=application/json"
+DIVISION_RELEASE_URL = "https://rest.ensembl.org/info/eg_version?content-type=application/json"
 
 
 class EnsemblUrlBuilder(ABC):
@@ -59,20 +60,15 @@ def __init__(self, species: Dict):
         self.url_root = "ensemblgenomes.org/pub/release-{assembly_version}/{short_division}"
         self.short_division = DIVISION_LOOKUP[species["division"]]
         self.assembly = species["assembly_name"].replace(" ", "_")
-        self.assembly_version = utils.requests_retry_session().get(DIVISION_RELEASE_URL).json()["version"]
-
-        # Some species are nested within a collection directory. If
-        # this is the case, then we need to add that extra directory
-        # to the URL, and for whatever reason the filename is not
-        # capitalized.
-        COLLECTION_REGEX = r"^(.*_collection).*"
-        match_object = re.search(COLLECTION_REGEX, species["dbname"])
-        if match_object:
-            self.species_sub_dir = match_object.group(1) + "/" + species["species"]
-            self.filename_species = species["species"]
-        else:
-            self.species_sub_dir = species["species"]
-            self.filename_species = species["species"].capitalize()
+
+        # For some reason the API is returning version 44, which doesn't seem to exist
+        # in the FTP servers: ftp://ftp.ensemblgenomes.org/pub/
+        # That's why the version is hardcoded below.
+        # self.assembly_version = utils.requests_retry_session().get(DIVISION_RELEASE_URL).json()["version"]
+        self.assembly_version = '43'
+
+        self.species_sub_dir = species["name"]
+        self.filename_species = species["name"].capitalize()
 
         # These fields aren't needed for the URL, but they vary between
         # the two REST APIs.
@@ -127,7 +123,8 @@ def __init__(self, species: Dict):
         self.species_sub_dir = species["name"]
         self.filename_species = species["name"].capitalize()
         self.assembly = species["assembly"]
-        self.assembly_version = utils.requests_retry_session().get(MAIN_RELEASE_URL).json()["release"]
+        self.assembly_version = utils.requests_retry_session().get(
+            MAIN_RELEASE_URL).json()["release"]
         self.scientific_name = self.filename_species.replace("_", " ")
         self.taxonomy_id = species["taxon_id"]
 
@@ -205,7 +202,7 @@ def _generate_files(self, species: Dict) -> None:
         url_builder = ensembl_url_builder_factory(species)
         fasta_download_url = url_builder.build_transcriptome_url()
         gtf_download_url = url_builder.build_gtf_url()
-
+        
         platform_accession_code = species.pop("division")
         self._clean_metadata(species)
 


Test patch:
diff --git a/foreman/data_refinery_foreman/surveyor/test_transcriptome_index.py b/foreman/data_refinery_foreman/surveyor/test_transcriptome_index.py
--- a/foreman/data_refinery_foreman/surveyor/test_transcriptome_index.py
+++ b/foreman/data_refinery_foreman/surveyor/test_transcriptome_index.py
@@ -27,7 +27,6 @@ def setUp(self):
         key_value_pair.save()
 
     @patch('data_refinery_foreman.surveyor.external_source.message_queue.send_job')
-    @skip("We're doing a staging test to see if the new salmon version works.")
     def test_survey(self, mock_send_job):
         surveyor = TranscriptomeIndexSurveyor(self.survey_job)
         surveyor.survey(source_type="TRANSCRIPTOME_INDEX")
@@ -67,7 +66,6 @@ def test_correct_index_location(self):
         for file in files:
             urllib.request.urlopen(file.source_url)
 
-    @skip("We're doing a staging test to see if the new salmon version works.")
     def test_correct_index_location_metazoa(self):
         """ Tests that the files returned actually exist.