Repository: AlexsLemonade/refinebio

Issue:
Get SRA sample metadata from BioSample database
### Context

Yesterday, we were taking a look at some zebrafish crunch RNA-seq samples.

### Problem or idea

To quote from the MetaSRA publication ([Bernstein, et al. _Bioinformatics._ 2017.](
https://doi.org/10.1093/bioinformatics/btx334))

> Unfortunately, [large-scale] studies are currently difficult due to the poor structure of the metadata associated with samples in the SRA. These metadata are centrally stored at the BioSample database ([Barrett et al., 2012](https://doi.org/10.1093/nar/gkr1163)), with each SRA sequencing experiment referencing a sample-specific metadata record within this database. The BioSample database organizes each sample’s metadata as sets of key-value pairs where the key is a property of the sample and the value is a property-value of the sample. 

We are not currently extracting metadata from the BioSample database ([documentation](https://www.ncbi.nlm.nih.gov/biosample/docs/); unclear how useful that will be). I've also found this: https://catalog.data.gov/dataset/biosample/resource/61cb64a9-14c1-4216-8af9-4a4d68286ef1 which seems a bit worrisome.

### Solution or next step

We should circle back to this following the initial release. [MetaSRA](http://metasra.biostat.wisc.edu) uses [SRAdb](https://github.com/seandavi/SRAdb), so I'm linking those here for reference. 

EMBL-EBI also has a [BioSamples database](https://www.ebi.ac.uk/biosamples/) which seems like it contains ENA data and has [API documentation](https://www.ebi.ac.uk/biosamples/docs/references/api), so if that covers all of SRA that may be an avenue to explore.

### New Issue Checklist

- [x] The title is short and descriptive.
- [x] You have explained the context that led you to write this issue.
- [x] You have reported a problem or idea.
- [x] You have proposed a solution or next step.


Discussion:
### Examples of BioSample documentation vs what is in Refine.bio

Most RNA-seq datasets appear to only have sample titles. BioSample has more information. If we want this data to be readily useable by the users, we really should try to get the BioSample metadata for RNA-seq samples. 

- Example: SRP066834
[refinebio entry](https://www.refine.bio/experiments/SRP066834/human-cerebral-organoids-recapitulate-gene-expression-programs-of-fetal-neocortex-development)
Sometimes sample titles in refine.bio will technically contain the meta information but in such an abbreviated fashion that it isn't clear what it means: eg `whOrg1_B12_hOrg33d_c2` and `fetal1_A10_fetal_12wpc_c1`
In `fetal1_A10_fetal_12wpc_c1` the `wpc` appears to be abbreviated for `weeks post conception` as said in it's [biosample entry](https://www.ncbi.nlm.nih.gov/biosample/SAMN04304316)

- Example: SRP071643
This example also tries to have all the info in the title but abbreviated. But this doesn't cover it as well:
[refinebio entry](https://www.refine.bio/experiments/SRP071643/sc3-consensus-clustering-of-single-cell-rna-seq-data)
[biosample entry](https://www.ncbi.nlm.nih.gov/biosample/SAMN06468864)

Ok so I think it would make sense to get the biosample data and associate it to the samples. We have run accessions for all of our samples. We can use https://www.ebi.ac.uk/ena/data/view/SRR3219272&display=xml to get a sample accession (SRS, ERS, DRS).

This SRA sample accession can then be used to hit https://www.ebi.ac.uk/ena/data/view/SRS1336128&display=xml/. That xml contains a biosample accession (SAMN04546047) which can then be used to hit https://www.ebi.ac.uk/biosamples/docs/references/api/search#_get_sample_by_accession.

We can then associate that metadata with each of the (refinebio) samples that belong to it. This is a bit confusing because our data model only has experiments and samples, where the bioproject/SRA data models have studies, experiments, samples, and runs. From a metadata perspective I think any bioproject/SRA study/experiment metadata should be associated with our Experiment table, while any bioproject/SRA samples/runs metadata should be associated with our Sample table.
After doing some digging around, it looks like we already were capable of getting SRA metadata, but a bug prevented us from doing so.

This is what the old code looked like for applying harmonized metadata to a sample:
```python
        for key, value in harmonized_sample.items():
            setattr(sample, key, value)
```

This code doesn't work, because the output  from `harmony.harmonize` actually looks like this: `{'STZ-3_RNA-Seq': {'disease': 'stz-induced diabetes', 'disease_stage': 'stz-induced diabetes', 'cell_line': 'sprague-dawley', 'specimen_part': 'kidney inner medulla', 'genetic_information': 'sprague-dawley', 'subject': 'stz-3, kidney inner medulla'}}`. It's a nested `dict`, indexed by title then by harmonized fields.

I changed the above code to this, and now I can see sample metadata for locally-surveyed SRA samples:

```python
        for key, value in harmonized_sample[sample.title].items():
            setattr(sample, key, value)

```

For reference, here is the (abbreviated) data from my local API versus the `refine.bio` server:

Local:
```json
{
            "title": "STZ-3_RNA-Seq",
            "accession_code": "SRR2050231",
            "source_database": "SRA",
            "organism": {
                "name": "RATTUS_NORVEGICUS",
                "taxonomy_id": 10116
            },
            "has_raw": true,
            "sex": "",
            "age": null,
            "specimen_part": "kidney inner medulla",
            "genotype": "",
            "disease": "stz-induced diabetes",
            "disease_stage": "stz-induced diabetes",
            "cell_line": "sprague-dawley",
            "treatment": "",
            "race": "",
            "subject": "stz-3, kidney inner medulla",
            "compound": "",
            "time": "",
        }
```

Production:
```json
{
            "title": "STZ-3_RNA-Seq",
            "accession_code": "SRR2050231",
            "source_database": "SRA",
            "organism": {
                "name": "RATTUS_NORVEGICUS",
                "taxonomy_id": 10116
            },
            "has_raw": true,
            "sex": "",
            "age": null,
            "specimen_part": "",
            "genotype": "",
            "disease": "",
            "disease_stage": "",
            "cell_line": "",
            "treatment": "",
            "race": "",
            "subject": "",
            "compound": "",
            "time": "",
}
```

It looks like our missing metadata also more-or-less matches what we would be getting from harmonizing BioSample metadata:

```json
"description title" : [ {
      "text" : "STZ-3_RNA-Seq"
    } ],
    "disease" : [ {
      "text" : "STZ-induced diabetes"
    } ],
    "organism" : [ {
      "text" : "Rattus norvegicus",
      "ontologyTerms" : [ "http://purl.obolibrary.org/obo/NCBITaxon_10116" ]
    } ],
    "source name" : [ {
      "text" : "STZ-3, kidney inner medulla"
    } ],
    "strain" : [ {
      "text" : "Sprague-Dawley",
      "ontologyTerms" : [ "http://www.ebi.ac.uk/efo/EFO_0001352" ]
    } ],
    "tissue" : [ {
      "text" : "kidney inner medulla"
    } ]
```

Resolving patch:
diff --git a/foreman/data_refinery_foreman/foreman/management/commands/update_metadata.py b/foreman/data_refinery_foreman/foreman/management/commands/update_metadata.py
new file mode 100644
--- /dev/null
+++ b/foreman/data_refinery_foreman/foreman/management/commands/update_metadata.py
@@ -0,0 +1,65 @@
+from django.core.management.base import BaseCommand
+import GEOparse
+import sys
+
+from data_refinery_common.models import Sample
+from data_refinery_common.logging import get_and_configure_logger
+from data_refinery_foreman.surveyor import harmony
+from data_refinery_foreman.surveyor.sra import SraSurveyor
+from data_refinery_foreman.surveyor.geo import GeoSurveyor
+from data_refinery_foreman.surveyor.array_express import ArrayExpressSurveyor
+
+logger = get_and_configure_logger(__name__)
+
+
+class Command(BaseCommand):
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "--source-database",
+            type=str,
+            help=("The name of a source database, such as ARRAY_EXPRESS, GEO, or SRA."
+                  "All samples from this source database will have their metadata refreshed."))
+
+    def handle(self, *args, **options):
+        """Refreshes the metadata for RNA-Seq samples
+        """
+        possible_source_databases = ["ARRAY_EXPRESS", "GEO", "SRA"]
+
+        if options["source_database"] is None:
+            samples = Sample.objects.all()
+        elif options["source_database"] in possible_source_databases:
+            source_database = options["source_database"]
+            samples = Sample.objects.filter(source_database=source_database)
+        else:
+            logger.error("Invalid source database \"{}\"".format(options["source_database"]) +
+                         "\nPossible source databases: {}".format(", ".join(possible_source_databases)))
+            sys.exit(1)
+
+        for sample in samples:
+            logger.debug("Refreshing metadata for a sample.",
+                         sample=sample.accession_code)
+            if sample.source_database == "SRA":
+                metadata = SraSurveyor.gather_all_metadata(
+                    sample.accession_code)
+                SraSurveyor._apply_harmonized_metadata_to_sample(
+                    sample, metadata)
+            elif sample.source_database == "GEO":
+                gse = GEOparse.get_GEO(
+                    sample.experiments.first().accession_code,
+                    destdir="/tmp/management",
+                    how="brief",
+                    silent=True)
+                preprocessed_samples = harmony.preprocess_geo(gse.gsms.items())
+                harmonized_samples = harmony.harmonize(preprocessed_samples)
+                GeoSurveyor._apply_harmonized_metadata_to_sample(
+                    sample, harmonized_samples[sample.title])
+            elif sample.source_database == "ARRAY_EXPRESS":
+                SDRF_URL_TEMPLATE = "https://www.ebi.ac.uk/arrayexpress/files/{code}/{code}.sdrf.txt"
+                sdrf_url = SDRF_URL_TEMPLATE.format(
+                    code=sample.experiments.first().accession_code)
+                sdrf_samples = harmony.parse_sdrf(sdrf_url)
+                harmonized_samples = harmony.harmonize(sdrf_samples)
+                ArrayExpressSurveyor._apply_harmonized_metadata_to_sample(
+                    sample, harmonized_samples[sample.title])
+
+            sample.save()
diff --git a/foreman/data_refinery_foreman/surveyor/array_express.py b/foreman/data_refinery_foreman/surveyor/array_express.py
--- a/foreman/data_refinery_foreman/surveyor/array_express.py
+++ b/foreman/data_refinery_foreman/surveyor/array_express.py
@@ -356,6 +356,12 @@ def update_sample_protocol_info(existing_protocols, experiment_protocol, protoco
 
         return (existing_protocols, is_updated)
 
+    @staticmethod
+    def _apply_harmonized_metadata_to_sample(sample: Sample, harmonized_metadata: dict):
+        """Applies the harmonized metadata to `sample`"""
+        for key, value in harmonized_metadata.items():
+            setattr(sample, key, value)
+
     def create_samples_from_api(self,
                                 experiment: Experiment,
                                 platform_dict: Dict
@@ -378,7 +384,6 @@ def create_samples_from_api(self,
                 Download the derived data and no-op it.
 
         See an example at: https://www.ebi.ac.uk/arrayexpress/json/v3/experiments/E-MTAB-3050/samples
-
         """
 
         created_samples = []
@@ -559,9 +564,7 @@ def create_samples_from_api(self,
 
                 # Directly assign the harmonized properties
                 harmonized_sample = harmonized_samples[title]
-                for key, value in harmonized_sample.items():
-                    setattr(sample_object, key, value)
-                sample_object.save()
+                ArrayExpressSurveyor._apply_harmonized_metadata_to_sample(sample_object, harmonized_sample)
 
                 sample_annotation = SampleAnnotation()
                 sample_annotation.data = sample_data
diff --git a/foreman/data_refinery_foreman/surveyor/geo.py b/foreman/data_refinery_foreman/surveyor/geo.py
--- a/foreman/data_refinery_foreman/surveyor/geo.py
+++ b/foreman/data_refinery_foreman/surveyor/geo.py
@@ -213,6 +213,12 @@ def get_sample_protocol_info(sample_metadata, sample_accession_code):
                                               + sample_accession_code)
         return protocol_info
 
+    @staticmethod
+    def _apply_harmonized_metadata_to_sample(sample: Sample, harmonized_metadata: dict):
+        """Applies the harmonized metadata to `sample`"""
+        for key, value in harmonized_metadata.items():
+            setattr(sample, key, value)
+
     def create_experiment_and_samples_from_api(self, experiment_accession_code) -> (Experiment, List[Sample]):
         """ The main surveyor - find the Experiment and Samples from NCBI GEO.
 
@@ -303,10 +309,7 @@ def create_experiment_and_samples_from_api(self, experiment_accession_code) -> (
 
                 self.set_platform_properties(sample_object, sample.metadata, gse)
 
-                # Directly assign the harmonized properties
-                harmonized_sample = harmonized_samples[sample_object.title]
-                for key, value in harmonized_sample.items():
-                    setattr(sample_object, key, value)
+                GeoSurveyor._apply_harmonized_metadata_to_sample(sample_object, harmonized_samples[sample_object.title])
 
                 # Sample-level protocol_info
                 sample_object.protocol_info = self.get_sample_protocol_info(sample.metadata,


Test patch: