Repository: AlexsLemonade/refinebio

Issue:
Record which samples end up making it into a compendia
### Context

As we build compendia we drop some samples that aren't suitable for various reasons. However the only record in our database that tracks what samples are in it is the dataset object which has every sample including those that get dropped.

### Problem or idea

We should track this... _somehow_

### Solution or next step

We should already have this information in the compendia's data because during the first pass over the samples we keep track of the gene ids, the microarray samples, and the rnaseq samples. It's just a matter of actually getting this into the database.


Discussion:
If we want this to be for users of the compendium but not the front end we could include one or more files that list the samples dropped and the stage at which they were dropped.
> If we want this to be for users of the compendium but not the front end we could include one or more files that list the samples dropped and the stage at which they were dropped.

Ooooh I like that! We should do that! That'll require keeping track of what samples we drop at each step that can drop samples.

But also, we should look at the matrix we create rather than the dataset in places like 
https://github.com/AlexsLemonade/refinebio/blob/dev/workers/data_refinery_workers/processors/smashing_utils.py#L884-L899 and https://github.com/AlexsLemonade/refinebio/blob/dev/workers/data_refinery_workers/processors/smashing_utils.py#L661-L666
I went over the code and found 4 places where we drop samples, @kurtwheeler let me know if I'm missing any:

- on `_prepare_input` we drop any samples without a smashable file or if we have an error downloading the file from S3

- on `_prepare_frames` we do two passes over all `computed_files, samples` and drop all the samples where we can't load the data from the computed file.

- on `_perform_imputation` we drop all the samples "with <50% present values"

Do we want to include the samples that are dropped on all of these steps? The ones that are dropped in the imputation step seem more important.

Also, since we have multiple reasons why samples can be dropped, how should we write this information into the metadata. Instead of using separate files I'm thinking we could also add an additional property to the metadata on the sample details.
I discussed this with @kurtwheeler and we came up with several options:

1. Only include the samples that made into the compendia in `aggregated_metadata.json`, and then have a different file (eg `missing_samples_metadata.json`) with the samples that didn't make it and the reason why. Something like:
    ```
    {
    "GSM1232": {
        "reason": "We failed while downloading the computed file",
    },
    "sample_accession_code": {
        "reason": "Sample had <50% present values"
    },
    [...]
    }
    ```
    We could also add all sample metadata there, and the experiment relations.

2. We can include **all** samples in `aggregated_metadata.json` and still keep a different file with the reasons why they didn't make it into the compendia (eg `missing_samples_metadata.json`)

3. We can include two new fields for each sample in the current `aggregated_metadata.json`: a boolean value specifying whether the sample is in the compendia, and a `reason` for the ones that didn't make it.

Which one do you think is better? @jaclyn-taroni @cgreene 



I like Option 1 with the term "filtered" instead of "missing", I think.
Agree with @cgreene 

Resolving patch:
diff --git a/workers/data_refinery_workers/processors/create_compendia.py b/workers/data_refinery_workers/processors/create_compendia.py
--- a/workers/data_refinery_workers/processors/create_compendia.py
+++ b/workers/data_refinery_workers/processors/create_compendia.py
@@ -20,7 +20,8 @@
                                          CompendiumResultOrganismAssociation,
                                          ComputedFile,
                                          Organism,
-                                         Pipeline)
+                                         Pipeline,
+                                         Sample)
 from data_refinery_common.utils import get_env_variable
 from data_refinery_workers.processors import smashing_utils, utils
 
@@ -59,6 +60,10 @@ def _prepare_input(job_context: Dict) -> Dict:
     all_samples = list(itertools.chain(*job_context["samples"].values()))
     job_context["samples"] = {job_context["primary_organism"]: all_samples}
 
+    # We'll store here all sample accession codes that didn't make it into the compendia
+    # with the reason why not.
+    job_context['filtered_samples'] = {}
+
     job_context = smashing_utils.prepare_files(job_context)
 
     # Compendia jobs only run for one organism, so we know the only
@@ -231,8 +236,8 @@ def _perform_imputation(job_context: Dict) -> Dict:
     # Remove genes (rows) with <=70% present values in combined_matrix
     thresh = combined_matrix.shape[1] * .7  # (Rows, Columns)
     # Everything below `thresh` is dropped
-    row_filtered_matrix = combined_matrix.dropna(axis='index',
-                                                 thresh=thresh)
+    row_filtered_matrix = combined_matrix.dropna(axis='index', thresh=thresh)
+
     del combined_matrix
     del thresh
 
@@ -254,6 +259,16 @@ def _perform_imputation(job_context: Dict) -> Dict:
     log_state("end drop NA genes", job_context["job"].id, drop_na_samples_start)
     replace_zeroes_start = log_state("start replace zeroes", job_context["job"].id)
 
+    for sample_accession_code in row_filtered_matrix.columns:
+        if sample_accession_code not in row_col_filtered_matrix_samples_columns:
+            sample = Sample.objects.get(accession_code=sample_accession_code)
+            sample_metadata = sample.to_metadata_dict()
+            job_context['filtered_samples'][sample_accession_code] = {
+                **sample_metadata,
+                'reason': 'Sample was dropped because it had less than 50% present values.',
+                'experiment_accession_code': smashing_utils.get_experiment_accession(sample.accession_code, job_context['dataset'].data)
+            }
+
     del row_filtered_matrix
 
     # # Visualize Row and Column Filtered
diff --git a/workers/data_refinery_workers/processors/create_quantpendia.py b/workers/data_refinery_workers/processors/create_quantpendia.py
--- a/workers/data_refinery_workers/processors/create_quantpendia.py
+++ b/workers/data_refinery_workers/processors/create_quantpendia.py
@@ -41,6 +41,7 @@ def create_quantpendia(job_id: int) -> None:
 
 @utils.cache_keys('time_start', 'num_samples', 'time_end', 'formatted_command', work_dir_key='job_dir')
 def _download_files(job_context: Dict) -> Dict:
+    job_context['filtered_samples'] = {}
     job_context['time_start'] = timezone.now()
 
     num_samples = 0
diff --git a/workers/data_refinery_workers/processors/smashing_utils.py b/workers/data_refinery_workers/processors/smashing_utils.py
--- a/workers/data_refinery_workers/processors/smashing_utils.py
+++ b/workers/data_refinery_workers/processors/smashing_utils.py
@@ -63,7 +63,9 @@ def prepare_files(job_context: Dict) -> Dict:
     """
     start_prepare_files = log_state("start prepare files", job_context["job"].id)
     found_files = False
+    job_context['filtered_samples'] = {}
     job_context['input_files'] = {}
+
     # `key` can either be the species name or experiment accession.
     for key, samples in job_context["samples"].items():
         smashable_files = []
@@ -74,6 +76,13 @@ def prepare_files(job_context: Dict) -> Dict:
                 smashable_files = smashable_files + [(smashable_file, sample)]
                 seen_files.add(smashable_file)
                 found_files = True
+            else:
+                sample_metadata = sample.to_metadata_dict()
+                job_context['filtered_samples'][sample.accession_code] = {
+                    **sample_metadata,
+                    'reason': 'This sample did not have a processed file associated with it in our database.',
+                    'experiment_accession_code': get_experiment_accession(sample.accession_code, job_context['dataset'].data)
+                }
 
         job_context['input_files'][key] = smashable_files
 
@@ -304,6 +313,14 @@ def process_frames_for_key(key: str,
                                computed_file=computed_file.id,
                                dataset_id=job_context['dataset'].id,
                                job_id=job_context["job"].id)
+
+                sample_metadata = sample.to_metadata_dict()
+                job_context['filtered_samples'][sample.accession_code] = {
+                    **sample_metadata,
+                    'reason': 'The file associated with this sample did not pass the QC checks we apply before aggregating.',
+                    'filename': computed_file.filename,
+                    'experiment_accession_code': get_experiment_accession(sample.accession_code, job_context['dataset'].data)
+                }
                 continue
 
             # Count how many frames are in each tech so we can preallocate
@@ -375,9 +392,16 @@ def process_frames_for_key(key: str,
 
         if frame_data is None:
             job_context['unsmashable_files'].append(computed_file.filename)
+            sample_metadata = sample.to_metadata_dict()
+            job_context['filtered_samples'][sample.accession_code] = {
+                **sample_metadata,
+                'reason': 'The file associated with this sample did not contain a vector that fit the expected dimensions of the matrix.',
+                'filename': computed_file.filename,
+                'experiment_accession_code': get_experiment_accession(sample.accession_code, job_context['dataset'].data)
+            }
             continue
-        else:
-            frame_data = frame_data.reindex(all_gene_identifiers)
+
+        frame_data = frame_data.reindex(all_gene_identifiers)
 
         # The dataframe for each sample will only have one column
         # whose header will be the accession code.
@@ -613,15 +637,25 @@ def compile_metadata(job_context: Dict) -> Dict:
         metadata['ks_warning'] = job_context.get("ks_warning", None)
         metadata['quantile_normalized'] = job_context['dataset'].quantile_normalize
 
+    filtered_samples = job_context['filtered_samples']
+
     samples = {}
     for sample in job_context["dataset"].get_samples():
+        if sample.accession_code in filtered_samples:
+            # skip the samples that were filtered
+            continue
         samples[sample.accession_code] = sample.to_metadata_dict()
 
     metadata['samples'] = samples
 
     experiments = {}
     for experiment in job_context["dataset"].get_experiments():
-        experiments[experiment.accession_code] = experiment.to_metadata_dict()
+        experiment_metadata = experiment.to_metadata_dict()
+        # exclude filtered samples from experiment metadata
+        all_samples = experiment_metadata['sample_accession_codes']
+        all_samples = [code for code in all_samples if code not in filtered_samples]
+        experiment_metadata['sample_accession_codes'] = all_samples
+        experiments[experiment.accession_code] = experiment_metadata
 
     metadata['experiments'] = experiments
 
@@ -643,18 +677,32 @@ def write_non_data_files(job_context: Dict) -> Dict:
 
     # Write samples metadata to TSV
     try:
-        tsv_paths = write_tsv_json(job_context)
-        job_context['metadata_tsv_paths'] = tsv_paths
+        write_tsv_json(job_context)
         # Metadata to JSON
         job_context['metadata']['created_at'] = timezone.now().strftime('%Y-%m-%dT%H:%M:%S')
-        with open(job_context["output_dir"] + 'aggregated_metadata.json',
-                  'w',
-                  encoding='utf-8') as metadata_file:
+        aggregated_metadata_path = os.path.join(job_context["output_dir"],
+                                                'aggregated_metadata.json')
+        with open(aggregated_metadata_path, 'w', encoding='utf-8') as metadata_file:
             json.dump(job_context['metadata'], metadata_file, indent=4, sort_keys=True)
+
+        if job_context['filtered_samples']:
+            # generate filtered samples file only if some samples were skipped
+            filtered_samples_path = os.path.join(job_context["output_dir"],
+                                                 'filtered_samples_metadata.json')
+            with open(filtered_samples_path, 'w',encoding='utf-8') as metadata_file:
+                json.dump(job_context['filtered_samples'], metadata_file, indent=4, sort_keys=True)
+
+            columns = get_tsv_columns(job_context['filtered_samples'])
+            filtered_samples_tsv_path = os.path.join(job_context["output_dir"],
+                                                     'filtered_samples_metadata.tsv')
+            with open(filtered_samples_tsv_path, 'w', encoding='utf-8') as tsv_file:
+                dw = csv.DictWriter(tsv_file, columns, delimiter='\t', extrasaction='ignore')
+                dw.writeheader()
+                for sample_metadata in job_context['filtered_samples'].values():
+                    dw.writerow(get_tsv_row_data(sample_metadata, job_context["dataset"].data))
+
     except Exception as e:
-        logger.exception("Failed to write metadata TSV!",
-                         job_id=job_context['job'].id)
-        job_context['metadata_tsv_paths'] = None
+        logger.exception("Failed to write metadata TSV!", job_id=job_context['job'].id)
 
     return job_context
 
@@ -769,14 +817,13 @@ def get_tsv_row_data(sample_metadata, dataset_data):
     return row_data
 
 
-def get_tsv_columns(job_context, samples_metadata):
+def get_tsv_columns(samples_metadata):
     """Returns an array of strings that will be written as a TSV file's
     header. The columns are based on fields found in samples_metadata.
 
     Some nested annotation fields are taken out as separate columns
     because they are more important than the others.
     """
-    tsv_start = log_state("start get tsv columns", job_context["job"].id)
     refinebio_columns = set()
     annotation_columns = set()
     for sample_metadata in samples_metadata.values():
@@ -822,7 +869,6 @@ def get_tsv_columns(job_context, samples_metadata):
     # always first, followed by the other refinebio columns (in alphabetic order), and
     # annotation columns (in alphabetic order) at the end.
     refinebio_columns.discard('refinebio_accession_code')
-    log_state("end get tsv columns", job_context["job"].id, tsv_start)
     return ['refinebio_accession_code', 'experiment_accession'] + sorted(refinebio_columns) \
         + sorted(annotation_columns)
 
@@ -836,7 +882,7 @@ def write_tsv_json(job_context):
     metadata = job_context['metadata']
 
     # Uniform TSV header per dataset
-    columns = get_tsv_columns(job_context, metadata['samples'])
+    columns = get_tsv_columns(metadata['samples'])
 
     # Per-Experiment Metadata
     if job_context["dataset"].aggregate_by == "EXPERIMENT":


Test patch:
diff --git a/workers/data_refinery_workers/processors/test_compendia.py b/workers/data_refinery_workers/processors/test_compendia.py
--- a/workers/data_refinery_workers/processors/test_compendia.py
+++ b/workers/data_refinery_workers/processors/test_compendia.py
@@ -77,6 +77,37 @@ def test_create_compendia(self):
         assoc.computed_file = computed_file
         assoc.save()
 
+        # Missing sample that will be filtered
+        sample = Sample()
+        sample.accession_code = 'GSM1487222'
+        sample.title = 'this sample will be filtered'
+        sample.organism = gallus_gallus
+        sample.technology="MICROARRAY"
+        sample.save()
+
+        sra = SampleResultAssociation()
+        sra.sample = sample
+        sra.result = result
+        sra.save()
+
+        esa = ExperimentSampleAssociation()
+        esa.experiment = experiment
+        esa.sample = sample
+        esa.save()
+
+        computed_file = ComputedFile()
+        computed_file.filename = "GSM1487222_empty.PCL"
+        computed_file.absolute_file_path = "/home/user/data_store/PCL/doesnt_exists.PCL"
+        computed_file.result = result
+        computed_file.size_in_bytes = 123
+        computed_file.is_smashable = True
+        computed_file.save()
+
+        assoc = SampleComputedFileAssociation()
+        assoc.sample = sample
+        assoc.computed_file = computed_file
+        assoc.save()
+
         # RNASEQ TECH
         experiment2 = Experiment()
         experiment2.accession_code = "SRS332914"
@@ -116,7 +147,7 @@ def test_create_compendia(self):
         assoc2.save()
 
         dset = Dataset()
-        dset.data = {'GSE1487313': ['GSM1487313'], 'SRX332914': ['SRS332914']}
+        dset.data = {'GSE1487313': ['GSM1487313', 'GSM1487222'], 'SRX332914': ['SRS332914']}
         dset.scale_by = 'NONE'
         dset.aggregate_by = 'SPECIES'
         dset.svd_algorithm = 'ARPACK'
@@ -128,11 +159,14 @@ def test_create_compendia(self):
         pjda.dataset = dset
         pjda.save()
 
-#        with self.assertRaises(ValueError):
         final_context = create_compendia.create_compendia(job.id)
 
         self.assertFalse(job.success)
 
+        # check that sample with no computed file was skipped
+        self.assertTrue('GSM1487222' in final_context['filtered_samples'])
+        self.assertEqual(final_context['filtered_samples']['GSM1487222']['experiment_accession_code'], 'GSE1487313')
+
 
     @tag('compendia')
     def test_create_compendia_danio(self):
@@ -252,6 +286,26 @@ def test_create_compendia_danio(self):
 
             rnas.append(file)
 
+        # Missing sample that will be filtered
+        sample = Sample()
+        sample.accession_code = 'GSM1487222'
+        sample.title = 'this sample will be filtered'
+        sample.organism = danio_rerio
+        sample.technology="RNASEQ"
+        sample.save()
+
+        sra = SampleResultAssociation()
+        sra.sample = sample
+        sra.result = result
+        sra.save()
+
+        esa = ExperimentSampleAssociation()
+        esa.experiment = experiment
+        esa.sample = sample
+        esa.save()
+
+        rnas.append(sample.accession_code)
+
         dset = Dataset()
         dset.data = {'GSE1234': micros, 'GSE5678': rnas}
         dset.scale_by = 'NONE'
@@ -280,6 +334,9 @@ def test_create_compendia_danio(self):
                 'DANIO_RERIO')
         self.assertEqual(final_context['compendium_result'].organisms.count(), 1)
 
+        # check that sample with no computed file was skipped
+        self.assertTrue('GSM1487222' in final_context['filtered_samples'])
+        self.assertEqual(final_context['filtered_samples']['GSM1487222']['experiment_accession_code'], 'GSE5678')
 
         # It's maybe not worth asserting this until we're sure the behavior is correct
         # self.assertEqual(final_context['merged_qn'].shape, (9045, 830))
diff --git a/workers/data_refinery_workers/processors/test_smasher.py b/workers/data_refinery_workers/processors/test_smasher.py
--- a/workers/data_refinery_workers/processors/test_smasher.py
+++ b/workers/data_refinery_workers/processors/test_smasher.py
@@ -1289,7 +1289,7 @@ def test_columns(self):
             'job': pj
         }
 
-        columns = smashing_utils.get_tsv_columns(job_context, self.metadata['samples'])
+        columns = smashing_utils.get_tsv_columns(self.metadata['samples'])
         self.assertEqual(len(columns), 22)
         self.assertEqual(columns[0], 'refinebio_accession_code')
         self.assertTrue('refinebio_accession_code' in columns)