internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: AlexsLemonade/refinebio

Issue:
[HOTFIX] Prevent compendia jobs from getting cleaned up and fix QN jobs
## Issue Number

#1728 
#1726 
#1727 


Discussion:

Resolving patch:
diff --git a/foreman/data_refinery_foreman/foreman/management/commands/create_compendia.py b/foreman/data_refinery_foreman/foreman/management/commands/create_compendia.py
--- a/foreman/data_refinery_foreman/foreman/management/commands/create_compendia.py
+++ b/foreman/data_refinery_foreman/foreman/management/commands/create_compendia.py
@@ -21,7 +21,8 @@ def create_job_for_organism(organism=Organism, quant_sf_only=False, svd_algorith
     experiments = Experiment.objects.filter(organisms=organism).prefetch_related('samples')
 
     for experiment in queryset_iterator(experiments):
-        data[experiment.accession_code] = list(experiment.samples.filter(organism=organism).values_list('accession_code', flat=True))
+        data[experiment.accession_code] = list(experiment.samples.filter(organism=organism)\
+            .values_list('accession_code', flat=True))
 
     job = ProcessorJob()
     job.pipeline_applied = ProcessorPipeline.CREATE_COMPENDIA.value
@@ -30,7 +31,8 @@ def create_job_for_organism(organism=Organism, quant_sf_only=False, svd_algorith
     dset = Dataset()
     dset.data = data
     dset.scale_by = 'NONE'
-    dset.aggregate_by = 'SPECIES'
+    # The quantpendias should be aggregated by species
+    dset.aggregate_by = 'EXPERIMENT' if quant_sf_only else 'SPECIES'
     dset.quantile_normalize = False
     dset.quant_sf_only = quant_sf_only
     dset.svd_algorithm = svd_algorithm
@@ -55,7 +57,7 @@ def add_arguments(self, parser):
         parser.add_argument(
             "--quant-sf-only",
             type=lambda x: x == "True",
-            help=("Whether to create a quantpendium or normal compendium."))
+            help=("Whether to create a quantpendium or normal compendium. Quantpendium will be aggregated by EXPERIMENT"))
 
         parser.add_argument(
             "--svd-algorithm",
@@ -78,19 +80,16 @@ def handle(self, *args, **options):
         # I think we could just use options["quant_sf_only"] but I
         # wanna make sure that values that are not True do not trigger
         # a truthy evaluation.
-        quant_sf_only = False
-        if options["quant_sf_only"] is True:
-            quant_sf_only = True
+        quant_sf_only = options["quant_sf_only"] is True
 
         # default algorithm to arpack until we decide that ranomized is preferred
         svd_algorithm = 'NONE' if quant_sf_only else 'ARPACK'
         if options["svd_algorithm"] in ['ARPACK', 'RANDOMIZED', 'NONE']:
             svd_algorithm = options["svd_algorithm"]
 
-        logger.debug(all_organisms)
+        logger.debug('Generating compendia for organisms', organisms=all_organisms)
 
         for organism in all_organisms:
-            logger.debug(organism)
             job = create_job_for_organism(organism, quant_sf_only, svd_algorithm)
             logger.info("Sending CREATE_COMPENDIA for Organism", job_id=str(job.pk), organism=str(organism))
             send_job(ProcessorPipeline.CREATE_COMPENDIA, job)


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
API Stats include timestamp of last broken cache
### Context

Our database has a 10 minute cache. 

### Problem or idea

It'd be nifty to be able to look at the stats endpoint response and figure out when it was last busted.

### Solution or next step

Include that timestamp.


Discussion:

Resolving patch:
diff --git a/api/data_refinery_api/views.py b/api/data_refinery_api/views.py
--- a/api/data_refinery_api/views.py
+++ b/api/data_refinery_api/views.py
@@ -922,6 +922,7 @@ def get(self, request, version, format=None):
     @classmethod
     def calculate_stats(cls, range_param):
         data = {}
+        data['generated_on'] = timezone.now()
         data['survey_jobs'] = cls._get_job_stats(SurveyJob.objects, range_param)
         data['downloader_jobs'] = cls._get_job_stats(DownloaderJob.objects, range_param)
         data['processor_jobs'] = cls._get_job_stats(ProcessorJob.objects, range_param)


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Improve docs and make deploy.sh easier to invoke correctly
## Issue Number

I don't think there's one that's quite specific to the types of documentation improvements this introduces.

## Purpose/Implementation Notes

Our README was getting behind. This brings it up to date and makes the deploy process easier to follow.

## Types of changes

- New feature (non-breaking change which adds functionality)
- Documentation

## Functional tests

None yet. This should be tested by spinning up a stack and destroying it.

## Checklist

- [x] Lint and unit tests pass locally with my changes
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] I have added necessary documentation (if appropriate)
- [x] Any dependent changes have been merged and published in downstream modules



Discussion:

Resolving patch:
diff --git a/api/data_refinery_api/views.py b/api/data_refinery_api/views.py
--- a/api/data_refinery_api/views.py
+++ b/api/data_refinery_api/views.py
@@ -675,7 +675,8 @@ def get_queryset(self):
         # case insensitive search https://docs.djangoproject.com/en/2.1/ref/models/querysets/#icontains
         filter_by = self.request.query_params.get('filter_by', None)
         if filter_by:
-            queryset = queryset.filter(Q(title__icontains=filter_by) |
+            queryset = queryset.filter(Q(accession_code__icontains=filter_by) |
+                                       Q(title__icontains=filter_by) |
                                        Q(sex__icontains=filter_by) |
                                        Q(age__icontains=filter_by) |
                                        Q(specimen_part__icontains=filter_by) |


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Quantpendia failed to upload to S3
### Context

We kicked off quantpendia jobs for all organisms but they weren't succeeding because they couldn't upload to S3.

### Problem or idea

This is probably just because the worker instances don't have access to the compendia S3 bucket. The smasher probably has those permissions, but it looks like the workers don't.

### Solution or next step

Give worker instances permissions to push to the compendia S3 bucket.


Discussion:

Resolving patch:
diff --git a/workers/data_refinery_workers/processors/create_quantpendia.py b/workers/data_refinery_workers/processors/create_quantpendia.py
--- a/workers/data_refinery_workers/processors/create_quantpendia.py
+++ b/workers/data_refinery_workers/processors/create_quantpendia.py
@@ -3,6 +3,7 @@
 import shutil
 import time
 from django.utils import timezone
+from django.conf import settings
 from typing import Dict, List, Tuple
 import psutil
 
@@ -114,7 +115,6 @@ def create_result_objects(job_context: Dict) -> Dict:
     archive_computed_file.sync_to_s3(S3_BUCKET_NAME, s3_key)
 
     job_context['result'] = result
-    job_context['computed_files'] = [archive_computed_file]
     job_context['success'] = True
 
     return job_context
@@ -123,7 +123,9 @@ def create_result_objects(job_context: Dict) -> Dict:
 def remove_job_dir(job_context: Dict):
     """ remove the directory when the job is successful. At this point
     the quantpendia was already zipped and uploaded. """
-    shutil.rmtree(job_context["job_dir"], ignore_errors=True)
+    # don't remove the files when running locally or for tests
+    if settings.RUNNING_IN_CLOUD:
+        shutil.rmtree(job_context["job_dir"], ignore_errors=True)
     return job_context
 
 def make_dirs(job_context: Dict):


Test patch:
diff --git a/workers/data_refinery_workers/processors/test_create_quantpendia.py b/workers/data_refinery_workers/processors/test_create_quantpendia.py
--- a/workers/data_refinery_workers/processors/test_create_quantpendia.py
+++ b/workers/data_refinery_workers/processors/test_create_quantpendia.py
@@ -96,3 +96,8 @@ def test_create_quantpendia(self):
         self.assertTrue(final_context['metadata']['quant_sf_only'])
         self.assertEqual(final_context['metadata']['num_samples'], 1)
         self.assertEqual(final_context['metadata']['num_experiments'], 1)
+
+        # test that archive exists
+        quantpendia_file = ComputedFile.objects.filter(is_compendia=True, quant_sf_only=True).latest()
+        self.assertTrue(os.path.exists(quantpendia_file.absolute_file_path))
+
Repository: AlexsLemonade/refinebio

Issue:
Extend processor exception classes to include additional arbitrary context
### Context

#1843 is dope. However right now those exceptions can only be used to log the job id. If we wanted to log a sample accession or what have you, we couldn't do it as it is.

### Problem or idea

We've created some new exception classes, and so we can add any properties/constructor parameters we want to them. I think it'd be helpful if we added a parameter that would store the additional context we want logged and then stored it on the ProcessorJobException object. Then where we handle those exceptions we can check that field and see if there's additional context we should be logging.

### Solution or next step

Implement what's above and use it in a few places. We can slowly transition our codebase to using this pattern more broadly over time once we have this implemented.


Discussion:

Resolving patch:
diff --git a/workers/data_refinery_workers/processors/utils.py b/workers/data_refinery_workers/processors/utils.py
--- a/workers/data_refinery_workers/processors/utils.py
+++ b/workers/data_refinery_workers/processors/utils.py
@@ -370,10 +370,11 @@ def run_pipeline(start_value: Dict, pipeline: List[Callable]):
             logger.info('Processor Job deleted itself.', reason=e.reason, processor_job=job_id)
             break
         except ProcessorJobError as e:
-            job.failure_reason = e.failure_reason
-            if e.success is not None:
-                job.succeess = e.success
-            logger.exception(e.failure_reason, processor_job=job.id)
+            e.update_job(job)
+            logger.exception(e.failure_reason, processor_job=job.id, **e.context)
+            if e.success is False:
+                # end_job will use this and set the value
+                last_result['success'] = False
             return end_job(last_result)
         except Exception as e:
             failure_reason = ("Unhandled exception caught while running processor"
@@ -400,11 +401,21 @@ def run_pipeline(start_value: Dict, pipeline: List[Callable]):
 
 class ProcessorJobError(Exception):
     """ General processor job error class. """
-    def __init__(self, failure_reason, *, success=None, original_exception=None):
+    def __init__(self, failure_reason, *, success=None, no_retry=None, **context):
         super(ProcessorJobError, self).__init__(failure_reason)
         self.failure_reason = failure_reason
         self.success = success
-        self.original_exception = original_exception
+        self.no_retry = no_retry
+        # additional context to be included when logging
+        self.context = context
+
+    def update_job(self, job):
+        job.failure_reason = self.failure_reason
+        if self.success is not None:
+            job.success = self.success
+        if self.no_retry is not None:
+            job.no_retry = self.no_retry
+        job.save()
 
 
 class ProcessorJobDeleteSelf(Exception):


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Compendium Missing Generated File
### Context

`RATTUS_NORVEGICUS/rattus_norvegicus.tsv` is missing.

<img width="832" alt="Screen Shot 2019-11-27 at 10 29 50 PM" src="https://user-images.githubusercontent.com/1075609/69775087-ae306e00-1165-11ea-9a64-1035c56ea079.png">

### Problem or idea

This should be included. I looked it up, I removed this when removing the s3 copy to remove the extra computed files.
This means that any compendia generated in the past 9 days are not available as well.

### Solution or next step

Deploy fix for compendia output file.
I will add a list of compendia here that need to be regenerated.
Regenerate those compendia.



Discussion:

Resolving patch:
diff --git a/workers/data_refinery_workers/processors/create_compendia.py b/workers/data_refinery_workers/processors/create_compendia.py
--- a/workers/data_refinery_workers/processors/create_compendia.py
+++ b/workers/data_refinery_workers/processors/create_compendia.py
@@ -358,7 +358,7 @@ def _perform_imputation(job_context: Dict) -> Dict:
     # visualized_merged_qn = visualize.visualize(job_context['merged_qn'].copy(), output_path)
 
     job_context['time_end'] = timezone.now()
-    job_context['formatted_command'] = "create_compendia.py"
+    job_context['formatted_command'] = ["create_compendia.py"]
     log_state("end prepare imputation", job_context["job"].id, imputation_start)
     return job_context
 
@@ -382,6 +382,10 @@ def _create_result_objects(job_context: Dict) -> Dict:
         return utils.handle_processor_exception(job_context, processor_key, e)
     result.save()
 
+    # Write the compendia dataframe to a file
+    job_context['csv_outfile'] = job_context['output_dir'] + job_context['organism_name'] + '.tsv'
+    job_context['merged_qn'].to_csv(job_context['csv_outfile'], sep='\t', encoding='utf-8')
+
     organism_key = list(job_context['samples'].keys())[0]
     annotation = ComputationalResultAnnotation()
     annotation.result = result
@@ -423,7 +427,6 @@ def _create_result_objects(job_context: Dict) -> Dict:
     # Compendia Result Helpers
     primary_organism = Organism.get_object_for_name(job_context['primary_organism'])
     organisms = [Organism.get_object_for_name(organism) for organism in job_context["all_organisms"]]
-    organisms_filter = Q(organisms__in=organisms)
     compendium_version = CompendiumResult.objects.filter(
                                                    primary_organism=primary_organism,
                                                    quant_sf_only=False


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Processing info: Sample is not available but only tximport is run on it
Ex:
https://www.refine.bio/experiments/SRP074593/gene-expression-profiling-of-drosophila-s2r-cells-following-rnai-mediated-knockdown-of-transcription-factors

![Screen Shot 2020-01-03 at 2 58 39 PM](https://user-images.githubusercontent.com/15315514/71745903-8e155e00-2e39-11ea-92d4-de1927766356.png)


We should just show the submitter supplied info, since the samples aren't really processed through our pipeline



Discussion:
@kurtwheeler could this be one of the experiments where we ran tximport when not all samples were processed? This looks like a data integrity issue.
#2056 added a fix on the code, I still have to add a migration to fix the data
Found another experiment with samples were we it seems we ran both salmon and tximport but the sample is not processed

https://www.refine.bio/experiments/ERP017365/impc-high-throughput-rna-seq-of-1135-lines

Resolving patch:
diff --git a/common/data_refinery_common/migrations/0052_remove_invalid_tximport_associations.py b/common/data_refinery_common/migrations/0052_remove_invalid_tximport_associations.py
new file mode 100644
--- /dev/null
+++ b/common/data_refinery_common/migrations/0052_remove_invalid_tximport_associations.py
@@ -0,0 +1,23 @@
+from django.db import migrations
+
+
+def remove_tximport_invalid_associations(apps, schema_editor):
+    """ We were associating tximport with all samples in an experiment
+    even though some of them were unprocessed.
+    Ref https://github.com/AlexsLemonade/refinebio/issues/2054 """
+    SampleResultAssociation = apps.get_model("data_refinery_common", "SampleResultAssociation")
+
+    SampleResultAssociation.objects.filter(
+        sample__is_processed=False, result__processor__name__iexact="tximport"
+    ).delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("data_refinery_common", "0051_remove_dataset_email_sent"),
+    ]
+
+    operations = [
+        migrations.RunPython(remove_tximport_invalid_associations),
+    ]


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Engagement bot thinks every user is a returning user
### Context

https://alexslemonade.slack.com/archives/CRK42AL1Y/p1587988808265500

### Problem or idea

@dvenprasad says 6 of those are new users. There must be a bug in the queries it uses or something.

### Solution or next step

Fix the engagement bot so it reports new users as new users.


Discussion:
Oh and if it's easy to do while you're at it, could we get `In the last 7 days, 17 users downloaded datasets from 16 locations.` to include the total like `In the last 7 days, 17 users X downloaded datasets from 16 locations.` where X is the total number of datasets downloaded in the last week?


Resolving patch:
diff --git a/api/data_refinery_api/management/commands/post_downloads_summary.py b/api/data_refinery_api/management/commands/post_downloads_summary.py
--- a/api/data_refinery_api/management/commands/post_downloads_summary.py
+++ b/api/data_refinery_api/management/commands/post_downloads_summary.py
@@ -42,28 +42,30 @@ def handle(self, *args, **options):
         unique_users = list(set(annotation.dataset.email_address for annotation in annotations))
         unique_ips = list(set(annotation.data["ip"] for annotation in annotations))
 
-        if unique_users:
-            fallback_text = "In the last {0} days, {1} users downloaded datasets from {2} locations.".format(
-                days, len(unique_users), len(unique_ips)
-            )
-        else:
-            fallback_text = "There were no downloads in the last {0} days.".format(days)
-
         new_users = ""
         returning_users = ""
+        total_downloads = 0
         for email in unique_users:
             user_annotations = annotation_queryset.filter(dataset__email_address=email)
-            total_downloads = user_annotations.count()
+            downloads = user_annotations.count()
+            total_downloads += downloads
             unique_locations = list(set(annotation.data["ip"] for annotation in user_annotations))
             locations = ", ".join(get_ip_location(ip) for ip in unique_locations)
-            is_new_user = DatasetAnnotation.objects.filter(
+            is_returning_user = DatasetAnnotation.objects.filter(
                 created_at__lt=start_time, dataset__email_address=email
             )
-            text = "{0} | {1} downloads from {2}\n".format(email, total_downloads, locations)
-            if is_new_user:
-                new_users += text
-            else:
+            text = "{0} | {1} downloads from {2}\n".format(email, downloads, locations)
+            if is_returning_user:
                 returning_users += text
+            else:
+                new_users += text
+
+        if total_downloads > 0:
+            fallback_text = "In the last {0} days, {1} users downloaded {2} datasets from {3} locations.".format(
+                days, len(unique_users), total_downloads, len(unique_ips)
+            )
+        else:
+            fallback_text = "There were no downloads in the last {0} days.".format(days)
 
         blocks = [
             {


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
There are two Illumina HiSeq 2000 platforms
<img width="259" alt="Screen_Shot_2020-05-05_at_3_37_37_PM" src="https://user-images.githubusercontent.com/15315514/81108350-93b5aa00-8ee6-11ea-98b1-8c572d59cb4d.png">

Are they the same thing? It's confusing. 


Discussion:
Also looks like Illumina HiSeq 2500 is also repeated. 
are they Ls and Is? If you convert them all to caps?
Yeah they L and Is and they are in the correct places
Whitespace that somehow is getting cleaned up for presentation?
The search endpoint returns the following platforms
`IlluminaHiSeq2000`
`Illumina HiSeq 2000`

Resolving patch:
diff --git a/common/data_refinery_common/migrations/0057_fix_platform_accessions_spaces.py b/common/data_refinery_common/migrations/0057_fix_platform_accessions_spaces.py
new file mode 100644
--- /dev/null
+++ b/common/data_refinery_common/migrations/0057_fix_platform_accessions_spaces.py
@@ -0,0 +1,29 @@
+from django.db import migrations
+from django.utils import timezone
+
+from data_refinery_common.utils import get_supported_rnaseq_platforms
+
+
+def remove_spaces_from_platform_accessions(apps, schema_editor):
+    Sample = apps.get_model("data_refinery_common", "Sample")
+
+    # get_supported_rnaseq_platforms() returns the platform names with
+    # spaces, but our platform accessions should have no spaces
+    for bad_accession in get_supported_rnaseq_platforms():
+        platform_accession = bad_accession.replace(" ", "")
+        bad_samples = Sample.objects.all().filter(platform_accession_code=bad_accession)
+
+        if not bad_samples:
+            continue
+
+        bad_samples.update(platform_accession_code=platform_accession, last_modified=timezone.now())
+        print("Updating platform accession from '%s' to '%s'" % (bad_accession, platform_accession))
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("data_refinery_common", "0056_auto_20200529_1230"),
+    ]
+
+    operations = [migrations.RunPython(remove_spaces_from_platform_accessions)]


Test patch:
Repository: AlexsLemonade/refinebio

Issue:
Add indices to new models for automatic accession gathering
### Context

We want to start adding indices to new models that we create. 


### Problem or idea

_The context should lead to something, an idea or a problem that you’re facing. Put that here._

### Solution or next step

_You can tag others or simply leave it for further investigation, but you must propose a next step towards solving the issue._



Discussion:

Resolving patch:
diff --git a/common/data_refinery_common/migrations/0072_auto_20220927_2235.py b/common/data_refinery_common/migrations/0072_auto_20220927_2235.py
new file mode 100644
--- /dev/null
+++ b/common/data_refinery_common/migrations/0072_auto_20220927_2235.py
@@ -0,0 +1,21 @@
+# Generated by Django 3.2.7 on 2022-09-27 22:35
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("data_refinery_common", "0071_gatheredaccession"),
+    ]
+
+    operations = [
+        migrations.AddIndex(
+            model_name="gatheredaccession",
+            index=models.Index(fields=["accession_code"], name="gathered_ac_accessi_5dde9a_idx"),
+        ),
+        migrations.AddIndex(
+            model_name="gatheredaccession",
+            index=models.Index(fields=["created_at"], name="gathered_ac_created_2efd0f_idx"),
+        ),
+    ]
diff --git a/common/data_refinery_common/models/gathered_accession.py b/common/data_refinery_common/models/gathered_accession.py
--- a/common/data_refinery_common/models/gathered_accession.py
+++ b/common/data_refinery_common/models/gathered_accession.py
@@ -9,6 +9,10 @@ class GatheredAccession(models.Model):
 
     class Meta:
         db_table = "gathered_accessions"
+        indexes = (
+            models.Index(fields=("accession_code",)),
+            models.Index(fields=("created_at",)),
+        )
 
     accession_code = models.TextField(unique=True)
     created_at = models.DateTimeField(auto_now_add=True)


Test patch:
Repository: craiga/will-of-the-prophets

Issue:
Clean up login form



Discussion:

Resolving patch:
diff --git a/will_of_the_prophets/settings/__init__.py b/will_of_the_prophets/settings/__init__.py
--- a/will_of_the_prophets/settings/__init__.py
+++ b/will_of_the_prophets/settings/__init__.py
@@ -42,6 +42,7 @@
     'django.contrib.messages',
     'django.contrib.staticfiles',
     'sass_processor',
+    'widget_tweaks',
     'bootstrap',
     'will_of_the_prophets',
 ]


Test patch: