Repository: AlexsLemonade/refinebio

Issue:
Add the biggest `failure_reason` to the dashboard?
### Context

I have been debugging some sample processing errors.

### Problem or idea

I think it would be useful to have a list of the top failure reasons and the number of samples that they affect. I'm not sure if the dashboard is a good place for this.

```
  ccc   |                                 job_failure_reason                                  
--------+-------------------------------------------------------------------------------------
 646262 | 
  47680 | Salmon timed out because it failed to complete within 3 hours.
  40654 | Encountered error in R code while running illumina.R pipeline during processing of 
  29566 | Unhandled exception caught while running processor function _prepare_files in pipel
  26986 | Shell call to salmon failed because: ### salmon (mapping-based) v0.13.1            +
        | ### [ progr
  10347 | Missing file for processor job but unable to recreate downloader jobs!
   8912 | Encountered error in R code while running AFFY_TO_PCL pipeline during processing of
   3682 | The tx index for this organism is broken right now.
   3010 | list index out of range
   2629 | ProcessorJob has already completed with a fail - why are we here again? Bad Nomad!
   2556 | Unhandled exception caught while running processor function start_job in pipeline: 
   2440 | Missing transcriptome index. (TRANSCRIPTOME_LONG)
   1919 | Unhandled exception caught while running processor function _convert_genes in pipel
   1886 | Missing transcriptome index. (TRANSCRIPTOME_SHORT)
   1699 | Failed to download or extract transcriptome index for organism ARABIDOPSIS_THALIANA
   1424 | Told to convert AFFY genes without an internal_accession - how did this happen?
   1200 | Unhandled exception caught while running processor function _detect_platform in pip
   1138 | Unhandled exception caught while running processor function _run_scan_upc in pipeli
    868 | Encountered error in R code while running gene_convert_illumina.R pipeline during p
    791 | Status code -9 from gene_convert.R: b"\nAttaching package: 'dplyr'\n\nThe following
(20 rows)
```

That's the top 20 failure reasons with a total of `835,649` potential samples. It's also very strange that there are so many samples with no `failure_reason` on their last processor job.

<details><summary>query</summary>

```sql
select count(*) as ccc, (
    select left(processor_jobs.failure_reason, 83)
    from original_file_sample_associations, processorjob_originalfile_associations, processor_jobs
    where samples.id=original_file_sample_associations.sample_id
        and original_file_sample_associations.original_file_id=processorjob_originalfile_associations.original_file_id
        and processorjob_originalfile_associations.processor_job_id=processor_jobs.id
    order by processor_jobs.start_time desc nulls last
    limit 1
  ) as job_failure_reason
from samples
where samples.is_processed='f'
group by job_failure_reason
order by ccc desc
limit 20;
```

</details>

I have been working on both:
```
   8912 | Encountered error in R code while running AFFY_TO_PCL pipeline during processing of
   2629 | ProcessorJob has already completed with a fail - why are we here again? Bad Nomad!
```

Maybe we can re-queue some of the samples with these errors? With https://github.com/AlexsLemonade/refinebio/pull/1541

### Solution or next step

Should we add this to the dashboard?




Discussion:
It would be amazing to add this to the dashboard if the numbers could be aligned with the time windows selected (one day, one week, etc). This way we could know what errors are happening now. We have this a little bit with Sentry, but we haven't found that to be useful for some reason. We will want to make sure this doesn't fall into the same trap.
This would increase the usefulness of the dashboard for me. Also, this ticket is awesome 👏 
I'll note that query seems like we're failing without a failure reason much more than we are. Add a couple filters and it doesn't look nearly so bad:

```
data_refinery=> select count(*) from processor_jobs where failure_reason is null and success='f' and num_retries=2;
 count  
--------
 112757
```

It doesn't seem entirely unlikely to me that a lot of these no-failure-reason-jobs just needed more RAM:
```
data_refinery=> select count(*), ram_amount from processor_jobs where failure_reason is null and success='f' and num_retries=2 group by ram_amount order by ram_amount;
 count | ram_amount 
-------+------------
 11835 |       2048
  5333 |       3072
   781 |       4096
  4610 |       8192
 15141 |      12288
 11777 |      16384
 54874 |      32768
  8406 |      65536
(8 rows)
```

Resolving patch:
diff --git a/api/data_refinery_api/urls.py b/api/data_refinery_api/urls.py
--- a/api/data_refinery_api/urls.py
+++ b/api/data_refinery_api/urls.py
@@ -39,7 +39,9 @@
     CompendiumResultDetails,
     ComputedFilesList,
     OriginalFileList,
-    AboutStats
+    AboutStats,
+    FailedDownloaderJobStats,
+    FailedProcessorJobStats
 )
 
 # This provides _public_ access to the /admin interface!
@@ -102,6 +104,8 @@ class AccessUser:
 
         # Dashboard Driver
         url(r'^stats/$', Stats.as_view(), name='stats'),
+        url(r'^stats/failures/downloader$', FailedDownloaderJobStats.as_view(), name='stats_failed_downloader'),
+        url(r'^stats/failures/processor$', FailedProcessorJobStats.as_view(), name='stats_failed_processor'),
         url(r'^stats-about/$', AboutStats.as_view(), name='stats_about'),
 
         # Transcriptome Indices and QN Targets
diff --git a/api/data_refinery_api/views.py b/api/data_refinery_api/views.py
--- a/api/data_refinery_api/views.py
+++ b/api/data_refinery_api/views.py
@@ -7,7 +7,7 @@
 from django.conf import settings
 from django.views.decorators.cache import cache_page
 from django.db.models import Count, Prefetch, DateTimeField, OuterRef, Subquery
-from django.db.models.functions import Trunc
+from django.db.models.functions import Trunc, Left
 from django.db.models.aggregates import Avg, Sum
 from django.db.models.expressions import F, Q
 from django.http import Http404, HttpResponse, HttpResponseRedirect, HttpResponseBadRequest
@@ -916,6 +916,72 @@ def get_queryset(self):
 # Statistics
 ###
 
+def get_start_date(range_param):
+    current_date = datetime.now(tz=timezone.utc)
+    return {
+        'day': current_date - timedelta(days=1),
+        'week': current_date - timedelta(weeks=1),
+        'month': current_date - timedelta(days=30),
+        'year': current_date - timedelta(days=365)
+    }.get(range_param)
+
+def paginate_queryset_response(queryset, request):
+    paginator = LimitOffsetPagination()
+    page_items = paginator.paginate_queryset(queryset, request)
+    return Response(data={
+                        'results': [x.to_dict() for x in page_items],
+                        'limit': paginator.limit,
+                        'offset': paginator.offset,
+                        'count': paginator.count
+                    },
+                    status=status.HTTP_200_OK)
+
+class FailedDownloaderJobStats(APIView):
+    @swagger_auto_schema(manual_parameters=[openapi.Parameter(
+        name='range', in_=openapi.IN_QUERY, type=openapi.TYPE_STRING,
+        description="Specify a range from which to calculate the possible options",
+        enum=('day', 'week', 'month', 'year',)
+    )])
+    @method_decorator(cache_page(10 * 60))
+    def get(self, request, version, format=None):
+        range_param = request.query_params.dict().pop('range', 'day')
+        start_date = get_start_date(range_param)
+        jobs = DownloaderJob.objects\
+            .filter(created_at__gt=start_date)\
+            .annotate(reason=Left('failure_reason', 80))\
+            .values('reason')\
+            .annotate(
+                job_count=Count('reason'),
+                sample_count=Count('original_files__samples', distinct=True, filter=Q(original_files__samples__is_processed=False))
+            )\
+            .order_by('-job_count')
+
+        return paginate_queryset_response(jobs, request)
+
+
+class FailedProcessorJobStats(APIView):
+    @swagger_auto_schema(manual_parameters=[openapi.Parameter(
+        name='range', in_=openapi.IN_QUERY, type=openapi.TYPE_STRING,
+        description="Specify a range from which to calculate the possible options",
+        enum=('day', 'week', 'month', 'year',)
+    )])
+    @method_decorator(cache_page(10 * 60))
+    def get(self, request, version, format=None):
+        range_param = request.query_params.dict().pop('range', 'day')
+        start_date = get_start_date(range_param)
+        jobs = ProcessorJob.objects\
+            .filter(created_at__gt=start_date)\
+            .annotate(reason=Left('failure_reason', 80))\
+            .values('reason')\
+            .annotate(
+                job_count=Count('reason'),
+                sample_count=Count('original_files__samples', distinct=True, filter=Q(original_files__samples__is_processed=False))
+            )\
+            .order_by('-job_count')
+
+        return paginate_queryset_response(jobs, request)
+
+
 class AboutStats(APIView):
     """ Returns general stats for the site, used in the about page """
     @method_decorator(cache_page(10 * 60))
@@ -976,6 +1042,7 @@ def _get_samples_available(self):
         ).distinct().count()
         return processed_samples + unprocessed_samples_with_quant
 
+
 class Stats(APIView):
     """ Statistics about the health of the system. """
 
@@ -1104,13 +1171,7 @@ def _get_job_stats(cls, jobs, range_param):
         start_filter = Q()
 
         if range_param:
-            current_date = datetime.now(tz=timezone.utc)
-            start_date = {
-                'day': current_date - timedelta(days=1),
-                'week': current_date - timedelta(weeks=1),
-                'month': current_date - timedelta(days=30),
-                'year': current_date - timedelta(days=365)
-            }.get(range_param)
+            start_date = get_start_date(range_param)
             start_filter = start_filter | Q(start_time__gte=start_date) | Q(start_time__isnull=True)
 
         result = jobs.filter(start_filter).aggregate(
@@ -1170,13 +1231,7 @@ def _get_intervals(cls, objects, range_param, field='last_modified'):
             'month': 'day',
             'year': 'month'
         }
-        current_date = datetime.now(tz=timezone.utc)
-        range_to_start_date = {
-            'day': current_date - timedelta(days=1),
-            'week': current_date - timedelta(weeks=1),
-            'month': current_date - timedelta(days=30),
-            'year': current_date - timedelta(days=365)
-        }
+        range_to_start_date = get_start_date(range_param)
 
         # truncate the `last_modified` field by hour, day or month depending on the `range` param
         # and annotate each object with that. This will allow us to count the number of objects


Test patch: