Repository: AlexsLemonade/refinebio

Issue:
Too many emails when dataset fails to smash
### Context
I tried to download a dataset from refine.bio which failed. (Nov 19, 20) 

### Problem or idea
I received 5 emails that day and 4 of them in a span of 20 minutes, all with the same failure reason. 
For another one, I tried to download I got 3 emails in a span of 2 minutes. 

I don't know if this is the normal behavior or there was something else going on that day which might have triggered the additional emails.

### Solution or next step

Users should receive one failure email.



Discussion:
I need some help here... when a dataset fails to process we have to send an email to the user that requested it, the problem is that datasets jobs are executed multiple times and from the smasher, we don't have a way to know which one will be the last time.

I see two possible solutions:

1. Add a way to determine when a smasher job won't be retried again, ie it has `num_retries=2`, and send the emails before finishing the job. 
The logic to determine if a job should be retried is in the foreman, so we would have to either duplicate it or add a dependency between these modules.

2. Execute the smasher jobs **a single time**, this way we can always send the email if processing fails.

I lean towards the second one since we allow users to retry datasets easily when they fail from the frontend. 

Does this make sense? Am I missing something here? @davidsmejia @kurtwheeler 
- Have the foreman queue a new job type that should handle failures for other jobs, in the case of the smasher here is where we would send the failure notification. But it could be useful for other jobs as well.

- Don't send failures notifications at all

Talked with @davidsmejia and we came up with additional options.



Also, if a dataset fails to smash, what's the likelihood that re-trying it would have a different effect?
I looked into all smasher failures to see if there might be some that make sense to retry. These are all the **different reasons**

```
'Caught either a SIGTERM or SIGINT signal.'
'Failure reason: Failed to upload /home/user/data_store/smashed/9b0127b5-e3c1-42ea-ae5a-58a945131a9c.zip to data-refinery-s3-results-circleci-prod/9b0127b5-e3c1-42ea-ae5a-58a945131a9c.zip: An error occurred (BadDigest) when calling the PutObject operation (reached max retries: 4): The Content-MD5 you specified did not match what we received.'

'Could not find QN target for Organism: MUS_MUSCULUS'
"Couldn't get any files to smash for Smash job!!"
"Couldn't get any files to smash for Smash job - empty all_sample_files"
'Failure reason: canceling statement due to statement timeout\n'

'Failure reason: Smasher received non-existent file path.'
"Unhandled exception caught while running processor function _notify in pipeline: 'result_url'"
"Unhandled exception caught while running processor function _prepare_files in pipeline: [Errno 28] No space left on device: '/home/user/data_store/smashed/985ca52b-579b-432f-98d8-78f4cbd43a4e/'"
'Unhandled exception caught while running processor function start_job in pipeline: processors.start_job called on job 11643 that has already been started!'

"Failure reason: [Errno 28] No space left on device: '/home/user/data_store/processor_job_37561'"
"Failure reason: [Errno 2] No such file or directory: '/home/user/data_store/smashed/9b0127b5-e3c1-42ea-ae5a-58a945131a9c/output/'"
'Failure reason: Failed Kolmogorov Smirnov test! Stat: 0.041362299592069066 PVal: 8.881784197001252e-16'
"Failure reason: 'float' object cannot be interpreted as an integer"
"Failure reason: 'merged_qn'"
"Failure reason: 'organism_name'"
"Failure reason: 'NoneType' object has no attribute 'transpose'"
'Failure reason: Problem occured during quantile normalization: No merged_qn'
'Failure reason: Found fewer columns than required for QN combinatorial - bad smash?'
```

Out of those, only the first two looks like it might yield a different result if we retry them. 

I think it should be safe to run `SMASHER` jobs only once. Assuming they don't get a `SIGTERM`. From the frontend is easy to retry them if users need to.

Also, only 5 jobs failed for these two reasons out of 783.

```
>>> ProcessorJob.objects.filter(pipeline_applied='SMASHER').count()
783
>>> ProcessorJob.objects.filter(pipeline_applied='SMASHER', failure_reason='Caught either a SIGTERM or SIGINT signal.').count()
3
>>> ProcessorJob.objects.filter(pipeline_applied='SMASHER', failure_reason__startswith='Failure reason: Failed to upload').count()
2
```


Do we try the md5 thing 4 times internally or is this an issue that came up on the 4th time? I also imagine those sigterm or sigint ones are primarily on large jobs (compendia) when we redeployed. Is that the case?

I agree with your take on it being safe to retry once.
> Do we try the md5 thing 4 times internally or is this an issue that came up on the 4th time?

No, we just try to upload it a single time. Two whole processor jobs failed because of that. I'm guessing the files could have been corrupted during the transfer.

> I also imagine those sigterm or sigint ones are primarily on large jobs (compendia) when we redeployed. Is that the case?

No, those were all dataset smasher jobs from users. I do think they might have failed because we deployed while we were processing them.

> I agree with your take on it being safe to retry once.

I'm going to make it so that the jobs are not retried once they send an email.



Resolving patch:
diff --git a/workers/data_refinery_workers/processors/smasher.py b/workers/data_refinery_workers/processors/smasher.py
--- a/workers/data_refinery_workers/processors/smasher.py
+++ b/workers/data_refinery_workers/processors/smasher.py
@@ -433,6 +433,11 @@ def _notify(job_context: Dict) -> Dict:
                                           exc_info=1,
                                           result_url=job_context['result_url'])
 
+    # We don't want to retry this dataset after we send a notification to users
+    # https://github.com/alexslemonade/refinebio/issues/1944
+    job_context['job'].no_retry = True
+    job_context['job'].save()
+
     return job_context
 
 


Test patch: