Repository: AlexsLemonade/refinebio

Issue:
Delete data when a processor job fails for the third time.
### Context

When a job fails we don't delete the input data for it in case we want to try it again. However at this point we're at a place where we don't really ever go back and retry a job more than three times anymore. This means that as a volume has more and more final failures, the disk space will slowly fill up.

### Problem or idea

We should delete the input data associated with a job when it fails for the third time.

### Solution or next step

So I think basically it should just be that in this function: https://github.com/AlexsLemonade/refinebio/blob/003003dc448b872cd2f80013a3defb6d67f5f9f4/foreman/data_refinery_foreman/foreman/main.py#L113
we need to add logic that does similar stuff to: https://github.com/AlexsLemonade/refinebio/blob/003003dc448b872cd2f80013a3defb6d67f5f9f4/workers/data_refinery_workers/processors/utils.py#L297



Discussion:

Resolving patch:
diff --git a/common/data_refinery_common/models/jobs.py b/common/data_refinery_common/models/jobs.py
--- a/common/data_refinery_common/models/jobs.py
+++ b/common/data_refinery_common/models/jobs.py
@@ -148,6 +148,13 @@ def save(self, *args, **kwargs):
         self.last_modified = current_time
         return super(ProcessorJob, self).save(*args, **kwargs)
 
+    def clean_up_original_files(self):
+        """ If a processor job fails too many times, we want to clean up the downloaded files to
+        save hard drive space.
+        """
+        for original_file in self.original_files.all():
+            original_file.delete_local_file()
+
     def __str__(self):
         return "ProcessorJob " + str(self.pk) + ": " + str(self.pipeline_applied)
 
diff --git a/foreman/data_refinery_foreman/foreman/main.py b/foreman/data_refinery_foreman/foreman/main.py
--- a/foreman/data_refinery_foreman/foreman/main.py
+++ b/foreman/data_refinery_foreman/foreman/main.py
@@ -121,6 +121,11 @@ def handle_repeated_failure(job) -> None:
     job.success = False
     job.save()
 
+    # If the job is a processor job and we want to stop retrying it,
+    # delete the original files to conserve disk space.
+    if isinstance(job, ProcessorJob):
+        job.clean_up_original_files()
+
     # At some point this should become more noisy/attention
     # grabbing. However for the time being just logging should be
     # sufficient because all log messages will be closely monitored


Test patch:
diff --git a/foreman/data_refinery_foreman/foreman/test_main.py b/foreman/data_refinery_foreman/foreman/test_main.py
--- a/foreman/data_refinery_foreman/foreman/test_main.py
+++ b/foreman/data_refinery_foreman/foreman/test_main.py
@@ -395,9 +395,10 @@ def test_requeuing_processor_job_w_more_ram(self, mock_send_job, mock_get_active
         self.assertEqual(original_job.ram_amount, 16384)
         self.assertEqual(retried_job.ram_amount, 32768)
 
+    @patch('os.remove')
     @patch('data_refinery_foreman.foreman.main.get_active_volumes')
     @patch('data_refinery_foreman.foreman.main.send_job')
-    def test_repeated_processor_failures(self, mock_send_job, mock_get_active_volumes):
+    def test_repeated_processor_failures(self, mock_send_job, mock_get_active_volumes, mock_os_remove):
         mock_send_job.return_value = True
         mock_get_active_volumes.return_value = {"1", "2", "3"}
 
@@ -426,6 +427,8 @@ def test_repeated_processor_failures(self, mock_send_job, mock_get_active_volume
         self.assertEqual(last_job.num_retries, main.MAX_NUM_RETRIES)
         self.assertFalse(last_job.success)
 
+        mock_os_remove.assert_called_with("nor this")
+
     @patch('data_refinery_foreman.foreman.main.get_active_volumes')
     @patch('data_refinery_foreman.foreman.main.send_job')
     def test_retrying_failed_processor_jobs(self, mock_send_job, mock_get_active_volumes):