Repository: Shopify/maintenance_tasks (Ruby)

Issue:
Don't rescue errors raised in callbacks
The documentation currently dictates that errors raised in any callback other than `after_error` will bubble up to be handled by the error handled, and will halt task execution: https://github.com/Shopify/maintenance_tasks#using-task-callbacks

This is not how things work though -- when refactoring the callbacks code in https://github.com/Shopify/maintenance_tasks/commit/7aff74ee00eaa21956a9e517f66bcca9f1adea1f , we started rescuing any errors raised inside any of the task callbacks. There may have been a reason we did this at the time, but it's not the desired behaviour today, nor what is documented in the README.

We shouldn't be rescuing any errors outside of the `after_error` callback. Users can choose to rescue callback errors themselves, but in all other cases the error should bubble up to the handler and stop the task. This is especially important because users might be explicitly using callbacks to dictate task flow / halt execution.


Discussion:

Patch:
diff --git a/app/models/maintenance_tasks/run.rb b/app/models/maintenance_tasks/run.rb
index 942e9e8bf..64c5d6193 100644
--- a/app/models/maintenance_tasks/run.rb
+++ b/app/models/maintenance_tasks/run.rb
@@ -140,7 +140,7 @@ def persist_error(error)
         backtrace: MaintenanceTasks.backtrace_cleaner.clean(error.backtrace),
         ended_at: Time.now,
       )
-      run_task_callbacks(:error)
+      run_error_callback
     rescue ActiveRecord::StaleObjectError
       reload_status
       retry
@@ -264,7 +264,7 @@ def running
     #   specified by the Task.
     def start(count)
       update!(started_at: Time.now, tick_total: count)
-      run_task_callbacks(:start)
+      task.run_callbacks(:start)
     rescue ActiveRecord::StaleObjectError
       reload_status
       retry
@@ -430,6 +430,12 @@ def task
 
     def run_task_callbacks(callback)
       task.run_callbacks(callback)
+    rescue Task::NotFoundError
+      nil
+    end
+
+    def run_error_callback
+      task.run_callbacks(:error)
     rescue
       nil
     end


Test patch:
diff --git a/test/jobs/maintenance_tasks/task_job_test.rb b/test/jobs/maintenance_tasks/task_job_test.rb
index aa2ec7744..55341a857 100644
--- a/test/jobs/maintenance_tasks/task_job_test.rb
+++ b/test/jobs/maintenance_tasks/task_job_test.rb
@@ -580,5 +580,21 @@ class << self
     ensure
       CustomTaskJob.race_condition_prepended_after_hook = nil
     end
+
+    test ".perform_now bubbles up callback errors" do
+      Maintenance::CallbackTestTask
+        .any_instance
+        .expects(:after_start_callback)
+        .raises("Callback error!")
+
+      Maintenance::CallbackTestTask.any_instance.expects(:process).never
+
+      run = Run.create!(task_name: "Maintenance::CallbackTestTask")
+      run.expects(:persist_error).with do |exception|
+        assert_equal "Callback error!", exception.message
+      end
+
+      TaskJob.perform_now(run)
+    end
   end
 end
diff --git a/test/models/maintenance_tasks/run_test.rb b/test/models/maintenance_tasks/run_test.rb
index b6077e440..7a293872d 100644
--- a/test/models/maintenance_tasks/run_test.rb
+++ b/test/models/maintenance_tasks/run_test.rb
@@ -172,6 +172,18 @@ class RunTest < ActiveSupport::TestCase
       run.persist_error(error)
     end
 
+    test "#persist_error can handle error callback raising" do
+      run = Run.create!(task_name: "Maintenance::CallbackTestTask")
+      error = ArgumentError.new("Something went wrong")
+      error.set_backtrace(["lib/foo.rb:42:in `bar'"])
+
+      run.task.expects(:after_error_callback).raises("Callback error!")
+
+      assert_nothing_raised do
+        run.persist_error(error)
+      end
+    end
+
     test "#persist_error does not raise on longer error class names" do
       run = Run.create!(task_name: "Maintenance::ErrorTask")
       error = ArgumentError.new("Something went wrong")