Repository: rails/solid_queue (Ruby)

Issue:
When the worker dies, the concurrency limit is not reset
I have a job that can run for an extended period — let’s call it `InfiniteSleepJob`:

```rb
class InfiniteSleepJob < ApplicationJob
  # limit only one simultaneous execution of operation per key
  limits_concurrency to: 1, key: ->(key, *_args) { key },  group: "my_group", duration: 1.day

  def perform(*args)
    # do smt
  end
end
```

When I add a few jobs to the queue and the worker unexpectedly dies during execution (e.g., due to an HA event or if I forcibly kill the Docker container with `docker kill container`), the concurrency limit gets stuck, even after the job is pruned. Block is released only when it expires

Here’s an example of a job that was pruned, but the concurrency limit wasn’t removed:
![Image](https://github.com/user-attachments/assets/3b0f0bc8-1200-4a5c-86ee-ca869ccb170e)

![Image](https://github.com/user-attachments/assets/732eccd8-0200-41fc-add7-9c0da937ccdb)

Here’s my `queue.yml`:

```yml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval:  120
  workers:
    - queues: "default"
      threads: 3
      processes: 1
      polling_interval: 1
```

I’d expect the queue to unlock as soon as the job fails, allowing the blocked jobs to proceed as expected.

Could you kindly clarify whether this is the expected behavior? Am I missing something?


Discussion:
Hey @OlegChuev, yeah, it's expected behaviour if the worker dies unceremoniously (eg. a `kill -9` signal). It wouldn't have a chance to clean up after itself, which includes the concurrency limit. The duration parameter determines how long the system will wait to unblock jobs that didn't get successfully unblocked for any reason, so if your long running job starts at, say 17:00, another one gets enqueued and blocked at 18:00, and the first one dies at 20:00, it won't be until the following day at 17:00 (`1.day` duration) that the blocked job will be considered to be unlocked. 
Although I think I could improve this case, when releasing claimed jobs whose worker got killed... 🤔 I'll work on this next week. 
Hello @rosa, I have tried to fix this issue in my PR https://github.com/rails/solid_queue/pull/547



Patch:
diff --git a/app/models/solid_queue/claimed_execution.rb b/app/models/solid_queue/claimed_execution.rb
index 94ee3593..8840505b 100644
--- a/app/models/solid_queue/claimed_execution.rb
+++ b/app/models/solid_queue/claimed_execution.rb
@@ -39,7 +39,10 @@ def release_all
     def fail_all_with(error)
       SolidQueue.instrument(:fail_many_claimed) do |payload|
         includes(:job).tap do |executions|
-          executions.each { |execution| execution.failed_with(error) }
+          executions.each do |execution|
+            execution.failed_with(error)
+            execution.unblock_next_job
+          end
 
           payload[:process_ids] = executions.map(&:process_id).uniq
           payload[:job_ids] = executions.map(&:job_id).uniq
@@ -67,7 +70,7 @@ def perform
       raise result.error
     end
   ensure
-    job.unblock_next_blocked_job
+    unblock_next_job
   end
 
   def release
@@ -90,6 +93,10 @@ def failed_with(error)
     end
   end
 
+  def unblock_next_job
+    job.unblock_next_blocked_job
+  end
+
   private
     def execute
       ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id))


Test patch:
diff --git a/test/unit/supervisor_test.rb b/test/unit/supervisor_test.rb
index c430544a..108ebb6f 100644
--- a/test/unit/supervisor_test.rb
+++ b/test/unit/supervisor_test.rb
@@ -133,6 +133,36 @@ class SupervisorTest < ActiveSupport::TestCase
     end
   end
 
+  test "fail orphaned executions by releasing their concurrency locks" do
+    result = JobResult.create!(queue_name: "default", status: "seq: ")
+    4.times { |i| ThrottledUpdateResultJob.set(queue: :new_queue).perform_later(result) }
+    process = SolidQueue::Process.register(kind: "Worker", pid: 42, name: "worker-123")
+
+    SolidQueue::ReadyExecution.claim("*", 5, process.id)
+
+    assert_equal 3, SolidQueue::ClaimedExecution.count
+    assert_equal 0, SolidQueue::ReadyExecution.count
+    assert_equal 1, SolidQueue::BlockedExecution.count
+
+    assert_equal [ process.id ], SolidQueue::ClaimedExecution.last(3).pluck(:process_id).uniq
+
+    # Simnulate orphaned executions by just wiping the claiming process
+    process.delete
+
+    pid = run_supervisor_as_fork(workers: [ { queues: "background", polling_interval: 10, processes: 2 } ])
+    wait_for_registered_processes(3)
+    assert_registered_supervisor(pid)
+
+    terminate_process(pid)
+
+    skip_active_record_query_cache do
+      assert_equal 0, SolidQueue::ClaimedExecution.count
+      assert_equal 3, SolidQueue::FailedExecution.count
+      assert_equal 0, SolidQueue::BlockedExecution.count
+      assert_equal 1, SolidQueue::ReadyExecution.count
+    end
+  end
+
   test "prune processes with expired heartbeats" do
     pruned = SolidQueue::Process.register(kind: "Worker", pid: 42, name: "worker-42")