Repository: sidekiq/sidekiq (Ruby)

Issue:
Dead queue stuck in retry loop after hitting "Retry All"
Ruby version: 3.2.5
Rails version: 7.1.4.1
Sidekiq / Pro / Enterprise version(s): sidekiq-pro 5.5.8

Dead queue sometimes gets stuck in a retry loop after hitting "Retry All" from the Dead page on Sidekiq UI. The dead jobs are enqueued & run, end up in the dead queue, then enqueued again on repeat.

I believe this is because [retry_all](https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/api.rb#L843-L847) loops while the [size of the dead queue](https://github.com/sidekiq/sidekiq/blob/main/lib/sidekiq/api.rb#L620-L623) is > 0. This means that the initial jobs running could fail and go back to the dead queue before the later jobs get run.

Reproducing on Rails console:
```
> dead_set = Sidekiq::DeadSet.new
=> #<Sidekiq::DeadSet:0x00007fb0edb85330 @_size=51, @name="dead">
> while dead_set.size > 0
     dead_set.each do |job|
       job.retry
     end
     print "."
   end
...........=> nil
```



Discussion: Brainstorming here...

Since the score of any new entries to the Dead set should be strictly larger than any existing score, we should be able to add logic like "break after enqueuing N elements" where N is the size at the start of the loop, instead of looping until size = 0.

PRs welcome, I'm working on Faktory right now.
Aren’t we already looping through each job and calling retry? Why can’t we just get rid of the size > 0 check?

I’m assuming it’s because we want to capture any incoming dead jobs as well, which I suppose begs the question of whether “Retry All” should retry what’s on the page or what’s in the Redis dead queue.
We're mutating a data structure while simultaneously iterating through it. Lots of room for data races and footguns here.

Since this is a relatively straightforward thing to test, I think we should create an integration test which exercises this case so we can test and verify the current semantics. There should be something close to that already in `test/api_test.rb`.
The size check is because iterating while mutating is unpredictable. If we remove an element, all other elements shift their index so we can't effectively use a cursor.

We could iterate backwards from the "tail" of the structure and use indexes to remove the tail element. That will only fail if another entry is pushed into the middle of the structure: unlikely for the dead set but quite common for the scheduled set.

I just tested and verified that using `each` breaks (retrying each job causes 500 out of 1000 jobs to be retried) but using `reverse_each` works exactly as expected. Unfortunately reverse_each loads the entire dataset into memory first. 😭 No easy solution yet.
I would suggest the easiest and best long-term choice is to rewrite these APIs to use ZPOPMIN[0], which we couldn't use until Sidekiq 7.0. Popping one entry at a time effectively rate limits the enqueue process and minimizes any data loss if the network happens to drop right at that instant.

[0]: https://redis.io/docs/latest/commands/zpopmin/
Let me know if you have any feedback RE: this PR. ^^^^^^^^^^^



Fix (patch):
diff --git a/Changes.md b/Changes.md
index 4916962064..7747e2a2de 100644
--- a/Changes.md
+++ b/Changes.md
@@ -5,6 +5,10 @@
 HEAD
 ----------
 
+- Reimplement `retry_all` and `kill_all` API methods to use ZPOPMIN,
+  approximately 30-60% faster. [#6481]
+- Add preload testing binary at `examples/testing/sidekiq_boot` to verify your Rails app boots correctly with Sidekiq Enterprise's app preloading.
+- Fix circular require with ActiveJob adapter [#6477]
 - Fix potential race condition leading to incorrect serialized values for CurrentAttributes [#6475]
 
 7.3.4
diff --git a/lib/sidekiq/api.rb b/lib/sidekiq/api.rb
index f5b8eabdb5..fa206d6570 100644
--- a/lib/sidekiq/api.rb
+++ b/lib/sidekiq/api.rb
@@ -668,6 +668,41 @@ def schedule(timestamp, job)
       end
     end
 
+    def pop_each
+      Sidekiq.redis do |c|
+        size.times do
+          data, score = c.zpopmin(name, 1)&.first
+          break unless data
+          yield data, score
+        end
+      end
+    end
+
+    def retry_all
+      c = Sidekiq::Client.new
+      pop_each do |msg, _|
+        job = Sidekiq.load_json(msg)
+        # Manual retries should not count against the retry limit.
+        job["retry_count"] -= 1 if job["retry_count"]
+        c.push(job)
+      end
+    end
+
+    # Move all jobs from this Set to the Dead Set.
+    # See DeadSet#kill
+    def kill_all(notify_failure: false, ex: nil)
+      ds = DeadSet.new
+      opts = {notify_failure: notify_failure, ex: ex, trim: false}
+
+      begin
+        pop_each do |msg, _|
+          ds.kill(msg, opts)
+        end
+      ensure
+        ds.trim
+      end
+    end
+
     def each
       initial_size = @_size
       offset_size = 0
@@ -765,10 +800,6 @@ def delete_by_jid(score, jid)
 
   ##
   # The set of scheduled jobs within Sidekiq.
-  # Based on this, you can search/filter for jobs.  Here's an
-  # example where I'm selecting jobs based on some complex logic
-  # and deleting them from the scheduled set.
-  #
   # See the API wiki page for usage notes and examples.
   #
   class ScheduledSet < JobSet
@@ -779,26 +810,12 @@ def initialize
 
   ##
   # The set of retries within Sidekiq.
-  # Based on this, you can search/filter for jobs.  Here's an
-  # example where I'm selecting all jobs of a certain type
-  # and deleting them from the retry queue.
-  #
   # See the API wiki page for usage notes and examples.
   #
   class RetrySet < JobSet
     def initialize
       super("retry")
     end
-
-    # Enqueues all jobs pending within the retry set.
-    def retry_all
-      each(&:retry) while size > 0
-    end
-
-    # Kills all jobs pending within the retry set.
-    def kill_all
-      each(&:kill) while size > 0
-    end
   end
 
   ##
@@ -811,20 +828,31 @@ def initialize
       super("dead")
     end
 
+    # Trim dead jobs which are over our storage limits
+    def trim
+      hash = Sidekiq.default_configuration
+      now = Time.now.to_f
+      Sidekiq.redis do |conn|
+        conn.multi do |transaction|
+          transaction.zremrangebyscore(name, "-inf", now - hash[:dead_timeout_in_seconds])
+          transaction.zremrangebyrank(name, 0, - hash[:dead_max_jobs])
+        end
+      end
+    end
+
     # Add the given job to the Dead set.
     # @param message [String] the job data as JSON
-    # @option opts [Boolean] :notify_failure  (true) Whether death handlers should be called
+    # @option opts [Boolean] :notify_failure (true) Whether death handlers should be called
+    # @option opts [Boolean] :trim (true) Whether Sidekiq should trim the structure to keep it within configuration
     # @option opts [Exception] :ex (RuntimeError) An exception to pass to the death handlers
     def kill(message, opts = {})
       now = Time.now.to_f
       Sidekiq.redis do |conn|
-        conn.multi do |transaction|
-          transaction.zadd(name, now.to_s, message)
-          transaction.zremrangebyscore(name, "-inf", now - Sidekiq::Config::DEFAULTS[:dead_timeout_in_seconds])
-          transaction.zremrangebyrank(name, 0, - Sidekiq::Config::DEFAULTS[:dead_max_jobs])
-        end
+        conn.zadd(name, now.to_s, message)
       end
 
+      trim if opts[:trim] != false
+
       if opts[:notify_failure] != false
         job = Sidekiq.load_json(message)
         if opts[:ex]
@@ -839,11 +867,6 @@ def kill(message, opts = {})
       end
       true
     end
-
-    # Enqueue all dead jobs
-    def retry_all
-      each(&:retry) while size > 0
-    end
   end
 
   ##