Repository: ruby/debug (Ruby)

Issue:
Debugger freezes when triggering timeout inside a timeout block with Puma server
**Your environment**

* `ruby -v`: `3.1.3`
* `rdbg -v`: `1.7.1`

**Describe the bug**

When the following conditions all match, the debugger has a high possibility to freeze:

- Inside a Rails app with Puma web server
- The breakpoint is inside a timeout call, like

    ```rb
      def hello
        Timeout.timeout(3600) do
          binding.b
        end
      end
    ```

- From the breakpoint, executes something that calls `Timeout.timeout` underneath

**To Reproduce**

I've documented reproduction steps in [this repo](https://github.com/st0012/debug-freeze-issue).

**Expected behavior**

The debugger doesn't freeze.

**Additional context**

It may not be the root cause, but `Timeout`'s thread being stopped by the debugger plays a major role here. 

If we somehow modify [this method](https://github.com/ruby/debug/blob/master/lib/debug/session.rb#L1654-L1664) to skip pausing the `Timeout` thread, like

```rb
    private def thread_stopper
      TracePoint.new(:line) do
        # run on each thread
        tc = ThreadClient.current
        next if tc.management?
        next unless tc.running?
        next if tc == @tc
        next if tc.thread.name && tc.thread.name.match?(/Timeout/)

        tc.on_pause
      end
    end
```

Then the issue doesn't occur.



Discussion: Thank you for the report.

* Is it difficult to repro within a single file? (without puma)
* Can you show the backtrace when thread_stopper's TP stoppes?

by modifying like that:

``` ruby
   private def thread_stopper
      TracePoint.new(:line) do
        # run on each thread
        tc = ThreadClient.current
        next if tc.management?
        next unless tc.running?
        next if tc == @tc
        STDERR.puts caller.inspect if tc.thread.name.match?(/Timeout/)

        tc.on_pause
      end
    end
```
Anyway, I assume that it is difficult to fix in theory.
> Is it difficult to repro within a single file? (without puma)

I tried it but didn't succeed. I suspect running it from a thread pool is one criteria and will experiment it later.

> Can you show the backtrace when thread_stopper's TP stoppes?

```rb
"/Users/hung-wulo/.gem/ruby/3.1.2/gems/timeout-0.3.1/lib/timeout.rb:112:in `block (2 levels) in create_timeout_thread'", 
"/Users/hung-wulo/.gem/ruby/3.1.2/gems/timeout-0.3.1/lib/timeout.rb:111:in `synchronize'", 
"/Users/hung-wulo/.gem/ruby/3.1.2/gems/timeout-0.3.1/lib/timeout.rb:111:in `block in create_timeout_thread'"
```

> Anyway, I assume that it is difficult to fix in theory.

Yeah I assumed that to be the case. 

In the short-term I'll try to skip the Timeout thread pause to workaround it. Because many code paths in our app is wrapped inside a timeout block (from libs), and our devs constantly hang in those code paths. 
Sorry for the delay, I managed to reproduce this with a single Ruby file:

```rb
# test.rb
def timeout
  Timeout.timeout(3) do
    puts "Something that needs timeout"
  end
end

Thread.new do
  Timeout.timeout(100) do
    binding.b
  end
end

# just to prevent the main thread from existing too early
sleep 100
```

```
$ rdbg -n test.rb

[4, 13] in test.rb
     4|   end
     5| end
     6| 
     7| Thread.new do
     8|   Timeout.timeout(100) do
=>   9|     binding.b
    10|   end
    11| end
    12| 
    13| # just to prevent the main thread from existing too early
=>#0    block in <main> (2 levels) at test.rb:9
  #1    block {|exc=#<Timeout::Error: Timeout::Error>|} in timeout at ~/.gem/ruby/3.1.2/gems/timeout-0.3.1/lib/timeout.rb:199
  # and 5 frames (use `bt' command for all frames)
(rdbg) timeout
Something that needs timeout
nil
(rdbg) timeout
# hangs
```

It produces like 9 out of 10 times.


And if it helps, here's the output with some debugging messages from my patch in `timeout`:

```
$ rdbg -n test.rb
QUEUE_MUTEX locked? false
[Timeout.timeout] acquiring QUEUE_MUTEX
[Timeout.timeout] released QUEUE_MUTEX
QUEUE_MUTEX locked? false
[Timeout thread] acquiring QUEUE_MUTEX
[4, 13] in test.rb
     4|   end
     5| end
     6| 
     7| Thread.new do
     8|   Timeout.timeout(100) do
=>   9|     binding.b
    10|   end
    11| end
    12| 
    13| # just to prevent the main thread from existing too early
=>#0    block in <main> (2 levels) at test.rb:9
  #1    block {|exc=#<Timeout::Error: Timeout::Error>|} in timeout at ~/.gem/ruby/3.1.2/gems/timeout-0.3.1/lib/timeout.rb:199
  # and 5 frames (use `bt' command for all frames)
(rdbg) timeout
QUEUE_MUTEX locked? false
[Timeout.timeout] acquiring QUEUE_MUTEX
[Timeout.timeout] released QUEUE_MUTEX
QUEUE_MUTEX locked? false
Something that needs timeout
nil
(rdbg) timeout
QUEUE_MUTEX locked? true
[Timeout.timeout] acquiring QUEUE_MUTEX
```

I think the main cause is that the [timeout thread](https://github.com/ruby/timeout/blob/1415dd1620d75d23ded0b0292ee91e3712f9e962/lib/timeout.rb#L101-L122) would [acquire the `QUEUE_MUTEX` and never let go](https://github.com/ruby/timeout/blob/master/lib/timeout.rb#L111), likely because the thread would be frozen by then. So the later `Timeout.timeout` calls wouldn't be able to [acquire the lock for synchronize](https://github.com/ruby/timeout/blob/1415dd1620d75d23ded0b0292ee91e3712f9e962/lib/timeout.rb#L184-L187).

But I don't understand why the first `timeout` call can still acquire the lock and it only deadlocks in the second call 🤔 


The same problem doesn't seem to happen to `byebug`, even though it stops all threads as well.
It seems to use [C-level APIs like `rb_thread_stop`](https://github.com/deivid-rodriguez/byebug/blob/master/ext/byebug/threads.c#L138) for thread-control. I wonder if that'll be the solution for `debug` too?
Now I understand the problem.

Simplified code:

```ruby
th0 = Thread.new{sleep}
m = Mutex.new
th1 = Thread.new do
  m.lock
  sleep 1
  m.unlock
end

sleep 0.5
debugger do: 'm.lock'
```

In this case, 

(1) `m` is locked by `th1`
(2) main thread entered the REPL
(3) REPL runs `m.lock` but `th1` doesn't resumed, and `m.unlock` is not called forever.
(*) Because of `th0`, deadlock detector doesn't raise an error because `th0` can make progress.

----

* Solution 1: Restart all threads for the evaluation on REPL.
* Solution 2: Do not stop all threads except the running thread (in this case, main thread).
* Solution 3: Make timeout thread exceptional (do not stop)

(Solution 3) doesn't make sense because there are many similar cases.
(Solution 2) maybe make it difficult to debug thread programs.

So Solution 1? In fact the earlier versions restart all threads.



Fix (patch):
diff --git a/lib/debug/session.rb b/lib/debug/session.rb
index ffd333a37..301dc96e2 100644
--- a/lib/debug/session.rb
+++ b/lib/debug/session.rb
@@ -345,7 +345,7 @@ def process_event evt
           opt = ev_args[3]
           add_tracer ObjectTracer.new(@ui, obj_id, obj_inspect, **opt)
         else
-          # ignore
+          stop_all_threads
         end
 
         wait_command_loop
@@ -902,13 +902,13 @@ def register_default_command
       # * `p <expr>`
       #   * Evaluate like `p <expr>` on the current frame.
       register_command 'p' do |arg|
-        request_tc [:eval, :p, arg.to_s]
+        request_eval :p, arg.to_s
       end
 
       # * `pp <expr>`
       #   * Evaluate like `pp <expr>` on the current frame.
       register_command 'pp' do |arg|
-        request_tc [:eval, :pp, arg.to_s]
+        request_eval :pp, arg.to_s
       end
 
       # * `eval <expr>`
@@ -919,7 +919,7 @@ def register_default_command
           @ui.puts "\nTo evaluate the variable `#{cmd}`, use `pp #{cmd}` instead."
           :retry
         else
-          request_tc [:eval, :call, arg]
+          request_eval :call, arg
         end
       end
 
@@ -930,7 +930,7 @@ def register_default_command
           @ui.puts "not supported on the remote console."
           :retry
         end
-        request_tc [:eval, :irb]
+        request_eval :irb, nil
       end
 
       ### Trace
@@ -1150,7 +1150,7 @@ def process_command line
         @repl_prev_line = nil
         check_unsafe
 
-        request_tc [:eval, :pp, line]
+        request_eval :pp, line
       end
 
     rescue Interrupt
@@ -1166,6 +1166,11 @@ def process_command line
       return :retry
     end
 
+    def request_eval type, src
+      restart_all_threads
+      request_tc [:eval, type, src]
+    end
+
     def step_command type, arg
       if type == :until
         leave_subsession [:step, type, arg]
diff --git a/lib/debug/thread_client.rb b/lib/debug/thread_client.rb
index cd79a3205..cbecdf54d 100644
--- a/lib/debug/thread_client.rb
+++ b/lib/debug/thread_client.rb
@@ -1230,7 +1230,15 @@ def wait_next_action_
     rescue SuspendReplay, SystemExit, Interrupt
       raise
     rescue Exception => e
-      pp ["DEBUGGER Exception: #{__FILE__}:#{__LINE__}", e, e.backtrace]
+      STDERR.puts e.cause.inspect
+      STDERR.puts e.inspect
+      Thread.list.each{|th|
+        STDERR.puts "@@@ #{th}"
+        th.backtrace.each{|b|
+          STDERR.puts " > #{b}"
+        }
+      }
+      p ["DEBUGGER Exception: #{__FILE__}:#{__LINE__}", e, e.backtrace]
       raise
     ensure
       @returning = false