Repository: Shopify/ruby-lsp-rails (Ruby)

Issue:
Ruby LSP fails to boot due to an error in Runner Client Ruby LSP Rails failed to initialize server: unexpected token at ''
### Description
Unable to get Ruby LSP client start due to an error in boot cycle w/ Runner client Ruby, this then switches to the noop/null client, not allowing cooler functionalities such as peak definition, etc.

I've done some debugging, and you can see the details in the sections after the stack traces.

### Stack Trace w/ General Public Release

```
2024-03-23 17:57:42.278 [info] (my-code-base) Ruby LSP> Skipping custom bundle setup since /home/my-code-base/.ruby-lsp/Gemfile.lock already exists and is up to date
2024-03-23 17:57:42.328 [info] (my-code-base) Ruby LSP> Running bundle install for the custom bundle. This may take a while...
Ruby LSP> Command: (bundle check || bundle install) 1>&2
2024-03-23 17:57:43.531 [info] (my-code-base) The Gemfile's dependencies are satisfied
2024-03-23 17:57:47.302 [info] (my-code-base) Starting Ruby LSP v0.14.6...
2024-03-23 17:57:47.342 [info] (my-code-base) Ruby LSP is ready
2024-03-23 17:57:47.343 [info] (my-code-base) Ruby LSP Rails booting server
2024-03-23 17:57:56.055 [info] (my-code-base) Ruby LSP Rails failed to initialize server: unexpected token at ''
/home/bento/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb:219:in `parse'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb:219:in `parse'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:138:in `read_response'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:62:in `initialize'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `new'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `create_client'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/addon.rb:29:in `block in activate'

2024-03-23 17:57:56.055 [info] (my-code-base) Server dependent features will not be available
```

### Stack Trace w/ Pre-release version

```
024-03-23 18:00:55.884 [info] (my-code-base) Discovered Ruby version 3.3.0
2024-03-23 18:00:55.885 [info] (my-code-base) Using cached Ruby path: /home/bhrgv/.rbenv/versions/3.3.0/bin
2024-03-23 18:00:55.973 [info] (my-code-base) Activated Ruby environment: gem_home=/home/bhrgv/.local/share/gem/ruby/3.3.0, version=3.3.0, yjit=null, gem_root=/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0
2024-03-23 18:00:57.032 [info] (my-code-base) Ruby LSP> Skipping custom bundle setup since /home/bhrgv/my-code-base/.ruby-lsp/Gemfile.lock already exists and is up to date

2024-03-23 18:00:57.071 [info] (my-code-base) Ruby LSP> Running bundle install for the custom bundle. This may take a while...
Ruby LSP> Command: (bundle check || bundle install) 1>&2

2024-03-23 18:00:58.313 [info] (my-code-base) https://github.com/mlarraz/dalli.git (at safe_get@0633e16) is not yet checked
out. Run `bundle install` first.

2024-03-23 18:01:00.514 [info] (my-code-base) Fetching gem metadata from https://rubygems.org/
2024-03-23 18:01:00.514 [info] (my-code-base) .
2024-03-23 18:01:01.435 [info] (my-code-base) .
2024-03-23 18:01:01.796 [info] (my-code-base) .
2024-03-23 18:01:02.508 [info] (my-code-base) .
2024-03-23 18:01:02.546 [info] (my-code-base) .
2024-03-23 18:01:02.553 [info] (my-code-base) .
2024-03-23 18:01:02.559 [info] (my-code-base) .
2024-03-23 18:01:06.227 [info] (my-code-base) 

2024-03-23 18:01:06.230 [info] (my-code-base) Fetching https://github.com/mlarraz/dalli.git

2024-03-23 18:01:07.184 [info] (my-code-base) Fetching https://github.com/instacart/sidekiq-unique-jobs.git

2024-03-23 18:01:08.060 [info] (my-code-base) Fetching https://github.com/iancanderson/ingreedy.git

2024-03-23 18:01:08.437 [info] (my-code-base) Fetching gem metadata from https://gems.contribsys.com/
2024-03-23 18:01:08.437 [info] (my-code-base) .
2024-03-23 18:01:08.455 [info] (my-code-base) .
2024-03-23 18:01:08.494 [info] (my-code-base) 

2024-03-23 18:01:08.616 [info] (my-code-base) Fetching https://github.com/coinbase/temporal-ruby.git

2024-03-23 18:01:09.981 [info] (my-code-base) Fetching gem metadata from https://rubygems.pkg.github.com/instacart/
2024-03-23 18:01:09.981 [info] (my-code-base) .
2024-03-23 18:01:10.131 [info] (my-code-base) .
2024-03-23 18:01:10.152 [info] (my-code-base) 

2024-03-23 18:01:10.939 [info] (my-code-base) Bundle complete! 227 Gemfile dependencies, 519 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

2024-03-23 18:01:14.919 [info] (my-code-base) Starting Ruby LSP v0.14.6...

2024-03-23 18:01:14.960 [info] (my-code-base) Ruby LSP is ready

2024-03-23 18:01:14.962 [info] (my-code-base) Ruby LSP Rails booting server

2024-03-23 18:01:24.207 [info] (my-code-base) Ruby LSP Rails failed to initialize server: unexpected token at ''
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb:219:in `parse'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/3.3.0/json/common.rb:219:in `parse'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:138:in `read_response'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:62:in `initialize'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `new'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `create_client'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `bind_call'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11288/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/home/bhrgv/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/addon.rb:29:in `block in activate'

2024-03-23 18:01:24.316 [info] (my-code-base) Server dependent features will not be available

```

### Observation
Looking at the code, it seems the issue is that Runner client is sending and expecting JSON data to be returned back but is getting back an empty string, which causes JSON.parse to fail.

Interestingly, if I just start `ruby-lsp` manually in my project via terminal (I did not install the gem myself and it was installed through the extension), it boots.

```
$ ruby-lsp
Ruby LSP> Skipping custom bundle setup since /home/bento/carrot/customers/customers-backend/.ruby-lsp/Gemfile.lock already exists and is up to date
Ruby LSP> Running bundle install for the custom bundle. This may take a while...
Ruby LSP> Command: ((bundle check && bundle update ruby-lsp ruby-lsp-rails) || bundle install) 1>&2
The Gemfile's dependencies are satisfied
Fetching gem metadata from https://rubygems.pkg.github.com/instacart/..
Fetching gem metadata from https://gems.contribsys.com/..
Fetching gem metadata from https://rubygems.org/.......
Resolving dependencies...
Bundler attempted to update ruby-lsp but its version stayed the same
Bundler attempted to update ruby-lsp-rails but its version stayed the same
Bundle updated!
Starting Ruby LSP v0.14.6...
```

Going further, it looks like the failure is happening due to ruby-lsp-rails. I checked the runnr client and saw that it POpens a process and communicates it. Running that server manually, I can see that it boots as expected.

```bash
$ ./bin/rails runner /home/bento/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/server.rb start

# bunch of output from rails booting
Content-Length: 27
{"result":{"message":"ok"}}
```

### Reproduction steps
1. **Operating System**
2. **Ruby Version**: 3.3.0
3. **Ruby Version Manager**: rbenv
4. **Editor**: VSCode running on remote (extension is installed on remote as well)

I nuked all my local extentions, as well as the remote server and re-installed it. Even the remote machine is newly provisioned.

LSP server fails while indexing files, but does not indicate which file failed
### Description

I installed the Ruby LSP extension in VS Cde
Launching Lsp-server gives the following error:

```
LSP Rails failed to initialize server: unexpected token at ''
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/json-2.7.1/lib/json/common.rb:219:in `parse'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/json-2.7.1/lib/json/common.rb:219:in `parse'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:138:in `read_response'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `bind_call'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:62:in `initialize'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `bind_call'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `new'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb:16:in `create_client'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `bind_call'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/sorbet-runtime-0.5.11311/lib/types/private/methods/_methods.rb:279:in `block in _on_method_added'
/Users/my_user/.rbenv/versions/3.2.1/lib/ruby/gems/3.2.0/gems/ruby-lsp-rails-0.3.3/lib/ruby_lsp/ruby_lsp_rails/addon.rb:29:in `block in activate'
Server dependent features will not be available
```

As a test I created an empty Rails project, and Lsp-server correctly indexes the project

Been looking for a way to get more verbose output, especially to get the path of the file that LSP fails to index. Could you help me to do so?

Thanks!




Discussion: Hi @bhargavrpatel, this should be solved by https://github.com/Shopify/ruby-lsp-rails/pull/310 once released.

Hi @bbouchet-trustpair,

From the logs this doesn't look related to indexing.

If you run `bin/rails console` does it start successfully?
Hey @andyw8 thanks for your reply
Yes rails' console (and server), run successfully.

(transferred to ruby-lsp-rails)
Do you have anywhere that prints to `stdout` while the app boots, e.g. a `puts` statement?
> Do you have anywhere that prints to `stdout` while the app boots, e.g. a `puts` statement?

Yes, there are a few `puts` statements. Is it not compatible?
This should be solved by https://github.com/Shopify/ruby-lsp-rails/pull/310 once released.
Thank you @andyw8 



Fix (patch):
diff --git a/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb b/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb
index b8dce414..d5a1f64e 100644
--- a/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb
+++ b/lib/ruby_lsp/ruby_lsp_rails/runner_client.rb
@@ -30,6 +30,9 @@ def create_client
 
       class InitializationError < StandardError; end
       class IncompleteMessageError < StandardError; end
+      class EmptyMessageError < StandardError; end
+
+      MAX_RETRIES = 5
 
       extend T::Sig
 
@@ -59,7 +62,16 @@ def initialize
         @stdout.binmode # for Windows compatibility
 
         $stderr.puts("Ruby LSP Rails booting server")
-        read_response
+        count = 0
+
+        begin
+          count += 1
+          read_response
+        rescue EmptyMessageError
+          $stderr.puts("Ruby LSP Rails is retrying initialize (#{count})")
+          retry if count < MAX_RETRIES
+        end
+
         $stderr.puts("Finished booting Ruby LSP Rails server")
 
         unless ENV["RAILS_ENV"] == "test"
@@ -134,7 +146,10 @@ def read_response
         headers = @stdout.gets("\r\n\r\n")
         raise IncompleteMessageError unless headers
 
-        raw_response = @stdout.read(headers[/Content-Length: (\d+)/i, 1].to_i)
+        content_length = headers[/Content-Length: (\d+)/i, 1].to_i
+        raise EmptyMessageError if content_length.zero?
+
+        raw_response = @stdout.read(content_length)
         response = JSON.parse(T.must(raw_response), symbolize_names: true)
 
         if response[:error]