Repository: Shopify/ruby-lsp (Ruby)

Issue:
setup_bundler's `rails_app?` incompatible with non-ascii characters
### Description

### Reproduction steps
Having the config/application.rb with Japanese characters (eg: あ)

In https://github.com/Shopify/ruby-lsp/blob/88b85b810061aafd7d8d7078df6b5aa713979e54/test/fixtures/rails_application.rb add `あ`
```ruby
module MyApp
  class Application < Rails::Application
    # あ
  end
end
```

### Details

From this change https://github.com/Shopify/ruby-lsp/pull/2218 ruby-lsp error out on 
https://github.com/Shopify/ruby-lsp/blob/88b85b810061aafd7d8d7078df6b5aa713979e54/lib/ruby_lsp/setup_bundler.rb#L292

Ruby-lsp: 0.17.14
Ruby: 3.3.4
Rails: 6.1.7.8


Error: `setup_bundler.rb:295:in 'match?': invalid byte sequence in US-ASCII (ArgumentError)`
Full output:
```
2024-08-20 17:32:33.321 [info] (app) Checking if chruby is available on the path with command: /bin/zsh -i -c 'chruby --version'
2024-08-20 17:32:35.982 [info] (app) Checking if rbenv is available on the path with command: /bin/zsh -i -c 'rbenv --version'
2024-08-20 17:32:36.935 [info] (app) Discovered version manager rbenv
2024-08-20 17:32:36.935 [info] (app) Running command: `rbenv exec ruby -W0 -rjson -e 'STDERR.print({env: ENV.to_h,yjit:!!defined?(RubyVM::YJIT),version:RUBY_VERSION}.to_json)'` in /Users/thomas/projects/app using shell: /bin/zsh
2024-08-20 17:32:38.382 [info] (app) /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:295:in `match?': invalid byte sequence in US-ASCII (ArgumentError)

      /class .* < (::)?Rails::Application/.match?(application_contents)
                                                  ^^^^^^^^^^^^^^^^^^^^
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:295:in `rails_app?'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `bind_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `validate_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/lib/ruby_lsp/setup_bundler.rb:53:in `initialize'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `bind_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/call_validation.rb:270:in `validate_call'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/sorbet-runtime-0.5.11528/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/exe/ruby-lsp:67:in `new'
	from /Users/thomas/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/ruby-lsp-0.17.14/exe/ruby-lsp:67:in `<top (required)>'
	from /Users/thomas/.rbenv/versions/3.3.4/bin/ruby-lsp:25:in `load'
	from /Users/thomas/.rbenv/versions/3.3.4/bin/ruby-lsp:25:in `<main>'

2024-08-20 17:32:38.390 [info] (app) [Error - 5:32:38 PM] Server process exited with code 1.
2024-08-20 17:32:38.391 [info] (app) [Error - 5:32:38 PM] Server initialization failed.
```

Using `application_contents.force_encoding("UTF-8")` solve the problem.


Discussion:
@D-system what does the following return in your app?

`file -mime config/application.rb`
I've also started seeing this error. My application.rb file has a single `’` non-ASCII character that has been there for some time:

```ruby
# Override Rails’ default public exception handling
config.exceptions_app = self.routes
```

```
$ file -mime config/application.rb
config/application.rb: Ruby script text, Unicode text, UTF-8 text
```
Can you show the output of `Encoding.default_external` and `Encoding.default_internal` in irb?
This problem doesn't seem to be isolated to multibyte characters. There's some encoding issue in play here. Adding Japanese or other multibyte characters to the fixture still passes the test because the file contents are read as UTF-8.

Even forcing the file encoding to be US-ASCII passes the test.
I created a revert https://github.com/Shopify/ruby-lsp/pull/2466 while we're trying to understand what's going on, since this blocks people from using the LSP.
If you run `RUBYOPT="--encoding US-ASCII" bundle exec ruby -Itest ./test/setup_bundler_test.rb --name 
SetupBundlerTest#test_ruby_lsp_rails_is_automatically_included_in_rails_apps`, the test will fail (if it contains non-ascii characters). Though, I'm not sure if forcing the encoding to UTF8 when reading the file would be the correct solution. 
It also fails with `LANG=anything bundle exec ruby -Itest ./test/setup_bundler_test.rb --name  SetupBundlerTest#test_ruby_lsp_rails_is_automatically_included_in_rails_apps`, so I'm wondering if something in the environment is setting that. 🤔 
https://github.com/Shopify/ruby-lsp/pull/2469 should solve this but I'd still like to understand why if/why `LANG` being changed.
```shell
$ echo $LANG                                                                               
en_US.UTF-8
```
@andyw8 
> what does the following return in your app?

```
❯ file -mime config/application.rb
config/application.rb: Ruby script text, Unicode text, UTF-8 text
```
> [#2469](https://github.com/Shopify/ruby-lsp/pull/2469) should solve this but I'd still like to understand why if/why `LANG` being changed.

I tried it with 0.17.4 and yes, it solved the issue.
And the 0.17.4 was able to update to 0.17.5 and runs great. Thanks for the reactivity.
Closing the ticket.

So the issue depends on the var env `LANG` which Rails solved the problem by forcing the encoding to UTF-8 https://github.com/rails/rails/blob/v7.1.3.3/railties/lib/rails.rb#L21-L25
> Can you show the output of `Encoding.default_external` and `Encoding.default_internal` in irb?

```
irb(main):001> Encoding.default_external
=> #<Encoding:UTF-8>
irb(main):002> Encoding.default_internal
=> nil
```
FYI: maintenance task had the same issue from ActiveStorage that reads CSV files in ASCII-8bit/binary even that file is text with UTF-8 encoding at upload time.
https://github.com/Shopify/maintenance_tasks/pull/1025



Patch:
diff --git a/lib/ruby_lsp/setup_bundler.rb b/lib/ruby_lsp/setup_bundler.rb
index b8a5eacc56..ca10ea7ced 100644
--- a/lib/ruby_lsp/setup_bundler.rb
+++ b/lib/ruby_lsp/setup_bundler.rb
@@ -286,7 +286,7 @@ def correct_relative_remote_paths
     sig { returns(T::Boolean) }
     def rails_app?
       config = Pathname.new("config/application.rb").expand_path
-      application_contents = config.read if config.exist?
+      application_contents = config.read(external_encoding: Encoding::UTF_8) if config.exist?
       return false unless application_contents
 
       /class .* < (::)?Rails::Application/.match?(application_contents)
diff --git a/sorbet/rbi/shims/pathname.rbi b/sorbet/rbi/shims/pathname.rbi
new file mode 100644
index 0000000000..774bc090be
--- /dev/null
+++ b/sorbet/rbi/shims/pathname.rbi
@@ -0,0 +1,16 @@
+# typed: false
+
+class Pathname
+  sig do
+    params(
+      external_encoding: T.any(String, Encoding),
+      internal_encoding: T.any(String, Encoding),
+      encoding: T.any(String, Encoding),
+      textmode: BasicObject,
+      binmode: BasicObject,
+      autoclose: BasicObject,
+      mode: String,
+    ).returns(String)
+  end
+  def read(external_encoding: T.unsafe(nil), internal_encoding: T.unsafe(nil), encoding: T.unsafe(nil), textmode: T.unsafe(nil), binmode: T.unsafe(nil), autoclose: T.unsafe(nil), mode: T.unsafe(nil)); end
+end


Test patch:
diff --git a/test/fixtures/rails_application.rb b/test/fixtures/rails_application.rb
index 27eff561a4..cb43f7bf47 100644
--- a/test/fixtures/rails_application.rb
+++ b/test/fixtures/rails_application.rb
@@ -1,4 +1,5 @@
 module MyApp
   class Application < Rails::Application
+    # あ
   end
 end
diff --git a/test/setup_bundler_test.rb b/test/setup_bundler_test.rb
index afe3d39104..c360a23856 100644
--- a/test/setup_bundler_test.rb
+++ b/test/setup_bundler_test.rb
@@ -506,6 +506,39 @@ def test_ruby_lsp_rails_is_automatically_included_in_rails_apps
     end
   end
 
+  def test_ruby_lsp_rails_detection_handles_lang_from_environment
+    with_default_external_encoding("us-ascii") do
+      Dir.mktmpdir do |dir|
+        FileUtils.mkdir("#{dir}/config")
+        FileUtils.cp("test/fixtures/rails_application.rb", "#{dir}/config/application.rb")
+        Dir.chdir(dir) do
+          File.write(File.join(dir, "Gemfile"), <<~GEMFILE)
+            source "https://rubygems.org"
+            gem "rails"
+          GEMFILE
+
+          capture_subprocess_io do
+            Bundler.with_unbundled_env do
+              # Run bundle install to generate the lockfile
+              system("bundle install")
+            end
+          end
+
+          Object.any_instance.expects(:system).with(
+            bundle_env(dir, ".ruby-lsp/Gemfile"),
+            "(bundle check || bundle install) 1>&2",
+          ).returns(true)
+          Bundler.with_unbundled_env do
+            run_script(dir)
+          end
+
+          assert_path_exists(".ruby-lsp/Gemfile")
+          assert_match('gem "ruby-lsp-rails"', File.read(".ruby-lsp/Gemfile"))
+        end
+      end
+    end
+  end
+
   def test_recovers_from_stale_lockfiles
     Dir.mktmpdir do |dir|
       custom_dir = File.join(dir, ".ruby-lsp")
@@ -575,6 +608,30 @@ def test_recovers_from_stale_lockfiles
 
   private
 
+  def with_default_external_encoding(encoding, &block)
+    ignore_warnings do
+      original_encoding = Encoding.default_external
+      begin
+        Encoding.default_external = encoding
+        block.call
+      ensure
+        Encoding.default_external = original_encoding
+      end
+    end
+  end
+
+  def ignore_warnings(&block)
+    # Since overwriting the encoding emits a warning
+    previous = $VERBOSE
+    $VERBOSE = nil
+
+    begin
+      block.call
+    ensure
+      $VERBOSE = previous
+    end
+  end
+
   # This method runs the script and then immediately unloads it. This allows us to make assertions against the effects
   # of running the script multiple times
   def run_script(path = Dir.pwd, expected_path: nil, **options)