Repository: ruby/rubygems (Ruby)

Issue:
RubyGems attempts to activate multiple versions of net-http
### Describe the problem as clearly as you can

With the addition of the `net-http` dependency in https://github.com/lostisland/faraday-net_http/pull/31 and release of `faraday-net_http`, our build pipelines started to fail with this error:

```
/usr/local/lib/ruby/3.1.0/rubygems/specification.rb:2284:in `check_version_conflict': can't activate net-http-0.1.1, already activated net-http-0.3.0 (Gem::LoadError)
	from /usr/local/lib/ruby/3.1.0/rubygems/specification.rb:1421:in `activate'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:119:in `block in require'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:118:in `each'
	from <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:118:in `require'
```

### Did you try upgrading RubyGems?

Yes.

### Post steps to reproduce the problem

I've distilled the failure into these steps:

1. First, create this `Gemfile`:

```ruby
source 'https://rubygems.org'

source 'https://packagecloud.io/cinc-project/stable' do
  gem 'chef', '~> 17.10.95'
  gem 'chef-utils', '= 17.10.95'
end

gem 'faraday', '1.10.3'
```

2. Then run:

```shell
docker run -it -v $(pwd):/build ruby:3.1 bash
cd /build
gem install net-http:0.1.1 faraday
bundle install
ruby -r "chef" -r 'net/http' -e "puts 1"
```

In this example, we have multiple versions of `net-http` and `faraday` installed:

```
# gem list net-http faraday

*** LOCAL GEMS ***

net-http (default: 0.3.0, 0.1.1)

*** LOCAL GEMS ***

faraday (2.9.0, 1.10.3)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-follow_redirects (0.3.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (3.1.0, 1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
```

I see two attempts to require `net/http`:

1. https://github.com/rubygems/rubygems/blob/8ffa73df3a250623b13c5bf7ab48ed8b8e84e46f/lib/rubygems/core_ext/kernel_require.rb#L60
2. https://github.com/rubygems/rubygems/blob/8ffa73df3a250623b13c5bf7ab48ed8b8e84e46f/lib/rubygems/core_ext/kernel_require.rb#L98

On the second attempt, RubyGems attempts to load `net-http` 0.1.1 instead of 0.3.0, and this is where it fails.

When `require 'chef'` runs above, it pulls in a number of other `require` statements, but there are a few unresolved dependencies (i.e. those that don't have an explicit `require`):

* `rexml`
* `faraday`

The first `require 'net/http'` seems to pull in 0.3.0 just fine.

However, in the second attempt, for each unresolved dependency, RubyGems attempts to [traverse the dependency tree in reverse](https://github.com/rubygems/rubygems/blob/v3.2.26/lib/rubygems/specification.rb#L2545-L2572) and activate them automatically **if the dependency includes `net/http` in the path**. For `faraday`, this means it will attempt to load `net-http` -> `faraday-net_http` -> `faraday`.

To do this, it tries to find all the gemspecs that satisfy the requirement: https://github.com/rubygems/rubygems/blob/8ffa73df3a250623b13c5bf7ab48ed8b8e84e46f/lib/rubygems/specification.rb#L2618

For `net-http`, this means that both 0.3.0 and 0.1.1 match: https://github.com/rubygems/rubygems/blob/8ffa73df3a250623b13c5bf7ab48ed8b8e84e46f/lib/rubygems/dependency.rb#L275-L282

If bundler were used, only one would matching gem.

It appears that for some reason, while recursively building up the trail, RubyGems constructs multiple paths for `net-http` -> `faraday-net_http` -> `faraday`. One includes `net-http` 0.3.0, and another includes `net-http` 0.1.1.

### Which command did you run?

See above. Running `ruby -r "chef" -r 'net/http' -e "puts 1"` results in the error.

### What were you expecting to happen?

No error. I would expect RubyGems to load only one version of `net-http`, but maybe this use case is why Bundler is needed.

### What actually happened?

See above. We can use `bundler` or use different paths to install gems, but it was quite a surprise that this simple dependency change broke in this way.

### Run `gem env` and paste the output below

```
RubyGems Environment:
  - RUBYGEMS VERSION: 3.5.4
  - RUBY VERSION: 3.1.4 (2023-03-30 patchlevel 223) [aarch64-linux]
  - INSTALLATION DIRECTORY: /usr/local/bundle
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/3.1.0
  - RUBY EXECUTABLE: /usr/local/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/local/bundle/bin
  - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /usr/local/etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - aarch64-linux
  - GEM PATHS:
     - /usr/local/bundle
     - /root/.local/share/gem/ruby/3.1.0
     - /usr/local/lib/ruby/gems/3.1.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
     - "install" => "--no-document"
     - "update" => "--no-document"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/bundle/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin
```


Discussion:
Thanks for the clean report!

It took me a while to figure it out, but I think #7379 should address it. Still working on a proper test and just opened a WIP PR to check with CI.

The culprit was [this guy](https://github.com/chef/chef/blob/4d66a5dc552a37f856dc59516a31d629d44a4769/lib/chef/http.rb#L29), plus a bunch of other conditions that I'm trying to simulate in a test.
@deivid-rodriguez Thanks so much!
I think I got a proper fix with regression test now!



Patch:
diff --git a/lib/rubygems.rb b/lib/rubygems.rb
index 47b1ce69d266..ad7ab1075636 100644
--- a/lib/rubygems.rb
+++ b/lib/rubygems.rb
@@ -1216,6 +1216,13 @@ def register_default_spec(spec)
     ##
     # Find a Gem::Specification of default gem from +path+
 
+    def find_default_spec(path)
+      @path_to_default_spec_map[path]
+    end
+
+    ##
+    # Find an unresolved Gem::Specification of default gem from +path+
+
     def find_unresolved_default_spec(path)
       default_spec = @path_to_default_spec_map[path]
       default_spec if default_spec && loaded_specs[default_spec.name] != default_spec
diff --git a/lib/rubygems/core_ext/kernel_require.rb b/lib/rubygems/core_ext/kernel_require.rb
index bbd7852e92f8..073966b696cc 100644
--- a/lib/rubygems/core_ext/kernel_require.rb
+++ b/lib/rubygems/core_ext/kernel_require.rb
@@ -39,7 +39,14 @@ def require(path) # :doc:
     RUBYGEMS_ACTIVATION_MONITOR.synchronize do
       path = File.path(path)
 
-      if spec = Gem.find_unresolved_default_spec(path)
+      # If +path+ belongs to a default gem, we activate it and then go straight
+      # to normal require
+
+      if spec = Gem.find_default_spec(path)
+        name = spec.name
+
+        next if Gem.loaded_specs[name]
+
         # Ensure -I beats a default gem
         resolved_path = begin
           rp = nil
@@ -57,8 +64,10 @@ def require(path) # :doc:
           rp
         end
 
-        Kernel.send(:gem, spec.name, Gem::Requirement.default_prerelease) unless
+        Kernel.send(:gem, name, Gem::Requirement.default_prerelease) unless
           resolved_path
+
+        next
       end
 
       # If there are no unresolved deps, then we can use just try


Test patch:
diff --git a/test/rubygems/test_require.rb b/test/rubygems/test_require.rb
index fc5f219490d7..f4dbfc477fff 100644
--- a/test/rubygems/test_require.rb
+++ b/test/rubygems/test_require.rb
@@ -529,6 +529,65 @@ def test_default_gem_prerelease
     assert_equal %w[default-3.0.0.rc2], loaded_spec_names
   end
 
+  def test_default_gem_with_unresolved_gems_depending_on_it
+    net_http_old = util_spec "net-http", "0.1.1", nil, "lib/net/http.rb"
+    install_gem net_http_old
+
+    net_http_default = new_default_spec "net-http", "0.3.0", nil, "net/http.rb"
+    install_default_gems net_http_default
+
+    faraday_1 = util_spec "faraday", "1", { "net-http" => ">= 0" }
+    install_gem faraday_1
+
+    faraday_2 = util_spec "faraday", "2", { "net-http" => ">= 0" }
+    install_gem faraday_2
+
+    chef = util_spec "chef", "1", { "faraday" => [">= 1", "< 3"] }, "lib/chef.rb"
+    install_gem chef
+
+    assert_require "chef"
+    assert_require "net/http"
+  end
+
+  def test_default_gem_required_circulary_with_unresolved_gems_depending_on_it
+    net_http_old = util_spec "net-http", "0.1.1", nil, "lib/net/http.rb"
+    install_gem net_http_old
+
+    net_http_default = new_default_spec "net-http", "0.3.0", nil, "net/http.rb"
+    net_http_default_path = File.join(@tempdir, "default_gems", "lib", "net/http.rb")
+    install_default_gems net_http_default
+    File.write(net_http_default_path, 'require "net/http"')
+
+    faraday_1 = util_spec "faraday", "1", { "net-http" => ">= 0" }
+    install_gem faraday_1
+
+    faraday_2 = util_spec "faraday", "2", { "net-http" => ">= 0" }
+    install_gem faraday_2
+
+    chef = util_spec "chef", "1", { "faraday" => [">= 1", "< 3"] }, "lib/chef.rb"
+    install_gem chef
+
+    assert_require "chef"
+
+    out, err = capture_output do
+      assert_require "net/http"
+    end
+
+    assert_empty out
+
+    circular_require_warning = false
+
+    err_lines = err.split("\n").reject do |line|
+      if line.include?("circular require")
+        circular_require_warning = true
+      elsif circular_require_warning # ignore backtrace lines for circular require warning
+        circular_require_warning = line.start_with?(/[\s]/)
+      end
+    end
+
+    assert_empty err_lines
+  end
+
   def loaded_spec_names
     Gem.loaded_specs.values.map(&:full_name).sort
   end