Repository: ruby/rubygems (Ruby)

Issue:
`gem install <gem-that-does-not-exist>` is very slow
### Describe the problem as clearly as you can

Not a huge issue but `gem install <gem-that-does-not-exist>` is very slow, unnecessarily.

### Did you try upgrading RubyGems?

Yes.

### Post steps to reproduce the problem

```
$ time sudo gem install --debug --verbose pony-gem
NOTE:  Debugging mode prints all exceptions even when rescued
HEAD https://index.rubygems.org/
200 OK
GET https://index.rubygems.org/info/pony-gem
404 Not Found
Exception `Gem::RemoteFetcher::FetchError' at /usr/local/lib/site_ruby/3.1.0/rubygems/remote_fetcher.rb:232 - bad response Not Found 404 (https://index.rubygems.org/info/pony-gem)
GET https://index.rubygems.org/prerelease_specs.4.8.gz
200 OK
GET https://index.rubygems.org/specs.4.8.gz
200 OK
# >>>>>>>>>>> IT HANGS HERE FOR A MINUTE <<<<<<<<<<<<
Exception `Gem::UnsatisfiableDependencyError' at /usr/local/lib/site_ruby/3.1.0/rubygems/resolver/installer_set.rb:82 - Unable to resolve dependency: user requested 'pony-gem (>= 0)'
ERROR:  Could not find a valid gem 'pony-gem' (>= 0) in any repository
GET https://rubygems.org/latest_specs.4.8.gz
200 OK
ERROR:  Possible alternatives: donsgem, moneygem, poc_gem, pokegem, pry-gem, tinygem

real    1m15.133s
user    0m0.010s
sys     0m0.021s
```

### Which command did you run?

`gem install` with a gem that does not exist.

### What were you expecting to happen?

After the gem is found to not exist in the compact index API, fail directly.

### What actually happened?

RubyGems fallback to trying to find the gem inside the full marshaled index, which is very slow. 



Discussion: It's not _as bad_ on my system, but it's 20-23 seconds for a nonexistent gem vs ~1s for an existent gem (in my case, I went with `time gem install --debug --verbose boop` since it has no dependencies).

---

Times for running `time gem install --debug --verbose pony-gem` on my FreeBSD 13.1 system, with different RubyGems versions.

All of these are using Ruby 3.2.1.

| Version | Time |
|---------|------|
| 3.3.26  | 22.445s |
| ...     |  |
| 3.4.16  | 21.279s |
| 3.4.17  | 21.369s |
| e2e2e7cff1 | 21.430s |

Compared to a gem that actually exists:

```shellsession
puppy@orthrus$ time gem install --debug --verbose boop                                                                                                                        NOTE:  Debugging mode prints all exceptions even when rescued
HEAD https://index.rubygems.org/                                                                                                                                                                                 200 OK
GET https://index.rubygems.org/info/boop                                                                                                                                                                         200 OK
[...]
Successfully installed boop-0.5.0
1 gem installed

real    0m1.078s
user    0m0.194s
sys     0m0.016s
puppy@orthrus$
```
From profiling with rbspy:

```
Time since start: 17s. Press Ctrl+C to stop.
Summary of profiling data so far:
% self  % total  name
 20.64    69.22  <=> - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/name_tuple.rb:95
 15.48    35.65  <=> [c function] - (unknown)
 14.96    20.23  <=> - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/version.rb:381
 12.93    12.93  sort_priority - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/platform.rb:52
  9.45    12.17  block in levenshtein_distance - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/text.rb:81
  4.23     5.10  load - <internal:marshal>:35
  4.17    73.39  sort [c function] - (unknown)
  2.72     2.72  min3 - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/text.rb:49
  2.09     2.09  == [c function] - (unknown)
  1.22     1.57  sort_by [c function] - (unknown)
  0.93     2.14  canonical_segments - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/version.rb:388
  0.81     0.81  === [c function] - (unknown)
  0.75     1.10  block (2 levels) in initialize - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/resolver/index_set.rb:29
  0.64    99.42  each [c function] - (unknown)
  0.64    82.26  new [c function] - (unknown)
  0.64     0.64  downcase [c function] - (unknown)
  0.58     0.70  initialize - /Users/segiddins/Development/github.com/rubygems/rubygems/lib/rubygems/name_tuple.rb:18
  0.58     0.58  scan [c function] - (unknown)
  0.52     0.52  wait_readable [c function] - (unknown)
  0.52     0.52  readline [c function] - (unknown)
```

It looks like sorting the name tuples (particularly the `:complete` ones) is the really slow part
This happens as well if you force rubygems to error on requests to the compact index api, I get a 17s successful install when I do that. Seems to be that it's the full index path that is very slow, rather than it being intrinsic to gems that dont exist.
Yep, it's the full index path that it's slow, and of course we could optimize that.

My point was more that if the compact index API returns 404, does it really make sense to check the full index? The fallback is meant for cases where the compact index is not implemented, down, or whatever, but in this case the compact index API is working just fine, so I don't think we need the fallback at all?
> Yep, it's the full index path that it's slow, and of course we could optimize that.
> 
> My point was more that if the compact index API returns 404, does it really make sense to check the full index? The fallback is meant for cases where the compact index is not implemented, down, or whatever, but in this case the compact index API is working just fine, so I don't think we need the fallback at all?

How does this work? Compact index API can return 404 for various reasons. For example when not implemented.
The signal for "not implemented" is that the `/versions` endpoint responds with 404. We will never request specific `/info` endpoints in that case.
> The signal for "not implemented" is that the `/versions` endpoint responds with 404. We will never request specific `/info` endpoints in that case.

Clear. Isn't the signal it returns anything other than success?

_We should definitely start documenting this gem/bundle commands server API needed._
Actually, I think 404 from `/versions` endpoint is how Bundler works, but I'm not fully sure how RubyGems work.

So my approach at #6990 may indeed be completely breaking the fallback, nice catch.

Another solution would be that `/info/<name>` on an server actually implementing the compact index API should not return 404, but 200 with some specific content, or with an empty body? Making it clear that the route exists and it's actually valid?
> Another solution would be that `/info/<name>` on an server actually implementing the compact index API should not return 404, but 200 with some specific content, or with an empty body? Making it clear that the route exists and it's actually valid?

Let's sync on this somewhere, we need to finally specify Compact index API behaviour and release final stable documentation for API "adopters".
This sounds interesting. Happy to help out when documenting since it would help me learn the API better.

I like the idea of returning something like a 204. It sounds odd at first to say "ok, no content" to a not found gem, but this url request is kind of like a query. You need a result for "successfully found no matches" that is separate from "wrong url 🤷‍♂️".
So I had another look into this and I think this is actually a bug introduced when RubyGems was migrated to use the Compact Index API. In https://github.com/rubygems/rubygems/commit/c9d157defb1783a7c5f5b890916cc3349dd1ffb7#diff-3a6811b74628a1ec350eb51728ad8c87e8110e8fb4d8938567dc50e9dcc0958eL83, the API path should've been updated from `./api/v1/dependencies` to `./versions` so that checking for availability of the API works like Bundler.

#8006 should fix this!



Fix (patch):
diff --git a/lib/rubygems/commands/install_command.rb b/lib/rubygems/commands/install_command.rb
index 2091634a2912..2888b6c55a83 100644
--- a/lib/rubygems/commands/install_command.rb
+++ b/lib/rubygems/commands/install_command.rb
@@ -224,10 +224,6 @@ def install_gems # :nodoc:
       rescue Gem::InstallError => e
         alert_error "Error installing #{gem_name}:\n\t#{e.message}"
         exit_code |= 1
-      rescue Gem::GemNotFoundException => e
-        show_lookup_failure e.name, e.version, e.errors, suppress_suggestions
-
-        exit_code |= 2
       rescue Gem::UnsatisfiableDependencyError => e
         show_lookup_failure e.name, e.version, e.errors, suppress_suggestions,
                             "'#{gem_name}' (#{gem_version})"
diff --git a/lib/rubygems/exceptions.rb b/lib/rubygems/exceptions.rb
index 0308b4687f9f..793324b875c7 100644
--- a/lib/rubygems/exceptions.rb
+++ b/lib/rubygems/exceptions.rb
@@ -104,9 +104,6 @@ class Gem::FormatException < Gem::Exception
 
 class Gem::GemNotFoundException < Gem::Exception; end
 
-##
-# Raised by the DependencyInstaller when a specific gem cannot be found
-
 class Gem::SpecificGemNotFoundException < Gem::GemNotFoundException
   ##
   # Creates a new SpecificGemNotFoundException for a gem with the given +name+
@@ -137,6 +134,8 @@ def initialize(name, version, errors=nil)
   attr_reader :errors
 end
 
+Gem.deprecate_constant :SpecificGemNotFoundException
+
 ##
 # Raised by Gem::Resolver when dependencies conflict and create the
 # inability to find a valid possible spec for a request.
diff --git a/lib/rubygems/resolver/best_set.rb b/lib/rubygems/resolver/best_set.rb
index a983f8c6b69f..57d0d0037576 100644
--- a/lib/rubygems/resolver/best_set.rb
+++ b/lib/rubygems/resolver/best_set.rb
@@ -29,10 +29,8 @@ def find_all(req) # :nodoc:
     pick_sets if @remote && @sets.empty?
 
     super
-  rescue Gem::RemoteFetcher::FetchError => e
-    replace_failed_api_set e
-
-    retry
+  rescue Gem::RemoteFetcher::FetchError
+    []
   end
 
   def prefetch(reqs) # :nodoc:
@@ -50,28 +48,4 @@ def pretty_print(q) # :nodoc:
       q.pp @sets
     end
   end
-
-  ##
-  # Replaces a failed APISet for the URI in +error+ with an IndexSet.
-  #
-  # If no matching APISet can be found the original +error+ is raised.
-  #
-  # The calling method must retry the exception to repeat the lookup.
-
-  def replace_failed_api_set(error) # :nodoc:
-    uri = error.original_uri
-    uri = Gem::URI uri unless Gem::URI === uri
-    uri += "."
-
-    raise error unless api_set = @sets.find do |set|
-      Gem::Resolver::APISet === set && set.dep_uri == uri
-    end
-
-    index_set = Gem::Resolver::IndexSet.new api_set.source
-
-    @sets.map! do |set|
-      next set unless set == api_set
-      index_set
-    end
-  end
 end
diff --git a/lib/rubygems/source.rb b/lib/rubygems/source.rb
index 4e5545982d64..bee5681dab29 100644
--- a/lib/rubygems/source.rb
+++ b/lib/rubygems/source.rb
@@ -79,7 +79,7 @@ def dependency_resolver_set # :nodoc:
       uri
     end
 
-    bundler_api_uri = enforce_trailing_slash(fetch_uri)
+    bundler_api_uri = enforce_trailing_slash(fetch_uri) + "versions"
 
     begin
       fetcher = Gem::RemoteFetcher.fetcher