rubygems/rubygems — pull request #435 by jonleighton (2013-01-25 19:03:39+00:00)
Title: Proof-of-concept for performance fix to avoid loading every gemspec

## Problem ##

Rubygems currently loads every single gemspec in basically all situations. This is slow and gets slower as you install more gems.

## Previous solutions ##

* #376 avoids evalling the gemspec by keeping a marshalled copy of it. This was deemed to be quite complex to get right in all edge cases, and didn't seem to show a significant improvement.
* #396 infers the gem name and version from the name of the gemspec. This data can then be used to compare gem versions before needing to load their gemspecs. This was deemed to be too brittle. #396 also only improves `Gem::Dependency#matching_specs`, which means that other code paths (for instance via `Kernel#require`) can still end up loading all gemspecs and therefore not being any faster.

## This solution ##

My patch is a proof of concept. It is not production ready but I want to get feedback on the approach first, and will then make all the code nice and pretty and non-hacky.

It's a continuation of #396, taking forward some of the suggestions in there. It also tackles the `Kernel#require` code path.

When a gem is installed, the gemspec is written out with the second line like this:

```
# stub: rails 3.2.11 ruby lib
```

The second line contains: name, version, platform, require_paths. require_paths are delimited by NUL bytes. (One limitation is that require_paths are assumed not to contain any newlines in the path.)

There is now a `Gem::Specification.stubs` method which returns a `Gem::StubSpecification` for each gemspec on the system. The `StubSpecification` objects can be used to perform a bunch of work without having to actually load the gemspec. This provides significant performance improvements.

My aim is to completely remove the use of `Gem::Specification._all` from the codebase, but I stop short of that in this PoC. As explained on the comments, I am currently leaning on `Gem::Specification` to provide some behaviour to `StubSpecification`. I would intend to extract this shared behaviour into a module which will be nicer and will make it more explicit about what code is or isn't used by `StubSpecification`.

At the moment all the test pass except for two to do with the uninstall command (I haven't looked at why, but the point is it mostly works).

## How to try it ##

```
$ cd /path/to/rubygems-repo
$ export RUBYOPT="-Ilib"
$ gem pristine --all
```

Now run some stuff

## My tests ##

I have 348 gems installed:

```
$ ls `ruby -e "puts Gem.dir"`/specifications | wc -l
348
```

I experimented with two tests:

### Test One ###

#### Before ####

```
$ time ruby -e ""

real	0m0.120s
user	0m0.063s
sys	0m0.041s

$ time m --help > /dev/null

real	0m0.450s
user	0m0.373s
sys	0m0.057s
```

#### After ####

```
$ time m --help > /dev/null

real	0m0.177s
user	0m0.100s
sys	0m0.061s
```

### Test Two ###

#### Before ####

````
$ time ruby -r active_record -e ""

real	0m0.853s
user	0m0.750s
sys	0m0.080s
```

#### After ####

```
$ time ruby -r active_record -e ""

real	0m0.656s
user	0m0.554s
sys	0m0.085s
```

## Conclusion ##

This speed up will be most beneficial for small command line scripts. But it will also give a modest boost to bigger tasks like booting rails.

I would like to get feedback on whether this is an acceptable approach. If so I will polish up and finish off the patch and hopefully get it merged.

Thanks!