square/shuttle — pull request #70 by RISCfuture (2014-03-06 08:30:36+00:00)
Title: Importing speedup

@RickCSong

1. Separate file parsing and key creation into separate workers
2. Change Blob key caching to a database table

1.

To speed importing, this commit extracts the key-creation code away from
`BlobImporter` and puts it in a new worker, `KeyCreator`, leaving
`BlobImporter` to strictly do file parsing.

Now, as `BlobImporter` parses a file, it builds an array of key data. This data
is segmented in groups of 100, and each group is sent to a `KeyCreator` worker
for processing. `KeyCreator` batch-creates the keys, and does the bulk
ElasticSearch updates and association updates.

In order to prevent `Commit#loading` from becoming false prematurely, the
`BlobImporter` adds the `KeyCreator`'s job ID to the commit's loading list the
same way that the `Commit` adds the `BlobImporter`'s JID to its loading list.
The `KeyCreator` has the same logic to remove itself from the loading list as
the `BlobImporter`. A commit will not finish loading until all `BlobImporter`s
_and_ `KeyCreator`s have finished.

All of the code dealing with creating keys has been moved from `Importer::Base`
to `KeyCreator` (the exception being locale imports, which I generally consider
to be deprecated at this point).

In order to continue to support inline imports, the `:inline` option must be
further passed down the import call chain. Now, `Importer::Base` has a
`#inline` attr_accessor that is set by `Blob#import_strings`, which is passed
the value by the `BlobImporter`.

2.

Formerly, keys found in a blob in the course of importing were cached in Redis,
the thought being that if the blob were again encountered in later commits, the
cached keys could simply be added to the new commit. This code never actually
worked.

So a `blobs_keys` table has been added and a `has_and_belongs_to_many`
association created between `Blob` and `Key`. Now, the first time a blob is
encountered, this association is populated with all the found keys. Future
imports will use those cached keys, bypassing the parsing and key creation
steps, unless the `:force` option is used.

The `cached_keys` column on `blobs` records whether the blobs_keys association
is fresh.