jrochkind/traject — pull request #13 by billdueber (2013-08-28 16:05:40+00:00)
Title: Only create the spec once, not for every single record

THIS IS FOR DISCUSSION ONLY AT THIS POINT

Included in this pull request is a rewrite of extract_marc and its dependencies (called, originally, "extract_marc2") that instead of returning a lambda that creates a Traject::MarcExtractor for every single call, creates it once and then exposes it to the indexer (essentially) via the lambda's closure.

This necessitated a rewrite of marc_extractor.rb to allow the extract* methods to take in a record, instead of relying on the originally-passed in record. The *right* way to do this is to change the Traject::MarcExtractor.initialize signature to only take a spec and an options hash, not a marc record, but for the moment this allows us to compare apples to apples. (You'll see I pass in a :junk value in the new code).

This makes semantic sense to me: the lambda returned by the spec is _in no way_ reliant on the particular marc record passed in for its behavioral definition. I think the way it is now mixes up data/code in a way that obscures the fact that the code (lambda) produced is independent of the MARC record, which is just stuffed into an instance variable and doesn't change the generated code at all (give or take; I know it's internal state, yada yada, but you see my point).

It also turns out, as I suspected, to be a lot faster.

You can run the given benchmarking code on your own machine: on mine, I get this on a 19K record marc file, using JSONWriter > /dev/null and the standard ruby-marc reader.

```
jruby 1.7.4 (1.9.3p392) 2013-05-16 2390d3b on Java HotSpot(TM) 64-Bit Server VM 1.7.0_40-ea-b27 [darwin-x86_64]
Rehearsal -------------------------------------------------------------
extract_marc_conf.rb (0) 62.530000 0.470000 63.000000 ( 50.865000)
extract_marc2_conf.rb (0) 33.260000 0.150000 33.410000 ( 32.260000)
extract_marc_conf.rb (3) 56.750000 0.390000 57.140000 ( 15.941000)
extract_marc2_conf.rb (3) 35.540000 0.180000 35.720000 ( 10.084000)
-------------------------------------------------- total: 189.270000sec

user system total real
extract_marc_conf.rb (0) 40.520000 0.170000 40.690000 ( 40.138000)
extract_marc2_conf.rb (0) 27.830000 0.110000 27.940000 ( 27.548000)
extract_marc_conf.rb (3) 48.420000 0.180000 48.600000 ( 13.086000)
extract_marc2_conf.rb (3) 34.590000 0.140000 34.730000 ( 9.645000)

```

By my math, that's a 30-35% speed increase. So, I'd say this is worth doing.