hudmol/archivesspace — pull request #348 by marktriggs (2013-03-24 06:54:53+00:00)
Title: Import performance

Today I've been working on getting hornstein.xml import performance back under control. There's probably still more we could do, but I've got it down to a few minutes on my machine (it was about 15 minutes when I started).

Things I've changed:

* update_from_json now has an optional parameter that lets you enable/disable whether linked records are applied. A lot of unnecessary work was being done when the importer did its second "update refs" pass, so this lets it request that linked records be left alone.

* Parsing JSONModels out of the request body was slower than it needed to be because it would grab/return a DB connection for every dynamic enum value it struck. Modified ensure_params to run within a DB.open, so it uses the same DB connection for every check.

* The "always create missing values" function of dynamic enums (used by imports) was very slow: every enum value check would trigger an INSERT statement to try to define that value. I modified this to track which values have been created for the current request so it doesn't do this every time.

* Rails' .to_json is slow. ActiveSupport adds a bunch of extra encoding and error handling that really starts to hurt when dealing with large JSON documents. I've centralised our JSON generation in ASUtils and have gently coerced everything into using that (bypassing ActiveSupport's JSON stuff where possible)

* JSON::Validator serialises schemas to generate a unique URI for them, and does so using ActiveSupport's JSON code when it's loaded. Monkey patched JSON::Validator to force it to use the regular JSON gem, which is much quicker.

* The XML importer's regex cache was growing without limit and using lots of memory for hornstein (100mb+). I switched in an LRU cache that lets us cap how many entries stick around in memory. It still seems to work, but I might need @lcdhoffman to bless this change.

* Increased the memory allocation of the launcher to 512mb. 2013 yo!

I haven't assigned this to anyone, but I'll wait to hear from @lcdhoffman before merging. If anyone else sees any problems please let me know, though :)