zdennis/activerecord-import — pull request #45 by saizai (2012-02-10 11:51:03+00:00)
Title: Profiling-based improvement to import#values_sql_for_columns_and_attributes - 80% speedup for large imports

https://plus.google.com/u/0/103112149634414554669/posts/EGghnZ1icVY for details.

Three issues corrected:

1. Calling ActiveRecord::Base#connection within the loop. This may well be an ActiveRecord WTF, but connection actually isn't memoized; every time you call it, it goes back to the connection pool and fetches things over again. Yikes. So I memoized it local to the function.

2. Calling sequence_name needlessly, for non-id columns. This is a result of doing and in the wrong order. Statements are evaluated left-to-right; this means that you should put the stuff that's cheapest to fail on the left, and the expensive stuff (or the bits whose real return values you want preserved) on the right. So I just swapped the order.

There's a deeper WTF here, in that sequence_name should be memoized within ActiveRecord, but something was breaking that. Not sure what, and I haven't fixed that, so it'll still be slow if you are using a primary key column. Serves you right for trying to insert explicit primary key values instead of leaving that up to the database at insertion time. :-P

3. Double-escaping. connection.quote already does appropriate type conversion and escaping; calling connection.type_cast first provides no better security, but slows things down significantly by making things be converted twice. So I just got rid of that.

These three changes resulted in an 80% performance improvement for the import call.