onyxfish/csvkit — Issue #142: Improve csvsql insert performance using a transaction
Author: migonzalvar
Created: 2012-01-18 17:53:15+00:00
State: open

I added a transaction in csvsql.py. Performance for big files improves a lot.

```diff
--- a/csvkit/utilities/csvsql.py
+++ b/csvkit/utilities/csvsql.py
@@ -56,8 +56,12 @@ class CSVSQL(CSVKitUtility):
insert = sql_table.insert()
headers = csv_table.headers()

+ conn = engine.connect()
+ trans = conn.begin()
for row in csv_table.to_rows():
- engine.execute(insert, [dict(zip(headers, row)), ])
+ conn.execute(insert, [dict(zip(headers, row)), ])
+ trans.commit()
+ conn.close()

# Writing to file
else:

```

AFAIK, every single SQLalchemy database backend includes transaction support.

Could this little patch be applied to the code? Do you prefer a pull request?