okfn/messytables — pull request #78 by rossjones (2013-07-19 15:40:26+00:00)
Title: Adds support for Open Spreadsheet format files (*.ods)

Unfortunately for large files existing libraries are not particularly good at handling them, large.ods shows a 2Mb ODS file that expands out to 10Mb and odfpy uses Gb's of RAM and takes over a minute and a half to load it (bug report on odfpy at https://joinup.ec.europa.eu/software/odfpy/issue/excessive-ram-usage-when-loading-2mb-ods-file).

Although not particularly pretty (regex, ugh) this version uses regex and XML of fragments parsing to pull the relevant information, and can process 26k rows in a 10Mb content.xml in approx 5-7s (still far, far too long :( )

This is far from a perfect solution, but it's currently the only way to get acceptable performance to just pull a single sheet from large ODS files without mulitple passes with the sax parsing (or trading huge amounts of memory to avoid to avoid that).

The XML parsing (using iterparse) requires a wrapper (on the Linux version of LXML) so that the namespaces are handled correctly - this isn't an issue on OSX but that works with the wrapper too.