crayZsaaron/df-legends-reader — pull request #5 by PT-Fort (2013-08-18 22:34:44+00:00)
Title: Handling for invalid files. Reduce memory usage. Progress indicators.

The invalid handling is needed because the XML output leaves ASCII characters in the file. I've seen bugs in other legend readers because of this, so this is valuable to others even now. The solution I've coded here is to detect the crash, and do a brute-force read, convert, and write. Luckily, it only takes about 5 seconds for the 266 MB file I have. So it should be fairly scalable. It creates a second XML file with '-fixed' appended at the end. The good news is that this file can then be used as-is, without further processing. The issue is that this is a second copy of the XML, so perhaps the user should be informed that a second (potentially many GB) file will be created.

I also added progress indicators because I needed them for the larger file. It takes about 50 seconds to load a 266 MB file.

I also looked at ways to reduce the memory usage of the parser. Right now my computer takes almost all of my 4 GB RAM, so that's not optimal. Documentation says to call element.clear() when done with an element. Before, it read in all the tags before getting to the dwarf_world tag. Thus it couldn't clear anything. Now, it looks for the high-level 'sites', 'regions' etc. tags, and processes those. This gives a large improvement in speed and memory usage. Another step would be to check for tags one level lower, since it is loading in a lot of historical figures data before processing it. This should be easy, as the XML file has a fixed format, so it would be easy to keep track of which high-level tag the parser is currently in. I think this would give another speed increase and memory decrease.