crayZsaaron/df-legends-reader — pull request #8 by PT-Fort (2013-08-23 01:37:54+00:00)
Title: Optimizing for memory usage and time to process

This is a large optimization of the memory used by the everything, as well as the time taken to build it. I have radically changed the parsing process, so I will wait for comments before checking this change in.

I ran the test.py script and loaded the dictionary in the GUI to test.

Three changes that impact the use of the dictionary:
1) All values that can be stored as integers will be. This comes at a slight speed decrease, but offers a large memory savings. I have added/removed casting in various locations in the code.
2) Similarly, all attribute values that were '-1' are now not added to the dictionary. This saves a large amount of space as well.
3) For the historical figures, in the dictionaries hf_links and entity_links, I have changed the keys. Instead of 'hfid' and 'entity_id', this is now 'id'. Instead of 'link_type', that key is now 'type'. This was done to save space in the dictionary, and slightly reduces the amount of code in the parser. I can change it back if it is a major issue.

Before commits After first commit After second commit
12 MB file
Time (s): - 2 1.5
Memory (MB): 60 51 40

266 MB file
Time (s): 41 49 36
Memory (MB): 1141 980 742