aaronstevenwhite/VALEXCorpusReader — pull request #1 by Copper-Head (2013-12-31 20:42:58+00:00)
Title: Alternative corpus loading strategy

Stuff that I've added:
- slightly changed the way the error log is initiated. nothing too big.
- ability to pickle a parsed corpus (this should save us time in the future so that we don't have to reparse the thing all the time). I think this should be very useful no matter what direction we go in ultimately.
- a proof-of concept alternative extracting information from the files using pyparsing; more on that below.

Parsing the Corpus with RegExp instead of PyParsing:
Once I set up a primitive pyparsing-based procedure to deal with the files (I basically just tweaked the stuff in your master branch so that all the variables lined up etc.), the program would take about 1.4 sec to process each file, which gave me a total runtime of about 150 minutes. That's kind of a lot.
So I looked at the files and found that they could be potentially parsed much quicker if we simply defined some regular expressions (these are located in the GETTERS dictionary currently) and sifted through each file using those.
When I did this, the whole corpus was parsed in 190 seconds and only the file for "disqualify" was not processed (somehow no error was recorded for that file, so I'll have do deal with that separately).
I then decided to try incorporating pyparsing into this algorithm, applying it only to some of the strings I collected using regexp, namely the subject and direct object entries.
This bumped my runtime back to about 1 second per file and simultaneously increased memory usage to the point where I'd fill up my 4GB of RAM after about 5 minutes of processing.
When I looked a the error log, I kept seeing "missing parens" messages.
Clearly, something's going on with the pyparsing step. I'm beginning to think we may want to deploy that module in very limited circumstances. Alternatively, we could try running a separate routine on a pickled "preprocessed" corpus. This leads us to the next section.

Things I have questions about:
#1 I'd like to hear what you think about the parsing situation.

#2 We do need to prioritize what functionality we want to get off the ground asap and what's not as important. At this point I've just been trying to go through the whole corpus extracting something as well as getting a feel for the format of the raw data and the tools we have at our disposal. I think I'm done with that now, however, so it's time for the next stage.

#2 It seems to me that we may want to approach this in a combined Functional and Object Oriented Programming way. I suggest this partially because I'm more used to FP myself, but also because that method should work pretty well here due to the lack of variability in data types (maybe I'm wrong, but seems to me that once we establish the core 3 or 4 classes, we won't need to set up any major new ones). I have currently written a bunch of functions, at some point we may want to decide whether to include these into classes or not. This isn't very high priority, as long as we have something that works and is readable at this point I'm fine.