funginstitute/patentprocessor — pull request #55 by laironald (2013-08-20 05:38:07+00:00)
Title: parse changes, moving files around for starcluster bulk

parse is the script Gabe provided
parse_2 is the modified script (plan is to integrate this as parse)
* note SQLite appears to perform faster than MySQL
* commit code in parse is not performing correctly
* generator function in parse has issues

==============================

#### parse

```
def parse_files(filelist):
if not filelist:
return []
parsed = itertools.imap(extract_xml_strings, filelist)
return itertools.chain.from_iterable(parsed)

def parse_patents(xmltuples):
if not xmltuples:
return []
return map(parse_patent, xmltuples)

def database_commit(patobjects, commit_frequency=commit_frequency):
num_objs = len(patobjects) # compute this once
for i, patobj in enumerate(patobjects):
alchemy.add(patobj)
if commit_frequency and ((i+1) % commit_frequency == 0 or i == num_objs):
alchemy.commit()
print " *", (i+1), datetime.datetime.now()
if not commit_frequency:
alchemy.commit()
```

the above 3 functions are consolidated to parse_2 per below
reason being, the generator function ends up with memory issues
(see Killed statements when executing against full dataset)

#### parse_2

```
def parse_files(filelist):
if not filelist:
return []
for filename in filelist:
print filename
for i, xmltuple in enumerate(extract_xml_strings(filename)):
patobj = parse_patent(xmltuple)
alchemy.add(patobj)
if commit_frequency and ((i+1) % commit_frequency == 0):
alchemy.commit()
print " *", (i+1), datetime.datetime.now()
alchemy.commit()
```

#### BENCHMARKING

I am processing two datasets:
2012/01/03 and 2013/04/16
* full datasets
* smaller datasets (random 25 record sample)

===========================

parse -- MySQL -- Full dataset
Mon Aug 19 22:19:50 CDT 2013
Killed
Mon Aug 19 22:26:37 CDT 2013

The job does not finish and runs out of memory, essentially "killing itself". This takes approximately 7 minutes to reach this conclusion.

------------------------------------------------------

parse_2 -- MySQL -- Full dataset
Mon Aug 19 22:46:15 CDT 2013
...
...
Mon Aug 19 23:33:19 CDT 2013

The job takes approximate 45 minutes with two files. 10,000 records or so.

------------------------------------------------------

parse -- sqlite - Full dataset
Mon Aug 19 23:54:03 CDT 2013
Killed
Tue Aug 20 00:00:22 CDT 2013

Similar to MySQL, the job does not complete. Takes a bit less time @ 6mins.

------------------------------------------------------

parse_2 -- sqlite -- Full dataset
Tue Aug 20 00:03:51 CDT 2013
...
...
Tue Aug 20 00:42:18 CDT 2013

The job takes a bit less than 40 minutes with two files. 10,000 records. Slight improvement over MySQL

------------------------------------------------------

parse -- MySQL -- Small dataset
Tue Aug 20 00:24:59 CDT 2013
Tue Aug 20 00:25:16 CDT 2013

Takes 17 seconds --- however this has commit issues (it doesn't properly commit)

------------------------------------------------------

parse_2 -- MySQL -- Small dataset
Tue Aug 20 00:26:25 CDT 2013
XML/out/ipg120103.xml
XML/out/ipg130416.xml
Tue Aug 20 00:26:39 CDT 2013

Takes 14 seconds --- with proper committing

------------------------------------------------------

parse -- sqlite - Small dataset
Tue Aug 20 00:23:07 CDT 2013
Tue Aug 20 00:23:17 CDT 2013

Smaller dataset takes 10 seconds
However, it appears the commit doesn't properly commit --
* there is an error with this parse/commit method

------------------------------------------------------

parse_2 -- sqlite -- Small dataset
Tue Aug 20 00:21:39 CDT 2013
XML/out/ipg120103.xml
XML/out/ipg130416.xml
Tue Aug 20 00:21:51 CDT 2013

Smaller dataset takes 12 seconds