doolin/patentprocessor — pull request #12 by alvinchang (2012-07-02 22:36:59+00:00)
Title: Text_to_db.py // benchmark-confirm.py 

Text_to_db.py
- Joins final.txt (Field1###Field2###Field3) into specified sqlite3 file and appends Field2, Field3 to them. (But, unclear whether or not we need to do this). Invnum is now set as the primary key to avoid duplicates. Insert now Ignores if there is a duplicate.
- Logs SQL errors
- Currently has the same results as zardoz (with invnum_n and invnum), surface tested pairwise with invpat_final.

TODO:
Need to script test for this.
Determine validity of Field2, Field3. If they are unnecessary, don't need to convert tuple to list to modify and change back from list to tuple (saves a lot of computational time).

Benchmark-confirm.py
- Splits the goldstandard.csv file, runs it against the output from zardoz to see if the database has the same entry.
- Logs any errors (ex. zardoz output does not have an match for an entry in the goldstandard.csv file. currently there are ~22/1333 errors).

TODO:
Investigate source of errors, although the number is small need to figure out why.
Run benchmark.py on it.