documentcloud/docsplit — pull request #92 by jhosteny (2013-08-28 02:31:00+00:00)
Title: Enable specification of a config file, and generate hocr output if option set

Guys,

Sorry it took so long to get to this, but I've opened this as a new pull request as it is substantially different from the prior one (#81). Please note the most important thing - this change adds a dependency on nokogiri.

First, I took the suggestion to make the command line option one to set a config file. If a config file is set, we look for the variable that enables hocr generation in the file itself. One thing to note is that tesseract allows you to use "+[config]" and it will look for the config file in a well-known location. I decided to just require the full path to be explicit when this option is specified, to avoid install issues.

When hocr generation is on, tesseract appears to only produce html output, not text. So, I've added some routines to generate a text file as well. Additionally, the original hocr output is annotated so that the word tags have two new data attributes set, "data-start" and "data-stop," which are the character start / stop positions of that word.

Lastly, note that the cleaning is a little simplified. I run the text from the xml node through the method that checks for a garbage word, but I didn't do anything fancy looking for too much whitespace. I figured this was good enough for now.