jlegewie/zotfile — pull request #1 by devietti (2012-01-03 21:12:02+00:00)
Title: using pdf.js for annotation extraction

Hi Joscha,

I've done some hacking on using pdf.js (http://andreasgal.com/2011/06/15/pdf-js/) for extracting note, highlight and underline annotations from PDFs. I integrated this into the most recent ZotFile version, replacing the existing code that uses the Mac-only PdfExtractor program. I've made some modifications to pdf.js to get this to work. I plan to try to get these modifications accepted into the pdf.js mainline, at which point I'll update my ZotFile fork as well.

The basic outline of the code is that the URL of a PDF whose annotations need to be extracted is passed to a function that resides on a special webpage, extract.html, which gets opened in a background tab (or new window in Zotero Standalone). The code on this page runs pdf.js to parse the PDF and extract its annotations. I tried for a good while to get this code to run as part of zotfile.js, but I couldn't. I think the issue is that the code in zotfile.js runs in a special non-webpage context, and the pdf.js code really expects to be running as part of a webpage. One issue was that creating new HTML nodes, which pdf.js does often, seemed to be silently broken. After the PDF extraction is done, a callback in zotfile.js gets called, and the next PDF gets processed, until all PDFs are processed.

I've tested with Firefox 9.0.1 on Mac OS X Lion, though the code should be cross-platform. I've tested using Zotero 2.1.10 and Zotero Standalone 3.0b3.2.

Not everything is yet working as well as I'd like:
1. Zotero Standalone works, but the interface is clunky. I can't figure out how to close or hide the window that pops up to run the extraction webpage.
2. Multi-line annotations don't get extracted correctly, due to a bug in pdf.js. This bug needs to be fixed on the pdf.js side, however, and won't affect the ZotFile code.

Still, I wanted to share what I had with you since most of the functionality is present. Let me know what you think!

I also had a few questions/comments about the ZotFile code.
1. When should the removeHyphens() function run? I couldn't figure out where the hyphens it's removing were coming from. Are they artifacts of the extraction process?
2. I simplified the representation of annotations, to be just a filename, page number, type and content string. I didn't understand the difference between the .text and .textMarkUp properties in getNoteContent(). It seemed like .text was only being used with "note" annotations, and .textMarkUp with highlight/underline annotations. Since there's already a type property, I merged these .text and .textMarkUp into a new .content property.

cheers,
joe