ananelson/oacensus — pull request #21 by cameronneylon (2014-07-25 09:51:38+00:00)
Title: Pub speed

I've done a couple of things here to enable me to use the codebase for a different project. Alongside this I've introduced a couple of fixes for problems I've found.

1. There are a bunch of new files in the examples folder, these aren't really necessary for anything, its just where I've been working but they might be useful examples.
2. models.py - New elements added to the Article model to enable tracking of submit and accept date based on pubmed information
3. pubspeed-excel report added for the external project
4. An openaire scraper and relevant tests (this was earlier work but should probably be incorporated)
5. scrapers/pubmed.py Various additions (see below)

In scrapers/pubmed.py I made a series of changes.

First of these is that pubmed contains invalid dates. I wrapped date parsing in a try: except: clause and set it to unknown if it fails. Not sure whether this is the best approach but it worked for what I needed. Line ~184

I added end_date as a parameter for fetch_batch, include the end_date where it is called and added code to incorporate it in the parameters. I don't know whether this matters or not but I was trying to solve a problem where date searching pubmed seems to be broken. Lines 160-200.

Modified Article.create() to use Article.create_or_update_by_doi() as the period batching was duplicating article creation (because of the date search issue giving the same article in multiple periods). If there is no DOI it defaults to the old approach.