SeerLabs/CiteSeerX — pull request #3 by fanchyna (2013-08-14 18:04:13+00:00)
Title: multiple update to cdi

Multiple features are added to the cdi.py module: described in the "UPDATE" file inside the folder. Some of them are listed
(*) Re-download PDF files using "wget" if they are not found from Heritrix mirror directory.
(*) Throw the _mysql_exceptions.Warnings exception when a URL is not able to be saved into the "url" field of database.
(*) Add a urlfilters.py module and add "urllengthrule" and "blacklistrule".