first sketches of uascraper
Fix metadata, publication and persisitent_identifier fields in narcis spider
add utwente to list of Narcis  repositories
update C links to work with more repositories/sites
A spider can now choose it's preferate pipeline(s).
In scrapy, by default, all spiders are forced to use all pipelines defined in settings.
A spider can now choose it's preferate pipeline(s).
In scrapy, by default, all spiders are forced to use all pipelines defined in settings.
add SaveFilePipeline class  to save the file in a folder with the db id as name. In construction!
WORK IN PROGRESS
more bug fixed
add validatiopn pipelines for Paper, Author, ResourceFile, and all 3 toghether
Finnished URLDedupFilter downloader middleware
autogenerate narcis URLs
branch 1.1
a
refactored URL dedup middleware
Add link_item and add_item decorator.
Refactored NarcisSpider accordingly.
Added more Link extractors for  'foreign' repositories inside Narcis
Update 1.2
Update 1.1
Updated 1.1