Package: scrapy-omdena-latam
Scrapy Tool for Omdena Latam LFR ChallengeWeb Crawling application runningScrapyTool, extracting official policies from the following sources:Characteristics of the information sourcesChile (LeyChile)Search type:Exhaustive, through API , limited by the pages_numSpeed:FastAmount of avaliable documents:10-100kDocument Type:HTMLMexico Distrito oficial de la FederaciónSearch type:Exhaustive, through scrapping (Xpath) , limited by years range.Speed:Terrible slow and buggy when you change pipelines orderAmount of avaliable documents:10-100kDocument Type:HTMLEl PeruanoSearch type:Speed:Amount of avaliable documents:Document Type:Setup Steps:Recommendations:Use a virtual environment not your python system to run and also to install the dependencies.Install dependenciespip install -r requirements.txtScrapy settings.pyhttps://drive.google.com/file/d/1bjbjYSXQqZQpJdwATRCLZFSRQciiULy-/view?usp=sharingWarning!S3 upload pipeline and MySQL insert pipeline doesn't work together.
Use:ITEM_PIPELINES = {
    # 'scrapy.pipelines.files.FilesPipeline': 100,
    'scrapy_official_newspapers.pipelines.ScrapyOfficialNewspapersMySQLPipeline': 200,
        }andITEM_PIPELINES = {
     'scrapy.pipelines.files.FilesPipeline': 100,
    #'scrapy_official_newspapers.pipelines.ScrapyOfficialNewspapersMySQLPipeline': 200,
        }The order doesn't mind.Database accessSetup the DB access inserting settings.json intoscrappy_official_newspapers.{
  "username": "username",
  "password": "password",
  "db_name": "db_name",
  "aws_endpoint": "your_db_instance_access"
}S3 AccessSetup scrapysettings.pylocated atscrappy_official_newspapersAWS_ACCESS_KEY_ID = "XXXXXXXXXXXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
FILES_STORE = 's3://wri-latin-test/'Runfrom repository root:cd scrapy_official_newspapersscrapy crawl leychilescrapy crawl MexicoDOFMonitorization/Debug/TestThrough mysql table inspection, you can check how the information is being inserted.Throughhttps://console.aws.amazon.com/console/homeafter authenticating you can navigate to S3 Service, you can check the files uploaded and their properties.GoalProvide a structured information system, from multiple variated sources formats containing raw official policy documents, keeping the reference to their attributes.Attributescountrygeo_codelevelsourcetitlereferenceauthorshipresumepublication_dateenforcement_date
urldoc_urldoc_namedoc_typefile_urls (Needed for Scrappy S3FilesPipeline, Not in DB Schema)Not implemented like that, but the idea is to also keep track of:file_raw_S3_urlfile_proccesed_task_1_S3_urlSo we are able to keep track of the policies with their attributes and also have the capability of maintaining the relationships and results of different processes.What is this information system:Policies Documents Stored and indexed through the integration of a relational MySQL database and AWS S3 media database service (FTP would work too), with most attributes such as the title of the act, a resume of the document, the date of publication...Documentation:https://docs.scrapy.org/en/2.2/https://docs.scrapy.org/en/2.2/topics/media-pipeline.html#enabling-your-media-pipelineRecomendations:Do not make an extensive use of the tool, probably information will be thrown because it is still in development, you can collapse the target's resources and engraving amazon's bill through this.