Package: opendatacrawler
A tool to craw data to your projects from open data portalsReport Bug·Request FeatureTable of ContentsAbout The ProjectGetting StartedrequirementsInstallationUsageRoadmapContributingLicenseContactAbout The ProjectOpen Data Crawler is a tool to extract data from open data portals and statistics portals. The community can contribute adding support to other data portals or adding new features.Features:Download datasets from open data portals or statsitics portalDownload metadata from resourcesFilter by data typeFilter by topic(back to top)Getting StartedThis is an example of how you may give instructions on setting up your project locally.
To get a local copy up and running follow these simple example steps.RequirementsYou need python 3.9 installedClone the repogitclonehttps://github.com/aberenguerpas/opendatacrawler.gitMove to root directorycdopendatacrawlerInstall the requirements from requirements.txtpip3install-rrequirements.txtSocrata portals requiere an app token to avoid throttling limits, you can obtain an api keyhereand set onconfig.iniInstallationRun from the project rootpython3setup.pyinstall(back to top)UsageUse this tool is very simple, you only need to specify the data source and the tool automatically detect the portal type and starts to dowload the data.ExamplesDowload all data from a portal:python opendatacrawler -d https://data.smartdublin.ie/Dowload all data with their metadata:python opendatacrawler -d https://data.smartdublin.ie/ -mDowload partial dataset (first 50 lines for csv files):python opendatacrawler -d https://data.smartdublin.ie/ -pdDowload specific fromat data. For example xls and csv:python opendatacrawler -d https://data.smartdublin.ie/ -t xls csvDowload specifics categories. For example xls and csv:python opendatacrawler -d https://data.smartdublin.ie/ -c tourism transportHelp with all posible commands:python opendatacrawler -hFor more examples, please refer to theDocumentation(back to top)Currently supported portals and sitesCKANSocratahttps://ec.europa.eu/eurostat(LIMITED ⚠️)*https://datacatalogapi.worldbank.org/(LIMITED ⚠️)*https://datos.gob.esOpenDataSoft* Works with restrictions or download limitationsSee theopen issuesfor a full list of proposed features (and known issues).(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion or add site/portal support that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestAdd support to other portalCreate a file with the name of the portal + crawler Ex.examplecrawler.pyinside the folderopendatacrawlerCreate a class ExampleCrawler who inherits fromOpenDataCrawlerInterfaceThe class must contain at least the functionsget_package_list()andget_package()Check the descriptions of the functions on theopendatacrawlerInterface.pyYou can also use or add some functions toutils.pyAdd in the functiondetect_dms()onodcrawler.pya way to detect the site you want to add.(back to top)LicenseDistributed under the MIT License. SeeLICENSEfor more information.(back to top)ColaboratorsContact🙋‍♂️Alberto Berenguer Pastor📱@aberenguerpas✉️alberto.berenguer@ua.es(back to top)