Package: ckan-harvesters
Harvester Next Generation for CKANInstallpip install ckan-harvestersUse data.json sourcesfromharvesters.datajson.harvesterimportDataJSONdj=DataJSON()dj.url='https://data.iowa.gov/data.json'try:dj.fetch()exceptExceptionase:print(e)valid=dj.validate(validator_schema='non-federal-v1.1')print(dj.errors)# ['Error validating JsonSchema: \'bureauCode\' is a required property ...# full dict with the sourceprint(dj.as_json())"""{'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld','@id': 'https://data.iowa.gov/data.json','@type': 'dcat:Catalog','conformsTo': 'https://project-open-data.cio.gov/v1.1/schema','describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json','dataset': [{'accessLevel': 'public','landingPage': 'https://data.iowa.gov/d/23jk-3uwr','issued': '2017-01-30','@type': 'dcat:Dataset',..."""# just headersprint(dj.headers)"""{'@context': 'https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld','@id': 'https://data.iowa.gov/data.json','@type': 'dcat:Catalog','conformsTo': 'https://project-open-data.cio.gov/v1.1/schema','describedBy': 'https://project-open-data.cio.gov/v1.1/schema/catalog.json',}"""fordatasetindj.datasets:print(dataset['title'])ImpairedStreams20142009-2010IowaPublicSchoolDistrictBoundaries2015-2016IowaPublicSchoolDistrictBoundariesImpairedStreams2010ImpairedLakes20142007-2008IowaPublicSchoolDistrictBoundariesImpairedStreams20122011-2012IowaPublicSchoolDistrictBoundariesActiveandCompletedWatershedProjects-IDALS2012-2013IowaPublicSchoolDistrictBoundaries2010-2011IowaPublicSchoolDistrictBoundaries2016-2017IowaPublicSchoolDistrictBoundaries2014-2015IowaPublicSchoolDistrictBoundariesImpairedLakes20082008-2009IowaPublicSchoolDistrictBoundaries2013-2014IowaPublicSchoolDistrictBoundariesImpairedLakes2010ImpairedLakes2012ImpairedStreams2008Use CSW sourcesfromharvesters.csw.harvesterimportCSWSourcec=CSWSource(url='http://data.nconemap.com/geoportal/csw?Request=GetCapabilities&Service=CSW&Version=2.0.2')csw.fetch()csw_info=csw.as_json()print('CSW title:{}'.format(csw_info['identification']['title']))# CSW title: ArcGIS Server Geoportal Extension 10 - OGC CSW 2.0.2 ISO APDevelopmentTo setup a develop environment, clone the repository and in a virtualenv install the dependenciespip install -r requirements.txtThis will install the library in development mode, and other libraries for tests.TestThen to run the test suite with pytest:pytestWe usepytest-vcrbased on the wonderfulVCRpy, to mock http requests. In this way, we don't need to hit the real internet to run our test (which is very fragile and slow), because there is a mocked version of a each response needed by tests, in vcr'scassettesformat.In order to update thesecassettesjust run as following:pytest --vcr-record=allTo actually hit the internet without use mocks, disable the pluginpytest --disable-vcrIn order to read from thesecassettesjust run as following:pytest --vcr-record=noneTests without a CKAN instancepython -m pytest tests

================ test session starts =============
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
rootdir: /home/hudson/dev/datopian/ckan-ng-harvester-core
plugins: vcr-1.0.2
collected 17 items                                                                                                                                                          

tests/test_csw_dataset_adapter.py ....      [ 23%]
tests/test_data_json.py .......             [ 64%]
tests/test_datajson_dataset_adapter.py .....[100%]

=============== 17 passed in 17.52s ==============Tests with a CKAN instance.You will need to copy settings.py file to local_settings.py file and fill the required values.You can use a local or remote CKAN instance.python -m pytest tests_with_ckan/test_harvest.py