Package: esridump
esri-dumpScrapes an Esri REST endpoint and writes a GeoJSON file.InstallationIf you just want to use the command line toolesri2geojson, the recommended way to install this package is to create avirtual environmentand install it there. This method does not require that yougit clonethis repository and can get you up and running quickly:virtualenv esridump
source esridump/bin/activate
pip install esridumpUsageCommand lineThis module will install a command line utility calledesri2geojsonthat accepts an Esri REST layer endpoint URL and a filename to write the output GeoJSON to:esri2geojsonhttp://cookviewer1.cookcountyil.gov/ArcGIS/rest/services/cookVwrDynmc/MapServer/11cookcounty.geojsonYou can write tostdoutby using the special output filename of-(a single dash character).You can also pass in the--jsonlinesoption to write newline-separated (\n) lines of GeoJSON features, which you can then pipe into other applications.Python moduleYou can use this module in your code to get GeoJSON Feature-shaped Pythondictsinto your code:importjsonfromesridump.dumperimportEsriDumperd=EsriDumper('http://example.com/arcgis/rest/services/Layer/MapServer/1')# Iterate over each featureforfeatureind:print(json.dumps(feature))d=EsriDumper('http://example.com/arcgis/rest/services/Layer/MapServer/2')# Or get all features in one listall_features=list(d)MethodologyThe module will do its best to find the most efficient method of retrieving data from the Esri server, giventhe capabilities of the server. There are several strategies we use to get the data, described here in most to least efficient order:resultOffsetPaginationIn ArcGIS REST API version 10.3, Esri added support for pagination directly with theresultOffsetandresultRecordCountparameters. Unfortunately, most servers don't support this feature because the backend SQL engine must also be configured to support it. So far, it seems that only the Esri-hosted layers support this feature reliably.objectIdField ChunkingIn ArcGIS REST API version 10.0, Esri added support for the server to return an exhaustive list of object IDs for all features in a layer. Once this list of object IDs is retrieved, we break it into chunks ofmaxRecordCountqueries using theobjectIdsparameter.objectIdStatisticswhere-clausesIn ArcGIS REST API version 10.1, Esri added support for performing various statistical queries on the server without requiring the client to download the whole dataset. On servers that support this and don't respond to theobjectIdsqueries, we will use a minimum and maximum statistics query to find the minimum and maximum values for theobjectIdcolumn, then build chunks ofwhere-clauses that narrow the range down toobjectIds between two fenceposts.Geometry Quadtree QueriesWhen a server does not support any of these methods, we'll make recursive quad-tree queries using bounding envelopes. We start with a query for the layer's entireextent. If the server returns exactly themaxRecordCountnumber of features, we split thatextentinto 4 equal rectangles and query those. If those smaller queries returnmaxRecordCountfeatures, we split the rectangle again and continue until the server returns something less than themaxRecordCount.DevelopmentTo suggest changes or improvements to this code, create a fork on Github and clone your repository locally:git clone git@github.com:openaddresses/pyesridump.git # replace with your fork
cd pyesridumpWe use Pipenv to manage dependencies for development. Make sure you havePipenv installedand then install the dependencies for development:pipenv install --dev
pipenv shellYour changes to the code will be reflected when you run theesri2geojsoncommand from within the virtual environment. You can also run (and add) tests to check that your changes didn't break anything:nosetestsSee AlsoThis Python module was extracted from OpenAddressesmachine, which was inspired by code fromkoop. A similar node/JavaScript module is available inesri-dump.