Package: geodatacrawler
pyGeoDataCrawlerThe tool crawls a data folder or tree. For each spatial file identified, it will process the file. Extract as many information as possible and store it on a sidecar metadata file.The tool can also look for existing metadata using common conventions. For metadata imports the tool wil useowslib, which supports some metadata formats.Several options exist for using the results of the generated index:The resulting indexed content can be converted to iso19139 or OGCAPI-records and inserted on an instance of pycsw, geonetwork or pygeoapi, to make it searchable.Automated creation of a mapserver mapfile to provide OGC services on top of the spatial files identified.InstallationThe tool requires GDAL 3.2.2 and pysqlite 0.4.6 to be installed. I recommend to usecondato install them.conda create --name pgdc python=3.9 
conda activate pgdc
conda install -c conda-forge gdal==3.3.2
conda install -c conda-forge pysqlite3==0.4.6Then run:pip install geodatacrawlerUsageThe tools are typically called from commandline or a bash script.Index metadatacrawl-metadata --mode=init --dir=/myproject/data [--out-dir=/mnt/myoutput]Mode explained:init; creates new metadata for files which do not have it yet (not overwriting)update; updates the metadata, merging new content on existing (not creating new)export; exports the mcf metadata to xml and stored it in a folder (to be loaded on pycsw) or on a database (todo)import-csv; imports a csv of metadata fiels into a series of mcf files, typically combined with a.j2 filewith same name, whichmapsthe csv-fields to mcf-fieldsThe export utility will merge any yaml file to a index.yml from a parent folder. This will allow you to create minimal metadata at the detailed level, while providing more generic metadata down the tree. The index.yml is also used as a configuration for any mapfile creation (service metadata).Most parameters are configured from the commandline, check --help to get explanation.
2 parameters can be set as an environment variablepgdc_host is the url on which the data will be hosted in mapserver or a webdav folder.pgdc_schema_path is a physical path to an override of the default iso19139 schema of pygeometa, containing jinja templates to format the exported xmlSome parameters can be set in index.yml, in a robot section. Note that config is inherited from parent folders.mcf:version 1.0robot:skip-subfolders:True# do not move into subfolders, typically if subfolder is a set of tiles, default: Falseskip-files:"temp.*"# do not process files matching a regexp, default: NoneOGR/GDAL formatsSome GDAL (raster) or OGR (vector) formats, such asFileGDB,GeoPackageandparquetrequire an additional plugin. Verify for each of the commom formats in your organisation, if the relevant GDAL plugins are installed.Create mapfileThe metadata identified can be used to create OGC services exposing the files. Currently the tool createsmapserver mapfiles, which are placed on a output-folder. Aindex.ymlconfiguraton file is expected at the root of the folder to be indexed, if not, it will be created.crawl-mapfile --dir=/mnt/data [--out-dir=/mnt/mapserver/mapfiles]Some parameters in the mapfile can be set using environment variables:ParamDescriptionExamplepgdc_out_dira folder where files are placed (can override with --dir-out)pgdc_md_urla pattern on how to link to metadata, use {0} to be substituted by record uuid, or empty to not include metadata linkhttps://example.com/{0}pgdc_ms_urlthe base url of mapserverhttp://example.com/mapspgdc_webdav_urlthe base url on which data files are published or empty if not publishedhttp://example.com/datapgdc_md_link_typeswhich service links to addOGC:WMS,OGC:WFS,OGC:WCS,OGCAPI:Featuresexportpgdc_webdav_url="https://example.com/data"Amapserver dockerimage is provided by Camp to Camp which is able to expose a number of mapfiles as mapservices, eghttp://example.com/{mapfile}?request=getcapabilities&service=wms. Each mapfile needs to be configured as alias inmapserver config file.Layer stylingYou can now set dedicated layer styling for grids and vectors.Add mapserver mapfile syntax to the mcf robot sectionrobot:map:styles:|CLASSNAME "style"STYLECOLOR 100 100 100SIZE 8WIDTH 1ENDENDOr style your layer explicitely. Note that you can define multiple styles per layer, the last is used as default:A range of colors (grid only), the min-max range of the first band is devided by the number of colors.robot:map:styles:-name:rainbowclasses:"#ff000,#ffff00,#00ff00,#00ffff,#0000ff"-name:graysclasses:"#00000,#333333,#666666,#999999,#cccccc,#ffffff"A range of distinct values, you can also use rgb colorsrobot:map:styles:-name:rainbowproperty:length# name of the column, vector onlyclasses:-label:Trueval:1color:"02550"-label:Falseval:0color:"25500"A range of classesrobot:map:styles:-name:Scaleproperty:length# name of the column, vector onlyclasses:-label:Lowmin:0max:100color:"#0000ff"-label:Mediummin:100max:200color:"#00ff00"-label:Highmin:200max:300color:"#ff0000"DevelopmentPython PoetryThe project is based on common coding conventions from the python poetry community.On the sources, either run scripts directly:poetry run crawl-mapfile --dir=/mnt/dataor run a shell in the poetry environment:poetry shellThe GDAL dependency has some installation issue on poetry, seeherefor a workaround> poetry shell
>> sudo apt-get install gdal
>> gdalinfo --version
GDAL 3.3.2, released 2021/09/01
>> pip install gdal==3.3.2
>> exitReleaseupdateinit.pyandpyporoject.tomlpush changestrigger semantic releasepoetry buildpoetry publish