Package: overturemapsdownloader
Overture Maps DownloaderThis repository is still under heavy development. Features may change, and documentation will be updated accordingly. Use at your own risk and feel free to contribute!OverviewOvertureMapsDownloader is a monorepo that contains two main projects aimed at simplifying the process of working with Overture Maps data in Parquet format.DuckDB-WASM with MapThe first project is a browser-based application that leverages DuckDB-WASM and mapping functionalities. This allows you to query Overture Maps data in Parquet format directly within your web browser. It offers a seamless and interactive way to work with geospatial data without the need for any backend services. The codebase for this project is located in the/appfolder.Data Manipulation/Downloading and ConversionThe second project focuses on data manipulation, downloading, and conversion of Overture Maps data. It utilizes technologies like DuckDB, Python, and IPython Notebooks, along with GDAL/OGR for data conversion. This project is containerized using Docker, making it easy to integrate into existing pipelines and ETL workflows. For hands-on examples, check out the Jupyter notebooks available in theexamplesfolder.Whether you're a data scientist, a geospatial analyst, or a developer, OvertureMapsDownloader offers comprehensive solutions for your geospatial data needs.Table of ContentsOverture Maps DownloaderOverviewDuckDB-WASM with MapData Manipulation/Downloading and ConversionTable of ContentsDuckDB-WASM in Browser (Developers)Getting StartedVisual DemoData Manipulation/Downloading and ConversionPrerequisitesInstallationthe docker image manily contains the following tools:UsageDownload Geospatial DataCommandsExamples of downloading data CLI:With Docker:Without Docker you can use the following commands with (pip install overturemapsdownloader):Convert Parquet to GeoPackageConvert Parquet to MBTiles (will support tippecanoe in the future)Convert Parquet to ESRI File Geodatabase vector (OpenFileGDB)Using Jupyter Notebooks (Data Scientists/GIS Analysts)ConfigurationRoadmapDuckDB-WASM in Browser (Developers) (ReactJS)Data Manipulation/Downloading and Conversion (Python)ContributingDuckDB-WASM in Browser (Developers)For developers interested in running DuckDB-WASM in the browser, the relevant codebase is located in the/appfolder at the root of the repository. This setup allows you to develop, run, and build the DuckDB-WASM component seamlessly.Getting StartedNavigate to the/appfolder:cd/appInstall the required packages:yarninstallNote: Make sure you have npm version 18 or above for compatibility.Start the development server:yarndevthen openhttp://localhost:5173/OvertureMapsDownloaderin your browserVisual DemoData Manipulation/Downloading and ConversionPrerequisitesDocker and Docker ComposeGeoJSON file containing the bounding box polygon (e.g.,bbox.geojson)you can make yours easly onhttps://geojson.ioor useexamples/bbox.jsonfor testingInstallationTo get started, you'll need to pull the Docker image from the GitHub Container Registry:dockerpullghcr.io/youssef-harby/overturemapsdownloader:latestthe docker image manily contains the following tools:GDAL/OGR v3.8.4Python v3.10 (and GDAL/OGR Python bindings + duckdb python bindings)DuckDB v0.10.0Jupyter.. and moreUsageDownload Geospatial DataCreate a folder and place your bounding box polygon in GeoJSON format inside it (e.g.,bbox.geojson).Navigate to the folder:cd/examples# e.g., cd /path/to/your/folder contains bbox.geojson and will contain the output fileRun the following command to download geospatial data:dockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themeplaces--ptypeplace--bbox/examples/bbox.geojson--output/examples/places.parquetCommandsThe omaps CommandOMDownloaderomaps[OPTIONS]options:--theme [admins|buildings|buildings|places|transportation]Theme of the data to download--ptype [locality|administrativeBoundary|building|place|water|connector|segment]Type of the data to download--bbox PATHBounding box polygon in GeoJSON format as a path to a file--output PATHOutput file path (e.g.,places.parquet)--helpShow this message and exit.Examples of downloading data CLI:With Docker:# admins/localitydockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themeadmins--ptypelocality--bbox/examples/bbox.geojson--output/examples/locality.parquet# admins/administrativeBoundarydockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themeadmins--ptypeadministrativeBoundary--bbox/examples/bbox.geojson--output/examples/admins.parquet# buildings/buildingdockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themebuildings--ptypebuilding--bbox/examples/bbox.geojson--output/examples/building.parquet# places/placedockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themeplaces--ptypeplace--bbox/examples/bbox.geojson--output/examples/place.parquet# transportation/connectordockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themetransportation--ptypeconnector--bbox/examples/bbox.geojson--output/examples/connector.parquet# transportation/segmentdockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestOMDownloaderomaps--themetransportation--ptypesegment--bbox/examples/bbox.geojson--output/examples/segment.parquetWithout Docker you can use the following commands with (pip install overturemapsdownloader):# admins/localityOMDownloaderomaps--themeadmins--ptypelocality--bboxexamples/bbox.geojson--outputexamples/locality.parquet# admins/administrativeBoundaryOMDownloaderomaps--themeadmins--ptypeadministrativeBoundary--bboxexamples/bbox.geojson--outputexamples/admins.parquet# buildings/buildingOMDownloaderomaps--themebuildings--ptypebuilding--bboxexamples/bbox.geojson--outputexamples/building.parquet# places/placeOMDownloaderomaps--themeplaces--ptypeplace--bboxexamples/bbox.geojson--outputexamples/place.parquet# transportation/connectorOMDownloaderomaps--themetransportation--ptypeconnector--bboxexamples/bbox.geojson--outputexamples/connector.parquet# transportation/segmentOMDownloaderomaps--themetransportation--ptypesegment--bboxexamples/bbox.geojson--outputexamples/segment.parquetConvert Parquet to GeoPackageTo convert the downloaded data to GeoPackage format, run the following command:dockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestogr2ogr/examples/output.gpkg/examples/places.parquetConvert Parquet to MBTiles (will support tippecanoe in the future)dockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestogr2ogr-dscoMAXZOOM=14/examples/output.mbtiles/examples/places.parquetConvert Parquet to ESRI File Geodatabase vector (OpenFileGDB)dockerrun-v$(pwd):/examplesghcr.io/youssef-harby/overturemapsdownloader:latestogr2ogr/examples/output.gdb/examples/places.parquetUsing Jupyter Notebooks (Data Scientists/GIS Analysts)If you prefer to use Jupyter notebooks for your geospatial data manipulation tasks, you can easily set up a Jupyter environment using Docker Compose.Clone the repository:gitclonehttps://github.com/Youssef-Harby/OvertureMapsDownloader.gitNavigate to the repository folder:cdOvertureMapsDownloaderRun the following Docker Compose command:dockercomposeupThis will start a Jupyter notebook server accessible athttp://localhost:8888/lab.ConfigurationFor advanced configurations, please refer to theconfig.ymlfile.RoadmapDuckDB-WASM in Browser (Developers) (ReactJS)Add Popup on click of the feautre to show the result of the queryAdd a button to use local parquet url instead of the remote oneAdd a button to download the result of the query as GeoJSON..etcAdd a loading spinner while the query is runningQuery by bbox drawn on the mapfeel free to add more ...Data Manipulation/Downloading and Conversion (Python)Resolve JSON values to be parsed to the correct data typeOSGeo/gdal#8227ContributingEveryone is invited to contribute to this project. Whether you're a beginner or experienced, your help is valuable. Don't worry about not getting it right the first time; the idea is to grow and learn together.You can contribute by:Commenting on or adding toexisting issuesStarting or participating indiscussionsIf you're new and have questions or need guidance, thediscussionspage is a great place to start.