Package: geobeam
geobeam adds GIS capabilities to your Apache Beam pipelines.What does geobeam do?geobeamenables you to ingest and analyze massive amounts of geospatial data in parallel usingDataflow.
geobeam provides a set ofFileBasedSourceclasses that make it easy to read, process, and write geospatial data, and provides a set of helpful
Apache Beam transforms and utilities that make it easier to process GIS data in your Dataflow pipelines.See theFull Documentationfor complete API specification.RequirementsApache Beam 2.46+Python 3.8+Note: Make sure the Python version used to run the pipeline matches the version in the built container.Supported input typesFile formatData typeGeobeam classtiffrasterRasterBlockSourceandRasterPolygonSourceshpvectorShapefileSourcegdbvectorGeodatabaseSourcejsonvectorGeoJSONSourceURLvectorESRIServerSourceIncluded librariesgeobeamincludes several python modules that allow you to perform a wide variety of operations and analyses on your geospatial data.ModuleVersionDescriptiongdal3.5.2python bindings for GDALrasterio1.3.2reads and writes geospatial raster datafiona1.8.21reads and writes geospatial vector datashapely1.8.4manipulation and analysis of geometric objects in the cartesian planeesridump1.11.0read layer from ESRI serverHow to Use1. Install the modulepip install geobeam2. Write your pipelineWrite a normal Apache Beam pipeline using one ofgeobeams file sources.
Seegeobeam/examplesfor inspiration.3. RunRun locallypython -m geobeam.examples.geotiff_dem \
  --gcs_url gs://geobeam/examples/dem-clipped-test.tif \
  --dataset examples \
  --table dem \
  --band_column elev \
  --runner DirectRunner \
  --temp_location <temp gs://> \
  --project <project_id>Note: Some of the provided examples may take a very long time to run locally...Run in DataflowWrite a DockerfileThis will run in Dataflow as acustom containerbased on thedataflow-geobeam/baseimage.
It is recommended that you publish your own container based on the Dockerfile in this repository and store it in your project's GCR registry.FROMgcr.io/dataflow-geobeam/baseRUNpipinstallgeobeamCOPYrequirements.txt.RUNpipinstall-rrequirements.txtCOPY..# build locally with dockerdockerbuild-tgcr.io/<project_id>/geobeam
dockerpushgcr.io/<project_id>/geobeam# or build with Cloud Buildgcloudbuildssubmit--timeout3600s--worker_machine_typen1-highcpu-8Start the Dataflow job# run the geotiff_soilgrid example in dataflow
python -m geobeam.examples.geotiff_soilgrid \
  --gcs_url gs://geobeam/examples/AWCh3_M_sl1_250m_ll.tif \
  --dataset examples \
  --table soilgrid \
  --band_column h3 \
  --runner DataflowRunner \
  --sdk_container_image gcr.io/dataflow-geobeam/base \
  --temp_location <temp bucket> \
  --service_account_email <service account> \
  --region us-central1 \
  --max_num_workers 2 \
  --worker_machine_type c2-standard-30 \ExamplesRead Raster as Blocksdefrun(options):fromgeobeam.ioimportRasterBlockSourcefromgeobeam.fnimportformat_rasterblock_recordwithbeam.Pipeline(options)asp:(p|'ReadRaster'>>beam.io.Read(RasterBlockSource(gcs_url))|'FormatRecord'>>beam.Map(format_rasterblock_record)|'WriteToBigquery'>>beam.io.WriteToBigQuery('geo.dem'))Validate and Simplify Shapefiledefrun(options):fromgeobeam.ioimportShapefileSourcefromgeobeam.fnimportmake_valid,filter_invalid,format_recordwithbeam.Pipeline(options)asp:(p|'ReadShapefile'>>beam.io.Read(ShapefileSource(gcs_url))|'Validate'>>beam.Map(make_valid)|'FilterInvalid'>>beam.Filter(filter_invalid)|'FormatRecord'>>beam.Map(format_record)|'WriteToBigquery'>>beam.io.WriteToBigQuery('geo.parcel'))Seegeobeam/examples/for complete examples.A number of example pipelines are available in thegeobeam/examples/folder.
To run them in your Google Cloud project, run the includedterraformfile to set up the Bigquery dataset and tables used by the example pipelines.Open up Bigquery GeoViz to visualize your data.Shapefile ExampleThe National Flood Hazard Layer loaded from a shapefile. Example pipeline atgeobeam/examples/shapefile_nfhl.pyRaster ExampleThe Digital Elevation Model is a high-resolution model of elevation measurements at 1-meter resolution. (Values converted to centimeters). Example pipeline:geobeam/examples/geotiff_dem.py.Included TransformsThegeobeam.fnmodule includes severalBeam Transformsthat you can use in your pipelines.ModuleDescriptiongeobeam.fn.make_validAttempt to make all geometries valid.geobeam.fn.filter_invalidFilter out invalid geometries that cannot be made validgeobeam.fn.format_recordFormat the (props, geom) tuple received from a vector source into adictthat can be inserted into the destination tablegeobeam.fn.format_rasterblock_recordFormat the output record for blocks read fromRasterBlockSourcegeobeam.fn.format_rasterpolygon_recordFormat the output record for blocks read fromRasterPolygonSourceExecution parametersEach FileSource accepts several parameters that you can use to configure how your data is loaded and processed.
These can be parsed as pipeline arguments and passed into the respective FileSources as seen in the examples pipelines.ParameterInput typeDescriptionDefaultRequired?skip_reprojectAllTrue to skip reprojection during readFalseNoin_epsgAllAnEPSG integerto override the input source CRS to reproject fromNoin_projAllAPROJ stringto override the input source CRSNoband_numberRasterThe raster band to read from1Noinclude_nodataRasterTrue to includenodatavaluesFalseNoreturn_block_transformRasterTrue to include rasteriotransformobject with each block to use withgeobeam.fn.format_rasterpixel_recordFalseNolayer_nameVectorName of layer to readYes, for shapefilesgdb_nameVectorName of geodatabase directory in a gdb zip archiveYes, for GDB filesLicenseThis is not an officially supported Google product, though support will be provided on a best-effort basis.Copyright 2023 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.