Package: geopyspark
GeoPySpark is a Python bindings library forGeoTrellis, a Scala
library for working with geospatial data in a distributed environment.
By usingPySpark, GeoPySpark is
able to provide an interface into the GeoTrellis framework.LinksDocumentationGitterA Quick ExampleHere is a quick example of GeoPySpark. In the following code, we take NLCD data
of the state of Pennsylvania from 2011, and do a masking operation on it with
a Polygon that represents an area of interest. This masked layer is then saved.If you wish to follow along with this example, you will need to download the
NLCD data and unzip it.. Running these two commands will complete these tasks
for you:curl -o /tmp/NLCD2011_LC_Pennsylvania.zip "https://s3-us-west-2.amazonaws.com/prd-tnm/StagedProducts/NLCD/data/2011/landcover/states/NLCD2011_LC_Pennsylvania.zip?ORIG=513_SBDDG"
unzip -d /tmp /tmp/NLCD2011_LC_Pennsylvania.zipimportgeopysparkasgpsfrompysparkimportSparkContextfromshapely.geometryimportbox# Create the SparkContextconf=gps.geopyspark_conf(appName="geopyspark-example",master="local[*]")sc=SparkContext(conf=conf)# Read in the NLCD tif that has been saved locally.# This tif represents the state of Pennsylvania.raster_layer=gps.geotiff.get(layer_type=gps.LayerType.SPATIAL,uri='/tmp/NLCD2011_LC_Pennsylvania.tif',num_partitions=100)# Tile the rasters within the layer and reproject them to Web Mercator.tiled_layer=raster_layer.tile_to_layout(layout=gps.GlobalLayout(),target_crs=3857)# Creates a Polygon that covers roughly the north-west section of Philadelphia.# This is the region that will be masked.area_of_interest=box(-75.229225,40.003686,-75.107345,40.084375)# Mask the tiles within the layer with the area of interestmasked=tiled_layer.mask(geometries=area_of_interest)# We will now pyramid the masked TiledRasterLayer so that we can use it in a TMS server later.pyramided_mask=masked.pyramid()# Save each layer of the pyramid locally so that it can be accessed at a later time.forpyramidinpyramided_mask.levels.values():gps.write(uri='file:///tmp/pa-nlcd-2011',layer_name='north-west-philly',tiled_raster_layer=pyramid)For additional examples, check out theJupyter notebook demos.RequirementsRequirementVersionJava>=1.8Scala>=2.11Python3.3 - 3.6Spark>=2.1.1Java 8 and Scala 2.11 are needed for GeoPySpark to work, as they are required by
GeoTrellis. In addition, Spark needs to be installed and configured with the
environment variableSPARK_HOMEset.You can test to see if Spark is installed properly by running the following in
the terminal:> echo $SPARK_HOME
/usr/local/bin/sparkIf the return is a path leading to your Spark folder, then it means that Spark
has been configured correctly. IfSPARK_HOMEis unset or empty, you’ll need to add it
to yourPATHafter noting where Spark is installed on your system. For example,
a MacOS installation of Spark 2.3.0 via HomeBrew would setSPARK_HOMEas follows:# In ~/.bash_profileexportSPARK_HOME=/usr/local/Cellar/apache-spark/2.3.0/libexec/InstallationBefore installing, check the aboveRequirementstable to make sure that the
requirements are met.Installing From PipTo install viapipopen the terminal and run the following:pip install geopyspark
geopyspark install-jarThe first command installs the python code and thegeopysparkcommand
from PyPi. The second downloads the backend jar file, which is too large
to be included in the pip package, and installs it to the GeoPySpark
installation directory. For more information about thegeopysparkcommand, see theGeoPySpark CLIsection.Installing From SourceIf you would rather install from source, clone the GeoPySpark repo and enter it.git clone https://github.com/locationtech-labs/geopyspark.git
cd geopyspark
make installThis will assemble the backend-endjarthat contains the Scala code,
move it to thejarssub-package, and then runs thesetup.pyscript.Note:If you have altered the global behavior ofsbtthis install may
not work the way it was intended.UninstallingTo uninstall GeoPySpark, run the following in the terminal:pip uninstall geopyspark
rm .local/bin/geopysparkContact and SupportIf you need help, have questions, or like to talk to the developers (let us
know what you’re working on!) you can contact us at:GitterMailing listAs you may have noticed from the above links, those are links to the GeoTrellis
gitter channel and mailing list. This is because this project is currently an
offshoot of GeoTrellis, and we will be using their mailing list and gitter
channel as a means of contact. However, we will form our own if there is a need
for it.GeoPySpark CLIWhen GeoPySpark is installed, it comes with a script which can be accessed
from anywhere on you computer. This script is used to facilitate management
of the GeoPySpark jar file that must be installed in order for GeoPySpark to
work correctly. Here are the available commands:geopyspark -h, --help // return help string and exit
geopyspark install-jar // downloads jar file to default location, which is geopyspark install dir
geopyspark install-jar -p, --path [download/path] //downloads the jar file to location specified
geopyspark jar-path //returns the relative path of the jar file
geopyspark jar-path -a, --absolute //returns the absolute path of the jar filegeopysparkinstall-jaris only needed when installing GeoPySpark throughpip; and itmustbe ran before using GeoPySpark. If no path is selected,
then the jar will be installed wherever GeoPySpark was installed.The second and third commands are for getting the location of the jar file.
These can be used regardless of installation method. However, if installed
throughpip, then the jar must be downloaded first or these commands
will not work.Developing GeoPySparkContributingFeedback and contributions to GeoPySpark are always welcomed.
A CLA is required for contribution, seeContributingfor more
information.Installing for Developersmake build
pip install -e .make buildwill assemble the back-endjarand move it thejarssub-package. The second command will install GeoPySpark in “editable” mode.
Meaning any changes to the source files will also appear in your system
installation.Within a virtualenvIt’s possible that you may run into issues when performing thepip install-e.described above with a Python virtualenv active. If you’re having trouble with
Python finding installed libraries within the virtualenv, try adding the virtualenv
site-packages directory to your PYTHONPATH:workon <your-geopyspark-virtualenv-name>
export PYTHONPATH=$VIRTUAL_ENV/lib/<your python version>/site-packagesReplace<your python versionwith whatever Python versionvirtualenvwrapperis set to. Once you’ve set PYTHONPATH, re-install
GeoPySpark using the instructions in “Installing for Developers” above.Running GeoPySpark TestsGeoPySpark uses thepytesttesting
framework to run its unittests. If you wish to run GeoPySpark’s unittests,
then you must first clone this repository to your machine. Once complete,
go to the root of the library and run the following command:pytestThis will then run all of the tests present in the GeoPySpark library.Note: The unittests require additional dependencies in order to pass fully.pyproj,colortools,
andmatplotlib(only for >=Python3.4) are needed to
ensure that all of the tests pass.Make Targetsinstall- install GeoPySpark python package locallywheel- build python GeoPySpark wheel for distributionpyspark- start pyspark shell with project jarsbuild- builds the backend jar and moves it to the jars sub-packageclean- remove the wheel, the backend jar file, and clean the
geotrellis-backend directoryDeveloping GeoPySpark With GeoNotebookNote: Before begining this section, it should be noted that python-mapnik,
a dependency for GeoNotebook, has been found to be difficult to install. If
problems are encountered during installation, a possible work around would be
to runmake wheeland then dodocker cpthewheelinto the
GeoPySpark docker container and install it from there.GeoNotebookis a Jupyter
notebook extension that specializes in working with geospatial data. GeoPySpark
can be used with this notebook; which allows for a more interactive experience
when using the library. For this section, we will be installing both tools in a
virtual environment. It is recommended that you start with a new environment
before following this guide.Because there’s already documentation on how to install GeoPySpark in a virtual
environment, we won’t go over it here. As for GeoNotebook, it also has a section
oninstallationso that will not be covered here either.Once you’ve setup both GeoPySpark and GeoNotebook, all that needs to be done
is go to where you want to save/have saved your notebooks and execute this
command:jupyter notebookThis will open up the jupyter hub and will allow you to work on your notebooks.It is also possible to develop with both GeoPySpark and GeoNotebook in editable mode.
To do so you will need to re-install and re-register GeoNotebook with Jupyter.pip uninstall geonotebook
git clone --branch feature/geotrellis https://github.com/geotrellis/geonotebook ~/geonotebook
pip install -r ~/geonotebook/prerequirements.txt
pip install -r ~/geonotebook/requirements.txt
pip install -e ~/geonotebook
jupyter serverextension enable --py geonotebook
jupyter nbextension enable --py geonotebook
make notebookThe defaultGeonotebook (Python 3)kernel will require the following environment variables to be defined:export PYSPARK_PYTHON="/usr/local/bin/python3"
export SPARK_HOME="/usr/local/apache-spark/2.1.1/libexec"
export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.10.4-src.zip:${SPARK_HOME}/python/lib/pyspark.zip"Make sure to define them to values that are correct for your system.
Themake notebookcommand also makes used ofPYSPARK_SUBMIT_ARGSvariable defined in theMakefile.GeoNotebook/GeoTrellis integration in currently in active development and not part of GeoNotebook master.
The latest development is on afeature/geotrellisbranch at<https://github.com/geotrellis/geonotebook>.Side Note For DevelopersAn optional (but recommended!) step for developers is to place these
two lines of code at the top of your notebooks.%load_extautoreload%autoreload2This will make it so that you don’t have to leave the notebook for your changes
to take affect. Rather, you just have to reimport the module and it will be
updated. However, there are a few caveats when usingautoreloadthat can be
readhere.Usingpip install-ein conjunction withautoreloadshould cover any
changes made, though, and will make the development experience much less
painful.