Package: satadd
satadd: CLI pipeline for Planet, Satellogic, Google Earth Engine and Digital Globe ImageryCite asSamapriya Roy. (2018, October 6). samapriya/satadd: satadd: CLI pipeline for Planet, Satellogic,
Google Earth Engine and Digital Globe Imagery (Version 0.0.3). Zenodo. http://doi.org/10.5281/zenodo.1450622Google Earth Engine opened the door for the possibility of getting a curated list of public datasets already ingested in an analysis platform. With over 450+ raster datasets alone, they form one of the most unique collections for publicly available datasets and are still growing. While this was happening for free and open source datasets more and more data was coming in and companies were opening their doors to open data and has missions to include researchers, users, developers and everyone else who wanted to use this datasets. This included but is not limited to companies like Planet, Digital Globe and Satellogic making large chunks of their datasets open for users. Also introduction of high temporal resolution datasets like PlanetScope, high spatial resolution like Skysat and Digital Globe and high spectral resolution images like hyperspectral data from Satellogic was changing our approach to problem solving. While there has been development in building standard API and data access methods there is still room for growth and standrardization and above all easy access to these resources. Planet'sOpen California Program, theEducation and Research Program, Digital Globe'sOpen Data ProgramandEducation and Research program under Satellogic and their Open Datait became obvious that questions we can ask from these sensors could get interesting.This tool was built with a focus on the same issues and borrow parts from my other projects such asppipefor handling Planet's datasets,gee2driveto handle download collections already available in Google Earth Engine (GEE),pygbdxwhich is a relatively new project to explore Digital Globe assets and I have now integrated tools to access and download Satellogic imagery. Core components from a lot of these tools have gone into buildingsataddbased on the idea of adding satelite data as needed. These tools include authentications setups for every account, and access to datasets, metadata among other tools. This was not build however for heavy lifting though I have tested this on hundreds and thousands of assets delivery so it behaves robustly for now. The tool is build and rebuilt as companies change their authentication protocal and delivery mechanisms and allow for improving many aspects of data delivery and preprocessing in the next iterations.While almost all of these tools allow for local export, GEE only exports for now to Google Drive or your Google Cloud Storage Buckets, though what is lost in the delivery endpoints is gained in the fact that GEE is already a mature platform to analyze and look at open datasets but also allows you to bring private datasets into GEE for analysis. So while data download and local analysis may have been the norm, it serves us well to think about posting analysis rather in analysis engines. But that is a discussion for a different time. At this point, I am hoping that this tool alows you to do exactly what the intentions might have been from different providers and to bring them together. Since this tool downloads data it is indeed bandwidth heavy and requires a steady internet connection. This tools handles authentication, downloading, and talking to different API end points and services. In the future I am hoping to include additional preprocessing and delivery to non local endpoints like existing ftp, servers or buckets.Table of contentsInstallationGetting startedsatadd Satellite Data Download AddonInitialize and AuthenticatePlanet ToolsGBDX ToolsSatellogic ToolsGEE ToolsInstallationThis assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and tryingpythonand thenpip listIf you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have released this as a python 2.7 but can be easily modified for python 3.This toolbox also uses some functionality from GDALFor installing GDAL in Ubuntusudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install gdal-bin
sudo apt-get install python-gdalFor Windows I found thisguidefrom UCLAIt has been brought to my notice that installing shapely on windows is not simplypip install shapelyso install Shapely separely anduse instructions from their pypi project pagefor Windows installationShapely is important requirement for the tool but since the installation varies based on the operating system but install it using the earlier instructions anyways before the next steps. On other operating systemspip install shapelyshould work just fine.To installsataddYou can install using two methodspip install sataddor you can also trygit clone https://github.com/samapriya/satadd.git
cd satadd
python setup.py installFor linux use sudo. This release also contains a windows installer which bypasses the need for you to have admin permission, it does however require you to have python in the system path meaning when you open up command prompt you should be able to type python and start it within the command prompt window. Post installation using the installer you can just call satadd using the command prompt similar to calling python. Give it a go post installation typesatadd -hInstallation is an optional step; the application can be also run directly by executing satadd.py script. The advantage of having it installed is being able to execute satadd as any command line tool. I recommend installation within virtual environment. If you don't want to install, browse into the satadd folder and trypython satadd.pyto get to the same result.Getting startedAs usual, to print help:usage: satadd.py [-h]
                 {planetkey,dginit,satinit,eeinit,dasync,savedsearch,metadata,simple_search,footprint,satraster,satmeta,metalist,reproject,refresh,idsearch,intersect,band
type,export}
                 ...

Simple CLI for piping Planet, Satellogic,GEE & GBDX Assets

positional arguments:
  {planetkey,dginit,satinit,eeinit,dasync,savedsearch,metadata,simple_search,footprint,satraster,satmeta,metalist,reproject,refresh,idsearch,intersect,bandtype,export}
    planetkey           Setting up planet API Key
    dginit              Initialize Digital Globe GBDX
    satinit             Initialize Satellogic Tokens
    eeinit              Initialize Google Earth Engine
    credrefresh         Refresh Satellogic & GBDX tokens


    dasync              Uses the Planet Client Async Downloader to download Planet Assets: Does not require activation
    savedsearch         Tool to download saved searches from Planet Explorer
    metadata            Tool to tabulate and convert all metadata files from Planet
                        Item and Asset types for Ingestion into GEE


    simple_search       Simple search to look for DG assets that intersect your AOI handles KML/SHP/GEOJSON
    metadata            Exports metadata for simple search into constitutent folders as JSON files
    footprint           Exports footprint for metadata files extracted earlier
                        and converts them to individual geometries (GeoJSON)and combined geometry (GeoJSON) file


    satraster           Filter and download Satellogic Imagery
    satlist             Get url for band list based on filtered Satellogic Imagery
    multiproc           Multiprocess based downloader based on satlist
    satmeta             Filter and download Satellogic Metadata
    metalist            Generates Basic Metadata list per scene for Satellogic Imagery
    reproject           Batch reproject rasters using EPSG code


    eerefresh           Refreshes your personal asset list and GEE Asset list
    idsearch            Does possible matches using asset name to give you asseth id/full path
    intersect           Exports a report of all assets(Personal & GEE) intersecting with provided geometry
    bandtype            Prints GEE bandtype and generates list to be used for export
    export              Export GEE Collections based on filter

optional arguments:
  -h, --help            show this help message and exitTo obtain help for a specific functionality, simply call it withhelpswitch, e.g.:satadd idsearch -h. If you didn't install satadd, then you can run it just by going tosatadddirectory and runningpython satadd.py [arguments go here]satadd Satellite Data Download AddonThis tool is designed to augment to the existing facilty of image export using a CLI, whereby you can pass it arguments to filter based on an area of interest geojson file, a start and end date for collectionInitialize and AuthenticateThis is an autosuggestive terminal which uses the gee2add package to perform all of the functions but has autosuggest for Earth Engine catalog and your own personal catalog. This way you can get access to image id without needing the catalog id in the javascript codeeditor.planetkey           Setting up planet API Key
dginit              Initialize Digital Globe GBDX
satinit             Initialize Satellogic Tokens
eeinit              Initialize Google Earth Engine
credrefresh         Refresh Satellogic & GBDX tokensEach of these authentication tools allow you to link and save credentials for each of these services you can check them by typing something likesatadd planetkey. Certain services require the authentication tokens to be refreshed you can simply access it usingsatadd credrefresh.Planet ToolsThe Planet Toolsets consists of tools required to access control and download planet labs assets (PlanetScope and RapidEye OrthoTiles) as well as parse metadata in a tabular form which maybe required by other applications. These tools are designed to interact withPlanet's Python Clientand the saved search featured embedded inPlanet Explorerand will allow you to access and download planet imagery and metadata as needed. This also allows you to process the metadata incase you are ingesting this to GEE.dasync              Uses the Planet Client Async Downloader to download Planet Assets: Does not require activation
savedsearch         Tool to download saved searches from Planet Explorer
metadata            Tool to tabulate and convert all metadata files from Planet
                    Item and Asset types for Ingestion into GEEGBDX ToolsThis is a simple cli to Digital Globe's GBDX platform, this was designed from the perspective of community user (the freely available tier). This platform allows you to access all of DG's Open data and also open Ikonos data along with Landsat and Sentinel datasets. You can create anotebook acccount here. The notebook setup offers additional tools, a GUI and interactive framework while CLI simplifies some of the operational needs of batch processing and performing calls using your own local machine. This tool will allow you to perform a simple seach using a geometry to get asset summary, export the metadata as json file and also image footprint as a combined and individual geojson files.simple_search       Simple search to look for DG assets that intersect your AOI handles KML/SHP/GEOJSON
metadata            Exports metadata for simple search into constitutent folders as JSON files
footprint           Exports footprint for metadata files extracted earlier
                    and converts them to individual geometries (GeoJSON)and combined geometry (GeoJSON) fileSatellogic ToolsThis tool allows you to access theopen data shared by Satellogicand filter and pass a geometry object to get both micro(multiband) and macro (hyperspectral) rasters, metadata and basic metadalist. The download tool is a multipart downloader to handle quick downloads. The metalist tool can be used to create a simple metadata list for you to batch upload imagery into GEE for analysis. The reproject tool is included to handle batch reprojections as needed. The tool uses geometry passed as a geojson object go togeojson.io. Satlist produces the band list urls and you can then use the multiproc tool to use multiprocessing to download the links.satraster           Filter and download Satellogic Imagery
satlist             Get url for band list based on filtered Satellogic Imagery
multiproc           Multiprocess based downloader based on satlist
satmeta             Filter and download Satellogic Metadata
metalist            Generates Basic Metadata list per scene for Satellogic Imagery
reproject           Batch reproject rasters using EPSG codeGEE ToolsThis tool allows you to use the gee2drive tool functionalities to explore, match and export existing collections in GEE. Export requires all the bandtypes to be of the same kind. For the past couple of months I havemaintained a catalog of the most current Google Earth Engine assets, within their raster data catalog. I update this list every week. This tool downloads the most current version of this list, and allows the user to explore band types and export a collection as needed.eerefresh           Refreshes your personal asset list and GEE Asset list
idsearch            Does possible matches using asset name to give you asseth id/full path
intersect           Exports a report of all assets(Personal & GEE) intersecting with provided geometry
bandtype            Prints GEE bandtype and generates list to be used for export
export              Export GEE Collections based on filterChangelogv0.0.4Fixed issue with Shapely install on windowsUpdated credrefresh to better refresh gbdx tokensv0.0.3Added better filename parsing for Satellogic imagesAdded error handling for multiprocessing download of Satellogic imagesv0.0.2Now searches for all DG and non DG assets available within GBDXAdded capability to create url list for rasters and download support using multiprocessing