Package: ppipe
Planet GEE Pipeline CLIThis tool is designed to facilitate moving data from Planet's API into Google Earth Engine and using a metadata library. The tool downloads data into a local storage and allows you to process the metadata before uploading into Google Earth Engine. This tool has been updated to account for metadata property type by going through each metadata column and then uses theupload manifestfunction to upload images for ingestion to EE. This tool also draws from an additional tool I created with is theGoogle Earth Engine Asset Manager AddonThis includes the batch upload feature, but now additional tools such as generating reports of Earth Engine assets and querying quota to name just a few. The ambition is apart from helping users with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as the need arises. I have now released this as aPyPI packagefor easy installation and this will be updated along with the the GitHub package.Though this tool is designed to download planet imagery, useporderto use the new ordersv2 API to download planet imagery. This includes additional filter capabilities along with capability to clip and download.Please note: This tool is in no way an official tool or Planet offering, but is a personal project created and maintained by Samapriya RoyIf you use this tool to download data for your research, and find this tool useful, star and cite it as belowSamapriya Roy. (2019, October 10). samapriya/Planet-GEE-Pipeline-CLI: Planet-GEE-Pipeline-CLI (Version 0.4.8). Zenodo.
http://doi.org/10.5281/zenodo.3479525Table of contentsInstallationGetting startedUsage examplesPlanet ToolsPlanet KeyPlanet QuotaDownload AsyncDownload Saved SearchesMetadata ParserEarth Engine Toolsselenium updategee selsetupEE UserEE QuotaCreateBatch uploaderAsset ListAsset SizeTask QueryAssets AccessCancel all tasksCreditsInstallationThis assumes that you have native python & pip installed in your system, you can test this by going to the terminal (or windows command prompt) and tryingpythonand thenpip listIf you get no errors and you have python 2.7.14 or higher you should be good to go. Please note that I have released this as a python 2.7 but can be easily modified for python 3.Make sure you install OpenSSL fromthis pageThis toolbox also uses some functionality from GDALFor installing GDAL in Ubuntusudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
sudo apt-get install gdal-bin
sudo apt-get install python-gdalFor Windows I found thisguidefrom UCLATo installPlanet-GEE-Pipeline-CLI:You can install using two methodspip install ppipeor you can also trygit clone https://github.com/samapriya/Planet-GEE-Pipeline-CLI.git
cd Planet-GEE-Pipeline-CLI
python setup.py installFor linux use sudo. This release also contains a windows installer which bypasses the need for you to have admin permission, it does however require you to have python in the system path meaning when you open up command prompt you should be able to type python and start it within the command prompt window. Post installation using the installer you can just call ppipe using the command prompt similar to calling python. Give it a go post installation typeppipe -hInstallation is an optional step; the application can be also run directly by executing ppipe.py script. The advantage of having it installed is being able to execute ppipe as any command line tool. I recommend installation within virtual environment. If you don't want to install, browse into the ppipe folder and trypython ppipe.pyto get to the same result.Getting startedAs usual, to print help:> ppipe -h
usage: ppipe [-h]
             {
             ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,ca
             ...

Planet Pipeline with Google Earth Engine Batch Addons

positional arguments:
  { ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
                        ---------------------------------------
                        -----Choose from Planet Tools Below-----
                        ---------------------------------------
    planetkey           Enter your planet API Key
    pquota              Prints your Planet Quota Details
    dasync              Uses the Planet Client Async Downloader to download
                        Planet Assets: Does not require activation
    savedsearch         Tool to download saved searches from Planet Explorer
    metadata            Tool to tabulate and convert all metadata files from
                        Planet or Digital Globe Assets
                        -------------------------------------------
                        ----Choose from Earth Engine Tools Below----
                        -------------------------------------------
    update              Updates Selenium drivers for firefox
    selsetup            Non headless setup for new google account, use if
                        upload throws errors
    ee_user             Get Earth Engine API Key & Paste it back to Command
                        line/shell to change user
    quota               Print Earth Engine total quota and used quota
    create              Allows the user to create an asset collection or
                        folder in Google Earth Engine
    selupload           Batch Asset Uploader for Planet Items & Assets using
                        Selenium
    lst                 List assets in a folder/collection or write as text
                        file
    assetsize           Prints collection size in Human Readable form & Number
                        of assets
    tasks               Queries current task status
                        [completed,running,ready,failed,cancelled]
    access              Sets Permissions for items in folder
    cancel              Cancel all running tasks

optional arguments:
  -h, --help            show this help message and exitTo obtain help for a specific functionality, simply call it withhelpswitch, e.g.:ppipe upload -h. If you didn't install ppipe, then you can run it just by going toppipedirectory and runningpython ppipe.py [arguments go here]Usage examplesUsage examples have been segmented into two parts focusing on both planet tools as well as earth engine tools, earth engine tools include additional developments in CLI which allows you to recursively interact with their python APIPlanet ToolsThe Planet Toolsets consists of tools required to access control and download planet labs assets (PlanetScope and RapidEye OrthoTiles) as well as parse metadata in a tabular form which maybe required by other applications.Planet KeyThis tool basically asks you to input your Planet API Key using a password prompt this is then used for all subsequent tools. This tool now includes an option for a quiet authentication using the API key incase it is unable to invoke an interactive environment such as in Google colaboratory. You can also performplanet initto set the key as the tool can read from it directly.usage: ppipe planetkey [-h] [--type TYPE] [--key KEY]

optional arguments:
  -h, --help   show this help message and exit

Optional named arguments:
  --type TYPE  For direct key entry type --type quiet
  --key KEY    Your Planet API KeyIf using on a private machine the Key is saved as a csv file for all future runs of the tool.Planet QuotaThis tool prints your Planet quota including allocation name, Total quota, quota used and quota remaining. Calling it is simpleppipe pquotaDownload AsyncThis tool is built as a wrapper around the Planet Client's own download tool. The tool included in the planet client is multithreaded and allows the user to activate, poll and download at the same time. This will allow you to pass geometry in terms of a geojson files or all filter like start and end date and time using the .json files you created using theaoijsontool earlier.usage: ppipe dasync [-h] [--infile INFILE] [--item ITEM] [--asset ASSET]
                       [--local LOCAL] [--start START] [--end END]
                       [--cmin CMIN] [--cmax CMAX]

optional arguments:
  -h, --help       show this help message and exit
  --infile INFILE  Choose a geojson from geojson.io or the aoi-json you
                   created earlier using ppipe aoijson
  --item ITEM      Choose from Planet Item types Example: PSScene4Band,
                   PSOrthoTile, REOrthoTile etc
  --asset ASSET    Choose an asset type example: anlaytic,
                   analytic_dn,analytic_sr,analytic_xml etc
  --local LOCAL    Local Path where Planet Item and asset types are saved

Optional named arguments:
  --start START    Start date filter format YYYY-MM-DD
  --end END        End date filter format YYYY-MM-DD
  --cmin CMIN      Cloud cover minimum between 0-1
  --cmax CMAX      Cloud cover maximum between 0-1A setup using geojson needs to include other filters too and a typical setup would beppipe dasync --infile "C:\Users\johndoe\geometry.geojson" --item "PSScene4Band" --asset "analytic" --local "C:\planet" --start "2018-06-01" --end "2018-08-01" --cmin 0 --cmax 0.4Using a stuctured json file that you might have created earlier means you don't have to pass additional filters everytimepython ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo"However, you can still decide to pass the filters and the filters you pass will overwrite existing filterspython ppipe.py dasync --infile "C:\Users\johndoe\geometry.json" --item "PSScene4Band" --asset "analytic_xml" --local "C:\planet_demo" --start "2018-06-01" --end "2018-08-01" --cmin "0" --cmax 0.4Download Saved SearchesDownload assets from saved searches which are saved in your planet explorer.usage: ppipe savedsearch [-h] [--name NAME] [--asset ASSET] [--local LOCAL]
                         [--limit LIMIT]

optional arguments:
  -h, --help     show this help message and exit
  --name NAME    Name of your saved search(It is case sensitive)
  --asset ASSET  Choose asset type analytic, analytic_xml, analytic_sr,
                 analytic_dn etc
  --local LOCAL  Local Path (full path address) where PlanetAssets are saved

Optional named arguments:
  --limit LIMIT  Choose number of assets you want to downloadMetadata ParserThe metadata tab is a more powerful tool and consists of metadata parsing for All PlanetScope and RapiEye Assets along with Digital Globe MultiSpectral and DigitalGlobe PanChromatic datasets. This was developed as a standalone to process xml metadata files from multiple sources and is important step is the user plans to upload these assets to Google Earth Engine.usage: ppipe metadata [-h] [--asset ASSET] [--mf MF] [--mfile MFILE]
                      [--errorlog ERRORLOG] [--dir DIR]

optional arguments:
  -h, --help           show this help message and exit
  --asset ASSET        Choose PS OrthoTile(PSO)|PS OrthoTile DN(PSO_DN)|PS
                       OrthoTile Visual(PSO_V)|PS4Band Analytic(PS4B)|PS4Band
                       DN(PS4B_DN)|PS4Band SR(PS4B_SR)|PS3Band
                       Analytic(PS3B)|PS3Band DN(PS3B_DN)|PS3Band
                       Visual(PS3B_V)|RE OrthoTile (REO)|RE OrthoTile
                       Visual(REO_V)|DigitalGlobe
                       MultiSpectral(DGMS)|DigitalGlobe
                       Panchromatic(DGP)|PolarGeospatial CenterDEM
                       Strip(PGCDEM)?
  --mf MF              Metadata folder?
  --mfile MFILE        Metadata filename to be exported along with Path.csv
  --errorlog ERRORLOG  Errorlog to be exported along with Path.csv

Optional named arguments:
  --dir DIR            Path to Image Directory to be used to get ImageTags
                       with metadata. use only with PS4B_SREarth Engine ToolsThe ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises. This is also a seperate package for earth engine users to use and can be downloadedhereselenium updateThis is a key step since all upload function depends on this step, so make sure you run this. This downloads selenium driver and places to your local directory for windows and Linux subsystems. This is the first step to use selenium supported upload.ppipe updategee selsetupOnce in a while the geckodriver requires manual input before signing into the google earth engine, this tool will allow you to interact with the initialization of Google Earth Engine code editor window. It allows the user to specify the account they want to use, and should only be needed once.geeup selsetupEE UserThis tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back. This command takes in no arguments.ppipe ee_userEE QuotaThis tool prints earthengine quota tools, used and remaining quota. No argument is requiredppipe quotaCreateThis tool allows you to create a collection or folder in your earth engine root directory. The tool uses the system cli to achieve this and this has been included so as to reduce the need to switch between multiple tools and CLI.usage: ppipe.py create [-h] --typ TYP --path PATH

optional arguments:
  -h, --help   show this help message and exit
  --typ TYP    Specify type: collection or folder
  --path PATH  This is the path for the earth engine asset to be created full
               path is needsed eg: users/johndoe/collectionBatch uploaderThe script creates an Image Collection from GeoTIFFs in your local directory. By default, the image name in the collection is the same as the local directory name; with the optional parameter you can provide a different name.You have to process the metadata for images, which is covered in the next section along with a manifest type for Planet image and asset:Metadata parser.usage: ppipe selupload [-h] --source SOURCE --dest DEST [-m METADATA]
                          [--large] [--nodata NODATA] [--bands BANDS]
                          [-u USER] [-b BUCKET]

optional arguments:
  -h, --help            show this help message and exit

Required named arguments.:
  --source SOURCE       Path to the directory with images for upload.
  --dest DEST           Destination. Full path for upload to Google Earth
                        Engine, e.g. users/pinkiepie/myponycollection
  -u USER, --user USER  Google account name (gmail address).

Optional named arguments:
  -m METADATA, --metadata METADATA
                        Path to CSV with metadata.
  --large               (Advanced) Use multipart upload. Might help if upload
                        of large files is failing on some systems. Might cause
                        other issues.
  --nodata NODATA       The value to burn into the raster as NoData (missing
                        data)
  --bands BANDS         Comma-separated list of names to use for the image
                        bands. Spacesor other special characters are not
                        allowed.
  -b BUCKET, --bucket BUCKET
                        Google Cloud Storage bucket name.Asset ListThis tool is designed to either print or output asset lists within folders or collections using earthengine ls tool functions.usage: ppipe lst [-h] --location LOCATION --typ TYP [--items ITEMS]
                 [--output OUTPUT]

optional arguments:
  -h, --help           show this help message and exit

Required named arguments.:
  --location LOCATION  This it the location of your folder/collection
  --typ TYP            Whether you want the list to be printed or output as
                       text[print/report]

Optional named arguments:
  --items ITEMS        Number of items to list
  --output OUTPUT      Folder location for report to be exportedAsset SizeThis tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size.usage: ppipe assetsize [-h] --asset ASSET

optional arguments:
  -h, --help     show this help message and exit
  --asset ASSET  Earth Engine Asset for which to get size propertiesTask QueryThis script counts all currently running,ready,completed,failed and cancelled tasks along with failed tasks. This is linked to the account you initialized with your google earth engine account. This takes no argument.usage: ppipe.py tasks [-h]

optional arguments:
  -h, --help  show this help message and exit

ppipe.py tasksAssets AccessThis tool allows you to set asset acess for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time.usage: ppipe.py access [-h] --asset ASSET --user USER --role ROLE

optional arguments:
  -h, --help     show this help message and exit
  --asset ASSET  This is the path to the earth engine asset whose permission
                 you are changing folder/collection/image
  --user USER    Full email address of the user, try using "AllUsers" to make
                 it public
  --role ROLE    Choose between reader, writer or deleteCancel all tasksThis is a simpler tool, can be called directly from the earthengine cli as wellearthengine cli command
earthengine task cancel allUsage using this toolusage: ppipe cancel [-h]

optional arguments:
  -h, --help  show this help message and exitCreditsJetStreamA portion of the work is suported by JetStream Grant TG-GEO160014.Also supported byPlanet Labs Ambassador ProgramOriginal upload function adapted fromLukasz's asset manager toolChangelogv0.4.9fixed issue with selsetupv0.4.8fixed raw_input vs input for py2 and py3 casesupdated selenium setup path for windows vs other operating systemsv0.4.7Download selenium drivers for all operating systemsupdated asset size, acl changer and other toolsfixed metadata tool to work with python3overall general improvements to the tool and updatesadded selenium setup toolsv0.4.6added fix for downloading selenium driversimproved error handlingv0.4.4Fixed selenium download pathStreamlined overall functions of the toolHandles metadata for clipped assets (useporderto clip)Updated selenium uploader to function with earthengine-apiOverall improvements and stability checkv0.4.3Added selenium uploader to upload images to Earth Engine after auth issuesOverall improvements to the tools and added notifications and contribution noticesv0.4.1Major improvements to earth engine tools including better task reporting, batch copy and moveImprovement to the access tool which allows you to add read/write permissions for entire EE folder and collections for specific usersv0.3.8Now include a tool to print your planet quota detailspquotaTool includesdasyncwhich uses thePlanet's Python Client Downloaderto activate, download using multithreadingv0.3.7Now allows users to download from saved searches in planet explorerv0.3.6Now handles complex geometry by using a bounding box for GeoJSON filesv0.3.5Fixed issues with space functionImproved file size parsingv0.3.2Generalized CLI argumentsFixed subprocess shell issue for uploadv0.3.0Allows for quiet authentication for use in Google Colab or non interactive environmentsImproved planet key entry and authentication protocolsv0.2.91Fixed issue with Surface Reflectance metadata and manifest libImproved ingestion support for (PSScene4Band analytic_Sr)[PS4B_SR]v0.2.9Fixed issues with generating id listImproved overall security of command callsv0.2.2Major improvements to ingestion using manifest ingest in Google Earth EngineContains manifest for all commonly used Planet Data item and asset combinationsAdded additional tool to Earth Engine Enhancement including quota check before upload to GEEv0.2.1Fixed initialization loop issuev0.2.0Metadata parser and Uploader Can now handle PlanetScope 4 Band Surface Reflectance DatasetsGeneral Improvementsv0.1.9Changes made to reflect updated GEE Addon toolsgeneral improvementsv0.1.8Minor fixes to parser and general improvementsPlanet Key is now stored in a configuration folder which is safer "C:\users.config\planet"Earth Engine now requires you to assign a field type for metadata meaning an alphanumeric column like satID cannot also have numeric values unless specified explicitly . Manifest option has been added to handle this (just use -mf "planetscope")Added capability to query download size and local disk capacity before downloading planet assets.Added the list function to generate list of collections or folders including reportsAdded the collection size tool which allows you to estimate total size or quota used from your allocated quota.ogr2ft feature is removed since Earth Engine now allows vector and table uploading.