Package: openmapflow
OpenMapFlow 🌍Rapid map creation with machine learning and earth observation data.Example projects:Cropland,Buildings,MaizeExample maps:Earth Engine scriptTutorialCreating a map from scratchGenerating a projectAdding dataTraining a modelCreating a mapAccessing existing datasetsTutorialColab notebook tutorial demonstrating data exploration, model training, and inference over small region. (video)Prerequisites:Github access token (obtainedhere)Forked OpenMapFlow repositoryBasic Python knowledgeCreating a map from scratchTo create your own maps with OpenMapFlow, you need toGenerate your own OpenMapFlow project, this will allow you to:Add your own labeled dataTrain a modelusing that labeled data, andCreate a mapusing the trained model.Generating a projectA project can be generated by either following the below documentation OR running the above Colab notebook.Prerequisites:Github repository- where your project will be storedGoogle/Gmail based account- for accessing Google Drive and Google CloudGoogle Cloud Project (create) - for accessing Cloud resources for creating a map (additional info)Google Cloud Service Account Key (generate) - for deploying Cloud resources from Github ActionsOnce all prerequisites are satisfied, inside your Github repository run:pipinstallopenmapflow
openmapflowgenerateThe command will prompt for project configuration such as project name and Google Cloud Project ID. Several prompts will have defaults shown in square brackets. These will be used if nothing is entered.After all configuration is set, the following project structure will be generated:<YOUR PROJECT NAME>
│   README.md
│   datasets.py             # Dataset definitions (how labels should be processed)
│   evaluate.py             # Template script for evaluating a model
│   openmapflow.yaml        # Project configuration file
│   train.py                # Template script for training a model
│   
└─── .dvc/                  # https://dvc.org/doc/user-guide/what-is-dvc
│       
└─── .github
│   │
│   └─── workflows          # Github actions
│       │   deploy.yaml     # Automated Google Cloud deployment of trained models
│       │   test.yaml       # Automated integration tests of labeled data
│       
└─── data
    │   raw_labels/                     # User added labels
    │   datasets/                       # ML ready datasets (labels + earth observation data)
    │   models/                         # Models trained using datasets
    |   raw_labels.dvc                  # Reference to a version of raw_labels/
    |   datasets.dvc                    # Reference to a version of datasets/
    │   models.dvc                      # Reference to a version of models/Github Actions SecretsBeing able to pull and deploy data inside Github Actions requires access to Google Cloud. To allow the Github action to access Google Cloud, add a new repository secret (instructions).In step 5 of the instructions, name the secret:GCP_SA_KEYIn step 6, enter your Google Cloud Service Account KeyAfter this the Github actions should successfully run.GCloud Bucket:
A Google Cloud bucket must be created for the labeled earth observation files. Assuming gcloud is installed run:gcloudauthlogin
gsutilmb-l<YOUR_OPENMAPFLOW_YAML_GCLOUD_LOCATION>gs://<YOUR_OPENMAPFLOW_YAML_BUCKET_LABELED_EO>Adding dataAdding already existing dataPrerequisites:Generated OpenMapFlow projectAdd reference to already existing dataset in your datasets.py:fromopenmapflow.datasetsimportGeowikiLandcover2017,TogoCrop2019datasets=[GeowikiLandcover2017(),TogoCrop2019()]Download and push datasetsopenmapflowcreate-datasets# Download datasetsdvccommit&&dvcpush# Push data to version controlgitadd.
gitcommit-m'Created new dataset'gitpushAdding custom dataData can be added by either following the below documentation OR running the above Colab notebook.Prerequisites:Generated OpenMapFlow projectEarthEngine account- for accessing Earth Engine and pulling satellite dataRaw labels - a file (csv/shp/zip/txt) containing a list of labels and their coordinates (latitude, longitude)Pull the latest datadvcpullMove raw label files into project's data/raw_labels folderWrite aLabeledDatasetclass indatasets.pywith aload_labelsfunction that converts raw labels to a standard format, example:label_col="is_crop"classTogoCrop2019(LabeledDataset):defload_labels(self)->pd.DataFrame:# Read in raw label filedf=pd.read_csv(PROJECT_ROOT/DataPaths.RAW_LABELS/"Togo_2019.csv")# Rename coordinate columns to be used for getting Earth observation datadf.rename(columns={"latitude":LAT,"longitude":LON},inplace=True)# Set start and end date for Earth observation datadf[START],df[END]=date(2019,1,1),date(2020,12,31)# Set consistent label columndf[label_col]=df["crop"].astype(float)# Split labels into train, validation, and test setsdf[SUBSET]=train_val_test_split(index=df.index,val=0.2,test=0.2)# Set country column for later analysisdf[COUNTRY]="Togo"returndfdatasets:List[LabeledDataset]=[TogoCrop2019(),...]Check your new datasetload_labelsfunctionopenmapflowverifyTogoCrop2019Run dataset creation (can be skipped if automated in CI e.g. inhttps://github.com/nasaharvest/crop-mask):earthengineauthenticate# For getting new earth observation datagcloudauthlogin# For getting cached earth observation dataopenmapflowcreate-datasets# Initiatiates or checks progress of dataset creationPush new data to remote storage and new code to Githubdvccommit&&dvcpush
gitadd.
gitcommit-m'Created new dataset'gitpushTraining a modelA model can be trained by either following the below documentation OR running the above Colab notebook.Prerequisites:Generated OpenMapFlow projectAdded labeled data# Pull in latest datadvcpull# Set model name, train model, record test metricsexportMODEL_NAME=<YOURMODELNAME>pythontrain.py--model_name$MODEL_NAMEpythonevaluate.py--model_name$MODEL_NAME# Push new models to data version controldvccommitdvcpush# Make a Pull Request to the repositorygitcheckout-b"$MODEL_NAME"gitadd.
gitcommit-m"$MODEL_NAME"gitpush--set-upstreamorigin"$MODEL_NAME"Now after merging the pull request, the model will be deployed to Google Cloud.Creating a mapPrerequisites:Generated OpenMapFlow projectAdded labeled dataTrained modelOnly available through above Colab notebook. Cloud Architecture must be deployed using the deploy.yaml Github Action.Accessing existing datasetsfromopenmapflow.datasetsimportTogoCrop2019df=TogoCrop2019().load_df(to_np=True)x=df.iloc[0]["eo_data"]y=df.iloc[0]["class_prob"]Citation@inproceedings{OpenMapFlow2023,
  title={OpenMapFlow: A Library for Rapid Map Creation with Machine Learning and Remote Sensing Data},
  author={Zvonkov, Ivan and Tseng, Gabriel and Nakalembe, Catherine and Kerner, Hannah},
  booktitle={AAAI},
  year={2023}
}