Package: pytorch-caney
pytorch-caneyPython package for lots of Pytorch tools.DocumentationLatest:https://nasa-nccs-hpda.github.io/pytorch-caney/latestObjectivesLibrary to process remote sensing imagery using GPU and CPU parallelization.Machine Learning and Deep Learning image classification and regression.Agnostic array and vector-like data structures.User interface environments via Notebooks for easy to use AI/ML projects.Example notebooks for quick AI/ML start with your own data.InstallationThe following library is intended to be used to accelerate the development of data science products
for remote sensing satellite imagery, or any other applications. pytorch-caney can be installed
by itself, but instructions for installing the full environments are listed under the requirements
directory so projects, examples, and notebooks can be run.Note: PIP installations do not include CUDA libraries for GPU support. Make sure NVIDIA libraries
are installed locally in the system if not using conda/mamba.moduleloadsingularity# if a module needs to be loadedsingularitybuild--sandboxpytorch-caney-containerdocker://nasanccs/pytorch-caney:latestWhy Caney?"Caney" means longhouse in Taíno.ContributorsJordan Alexis Caraballo-Vega,jordan.a.caraballo-vega@nasa.govCaleb Spradlin,caleb.s.spradlin@nasa.govContributingPlease see ourguide for contributing to pytorch-caney.SatVisionnamepretrainresolution#paramsSatVision-BMODIS-1.9-M192x19284.5MSatVision Datasetsnamebandsresolution#chipsMODIS-Small7128x1281,994,131Pre-training with Masked Image ModelingTo pre-train the swinv2 base model with masked image modeling pre-training, run:torchrun--nproc_per_node<NGPUS>pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py--cfg<config-file>--dataset<dataset-name>--data-paths<path-to-data-subfolder-1>--batch-size<batch-size>--output<output-dir>--enable-ampFor example to run on a compute node with 4 GPUs and a batch size of 128 on the MODIS SatVision pre-training dataset with a base swinv2 model, run:singularityshell--nv-B<mounts>/path/to/container/pytorch-caney-container
Singularity>exportPYTHONPATH=$PWD:$PWD/pytorch-caney
Singularity>torchrun--nproc_per_node4pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py--cfgpytorch-caney/examples/satvision/mim_pretrain_swinv2_satvision_base_192_window12_800ep.yaml--datasetMODIS--data-paths/explore/nobackup/projects/ilab/data/satvision/pretraining/training_*--batch-size128--output.--enable-ampThis example script runs the exact configuration used to make the SatVision-base model pre-training with MiM and the MODIS pre-training dataset.singularityshell--nv-B<mounts>/path/to/container/pytorch-caney-container
Singularity>cdpytorch-caney/examples/satvision
Singularity>./run_satvision_pretrain.shFine-tuning Satvision-baseTo fine-tune the satvision-base pre-trained model, run:torchrun--nproc_per_node<NGPUS>pytorch-caney/pytorch_caney/pipelines/finetuning/finetune.py--cfg<config-file>--pretrained<path-to-pretrained>--dataset<dataset-name>--data-paths<path-to-data-subfolder-1>--batch-size<batch-size>--output<output-dir>--enable-ampSee example config files pytorch-caney/examples/satvision/finetune_satvision_base_*.yaml to see how to structure your config file for fine-tuning.TestingFor unittests, run this bash command to run linting and unit test runs. This will execute unit tests and linting in a temporary venv environment only used for testing.gitclonegit@github.com:nasa-nccs-hpda/pytorch-caney.gitcdpytorch-caney;bashtest.shor run unit tests directly with container or anaconda envgitclonegit@github.com:nasa-nccs-hpda/pytorch-caney.git
singularitybuild--sandboxpytorch-caney-containerdocker://nasanccs/pytorch-caney:latest
singularityshell--nv-B<mounts>/path/to/container/pytorch-caney-containercdpytorch-caney;python-munittestdiscoverpytorch_caney/testsgitclonegit@github.com:nasa-nccs-hpda/pytorch-caney.gitcdpytorch-caney;condaenvcreate-frequirements/environment_gpu.yml;condaactivatepytorch-caney
python-munittestdiscoverpytorch_caney/testsReferencesPytorch LightningSwin TransformerSimMIM