Package: pyspacer
PySpacerPySpacer (AKA spacer) provides utilities to extract features from random point
locations in images and then train classifiers over those features.
It is used in the vision backend ofhttps://github.com/coralnet/coralnet.Spacer currently supports python >=3.8.InstallationThe spacer repo can be installed in three ways.Pip install -- for integration with other Python projects.Local clone -- ideal for testing and development.From Dockerfile -- the only option that supports Caffe, which is used for the legacy feature-extractor.ConfigSetting spacer config variables is only necessary when using certain features. If you don't need S3 storage, and you won't load extractors remotely, you can skip this section.SeeCONFIGURABLE_VARSinconfig.pyfor a full list of available variables, and for an explanation of when each variable must be configured or not.Spacer's config variables can be set in any of the following ways:As environment variables; recommended if youpip installthe package. Each variable name must be prefixed withSPACER_:export SPACER_AWS_ACCESS_KEY_ID='YOUR_AWS_KEY_ID'export SPACER_AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY'export SPACER_AWS_REGION='us-west-2'export SPACER_EXTRACTORS_CACHE_DIR='/your/cache'In asecrets.jsonfile in the same directory as this README; recommended for Docker builds and local clones. Examplesecrets.jsoncontents:{"AWS_ACCESS_KEY_ID":"YOUR_AWS_KEY_ID","AWS_SECRET_ACCESS_KEY":"YOUR_AWS_SECRET_KEY","AWS_REGION":"us-west-2","EXTRACTORS_CACHE_DIR":"/your/cache"}As a Django setting; recommended for a Django project that uses spacer. Example code in a Django settings module:SPACER={'AWS_ACCESS_KEY_ID':'YOUR_AWS_KEY_ID','AWS_SECRET_ACCESS_KEY':'YOUR_AWS_SECRET_KEY','AWS_REGION':'us-west-2','EXTRACTORS_CACHE_DIR':'/your/cache',}Spacer supports the following schemes of using multiple settings sources:Check environment variables first, then use secrets.json as a fallback.Check environment variables first, then use Django settings as a fallback.However, spacer will not read from multiple file-based settings sources; so if a secrets.json file is present, then spacer will not check for Django settings as a fallback.To debug your configuration, try opening a Python shell and runfrom spacer import config, thenconfig.check().Docker buildThe docker build is used in coralnet's deployment.Install docker on your systemClone this repo to a local folder; let's say it's/your/local/pyspacerSet up configuration as detailed above.Choose a local folder for caching extractor files; let's say it's/your/local/cacheBuild image:docker build -f /your/local/pyspacer/Dockerfile -t myimagenameRun:docker run -v /your/local/cache:/workspace/cache -v /your/local/pyspacer:/workspace/spacer -it myimagenameThe-v /your/local/cache:/workspace/cachepart ensures
that all build attempts use the same cache folder of your host storage.The-v /your/local/pyspacer:/workspace/spacermounts your local spacer clone (includingsecrets.json, if used) so that the container has the right permissions.Overall, this runs the default CMD command specified in the dockerfile
(unit-test with coverage). If you want to enter the docker container,
run the same command but appendbashin the end.Pip installpip install pyspacerSet up configuration.secrets.jsonisn't an option here, so either use environment variables, or Django settings if you have a Django project.Local cloneClone this repoIf using Windows: turn Git'sautocrlfsetting off before your initial checkout. Otherwise, pickled classifiers inspacer/tests/fixtureswill get checked out with\r\nnewlines, and the pickle module will fail to load them, leading to test failures. However, autocrlf should be left on when adding any new non-pickle files.pip install -r requirements.txtSet up configurationCode overviewSpacer executes tasks as defined in messages. The message types are defined
inmessages.pyand the tasks intasks.py. Several data types which can be used for input and output serialization are defined
indata_classes.py.For examples on how to create spacer tasks, refer to the Core API section below, and the unit tests intest_tasks.py.Tasks can be executed directly by calling the methods in tasks.py.
However, spacer also supports an interface with AWS Batch
handled byenv_job()inmailman.py.Spacer supports four storage types:s3,filesystem,memoryandurl.
Refer tostorage.pyfor details. The memory storage is mostly used for
testing, and the url storage is read only.config.pydefines configurable variables/settings and various constants.Core APIThetasks.pymodule has four functions which comprise the main interface of pyspacer:extract_featuresThe first step when analyzing an image, or preparing an image as training data, is extractingfeaturesfrom the image. For this step, you specify a set of points (pixel locations) in the image which you want to analyze. At each point, spacer will crop a square of pixels centered around that location and extract features based on that square.You'll also need a feature extractor, but spacer does not provide one out of the box. Spacer'sextract_features.pyprovides the Python classesEfficientNetExtractorfor loading EfficientNet extractors in PyTorch format (CoralNet 1.0's default extraction scheme), andVGG16CaffeExtractorfor loading VGG16 extractors in Caffe format (CoralNet's legacy extraction scheme).You'll either want to match one of these schemes so you can use the provided classes, or you'll have to write your own extractor class which inherits from the base classFeatureExtractor. Between the provided classes, the easier one to use will probably beEfficientNetExtractor, because Caffe is old software which is more complicated to install.If you're loading the extractor files remotely (from S3 or from a URL), the files will be automatically cached to your configuredEXTRACTORS_CACHE_DIRfor faster subsequent loads.The output ofextract_features()is a single feature-vector file, which is a JSON file that is deserializable using thedata_classes.ImageFeaturesclass. Example usage:fromspacer.extract_featuresimportEfficientNetExtractorfromspacer.messagesimportDataLocation,ExtractFeaturesMsgfromspacer.tasksimportextract_featuresmessage=ExtractFeaturesMsg(# This token is purely for your bookkeeping; you may find it useful if you# choose to track tasks by saving these task messages. For example, you# can make the token something that uniquely identifies the input image.job_token='image1',# Instantiated feature extractor. Each extractor class defines the# data_locations which must be specified. In EfficientNetExtractor's case,# a PyTorch 'weights' file is required.extractor=EfficientNetExtractor(data_locations=dict(weights=DataLocation('filesystem','/path/to/weights.pt'),),),# (row, column) tuples specifying pixel locations in the image.# Note that row is y, column is x.rowcols=[(2200,1000),(1400,1500),(3000,450)],# Where the input image should be read from.image_loc=DataLocation('filesystem','/path/to/image1.jpg'),# Where the feature vector should be output to.# CoralNet uses a custom .featurevector extension for these, but the# format is just JSON.feature_loc=DataLocation('filesystem','/path/to/image1.featurevector'),)return_message=extract_features(message)print("Feature vector stored at: /path/to/image1.featurevector")print(f"Extraction runtime:{return_message.runtime:.1f}s")train_classifierTo train a classifier, you need:Feature vectors; each vector corresponds to a set of point locations in one imageGround-truth labels (typically applied/confirmed by human inspection) for each of the points specified in those feature vectorsTraining parametersThe labels must be split into training, reference, and validation sets:The training set (train) is the data that actually goes into the classifier training algorithm during each training epoch. This is generally much larger than the other two sets.The reference set (ref) is used to evaluate and calibrate the classifier between epochs.The validation set (val) is used to evaluate the final classifier after training is finished.This three-set split is known by other names elsewhere, such astraining, validation, and test setsrespectively, ortraining, development, and test setsrespectively.There are a few ways to create thelabelsstructure. Each way involves creating one or more instances ofdata_classes.ImageLabels:fromspacer.data_classesimportImageLabelsimage_labels=ImageLabels({# Labels for one feature vector's points.'/path/to/image1.featurevector':[# Point location at row 1000, column 2000, labeled as class 1.(1000,2000,1),# Point location at row 3000, column 2000, labeled as class 2.(3000,2000,2),],# Labels for another feature vector's points.'/path/to/image2.featurevector':[(1500,2500,3),(2500,500,1),],})Thelabelsargument ofTrainClassifierMsgexpects an instance ofdata_classes.TrainingTaskLabels. There are a few ways to create this:Pass a single ImageLabels instance to thetask_utils.preprocess_labels()function. preprocess_labels() then decides how to split up your labels into train, ref, and val (while doing error checks in the meantime), and creates a TrainingTaskLabels instance from there.Pass three ImageLabels instances to the TrainingTaskLabels constructor: one instance for each of train, ref, and val.Do method 1, but also specify theaccepted_classesargument to preprocess_labels(); this makes the function filter out any labels that aren't in the desired set of classes.Do method 2, but also pass the TrainingTaskLabels through preprocess_labels(). This allows you to use the error-checking and accepted_classes parts of preprocess_labels(), and the train/ref/val split you defined will remain intact.fromspacer.data_classesimportImageLabelsfromspacer.messagesimportTrainingTaskLabelsfromspacer.task_utilsimportpreprocess_labels# 1labels=preprocess_labels(ImageLabels(...))# 2labels=TrainingTaskLabels(train=ImageLabels(...),ref=ImageLabels(...),val=ImageLabels(...))# 3labels=preprocess_labels(ImageLabels(...),accepted_classes={...})# 4labels=preprocess_labels(TrainingTaskLabels(...),accepted_classes={...})So, pass that and the other required arguments to TrainClassifierMsg, and then pass that message totrain_classifier(), which produces:A classifier as ansklearn.calibration.CalibratedClassifierCVinstance, stored in a pickle (.pkl) file. spacer'sstorage.load_classifier()function can help with loading classifiers that were created in older scikit-learn versions.Classifier evaluation results as a JSON file.Example:fromspacer.data_classesimportImageLabelsfromspacer.messagesimportDataLocation,TrainClassifierMsgfromspacer.tasksimporttrain_classifierfromspacer.task_utilsimportpreprocess_labelsmessage=TrainClassifierMsg(# For your bookkeeping.job_token='classifier1',# 'minibatch' is currently the only trainer that spacer defines.trainer_name='minibatch',# How many iterations the training algorithm should run; more epochs# = more opportunity to converge to a better fit, but slower.nbr_epochs=10,# Classifier types available:# 1. 'MLP': multi-layer perceptron; newer classifier type for CoralNet# 2. 'LR': logistic regression; older classifier type for CoralNet# Both types are run with scikit-learn.clf_type='MLP',# Point-locations to ground-truth-labels (annotations) mappings# used to train the classifier.# The dict keys must be the same as the `key` used in the# extract-features task's `feature_loc`.# The dict values are lists of tuples of (row, column, label ID).# You'll need to be tracking a mapping of integer label IDs to the# labels you use.# preprocess_labels() can automatically split the data into training,# reference, and validation sets. However, you may also define how to# split it yourself; for details, see `TrainingTaskLabels` comments# in messages.py.labels=preprocess_labels(ImageLabels({'/path/to/image1.featurevector':[(1000,2000,1),(3000,2000,2)],'/path/to/image2.featurevector':[(1000,2000,3),(3000,2000,1)],'/path/to/image3.featurevector':[(1234,2857,11),(3094,2262,25)],})),# All the feature vectors should use the same storage_type, and the same# S3 bucket_name if applicable. This DataLocation's purpose is to describe# those common storage details. The key arg is ignored, because that will# be different for each feature vector.features_loc=DataLocation('filesystem',''),# List of previously-created models (classifiers) to also evaluate# using this validation set, for informational purposes only.# This can be handy for comparing classifiers.previous_model_locs=[DataLocation('filesystem','/path/to/oldclassifier1.pkl'),DataLocation('filesystem','/path/to/oldclassifier2.pkl'),],# Where the new model (classifier) should be output to.model_loc=DataLocation('filesystem','/path/to/classifier1.pkl'),# Where the detailed evaluation results of the new model should be stored.valresult_loc=DataLocation('filesystem','/path/to/valresult.json'),)return_message=train_classifier(message)print("Classifier stored at: /path/to/classifier1.pkl")print("Evaluation results stored at: /path/to/valresult.json")print(f"New model's accuracy (0.0 = 0%, 1.0 = 100%):{return_message.acc}")print(f"Previous models' accuracies on the validation set:"f"{return_message.pc_accs}")print("New model's accuracy progression (calculated on the reference set)"f" after each epoch of training:{return_message.ref_accs}")print(f"Training runtime:{return_message.runtime:.1f}s")Evaluation results consist of three arrays:gt: Ground-truth label IDs for each point in the validation set.est: Estimated (classifier-predicted) label IDs for each point in the validation set.scores: Classifier's confidence scores (0.0 = 0%, 1.0 = 100%) for each estimated label ID.Theith element ofgt,ith element ofest, andith element ofscorescorrespond to each other. But the elements are otherwise in an undefined order.Accuracy is defined as the percentage ofgtlabels that match the correspondingestlabels.Here's a snippet which lists out the evaluation results:importjsonfromspacer.data_classesimportValResultswithopen('/path/to/valresult.json')asf:valresult=ValResults.deserialize(json.load(f))forground_truth_i,prediction_i,scoreinzip(valresult.gt,valresult.est,valresult.scores):print(f"Actual ={valresult.classes[ground_truth_i]},"f" Predicted ={valresult.classes[prediction_i]},"f" Confidence ={100*score:.1f}%")classify_featuresTakes a feature vector (representing points in an image) to classify, and a classifier trained on the same type of features (EfficientNet or VGG16). Produces prediction results (scores) for the image points, as posterior probabilities for each class. Example:fromspacer.messagesimportDataLocation,ClassifyFeaturesMsgfromspacer.tasksimportclassify_featuresmessage=ClassifyFeaturesMsg(# For your bookkeeping.job_token='image1',# Where the input feature-vector should be read from.feature_loc=DataLocation('filesystem','/path/to/image1.featurevector'),# Where the classifier should be read from.classifier_loc=DataLocation('filesystem','/path/to/classifier1.pkl'),)return_message=classify_features(message)print(f"Classification runtime:{return_message.runtime:.1f}s")print(f"Classes (recognized labels):{return_message.classes}")print("Classifier's scores for each point in the feature vector;"" scores are posterior probabilities of each class, with classes"" ordered as above:")forrow,col,scoresinreturn_message.scores:print(f"Row{row}, column{col}:{scores}")The label which has the highest score for a particular point (row-column position) can be considered the classifier's predicted label for that point.One possible usage strategy is to trust the classifier's predictions for points where the highest confidence score is above a certain threshold, such as 0.8 (80%), and have human annotators check all other points.classify_imageThis basically doesextract_featuresandclassify_featurestogether in one go, without needing to specify a storage location for the feature vector.Takes an image, a list of pixel locations on that image, a feature extractor, and a classifier. Produces prediction results (scores) for the image points, as posterior probabilities for each class. Example:fromspacer.extract_featuresimportEfficientNetExtractorfromspacer.messagesimportDataLocation,ClassifyImageMsgfromspacer.tasksimportclassify_imagemessage=ClassifyImageMsg(# For your bookkeeping.job_token='image1',# Where the input image should be read from.image_loc=DataLocation('filesystem','/path/to/image1.jpg'),# Instantiated feature extractor.extractor=EfficientNetExtractor(data_locations=dict(weights=DataLocation('filesystem','/path/to/weights.pt'),),),# (row, column) tuples specifying pixel locations in the image.# Note that row is y, column is x.rowcols=[(2200,1000),(1400,1500),(3000,450)],# Where the classifier should be read from.classifier_loc=DataLocation('filesystem','/path/to/classifier1.pkl'),)return_message=classify_image(message)print(f"Runtime:{return_message.runtime:.1f}s")print(f"Classes (recognized labels):{return_message.classes}")print("Classifier's scores for each point in rowcols;"" scores are posterior probabilities of each class, with classes"" ordered as above:")forrow,col,scoresinreturn_message.scores:print(f"Row{row}, column{col}:{scores}")Unit testsIf you are using the docker build or local install, you can run the test suite by runningpython -m unittestfrom thespacerdirectory.Expect many tests to be skipped, since most test fixtures aren't set up for public access yet.Run just a single test module with a command likepython -m unittest tests.test_tasks, or justpython -m tests.test_tasks(the latter invokes theif __name__ == '__main__':part of the module).You can get logging output during test runs with theLOG_DESTINATIONandLOG_LEVELvars in config.py.You can check code coverage like so:coverage run --source=spacer --omit=spacer/tests/* -m unittest    
coverage report -m
coverage html