Package: alectio-sdk
RequirementsPython3 (Required)PIP3    (Required)Ubuntu 16.04+ / MacOS / Windows 10GCC / C++ (Will depend on OS you are using. Ubuntu, MacOS it comes default. Some falvours of linux distribution like Amazon Linux/RED Hat linux might not have GCC or C++ realted libraires installed)For this tutorial, we are assuming you are using Python3 and PIP3. Also, make sure you have the necessary build tools installed (might vary from OS to OS). If you get any errors while installing any dependent packages feel free to reach out to us but most of it can quickly be solved by a simple Google search.Installation1. Key ManagementIf you have not already created your Client ID and Client Secret then do so by visiting:openhttps://pro.alectio.com/Login there and create project and an experiment.An experiment token will be generated.Enter your experiment token in main.py (refer example in step 3) to authenticate.2. Set up a virtual environmentWe recommend to set-up a virtual environment.For example, you can use python's built-in virtual environment via:python3 -m venv env
source env/bin/activate3. Download the examplesAll examples available inexamplesdirectory4. Install the requirements in examplespip install -r requirements.txt5. Run ExamplesThe remaining installation instructions are detailed in theexamplesdirectory. We cover one example forimage classification.Alectio SDKAlectioSDK is a package that enables developers to build an ML pipeline as a Flask app to interact with Alectio's platform. It is designed for Alectio's clients, who prefer to keep their model and data on their own server.The package is currently under active development. More functionalities that aim to enhance robustness will be added soon, but for now, the package provides a classalectio_sdk.sdk.Pipelinethat interfaces with customer-side processes in a consistent manner. Customers need to implement 4 processes as python functions:A process to train the modelA process to test the modelA process to apply the model to infer unlabeled dataA process to assign each data point in the dataset to a unique index (Refer to one of the examples to know how)A Pipeline can be created inside themain.pyfile using the following syntax:importyamlfromalectio_sdk.sdkimportPipelinefromprocessesimporttrain,test,infer,getdatasetstate# All the variables can be declared inside the .yaml filewithopen("./config.yaml","r")asstream:args=yaml.safe_load(stream)# Initialising the Experiment PipelineAlectioPipeline=Pipeline(name=args["exp_name"],train_fn=train,# A process to train the modeltest_fn=test,# A process to test the modelinfer_fn=infer,# A process to apply the model to infer unlabeled datagetstate_fn=getdatasetstate,# A process to assign each data point in the dataset to a unique indexargs=args,# Any arguments that user ants to use inside his train, test, infer functions.token="xxxxxx7041a6xxxxx7948cexxxxxxxx",# Experiment tokenmultiple_initialisations={"seeds":[],"limit_value":0},# Multiple seed initialisation feature)Refer to the alectio examples for more clarity on the use of the Pipeline class.Train the ModelThe logic for training the model should be implemented in this process. The function should look like this:deftrain(args,labeled,resume_from,ckpt_file):"""Training FunctionInput args:args* # Arguments passed to Alectio Pipelinelabeled: list # List of labeled indices for trainingresume_from: str # Path to last checkpoint fileckpt_file: str # Path to saved modelReturns:Noneoroutput_dict: dict # Labels and Hyperparams"""# implement your logic to train the model# with the selected data indexed by `labeled`# lbs <- dictionary of indices of train data and their ground-truthreturn{'labels':lbs,'hyperparams':hyperparameters}The name of the function can be anything you like. It takes an argument as shown in the example above.keyvalueresume_froma string that specifies which checkpoint to resume fromckpt_filea string that specifies the name of checkpoint to be saved for the current looplabeleda list of indices of selected samples used to train the model in this loopDepending on your situation, the samples indicated in labeled might not be labeled (despite the variable name). We call it labeled because, in the active learning setting, this list represents the pool of samples iteratively labeled by the human oracle.Test the ModelThe logic for testing the model should be implemented in this process. The function representing this process should look like this:deftest(args,ckpt_file):"""testing functionInput args:args* # Arguments passed to Alectio Pipelineckpt_file: str # Path to saved modelReturns:output_dict: dict # Preds and Labels"""# implement your testing logic here# put the predictions and labels into# two dictionaries# lbs <- dictionary of indices of test data and their ground-truth# prd <- dictionary of indices of test data and their predictionreturn{'predictions':prd,'labels':lbs}The test function takes arguments as shown in the example above.keyvalueckpt_filea string that specifies which checkpoint to test modelThe test function needs to return a dictionary with two keys:keyvaluepredictionsa dictionary of an index and a prediction for each test samplelabelsa dictionary of an index and a ground truth label for each test sampleThe format of the values depends on the type of ML problem. Please refer to the examples directory for details.Apply InferenceThe logic for applying the model to infer the unlabeled data should be implemented in this process. The function representing this process should look like this:definfer(args,unlabeled,ckpt_file):"""Inference FunctionInput args:args* # Arguments passed to Alectio Pipelineunlabeled: list # List of labeled indices for inferenceckpt_file: str # Path to saved modelreturns:output_dict: dict"""# implement your inference logic here# outputs <- save the output from the model on the unlabeled data as a dictionaryreturn{'outputs':outputs}The infer function takes an argument payload, which is a dictionary with 2 keys:keyvalueckpt_filea string that specifies which checkpoint to use to infer on the unlabeled dataunlabeleda list of of indices of unlabeled data in the training setThe infer function needs to return a dictionary with one key.keyvalueoutputsa dictionary of indexes mapped to the models output before an activation function is appliedFor example, if it is a classification problem, return the outputbeforeapplying softmax.
For more details about the format of the output, please refer to theexamplesdirectory.config.yamlPut in all the requirements that are required for the model to train. This will be read and used in processes.py when the model trains. For example if config.yaml looks like this:LOG_DIR:"./log"DATA_DIR:"./data"EXPT_DIR:"./log"exptname:"ManualAL"# Model configsbackbone:"Resnet101"description:"Pedestrian detection"epochs:10..You can access them inside your any of the above 4 processes as lets say args["backbone"] , args["description"] etc.SDK- Features1. Tracking CO2 emissionsThe alectio SDK is capable of tracking the CO2 emissions during the experiment. The SDK uses an open-source package called code carbon to track the CO2 emissions along with the (CPU, GPU, and RAM) usage. This data is tracked and synced, once the experiment ends, with the user account where the user can see the total CO2 emission on his dashboard.2. Time-Saved InformationThe SDK uses linear interpolation to estimate the time that a user saved to train his model in each active learning cycle. The time-saved information is logged after each AL cycle and gets synced with the platform at the end of the experiment. The time-saved insights can be seen on the user dashboard.3. Storing HyperparametersThe SDK has the ability to track the hyperparameters for each AL cycle. To use this feature the user just needs to return a dictionary of their hyperparameters. Currently, the SDK supports a limited number of hyperparameters, the list of these parameters is shown below:hyperparameter_names=["optimizer_name",# Name of the optimizer used"loss",# Loss of the training process"running_loss",# Running Loss"epochs",# Number of epochs for which the model was trained"batch_size",# batch size on which the model was trained"loss_function",# name of loss function used for training"activation",# List of activation functions used"optimizer",# Can be a state_dict in case of Pytorch]The syntax for storing these values is shown in the train function section.4. Running Multiple Seed InitializationThe SDK can also help the user choose the right seed for his experiment by training his model on a range of seed values and selecting the best seed depending on the performance of models on these seed values. In order to use this feature the user can just use the multiple_initialisations argument of the Alectio Pipeline. The syntax is as shown below:fromalectio_sdk.sdkimportPipelineAlectioPipeline=Pipeline(name=args["exp_name"],train_fn=train,test_fn=test,infer_fn=infer,getstate_fn=getdatasetstate,args=args,token="xxxxxx7041a6xxxxx7948cexxxxxxxx",multiple_initialisations={"seeds":[10,42,36,78],"limit_value":4000},)The input of this argument is a dict with 2 keys.keyvalueseeda list containing different seed values you want to test your model on.limit_valueThe number of samples from which you want to select the training samples from.5. Accessing Alectio Public DatasetsThe user can access Alectio Public Datasets usin the Alectio SDK. The user needs to select the public dataset he wants to use during creating his/her project on the Alectio platform. Alectio Public datasets contain training, validation and testing data. The code snippet to use the Public datasets is as given below.1. Pytorch# Pytorch Syntaximporttorchvisionfromtorchvisionimporttransformsfromalectio_sdk.sdk.alectio_datasetimportAlectioDatasetfromtorch.utils.dataimportDataLoader,Subset# create a public dataset object# token = experiment token# root = directory in which you want to download your dataset# framework = pytorch/tensorflowalectio_dataset=AlectioDataset(token="your_exp_token_goes_here",root="./data",framework="pytorch")# train datasettrain_transforms=transforms.Compose([transforms.Resize((224,224)),transforms.RandomHorizontalFlip(),transforms.ToTensor(),transforms.Normalize(mean=[0.485,0.456,0.406],std=[0.229,0.224,0.225]),])# call the get dataset function# dataset_type = train/test/validation# transforms = augmentations/transformations you want to perform# Returns# DataLoader Object | Length of dataset | Mapping of labels and indicestrain_dataset,train_dataset_len,train_class_to_idx=alectio_dataset.get_dataset(dataset_type="train",transforms=train_transforms)2. Tensorflow# Tensorflow Syntaximporttensorflowastffromalectio_sdk.sdk.alectio_datasetimportAlectioDataset# create a public dataset object# token = experiment token# root = directory in which you want to download your dataset# framework = pytorch/tensorflowalectio_dataset=AlectioDataset(token="your_exp_token_goes_here",root="./data",framework="tensorflow")# train dataset# all transforms supported by Tensoflow ImageDataGenerator can be added to the transform dicttrain_transforms=dict(featurewise_center=False,samplewise_center=False,featurewise_std_normalization=False,samplewise_std_normalization=False,zca_whitening=False,channel_shift_range=0.0,fill_mode='nearest',cval=0.0,horizontal_flip=False,vertical_flip=False,rescale=None,preprocessing_function=None,data_format=None,)# call the get dataset function# dataset_type = train/test/validation# transforms = dict of augmentations/transformations you want to perform# Returns# Imagedatagenerator Object | Length of dataset | Mapping of labels and indicestrain_dataset,train_dataset_len,train_class_to_idx=alectio_dataset.get_dataset(dataset_type="train",transforms=train_transforms)