Package: pipeline-vilma
Pipeline VilmaIntroductionPipeline Vilma is a Data Pipeline designed to be used in Machine Learning
projects. The main goal is to create a pipeline which allows user to only
annotate information and let the process to the pipeline.The design is based on components communicating through messages, therefore it
is possible to use them all together, as we design or connect the components to
your current data pipeline.ComponentsProviderclassProvider:"""An abstract class used to represent a data provider.Examples of data providers:- RGB Camera, providing frames- Humidity sensor, providing actual humidity level- Termographic camera, providing a heatmap...Attributes----------queue_server : strthe IPv4 of a RabbitMQ queue serverqueue_output : strthe name of the queue where provider will publish their datasensor_id : stran unique identification of the provider, e.g, rgbcamera, microphone, etcMethods-------pack_data(self, data_type, content)Based on the data type, choose the right package format for the contentrun(self, loop_interval=0.016)Main loop method that should be implemented, in order to describe a providerbehaviourforward(self, message)Forward the package to the next component of the pipeline (queue_output)"""CollectorclassCollector:"""An abstract class used to represent a data collector.The component is the responsible for creating a dataset from providers input...Attributes----------bypass : boolsome applications cannot store data, and or create dataset for privacyreasons, therefore collector can work in bypass mode where it only forwardsthe received dataratio : floatclassthe percentage (between [0,1]) number of samples to be collected, i.e.,if "ratio" is set to 0.5, half of the samples received on collector will bestored for dataset creationqueue_server : strthe IPv4 of a RabbitMQ queue serverqueue_input: strthe name of the queue from where collector consumequeue_output : strthe name of the queue where provider will publish their datadata_api_config: dictexpects a Python dictionary with the endpoints to map the class to an userimplemented API. It allows the component to support different data APIsolutions (like different databases, languages, implementations) as longas the expected endpoints are supported.Expected dictionary:{"base_url": "http://192.168.0.1" # IP where the API is running"port": 3333,                    # The API's server portclass"endpoints": {"add_item": "/collection","get_item": "/collection/<ID>","mark_trained": "/collection/<ID>","count_labeled": "/collector/counter","get_training_set": "/dataset/"}}Endpoints description:- add_item- Method type: HTTP POST- URL format: http://host:port/resource- Data: support the JSON messager structure, e.g. ImageBase64- Request example:- http://192.168.0.1:3333/collection- Data: {"x": data, "y": data }- Response example:- The JSON messager sent- get_item- Method type: HTTP GET- URL format: http://host:port/resource/<ID>- Request example:- http://192.168.0.1:3333/collection/1- Response example:- A JSON pipeline supported message item- mark_trained- Method type: HTTP PUT- URL format: http://host:port/resource/<ID>- Data: {"status": "readyForTraining"}- Request example:- http://192.168.0.1:3333/collection/1- Data: {"status": "readyForTraining"}- Response example:{"message": "success"}- count_labeled:- Method type: HTTP GET- URL format: http://host:port/resource- Request example:- http://192.168.0.1:3333/collection-counter- Response example:- { "readyForTraining": 123 }- get_training_set:- Method type: HTTP GET- URL format: http://host:port/resource- Request example:- http://192.168.0.1:3333/training-set- Response example:- { "readyForTraining": ["itemId1", "itemId2"] }Methods-------run(self)Starts consuming collectors input queueon_new_message(self, ch, method, properties, body)Based on component's configuration, perform the actions on eachreceived message.Finally, forwards the current message state to the next componentof the pipeline.receive(self, callback)Creates a listener on collector's input queueforward(self, sensor_id, message)Forward the message to an individual queue related to the uniquesensor id"""CollectorDataApiclassCollectorDataApi:"""A class used to link an HTTP Data API to Collector component.It is reponsible to allow users to create different APIsimplementation and by that the usage of any database.To connect it to the pipeline, it is required theimplementation of the APIs described in Collector....Statics----------STATUS_FOR_LABELING : strdefault value: 'labeling-required'an internal status used to notify other components about adata item waiting for labelingSTATUS_READY_FOR_TRAINING : strdefault value: 'readyForTraining'an internal status used to notify other componentsabout a data item ready for trainingAttributes----------storage_api : dictexpects a Python dictionary with the endpoints to an APIimplemented to store the data on a database or any otherpersistence systemAn API, persisting the data on a MongoDB is available on the examples.Methods-------def _add_labeling_properties(self, message)Include on the message the labeling status and creation timestampdef _insert_on_db(self, message)Make a request to the API to persist the messagedef add_item(self, message)def get_item(self, item_id)def mark_trained(self, item_id, status)def get_training_data(self)def count_labeled(self)def store(self, message)public method responsible to execute the private ones and storethe message on the database"""FileManagerclassFileManager:"""A class used to create and manage required files for training and evaluationprocess.To connect it to the pipeline, it is required the implementation of the APIendpoints described in CollectorDataApi....Attributes----------queue_server : strthe IPv4 of a RabbitMQ queue serverqueue_output : strthe name of the queue where FileManager will publish their dataretrain_counter : numberFileManager is designed to work together with a Trainer component,therefore, it is possible to define a minimum number of files to triggerthis interaction. If retrain_counter is set to 20, every new 20 labeledinstances are ready for training, this data is turned into physical filesand then forwarded to training process.data_api : dictexpects a Python dictionary with the endpoints to an API implementedto store the data on a database or any other persistence systemAn API, persisting the data on a MongoDB is available on the examples.Methods-------def run(self, loop_interval=0.016)Run the component with the following behaviour:- Get the number of ready for training items through DataApi- In case, the number of items is greater than the retrain_counter, downloadthe data and forward it to the next component which is usually a trainingcomponent- Mark the forwarded items as sent to trainingdef mark_data_as_included_in_the_training_process(self, item_ids)Set the items on the DataApi as "already in training set"def _download_data_and_labels(self)Download the information from DataApi and call a method to be implemented tostore this data in a physical file, i.e., it is expected from userto develop how the file should be, e.g. for images an JPG, for sound MP3,etc.def store_on_filemanager(self, name, x, y)Store an item identified by its name, together with its X data, e.g., forimage the Bas64 encoded imageand its Y, e.g., for image detection, the annotated bounding boxesdef get_dataset_paths(self)Provide the absolute path where the items and their annotations wherestored. It is expected to return an tuple in the format (items, labels)def forward_to_trainer(self, message)Forward the set of ready items to a training component"""InstancerclassInstancer:"""An abstract class used to create instances based on multiple sensors (providers)The instancer is responsible to create data relationship from the differentproviders, based on the necessities of the estimator componentAttributes----------number_of_providers : intthe current number of providers included in the pipelinequeue_server : strthe IPv4 of a RabbitMQ queue serverqueue_input: strthe name of the queue from where collector consumequeue_output : strthe name of the queue where provider will publish their dataMethods-------run(self)Should be implemented from the user, in order to describe the providerbehaviourcreate_custom_instance(self)Should be implemented from the user, in order to describe the providerbehaviourcreate_simple_instance(self)Create an instance directly from the data from the input queuecreate_instance_by_time_window(self, sensor_id, time_window_s)Create an instance based on the list of messages received in a time windowcreate_instance_by_repetition(self, sensor_id, number_of_messages)Create an instance based on a defined number of received messages.e.g.: a model requires 7 images as its input.create_instance_by_syncing_all_providers(self)Synchronize all the providers and create an instance including all of them.forward(self, message)Forward the message to the next component of the pipeline"""MessagerclassMessager:"""An class responsible to define the communication between componentsIt is a wrapper on RabbitMQ library pikaAttributes----------server_url : strthe IPv4 of a RabbitMQ queue serverqueue_name: strthe name of the queue to be interactedMethods-------validate(self, message)Validate the received message, based on the defined schemapublish_to_sibling_queue(self, message, unique_id, validate=True)Send a message to a fragmented queue component, i.e., to a component whichexpectsmultiple queues as its input. Used to publish providers' data in individualqueuespublish(self, message, exchange='', validate=True)Publish the data to the Messager's queueget_message(self)Get a message from the queue, FIFO-based.consume(self, callback)Provide a listener to the queue, so every new message on the queue, thecallback (implemented by the user) is called"""EstimatorclassEstimator:"""An class responsible to define the behavior of an estimator responsible toperform inferences on the pipeline's dataAttributes----------queue_server : strthe IPv4 of a RabbitMQ queue serverinput_queue_instancer : strthe name of the queue to be interactedinput_evaluator_queue : strthe name of the queue used from evaluator to inform estimator aboutnew modelsoutput_queue : strthe name of the queue to be interactedMethods-------get_estimator_model(self)Should be implemented from the user, in order to describe the desiredbehaviourset_estimator_model(self, message)Should be implemented from the user, in order to describe the desiredbehaviourupdate_model(self)Should be implemented from the user, in order to describe the desiredbehaviourestimate(self, message)Should be implemented from the user, in order to describe the desiredbehaviourrun(self)Provide the component's behaviour:- Check for new models available- Get a message from the queue- Make an inference- Forward the resultsforward(self, message)Forward the message to the next component of the pipeline"""EvaluatorclassEvaluator:"""An abstract class responsible to evaluate the quality of trained models.Get a list of models from a trainer component and forwards the best one for anEstimator component, therefore, the inference is always done with the bestpossible results.Constructor----------Attributes----------queue_server : strthe IPv4 of a RabbitMQ queue serverinput_queue : Messagerthe name of the queue to be consumedoutput_queue : Messagerthe name of the queue which Evaluator publish its dataevaluated : list(key,value)a list of already evaluated modelscurrent_best : (key,value)the current best model, e.g. weights, evaluated, i.e., the one recommendedto be used on an Estimator componentMethods-------evaluate_models(self, message)Should be implemented from the user, in order to describe the providerbehaviourinclude_evaluation_to_list(self, key, value)self.evaluated.append((key, value))is_included_in_evaluation_list(self, key):check if a model, identified by its key, was already evaluatedthis can reduce drastically the processing time of the componentget_current_best_evaluated(self):return the actual best model evaluated in the format (key,value)key: model pathvalue: it's evaluation value, e.g. mAPset_current_best_evaluated(self, key, value):update the current best evaluated model that is informed to the interestedcomponents by the queue systemrun(self):implements the behaviour of the componentforward(self, message):publish the evaluated data to next componentthe message follows the format:{"key": "uniqueFilePath","value" "the evaluation value of the model","params": {"modelDefinition": "some extra data about the results"}}"""TrainerDetectionclassTrainerDetection:"""An abstract class responsible to evaluate the quality of trained models.Get a list of models from a trainer component and forwards the best one for anEstimator component, therefore, the inference is always done with the bestpossible results.Constructor----------Attributes----------queue_server : strthe IPv4 of a RabbitMQ queue serverqueue_input : strthe name of the queue to be consumedqueue_output : strthe name of the queue which Evaluator publish its data"""defget_new_training_samples(self):""""""deftrain(self,pretrained):"""Should be implemented from the user, in order to describe the component'sbehaviour"""defcreate_train_and_validation_files(self,message):"""Should be implemented from the user, in order to describe the component'sbehaviour"""defget_latest_pretrained_weights(self):"""Should be implemented from the user, in order to describe the component'sbehaviour"""defsend_weights_to_evaluator(self):"""Should be implemented from the user, in order to describe the component'sbehaviour"""defrun(self,loop_interval=0.016):"""Run the component with the following behaviour:- Check for new samples for training- In case there is new samples, split them on validation and training- Retrain- Forward the trained modelsthe data and forward it to the next component which is usually a trainingcomponent- Mark the forwarded items as sent to training"""defforward(self,message):"""Forward the message to the next component of the pipeline"""