Package: qoa4ml
QoA4ML - Quality of Analytics for MLSource codehttps://github.com/rdsea/QoA4MLMonitoring ClientQoA Client: an object that observes metrics, generates metric reports, and sends them to the Observation service via a list of connectors (e.g., messaging connector: RabbitMQ).The developers only need to init a QoAClient at the beginning and use it to observe/evaluate metrics by self-instrumentation (calling its functions) at the right place in the source code.To initiate a QoA Client, developers can specify a configuration file path or refer to a configuration as a dictionary, or give the registration service (URL) where the client can get its configuration.Theconfigurationcontains the information about the client and its configuration in form of dictionaryExample:clientConf={"client":{"userID":"aaltosea1","instance_name":"ML02","stageID":"ML","method":"REST","application":"test","role":"ml"},"connector":{"amqp_connector":{"class":"amqp","conf":{"end_point":"localhost","exchange_name":"qoa4ml","exchange_type":"topic","out_routing_key":"qoa.report.ml"}}}}qoaClient=QoaClient(config_dict=clientConf)Theconnectoris the dictionary containing multiple connector configuration (amqp, mqtt, kafka)
If 'connector' is not define, developer must give 'registration_url'
The 'registration_url' specify the service where the client register for monitoring service. If it's set, the client register with the service and receive connector configuration.
For example: "http://localhost:5001/registration"Via this client, developers can call different monitoring probes to measure desired metrics and categorize them into data quality, service performance or inference quality.By using our probes (e.g.,observeErronous,observeMissing, andobserveInferenceMetric), the metrics are already categorized in the quality report.For unsupported metrics or user-defined metrics, the developers can report them by usingobserveMetricproviding metric's names and their expected categories. For exampleqoaClient.observeMetric(metric_name="image_width", value=200, category=1).Category: metrics are categorized into following groups:0 - Quality: Performance (metrics for evaluating service performance e.g., response time, throughput)1 - Quality: Data (metrics for evaluating data quality e.g., missing, duplicate, erroneous)2 - Quality: Inference (metrics for evaluating quality of ML inference, measured from inferences e.g., accuracy, confidence)3 - Resource: metrics for evaluating resource utilization e.g. CPU, MemoryTo send the quality report to the observation service, the developers can callreportfrom the QoAClient. For example:qualityReport = qoaClient.report(), the function will additionally return thereportat current stage and save it toqualityReportTo aggregate reports from previous stage (in a pipeline) for building the computation graphs, the client can callimportPReport. For exampleqoaClient.importPReport(previousReport)ProbesQoA4ML Probes: libraries and lightweight modules capturing metrics. They are integrated into suitable ML serving frameworks and ML codeProbe properties:Can be written in different languages (Python, GoLang)Can have different communications to monitoring systems (depending on probes and its ML support)Capture metrics with a clear definition/scopee.g., Response time for an ML stage (training) or a service call (of ML APIs)Thus output of probes must be correlated to objects to be monitored and the tenantSupport high or low-level metrics/attributesdepending on probes implementationCan be instrumented into source code or standloneMetricWe support some metric classes for collecting different types of metric: Counter, Gauge, Summary, HistogramMetric: an original class providing some common functions on an metric object.Attribute:metric_namedescriptionvalueFunction:__init__: let user define the metric name, description and default value.set: set itsvalueto a specific valueget_val: get current valueget_name: return metric nameget_des: return metric description__str__: return information about the metric in form of stringto_dict: return information about the metric in form of dictionaryCounterAttribute: same asMetric& on further developingFunction:inc: increase the value of the metric by the given number/by 1 by default.reset: set the value back to zero.GaugeAttribute: same asMetric& on further developingFunction:inc: increase the value of the metric by a given number/by 1 by default.dec: decrease the value of the metric by a given number/by 1 by default.set: set the value to a given number.SummaryAttribute: same asMetric& on further developingFunction:inc: increase the value of the metric by a given number/by 1 by default.dec: decrease the value of the metric by a given number/by 1 by default.set: set the value to a given number.HistogramAttribute: same asMetric& on further developingFunction:inc: increase the value of the metric by a given number/by 1 by default.dec: decrease the value of the metric by a given number/by 1 by default.set: set the value to a given number.QoA4ML ReportsThis module definesQoA_Report, an object that provide functions to export monitored metric to the following schema:{"computationGraph":{"instances":{"@instance_id":{"instance_name":"@name_of_instance","method":"@method/task/function","previous_instance":["@list_of_previous_instance"]},...},"last_instance":"@name_of_last_instance_in_the_graph"},"quality":{"data":{"@stageID":{"@metric_name":{"@instance_id":"@value"}}},"performance":{"@stageID":{"@metric_name":{"@instance_id":"@value"}}},"inference":{"@inference_id":{"value":"@value","confident":"@confidence","accuracy":"@accuracy","instance_id":"@instance_id","source":["@list_of_inferences_to_infer_this_inference"]}}}}The example is shown inexample/reports/qoa_report/example.txtAttribute:previous_report_instance= list previous servicesreport_list: list of reports from previous servicesprevious_inference: list previous inferencesquality_report: report all quality (data, service, inference qualtiy) of the serviceexecution_graph: report the execution graphreport: the final report to be sentFunction:__init__: init as empty report.import_report_from_file: init QoA Report fromjsonfile.importPReport: import reports from previous service to build the execution and inference graphbuild_execution_graph: build execution graph from list of previous reportsbuild_quality_report: build the quality report from metrics collected in runtimegenerateReport: return the final report.observeMetric: observe metrics in runtime with 3 categories: service quality, data quality, inference qualtiy. This can be extended to observe resource metrics.Exampleshttps://github.com/rdsea/QoA4ML/tree/main/exampleOverviewProbes will be integrated to client program or system service to collect metrics at the edge
Probes will generate reports and sent to message broker using different connector. Coresponding collector should be used to acquire the metrics.CollectorThe manager/orchestrator have to integrate collector to collect metric using different protocols for further analysis.Attribute:Function:__init__: take a configuration as adictcontaining information about the data source, eg. broker, channel, queue, etc. It can take anobjectas an attributehostto return the message for further processing.If the collector is initiated by an object inherited class, this class must implementmessage_processingfunction to process the message returned by the collector. Otherwise, the collector will print the message to the console.on_request: handle message from data source (message broker,...)start&stop: start and stop consuming messageget_queue: return the queue name.ConnectorConnectors are implement with different protocols for sending report. Example: sending report to message broker - AMQP/MQTTAttribute:Function:__init__: take a configuration as adictcontaining information about the data sink, eg. broker, channel, queue, etc. It can take aboolparameterlogfor logging messages for further processing.send_data: a function to send data to specifiedrouting_key/queuewith a corresponding keycorr_idto trace back message.UtilitiesA module provide some frequently used functions and some function to directly collect system metrics.Noteeva_duplicate,eva_erronous,eva_missing, anddetect_outlierprobes are using ydata-quality library, which is only available for Python 3.8For using ML quality probes, you may need to install a few more dependencies, e.g., tensorflow and Pillow.QoaClient uses AMQP protocol by default. To use MQTT, you may need to installpaho-mqtt.To monitor Docker stats, you need to installdockerpython client.To connect with Prometheus, you need to installprometheus-clientChange Log0.0.13 (18/04/2022)First Release0.0.18 (10/05/2022)Update system metric0.0.19 (31/05/2022)Update process metric0.0.54 (20/09/2022)Update monitoring system/process/docker0.0.62 (20/09/2022)Add metric and modify format of system/process reports0.0.64 (15/03/2022)Add metric and modify format of system/process reports0.0.72 (22/05/2023)Refactor source code