Package: thetis
Even if your AI is as strong as Achilles, Thetis will certainly know about its weaknesses.Thetis is a service to examine safety-relevant aspects of AI algorithms (e.g., machine learning or deep learning models)
for uncertainty consistency (calibration), fairness, data quality, and robustness.Please visit theAPI Documentation Pagefor the
complete documentation and detailed backgrounds to the several safety aspects.InstallationThetis is installed via the Python Package Index (PyPI) using Python's PIP-Tool. Simply type$pipinstallthetisto install the latest version of Thetis for your Python environment.Obtain LicenseRequest yourfree evaluation licenseatefs-techhub.com.
Alternatively, use the included demo-licenses if you only wish to follow along with theexamples.Place the license file either in the working directory of your application or at:Windows:<User>\\AppData\\Local\\Thetis\\license.datUnix:~/.local/thetis/license.datFor further information about subscription models and license settings, seeSubscription.Your feedback is highly appreciated!QuickstartClassification ExampleThetis can evaluate the AI safety of classifier models.
In a first step, we demonstrate how to evaluate and rate your AI model using a basic classification
example fromscikit-learn.
The instructions below should be easy do adapt to your own use-case.Example Data and Model PreparationClick to show/hide example code for data and model preparation.To start with our basic classification example, we need to load some data. In this tutorial, we use theAdult data set.
This data set demonstrates a prediction task that determines whether a person earns over 50K a year.
Let us now load the data set using tools from the scikit-learn library.Note:If your machine is behind a proxy server, downloading the example data as shown here may not work.
If that is the case for you, check out our detection example below.importpandasaspdfromsklearn.datasetsimportfetch_openmltestset_size=10000# use "fetch_openml" by scikit-learn to load "Adult" dataset from OpenMLdataset,target=fetch_openml(data_id=1590,return_X_y=True,parser="auto")df_train,df_test=dataset.iloc[:-testset_size],dataset.iloc[-testset_size:]target_train,target_test=target.iloc[:-testset_size],target.iloc[-testset_size:]# drop columns with sensitive attributes from classifier input and convert categorical attributes to one-hotdf_train_cleared=df_train.drop(columns=["education","race","sex","native-country","relationship","marital-status"])df_test_cleared=df_test.drop(columns=["education","race","sex","native-country","relationship","marital-status"])# convert categorical columns to class codes with integer representationcategorical_columns=["workclass","occupation"]df_train_cleared[categorical_columns]=df_train_cleared[categorical_columns].apply(lambdacol:pd.Categorical(col).codes)df_test_cleared[categorical_columns]=df_test_cleared[categorical_columns].apply(lambdacol:pd.Categorical(col).codes)This yields twoPandasdata frames with a reduced set of information.In the next step, we train a simple Random Forest classifier on the training data using scikit-learn.
Furthermore, we make predictions on the test data using the trained model:fromsklearn.ensembleimportRandomForestClassifier# initialize a Random Forest classifier and fit to training dataclassifier=RandomForestClassifier(verbose=True)classifier.fit(pd.get_dummies(df_train_cleared),target_train)# finally, make predictions on the validation data setconfidence=classifier.predict_proba(pd.get_dummies(df_test_cleared))labels=classifier.predict(pd.get_dummies(df_test_cleared))Running AI Safety Evaluation with ThetisFor details of Thetis configuration, see sectionConfiguration.
For the current example, you can download thedemo configuration filefrom this repository orclick here.fromthetisimportthetis# use sensitive attributes during safety evaluationannotations=pd.DataFrame({"target":target_test,"race":df_test["race"],"sex":df_test["sex"]})predictions=pd.DataFrame({"labels":labels,"confidence":confidence[:,1]},index=annotations.index)result=thetis(config="examples/demo_config_classification.yaml",annotations=annotations,predictions=predictions,output_dir="./output",license_file_path="examples/demo_license_classification.dat",)Thetis expects two Pandas data frames to run an evaluation:Annotations:pd.DataFramewith ground-truth information about the data set. The columntargetis required, holding
the ground-truth target information. Furthermore, columns for sensitive attributes are expected that have been
configured for the AI Fairness evaluation.Predictions:pd.DataFramewith the AI predictions for each sample in the data set. The columnslabelsandconfidenceare required, holding information about the predicted label and the respective prediction
probability (model uncertainty or confidence).Note that the indices of the data frames for annotations and predictions must match.Thetis returns its findings, the final rating, and recommendations for mitigation strategies as a JSON-like dictionary.
We capture the dictionary asresultand can access the different evaluation aspects:result[<task>]['rating_score']for the rating score of the selected task (e.g., 'fairness' or 'uncertainty').result[<task>]['recommendations']for the recommendations to mitigate possible issues of the selected task.result[<task>]['rating_enum']for a categorization of the actual aspect into'GOOD','MEDIUM',
or'BAD'depending on the rating score.Object Detection Example (Image-based)Thetis is also capable of evaluating the AI safety of modern, image-based object detector models.
We utilize aFaster R-CNN model by Torchvisionin conjunction
with a demo data set (Download here) to demonstrate the evaluation process for
object detectors. The instructions below should be easy do adapt to your own use-case.Running Inference with PyTorch Object DetectorClick to show/hide example code for data and model preparation.First, we need to load and initialize theFaster R-CNN by Torchvision.Note:If your machine is behind a proxy, you likely need to configure your environment so that the following code
can download the pre-trained model and data set. Set theHTTP_PROXYandHTTPS_PROXYenvironment variables to point
to your proxy server and port before launching python or jupyter.importnumpyasnpfromtorchvision.ioimportread_image,ImageReadModefromtorchvision.models.detectionimportfasterrcnn_resnet50_fpn_v2,FasterRCNN_ResNet50_FPN_V2_Weights# initialize object detection model from torchvision model zooweights=FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULTmodel=fasterrcnn_resnet50_fpn_v2(weights=weights)model.eval()# retrieve necessary image transformations (e.g., normalization, etc.) and available categoriespreprocess=weights.transforms()categories=np.array(weights.meta["categories"])Note that the model is pre-trained on the MS COCO data set with several categories. In our example, we only
work with the categories "person", "bicycle", and "car". In the next step, download and extract
theDemo Detection Data Setwhich is artificially generated using
theCarla simulation engine. After download and extraction, we can load the JSON annotation
files and run inference with the Torchvision model:importosfromglobimportglobfromtqdmimporttqdmimportjsonimporttorch# set the inference device - if you have a CUDA device, you can set it to "cuda:<idx>"device="cpu"# device = "cuda:0"model.to(device)# get a list of all JSON filesannotation_files=glob(os.path.join("demo_detection","annotations","*.json"))data=[]# iterate over all JSON files and retrieve annotationsforfilenameintqdm(annotation_files,desc="Running inference on images ..."):withopen(filename,"r")asopen_file:anns=json.load(open_file)# load respective image, run preprocessing (transformation) and finally run inferenceimg=read_image(os.path.join("demo_detection","img",anns["image_file"]),ImageReadMode.RGB)img=[preprocess(img).to(device=device)]# make inference and copy back to CPU (if CUDA device has been used for inference)withtorch.no_grad():pred=model(img)[0]pred={k:v.cpu()fork,vinpred.items()}# store predicted and target data for current framedata.append((pred,anns))Expected Data Format for Object DetectionAfter loading the ground-truth information and running inference using an AI model (see example above),
we must format our predictions and annotations in a way that can be ingested by Thetis. In object detection evaluation mode,
Thetis expects a Python dictionary for the predictions and annotations, where the keys represent the image identifiers
(e.g., image name) and the values represent the individual (predicted or ground-truth) objects within a single frame.importpandasaspd# Thetis expects a dictionary with image name as key and a pd.DataFrame with predicted information as value.# A similar format is also expected for the ground-truth annotations with extra sensitive attributes# used for fairness evaluation. The field "__meta__" is always required with meta information for each frame.annotations={"__meta__":pd.DataFrame(columns=["width","height"])}predictions={}# iterate over all frames with predicted and target informationforpred,annsindata:# retrieve predicted labels, bounding boxes, and filter predictions by labelpredicted_labels=categories[pred["labels"].numpy()]predicted_boxes=pred["boxes"].numpy().reshape((-1,4))target_boxes=np.array(anns["boxes"]).reshape((-1,4))filter=np.isin(predicted_labels,["person","bicycle","car"])filename=anns["image_file"]# add predicted information as pd.DataFramepredictions[filename]=pd.DataFrame.from_dict({"labels":predicted_labels[filter],"confidence":pred["scores"].numpy()[filter],"xmin":predicted_boxes[:,0][filter],"ymin":predicted_boxes[:,1][filter],"xmax":predicted_boxes[:,2][filter],"ymax":predicted_boxes[:,3][filter],})# add ground-truth information also as pd.DataFrame with additional sensitive attributesannotations[filename]=pd.DataFrame.from_dict({"target":anns["classes"],"gender":anns["gender"],"age":anns["age"],"xmin":target_boxes[:,0],"ymin":target_boxes[:,1],"xmax":target_boxes[:,2],"ymax":target_boxes[:,3],})# some additional meta information such as image width and height are also requiredannotations["__meta__"].loc[filename]=[anns["image_width"],anns["image_height"]]Important: The dictionary for the ground-truth annotations requires a key__meta__which holds width and height
information for each image within the data set, provided as Pandas DataFrame. Note that the index of the entries within
this DataFrame must match with the keys (i.e. image identifiers) of the Python dictionaries.Running AI Safety Evaluation with ThetisGiven your data is in the right format, simply call Thetis with the predictions, the ground-truth information and the
prepared configuration file:fromthetisimportthetis# finally, we can call the Thetis evaluation service similarly to the classification caseresult=thetis(config="examples/demo_config_detection.yaml",annotations=annotations,predictions=predictions,output_dir="./output",license_file_path="examples/demo_license_detection.dat",)For details of Thetis configuration, see sectionConfiguration.
For the current example, you can download thedemo configuration filefrom this repository orclick here.Thetis returns its findings, the final rating and recommendations for mitigation strategies as a JSON-like dictionary.
We capture the dictionary asresultand can access the different evaluation aspects:result[<task>]['rating_score']for the rating score of the selected task (e.g., 'fairness' or 'uncertainty').result[<task>]['recommendations']for the recommendations to mitigate possible issues of the selected task.result[<task>]['rating_enum']for a categorization of the actual aspect into'GOOD','MEDIUM',
or'BAD'depending on the rating score.Note that the remaining evaluation metrics are grouped by the specified IoU scores which are used for the matching
of predicted objects with ground-truth ones (e.g., an IoU score of 0.5 might be used to decide if a prediction
has matched an existing ground-truth object or not). In the configuration file, you can specify multiple IoU scores
that are taken into account for the final evaluation process.