Package: efemarai
Efemarai websiteEfemarai platformUse now!An SDK for interacting with the Efemarai MLtesting platform. Make your R&D model production ready.📘Documentation|🛠️Installation|👀Break YOLO|🚀Join Community|😞Reporting IssuesIntroductionEfemarai is the easiest to integrate open source platform for testing and validating Computer Vision ML models. It works with any framework and model (PyTorch, TensorFlow, Keras, sklearn, Detectron2, OpenMMLab, YOLO, etc) and in 5 function calls finds examples that break your model.Major features🔮 Operational DomainFinetune how the images should be transformed, such that they cover the variablity the model is exepected to see in the real world.👨‍💻 Support any Input and Output typesNot only do we support tasks such asclassification,object detection,instance segmentation,keypoints detection,regression, but also any combination thereoff, with any type of input - single image, multi-image, video, text, or anything that combines those.📈 High efficiencyDon't waste time on randomly augmenting data, with Efemarai you are narrowing down failure modes in your model that are informative and you can fix.Example WorksFind issues with a COCO detectorApply advanced transformations, copy, edit, delete any part of the image, reimagine how things should vary in the real world.Break a face detectorWith theFaceWorksmodule, you can perform relighting, face rotation, skin re-toning, etc to find conditions that break your model.OtherIf you work in the medical, aerospace and defence, security or ag domain, we provide custom capabilities that are domain specific. For these or other enquiries, drop usa message.SetupInstall withpipinstall-Uefemaraithen runefemaraiinitand follow the instructions to connect your account tohttps://ci.efemarai.com.Example UsageCreate a Bounding Box ProjectWhen your project depends on bounding boxes, the uploaded dataset needs to contain the required bounding box information alongside each image as part of a single datapoint.First we will create a dataset, and later on, a dummy model that returns bounding box information.Create datasetA convenient approach for creating a bounding box dataset is by ensuring that the local format is in COCO-compatible format.importefemaraiasef# Create a projectproject=ef.Session().create_project(name="Example Bounding Box Project (COCO)",description="Example project using the COCO dataset format.",exists_ok=True,)dataset=project.create_dataset(name="Bounding Box dataset",data_url="./data/coco/test",annotations_url="./data/coco/annotations/test_instances.json",stage=ef.DatasetStage.Test,format=ef.DatasetFormat.COCO,)If your dataset is remote or part of an existing database with custom formats, you can easily upload it to the system by (1) iterating over the dataset and (2) creating datapoints containing the images and required targets. You can find a code examplehere.After wrapping up any processing, you can confirm the status in the UI and explore the inputs and annotations.Create a modelA model that works with bounding boxes dataset will need to return a list ofef.BoundingBoxobjects that will be matched to the ones stored in the dataset. In a filedummy_model.pysave the following code:importefemaraiasefimportnumpyasnpclassDummyModel:"""A DummyModel returning a random bbox"""def__init__(self,device):self.device=device# Move model to devicedef__call__(self,image):return{"class_id":np.random.randint(0,3),"bbox":[100,150,250,350],"score":np.random.random(),}defpredict_images(datapoints,model,device):outputs=[]fordatapointindatapoints:image=datapoint.get_input("image")# This corresponds to the key from the datapoint input creation dictimage_post_process=image.data/255-0.5# perform any pre-processingoutput=model(image_post_process)# Here again the label can be referenced by name or class# label = ef.AnnotationClass(name=output["class_name"])label=ef.AnnotationClass(id=output["class_id"])outputs.append([ef.BoundingBox(xyxy=output["bbox"],confidence=output["score"],# Confidence of detectionref_field=image,# Say which image this output refers tolabel=label,# And what label it has),])returnoutputsdefload_model(device):model=DummyModel(device)returnmodelThat's the two things you need to define - how to load the model and how to perform an inference on a batch!efemarai.yaml fileTo run the model, you need to have defined the loading and inference capabilities in the efemarai.yaml file. This way we can work with any model from any platfrom or framework.Here’s the one corresponding to the dummy model.project:name:"ExampleBoundingBoxProject"models:-name:Dummy Modeldescription:This is a dummy model to show consuming inputs and outputsruntime:image:python:3.10-slim-busterdevice:"gpu"batch:max_size:10load:entrypoint:dummy_model:load_modelinputs:-name:devicevalue:${model.runtime.device}output:name:modelpredict:entrypoint:dummy_model:predict_imagesinputs:-name:datapointsvalue:${datapoints}-name:modelvalue:${model.runtime.load.output.model}-name:devicevalue:${model.runtime.device}output:name:predictionskeys:-bboxRegister the modelTo register the model, use the CLI to upload it by going into the root of the file directory, next to theefemarai.yaml.efmodelcreate.Now you should be able to see the model uploaded and active with this project.Create a domainLet's use the UI to quickly create a domain (Example) that you expect your model to operate in.You can find more information in thedocs.Working with stress testsNow that you have defined your model, domain and dataset, you are ready to stress test your model and validate how well it works!# Create a new stress testtest=project.create_stress_test(name="Test via SDK",model=project.model("Dummy Model"),domain=project.domain("Example"),dataset="Example Bounding Box Project (COCO)",# Just a name also works)# Load an existing stress testtest=project.stress_test("Test via SDK")# Download dataset with discovered vulnerabilitiesdataset_filepath=test.vulnerabilities_dataset()# Check test run stateprint(f"Running:{test.running}Failed:{test.failed}Finished:{test.finished}")Models, domains and datasets can be easily created programatically, but
they require quite a few configuration paramaters to be provided. That's
why the most convenient way to create a project with multiple models, domains
and datasets is to put everything into a config file.