Package: ai-dive
🤿 AI DiveAIDataIngestor,Verifier andEncoder.This library is meant to be practical examples of how to run real world AI Models given data.Installationpip install ai-diveWhy build AI-DiveIn the age of theAI Engineer, it is more likely that you will start by grabbing an off the shelf model as a starting point than training your own from scratch. That is not to say you will never train a model. It is just to say, let's verify state of the art before we go building.A wide range of AI tasks that used to take 5 years and a research team to accomplish in 2013, now just require API docs and a spare afternoon in 2023.🤿 AI-Dive let's you easily dive into the results of a model to decide whether it is worth building upon. It also gives a simple and consistent interface to run in your app or implement new models.ModelTODO: Breakup below into each partModelDatasetDiverSaverDatasetTODODiveTODOSaveTODOAll Together NowTODOModel & DatasetThere are only a two interfaces to implement to get up and running on any model or dataset.Dataset- How to iterate over dataModel- How to predict given each data pointDive & SaveThere are two helper classes to run your model given a datasetDiver- How to run each datapoint from your dataset through the model.Saver- How to save off the results of the run. Running the model and not saving the results can cost time and money.ModelsAI-Dive provides a wrapper around existing models to make them easy to run on your own data. We are not writing models from scratch, we are simply wrapping them with a consistent interface so they can be evaluated in a consistent way.fromai.dive.models.vitimportViTmodel=ViT()data={"full_path":"images/shiba_inu_1.jpg"}output=model.predict(data)print(output)There are a few models implemented already, we are looking to extend this list to new models as the come out, or allow this interface to be implemented in your package to save you time evaluating.HELP US BUILD OUT OUR MODEL LIBRARY OR IMPLEMENT YOUR OWN
TODO: Show how to do eitherVision Transformer (ViT)Llama-2Mistral-7bDalle-3Stable DiffusionMagic AnimateDatasetsModels are worthless without the data to run and evaluate them on. Sure you can poke your model with a stick by running on a single example, but the real insights come from running your model given a dataset.fromai.dive.models.vitimportViTfromai.dive.data.directory_classificationimportDirectoryClassification# Instantiate the model and datasetmodel=ViT()dataset=DirectoryClassification(data_dir="/path/to/images")# Use a Saver to write the results to a csvsaver=Saver("output.csv",output_keys=['filename','class_name','prediction','probability'],save_every=10)# Run the model on the dataset, and save the results as we godiver=Diver(model,dataset,saver=saver)results=diver.run()# The output will be a list of all the predictionsprint(results)TheDiverobject saves you the work of processing each row in the dataframe and theSavertakes care of writing all the results to disk so you can compare them across runs.With plug and play models and datasets, the hope is anyone can evaluate a model against any dataset and share the results quickly and effectively.Model InterfaceTODODataset InterfaceA dataset has to implement two methods__len__and__getitem__so that we can iterate over it. If it implements_build, you can load everything into memory to make the other calls faster.Here is an example dataset that iterates over a directory of images with the folder names as classnames.Example directory structure:images/
    cat/
        1.jpg
        2.jpg
    dog/
        1.jpg
        2.jpg
        3.jpgExample data loader:fromai.dive.data.datasetimportDatasetimportosclassDirImageClassification(Dataset):def__init__(self,data_dir):super().__init__()self.data_dir=data_dir# For iterating over the datasetdef__len__(self):returnlen(self.filepaths)# For iterating over the datasetdef__getitem__(self,idx):return{"filepath":self.filepaths[idx],"class_name":self.labels[idx]}# Override this function to load the dataset into memory for fast accessdef_build(self):# iterate over files in directory, taking the directory name as the labellabels=[]filepaths=[]forroot,dirs,filesinos.walk(self.data_dir):forfileinfiles:iffile.endswith(".jpg")orfile.endswith(".png"):labels.append(os.path.basename(root))filepaths.append(os.path.join(root,file))self.labels=labelsself.filepaths=filepaths