Package: imgbeddings
imgbeddingsA Python package to generate embedding vectors from images, usingOpenAI's robustCLIP modelviaHugging Facetransformers. These image embeddings, derived from an image model that has seen the entire internet up to mid-2020, can be used for many things: unsupervised clustering (e.g. viaumap), embeddings search (e.g. viafaiss), and using downstream for other framework-agnostic ML/AI tasks such as building a classifier or calculating image similarity.Theembeddings generation modelsare ONNX INT8-quantized, meaning they're 20-30% faster on the CPU, much smaller on disk, and doesn't require PyTorch or TensorFlow as a dependency!Works for many different image domains thanks to CLIP's zero-shot performance.Includes utilities for usingprincipal component analysis (PCA)to reduces the dimensionality of generated embeddings without losing much info.Real-World Demo NotebooksYou can read how to use imgbeddings for real-world use cases in these Jupyter Notebooks:Cats vs. Dogs: image clustering and building a cat/dog classifierPokémon: most-similar image searchImage Augmentation: generated embedding resilience to altered inputsInstallationaitextgen can be installed from PyPI:pip3installimgbeddingsQuick ExampleLet's say you want to generate an image embedding for acute cat photo. First you can download the photo:importrequestsfromPILimportImageurl="http://images.cocodataset.org/val2017/000000039769.jpg"image=Image.open(requests.get(url,stream=True).raw)Then, you can load imgbeddings. By default, imgbeddings will load a 88MB model based on the patch32 variant of CLIP, which separates each image into 49 32x32 patches.fromimgbeddingsimportimgbeddingsibed=imgbeddings()You can also load the patch16 model by passingpatch_size = 16toimgbeddings()(more granular embeddings but takes about 3x longer to run), or the "large" patch14 model withpatch_size = 14(3.5x model size, 3x longer than patch16).Then to generate embeddings, all you have to is pass the image toto_embeddings()!embedding=ibed.to_embeddings(image)embedding[0][0:5]# array([ 0.914541, 0.45988417, 0.0350069 , -0.9054574 , 0.08941309], dtype=float32)This returns a 768Dnumpyvector for each input, which can be used for pretty much anything in the ML/AI world. You can also pass a list of filename and/orPILImages for batch embeddings generation.See the Demo Notebooks above for more advanced parameters and real-world use cases. More formal documentation will be added soon.EthicsTheofficial paper for CLIPexplicitly notes that there are inherent biases in the finished model, and that CLIP shouldn't be used in production applications as a result. My perspective is that having better tools free-and-open-source todetectsuch issues and make it more transparent is an overall good for the future of AI, especially since there are less-public ways to create image embeddings that aren't as accessible. At the least, this package doesn't do anything that wasn't already available when CLIP was open-sourced in January 2021.If you do use imgbeddings for your own project, I recommend doing a strong QA pass along a diverse set of inputs for your application, which is something you should always be doing whenever you work with machine learning, biased models or not.imgbeddings is not responsible for malicious misuse of image embeddings.Design NotesNote that CLIP was trained on square images only, and imgbeddings will pad and resize rectangular images into a square (imgbeddings deliberately does not center crop). As a result, images too wide/tall (e.g. more than a 3:1 ratio of largest dimension to smallest) will not generate robust embeddings.This package only works with image data intentionally as opposed to leveraging CLIP's ability to link image and text. For downstream tasks, using your own text in conjunction with an image will likely give better results. (e.g. if training a model on an image embeddings + text embeddings, feed both and let the model determine the relative importance of each for your use case)For more miscellaneous design notes, seeDESIGN.md.Maintainer/CreatorMax Woolf (@minimaxir)Max's open-source projects are supported by hisPatreonandGitHub Sponsors. If you found this project helpful, any monetary contributions to the Patreon are appreciated and will be put to good creative use.See AlsoSentence Transformers, which has awrapper around CLIPthat supports Image-to-Image search.LicenseMIT