Package: image-embeddings
image_embeddingsUsing efficientnet to provide embeddings for retrieval.Why this repo ? Embeddings are a widely used technique that is well known in scientific circles. But it seems to be underused and not very well known for most engineers. I want to show how easy it is to represent things as embeddings, and how many application this unlocks. Checkout thedemofirst!Workflowdownload some picturesrun inference on them to get embeddingssimple knn example, to understand what's the point : click on some pictures and see KNNSimple InstallRunpip install image_embeddingsExample workflowrunimage_embeddings save_examples_to_folder --images_count=1000 --output_folder=tf_flower_images, this will retrieve 1000 image files fromhttps://www.tensorflow.org/datasets/catalog/tf_flowers(but you can also pick any other dataset)produce tf records withimage_embeddings write_tfrecord --image_folder=tf_flower_images --output_folder=tf_flower_tf_records --shards=10run the inference withimage_embeddings run_inference --tfrecords_folder=tf_flower_tf_records --output_folder=tf_flower_embeddingsrun a random knn search on themimage_embeddings random_search --path=tf_flower_embeddingsOptionally if you want to use the embeddings in numpy (in other languages), runimage_embeddings embeddings_to_numpy --input_path=tf_flower_embeddings --output_path=tf_flower_numpy. In particular this can be used in theweb demo$ image_embeddings random_search --path=tf_flower_embeddings
image_roses_261
160.83 image_roses_261
114.36 image_roses_118
102.77 image_roses_537
92.95 image_roses_659
88.49 image_roses_197Explore theSimple notebookfor more details.You can try it locally ortry it in colabTheFrom scratchnotebook provides an explanation on how to build this from scratch.APIimage_embeddings.downloaderDownloader from tensorflow datasets. Any other set of images could be used insteadimage_embeddings.downloader.save_examples_to_folder(output_folder, images_count=1000, dataset="tf_flowers")Savehttps://www.tensorflow.org/datasets/catalog/tf_flowersto folder
Also works with other tf datasetsimage_embeddings.inferenceCreate tf recors from images files, and apply inference with an efficientnet model. Other models could be used.image_embeddings.inference.write_tfrecord(image_folder, output_folder, num_shards=100)Write tf records from an image foldersimage_embeddings.inference.run_inference(tfrecords_folder, output_folder, batch_size=1000)Run inference on provided tf records and save to folder the embeddingsimage_embeddings.knnConvenience methods to read, build indices and apply search on them. These methods are provided as example.
Usefaissdirectly for bigger datasets.image_embeddings.knn.read_embeddings(path)Run embeddings from path and return a tuple withembeddings as a numpy matrixan id to name dictionarya name to id dictionaryimage_embeddings.knn.build_index(emb)Build a simple faiss inner product index using the provided matrix of embeddingsimage_embeddings.knn.search(index, id_to_name, emb, k=5)Search the query embeddings and return an array of (distance, name) imagesimage_embeddings.knn.display_picture(image_path, image_name)Display one picture from the given path and image name in jupyterimage_embeddings.knn.display_results(image_path, results)Display the results from search methodimage_embeddings.knn.random_search(path)Load the embeddings, apply a random search on them and display the resultimage_embeddings.knn.embeddings_to_numpy(input_path, output_folder)Load the embeddings from the input folder as parquet and save them asjson for the id -> name mappingnumpy for the embeddingsParticularly useful to read the embeddings from other languagesAdvanced InstallationPrerequisitesMake sure you usepython>=3.6and an up-to-date version ofpipandsetuptoolspython --version
pip install -U pip setuptoolsIt is recommended to installimage_embeddingsin a new virtual environment. For
examplepython3 -m venv image_embeddings_env
source image_embeddings_env/bin/activate
pip install -U pip setuptools
pip install image_embeddingsUsing Pippip install image_embeddingsFrom SourceFirst, clone theimage_embeddingsrepo on your local machine withgit clone https://github.com/rom1504/image_embeddings.git
cd image_embeddings
make installTo install development tools and test requirements, runmake install-devTestTo run unit tests in your current environment, runmake testTo run lint + unit tests in a fresh virtual environment,
runmake venv-lint-testLintTo runblack --check:make lintTo auto-format the code usingblackmake blackTaskssimple downloader in pythonsimple inference in python usinghttps://github.com/qubvel/efficientnetbuild python basic knn example usinghttps://github.com/facebookresearch/faissbuild basic ui using lit element and some brute force knn to show what it does, put in github pagesuse to illustrate embeddings blogpost