Package: vector-forge
🐍 Vector Forge is a Python package designed for easy transformation of various data types into feature vectors.💡Core ideas🌄 For image embeddings, Vector Forge uses pre-trained networks, which means the models have already learned features
from
a large set of images calledImageNet. When we use these models in Vector Forge, we skip
the part that
identifies objects, and instead, we use
the part that understands the image features. This way, we get a bunch of numbers (a vector) representing the image,
which can be used
in many different tasks like finding similar images, clustering, classification and many more.📄 Text embeddings are a way to convert words or sentences into numbers, making it possible for computers to understand
and
process them. In Vector Forge, theCLIP ViT-B/32model is
utilized to generate these embeddings. When you provide any text, be it
a single word or a sentence, to CLIP, it transforms this text into a fixed-size vector. Each vector has a consistent
length, no matter how long or short the original text is. This consistency in size is valuable, especially when
comparing different pieces of text or measuring how similar a piece of text is to an image.🧩 FeaturesImage to Vector conversion: Easily convert individual images into feature vectors by specifying your desired model
to extract meaningful representations.Batch processing: Provide a folder path to process multiple images in bulk. Select your preferred model and let
Vector Forge swiftly handle all the images in the specified directory.Text to Vector transformation: Effortlessly convert textual data into vectors. Choose your model, and Vector Forge
will transform your text input into a high-dimensional vector representation.Support for multiple models: Vector Forge supports various models for vectorization, including CLIP ViT-B/32, CLIP
ViT-L/14, Xception,
VGG16 and VGG19, to provide flexibility in handling different data types.⚙️ RequirementsPython >= 3.10📦 Supported modelsModel NameImplementationParameter ValueSupports ImageSupports TextEmbedding SizeCLIP ViT-B/32PyTorchModels.CLIP_B_P32✅✅(512,)CLIP ViT-L/14PyTorchModels.CLIP_L_P14✅✅(768,)CLIP ViT-B/32 OpenVino™OpenVino™Models.CLIP_B_P32_OV✅✅(512,)CLIP ViT-L/14 OpenVino™OpenVino™Models.CLIP_L_P14_OV✅✅(768,)VGG16KerasModels.VGG16✅❌(512,)VGG19KerasModels.VGG19✅❌(512,)XceptionKerasModels.Xception✅❌(2048,)🎛️ UsageYou can work on many pictures at once or just one piece of text with simple commands, making it a breeze to get your
data ready for further use or analysis🔧 Installationpipinstallvector_forge🔌 Create a vectorizerImport the necessary classes or functionsfromvector_forgeimportVectorizerDefault vectorizerBy default, the vectorizer isCLIP ViT-B/32, as it works for text
and images.vectorizer=Vectorizer()Text to VectorExample how to convert a text prompt to a vector.text_embedding=vectorizer.text_to_vector("Nice text!")Image to VectorExample how to convert to convert image from path to vector.image_embedding=vectorizer.image_to_vector("/path/to/image.jpg")Change the vectorizer to use a different modelExample how to change the vectorizer model, in this example
toXception.Keep in mind, that not all models work for for text prompts. If you want to compare image and texts, I recommend
usingCLIP ViT-B/32.fromvector_forgeimportVectorizer,Modelsvectorizer=Vectorizer(model=Models.Xception)Return typesIn Vector Forge, you have the flexibility to choose the format in which the vectors are returned. This is controlled by
thereturn_typeparameter available in theimage_to_vectorandtext_to_vectormethods of the Vectorizer
class. Here are
the available return types along with examples:a)return_type="numpy"This is the default return type. Vectors are returned asNumPyarrays.image_embedding=vectorizer.image_to_vector("/path/to/image.jpg",return_type="numpy")# Output: array([0.0234, 0.0345, ..., 0.0456])# Shape: (2048,) for Xception, (512,) for VGG16, VGG19 and CLIP ViT-B/32, (768, ) for CLIP ViT-L/14b)return_type="str"Vectors are returned as a string representation of the NumPy array.image_embedding=vectorizer.image_to_vector("/path/to/image.jpg",return_type="str")# Output: "[0.0234, 0.0345, ..., 0.0456]"c)return_type="list"Vectors are returned as a list of values.image_embedding=vectorizer.image_to_vector("/path/to/image.jpg",return_type="list")# Output: [0.0234, 0.0345, ..., 0.0456]d)return_type="2darray"Vectors are returned as a 2-dimensional NumPy array, where each vector is a row in the array. This format is especially
useful when you want to compute similarities or perform other vectorized operations.image_embedding=vectorizer.image_to_vector("/path/to/image.jpg",return_type="2darray")# Output: array([[0.0234, 0.0345, ..., 0.0456]])# Shape: (1, 2048)  # for Xception, (1, 512) for VGG16, VGG19 and CLIP ViT-B/32, (1, 768) for CLIP ViT-L/14Batch Processing for imagesVector Forge can process multiple images in a folder in one go. Just provide the folder path, and theload_from_foldermethod will handle the rest.# Convert all valid images in a folder to vectorsforvectorinvectorizer.load_from_folder("/path/to/folder"):print(vector.shape)You can specify thereturn_type,save_to_index, andfile_info_extractorparameters to control the output format,
to save the file paths of processed images to an index file, and to execute a custom function on each file for
additional information extraction, respectively.# Example with return_type and save_to_indexforvectorinvectorizer.load_from_folder("/path/to/folder",return_type="2darray",save_to_index="paths.txt"):print(vector.shape)fromvector_forge.info_extractorsimportget_file_info# Example with additional information to each fileforvector,dimensioninvectorizer.load_from_folder("/path/to/folder",file_info_extractor=get_file_info):print(vector.shape)Image preprocessingVector Forge provides a collection of image preprocessing functions to help prepare images for vectorization. These
functions can be found in theimage_preprocessors.
You can also specify your own custom image preprocessing function.fromvector_forge.image_preprocessorsimportresize_image# Create a Vectorizer instance with the resize_image function as the image preprocessorresize_fn=lambdaimg:resize_image(img,width=600)vectorizer=Vectorizer(image_preprocessor=resize_fn)vector=vectorizer.image_to_vector(input_image='/path/to/image.jpg')fromvector_forge.image_preprocessorsimportconvert_to_grayscale# Create a Vectorizer instance with the convert_to_grayscale function as the image preprocessorvectorizer=Vectorizer(image_preprocessor=convert_to_grayscale)vector=vectorizer.image_to_vector(input_image='/path/to/image.jpg')🧪 A complete examplefromvector_forgeimportVectorizerfromvector_forge.image_preprocessorsimportresize_imagefromsklearn.metrics.pairwiseimportcosine_similaritydefcompute_similarity(vectorizer,text,image_path):# Convert text and image to vectors with return type '2darray'text_embedding=vectorizer.text_to_vector(text,return_type="2darray")image_embedding=vectorizer.image_to_vector(image_path,return_type="2darray")# Compute cosine similaritysimilarity=cosine_similarity(text_embedding,image_embedding)[0][0]returnsimilarity# Create a vectorizer with the default CLIP ViT-B/32 model and a custom image preprocessorresize_fn=lambdaimg:resize_image(img,width=600)vectorizer=Vectorizer(image_preprocessor=resize_fn)# Define text and image pathstext="A couple of birds"image_path_1="vector_forge/test_data/birds.jpg"# adapt paths accordinglyimage_path_2="vector_forge/test_data/sample.jpg"# adapt paths accordingly# Compute and print similarity scoressimilarity_1=compute_similarity(vectorizer,text,image_path_1)similarity_2=compute_similarity(vectorizer,text,image_path_2)print(f"Similarity between text and first image:{similarity_1}")print(f"Similarity between text and second image:{similarity_2}")Complete example how to usefile_info_extractor, which can extract some valuable information from files.fromvector_forgeimportVectorizer,Modelsfromvector_forge.info_extractorsimportget_colors# Create a Vectorizer instancevectorizer=Vectorizer(model=Models.Xception)# Define the path to your folder containing imagesfolder_path='/path/to/images'# Process all images in the specified folderforvector,colorsinvectorizer.load_from_folder(folder_path,file_info_extractor=get_colors):# Print the vector shape and image dimensionsprint(f'Vector shape:{vector.shape}')print(f'Image colors:{colors}')⚠️ DisclaimerVector Forge is provided as-is, without warranty of any kind. Users should employ the library at their own risk. It's
important to test and validate the library's results in your specific context to ensure it meets your needs. Performance
and accuracy can vary based on data and use cases. We encourage all users to thoroughly verify the library's outputs and
consider them as one of many tools in their toolkit.🔮 Future featuresMake inference APIs which hold the models in memoryImagesAdd support for VGG19Add possibility for index creation when usingload_from_folderAdd support forlarger CLIP modelOptimize CLIP generation
withOpenVino IRBatch support forload_from_folderoperationsAdd support for custom type of Keras modelsTextsAdd support for GloVeAdd text preprocessors