Package: uform
UFormPocket-Sized Multimodal AIFor Content Understanding and GenerationWelcome to UForm, amultimodalAI library that's as versatile as it is efficient.
UFormtiny embedding modelswill help you understand and search visual and textual content across various languages.
UFormsmall generative models, on the other hand, don't only support conversational and chat use-cases, but are also capable of image captioning and Visual Question Answering (VQA).
With compactcustom pre-trained transformer models, this can run anywhere from your server farm down to your smartphone.FeaturesThroughput: Thanks to the small size, the inference speed is2-4x fasterthan competitors.Tiny Embeddings: 256-dimensional vectors are 2-3x quicker tosearchthan from CLIP-like models.Quantization Aware: Downcasted embeddings fromf32toi8without losing much recall.Multilingual: Trained on a balanced dataset, the recall is great across over20 languages.Hardware Friendly: Whether it's Apple's CoreML or ONNX,we've got you covered.ModelsEmbedding ModelsModelParametersLanguagesArchitectureuform-vl-english143M12 text layers, ViT-B/16, 2 multimodal layersuform-vl-multilingual-v2206M218 text layers, ViT-B/16, 4 multimodal layersuform-vl-multilingual206M128 text layers, ViT-B/16, 4 multimodal layersGenerative ModelsModelParametersPurposeArchitectureuform-gen2-qwen-500m1.2BChat, Image Captioning, VQAqwen1.5-0.5B, ViT-H/14uform-gen1.5BImage Captioning, VQAllama-1.3B, ViT-B/16Quick StartOnce youpip install uform, fetching the models is as easy as:importuformmodel=uform.get_model('unum-cloud/uform-vl-english')# Just Englishmodel=uform.get_model('unum-cloud/uform-vl-multilingual-v2')# 21 LanguagesProducing EmbeddingsfromPILimportImageimporttorch.nn.functionalasFtext='a small red panda in a zoo'image=Image.open('red_panda.jpg')image_data=model.preprocess_image(image)text_data=model.preprocess_text(text)image_features,image_embedding=model.encode_image(image_data,return_features=True)text_features,text_embedding=model.encode_text(text_data,return_features=True)similarity=F.cosine_similarity(image_embedding,text_embedding)To search for similar items, the embeddings can be compared using cosine similarity.
The resulting value will fall within the range of-1to1, where1indicates a high likelihood of a match.
Once the list of nearest neighbors (best matches) is obtained, the joint multimodal embeddings, created from both text and image features, can be used to better rerank (reorder) the list.
The model can calculate a "matching score" that falls within the range of[0, 1], where1indicates a high likelihood of a match.joint_embedding=model.encode_multimodal(image_features=image_features,text_features=text_features,attention_mask=text_data['attention_mask'])score=model.get_matching_scores(joint_embedding)Chat, Image Captioning and Question AnsweringThe generative model can be used to caption images, answer questions about them. Also it is suitable for a multimodal chat.fromtransformersimportAutoModel,AutoProcessormodel=AutoModel.from_pretrained("unum-cloud/uform-gen2-qwen-500m",trust_remote_code=True)processor=AutoProcessor.from_pretrained("unum-cloud/uform-gen2-qwen-500m",trust_remote_code=True)prompt="Question or Instruction"image=Image.open("image.jpg")inputs=processor(text=[prompt],images=[image],return_tensors="pt")withtorch.inference_mode():output=model.generate(**inputs,do_sample=False,use_cache=True,max_new_tokens=256,eos_token_id=151645,pad_token_id=processor.tokenizer.pad_token_id)prompt_len=inputs["input_ids"].shape[1]decoded_text=processor.batch_decode(output[:,prompt_len:])[0]You can check examples of different prompts in ourdemo spaceImage Captioning and Question AnsweringIt is the instruction for the first version of UForm-Gen model. We highly recommend you use the new model, instructions for which you can find above.The generative model can be used to caption images, summarize their content, or answer questions about them.
The exact behavior is controlled by prompts.fromuform.gen_modelimportVLMForCausalLM,VLMProcessormodel=VLMForCausalLM.from_pretrained("unum-cloud/uform-gen")processor=VLMProcessor.from_pretrained("unum-cloud/uform-gen")# [cap] Narrate the contents of the image with precision.# [cap] Summarize the visual content of the image.# [vqa] What is the main subject of the image?prompt="[cap] Summarize the visual content of the image."image=Image.open("zebra.jpg")inputs=processor(texts=[prompt],images=[image],return_tensors="pt")withtorch.inference_mode():output=model.generate(**inputs,do_sample=False,use_cache=True,max_new_tokens=128,eos_token_id=32001,pad_token_id=processor.tokenizer.pad_token_id)prompt_len=inputs["input_ids"].shape[1]decoded_text=processor.batch_decode(output[:,prompt_len:])[0]Multimodal ChatThe generative models can be used for chat-like experiences, where the user can provide both text and images as input.
To use that feature, you can start with the following CLI command:uform-chat--modelunum-cloud/uform-gen-chat--image=zebra.jpg
uform-chat--modelunum-cloud/uform-gen-chat\--image="https://bit.ly/3tIVg9M"\--device="cuda:0"\--fp16Multi-GPUTo achieve higher throughput, you can launch UForm on multiple GPUs.
For that pick the encoder of the model you want to run in parallel (text_encoderorimage_encoder), and wrap it innn.DataParallel(ornn.DistributedDataParallel).importuformmodel=uform.get_model('unum-cloud/uform-vl-english')model_image=nn.DataParallel(model.image_encoder)device=torch.device("cuda"iftorch.cuda.is_available()else"cpu")model_image.to(device)_,res=model_image(images,0)EvaluationEmbedding ModelsFew retrieval benchmarks exist for multimodal embeddings.
The most famous ones for English are "MS-COCO" and "Flickr30k".
Evaluatinguform-vl-englishmodel, one can expect the following numbers for search quality.DatasetRecall @ 1Recall @ 5Recall @ 10Flickr0.7270.9150.949MS-COCO¹0.5100.7610.838For multilingual benchmarks, we've created theunum-cloud/coco-smrepository².
Evaluating theunum-cloud/uform-vl-multilingual-v2model, one can expect the following metrics for text-to-image search, compared againstxlm-roberta-base-ViT-B-32OpenCLIPmodel.LanguageOpenCLIP @ 1UForm @ 1OpenCLIP @ 5UForm @ 5OpenCLIP @ 10UForm @ 10SpeakersEnglish 🇺🇸37.837.763.565.073.575.91'452 MChinese 🇨🇳27.332.251.359.062.170.51'118 MHindi 🇮🇳20.731.342.557.953.769.6602 MSpanish 🇪🇸32.635.658.062.868.873.7548 MArabic 🇸🇦22.731.744.957.855.869.2274 MFrench 🇫🇷31.335.456.562.667.473.3274 MAll languages.LanguageOpenCLIP @ 1UForm @ 1OpenCLIP @ 5UForm @ 5OpenCLIP @ 10UForm @ 10SpeakersArabic 🇸🇦22.731.744.957.855.869.2274 MArmenian 🇦🇲5.622.014.344.720.256.04 MChinese 🇨🇳27.332.251.359.062.170.51'118 MEnglish 🇺🇸37.837.763.565.073.575.91'452 MFrench 🇫🇷31.335.456.562.667.473.3274 MGerman 🇩🇪31.735.156.962.267.473.3134 MHebrew 🇮🇱23.726.746.351.857.063.59 MHindi 🇮🇳20.731.342.557.953.769.6602 MIndonesian 🇮🇩26.930.751.457.062.768.6199 MItalian 🇮🇹31.334.956.762.167.173.167 MJapanese 🇯🇵27.432.651.559.262.670.6125 MKorean 🇰🇷24.431.548.157.859.269.281 MPersian 🇮🇷24.028.847.054.657.866.277 MPolish 🇵🇱29.233.653.960.164.771.341 MPortuguese 🇵🇹31.632.757.159.667.971.0257 MRussian 🇷🇺29.933.954.860.965.872.0258 MSpanish 🇪🇸32.635.658.062.868.873.7548 MThai 🇹🇭21.528.743.054.653.766.061 MTurkish 🇹🇷25.533.049.159.660.370.888 MUkranian 🇺🇦26.030.649.956.760.968.141 MVietnamese 🇻🇳25.428.349.253.960.365.585 MMean26.5±6.431.8±3.549.8±9.858.1±4.560.4±10.669.4±4.3-Google Translate27.4±6.331.5±3.551.1±9.557.8±4.461.7±10.369.1±4.3-Microsoft Translator27.2±6.431.4±3.650.8±9.857.7±4.761.4±10.668.9±4.6-Meta NLLB24.9±6.732.4±3.547.5±10.358.9±4.558.2±11.270.2±4.3-Generative ModelsModelLLM SizeSQAMMEMMBenchAverage¹UForm-Gen2-Qwen-500m0.5B45.5880.142.029.31MobileVLM v21.4B52.11302.857.736.81LLaVA-Phi2.7B68.41335.159.842.95For captioning evaluation we measure CLIPScore and RefCLIPScore³.ModelSizeCaption LengthCLIPScoreRefCLIPScorellava-hf/llava-1.5-7b-hf7BLong0.8780.529llava-hf/llava-1.5-7b-hf7BShort0.8860.531Salesforce/instructblip-vicuna-7b7BLong0.9020.534Salesforce/instructblip-vicuna-7b7BShort0.8480.523unum-cloud/uform-gen1.5BLong0.8470.523unum-cloud/uform-gen1.5BShort0.8420.522unum-cloud/uform-gen-chat1.5BLong0.8600.525unum-cloud/uform-gen-chat1.5BShort0.8580.525Results for VQAv2 evaluation.ModelSizeAccuracyllava-hf/llava-1.5-7b-hf7B78.5unum-cloud/uform-gen1.5B66.5¹ Train split was in training data.² Lacking a broad enough evaluation dataset, we translated theCOCO Karpathy test splitwith multiple public and proprietary translation services, averaging the scores across all sets, and breaking them down in the bottom section.³ We usedapple/DFN5B-CLIP-ViT-H-14-378CLIP model.SpeedOn Nvidia RTX 3090, the following performance is expected on text encoding.ModelMultilingualSpeedSpeedupbert-base-uncasedNo1'612 sequences/seconddistilbert-base-uncasedNo3'174 sequences/secondx 1.96sentence-transformers/all-MiniLM-L12-v2Yes3'604 sequences/secondx 2.24unum-cloud/uform-vl-multilingual-v2Yes6'809 sequences/secondx 4.22On Nvidia RTX 3090, the following performance is expected on text token generation usingfloat16, equivalent PyTorch settings, and greedy decoding.ModelSizeSpeedSpeedupllava-hf/llava-1.5-7b-hf7B~ 40 tokens/secondSalesforce/instructblip-vicuna-7b7B~ 40 tokens/secondunum-cloud/uform-gen1.5B~ 140 tokens/secondx 3.5Given the small size of the model it also work well on mobile devices.
On Apple M2 Arm chips the energy efficiency of inference can exceed that of the RTX 3090 GPU and other Ampere-generation cards.DeviceSpeedDevice TDPEfficiencyNvidia RTX 3090~ 140 tokens/second< 350W0.40 tokens/jouleApple M2 Pro unplugged~ 19 tokens/second< 20W0.95 tokens/jouleApple M2 Max unplugged~ 38 tokens/second< 36W1.06 tokens/jouleApple M2 Max plugged~ 56 tokens/second< 89W0.63 tokens/joule[!WARNING]
The above numbers are for reference only and are not guaranteed to be accurate.LicenseAll models come under the same license as the code - Apache 2.0.