Question: I am trying to use CLIP to calculate the similarities between strings. (I know that CLIP is usually used with text and images but it should work with only strings as well.) I provide a list of simple text prompts and calculate the similarity between their embeddings. The similarities are off but I can't figure what I'm doing wrong. import torch import clip from torch.nn import CosineSimilarity cos = CosineSimilarity(dim=1, eps=1e-6) def gen_features(model, text): tokens = clip.tokenize([text]).to(device) text_features = model.encode_text(tokens) return text_features def dist(v1, v2): #return torch.dist(normalize(v1), normalize(v2)) # euclidean distance #return cos(normalize(v1), normalize(v2)).item() # cosine similarity similarity = (normalize(v1) @ normalize(v2).T) return similarity.item() device = "cuda" if torch.cuda.is_available() else "cpu" model_name = "ViT-B/32" model, _ = clip.load(model_name, device=device) sentences = ["A cat", "A dog", "A labrador", "A poodle", "A wolf", "A lion", "A house"] with torch.no_grad(): embeddings = [(sentence, gen_features(model, sentence)) for sentence in sentences] for label1, embedding1 in embeddings: for label2, embedding2 in embeddings: print(f"{label1} -> {label2}: {dist(embedding1, embedding2)}") Output A cat -> A cat: 0.9999998211860657 A cat -> A dog: 0.9361147880554199 A cat -> A labrador: 0.8170720934867859 A cat -> A poodle: 0.8438302278518677 A cat -> A wolf: 0.9086413979530334 A cat -> A lion: 0.8914517164230347 A cat -> A house: 0.8724125027656555 A dog -> A cat: 0.9361147880554199 A dog -> A dog: 1.0000004768371582 A dog -> A labrador: 0.8481228351593018 A dog -> A poodle: 0.9010260105133057 A dog -> A wolf: 0.9260395169258118 A dog -> A lion: 0.886112630367279 A dog -> A house: 0.8852840662002563 A labrador -> A cat: 0.8170720934867859 A labrador -> A dog: 0.8481228351593018 A labrador -> A labrador: 1.000000238418579 A labrador -> A poodle: 0.7722526788711548 A labrador -> A wolf: 0.8111101984977722 A labrador -> A lion: 0.783727765083313 A labrador -> A house: 0.7569846510887146 A poodle -> A cat: 0.8438302278518677 A poodle -> A dog: 0.9010260105133057 A poodle -> A labrador: 0.7722526788711548 A poodle -> A poodle: 0.999999463558197 A poodle -> A wolf: 0.8539597988128662 A poodle -> A lion: 0.8460092544555664 A poodle -> A house: 0.8119628429412842 A wolf -> A cat: 0.9086413979530334 A wolf -> A dog: 0.9260395169258118 A wolf -> A labrador: 0.8111101984977722 A wolf -> A poodle: 0.8539597988128662 A wolf -> A wolf: 1.000000238418579 A wolf -> A lion: 0.9043934941291809 A wolf -> A house: 0.860664427280426 A lion -> A cat: 0.8914517164230347 A lion -> A dog: 0.886112630367279 A lion -> A labrador: 0.783727765083313 A lion -> A poodle: 0.8460092544555664 A lion -> A wolf: 0.9043934941291809 A lion -> A lion: 1.0000004768371582 A lion -> A house: 0.8402873873710632 A house -> A cat: 0.8724125027656555 A house -> A dog: 0.8852840662002563 A house -> A labrador: 0.7569846510887146 A house -> A poodle: 0.8119628429412842 A house -> A wolf: 0.860664427280426 A house -> A lion: 0.8402873873710632 A house -> A house: 0.9999997615814209 The results show that a dog is closer to a house than it is for a labrador 0.885 vs 0.848 which doesn't make sense. I've tried cosine similarity and euclidean distance to check whether the distance measure was wrong, but the results are similar. Where am I going wrong? 

Answer: If you use the text embeddings from the output of CLIPTextModel ([number of prompts, 77, 512]), flatten them ([number of prompts, 39424]) and the apply cosine similarity, you'll get improved results. This code lets you test both solutions ([1,512] and [77,512]). I'm running it on Google Colab.  !pip install -U torch transformers import torch from torch.nn import CosineSimilarity from transformers import CLIPTokenizer, CLIPModel, CLIPTextModel cossim = CosineSimilarity(dim=0, eps=1e-6) def dist(v1, v2): return cossim(v1, v2) torch_device = "cuda" if torch.cuda.is_available() else "cpu" models = [ 'openai/clip-vit-base-patch16', 'openai/clip-vit-base-patch32', 'openai/clip-vit-large-patch14', ] model_id = models[1] tokenizer = CLIPTokenizer.from_pretrained(model_id) text_encoder = CLIPTextModel.from_pretrained(model_id).to(torch_device) model = CLIPModel.from_pretrained(model_id).to(torch_device) prompts = [ "A cat", "A dog", "A labrador", "A poodle", "A wolf", "A lion", "A house", ] text_inputs = tokenizer( prompts, padding="max_length", return_tensors="pt", ).to(torch_device) text_features = model.get_text_features(**text_inputs) text_embeddings = torch.flatten(text_encoder(text_inputs.input_ids.to(torch_device))['last_hidden_state'],1,-1) print("\n\nusing text_features") for i1, label1 in enumerate(prompts): for i2, label2 in enumerate(prompts): if (i2>=i1): print(f"{label1} <-> {label2} = {dist(text_features[i1], text_features[i2]):.4f}") print("\n\nusing text_embeddings") for i1, label1 in enumerate(prompts): for i2, label2 in enumerate(prompts): if (i2>=i1): print(f"{label1} <-> {label2} = {dist(text_embeddings[i1], text_embeddings[i2]):.4f}")  You'll get the same values for the [1,512] embedding A cat <-> A cat = 1.0000 A cat <-> A dog = 0.9361 A cat <-> A labrador = 0.8171 A cat <-> A poodle = 0.8438 A cat <-> A wolf = 0.9086 A cat <-> A lion = 0.8915 A cat <-> A house = 0.8724 A dog <-> A dog = 1.0000 **A dog <-> A labrador = 0.8481** A dog <-> A poodle = 0.9010 A dog <-> A wolf = 0.9260 A dog <-> A lion = 0.8861 **A dog <-> A house = 0.8853** A labrador <-> A labrador = 1.0000 A labrador <-> A poodle = 0.7723 A labrador <-> A wolf = 0.8111 A labrador <-> A lion = 0.7837 A labrador <-> A house = 0.7570 A poodle <-> A poodle = 1.0000 A poodle <-> A wolf = 0.8540 A poodle <-> A lion = 0.8460 A poodle <-> A house = 0.8120 A wolf <-> A wolf = 1.0000 A wolf <-> A lion = 0.9044 A wolf <-> A house = 0.8607 A lion <-> A lion = 1.0000 A lion <-> A house = 0.8403 A house <-> A house = 1.0000  But the results have improved with the [1,77,512] embedding, and now the dog is closer to the labrador than to the house. Still, you'll get funny results such as the cat being more similar to a house than to a poodle. A cat <-> A cat = 1.0000 A cat <-> A dog = 0.8880 A cat <-> A labrador = 0.8057 A cat <-> A poodle = 0.7579 A cat <-> A wolf = 0.8558 A cat <-> A lion = 0.8358 A cat <-> A house = 0.8024 A dog <-> A dog = 1.0000 **A dog <-> A labrador = 0.8794** A dog <-> A poodle = 0.8583 A dog <-> A wolf = 0.8888 A dog <-> A lion = 0.8265 **A dog <-> A house = 0.8294** A labrador <-> A labrador = 1.0000 A labrador <-> A poodle = 0.8006 A labrador <-> A wolf = 0.8182 A labrador <-> A lion = 0.7958 A labrador <-> A house = 0.7608 A poodle <-> A poodle = 1.0000 A poodle <-> A wolf = 0.7928 A poodle <-> A lion = 0.7735 A poodle <-> A house = 0.7623 A wolf <-> A wolf = 1.0000 A wolf <-> A lion = 0.8496 A wolf <-> A house = 0.8063 A lion <-> A lion = 1.0000 A lion <-> A house = 0.7671 A house <-> A house = 1.0000