Update README.md

README.md

@@ -1,3 +1,142 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+# Marqo E-commerce Embedding Models
+In this work, we introduce two state-of-the-art embedding models for e-commerce: 
+Marqo-Ecommerce-B and Marqo-Ecommerce-L. 
+They are over 30% better compared to Amazon Titan Embedding services for e-commerce retrieval tasks.
+
+**Released Content**: 
+1) Marqo-Ecommerce-B and Marqo-Ecommerce-L embedding models
+2) GoogleShopping-1m and AmazonProducts-3m for evaluation
+3) Evaluation Code
+
+<img src="performance.png" alt="multi split visual" width="500"/>
+
+## Models
+
+| **Embedding Model** | **#Params (m)** | **Dimension** | **HuggingFace**                    | **Download .pt**                                                                                            |
+|---------------------| --- |---------------|------------------------------------|-------------------------------------------------------------------------------------------------------------|
+| Marqo-Ecommerce-B   | 203 | 768           | Marqo/marqo-ecommerce-embeddings-B | [link](https://marqo-gcl-public.s3.us-west-2.amazonaws.com/marqo-general-ecomm/marqo-ecomm-embeddings-b.pt) |
+| Marqo-Ecommerce-L   | 652 | 1024          | Marqo/marqo-ecommerce-embeddings-L | [link](https://marqo-gcl-public.s3.us-west-2.amazonaws.com/marqo-general-ecomm/marqo-ecomm-embeddings-l.pt)                                       |
+
+### HuggingFace with OpenCLIP
+```
+pip install open_clip_torch
+```
+```python
+from PIL import Image
+import open_clip
+import requests
+import torch
+
+# Specify model from Hugging Face Hub
+model_name = 'hf-hub:Marqo/marqo-ecommerce-embeddings-B'
+model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms(model_name)
+tokenizer = open_clip.get_tokenizer(model_name)
+
+# Preprocess the image and tokenize text inputs
+# Load an example image from a URL
+img = Image.open(requests.get('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png', stream=True).raw)
+image = preprocess_val(img).unsqueeze(0)
+text = tokenizer(["a hat", "a t-shirt", "shoes"])
+
+# Perform inference
+with torch.no_grad(), torch.cuda.amp.autocast():
+    image_features = model.encode_image(image, normalize=True)
+    text_features = model.encode_text(text, normalize=True)
+
+    # Calculate similarity probabilities
+    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
+
+# Display the label probabilities
+print("Label probs:", text_probs)
+# [9.9955e-01, 4.4712e-04, 4.4010e-06]]
+```
+### HuggingFace with transformers
+```python
+from transformers import AutoModel, AutoProcessor
+import torch
+from PIL import Image
+import requests
+# model_name= 'Marqo/marqo-ecommerce-embeddings-L'
+model_name = 'Marqo/marqo-ecommerce-embeddings-B'
+
+model_1 = AutoModel.from_pretrained(model_name, trust_remote_code=True)
+processor_1 = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
+
+img = Image.open(requests.get('https://raw.githubusercontent.com/marqo-ai/marqo-FashionCLIP/main/docs/fashion-hippo.png', stream=True).raw).convert("RGB")
+image_1 = [img]
+text_1 = ["a hat", "a t-shirt", "shoes"]
+processed_1 = processor_1(text=text_1, images=image_1, padding='max_length', return_tensors="pt")
+processor_1.image_processor.do_rescale = False
+with torch.no_grad():
+    image_features_1 = model_1.get_image_features(processed_1['pixel_values'], normalize=True)
+    text_features_1 = model_1.get_text_features(processed_1['input_ids'], normalize=True)
+
+    text_probs_1 = (100 * image_features_1 @ text_features_1.T).softmax(dim=-1)
+    
+print(text_probs_1)
+# [9.9955e-01, 4.4712e-04, 4.4010e-06]]
+```
+
+### Evaluation with GCL
+```
+git clone https://github.com/marqo-ai/GCL
+```
+Install the packages required by GCL. 
+```
+cd ./GCL
+MODEL=hf-hub:Marqo/marqo-ecommerce-B
+outdir=/MarqoModels/GE/marqo-ecommerce-B/gs-title2image2
+hfdataset=Marqo/google-shopping-general-eval
+python  evals/eval_hf_datasets_v1.py \
+      --model_name $MODEL \
+      --hf-dataset $hfdataset \
+      --output-dir $outdir \
+      --batch-size 1024 \
+      --num_workers 8 \
+      --left-key "['title']" \
+      --right-key "['image']" \
+      --img-or-txt "[['txt'], ['img']]" \
+      --left-weight "[1]" \
+      --right-weight "[1]" \
+      --run-queries-cpu \
+      --top-q 4000 \
+      --doc-id-key item_ID \
+      --context-length "[[64], [0]]"
+```
+
+
+## Detailed Performance
+**GoogleShopping-Text2Image Retrieval.**
+
+| **Embedding Model** | **mAP** | **P@10** | **R@10** | **MRR** |
+| --- | --- | --- | --- | --- |
+| Marqo-Ecommerce-L | **0.682** | **0.089** | **0.878** | **0.683** |
+| Marqo-Ecommerce-B | 0.623 | 0.084 | 0.832 | 0.624 |
+| Amazon-Titan-MultiModal | 0.475 | 0.065 | 0.648 | 0.475 |
+| ViT-B-16-SigLip | 0.476 | 0.067 | 0.660 | 0.477 |
+| ViT-L-16-SigLip | 0.540 | 0.073 | 0.722 | 0.540 |
+
+
+**GoogleShopping-Category2Image Retrieval.**  
+
+| **Embedding Model** | **mAP** | **P@10** | **MRR** | **nDCG@10** |
+| --- | --- | --- | --- | --- |
+| Marqo-Ecommerce-L | **0.463** | **0.652** | **0.822** | **0.666** |
+| Marqo-Ecommerce-B | 0.423 | 0.629 | 0.810 | 0.644 |
+| Amazon-Titan-MultiModal | 0.246 | 0.429 | 0.642 | 0.446 |
+| ViT-B-16-SigLip | 0.277 | 0.458 | 0.660 | 0.473 |
+| ViT-L-16-SigLip | 0.324 | 0.497 | 0.687 | 0.509 |
+
+**AmazonProducts-Text2Image Retrieval.** 
+
+| **Embedding Model** | **mAP** | **P@10** | **R@10** | **MRR** |
+| --- | --- | --- | --- | --- |
+| Marqo-Ecommerce-L | **0.658** | **0.096** | **0.854** | **0.663** |
+| Marqo-Ecommerce-B | 0.592 | 0.089 | 0.795 | 0.597 |
+| Amazon-Titan-MultiModal | 0.456 | 0.064 | 0.627 | 0.457 |
+| ViT-B-16-SigLip | 0.480 | 0.070 | 0.650 | 0.484 |
+| ViT-L-16-SigLip | 0.544 | 0.077 | 0.715 | 0.548 |
+