J-MADRAL/J-BiBERT

J-BiBERT is a BERT-sized dense retriever initialized from BERT public checkpoint, further pre-trained on e-commerce product and review data, and fine-tuned jointly on product and review search retrieval tasks on the Amazon ESCI and Search ESCI datasets. We use symmetric encoder architecture, with a single shared encoder for both queries and products. The similarity function is dot product.

Paper and Repository

J-BiBERT has been described in the Multi-Aspect Joint Retrieval for E-Commerce: Bridging Product Catalogs and Customer Reviews paper. The associated GitHub repository is available at https://anonymous.4open.science/r/J-MADRAL-C4CC.

Usage (HuggingFace Transformers)

Using the model directly in HuggingFace transformers requires additional code available in the repository.

import modeling
import torch
import transformers

# We use a couple of training queries from Amazon ESCI and Search ESCI as an example.
queries = [
    "iphone 11 pro max case"
    "cotton summer dress care instructions"
]
products_and_reviews = [
    "OtterBox Symmetry Series Case for iPhone 11 Pro Max - Black [...]",
    "Camera Lens Protector for iPhone 11 Pro/Pro Max, Tempered Glass 9H Hardness Anti-Scratch Camera Screen Protective [...]"
    "Cute, cool and comfy summer dress [...] Hand wash and line drys easily, material is crinkly so no ironing needed. [...]",
    "Excellent machine JET J-2530 15-Inch 3/4-Horsepower Bench Drill Press. Two common Amazon reviewer complaints about higher-end drill presses [...]"
]

# Load the tokenizer and model.
tokenizer = transformers.AutoTokenizer.from_pretrained("J-MADRAL/J-BiBERT")
model = modeling.BiEncoderModel.from_pretrained("J-MADRAL/J-BiBERT")

# Tokenize the input data.
q_input = tokenizer(queries,
                    add_special_tokens=True,
                    truncation=True,
                    padding=True,
                    max_length=128,
                    return_tensors="pt")
pr_input = tokenizer(products_and_reviews,
                     add_special_tokens=True,
                     truncation=True,
                     padding=True,
                     max_length=128,
                     return_tensors="pt")

# Compute embeddings: take the "pooled_output".
q_emb = model(**q_input).pooled_output
pr_emb = model(**pr_input).pooled_output

# Compute similarity scores, using dot product similarity.
scores = torch.matmul(q_emb, pr_emb.transpose(0, 1))

Training Hyperparameters

Training Stage	Num. Epochs	Learning Rate	AP Scaling Factor	Max Num Tokens	Batch Size	Num Negatives
Pre-training	20	1e-4	0.10	128	64	---
Fine-tuning	20	5e-6	0.05	128	64	7

The data used for fine-tuning is available at https://huggingface.co/datasets/J-MADRAL/TrainingData.

Evaluation Results

Amazon ESCI

Model	R@100	R@500	MRR	nDCG@10	nDCG@50
BM25	0.4949	0.6603	0.4056	0.2599	0.3160
DRAGON	0.5490	0.7155	0.4541	0.2929	0.3540
P-BiBERT	0.6018	0.7640	0.4939	0.3276	0.3967
P-MADRAL	0.6235	0.7806	0.5060	0.3382	0.4104
J-BiBERT	0.5889	0.7556	0.4808	0.3195	0.3858
J-MADRAL	0.6083	0.7729	0.4988	0.3330	0.4028

TREC Product Search 2023

Model	R@100	R@500	MRR	nDCG@10	nDCG@50
BM25	0.7252	0.8650	0.7746	0.6231	0.5988
DRAGON	0.7373	0.8692	0.8135	0.6526	0.6293
P-BiBERT	0.7432	0.8757	0.8256	0.6612	0.6287
P-MADRAL	0.7547	0.8840	0.8337	0.6719	0.6480
J-BiBERT	0.7432	0.8684	0.8114	0.6487	0.6213
J-MADRAL	0.7527	0.8888	0.8333	0.6702	0.6411

Search ESCI

Model	R@100	R@500	MRR	nDCG@10	nDCG@50
BM25	0.5875	0.7288	0.2539	0.2766	0.3101
DRAGON	0.5451	0.6751	0.2347	0.2567	0.2873
R-BiBERT	0.5972	0.7350	0.2602	0.2855	0.3176
R-MADRAL	0.6405	0.7626	0.2944	0.3215	0.3541
J-BiBERT	0.6300	0.7593	0.2879	0.3140	0.3474
J-MADRAL	0.6488	0.7729	0.3007	0.3281	0.3611