Neuronx model for BAAI/bge-base-en-v1.5
This repository contains are AWS Inferentia2 and neuronx
compatible checkpoint for BAAI/bge-base-en-v1.5. You can find detailed information about the base model on its Model Card.
Usage on Amazon SageMaker
coming soon
Usage with optimum-neuron
from optimum.neuron import NeuronModelForFeatureExtraction
from transformers import AutoTokenizer
import torch
import torch_neuronx
# Load Model from Hugging Face repository
model = NeuronModelForFeatureExtraction.from_pretrained("aws-neuron/bge-base-en-v1-5-seqlen-384-bs-1")
tokenizer = AutoTokenizer.from_pretrained("aws-neuron/bge-base-en-v1-5-seqlen-384-bs-1")
# sentence input
inputs = "Hello, my dog is cute"
# Tokenize sentences
encoded_input = tokenizer(inputs,return_tensors="pt",truncation=True,max_length=model.config.neuron["static_sequence_length"])
# Compute embeddings
with torch.no_grad():
model_output = model(*tuple(encoded_input.values()))
# Perform pooling. In this case, cls pooling.
sentence_embeddings = model_output[0][:, 0]
# normalize embeddings
sentence_embeddings = torch.nn.functional.normalize(sentence_embeddings, p=2, dim=1)
input_shapes
{
"sequence_length": 384,
"batch_size": 1
}
- Downloads last month
- 4,460
Evaluation results
- accuracy on MTEB AmazonCounterfactualClassification (en)test set self-reported76.149
- ap on MTEB AmazonCounterfactualClassification (en)test set self-reported39.323
- f1 on MTEB AmazonCounterfactualClassification (en)test set self-reported70.169
- accuracy on MTEB AmazonPolarityClassificationtest set self-reported93.387
- ap on MTEB AmazonPolarityClassificationtest set self-reported90.213
- f1 on MTEB AmazonPolarityClassificationtest set self-reported93.377
- accuracy on MTEB AmazonReviewsClassification (en)test set self-reported48.846
- f1 on MTEB AmazonReviewsClassification (en)test set self-reported48.146
- map_at_1 on MTEB ArguAnatest set self-reported40.754
- map_at_10 on MTEB ArguAnatest set self-reported55.761
- map_at_100 on MTEB ArguAnatest set self-reported56.331
- map_at_1000 on MTEB ArguAnatest set self-reported56.334
- map_at_3 on MTEB ArguAnatest set self-reported51.920
- map_at_5 on MTEB ArguAnatest set self-reported54.011
- mrr_at_1 on MTEB ArguAnatest set self-reported41.181
- mrr_at_10 on MTEB ArguAnatest set self-reported55.968
- mrr_at_100 on MTEB ArguAnatest set self-reported56.538
- mrr_at_1000 on MTEB ArguAnatest set self-reported56.542
- mrr_at_3 on MTEB ArguAnatest set self-reported51.980
- mrr_at_5 on MTEB ArguAnatest set self-reported54.209
- ndcg_at_1 on MTEB ArguAnatest set self-reported40.754
- ndcg_at_10 on MTEB ArguAnatest set self-reported63.605
- ndcg_at_100 on MTEB ArguAnatest set self-reported66.052
- ndcg_at_1000 on MTEB ArguAnatest set self-reported66.120
- ndcg_at_3 on MTEB ArguAnatest set self-reported55.708
- ndcg_at_5 on MTEB ArguAnatest set self-reported59.452
- precision_at_1 on MTEB ArguAnatest set self-reported40.754
- precision_at_10 on MTEB ArguAnatest set self-reported8.841
- precision_at_100 on MTEB ArguAnatest set self-reported0.991
- precision_at_1000 on MTEB ArguAnatest set self-reported0.100
- precision_at_3 on MTEB ArguAnatest set self-reported22.238
- precision_at_5 on MTEB ArguAnatest set self-reported15.149
- recall_at_1 on MTEB ArguAnatest set self-reported40.754
- recall_at_10 on MTEB ArguAnatest set self-reported88.407
- recall_at_100 on MTEB ArguAnatest set self-reported99.147
- recall_at_1000 on MTEB ArguAnatest set self-reported99.644
- recall_at_3 on MTEB ArguAnatest set self-reported66.714
- recall_at_5 on MTEB ArguAnatest set self-reported75.747
- v_measure on MTEB ArxivClusteringP2Ptest set self-reported48.749
- v_measure on MTEB ArxivClusteringS2Stest set self-reported42.808
- map on MTEB AskUbuntuDupQuestionstest set self-reported62.128
- mrr on MTEB AskUbuntuDupQuestionstest set self-reported74.281
- cos_sim_pearson on MTEB BIOSSEStest set self-reported89.246
- cos_sim_spearman on MTEB BIOSSEStest set self-reported86.938
- euclidean_pearson on MTEB BIOSSEStest set self-reported87.586
- euclidean_spearman on MTEB BIOSSEStest set self-reported88.056
- manhattan_pearson on MTEB BIOSSEStest set self-reported87.559
- manhattan_spearman on MTEB BIOSSEStest set self-reported88.197
- accuracy on MTEB Banking77Classificationtest set self-reported86.951
- f1 on MTEB Banking77Classificationtest set self-reported86.925
- v_measure on MTEB BiorxivClusteringP2Ptest set self-reported39.103