Edit model card

Model

Cross-Encoder Model for sentence-similarity

This model was is an improvement over the dangvantuan/CrossEncoder-camembert-large offering greater robustness and better performance

Training Data

This model was trained on the STS benchmark dataset and has been combined with Augmented SBERT. The model benefits from Pair Sampling Strategies using two models: CrossEncoder-camembert-large and dangvantuan/sentence-camembert-large. The model will predict a score between 0 and 1 how for the semantic similarity of two sentences.

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

pip install -U sentence-transformers

Then you can use the model like this:

from sentence_transformers import CrossEncoder
model = CrossEncoder('Lajavaness/CrossEncoder-camembert-large', max_length=512)
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ])

Evaluation

The model can be evaluated as follows on the French test data of stsb.

from sentence_transformers.readers import InputExample
from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator
from datasets import load_dataset
def convert_dataset(dataset):
    dataset_samples=[]
    for df in dataset:
        score = float(df['similarity_score'])/5.0  # Normalize score to range 0 ... 1
        inp_example = InputExample(texts=[df['sentence1'], 
                                    df['sentence2']], label=score)
        dataset_samples.append(inp_example)
    return dataset_samples

# Loading the dataset for evaluation
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev")
df_test = load_dataset("stsb_multi_mt", name="fr", split="test")

# Convert the dataset for evaluation

# For Dev set:
dev_samples = convert_dataset(df_dev)
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev')
val_evaluator(model, output_path="./")

# For Test set, the Pearson and Spearman correlation are evaluated on many different benchmark datasets:

test_samples = convert_dataset(df_test)
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test')
test_evaluator(models, output_path="./")

Test Result: The performance is measured using Pearson and Spearman correlation:

  • On dev
Model Pearson correlation Spearman correlation #params
Lajavaness/CrossEncoder-camembert-large 90.34 90.15 336M
dangvantuan/CrossEncoder-camembert-large 90.11 90.01 336M
  • On test:

Pearson score

Model STS-B STS12-fr STS13-fr STS14-fr STS15-fr STS16-fr SICK-fr
Lajavaness/CrossEncoder-camembert-large 88.63 90.76 88.24 90.22 92.23 82.31 84.61
dangvantuan/CrossEncoder-camembert-large 88.16 90.12 88.36 89.86 92.04 82.01 84.23

Spearman score

Model STS-B STS12-fr STS13-fr STS14-fr STS15-fr STS16-fr SICK-fr
Lajavaness/CrossEncoder-camembert-large 88.03 84.87 87.88 89.10 92.16 82.50 80.78
dangvantuan/CrossEncoder-camembert-large 87.57 84.24 88.01 88.62 91.99 82.16 80.38
Downloads last month
257
Inference API
Inference API (serverless) does not yet support transformers models for this pipeline type.

Dataset used to train Lajavaness/CrossEncoder-camembert-large

Evaluation results

  • Test Pearson correlation coefficient on Text Similarity fr
    self-reported
    90.340