Model: joshuapb/fine-tuned-matryoshka-100
Author: joshuapb
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:100, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, en, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:BAAI/bge-base-en-v1.5, base_model:finetune:BAAI/bge-base-en-v1.5, license:apache-2.0, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: BAAI/bge-base-en-v1.5
Card metadata: base_model: BAAI/bge-base-en-v1.5
language: en
library_name: sentence-transformers
license: apache-2.0
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
pipeline_tag: sentence-similarity
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:100, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
widget: source_sentence: Fig. 8. The accuracy of instruct-GPT series models of different sizes (left to right, small to large). Larger model doing better on binary classification of answerable and unanswerable questions in SelfAware eval. (Image source: Yin et al. 2023)
Another way to assess the model’s awareness of unknown knowledge is to measure the model’s output uncertainty. When a question is in-between known and unknown, the model is expected to demonstrate the right level of confidence.
The experiment by Kadavath et al. (2022) showed that LLMs are shown to be well calibrated in their estimation probabilities of answer correctness on diverse multiple choice questions in a format with visible lettered answer options (MMLU, TruthfulQA, QuALITY, LogiQA), meaning that the predicted probability coincides with the frequency of that answer being true. RLHF fine-tuning makes the model poorly calibrated, but higher sampling temperature leads to better calibration results.
sentences: What effect does the slower acquisition of new knowledge compared to established knowledge have on the effectiveness of large language models in practical scenarios?, How do discrepancies identified during the final output review phase affect the overall quality of the generated responses?, What effect does reinforcement learning from human feedback (RLHF) fine-tuning have on how well large language models assess the accuracy of their answers?, source_sentence: Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)
Some interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.

Unknown examples are fitted substantially slower than Known.
The best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.
Among Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.
sentences: What is the relationship between the structural formatting of inquiries and the occurrence of calibration errors in artificial intelligence models, and in what ways can this understanding contribute to the optimization of model training processes?, What are the benefits of integrating a pretrained Natural Language Inference (NLI) model with MPNet when assessing the reliability of reasoning paths in knowledge retrieval?, In what ways do the classifications of Known versus Unknown examples influence the propensity of AI models to generate hallucinations during their training processes?, source_sentence: Fig. 3. The evaluation framework for the FactualityPrompt benchmark.(Image source: Lee, et al. 2022)
Given the model continuation and paired Wikipedia text, two evaluation metrics for hallucination are considered:

Hallucination NE (Named Entity) errors: Using a pretrained entity detection model and document-level grounding, this metric measures the fraction of detected named entities that do not appear in the ground truth document.
Entailment ratios: Using a RoBERTa model fine-tuned on MNLI and sentence-level knowledge grounding, this metric calculates the fraction of generated sentences that are marked as relevant to the paired Wikipedia sentence by the entailment model.
sentences: What impact does the implementation of a pretrained query-document relevance model have on the process of document selection in research methodologies?, In what ways does the sequence in which information is delivered in AI-generated responses influence the likelihood of generating inaccuracies or hallucinations?, In what ways does the FactualityPrompt benchmark assess the performance of named entity detection models, particularly in relation to errors arising from hallucinated named entities?, source_sentence: Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)
Some interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.

Unknown examples are fitted substantially slower than Known.
The best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.
Among Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.
sentences: In what ways does the inherently adversarial structure of TruthfulQA inquiries facilitate the detection of prevalent fallacies in human cognitive processes, and what implications does this have for understanding the constraints of expansive language models?, In what ways do MaybeKnown cases influence the performance of a model when contrasted with HighlyKnown examples, particularly in relation to the occurrence of hallucinations?, In what ways does the Self-RAG framework leverage reflection tokens to enhance the quality of its generated outputs, and what implications does this have for the overall generation process?, source_sentence: Fine-tuning New Knowledge#
Fine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common technique for improving certain capabilities of the model like instruction following. Introducing new knowledge at the fine-tuning stage is hard to avoid.
Fine-tuning usually consumes much less compute, making it debatable whether the model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge encourages hallucinations. They found that (1) LLMs learn fine-tuning examples with new knowledge slower than other examples with knowledge consistent with the pre-existing knowledge of the model; (2) Once the examples with new knowledge are eventually learned, they increase the model’s tendency to hallucinate.
sentences: How does the IsRel token function in the retrieval process, and what impact does it have on the relevance of generated content to reduce hallucination?, What is the relationship between the calibration of AI models and the effectiveness of verbalized probabilities when applied to tasks of varying difficulty levels?, How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?
model-index: name: BGE base Financial Matryoshka
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 768
type: dim_768
metrics: type: cosine_accuracy@1
value: 0.828125
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9635416666666666
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9739583333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.828125
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3211805555555556
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.1947916666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.828125
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9635416666666666
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9739583333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9220150687007592
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.8976707175925925
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.8981047453703703
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 512
type: dim_512
metrics: type: cosine_accuracy@1
value: 0.8020833333333334
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9635416666666666
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9739583333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9895833333333334
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8020833333333334
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3211805555555556
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.1947916666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09895833333333333
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8020833333333334
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9635416666666666
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9739583333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9895833333333334
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9077325270335209
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.880220734126984
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.8810414411976911
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 256
type: dim_256
metrics: type: cosine_accuracy@1
value: 0.796875
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9583333333333334
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.96875
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9791666666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.796875
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3194444444444445
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19374999999999998
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09791666666666665
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.796875
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9583333333333334
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.96875
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9791666666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9011377823848584
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.8746155753968253
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.8757564484126984
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 128
type: dim_128
metrics: type: cosine_accuracy@1
value: 0.7864583333333334
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9322916666666666
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9635416666666666
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9635416666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.7864583333333334
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3107638888888889
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19270833333333334
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09635416666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.7864583333333334
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9322916666666666
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9635416666666666
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9635416666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.888061438431803
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.8623263888888889
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.8647421480429293
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 64
type: dim_64
metrics: type: cosine_accuracy@1
value: 0.6875
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.8645833333333334
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9270833333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.96875
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.6875
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.2881944444444445
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.18541666666666665
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09687499999999999
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.6875
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.8645833333333334
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9270833333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.96875
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.8335872598831777
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.7895895337301586
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.7917890681938919
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 9
Created: 2024-08-23 22:45:47