Model: joshuapb/fine-tuned-matryoshka-200
Author: joshuapb
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:200, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, en, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:BAAI/bge-base-en-v1.5, base_model:finetune:BAAI/bge-base-en-v1.5, license:apache-2.0, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: BAAI/bge-base-en-v1.5
Card metadata: base_model: BAAI/bge-base-en-v1.5
language: en
library_name: sentence-transformers
license: apache-2.0
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
pipeline_tag: sentence-similarity
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:200, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
widget: source_sentence: Verbalized number or word (e.g. “lowest”, “low”, “medium”, “high”, “highest”), such as "Confidence: 60% / Medium".
Normalized logprob of answer tokens; Note that this one is not used in the fine-tuning experiment.
Logprob of an indirect "True/False" token after the raw answer.
Their experiments focused on how well calibration generalizes under distribution shifts in task difficulty or content. Each fine-tuning datapoint is a question, the model’s answer (possibly incorrect), and a calibrated confidence. Verbalized probability generalizes well to both cases, while all setups are doing well on multiply-divide task shift.  Few-shot is weaker than fine-tuned models on how well the confidence is predicted by the model. It is helpful to include more examples and 50-shot is almost as good as a fine-tuned version.
sentences: What is the relationship between the calibration of AI models and the effectiveness of verbalized probabilities when applied to tasks of varying difficulty levels?, In what ways does the F1 @ K metric contribute to evaluating the factual accuracy and comprehensiveness of outputs generated by long-form language models?, What impact does the implementation of a pretrained query-document relevance model have on the process of document selection in research methodologies?, source_sentence: Fig. 4. Overview of SAFE for factuality evaluation of long-form LLM generation. (Image source: Wei et al. 2024)
The SAFE evaluation metric is F1 @ K. The motivation is that model response for long-form factuality should ideally hit both precision and recall, as the response should be both

factual : measured by precision, the percentage of supported facts among all facts in the entire response.
long : measured by recall, the percentage of provided facts among all relevant facts that should appear in the response. Therefore we want to consider the number of supported facts up to $K$.

Given the model response $y$, the metric F1 @ K is defined as:
sentences: What methodologies does the agreement model employ to identify discrepancies between the original and revised text, and how do these methodologies impact the overall editing workflow?, In what ways does the SAFE evaluation metric achieve a harmonious equilibrium between precision and recall in the context of evaluating the factual accuracy of long-form outputs generated by large language models?, In what ways does the inherently adversarial structure of TruthfulQA inquiries facilitate the detection of prevalent fallacies in human cognitive processes, and what implications does this have for understanding the constraints of expansive language models?, source_sentence: Non-context LLM: Prompt LLM directly with <atomic-fact> True or False? without additional context.
Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context.
Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction.
Retrieval→LLM + NP: Ensemble of two methods.

Some interesting observations on model hallucination behavior:

Error rates are higher for rarer entities in the task of biography generation.
Error rates are higher for facts mentioned later in the generation.
Using retrieval to ground the model generation significantly helps reduce hallucination.
sentences: In what ways does the Rethinking with Retrieval (RR) methodology leverage Chain-of-Thought (CoT) prompting to enhance the efficacy of external knowledge retrieval, and what implications does this have for the precision of predictive outcomes generated by models?, In what ways does the retrieval of related passages contribute to minimizing hallucinations in large language models, and what specific techniques can be employed to evaluate the impact of this approach?, What are the benefits of using retrieval methods in biography generation to minimize inaccuracies, especially when compared to traditional prompting techniques that lack context?, source_sentence: Yin et al. (2023) studies the concept of self-knowledge, referring to whether language models know what they know or don’t know.
SelfAware, containing 1,032 unanswerable questions across five categories and 2,337 answerable questions. Unanswerable questions are sourced from online forums with human annotations while answerable questions are sourced from SQuAD, HotpotQA and TriviaQA based on text similarity with unanswerable questions. A question may be unanswerable due to various reasons, such as no scientific consensus, imaginations of the future, completely subjective, philosophical reasons that may yield multiple responses, etc. Considering separating answerable vs unanswerable questions as a binary classification task, we can measure F1-score or accuracy and the experiments showed that larger models can do better at this task.
sentences: What is the relationship between model size and performance metrics, such as F1-score and accuracy, in the context of classifying questions into answerable and unanswerable categories?, How does the introduction of stochastic perturbations in synthetic training data contribute to the enhancement of editor model efficacy within LangChain frameworks?, How do the various output values linked to reflection tokens in the Self-RAG framework impact the generation process, and why are they important?, source_sentence: Fig. 1. Knowledge categorization of close-book QA examples based on how likely the model outputs correct answers. (Image source: Gekhman et al. 2024)
Some interesting observations of the experiments, where dev set accuracy is considered a proxy for hallucinations.

Unknown examples are fitted substantially slower than Known.
The best dev performance is obtained when the LLM fits the majority of the Known training examples but only a few of the Unknown ones. The model starts to hallucinate when it learns most of the Unknown examples.
Among Known examples, MaybeKnown cases result in better overall performance, more essential than HighlyKnown ones.
sentences: In what ways does the fitting speed of examples that are not previously encountered differ from that of familiar examples, and how does this variation influence the overall accuracy of the model on the development set?, What role do reflection tokens play in enhancing the efficiency of document retrieval and generation within the Self-RAG framework?, How do the results presented by Gekhman et al. in their 2024 study inform our understanding of the reliability metrics associated with large language models (LLMs) when subjected to fine-tuning with novel datasets?
model-index: name: BGE base Financial Matryoshka
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 768
type: dim_768
metrics: type: cosine_accuracy@1
value: 0.8802083333333334
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.96875
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.984375
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8802083333333334
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3229166666666667
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.196875
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8802083333333334
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.96875
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.984375
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9433275174124347
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9261284722222224
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9264025950292397
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 512
type: dim_512
metrics: type: cosine_accuracy@1
value: 0.8697916666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9739583333333334
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9739583333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8697916666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3246527777777778
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.1947916666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8697916666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9739583333333334
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9739583333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.939968526552219
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9216269841269841
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9220610119047619
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 256
type: dim_256
metrics: type: cosine_accuracy@1
value: 0.8697916666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9739583333333334
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.984375
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8697916666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3246527777777778
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.196875
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09999999999999999
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8697916666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9739583333333334
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.984375
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9419747509776967
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.922676917989418
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.922676917989418
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 128
type: dim_128
metrics: type: cosine_accuracy@1
value: 0.8541666666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9583333333333334
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.96875
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8541666666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3194444444444445
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19374999999999998
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8541666666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9583333333333334
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.96875
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9306358745697197
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9094328703703702
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9098668981481483
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 64
type: dim_64
metrics: type: cosine_accuracy@1
value: 0.7916666666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.953125
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9739583333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9895833333333334
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.7916666666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3177083333333333
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.1947916666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09895833333333333
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.7916666666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.953125
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9739583333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9895833333333334
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9003914274568845
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.8705935846560847
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.8713150853775854
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 113
Created: 2024-08-23 22:47:18