Model: joshuapb/fine-tuned-matryoshka-500
Author: joshuapb
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:500, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, en, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:BAAI/bge-base-en-v1.5, base_model:finetune:BAAI/bge-base-en-v1.5, license:apache-2.0, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: BAAI/bge-base-en-v1.5
Card metadata: base_model: BAAI/bge-base-en-v1.5
language: en
library_name: sentence-transformers
license: apache-2.0
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
pipeline_tag: sentence-similarity
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:500, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
widget: source_sentence: Non-context LLM: Prompt LLM directly with <atomic-fact> True or False? without additional context.
Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context.
Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction.
Retrieval→LLM + NP: Ensemble of two methods.

Some interesting observations on model hallucination behavior:

Error rates are higher for rarer entities in the task of biography generation.
Error rates are higher for facts mentioned later in the generation.
Using retrieval to ground the model generation significantly helps reduce hallucination.
sentences: What is the impact of infrequent entities on the efficacy of language models in the context of biography generation?, In what ways does FActScore enhance the assessment of factual accuracy in long-form content generation when compared to conventional evaluation techniques?, What approaches does SelfCheckGPT implement when faced with questions it cannot answer, and how does this influence its overall reliability in delivering accurate information?, source_sentence: Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$.

(1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$.
(2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$.
(3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$.




Fig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022)
When evaluating the revised text $y$, both attribution and preservation metrics matter.
sentences: What impact does adjusting the sampling temperature have on the calibration of large language models, and how does this influence the uncertainty of their outputs?, How do unanswerable questions differ from answerable ones in the context of a language model's understanding of its own capabilities?, In what ways does the agreement model evaluate discrepancies between the provided evidence and the updated text, and how does this evaluation impact the reliability of AI-generated content modifications?, source_sentence: Non-context LLM: Prompt LLM directly with <atomic-fact> True or False? without additional context.
Retrieval→LLM: Prompt with $k$ related passages retrieved from the knowledge source as context.
Nonparametric probability (NP)): Compute the average likelihood of tokens in the atomic fact by a masked LM and use that to make a prediction.
Retrieval→LLM + NP: Ensemble of two methods.

Some interesting observations on model hallucination behavior:

Error rates are higher for rarer entities in the task of biography generation.
Error rates are higher for facts mentioned later in the generation.
Using retrieval to ground the model generation significantly helps reduce hallucination.
sentences: In what ways can the acknowledgment of uncertainty by large language models (LLMs) contribute to the mitigation of hallucinations and enhance the overall factual accuracy of generated content?, In what ways does the process of retrieving related passages contribute to minimizing hallucinations in the outputs generated by language models, and how does this approach differ from the application of nonparametric probability methods?, How does the triplet structure $(c, y, y^*)$ play a crucial role in the categorization of errors, and in what ways does it enhance the training process of the editor model?, source_sentence: Fine-tuning New Knowledge#
Fine-tuning a pre-trained LLM via supervised fine-tuning and RLHF is a common technique for improving certain capabilities of the model like instruction following. Introducing new knowledge at the fine-tuning stage is hard to avoid.
Fine-tuning usually consumes much less compute, making it debatable whether the model can reliably learn new knowledge via small-scale fine-tuning. Gekhman et al. 2024 studied the research question of whether fine-tuning LLMs on new knowledge encourages hallucinations. They found that (1) LLMs learn fine-tuning examples with new knowledge slower than other examples with knowledge consistent with the pre-existing knowledge of the model; (2) Once the examples with new knowledge are eventually learned, they increase the model’s tendency to hallucinate.
sentences: How do the intentionally designed questions in TruthfulQA highlight prevalent misunderstandings regarding AI responses in the healthcare domain?, What effect does the slower acquisition of new knowledge compared to established knowledge have on the effectiveness of large language models in practical scenarios?, How do the RARR methodology and the FAVA model compare in their approaches to mitigating hallucination errors in generated outputs, and what key distinctions can be identified between the two?, source_sentence: Revision stage: Edit the output to correct content unsupported by evidence while preserving the original content as much as possible. Initialize the revised text $y=x$.

(1) Per $(q_i, e_{ij})$, an agreement model (via few-shot prompting + CoT, $(y, q, e) \to {0,1}$) checks whether the evidence $e_i$ disagrees with the current revised text $y$.
(2) Only if a disagreement is detect, the edit model (via few-shot prompting + CoT, $(y, q, e) \to \text{ new }y$) outputs a new version of $y$ that aims to agree with evidence $e_{ij}$ while otherwise minimally altering $y$.
(3) Finally only a limited number $M=5$ of evidence goes into the attribution report $A$.




Fig. 12. Illustration of RARR (Retrofit Attribution using Research and Revision). (Image source: Gao et al. 2022)
When evaluating the revised text $y$, both attribution and preservation metrics matter.
sentences: What mechanisms does the editing algorithm employ to maintain fidelity to the source material while simultaneously ensuring alignment with the supporting evidence?, What is the impact of constraining the dataset to a maximum of $M=5$ instances on the accuracy and reliability of the attribution report $A$ when analyzing AI-generated content?, In what ways does the implementation of a query generation model enhance the research phase when it comes to validating the accuracy of information?
model-index: name: BGE base Financial Matryoshka
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 768
type: dim_768
metrics: type: cosine_accuracy@1
value: 0.8802083333333334
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.96875
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9895833333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8802083333333334
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3229166666666667
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19791666666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09999999999999999
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8802083333333334
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.96875
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9895833333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9477255159324969
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9301711309523809
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.930171130952381
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 512
type: dim_512
metrics: type: cosine_accuracy@1
value: 0.875
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.96875
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9947916666666666
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.875
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3229166666666667
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19895833333333335
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09999999999999999
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.875
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.96875
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9947916666666666
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9459628876705072
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9277405753968253
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9277405753968253
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 256
type: dim_256
metrics: type: cosine_accuracy@1
value: 0.8802083333333334
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.96875
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9947916666666666
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8802083333333334
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.3229166666666667
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19895833333333335
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09999999999999999
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8802083333333334
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.96875
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9947916666666666
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9458393511377685
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9277405753968254
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9277405753968253
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 128
type: dim_128
metrics: type: cosine_accuracy@1
value: 0.8697916666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.984375
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9895833333333334
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8697916666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.328125
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19791666666666666
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8697916666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.984375
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9895833333333334
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9440191417149189
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9265252976190478
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.92687251984127
name: Cosine Map@100
verified: no, task: type: information-retrieval
name: Information Retrieval
dataset: name: dim 64
type: dim_64
metrics: type: cosine_accuracy@1
value: 0.8541666666666666
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.984375
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 0.9947916666666666
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 0.9947916666666666
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8541666666666666
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.328125
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.19895833333333335
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.09947916666666667
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8541666666666666
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.984375
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 0.9947916666666666
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 0.9947916666666666
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9380774892768095
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9184027777777778
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9186111111111112
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 9
Created: 2024-08-23 23:20:38