Model: chelleboyer/llm-evals-2-79b954ef-4798-4994-be72-a88d46b8ecca
Author: chelleboyer
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:400, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:2205.13147, arxiv:1705.00652, base_model:Snowflake/snowflake-arctic-embed-l, base_model:finetune:Snowflake/snowflake-arctic-embed-l, model-index, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: Snowflake/snowflake-arctic-embed-l
Card metadata: tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:400, loss:MatryoshkaLoss, loss:MultipleNegativesRankingLoss
base_model: Snowflake/snowflake-arctic-embed-l
widget: source_sentence: What is the title of the dataset introduced by Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu in their 2019 publication?
sentences: TechQA [3], Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu.


PubMedQA: A dataset for biomedical research question answering.


In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.


doi: 10.18653/v1/D19-1259., Our contributions address the need for standardized benchmarks and methodologies, enabling more precise and actionable insights into the strengths and weaknesses of different RAG systems. This, in turn, will facilitate iterative improvement of RAG models, driving forward the capabilities of retrieval-augmented generation in real-world applications.



References


Adlakha et al. [2023]

V. Adlakha, P. BehnamGhader, X. H. Lu, N. Meade, and S. Reddy., source_sentence: What does the 2024 paper by Es et al. propose regarding the evaluation of retrieval augmented generation?
sentences: doi: 10.18653/v1/2023.findings-acl.60.


URL https://aclanthology.org/2023.findings-acl.60.




Dinan et al. [2019]

E. Dinan, S. Roller, K. Shuster, A. Fan, M. Auli, and J. Weston.


Wizard of wikipedia: Knowledge-powered conversational agents, 2019.




Es et al. [2024]

S. Es, J. James, L. Espinosa Anke, and S. Schockaert.


RAGAs: Automated evaluation of retrieval augmented generation., Source Domains

RAGBench comprises five distinct domains: bio-medical research (PubmedQA, CovidQA), general knowledge (HotpotQA, MS Marco, HAGRID, ExperQA), legal contracts (CuAD), customer support (DelucionQA, EManual, TechQA), and finance (FinBench, TAT-QA). We select these specific domains based on availability of data, and applicability to real-world RAG applications across different industry verticals. For detailed descriptions of each component data source, refer to Appendix 7.2., The overall_supported_explanation field is a string explaining why the response
*as a whole* is or is not supported by the documents. In this field, provide a
step-by-step breakdown of the claims made in the response and the support (or
lack thereof) for those claims in the documents. Begin by assessing each claim
separately, one by one; don’t make any remarks about the response as a whole
until you have assessed all the claims in isolation., source_sentence: What are some common sources of questions in research or surveys?
sentences: Kwiatkowski et al. [2019]

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov.


Natural questions: a benchmark for question answering research.


Transactions of the Association of Computational Linguistics, 2019.




Laurer et al. [2022]

M. Laurer, W. van Atteveldt, A. Casas, and K. Welbers., Question Sources, the overall RAG system performance, with the potential to provide granular, actionable insights to the RAG practitioner., source_sentence: What evaluation metrics are reported for the response-level hallucination detection task?
sentences: 4.3 Evaluation

Our granular annotation schema allows for various evaluation setups. For example, we could evaluate either span-level or example/response-level predictions. For easy comparison with existing RAG evaluation approaches that are less granular, we report area under the receiver-operator curve (AUROC) on the response-level hallucination detection task, and root mean squared error (RMSE) for example-level context Relevance and Utilization predictions., EManual is a question answer dataset comprising consumer electronic device manuals and realistic questions about them composed by human annotators. The subset made available at the time of writing amounts to 659 unique questions about the Samsung Smart TV/remote and the accompanying user manual, segmented into 261 chunks. To form a RAG dataset, we embed the manual segments into a vector database with OpenAI embedding and retrieve up to 3 context documents per question from it. For each, Table 3: Benchmark evaluation on test splits. Reporting AUROC for predicting hallucinated responses (Hal), RMSE for predicting Context Relevance (Rel) and utilization (Util). ∗ indicates statistical significance at 95% confidence intervals, measured by bootstrap comparing the top and second-best results. RAGAS and Trulens do not evaluate Utilization.




GPT-3.5
RAGAS
TruLens
DeBERTA


Dataset


Hal↑↑\uparrow↑




Rel↓↓\downarrow↓


Util↓↓\downarrow↓



Hal↑↑\uparrow↑




Rel↓↓\downarrow↓, source_sentence: What is the main contribution of Kwiatkowski et al. [2019] in the field of question answering research?
sentences: The sentence_support_information field is a list of objects, one for each sentence
in the response. Each object MUST have the following fields:
- response_sentence_key: a string identifying the sentence in the response.
This key is the same as the one used in the response above.
- explanation: a string explaining why the sentence is or is not supported by the
documents.
- supporting_sentence_keys: keys (e.g. ’0a’) of sentences from the documents that, Kwiatkowski et al. [2019]

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov.


Natural questions: a benchmark for question answering research.


Transactions of the Association of Computational Linguistics, 2019.




Laurer et al. [2022]

M. Laurer, W. van Atteveldt, A. Casas, and K. Welbers., with consistent annotations. To best represent real-world RAG scenarios, we vary a number parameters to construct the benchmark: the source domain, number of context documents, context token length, and the response generator model Figure 1 illustrates where these variable parameters fall in the RAG pipeline.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics: cosine_accuracy@1, cosine_accuracy@3, cosine_accuracy@5, cosine_accuracy@10, cosine_precision@1, cosine_precision@3, cosine_precision@5, cosine_precision@10, cosine_recall@1, cosine_recall@3, cosine_recall@5, cosine_recall@10, cosine_ndcg@10, cosine_mrr@10, cosine_map@100
model-index: name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-l
results: task: type: information-retrieval
name: Information Retrieval
dataset: name: Unknown
type: unknown
metrics: type: cosine_accuracy@1
value: 0.8571428571428571
name: Cosine Accuracy@1
verified: no, type: cosine_accuracy@3
value: 0.9642857142857143
name: Cosine Accuracy@3
verified: no, type: cosine_accuracy@5
value: 1
name: Cosine Accuracy@5
verified: no, type: cosine_accuracy@10
value: 1
name: Cosine Accuracy@10
verified: no, type: cosine_precision@1
value: 0.8571428571428571
name: Cosine Precision@1
verified: no, type: cosine_precision@3
value: 0.32142857142857145
name: Cosine Precision@3
verified: no, type: cosine_precision@5
value: 0.20000000000000004
name: Cosine Precision@5
verified: no, type: cosine_precision@10
value: 0.10000000000000002
name: Cosine Precision@10
verified: no, type: cosine_recall@1
value: 0.8571428571428571
name: Cosine Recall@1
verified: no, type: cosine_recall@3
value: 0.9642857142857143
name: Cosine Recall@3
verified: no, type: cosine_recall@5
value: 1
name: Cosine Recall@5
verified: no, type: cosine_recall@10
value: 1
name: Cosine Recall@10
verified: no, type: cosine_ndcg@10
value: 0.9385586452838898
name: Cosine Ndcg@10
verified: no, type: cosine_mrr@10
value: 0.9178571428571428
name: Cosine Mrr@10
verified: no, type: cosine_map@100
value: 0.9178571428571428
name: Cosine Map@100
verified: no
Likes: 0 | All-time downloads: 6
Created: 2025-05-13 12:56:29