RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 2: Prompt excerpts for the question generation task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .; Prompt excerpts for the question generation task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .
Figure 3: Prompt excerpts for the question answering task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .; Prompt excerpts for the question answering task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .
Figure 4: Prompt excerpts for the query generation task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .; Prompt excerpts for the query generation task for the “hyper” setting. The red text is the expected model generation for the given prompt. The complete prompt contains four examples and is included in Appendix B .
Figure 5: Average dataset performance for HotpotQA, MuSiQue, 2WikiQA, and FEVER. We vary the amount of finetuning data and model sizes. We report model performance using SelfAsk when the amount of finetuning data equals to zero.; Average dataset performance for HotpotQA, MuSiQue, 2WikiQA, and FEVER. We vary the amount of finetuning data and model sizes. We report model performance using SelfAsk when the amount of finetuning data equals to zero.
Dataset statistics for synthetic data generated in this work. We omit the average length of answers for fact verification as it is a classification task. SQ=Single-Query. TQ=Two-Queries.
Numbers of evaluation data and documents in retrieval corpus used in this work.
Few-shot results on multi-hop question answering and fact verification benchmarks. We list the model size of GPT-3.5 as 175B since prior work uses the DaVinci model, which was estimated to have 175B parameters (Gao, 2021 ) . We note that the results from prior work are not directly comparable to ours mostly due to the differences in the sizes of evaluation datasets, retrieval corpus, and underlying base models. * indicates our re-implementation. We boldface the best results for GPT-3.5 and our work in each column.
Multi-hop question answering results comparing our work to prior work on few-shot multi-hop question generation. We obtain these results by finetuning LLaMA 7B on 100k data for each setting.
Multi-hop question answering results comparing our work to prior work on query generation. We additionally report precision (prec.) and recall (rec.) of the top-ranked documents for each task to measure retrieval performance. We obtain these results by finetuning LLaMA 7B on 100k data for each setting.
Results comparing with or without using the filtering steps. We obtain these results by finetuning LLaMA 7B on 100k data for each setting.
Results when finetuning LLaMA 7B on 100k data which consist of (1) both “hyper” and “topic” QA pairs, (2) “hyper” QA pairs only, and (3) “topic” QA pairs only.
Complete prompt for the question generation task in the “topic” setting.
Complete prompt for the question generation task in the “hyper” setting.
Complete prompt for the question answering task in the “topic” setting.
Complete prompt for the question answering task in the “hyper” setting.
Complete prompt for the query generation task in the “topic” setting.
Complete prompt for the query generation task in the “hyper” setting.
Complete prompt for the claim verification task for fact verification.
Complete prompt for the claim generation task for fact verification.
Complete prompt for the query generation task for fact verification.