TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
Authors: Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong
Accepted at: acl25


Reviewer paper summaries: The authors consider the problem of tracing sentences precisely to a text they originate from, together with a fine-granularity relationship among quotation, compression, inference, and others (that can potentially encompass negation). 
They introduce the problem, and build a benchmark for the task, considering both Chinese and English, and various lengths of text. They compare 11 LLMs on the task, and report macro- and micro- precision and recall, as well as an overall F1.

This paper proposes a benchmark, Text pROVEnance (TROVE), for identifying source texts given the target texts and providing the relations between the src and target sentences. The dataset is constructed on top of three existing datasets, and the annotations are initialized with GPT-4 predictions and then verified by human annotators.  11 LLMs are benchmarked, and the relation classification task remains challenging even for close-sourced LLMs. Retrieval augmented tracing is found to be important in the proposed task.
Summary of strengths: The problem appears novel and useful. The experimental evaluation shows that the problem is not solved yet, and remains challenging including for large models. 
Having considered both English and Chinese is a plus.
The authors show that retrieval-augmented tracing (RA) performs better than direct prompting (DP). 

The authors provide the complete prompts in the Appendix.

- The paper introduces the task of identifying finer-grained provenance with four types of relationships.
- The authors collected human annotations on both sentence alignment and relations, which improves the quality of the dataset.
- Large-scale experiments are conducted across 11 LLMs covering both open-sourced and close-sourced models, offering a fair comparison to evaluate how different LLMs perform on this task.
Summary of weaknesses: The authors acknowledge (end of 4.1) that "some subsets inevitably remain underrepresented". At this point, the reader would like to know more (how bad this problem is). It would be good to point perhaps to the Appendix or another place where this imbalance is documented.

The writing may need to be improved in places, e.g., 4.2:
"[we] then perform provenance based on these sentences" - "to perform provenance" is not correct English, please reformulate

"We have presented ablation studies" should read "We present ablation studies".


Figures 3 and 5 are unreadable (the font is way too small). Also, insufficient contrast is used in the colors, which means a B/W printout cannot be read. This is a pity because they are interesting, i.e., I comment the authors for having added them, but I can't appreciate them on paper. I understand any PDF can be enlarged, but there are also some common limits to the size of the font used. 

Under Table 3, in the paragraph "Precision-Recall Trade-offs across Models", the last two phrases ("In real-world applications, ... is a priority") seem to state very general things (they would be right regardless of the problem at hand). I suggest removing them to make space for making Figures 3 and 5 more readable. 

Also, the beginning of the "Relationship Classification" paragraph feels repetitive. It could start directly with: "Table 3 shows that Larger models..." (then the existing text) 

Appendix C shows that retrieval has a particularly dramatic impact for Gemini and Kimi, than for ChatGPT. It would be nice if the authors could comment about possible reasons for that.  

The authors do not provide the benchmark nor do they (as far as I can tell) promise to release it. Maybe they mean to do it upon acceptance, but this has not been clear based on the paper. Further, they built their dataset by selecting some content from a set of prior corpora. Currently, a motivated reader could not redo this task because of lack of information on how the choice was made.

- While the major motivation is to facilitate future research within high-stakes domains like legal and medical, the benchmark is still built mainly on news and dialogue datasets (only 37 documents belong to GovReport/arXiv in Table 4). I wonder whether the takeaways can be transferable to the legal/medical domains where domain knowledge is more vital. 
- One of the main contributions is the human revision of GPT-4 predicted provenance. However, the details on the revision are missing.
The dataset's description is a bit unclear. Human annotations are introduced as the "provenance of 4,388 English and 811 Chinese sentences." I assume you are talking about the number of target sentences. The authors repetitively use "provenance sentences" in the paper when referring to the candidate source sentences. 
- In Section 5 ("Retrieval Augment," lines 370-372), the methodology is unclear. Specifically, it is not specified whether the authors directly incorporated a retrieval prompt or provided the retrieved candidates in Equation 1.
- Analysis of how models perform differently across different relations would be more helpful.
Comments, suggestions and typos: Numbers are usually right-aligned so that the decimals of the same order are one under another, units one under another, hundreds one under another, etc. Please right-align numbers in Table 1. 
(Table 2 is already OK, although there is just something surprising with the Macro R-Ps for Vicuna, which are not exactly one under the other, not clear why).

- GPT4 appears multiple times in the paper and is used as the primary model to initialize the provenance sentences, but its performances and details are missing. What is the model you use here, and what is its performance?
- What is the error distribution across different relationship types? Are some relation types more difficult for models to classify than others? A confusion matrix analysis could provide insights into common error patterns and model weaknesses.
Overall assessment scores: 4

3
Metareview score: 4
Metareview: The paper proposes a new benchmark, TROVE, for fine-grained text provenance, focusing on tracing each sentence in a generated text back to its specific sources and classifying the relationship. This fills a gap between coarse-grained citation tasks and deeper semantic analyses. The authors curate a high-quality dataset and systematically evaluate multiple LLMs with both direct prompting and retrieval-augmented approaches.
Summary of reasons to publish: 1. TROVE covers various domains, languages, document lengths, and multi-document settings, providing broad coverage for text provenance.

2. The classification of relationships goes beyond simpler tasks that often only check source presence, thereby enabling deeper semantic analysis.

3. The paper includes experiments with a range of open-source and closed-source LLMs, revealing how retrieval augmentation substantially improves provenance accuracy. The authors also highlight trade-offs in precision versus recall, model size, and direct vs. retrieval prompting.