Chunking-based Factual Error Correction with Part-of-Speech Prompting
Authors: Shuqin Jiang, Zhenyun Deng, Xiaofeng Zhu


Reviewer paper summaries: The paper presents an easy to use method for factual error correction (FEC). The method, called CPP, replaces the commonly used token-based text infilling module in FEC models with chunking-based infilling. This change aims to maintain faithfulness to the original claim. Additionally, CPP incorporates PoS tag information to minimize false positive tokens that may arise during correction. The authors benchmark CPP against various relevant baselines and evaluation metrics, demonstrating its effectiveness. Their ablation study further underscores the significance of each module. Although the paper is well-written and presents comprehensive experimentation and results, there are specific concerns highlighted in the weaknesses section. I look forward to the authors' responses to the queries raised in the following section.
Summary of strengths: 1. The introduced method for factual error correction is simple, easy to use, and gives notable gains over existing baselines.
2. The in-depth analysis, particularly in Section 5, and the comprehensive ablation studies effectively emphasize the importance of each sub-module within the method. These studies also substantiate the authors' rationale for introducing each component.
Summary of weaknesses: 1. While the paper compares CPP with domain specific baselines, it lacks comparison with current SOTA LLMs such as GPT, Calude, Llama, and Gemini series. Given their growing dominance, it would be valuable to understand their performance in FEC and see whether task-specific methods like CPP are necessary or not moving forward.
2. FactEdit proposed by Balachandran et al., 2022 (https://arxiv.org/pdf/2210.12378) and SciFix proposed by Ashok et al., 2023 (https://arxiv.org/pdf/2305.14707) also provide effective ways to tackle FEC. How does CPP fare against these methods?
3. The paper mentions the VENCE model by Chen et al., 2023 (https://arxiv.org/pdf/2211.12130) in the related works section but does not use it as a baseline for comparison. What is the rationale behind this decision?
Comments, suggestions and typos: **Typos and Questions:**

1. The citation for “Evidence-based factual error correction” appears twice in the references. I recommend using the official ACL citation.
2. Line 198 and 287 - ROUGH should be corrected to ROUGE.
3. The results of CPP and CPP$_{ALL}$ are pretty similar. Are these gains statistically significant? 
4. Can the authors also provide exact match scores for all the methods?

**Relevant Missing References:**

1. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual Error Correction for Abstractive Summarization Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258, Online. Association for Computational Linguistics. 
2. Hwanhee Lee, Cheoneum Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Juae Kim, and Kyomin Jung. 2022. Factual Error Correction for Abstractive Summaries Using Entity Retrieval. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 439–444, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. 
3. Vidhisha Balachandran, Hannaneh Hajishirzi, William Cohen, and Yulia Tsvetkov. 2022. Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9818–9830, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. 
4. Dhananjay Ashok, Atharva Kulkarni, Hai Pham, and Barnabas Poczos. 2023. The student becomes the master: Outperforming GPT3 on Scientific Factual Error Correction. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6762–6778, Singapore. Association for Computational Linguistics. 
5. Wang Y, Reddy RG, Mujahid ZM, Arora A, Rubashevskii A, Geng J, Afzal OM, Pan L, Borenstein N, Pillai A, Augenstein I. Factcheck-GPT: End-to-End Fine-Grained Document-Level Fact-Checking and Correction of LLM Output. arXiv preprint arXiv:2311.09000. 2023 Nov 15. 
6. He X, Zhang Q, Jin AL, Ma J, Yuan Y, Yiu SM. Improving Factual Error Correction by Learning to Inject Factual Errors. InProceedings of the AAAI Conference on Artificial Intelligence 2024 Mar 24 (Vol. 38, No. 16, pp. 18197-18205). 
7. Miaoran Li, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhu Zhang. 2024. Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 163–181, Mexico City, Mexico. Association for Computational Linguistics.
Overall assessment scores: 3.5