OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs
Authors: Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi Georgiev, Jiahui Geng, Preslav Nakov


Reviewer paper summaries: Proposed OpenFactCheck, a unified factuality evaluation framework for LLMs consisting of three modules: (i) CUSTCHECKER: allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims, (ii) LLMEVAL, a unified evaluation framework that assesses LLM’s factuality ability from various perspectives, and (iii) CHECKEREVAL is an extensible solution for gauging the reliability of automatic fact-checkers verification results using human-annotated datasets.
Summary of strengths: Since different papers use disparate evaluation benchmarks and measurements, this paper introduces a unified factuality evaluation framework for LLMs.
Summary of weaknesses: 1. The scientific novelty of this work is very limited. Some existing methods are combined without proposing any new solutions.
2. The reproducibility is a concern: how was the response for GPT-4 and LLaMA-2 collected? What were the prompts, and decoding parameters?
3. The evaluation is also not comprehensive. Since this work lacks scientific novelty, should at least contain extensive experiments to validate various components of the proposed method. However, the evaluation is only limited to GPT-4 and LLaMA-2. 
4. Moreover, the paper is quite difficult to follow. The presentation should be further improved with better coherence. At this moment, the proposed system seems quite complicated.
Comments, suggestions and typos: I believe this paper lacks scientific contribution and does not fit well with the research track. It may probably fit better if submitted to the System Demonstration track.
Overall assessment scores: 2.5