FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure

## Abstract

Abstract Accurate and transparent financial information disclosure is crucial in the fields of accounting and finance, ensuring market efficiency and investor confidence. Among many information disclosure platforms, the Chinese stock exchanges’ investor interactive platform provides a novel and interactive way for listed firms to disclose information of interest to investors through an online question-and-answer (Q&A) format. However, it is common for listed firms to respond to questions with limited or no substantive information, and automatically evaluating the quality of financial information disclosure on large amounts of Q&A pairs is challenging. This paper builds a benchmark FinTruthQA, that can evaluate advanced natural language processing (NLP) techniques for the automatic quality assessment of information disclosure in financial Q&A data. FinTruthQA comprises 6,000 real-world financial Q&A entries and each Q&A was manually annotated based on four conceptual dimensions of accounting: question identification, question relevance, answer readability, and answer relevance . We benchmarked various NLP techniques on FinTruthQA, including statistical machine learning models, pre-trained language model and their fine-tuned versions, as well as the large language model (LLM) GPT-4. Experiments showed that existing NLP models have strong predictive ability for question identification and question relevance tasks, but are suboptimal for answer readability and answer relevance tasks. By establishing this benchmark, we provide a robust foundation for the automatic evaluation of information disclosure, significantly enhancing the transparency and quality of financial reporting. FinTruthQA can be used by auditors, regulators, and financial analysts for real-time monitoring and data-driven decision-making, as well as by researchers for advanced studies in accounting and finance, ultimately fostering greater trust and efficiency in the financial markets.

## 1 Introduction

Within the financial market, enhancing transparency and timeliness in information disclosure can mitigate information friction, empower investors to make informed decisions, and foster market efficiency.
According to Lee and Zhong (), retail investors struggle to integrate the existing public information, and increased online platform activities are associated with more active market transactions, such as greater trading volume, return volatility, and market liquidity. The investor interactive platforms established by the Shanghai Stock Exchange (SSE)(^1^11[http://www.sse.com.cn/](http://www.sse.com.cn/)) and Shenzhen Stock Exchange (SZSE)(^2^22[https://www.szse.cn/index/index.html](https://www.szse.cn/index/index.html)) offer a unique avenue for retail investors to engage directly with listed firms. In contrast to conventional modes of disclosure, such as mandatory disclosures and traditional voluntary disclosures like conference calls, which follow specific procedures and formats, this format enables direct, interactive, near real-time communication between firms and ordinary investors through the format of Q&A. By analyzing the interactions on these platforms, researchers can gain insights into how effectively companies communicate with investors, the types of information that are most valued by investors, and how the disclosure practices of listed firms may impact their corresponding trading patterns in financial markets. This will enhance financial market regulation and provide deeper insights into corporate disclosure practices.

The interactive nature of these platforms provides a rich dataset for assessing the quality of information disclosure. However, the sheer volume of data and the diversity in the quality of questions and responses pose significant challenges for large-scale analysis. Leuz and Wysocki () summarize that the commonly used measures of disclosure in the financial sector primarily capture the quantity rather than the quality of the information provided. However, it is possible to construct the quality dimension of disclosure by focusing on specific disclosures, and Natural Language Processing (NLP) emerges as a promising solution to evaluate the quality of information disclosure at scale. Existing applications of NLP in automatic financial and accounting text analysis have demonstrated success in various domains, such as automatic analysis of annual financial reports(Smailović et al., ) and automatic sentiment analysis system on financial news articles(Van de Kauter et al., ). However, to the best of our knowledge, there is no standard dataset that evaluates the quality of financial information disclosure in these Q&A platforms.

In this work, we present a Q&A information disclosure quality assessment benchmark that evaluates Chinese Q&A data sourced from interactive platforms for communication between investors and listed companies, as established by SSE and SZSE. In developing this model, we have explored several technical challenges, including the interpretation of natural language, context sensitivity, and the subjective nature of relevance and readability assessments. This paper details the development, implementation, and evaluation of our NLP model, emphasizing its potential impact on the financial sector and the broader implications for automated quality assessment in investor communications. Datasets and code are available at [https://github.com/bethxx99/FinTruthQA](https://github.com/bethxx99/FinTruthQA).

Figure: Figure 1: Workflow of the financial Q&A quality evaluation process.
Refer to caption: x1.png

## 2 Related work

##### Disclosure measurement

The measurement issue pertaining to disclosures has been a persistent challenge in empirical accounting research(Leuz and Wysocki, ). Predominantly, comprehensive measures have focused on quantifying disclosures, encompassing criteria such as the presence or absence of quarterly and annual reports, as well as numerical counts of activities like conference calls and earnings forecasts(Lee and Zhong, ). Another commonly used measure is the Association for Investment Management and Research (AIMR) rankings, based on annual surveys of financial analysts who evaluate U.S. firms’ disclosure practices(Nagar et al., ; Healy et al., ; Welker, ). However, they are limited to large U.S. firms and cover only a specific time period. Furthermore, alternative measures designed for specific disclosures typically employ a keyword-based approach to analyze the tone and sentiment of well-established, traditional disclosures such as earnings forecasts and conference call transcripts(Bochkay et al., ; Frankel et al., ; Li, ).

##### Financial NLP

Financial documents, such as annual reports, financial news articles, and economic policy statements, are typically composed in technical language in a non-structured text format and possess unique features distinct from other domains.(Chang et al., ) Recent studies have focused on developing pre-trained models tailored for the financial domain (Araci, ; Yang et al., ; Liu et al., ), with a primary emphasis on the English language. These efforts are predominantly oriented towards sentiment classification tasks in the financial context.(Wan et al., ) Additional research in this field has concentrated on question-answering tasks, involving datasets that combine textual and tabular information, where numerical analysis is often essential to derive answers (Zhu et al., ; Zhao et al., ; Li et al., ). To our knowledge, there has been no prior research or dataset dedicated to the automatic quality assessment of financial Q&A data.

## 3 Dataset construction

### 3.1 Data collection

The data for this study was obtained from the interactive platforms for communication between investors and listed companies established by the Shanghai Stock Exchange and the Shenzhen Stock Exchange. We collected over 900,000 Q&A entries, among which we randomly selected 6,000 samples for manual annotation, with 3,000 entries from each exchange’s platform. The data collection process involved using web scraping techniques to extract the Q&A entries from the official platforms. The data was then manually reviewed and processed to ensure quality and consistency. The ethics statement and relevant discussion can be found in Supplement Section A.

Figure: Figure 2: Example of Q&A pair and quality evaluation criteria.
Refer to caption: x2.png

### 3.2 Data annotation

We focuses on four key information disclosure quality evaluation criteria: question identification, question relevance, answer relevance, and answer readability. These criteria are widely recognized as crucial indicators of information quality and are important for investors to consider when evaluating Q&A information. Specifically, the criteria in FinTruthQA are defined as follows:

- 1.
Question identification: This criterion assesses whether the question from an investor is a genuine question, as opposed to a suggestion or complaint. Each question is assigned a binary label: Positive, indicating the entry is a real question, or Negative, indicating the entry is not a real question. Data entries with questions that do not meet the definition of a real question are excluded from further evaluation based on other criteria.
- 2.
Question relevance: This criterion evaluates whether a question is pertinent to the investment topic, labeling questions as Positive if they are relevant and Negative if they are not. It assesses whether the question is related to the company’s shareholders, financial indicators, industry conditions, or other relevant investment topics.
- 3.
Answer readability: This criterion evaluates whether an answer is easy to understand and well-written. The readability score ranges from 1 to 4, with readability decreasing from 4 to 1. A score of 4 indicates that the answer is very readable, clear, and concise, free of ambiguity. A score of 3 indicates moderate readability with some issues. A score of 2 indicates significant readability issues. A score of 1 indicates the answer is unreadable, with major ambiguities, incoherent sentences, and content unrelated to the question.
- 4.
Answer relevance: This criterion evaluates whether an answer provides a relevant and informative response to the question asked. It assesses whether the answer provides enough detail and explanation to address the question. Answers are classified into three categories: 1, indicating the answer is completely unrelated to the question (none of the questions were answered); 2, indicating the answer is partially related (some questions were answered) or difficult to judge; and 3, indicating the answer is fully related to the question (all questions were answered).

The importance of these criteria stems from their collective ability to enhance the quality and reliability of information on investor communication platforms. Annotators with finance and accounting backgrounds were engaged to annotate the Q&A data. Figure 2 illustrates an example of the annotation. A detailed annotation process is listed in the Supplement Section B.

### 3.3 Quality control

For the answer relevance task, we observed that the raw annotations included noise due to the subjectivity of this task. Hence, we conducted an additional quality control step to correct the raw annotations. We implemented an enhanced correction process inspired by Confident Learning (Northcutt et al., ) principles. Sample data points demonstrating inaccuracies and implementation details are presented in Supplement Section C.

### 3.4 Data pre-processing

We found that the Shanghai Stock Exchange (SSE) data includes the company name and stock code at the beginning of the question texts, while the Shenzhen Stock Exchange (SZSE) data does not. We compared experiments on Q&A pairs with and without the company name as the prefix, and observed negligible differences. We removed these prefixes from the SSE data in our experiments.

### 3.5 Dataset statistics

The complete annotated dataset comprises 6,000 samples labeled for question identification, question relevance, answer relevance, and answer readability tasks, with each data point having four distinct labels corresponding to these tasks. The dataset was randomly divided into training, validation, and testing sets in a ratio of 4:1:1. The label distributions for each task are detailed in Tables 1. The provided statistics reflect the data after undergoing our quality control process. Note that our later analysis focuses exclusively on real questions. Only for the first task (question identification), did we utilize the complete set of 6,000 data samples. In the subsequent three tasks, we removed samples that were not questions. This decision was based on the rationale that non-question data could skew the model’s understanding and performance, especially for tasks like question relevance, where the context of a real question is crucial for accurate assessment. It is worth noticing that the labels of task question relevance are highly imbalanced, with only 6 and 7 negative labels in the validation and test dataset, respectively. We decided to keep this task as this reflects the real-world data distribution, and the concept of question relevance is important in financial information disclosure. Figure  3 shows the length distribution of questions and answers in characters of all the 6,000 QA pairs.

**Table 1: Dataset Statistic of Different Tasks.**
| Task | Label | Train | Validation | Test | Label_Num | Freq | Total |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Question identification | Negative | 327 | 90 | 81 | 498 | 8.30% | 6000 |
|  | Positive | 3673 | 910 | 919 | 5502 | 91.70% |  |
| Question relevance | Negative | 714 | 6 | 7 | 27 | 0.49% | 5502 |
|  | Positive | 3654 | 911 | 910 | 5475 | 99.51% |  |
| Answer readability | 1 | 106 | 190 | 198 | 1094 | 19.88% | 5502 |
|  | 2 | 26 | 3 | 6 | 35 | 0.64% |  |
|  | 3 | 68 | 19 | 20 | 107 | 1.94% |  |
|  | 4 | 2868 | 705 | 693 | 4266 | 77.54% |  |
| Answer relevance | 1 | 731 | 194 | 206 | 1131 | 20.63% | 5502 |
|  | 2 | 533 | 136 | 130 | 779 | 14.21% |  |
|  | 3 | 2404 | 587 | 581 | 3572 | 65.16% |  |

Figure: Figure 3: Length distributions of questions and answers (in characters).
Refer to caption: extracted/5665297/figs/ques.png

## 4 Benchmarks

### 4.1 Model selection

To fully evaluate the performance of different methods in FinTruthQA, we benchmark several models that cover Machine Learning (ML)-based, Pre-trained Language Model (PLM)-based, PLM with continued Pre-Taining (CPT)-based, and LLM-based.
Given the distinct nature and varying difficulty of different tasks, we applied different approaches accordingly.

ML-based.
We modeled these tasks as text classification and employed various advanced machine learning models, including Logistic Regression (LR)(Hosmer Jr et al., ), Support Vector Machine (SVM)(Hearst et al., ), K-nearest neighbor (KNN)(Peterson, ), and Random Forest (RF)(Breiman, ).
For semantic feature extraction, we first utilized the Jieba toolkit (^3^33[https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)) for Chinese word segmentation and then extracted the weighted representation of the input text using TF-IDF Sparck Jones ().
To achieve optimal performance, we used cross-validation and GridSearch to rigorously explore the best hyperparameters. Additionally, the best-performing classifiers and hyperparameters varied across tasks, reflecting the distinct nature of each task. Consequently, we conducted task-specific hyperparameter searches for each task respectively.

PLM-based.
We first investigated BERT and its various variants for the tasks of answer relevance and answer readability, as these are more challenging compared to the first two tasks.
Then, we explored BERT-based models specifically trained on financial text data, such as financial news, research reports, and financial encyclopedia entries (FinBERT,(^4^44[https://github.com/valuesimplex/FinBERT](https://github.com/valuesimplex/FinBERT)) Mengzi-fin (Zhang et al., ), and DKPLM (Zhang et al., )).
Furthermore, we examined whether fine-tuned PLMs on specific tasks, such as NLI (Natural Language Inference) and QA (Question Answering), would offer any benefits for these tasks (SBERT-nli (Reimers and Gurevych, ) and RoBERTa-extractive-qa (Liu et al., )).
Finally, we also examined the performance of a large BERT model (BERT (Large)).(^5^55[https://huggingface.co/yechen/bert-large-chinese](https://huggingface.co/yechen/bert-large-chinese))
Notably, unless otherwise specified, all PLMs employed in this study are the base and Chinese versions. Links for these PLMs can be found in Supplement Section D.

PLM with CPT-based.
Given the prevalence of abbreviations and specialized terms in financial texts, general domain PLMs often struggle with comprehension, especially for the challenging task of answer relevance.
To alleviate this issue, we curated a substantial volume of domain-specific texts and conducted continued pre-training on general PLMs.
Specifically, we collected over 900,000 pairs of Q&A from the SSE and SZSE platforms by web scraping. High-performing models from previous experiments were then selected for continued pretraining via masked language modeling.
This pre-training on a large-scale, domain-specific dataset enhances the PLM’s ability to understand the financial context, capture more specialized information, and improve performance in subsequent downstream tasks.

LLM-based.
We further evaluate advanced LLMs in the most challenging task of answer relevance.
Different from previous methods, the LLM-based method leverages LLMs’ in-context learning ability to perform the task with zero-shot instead of specific fine-tuning.
We select the most advanced LLM (GPT-4) and instruct it with task description and evaluation criteria, requiring the model to directly output relevance score (Direct score). Moreover, we incorporate the Chain of Thought (CoT) (Wei et al., ) technique, which instructs the GPT-4 to provide specific reasoning along with the relevance scores (Score with CoT). Prompt and details are listed in Supplement Section G.

### 4.2 Settings and metrics

Implementation details. For ML-based models, we utilized the standard scikit-learn(^6^66[https://scikit-learn.org/stable/](https://scikit-learn.org/stable/)) library. For all PLM-based methods, we uniformly used PyTorch and HuggingFace’s framework(^7^77[https://github.com/HuggingFace](https://github.com/HuggingFace)) for downstream fine-tuning and evaluation. The hyperparameter settings of our experiments using PLM-based model can be found in Supplement Section E.
The tasks of question identification and question relevance encode only the questions as input, while answer readability and answer relevance required encoding both the question and the answer. The sequence starts with a special classification token [CLS], and the question and answer are separated by a special token [SEP].
During the continued pre-training, we added two special tokens, [Q] and [A], into the vocabulary and prefixed them to the question and answer, respectively. This helps the PLMs better identify the components of each sentence. The continued pre-training was performed based on the UER framework (Zhao et al., ).
All tests of GPT-4 were conducted by calling OpenAI’s API with version GPT-4-0613.

##### Metrics

For the first two binary classification tasks, we used accuracy, precision, recall, and F1-score metrics.
For the last two multi-class classification tasks, we calculated both micro and macro F1-scores, along with an additional metric: Quadratic Weighted Kappa (QWK) (Dong et al., ).

### 4.3 Results and analysis

Tables  2,  3, and  4 present the performance of different methods in FinTruthQA. Experimental results reported with error bars can be found in Supplement Section F.

#### 4.3.1 Task 1: Question identification

**Table 2: Model evaluation results of question identification and question relevance task.**
|  |  | Question identification | Question relevance |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Model | Accuracy | F1 | Precision | Recall | Accuracy | F1 | Precision | Recall |
| ML-based | TF-IDF + LR | 0.9410 | 0.9688 | 0.9452 | 0.9935 | 0.9924 | 0.9962 | 0.9924 | 1.0000 |
| TF-IDF + RF | 0.9370 | 0.9667 | 0.9423 | 0.9924 | 0.9924 | 0.9962 | 0.9924 | 1.0000 |  |
| TF-IDF + SVM | 0.9390 | 0.9678 | 0.9424 | 0.9946 | 0.9924 | 0.9962 | 0.9924 | 1.0000 |  |
| TF-IDF + KNN | 0.9330 | 0.9645 | 0.9411 | 0.9891 | 0.9924 | 0.9962 | 0.9924 | 1.0000 |  |
| PLM-based | BERT | 0.9617 | 0.9795 | 0.9735 | 0.9857 | 0.9983 | 0.9992 | 0.9983 | 1.0000 |

**Table 3: Model evaluation results of answer readability task.**
| Method | Model | F1 (Micro) | F1 (Macro) | F1 (Weighted) | QWK |
| --- | --- | --- | --- | --- | --- |
| ML-based | TF-IDF + LR | 0.8135 | 0.3510 | 0.7878 | 0.4152 |
| TF-IDF + RandomForest | 0.7950 | 0.3156 | 0.7540 | 0.2912 |  |
| TF-IDF + SVM | 0.8113 | 0.3362 | 0.7763 | 0.3669 |  |
| TF-IDF + KNN | 0.7852 | 0.3015 | 0.7395 | 0.2418 |  |
| PLM-based | BERT | 0.8550 | 0.4203 | 0.8429 | 0.6293 |
| RoBERTa | 0.8539 | 0.4035 | 0.8405 | 0.6267 |  |
| FinBERT | 0.8790 | 0.4189 | 0.8629 | 0.6889 |  |
| Mengzi-fin | 0.8648 | 0.4109 | 0.8495 | 0.6512 |  |
| DKPLM | 0.8713 | 0.4355 | 0.8572 | 0.6666 |  |
| SBERT-nli | 0.8451 | 0.3914 | 0.8274 | 0.5776 |  |
| RoBERTa-extractive-qa | 0.8648 | 0.4109 | 0.8495 | 0.6512 |  |
| BERT (Large) | 0.8582 | 0.4035 | 0.8407 | 0.6209 |  |
|  | RoBERTa (Large) | 0.8571 | 0.4064 | 0.8441 | 0.6387 |

Table 2 shows the results for task question identification. In this task, we evaluated the performance of various models of ML-based and PLM-based. Among the ML-based methods, the LR classifier showed the best performance with an accuracy of 94.10% and an F1 score of 96.88%. Meanwhile, the BERT outperformed all ML-based models, achieving an accuracy of 96.17%, an F1 score of 97.95%, a precision of 97.35%, and a recall of 98.57%.
The results suggest that it not only correctly identifies a higher proportion of real questions (high recall) but also maintains a high level of correctness in its positive predictions (high precision). This balanced performance in precision and recall underscores the model’s capability to handle the nuances in question identification effectively.
This may be attributed to the PLM’s superior ability to encode text compared to TF-IDF.

#### 4.3.2 Task 2: Question relevance

The evaluation results for the task question relevance are detailed in Table 2. The PLM-based model outperformed the ML-based models across all metrics, achieving an accuracy of 99.83%, an F1 score of 99.92%, a precision of 99.83%, and a recall of 100%. This indicates that the model effectively identifies relevant questions while also correctly classifying the few negative samples, showcasing a balanced performance. In contrast, the TF-IDF-based models all demonstrated identical performance metrics and tended to predict all samples as positive. This is because the label distribution for this task is highly skewed (shown in Table 1), with 910 positive samples and only 7 negative samples in the test set. This imbalance poses a significant challenge for accurate classification, as models can easily achieve high recall by predicting all samples as positive, but this approach would fail to identify the few negative samples correctly.

#### 4.3.3 Task 3: Answer readability

The evaluation results for the task answer readability are shown in Table 3. Among the ML-based models, the LR classifier exhibited the highest performance, achieving an F1 Micro score of 81.35% and a QWK of 0.4152. While LR provided reasonable performance, the relatively low F1 Macro score highlights difficulties in handling class imbalance effectively. In contrast, the PLM-based models demonstrated a substantial improvement over the ML-based models. The BERT achieved an F1 Micro score of 85.50% and a QWK of 0.6293, showcasing its ability to manage the complexity and variability in readability assessment.
FinBERT outperformed all other models with an F1 Micro score of 87.90%, an F1 Macro score of 41.89%, an F1 Weighted score of 86.29%, and the highest QWK of 0.6889. FinBERT’s pre-training on financial text gave it a distinct advantage in this domain-specific task, allowing it to achieve the highest accuracy and alignment with human judgments. We also visualize the confusion matrix for our best-performing model (FinBERT) in this task in Figure 4(a). While the model performs well in identifying answers with the best readability, it needs significant improvement in distinguishing lower readability levels.

#### 4.3.4 Task 4: Answer relevance

**Table 4: Model evaluation results of answer relevance task.**
| Method | Model | F1 (Micro) | F1 (Macro) | F1 (Weighted) | QWK |
| --- | --- | --- | --- | --- | --- |
| ML-based | TF-IDF + LR | 0.7001 | 0.5032 | 0.6549 | 0.4117 |
| TF-IDF + RF | 0.6979 | 0.4366 | 0.6241 | 0.3627 |  |
| TF-IDF + SVM | 0.7056 | 0.4496 | 0.6339 | 0.3906 |  |
| TF-IDF + KNN | 0.6521 | 0.3863 | 0.5787 | 0.2322 |  |
| PLM-based | BERT | 0.7797 | 0.6638 | 0.7638 | 0.6479 |
| RoBERTa | 0.7710 | 0.6503 | 0.7534 | 0.6177 |  |
| FinBERT | 0.7688 | 0.6710 | 0.7623 | 0.6496 |  |
| Mengzi-fin | 0.7710 | 0.6705 | 0.7635 | 0.6484 |  |
| DKPLM | 0.7612 | 0.6540 | 0.7541 | 0.6375 |  |
| SBERT-nli | 0.7426 | 0.5790 | 0.7086 | 0.5258 |  |
| RoBERTa-extractive-qa | 0.7841 | 0.6795 | 0.7732 | 0.6641 |  |
| BERT (Large) | 0.7721 | 0.6800 | 0.7664 | 0.6331 |  |
| RoBERTa (Large) | 0.7841 | 0.6916 | 0.7772 | 0.6565 |  |
| PLM with CPT-based | FinBERT | 0.8026 | 0.6958 | 0.7886 | 0.6951 |
| RoBERTa-extractive-qa | 0.7786 | 0.6626 | 0.7662 | 0.6619 |  |
| BERT (Large) | 0.7722 | 0.6756 | 0.7639 | 0.6212 |  |
| RoBERTa (Large) | 0.7328 | 0.6195 | 0.7214 | 0.5372 |  |
| LLM-based | Direct score (GPT-4) | 0.6739 | 0.6311 | 0.7014 | 0.5906 |
| Score with CoT (GPT-4) | 0.5976 | 0.5845 | 0.6428 | 0.5751 |  |

Table 4 shows the experiment results on task answer relevance. TF-IDF + SVM performs the best among ML-based models, with an F1 Micro score of 70.56% and a QWK of 0.3906. Regarding PLM-based models, RoBERTa-extractive-qa stood out with an F1 Micro score of 78.41% and a QWK of 0.6641, demonstrating significantly better performance than ML-based models. This can be attributed to its pretraining on QA datasets, which likely enhanced its ability to understand and evaluate answer relevance effectively. According to the performance of the PLMs with continued pretraining presented in Table 4, further pertaining brings the most improvement on FinBERT, which was initially trained on financial corpora, achieving a Micro F1 score of 80.26% and 0.6951 QWK score. On the other hand, continued pretrained does not improve model performance on several other PLMs (e.g. RoBERTa-extractive-qa, BERT large, RoBERTa large). The results could be attributed to the fact that FinBERT’s initial training presumably provided it with a foundational understanding of the financial lexicon and contexts, which was further enriched through pre-training. Other models might suffer from a misalignment between the model’s learned representations and the domain-specific requirements. The results of GPT-4 with and without CoT are presented at the bottom of Table 4, GPT-4 models are largely behind PLM-based models, highlighting the challenge of state-of-the-art LLM on Chinese financial information disclosure understanding. Direct GPT-4 scoring yielded better results compared to GPT-4 with CoT. A potential reason is that GPT-4 was primarily trained on English corpora; it lacks sufficient financial domain knowledge in Chinese. This may result in knowledge gaps and hallucinations during the CoT process, thereby reducing the model’s performance. Another study also observed a similar phenomenon where CoT failed in healthcare domain prediction with Chinese text. (Liu et al., ) We anticipate that further domain adaptation methods, such as few-shot learning or retrieval-augmented generation (Lewis et al., ), may improve the performance of LLMs on this task. Figure 4(b) shows the confusion matrix for the best-performing model (FinBERT with continued pretraining). It demonstrates that the model effectively captures answers that are completely related to the questions (labeled as 3). However, the model struggles to distinguish partially unrelated answers (labeled as 2) from completely related ones, where the nuanced differences are subtle and challenging.

Figure: (a) FinBERT for answer readability task.
Refer to caption: extracted/5665297/figs/conf3.png

## 5 Limitations

There are several limitations in this study. First, the Chinese cultural and regulatory context of SSE and SZSE may limit the generalizability of the findings to other markets. Future research is needed to explore the applicability of the developed models and methodologies in different contexts and validate the findings across diverse settings. Second, the annotation process inherently involves a degree of subjectivity, as differences in annotators’ interpretations can introduce variability in the labeled data. To mitigate this, we conducted multiple rounds of pre-annotation and formal annotation with professional annotators, along with thorough error analysis and expert reviews to ensure consistency. Third, while precision, recall, F1-score, and quadratic weighted kappa (QWK) are standard metrics for evaluating model performance, they may not fully capture the qualitative aspects of information disclosure quality. The reliance on these metrics could lead to an overemphasis on quantitative evaluation while neglecting interpretability. Lastly, the task of question relevance is highly imbalanced, and models on this task are not well compared. We decide to keep this task in our benchmark to reflect the real-world data distribution.

## 6 Conclusion

In this study, we introduced FinTruthQA, a benchmark designed to evaluate advanced NLP techniques for the automatic quality assessment of information disclosure in financial Q&A data. We created a dataset comprising 6,000 real-world financial Q&A entries, each manually annotated based on four conceptual dimensions of accounting. Experiments revealed the effectiveness of pre-trained BERT models in tackling these tasks. In particular, we observed that models pre-trained on financial corpora, as well as those designed for related tasks such as QA, outperformed other variants of BERT and GPT-4. This emphasized the importance of domain-specific and task-oriented training in improving performance. Although existing NLP models showed reasonable performance on this benchmark, there is still significant room for improvement, especially in the task of answer relevance, which calls for better NLP models. This benchmark can serve as a new text understanding for evaluating LLMs and it provides a novel, valuable, and practical sandbox for improving financial information disclosure with AI and data-driven approaches.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] See Section 5. Limitation.
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Supplement Section A.
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] More discussion can be found in Supplement Section A.
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[N/A] This study aims for the construction and evaluation of a new benchmark. No theoretical assumptions.
(b)
Did you include complete proofs of all theoretical results?
[N/A] This study aims for the construction and evaluation of a new benchmark. No theoretical assumptions.
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] All the datasets, benchmarks and code are available at [https://github.com/bethxx99/FinTruthQA](https://github.com/bethxx99/FinTruthQA)
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 3.5 for data splits and Supplement Section E for hyperparameter choices.
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The detailed results can be found in the supplementary material.
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Supplement Section E.
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes] We used code from several models in our benchmarks, all the sources were properly cited in this paper.
(b)
Did you mention the license of the assets?
[Yes] Our code and data are open access and license information is listed in GitHub.
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Supplementary materials.
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] This data used in this study are open-available.
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Supplement Section A.
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] This study does not involve crowdsourcing and human subjects.
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] This study does not involve crowdsourcing and human subjects.
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] This study does not involve crowdsourcing and human subjects.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See weakness
soundness: 2
strengths: 1. Novel Benchmark Dataset: The paper introduces FinTruthQA, a unique and practical dataset for evaluating the quality of financial information disclosure in Q&A format on investor platforms. This dataset fills a gap in financial NLP, providing a structured way to evaluate both question and answer quality, which is valuable for real-world applications in finance.

2. Comprehensive Model Evaluation: The authors benchmark a range of NLP techniques, from traditional ML-based models to pre-trained language models (PLMs) and LLMs, demonstrating the utility and limitations of each in assessing Q&A quality.
summary: This paper presents a dataset designed to assess the quality of responses on Chinese stock exchange investor platforms, specifically the Shanghai and Shenzhen exchanges. The FinTruthQA dataset includes 6,000 Q&A entries, each manually annotated across four evaluation criteria: question identification, question relevance, answer readability, and answer relevance. The authors benchmark various NLP models, including machine learning methods, pre-trained language models, and large language models, demonstrating that while models perform well on question-related tasks, challenges persist in evaluating answer readability and relevance.
weaknesses: 1. Limited Evaluation Settings: The evaluation framework is somewhat restrictive, as only ML-based and PLM-based models are assessed across most tasks, while LLMs are evaluated solely on the answer relevance task in a zero-shot setting. A comprehensive comparison across all tasks is necessary to fully understand the strengths and limitations of ML-based models, PLMs, and LLMs. Moreover, applying only zero-shot results for LLMs on answer relevance is inconsistent with the supervised fine-tuning used for ML and PLM models, which could skew the comparative assessment.

2. Copyright and Data Usage Concerns: The paper lacks clarity regarding copyright compliance for the data sources. It is unclear if legal permissions have been obtained to use and openly release the Q&A entries from stock exchange platforms for research purposes. Explicit details on copyright considerations are essential to ensure the ethical and legal viability of this dataset for the broader research community.

3. Exclusion of Financial LLMs: While the paper evaluates a range of models, it does not include financial domain-specific LLMs, which are highly relevant for the specialized nature of this task. Incorporating financial LLMs would provide valuable insights into the performance gains from domain-specific pre-training and would create a more representative assessment of available model options.

4. Class Imbalance in Dataset: The dataset suffers from significant class imbalance, such as in tasks like question identification, where the majority of samples are labeled as positive.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: See in Weaknesses.

In addition, the authors still use traditional metrics when evaluating. I'm not sure if this is entirely appropriate for evaluation in the financial sector.
soundness: 3
strengths: 1. This paper targets a real-world problem in financial markets about how to improve transparency and quality of financial reporting. 

2. The annotation process is comprehensive in that the annotation team takes multiple rounds and quality control, comprehensive evaluation using multiple metrics, and documentation of the data collection and processing steps. 

3. This paper evaluated multiple types of models (ML, PLM, LLM), including domain-specific models, and continued pre-training models. The author also showed a detailed error analysis and performance comparisons. 

4. The experiment results demonstrated that domain-specific models, FinBERT with continued pre-training, perform better on complex tasks. This insight is useful to the domain.
summary: The authors discuss the relationship between the quality of disclosure in financial Q&A data and the ability of investors to make informed decisions. In this paper, the author created a new benchmark dataset (FinTruthQA) containing 6,000 manually annotated financial Q&A pairs from Chinese stock platforms. The author benchmarked various approaches including ML-based models, pre-trained language models, and large language models.
weaknesses: 1. The FinTruthQA dataset is exclusively from Chinese markets. In addition, the proposed dataset focuses only on the Q&A format, excluding other types of financial disclosure (like, long reports, and analytic reports).

2. In question relevance, the label is imbalanced (99.51% positive), and limited number of samples for some categories. This may affect the validity and generalizability of the benchmark results.

3. Limited overall technical contribution to financial Q&A or general domain Q&A.

4. Experiment detail is not comprehensive, such as no details such as temperature were reported.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: None
soundness: 3
strengths: 1. Dataset: The paper introduces a  dataset, FinTruthQA, specifically tailored for assessing financial information disclosure quality, covering critical criteria such as question relevance and answer readability. 

2. Real-world Applicability: The dataset and methods are designed with practical applications in mind, aiding auditors, analysts, and regulators in real-time monitoring and enhancing transparency in financial disclosures.
summary: The paper introduces a dataset designed to assess the quality of financial information disclosure using natural language processing (NLP) techniques. This dataset, FinTruthQA, consists of 6,000 annotated Q&A entries from Chinese stock exchanges, focusing on four evaluation criteria: question identification, question relevance, answer readability, and answer relevance. The authors benchmarked various language models, finding that while models perform well in identifying and categorizing questions, they struggle with answer readability and relevance.
weaknesses: 1. Title Discrepancy: The title of the paper suggests a focus on the truthfulness of the responses within the dataset. However, the actual content primarily addresses other aspects such as readability and relevance. A title that more accurately reflects the paper's content could improve its alignment with the reader's expectations. 

2. Baseline Methodologies for Readability: The paper could be strengthened by including more established baselines for evaluating readability, such as the Gunning fog index. For a comprehensive view on this, the authors might consider reviewing literature from targeted workshops like the TSAR workshop on readability (https://tsar-workshop.github.io/). 

3. Label Distribution and Utility: The current label selection, particularly for answer readability, seems suboptimal. Labels 2 and 3 are underrepresented, appearing in less than 3% of the QA pairs. This distribution is problematic, as evidenced by the classification challenges shown in Figure 4, where Label 4 is frequently misclassified as Label 1, but rarely as Labels 2 or 3. Refining these categories could enhance the model's learning and classification accuracy. 

4. Definition of Readability: The paper's definition of readability focuses on ease of understanding and quality of writing. However, readability could be more effectively assessed by considering additional components such as structure, clarity, style, and grammar. A more detailed framework for readability would likely improve the assessment’s depth and utility.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See above
soundness: 3
strengths: The authors address the crucial challenge of evaluating the quality of financial disclosures on platforms where companies communicate with investors. The interactive nature of these disclosures and their unstructured format make this a valuable problem.

By focusing on investor-platform Q&A interactions, the paper explores a unique and under-researched form of financial communication that differs from more structured formats like reports or calls.

The experiments cover a wide range of ML and PLM-based models, and the paper offers insights into their performance, especially emphasizing the limitations of current models in answer readability and relevance tasks. This provides a strong foundation for future improvements.
summary: The paper introduces FinTruthQA, a new benchmark dataset developed to evaluate the quality of financial information disclosures on interactive platforms for investors in Chinese stock exchanges (Shanghai Stock Exchange and Shenzhen Stock Exchange). The dataset comprises 6,000 real-world Q&A entries and assesses disclosure quality across four key criteria: question identification, question relevance, answer readability, and answer relevance. The authors benchmark various machine learning (ML) and pre-trained language models (PLMs) and examine their effectiveness on these tasks.
weaknesses: Although the dataset focuses on investor Q&A in the Chinese stock exchange context, the lack of testing across different regulatory and cultural contexts limits its generalizability. Without broader applicability, the impact of FinTruthQA as a global benchmark remains restricted.

The reliance on human annotation for subjective criteria (e.g., answer readability and relevance) introduces potential biases. While the authors conducted quality control, it is unclear how consistently these subjective labels would hold across diverse annotator groups or application settings.

Although the paper provides an overview of performance results, it lacks an in-depth analysis of errors made by models. Understanding specific failure cases would offer valuable insights into the dataset’s complexity and guide future model improvements.

Many releted papers are not refered, such as 
The FinBen: An Holistic Financial Benchmark for Large Language Models
FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models
Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency
FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Making
Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Meta-review: This paper presents a new dataset for assessing the quality of responses on Chinese stock exchange investor platforms. In particular, the proposed FinTruthQA dataset includes 6,000 Q&A entries, each manually annotated across four evaluation criteria: question identification, question relevance, answer readability, and answer relevance. Although the new dataset would make a very meaningful contribution to the research community, reviewers found that there still major limitations in the current version, such as limited evaluation settings, definition of readability, label distribution, lack of in-depth analysis, etc. Although some of these issues have been addressed by the authors' rebuttal, during the post-rebuttal discussions, reviewers believed that some remaining issues (such as the class imbalance issue) cannot be addressed by a minor revision. Therefore, this paper in its current version is not ready for publication at ICLR.

Decision: Reject