A Compare-and-contrast Multistage Pipeline for Uncovering Financial Signals in Financial Reports
Year: 2023

Abstract:
In this paper, we address the challenge of discovering financial signals in narrative financial reports. As these documents are often lengthy and tend to blend routine information with new information, it is challenging for professionals to discern critical financial signals. To this end, we leverage the inherent nature of the year-to-year structure of reports to define a novel signal-highlighting task; more importantly, we propose a compare-andcontrast multistage pipeline that recognizes different relationships between the reports and locates relevant rationales for these relationships. We also create and publicly release a humanannotated dataset for our task. Our experiments on the dataset validate the effectiveness of our pipeline, and we provide detailed analyses and ablation studies to support our findings.

heading: 1 Introduction
text: With the rapid growth of information, many tasks in the field of natural language processing (NLP) involve streamlining information comprehension. One such task is summarization, which selects a subset of sentences or generates new content that best represents the given document (Hermann et al., 2015; See et al., 2017; Cohan et al., 2018). This task helps humans save time and effort by identifying important information in a text. In the finance context, comprehending regulatory narrative reports is a classic example of efficiently mining signals from a large amount of text. As these reports often contain rich information concerning specific financial entities, discovering valuable insights is crucial for academia and the finance industry.
Much research has shown that textual features from financial reports contain valuable financial signals about future firm performance and market reactions (e.g., Badertscher et al., 2018; Ertugrul et al., 2017; You and Zhang, 2009). However, authorities such as the Securities and Exchange Commission
(SEC) require that companies provide comprehensive and detailed information about their current status in these reports, which often contain much unimportant and already-known information. For example, the token overlap ratio between annual 10-K reports of the same company between adjacent years is often high,1 making it a challenging and tedious task to acquire important signals in new reports (termed as the overlapping characteristic hereafter).
Recent advances in NLP technology have included attempts to efficiently and effectively comprehend lengthy financial documents. One approach to address this problem is through summarization (e.g., Zmandar et al., 2021b; Orzhenovskii, 2021; Gokhan et al., 2021). Other approaches additionally leverage numerical metrics, such as stock return volatility and abnormal trading volumes, to locate essential financial signals in reports (e.g., Kogan et al., 2009; Tsai and Wang, 2017; Rekabsaz et al., 2017; Agrawal et al., 2021; Lin et al., 2021). However, these approaches often require high-quality human annotation or suitable financial measures, which poses significant limitations in practical scenarios.
In this study, we approach financial report comprehension from a novel perspective by leveraging the intrinsic year-to-year characteristic of reports (i.e., the overlapping characteristics). Specifically, for a particular company, we use the document published in the previous year as an information anchor (i.e., the reference) to construct a year-to-year structure and locate important financial signals in the report of the subsequent year (i.e., the target). This inherent structure enables us to mine financial signals in a compare-and-contrast self-supervised manner, compared to existing supervised approaches.
Based on the year-to-year structure, we propose
1The overlap ratio calculated from Item 7 of the reports of the 3,849 companies from 2011 to 2018 (see FINAL dataset in Section 4) is around 0.826 on average.
14307
a compare-and-contrast multistage pipeline to effectively locate financial signals in reports. We first identify a few types of relationships between reference and target financial reports at the segment level. Then, using these recognized relationships, we present a novel financial signal-highlighting task together with a domain-adaptive highlighting model. The goal of this task is to identify the rationales, represented by the importance of certain words, for a specific pair of year-to-year segments. Therefore, the words with high importance are deemed to be crucial financial signals in these reports. For experiments, we present a synthetic dataset consisting of 30,400 reference-to-target segment pairs for financial signal highlighting.2 Experimental results validate the effectiveness of the proposed pipeline; detailed analyses and ablation studies are also provided.

heading: 2 Problem Definition
text: The year-to-year nature of financial reports allows us to take advantage of the differences between a company’s documents in consecutive years. These differences may reveal complex but insightful relationships within a pair of documents. To better understand these relationships, we investigate them through rationales (represented by the word importance), which are considered essential signals in financial reports.

heading: 2.1 Reference-to-target Structure
text: Formally, for each company, Dℓ is a set containing all segments in its financial report at year ℓ, where each element d ∈ Dℓ refers to a single segment. While we regard a focal company’s financial report at year ℓ, Dℓ, as the target document, we view the same company’s report at year ℓ− 1, Dℓ−1, as the reference document. Given the annual nature (i.e., the reference-to-target structure) of financial reports, we further break down the document-todocument relationship between Dℓ and Dℓ−1 into enumerated segment-to-segment relationships. We denote the set of enumerated segment pairs as T̄ .3
However, as T̄ includes all pairs of segments enumerated from Dℓ, and Dℓ−1 (i.e., |Dℓ||Dℓ−1| pairs), intuitively, most segment pairs in T̄ have no interesting relationship. Hence, we reduce the
2The dataset and codes are available at https:// github.com/cnclabs/codes.fin.highlight.
3Note that each (Dℓ,Dℓ−1) pair corresponds to a set of segment pairs T̄ ; to simplify the notation, we do not use the subscript for T̄ to characterize the different sets.
set T̄ to T by removing irrelevant segment pairs based on their syntactical similarities. Specifically, for each target segment t ∈ Dℓ, we calculate the ROUGE-2 (Lin, 2004) scores between the target segment t and all reference segments r ∈ Dℓ−1 and sort the reference segments according to their scores in descending order as S̄(t) = ( r1, r2, . . . , r|Dℓ−1| ) .4 With S̄(t), we then discard reference segments that fall behind the largest ROUGE-2 difference out of all possible ROUGE-2 differences, resulting in a truncated set S(t).5 Note that the difference is calculated between the two consecutive ROUGE-2 scores in S̄(t). Finally, with S(t), the reduced segment pair set is T = {(r, t)|(r, t) ∈ T̄ ∧ r ∈ S(t)}.
To locate meaningful financial signals revealed by segment pair differences, we further classify each pair (r, t) ∈ T into the following two sets:
1. T β contains reference-to-target segment pairs with largely similar meanings (see Table 1(a)). Generally, there is no additionally noteworthy content in target segment t compared to reference segment r.
2. T α = T \ T β contains segment pairs with dissimilar meanings (see Table 1(b)). Pairs in T α are further classified into two types based on their syntactic and semantic similarity, as discussed in Section 3.2.
Note that all the aforementioned terminologies and notations can also be found in Figure 1; they will be used throughout the following sections of this paper.
4The round parentheses represent the ordered set. 5Empirically, there are often one to five remaining refer-
ence segments in the truncated set for a target segment t.

heading: 2.2 Highlighting Task
text: We consider pairs in T α as the pairs of interest and provide rationales of underlying pairwise relationships by predicting the word importance for each segment pair (r, t) ∈ T α as
R =∆ Pf (t|r), (1)
where R indicates the word importance of a target segment t conditioned on reference segment r, and the highlighting model is denoted as f (detailed in Sections 3.3 and 3.4).

heading: 3 Proposed Pipeline
text: Here we describe the proposed multistage pipeline for discovering the rationale behind the referenceto-target structure in financial reports, as illustrated in Figure 1.
3.1 S0: Document Segmentation
Financial reports are multimodal, often covering multiple aspects and topics; each aspect or topic usually uses one to three consecutive sentences to convey its meaning. Therefore, instead of considering sentences as the basic unit of text, we here regard uni-modal segments as the smallest unit for financial documents. We first use spaCy API for sentence segmentation.6 Then, we utilize the finetuned cross-segment BERT (Lukasik et al., 2020) to obtain coherent uni-modal segments. Note that some studies show that breaking a document into uni-modal segments benefits downstream applications (Shtekh et al., 2018; Qiu et al., 2022; Chivers et al., 2022).
6https://spacy.io/api/sentencizer
3.2 S1: Relation Recognition In this stage, a systematic procedure manages relation types T β and T α with semantic and syntactic similarity. Specifically, we use two functions, ROUGE-2 and Sentence-BERT (Reimers and Gurevych, 2019) cosine similarity,7 to assess the syntactic and semantic similarity between each reference-to-target pair (r, t) ∈ T .8 The scores for the syntactic and semantic similarity are denoted as ϕsyn(r, t) and ϕsem(r, t), respectively.9 We empirically design a rule-based procedure and classify each segment pair into three types.
1. Insignificant relations (T β) correspond to uninformative segment pairs with highly similar syntactic and semantic meanings between target and reference segment (i.e., ϕsyn > ϵsyn and ϕsem > ϵsem).
2. Revised relations (T α1 ) correspond to segment pairs that differ in some words only but disclose quite different meanings, resulting in a high ϕsyn(r, t) but a relatively low ϕsem(r, t) (i.e., ϕsyn > ϵsyn and ϕsem < ϵsem).
3. Mismatched relations (T α2 ) correspond to segment pair meanings that are to some extent mutually exclusive, resulting in a low ϕsyn(r, t) (i.e., ϕsyn < ϵsyn).
The procedure and the setting of the two thresholds (ϵsem and ϵsyn) are also summarized in Figure 4 in Appendix C.
7We derive segment embeddings using average pooling. 8Note that before the following procedure, we first reduce the set T̄ to T by removing irrelevant segment pairs (see Section 2.1).
9Note that the scoring functions are not limited to these two but can be replaced with other suitable functions.
3.3 S2: Out-of-domain Fine-tuning Here we pinpoint financial signals for segment pairs in T α = T α1 ∪ T α2 . Specifically, for each segment pair (r, t) ∈ T α, we discover rationales through predicted word importance in target segment t, where the rationales are inferred conditioned on reference segment r (see Eq. (1)).
Binary token classification To accomplish this, we cast the word importance prediction as supervised binary token classification. First, we leverage the pre-trained BERT (Devlin et al., 2019) model to construct contextualized reference-to-target pair representations, where each pair of interest constitutes an input with special tokens as
h(r,t) = BERT([CLS]r[SEP]t),
where h(r,t) ∈ Rn×d is the contextualized token representation of the pair, d is the dimension of each token representation, and n is the number of tokens in segment pair (r, t). Second, on top of the token representation h(r,t), we add a highlighting model f(·) (an MLP layer) with softmax function. The resultant conditional word importance P jf (t|r) for the j-th word in target segment t is
P jf (t|r) = exp
(( f ( hj(r,t) ) [1] ) /τ )
∑2 i=1 exp (( f ( hj(r,t) ) [i] ) /τ ) ,
(2) where hj(r,t) denotes the token representation of the j-th word in target segment t (i.e., the j-th row vector of h(r,t)), f(·) : Rd → R2, and τ is a hyperparameter that controls the probability distribution.
Signal highlighting warm-up As we view signal highlighting as binary token classification, we first fine-tune the model f(·) on e-SNLI (Camburu et al., 2018), an external human-annotated dataset, to obtain a zero-shot model. Note that e-SNLI was compiled for explanation generation with human-annotated rationales to distinguish relations of aligned sentence pairs (r′, t′) (i.e., premise and hypothesis) in natural language inference. We then treat the annotated words as the ground truth for the premise-to-hypothesis relation,10 which is similar to our reference-to-target structure. Formally, we adopt the binary cross-entropy objective for each token in hypothesis t′ to fine-tune the
10Here, we specifically select contradiction pairs in e-SNLI as this relationship is closer to our goal than the other two.
BERT token representations and the highlighting model f(·) as
LCE = ∑
j
− ( Y jt′ logP j f ( t′|r′ ))
+ ( 1− Y jt′ ) log ( 1− P jf (t′|r′) ) ,
where Yt′ is a vector in which each element Y j t′ indicates the binary label of word importance for the j-th word in hypothesis t′. For instance, Y jt′ = 1 implies the j-th word in t′ is annotated as an important word conditioned on the given premise sentence r′. We thus construct the out-of-domain zero-shot highlighting model by fine-tuning on eSNLI, which is regarded as a baseline to proceed with the following financial domain adaptation (see Figure 1).
3.4 S2+: In-domain Fine-tuning Generally, for applications, particularly in niche domains like finance, models with a zero-shot setting may not be effective enough. Also, several studies show that language models exhibit poor performance under domain shift scenarios (Ben-David et al., 2006; Han and Eisenstein, 2019; Gururangan et al., 2020; Li et al., 2022). We account for this by equipping the proposed pipeline with an extra in-domain fine-tuning stage to enable our highlighting model to adapt properly to the financial domain. Specifically, we construct a domain-adaptive financial signal highlighting model f+(·) with the following learning strategies: (1) pseudo-labeling with revised segment pairs in T α1 , and (2) further fine-tuning with soft labels.
Pseudo-labeling with revised segment pairs We introduce a simple yet effective pseudo-labeling approach that uses revised segment pairs (i.e., T α1 ) collected from stage S1 (see Section 3.2). Recall that these segment pairs differ in some words only but have quite different meanings. Given such a property, we establish a heuristic labeling approach for pseudo-labels of financial signals. Intuitively, we treat all revised words in target segment t as important words and mark them as positive, and randomly sample other words as negative ones.11
Further fine-tuning with soft labels To compensate for deficiencies in such assertive binary pseudo-labels, we use soft labeling to make the
11We set the number of negative labels to three times that of the positive ones.
token representations more generalized. Initially, as illustrated in Figure 1, we leverage the zeroshot highlighting model f(·) learned at stage S2 to calculate the approximate word importance of the revised segment pairs, the results of which are regarded as soft labels compared to the assertive pseudo-binary labels. We then construct the softlabeling objective LSL as
LSL = γLCE + (1− γ)LKL, (3)
where
LKL = ∑
j
−KL ( P jf (t|r) ∥∥∥P jf+(t|r) )
(4)
and γ is a hyperparameter that controls the impact of soft labeling. In Eqs. (3) and (4), KL(·) denotes Kullback–Leibler (KL) divergence, and Pf (t|r) and Pf+(t|r) indicate the estimated probability distributions predicted by f(·) and f+(·), respectively. Finally, we fine-tune the highlighting model f+(·) with the pseudo-labels annotated on segments in T α1 by optimizing LSL in Eq. (3). Note that we not only utilize probabilities Pf (t|r) as our training targets (i.e., soft labels) for LKL but we also adopt the warm-start token representations and highlighting layer f(·) as the initial checkpoint for fine-tuning f+(·). In addition, we discover that hyperparameters τ and γ affect the performance significantly. We discuss the hyperparameter search in Appendix B.

heading: 4 The FINAL Dataset
text: We constructed FINAL (FINancial-ALpha), a financial signal highlighting dataset, consisting of 30,400 reference-to-target segment pairs in ∈ T α.

heading: 4.1 Financial 10-K Corpus Preprocessing
text: We used Form 10-K filings collected from the Software Repository for Accounting and Finance,12 where a Form 10-K is an annual report required by the U.S. SEC. Specifically, we used 10-K filings from 2011 to 2018, which comprise 63,336 filings from 12,960 public companies. To make the best use of the year-to-year information, we discarded companies for which the reports in some years were missing during the period; 3,849 companies (3,849×8=30,792 reports total) remained after this filtering. We then randomly sampled 200 companies from the 3,849 companies with their
12https://sraf.nd.edu/sec-edgar-data/
annual reports to construct the dataset. In addition, while every 10-K annual report contains 15 schedules (e.g., Items 1, 1A, 1B, 2, 3, . . . , 7, 7A, . . . , 15), 13 we extracted only Item 7 (Management’s Discussion and Analysis of Financial Conditions and Results of Operations (“MD&A”)) to form the FINAL dataset.14 Finally, we aligned each document Dℓ with its corresponding last-year document Dℓ−1, resulting in 1,400 reference-to-target document pairs (i.e., 200 companies × 7 year-to-year pairs).

heading: 4.2 Year-to-year Segment Pair Generation
text: After preprocessing, we followed the proposed multistage pipeline by first passing each document pair through stage S0 to obtain an enumerated set of segment pairs T̄ ; we then reduced T̄ to T by removing irrelevant segment pairs (see Section 2.1). Next, we followed the relation recognition stage S1 in Section 3.2 to obtain the two groups of segment pairs: T α1 and T α2 . From each of these two groups, we randomly sampled 200 pairs for human annotation as our evaluation sets. Likewise, we randomly sampled 30,000 pairs from the rest of the revised segment pairs (i.e., T α1 ) as the training set for the pseudo-labeling approach in Section 3.4.

heading: 4.3 Human Annotation
text: To evaluate the empirical effectiveness of the proposed pipeline, we manually annotated the sampled 400 segment pairs. For each segment pair (r, t), we collected the labels of rationales from three annotators. Specifically, the annotators were to distinguish which words in each target segment t to regard as important financial signals according to the context of the corresponding reference segment r. That is, the words with positive labels
13https://en.wikipedia.org/wiki/Form_ 10-K
14This setting follows most of the literature regarding textual analysis of financial reports.
were to characterize the reference-to-target relationship or disclose extra information of interest,15 whereas the rest of the words in t were labeled as negative. We further assessed the inter-rater reliability of the three annotations with Fleiss’ κ (Fleiss, 1971). For simplicity, we treat the prediction for the importance of each word in the target segment as independent classification tasks (containing roughly 12K words in the 400 evaluation pairs): for evaluation pairs from T α1 , κ = 0.71; for those from T α2 , κ = 0.60. The training and evaluation sets are described in Table 2(a), where Avg. |t| and Avg. |r| are the average lengths of target and reference segments, respectively, and Avg. #w+ and Avg. #w− are the average numbers of words annotated as positive and negative, respectively.

heading: 5 Experiments

heading: 5.1 Evaluation Datasets
text: FINAL We evaluated the highlighting performance on the two evaluation sets with the humanannotated ground truth (see Table 2(a)).
e-SNLIc We additionally evaluated the performance on e-SNLI. Particularly, in this paper, we used only the premise-to-hypothesis sentence pairs labeled as contradiction (denoted as e-SNLIc) in the test set of the e-SNLI dataset for evaluation (see Table 2(b)).

heading: 5.2 Evaluation Metrics
text: Recall-sensitive metric In practice, financial practitioners are usually concerned more about the recall of the discovered signals than their precision due to the high cost of missing signals. Accordingly, we borrow the idea of R-precision (Buckley and Voorhees, 2000), a metric from the information retrieval field. In our case, R-precision (R-Prec) is the precision at R, where R is the number of annotated words in each target segment: if there are r annotated words among the top-R predicted words, then the R-precision is r/R.
Sequence agreement of word importance In addition, we measure the agreement between the predicted importance of words for each target segment (considered as a number sequence) and its corresponding ground-truth sequence. Specifically, we use the Pearson correlation coefficient (PCC) for evaluation.
15The annotation guidelines are provided in Appendix D.
Note that for R-Prec, we use majority voting to derive single ground-truth labels from the three annotators, whereas for PCC, we take the mean agreement of the three annotations as the ground truth. Note also that neither of the above two metrics requires a hard threshold to determine the important words for evaluation. Whereas R-Prec considers the words with the top-R highest predicted probabilities, PCC directly leverages the predicted probabilities of words as the importance of words for calculation.

heading: 5.3 Compared Methods
text: Zero-shot We fine-tuned the BERT-base model on the e-SNLIc training set with the binary token classification cross-entropy objective (See Section 3.3 for details) and used this as a zero-shot approach for financial signal highlighting.
Pseudo few-shot Instead of using e-SNLIc, we fine-tuned the BERT-base model on the 30,000 revised segment pairs in T α1 (see the “Train” data in Table 2(a)) with the pseudo-label tokens (see pseudo-labeling introduced in Section 3.4) and use this as a pseudo few-shot approach.
Domain-adaptive Using the zero-shot highlighting model as the initialization, we further performed in-domain fine-tuning (see stage S+2 in Section 3.4) for domain adaptation.

heading: 5.4 Empirical Results

heading: 5.4.1 Main Results for Signal Highlighting
text: Performance on FINAL Table 3 tabulates the highlighting performance under four conditions (i.e., #1–#4), where W.U. denotes that e-SNLIc is used for warm-up fine-tuning (i.e., the zero-shot highlighting model), P and S denote pseudo and soft labeling, respectively, and ’∗’ denotes statistical significance with respect to the performance of
zero-shot learning (#1) under a paired t-test with p < 0.05.
We first focus on the results of the main task on FINAL, where the listed results are those evaluated on the union of the two evaluation sets (including 400 segment pairs in total). As shown in the table, the proposed domain-adaptive approach using both pseudo and soft labeling techniques (i.e., condition #4) achieves the best R-Prec of 0.7865 and PCC of 0.7290. In addition, from the performance increase from condition #2 to #3, we observe that warm-up fine-tuning (W.U.) plays an essential role in financial signal highlighting. Similarly, soft labeling is also beneficial for our task, bringing a 10% performance improvement in both evaluation metrics (by comparing the results of conditions #3 and #4). However, from the results of conditions #1 and #3, we observe that adopting pseudo-labeling alone might not be helpful for this task, perhaps because the pseudo-labels constructed by the proposed heuristic approach (see Section 3.3) are too aggressive for unimportant tokens, resulting in a biased highlighting model. In sum, we offer two main observations from Table 3.
• The proposed domain-adaptive fine-tuning with pseudo and soft labeling is effective for signal highlighting in financial reports.
• Warm-up fine-tuning and soft labeling are two crucial components to constructing an effective domain-adaptive highlighting model.
Generalization ability between domains Table 3 also lists the results on the e-SNLIc testing data: only the model with condition #4 performs on par with or even outperforms that with condition #1 (i.e., zero-shot), showing that the highlighting model fine-tuned by the propose domainadaptive approach exhibits good generalizability.

heading: 5.4.2 Analyses on Different Types of Relationships
text: To better understand the empirical advantages of the domain-adaptive approach, we further investigate the highlighting performance for different kinds of reference-to-target relations, T α1 (revised) and T α2 (mismatched). Figure 2 compares the results of the zero-shot (#1) and domain-adaptive (#4) methods in terms of two metrics. We here focus on PCC, as R-precision considers only the set of important words (i.e., labeled as positive) instead of all the words in each target segment. In
the figure, we see that despite the significant PCC improvements on both revised and mismatched pairs, the benefit of domain adaptation on mismatched pairs is markedly greater than that on revised pairs, yielding a PCC improvement of approximately 23%. Perhaps the important words in the mismatched pairs are more uncertain, necessitating intensive domain adaptation more than those in the revised pairs. Note that we fine-tuned the model on only 30,000 revised segment pairs in T α1 for domain adaptation; however, the highlighting results of mismatched pairs T α2 exhibit more significant improvement. This suggests that the proposed domain-adaptive approach addresses domain shift and yields a superior ability to infer word importance even for unfamiliar (unseen) relationships (See Appendix E also).

heading: 5.5 Ablation Studies

heading: 5.5.1 Impact of Referenced Sources
text: We first determined the impact of the reference segment, which is viewed as the context of a given target segment in terms of discovering the financial signals in the target segment. To this end, for each reference-to-target pair (r, t), we substituted the original reference segment r (i.e., the most syntactically similar segment in the previous years’ document Dℓ−1) for other text and constructed a few variants of variant-to-target segment pairs for inference using the highlighting model. Specifically, we fixed the target segment but recast the BERT contextualized representation of variant pairs as
• Empty: A single [PAD] token is used as the reference segment (implying none in BERT);
• Same: The target segment is used as the reference segment;
• Random: A randomly selected segment is used as the reference segment.
In Table 4, the original setup significantly outperforms the other three settings in both FINAL and e-SNLIc, showing that the knowledge provided by the reference segments is critical for capturing important financial signals in the corresponding target segment.

heading: 5.5.2 Effect of Lexicon-based labeling
text: Recall that in Section 3.4, we introduced a heuristic pseudo-labeling approach that views all revised words in target segment t as important words and marks them as positive while we randomly sample other words as negative words. We here test the effect of additionally incorporating an external financial lexicon for pseudo-labeling. Specifically, we adopt the most representative financial sentiment lexicon—the Loughran–McDonald Master Dictionary (Loughran and Mcdonald, 2011)—and assume that in addition to the revised words in the heuristic approach, the 3,872 sentiment words in the dictionary also reveal important financial signals (i.e., are labeled as positive). Additionally, we treat the 20K most frequently-occurring words, as well as the standard stopwords, as negative words.
As shown in Table 5, surprisingly, adding the lexicon for pseudo-labeling does not improve performance but instead worsens the highlighting results. Although these financial sentiment words convey important financial signals, they are globally important among all financial reports. However, this characteristic precludes the use of the lexicon for company-specific reference-to-target highlighting, which is focused more on local relationships between a pair of segments.

heading: 6 Related Work
text: Research on financial report analysis has been ongoing for many years, with various studies utilizing both textual and numerical features to identify signals in reports. For instance, some researchers
have used the relationship between tokens and quantitative indicators from the financial market to identify financial risks (e.g., Kogan et al., 2009; Tsai and Wang, 2017; Lin et al., 2021). Others have adapted unsupervised methods to recognize information and classify risk factors in financial reports (e.g., Huang and Li, 2011; Lin et al., 2011). However, previous research has mostly focused on risk factors in a global context rather than companyspecific signals, which is the focus of this study.
Recently, transformer-based language models such as BERT, GPT-3, and T5 (Devlin et al., 2019; Brown et al., 2020; Raffel et al., 2020) have made significant strides in the summarization task. In 2020, Zmandar et al. (2021a) proposed the Financial Narrative Summarization shared task (FNS 2020), which aims to summarize annual UK reports. While some methods for this task have achieved satisfactory performance using ROUGE as a metric (e.g., Zmandar et al., 2021b; Orzhenovskii, 2021; Gokhan et al., 2021), they have been criticized for sometimes omitting essential signals under a ROUGE-guided policy. Additionally, the signals discovered through these approaches are heavily dependent on high-quality human-annotated summaries, making it challenging to apply them in real-world scenarios.
In the field of NLP, some research has focused on developing rationalizing models related to the concept of our highlighting model. For example, Lei et al. (2016) proposed a method for learning the rationale (words) to justify a model’s prediction by selecting a subset of text inputs. More recently, some studies have proposed methods that can rationalize the relationship of sentence pairs, such as natural language inference (Jiang et al., 2021) and query-document relevance (Kim et al., 2022). Additionally, DeYoung et al. (2020) released a benchmark to facilitate the development of interpretable NLP models with faithfulness.

heading: 7 Conclusion
text: This paper addresses the task of identifying rationales as insightful financial signals between two narrative financial reports in consecutive years. We use the reference-to-target structure of financial reports to develop a compare-and-contrast multistage pipeline, comprising mainly of relation recognition and signal highlighting stages. In particular, we propose domain-adaptive learning strategies for the financial signal highlighting task, including out-ofdomain warm-up and in-domain fine-tuning. Our empirical results confirm the effectiveness of the proposed approaches. We also present the newly constructed FINAL dataset.
Some future works include (a) improving highlighting effectiveness by developing multitask learning on large financial corpora as financial pretrained representations; (b) increasing efficiency by integrating dense retrieval methods in the Relation Recognition stage; (c) analyzing broader relationships beyond the year-to-year ones (e.g., crosscompany); (d) identifying important words in both reference segments and target segments (i.e., twoway rationalization), which may provide a more in-depth financial analysis. And we believe this research can facilitate NLP techniques applied in finance domain.

heading: 8 Limitations
text: We identify crucial financial signals in reports which can help financial practitioners to digest long financial documents efficiently. However, factors such as macroeconomics, stock prices, and public policies may affect how a financial practitioner views financial reports in practice. Confidential intelligence or social media may greatly affect the analysis results. Therefore, we limit our task to the scenario in which the content in the reports is the sole information available to users. Accordingly, to prevent bias in the annotation process, we acquire annotations from annotators under similar scenarios (graduate students majoring in accounting or other related fields) rather than from financial professionals. In addition, language partially constrains our methods since the data we used in stage S2 is in English; adding a machine translation module may have sub-optimal effectiveness of financial signal highlighting. This is mainly because the financial signals highly depend on many languagespecific knowledge or country regulations.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for the reviews. The work was supported by the grants from the National Science and Technology Council (Grant number NSTC 111-3111-E-002003 and MOST 111-2622-8-002-028-111C2415), National Taiwan University (Grant number NTUCC-112L891104), and NTU IoX Center (Grant number 109-3111-8-002-002-09HZA27003F and MOST 111-2622-8-002-028-111C2415).

heading: A Training Detail
text: All model fine-tuning and inference (in Section 5 and Section 5.5) were conducted on an NVIDIA Tesla V100 32GB GPU. Each model fine-tuning can be done within three hours. We also ran all of the models with shared training settings, including the number of training steps, optimizers, and token batch sizes; we set other related training parameters as the settings in Huggingface Trainer.16

heading: B Hyperparameter Search
text: Recall that while the hyperparameter τ in Eq. (2) controls the probability distribution of the word importance, γ in Eq. (3) controls the impact of soft labeling. Figure 3 shows the performance in terms of R-Prec with different hyperparameter settings, where the left panel shows the results of τ fixed at 1 with γ ranging from 0 to 1, and the right panel shows that of γ fixed at 0.1 with τ ranging from 0.1 to 2. In the left panel of the figure, on FINAL, we see that solely adopting cross-entropy loss LCE (γ = 1) is not effective for fine-tuning the signal highlighting model, nor is adopting KL loss LKL (γ = 0) (see Eq. (3)); γ = 0.1 achieves the best R-Prec. These empirical results again validate the effectiveness of the proposed soft labeling for our highlighting task. In addition, we froze γ at 0.1 and experimented with different settings for the temperature parameter τ , the results of which are shown in the right panel of Figure 3, showing that τ = 2 is the most effective setting. We thus set our final hyperparameters to τ = 2 and γ = 0.1 to yield the best performance.

heading: C Empirical Thresholds
text: For the relation recognition procedure in S1 (see Section 3.2 and Figure 4), we empirically set the
16https://huggingface.co/docs/ transformers/main_classes/trainer
thresholds ϵsyn = 0.6296 and ϵsem = 0.9011. Both numbers are the 50 percentiles of the corresponding similarity scores calculated from the reduced segment pair set (T ). Note that, in this work, we adopt a rule-based heuristic method for recognizing relations using similarity functions with hard thresholds. We leave the exploration of other similarity functions, thresholds, and approaches to future work.

heading: D Annotation Guidelines
text: For each segment pair, the annotators were to focus on the semantic difference regarding the referenceto-target relationship and annotate words in the target segment as positive when the words were considered important financial signals. The following guidelines were given for the annotators’ reference.
• Changes: Changing numbers or objects are important signals in financial reports (e.g., sales, cost, partnership, products, etc.). • Opposition: Descriptive phrases that indicate distant semantic meanings (e.g., increased/decreased, effective/ineffective, etc.). • Precise: Labeling words with high confidence as positive only (i.e., leaving ambiguous words as negative). • Extra information: Identifying new information according to the context, for which the annotators considered the reference segment as the context (e.g., new policy, canceled deals, newly published products, etc.).

heading: E Empirical Cases
text: In Table 6, we take few revised segment pairs (T α1 ) and mismatched segment pairs (T α2 ) as examples. The underlined words are with the top-k highest importance predicted by the proposed pipeline.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 8.
7 A2. Did you discuss any potential risks of your work? We have checked and discussed the ethical policies; we didn not find potential risks in our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1.
3 A4. Have you used AI writing assistants when working on this paper? We used Grammarly to check if there are any misused grammar, typos in very beginning manuscripts.
B 3 Did you use or create scientific artifacts? Left blank.
3 B1. Did you cite the creators of artifacts you used? Section 3 (spaCy, e-SNLI dataset); Section 4(Software Repository for Accounting and Finance(SRAF))
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We will update the license or terms for use of our released dataset if we open-sourced our codes and data (they are in anonymous repository now.) According to Software Repository for Accounting and Finance(SRAF), all software and data are provided without warranties, and are for non-commercial purposes.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3.3 (the detail settings of our intended use of external data.); Section 4(our created dataset). We will update the license or terms for use of our released dataset if we open-sourced our codes and data (they are in anonymous repository now.)
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We used the original corpus from SRAF without any revisions or other related resources. We believe the contents in this data do not contain the sensitive information about ethical issues.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? To our knowledge, the raw corpus we used is from 10-K financial report which is regulated by U.S. SEC; therefore, the data is covered in financial domain and in English only.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5.4 (main results); Section 5.5 (ablation studies).
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A (training detail).
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B (hyperparameters search).
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In section 5, we reports the experimental results; we also report the p-value of pair t-test in the results table.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We cite the URL of spacy’s API we used. ROUGE setup is the default setting from original paper.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.3.
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix D.
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4.3 (human annotation); Section 8 (limitation).
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We have informed and acquired their approval for the further usage in this paper. The annotators were agreed and awared of the purpose of our work. We have also acquired their approval for releasing the dataset with their annotations.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We have checked and followed the ethical policies in the process of this work.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section 8.

heading: Future Work
text: We leave the exploration of other similarity functions, thresholds, and approaches to future work