FinNum-3 –Investor's and Manager’s Fine-grained
Claim Detection

Hsin-Hsi Chen, Department of Computer Science and Information Engineering, National Taiwan
UniversityHiroya Takamura, AIRC, National Institute of Advanced Industrial Science and Technology
Yu-Lieh Huang, Department of Quantitative Finance, National Tsing Hua University
Hen-Hsen Huang, Department of Computer Science, National Chengchi University
Chung-Chi Chen, MOST Joint Research Center for AI Technology and All Vista Healthcare

1

FinNum-3• Manager’s and Investor's Fine-grained Claim Detection• Chinese financial analysis reports (investor)• English earnings conference call (manager)• Information in NumClaim 2.0• Given• Target numeral• Context of Target numeral• Model Output• Category of target numeral (FinNum-1)• In-claim or out-of-claim (FinNum-3)• Metrics• Micro-F1 and Macro-F1 Scores• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html

2Motivation• FinNum-1 & FinNum-2 Social Media• FinNum-3  Formal Documents• Argument mining in finance• Over 58.47% of sentences in analysis report contain at least onenumeral• Investors always make a claim with an estimation• (X) We estimate that the sales may growth• (O) We estimate that the sales growth rate may exceed 40%• The importance of fine-grained claims and the numerals.
• We estimate that the sales growth rate may exceed 5%• We estimate that the sales growth rate may exceed 40%

3NumClaim• Chinese financial analysis reports (investor)• The annotators work in the financial industry (bank’s treasury
department and hedge fund)• The Cohen’s kappa agreements between the experts are 88.31%• 5,144 instances: 23.78% “In-claim” and 76.22% “Out-of-claim”

Chen, Chung-Chi, Hen-Hsen Huang, and Hsin-Hsi Chen. "NumClaim: Investor's Fine-grained ClaimDetection." Proceedings of the 29th ACM International Conference on Information & Knowledge
Management. 2020.

4

Auxiliary Task –Numeral Understanding (FinNum-1)• The Cohen’s kappa agreements between the experts are 89.55%

Chung-Chi Chen, Hen-Hsen Huang, Yow-Ting Shiue, and Hsin-Hsi Chen. 2018. Numeral understanding in financial tweets for fine-grained
crowd-based forecasting. In IEEE/WIC/ACM International Conference on Web Intelligence5

Statistics[12] Steffen Eger, Johannes Daxenberger, and Iryna Gurevych. 2017. Neural End-to-End Learning for Computational
Argumentation Mining. In ACL[13] Steffen Eger, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2018.Cross-lingual Argumentation Mining:
Machine Translation (and a bit of Projection) is All You Need!. In COLING.

6

Experimental Results• Encoding: BERT
• Baseline: CNN, BiGRU, CapsNet• Metrics: Macro-F1• Class Weight (CW)
• Numeral Encoder• Represent the digit (0-9) and the decimal point as a 11-dimension tensor, and concatenate it with a tensor for theinter-numeral position information.
• Joint Learning with Category Classification Task (CG)7

Earnings Conference Call•

Modeling Financial Analysts' Decision Making•Keith, Katherine, and Amanda Stent. "Modeling Financial Analysts’ Decision Making
via the Pragmatics and Semantics of Earnings Calls." ACL-2019.

•Risk Forecasting•

Ye, Zhen, Yu Qin, and Wei Xu. "Financial Risk Prediction with Multi-Round Q&AAttention Network.“ IJCAI-2020.

•Sawhney, Ramit, et al. "VolTAGE: Volatility Forecasting via Text-Audio Fusion withGraph Convolution Networks for Earnings Calls." EMNLP-2020.
•Qin, Yu, and Yi Yang. "What you say and how you say it matters: Predicting financial
risk using verbal and vocal cues." ACL-2019•Yang, Linyi, et al. "HTML: Hierarchical Transformer-based Multi-task Learning for

Volatility Prediction." Proceedings of The Web Conference 2020.
•Li, Jiazheng, et al. "MAEC: A Multimodal Aligned Earnings Conference Call Dataset for
Financial Risk Prediction." CIKM-2020.

8

Questions from Reviewers (1/3)• The organisers ran a related task outside NTCIR that involves a cash
prize. If that task involves a cash prize, and the NTCIR task does
not, what would be the motivation for researchers to sign up withthe latter?
• Originally, we also plan to prepare a cash prize for FinNum-2
participants. However, after discussing with chairs and otherorganizers, we decide to cancel the cash prize. Thus, we did notshow the cash prize in the proposal of FinNum-3.
• The size of the listed categories is highly unbalanced. Therefore, theauthors should consider focusing on some major categories instead ofusing all of them in the second task.
• Because category information is prepared as the auxiliary task forenhancing the performance of claim detection, our participants can

explore different settings.

9Questions from Reviewers (2/3)•

It was not very clear how beneficial it is to distinguish in-claim and out-claim. It
should be better motivated. The lack of any end-user task (the proposal assertsthat there will be beneficial effects on “downstream tasks”, but no such tasks

are named)•Argumentation Strategies•

Argumentation Structures•Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021.

Evaluating the Rationales of Amateur Investors. In Proceedings of The
Web Conference 2021 (WWW'21)•It is not clear whether the macroaverage will be taken over types of subtypes inthe type classification task, but the real puzzle is what the macroaverage will
be computed over in the “in-claim” detection task.

•Calculate metrics for each label, and find their unweighted mean. This
does not take label imbalance into account.

10

Questions from Reviewers (3/3)• There is another concern for in- and out-claim classification: Is it aproblem of numerals?
• Based on the results of the pilot exploration, we find that numeral
understanding can be used for better comprehending financial
documents. Claim detection is the first step of argument mining. We
hope to use FinNum-3 to link with the next shared task series weplan to propose in NTCIR-16-18 – Argument Mining in Financial

Narratives (FinArg)• Can Chinese Analyst’s Report and English Earnings Conference Callbe released under the license?
• Chinese Analyst’s Report  Can be shared by request for academicusage• English Earnings Conference Call  Publicly-available

11

Schedule• 2021• Apr 30 Chinese Training and Development Set Release• Jul 31 English Training and Development Set Release• Dec 1 Test Set Release• 2022• Jan 3 Task Registration Due• Jan 10 Participants’ Results Submission Due• Feb 1 Evaluation Result Release & Draft Task Overview Paper Release• Mar 1 Participants’ Papers Submission Due• May 1 Camera-ready Paper• Jun NTCIR-16 Conference in NII, Tokyo, Japan12

Feel free to contact us if you have any questions.
March & April  Hen-Hsen Huang: hhhuang@nccu.edu.tw
From May  Chung-Chi Chen: cjchen@nlg.csie.ntu.edu.tw