QuanSIRA: The Quantitative Investment Risk Modeling in Stock Markets with Large Language Models
Authors: Zheyang Luo, Jiangming Liu


Reviewer paper summaries: This paper introduces QuanSIRA, a dataset for investment risk analysis using language models. It is based on two years of Twitter and market data for 69 stocks. The authors propose a composite metric that integrates multiple risk measurements to assess stock risk levels. They also present a novel risk prediction model based on pre-trained language models (PLMs), utilizing both textual and market data as inputs. Benchmarks were conducted on both in-stock and cross-stock risk predictions using the proposed datasets and risk prediction model.
Summary of strengths: 1. The problem of risk modeling is important in the real-world market and was overlooked in previous NLP works
2. The composite risk level metric is interesting and includes comprehensive risk measurements
3. The authors proposed a novel framework for risk prediction that combines news and market data using LMs
Summary of weaknesses: 1. The dataset is small contains only 69 stocks and has a short time span, especially since the test split is only 72 days for in-stock and 9 stocks for cross-stock settings. Moreover, the contribution of the authors is limited, they didn't really collect new data as the stock and tweet are directly reused from Xu and Cohen (2018), and the authors only annotate them with their proposed risk level metric. It is hard to be regarded as proposing a "new" dataset. The link to the code and data is also broken in line 102.
2. The authors claimed LLMs in their paper, but there are no LLMs used in their method. They only analyzed PTMs like Bert and Roberta. 
3. The clarity of the methods can be improved as well as the design of the methods. See the suggestions for details.
Comments, suggestions and typos: 1. 5.1, how many ticks are used to construct D_num, and what is the frequency? Whether the model is aware of the frequency? 
2. 5.2, what is the n in the equations? How do they connect to N? All q_i are computed from the same D_text?
3. Whether the model predict the risk of the next trading day? Does it model the intra-day risk?
4. How was the cross-stock prediction performed? Input the data of one stock predict for another? What if the correlation between two stocks is low?
5. No illustration about the training details such as hyperparams.
6. Sec 5.1: Why does using LM to process time series data directly make sense? Given that the processing of numbers is still challenging in LMs. Are there any references to it that can support your design?
7. There is no discussion of the threshold selection for low, medium, and high risks. Different metrics may have different mean and variance, thus the importance of each metric may be implicitly unequally weighed while using an equal weight in the composed risk-level metric.
Overall assessment scores: 1.5