ContraSim: Contrastive Similarity Space Learning for Financial Market Predictions

Authors: Nicholas Vinden, Raeid Saqur, Zining Zhu, Frank Rudzicz
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Learning Representations, Large Language Models, Financial Forecasting

Abstract:
We introduce the Contrastive Similarity Space Embedding Algorithm (ContraSim), a novel framework for uncovering the global semantic relationships between daily financial headlines and market movements. ContraSim operates in two key stages: 
(i) Weighted Headline Augmentation, which generates augmented financial headlines along with a semantic fine-grained similarity score, and (ii) Weighted Self-Supervised Contrastive Learning (WSSCL), an extended version of classical self-supervised contrastive learning that uses the similarity metric to create a refined weighted embedding space. This embedding space clusters semantically similar headlines together, facilitating deeper market insights. Empirical results demonstrate that integrating ContraSim features into financial forecasting tasks improves classification accuracy from WSJ headlines by 7%. Moreover, leveraging an information density analysis, we find that the similarity spaces constructed by ContraSim intrinsically cluster days with homogeneous market movement directions, indicating that ContraSim captures market dynamics independent of ground truth labels. Additionally, ContraSim enables the identification of historical news days that closely resemble the headlines of the current day, providing analysts with actionable insights to predict market trends by referencing analogous past events.

Decision: reject
Reviewer scores: 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730125805749

thread_type:
official_review

<thread_object>
creator: reviewer_yax5

<content>
summary:
The paper introduces ContraSim, an innovative approach for predicting financial market movements that employs a two-part method: it first creates semantically varied financial headlines through Weighted Headline Augmentation, and then applies Weighted-Self Supervised Contrastive Learning (WSSCL) to refine the embedding space. This combination allows ContraSim to effectively group newslines that reflect similar market trends, achieving a notable 7% improvement in classification accuracy for financial forecasts. Additionally, the authors present a novel metric called Info-kNN to assess how well the embedding space captures significant semantic relationships within financial news.

strengths:
It presents a methodology that integrates advanced techniques such as Weighted Headline Augmentation and Weighted-Self Supervised Contrastive Learning (WSSCL), providing a fresh perspective on the interplay between financial news and market behavior.  The introduction of the Info-kNN metric to evaluate semantic clustering in the embedding space is another key contribution, offering researchers a robust tool for assessing how well models capture the complexities of financial data.

weaknesses:
First, the paper does not inform the reader how ContraSim contributes to the field relative to other approaches. Second, apparently LLMs bring their own biases to the model. Any biases inherited from the LLM can affect the quality of the data and predictions. Predictions based on headlines provides limited scope and prediction errors as headlines are usually written to take attention.

questions:
What are the types of augmentations utilized?
What are the characteristics of the learned embedding space that influences the interpretability of financial predictions? 
Can you elaborate the hyperparameter tuning process in more detail?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730249539622

thread_type:
official_review

<thread_object>
creator: reviewer_lyzi

<content>
summary:
The paper presents ContraSim, a contrastive learning framework for financial market prediction that creates a dense embedding space for financial news headlines. The approach has two main components:

1. Weighted Headline Augmentation: This technique generates variations of financial headlines with known semantic distances, including reworded, slightly ablated, and negative versions, allowing the model to capture rich semantic relationships between headlines.
2. Weighted Self-Supervised Contrastive Learning (WSSCL): Extending traditional contrastive learning, WSSCL uses augmented headlines to establish a continuous similarity space, clustering semantically similar headlines and separating dissimilar ones.

The framework introduces a novel metric, Info-kNN, to measure the density of semantically similar clusters. When integrated with a language model-based classifier, ContraSim achieves a 7% improvement in classification accuracy and a 13% boost in balanced accuracy over the baseline. Additionally, it provides practical support for financial analysts by identifying similar historical market days, offering valuable insights for forecasting.

strengths:
1. Innovative and Practical Framework: The integration of contrastive learning through Weighted Headline Augmentation and WSSCL in financial market prediction presents a novel approach that assists financial analysts by enabling the identification of historical market conditions.
2. Enhanced Performance: The framework’s substantial improvements in classification accuracy (+7%) and balanced accuracy (+13%) validate its effectiveness.
3. Info-kNN Metric: The introduction of the Info-kNN metric, which evaluates clustering in the embedding space using information-theoretic principles, is a notable contribution. This metric provides a new way to assess contrastive learning models and has potential applications beyond financial data.

weaknesses:
1. Limited Ablation Studies: The current ablation studies are limited, particularly in assessing the individual impact of the headline augmentations (reworded, slightly ablated, and negative). Examining how each augmentation affects accuracy and clustering—especially by showing the effect of their removal—would strengthen the claims. A similar analysis could be extended to Info-kNN and WSSCL components.
2. Narrow Scope in Experiments: Although the authors mention plans for broader application, the current validation is confined to financial data, which limits the immediate generalizability of the findings. Including experiments from other domains or providing a more in-depth discussion of potential applications would enhance the broader impact of this work.

questions:
Could you provide more detailed insights into how the different headline augmentation types (reworded, slightly
ablated, negative) contribute individually to the observed improvements in classification accuracy and clustering
effectiveness? While the paper outlines these augmentations, it would be helpful to understand the individual
impact of each augmentation type on the model's learning process. Additionally, a more detailed ablation study
of key components, such as augmentation types, Info-kNN, and WSSCL, would offer a clearer understanding of
their respective contributions to the overall performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650950094

thread_type:
official_review

<thread_object>
creator: reviewer_kict

<content>
summary:
This paper proposes a model for stock movement prediction.

strengths:
N/A

weaknesses:
1. Low-quality presentation.
2. Lack of serious experiments.
3. Lack of contributions.

questions:
The presentation of this paper is poor, which directly affects readers' understanding of the model design and problem formulation. It is unclear whether some model details are omitted or buried under misleading representations
1. There are numerous grammar mistakes and typos in the first paragraph of Section 3.1, with almost every sentence containing issues. This makes understanding the motivation behind building the proposed embedding space difficult.
2. It is unclear why the authors chose to use 10 and 30 to control the number of news articles sampled per day. Are these numbers theoretically justified, suggested by domain experts, or supported by empirical results? Also, what happens if there are fewer than 10 financial-related news articles in a day?
3. What is the motivation behind using random sampling when there are more than 30 news articles per day? The use of randomness makes it difficult to ensure that the space will cover all topics described in the news for that day. In the worst-case scenario, important topics might be completely omitted.
4. The authors claim that they only keep newslines and ignore tabular data. What does this mean? This statement seems to come out of nowhere.
5. In the description of creating headlines, it is unclear why "prompts" are sometimes created and, at other times, "headlines" are created. Is there a specific reason for this, or are these typos?
6. In Equation 3, it is unclear what the text refers to.
7. The notation $N_i$ in the description below Equation 3 should be $N_k$
8. The symbol $h$ in Equation 3 should be $\hat{h}$
9. The augmentation method lacks a quality control mechanism for headline generation. Also, the definitions of "slight ablation" and "negative" are unclear. In the given example, it appears that the authors only used a language model to generate fake company names, which lacks a clear motivation.
10. The section on "generating newsline buckets" is difficult to understand due to grammar mistakes and undefined terms, making the entire subsection challenging to follow.
11. The method used by the authors to achieve "known semantic distance" lacks motivation. It appears that the authors manually assigned values of 1, 0.5, and 0 to three different augmentations without justification.
12. Grammar mistakes and typos continue throughout the rest of the paper, making it challenging to list them all. However, there are some critical ones. For example, in the WSSCL section, the authors state that there are only three movements: Fall, Natural, and Fall again.
13. The experiment section lacks serious comparisons to justify the performance of the proposed model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731563729415

thread_type:
official_review

<thread_object>
creator: reviewer_lvgh

<content>
summary:
In this paper, a novel Weighted Self-Supervised Contrastive Learning (WSSCL) method is introduced to cluster news-lines based on learned semantic embeddings. LLMs are used to create modified prompts containing semantically identical or augmented news-lines. An enhanced version of KNN algorithm based on information theory is also introduced for clustering, in order to handle imbalanced labelled classes. Experiments on the task of stock market direction prediction show the effectiveness of the proposed method.

strengths:
The paper focuses on an interesting problem, that is stock market direction prediction. The proposed method based on contrastive learning of text embeddings is reasonable. The example about rephrasing, slight ablation, and negative modification of the headline in Table 1 is illustrative and to the point.

weaknesses:
Only one dataset (the NIFTY dataset) is used. It'd be great if the authors could use more datasets for evaluation.

questions:
The estimated baseline values are a mean of random samples following the (23%, 60%, 17%) label split in Table 3. Does it mean if a trivial classifier always predicts the label of a sample is Neutral, it would achieve 60% accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732556848877

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
[Update] Added Experiments to Address Dataset Limitation and Empirical Scope

comment:
We thank the reviewers for the thoughful reviews and suggestions to improve our manuscript. One common weakness suggested by multiple reviewers  (`LVGH` and `LYzi`) was the lack of additional datasets and/or any evaluation for generalizability (broader application) of our proposed approach and/or need of more experiments (`kiCt`) across other tasks/domains.

### Updated (Additional) Datasets
To address this limitation, we have now conducted experiments on two additional datasets: i) BigData22 (which is another financial news headlines dataset, but contains many different asset classes and tickers, unlike the NIFTY dataset that tracks the overall S&P500 market), and ii) the popular IMDB movie review classification dataset - as the name implies, is an orthogonal dataset/experiment in terms of domain and task. 

| **Name of the Dataset** | **Problem Domain**     | **Headlines** | **Days/Movies** | **Date Range**              |
|--------------------------|------------------------|---------------|-----------------|-----------------------------|
| NIFTY-SFT               | Financial Headlines   | 106,271       | 696             | 2014-01-02 to 2015-12-30    |
| BigData22               | Financial Tweets      | 272,762       | 362             | 2019-07-05 to 2020-06-30    |
| IMDB Review             | Movie Reviews         | 955,788       | 352             | 2017-01-03 to 2017-12-28    |

### Updated Experiment Results 

Our manuscript is now updated with the results of these additional experiments (in Section 4). These addition results further bolster the efficacy of our proposed approach.

---

| **Model**        | **NIFTY-SFT (Accuracy / F1)** | **BigData22 (Accuracy / F1)** | **IMDB (Accuracy / F1)** |
|-------------------|-------------------------------|--------------------------------|---------------------------|
| Baseline         | 0.3333 / 0.3333              | 0.5000 / 0.5000               | 0.3333 / 0.3333          |
| $Class_{Proj}$   | 0.3434 / 0.3389              | 0.5005 / 0.5016               | 0.3900 / 0.3897          |
| $Class_{LLM}$    | 0.3522 / 0.3833              | 0.5150 / **0.5094**           | 0.4518 / 0.4124          |
| $Class_{Both}$   | **0.3774** / **0.4670**      | **0.5156** / _0.5089_         | **0.5198** / **0.4620**  |

Table 4: Ablation study on market movement prediction, comparing classification models using ContraSim embedding vectors. $Class_{Proj}$ relies solely on ContraSim embeddings for predictions, $Class_{LLM}$ uses an LLM fine-tuned on market movement tasks, and $Class_{Both}$ combines both features. Results highlight that $Class_{Both}$ consistently outperforms other configurations across datasets. Notably, $Class_{Both}$ achieves the highest Accuracy and F1 scores on all datasets, including the IMDB task, where movie ratings (low, medium, high) are predicted using reviews. This demonstrates the effectiveness of combining similarity space projections with LLM embeddings for diverse classification tasks.

| **Dataset**     | **Model**             | **Info-KNN (k=5)** | **KNN (k=5)** | **KL-Divergence** | **JSD**  |
|------------------|-----------------------|--------------------|---------------|-------------------|----------|
| **NIFTY-SFT**   | Baseline              | .5916              | .4668         | .3539             | .1054    |
|                  | $\mathcal{L}_{CWCL}$ | .7647              | .4732         | **.3821**         | **.1164**|
|                  | $\mathcal{L}_{WSCL}$ | **.7219**          | **.5205**     | .3740             | .1144    |
| **BigData22**    | Baseline              | .7951              | .5506         | .1499             | .0452    |
|                  | $\mathcal{L}_{CWCL}$ | **.9084**          | **.7101**     | .2030             | .0607    |
|                  | $\mathcal{L}_{WSCL}$ | .8590              | .5507         | **.2246**         | **.0640**|
| **IMDB**         | Baseline              | .7456              | .5781         | .2919             | .0818    |
|                  | $\mathcal{L}_{CWCL}$ | .7626              | **.7500**     | **.3957**         | **.1120**|
|                  | $\mathcal{L}_{WSCL}$ | **.8252**          | .6875         | .3024             | .0908    |

Table 5: Results demonstrate that ContraSim embeddings effectively cluster real WSJ headlines (NIFTY-SFT) and Financial Twitter posts (BigData22) based on market movement directions (falling, neutral, rising) without explicit ground truth labels. Additionally, a similar clustering effect is observed with movie reviews, where reviews of movies with similar ratings (low, medium, high) are grouped together.

---

We would be most grateful if the reviewers could kindly review the update version w/ these significant additional experiments to (we hope, conclusively) address the main limitation/concern raised, and update the overall score of the paper if they are satisfactory. 

Sincerely, 

the Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a novel Weighted Self-Supervised Contrastive Learning (WSSCL) method for financial market prediction. The proposed framework introduces a novel metric, Info-kNN, to measure the density of semantically similar clusters. Evaluations on a dataset show the food performance of the proposal. The reviewers have concerns on the insufficient novelty of the paper, the weak evaluation, and the lack of clear explanations on ContraSim's contribution.