# A Framework for Market State Prediction with Ontological Asset Selection: A Multimodal Approach
## 4 Materials and Methods
### Ontology-Based Stock Selection
#### 4.2.6 Data Preprocessing and Noise Handling  
Effective data preprocessing is critical for ensuring the reliability of machine learning models, particularly in financial applications where datasets often contain noise and inconsistencies. In this study, several preprocessing steps were employed to enhance data quality and reduce the impact of noise:  
* Data Cleaning: Missing values in financial indicators (e.g., Illiquidity Index, VIX) were addressed through a combination of interpolation and forward-fill techniques. This approach ensures continuity in time series data while minimizing the introduction of artificial trends. - Rare Events and Structural Changes: Outliers frequently reflect significant market responses to macroeconomic events, such as interest rate announcements or unexpected geopolitical developments. For example, a sudden spike in the VIX might indicate heightened uncertainty due to a global crisis. As highlighted by [PERSON] et al. (2023) [28], such events provide crucial insights into systemic market disruptions, particularly during crises when asset correlations weaken, revealing structural changes in financial systems. - Market Insights from Extremes: [PERSON] and [PERSON] (2021) [29] demonstrated that outliers, such as those observed during \"flash crashes\", often indicate unbounded variance in trading volume, reflecting systemic risks and feedback loops. These events are not noise but critical manifestations of market stress and self-organized criticality. - Contextual Validity: Outliers are not inherently errors; they may legitimately represent extreme but relevant behaviors within financial markets. Removing them indiscriminately risks losing critical insights into market dynamics under extreme conditions. - Normalization and Scaling: To maintain consistency across variables with different scales (e.g., sentiment scores, volatility indices), data were normalized using min-max scaling. This step standardizes inputs to ensure that all features contribute equally to the model's performance. * Text Preprocessing for Sentiment Analysis: Financial news texts were tokenized, lowercased, and stripped of stopwords before being fed into the BERT model. This preprocessing ensures that irrelevant tokens do not dilute the sentiment signals extracted by the model, as suggested by [PERSON] et al. (2020) [7]. The importance of robust preprocessing in financial datasets has been emphasized in the literature. [PERSON] et al. (2020) [7] demonstrated that noise in textual data can significantly impact sentiment analysis outcomes, highlighting the need for effective text-cleaning pipelines. Similarly, [PERSON] et al. (2022) [30] underscored the role of scaling and proper handling of outliers in improving the reliability of financial models. [PERSON] et al. (2023) [28] and [PERSON] and [PERSON] (2021) [29] further advocate for retaining outliers when they represent rare but critical market events, ensuring that systemic insights are preserved. By implementing these preprocessing steps, this study ensures that the inputs to the framework--ranging from sentiment scores to financial indicators--are both accurate and representative, minimizing the risks associated with noisy or incomplete data while preserving critical market insights. - Market State Prediction
- Construction of the Sentiment Index The Sentiment Index is derived from a combination of financial variables, such as amplitude, volume, turnover, and others, which collectively capture market sentiment. Given the high dimensionality of these variables, dimensionality reduction techniques are applied to ensure computational efficiency and enhance the signal-to-noise ratio. Dimensionality reduction is crucial when dealing with datasets containing numerous interdependent variables. [PERSON] et al. (2020) [31] provided a comparative study of various dimensionality reduction techniques, highlighting their respective strengths in preserving essential features while minimizing redundancy.