## 5 Results and Analysis
### Interpretability, Calibration, and Error Analysis  
Beyond quantitative metrics, understanding the model's internal working mechanisms, assessing the reliability of its predictions, and analyzing its failure cases are equally important.  
Although Hyena uses long convolutions rather than self-attention, its implicitly parameterized convolution kernels \(h_{n}\) can still be analyzed. By visualizing the weights of the trained kernels or their response to inputs (analogous to attention maps but along the time/sequence dimension), we can investigate which parts or patterns in the input text the model focuses on.  
As in Figure 10, analysis of the model's internal mechanisms (like kernel responses or equivalent contribution scores) suggests a tendency to focus on keywords and phrases directly related to monetary policy stance and economic conditions. For instance, terms like \"inflation risks\", \"tigther monetary policy\", \"restrictive stance\", and \"raise the target range\" often receive higher contribution scores, particularly when predicting an \"up\" rate change. Conversely, words like \"accommodative\", \"patient\", or \"softening labor market\" are more frequently associated with \"down\" or \"stable\" predictions. This pattern indicates that the model is indeed learning to understand the economic meaning and policy signals within the text, rather than relying solely on superficial statistical features.  
A reliable predictive model should not only be accurate, but also produce probability scores that reflect true confidence levels. We use the Brier score to measure overall calibration error and visualize calibration using a reliability diagram.  
Figure 11 shows the calibration curve for the model's predicted probabilities for the \"up\" class. Ideally, the curve should closely follow the diagonal line (perfect calibration). Our model's curve (blue line) is reasonably close to the diagonal, suggesting that its predicted probabilities are largely reliable. For example, when the model predicts an \"up\" probability around 0.8, the actual frequency of \"up\" events is also close to 80%. The calculated Brier score is relatively low, further confirming the model's good overall calibration. This implies that the model's output probabilities can be used with reasonable confidence for risk assessment or decision-making.  
Figure 10: Heatmap of Hyena convolution kernel contribution response to specific vocabulary in an example sentence. Higher scores indicate greater model focus.  
To evaluate the three-class classification performance in more detail, we plot ROC curves and precision-recall (PR) curves for each class.  
Figure 12 presents the receiver operating characteristic (ROC, left) and precision-recall (PR, right) curves for the three output classes ('up', 'down', and'stable') evaluated on the FOMC test set. The ROC curves indicate that the model performs significantly better than random chance for all classes. Notably, the area under the curve (AUC) scores for the directional change classes, 'up' (AUC = 0.86) and 'down' (AUC = 0.86), suggest a reasonable ability to distinguish these instances from others based on the model's score rankings. However, the lower AUC for the'stable' class (AUC = 0.73) points to greater difficulty in uniquely identifying neutral or ambiguous policy stances from potentially directional ones.  
Figure 11: Model prediction probability calibration curve (reliability diagram) for the ‘up’ class.  
Figure 12: Receiver operating characteristic (ROC) and precision–recall (PR) curves per class for the model on the FOMC test set. **(a)** ROC curves for all three classes (‘up’, ‘down’, and ‘stable’) are well above the diagonal, with high AUC values, indicating strong discriminative ability. **(b)** PR curves show high average precision (AP) scores for all classes, including the minority ‘up’ and ‘down’ classes.