Investigating the Impact of Feature Reduction for Deep Learning-based Seasonal Sea Ice Forecasting

Venue: NLDL_org_2025_Conference
Published: 2024-09-09 13:54:29

Abstract: With the state-of-the-art IceNet model, deep learning has contributed to an important aspect of climate research by leveraging a range of climate inputs to provide accurate forecasts of Arctic sea ice concentration (SIC).
The deep learning subfield of eXplainable AI (XAI) has gained enormous attention in order to gauge feature importance of neural networks, for instance by leveraging network gradients.
In recent work, an XAI study of the IceNet was conducted, using gradient saliency maps to interrogate its feature importance.
A majority of XAI studies provide information about feature importance as revealed by the XAI method, but rarely provide thorough analysis of effects from reducing the number of input variables. 
In this paper, we train versions of the IceNet with drastically reduced numbers of input features according to results of XAI and investigate the effects on the sea ice predictions, on average and with respect to specific events.
Our results provide evidence that the model generally performs better when less features are used, but in case of anomalous events, a larger number of features is beneficial.
We believe our thorough study of the IceNet in terms of feature importance revealed by XAI may give inspiration for other deep learning-based problem scenarios and application domains.

Main paper: 
 : 
 input features and provides very accurate forecasts, in particular for anomalous events.However, the predictions are not easy to interpret and the question was posed from which features the network draws the information that leads to its accurate forecasts.Joakimsen et al. [9] leverage a gradient based method to provide an extensive deep learning XAI [10] analysis of the IceNet's feature importance. Thereby, they focus on the forecast for the anomalous month September 2013, as the IceNet showed a particularly high accuracy in this prediction.The results yield detailed information about the impact of the individual features with spatial resolution and with respect to lead times. Based on their results, Joakimsen et al. conclude that only a fraction of the input features provide a relevant contribution to the forecast and suggest that a model trained with only a few features should maintain a high accuracy.Convolutional neural networks are computationally demanding and typically require substantial storage capacity [11] . There has been a lot of effort to leverage feature importance scores to prune parameters and reduce redundancy, as this offers a way to reduce storage requirements and computation costs while maintaining a high accuracy [12] . In contrast to previous studies that often cut back individual connections, node, etc. [13] , we want to examine a more radical approach by completely discarding the features with low importance scores. This has a distinct advantage because it entirely removes the need for a portion of the input features, that in many cases might be hard to come by. Inspired by the findings of Joakimsen et al. [9] , we conduct a novel analysis where we train model variations of the IceNet with different configurations of input features. We compare the performance for the different configurations for the case that was studied by Joakimsen et al. in detail and investigate how the results generalize for all predictions. Finally, we separate a set of anomalous events to examine how the models compare when it comes to predict outliers. The model itself is a convolutional neural network with a U-Net [14] architecture (see Figure 1 ). A transfer learning approach is used to train the model. First, the model is pretrained on climate simulation data (CMIP6) from 1850 to 2100. Then, the training is continued on monthly averaged observation data (era5) from 1980 to 2012. Detailed information about the type, origin and preprocessing of the data can be found in [4] and on https: //github.com/tom-andersson/icenet-paper.Triggered by the accurate forecasts of IceNet for extreme events, Joakimsen et al. published an XAI study with the aim to identify the features, that are most relevant for these results.There are several approaches on how to estimate 150 feature importance for a deep neural network [15-151 17] . Gradient based saliency maps [18] , as they 152 are used in by Joakimsen et al. [9] , offer a way to 153 not only assign importance scores to the individual 154 features, but also provide information on whether 155 or not features have a positive or negative impact 156 on the predictions. Furthermore, this method is 157 spatially resolved, which is particularly useful when 158 there are regions of special interest in the forecasts. 159The gradient of a function can be seen as a mea-160 sure of its sensitivity with respect to small changes of 161 the input variables. Let x = {x 1 , ..., x K } be a set of 162 K input features that result in a prediction f (x) mn 163 for the grid cell (m, n), with m ∈ {1, ..., M } and 164 n ∈ {1, ..., N }, whereas M and N are the number of 165 rows and columns of the grid. A gradient saliency 166 map can be created with respect to a distinct fea-167 ture x k , by computing the gradients of the predic-168 tion f (x) mn with respect to all spatial components 169 x k (i, j) of the input feature x k and accumulating 170 over the spatial components (m, n) of the prediction 171 as follows:172 R(x k (i, j)) = ∂ ∂x k (i, j) M m=1 N n=1 f (x) mn . (1) 173The value of R(x k (i, j)) yields information on how 174 a change of the (i, j)-component of feature x k influ-175 ences the overall prediction. In order to get a single 176 value R(x k ) to rank the feature importance, it is 177 summed over the spatial components (i, j):178 R(x k ) = M i N j R(x k (i, j)).( Consistent with [4] , the performance of the trained model is evaluated using a binary accuracy measure, based on the 15 % threshold, which is a common metric to measure differences in sea ice extent [24] . Each cell is regarded as either ice (SIC > 15 %) or no ice (SIC ≤ 15 %). As for the predictions, that means if the accumulated probability of the classes 2 (marginal ice) and 3 (full ice) is above 50 % the cell is regarded as ice, otherwise it is considered to be no ice. The binary accuracy calculates as the percentage of correctly classified grid cells for every individual prediction. In addition to the pure measure of accuracy, we use the standard deviation between ensemble members to provide a brief uncertainty estimation for the different IceNet configuration in Appendix A.To get a deeper understanding of how the feature reduction affects the predictions beyond simply measuring the average accuracy, we look at different cases separately. The analysis of feature importance by Joakimsen et al. was performed on the prediction for September 2013, as this was a particularly anomalous but accurately predicted event. Thus, we will first compare how the reduced input features affect the model predictions for this particular month in detail. Next, we look at the general case, where we include all predictions to see if the results that were obtained for September 2013 generalize. Last we separate a set of predictions for that we classify as anomalous months and compare the model performances for these predictions. With this set of experiments we aim to analyze the impact of feature reduction on general predictions but also to uncover how anomalous events relate to that, as they are of particular interest. Further, we can put the prediction for September 2013 into context and use it to reveal some details of how the different predictions differ. In this section we provide the results of three exper-279 iments and evaluate the results with respect to the 280 impact of feature reduction in different scenarios. Figure 4 shows the binary accuracies for the predictions of all models for lead times from 1 to 6 months. Supplementary figures of the binary accuracies which include uncertainty estimations can be found in Appendix A. The accuracies between the models for a given lead time vary slightly but remain in the same domain and thus, support the results of Figure 3 . It is notable that the binary accuracies of the reduced models both exceed the accuracy of the original model with 1.2 and 1.0 percentage points (pp.) for a lead time of 1 month. These results strongly support the hypothesis that IceNet's good performance for the prediction of the extreme event in September 2013 is mainly based on previous SICs. Also the observation that the accuracy of the reduced models increases relative to the original model matches the results of Joakimsen et al.Next we examine whether this behavior also extends to the general model performance, apart from this individual extreme event. For this purpose, we average the binary accuracies for each lead time over all predictions from 2012 to 2020. Figure 5 shows the resulting average accuracy versus lead time for each configuration. The original IceNet configuration yields a slightly (ca. 0.5 pp.) lower accuracy than observed by Andersson et al., but this can be explained by discarding the pre-training and the lower number of 10 ensemble members in our experiment compared to 25 members in the experiments of Andersson et al. Remarkably, while decreasing in the same manner, the reduced IceNet is 0.1 -0.4 pp. more accurate over all lead times, with a maximum accuracy of 95.7 % for 1 month lead time. Even the minimal configuration of the model shows higher accuracy than the original version for lead times up to 2 months. From lead times of 3 months and up, the accuracy drops below the original one. While nearly matching the accuracy of the reduced model for small lead times, the minimal model's accuracy is clearly the lowest for large lead times and thus, decreases faster with increasing lead time. These results show that Joakimsen et al.'s hypothesis, which corresponds to the reduced model, holds true even for the general case. For longer lead times it seems that not only the LTF is relevant, but also the monthly SICs of the preceding year, as the minimal model's accuracy decreases quicker compared to the configurations that include these SICs. It should be noted that the LTF itself already provides a good estimate for the future SIC [4] , particularly for non-extreme events. Thus, being able to predict 374 anomalous events with a high accuracy holds more 375 value than regular predictions. the LTF. The figure shows that IceNet's accuracies are generally lower when also the LTF accuracy is low. But at the same time the accuracies distinguish more from the LTF line, for low LTF accuracies. In other words, the more the observed sea ice deviates from its usual extend for a given month, the more superior are the IceNet predictions compared to the LTF. While this view makes it hard to draw general conclusions about the differences between the IceNet configurations, the figure shows that for most extreme events the original configuration performs better than the reduced versions and that the predictions for September 2013 (marked in the figure) is just an exception.To evaluate the performance for extreme events more quantitatively, we classify the 10 % lowest LTF accuracies as anomalous / extreme and assess the performance for these months separately. That corresponds to the predictions left of the red dotted line in Figure 6 . Figure 7 shows the average accuracy for these extreme events versus the lead time for of input features. For these models we provided an extensive performance analysis including different sets of predictions. Our results show that averaged over all predictions, the reduced model yields the highest accuracy for all lead times. The minimal model shows an increased accuracy for lead times up to two months but drops below the original model for larger lead times. For the particular event of September 2013, we also demonstrated that the reduced versions capture properties of the ice structure, that the original version missed. In the end we show that the original model remains superior in cases that deviate a lot from the usual SIC for a given month.We conclude that XAI studies as provided by Joakimsen et al. [9] can be leveraged to effectively minimize the amount of input features for deep learning models, by maintaining overall high accuracy, or even increasing it. This yields a practical and straightforward method, e.g. for cases when certain data is not easily obtainable or data storage is an issue. For the generalization to extreme events and outliers, however, models might still benefit from additional features.Future work might investigate the computational benefits of decreasing the number of features. Further studies might benefit from more extensive underlying XAI studies that, e.g. include different methods to estimate feature importance to increase reliability. Additionally, the robustness of the models might be analyzed by introducing perturbations to the model. Interesting insights could also be gained by going deeper into the uncertainty estimation, for example by training several ensembles per configuration and compare the accuracy deviations of the ensembles within one configuration.In section 4, we discussed the performance of different IceNet configurations in terms of the binary accuracy, using the predictions given by an ensemble individual accuracy score per predicted month and 651 lead time. According to our performance analysis, 652 the calculation of the standard deviation should be 653 performed in such a way that we get distinct results 654 per lead time and set of predictions. We could calcu-655 late the standard deviation between the individual 656 model accuracies, taking into account all of their 657 predictions for a set of dates and fixed lead time at 658 once. However, to reduce the effect of the size of 659 the data set, i.e. the number of dates included into 660 the calculation, we decide to calculate the standard 661 deviation of an ensemble for each lead time and each 662 prediction at a time. Thus, for each ensemble we get 663 one value per prediction month and lead time. For 664 the evaluation of a prediction set, we average over 665 the respective standard deviations. To show the 666 results of our uncertainty estimation, we reproduce 667  
 
 Appendix:  


Decision: Quality:
The paper is methodologically sound, grounded in significant prior work, and provides detailed experimental setups and analyses. It introduces a comprehensive evaluation of feature reduction in sea ice forecasting using the IceNet model, examining reduced feature sets in normal and anomalous data scenarios. However, suggestions for improvement include an ablation study to determine specific feature contributions and uncertainty quantification for ensemble results.

Pros:
- Effective reduction in the feature space, achieving high accuracy on general data while maintaining performance on anomalous events.
- Clear motivation and presentation of experiments with thoughtful observations on expected versus observed outcomes.
- Practical implications for climate change modeling and policy-making in the Arctic.

Cons:
- The reliance on saliency-based methods from earlier work (Joakimsen et al., Simonyan et al.) has limitations, as gradient-based saliency maps may lack causality with model predictions.
- The study could benefit from using more robust XAI techniques (e.g., GradCAM, SHAP, or LIME) and a robustness analysis under perturbations.
- The incremental nature of the study could have been clarified more, especially for readers outside this niche field.

Clarity:
- The work is well-written with clear motivations and structured analyses, but the introduction and abstract could benefit from a clearer distinction of the novel contributions.

Originality:
- The work is novel in its application of feature reduction in Arctic sea ice forecasting using XAI-driven methods. However, it heavily builds on previous studies, which may make the contributions seem incremental.

Significance:
- The research is of moderate significance, primarily valuable for the climate modeling and forecasting community. The exploration of feature reduction in the IceNet model holds practical implications, but further robustness analyses could enhance its broader applicability.

Reviews: review: title: title: Study of IceNet models under saliency-based feature reduction
paper_summary: paper_summary: This work investigates the impact of XAI methods in conjunction with IceNet (Andersson et al. 2021), a UNet model for forecasting sea-ice concentration (SIC) levels in the arctic. In particular, the work looks to investigate the impact of saliency based occlusion at the feature level to study the robustness and model performance under limitations of a reduced feature space. The authors base their approach on a gradient based saliency method developed explicitly for the purpose of SIC (Joakimsen et al. 2022), which in turn is largely based on early gradient based saliency via backpropagation (Simonyan et al. 2013). The manuscript makes a strong case for the adaptability of reduced feature models in improving forecast accuracy without compromising on performance, even in scenarios involving anomalous events.
main_review: main_review: The authors improved an already well formulated work with the feedback from the review. Our score still stands, and we recommend to accept the paper to the conference.
strength_weakness: strength_weakness: - W1: The basis for saliency underlying the approach by Simonyan et al., and subsequently also for Joakimsen et al. has been shown to be more or less causally independent on the predictions of the model under scrutiny (Adebayo et al. 2018). In this work, the authors demonstrate that the saliency maps produced by gradient based methods demonstrate little change when replacing most of the layers of a convolutional network with randomly initialized weights. While this can be said to be a weakness of the aforementioned works, the reliance on this approach in the current work is also affected by proxy. This reviewer would have liked to seen applications of methods that are more robust in this regard, e.g., GradCAM or occlusion based methods, such as SHAP or LIME, particularly given the occlusion based approach of the study.
- W2: Table 1 shows accuracies over what is presumably the full ensemble of 10 models. Given the ensemble, is should be relatively straightforward to estimate the uncertainty of these predictions. This would, in this reviewers opinion, improve the presentation of the results, and provide the reader with an insight into the stability, reliability, and overall significance of the reported results.
- W3: The work seems to be heavily based on previous works, and could be presented with more clear delineation as a separate study, particularly in the abstract and introduction. On the first read through, the reader is left wondering as to the exact contributions of the current work. While the work "goes further" than the previous work, the extent of the scope of the current work is not too clear, particularly for researchers outside the niche of SIC forecasting. As it stands, the work is to be seen as an incremental study, as opposed to a significant step forward.
- W4: Given the nature of the study, it would be reasonable to ask for a robustness analysis of the model under more challenging perturbations. Given that convolutional models generally tend to be sensitive to perturbations, having estimates of the robustness of the trained models would ensure the reader that the results are not due to anomalies.
title: title: Sound manuscript with valuable evaluations and results for the machine learning sea ice coverage prediction audience
paper_summary: paper_summary: The manuscript presents a study that expands on the previously published machine learning model IceNet for sea ice concentration forecasts ranging from one through six months lead time in the northern hemisphere. Results suggest that a meaningful reduction of the number of input features (based on their importance score) can improve the forecast quality under normal conditions. In extreme situations and anomalies, though, the full set of input features results in an improved forecasts of sea ice cover compared to when using a reduced or minimal set of input features.
main_review: main_review: The authors carefully addressed and solved my concerns. During the rebuttal, one downside became clear, namely a large overlap of standard deviations for different configurations. This weakens the relevance of different input configurations. Nevertheless, the analyses and results are worth sharing with the community.
strength_weakness: strength_weakness: ### Structure
The manuscript would benefit from a structural rework. For example, the transition from Methodology to Results appears somewhat abrupt and the Methods section seems more like Related Work or Foundations. I'd thus suggest to rename section 2 into Related Work and Methodology (or to create two separate sections) and outline the manipulations or modifications this manuscript adds to prior work (such as currently contained at the beginning of the Results section). Also, the Methods section could be subsectioned to outline the conceptualizations of the two core aspects of this work: (a) retraining IceNet with a reduced set of features, and (b) investigating IceNet's generalization performance. The Results section could be subsectioned accordingly.

### Clarity
- Can authors please include information about how IceNet generates predictions out to 6 months? That is, does it run autoregressively, are separate models trained for each lead time, or does the model generate all outputs at once?
- Also, it is unclear how the Linear Trend Forecast models is designed and where the data comes from. In this vein, lines 104-105 could be extended to contain concrete information about the satellites that recored the data, where it is available, and how it has been preprocesses.
- Figures could benefit uppon inclusion of a grid. Once done, concrete values can be extracted from the Figures and Table 1 could be converted into a figure similar to Figure 4 to improve the presentation of results and make it more coherent.
- Does Figure 5 contain only extreme events (as suggested in lines 298-302)? This might be emphasized around lines 286-288 and also added to the figure caption. If Figure 5 does not only contain extreme events, can the authors mark extreme events explicitly?


### Minor remarks and typos
- Line 143 remove one "the"
- In Line 167, add "(land masks)" to the text again to repeat what is meant with metadata.
- When talking about less ensemble members compared to Andersson et al., please add the number of ensembles used there.
- Missing "s" at "drop" in line 316?