## Model Interpretation Considering Both Time and Frequency Axes Given Time Series Data  
**[PERSON] 1\({}^{\copyright}\)**, **[PERSON] 2\({}^{\copyright}\)**, **[PERSON] 2\({}^{\copyright}\)** and **[PERSON] 2\({}^{\copyright,\copyright}\)**  
Footnote 1: footnotemark:  
Footnote 2: footnotemark:  
Footnote 3: footnotemark:  
Footnote 4: footnotemark:  
Footnote 5: footnotemark:  
Footnote 6: footnotemark:  
Footnote  
+++ ==WARNING: Truncated because of repetitions==
8: footnotemark:  
++  
Since the interpretation methods for deep learning have been proposed, such as Grad-CAM [18] the gradient-based visualization, KernelSHAP [19] the Shapley value-based method from game theory and LIME, which exploits surrogate model [20], the interpretations are also applied to the medical domain [21; 22; 23]. However, Grad-CAM-based visualization is only appropriate for image data. Besides, the exact Shapley value is difficult to earn, because the Shapley value is calculated from all combinations of the features [19]. Therefore, for one-dimensional data, LIME is utilized in the state-of-the-art interpretations, which are named NEVES [24] and LIMESegment [25]. Given a target model that is to be explained and the input data, LIME exploits a surrogate model that is considered easy to interpret, such as linear model or decision tree [20]. The explanation is obtained from training the surrogate model using perturbed samples and associated logits predicted by the target model. The perturbed samples are a randomly permuted subset of the input. For example, given an image of skin cancer, the image can be divided into lesions and normal skin of the image. Then, the perturbed samples can be the image of lesions, the image of normal skin and the image of both. Because the characteristics of LIME, the explanation depends on how to split the input instance. Former works exploit LIME, split the input time series at the same interval and fill it with interpolation [24] or divide from the change point, which is the boundary of the most dissimilar between window slides [25]. They are, however, limited in that they consider the signal changing along the time axis whereas physicians consider the characteristics of signals along the time and frequency axes in practice [26; 27; 28; 29]. In this work, we propose the method to interpret models based on deep learning given time series medical data. Specifically, we consider the medical signals, which are that not only the change point along the time axis but also the frequency plays an important role in diagnoses such as electroencephalography (EEG), electromyography (EMG) and electrocardiography (ECG) [30; 31; 32]. Given the input signal, our suggestion considers both time and frequency axes. We transform the input signal using short-time Fourier transform (STFT) and divide it into aspects of both axes. The divided signal is randomly permuted and inverted by inverse STFT (ISTFT). Finally, our method produces the explanation of the input as depicted in Figure 1. In the experiments, we show that our method figures out a significantly more critical explanation than the state-of-the-art and it generates a comparative explanation as a physician's decision. Our contributions are as follows:  
* We propose the novel approach to interpret a deep learning classifier given medical signals such as EEG considering both aspects of time and frequency while physicians consider both of them to diagnose in practice. * In the experiments, we confirm the suggestion from this work with the real-world dataset, which is EEG signals recorded from patients during polysomnogrphic studies. * We show that our suggestion captures the probable explanations such as K-complexes and delta waves, which are considered strong evidence of the second sleep stage and the third sleep stage, respectively. The rest of our paper is organized as follows. In Section 2, we review existing interpretations for deep learning models and their limitations in the general and medical domains. We define our problem and introduce our method in Section 3.