Objective Evaluation of Deep Visual Interpretations on Time Series Data

1 INTRODUCTION . Due to its high performance on complex multi-modal data , deep learning ( DL ) becomes increasingly popular in many real-world applications that process time series data ( Fawaz et al. , 2019b ) . While we fundamentally rely on their classification accuracy in many safety-relevant applications ( Berkenkamp et al. , 2017 ) they remain difficult to interpret . Typical applications are the monitoring of industrial processes ( Löffler et al. , 2021 ) , the support of health care and sports ( Dorschky et al. , 2020 ) , or safety in autonomous driving ( Schmidt et al. , 2021 ) . The need for improved model understanding ( Carvalho et al. , 2019 ) , along with regulatory guidelines ( Goodman & Flaxman , 2017 ) , led to a myriad of new approaches to the visual interpretation problem ( Zhang & Zhu , 2018 ) . Post-hoc visual interpretation allows a domain expert to validate and understand how ( almost ) arbitrary deep learning models operate . Their central idea lies in highlighting features on the input that are ” relevant ” for the prediction of a learned model ( Adebayo et al. , 2018 ) . Many of these techniques do not require a modification of the original model ( Simonyan et al. , 2014 ; Ribeiro et al. , 2016 ) and are compatible with different architectures , which makes them useful as a general-purpose validation tool for neural networks across different tasks ( Arrieta et al. , 2020 ) . However , while visual interpretation yields intuitive and correct explanations for images ( Samek et al. , 2021 ) , the application of these methods on time series data is still an unsolved problem ( Rojat et al. , 2021 ) . Time series inherently are more diverse ( Rojat et al. , 2021 ) ( because they may originate from a variety of different sensors and processes ) , and often do not allow an obvious patch- or texture-based localization of critical features for human observers . This makes the application and the evaluation of visual interpretability methods difficult . A domain expert can not easily judge if explanations are correct in ( i ) explaining the reason for the decision process in the DL model and in ( ii ) capturing the real features in the dataset that lead to a correct classification . Hence , it is important not to blindly apply different visualization methods . This requires quality metrics that evaluate visual interpretations and that enable an expert to select a suitable visualization for a given model and task at hand . However , both state-of-the-art visualization techniques and metrics that evaluate visual interpretations ( e.g . Pixel Flipping ( Samek et al. , 2017 ) , Sanity Check ( Adebayo et al. , 2018 ) , and sensitivity checks ( Rebuffi et al. , 2020 ) ) have so far only been examined on images ( Rojat et al. , 2021 ) or on NLP tasks ( Arras et al. , 2017 ) . This lack of objective and subjective evaluation seriously limits the application and utility of them for time series . In this paper , we propose to evaluate visual interpretations for time series combining six orthogonal metrics : ” sanity ” ( Adebayo et al. , 2018 ) , ” faithfulness ” ( Alvarez Melis & Jaakkola , 2018 ) , ” sensitivity ” ( Rebuffi et al. , 2020 ) , ” robustness ” ( Yeh et al. , 2019 ) , ” stability ” ( Fel & Vigouroux , 2020 ; Li et al. , 2021 ) , and a novel metric based on human preferences : ” localization ” . These metrics rate and validate distinct qualities of saliency . Our metrics are both based on the functional perspective , i.e. , based on the model-specific operation , and on how well they represent annotators ’ semantics . In an extensive evaluation , we train four different architectures on two different types of tasks : UTime model ( Perslev et al. , 2019 ) and bidirectional Long Short-Term Memory ( bi-LSTM ) ( Schuster & Paliwal , 1997 ) on segmentation tasks , and Fully Convolutional Network ( FCN ) ( Long et al. , 2015 ) and Temporal Convolutional Network ( TCN ) ( Bai et al. , 2018 ) on classification tasks . We use diverse datasets from the UCR repository ( Dau et al. , 2018 ) ( GunPointAgeSpan , FordA , FordB , ElectricDevices , MelbournePedestrian , NATOPS ) and for segmentation the more complex real-world tool tracking dataset ( Löffler et al. , 2021 ) . The experiments show the necessity of all categories to create an objective rating for methods , models and tasks , and allow domain experts to understand , rate , and validate saliency for time series in safety-critical applications . The rest of the paper is structured as follows . Section 2 discusses background and related work . Section 3 introduces extended and novel metrics for both the classification and segmentation task . Section 4 discusses the experimental results and Section 5 proposes recommendations . 2 BACKGROUND AND RELATED WORK . Interpretation methods for DL models can be divided into ante-hoc methods , i.e. , methods that are inherently part of the model , and post-hoc methods , i.e. , methods that provide the interpretation after training ( Rojat et al. , 2021 ) . We focus on post-hoc methods and divide them into gradient-based and perturbation-based methods ( Li et al. , 2021 ; Warnecke et al. , 2020 ; Ismail et al. , 2020 ) . Gradient-based methods compute the relevance for all input features by passing gradients backwards through the neural network ( Ancona et al. , 2018 ) . Gradient ( Simonyan et al. , 2014 ) computes a class c ’ s saliency map M c using the derivative of the class score P c of the model with respect to the input sample x , as M c ( x ) = ∂P c ∂x . The gradient indicates the importance of points in the input sequence for predictions . The advantage of gradient-based methods lies in their computational efficiency , as they use only a small number of backward passes to compute M c ( x ) . Perturbation-based methods perturb known input samples and measure the effects of specific perturbations on the predicted class via a forward pass through the network . For instance , Local Interpretable Model-Agnostic Explanations ( LIME ) ( Ribeiro et al. , 2016 ) fits a local surrogate model ( e.g. , a linear regression ) as an explanation , and calculates relevance based on this surrogate . Perturbation-based methods are computationally expensive as they require multiple forward passes per sample . However , they do not need gradient information and work with black-box models . 2.1 METRICS FOR SALIENCY ON TIME SERIES . Most interpretation methods were originally designed for image or text data . Understanding and comparing visual interpretation is intuitive on image data , compared to more abstract time series . Furthermore , the diversity of interpretation methods complicates an objective choice for the model and task ( Rojat et al. , 2021 ) . For example , when Wang et al . ( 2017 ) and Fawaz et al . ( 2019b ) apply Class Activation Maps ( CAM ) ( Zhou et al. , 2016 ) on well-known UCR datasets ( Dau et al. , 2018 ) , they notice a qualitative difference of CAM interpretations between network architectures . Similarly , other work relies on domain experts that perform a ( costly ) qualitative evaluation ( Strodthoff & Strodthoff , 2019 ; Fawaz et al. , 2019a ; Oviedo et al. , 2019 ) . Hence , there is an increasing need for objective evaluation metrics to make the interpretations measurable and comparable . Metrics . There exists a large variety of different metrics . First , the stability of saliency maps can be evaluated with respect to the sensitivity of the majority class ( Yeh et al. , 2019 ) . This metric evaluates the change of interpretation when the input samples are attacked by adversarial noise . It is a special form of perturbation , that was so far only applied to image data . Second , Rebuffi et al . ( 2020 ) measure class sensitivity by comparing saliency maps of a min- and max-class . Third , Cho et al . ( 2020 ) perturb the unimportant input samples and preserve only the important inputs . If the predictions are stable compared to the unperturbed input , the interpretation is robust . Similarly , Ates et al . ( 2021 ) evaluate the local robustness ( i.e. , similar samples should lead to similar interpretations ) of the interpretation by measuring the ratio of change of the interpretation compared to the amount of input perturbation . Finally , faithfulness is a perturbation-based metric that aims to measure the relationship between saliency and predictive features ( Alvarez Melis & Jaakkola , 2018 ) . Schlegel et al . ( 2019 ) focus on continuous univariate time series . Sub-sequences with high relevance values are perturbed either by swapping their samples or replacing them with a mean value . Similarly , Ismail et al . ( 2020 ) propose input perturbations to evaluate models on synthetic time series . As they perturb inputs point-wise instead of sub-sequences their metric does not consider time dependency between consecutive points . Our faithfulness metric perturbs sub-sequences of high saliency ( Schlegel et al. , 2019 ) . We adapt each type of metric into our set to provide a visual interpretation of time series . Categories of metrics . Metrics can be divided into different categories , depending on the question they answer . Doshi-Velez & Kim ( 2017 ) propose a distinction between human-grounded and functional metrics . The former involve human perception and intuition with the goal of generating qualitative , intuitive visualizations , e.g. , bounding boxes for image object detection for testing the localization of saliency maps ( Jianming et al. , 2016 ) , or questionnaires to indicate testers ’ opinions on the quality of explanations ( Li et al. , 2021 ) . Functional metrics provide statistical measures , e.g. , to aggregate performance metrics automatically ( Rojat et al. , 2021 ) , and make use of proxy tasks to generate a quantitative evaluation . Another categorization ( Li et al. , 2021 ) identifies multiple broad categories for image data , that we may transfer to time series , i.e. , the faithfulness of salient features , the class sensitivity of an explanation , and the stability of explanations given noisy inputs . This paper proposes a framework of metrics with orthogonal categories , specifically to time series . We adapt and extend metrics to ( multivariate ) time series and propose an intra-class stability metric and a concept of relevance localization : we build on the pointing game ( Jianming et al. , 2016 ) and combine it with the precision and recall for time series framework ( Tatbul et al. , 2018 ) . To our knowledge we are the first to apply and evaluate sanity check ( Adebayo et al. , 2018 ) on time series . 3 SCORING CATEGORIES . We propose a set of six distinct categories ( sanity , faithfulness , sensitivity , robustness , stability , and localization ) to assess visual interpretation methods and to determine their performance and trustworthiness in classification or segmentation tasks on time series . For each of them we propose a metric that enables a comparative evaluation of diverse types of visual interpretation methods . Why do we need six scoring categories ? It seems tempting to rely on a single metric or on a single aggregated score across multiple metrics to assess the quality of a visual interpretation . However , we show that the six presented categories are inherently orthogonal ( see Fig . 2 ) and capture distinct qualities . Interpretations depend on model parameters ( sanity check ( Adebayo et al. , 2018 ) ) , predictive features ( faithfulness ( Alvarez Melis & Jaakkola , 2018 ) ) , coherence of class predictions ( intra-class stability ( Fel & Vigouroux , 2020 ) ) , robustness against noise ( max sensitivity with adversarial noise ( Yeh et al. , 2019 ) ) , specific sequences ( or even points ) in a time series like by Tatbul et al . ( 2018 ) ( novel localization metric ) , and the relevance map ’ s specificity ( interclass sensitivity ( Rebuffi et al. , 2020 ) ) . It is important to assess if a given interpretation accurately captures these dependencies . We give a brief overview of our frameworks ’ functional ( Fig . 1a-e ) and human-grounded metrics ( Fig . 1f ) that we evaluate with time series data . ( a ) sane saliency depends on network parameters and is structurally different after randomizing the network ’ s weights ρi for layers [ 1 , 2 , 3 ] ( Adebayo et al. , 2018 ) . ( b ) faithful saliency correlates with predictive accuracy and perturbing a percentage of the input sequence with high saliency decreases accuracy ( Alvarez Melis & Jaakkola , 2018 ) . ( c ) saliency is sensitive when the predicted ( max ) class in one sample is sufficiently different from any other ( min ) class ( Rebuffi et al. , 2020 ) . ( d ) saliency is robust , if small changes to the input cause only small changes to the saliency ( Yeh et al. , 2019 ) . ( e ) saliency is stable if it has a low variance and standard deviation for all samples of a class , with respect to a suitable distance metric ( Fel & Vigouroux , 2020 ) . ( f ) saliency should be localized on the predicted classes segment ( t0 to t1 ) . We will define those categories on time series after introducing a unified notation . Notation . We follow the notation adapted from Fawaz et al . ( 2019b ) : A multivariate time series is defined byX = [ X1 , ... , XH ] , whereH is the number of input channels , Xi = ( xi1 , ... , x i T ) ∈ RT is an ordered set of real values , and T denotes the number of time steps . For H equal to 1 , we consider a univariate time series , otherwise we consider a multivariate time series . Time series often include complex temporal dependencies , i.e. , distinct points are not independent . Time series classification defines a mapping X → y that minimizes the error on a dataset D = { ( X1 , y1 ) , ... , ( XN , yN ) } , where N is the number of data samples , X ∈ D is a time series , yi ∈ RC denotes the one-hot vector of a class label the input belongs to , and C is the number of classes . In time series segmentation , we search X → Y that maps an input sample to a dense classification Y = [ y1 , ... , yT ] ∈ RC×T ( Perslev et al. , 2019 ) , i.e. , a class label is predicted for each time step . Post-hoc visual interpretation methods compute a relevance map M cm ∈ RH×T , M cm ( X ) = [ R1 , ... , RH ] , where Ri = ( ri1 , ... , r i T ) , representing the importance of each input feature with respect to the class c and a model m , for each time step . We use M as a function to produce the saliency map . For clarity , we will omit the dependency onm , i.e. , M cm ≡M c , if it is not explicitly required . An evaluation metric for visual interpretation methods defines a score Smetric ( · ) that rates the quality of the relevance map M at a sample X given a model m and optional parameters . We provide a unified view , so that for all scores , a higher score corresponds to a better visualization according to the perspective .

Summary: This study deals with the issue of interpretability in the analysis of time series data using deep learning models. Six metrics for evaluating interpretation methods are introduced and nine existing interpretation methods are evaluated. One of these metrics is proposed by the author. One of the metrics is based on an experimental evaluation, which shows that none of the methods is superior in all the metrics.