## 3 Attempts at Model Interpretation
### DNN Model Interpretation
#### 3.2.2 Global Interpretation  
A second approach tries to 'uncover qualities of the data at large that affect model behaviors' [4]. For example, it is analyzed how input features propagate within a neural network [20; 23], or how models transform input features using mechanisms similar to basis changes in vector spaces [24; 25]. Another more recent approach seeks to characterize dNN nodes in terms of activation patterns to interpret or visualize how the models operate [26; 27]. Analyses of this sort are often performed for dNNs that operate on image data or that play games which are presented to the human player as a series of images. This is because here results of the analysis can be visualized and thereby interpreted in some sense by appealing to visual intuition, by analogy with the way we describe the perceptual environment by appealing to its affordance character [28]. Predictor Decomposition  
[PERSON] et al. [23] proposed a recursive mechanism to map the predictions of a functional \(f\) from its range to its domain space, i.e., they defined a basis-changing inverse functional \(g:\mathbb{R}\to\mathbb{R}^{k}\) (see Equation (1)) with \(g(f(\mathbf{x}))=\mathbf{r}\in\mathbb{R}^{k}\), constrained as follows:  
\[\sum_{p=1}^{k}r_{p}=f(\mathbf{x}), \tag{5}\]  
where \(k\) is again the dimension of the domain of \(f\) and each \(r_{p}\) is obtained by summing over a subset of the nodes of a neural network. The vector \(\mathbf{r}=(r_{1},\ldots,r_{p})\) can be understood as a decomposition (basis change) of the range of the function that is obtained by harvesting and summarising the structure of the neural network. It can be used to visualize prediction-relevant features in the input and in the layers of the neural network to obtain'relevance maps' (heatmaps) that indicate which features of the input space contribute to the classification output of the network [20] (Suppl. 3.2-3.3). Such maps can be inspected to identify which features of the image are used by the classification algorithm. [PERSON] et al. went one step further and clustered the heatmaps from classification tasks to obtain sets of images with similar classification-relevant properties [20] (Methods Section). The inspection and clustering of heatmaps generated from image-classification and Atari-game playing dNNs revealed the features that the dNN had learned. For example, it was shown that dNNs learn to classify horses from notes printed on pictures that designate them as such. If these notes are transposed to pictures of cars, then the cars get classified as horses as well. In another setting, the authors identified that the dNN uses the table nudging mechanism built into a virtual pinball game to move the ball without using the flippers at all in order to obtain an infinite number of points. This strategy would not work if the simulation was adapted to model a real pinball machine more closely, for there the machine tilts upon repeated manipulation of the table in order to thwart the player's attempt to manipulate the game. Node and Activation Characterisations  
Several groups have proposed procedures to assign descriptive concepts to individual nodes of a neural network in order to provide semantics that would make them interpretable for users [26; 29]. For example, nodes used in the classification of images of water landscape pictures can be associated with concepts such as blue or river or water. As proposed by [PERSON] et al., dNN nodes can either be associated with atomic concepts \(\mathcal{C}\) from a vocabulary \(\mathcal{C}\)[29] or they can be assigned to formulae \(\mathcal{L}(\mathcal{C})\) consisting of combinations of concepts obtained by applying connectives (\(\wedge,\vee,\) -) of propositional logic over \(\mathcal{C}\) to yield node interpretation semantics more adequate to the behavior of input-distal nodes [26]. For example, a node can be associated with a propositional formula such as \((\texttt{water}\vee\texttt{river})\wedge\texttt{-blue})\).