Question: The authors claim that the method can be used in interpretability-related tasks. Can you elaborate on that or give some examples?
Evidence: Overview & Motivation. To preserve the semantic meaning of conv layers, we represent each filter as a node, and link filters in consecutive layers via weighted edges (we define the weights below), as shown in Fig.. The motivation behind this approach is that for larger networks, there is an explosion of nodes by unrolling the convolutions (unrolled representation explained in § 2.3), and the integrity of a unit (a single filter or kernel) becomes untenable. Also, by mapping the nodes to specific entities of NNs, our proposed graph model is more interpretable than the unrolled model, and thus it is easier to interpret the outputs of downstream graph analysis on our graph representation (e.g., computing node features, tracking the evolution of the graph).
Detailed Description. Formally, let tensor K i be a kernel in layer i with f i filters, each with c i channels and dimensions h i ×w i . We use bracket notation to index into the kernel: for example, K i [l, : , :, :] indexes the l th filter of kernel K i . We create f i nodes representing each filter {v
fi }, with features defined as the corresponding biases in that layer. Let K j be the next convolutional layer defined analogously. While edge weights between neurons in FC layers are defined in the standard way, we define the edge weights between conv layers as the norm over each kernel's channels. The edge between node v (i) k representing the k th filter in layer i (i.e., K i [k, :, :, :]) and node/filter v
which is the norm of the k th channel of the l th filter in the j th layer. Though the proposed representations enable alternative edge weight configurations, we focus on the norm of filters as other studies, including those on pruning NNs, have demonstrated that this metric has strong correlation with filter importance. In the case of two conv layers, the resultant graph is an attributed bipartite graph with f i + f j nodes and f i × f j edges, where node attributes include flattened weight vectors or filter maps. Other information such as average gradients or the bias vector can also be used for node features.
We cover the most relevant work here, and dynamic graph mining in App. A.5. Some recent research efforts are devoted to modeling the NN architectures as graphs due to their topological identity and study their graph properties Rieck et al. (
Overview & Motivation. To preserve the seman_x005f_x0002_tic meaning of conv layers, we represent each filter as a node, and link filters in consecutive layers via weighted edges (we define the weights below), as shown in Fig. 3. The motivation behind this approach is that for larger networks, there is an explosion of nodes by unrolling the convolutions (unrolled rep_x0002_resentation explained in § 2.3), and the integrity of a unit (a single filter or kernel) becomes untenable. Also, by mapping the nodes to specific entities of NNs, our proposed graph model is more interpretable than the unrolled model, and thus it is easier to interpret the outputs of downstream graph analysis on our graph representation (e.g., computing node features, tracking the evolution of the graph).****We cover the most relevant work here, and dynamic graph mining in App. A.5. Some recent research efforts are devoted to modeling the NN architectures as graphs due to their topological identity and study their graph properties Rieck et al. (2019); Filan et al. (2020); You et al. (2020). For example, You et al. You et al. (2020) propose a relational graph representation to model message exchange between layers, and empirically show the common properties shared by NNs with significantly improved predictive performance in terms of graph clustering coefficients and average path lengths. Rieck et al. Rieck et al. (2019) propose a complexity measure related to NN performance—neural persistence—based on topological data analysis on weighted stratified graph. Filan et al. Filan et al. (2020) present an exploratory study of the NN modularity. Gebhart et al. (2019) proposes to compute persistent homology over the activation graph of an NN. The output is a graded set of subgraphs, which are shown to be related to the task-specific semantic that are captured by original NN.
Answer: We have updated our paper to phrase our statements regarding interpretability more clearly (Section 3.1 and Section 5); we realize that our previous claims could be confused with “NN interpretability”, a topic that has a different focus than our work.
Our main claim is that our proposed rolled graph representation is more interpretable than the unrolled representation---i.e., it is easier to understand what it represents. In our proposed rolled graph representation, the nodes are defined to represent filters in convolutional layers; on the other hand, the nodes represent feature maps in the unrolled representation introduced by (Rieck et al, 2019). Thus, we argue that it is easier to interpret the outputs of downstream graph analysis on our graph representation (e.g., computing node properties, tracking the evolution of the graph, performing community detection). For example, when we calculate the eigenvector centrality, it is clear which filters (from a specific layer) are the most influential filters; if we were to do the same calculation on the previously introduced unrolled representation, it would not be possible to draw such a conclusion. Furthemore, we believe that our graph representation can be a powerful tool for future studies that aim to understand the NN dynamics.