Title: EXPLAINING ALPHAGO: INTERPRETING CONTEXTU-AL EFFECTS IN NEURAL NETWORKS

Abstract: In this paper, we propose to disentangle and interpret contextual effects that are encoded in a pre-trained deep neural network. We use our method to explain the gaming strategy of the alphaGo Zero model. Unlike previous studies that visualized image appearances corresponding to the network output or a neural activation only from a global perspective, our research aims to clarify how a certain input unit (dimension) collaborates with other units (dimensions) to constitute inference patterns of the neural network and thus contribute to the network output. The analysis of local contextual effects w.r.t. certain input units is of special values in real applications. Explaining the logic of the alphaGo Zero model is a typical application. In experiments, our method successfully disentangled the rationale of each move during the Go game.

1. INTRODUCTION:

Interpreting the decision-making logic hidden inside neural networks is an emerging research direction in recent years. The visualization of neural networks and the extraction of pixel-level inputoutput correlations are two typical methodologies. However, previous studies usually interpret the knowledge inside a pre-trained neural network from a global perspective. For example, (Ribeiro et al., 2016; Lundberg & Lee, 2017; Kindermans et al., 2018) mined input units (dimensions or pixels) that the network output is sensitive to; (Bau et al., 2017) visualized receptive fields of filters in intermediate layers; (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015; Simonyan et al., 2013; Dosovitskiy & Brox, 2016; Fong & Vedaldi, 2017; Selvaraju et al., 2017) illustrated image appearances that maximized the score of the network output, a filter's response, or a certain activation unit in a feature map.

However, instead of visualizing the entire appearance that is responsible for a network output or an activation unit, we are more interested in the following questions.

• How does a local input unit contribute to the network output? Here, we can vectorize the input of the network into a high-dimensional vector, and we treat each dimension as a specific "unit" without ambiguity. As we know, a single input unit is usually not informative enough to make independent contributions to the network output. Thus, we need to clarify which other input units the target input unit collaborates with to constitute inference patterns of the neural network, so as to pass information to high layers. • Can we quantitatively measure the significance of above contextual collaborations between the target input unit and its neighboring units?

Method: Therefore, given a pre-trained convolutional neural network (CNN), we propose to disentangle contextual effects w.r.t. certain input units.

As shown in Fig. 1 , we design two methods to interpret contextual collaborations at different scales, which are agnostic to the structure of CNNs. The first method estimates a rough region of contextual collaborations, i.e. clarifying whether the target input unit mainly collaborates with a few neighboring units or most units of the input. This method distills knowledge from the pre-trained network into a mixture of local models (see Fig. 2 ), where each model encodes contextual collaborations within a specific input region to make predictions. We hope that the knowledge-distillation strategy can help people determine quantitative contributions from different regions. Then, given a model for local collaborations, the second method further analyzes the significance of detailed collaborations between each pair of input units, when we use the local model to make predictions on an image.

Application, explaining the alphaGo Zero model: The quantitative analysis of contextual collaborations w.r.t. a local input unit is of special values in some tasks. For example, explaining the alphaGo model (Silver et al., 2016; Hassabis & Silver, 18 October 2017) is a typical application.

The alphaGo model contains a value network to evaluate the current state of the game-a high output score indicates a high probability of winning. As we know, the contribution of a single move (i.e. placing a new stone on the Go board) to the output score during the game depends on contextual shapes on the Go board. Thus, disentangling explicit contextual collaborations that contribute to the output of the value network is important to understand the logic of each new move hidden in the alphaGo model.

More crucially, in this study, we explain the alphaGo Zero model (Hassabis & Silver, 18 October 2017) , which extends the scope of interests of this study from diagnosing feature representations of a neural network to a more appealing issue letting self-improving AI teach people new knowledge.

The alphaGo Zero model is pre-trained via self-play without receiving any prior knowledge from human experience as supervision. In this way, all extracted contextual collaborations represent the automatically learned intelligence, rather than human knowledge.

As demonstrated in well-known Go competitions between the alphaGo and human players (alp, Retrieved 17 March 2016; 2017-05-27) , the automatically learned model sometimes made decisions that could not be explained by existing gaming principles. The visualization of contextual collaborations may provide new knowledge beyond people's current understanding of the Go game. 

2. RELATED WORK:

Understanding feature representations inside neural networks is an emerging research direction in recent years. Related studies include 1) the visualization and diagnosis of network features, 2) disentangling or distilling network feature representations into interpretable models, and 3) learning neural networks with disentangled and interpretable features in intermediate layers.

Network visualization: Instead of analyzing network features from a global view (Wolchover, 2017; Schwartz-Ziv & Tishby, 2017; Rauber et al., 2016) , (Bau et al., 2017) defined six types of semantics for middle-layer feature maps of a CNN, i.e. objects, parts, scenes, textures, materials, and colors. Usually, each filter encodes a mixture of different semantics, thus difficult to explain.

Visualization of filters in intermediate layers is the most direct method to analyze the knowledge hidden inside a neural network. (Zeiler & Fergus, 2014; Mahendran & Vedaldi, 2015; Simonyan et al., 2013; Dosovitskiy & Brox, 2016; Yosinski et al., 2015; Dong et al., 2017; Zhang et al., 2016) showed the appearance that maximized the score of a given unit. (Dosovitskiy & Brox, 2016) used up-convolutional nets to invert CNN feature maps to their corresponding images.

Pattern retrieval: Some studies retrieved certain units from intermediate layers of CNNs that were related to certain semantics, although the relationship between a certain semantics and each neural unit was usually convincing enough. People usually parallel the retrieved units similar to conventional mid-level features (Singh et al., 2012) of images. (Zhou et al., 2015; 2016) selected units from feature maps to describe "scenes". (Simon & Rodner, 2015) discovered objects from feature maps.

Model diagnosis and distillation: Model-diagnosis methods, such as the LIME (Ribeiro et al., 2016) , the SHAP (Lundberg & Lee, 2017), influence functions (Koh & Liang, 2017) , gradientbased visualization methods (Fong & Vedaldi, 2017; Selvaraju et al., 2017) , and (Kumar et al., 2017) extracted image regions that were responsible for network outputs. (Vaughan et al., 2018; Zhang et al., 2018b) distilled knowledge from a pre-trained neural network into explainable models to interpret the logic of the target network. Such distillation-based network explanation is related to the first method proposed in this paper. However, unlike previous studies distilling knowledge into explicit visual concepts, our using distillation to disentangle local contextual effects has not been explored in previous studies.

Learning interpretable representations::

A new trend is to learn networks with meaningful feature representations in intermediate layers (Hu et al., 2016; Stone et al., 2017; Liao et al., 2016) in a weakly-supervised or unsupervised manner. For example, capsule nets (Sabour et al., 2017) and interpretable RCNN (Wu et al., 2017) learned interpretable middle-layer features. InfoGAN (Chen et al., 2016) and β-VAE (Higgins et al., 2017) learned meaningful input codes of generative networks. (Zhang et al., 2018a ) developed a loss to push each middle-layer filter towards the representation of a specific object part during the learning process without given part annotations.

All above related studies mainly focused on semantic meanings of a filter, an activation unit, a network output. In contrast, our work first analyzes quantitative contextual effects w.r.t. a specific input unit during the inference process. Clarifying explicit mechanisms of how an input unit contributes to the network output has special values in applications.

3. ALGORITHM:

In the following two subsections, we will introduce two methods that extract contextual collaborations w.r.t. a certain input unit from a CNN at different scales. Then, we will introduce the application that uses the proposed methods to explain the alphaGo Zero model.

3.1. DETERMINING THE REGION OF CONTEXTUAL COLLABORATIONS w.r.t. AN INPUT UNIT:

Since the input feature usually has a huge number of dimensions (units), it is difficult to accurately discover a few input units that collaborate with a target input unit. Therefore, it is important to first approximate the rough region of contextual collaborations before the unit-level analysis of contextual collaborations, i.e. clarifying in which regions contextual collaborations are contained.

Given a pre-trained neural network, an input sample, and a target unit of the sample, we propose a method that uses knowledge distillation to determine the region of contextual collaborations w.r.t. the target input unit. Let I ∈ I denote the input feature (e.g. an image or the state in a Go board).

Note that input features of most CNNs can be represented as a tensor I ∈ R H×W ×D , where H and W indicate the height of the width of the input, respectively; D is the channel number. We clip different lattices (regions) Λ 1 , Λ 2 , . . . , Λ N ∈ Λ from the input tensor, and input units within the i-th lattice are given as

I Λi ∈ R h×w×D , h ≤ H, w ≤ W . Different lattices overlap with each other.

The core idea is that we use a mixture of models to approximate the function of the given pretrained neural network (namely the teacher net), where each model is a student net and uses input information within a specific lattice I Λi to make predictions. More crucially, given different inputs, the importance for the same lattice may also change. For example, as shown in (Selvaraju et al., 2017) , the head appearance is the dominating feature in the classification of animal categories. Thus, if a lattice corresponds to the head, then this lattice will contribute more than other lattices, thereby having a large weight α i . Therefore, our method estimates a specific weight α i for each input I, i.e. α i is formulated as a function of I (which will be introduced later).

ŷ ≈ α1 • y1 + α2 • y2 + . . . + αn • yn (1)

Significance of contextual collaborations: Based on the above equation, the significance of contextual collaborations within each lattice Λ i w.r.t. an input unit can be measured as

s i . ∆ŷ ≈ α 1 • ∆y 1 Impacts from the first lattice Λ1 +α 2 • ∆y 2 + . . . + α n • ∆y n , s i = |α i • ∆y i | (2)

where we revise the value of the target unit in the input and check the change of network outputs, ∆ŷ = f (I new ) -f (I) and ∆y i = f i (I new Λi ) -f i (I Λi ). If contextual collaborations w.r.t. the target unit mainly localize within the i-th lattice Λ i , then α i • ∆y i can be expected to contribute the most to the change of ŷ.

We conduct two knowledge-distillation processes to learn student nets and a model of determining {α i }, respectively.

Student nets::

The first process distills knowledge from the teacher net to each student net f i with parameters θ i based on the distillation loss min θi I∈I y I,i -ŷI 2 , where the subscript I indicates the output for the input I. Considering that Λ i only contains partial information of I, we do not expect y I,i to reconstruct ŷI without any errors.

Distilling knowledge to weights: Then, the second distillation process estimates a set of weights α = [α I,1 , α I,2 , . . . , α I,n ] for each specific input I. We use the following loss to learn another neural network g with parameters θ g to infer the weight.

α = g(I), min θg Loss(θ g ), Loss(θ g ) = I∈I ∆ŷ I - N i=1 α I,i • ∆y I,i 2 (3) 3.2 FINE-GRAINED CONTEXTUAL COLLABORATIONS w.r.t. AN INPUT UNIT

In the above subsection, we introduce a method to distill knowledge of contextual collaborations into student nets of different regions. Given a student net, in this subsection, we develop an approach to disentangling from the student net explicit contextual collaborations w.r.t. a specific input unit u, i.e. identifying which input unit v collaborates with u to compute the network output.

We can consider a student net as a cascade of functions of N layers, i.e. x (l) = φ l (x (l-1) ) (or x (l) = φ l (x (l-1) ) + x (l-m) for skip connections), where x (l) denotes the output feature of the l-th layer. In particular, x (0) and x (n) indicate the input and output of the network, respectively. We only focus on a single scalar output of the network (we may handle different output dimensions separately if the network has a high-dimensional output). If the sigmoid/softmax layer is the last layer, we use the score before the softmax/sigmoid operation as x (n) to simplify the analysis.

3.2.1. PRELIMINARIES, THE ESTIMATION OF QUANTITATIVE CONTRIBUTION:

As preliminaries of our algorithm, we extend the technique of (Shrikumar et al., 2016) to estimate the quantitative contribution of each neural activation in a feature map to the final prediction. We use C x ∈ R H l ×W l ×D l to denote the contribution distribution of neural activations on the l-th layer x ∈ R H l ×W l ×D l . The score of the i-th element C xi denotes the ratio of the unit x i 's score contribution w.r.t. the entire network output score. Because x (n) is the scalar network output, it has a unit contribution C x (n) = 1. Then, we introduce how to back-propagate contributions to feature maps in low layers.

The method of contribution propagation is similar to network visualization based on gradient backpropagation (Mahendran & Vedaldi, 2015; Yosinski et al., 2015) . However, contribution propagation reflects more objective distribution of numerical contributions over {x i }, instead of biasedly boosting compacts of the most important activations.

Without loss of generality, in this paragraph, we use o = φ(x) to simplify the notation of the function of a certain layer. If the layer is a conv-layer or a fully-connected layer, then we can represent the convolution operation for computing each elementary activation score o i of o in a vectorized form1 o i = j x j w j + b. We consider x j w j as the numerical contribution of x j to o i . Thus, we can decompose the entire contribution of o i , C oi , into elementary contributions of x j , i.e.

C oi→xj = C oi • xj wj

oi+max{-b,0} , which satisfies C oi→xj ∝ x j w j (see the appendix for details). Then, the entire contribution of x j is computed as the sum of elementary contributions from all o i in the above layer, i.e.

C xj = i C oi→xj .

A cascade of a conv-layer and a batch-normalization layer can be rewritten in the form of a single conv-layer, where normalization parameters are absorbed into the conv-layer 1 . For skip connections, a neural unit may receive contributions from different layers, C x (l)

j = i C o (l+1) i →x (l) j + C x (l+m) j . If

the layer is a ReLU layer or a Pooling layer, the contribution propagation has the same formulation as gradient back-propagations of those layers 1 .

3.2.2. THE EXTRACTION OF CONTEXTUAL COLLABORATIONS:

As discussed in (Bau et al., 2017) , each neural activation o i of a middle-layer feature o can be considered as the detection of a mid-level inference pattern. All input units must collaborate with neighboring units to activate some middle-layer feature units, in order to pass their information to the network output.

Therefore, in this research, we develop a method to 1. determine which mid-level patterns (or which neural activations o i ) the target unit u constitutes; 2. clarify which input units v help the target u to constitute the mid-level patterns; 3. measure the strength of the collaboration between u and v.

Let o bfr and o denote the feature map of a certain conv-layer o = f (x) when the network receives input features with the target unit u being activated and the feature map generated without u being activated, respectively. In this way, we can use |o -o bfr | to represent the absolute effect of u on the feature map o. The overall contribution of the i-th neural unit C oi depends on the activation score o i , C oi ∝ max{o i , 0}, where max{o i , 0} measures the activation strength used for inference. The proportion of the contribution is affected by the target unit u can be roughly formulated as Co .

EQUATION

where C oi = 0 and thus Coi = 0 if o i ≤ 0, because negative activation scores of a conv-layer cannot pass information through the following ReLU layer (o is not the feature map of the last conv-layer before the network output).

In this way, Coi highlights a few mid-level patterns (neural activations) related to the target unit u. Co measures the contribution proportion that is affected by the target unit u. We can use Co to replace C o and use techniques in Section 3.2.1 to propagate Co back to input units x (0) . Thus, Cx (0) represents a map of fine-grained contextual collaborations w.r.t. u. Each element in the map Cx (0) j is given as x (0) j 's collaboration with u.

We can understand the proposed method as follows. The relative activation change |oi-o bfr i | oi can be used as a weight to evaluate the correlation between u and the i-th activation unit (inference pattern). In this way, we can extract input units that make great influences on u's inference patterns, rather than affect all inference patterns. Note that both u and v may either increase or decrease the value of o i . It means that the contextual unit v may either boost u's effects on the inference pattern, or weaken u's effects.

3.3. APPLICATION: EXPLAINING THE ALPHAGO ZERO MODEL:

We use the ELF OpenGo (Tian et al., 2018; 2017) as the implementation of the alphaGo Zero model. We combine the above two methods to jointly explain each move's logic hidden in the value net of the alphaGo Zero model during the game. As we know, the alphaGo Zero model contains a value net, policy nets, and the module of the Monte-Carlo Tree Search (MCTS). Generally speaking, the superior performance of the alphaGo model greatly relies on the enumeration power of the policy net and the MCTS, but the value net provides the most direct information about how the model evaluates the current state of the game. Therefore, we explain the value net, rather than the policy net or the MCTS. In the ELF OpenGo implementation, the value net is a residual network with 20 residual blocks, each containing two conv-layers. We take the scalar output2 before the final (sigmoid) layer as the target value to evaluate the current state on the Go board.

Given the current move of the game, our goal is to estimate unit-level contextual collaborations w.r.t. the current move. I.e. we aim to analyze which neighboring stones and/or what global shapes help the current move make influences to the game. We distill knowledge from the value net to student networks to approximate contextual collaborations within different regions. Then, we estimate unitlevel contextual collaborations based on the student net.

Determining local contextual collaborations::

We design two types of student networks, which receive lattices at the scales of 13 × 13 and 10 × 10, respectively. In this way, we can conduct two distillation processes to learn neural networks that encode contextual collaborations at different scales.

As shown in Fig. 2 , we have four student nets {f i |i = 1, . . . , 4} oriented to 13 × 13 lattices. Except for the output, the four student nets have the same network structure as the value net. The four student nets share parameters in all layers. The input of a student net only has two channels corresponding to maps of white stones and black stones, respectively, on the Go board. We crop four overlapping lattices at the four corners of the Go board for both training and testing. Note that we rotate the board state within each lattice I Λi to make the top-left position corresponds to the corner of the board, before we input I Λi to the student net. The neural network g has the same settings as the value net. g receives a concatenation of [I Λ1 , . . . , I Λ4 ] as the input. g outputs four scalar weights {α i } for the four local student networks {y i }. We learn g via knowledge distillation.

Student nets for 10×10 lattices have similar settings as those for 13×13 lattices. We divide the entire Go board into 3 × 3 overlapping 10 × 10 lattices. Nine student nets encode local knowledge from nine local lattices. We learn another neural network g, which uses a concatenation of [I Λ1 , . . . , I Λ9 ] to weight for the nine local lattices.

Finally, we select the most relevant 10 × 10 lattice and the most relevant 13 × 13 lattice, via max i s i , for explanation.

Estimating unit-level contextual collaborations: In order to obtain fine-grained collaborations, we apply the method in Section 3.2.2 to explain two student nets corresponding to the two selected relevant lattices. We also use our method to explain the value net. We compute a map of contextual collaborations for each neural network and normalize values in the map. We sum up maps of the three networks together to obtain the final map of contextual collaborations Ĉ.

More specifically, given a neural network, we use the feature of each conv-layer to compute the initial Co in Equation ( 4) and propagated Co to obtain a map of collaborations Cx (0) . We sum up maps based on the 1st, 3rd, 5th, and 7th conv-layers to obtain the collaboration map of the network.

4. EXPERIMENTS:

In experiments, we distilled knowledge of the value network to student nets, and disentangled finegrained contextual collaborations w.r.t. each new move. We compared the extracted contextual collaborations and human explanations for the new move to evaluate the proposed method.

4.1. EVALUATION METRIC:

In this section, we propose two metrics to evaluate the accuracy of the extracted contextual collaborations w.r.t. the new move. Note that considering the high complexity of the Go game, there is no exact ground-truth explanation for contextual collaborations. Different Go players usually have different analysis of the same board state. More crucially, as shown in competitions between the alphaGo and human players (alp, Retrieved 17 March 2016; 2017-05-27) , the knowledge encoded in the alphaGo was sometimes beyond humans' current understanding of the Go game and could not be explained by existing gaming principles.

In this study, we compared the similarity between the extracted contextual collaborations and humans' analysis of the new move. The extracted contextual collaborations were just rough explanations from the perspective of the alphaGo. We expected these collaborations to be close to, but not exactly the same as human understanding. More specifically, we invited Go players who had obtained four-dan grading rank to label contextual collaborations. To simplify the metric, Go players were asked to label a relative strength value of the collaboration between each stone and the target move (stone), no matter whether the relationship between the two stones was collaborative or adversarial. Considering the double-blind policy, the paper will introduce the Go players if the paper is accepted.

Let Ω be a set of existing stones except for the target stone u on the Go board. p v ≥ 0 denotes the labeled collaboration strength between each stone v ∈ Ω and the target stone u. q v = | Ĉv | is referred to as the collaboration strength estimated by our method, where Ĉv denotes the final estimated collaboration value on the stone v. We normalized the collaboration strength, pv = p v / v p v , qv = q v / v q v and computed the Jaccard similarity between the distribution of p and the distribution of q as the similarity metric.

In addition, considering the great complexity of the Go game, different Go players may annotate different contextual collaborations. Therefore, we also required Go players to provide a subjective rating for the extracted contextual collaborations of each board state, i.e. selecting one of the five ratings: 1-Unacceptable, 2-Problematic, 3-Acceptable, 4-Good, and 5-Perfect.

4.2. EXPERIMENTAL RESULTS AND ANALYSIS:

Fig. 3 shows the significance of the extracted contextual collaborations, as well as possible explanations for contextual collaborations, where the significance of the stone v's contextual collaboration was reported as the absolute collaboration strength q v instead of the original score Ĉv in experiments. Without loss of generality, let us focus on the winning probability of the black. Considering the complexity of the Go game, there may be two cases of a positive (or negative) value of the collaboration score Ĉv . The simplest case is that when a white stone had a negative value of Ĉv , it means that the white stone decreased the winning probability of the black. However, sometimes a white stone had a positive Ĉv . It may be because that this white stone did not sufficiently exhibit its power due to its contexts. Since the white and the white usually had a very similar number of stones in the Go board, putting a relatively ineffective white stone in a local region also wasted the opportunity of winning advantages in other regions in the zero-sum game. Similarly, the black stone may also have either a positive or a negative value of Ĉv .

The Jaccard similarity between the extracted collaborations and the manually-annotated collaborations was 0.3633. Nevertheless, considering the great diversity of explaining the same game state,

Result:

The black stone at (7,2) has a high value, because it collaborates with the new stone to escape from the surrounding of the white. The white stone at (7,4) has a high value, because it is about to be eaten by the new black stone. The black stone at (8,4) has a high value, because it collaborates with the new stone to eat the white stone at (7,4).

(0,0) (18, 18) Explanations for the estimated collaborations (0,0) (18, 18) The black stone at (8,6) has a high value, because it collaborates with the new stone, because it collaborates with the new black stone to get a head out of the white's regime. The two black stone also communicate with black stones on the top. The white stone at (7,7) has a high value, because future white stones can only be placed to the right to escape from the regime of the new black stone.

(0,0) (18, 18) The black stone at (2,7) has a high value, because it collaborates with the new black stone to separate white stones into the left and right groups, which increases the probability of attacking white stones on the left in the future. The white stone at (1,6) has a high value, because the new black stone reduces the white stone's space of "making eyes" in the future.

(0,0) (18, 18) The black stone at (4,3) has a high value, because the new black stone helps this black stone to get a head out of the white's regime. The white stone at (5,2) has a high value, because the new black stone limits the potential of the white's future development in its neighboring area. the average rating score that was made by Go players for the extracted collaborations was 3.7 (between 3-Acceptable and 4-Good). Please see the appendix for more results.

5. CONCLUSION AND DISCUSSIONS:

In this paper, we have proposed two typical methods for quantitative analysis of contextual collaborations w.r.t. a certain input unit in the decision-making of a neural network. Extracting fine-grained contextual collaborations to clarify the reason why and how an input unit passes its information to the network output is of significant values in specific applications, but it has not been well explored before, to the best of our knowledge. In particular, we have applied our methods to the alphaGo Zero model, in order to explain the potential logic hidden inside the model that is automatically learned via self-play without human annotations. Experiments have demonstrated the effectiveness of the proposed methods.

Note that there is no exact ground-truth for contextual collaborations of the Go game, and how to evaluate the quality of the extracted contextual collaborations is still an open problem. As a pioneering study, we do not require the explanation to be exactly fit human logics, because human logic is usually not the only correct explanations. Instead, we just aim to visualize contextual collaborations without manually pushing visualization results towards human-interpretable concepts. This is different from some previous studies of network visualization (Mahendran & Vedaldi, 2015; Yosinski et al., 2015) that added losses as the natural image prior, in order to obtain beautiful but biased visualization results. In the future, we will continue to cooperate with professional Go players to further refine the algorithm to visualize more accurate knowledge inside the alphaGo Zero model.

SUPPLEMENTARY MATERIALS FOR THE CONTRIBUTION PROPAGATION:

Let o = ω ⊗ x + β denote the convolutional operation of a conv-layer. We can rewrite the this equation in a vectorized form as o = W x + b, o, b ∈ R 1×N , W ∈ R N ×N . For each output element o i , o i = j x j W ij + b i . If the conv-layer is a fully-connected layer, then each element W ij corresponds to an element in ω. Otherwise, W is a sparse matrix, i.e. W ij = 0 if o i and x j are too far way to be covered by the convolutional filter. Thus, we can write o i = j x j w j + b to simplify the notation. Intuitively, we can propagate the contribution of o i to its compositional elements x j based on their numerical scores. Note that we only consider the case of o i > 0, because if o i ≤ 0, o i cannot pass information through the ReLU layer, and we obtain C oi = 0 and thus C oi→xj = 0. In particular, when b ≥ 0, all compositional scores just contribute an activation score o i -b, thereby receiving a total contribution of C oi For ReLU layers and Pooling layers, the formulation of the contribution propagation is identical to the formulation for the gradient back-propagation, because the gradient back-propagation and the contribution propagation both pass information to neural activations that are used during the forward propagation. 

:

Please see the Appendix for details.

The value net uses the current state, as well as seven most recent states, to output eight values for the eight states. To simplify the algorithm, we take the value corresponding to the current state as the target value.

Figures and Tables:

Figure 1: Explaining the alphaGo model. Given the state of the Go board and the next move, we use the alphaGo model to explain the rationale of the move. We first estimate a rough region of contextual collaborations w.r.t. the current move by distilling knowledge from the value net to student nets that receive different regions of the Go board as inputs. Then, given a student net, we analyze fine-grained contextual collaborations within its region of the Go board. In this figure, we use a board state from a real Go game between humans for clarity.

Figure 2: Division of lattices for two types of student nets. We distill knowledge from the value net into a mixture of four/nine student nets to approximate decision-making logic of the value net.

Figure 3: Significance of contextual collaborations w.r.t. the new black stone (the black star). Go players provided possible explanations for contextual collaborations. The red/blue color indicates a significant/insignificant contextual collaboration. Please see the appendix for more results.

b < 0, we believe the contribution of C oi all comes from elements of {x j }, and each element's contribution is given a C oi • xj wj oi-b . Thus, we getC oi→xj = C oi • x j w j o i + max{-b, 0}When a batch-normalization layer follows a conv-layer, then the function of the two cascaded layers can be written as o i =γ ( j x j w j + b)can absorb parameters for the batch normalization into the conv-layer, i.e. w j ←

Figure 5: We compared the extracted contextual collaborations at different scales (the second, third, fourth, and fifth columns) with annotations made by Go players.

Peer review (iclr2019):
 The paper uses a mixture of student models to learn from the pre-trained AlphaGoZero model (ELF OpenGo). Each student model only covers a faction of the board (13x13 or 10x10). With the student model, given one referenced stone u, its relationship with its surrounding stones is studied (i.e., contextual collaboration), by checking the difference between activation with/without the reference stone u. The relationship is in terms of values. High-value means the connection favors black and/or damaging white. Based on the proposed approach, the paper gives case studies of multiple situations in the Go game, as well as human player evaluation. The paper is somehow confusing, leaving a lot of issues to be addressed. In the first part of the paper, student models are trained to mimic the teacher*s output, after that student models are used to compute the contextual collaboration using some metric (e.g. how strong the role is played by one activation). I didn*t see why not just use the original 19x19 model directly to understand contextual collaboration (which should be much more accurate since ELF OpenGo is quite strong), and what would be the difference compared to using a model with smaller receptive field. Also how good is the local model in predicting the value function? The value function of Go is a global quantity and a few changes in the local stone patterns will completely change the value from black 100% win to white 100% win. Such information cannot be carried by a single value output by the local students (Eqn. 2 and 3). The experiments are unconvincing. There are a few visualizations, a few human evaluations, but the conclusion is a bit vague. What to get from the paper? Did we have a better understanding of the model? From Fig. 3 it looks like the model indeed captures some common basic patterns in the game, but this is also the case for methods without convolutional network (e.g., "Bayesian Pattern Ranking for Move Prediction in the Game of Go"). The sentence in (footnote 2, page 6: "The value net uses the current state, as well as seven most recent states, to output eight values for the eight states. To simplify the algorithm, we take the value corresponding to the current state as the target value ") is not right. AlphaGoZero model (and ELF OpenGo) takes the current state and the last 7 most recent states as input, and only output a single value for the current state. Overall I feel that the paper made an interesting initial attempt but there are still a lot of work to do. Detailed questions: 1. Experiments: - How many board situations are annotated? - It seems the heatmaps are very different yet the players gives perfect score. Is this metric too subjective? - For 13x13 splitting, the four sub-boards are rotated so that all corners are aligned with top-left. How about the nine 10x10 boards? The sub-board in the middle of the 19x19 board should not behave the same as the sub-board at the corners. 2. The paper shows stones that have strong correlations with the probe stone. Are there other stones in similar situations but are not picked up by this method? 3. What is the error/loss to estimate the whole board values using local region values? If the error is high then using student model for analysis might raise issues. 4. Can you show weights and values learned by other subregions to show how discriminative they are? 5. All examples are in the middle of some unfinished sequences, so just based on last move it will be a very strong baseline (e.g. this will be easily learned by convnets even if not specifically provided.) Can you provide some more game states such that the target move is far away from the last move region? (Applicable to both experiments) Minor issues: in 4.2 the white and the white -> the black and the white