Title: Chess Game Concepts Emerge under Weak Supervision: A Case Study of Tic-tac-toe
Authors: Hao Zhao
Ming Lu
Anbang Yao
Yurong Chen
Li Zhang
Venue: ICLR 2017
Keywords: Semi-Supervised Learning

Abstract: This paper explores the possibility of learning chess game concepts under weak supervision with convolutional neural networks, which is a topic that has not been visited to the best of our knowledge. We put this task in three different backgrounds: (1) deep reinforcement learning has shown an amazing capability to learn a mapping from visual inputs to most rewarding actions, without knowing the concepts of a video game. But how could we confirm that the network understands these concepts or it just does not? (2) cross-modal supervision for visual representation learning draws much attention recently. Is this methodology still applicable when it comes to the domain of game concepts and actions? (3) class activation mapping is widely recognized as a visualization technique to help us understand what a network has learnt. Is it possible for it to activate at non-salient regions? With the simplest chess game tic-tac-toe, we report interesting results as answers to those three questions mentioned above. All codes, pre-processed datasets and pre-trained models will be released.

TL;DR: investigating whether a CNN understands concepts from a new perspective

Review discussion:
* Could more quantitative analysis be provided to support the claims about model behavior? How often does the CAM localization pick the correct winning square? Can the learned features be used to predict whose turn it is, where they should move, and if anyone can win?
* The chessboard domain is much simpler than the domain of natural images, but a pre-trained AlexNet was used on this domain. Architecture choices are necessarily somewhat arbitrary, but is there a reason a network tuned to natural images was used? Why not a randomly initialized AlexNet or a simpler architecture?
* This work shows a way in which the behavior of a model can be interpreted in a toy scenario. Aside from the original use of CAM for natural images, can this work provide insights about existing model classes deployed in real world problem or about how to design procedures that make non-toy models interpretable?
* The observation that CAM localizes regions of constant pixel intensity is interesting. Have the authors considered if there is a way to identify the relevant context?
* Could more quantitative analysis be provided to support the claims about model behavior? How often does the CAM localization pick the correct winning square? Can the learned features be used to predict whose turn it is, where they should move, and if anyone can win?
We did quantitative evaluations with two protocols as described in the 7th section. Generally speaking, the accuracy is at least 70%.
* The chessboard domain is much simpler than the domain of natural images, but a pre-trained AlexNet was used on this domain. Architecture choices are necessarily somewhat arbitrary, but is there a reason a network tuned to natural images was used? Why not a randomly initialized AlexNet or a simpler architecture?
We use alexnet just out of convenience ... experiments with arbitrary architectures coming soon.
* This work shows a way in which the behavior of a model can be interpreted in a toy scenario. Aside from the original use of CAM for natural images, can this work provide insights about existing model classes deployed in real world problem or about how to design procedures that make non-toy models interpretable?
About non-toy results, it is not quite feasible for us to experiment with complicated games like Go.
But we may provide results on non-rendered tic-tac-toe chessboards.
* The observation that CAM localizes regions of constant pixel intensity is interesting. Have the authors considered if there is a way to identify the relevant context?
Thanks for your positive feedback. We have not thought about it and maybe in the future we may figure out a solution to identify the relevant context.
If I understand correctly, the CNN is trained to classify one of the 1029 states into 18 categories. These 18 categories and each of them correspond to the location of putting the piece that will make one of the 2 players win. Why is it surprising or interesting that a CNN activations fire at the correct location?
More specifically consider the following case - the image consists of either letters A or B.
Then I train a CNN to predict does the image have A or B and then I find activations of the CNN are on the character. This is expected. How is what you train for (functionally) different from this?
* If I understand correctly, the CNN is trained to classify one of the 1029 states into 18 categories. These 18 categories and each of them correspond to the location of putting the piece that will make one of the 2 players win. Why is it surprising or interesting that a CNN activations fire at the correct location? *
In order to answer this question, we have added one more experiment IV in the paper.
We argue that 'firing at the correct location' is not the only choice to achieve this classification task (and thus not a trivial finding). Actually there are two potential ways:
(1) classify by paying attention to the visual patterns formed by existing pieces. (without knowing the concepts of the game)
(2) classify by paying attention at the 'correct location'
experiment IV shows that at an earlier stage the model behaves like (1) and finally gains the ability to 'fire at the correct location'.
* More specifically consider the following case - the image consists of either letters A or B.
Then I train a CNN to predict does the image have A or B and then I find activations of the CNN are on the character. This is expected. How is what you train for (functionally) different from this? *
It is highly related to the third point we claim in the paper: "can CAM activate at non-salient regions".
If there is only A/B in the visual inputs, activating at their locations is trivial because there are gradients at the boundaries of 'visual A/B'.
However, in this experiment, totally information-free regions are activated.
So we argue that the 'A/B experiment' is not analogous to our experiments.
This paper cites the 2015 DeepMind work on game playing but not the more related 2016 Nature paper on AlphaGo( Mastering the game of Go with deep neural networks and tree search. David Silver et al., Nature, 2016). AlphaGo's SL component seems similar to this work. Can you elaborate on the relationship?
The goal of AlphaGo's SL component is to predict the expert's move.
The goal of this paper is to investigate why the network can accurately predict the (expert's or right) move.
And there are two potential answers to the question 'why the network can accurately predict the (expert's or right) move' :
(1) The network does not understand the concepts of the game. It predicts the right move according to existing pieces' patterns.
(2) The network somewhat understands the concepts of the game. It predicts the right move according to the 'empty space' on the chessboard.
And we provide several experiments to support that: the second answer is right.
This is what we are trying to claim here ... and it's quite different from what AlphaGo claims so we did not cite AlphaGo in the draft.
We can cover AlphaGo in the background discussion.
1029 tic-tac-toe boards are rendered (in various ways). These 1029 boards are legal boards where the next legal play can end the game. There are 18 categories of such boards -- 9 for the different locations of the next play, and 2 for the color of the next play. The supervision is basically saying "If you place a black square in the middle right, black will win" or "if you place a white square in the upper left, white will win". A CNN is trained to predict these 18 categories and can do so with 100% accuracy.
The focus of the paper is using Zhou et al's Class Activation Mapping to show where the CNN focuses when making it's decision. As I understand it, an input to CAM is the class of interest.
So let's say it is class 1 (black wins with a play to the bottom right square, if I've deciphered figure 2 correctly. Figure 2 should really be more clear about what each class is).
So we ask CAM to determine the area of focus of the CNN for deciding whether class 1 is exhibited. The focus ends up being on the empty bottom right square (because certainly you can't exhibit class 1 if the bottom right square is occupied). The CNN also needs to condition its decision on other parts of the board -- it needs to know whether there will be 3 in a row from some direction.
But maybe that conditioning is weaker?
That's kind of interesting but I'm not sure about the deeper statements about discovering game rules that the paper hints at. I'm also not sure about the connection of this work to weakly supervised learning or multi-modal learning.
The paper is pretty well written, overall, with some grammatical mistakes, but I simply don't see the surprising discovery of this work.
I also have some concerns about how contrived this scenario is -- using a big, expressive CNN for such a simple game domain and using a particular CNN visualization method.
I am not an expert in reinforcement learning (which isn't happening in this paper, but is in related works on CNN game playing), so maybe I'm not appreciating the paper appropriately.
Dear reviewer_2,
As a pilot study on a very strange topic, I think you have already provided fair reviews on it. Thx for your time.
But I have two more questions that I am really interested in:
(a) If activating at the right location is trivial/not surprising/natural, do you have any comments on RAC in Table 1 ?
Shouldn't RAC for expriment I/II/III all be 100% ? (or at least very close to each other ?)
(b) Do you have any improvement/extension/resubmission suggestions ?
Summary
This paper presents tic-tac-toe as toy problem for investigating CNNs.
A dataset is created containing tic-tac-toe boards where one player is one
move away from winning and a CNN is trained to label boards according
to (1) the player who can win (2 choices) and (2) the position they may move
to win (9 choices), resulting in 18 labels. The CNN evaluated in this paper
performs perfectly at the task and the paper's goal is to inspect how the
CNN works.
The fundamental mechanism for this inspection is Class Activation
Mapping (CAM) (Zhou et. al. 2016), which identifies regions of implicit attention
in the CNN. These implicit attention maps (localization heat maps) are used to
derive actions (which square each player should move). The attention maps
(1) attend to squares in the tic-tac-toe board rather than arbitrary
blobs, despite the fact that one square in a board has uniform color, and
(2) they can be used to pick correct (winning) actions.
This experiment are used to support assertions that the network understands
(1) chess (tic-tac-toe) boards
(2) a rule for winning tic-tac-toe
(3) that there are two players.
Some follow up experiments indicate similar results under various renderings
of the tic-tac-toe boards and an incomplete training regime.
More Clarifying Questions
* I am not quite sure precisely how CAM is implemented here. In the original CAM
one must identify a class of interest to visualize (e.g., cat or dog). I don't
think this paper identifies such a choice. How is one of the 18 possible classes
chosen for creating the CAM visualization and through that visualization
choosing an action?
* How was the test set for this dataset for the table 1 results created?
How many of the final 1029 states were used for test and was the
distribution of labels the same in train and test?
* How is RCO computed? Is rank correlation or Pearson correlation used?
If Pearson correlation is used then it may be good to consider rank correlation,
as argued in "Human Attention in Visual Question Answering: Do Humans and
Deep Networks Look at the Same Regions?" by Das et. al. in EMNLP 2016.
In table 1, what does the 10^3 next to RCO mean?
Pros
* The proposed method, deriving an action to take from the result of a
visualization technique, is very novel.
* This paper provides an experiment that clearly shows a CNN relying on context
to make accurate predictions.
* The use of a toy tic-tac-toe domain to study attention in CNNs
(implicit or otherwise) is a potentially fruitful setting that may
lead to better understanding of implicit and maybe explicit attention mechanisms.
Cons
* This work distinguishes between predictions about "what will happen"
(will the white player win?) and "what to do" (where should the white
player move to win?). The central idea is generalization from "what will happen"
to "what to do" indicates concept learning (sec. 2.1). Why should an ability to
act be any more indicative of a learned concept than an ability to predict
future states. I see a further issue with the presentation of this approach and
a potential correctness problem:
1. (correctness)
In the specific setting proposed I see no difference between "what to do"
and "what will happen."
Suppose one created labels dictating "what to do" for each example in the
proposed dataset. How would these differ from the labels of "what will happen"
in the proposed dataset? In this case "what will happen" labels include
both player identity (who wins) and board position (which position they move
to win). Wouldn't the "what to do" labels need to indicate board position?
They could also chosen to indicate player identity, which would make them
identical to the "what will happen" labels (both 18-way softmaxes).
2. (presentation)
I think this distinction would usually be handled by the Reinforcement Learning
framework, but the proposed method is not presented in that framework or
related to an RL based approach. In RL "what will happen" is the reward an
agent will receive for making a particular action and "what to do" is the
action an agent should take. From this point of view, generalization from
"what will happen" to "what to do" is not a novel thing to study.
Alternate models include:
* A deep Q network (Mnih. et. al. 2015) could predict the value of
every possible action where an action is a (player, board position) tuple.
* The argmax of the current model's softmax could be used as an action
prediction.
The deep Q network approach need not be implemented, but differences between
methods should be explained because of the uniqueness of the proposed approach.
* Comparison to work that uses visualization to investigate deep RL networks
is missing.
In particular, other work in RL has used Simonyan et. al.
(arXiv 2013) style saliency maps to investigate network behavior.
For example,
"Dueling Network Architectures for Deep Reinforcement Learning" by Wang et. al.
in (ICML 2016) uses saliency maps to identify differences between their
state-value and advantage networks. In "Graying the black box:
Understanding DQNs" by Zahavy et. al. (ICML 2016) these saliency maps are
also used to analyze network behavior.
* In section 2.3, saliency maps of Simonyan et. al. are said to not be able to
activate on grid squares because they have constant intensity, yet no empirical
or theoretical evidence is provided for this claim.
On a related note, what precisely is the notion of information referenced in
section 2.3 and why is it relevant? Is it entropy of the distribution of pixel
intensities in a patch? To me it seems that any measure which depends only
on one patch is irrelevant because the methods discussed (e.g., saliency maps)
depend on context as well as the intensities within a patch.
* The presentation in the paper would be improved if the results in section 7
were presented along with relevant discussion in preceding sections.
Overall Evaluation
The experiments presented here are novel, but I am not sure they are very
significant or offer clear conclusions. The methods and goals are not presented
clearly and lack the broader relevant context mentioned above.
Furthermore, I
find the lines of thought mentioned in the Cons section possibly incorrect
or incomplete. As detailed with further clarifying questions, upon closer
inspection I do not see how some aspects of the proposed approach were
implemented, so my opinion may change with further details.
Dear reviewer_1,
* I am not quite sure precisely how CAM is implemented here. In the original CAM
one must identify a class of interest to visualize (e.g., cat or dog). I don't
think this paper identifies such a choice. How is one of the 18 possible classes
chosen for creating the CAM visualization and through that visualization
choosing an action?
We do identify such a choice. That choice is the network classification output (which is 100% correct in all experiments).
Our CAM implementation can be found here: https://github.com/Fromandto/marvin_cam
To the best of our knowledge, it's technically same as Zhou's caffe version.
* How was the test set for this dataset for the table 1 results created?
How many of the final 1029 states were used for test and was the
distribution of labels the same in train and test?
We did not split training and testing sets. And we think training and testing on the same set is fair here because finding out what the network has learnt from the training set is exactly what we are trying to do here.
In a future re-submission version, we will do a 5-fold cross validation (with label distribution regularization).
* How is RCO computed? Is rank correlation or Pearson correlation used?
If Pearson correlation is used then it may be good to consider rank correlation,
as argued in "Human Attention in Visual Question Answering: Do Humans and
Deep Networks Look at the Same Regions?" by Das et. al. in EMNLP 2016.
In table 1, what does the 10^3 next to RCO mean?
(10^3) means -8.096 * 10^3.
I guess you are confused about the numerical scale of this correlation.
To clarify, the ideal representation kernel is generated by this matlab code:
kernel = zeros(180 , 180 , 9);
center = [150 , 150 ; 150 , 90 ; 150 , 30 ; 90 , 150 ; 90 , 90 ; 90 , 30 ; 30 , 150 ; 30 , 90 ; 30 , 30];
sigma = 30;
for i = 1 : 9
c = center(i , :);
for j = 1 : 180
for k = 1 : 180
kernel(j , k , i) = 2 * exp(-(norm(c - [j , k]) / sigma)) - 1;
end
end
end
the amplitude of the gaussian kernel is 2 and we add -1 bias (in order to penalize activations on wrong locations).
the CAM representations are normalized into [0,1].
So ... the numerical scale of RCO is around -10000.
* PROS *
Thank you so much for your positive feedbacks.
As a pilot sduty, we are eager to know what other researchers find interesting in it so that we can work on these points in the future.
* This work distinguishes between predictions about "what will happen"
(will the white player win?) and "what to do" (where should the white
player move to win?). The central idea is generalization from "what will happen"
to "what to do" indicates concept learning (sec. 2.1). Why should an ability to
act be any more indicative of a learned concept than an ability to predict
future states. I see a further issue with the presentation of this approach and
a potential correctness problem:
We admit that our interpretations about 'what will happen' and 'what to do' are somewhat overstated.
Thank you for pointing out this and we will make these claims carefully in a future version.
* Comparison to work that uses visualization to investigate deep RL networks
is missing.
In particular, other work in RL has used Simonyan et. al.
(arXiv 2013) style saliency maps to investigate network behavior.
For example,
"Dueling Network Architectures for Deep Reinforcement Learning" by Wang et. al.
in (ICML 2016) uses saliency maps to identify differences between their
state-value and advantage networks. In "Graying the black box:
Understanding DQNs" by Zahavy et. al. (ICML 2016) these saliency maps are
also used to analyze network behavior.
Thank you very much for referring these two ICML 2016 papers.
We have went through CVPR/ECCV/ICCV/NIPS/ICLR (although not exhaustively) for such papers but just didn't find any highly-related ones.
* In section 2.3, saliency maps of Simonyan et. al. are said to not be able to
activate on grid squares because they have constant intensity, yet no empirical
or theoretical evidence is provided for this claim.
Yes, this claim does not stand.
On a related note, what precisely is the notion of information referenced in
section 2.3 and why is it relevant? Is it entropy of the distribution of pixel
intensities in a patch? To me it seems that any measure which depends only
on one patch is irrelevant because the methods discussed (e.g., saliency maps)
depend on context as well as the intensities within a patch.
Yes, we will make these claims more carefully in a future version.
* The presentation in the paper would be improved if the results in section 7
were presented along with relevant discussion in preceding sections.
Paper structure will be improved in a future version.
Finally,
Thank you again for your valuable reviews !
Game of tic-tac-toe is considered. 1029 tic-tac-toe board combinations are chosen so that a single move will result into victory of either the black or the white player. There are 18 possible moves - 2 players x 9 locations. A CNN is trained from a visual rendering of the game board to these 18 possible outputs. CAM technique is used to visualize the salient regions in the inputs responsible for the prediction that CNN makes. Authors find that predictions correspond to the winning board locations.
Authors claim that this:
1. is a very interesting finding.
2. CNN has figured out game rules.
3. Cross modal supervision is applicable to higher-level semantics.
I don't think (2) be can be claimed because the knowledge of game rules is not tested by any experiment. There is only "one" stage of a game - i.e.
last move that is considered.
Further, the results are on the training set itself - the bare minimum requirement of any implicit or explicit representation of game rules is the ability to act in previously unseen states (i.e. generalization).
Even if the CNN did generalize, I would avoid making any claims about knowledge of game rules.
For (3), author's definition of cross-modal seems to be training from images to games moves. In image-classification we go from images --> labels (i.e. between two different domains). We already know CNNs can perform such mappings. CNNs have been used to map images to actions such as in DQN my Mnih et al., or DDPG by Lillicrap et al. and a lot of other classical work such as ALVIN. It's unclear what points authors are trying to make.
For (1): how interesting is an implicit attention mechanism is a subjective matter. The authors claim a difference between the concepts of "what do do" and "what will happen". They claim by supervising for "what will happen", the CNN can automatically learn about "what to do". This is extensively studied in the model predictive control literature. Where model is "what will happen next", and the model is used to infer a control law - "what to do".
However, in the experimental setup presented in the paper what will happen and what to do seem to be the exact same things.
For further analysis of what the CNN has learnt I would recommend:
(a) Visualizing CAM with respect to incorrect classes. For eg, visualize the CAM with respect to player would lose (instead of winning).
(b) Split the data into train/val and use the predictions on the val-set for visualization. These would be much more informative about what kind of "generalizable" features the CNN pays attention to.
In summary, understanding why CNN's make what decisions they make is a very interesting area of research.
While the emergence of an implicit attention mechanism may be considered to be an interesting finding by some, many claims made by the authors are not supported by experiments (see comments above).
(1) yes, those claims about game rules and cross-modal supervision don't stand.
(2) with all due respect, still we think the experiments presented here (RACs for experiment I/II/III are not 100% but vary a lot) cannot be perfectly explained by any existing theories.
*AND* this (a phenomenon that cannot be explained) is the real reason why we claim that it is interesting (but *NOT* a 'subjective' judgement).
After all, Mendel observed that ‘3:1’ inheritance law in 19th century yet it got fully explained by DNA structure in 1950s.
(3) Thanks for your improvement suggestions ! visualizing with wrong classes is a good idea.

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, all reviewers are leaning against accepting the paper. Authors are encouraged to incorporate reviewer feedback in future iterations of this work.

Decision: Reject