Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This paper cites the 2015 DeepMind work on game playing but not the more related 2016 Nature paper on AlphaGo( Mastering the game of Go with deep neural networks and tree search. David Silver et al., Nature, 2016). AlphaGo's SL component seems similar to this work. Can you elaborate on the relationship?
review_title: Relation to AlphaGo

final_score: rating: 3: Clear rejection
review_content: Game of tic-tac-toe is considered. 1029 tic-tac-toe board combinations are chosen so that a single move will result into victory of either the black or the white player. There are 18 possible moves - 2 players x 9 locations. A CNN is trained from a visual rendering of the game board to these 18 possible outputs. CAM technique is used to visualize the salient regions in the inputs responsible for the prediction that CNN makes. Authors find that predictions correspond to the winning board locations. 

Authors claim that this:
1. is a very interesting finding. 
2. CNN has figured out game rules. 
3. Cross modal supervision is applicable to higher-level semantics. 

I don't think (2) be can be claimed because the knowledge of game rules is not tested by any experiment. There is only "one" stage of a game - i.e. last move that is considered. Further, the results are on the training set itself - the bare minimum requirement of any implicit or explicit representation of game rules is the ability to act in previously unseen states (i.e. generalization). Even if the CNN did generalize, I would avoid making any claims about knowledge of game rules. 

For (3), author's definition of cross-modal seems to be training from images to games moves. In image-classification we go from images --> labels (i.e. between two different domains). We already know CNNs can perform such mappings. CNNs have been used to map images to actions such as in DQN my Mnih et al., or DDPG by Lillicrap et al. and a lot of other classical work such as ALVIN. It's unclear what points authors are trying to make. 

For (1): how interesting is an implicit attention mechanism is a subjective matter. The authors claim a difference between the concepts of "what do do" and "what will happen". They claim by supervising for "what will happen", the CNN can automatically learn about "what to do". This is extensively studied in the model predictive control literature. Where model is "what will happen next", and the model is used to infer a control law - "what to do". However, in the experimental setup presented in the paper what will happen and what to do seem to be the exact same things. 

For further analysis of what the CNN has learnt I would recommend:
(a) Visualizing CAM with respect to incorrect classes. For eg, visualize the CAM with respect to player would lose (instead of winning).

(b) Split the data into train/val and use the predictions on the val-set for visualization. These would be much more informative about what kind of "generalizable" features the CNN pays attention to. 

In summary, understanding why CNN's make what decisions they make is a very interesting area of research. While the emergence of an implicit attention mechanism may be considered to be an interesting finding by some, many claims made by the authors are not supported by experiments (see comments above). 


 




review_title: Unclear 

final_score: rating: 3: Clear rejection
review_content: Summary
===
This paper presents tic-tac-toe as toy problem for investigating CNNs.
A dataset is created containing tic-tac-toe boards where one player is one
move away from winning and a CNN is trained to label boards according
to (1) the player who can win (2 choices) and (2) the position they may move
to win (9 choices), resulting in 18 labels. The CNN evaluated in this paper
performs perfectly at the task and the paper's goal is to inspect how the
CNN works.

The fundamental mechanism for this inspection is Class Activation
Mapping (CAM) (Zhou et. al. 2016), which identifies regions of implicit attention
in the CNN. These implicit attention maps (localization heat maps) are used to
derive actions (which square each player should move). The attention maps  

(1) attend to squares in the tic-tac-toe board rather than arbitrary
blobs, despite the fact that one square in a board has uniform color, and

(2) they can be used to pick correct (winning) actions.

This experiment are used to support assertions that the network understands
(1) chess (tic-tac-toe) boards
(2) a rule for winning tic-tac-toe
(3) that there are two players.

Some follow up experiments indicate similar results under various renderings
of the tic-tac-toe boards and an incomplete training regime.


More Clarifying Questions
===

* I am not quite sure precisely how CAM is implemented here. In the original CAM
one must identify a class of interest to visualize (e.g., cat or dog). I don't
think this paper identifies such a choice. How is one of the 18 possible classes
chosen for creating the CAM visualization and through that visualization
choosing an action?

* How was the test set for this dataset for the table 1 results created?
How many of the final 1029 states were used for test and was the
distribution of labels the same in train and test?

* How is RCO computed? Is rank correlation or Pearson correlation used?
If Pearson correlation is used then it may be good to consider rank correlation,
as argued in "Human Attention in Visual Question Answering: Do Humans and
Deep Networks Look at the Same Regions?" by Das et. al. in EMNLP 2016.
In table 1, what does the 10^3 next to RCO mean?


Pros
===

* The proposed method, deriving an action to take from the result of a
visualization technique, is very novel.

* This paper provides an experiment that clearly shows a CNN relying on context
to make accurate predictions.

* The use of a toy tic-tac-toe domain to study attention in CNNs
(implicit or otherwise) is a potentially fruitful setting that may
lead to better understanding of implicit and maybe explicit attention mechanisms.


Cons
===

* This work distinguishes between predictions about "what will happen"
(will the white player win?) and "what to do" (where should the white
player move to win?). The central idea is generalization from "what will happen"
to "what to do" indicates concept learning (sec. 2.1). Why should an ability to
act be any more indicative of a learned concept than an ability to predict
future states. I see a further issue with the presentation of this approach and
a potential correctness problem:

1. (correctness)
In the specific setting proposed I see no difference between "what to do"
and "what will happen."

Suppose one created labels dictating "what to do" for each example in the
proposed dataset. How would these differ from the labels of "what will happen"
in the proposed dataset? In this case "what will happen" labels include
both player identity (who wins) and board position (which position they move
to win). Wouldn't the "what to do" labels need to indicate board position?
They could also chosen to indicate player identity, which would make them
identical to the "what will happen" labels (both 18-way softmaxes).

2. (presentation)
I think this distinction would usually be handled by the Reinforcement Learning
framework, but the proposed method is not presented in that framework or
related to an RL based approach. In RL "what will happen" is the reward an
agent will receive for making a particular action and "what to do" is the
action an agent should take. From this point of view, generalization from
"what will happen" to "what to do" is not a novel thing to study.

Alternate models include:
    * A deep Q network (Mnih. et. al. 2015) could predict the value of
      every possible action where an action is a (player, board position) tuple.
    * The argmax of the current model's softmax could be used as an action
      prediction.
The deep Q network approach need not be implemented, but differences between
methods should be explained because of the uniqueness of the proposed approach.


* Comparison to work that uses visualization to investigate deep RL networks
is missing. In particular, other work in RL has used Simonyan et. al.
(arXiv 2013) style saliency maps to investigate network behavior. For example, 
"Dueling Network Architectures for Deep Reinforcement Learning" by Wang et. al.
in (ICML 2016) uses saliency maps to identify differences between their
state-value and advantage networks. In "Graying the black box:
Understanding DQNs" by Zahavy et. al. (ICML 2016) these saliency maps are
also used to analyze network behavior.


* In section 2.3, saliency maps of Simonyan et. al. are said to not be able to
activate on grid squares because they have constant intensity, yet no empirical
or theoretical evidence is provided for this claim.

On a related note, what precisely is the notion of information referenced in
section 2.3 and why is it relevant? Is it entropy of the distribution of pixel
intensities in a patch? To me it seems that any measure which depends only
on one patch is irrelevant because the methods discussed (e.g., saliency maps)
depend on context as well as the intensities within a patch.


* The presentation in the paper would be improved if the results in section 7
were presented along with relevant discussion in preceding sections.


Overall Evaluation
===
The experiments presented here are novel, but I am not sure they are very
significant or offer clear conclusions. The methods and goals are not presented
clearly and lack the broader relevant context mentioned above. Furthermore, I
find the lines of thought mentioned in the Cons section possibly incorrect
or incomplete. As detailed with further clarifying questions, upon closer
inspection I do not see how some aspects of the proposed approach were
implemented, so my opinion may change with further details.
review_title: Novel experiments, but the results and significance are not clear

final_score: rating: null
review_content: * Could more quantitative analysis be provided to support the claims about model behavior? How often does the CAM localization pick the correct winning square? Can the learned features be used to predict whose turn it is, where they should move, and if anyone can win?

* The chessboard domain is much simpler than the domain of natural images, but a pre-trained AlexNet was used on this domain. Architecture choices are necessarily somewhat arbitrary, but is there a reason a network tuned to natural images was used? Why not a randomly initialized AlexNet or a simpler architecture?

* This work shows a way in which the behavior of a model can be interpreted in a toy scenario. Aside from the original use of CAM for natural images, can this work provide insights about existing model classes deployed in real world problem or about how to design procedures that make non-toy models interpretable?

* The observation that CAM localizes regions of constant pixel intensity is interesting. Have the authors considered if there is a way to identify the relevant context?
review_title: Initial Questions

final_score: rating: null
review_content: If I understand correctly, the CNN is trained to classify one of the 1029 states into 18 categories. These 18 categories and each of them correspond to the location of putting the piece that will make one of the 2 players win. Why is it surprising or interesting that a CNN activations fire at the correct location? 

More specifically consider the following case - the image consists of either letters A or B. Then I train a CNN to predict does the image have A or B and then I find activations of the CNN are on the character. This is expected. How is what you train for (functionally) different from this? 
review_title: Regarding interpretation of results

final_score: rating: 3: Clear rejection
review_content: 1029 tic-tac-toe boards are rendered (in various ways). These 1029 boards are legal boards where the next legal play can end the game. There are 18 categories of such boards -- 9 for the different locations of the next play, and 2 for the color of the next play. The supervision is basically saying "If you place a black square in the middle right, black will win" or "if you place a white square in the upper left, white will win". A CNN is trained to predict these 18 categories and can do so with 100% accuracy.

The focus of the paper is using Zhou et al's Class Activation Mapping to show where the CNN focuses when making it's decision. As I understand it, an input to CAM is the class of interest. So let's say it is class 1 (black wins with a play to the bottom right square, if I've deciphered figure 2 correctly. Figure 2 should really be more clear about what each class is). So we ask CAM to determine the area of focus of the CNN for deciding whether class 1 is exhibited. The focus ends up being on the empty bottom right square (because certainly you can't exhibit class 1 if the bottom right square is occupied). The CNN also needs to condition its decision on other parts of the board -- it needs to know whether there will be 3 in a row from some direction. But maybe that conditioning is weaker?

That's kind of interesting but I'm not sure about the deeper statements about discovering game rules that the paper hints at. I'm also not sure about the connection of this work to weakly supervised learning or multi-modal learning.

The paper is pretty well written, overall, with some grammatical mistakes, but I simply don't see the surprising discovery of this work. 

I also have some concerns about how contrived this scenario is -- using a big, expressive CNN for such a simple game domain and using a particular CNN visualization method.

I am not an expert in reinforcement learning (which isn't happening in this paper, but is in related works on CNN game playing), so maybe I'm not appreciating the paper appropriately.
review_title: Still not sure what to take away from these experiments

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, all reviewers are leaning against accepting the paper. Authors are encouraged to incorporate reviewer feedback in future iterations of this work.
Decision: Reject