Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The idea of using the confusion matrix to augment the structure of the network is not new. This shown by Hinton et al. "Distilling the Knowledge in a Neural Network" as well as by Farley et al. "Self-informed neural network structure learning."
The extraction of structure from the confusion matrix itself is rather weak. The proposed method of splitting categories into confused and not super classes isn't robust as a given class c can end up in multiple super classes. Furthermore the decision of creating super classes is done locally (per row) without regard of the global structure of the problem. Employing sprectral graph theory to reason about the structure of the confusion matrix, its affinity counterpart actually, is much more principled.

The application of the ensemble of specialists to detecting adversarial examples is novel. However it is not very different from just adding a "unknown" class to a regular CNN model with multiple specialists.

I recommend the authors to pursue this area of research and find provably optimal specialist selection criteria. Furthermore, the proposed approach, leads to significant redundancy of computation as the lower layers across specialists and generalist are likely very similar if not identical. 
review_title: Structure from confusion

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes a novel specialist+1 ensemble which seems to provide robust confidence estimations for deep networks in the presence of adversarial examples. This is an important and interesting task and the results look promising, though I found the presentation lacking.

The technique presented is sufficiently novel, but draws significant motivation from 'Distilling the knowledge in a neural network' as the authors note. The paper has many grammatical errors throughout, but the overall narrative is reasonable. 

Overall the approach and results seem promising; however, much of the critical experimental setup and results are excused to the supplement (which itself is as long as the initial submission). I think this idea could spark interesting discussion in the community but I would lean towards rejection for now and encourage the authors to develop this work further.
review_title: A Promising Start

Decision: Accept