Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a simple variation on the recently proposed "temporal ensembling" method for semi-supervised learning. Temporal ensembling smoothes label predictions (using the same model on unlabeled data) to produce additional labeled data. Instead, the authors propose to smooth the model weights during training (exponential moving average) to yield a "teacher network" which can be used to generate new labeled data in an online way.

The approach is well motivated. The main advantage over temporal ensembling is that this method admits online training vs temporal ensembling which only yields new labels once per epoch. The experimental results seem believable, and indicate an improvement over temporal ensembling.

Overall, it appears to be a simple and interesting twist on temporal ensembling which yields better results in the semi-supervised setting. 
review_title: Simple, interesting semi-supervised training technique

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a new way to do semi-supervised learning. It takes inspiration from the temporal ensembling work. But instead of having an exponential moving average of predictions computed every epoch, and using those as labels/targets, this method use a moving average of the weights of the network and form a teach net that gives labels. This essentially extends the temporal ensembling method to work in an online fashion.

This is a simple and potentially powerful way to do semi-supervised learning. I am curious why the authors used batch norm instead of weight normalization. It's unclear to me what the validation set was used for (presumably hyper-parameter selection). It'd be nice to see some exploration for \alpha and its effects. Otherwise, the SVHN results look pretty compelling.
review_title: Interesting extension of the temporal ensembling

Decision: Accept