Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents ReMixMatch an improved version of MixMatch. The main contributions are the distribution alignment and the augmentation anchoring. Distribution alignment rescales the predictions based on the difference between the model marginals and the ground truth running average estimation. Augmentation anchoring instead of computing the guessed probabilities on unlabelled data as the average probabilities on transformed samples (as in MixMatch), it considers as guessed labels the average probabilities obtained from weak transformations (flip+crop) even when using stronger transformations (Autoaugment like). 

The paper is well written, has interesting experiments and very impressive results. 
However, there are some negative points that the authors should clarify:
- The final method is a mixup of many different techniques, thus, not a strong contribution, but many smaller contributions. 
- As shown in the ablation study, the main contribution on the obtained results seems to be the use of stronger transformations than in MixMatch. This is not so interesting, even though results are impressive. If this is the case, authors should state it more clearly in the paper that a large proportion of the gap in performance between MixMatch and ReMixMatch is the introduction of stronger transformations (Autoaugment style).

Overall the paper is well presented and contributes to further improve the performance on semi-supervised learning. I there fore recommend it for acceptance. However, I would like to see in the paper a more general overview on the fact that strong transformations can further improve semi-supervised methods and ReMixMatch is a way to leverage those transformations.


Additional comments:
- Instead of using the rescaling trick for distribution alignment, what about enforcing the marginal distribution on the annotated data and the marginal distribution of the model to be similar with KL divergence? Would it be better or worse than the proposed approach?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary
The authors make three major contributions that improve MixMatch and achieve state-of-the-art in a semi-supervised image classification task. The major contributions include: (1) distribution alignment to calibrate the predicted distribution of unlabeled data; (2) augmentation anchoring to allow more aggressive data augmentation; and (3) CTAugment to train the augmentation policy alongside the semi-supervised model.
The authors conduct experiments on SVHN, CIFAR-10 and STL, and show significant improvements over the MixMatch baseline. They also show good results (15.08% error rate) of training with 40 labeled data, in spite of very high variation. In the ablation study, they show the error rate drops as K (number of augmentation) increases. They also conduct ablation studies on the design choices of their method.

Decision
The decision for this paper is borderline, tending towards a weak accept. Overall, the paper proposes some simple but interesting ideas, e.g. distribution environments. However, although the proposed method achieves good performance over various (smaller) benchmarks, the method seems ad-hoc and complicated. As pointed out in the weakness section, many design choices are not well motivated, and the effects of those designs are not well studied. The tendency to accept is due to the overall strong results. 

Strength
1. Significant improvement over MixMatch baseline.
2. The proposed augmentation anchoring and distribution alignment can be easily integrated into existing work.
3. The proposed CTAugment method lifts the burden of training an RL data augmentation policy. 

Weakness
1. The objective of the update equation of CTAugment’s learned weights seems contradicted with the purpose of how data augmentation is used in the consistency-based SSL method. In other words, the objective of the update equation encourages higher weights for the distortion parameter that leads to lower variation in the predicted distribution. However, the idea of aggressive data augmentation is to generate data that has high variation in the model prediction, and then penalize the variation in the form of consistency loss. The variation induced by aggressive augmentation is the root of the consistency loss that helps regularize the model.
2. The authors should provide ablation study and analysis of their CTAugment. For example, they should compare with simple random augmentation policy. It is also recommended to show the learned weights of the distortion parameter. Also does larger K value when applied for vanilla MixMatch approach the results in ReMixMatch? 
3. The authors should provide more detail of the setting in the ablation study. For example, the setting of “No strong aug.” and “No weak aug.” are not clear. 
4. The authors hypothesize that “stronger augmentation can result in disparate predictions, so their average may not be a meaningful target.” However, they do not show any analysis to support this hypothesis.
5. It is recommended to evaluate the method on larger datasets such as CIFAR-100. It is not clear how well these methods scale, and for example using k=8 adds computation which may hinder training scalability. 

Minor Comments
1. For Table 2 and Table 3, it should be “error rate” rather than “accuracy”.
2. How is the loss weight λr tuned in the 40 labeled setting? How are the hyper-parameters tuned in general?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes two modifications for the MixMatch method [1] and achieves improved accuracy on a range of semi-supervised benchmarks. The first modification enforces the distribution of predicted labels to match the distribution of labeled data. The second modification is adding a learned data augmentation strategy, and adapting the method to work with strong data augmentation. The final method is titled ReMixMatch, and improves significantly over MixMatch, especially in low-data regime. 

The main contribution of the paper is really strong empirical results. The method achieves state of the art results or close to that on multiple benchmarks, with especially large gains in settings with very scarce labeled data, like 40 labels on CIFAR-10. 

Another important contribution is the learned data augmentation strategy, which as far as I understand is novel and overcomes some of the limitations of  existing learned data augmentation techniques. However, the explanation of the strategy wasn’t very clear for me, and the authors didn’t frame it as a major contribution.

The main drawback of the paper is that it seems to be more engineering-focused, and doesn’t provide much insight into semi-supervised learning. The paper can be summarized as adding two modifications to mix-match, and getting better results. The final method becomes fairly involved. Mix-Match is already an elaborate method, and ReMixMatch additionally introduces learned data augmentation, an additional loss term for matching label distributions between labeled and unlabeled data, consistency-loss, and a self-supervised loss (section 3.3). 

For the reasons above, I think the paper is borderline, but I am currently voting for acceptance based on the strong empirical performance. At the same time, I think the paper can be made stronger and more interesting to read, if the authors added some experiments aimed at understanding the proposed modifications. 

One set of experiments that I think would be interesting is aimed at understanding the distribution-matching part. For example, it would be great if the author could demonstrate that without this loss term the distribution of the predicted classes is wrong in the experiments from Section 4. It would also be interesting to see an experiment where the labeled data has a skewed distribution of classes, but we provide the method with information about the true class distribution, and demonstrating that this information helps predictive performance. 

For the learnable data augmentation it would be great if the authors could provide more insight into the method, how it works, and why is it better than the alternatives. Just analyzing the learned data augmentation in different settings and adding more intuition for what happens would make the paper more insightful and interesting to read. 

On a more minor note, the paper [1] seems to report 4.95 accuracy for MixMatch on CIFAR-10 with 4k labels, while in this paper it’s being reported as 6.24. What is the reason for the difference? Another paper, [2], reports very competitive results on CIFAR-10 for 4k labels. I would recommend discussing these results briefly in the paper. At the same time the empirical performance of ReMixMatch is really impressive, and I don’t think the results in [1] and [2] affect their significance.

[1] MixMatch: A Holistic Approach to Semi-Supervised Learning
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel

[2] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson

review_title: Official Blind Review #2

Meta-review: This works improves the MixMatch semi-supervised algorithm along the two directions of distribution alignment and augmentation anchoring, which together make the approach more data-efficient than prior work.
All reviewers agree that the impressive empirical results in the paper are its main strength, but express concern that the method is overly complicated and hacking together many known pieces, as well as doubt as to the extent of the contribution of the augmentation method itself, with requests for better augmentation controls.
While some of these concerns have not been addressed by authors in their response, the strength of empirical results seems enough to justify an acceptance recommendation.
Decision: Accept (Poster)