Venue: ML_Reproducibility_Challenge 2021 Fall

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Authors have done a very good job in reproducing the paper - although there has been some deviations in the results achieved, the authors managed to get very similar results, and they tried to contact the authors of the original paper to get some clarifications; with mixed success. Some of the original experiments are practically impossible to reproduce, especially when scaled at the size of Imagenet.

Overall a great effort.
review_title: Well written review and reproducibility report

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Thanks for your great paper!

Summary: The authors re-implemented the codes for most experiments and managed to validate the 3 claims of the original paper: high-quality counterfactuals, inductive bias requirements, and out-of-distribution robustness. Besides, the authors also improved CGN training on the MNIST dataset and provided a new insight of explainability analysis for invariant classifiers.

Pros:

- The paper is well written and the structure of the paper is very consistent
- The paper gives enough details about the methodology of the original paper, making it easy to read even without the original paper
- The details of hyperparameters and experimental setup are clearly stated for further reproduction
- The paper proposes an interesting view to visualize the latent feature space and model-focused features using t-SNE and GradCAM respectively
- The paper also finds that the finetuned CGN-based ensemble performs worse on all specified OOD-benchmarks
- The authors provide a detailed supplementary material
review_title: The paper is really well-written and should be accepted definitely!

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ### Reproducibility Summary
The authors have the mandatory first page and the contents are good.

### Scope of reproducibility
The scope is clear and the evidence is provided for all claims, although some experiments were only partially reproduced due to missing specification of hyperparameters and resource constraints.

### Code
The reproducibility authors re-used the source code by the original paper authors and released their own in https://github.com/anonymous-user-256/mlrc-cgn .

### Communication with original authors
The report mentions contacting and communicating with the original paper's authors but getting a late response.

### Hyperparameter Search
The authors reused the original paper's or original codebase' values when available, and only run experiments once.

### Ablation Study
The report includes ablations and further experiments

### Discussion on results
The report discusses the results obtained.

### Recommendations for reproducibility
There are recommendations for reproducibility, mainly things that were misspecified or underspecified, and trying low-compute experiments for ease of reproducibility.

### Overall organization and clarity
Great
review_title: Well done!

Meta-review: Following the recommendation of reviewers and meta-reviewer, the paper is accepted for ML Reproducibility Challenge 2021, and will be published in the upcoming special edition of ReScience Journal.

After several rounds of discussion, debate and incorporating recommendations from the Area Chairs and Program Chairs, the report has been granted the **Best Paper Award** of ML Reproducibility Challenge, 2021, due to its very high quality quality of all-round reproducibility effort and presentation. Congratulations!
Decision: Accept (Outstanding Paper)