Venue: ML_Reproducibility_Challenge 2020

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript provides a pytorch implementation about the paper "Fairness without demographics through adversarially reweighted learning". The authors do not perfectly reproduce the results, but the trend seems to be consistent with the original paper. Hyperparameter tuning is sufficient. Additional contribution of this paper is to apply this method to image data.
review_title: A good reproducibility paper with additional applications, but not completely consistent with original paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: The author reimplements the paper "Fairness without demographics through Adversarially Reweighted Learning".

On one hand, the good news is the author reproduces ARL's performance in the original paper. The frustrating news is that the author finds the baseline DRO's performance is much higher compared to that originally reported.

The author says the original paper does not provide the grid search results. Fortunately, I find that the authors of the original paper publish their code and the optimal hyperparameters in their [GitHub repository](https://github.com/google-research/google-research/tree/master/group_agnostic_fairness).

The optimal hyper-parameters from this paper's Appendix B are very different from the original paper's grid search result (see GitHub).

So I think the author should revisit the experiment and try to give some insight on what are the optimal hyperparameters.
review_title: Please check the original paper's optimal hyperparameters.

Meta-review: Overall reviews and/or the paper content not good enough for the AC to recommend to the journal.
Decision: Reject