[13] R. S. Geiger, D. Cope, J. Ip, M. Lotosh, A. Shah, J. Weng, and R. Tang, ““garbage in, garbage out” revisited: What do machine learning application papers report about human-labeled training data?” Quantitative Science Studies, vol. 2, no. 3, pp. 795–827, 2021. [Online]. Available: https://doi.org/10.1162%2Fqss_a_00144
[14] A. F. Cooper, Y. Lu, J. Forde, and C. M. De Sa, “Hyperparameter optimization is deceiving us, and how to stop it,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 3081–3095. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/17fafe5f6ce2f1904eb09d2e80a4cbf6-Paper.pdf
[15] X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Mohammadi Sepahvand, E. Raff, K. Madan, V. Voleti, S. Ebrahimi Kahou, V. Michalski, T. Arbel, C. Pal, G. Varoquaux, and P. Vincent, "Accounting for variance in machine learning benchmarks," in Proceedings of Machine Learning and Systems, A. Smola, A. Dimakis, and I. Stoica, Eds., vol. 3, 2021, pp. 747-769. [Online]. Available: https://proceedings.mlsys.org/paper_files/paper/2021/file/0184b0cd3cfb185989f858a1d9f5c1eb-Paper.pdf
[16] R. Dror, G. Baumer, M. Bogomolov, and R. Reichart, "Replicability analysis for natural language processing: Testing significance with multiple datasets," Transactions of the Association for Computational Linguistics, vol. 5, pp. 471-486, 2017. [Online]. Available: https://aclanthology.org/Q17-1033
[17] A. Benavoli, G. Corani, and F. Mangili, “Should we really use post-hoc tests based on mean-ranks?” Journal of Machine Learning Research, vol. 17, no. 5, pp. 1–10, 2016. [Online]. Available: http://jmlr.org/papers/v17/benavoli16a.html
[18] J. Demšar, "Statistical comparisons of classifiers over multiple data sets," Journal of Machine Learning Research, vol. 7, no. 1, pp. 1-30, 2006. [Online]. Available: http://jmlr.org/papers/v7/demsar06a.html
[19] K. Ahn, P. Jain, Z. Ji, S. Kale, P. Netrapalli, and G. I. Shamir, "Reproducibility in optimization: Theoretical framework and limits," in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 18022-18033. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/7274ed909a312d4d869cc328ad1c5f04-Paper-Conference.pdf
[20] E. Raff, “A step toward quantifying independently reproducible machine learning research,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2019/file/c429429bf1f2af051f2021dc92a8ebea-Paper.pdf
[21] X. Bouthillier, C. Laurent, and P. Vincent, “Unreproducible research is reproducible,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 725–734. [Online]. Available: https://proceedings.mlr.press/v97/bouthillier19a.html
[22] E. Raff and A. L. Farris, “A siren song of open source reproducibility, examples from machine learning,” in Proceedings of the 2023 ACM Conference on Reproducibility and Replicability, ser. ACM REP '23. New York, NY, USA: Association for Computing Machinery, 2023, p. 115–120. [Online]. Available: https://doi.org/10.1145/3589806.3600042
[23] M. Ferrari Dacrema, P. Cremonesi, and D. Jannach, “Are we really making much progress? a worrying analysis of recent neural recommendation approaches,” in Proceedings of the 13th ACM Conference on Recommender Systems, ser. RecSys '19. New York, NY, USA: Association for Computing Machinery, 2019, p. 101-109. [Online]. Available: https://doi.org/10.1145/3298689.3347058
[24] F. Lu, E. Raff, and J. Holt, “A coreset learning reality check,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 7, pp. 8940-8948, Jun. 2023. [Online]. Available: https://doi.org/10.1609/aaai.v37i7.26074