Towards Efficient Automated Machine Learning

Machine learning is widely used in a variety of different disciplines to develop predictive models for variables of interest. However, building such solutions is a time consuming and challenging discipline that requires highly trained data scientists and domain experts. In response, the field of automated machine learning (AutoML) aims to reduce human effort and speedup the development cycle through automation. Due to the ubiquity of hyperparameters in machine learning algorithms and the impact that a well-tuned hyperparameter configuration can have on predictive performance, hyperparameter optimization is a core problem in AutoML. More recently, the rise of deep learning has motivated neural architecture search (NAS), a specialized instance of a hyperparameter optimization problem focused on automating the design of neural networks. Naive approaches to hyperparameter optimization like grid search and random search are computationally intractable for large scale tuning problems. Consequently, this thesis focuses on developing efficient and principled methods for hyperparameter optimization and NAS. In particular, we make progress towards answering the following questions with the aim of developing algorithms for more efficient and effective automated machine learning: 1. Hyperparameter Optimization (a) How can we effectively use early-stopping to speed up hyperparameter optimization? (b) How can we exploit parallel computing to perform hyperparameter optimization in the same time it takes to train a single model in the sequential setting? (c) For multi-stage machine learning pipelines, how can we exploit the structure of the search space to reduce total computational cost? 2. Neural Architecture Search (a) What is the gap in performance between state-of-the-art weight-sharing NAS methods and random search baselines? (b) How can we develop more principled weight-sharing methods with provably faster convergence rates and improved empirical performance? (c) Does the weight-sharing paradigm commonly used in NAS have applications to more general hyperparameter optimization problems? Given these problems, this thesis is organized into two parts. The first part focuses on progress we have made towards efficient hyperparameter optimization by addressing Problems 1a, 1b, and 1c. The second part focuses on progress we have made towards understanding and improving weight-sharing for neural architecture search and beyond by addressing Problems 2a, 2b, and 2c. Acknowledgments I am grateful to have had the opportunity to work with so many amazing people throughout my graduate studies. The work presented in this thesis would not have been possible without them and the unwavering support from my loving family and friends. First and foremost, I would like to thank my advisor Ameet Talwalkar. Ameet has been a constant source of support, encouragement, and guidance throughout graduate school. From our early days in UCLA to the past few years at CMU, Ameet has been my biggest advocate and believed in me even when I doubted myself. His confidence in me has been instrumental in helping me navigate the challenges I faced to get to where I am today. Thank you Ameet. Special thanks to my close collaborators Professor Kevin Jamieson and Afshin Rostamizadeh for mentoring me when I first started graduate school and being great sources of inspiration and insight. It has also been great to work with Professor Nina Balcan on more recent projects and I have learned so much from her. Other professors I would like to thank include my co-authors Benjamin Recht, Moritz Hardt, and Virginia Smith; Katerina Fragkiadaki and Aarti Singh, course instructors for classes I TAed; and Ryan Tibshirani, for serving on the MLD Wellness Network. Thanks also to Professor Jamieson, Balcan, and Schneider for serving on my thesis committee. It has been great to have worked with collaborators from multiple institutions. Special thanks to my close collaborators Misha Khodak and Maruan Al-Shedivat, both of whom are tremendously knowledgeable and taught me so much. Thanks also to Tian Li, Evan Sparks, Giulia DeSalvo, Haoping Bai, Nick Roberts, Jonathan Ben-Tzur, and Katerina Gonina. I am also grateful for the friends that I have made throughout my graduate school including Chris Wu, Nolan Donoghue, Beer Changpinyo, and Ray Zhang from my time at UCLA; Misha, Greg Plumb, Sebastian Caldas, Jeffrey Li, Jeremy Cohen, Joon Kim, and Lucio Dery from the recently renamed Sage Lab; Maruan, Mariya Toneva, Anthony Platanios, and Simon Du, who were instrumental in launching the MLD Blog; Ifigeneia Apostolopoulou, Ritesh Noothigattu, Chieh Lin, Lisa Lee, and Leqi Liu from CMU; and Xavier Bouthillier, Jean-François Kagy, David Rosenberg, and Ningshan Zhang. Thanks also to Eric Forister, Maria Salgado, and Robert Sherman for helping me get into graduate school in the first place. The administrative staff at CMU has also been a great resource: special thanks to Diane Stidle for always being there when I had questions and helping me feel at home at CMU and to Christy Melucci for helping our lab run so smoothly. Finally, my family has been there for me from the very beginning: my mom Yuhong Chen, sister Grace Cantarella, Anthony and Grazia Cantarella, Jun Li, Du Chun, Louisa and Hansen Li; their love keeps me going. I dedicate this thesis to them and to my wife Seon Hee, who encouraged me to pursue my dreams, drove cross country with me through multiple blizzards to help me achieve them, and continues to be my greatest source of strength.