Training models on CPU memory can take hours or even days. Hyperparameter tuning adds even more time to this equation.
HPO libraries (e.g., Ray Tune, Optuna, Amazon SageMaker, and Spearmint) accelerate tuning through automation. They provide sophisticated tuning methods (such as Bayesian optimization) and are often equipped with advanced scheduling functionality. This enables the termination of unpromising trials, thus reaching an optimal solution faster and more efficiently. When combined with distributed training, these solutions are even more powerful.
Distributing training splits the training workload across multiple GPUs that run in parallel. This computational approach not only saves valuable training time but can also be used to dramatically speed up HPO. Automatic queuing tools, such as Run:AI and Celery, can be used together with HPO libraries to calculate hyperparameters across multiple GPUs in parallel.
The combination of HPO with multiple GPUs offers major benefits:
Scaling: Data scientists can scale the HPO process to hundreds or even thousands of GPUs that work simultaneously, allowing experiments to run in parallel rather than sequentially. This approach enables teams to retain the highest-performing experiments without having to wait for the results of less-successful experiments and dramatically accelerates training time.
Reduced management overhead: Data scientists no longer need to worry about managing underlying resources. Instead of using manual scripts, these tools automatically manage the queue, launching thousands of runs when the resources are available. This greatly simplifies HPO and ensures the cluster is always fully utilized.
Accelerated training: Replacing the tedious, slow, and serial manual methods with automation and parallel computing significantly increases HPO efficiency and reduces overall training time.
Data science teams have begun to use this approach and have achieved impressive results. In the Netflix example referenced above, the company used Spearmint to perform Bayesian optimization and used Celery to manage the distributed training across multiple GPUs. This enabled Netflix to tune, train, and update multiple neural networks for each of its international regions on a daily basis. In another example, a random forest classifier was used to perform HPO using a distributed approach. As a result, tuning was achieved 30 times faster, and classifier accuracy increased by 5%.