Engineering Practices for Machine Learning Lifecycle at Google and Microsoft
By: Geraldi Dzakwan
Tags: Hyperparameter Tuning, Deployment, Systems Engineering, Machine Learning, Infrastructure

Technical Contributions and Their Importance

Paper [1]: Google Vizier, A Service for Black-Box Optimization

Introduction

Black-Box Optimization

Before diving further into what Google Vizier is, we would like to give you a little introduction to black-box optimization. Black-box simply means that we could only evaluate a function f(x) but we don’t have direct access to the internal of the function, e.g. the function’s gradients or Hessian.

Put it together with optimization, a black-box optimization is a process of finding the best operating parameters by evaluating those parameters against the function. For companies like Google which uses a lot of heavy DNN models, the evaluation is more often than not expensive. Thus, it’s in their very best interest to generate a sequence of x that is more likely to reach the global optimum quicker than any other sequence of x in the parameter space. There are existing algorithms on this matter, such as Random Search, Grid Search, Simulated Annealing, Genetic Algorithm, and Bayesian Optimization, which I believe many of you have used, especially the first two.

Google Vizier Terms

There are three terms that I will use in explaining the concepts of this paper:

Trial, a list of parameter values, i.e. the x, that leads to a single evaluation of f(x) Study, a single optimization ran over a parameter space, i.e. a set of Trials Worker, the process/instance which is responsible to evaluate Trials

System Overview

Basic User Workflow

There is no better way to understand a tool than looking directly at the code. So, take a look at a snippet below that gives you a basic user workflow for Google Vizier. I’ve also written comments for you to understand the lines.

# Register this client with the Study, creating it if necessary

# Supply the study_config (name, owner, etc.)

# and the worker handle, i.e. the process ID

client.LoadStudy(study_config, worker_handle) # Study is not Done if there are pending Trials still

while (not client.StudyIsDone()):



# Obtain a Trial to evaluate

trial = client.GetSuggestion() # Evaluate the objective function at the trial parameters

metrics = RunTrial(trial) # Report back the results and mark Trial as Complete

client.CompleteTrial(trial, metrics)

Google Vizier Architecture

Below is the architecture of Google Vizier:

Figure 1: Google Vizier Architecture

There are 6 main components:

Evaluation Workers, a component provided by the client/user which will execute Trials Vizier API, a component that connects Evaluation Workers with Google Vizier functionalities Persistent Database, an internal Google Vizier component that holds the current state of all Studies, i.e. the Complete and Pending Trials Suggestion Service, a component that returns a new Trial to be executed by Evaluation Workers Early Stopping Service, a component that helps terminate a Trial early by analyzing several conditions Dangling Work Finder, a component that restarts works lost due to preemptions

Google Vizier Algorithms

Google Vizier provides the algorithm below that can be used by the clients to design their hyperparameter tuning experiments.

Automated Early Stopping

Performance Curve Stopping Rule, this rule performs regression on the performance (loss/accuracy) curves to predict the final objective value of a Trial given a set of previous Trials so we could base our Early Stopping mechanism on that predicted value. Median Stopping Rule, this rule stops a pending Trial when the Trial objective value at step s is strictly worse than the median value of the running averages at step s-1.

Transfer Learning

The idea is pretty much the same as Transfer Learning in general. Suppose we have tuned the Learning Rate and Regularization hyperparameters for a certain ML system. Then, we can use that Study/Studies as a prior to tuning the same ML system on a different dataset. Note that we could use more than one Study which isn’t a common thing in Transfer Learning in general.

The scheme is as below:

Build a stack of Gaussian Process regressors, i.e. a stack of Studies. Priors are placed on the bottom and its residual will be used for the Studies in the upper part of the stack. This approach works best when priors have many well-spaced data points but the top-level regressor has relatively little training data. This is exactly the same scenario as Transfer Learning in general, e.g. use ImageNet at the bottom and a small, domain-specific image dataset at the top.

Algorithm Playground

In the case that you need to customize your own algorithm, i.e. no algorithms implemented in Google Vizier match your needs, then you can do that through this Algorithm Playground.

Below is the architecture of the Algorithm Playground.

Figure 2: Algorithm Playground Architecture [1]

There are 4 main components:

Vizier API, same as before Persistent Database, same as before Abstract Policy, an abstract class, containing two abstract methods to be implemented Custom Policy, a class that inherits from the Abstract Policy and implements the two abstract methods Playground Binary, a binary executable compiled from the Custom Policy that is based on the demands reported by Vizier API Evaluation Workers, same as before, but note that these workers are unaware of the Playground Binary since it is bridged with Vizier API, so clients don’t need to adjust Evaluation Workers when they use Algorithm Playground

The clients need to implement the two abstract methods from Abstract Policy class in a way that suits their needs. Those two functions are:

# 1. Function to generate num_suggestions number of new TrialsGetNewSuggestions(trials, num_suggestions) # 2. Function that returns a list of Pending Trials that should be stopped earlyGetEarlyStoppingTrials(trials)

Google Vizier Use Cases