Question: What applications are Overton used in?
Evidence: In the life cycle of many production machine-learning applications, maintaining and improving deployed models is the dominant factor in their total cost and effectiveness–much greater than the cost of de novo model construction. Yet, there is little tooling for model life-cycle support. For such applications, a key task for supporting engineers is to improve and maintain the quality in the face of changes to the input distribution and new production features. This work describes a new style of data management system called Overton that provides abstractions to support the model life cycle by helping build models, manage supervision, and monitor application quality.
Overton is used in both near-real-time and backend production applications. However, for concreteness, our running example is a product that answers factoid queries, such as “how tall is the president of the united states?” In our experience, the engineers who maintain such machine learning products face several challenges on which they spend the bulk of their time.
Fine-grained Quality Monitoring While overall improvements to quality scores are important, often the week-to-week battle is improving fine-grained quality for important subsets of the input data. An individual subset may be rare but are nonetheless important, e.g., 0.1% of queries may correspond to a product feature that appears in an advertisement and so has an outsized importance. Traditional machine learning approaches effectively optimize for aggregate quality. As hundreds of such subsets are common in production applications, this presents data management and modeling challenges. An ideal system would monitor these subsets and provide tools to improve these subsets while maintaining overall quality.
Support for Multi-component Pipelines Even simple machine learning products comprise myriad individual tasks. Answering even a simple factoid query, such as “how tall is the president of the united states?” requires tackling many tasks including (1) find the named entities (`united states', and `president'), (2) find the database ids for named entities, (3) find the intent of the question, e.g., the height of the topic entity, (4) determine the topic entity, e.g., neither president nor united states, but the person Donald J. Trump, who is not explicitly mentioned, and (5) decide the appropriate UI to render it on a particular device. Any of these tasks can go wrong. Traditionally, systems are constructed as pipelines, and so determining which task is the culprit is challenging.
Updating Supervision When new features are created or quality bugs are identified, engineers provide additional supervision. Traditionally, supervision is provided by annotators (of varying skill levels), but increasingly programmatic supervision is the dominant form of supervision BIBREF0, BIBREF1, which includes labeling, data augmentation, and creating synthetic data. For both privacy and cost reasons, many applications are constructed using programmatic supervision as a primary source. An ideal system can accept supervision at multiple granularities and resolve conflicting supervision for those tasks.
There are other desiderata for such a system, but the commodity machine learning stack has evolved to support them: building deployment models, hyperparameter tuning, and simple model search are now well supported by commodity packages including TensorFlow, containers, and (private or public) cloud infrastructure. By combining these new systems, Overton is able to automate many of the traditional modeling choices, including deep learning architecture, its hyperparameters, and even which embeddings are used.
Overton provides the engineer with abstractions that allow them to build, maintain, and monitor their application by manipulating data files–not custom code. Inspired by relational systems, supervision (data) is managed separately from the model (schema). Akin to traditional logical independence, Overton's schema provides model independence: serving code does not change even when inputs, parameters, or resources of the model change. The schema changes very infrequently–many production services have not updated their schema in over a year.
Overton takes as input a schema whose design goal is to support rich applications from modeling to automatic deployment. In more detail, the schema has two elements: (1) data payloads similar to a relational schema, which describe the input data, and (2) model tasks, which describe the tasks that need to be accomplished. The schema defines the input, output, and coarse-grained data flow of a deep learning model. Informally, the schema defines what the model computes but not how the model computes it: Overton does not prescribe architectural details of the underlying model (e.g., Overton is free to embed sentences using an LSTM or a Transformer) or hyperparameters, like hidden state size. Additionally, sources of supervision are described as data–not in the schema–so they are free to rapidly evolve.
As shown in Figure FIGREF7, given a schema and a data file, Overton is responsible to instantiate and train a model, combine supervision, select the model's hyperparameters, and produce a production-ready binary. Overton compiles the schema into a (parameterized) TensorFlow or PyTorch program, and performs an architecture and hyperparameter search. A benefit of this compilation approach is that Overton can use standard toolkits to monitor training (TensorBoard equivalents) and to meet service-level agreements (Profilers). The models and metadata are written to an S3-like data store that is accessible from the production infrastructure. This has enabled model retraining and deployment to be nearly automatic, allowing teams to ship products more quickly.
In retrospect, the following three choices of Overton were the most important in meeting the above challenges.
(1) Code-free Deep Learning In Overton-based systems, engineers focus exclusively on fine-grained monitoring of their application quality and improving supervision–not tweaking deep learning models. An Overton engineer does not write any deep learning code in frameworks like TensorFlow. To support application quality improvement, we use a technique, called model slicing BIBREF3. The main idea is to allow the developer to identify fine-grained subsets of the input that are important to the product, e.g., queries about nutrition or queries that require sophisticated disambiguation. The system uses developer-defined slices as a guide to increase representation capacity. Using this recently developed technique led to state-of-the-art results on natural language benchmarks including GLUE and SuperGLUE BIBREF4.
(2) Multitask Learning Overton was built to natively support multitask learning BIBREF5, BIBREF6, BIBREF7 so that all model tasks are concurrently predicted. A key benefit is that Overton can accept supervision at whatever granularity (for whatever task) is available. Overton models often perform ancillary tasks like part-of-speech tagging or typing. Intuitively, if a representation has captured the semantics of a query, then it should reliably perform these ancillary tasks. Typically, ancillary tasks are also chosen either to be inexpensive to supervise. Ancillary task also allow developers to gain confidence in the model's predictions and have proved to be helpful for aids for debugging errors.
(3) Weak Supervision Applications have access to supervision of varying quality and combining this contradictory and incomplete supervision is a major challenge. Overton uses techniques from Snorkel BIBREF1 and Google's Snorkel DryBell BIBREF0, which have studied how to combine supervision in theory and in software. Here, we describe two novel observations from building production applications: (1) we describe the shift to applications which are constructed almost entirely with weakly supervised data due to cost, privacy, and cold-start issues, and (2) we observe that weak supervision may obviate the need for popular methods like transfer learning from massive pretrained models, e.g., BERT BIBREF8–on some production workloads, which suggests that a deeper trade-off study may be illuminating.
In summary, Overton represents a first-of-its kind machine-learning lifecycle management system that has a focus on monitoring and improving application quality. A key idea is to separate the model and data, which is enabled by a code-free approach to deep learning. Overton repurposes ideas from the database community and the machine learning community to help engineers in supporting the lifecycle of machine learning toolkits. This design is informed and refined from use in production systems for over a year in multiple machine-learned products.
Overton is used in both near-real-time and backend production applications.
Answer: Near-real-time and backend production applications.