An Open-Source Recipe for Real-Time , Adaptive Classifiers in a Planetary-Scale Network

This paper demonstrates how to use open-source software to deploy a set of adaptive classifiers that satisfy the unique demands of Facebook’s Traffic Infrastructure. We combine techniques from sub-disciplines of Machine Learning (ML), ServiceOriented Architecture (SOA) and control systems to embed an ML microservice tier that supports multiple simultaneous classifier models, tolerates version skew in clients and servers, and adapts classifier precision and recall in real time to match operational conditions such as network congestion and available cache space. 1 Embedding Machine Learning in the Network The load balancers, caches, and optical networks that make up Facebook’s Traffic infrastructure form a unique set of challenges for deploying ML classifiers. The hardware and software run under very high load, serving hundreds of billions of latency-sensitive requests for the people using our services each day. This infrastructure is shared over multiple products including Oculus, WhatsApp, Instagram, and the web and mobile versions of Facebook. These services are always-on and are maintained through a process of continuously deployed binaries and configurations. Finally, the traffic infrastructure is distributed in thousands of cities around the world, with unique physical and technical constraints in each location. The contributions of this paper can therefore be summarized as a low-cost approach to satisfy the requirements of this environment, including: • Support for continuous deployment and A/B testing via robust version-skew handling • Extrinsic adaptation of the classification threshold based on operational conditions • A multi-tenant classifier microservice with per-product features and classifiers • Emphasis on low-latency classification (e.g. 1% to a networking Round-Trip-Time) • Emphasis on low-cost classification (e.g. less than 5% load to existing compute clusters) • Operational safe-guards including an easy-to-find “Off Switch” 2 Adapting ML Classifiers as an SOA Microservice A service refers to a client/server application running on one or more machines and conforming to a service description. This paper demonstrates how to bring the robustness, maintainability, and deployability that are expected of production services to systems using open-source ML libraries, whose typical use cases are biased towards offline analyses in the data warehouse. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. 1 / / T h r i f t S t r u c t 2 s t r u c t Work { 3 1 : i 3 2 num1 = 0 , 4 2 : i 3 2 num2 , 5 3 : O p e r a t i o n op , 6 4 : o p t i o n a l s t r i n g comment , 7 } 1 s e r v i c e C a l c u l a t o r e x t e n d s 2 s h a r e d . S h a r e d S e r v i c e { 3 i 3 2 c a l c u l a t e ( 4 1 : i 3 2 l o g i d , 5 2 : Work w, 6 ) t h ro ws ( 1 : I n v a l i d O p e r a t i o n ouch ) , 7 } Figure 1: A Thrift Structure and Service Declaration from the Thrift Tutorial [17] We start by adopting an Interface Description Languages (IDL) like Apache Thrift [18], Google’s Protocol Buffers [16], or Microsoft Bond [10]. Each of these libraries benefits from years of experience with service-oriented architectures deployed in their respective companies. One key feature of these IDLs is their support for version skew for continuous deployment environments where clients and servers must continue to operate correctly in spite of multiple versions of each being active at any time. This IDL syntax prefixes the member variable of structures and method signatures with an integer identifier, in addition to its name and type, so that clients and servers can easily detect which version of a thrift interface is being used by the other. 2.1 Allowing Version Skew in a Classifier Service Machine Learning and Data Mining libraries like WEKA do not typically support concepts like version skew. In order to deploy a robust classifier microservice, these libraries needed to be augmented with this capability. WEKA libraries can import training and test data in CSV format or using its . arff file format. CSV files do not contain any kind of implicit schema to describe the types of the different columns of data. WEKA’s . arff file format has an in-file header to achieve this, but with multiple files from multiple sources, the in-file header is repeated in each file and creates incompatibilities more often than not. The ideal is to have a shared schema, rather than one copy of the schema in each file. Our ML service framework creates a WEKA . arff containing only a single Instance row in addition to the schema header. Subsequently, for each training generation of the model, this schema.arff file is revised in a reverse-compatible way by extending the lists of attributes and symbolic values for each nominal attribute. 2.2 Robust Processing of Instance Attributes In addition to sharing a data-type description over multiple files and over multiple training cycles, a classification service can use its schema to protect a model from client requests that are incompatible with that schema. This can occur cases where client includes an attribute that is not yet part of the trained model, where the client does not include an attribute that the model is expecting, or where the client uses an unknown value for a nominal attribute. Some classifiers allow empty or missing attributes in both training and in classification. Naive Bayes and BayesNet in particular offer a graceful handling of missing values, both in training and in classification, by operating on only the sub-set of attributes that are available and ignoring the missing ones. The general recommendation for missing values is based on the significance of the value’s absence. If the absence of a value has significant predictive value, then missing values should be marked as a placeholder value such as -1 for numerics or “missing-value” for nominals. Since, in all of the missing-value cases above, there is no predictive significance to version skew, then the best classification results will be obtained by using one of the classifiers that can operate on a sub-set of attributes.