Patent Description:
Deep learning is a specialized area of machine learning and artificial intelligence that may be used in different areas, such as computer vision, speech recognition, and text translation. In computer vision, the computer learns how to interpret images to detect persons, and identify objects or scenes. Deep learning models typically use extensive resources like memory and GPU power. Having a simpler client, such as a smartphone, a digital assistant, a robot or even a PC with low-end graphics, to run those models may limit the size, accuracy, and the number of models a user can run at the same time. If the user wants a frame by frame analysis from several video sources, this may be beyond the capabilities of the device.

Even though machine learning/deep learning applications may be deployed in the cloud, some applications have specific issues that motivate local deployment, such as privacy, security, data bandwidth, and real-time low latency decisions. In terms of privacy and security, sometimes there is a concern whether the information leaves the local network of a home (e.g., video or speech of the family) or an office (e.g., videos or speech of sensitive information). Regarding data bandwidth and latency, for the cases that involve processing video streams, sending data from high resolution frames constantly to the cloud involves large bandwidth and makes it difficult to have real-time (or close to real-time) results. The dependency on the external network conditions may result in the inference (and therefore, the decision) not being made in real-time.

Some edge devices may be capable of processing machine learning at the edge. However, such devices may be insufficient if multiple tasks are to be performed. For example, if a user wants to execute object detection, face recognition, and semantic segmentation in multiple camera streams of a house, the edge device might be able to execute one of these tasks, but may not be able to execute all of them. Replicating them indefinitely may become inefficient and cumbersome.

Some examples of this disclosure are directed to a local server, called a deep learning server (DLS), to provide access to several instances of machine learning, and being scalable for more instances. The deep learning server system may include multiple computers. The deep learning server provides an interface through which many clients can request inferences from a plurality of different machine learning models based on sensor data from a plurality of different sensors, without sending data outside the local network, and without depending on bandwidth and latency of an external network service. The deep learning server may have a customizable physical and logical architecture. The deep learning server can monitor several video sources on the local network and notify clients when a prediction or inference over a video source occurs. The deep learning server can connect to several machine learning models running distributed or on the same server, and provides a robust and flexible video preprocessing pipeline, optimizing the resources for several different clients. The clients may involve many different types of devices, including robots, printers, mobile phones, assistants/kiosks, etc..

Some examples of the deep learning server essentially combine client requests for the same machine learning model and the same sensor to improve efficiency. When multiple clients request the same machine learning model over the same data source, the deep learning server identifies that situation and makes a single call to the model server. Some examples of the deep learning server use a configuration file (e.g., a JavaScript Object Notation (JSON) configuration file) to create a pipeline that communicates with the model server and performs preprocessing on sensor data before the data is provided to the machine learning model. Some examples of the deep learning server run on fast HTTP/<NUM> with gRPC protocol with binary data transfers for achieving high frame rates in predictions and inferences. The gRPC protocol is an open source remote procedure call (RPC) protocol that uses HTTP/<NUM> for transport, and protocol buffers as the interface description language.

<FIG> is a block diagram illustrating a machine learning system <NUM> including a deep learning server <NUM> according to one example. System <NUM> includes client computing devices <NUM>(<NUM>) and <NUM>(<NUM>) (collectively referred to as clients <NUM>), deep learning server <NUM>, model servers <NUM>(<NUM>) and <NUM>(<NUM>) (collectively referred to as model servers <NUM>), and sensors <NUM>(<NUM>) and <NUM>(<NUM>) (collectively referred to as sensors <NUM>).

Sensors <NUM> provide sensor data to deep learning server <NUM>. The sensor data may provide an explicit indication of an event occurring (e.g., a door sensor providing an indication that a door has been opened), or the sensor data may be data that can be provided to a machine learning model that is trained to make an inference regarding the data (e.g., a video stream that is analyzed to perform a face detection). The term "machine learning model" as used herein generally refers to a trained machine learning model that has previously undergone a training process and is configured to make inferences from received data. Each of the model servers <NUM> includes at least one machine learning model <NUM>. The clients <NUM> may send requests to the deep learning server <NUM> to monitor certain ones of the sensors <NUM> and provide the clients <NUM> with event notifications when those sensors <NUM> detect an event. The clients <NUM> may also send requests to the deep learning server <NUM> to apply a specific one of the machine learning models <NUM> to the sensor data from a specific one of the sensors <NUM>, and return the results to the clients <NUM>.

<FIG> is a block diagram illustrating elements of the deep learning server <NUM> shown in <FIG> according to one example. Deep learning server <NUM> includes at least one processor <NUM>, a memory <NUM>, input devices <NUM>, output devices <NUM>, and display <NUM>. In the illustrated example, processor <NUM>, memory <NUM>, input devices <NUM>, output devices <NUM>, and display <NUM> are communicatively coupled to each other through communication link <NUM>.

Input devices <NUM> include a keyboard, mouse, data ports, and/or other suitable devices for inputting information into server <NUM>. Output devices <NUM> include speakers, data ports, and/or other suitable devices for outputting information from server <NUM>. Display <NUM> may be any type of display device that displays information to a user of server <NUM>.

Processor <NUM> includes a central processing unit (CPU) or another suitable processor. In one example, memory <NUM> stores machine readable instructions executed by processor <NUM> for operating the server <NUM>. Memory <NUM> includes any suitable combination of volatile and/or non-volatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory. These are examples of non-transitory computer readable storage media. The memory <NUM> is non-transitory in the sense that it does not encompass a transitory signal but instead is made up of at least one memory component to store machine executable instructions for performing techniques described herein.

Memory <NUM> stores interface module <NUM>, event publication-subscription (pub-sub) manager module <NUM>, preprocessing pipeline manager module <NUM>, and preprocessing pipelines <NUM>. Processor <NUM> executes instructions of modules <NUM>, <NUM>, and <NUM>, and preprocessing pipelines <NUM>, to perform techniques described herein. It is noted that some or all of the functionality of modules <NUM>, <NUM>, and <NUM>, and preprocessing pipelines <NUM>, may be implemented using cloud computing resources.

Interface module <NUM> manages communications between the server <NUM> and the clients <NUM>, and between the server <NUM> and the model servers <NUM>. Event pub-sub manager module <NUM> manages subscription requests from clients <NUM> to subscribe to certain event notifications, and publishes those event notifications to the clients <NUM>. Preprocessing pipeline manager module <NUM> generates preprocessing pipelines <NUM> based on received configuration files (e.g., JSON files). The preprocessing pipelines <NUM> perform preprocessing on sensor data from certain ones of the sensors <NUM> (<FIG>) prior to providing the data to the machine learning models <NUM>. The functions performed by modules <NUM>, <NUM>, and <NUM> are described in further detail below.

In one example, the various subcomponents or elements of the server <NUM> may be embodied in a plurality of different systems, where different modules may be grouped or distributed across the plurality of different systems. To achieve its desired functionality, server <NUM> may include various hardware components. Among these hardware components may be a number of processing devices, a number of data storage devices, a number of peripheral device adapters, and a number of network adapters. These hardware components may be interconnected through the use of a number of busses and/or network connections. The processing devices may include a hardware architecture to retrieve executable code from the data storage devices and execute the executable code. The executable code may, when executed by the processing devices, cause the processing devices to implement at least some of the functionality disclosed herein.

<FIG> is a block diagram illustrating an example system implementation <NUM> of the machine learning system <NUM> shown in <FIG>. System <NUM> includes client <NUM>, interface <NUM>, Real Time Streaming Protocol (RTSP) cameras <NUM>(<NUM>) and <NUM>(<NUM>) (collectively referred to as RTSP cameras <NUM>), event pub-sub manager <NUM>, first machine learning inference service ("Service A") <NUM>, sensors <NUM>(<NUM>)-<NUM>(<NUM>) (collectively referred to as sensors <NUM>), interface <NUM>, and second machine learning inference service ("Service B") <NUM>. Sensors <NUM> include a presence sensor <NUM>(<NUM>), a temperature sensor <NUM>(<NUM>), and a door sensor <NUM>(<NUM>). The interface <NUM> is communicatively coupled to the RTSP cameras <NUM>, event pub-sub manager <NUM>, machine learning inference service <NUM>, and interface <NUM> via communication link <NUM>. RSTP cameras <NUM> stream video through an RTSP protocol.

In the illustrated example, first machine learning service <NUM> is a scene recognition service, and second machine learning service <NUM> is a face detection and image classification service. Dashed line <NUM> represents a network boundary, indicating that machine learning inference services, such as service <NUM>, may be provided from outside the local network. The other elements shown in <FIG> are within the local network. Client <NUM> corresponds to one of the clients <NUM> (<FIG>). Interface <NUM> corresponds to interface module <NUM> (<FIG>). RSTP cameras <NUM> and sensors <NUM> correspond to sensors <NUM> (<FIG>). Event pub-sub manager <NUM> corresponds to event pub-sub manager <NUM> (<FIG>). Machine learning services <NUM> and <NUM> each correspond to one of the model servers <NUM> (<FIG>).

The interface <NUM> accepts connections from clients, such as client <NUM>, using the gRPC protocol. In one example, the client <NUM> and the interface <NUM> use standard call/response definitions for inference. The client <NUM> may ask the interface <NUM> what machine learning models it serves, and what sensors it can monitor, and the interface <NUM> will provide responsive information. The client <NUM> may also ask the interface <NUM> to monitor a specific sensor using a specific machine learning model and a set of parameters provided by the client <NUM>, and to return detections (e.g., monitor sensor A with model M and return to me all detections with <NUM>% accuracy).

The event pub-sub manager <NUM> is responsible for notifying client <NUM> of subscribed events from sensors <NUM>, and from customized rules. When an event is detected by one of the sensors <NUM>, the event pub-sub manager <NUM> sends an event notification <NUM>, via interface <NUM>, to all of the clients that have subscribed to that event. An event may be, for example, that a person was detected by presence sensor <NUM>(<NUM>), or that the temperature sensed by temperature sensor <NUM>(<NUM>) raised above a certain threshold, or that a door monitored by door sensor <NUM>(<NUM>) was just opened. The client <NUM> may send a subscription request <NUM> (e.g., a subscription request for presence detection from presence sensor <NUM>(<NUM>)) to the event pub-sub manager <NUM> via interface <NUM> to subscribe to a specified event. Events can be simple verifications, like in the case of the temperature, but can also be something that originated after a machine learning inference was calculated. The client <NUM> may send a subscription request <NUM> to subscribe to events that originate from any of the sensors <NUM>, as well as events that result from inferences performed by machine learning services <NUM> and <NUM>, and the interface <NUM> sends event notifications <NUM> to the client <NUM> for all events to which the client <NUM> has subscribed. This may be accomplished in a publication-subscription manner.

Client <NUM> may also engage in a bidirectional communication <NUM> with interface <NUM>, which includes the client <NUM> sending an inference request to interface <NUM>, and in response, the interface <NUM> sending inference results to the client <NUM>. In the inference request, the client <NUM> may identify the video stream from RTSP camera <NUM>(<NUM>) (or another camera) and which machine learning inference service <NUM> or <NUM> to use, and may also specify that the inference is to fall within certain parameters before the client is to be notified.

In one example, the deep learning server <NUM> accesses the video streams for specific inference requests and subscribed events via interface <NUM>; captures frames from these video streams; performs preprocessing on the captured frames using preprocessing pipelines <NUM> (<FIG>); and sends the preprocessed image data to machine learning inference services <NUM> and <NUM> via interface <NUM> (and interface <NUM> for service <NUM>). For the sake of efficiency, the same preprocessed frame could be applied to multiple machine learning inference services <NUM> and <NUM>. In response, the machine learning inference services <NUM> and <NUM> apply a machine learning model to the received image data, and send resulting inferences to the interface <NUM>. The deep learning server <NUM> then sends, via interface <NUM>, inference results to client <NUM> (and potentially other clients) based on the inferences received from the machine learning inference services <NUM> and <NUM>.

The interface <NUM> allows a machine learning model to be served by a machine other than the deep learning server <NUM>. Multiple machines with multiple GPUs can be arranged in a way that, from the point of view of the server <NUM>, the models X, Y and Z are available at a particular IP address and port. Therefore, if more resources are desired, more machines can be added to this topology and more models can be installed on them.

Multiple clients requesting predictions for multiple video streams from remote cameras (e.g. RTSP cameras) using multiple machine learning models, creates some issues, such as: (<NUM>) Multiple connections to the same RTSP stream may result in reduced efficiency; (<NUM>) multiple clients asking for the same prediction (e.g., image classification) for the same stream may result in reduced efficiency; (<NUM>) a single client asking for multiple predictions over the same stream may result in reduced efficiency. Examples of the deep learning server <NUM> disclosed herein address these issues, including handling multiple clients asking for multiple inferences over multiple video streams by coalescing requests and leveraging model batching to consume computational resources efficiently.

In some examples, the deep learning server <NUM> may use the same video stream and the same machine learning model for several different clients. For example, a client may connect to the deep learning server <NUM> and ask for notifications when there is a person in the kitchen. The deep learning server <NUM> then connects to the kitchen camera and starts monitoring its RTSP stream, and evaluates each frame on a model server with a person detection machine learning model. When a person is detected, the deep learning server <NUM> notifies that client, and sends back the frame where it occurred. If a second client connects to the deep learning server <NUM> and asks for person detection on the same camera, the deep learning server <NUM> may use the same inference result to reply to the second client, since inferences for the given camera are already being made.

Different machine learning models may operate on different types of data. For example, an object detection model may operate on 299x299 images, with three color channels, standardized with a certain standard deviation. A preprocessing pipeline may be used to convert data into a format that is appropriate for a particular machine learning model. Since each machine learning model can involve a different preprocessing, due to its input expectations, the deep learning server <NUM> provides users with the ability to specify a preprocessing pipeline for any given machine learning model. The process of defining a preprocessing pipeline is described in further detail below with reference to <FIG>.

<FIG> is a diagram illustrating a preprocessing pipeline according to one example. A class diagram <NUM> of a preprocessing pipeline includes a processing unit abstract class <NUM> and concrete subclasses <NUM>, <NUM>, <NUM>, and <NUM>. Concrete subclass <NUM> is an image resize processing unit to resize an image. Concrete subclass <NUM> is a grayscale processing unit to convert an image to a grayscale image. Concrete subclass <NUM> is a threshold processing unit to convert an image to a binary threshold image. Concrete subclass <NUM> is an extra example that indicates that additional concrete subclass may be provided. In one example, the deep learning server <NUM> includes a set of existing subclasses that a user may use to create a preprocessing pipeline, and also provides the user with the ability to create custom subclasses.

Selected ones of the concrete subclasses <NUM>, <NUM>, <NUM>, and <NUM> may be instantiated and linked together in a specified manner to generate an instantiated pipeline <NUM>. As shown in <FIG>, the instantiated pipeline <NUM> includes processing unit instances <NUM>(<NUM>)-<NUM>(<NUM>) (collectively referred to as processing unit instances <NUM>) that are linked together as shown. Each of the processing unit instances <NUM> is an instantiation of one of the concrete subclasses <NUM>, <NUM>, <NUM>, and <NUM>. In one example, each of the processing unit instances <NUM> receives a data vector as input, processes the input data vector, and outputs another vector to all processing unit instances connected to the output of that processing unit instance. Multiple outputs may be generated from a single input. Processing unit instance <NUM>(<NUM>) receives input data for the pipeline <NUM>, and processing unit instances <NUM>(<NUM>) and <NUM>(<NUM>) generate output data for the pipeline <NUM>. The configuration for an instantiated pipeline, such as pipeline <NUM>, may be defined by a configuration file (e.g., a JSON configuration file). The following Pseudocode Example I provides an example of a JSON configuration file for defining a preprocessing pipeline:
<IMG>
<IMG>.

Deep learning server <NUM> may dynamically reuse preprocessing pipelines for multiple clients as long as the clients ask for the same machine learning model over the same sensor data. For example, if three clients ask for an inference over the same video stream, deep learning server <NUM> may automatically make use of the same pipeline and attach new clients to the end of the pipeline to receive the results from the machine learning model. The preprocessing pipelines make preprocessing more efficient as processing components may be reused and, by decreasing resource utilization via resource sharing, more machine learning inferences may be provided for clients with the same available hardware at the edge. The preprocessing pipelines are also flexible enough to accept different types of data sources, such as audio or video.

In addition, when there are multiple machine learning models working on the same video stream, but with different preprocessing functions, the deep learning server <NUM> can handle this situation using the preprocessing pipelines. <FIG> is a diagram illustrating a preprocessing pipeline <NUM> to receive a single video stream and preprocess the video stream for two different machine learning models <NUM> and <NUM> according to one example. The preprocessing pipeline <NUM> includes an input feeder <NUM>, an image resize processing unit <NUM>, a first tensor processing unit <NUM>, and a second tensor processing unit <NUM>. The dashed line <NUM> represents a boundary between the deep learning server <NUM> and the model servers for models <NUM> and <NUM>. In the illustrated example, a single connection to the video stream <NUM> from camera <NUM> is provided by input feeder <NUM>, which provides the video stream to image resize processing unit <NUM>. Unit <NUM> resizes each frame in the received video stream, and then has an output that branches to units <NUM> and <NUM> to convert each resized frame to two different types. Each of the machine learning models <NUM> and <NUM> has its own specific input format for images. In the illustrated example, unit <NUM> converts each resized frame to a uint8 type frame for machine learning model <NUM>, and unit <NUM> converts each resized frame to a float32 type frame for machine learning model <NUM>. Machine learning model <NUM> is an object detection model that detects objects in received frames, and outputs an image <NUM> that identifies detected objects <NUM> (i.e., a person, a dog, and two chairs). Machine learning model <NUM> is an image segmentation model that segments received frames and outputs a segmented image <NUM>.

<FIG> is a block diagram illustrating elements of the deep learning server <NUM> coupled to machine learning model servers according to one example. The deep learning server <NUM> includes a preprocessing pipeline <NUM>, which includes three processing units <NUM>, <NUM>, and <NUM> at the end of the pipeline <NUM>.

In the illustrated example, processing unit <NUM> is a Tensorflow Serving processing unit that is configured to send sensor data and inference requests to a Tensorflow serving model server <NUM>. The Tensorflow serving model server <NUM> accesses Tensorflow Runtime <NUM> and machine learning models <NUM> to provide received sensor data to the models <NUM>, and returns resulting inferences to the Tensorflow Serving processing unit <NUM>. The Tensorflow Serving processing unit <NUM> may provide inference results to multiple clients. If inferences for a different video stream are requested by a client, a new instance of the Tensorflow Serving processing unit may be created for that video stream.

The Tensorflow serving model server <NUM> also accesses Keras Runtime <NUM> and machine learning models <NUM> to provide received sensor data to the models <NUM>, and returns resulting inferences to the Tensorflow Serving processing unit <NUM>. The Tensorflow Serving processing unit <NUM> enables the deep learning server <NUM> to serve Tensorflow and Keras machine learning models, and also any C++ class that implements the Servable interface.

Processing unit <NUM> is a Caffe serving processing unit that is configured to send sensor data and inference requests to a Caffe serving model server <NUM>. The Caffe serving model server <NUM> provides received sensor data to machine learning models, and returns resulting inferences to the processing unit <NUM>. Processing unit <NUM> is a cloud API processing unit that is configured to send sensor data and inference requests to cloud APIs <NUM>. The cloud APIs <NUM> provide received sensor data to machine learning models, and return resulting inferences to the processing unit <NUM>. The flexibility of the preprocessing pipeline <NUM> allows any backend to be plugged into the pipeline. Thus, other model serving solutions may be added when they are created.

Deep learning server <NUM> is a flexible and scalable system with resources to allow the deployment of multiple machine learning models, and includes components to provide efficient management and a communication interface to allow clients to access its features. Some examples disclosed herein provide execution of machine learning methods for edge devices, without the burden and the risks of communicating with the cloud. Some examples of deep learning server <NUM> provide the following features: (<NUM>) Avoid sending private data to the cloud; (<NUM>) avoid the issue of setting up a secure connection with the cloud; (<NUM>) avoid the dependency on network latency to enable real-time decisions (e.g., for a robot); (<NUM>) avoid the cost of high bandwidth to send all the data for some inferences to the cloud; (<NUM>) enable machine learning inferences for devices with limited compute resources (e.g., mobile phones, robots, televisions, printers, etc.); (<NUM>) manage multiple inference requests and notifications efficiently; (<NUM>) enable new applications (e.g., an application where the client requests to be notified of the presence of a person in the front door, in which case, the client points the server to that camera video stream and requests a person detection inference and to be notified of an event); (<NUM>) simplify deployment of new inference models; (<NUM>) efficient communication infrastructure based on gRPC and HTTP/<NUM>; (<NUM>) efficient management of multiple data sources, like home/office cameras; and (<NUM>) efficient data preparation prior to model inference computation provided by a customizable preprocessing pipelines.

One example of the present disclosure is directed to a method of responding to machine learning requests from multiple clients. <FIG> is a flow diagram illustrating a method <NUM> of responding to machine learning requests from multiple clients according to one example. In one example, deep learning server <NUM> is configured to perform method <NUM>. At <NUM> in method <NUM>, a computing device receives a first client request from a first client that identifies a machine learning model and a sensor. At <NUM>, the computing device sends a call to a server to apply the identified machine learning model to a set of data from the identified sensor, in response to the first client request. At <NUM>, the computing device receives a second client request from a second client that identifies a same machine learning model and sensor as the first client request. At <NUM>, the computing device sends response data from the identified machine learning model to both the first client and the second client without sending an additional call to the server in response to the second client request.

Note that the response data that is sent to both the first client and the second client in method <NUM> is based on client requests involving partially overlapping intervals of sensor data, and additional response data may be sent to either the first client or the second client. For example, if the sensor generates a video stream, the video stream may be separated into frames, and the first and second client requests may specify partially but not completely overlapping sets of frames. Response data based on the overlapping portion is reported to both clients, while additional response data based on nonoverlapping portions may be reported only to the client that identified those portions.

The method <NUM> may further include receiving, with the computing device, the set of data from the identified sensor; and performing, with the computing device, preprocessing on the set of data to generate preprocessed data. The method <NUM> may further include sending, with the computing device, the preprocessed data to the model server to apply the identified machine learning model to the preprocessed data.

The preprocessing on the set of data in method <NUM> may be performed by a preprocessing pipeline in the computing device, and the preprocessing pipeline may include a plurality of processing units linked together in a manner defined by a configuration file. Each of the processing units may receive an input vector, process the input vector, and output an output vector. A first one of the processing units at a beginning of the preprocessing pipeline may receive the set of data from the identified sensor, and a last one of the processing units at an end of the preprocessing pipeline may communicate with the model server. The last one of the processing units may send the preprocessed data to the model server and receive the response data.

The computing device, the first client, and the second client in method <NUM> may all be part of a same local network. The model server may be part of the same local network, or not part of the same local network. The identified sensor in method <NUM> may be a camera sensor and the set of data from the identified sensor may be a video stream.

Another example of the present disclosure is directed to a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to: receive a first machine learning client request from a first client that identifies a machine learning model and a sensor; cause a machine learning model server to apply the identified machine learning model to a set of sensor data from the identified sensor and receive responsive inference data, in response to the first machine learning client request; receive at least one additional machine learning client request from at least one additional client that identifies a same machine learning model and sensor as the first machine learning client request; and send the responsive inference data received in response to the first machine learning client request to both the first client and the at least one additional client.

The non-transitory computer-readable storage medium may further store instructions that, when executed by the processor, further cause the processor to: receive the set of sensor data from the identified sensor; perform preprocessing on the set of sensor data to generate preprocessed data; and send the preprocessed data to the machine learning model server to apply the identified machine learning model to the preprocessed data.

Yet another example of the present disclosure is directed to a system, which includes a server computing device including an interface to receive a first client request from a first client that identifies a machine learning model and a sensor, and a second client request from a second client that identifies a same machine learning model and sensor as the first client request. The system includes a preprocessing pipeline in the server computing device to, in response to the first client request, perform preprocessing on sensor data from the identified sensor to generate preprocessed data, send the preprocessed data to a model server to apply the identified machine learning model to the set of preprocessed data, and receive responsive inference data from the model server. The interface sends the responsive inference data to both the first client and the second client. The preprocessing pipeline may include a plurality of processing units linked together, and each of the processing units may receive an input vector, process the input vector, and output an output vector.

Claim 1:
A method (<NUM>), comprising:
receiving (<NUM>), by a computing device (<NUM>), a first client request from a first client (<NUM>) that identifies a machine learning model and a sensor (<NUM>);
sending (<NUM>), by the computing device, a call to a model server (<NUM>) to apply the identified machine learning model to a set of data from the identified sensor, in response to the first client request;
characterized in that it also comprises:
receiving (<NUM>), by the computing device, a second client request from a second client that identifies a same machine learning model and sensor as the first client request, said first and second client requests involving at least partially overlapping intervals of the set of data from the identified sensor; and
sending (<NUM>), by the computing device, response data based on the at least partially overlapping intervals from the identified machine learning model to both the first client and the second client without sending an additional call to the model server in response to the second client request.