NoScope: Scaling Neural Network Queries to the Real-World
By: Kaushik Mahorker
Tags: Machine Learning, Deep Learning, Research, Video Analytics

Neural networks have expanded the breadth of tasks we can perform and revolutionized the accuracies at which we can achieve them. However, they are computationally expensive and thus make it difficult to perform tasks at volume and scale — a necessity for most real world applications. This article explores one of the various state of the art research that provides system level and architectural optimizations to scale these payloads into real-world applications. In particular, we will be exploring queries over video data, a medium that increasingly needs real-time responses and carries the challenge of large file sizes.

Let’s start with a scenario.

Your company provides traffic analytic services at city intersections and has recently trained a neural network to identify various traffic scenarios such as accidents, stopped cars, red-light violations, etc from security cameras. As your business grows, you deal with and process more footage from more security cameras.

Now, you’re approached by a city to help them track accidents at certain key intersections. What optimizations can we make to identify accidents more efficiently?

NoScope: Optimizing Neural Network Queries over Video at Scale — Kang et al.

NoScope introduces a system for querying videos that can significantly lower the cost of analysis via inference-optimized model search. This system leverages the reference neural network and input video to generate a cascade of models consisting of specialized neural networks and difference detectors. NoScope generates results at the same levels of accuracy, but at almost one-third the original cost.

Architecture

NoScope Architecture

As shown in the upper half of the architecture diagram, NoScope applies the references model to a subset of the target video and generates labeled examples. In our presented scenario, NoScope would identify frames as “accident” or “no accident”.

Next, NoScope trains a specialized model that is much shallower than the reference model. Specialized models forego the generality of a reference model, but offer similar behavior on a specific task. Specialized models at one intersection are unlikely to generalize to other intersections or other queries, but a new specialized model can be generated at each intersection.

Frame differencing

NoScope also generates a difference detector that checks to see whether something new has happened in the frame. Security cameras generally have fixed orientation, so a fixed reference image can be used to determine whether the current frame is empty. If there is no difference between the current frame and reference image, this frame is simply ignored as it is also empty.

If there is a difference between the current frame and the reference image (i.e. the figure above), the current frame is also compared against an earlier frame that we know the label for. If these frames are similar, then the same label is applied without tasking the neural network, otherwise the specialized network is used and the label is stored.

These two points of differencing allow for short-circuit evaluation and NoScope reveals that omitting frames by leveraging historical information can help significantly reduce load.

Evaluation

Accuracy vs speedup achieved by NoScope on various datasets

The authors evaluated multiple datasets on binary classification tasks with a Tesla P100 GPU and 32 Intel Xeon E5–2698 v4 cores in each experiment. Clearly, these results depict that NoScope is able to achieve many orders of magnitude of speedup with none to little loss in accuracy. It also shows that this approach is just as effective between a variety of datasets and target objects.

Conclusion

NoScope helps bridge the gap between deep learning research and real-world applications by making neural networks significantly cost effective. It leverages specialized neural networks and intelligently selects frames to task for neural network evaluation to deliver the same results at about the third of the cost.

Revisiting the scenario. Your traffic analytics business can leverage the techniques presented in NoScope to specialize its neural network for identifying accidents. Using a specialized neural network for each intersection and NoScope’s difference detection, the load from many cameras can become distributed and significantly reduced.

NoScope delivers incredible savings on computation for real-world applications and has inspired more research into bringing powerful neural networks onto compute limited platforms.