VideoFlow: A Framework for Building Visual Analysis Pipelines
Authors: Yue Wu
Jianqiang Huang
Jiangjie Zhen
Guokun Wang
Chen Shen
Chang Zhou
Xian-Sheng Hua
Venue:  (2021)
Keywords: Computation graph
Resource
Computer vision
Deep learning
Framework
Software

Abstract: The past years have witnessed an explosion of deep learning frameworks like PyTorch and TensorFlow since the success of deep neural networks. These frameworks have significantly facilitated algorithm development in multimedia research and production. However, how to easily and efficiently build an end-to-end visual analysis pipeline with these algorithms is still an open issue. In most cases, developers have to spend a huge amount of time tackling data input and output, optimizing computation efficiency, or even debugging exhausting memory leaks together with algorithm development. VideoFlow aims to overcome these challenges by providing a flexible, efficient, extensible, and secure visual analysis framework for both the academia and industry. With VideoFlow, developers can focus on the improvement of algorithms themselves, as well as the construction of a complete visual analysis workflow. VideoFlow has been incubated in the practices of smart city innovation for more than three years. It has been widely used in tens of intelligent visual analysis systems. VideoFlow will be open-sourced at \url{https://github.com/xxx/videoflow}.

Under review as a conference paper at ICLR 2021
VIDEOFLOW: A FRAMEWORK FOR BUILDING VISUAL ANALYSIS PIPELINES
Anonymous authors Paper under double-blind review
ABSTRACT
The past years have witnessed an explosion of deep learning frameworks like Py- Torch and TensorFlow since the success of deep neural networks. These frame- works have significantly facilitated algorithm development in multimedia research and production. However, how to easily and efficiently build an end-to-end visual analysis pipeline with these algorithms is still an open issue. In most cases, de- velopers have to spend huge amount of time tackling data input and output, optimizing computation efficiency, even debugging exhausting memory leaks together with algorithm development. VideoFlow aims to overcome these chal- lenges by providing flexible, efficient, extensible, and secure visual analysis framework for both the academia and industry. With VideoFlow, developers can focus the improvement of algorithms themselves, as well as the construction of a complete visual analysis workflow. VideoFlow has been incubated in the prac- tices of smart city innovation for more than three years. It has been widely used in tens of intelligent visual analysis systems. VidcoFlow will be open-sourced at nttps://gitt
com xxx http///xxxvvdieloo..vo
INTRODUCTION
The success of computer vision techniques is spawning intelligent visual analysis systems in real applications. Rather than serving individual models, these systems are often powered by a workflow of image/video decoding, several serial or parallel algorithm processing stages, well sinking analysis results. The varied visual analysis requirements in different real scenarios put forward a high demand on framework for fast algorithm development, flexible pipeline construction, efficient workflow execution, as well as secure model protection.
There exist frameworks approaching Stream (Purandare] 2018) and MediaPipe (Lugaresi et al., (2019). DeepStream on top of GStreamer (GSTREAMER 1999), which primarily targets audio/video media editing rather than analysis. MediaPipe can be used to build prototypes to polished cross-platform applications and measure performance. Though it is flexible and extensible on calculators, efficiency, model security, and extension on more aspects are expected by real online services in industry.
of the above mentioned targets, like Deep-
In this paper, we present VideoFlow, meet the visual analysis requirements for both algorithm development and deployment in real systems with the following highlights.
Flexibility. VideoFlow is designed around stateful Computation Graph and stateless Resource, Computation graph abstracts the visual processing workflow into a stateful directed acyclic graph. Developers can focus on the implementation of processing units (graph nodes) and the construction of the whole workflow. Resource is stateless shared computation module of computation graphs. The most typical resource deep learning model inference. Resource decouples the stateless visual processing components from the whole complicated visual analysis pipeline, helping developers focus on the optimization these computation or Input/Outpui((OO intensive implementation.
Efficiency. VideoFlow is designed for better efficiency from four levels. (1) Resource-level: sources can aggregate the scattered computation requests from computation graph instances into intensive processing for better efficiency. (2) Video-level: all videos analyzed in parallel in shared execution engine. (3) Frame-level: video frames are parallelized on operations which are irrelevant to frame orders. (4) Operator-level: visual analysis is multi-branch pipeline in most

---

Under review as a conference paper at ICLR 2021
cases. The different branches and different operators of each branch without sequential dependency are analyzed in parallel.
Extensibility. VidcoFlow is designed from the beginning to be modular possible to allow easy extension to almost all its components. It can be extended to different hardware devices like Graphic Processing Units(GPU), Neural Processing Unit (NPU), etc. It can be hosted either x86 or ARM platforms. Developers can customize their own implementations with VideoFlow as a dependent library. The extended implementations can be registered back to VideoFlow as plugins at runtime.
Security. Model protection is an important problem in industry. VideoFlow encodes model files into encrypted binary codes part of the compiled library. The secret key can be obscured into the same library, or exported to separate key management service (KMS). At runtime, VideoFlow decrypts the models and verifies authorization from remote service periodically.
VideoFlow has been incubated in the practices of the smart city innovation for more than three years. It is designed for computer vision practitioners, including engineers, researchers, students, and software developers. The targets of VideoFlow include: 1) free developers from the exhausting data loading/sinking, parallel programming and debugging to the optimization of algorithms; 2) enable easy extension of video decoding, deep model inference and algorithm implementation; 3) provide highly efficient framework for large scale visual processing in industry rather than just experimental prototypes. 4) protect the intellectual property of models and algorithms to make sure that they can only work with authorization.
2 RELATED WORK
2. DEEP LEARNING FRAMEWORKS
Almost all existing deep learning frameworks like Caffe (Jia et al. 2014). TensorFlow (Abadi et al. 2016), PyTorch (Paszke al. 2017), MXNet (Chen et al.) 2015) describe networks in directed graphs even dynamic graphs VideoFlow draws lessons from this successful design for visual analysis. The difference is that the basic units in deep networks are low level operations like convo- lutions, compared to higher level processing like object tracking in VideoFlow. The data transferred between operators in VideoFlow is also much more complex than the Tensor in deep learning.
As to model inference, there are some specially optimized engines, like TensorRT (Vanholder (016) and MKL-DNN/nnAAPI (Intel) by hardware manufactures. In the open source community, devel- opers put forward TVM for easy extension to different hardware for more effective inference (Chen) jet al. 2017). On top of these engines, there are some serving platforms for individual models rather workflow construction, like tensorflow serving (Google 2016), NVIDIA Triton Inference Server (Goodwin & Jeong 2019). VideoFlow integrates these inference engines as Resources with their C++ interfaces.
2.2 VISUAL ANALYSIS FRAMEWORKS
The recent has witnessed some visual analysis frameworks. Nvidia launches the DeepStream project in the early days for video analysis on GPU (Purandare) 2018). It is oriented as well as optimized on GPU and TensorRT, regardless of the bustling heterogeneous hardware devices. Besides, it is built on top of GStreamer (GSTREAMER 1999). which primarily targets audio/video media editing rather than analysis, limiting its flexibility and extensibility. The nn--vidoo-anllttin project (Intel) 2019) is also built on top of GStreamer Deuermeyer & Andrey). Google proposed MediaPipe by building graphs for arbitrary streaming data processing with computation graph as well (Lugaresi et al., 2019). MediaPipe can be used to build prototypes to polished cross-platform applications and measure performance. Though it is flexible and extensible on calculators, real online visual analysis expects extension on more aspects, more efficiency optimization, and model security pro- tection. Compared to MediaPipe, VideoFlow features these advantages for better application in both academia and industry. Another framework also named Videoflow (de Armas 2019) is designed to facilitate easy and quick definition of computer vision stream processing pipelines. However, it is just prototype experimental platform, with limitations extensibility, efficiency, and security.
2

---

Under review as a conference paper at ICLR 2021
Operator (Stataful, Ordwwad
Computation (Statelu))
Dacss
Deteckul
Treckng
F-ldirker
Deled
Erannice
Ealerakn
20A
Pbel Convention
Oynane ROUER MO
Pre-Proeemitg Exteramn
Culla
Turnoft!
(2ee
Dalinase Connecty
Gragh Consdmatiion
APRcsit Osnnecto
Kawer
Piscesung Extersion
Oaee Storago Chere
Witeekeo
Premneiinal
Ozerslor-ensl
Foward Pamileito
In-dercce sisonsoin
Ersinsiee
Badkword
Video Decoding
CPU. GPU, NPU.
Resource (Stateless, ComputationiO Hnnssive
W.U. IPGA.
Execution Scheduzing
Figure I: Framework of VideoFlow.
3 ARCHITECTURE
VideoFlow is oriented around stateful Computation Graph and stateless Resource with well- optimized execution engine. Computation graph is a directed acyclic graph describing the whole workflow of video image analysis. As the two main components of a graph, Node and Edge de- note visual processing operators and data flow between operators, respectively. Resource is shared for graph irrelevant computation. The architecture is shown in Figure T
3.1 OPERATOR
Operator is the basic unit of visual analysis workflow. An operator depends on the outputs of its par- ent operators. Its own outputs can be consumed by arbitrary number of child operators. According the number inputs an outputs, operators are categorized follows:
Entrypoint: operators that have zero inputs. This is the start of computation graph. Each graph can have only one entrypoint.
Processor: operators that have at least one input and at least one output. Processors occupy most of the workflow of visual analysis. It's also the main kind of operator with the highest demand on easy extension.
Sinker: operators that have zero outputs. This is the end of a computation graph. A graph can have multiple sinkers.
3.2
DATA FLOW
Data flow is the edge connection between two operators (nodes). An operator may generate several number of data with different types for its child nodes. Data flow is a collection of arbitrary number of data pointers of arbitrary type (vector<voing idw>> in C++ implementation) in VideoFlow. VideoFlow guarantees that the incoming data pointers are always safe to be read. Developers do not need care how many other operators are also consuming the data, or whether the data should be released during the workflow.
3.3 RESOURCE
Resource is the stateless computation unit shared by graphs. The most representative resource is deep model inference. Resource is abstracted due to three main reasons. Firstly, many operations like deep model inference and data sinking to databases have their own independent semantics. They are irrelevant to whether it is used for video or image processing, which step of the whole pipeline invokes the operation, or how the outputs will be post-processed. Secondly, these operations are often computation or IO intensive. Leaving them the operators will incur bottlenecks on CPU, memory or network bandwidth due to large amount of resouree competition. Gathering the scattered but shared requests from different graphs for uniform processing proves to be a good practice to improve efficiency. Thirdly, resource can be improved without affecting the visual analysis logic.
3

---

Under review as a conference paper at ICLR 2021
Video
Video
Forward Function
Backward function
forward
forward
ready
ready?
<run>
entry?
sinker
k
Self backwerd
forward
Orderk
Forward
Parent Backward
Odeer
Orter
Thread
Pool
task queue
Figure 2: Execution Scheduling of VideoFlow.
For example, we can accelerate the inference speed of a PyTorch model by switching TensorRT worker. We can change to a more efficient database connector for more real-time data sinking. Without the abstraction of these resources, all affected operators have to be re-implemented to the benefits.
3.4 GRAPH CONSTRUCTION
Computation graph is described in json format with the following fields.
"resource" describes the resources that will be used by operators. Each resource should have two fields: "type" to create the correct class instance and "params" to specify the resource configurations.
"ops" describes the operators that will be used to construct computation graphs. Operators be used multiple times by different graphs. As the same to resource, each operator should have two fields: "type" and "params".
"graph" is the place to define computation graphs. Each graph definition a json dictionary of key-value pairs. Key is the operator name. Its value list of operator names as child nodes.
"subgraphs""optiona is used to re-use resources, operators and graphs from other graph configu- ration files.
specifies external dynamic libraries that should be loaded by VideoFlow, especially the extended libraries in Figure 3
"libs"[optional]
"config" | is for global settings, currently including number of parallel image processing threads and number of frames for parallel video processing.
An example file is provided in the supplementary material to show the person reidentification work- flow (Section 5)
3.5 EXECUTION SCHEDULING
With the graph defined and constructed, execution scheduling determines which operator should be calculated. In real cases, there can be multiple computation graph instances running in parallel, each with either shared or different structures. Figure shows the execution scheduling of these graphs. Each graph has several replicas, with each replica called actually processed these graph replicas/orders. The replicas are processed in parallel for frame- level parallelism. Each order starts from the Forward function of the entrypoint node.
order. Video frames
Forward. As Figure 2 shows, the forward function first checks if the current operator is ready to be executed. The readiness checking includes: 1) all parents of the current node have finished this order. 2) the previous order of the current node has been executed if the current node is an ordered operator. ready, the forward function puts its processing function into the task queue

---

Under review as a conference paper at ICLR 2021
User Customiztation Area
buildsmme
runtime
Extended Implementations
</> CPP
1011
1011
DLL
Comple
Model Files
Source Codes
register
vf-codify
include/vf
libvideoflow.so
EXE
1DDO
VideoFlow
>.
Illustration of extension. Developers depend VideoFlow to build their library at build- time. At runtime, the generated library is loaded by VideoFlow dynamically, Extended operators, resources and other modules are registered back VideoFlow.
Figure 3:
of the exccution engine, waiting to be executed. The processing function first finishes the internal processing logic of the operator. After that, it calls the Forward function of its following operators. it is the leaf operator, it calls the Backward function of its own. Forward of entrypoints is specially implemented with separate thread retrieving and dispatching data to idle orders.
Backward is the process to reset the node to be ready to process latter frames. The backward function first checks if all its children have been reset. If so, it resets its forward status. Then it continues to call the Backward of all its parents.
Execution Engine. The processing functions of operators put into task queue of the execution engine. All processing units share the same interface. The execution engine does not know which order of which graph processing function comes from. All orders and all graphs are executed concurrently once they are put into the queue. Inside the engine there is thread pool, with all threads fetching and executing tasks from the queue.
3.6 DEEP MODEL INFERENCE
Deep model inference the most typical computation intensive resource in visual analysis. The built-in implementation covers deep learning frameworks (Caffe, Py PyTorch, MXNet and accelera- tion toolkits (TensorRT, TVM, etc.). Developers can customize their own workers easily. VideoFlow adopts dynamic batching and in in-device processing to fully utilize the capability of heterogeneous hardware devices. On GPU, it supports multi-stream for better performance.
Dynamic Batching. Heterogeneous hardware devices often need work in the batch mode for full utilization. A single call of a batch of data is much more efficient than multiple calls single data. In VideoFlow, the input of deep learning inference defined for deep learning input and output of single task. VideoFlow defines a thread-safe queue collect the scattered DLTask from operators form DLTaskBatch. Deep inference workers request DLTaskBatchh from the queue with a timeout mechanism. The timeout is essential since videos, frames, and operators are not strictly time aligned. The timeout can try to collect as many tasks as possible in the limited period, while keeping the processing latency limited at the same time.
batch of DLTask, which is structure
In-device processing. Data transfer across host and device memory is cost expensive and time consuming. In VideoFlow, frames are defined with hardware contexts. It provides hardware-specific image operations like resizing, cropping, pixel format conversion, mean subtraction, scaling, etc. All these operations including dynamic batching are conducted their most appropriate hardware devices according to their hardware contexts. GPU-decoded frames are pre- pre-processed and analyzed on the same GPU card. CPU-decoded frames are pre-process propproceseed CPU, but analyzed device with the lowest expected latency. The in-device processing lowers the system cost significantly, especially the CPU cost, as will be verified on the Section
Currently TensorFlow is partially supported due difficulties in integrating its C++ interface and library.
5

---

Under review as a conference paper at ICLR 2021
3.7 EXTENSION OF VIDEOFLOW
Library extension is often intrusive. Developers have to write and compile their code together within the libraries. Though it works, the drawbacks are: 1) The total compilation time is getting longer with the library becomes more and more complex; 2) It is really hard to integrate extensions from different developers except that they share the same code repository. However, the different exten- sion may come from different teams, with their concern for code and model protection.
VideoFlow provides more convenient way for extension as Figure 3 shows. Developers can tomize their own implementation and generate their own libraries. These libraries can have their own model protection braries specified in the "libs" field of the graph configuration file at runtime. Throughout VideoFlow, registration is widely used for extension of all modules. By this way, VideoFlow enables coding sep- arately, but working together.
authorization. With the registration mechanism, VideoFlow will load li-
3.8 OPERATOR EXTENSION
Operator interface is highly simplified for easy customization. All operators derive from the same base class Op. As the start of graphs, customization of entrypoints requires little bit more care of data input and workflow interruption. These are highly dependent on how the graph will interact with outside callers. The built-in entrypoints have covered most cases of visual analysis. Except for entrypoints, there are only five steps to customize a new operator: construccion/ddesrrutttin initialization, auxiliary memory management, processing, and registration. We provide detailed example code an object detection operator in the supplementary material.
constrcciionDestruccion In construction function, developers should firstly make clear whether the current operator can be parallelized on the frame-level, which called ordered. Frame sequence order independent processing should all be declared unordered for better parallelism to boost the overall efficiency. The second is the number of parent operators and the input data type list. The last is the output type list of the current operator. Note that "*" is allowed during type specification for wildcard type. The output types of the parent operators will be checked to see whether they match the input types of child operators during graph construction.
Initialization. The Init function is used to initialize some settings after the graph has been con- structed, but before the actual analysis begins.
Auxiliary Memory Management. Auxiliary memory is the frequently used temporal memory during processing. The life-cycle of auxiliary memory is the same with the graph. Auxiliary memory can be used as the output data. To be specific, developers need to override the Ma MallocResource and ReleaseResouree
functions for auxiliary memory management.
Processing. This is the place to process the input data and generate the output data. The run function will be called again and again for visual analysis. All parallelism are optimized around this function, though developers do not need to care about the detailed mechanism. They only need to remember not to write shared memories (like class member variables or global variables) if the operator is declared as unordered for thread-safety, since there can be multiple threads executing the same run function on different frames. Note that the auxiliary memory will be allocated for each order of the graph. It is safe to operate on the auxiliary memory without thread-safety concern no matter the operator is ordered or not.
Registration. Registration is just macro register the operator to VideoFlow that it can be constructed according to its name.
3.9 DECODER EXTENSION
Video decoding needs to tackle various video sources like online camera recording, web streaming, local files, or even non-standard video transfer protocols. Besides, it should make full use of the hardware decoding modules which are widely equipped with modern heterogeneous hardware de- vices. VideoFlow abstracts video decoding into frontend and backend. The frontend tackles various video sources. For backend, there be different implementation different hardware devices.

---

Under review as a conference paper at ICLR 2021
These backends register to DecoderBackend with their supported codecs, decoding capabilities, and priority.
To extend video decoding, developers choose to implement new video frontend new de- coding backend. Interfaces of both the frontend and the backend are simple (Open, Put, Get, close). VideoFlow makes that frontend developers do not need to care about decoding accel- eration with hardware devices. Backend developers do not need to care about where the video data comes from or how demux the data packets.
3.10 DEEP MODEL INFERENCE EXTENSION
VideoFlow provides built-in inference support for Caffe, MXNet, and Py Torch, as well as TensorRT and TVM (Chen al. 2017). Nevertheless, there are still three main scenarios extend deep model inference: new inference backend, output post-processing, and input pre-processing.
Inference Backend means extend new deep learning or hardware acceleration frameworks. There are three functions to be overridden: Init, FeedData a, and process. The Init function should determine the hardware context, parse the models. check model imput and output, and allocate in- put/output memory buffers. FeedData a is used to pre-proces process and feed a batch of data to the inference framework. process is private function invoke the inference after the input data ready. The output is written back to DLTask in this step.
Output Post-processing. This quite common for frameworks like TensorRT since there are still quite large number of operators not supported. In many cases, developers can accelerate the back- bone network with these acceleration toolkits. The left unsupported procedures can be implemented by overriding the process function of existing inference backends.
Input Pre-processing. VideoFlow provides scaling with mean subtraction and data copy as the built-in pre-processing methods for input image data. To customize VideoFlow defines an empty template structure named PreP rocessF Func unc with device type, pixel format, and pre-prooe rre-pocesssing type its template parameters. Developers just need to implement new template specialization for their own pre-processing.
pre-processing function,
3.11 SECURITY
Model protection is key challenge in deployment. VideoFlow provides security mechanism as FigureB]showe vf-codify converts model files to encrypted source. The encryption key is obfuscated in the source codes avoid library file parsing. In real production deployment, the key should be deposited into a Key-Manngement-Serve
(KMS) for higher security level.
At runtime, the security module tries to request authorization and decryption key from remote service. If authorized successfully, it will decrypt the models into memory. Users may still have the concern if others can peep their models from the memory. This problem does exist real cases. Attackers may dump the whole memory and steal the model parameters. A possible solution may be that hardware manufactures provide a safe memory region for model parameters, or part of model parameters. Anyway, security is endless game of attack and defense. We welcome more open source contributions for better security.
TOOLS
VideoFlow provides a graph editor to help users write their computation graphs, visualizer to visualize the real-time analysis results of algorithms, like the detected objects, object trajectories, well as profiler to show the running status of each video channel and deep learning models. These tools can significantly benefit users for fast pipeline construction and optimization. Detailed illustration of the tools is shown in the appendix.
5 PERSON REID APPLICATION EXAMPLE
In this section, we show an application example with a person reidentification (ReID) system. We test the performance benchmark of some measurable aspects of VideoFlow verify its efficiency.

---

Under review as a conference paper at ICLR 2021
Table 1: Models used for Person RelD. Model
Task
Framework
Acceleration
Person Detection
SSD
Liu et al.
2016)
Caffe PyTore
TensorRT
Person Attribute
APR
Lin et al.
2019)
TensorRT
Person Feature
Resnet50 Guo
2020
MXNe
TensorRT
Table 2: Evaluation of Person ReID Pipeline.
Optimization
Resource Consumption
Frame Parallel
GPU
In-device
GPU Memory
Video Parallel
Average FPS
TensorRT
CPU
Memory
GPU
Decode Processing
95%
9.0%
30%
12%
17
4
290%
9.5%
70%
30%
2
25
4
1560%
9.7%
56%
30%
25
4
1100%
10.2%
70%
39%
25
4
800%
10.2%
70%
39%
25
8
950%
11.1%
84%
45%
12
25
Person ReID is widely explored in recent computer vision community (Song et al. 2019 Yu al. 2019
Sun et al. 2018 Wei et al.] 2018 Zhao et al.) 2017: Wei et al.) 2017 Hermans et al. 2017). typical person ReID video processing pipeline is shown Figure in the appendix. We sam- ple once every 5 frames due large content similarity. The decoded frames
processed with person detection and person tracking. After then, two branches extract the ReID feature and the person attributes, respectively. Sinkers include Vi sual 1.S. Sinker for real-time visualization and rPSSinker to calculate the processing speed. The pipeline is deployed on a cloud virtual machine with Intel Xeon E5-2682 CPU (16 cores), 64 GB memory, and Nvidia T4 GPU card.
Data. We choose MOT16-07 video sequence from mot challenge (Milan et al., 2016). We convert its resolution 1920x 1080 and push it with FFmpeg in loop rtsp video service (Schulzrinne et al. 1998). Different channels of VideoFlow pull the same video stream to mimic parallel video analysis.
Models.
The three models are trained with Caffe, PyTorch, and MXNet to verify the wide support of deep learning frameworks, as shown Table These models will be accelerated with TensorRT for fast inference.
Pipeline Benchmark. Table 2 shows the efficiency evaluation of the whole pipeline. We choose some optimization aspects that can be quantified to verify the efficiency. Without frame parallel processing, even single channel video cannot be processed in real-time. Note that the workflow in this example is not that complicated. With four orders of frame parallelism, VideoFlow can process 2 video streams. TensorRT significantly boost the capability to channels, as well as the CPU cost to almost 16 cores. GPU decoding reduces 5 cores of CPU consumption, with still 11 cores left mainly for person tracking. With in-device processing and larger frame parallelism, we can reduce about 1.5 cores of CPU consumption while improving concurrent real-time video processing to 12 channels.
6 CONCLUSION
In this paper, we present VideoFlow, a computation graph and resource based framework for visual analysis. We illustrate its superiority from the aspects of flexibility, efficiency, extensibility, and security. VideoFlow can help developers focus on algorithm improvement and the construction of visual analysis workflow. It is a carefully designed and implemented framework for the ease of visual analysis without bias to any hardware manufacturers, devices, platforms, or computer vision frameworks. can be used in both academic and industrial scenarios.
VideoFlow has been widely used in intelligent visual analysis systems. We will open-source the project welcome the contribution of the community for more implementation of operators, more adaptation of different hardware devices, as well as better optimization of the framework itself.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. Tensorflow: system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Tianqi Chen, Mu Li, Yutian Li. Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 2015.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, and Haichen Shen. Tvm: An end to end ir stack for deploying deep learning workloads on hardware platforms, 2017.
Jadiel de Armas. Videoflow. httpppss/gg..tu...ooooocooeoeeea
2019.
Deanne Deuermeyer and Z Andrey. Openvino toolkit release notes.
David Goodwin and Soyoung Jeong. Maximizing utilization for data center inference with tensorrt inference server. Nvidia GTC Silicon Valley, 2019.
Google. Tensorflow serving. hutpp::/ggLLuub..cm//eeeeerrerre
2016.
GSTREAMER. Gstreamer: open source multimedia framework. 1999.
URL. https://
gstreamer. t freedesktop.corg/
Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, et al. Gluonev and gluonnlp: Deep learning in computer vision and natural language processing. Journal of Machine Learning Research, 21(23))1-7, 2020.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv preprint arXiv: 2017.
Intel. oneapi deep neural network library. URL https::/c1..rg////
Intel. The gstreamer video analytics plugin, part of openvino toolkit dl streamer. https: //github. wwwp///pww.//stttoooo.oo... /ttpttataa..cccoo//aoa/aaeeeaeta -at
2019.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. Proceedings of the 22nd ACM international conference on Multimedia, pp. 675-678, 2014.
Yutian Lin, Liang Zheng, Zhedong Zheng, Yu Wu, Zhilan Hu, Chenggang Yan, and Yi Yang. Im- proving person re-identification by attribute and identity learning. Pattern Recognition, 95:151- 161, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pp. 21-37. Springer, 2016.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja. Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Man- fred Georg, and Matthias Grundmann. Mediapipe: framework for building perception pipelines. CoRR, abs/1906.08 17,1,,2,00 2019.
Anton Milan, Laura Taixé. Ian Reid, Stefan Roth, and Konrad Schindler. Mot16; A benchmark for multi-object tracking. arXiv preprint arXiv: 00831, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Kaustubh Purandare. An introduction to deepstream sdk. Nvidia GTC, 2018. URL https:// developer. nvidia com dwwpstepptttteeeee ream- sdk

---

Under review as a conference paper at ICLR 2021
Henning Schulzrinne, Anup Rao, and Robert Lanphier. Real time streaming protocol (rtsp). 1998.
Jifei Song, Yongxin Yang. Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Generalizable person re-identificatiin by domain-invariant mapping network. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 719-728, 2019.
Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In The European Con- ference on Computer Vision (ECCV), September 2018.
Han Vanholder. Efficient inference with tensorrt, 2016.
Longhui Wei, Shiliang Zhang, Hantao Yao, Wen Gao, and Qi Tian, Glad: Giobal-local-alignmeet descriptor for pedestrian retrieval. In Proceedings of the 2017 ACM Multimedia Conference, pp. 420-428. ACM, 2017.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person trasfer gan to bridge domain gap for person re-identification. Computer Vision and Pattern Recognition, IEEE Conference on, 2018.
Tianyuan Yu, Da Li, Yongxin Yang, Timothy M Hospedales, and Tao Xiang. Robust person re- identification by modelling feature uncertainty. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 552-561, 2019.
Haiyu Zhao, Maoqing Tian, Shuyang Sun, Jing Shao, Junjie Yan, Shuai Yi, Xiaogang Wang, and Xiaoou Tang. Spindle net: Person re-identification with human body region guided feature de- composition and fusion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1077-1085, 2017.
10

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
The appendix describes the tools provided by VideoFlow for computation graphs construction, real- time analysis visualization, and operator performance profiling.
A.1 GRAPH EDITOR
Graph editor aims to help users write their computation graphs. As shown in Figure it is comprised of three main panels: operator list, graph editor, and graph visualizer. The operator list panel shows the list of all operators inside library. Users can import several library metadata into this tool. Libraries can switched in this panel. The graph editor panel supports smart editing of graphs with auto-complction, grammar checking and auto-formatting. The graph visualizer panel synchronizes the graph definition the editor into a graphical view to help users review the whole processing workflow.
arratot list
Graph editor
Graph visualizer
Takke wdt's Ptraaaaaawcoeeett........ jee", wwn.ect
Mccad
Sibicton-1
Natect
"erpct Pfeatue,, "attributell,
"Aut-anth-iysoe Vpeddioer "sousliee
Ve be
Ep
15
9
Cantrowbbaaoor
"decodet:
"Lar" "abkdeoeen"
Deeptortadeer
1
Neightt Fall
Scate 1.3%
Simtune's
"sstect:
"Lpe "tetectts
Figure 4: Computation Graph Editor & Visualizer.
A.2 VISUALIZER
Visualizer is actually an image stream player, with abilities to draw rectangles and texts on images. It is used to visualize the real-time analysis results of algorithms, like the detected objects, object trajectories, object attributes shown in Figure [ It especially useful during the algorithm development period. In production, it can be used to check if algorithms are running or if algorithms perform well. Currently analysis results. A video player is expected from the community with the functionality to display frames together with the time aligned elements from algorithms.
an image stream player for better alignment of video frames and
A.3 PROFILER
One of the major things in development is to profile the performance of the overall system well as each of its components. Profiler of VideoFlow has two views: channel view and resource view. Channel view displays the running status of each video channel. As shown in Figure 6 there 12 orders for the selected channel. The horizontal axis is the timeline. The green blocks are the executed operators well as their execution time. The figure also verifies frame-level parallelism. The resource view currently displays the running status of deep learning models. With the horizontal axis denoting the timeline and vertical axis denoting the batch size of each inference. This can help

---

Under review as a conference paper at ICLR 2021
IEER
REVIEW
videoflow test person videoflow test-O
person
person
person
videoflow_tes test-
handbag no
no upred
videoflow Lestswweess1100
skirt no
vidcoflow_test test 2
adult
hat
no
videoflow_test-3
videoflow test
no
50
cotta
Figure 5: Real-time Analysis Visualizer.
Chunnei
Con Gociene
Durston
Griph
Order
DLP Pocessor
Channel
Resource List
11
Legand
Detector 2
MilonCooddrn1
Visua Briker8
Profiling of Video Orders
Figure 6: Performance Profiler.
to check heterogeneous hardware resources are fully utilized or overloaded. In cases where graph uses many deep models, it can help to analyze the bottleneck of the overall system throughout.
12


Reviews:
[{"review_id": "wYvd3_7Xb6j", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript describes a framework, VideoFlow, for deploying and analyzing video execution flows. VideoFlow abstracts the workload into computation graph and resource, and dynamically executes the graph given the resource.\n\nAlthough this work seems helpful for system integration and deployment, I did not see a strong connection between it and the topic of this conference. In this framework, learned models are treated as an atomic node and executed on other DL libraries. The graph mentioned in this paper, as well as terms like forward/backward, has nothing to do with the graph in machine learning concept. Therefore, I do not see this paper as a good candidate for machine learning community, but a better fit for system communities.\n\nThere are some additional comments:\n1. Figures should come with captions to illustrate themselves, instead of just a title.\n2. Figure 2 is confusing. It is hard to infer from the figure what forward function is. The authors need to emphasize this forward/backward function is different from DL models' forward/backward.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ErXNDYINUT0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of contribution:\n\nThis paper presents VideoFlow, a domain-specific deep learning framework that focuses on building pipelines for video analytics. The goal of VideoFlow is to facilitate both development and deployment of visual data analysis pipelines for real-world applications. The paper describes the design and implementation of this framework. VideoFlow seems like a useful tool, but its novelty and contributions to the state of the art have not been clearly articulated in the paper.\n\nStrong points:\n- The need for VideoFlow is well motivated.\n- Flexibility, efficiency, extensibility, and security have been identified to be the key design goals for VideoFlow's architecture.\n- VideoFlow has been in practical use in the smart city domain for the past few years.\n- VideoFlow supports utilizing heterogeneous hardware.     \n\nWeak points:\n- The paper reads more like a system demo paper. Its contribution as a research paper is not clear.\n- Good coverage of related work, but it is not clear how exactly VideoFlow compares to the state of the art. In what ways is it superior? No experimental evidence and/or user/case studies have been provided to show this.\n- There is not enough experimental evidence in the paper that the four design goals are all completely met.\n- It would have been nice to include more experience from the smart city deployment.\n\nAdditional comments:\n- Open-sourcing VideoFlow beyond its current incubated use would help enable community's use, feedback, and contributions.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BfSI5ZHhobA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a tutorial to a video analysis platform software, i.e., VideoFlow, which represents a video analysis task as a computation graph, provides common functions like video decoding and database storage, integrates deep learning frameworks, e.g. Caffe/Pytorch/MXNet as built-in inference engines, and supports heterogeneous hardware such as CPU/GPU/FPGA. VideoFlow also allows the customers to develop operator, decoder, and model inference extensions. The paper presents an example application of person ReID using the VideoFlow platform. The paper claims this VideoFlow software could be used in both academic and industrial scenarios. \n\nThe VideoFlow platform software is certainly a great development tool for video analysis tasks. The major concern is that if it is appropriate for ICLR to publish this tutorial which may be regarded as an endorsement to this software. Perhaps, VideoFlow could demonstrate in the exhibition of academic conferences to promote the usage in academia. \n\nSome detailed comments:\n\n1)\tIs VideoFlow a commercial software or a free open-source software? What kind of license does VideoFlow follow? As it integrates some functions of other software like Caffe/Pytorch/MxNet. \n\n2)\tThe major issue of adopting VideoFlow could be the learning curve how to use and debug applications on VideoFlow. VideoFlow appears a powerful and comprehensive video analysis platform, which abstracts and defines the interfaces among the components in a computation graph. For most of academia users, PyTorch/TensorFlow may be good to develop and verify some new deep model or algorithms. For industrial users, debugging any problem on VideoFlow requires a profound understanding of the platform. Deployment of an application could encounter issues on run-time library dependency, cross-compilation, resource limitation, e.g, on an embedded system. It would be a critical decision to adopt an open-sourced development platform for commercial applications for industrial users.\n\n3)\tSome minor questions:  As there is already a software named VideoFlow (de Armas, 2019), why not choose a different name for this software?  Vector<void*> appears a C++ implementation of a C-style concept, which implicitly requires the caller to agree with a protocol specified in other documents. The users may make mistakes that are hard to debug.  \n\nOverall, the proposed VideoFlow software is a great development tool for video analysis tasks using deep learning models. I encourage the authors to promote this software in the open-source community and demonstrate it in conference exhibition and trade shows.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9f_CIX4_5l", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\n\nThis paper introduce a framework for visual analysis (so-called VideoFlow) that will be open-sourced. VideoFlow is introduced with providing a flexible, efficient, extensible, and secure visual analysis framework for both academia and industry. It has been adopted in tens of intelligent visual analysis systems.\n\nHowever, through the whole paper, the key contributions of the VideoFlow should be only counted as engineering efforts rather than any novelty in the scientific or research perspective.\n\nTherefore, ICLR would be not an appropriate venue for the submitted paper to be published. It is more valuable for authors to submit to other venues such as OSDI (USENIX Symposium on Operating Systems Design and Implementation).\n\nStrengths\n---------\n\n- The paper introduces the architecture of the proposed framework (VideoFlow) in detail, including operator, data flow, resource, graph construction, execution scheduling, etc. These details should be helpful for users to understand the logic of its working flow and how the framework should be used.\n\n- The paper also presents the evaluation of person re-identification (ReID) task on VideoFlow to show its efficiency. \n\n\nWeaknesses\n----------\n\n- The motivation of presenting such a practical framework is clear and understandable, however, as a research paper, scientific contributions are required to prove its novelty rather than the only engineering efforts. For such a purpose, the following points could be considered to improve the current paper towards a more scientific-like research paper.\n\n\t- Address some scientific issues that were made in the architecture of VideoFlow: original methods solving the computation/IO intensive problem, theoretical efforts for graph construction and the design of execution scheduling.\n\n\t- Provide some theoretical or mathematical guarantees of the efficiency in the working flow of execution scheduling (Fig. 2).\n\t\n- The evaluation is not sufficient to show the advantages (flexible, efficient, extensible, and secure) that claimed by authors. In this paper, only the evaluation of ReID task is presented, but how about its efficiency and other advantages for other computer vision tasks such as object detection, segmentation, face recognition, and so on.\n\n\n\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2IabjocZRYY", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out the fact that just developing models using deep learning frameworks is not the end of the story when it comes to building end-to-end visual pipelines. The paper introduces VideoFlow which is a framework that aims to improve the development process of streaming pipelines.\n\nCompared to the widely used TensorFlow, PyTorch, and MXNet, VisualFlow focuses on more coarse-grained blocks like a \"whole network\" itself instead of \"layers.\" As such, the work is developed around a rather different units of data compared to Tensors in DNNs. This work also incorporates a GUI that lets the user edit the computation graphs. I believe this work in full fledged form may help the productivity while building visual analysis applications.\n\nSadly, there are many shortcomings of the paper. First, the paper literally spends over 60% of the paper to describe the implementation details which does not lead to much intellectual insights. This looks more like a technical report than a research paper.\n\nFurthermore, the paper lacks on evaluation in many aspects. For example, there is no evaluation about the potential overhead of the framework. I have listed a number of questions below with my comments.\n\nThe paper began with a luring abstract; however, after reading through the paper, the reader is left with only minor insights. Up to this point, the paper seems to be an amalgamation of various libraries and frameworks with a GUI wrapper. While I do believe the paper has some prospect as it does touch upon a real problem that does indeed take up a lot of resources in industry, I believe the paper is yet in a premature state to merit a publication at ICLR.\n\nAlso, I believe the paper would receive a more pertinent evaluation from a systems community like OSDI, SOSP, ATC. This is because these frameworks should not be just about putting things together to make something working, but should also entail a through experimentation of the performance and overhead.\n\nQuestions:\n- Could you please provide running examples of applications that have graphs with a complex topology?\n- How does the resource management behave for different scenarios such as underutilization? Could you show some visualization of that? For example, it would be very interesting to see how Dynamic Batching dynamically improves the utilization and the performance at runtime.\n- How can this be used for on-device scenarios, or cases where there are numerous devices? For example how does this compare to NNStreamer [1]?\n- If this framework were to be used in cases like Inference-as-a-Service scenario, how would this perform in terms of various QoS metrics?\n\n[1] NNStreamer, https://nnstreamer.ai", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciate the framework described in the paper and say it is a "useful tool", a "flexible, efficient, extensible, and secure visual analysis framework"  and "in full fledged form may help the productivity while building visual analysis applications."

However, the reviewers also point to significant shortcomings in terms of fit to ICLR, e.g. "looks more like a technical report than a research paper", "key contributions of the VideoFlow should be only counted as engineering efforts rather than any novelty in the scientific or research perspective", or "major concern is that if it is appropriate for ICLR to publish this tutorial which may be regarded as an endorsement to this software".

The authors did not reply to the reviewers comments.

Overall the paper does not seem to contain sufficient scientific contributions for being accepted.
Decision: Reject