VideoFlow: A Framework for Building Visual Analysis Pipelines

1 INTRODUCTION . The success of computer vision techniques is spawning intelligent visual analysis systems in real applications . Rather than serving individual models , these systems are often powered by a workflow of image/video decoding , several serial or parallel algorithm processing stages , as well as sinking analysis results . The varied visual analysis requirements in different real scenarios put forward a high demand on a framework for fast algorithm development , flexible pipeline construction , efficient workflow execution , as well as secure model protection . There exist some frameworks approaching some of the above mentioned targets , like DeepStream ( Purandare , 2018 ) and MediaPipe ( Lugaresi et al. , 2019 ) . DeepStream is on top of GStreamer ( GSTREAMER , 1999 ) , which primarily targets audio/video media editing rather than analysis . MediaPipe can be used to build prototypes to polished cross-platform applications and measure performance . Though it is flexible and extensible on calculators , efficiency , model security , and extension on more aspects are expected by real online services in industry . In this paper , we present VideoFlow , to meet the visual analysis requirements for both algorithm development and deployment in real systems with the following highlights . Flexibility . VideoFlow is designed around stateful Computation Graph and stateless Resource . Computation graph abstracts the visual processing workflow into a stateful directed acyclic graph . Developers can focus on the implementation of processing units ( graph nodes ) and the construction of the whole workflow . Resource is a stateless shared computation module of computation graphs . The most typical resource is deep learning model inference . Resource decouples the stateless visual processing components from the whole complicated visual analysis pipeline , helping developers focus on the optimization of these computation or Input/Output ( IO ) intensive implementation . Efficiency . VideoFlow is designed for better efficiency from four levels . ( 1 ) Resource-level : resources can aggregate the scattered computation requests from computation graph instances into intensive processing for better efficiency . ( 2 ) Video-level : all videos are analyzed in parallel in a shared execution engine . ( 3 ) Frame-level : video frames are parallelized on operations which are irrelevant to frame orders . ( 4 ) Operator-level : visual analysis is a multi-branch pipeline in most cases . The different branches and different operators of each branch without sequential dependency are analyzed in parallel . Extensibility . VideoFlow is designed from the beginning to be as modular as possible to allow easy extension to almost all its components . It can be extended to different hardware devices like Graphic Processing Units ( GPU ) , Neural Processing Unit ( NPU ) , etc . It can be hosted on either x86 or ARM platforms . Developers can customize their own implementations with VideoFlow as a dependent library . The extended implementations can be registered back to VideoFlow as plugins at runtime . Security . Model protection is an important problem in industry . VideoFlow encodes model files into encrypted binary codes as part of the compiled library . The secret key can be obscured into the same library , or exported to a separate key management service ( KMS ) . At runtime , VideoFlow decrypts the models and verifies authorization from a remote service periodically . VideoFlow has been incubated in the practices of the smart city innovation for more than three years . It is designed for computer vision practitioners , including engineers , researchers , students , and software developers . The targets of VideoFlow include : 1 ) free developers from the exhausting data loading/sinking , parallel programming and debugging to the optimization of algorithms ; 2 ) enable easy extension of video decoding , deep model inference and algorithm implementation ; 3 ) provide highly efficient framework for large scale visual processing in industry rather than just experimental prototypes . 4 ) protect the intellectual property of models and algorithms to make sure that they can only work with authorization . 2 RELATED WORK . 2.1 DEEP LEARNING FRAMEWORKS . Almost all existing deep learning frameworks like Caffe ( Jia et al. , 2014 ) , TensorFlow ( Abadi et al. , 2016 ) , PyTorch ( Paszke et al. , 2017 ) , MXNet ( Chen et al. , 2015 ) describe networks in directed graphs or even dynamic graphs . VideoFlow draws lessons from this successful design for visual analysis . The difference is that the basic units in deep networks are low level operations like convolutions , compared to higher level processing like object tracking in VideoFlow . The data transferred between operators in VideoFlow is also much more complex than the Tensor in deep learning . As to model inference , there are some specially optimized engines , like TensorRT ( Vanholder , 2016 ) and MKL-DNN/oneAPI ( Intel ) by hardware manufactures . In the open source community , developers put forward TVM for easy extension to different hardware for more effective inference ( Chen et al. , 2017 ) . On top of these engines , there are some serving platforms for individual models rather workflow construction , like tensorflow serving ( Google , 2016 ) , NVIDIA Triton Inference Server ( Goodwin & Jeong , 2019 ) . VideoFlow integrates these inference engines as Resources with their C++ interfaces . 2.2 VISUAL ANALYSIS FRAMEWORKS . The recent has witnessed some visual analysis frameworks . Nvidia launches the DeepStream project in the early days for video analysis on GPU ( Purandare , 2018 ) . It is oriented as well as optimized on GPU and TensorRT , regardless of the bustling heterogeneous hardware devices . Besides , it is built on top of GStreamer ( GSTREAMER , 1999 ) , which primarily targets audio/video media editing rather than analysis , limiting its flexibility and extensibility . The gst-video-analytics project ( Intel , 2019 ) is also built on top of GStreamer ( Deuermeyer & Andrey ) . Google proposed MediaPipe by building graphs for arbitrary streaming data processing with a computation graph as well ( Lugaresi et al. , 2019 ) . MediaPipe can be used to build prototypes to polished cross-platform applications and measure performance . Though it is flexible and extensible on calculators , real online visual analysis expects extension on more aspects , more efficiency optimization , and model security protection . Compared to MediaPipe , VideoFlow features these advantages for better application in both academia and industry . Another framework also named Videoflow ( de Armas , 2019 ) is designed to facilitate easy and quick definition of computer vision stream processing pipelines . However , it is just a prototype experimental platform , with limitations on extensibility , efficiency , and security . 3 ARCHITECTURE . VideoFlow is oriented around stateful Computation Graph and stateless Resource with a welloptimized execution engine . Computation graph is a directed acyclic graph describing the whole workflow of video or image analysis . As the two main components of a graph , Node and Edge denote visual processing operators and data flow between operators , respectively . Resource is shared for graph irrelevant computation . The architecture is shown in Figure 1 . 3.1 OPERATOR . Operator is the basic unit of visual analysis workflow . An operator depends on the outputs of its parent operators . Its own outputs can be consumed by arbitrary number of child operators . According to the number of inputs an outputs , operators are categorized as follows : • Entrypoint : operators that have zero inputs . This is the start of a computation graph . Each graph can have only one entrypoint . • Processor : operators that have at least one input and at least one output . Processors occupy most of the workflow of visual analysis . It ’ s also the main kind of operator with the highest demand on easy extension . • Sinker : operators that have zero outputs . This is the end of a computation graph . A graph can have multiple sinkers . 3.2 DATA FLOW . Data flow is the edge connection between two operators ( nodes ) . An operator may generate several number of data with different types for its child nodes . Data flow is a collection of arbitrary number of data pointers of arbitrary type ( vector < void * > in our C++ implementation ) in VideoFlow . VideoFlow guarantees that the incoming data pointers are always safe to be read . Developers do not need to care how many other operators are also consuming the data , or whether the data should be released during the workflow . 3.3 RESOURCE . Resource is the stateless computation unit shared by graphs . The most representative resource is deep model inference . Resource is abstracted due to three main reasons . Firstly , many operations like deep model inference and data sinking to databases have their own independent semantics . They are irrelevant to whether it is used for video or image processing , which step of the whole pipeline invokes the operation , or how the outputs will be post-processed . Secondly , these operations are often computation or IO intensive . Leaving them in the operators will incur bottlenecks on CPU , memory or network bandwidth due to large amount of resource competition . Gathering the scattered but shared requests from different graphs for uniform processing proves to be a good practice to improve efficiency . Thirdly , resource can be improved without affecting the visual analysis logic . For example , we can accelerate the inference speed of a PyTorch model by switching to a TensorRT worker . We can change to a more efficient database connector for more real-time data sinking . Without the abstraction of these resources , all affected operators have to be re-implemented to earn the benefits . 3.4 GRAPH CONSTRUCTION . Computation graph is described in json format with the following fields . “ resource ” describes the resources that will be used by operators . Each resource should have two fields : “ type ” to create the correct class instance and “ params ” to specify the resource configurations . “ ops ” describes the operators that will be used to construct computation graphs . Operators can be used multiple times by different graphs . As the same to resource , each operator should have two fields : “ type ” and “ params ” . “ graph ” is the place to define computation graphs . Each graph definition is a json dictionary of key-value pairs . Key is the operator name . Its value is a list of operator names as its child nodes . “ subgraphs ” [ optional ] is used to re-use resources , operators and graphs from other graph configuration files . “ libs ” [ optional ] specifies external dynamic libraries that should be loaded by VideoFlow , especially the extended libraries in Figure 3 . “ config ” [ optional ] is for global settings , currently including number of parallel image processing threads and number of frames for parallel video processing . An example file is provided in the supplementary material to show the person reidentification workflow ( Section 5 ) . 3.5 EXECUTION SCHEDULING . With the graph defined and constructed , execution scheduling determines which operator should be calculated . In real cases , there can be multiple computation graph instances running in parallel , each with either shared or different structures . Figure 2 shows the execution scheduling of these graphs . Each graph has several replicas , with each replica called as an order . Video frames are actually processed in these graph replicas/orders . The replicas are processed in parallel for framelevel parallelism . Each order starts from the Forward function of the entrypoint node . Forward . As Figure 2 shows , the forward function first checks if the current operator is ready to be executed . The readiness checking includes : 1 ) all parents of the current node have finished on this order . 2 ) the previous order of the current node has been executed if the current node is an ordered operator . If ready , the forward function puts its own processing function into the task queue of the execution engine , waiting to be executed . The processing function first finishes the internal processing logic of the operator . After that , it calls the Forward function of its following operators . If it is the leaf operator , it calls the Backward function of its own . Forward of entrypoints is specially implemented with a separate thread retrieving and dispatching data to idle orders . Backward is the process to reset the node to be ready to process latter frames . The backward function first checks if all its children have been reset . If so , it resets its forward status . Then it continues to call the Backward of all its parents . Execution Engine . The processing functions of operators are put into a task queue of the execution engine . All processing units share the same interface . The execution engine does not know which order of which graph a processing function comes from . All orders and all graphs are executed concurrently once they are put into the queue . Inside the engine there is a thread pool , with all threads fetching and executing tasks from the queue .

Summary: The paper presents a tutorial to a video analysis platform software, i.e., VideoFlow, which represents a video analysis task as a computation graph, provides common functions like video decoding and database storage, integrates deep learning frameworks, e.g. Caffe/Pytorch/MXNet as built-in inference engines, and supports heterogeneous hardware such as CPU/GPU/FPGA. VideoFlow also allows the customers to develop operator, decoder, and model inference extensions. The paper presents an example application of person ReID using the VideoFlow platform. The paper claims this VideoFlow software could be used in both academic and industrial scenarios.