# Inference Acceleration with Adaptive Distributed DNN Partition over Dynamic Video Stream
## 4 The Proposed ADDP Framework  
### Framework Overview  
Figure 5 depicts the DNN collaborative inference acceleration using ADDP. Currently, ADDP supports two major existing feature extraction networks, including the VGG and ResNet families. In this section, the task and device situation in a smart city scenario is described with the various modules of the framework and the data flow between them. The specific model cutting algorithm is introduced in the next part of this section. Figure 4: Cross -device DNN computation graph. **Device topology and tasks**. There are many devices and a high task complexity in a smart city, which cannot be done in real-time by end inference. Although the computational resources at the edge are richer than those at the end devices, completing the entire inference process at the edge cannot satisfy the real-time response constraint of each task. Edge-end collaboration, as a method that can effectively utilize the computing resources of devices and expand the total computing capacity while guaranteeing the inference delay, is the best technical means to process data in the smart city in real-time. We analyzed the best way to provide intelligent services for smart city stations based on the high number of cameras and uneven distribution of pedestrian density and devices. The distribution characteristics of smart city devices and pedestrian flow cause differences in the quantity of data collected by the devices. The quantity of data collected by devices distributed near dense crowds is greater than that collected by devices in other locations, but the computing power of the devices is fixed. Therefore, the best technical means to process data in a smart city in real-time is to reason in parallel between devices and then collaborate with the edge. For a smart city, 1-to-N technology can realize the carrier technology of this model's reasoning method. The devices are able to communicate with each other, the data volume can be balanced, and the devices can be aggregated into one computing node from the physical connection, and then the edge-end collaboration is carried out. **Resource config**. We abstracted the resource information of the device and the network condition into static offline and dynamic online configurations. Offline resource information includes the CPU information of all end devices and measures the computational power in the subsequent process without further change. Online updated resource information includes the edge information, such as GPU arithmetic power, disk capacity, memory size, the quantity of data collected on each device, and the bandwidth. Offline and online resource information are both used as configuration information and input to the regression model to predict the inference time of each layer of the DNN model. **Regression model**. Based on the resource information of the device, we applied the method of Section 3 to predict the computation time of each layer of the DNN model. We also dynamically adjusted the prediction results according to the changes of online resource information in resource config to ensure the optimal prediction of the decision module's results to the maximum extent. The regression model sends the generated infer table to the master node and edges involved in the resource config. **DNN model**. This part includes parameters, structure, and weights. Parameters describe the type of operators used in the DNN model, the size of the operators, and the  
Figure 5: ADDP framework. data dimension. Structure stores the DNN model computational graph and data flow. Weights hold the parameters necessary for the model to be computed during the inference process and are only loaded during the model inference process. **Decision module**. The decision module determines the selection of the model split points, which is the core part of the ADDP framework, including two main modules, feature-map layer partition (FLP) and feature-map size partition (FSP).