SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chanakya uses a multi-layer perceptron for predicting  $Q_{\theta}(z, a_i)$  for each sub-action  $a_i$  and given context  $z$ , and employs action branching architecture [40] to separately predict each sub-action, while operating on a shared intermediate representation. This significantly reduces the number of action-values to be predicted from  $\prod_{d \in \mathbb{D}} |d|$  to  $\sum_{d \in \mathbb{D}} |d|$ .
Real-time Scheduler. Choosing which frames to process is critical to achieve good streaming performance. Temporal aliasing, i.e., the mismatch between the output and the input streams, reduces performance. [6] prove optimality of shrinking tail scheduler. However, this scheduler relies on the assumption that the runtime  $\rho$  of the algorithm  $f$  is constant. This is reasonable as runtime distributions are unimodal, without considering resource contention.
Chanakya changes configurations at runtime, and corresponding  $\rho$  changes. Fortunately, our space of configurations is discrete, and  $\rho$ 's (for every configuration) can be assumed to be constant. Thus, a fragment of the sequence where a particular configuration is selected is a scenario where shrinking tail scheduler using that runtime will hold. We cache configuration runtimes of algorithm  $f$  and modify the shrinking tail scheduler to incorporate this assumption.
Asynchronous Training. Chanakya is trained in such a way that real-time stream processing is simulated, akin to challenges presented in [19, 20]. Thus, a key practical challenge is to factor in average latency for generating the context from the observations, computing the controller's action during training. We also need to ignore the time taken to train the controller as these would disrupt and change the system performance. Training the controller concurrently on a different GPU incurred high communication latency overhead.
Algorithms 1 and 2 describe the training strategy. We train the controller intermittently - after a sequence has finished processing, the controller's parameters are fixed while the sequence is streamed. While the sequence is processing, we store the  $(z,a,t)$  tuples in a buffer  $S$  along with all the algorithm's predictions (to construct the reward  $r$ ). During training, first we add these tuples to controller's buffer  $B$  and, sample from  $B$  to update  $Q_{\theta}(z,a)$ .
Training for edge devices. We emulate training the controller for edge devices on a server by using the measured runtimes for each configuration on the edge device, since Chanakya only requires the runtimes of the algorithm  $f_{\pi}$  on the target device to train the controller.
Table 2: Handling large decision spaces. We incorporate a tracker in our predefined perception system. Choice of detector scale and proposals, tracker scale and stride, combinatorially expands our decision space. Chanakya is able to learn a competitive policy compared to best static policies [6].
Approach	sAP
Static Policy (s=900, ts=600, k=5)	17.8
Static Policy (s=600, ts=600, k=5)	19.0
Chanakya (s1, np1, ts, k, R2)	19.4
Table 3: Learning decisions from scratch. Chanakya is able to decide model and other metaparameter choices even when optimal algorithmic components are unknown.
Approach	sAP
Static Policy (m=fcos, s=600)	16.7
Static Policy (m=yolov3, s=600)	20.2
Static Policy (m=frcnn, s=600)	20.4
Chanakya (m=m, s1, np1, R=R2)	20.7
We analyse and evaluate Chanakya through the following experiments. We employ sAP metric [6] that coherently evaluates real-time perception, combining latency and accuracy into a single metric. It is non-trivial to compare traditional systems that use mAP and latency for evaluation as their algorithms would require major modifications. However, we do augment one such method, AdaScale [2] by incorporating Kalman Filter and our scheduler. Apart from this, we compare our approach with static policies obtained from benchmarking. Please note Chanakya's improvements are complementary and in addition to the system components themselves, newer models would lead to further improvements. All the results are reported on Argoverse-HD, unless stated otherwise.
Improvements on a predefined system. For a real-time perception system on a given hardware (P40 GPU), let's assume system components - detector (Faster R-CNN), scheduler (Modified Shrinking Tail) and forecaster (a Kalman Filter) are known. We optimize configurations across two decision dimensions  $\mathbb{D} = \{D_s:\{360,480,540,640,720\},D_{np}:\{100,300,500,1000\}\}$ , i.e., detector scale and number of proposals. We evaluate Chanakya using both variants of rewards described in Section 3.4, and only intrinsics are provided as context. We compare with best performing configurations reported by [6] (static policy). Further, we asked an expert who employed domain knowledge [22] to optimize the proposal decision dimension via benchmarking (static expert policy). We also compare with a dynamic traditional policy, AdaScale [2] which predicts input scale and is trained to minimize single frame latency as it is situated in traditional latency budget paradigm. Upper bound takes the best configuration of the static policy from [6] and simulate a detector with 0ms latency for every frame. Table 1 shows that we outperform all of these methods, as Chanakya can adapt dynamically (See Figure 7) via context which explains improvement over static policies and considers the real-time constraint holistically. Chanakya does not solely minimize single frame latency which explains improvements over the traditional policy. Similar trends hold for ImageNet-VID, please see Appendix A.
Upper bound takes the best configuration of the static policy from [6] and simulate a detector with 0ms latency for every frame. Learning decisions from scratch. Consider when optimal components (say, the detection model or tracker) are unknown for a given hardware. We will now introduce the third dimension, which is the choice of the model (FCOS or YOLOv3 or Faster R-CNN). Chanakya has to pick configurations across model choices, scale choices and proposal choices (if applicable, decision dimension is ignored otherwise). Using an alternative detector, i.e., FCOS and with the best scale obtained via benchmarking (paired with Streamer [6]), the static policy has an sAP of 16.7, while Chanakya has sAP of 20.7 (23% improvement). Similar performance improvement (+5.5 sAP) was also seen on ImageNet-VID.
Handling large decision spaces. We extend our pre-defined system (Faster R-CNN + Scheduler + Kalman Filter) with a tracker (Tracktor [36]). Tracker supports configuration of tracker scale and temporal stride, i.e. number of times it's executed versus the detector, apart from detector scale, number of proposals, increases the decision space to 500 configurations. It is not feasible to explore the entire decision space using brute-force if we consider deployment to many hardware platforms. In contrast, our approach learned the execution policy in 5 training epochs. As shown in Table 2, Chanakya learns a competitive policy that outperforms the best static policy from [6] by  $+1.6$  sAP.
Figure 7: Dynamic Configurations executed on P40 GPU. For the predefined system, Chanakya changes the configuration depending on the intrinsics and achieves better performance.
Figure 8: Chanakya adapts to extrinsic factors. Compared to static execution policies, Chanakya has better performance at different levels of GPU contention and degrades less at higher contention.
(a) Xavier NX
(b) Xavier AGX
Figure 9: Configuration shift on edge devices. The predefined system is deployed on the edge devices. Chanakya is trained via emulation, and we observe a shift in chosen configurations that matches our expectations for these hardware platforms.
Performance and configuration shift on edge devices. We consider deploying the predefined system discussed earlier on two edge devices - Jetson Xavier NX and Xavier AGX. We employ emulated training procedure discussed in Section 3.5. From Table 4 comparing performance on Xavier NX, Chanakya improves sAP by  $12\%$  and  $28\%$  Argoverse-HD and ImageNet-VID respectively over static policies found by benchmarking. In Figure 9, we observe shifts in configurations selected for the two devices (compared to Figure 7 evaluated on a server GPU P40). Xavier AGX is more powerful, and we observe more frequent selection of higher detection scale ( $s = 480$ ) compared to Xavier NX, in line with our expectations.
Adapting to Extrinsic Factors. In our predefined system, we consider contention as our extrinsic factor and study Chanakya's behaviour, assuming black-box resource contention levels available at run-time. This scenario is common on smartphones/edge devices [4] where multiple processes employ the same resource. Static policies, wherein decisions are not made considering contention would degrade quickly (See Section 3.3).
Chanakya adapts to sensed contention level (provided as context) by choosing decisions which correspond to the sweet-spot for that contention level, with no additional modifications. Critically, Chanakya is flexible and adapts decisions to the contention level unlike [4] who explicitly model the problem as a rule-based system (modeling latency and accuracy with quadratic regression models) and handcraft a contention-aware scheduler. Performance degrades across all approaches with increasing contention and Figure 8 Chanakya is able to adapt better than static policies.
New hardware and action spaces. Consider the task of migrating a streaming perception system from a device employing P40 GPU, to newer V100 GPU. New hardware has additional features, such as faster inference at lower precision levels (P40 in comparison does offer limited support; few FP16 operations are faster). As the tradeoffs have shifted, we do not assume that components and hardware optimizations are known. We consider additional decisions – choice between inference at FP32 and FP16 and choice between models. We employed Chanakya with the following decision space:
Table 4: Performance on edge devices. Comparisons on Xavier-NX edge device. We can observe that Chanakya improves performance over static policies on both the datasets.
Approach	Argoverse-HD (sAP)	ImageNet-VID (sAP)
Streamer (s=560, np=500) (Static Policy) [6]	7.8	18.8
Streamer (s=640, np=300) (Static Policy) [6]	7.4	20.0
Streamer with Random Selection	7.6	17.5
Chanakya (R = R1)	8.1	19.4
Chanakya (R = R2)	8.3	22.4
Table 5: Importance of Context. All components of the considered context contribute to improved real-time performance.
Dropped feature	sAP
None	21.3
Adaptive Scale	19.2
Category aggregates	19.4
Confidence aggregates	20.3
Object size aggregates	20.4
Table 6: Resilience. Chanakya learns perform-. mant decisions despite using a bad fixed policy. Further, minimal performance degradation is observed between  $s_1$  and  $s_2$
Approach	sAP
Ours (s1, np1, R2, πfix = (640, 300))	21.3
Ours (s1, np1, R2, πfix = (480, 300))	21.3
Ours (s1, np1, R2, πfix = (360, 1000))	21.1
Ours (s2, np1, R1)	18.2
Ours (s2, np1, R2)	20.4
$\mathbb{D} = \{D_m:\{\text{resnet\_frcnn}, \text{swin\_frcnn}, \text{resnet\_cascade\_rcnn}, \text{swin\_cascade\_rcnn}\}, D_p:\{FP32, FP16\}, D_s:\{800, 880, 960, 1040, 1120, 1200\}, D_{np}:\{100, 300, 500, 1000\}\}$ . Without benchmarking and domain expertise, it's non-trivial to decide these meta-parameters. We consider the best known static policy (Faster R-CNN with Resnet-50 at  $s = 900$ ) proposed by [6] for V100 GPU as our baseline which has 18.3 sAP. Chanakya obtained 27.0 sAP, i.e., +8.7 sAP (47%) improvement, the highest improvement on the Streaming Perception Challenge without additional data.
4.1 Ablation Studies
We discuss some of the considerations of our approach using Chanakya as our perception system.
Importance of Context. Table 5 shows that when a context feature is dropped performance reduces by  $\approx 1$  sAP. Context overheads can be viewed in Appendix C.
Resilience. Table 6 shows that Chanakya is resilient to the choice of the initial fixed policy  $\pi_{fixed}$  used in our reward function  $R_{2}$ . Chanakya arrives at a competitive policy even if a bad fixed policy is chosen, (360, 1000), which employs very small scale and the maximum number of proposals. Chanakya is resilient to the choice of action space as changing scale space from  $s_{1}$  to  $s_{2}$  yielded competitive performance.
Limitations and Broader Impact. While our method does not directly enable unethical applications, it might be employed to deploy and improve such applications. Further study of regulatory mechanisms for safe ML deployments and the impact of methods like Chanakya is required.
We presented Chanakya, a learned runtime execution framework to optimize runtime decisions for real-time perception systems. Our learned controller is efficient, flexible, performant and resilient in nature. Chanakya appears useful for many real-time perception applications in the cloud and edge. We plan to extend this work to scenarios wherein applications require hard performance guarantees.
We thank Harish YVS for his insightful discussions and the reviewers for their detailed comments.
[41] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[43] A visual walkthrough of streaming perception solutions. https://mtli.github.io/ streaming/streaming-visuals.html, 2021.
[44] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
[45] Kai Kang, Hongsheng Li, Junjie Yan, Xingyu Zeng, Bin Yang, Tong Xiao, Cong Zhang, Zhe Wang, Ruohui Wang, Xiaogang Wang, et al. T-cnn: Tubelets with convolutional neural networks for object detection from videos. Transactions on Circuits and Systems for Video Technology, 2017.
[46] Akshay Uttama Nambi, Adtiya Virmani, and Venkata N Padmanabhan. Farsight: A smartphone-based vehicle ranging system. IMWUT, 2018.
[47] Akshay Uttama Nambi, Ishit Mehta, Anurag Ghosh, Vijay Lingam, and Venkata N Padmanabhan. Alt: towards automating driver license testing using smartphones. In Proc. SenSys, 2019.
[48] Songtao He, Favyen Bastani, Arjun Balasingam, Karthik Gopalakrishna, Ziwen Jiang, Mohammad Alizadeh, Hari Balakrishnan, Michael Cafarella, Tim Kraska, and Sam Madden. Beecluster: drone orchestration via predictive optimization. In MobiSys, 2020.
[49] Arjun Balasingam, Karthik Gopalakrishnan, Radhika Mittal, Mohammad Alizadeh, Hamsa Balakrishnan, and Hari Balakrishnan. Toward a marketplace for aerial computing. In Proceedings of the 7th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications, 2021.
[50] Srinivasan Iyengar, Ravi Raj Saxena, Joydeep Pal, Bhawana Chhaglani, Anurag Ghosh, Venkata N Padmanabhan, and Prabhakar T Venkata. Holistic energy awareness for intelligent drones. In BuildSys, 2021.