Title: Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception
Venue: NeurIPS 2023
Accepted: yes
Keywords: approximate execution framework; real time perception; latency-accuracy tradeoffs
Abstract: Real-time perception requires planned resource utilization. Computational planning in real-time perception is governed by two considerations -- accuracy and latency. There exist run-time decisions (e.g. choice of input resolution) that induce tradeoffs affecting performance on a given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics. 

Earlier runtime execution frameworks employed rule-based decision algorithms and operated with a fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate execution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoffs instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-art static and dynamic execution policies on public datasets on both server GPUs and edge devices.

Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception

Abstract

Real-time perception requires planned resource utilization. Computational planning in real-time perception is governed by two considerations - accuracy and latency. There exist run-time decisions (e.g. choice of input resolution) that induce tradeoffs affecting performance on a given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics. Earlier runtime execution frameworks employed rule-based decision algorithms and operated with a fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate execution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoffs instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-art static and dynamic execution policies on public datasets on both server GPUs and edge devices. Code can be viewed at https://github.com/microsoft/chanakya .

1 Introduction

Real-time perception is an important precursor for developing intelligent embodied systems. These systems operate on sensory data on top of a hardware substrate, viz , a mix of resource-constrained, embedded and networked computers. Resource planning is a critical consideration in such constrained scenarios. Further, these systems need to be carefully designed to be latency sensitive and ensure safe behaviour (See Figure 2).

Many runtime execution decisions exist, e.g. the spatial resolution (scale) to operate at, temporal stride of model execution, choice of model architecture etc. The runtime decisions are influenced by (a) intrinsic context derived from sensor data (e.g. image content) (b) extrinsic context observed from system characteristics, e.g. contention from external processes (other applications). A execution framework is needed to use available context, jointly optimize accuracy and latency while taking these decisions.

Traditional execution frameworks [1, 2, 3, 4] operate in the following paradigm - optimize model to operate within a latency budget. Largely, they do not jointly optimize accuracy and latency for real-time tasks. Instead, they optimize proxies like latency budget [1], input resolution [2], latency from resource contention [4] or energy [5]. Their approach requires rule-based decision making heuristic design for every characteristic, instead of learning the decision function. These frameworks explicitly handcraft rules to accommodate hardware capabilities. Moreover, traditional strategies (Figure 2) budgeting latency are suboptimal when operating on streaming data [6].

(a)

(b)

(c)

(d)

(a)

(c)

(a)

(b)

(c)

(d)

(a)

(b)

Misaligned Truck

Missed Pedestrian

Detected Pedestrian

(c)

(d)

Operating in the new streaming perception paradigm, we propose Chanakya 2 - a novel learning-based approximate execution framework to learn runtime decisions for real-time perception. Chanakya operates in the following manner - (a) Content and system characteristics are captured by intrinsic and extrinsic context which guide runtime decisions. (b) Decision classes are defined, such as the input resolution (scale) choices, the model choices etc. Chanakya handles these interacting decisions and their tradeoffs which combinatorially increase. (c) Accuracy and latency are jointly optimized using a novel reward function without approximating either objectives. (d) Execution policy is learnt from streaming data, and is dynamic, i.e. configuration changes during real-time execution of the perception system.

Chanakya learns performant execution policies and can incorporate new decision dimensions, stochastic system context and be ported to different hardware. Importantly, our improvements are in addition to, and complementary to models and the decision dimensions themselves - faster & more accurate models [9, 10], vision-guided runtime decisions [11, 12, 13] and system-focused decisions [14, 15] can be easily incorporated.

2 Related Work

Flexible Real-Time Decision Making. Intelligent systems that are reactive [16] and flexible [17] are well studied. Our work is related to classical resource utilization and scheduling literature [18] and theoretical frameworks for real-time reinforcement learning in asynchronous environments [19, 20]. Contrasting with prior RL based execution frameworks [3, 21] which are operating in the suboptimal traditional latency budget paradigm [6], Chanakya is situated in the streaming perception paradigm. The key challenge for all these works has been the difficulty in designing rewards to jointly optimize accuracy and latency, and previous frameworks have learnability issues (See Section 3.4). We propose Chanakya with a novel reward and asynchronous training procedure for training approximate execution frameworks on large public datasets.

Real-time Perception and Video Analytics. In the vision community, accuracy-latency pareto decisions or 'sweet spots' for vision models have been studied [22, 6]. Drawing from these inferences, proposed design decisions exploit spatial [11, 13] and temporal [10] redundancies and improve real-

(b)

(d)

Backbone

Forecaster

Configuration

Controller

Intrinsic Context

Perception System

Detector

Real Time Input Stream

Real Time Output Stream

V100

....

Jetson

Perception Hardware

time perception. These works are complementary to our work, and their meta-parameters can be learned via Chanakya, and changed at runtime (static in their work). Improved models [23, 24, 9], say via found by architecture search [25, 26] during training, compression techniques like quantization [27] and pruning [28] are also complementary.

In the systems community, Chanakya is related to video analytics efforts for the edge and cloud, wherein works aim to optimize design decisions [15, 29, 14, 30, 31] during training and inference. Chanakya can incorporate their decision designs during inference (like AdaScale [2]). For example, LegoDNN [15] decides which NN blocks to use via a novel optimization formulation, which instead can be learned via Chanakya. Mistify and Ekya [14, 31] tailor models considering offline accuracy, latency and concept drift and these model choices can be selected by Chanakya at runtime.

Approximate Execution Frameworks. Many approximate execution frameworks have been designed for real-time perception. However, instead of maximizing real-time performance, they optimize proxies such as energy [5], latency from contention [4], or latency budget [1] and employ handcrafted rule-based heuristics. Previous works [4] have also highlighted the need for reinforcement learning (RL) based methods to learn runtime decisions. Chanakya instead learns runtime decisions to maximize streaming performance conditioned on intrinsic and extrinsic context, operating on a variety of decision dimensions. Many systems frameworks describe runtime decisions such as changing input resolution [2, 32], switching between models [1, 32], changing model configurations with latency [4], selecting frames to execute models vs trackers [33, 5], deciding between remote execution on the cloud and local execution on the edge [33, 34, 35]. These decisions can be learnt by Chanakya.

3 Chanakya: Learned Execution Framework for Real-Time Perception

3.1 Background on Streaming Perception

Traditionally, for real-time perception, model is executed at input video's frequency, say, 30 FPS. Thus the model is constrained to process a frame within a latency budget, in this case, 33 ms per frame. Streaming perception [6] instead places a constraint on the algorithm's output, no constraints are placed on the underlying algorithm. The constraint is anytime the algorithm is queried, it must report the current world state estimate at that time instant.

Formal Definition: Input stream comprises of sensor observation x , ground truth world state y (if available) and timestamp t , denoted by { O i : ( x i , y i , t i ) } T i =1 . Till time t , algorithm f is provided { ( x i , t i ) | t i < t } . Algorithm f generates predictions { ( f ( x j ) , s j ) } N j =1 , where s j is the timestamp when a particular prediction f ( x j ) is produced. This output stream is not synchronised with input stream and is not necessarily operating on every input. Real-time constraint is enforced by comparing the most recent output f ( x φ ( t ) ) to ground truth y i , where φ ( t ) = argmax j s j ≤ t .

Throughput vs Latency. Assuming two hardware setups, one with a single GPU available and another with infinite GPU's available (simulated), we employ a Faster R-CNN detector (no forecaster)

22

Tracker scale

20

360.0

480.0

mAP

18

720.0

600.0

Detector scale

480.0

16

720.0

600.0

Stride

14

5.0

10.0

15.0

30

40

50

60

Time (avg)

21

20

480

560

640

Detector Scale

720

SAP

19

Detector Props

100

300

■

500

18

+

Chanakya

1000

17

22

24

26

28

mAP

0.20

Resolution

Latency

S

0.15

420

240

300

360

0.10

480

600

0.05

0

20

40

60

80

Contention Level(%)

on a P40 GPU. The sAP scores obtained are 13.9 sAP and 15.7 sAP respectively. Additional compute (GPU's) improves system throughput, but does not improve latency of real-time execution. Thus it is not a solution for improving real-time perception.

3.2 High Level Overview

We design a learned execution framework for real-time perception that jointly optimizes accuracy and latency. We consider object detection to be the instantiation of our perception problem, however, the described principles apply without loss of generality.

Consider real-time object detection. The real-timeness requirement of streaming perception is fulfilled by pairing our detector (say Faster R-CNN) with object tracker (Tracktor [36]) and state estimator (Kalman Filter). We can change meta-parameters like detector's and tracker's scale, temporal stride of tracker along with internal parameters (for instance, number of proposals in the detector) and generalize this to include the choice of components themselves (the detector/tracker etc). Consider these decisions to be configuration of the algorithm f , and a policy π that emits this configuration. We wish to learn a controller that governs the execution policy π ⋆ , which is triggered periodically to decide the runtime configuration. In this work, we assume that all the other components (detector, tracker etc) have been trained already and we solely learn the controller.

We formulate learning the runtime decision as a deep contextual bandit problem, where a sequence of independent trials are conducted and in every trial the controller chooses a configuration (i.e. action) a from the set of possible decisions D along with the provided context z . After a trial the controller receives a reward r as feedback for the chosen configuration and updates the execution policy. As we will see in Section 3.3, intrinsic and extrinsic factors affect policy π . Thus we define context z encapsulating these factors which is provided to the controller. We learn a Q -function Q θ ( z, a ) , for every action a given context z , where Q -values define the policy π to configure f .

3.3 Intrinsic and Extrinsic Context Guide Runtime Decisions

Numerous factors inform runtime decisions in real-time perception. Figure 3 depicts configurations for a predefined perception system. Randomly picking a 'good' configuration (marked by red ellipse) can result in a sub-optimal decision, causing a drop of at least 3 mAP in comparison to the 'optimal' configuration. Optimal decision shifts with hardware change, on Argoverse-HD using a P40 GPU, the best scale turns out to be 600 instead of 900 on a 1080Ti GPU [6]. Selection of algorithm components (say, the detector) is itself a decision with trade-offs [22]. It should be noted that while recent models [10] satisfy the streaming perception constraint, they are optimized for specific hardware [13]. Detection models exhibit different kinds of errors [37, 38], and changing the scale affects their performance. Lastly, offline and real-time perception performance are not correlated (Figure 4).

(a) R for sequences A and B .

(b) R fixed _ adv for sequences A and B .

50

SequenceA

SequenceB

50

SequenceB

SequenceA

40

40

Count

30

Count

30

20

20

10

10

0

0

0.0

0.2

0.4

0.6

0.8

0.25

0.50

Rewards

Rewards

Intrinsic factors like image content influence decisions - static configurations are sub-optimal. E.g., We can optimize scale at runtime [2] based on video content to improve throughput while maintaining accuracy. Extrinsic factors introduce additional challenges. Optimal decisions change with the stochastic nature of the external process, such as resource contention induced by other running processes. Figure 5 shows change in latency with contention levels and this relationship is non-linear. Other such extrinsic factors include network conditions and power considerations. Extrinsic context are non-differentiable, thus we do not use an end-to-end network to learn the overall context z .

We concatenate outputs of the following functions to form the overall context to the controller,

Switchability Classifier: Models exhibit different error characteristics, motivating the need to switch models at runtime [1]. Switchability classifier guides model switching at runtime.

Adaptive Scale Regressor: AdaScale [2] predicts the optimum scale an image should be inferred at, such that single frame latency is minimized. We use their regressor as an intrinsic context.

Contention Sensor: Process Contention affects model latency. We employ the GPU contention generator and sensor from [4], simulating discrete contention levels.

Scene aggregates: We also compute frame-level aggregates: (i) Confidence Aggregates : We compute the mean and the standard deviation of the confidence value associated with the predictions. The intuition is if the average confidence is low, which might indicate execution of an 'expensive' configuration. (ii) Category Aggregates : We count of instances of each category in the predictions. This might capture rare objects, which may require additional resources to be localized. (iii) Object Size Aggregates : We categorize and aggregate predictions as small, medium and large detections (like COCO thresholds [39]). This is useful as small objects are considered difficult to detect.

3.4 Reward Jointly Optimizes Latency And Accuracy

Traditionally, output ˆ y i is emitted for observation O i at time ˆ t i ( > t i ), and the reward [3, 21] is defined as

considering accuracy via ϕ and latency via ξ separately. However, hyperparameter λ for this additive reward is globally inconsistent (i.e. changes depending on intrinsic/extrinsic context) which doesn't work for large datasets. In our experiments we observed that controller collapsed to predict a single configuration when rewards of this form were provided (depending on λ ), and did not learn.

Now, consider the streaming perception setup described in Section 3.1. Deriving from this paradigm, our rewards optimize accuracy and latency simultaneously. Intuitively, reward R compares the model output with a ground truth stream of the actual world state. Assume algorithm f is governed by policy π receives observation O x , controller performs action a x at time t x to configure f and another

Algorithm 2: Training the Controller

Update Policy

observation O y with action a y is taken at t y ( t y > t x ). We propose a reward,

where L is an arbitrary single frame loss. Intuitively, the reward for action a x is streaming accuracy obtained for a stream sequence segment until a new action a y is taken in the future and φ ensures only real-time results are accounted for. Reward R ( t x , t y ) ensures learned controller implicitly considers real-time constraints and accuracy of the decision a x , not assuming any properties of the algorithm f or loss L . Thus, it can applied to other perception tasks.

While [6] defined L streaming = L ( { y t k , ˆ y φ ( t k ) } N k =0 ) and corresponds to R (0 , T ) (Loss over N observations from a video stream of length T), L streaming was employed to characterize performance of f on the sensor stream. Directly employing it as reward made training the controller sampleinefficient, and did not converge before we exhausted our training budget (of 10 epochs).

While reward proposed in Equation 1 is learnable, it is biased towards the inherent characteristics of the video sequence. Consider two video sequences A and B , Sequence A with one large object that can be easily detected would have much higher average rewards than Sequence B with many occluding and small objects. To normalize for this factor, we consider a fixed policy π fixed , i.e., a fixed configuration of algorithm f and propose,

where R π fixed is the reward obtained by executing algorithm f following a fixed policy. To implement R fixed _ adv , f with fixed policy π fixed are executed and prefetched the predictions and the timestamps are stored. During training, we compute R π and load π fixed outputs to simulate R π fixed .

This eliminates the bias in the inital proposed reward, and provides the advantage of learned policy over fixed policy (See Figure 6). In Section 4.1, we show any fixed policy configuration ( π fixed ) suffices and the policy learned is resilient to variations of this selection. We refer to the reward in Equation 1 as R 1 and Equation 2 as R 2 in our experiments for simplicity.

3.5 Controller Design Decisions

Our controller design decisions (such as employing deep contextual bandits) are motivated by severe real-time constraints. For e.g., we considered individual frames as independent trials while learning our controller, i.e. we compute context from a single frame to incur minimal overhead (amortized overhead of around 1%). Considering sequences as controller's input incurred significant overheads and impeded performance.

Combinatorially Increasing Configurations. Consider decision space D = { D 1 , D 2 ..D m } , with dimensionality M = | D | , where each decision dimension D i corresponds to a configuration parameter of the algorithm f discretized as d k . Thus, an action a has M decision dimensions and each discrete sub-action has a fixed number of choices. For example, if we consider 2 decision dimensions, D = { D scale , D model } , the potential configurations would be D scale = { 720 , 600 , 480 , 360 , 240 } , D model = { yolo, fcos, frcnn } . Using conventional discrete-action algorithms, ∏ d ∈ D | d | possible actions need to be considered. Efficiently exploring such large action spaces is difficult, rendering naive discrete-action algorithms like [3, 21] intractable.

Chanakya uses a multi-layer perceptron for predicting Q θ ( z, a i ) for each sub-action a i and given context z , and employs action branching architecture [40] to separately predict each sub-action, while operating on a shared intermediate representation. This significantly reduces the number of action-values to be predicted from ∏ d ∈ D | d | to ∑ d ∈ D | d | .

Real-time Scheduler. Choosing which frames to process is critical to achieve good streaming performance. Temporal aliasing, i.e., the mismatch between the output and the input streams, reduces performance. [6] prove optimality of shrinking tail scheduler. However, this scheduler relies on the assumption that the runtime ρ of the algorithm f is constant. This is reasonable as runtime distributions are unimodal, without considering resource contention.

Chanakya changes configurations at runtime, and corresponding ρ changes. Fortunately, our space of configurations is discrete , and ρ 's (for every configuration) can be assumed to be constant. Thus, a fragment of the sequence where a particular configuration is selected is a scenario where shrinking tail scheduler using that runtime will hold. We cache configuration runtimes of algorithm f and modify the shrinking tail scheduler to incorporate this assumption.

Asynchronous Training. Chanakya is trained in such a way that real-time stream processing is simulated, akin to challenges presented in [19, 20]. Thus, a key practical challenge is to factor in average latency for generating the context from the observations, computing the controller's action during training. We also need to ignore the time taken to train the controller as these would disrupt and change the system performance. Training the controller concurrently on a different GPU incurred high communication latency overhead.

Algorithms 1 and 2 describe the training strategy. We train the controller intermittently - after a sequence has finished processing, the controller's parameters are fixed while the sequence is streamed. While the sequence is processing, we store the ( z, a, t ) tuples in a buffer S along with all the algorithm's predictions (to construct the reward r ). During training, first we add these tuples to controller's buffer B and, sample from B to update Q θ ( z, a ) .

Training for edge devices. We emulate training the controller for edge devices on a server by using the measured runtimes for each configuration on the edge device, since Chanakya only requires the runtimes of the algorithm f π on the target device to train the controller.

4 Experiments and Results

Weanalyse and evaluate Chanakya through the following experiments. We employ sAP metric [6] that coherently evaluates real-time perception, combining latency and accuracy into a single metric. It is non-trivial to compare traditional systems that use mAP and latency for evaluation as their algorithms would require major modifications. However, we do augment one such method, AdaScale [2] by incorporating Kalman Filter and our scheduler. Apart from this, we compare our approach with static policies obtained from benchmarking. Please note Chanakya's improvements are complementary and in addition to the system components themselves, newer models would lead to further improvements. All the results are reported on Argoverse-HD, unless stated otherwise.

Improvements on a predefined system. For a real-time perception system on a given hardware (P40 GPU), let's assume system components - detector (Faster R-CNN), scheduler (Modified Shrinking Tail) and forecaster (a Kalman Filter) are known. We optimize configurations across two decision dimensions D = { D s : { 360 , 480 , 540 , 640 , 720 } , D np : { 100 , 300 , 500 , 1000 }} , i.e., detector scale and number of proposals. We evaluate Chanakya using both variants of rewards described in Section 3.4, and only intrinsics are provided as context. We compare with best performing configurations reported by [6] ( static policy ). Further, we asked an expert who employed domain knowledge [22] to optimize the proposal decision dimension via benchmarking ( static expert policy ). We also compare with a dynamic traditional policy , AdaScale [2] which predicts input scale and is trained to minimize single frame latency as it is situated in traditional latency budget paradigm. Upper bound takes the best configuration of the static policy from [6] and simulate a detector with 0ms latency for every frame. Table 1 shows that we outperform all of these methods, as Chanakya can adapt dynamically (See Figure 7) via context which explains improvement over static policies and considers the real-time constraint holistically. Chanakya does not solely minimize single frame latency which explains improvements over the traditional policy. Similar trends hold for ImageNet-VID, please see Appendix A.

Upper bound takes the best configuration of the static policy from [6] and simulate a detector with 0ms latency for every frame. Learning decisions from scratch. Consider when optimal components (say, the detection model or tracker) are unknown for a given hardware. We will now introduce the third dimension, which is the choice of the model (FCOS or YOLOv3 or Faster R-CNN). Chanakya has to pick configurations across model choices, scale choices and proposal choices (if applicable, decision dimension is ignored otherwise). Using an alternative detector, i.e, FCOS and with the best scale obtained via benchmarking (paired with Streamer [6]), the static policy has an sAP of 16.7, while Chanakya has sAP of 20.7 (23% improvement). Similar performance improvement (+5.5 sAP) was also seen on ImageNet-VID.

Handling large decision spaces. We extend our pre-defined system (Faster R-CNN + Scheduler + Kalman Filter) with a tracker (Tracktor [36]). Tracker supports configuration of tracker scale and temporal stride, i.e. number of times it's executed versus the detector, apart from detector scale, number of proposals, increases the decision space to 500 configurations. It is not feasible to explore the entire decision space using brute-force if we consider deployment to many hardware platforms. In contrast, our approach learned the execution policy in 5 training epochs. As shown in Table 2, Chanakya learns a competitive policy that outperforms the best static policy from [6] by +1.6 sAP.

Scales

Proposals

720

0.00

0.00

0.02

0.00

640

0.09

0.14

0.41

0.00

560

0

0.01

0.17

0.16

0.01

0

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

100

300

500

1000

Scales

Proposals

720

0.00

0.00

0.06

0.00

640

0.00

0.00

0.00

0.00

0.00

0.00

0.04

0.00

8

0.00

0.03

0.55

0.06

360

0.03

0.01

0.18

0.03

100

300

500

1000

(a) Xavier NX

18

16

SAP

Resolution

14

Static(360)

Static(600)

12

Static(720)

Random

10

Ours

0

20

40

60

80

ContentionLevel

Scales

Proposals

720

0.00

0.00

0.02

0.01

640

0.00

0.01

0.05

0.00

0.00

0.01

0.00

0.00

8

0.01

0.03

0.78

0.05

360

0.00

0.00

0.03

0.00

100

300

500

1000

Performance and configuration shift on edge devices. Weconsider deploying the predefined system discussed earlier on two edge devices - Jetson Xavier NX and Xavier AGX. We employ emulated training procedure discussed in Section 3.5. From Table 4 comparing performance on Xavier NX, Chanakya improves sAP by 12% and 28% Argoverse-HD and ImageNet-VID respectively over static policies found by benchmarking. In Figure 9, we observe shifts in configurations selected for the two devices (compared to Figure 7 evaluated on a server GPU P40). Xavier AGX is more powerful, and we observe more frequent selection of higher detection scale (s = 480 ) compared to Xavier NX, in line with our expectations.

Adapting to Extrinsic Factors. In our predefined system, we consider contention as our extrinsic factor and study Chanakya's behaviour, assuming black-box resource contention levels available at run-time. This scenario is common on smartphones/edge devices [4] where multiple processes employ the same resource. Static policies, wherein decisions are not made considering contention would degrade quickly (See Section 3.3).

Chanakya adapts to sensed contention level (provided as context) by choosing decisions which correspond to the sweet-spot for that contention level, with no additional modifications. Critically, Chanakya is flexible and adapts decisions to the contention level unlike [4] who explicitly model the problem as a rule-based system (modeling latency and accuracy with quadratic regression models) and handcraft a contention-aware scheduler. Performance degrades across all approaches with increasing contention and Figure 8 Chanakya is able to adapt better than static policies.

New hardware and action spaces. Consider the task of migrating a streaming perception system from a device employing P40 GPU, to newer V100 GPU. New hardware has additional features, such as faster inference at lower precision levels (P40 in comparison does offer limited support; few FP16 operations are faster). As the tradeoffs have shifted, we do not assume that components and hardware optimizations are known. We consider additional decisions - choice between inference at FP32 and FP16 and choice between models. We employed Chanakya with the following decision space:

D = { D m : { resnet _ frcnn, swin _ frcnn, resnet _ cascade _ rcnn, swin _ cascade _ rcnn } , D p : { FP 32 , F P 16 } , D s : { 800 , 880 , 960 , 1040 , 1120 , 1200 } , D np : { 100 , 300 , 500 , 1000 }} . Without benchmarking and domain expertise, it's non-trivial to decide these meta-parameters. We consider the best known static policy (Faster R-CNN with Resnet-50 at s = 900 ) proposed by [6] for V100 GPU as our baseline which has 18.3 sAP . Chanakya obtained 27.0 sAP, i.e., +8.7 sAP (47%) improvement, the highest improvement on the Streaming Perception Challenge without additional data.

4.1 Ablation Studies

We discuss some of the considerations of our approach using Chanakya as our perception system.

Importance of Context. Table 5 shows that when a context feature is dropped performance reduces by ≈ 1 sAP. Context overheads can be viewed in Appendix C.

Resilience. Table 6 shows that Chanakya is resilient to the choice of the initial fixed policy π fixed used in our reward function R 2 . Chanakya arrives at a competitive policy even if a bad fixed policy is chosen, (360 , 1000) , which employs very small scale and the maximum number of proposals. Chanakya is resilient to the choice of action space as changing scale space from s 1 to s 2 yielded competitive performance.

5 Conclusion

Limitations and Broader Impact. While our method does not directly enable unethical applications, it might be employed to deploy and improve such applications. Further study of regulatory mechanisms for safe ML deployments and the impact of methods like Chanakya is required.

We presented Chanakya, a learned runtime execution framework to optimize runtime decisions for real-time perception systems. Our learned controller is efficient, flexible, performant and resilient in nature. Chanakya appears useful for many real-time perception applications in the cloud and edge. We plan to extend this work to scenarios wherein applications require hard performance guarantees.

References

Seungyeop Han, Haichen Shen, Matthai Philipose, Sharad Agarwal, Alec Wolman, and Arvind Krishnamurthy. Mcdnn: An approximation-based execution framework for deep stream processing under resource constraints. In MobiSys , 2016.

Ting-Wu Chin, Ruizhou Ding, and Diana Marculescu. Adascale: Towards real-time video object detection using adaptive scaling. In MLSys , 2019.

Ning Chen, Siyi Quan, Sheng Zhang, Zhuzhong Qian, Yibo Jin, Jie Wu, Wenzhong Li, and Sanglu Lu. Cuttlefish: Neural configuration adaptation for video analysis in live augmented reality. Transactions on Parallel and Distributed Systems , 2020.

Ran Xu, Chen-lin Zhang, Pengcheng Wang, Jayoung Lee, Subrata Mitra, Somali Chaterji, Yin Li, and Saurabh Bagchi. ApproxDet: content and contention-aware approximate object detection for mobiles. In SenSys , 2020.

Kittipat Apicharttrisorn, Xukan Ran, Jiasi Chen, Srikanth V Krishnamurthy, and Amit K RoyChowdhury. Frugal following: Power thrifty object detection and tracking for mobile augmented reality. In SenSys , 2019.

Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. In ECCV , 2020.

Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR , 2019.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, and Jian Sun. Real-time object detection for streaming perception. In CVPR , 2022.

Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. Fovea: Foveated image magnification for autonomous navigation. In ICCV , 2021.

Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Porikli, and Amir Ghodrati. Salisa: Saliency-based input sampling for efficient video object detection. In ECCV , 2022.

Anurag Ghosh, N Dinesh Reddy, Christoph Mertz, and Srinivasa G Narasimhan. Learned two-plane perspective prior based image resampling for efficient object detection. In CVPR , 2023.

Peizhen Guo, Bo Hu, and Wenjun Hu. Mistify: Automating DNN model porting for On-Device inference at the edge. In NSDI , 2021.

Rui Han, Qinglong Zhang, Chi Harold Liu, Guoren Wang, Jian Tang, and Lydia Y Chen. Legodnn: block-grained scaling of deep neural networks for mobile vision. In MobiCom , 2021.

Michael P Georgeff and Amy L Lansky. Reactive reasoning and planning. In AAAI , 1987.

Eric J Horvitz. Computation and action under bounded resources . PhD thesis, 1990.

Mark Boddy and Thomas L Dean. Deliberation scheduling for problem solving in timeconstrained environments. Artificial Intelligence , 1994.

Simon Ramstedt and Chris Pal. Real-time reinforcement learning. NeurIPS , 2019.

Jaden B Travnik, Kory W Mathewson, Richard S Sutton, and Patrick M Pilarski. Reactive reinforcement learning in asynchronous environments. Frontiers in Robotics and AI , 2018.

Young Geun Kim and Carole-Jean Wu. Autoscale: Energy efficiency optimization for stochastic edge inference using reinforcement learning. In MICRO , 2020.

Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In CVPR , 2017.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV , 2019.

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR , 2018.

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. In CVPR , 2021.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML , 2019.

Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 , 2015.

Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS , 2017.

Shiqi Jiang, Zhiqi Lin, Yuanchun Li, Yuanchao Shu, and Yunxin Liu. Flexible high-resolution object detection on edge devices with tunable latency. In MobiCom , 2021.

Xu Zhang, Yiyang Ou, Siddhartha Sen, and Junchen Jiang. SENSEI: Aligning video streaming quality with dynamic user sensitivity. In NSDI , 2021.

Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, and Ion Stoica. Ekya: Continuous learning of video analytics models on edge compute servers. In NSDI , 2022.

Ran Xu, Jinkyu Koo, Rakesh Kumar, Peter Bai, Subrata Mitra, Ganga Meghanath, and Saurabh Bagchi. Approxnet: Content and contention aware video analytics system for the edge. arXiv preprint arXiv:1909.02068 , 2019.

Tiffany Yu-Han Chen, Lenin Ravindranath, Shuo Deng, Paramvir Bahl, and Hari Balakrishnan. Glimpse: Continuous, real-time object recognition on mobile devices. In SenSys , 2015.

Xukan Ran, Haolianz Chen, Xiaodan Zhu, Zhenming Liu, and Jiasi Chen. Deepdecision: A mobile deep learning framework for edge video analytics. In INFOCOM , 2018.

Anurag Ghosh, Srinivasan Iyengar, Stephen Lee, Anuj Rathore, and Venkata N Padmanabhan. React: Streaming video analytics on the edge with asynchronous cloud support. In IoTDI , 2023.

Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In ICCV , 2019.

Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. Tide: A general toolbox for identifying object detection errors. In ECCV , 2020.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR , 2016.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. In AAAI , 2018.

Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In CVPR , 2019.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015.

A visual walkthrough of streaming perception solutions. https://mtli.github.io/ streaming/streaming-visuals.html , 2021.

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 , 2019.

Kai Kang, Hongsheng Li, Junjie Yan, Xingyu Zeng, Bin Yang, Tong Xiao, Cong Zhang, Zhe Wang, Ruohui Wang, Xiaogang Wang, et al. T-cnn: Tubelets with convolutional neural networks for object detection from videos. Transactions on Circuits and Systems for Video Technology , 2017.

Akshay Uttama Nambi, Adtiya Virmani, and Venkata N Padmanabhan. Farsight: A smartphonebased vehicle ranging system. IMWUT , 2018.

Akshay Uttama Nambi, Ishit Mehta, Anurag Ghosh, Vijay Lingam, and Venkata N Padmanabhan. Alt: towards automating driver license testing using smartphones. In Proc. SenSys , 2019.

Songtao He, Favyen Bastani, Arjun Balasingam, Karthik Gopalakrishna, Ziwen Jiang, Mohammad Alizadeh, Hari Balakrishnan, Michael Cafarella, Tim Kraska, and Sam Madden. Beecluster: drone orchestration via predictive optimization. In MobiSys , 2020.

Arjun Balasingam, Karthik Gopalakrishnan, Radhika Mittal, Mohammad Alizadeh, Hamsa Balakrishnan, and Hari Balakrishnan. Toward a marketplace for aerial computing. In Proceedings of the 7th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications , 2021.

Srinivasan Iyengar, Ravi Raj Saxena, Joydeep Pal, Bhawana Chhaglani, Anurag Ghosh, Venkata N Padmanabhan, and Prabhakar T Venkata. Holistic energy awareness for intelligent drones. In BuildSys , 2021.

Peer reviews:
comments: SUMMARY:
This paper proposes a learned approximate execution framework named Chanakya. Chanakya considers intrinsic context like images and extrinsic context like latency and predicts runtime decisions. The reward function helps to learn a better trade-off online. Extensive experiments on the Argoverse-HD dataset show that Chanakya outperforms static policy and can be applied to new hardware and action spaces.

STRENGTHS:
1. This paper jointly optimizes accuracy and latency for real-time tasks. The reward function proposed considers both the simultaneous optimization between accuracy and latency, and different characteristics of the video sequence.
2. The detailed analysis of experiment results demonstrates the performance improvements of Chanakya.
3. This paper is well-organized and well-written to read.

WEAKNESSES:
1. Some settings of experiments are not clarified, like offline upper bound and RL training method.
2. Some details of learning the controller should be introduced in the main part of the paper.

QUESTIONS:
1. Why did employing L_{streaming} not converge which is mentioned in Line 165?
2. In table 1, how do you get sAP for offline upper bound?

LIMITATIONS:
The authors just mention that their method might be employed to improve unethical applications.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Chanakya, a framework for computation planning in real-time (streaming) perception. Chanakya uses a novel reward to make run-time decisions based on content- and system-based characteristics, simultaneously optimizing for both accuracy and latency. The proposed controller is learned on individual frames and uses a normalized reward to reduce bias. Chanakya’s performance is evaluated on a real-time perception stack consisting of a detector, scheduler, forecaster and tracker; these individual components are made part of the search space (eg: choice of model), along with their associated configurations (scale, #proposals, etc.). Results are reported across edge devices, runtime contention levels, and for cases where the perception stack has been migrated from one GPU to another.\n\nSTRENGTHS:\n* This is a well-written paper, and the core ideas have been explained clearly.\n* Impact & relevance: the paper targets real-time perception, which is an important component of modern AV and embodied systems.\n* Strong results: Chanakya appears to outperform SOTA static and dynamic approaches, and ones designed by domain experts.\n\nWEAKNESSES:\n* It is not clear how well Chanakya scales with increasing search space size. While the paper includes an evaluation of a reasonable number of decision dimensions, it would be interesting to get an understanding of Chanakya's scalability limits, especially since the authors claim in Section 2 that the search space could potentially be expanded by trying to account for varying input resolutions or via techniques such as pruning and/or quantization.\n* No comparisons to non-RL-based approaches, including rule-based and purely latency-focused systems.\n\nQUESTIONS:\nThe comparison in Figure 1(c) is to RetinaNet, which is a network from 2017. Do more modern networks (eg: ConvNeXt/Swin, CoAt-Net, MaxViT, etc.) fulfill the real-time/streaming constraint by being able to handle higher resolutions?\n\nLIMITATIONS:\nLimitations have been adequately addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper looks at the problem of unpredictable compute requirements for real-time perception. It addresses this in term of a multi-objective optimisation problem (quality of results and latency). The authors propose using RL in order to optimise the selection of various characteristics in order to achieve an optimal result.

STRENGTHS:
The paper is well written and provides a good introduction for the reader and guides them through the work.

WEAKNESSES:
The proposed approach does not come across as that novel. There is a lot of prior use of RL techniques for multi-objective optimisation which the authors don't appear to have looked at.

QUESTIONS:
- The authors spend a lot of the introduction naming and saying how good their approach is. This should be more analytical than what is presented.

- Lines 89-94 give a formal definition of the perception problem. However, most of the introduced terminology does not get used further in the paper. This space could be used for other material.

- Figures 2,3&4 present results, but it is not clear where these results came from and how they were produced.

- "In our experiments we observed that controller collapsed to predict a single configuration when rewards of this form were provided (depending on λ), and did not learn" - which experiments?

-

LIMITATIONS:
Small amount of discussion of limitations in conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

comments: SUMMARY:
This work provides a novel learning-based approximate execution framework to learn runtime decisions for real-time perception. The learned controller proves to be efficient and performant, which appears to be useful for many real-time perception applications in the cloud and edge.

STRENGTHS:
1. The focus of the work on real-time perception system's runtime execution decisions is important and timely.
2. The proposed Chanakya proves to learn performant execution policies and can incorporate new decision dimensions and be ported to different hardwares.

WEAKNESSES:
The experimental results are only reported on detection tasks, and other perception task like segmentation might be reported.
The decision dimention seems to be discrete and limited.

QUESTIONS:
How to evaluate and compare between the controllers on different hardware, which can have different features?

LIMITATIONS:
As listed in weakness part, more experimental results on other perception task can be reported, and not sure whether the decision space is valid.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author designs Chanakya, a learned execution framework for streaming perception that jointly optimizes accuracy and latency. To achieve so, the framework captures both intrinsic and extrinsic contexts and utilizes a novel reward function to train a learned controller.\n\nSTRENGTHS:\n1. The motivation is clear and the discussion of related work is comprehensive.\n2. The description of experimental setting is detailed and results of ablation studies are provided.\n\nWEAKNESSES:\n1. Novelty is somewhat limited. The methodological innovation in this paper is limited to use both intrinsic and extrinsic contexts to better learn runtime decisions with reinforcement learning based methods. The fundamental methodology is mostly shared with prior work.\n2. Presentation can be improved. Table 1 is at the top of page 6, but no context related to it appears until page 7. Asynchronous Training and Training for edge devices in Section 3.5 seems unnecessary to discuss.\n3. Baselines are incomplete. I think the experiment should at least have vanilla RL-based baselines and then show the effectiveness of the proposed framework step by step.\n4. I'm not sure it's a good idea to incorporate all factors into a learning controller. For single GPU program, I believe rule-based decision algorithms can adapt extrinsic factors (e.g. software and hardware status) very well.\n\nQUESTIONS:\nHow does your approach compare to hybrid solutions that employ handcrafted rule-based heuristics for extrinsic factors and learning-based decision algorithms for intrinsic factors?\n\nLIMITATIONS:\nIn the subsection \"Limitations and Broader Impact\", the authors point out that their framework can be used to deploy unethical applications, and I think this is just a potential negative societal impact, not a limitation. Therefore, I encourage authors to talk about the limitations of your work and potential solutions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

comments: SUMMARY:
The paper presents Chanakya, a learned framework for real-time perception that automatically balances accuracy and latency. Unlike previous fixed rule-based methods, Chanakya considers both intrinsic and extrinsic factors and is trained to make flexible decisions. evaluations show that it outperforms existing execution policies on both server GPUs and edge devices with low overhead.

STRENGTHS:
1. It proposes a learning based approximate execution framework to learn runtime decisions such as the resolutions of input, which model to use, etc.
2. The proposed framework considers multiple context both intrinsic and extrinsic to improve the reliability of the runtime decisions.

WEAKNESSES:
1. It is unclear how generalizable is the trained policy. More discussions on when train from scratch and when transfer learning and the corresponding cost when the system is in a different environment is needed.

QUESTIONS:
1. It would be good to have a diagram figure to show the high level overview of the whole framework.
2. There exists some work that use elastic model setting for different scenarios such as [1]. More discussions and comparisons are needed.

[1] Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "Scaled-yolov4: Scaling cross stage partial network." Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. 2021.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-30