Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception
Authors: Anurag Ghosh
Vaibhav Balloli
Akshay Nambi
Aditya Singh
Tanuja Ganu
Venue: NeurIPS 2023 poster (2023)
Keywords: approximate execution framework; real time perception; latency-accuracy tradeoffs
TL;DR: A learning-based approximate execution framework to learn runtime decisions for real-time perception.

Abstract: Real-time perception requires planned resource utilization. Computational planning in real-time perception is governed by two considerations -- accuracy and latency. There exist run-time decisions (e.g. choice of input resolution) that induce tradeoffs affecting performance on a given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics. 

Earlier runtime execution frameworks employed rule-based decision algorithms and operated with a fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate execution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoffs instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-art static and dynamic execution policies on public datasets on both server GPUs and edge devices.

Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception
Anurag Ghosh Vaibhav Balloli Akshay Nambi Aditya Singh 3+ Tanuja Ganu
*Carnegie Mellon University UUiversityy of Michigan *Microsoft Research India
amuraggh@cs. cmu, edu, vballoli@umich edu, aditya. aditya.singhh...... wingh.matt70iii.... mat.7@iittbbttu ac. in,
{akshayn, taganu)@miccrsoon crosoft.
Abstract
Real-time perception requires planned resource utilization. Computational planning in real-time perception governed by considerations accuracy and latency. There exist run-time decisions (e.g. choice input resolution) that induce tradeoffs affecting performance
given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics.
Earlier runtime execution frameworks employed rule-based decision algorithms and operated with fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate exe- cution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoff's instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-arr static and dynamic execu- tion policies on public datasets on both server GPUs and edge devices. Code can be viewed at https: //gi rithub..... cow htmm/microooof/.omommmmmmmo www.cooooooooooaaacaaaaaaaa
Introduction
Real-time perception is an important precursor for developing intelligent embodied systems. These systems operate on sensory data on top of hardware substrate, viz, a mix of essource-onnstrineee embedded and networked computers. Resource planning is critical consideration in such constrained scenarios. Further, these systems need to be carefully designed to be latency sensitive and ensure safe behaviour (See Figure 2).
Many runtime execution decisions exist, e.g. the spatial resolution (scale) to operate at, temporal stride of model execution, choice of model architecture etc. The runtime decisions are influenced by (a) intrinsic context derived from sensor data (e.g. image content) (b) extrinsic context observed from system characteristics, e.g. contention from external processes (other applications). A execution framework is needed to use available context, jointly optimize accuracy and latency while taking these decisions.
Traditional execution frameworks [1, 2, 3, 4] operate in the following paradigm optimize model to operate within a latency budget. Largely, they do not jointly optimize accuracy and latency for real-time tasks. Instead, they optimize proxies like latency budget [1], input resolution [2], latency from resource contention [4] or energy [5]. Their approach requires rule-based decision making heuristic design for every characteristic, instead of learning the decision function. These frameworks explicitly handcraft rules to accommodate hardware capabilities. Moreover, traditional strategies (Figure 2) budgeting latency are suboptimal when operating on streaming data [6].
"Wok done at Microsoft Research India
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Misaligned
Miesed Pedestrian
Detacted Pedestrian
Figure I: Optimizing real-time perception inference. (a) A powerful detector (HTC [7]) has very good offline (non real-time) (38.2 m.P) performance. (b) However, by the time it finishes execution (750ms per frame), inferences are not aligned with objects (6.2 sAP). (c) Solely optimizing for latency via a faster model (30ms, RetinaNet [8] at low resolution) leads to suboptimal and unsafe behavior (6.0 sAP),, like missing pedestrians. (d) Learning decisions with favourable inference tradeoffs with Chanakya (21.3 sAP) achieves better real-time performance.
Operating in the new streaming perception paradigm, we propose Chanakya a novel learning-base approximate execution framework to learn runtime decisions for real-time perception. Chanakya operates in the following manner (a) Content and system characteristies are captured by intrinsic and extrinsic context which guide runtime decisions. (b) Decision classes are defined, such as the input resolution (scale) choices, the model choices etc. Chanakya handles these interacting decisions and their tradeoffs which combinatorially increase. (c) Accuracy and latency are jointly optimized using novel reward function without approximating either objectives.
(d) Execution policy is learnt from streaming data, and dynamic, i.e. configuration changes during real-time execution of the perception system.
Chanakya learns performant execution policies and can incorporate new decision dimensions, stochas- tic system context and be ported to different hardware. Importantly, our improvements are in addition to, and complementary to models and the decision dimensions themselves faster & more accurate models [9, 10], rision-guides runtime decisions [11, 12, 13] and system- foccued decisions [14, 15] can be easily incorporated.
2 Related Work
Flexible Real-Time Decision Making, Intelligent systems that are reactive [16] and flexible [17] are well studied. Our work is related to classical resource utilization and scheduling literature [18] and theoretical frameworks for real-time reinforcement learning in asynchronous environments [19, 20]. Contrasting with prior RL based execution frameworks [3, 21] which operating in the suboptimal traditional latency budget paradigm [6], Chanakya is situated in the streaming perception paradigm. The key challenge for all these works has been the difficulty in designing rewards to jointly optimize accuracy and latency, and previous frameworks have learnability issues (See Section 3.4). We propose Chanakya with novel reward and asynchronous training procedure for training approximate execution frameworks on large public datasets.
Real-time Perception and Video Analytics. In the vision community, accuracy-latency pareto decisions or "sweet spots" for vision models have been studied [22, 6]. Drawing from these inferences, proposed design decisions exploit spatial [11, 13] and temporal [10] redundancies and improve real-
"Chanaya was an ancient Indian polymath, known for good governance strategies balancing economic development and social justice.

---

Real Time Input Stream
Conf ienffiuurtiin
Real
Time
Output
Stream
Detector
Perception Syaten
BLERR
V100
Controller
Jetson
Retrinate Conters
Perception Hardware
Figure 2: High level overview. Chanakya is learned execution framework for real-time time perception that jointly optimizes accuracy and latency. Assuming a perception system operating video stream, we describe meta- pettaeerreeters that are modified during execution they depend on the scene (intrinsic context) and hardware considerations (extrinsic context). A specific choice of meta-parameters corresponds to a runtime execution configuration decided by the learned controller. The controller emits a configuration given context so that real-time accuracy is maximized.
real-time
time perception. These works complementary to work, and their meta- vetta-arameters can be learned via Chanakya, and changed at runtime (static in their work). Improved models [23, 24, 9], say via found by architecture search [25, 26] during training, compression techniques like quantization [27] and pruning [28] are also complementary.
In the systems community, Chanakya is related to video analytics efforts for the edge and cloud, wherein works aim to optimize design decisions [15, 29, 14, 30, 31] during training and inference. Chanakya can incorporate their decision designs during inference (like AdaScale [2]). For example, LegoDNN [15] decides which NN blocks to use via a novel optimization formulation, which instead can be learned via Chanakya. Mistify and Ekya [14, 31] tailor models considering offline accuracy, latency and concept drift and these model choices can be selected by Chanakya at runtime.
Approximate Execution Frameworks. Many approximate execution frameworks have been de- signed for real-time perception. However, instead of maximizing real-time performance, they optimize proxies such as energy [5], latency from contention [4], or latency budget [1] and employ handcrafted rule-based heuristics. Previous works [4] have also highlighted the need for reinforcement learning (RL) based methods to learn runtime decisions. Chanakya instead learns runtime decisions to maxi- mize streaming performance conditioned on intrinsic and extrinsic context, operating on a variety of decision dimensions. Many systems frameworks describe runtime decisions such as changing input resolution [2, 32), switching between models [1, 32], changing model configurations with latency [4], selecting frames to execute models vs trackers [33, 5], deciding between remote execution on the cloud and local exccution on the edge [33, 34, 35]. These decisions can be learnt by Chanakya.
3 Chanakya: Learned Execution Framework for Real-Time Perception
3.1 Background on Streaming Perception
Traditionally, for real-time perception, model is executed at input video's frequency, say, 30 FPS. Thus the model is constrained to process a frame within a latency budget, in this case, 33 ms per frame. Streaming perception [6] instead places a constraint on the algorithm's 's output, no constraints are placed on the underlying algorithm. The constraint is anytime the algorithm is queried, it must report the current world state estimate at that time instant.
Input stream comprises of sensor observation x, ground truth world state (if available) and timestamp I, denoted by (O; (xi, Vi, 4)) Till time I, algorithm f is provided {(x,, 44 t). Algorithm generates predictions {(f(z)) s;), where 8; is the timestamp when a particular prediction f(z,) is produced. This output stream is not synchronised with input stream and is not necessarily operating on every input. Real-time constraint is enforced by comparing the most recent output (())(11) to ground truth yi, where (p(1) = argmax, 8j S I.
Formal Definition:
Throughput vs Latency. Assuming two hardware setups, one with a single GPU available and another with infinite GPU's available (simulated), employ a Faster R-CNN detector (no forecaster)
3

---

020
Delecde Gooe
Latenoy 12
Chi
18-
120.
100
450
1002
Gontention Lovel (75)
30
40
MAP
Time (avg)
Figure 5: Extrinsic Factors.
Figure Offline vs Real-time Perception. mAP vs sAP for different configurations of same system. An increase in offline performance (mAP) doesn't im- ply increase in real-time per- formance (sAP).
Figure 3: Configuration trade- offs of perception system. Many configurations (tracker scale, detector scale, temporal stride, number of proposals). Configs. in red ellipse satisfy real-time budget 33ms.
Process contention is dynamic, and induces latency differences for detector (Faster-RCNN) a given scale. Notice that this relationship is non-linear. Other components follow sim- ilar trends.
on a P40 GPU. The sAP scores obtained are 13.9 sAP and 15.7 sAP respectively. Additional compute (GPU's) improves system throughput, but does not improve latency of real-time execution. Thus is not a solution for improving real-time perception.
3.2 High Level Overview
We design learned execution framework for real-time perception that jointly optimizes accuracy and latency. We consider object detection to be the instantiation of our perception problem, however, the described principles apply without loss of generality.
Consider real-time object detection. The real-timeness requirement of streaming perception fulfilled by pairing our detector (say Faster R-C R-CNN) with object tracker (Tracktor [36]) and state estimator (Kalman Filter). We can change meta-paramettrs like detector's ss and tracker's 's scale, temporal stride of tracker along with internal parameters (for instance, number of proposals in the detector) and generalize this to include the choice of components themselves (the detector/tracker etc). Consider these decisions to be configuration of the algorithm f, and policy that emits this configuration. We wish to learn a controller that governs the execution policy π which is triggered periodically to decide the runtime configuration. In this work, assume that all the other components (detector, tracker etc) have been trained already and we solely learn the controller.
We formulate learning the runtime decision as deep contextual bandit problem, where a sequence of independent trials are conducted and in every trial the controller chooses a configuration (i.e. action) from the set of possible decisions D along with the provided context z. After trial the controller receives a reward r as feedback for the chosen configuration and updates the execution policy. As will see in Section 3.3, intrinsic and extrinsic factors affect policy я. Thus define context encapsulating these factors which is provided to the controller. We learn a Q-function Q0(2, a), for every action given context where Q-values define the policy to configure f.
3.3 Intrinsic and Extrinsic Context Guide Runtime Decisions
Numerous factors inform runtime decisions real-time perception. Figure depicts configurations for a predefined perception system. Randomly picking "good" configuration (marked by red ellipse) can result sub-optimal decision, causing a drop of at least mAP in comparison the "optimal" configuration. Optimal decision shifts with hardware change, on Argoverse-Hl HD using P40 GPU, the best scale turns out to be 600 instead of 900 on a 1080Ti GPU [6]. Selection of algorithm components (say, the detector) is itself a decision with trade-offs [22]. It should be noted that while recent models [10] satisfy the streaming perception constraint, they are optimized for
specific hardware [13]. Detection models exhibit different kinds of errors [37, 38), and changing
the scale affects their performance. Lastly, offline and real-time perception performance are not correlated (Figure 4).

---

Sequence A Sequence
50
Sequence
40
Count
Count
30
20-
10-
10-
0.0
0.4
-130 -1.25 -1.00
-0.75
-050
-9.25
0.00
0.5
0.0
Rewards
Rewards
R for sequenees and B.
(b) Rfixed adv for sequences A and B.
Figure 6: Understanding Rfized_dde adv- In many video sequences, performance measure (single frame loss L) high for all configurations (e-g. all objects are large, highly visible), such as Sequence Compared to sequences (like B) where performance is lower and more heavily dependent on detector type and scale (e.g. objects are small or occluded). Rifieed adv adv corrects this bias.
Intrinsic factors like image content influence decisions static configurations are sub-optimal. E.g.. We can optimize scale runtime [2] based on video content to improve throughput while maintaining accuracy. Extrinsic factors introduce additional challenges. Optimal decisions change with the stochastic nature of the external process, such as resource contention induced by other running processes. Figure 5 shows change in latency with contention levels and this relationship is non-linear. Other such extrinsic factors include network conditions and power considerations. Extrinsic context are non-differentiable, thus we do not use an end-to-end network to learn the overall context 2.
We concatenate outputs of the following functions to form the overall context to the controller,
Switchability Classifier: Models exhibit different error characteristics, motivating the need to switch models at runtime [1]. Switchability classifier guides model switching at runtime.
ddaptive Scale Regressor: AdaScale [2] predicts the optimum scale an image should be inferred at, such that single frame latency is minimized. We use their regressor as intrinsic context.
3. Contention Sensor: Process Contention affects model latency. We employ the GPU contention generator and sensor from [4], simulating discrete contention levels.
4. Scene aggregates: We also compute frame-level aggregates: (i) Confidence Aggregates: We compute the mean and the standard deviation of the confidence value associated with the predictions. The intuition is if the average confidence is low, which might indicate execution of an "expensive" configuration. (ii) Category Aggregates: We count of instances of each category in the predictions. This might capture rare objects, which may require additional resources be localized. (iii) Object Size Aggregates: We categorize and aggregate predictions as small, medium and large detections (like COCO thresholds [39]). This useful small objects considered difficult to detect.
3.4 Reward Jointly Optimizes Latency And Accuracy
Traditionally, output 9 is emitted for observation O; at time E (> t;), and the reward [3, 21] is defined as
Revad = plys, (t) + XE (4)
considering accuracy via o and latency via E separately. However, hyperparameter for this additive reward is globally inconsistent (i.e. changes depending intrinsic/extrinsic context) which doesn't 't work for large datasets. In our experiments we observed that controller collapsed to predict a single configuration when rewards of this form were provided (depending A), and did not learn
Now, consider streaming perception setup described Section 3.1. Deriving from this paradigm, our rewards optimize accuracy and latency simultaneously. Intuitively, reward R compares the model output with ground truth stream of the actual world state. Assume algorithm f is governed by policy æ receives observation Og, controller performs action ag at time tx to configure f and another
5

---

Algorithm 2: Training the Controller
Algorithm 1: Obtaining Observations From
Streaming Perception System
Function trein, cow.iin_cotrooeercee (st: Policy, S: Stream Replay
Buffer:;
Initialize policy at
Unpack length sequeence
Set change system configuration probability p
[(= a² c'.t))--- aa [tR) from replay bufler S:
for 1,...
do
for 1. 1..,,
Reset simulator;
r" R(t", 2 12);;
for every seguenee de
Store (2", ,,","" controller's replay buffer
Initialize empty replay buffer
= 1:
update(m)
while streaming
// Systen executes on streaning data
Update Policy
salue Draw from Bernoullipp):
Function updata Policy):
if volue == fru then
// Optinize the Q function
Observe from the stream;
= contezt(xx))
Sample N tuples
=
{[=] .,,r"), (.r).... p..... (2 a , uniformly from buffer B:
Change system config. using
far -ttme.mouul
for ..., N do
Store dr. fay) replay huffer
// Sat. the targets V" = r"
8
// Calculate
for
i
C
train (otpinooottooolr(.r.....
> 1िz Edean lv" Q=(=", d)]²
Use optimizer to optimize to minimize C;
observation Oy with action ay is taken at ty (t, > tz). We propose a reward,
R(12, bg) = L(Lve, (p(()))
(I)
where L is arbitrary single frame loss. Intuitively, the reward for action 0₂ is streaming accuracy obtained for a stream sequence segment until new action ay is taken in the fature and y ensures only real-time results are accounted for. Reward R(tz, ty) ensures learned controller implicitly considers
real-time constraints and accuracy of the decision ag, not assuming any properties the algorithm f or loss L. Thus, it can applied to other perception tasks.
While [6] defined Lstreaming observations from video stream of length T), Estreaming was employed to characterize performance of f on the sensor stream. Directly employing as reward made training the controller sample- inefficient, and did not converge before we exhausted our training budget (of 10 epochs).
L( ) and corresponds to R(O, T) (Loss over N
While reward proposed in Equation leamable, it biased towards the inherent characteristics of the video sequence. Consider two video sequences A and B, Sequence A with one large object that can be easily detected would have much higher average rewards than Sequence with many occluding and small objects. To normalize for this factor, we consider fixed policy т fized, i.c., fixed configuration of algorithm and propose,
Rfized adv (tws by) = Rs (42,,t,) -
Ray (tey ty)
where Rayang is the reward obtained by executing algorithm f following fixed policy. To implement Riized adus ff with fixed policy π fized executed and prefetched the predictions and the timestamps are stored. During training, we compute Rt and load fixed outputs to simulate Ra
This eliminates the bias in the inital proposed reward, and provides the advantage of learned policy over fixed policy (See Figure 6). In Section 4.1, we show any fixed policy configuration (i (izd) suffices and the policy learned is resilient to variations of this selection. We refer to the reward in Equation as RS and Equation 2 as R2 in our experiments for simplicity.
3.5 Controller Design Decisions
Our controller design decisions (such as employing deep contextual bandits) are motivated by severe real-time constraints. For e.g., we considered individual frames as independent trials while learning our controller, i.e. we compute context from a single frame to incur minimal overhead (amortized overhead of around 1 %.). Considering sequences as controller's input incurred significant overheads and impeded performance.

---

Table 1: Improvements on predefined system. Chanakya outperforms competing execution
policies for a predefined system. All the execution policies operate on top of a perception system employing the same components:
Faster R-CNN and Kalman Filter.
Approach
sAP
s.AP30
sAPTs
Streamer (s=900) [6] (Static Policy)
18.2
35.3
16.8
2. Streamer (s=600) [6] (Static Policy)
20.4
35.6
20.8
3. Streamer (s=600, np=300) (Static-Expert Policy)
20.8
36.0
20.9
Streamer (s=600, mp=500) (Static-Expert Policy)
20.9
35,9
20.9
5. Streamer + AdaScale [2] Crrnamic-Traditiinas Policy)
13.4
23.1
13.8
Streamer + AdaScale + Our Scheduler (Dynamic- Policy)
13.8
23.4
14.3
Chanakya (81, np,,, R=Rt))
21.0
36.8
21.2
8. Chanakya (81, npp,, R=R2, (s=2ed=(s=488, np=300)
21.3
37.3
21.1
9. Offline Upper Bound (s=600, Latency Ignored)
24.3
38.9
26.1
Combinatorially Increasing Configurations. Consider decision space D (D1, D2..Dm with dimensionality M [ where each decision dimension D; corresponds to a configuration parameter of the algorithm f discretized dg. Thus, an action has M decision dimensions and each discrete sub-action has a fixed number of choices. For example, if we consider 2 decision dimensions, D = { seaale ), the potential configurations would be Dscale (720, 600, 480, 360, 240), D Doodel = {yolo, fcos, frenn). ) Using conventional discrete-action algorithms, IL [d] possible actions need to be considered. Efficiently exploring such large action spaces is difficult, rendering naive discrete-action algorithms like [3, 21J intractable.
Chanakya uses multi-layer perceptron for predicting Qn(z, as)) for each sub-action a, and given context 2, and employs action branching architecture [40] to separately predict each sub-action, while operating shared intermediate representation. This significantly reduces the number of action-values to be predicted from IeeD [d] to EeEr |d].
Real-time Scheduler. Choosing which frames to process is critical to achieve good streaming performance. Temporal aliasing, i.e., the mismatch between the output and the input streams, reduces performance. [6] prove optimality of shrinking tail scheduler. However, this scheduler relies on the assumption that the runtime / of the algorithm f constant. This is reasonable as runtime distributions are unimodal, without considering resource contention.
Chanakya changes configurations at runtime, and corresponding p changes. Fortunately, our space of configurations is discrete, and p's (for every configuration) can be assumed to be constant. Thus, fragment of the sequence where particular configuration selected scenario where shrinking tail scheduler using that runtime will hold. We cache configuration runtimes of algorithm f and modify the shrinking tail scheduler to incorporate this assumption.
Asynchronous Training. Chanakya is trained such a way that real-time stream processing is simulated, akin to challenges presented in [19, 20]. Thus, key practical challenge is to factor in average latency for generating the context from the observations, computing the controller's action during training. We also need to ignore the time taken train the controller these would disrupt and change the system performance. Training the controller concurrently on a different GPU incurred high communication latency overhead.
Algorithms and 2 describe the training strategy. We train the controller intermittently after fixed while the sequence is
sequence has finished processing, the controller's parameters streamed. While the sequence is processing, we store the (z, a, t) tuples in a buffer S along with all the algorithm's 's predictions (to construct the reward r). During training, first add these tuples to controller's s buffer B and, sample from to update Q2(2.a).
Training for edge devices. We emulate training the controller for edge devices on server by using the measured runtimes for each configuration on the edge device, since Chanakya only requires the runtimes of the algorithm far on the target device to train the controller.
7

---

Table 2: Handling large decision spaces. We in- corporate a tracker in our predefined perception system. Choice of detector scale and proposals, tracker scale and stride, combinatorially expands our decision space. Chanakya able to learn a competitive policy compared to best static poli- cies [6].
Table 3: Learning decisions from scratch. Chanakya is able to decide model and other meta- parameter choices even when optimal algorith- mic components are unknown.
Approach
SAP
Static Policy (m=fcos, s=600)
16.7
Approach
SAP
Static Policy (m=yoloo3, s=600)
20.2
Static Policy (s=900, ts=600, k=5)
17.8
Statie Policy (m=frenm, s=600)
20.4
Static Policy (s=600, ts=600, k=5)
19.0
Chanakya (m=m, 81, np1, R-R2)
20.7
Chanakya (81, np1, ts, k, R2)
19.4
Experiments and Results
We analyse and evaluate Chanakya through the following experiments. We employ sAP metric [6] that coherently evaluates real-time perception, combining latency and accuracy into a single metric. It is non-trivial to compare traditional systems that use mAP and latency for evaluation as their algorithms would require major modifications. However, we do augment one such method, AdaScale [2] by incorporating Kalman Filter and our scheduler. Apart from this, we compare our approach with static policies obtained from benchmarking. Please note Chanakya's 's improvements are complementary and addition to the system components themselves, newer models would lead to further improvements. All the results are reported on ArgoverseHHD, unless stated otherwise.
Improvements on predefined system. For a real-time perception system on a given hardware (P40 GPU), let's assume system components detector (Faster R- R-CNN), scheduler (Modified Shrinking Tail) and forecaster (a Kalman Filter) are known. We optimize configurations across two decision dimensions D = {D, [360, 480, 540, (20). Dap {100, 300, 500, 1000]}, i.c., detector scale and number of proposals. We evaluate Chanakya using both variants of rewards described in Section 3.4, and only intrinsics are provided as context. We compare with best performing configurations reported by [6] (static policy). Further, we asked an expert who employed domain knowledge [22] to optimize the proposal decision dimension via benchmarking (static expert policy). We also compare with a dynamic traditional policy, AdaScale [2] which predicts input scale and is trained to minimize single frame latency as it is situated in traditional latency budget paradigm. Upper bound takes the best configuration of the static policy from [6] and simulate a detector with Oms latency for every frame. Table 1 shows that we outperform all of these methods, as Chanakya can adapt dynamically (See Figure 7) via context which explains improvement over static policies and considers the real-time constraint holistically. Chanakya does not solely minimize single frame latency which explains improvements over the traditional policy. Similar trends hold for ImageNet-VID. please see Appendix A.
Upper bound takes the best configuration of the static policy from [6] and simulate a detector with Oms latency for every frame. LearninL decisions from scratch. Consider when optimal components (say, the detection model or tracker) are unknown for a given hardware. We will now introduce the third dimension, which is the choice of the model (FCOS or YOLOv3 or Faster R- R-CNN). Chanakya has to pick configurations across model choices, scale choices and proposal choices (if applicable, decision dimension is ignored otherwise). Using an alternative detector, i.e, FCOS and with the best scale obtained via benchmarking (paired with Streamer [6]), the static policy has an sAP of 16.7, while Chanakya has sAP of 20.7 (23% improvement). Similar performance improvement (+5.5 sAP) was also seen on ImageNet-VIL
Handling large decision spaces. We extend our pre-defined system (Faster R-CNN Scheduler + Kalman Filter) with a tracker (Tracktor [36]). Tracker supports configuration of tracker scale and temporal stride, i.e. number of times it's executed versus the detector, apart from detector scale, number proposals, increases the decision space to 500 configurations. It is not feasible to explore the entire decision space using brute-force if we consider deployment to many hardware platforms. In contrast, our approach learned the execution policy in 5 training epochs. As shown in Table 2, Chanakya learns competitive policy that outperforms the best static policy from [6] by +1.6 sAP.
8

---

0.00
0.00
0.02
0.00
540
0.09
0.14
0.41
0.00
Ste
560
0.01
0.17
0.16
0.01
Statoj000
State/20)
480
0.00
0.00
0.00
0.00
Random
360
0.00
0.00
0.00
0.00
0
50 Contention Level
40
100
300
500
1000
Proposals
Figure 8: Chanakya adapts to extrinsic fac- tors. Compared static execution policies, Chanakya has better performance at different levels of GPU contention and degrades less at higher contention.
Figure 7: Dynamic Configurations executed P40 GPU. For the predefined system, Chanakya changes the configuration depending on the in- trinsics and achieves better performance.
720
720
640
360 480 560 640
0.01
Scales
Scales
360 480 560
0.00
0.00
0.04
0.00
0.00
0.01
0.00
0.00
0.00
0.03
0.55
0.06
0.01
0.03
0.78
0.05
0.03
0.01
0.18
0.03
0.00
0.00
0.03
0.00
Proposals
Proposals
Xavier NX
(b) Xavier AGX
Figure 9: Configuration shift on edge devices. The predefined system is deployed on the edge devices. Chanakya trained via emulation, and we observe shift chosen configurations that matches our expectations for these hardware platforms.
Performance and configuration shift edge devices. We consider deploying the predefined system discussed earlier on two edge devices Jetson Xavier NX and Xavier AGX. We employ emulated training procedure discussed in Section 3.5. From Table comparing performance on Xavier NX, Chanakya improves SAP by 12% and 28% Argoverse-HD HD and ImageNet- VID respectively over static policies found by benchmarking. In Figure devices (compared to Figure 7 evaluated on a server GPU P40). Xavier AGX is more powerful, and observe more frequent selection of higher detection scale 480) compared to Xavier NX, in line with our expectations.
observe shifts in configurations selected for the two
Adapting to Extrinsic Factors. In predefined system, consider contention factor and study Chanakya's behaviour, assuming black-box resource contention levels available run-time. This scenario is common smartphones/edge devices [4] where multiple processes employ the same resource. Static policies, wherein decisions are not made considering contention would degrade quickly (See Section 3.3).
extrinsic
Chanakya adapts to sensed contention level (provided as context) by choosing decisions which correspond to the sweet-spot for that contention level, with no additional modifications. Critically, Chanakya is ficxible and adapts decisions to the contention level unlike [4] who explicitly model the problem as rule-
system (modeling latency and accuracy with quadratic regression models) and handcraft a contention- aware scheduler. Performance degrades across all approaches with increasing contention and Figure 8 Chanakya is able to adapt better than static policies.
New hardware and action spaces. Consider the task of migrating streaming perception system from a device employing P40 GPU, to newer V100 GPU. New hardware has additional features, such as faster inference lower precision levels (P40 in comparison does offer limited support; few FP16 operations are faster). As the tradeoffs have shifted, we do not assume that components and hardware optimizations are known. We consider additional decisions choice between inference FP32 and FP16 and choice between models. We employed Chanakya with the following decision space:

---

Table Performance on edge devices. Comparisons on Xavier-NX edge device. We can observe that Chanakya improves performance over static policies on both the datasets.
Approach
Argoverse-Hi HD (sAP)
ImageNet- VID (sAP)
Streamer (s=560, np=500) (Static Policy) [6]
7.8
18.8
Streamer (s=640, np=300) (Static Policy) [6] Streamer with Random Selection
7.4
20.0
7.6
17.5
Chanakya RR= R1) Chanakya (R R2)
8.1
19.4
8.3
22.4
Table 6: Resilience. Chanakya learns perfor-
Table 5: Importance of Context. All compo- nents of the considered context contribute to im- proved real-time performance.
mant decisions despite using a bad fixed policy. Further, minimal performance degradation is ob- served between 81 and 82
Dropped feature
sAP
Approach
sAP
None
21.3
Ours (81, Rip,, R2, fin (640, 300))
21.3
Adaptive Scale
19.2
Ours (81, np1, R2, fin (480, 300)
21.3
Category aggregates
19.4
Ours (81, np1, R2, fin (360, 1000))
21.1
Confidence aggregates
20.3
Object size aggregates
20.4
Ours (82, np1, R1)
18.2
Ours (82, np1, R2)
20.4
(Dm : (resmet_frcnm, swin_frenn, rewnee_ccsscueerreeeeeeeeeee (FP32, FP1)), D2 {800, 880, 960, 1040, 1120, 1200}. Dmp {100, 300, 500, 1000)). Without benchmarking and domain expertise, it's non-trivial to decide these meta- best known static policy (Faster R-CNN with Resnet-50 at 900) proposed by [6] for V100 GPU as our baseline which has 18.3 8 sAP. Chanakya obtained 27.0 sAP, i.e., +8.7 sAP (47%) improvement, the highest improvement the Streaming Perception Challenge without additional data.
cascade_renn, swwn_ccsscuee_reeeeeeooo renn]. : Dp
meta-parameters, parameters. We consider the
4.1 Ablation Studies
We discuss some of the considerations of our approach using Chanakya our perception system. Importance of Context. Table 5 shows that when context feature dropped performance reduces by ~ sAP. Context overheads can be viewed in Appendix C.
Resilience. Table 6 shows that Chanakya is resilient to the choice of the initial fixed policy π fixed used our reward function R2. Chanakya arrives at competitive policy even if a bad fixed policy chosen, (360, 1000), which employs very small scale and the maximum number of proposals. Chanakya is resilient to the choice of action space as changing scale space from 81 to 82 yielded competitive performance.
Conclusion
Limitations and Broader Impact. While our method does not directly enable unethical applica- tions, might be employed to deploy and improve such applications. Further study of regulatory mechanisms for safe MI deployments and the impact of methods like Chanakya is required.
We presented Chanakya, a learned runtime execution framework optimize runtime decisions for real-time perception systems. Our learned controller is efficient, flexible, performant and resilient in nature. Chanakya appears useful for many real-time perception applications in the cloud and edge. We plan to extend this work to scenarios wherein applications require hard performance guarantees.
Acknowledgements
We thank Harish YVS for his insightful discussions and the reviewers for their detailed comments.
10

---

References
[1] Seungyeop Han, Haichen Shen, Matthai Philipose, Sharad Agarwal, Alec Wolman, and Arvind Krishnamurthy. Mcdnn: An approximation-based execution framework for deep stream pro- cessing under resource constraints. In MobiSys, 2016.
[2] Ting-Wu Chin, Ruizhou Ding, and Diana Marculescu. Adascale: Towards real-time video object detection using adaptive scaling. MLSys, 2019.
[3] Ning Chen, Siyi Quan, Sheng Zhang, Zhuzhong Qian, Yibo Jin, Jie Wu, Wenzhong Li, and Sanglu Lu. Cuttlefish: Neural configuration adaptation for video analysis in live augmented reality. Transactions on Parallel and Distributed Systems, 2020.
[4] Ran Xu, Chen-lin Zhang, Pengcheng Wang, Jayoung Lee, Subrata Mitra, Somali Chaterji, Yin Li, and Saurabh Bagchi. ApproxDet: content and contention-aware approximate object detection for mobiles. In SenSys, 2020.
[5] Kittipat Apicharttrisom, Xukan Ran, Jiasi Chen, Srikanth Krishnamurthy, and Amit Roy- Chowdhury. Frugal following: Power thrifty object detection and tracking for mobile augmented reality. In SenSys, 2019.
[6] Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. ECCV, 2020.
[7] Kai Chen, Jiangmiao Pang. Jiaqi Wang. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019.
[8] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[9] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[10] Jinrong Yang. Songtao Liu, Zeming Li, Xiaoping Li, and Jian Sun. Real-time object detection for streaming perception. In CVPR, 2022.
[11] Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan, Fovea: Foveated image magnification for autonomous navigation. In ICCV, 2021.
[12] Babak Ehteshami Bejnordi, Amirhossein Habibian, Fatih Porikli, and Amir Ghodrati. Salisa: Saliency-based input sampling for efficient video object detection. In ECCV, 2022.
[13] Anurag Ghosh, N Dinesh Reddy, Christoph Mertz, and Srinivasa G Narasimhan. Learned two-plane perspective prior based image resampling for efficient object detection. CVPR, 2023.
[14] Peizhen Guo, Bo Hu, and Wenjun Hu. Mistify: Automating DNN model porting for On-Device inference at the edge. In NSDI, 2021.
[15] Rui Han, Qinglong Zhang, Chi Harold Liu, Guoren Wang, Jian Tang, and Lydia Y Chen. Legodnn: block- -raineg scaling of deep neural networks for mobile vision. In MobiCom, 2021.
[16] Michael P Georgeff and Amy L Lansky. Reactive reasoning and planning, In AAAI, 1987.
[17] Eric J Horvitz. Computation and action under bounded resources. PhD thesis, 1990.
[18] Mark Boddy and Thomas Dean. Deliberation scheduling for problem solving time- constrained environments. Artificial Intelligence, 1994.
[19] Simon Ramstedt and Chris Pal. Real-time reinforcement learning. NeurlPS, 2019.
[20] Jaden B Travnik, Kory W Mathewson, Richard Sutton, and Patrick M Pilarski. Reactive reinforcement learning in asynchronous environments. Frontiers in Robotics and AI, 2018.

---

[21] Young Geun Kim and Carole-Jean Wu. Autoscale: Energy efficiency optimization for stochastic edge inference using reinforcement learning. MICRO, 2020.
[22] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/aceuracy trade-offs for modern convolutional object detectors. In CVPR, 2017.
[23] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In ICCV, 2019.
[24] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
[25] Chien-Ye Cao Wang, Alexey Bochkovskiy. and Hong- Yuan Mark Liao. Scaled-yolov4: Scaling cross stage partial network. In CVPR, 2021.
[26] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML,, 2019.
[27] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[28] Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurlPS, 2017.
[29] Shiqi Jiang, Zhiqi Lin, Yuanchun Li, Yuanchao Shu, and Yunxin Liu. Flexible high-resolution object detection on edge devices with tunable latency. In MobiCom, 2021.
[30] Xu Zhang, Yiyang Ou, Siddhartha Sen, and Junchen Jiang. SENSEL: Aligning video streaming quality with dynamic user sensitivity. In NSDI, 2021.
[31] Romil Bhardwaj. Zhengxu Xia, Ganesh Ananthanarayanan, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, and Ion Stoica. Ekya: Continuous learning of video analytics models on edge compute servers. In NSDI, 2022.
[32] Ran Xu, Jinkyu Koo, Rakesh Kumar, Peter Bai, Subrata Mitra, Ganga Meghanath, and Saurabh Bagchi. Approxnet: Content and contention aware video analytics system for the edge. arXiv preprint 190900900000006 2019.
[33] Tiffany Yu-Han Chen, Lenin Ravindranath, Shuo Deng, Paramvir Bahl, and Hari Balakrishnan. Glimpse: Continuous, real-time object recognition on mobile devices. In SenSys, 2015.
[34] Xukan Ran, Haolianz Chen, Xiaodan Zhu, Zhenming Liu, and Jiasi Chen. Deepdecision: A mobile deep learning framework for edge video analytics. In INFOCOM, 2018.
[35] Anurag Ghosh, Srinivasan Iyengar, Stephen Lee, Anuj Rathore, and Venkata N Padmanabhan. React: Streaming video analytics on the edge with asynchronous cloud support. In loTDI, 2023,
[36] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In ICCV, 2019.
[37] Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. Tide: identifying object detection errors. In ECCV, 2020.
A general toolbox for
[38] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[40] Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. In AAAI, 2018
12

---

[41] Ming-Fam Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang. Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In CVPR, 2019.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. I/CV, 2015.
[43] A visual walkthrough of streaming perception solutions. https://mtli.g: ieaiiiiggttbbb. io/ html, 2021.
8 wtwp:/ggsstteaiiiniiiiii
[44] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:/906.011 07155, 2019.
[45] Kai Kang, Hongsheng Li, Junjie Yan, Xingyu Zeng, Bin Yang, Tong Xiao, Cong Zhang, Zhe Wang. Ruohui Wang, Xiaogang Wang, et al. T-enn: Tubelets with convolutional neural networks for object detection from videos. Transactions on Circuits and Systems for Video Technology, 2017.
[46] Akshay Uttama Nambi, Adtiya Virmani, and Venkata Padmanabhan. Farsight: A smartphone- based vehicle ranging system. IMWUT, 2018.
[47] Akshay Uttama Nambi, Ishit Mehta, Anurag Ghosh, Vijay Lingam, and Venkata N Padman- abhan. Alt: towards automating driver license testing using smartphones. In Proc. SenSys, 2019.
[48] Songtao He, Favyen Bastani, Arjun Balasingam, Karthik Gopalakrishna, Ziwen Jiang, Moham- mad Alizadeh, Hari Balakrishnan, Michael Cafarella, Tim Kraska, and Sam Madden. Beecluster: drone orchestration via predictive optimization. In MobiSys, 2020.
[49] Arjun Balasingam, Karthik Gopalakrishnan, Radhika Mittal, Mohammad Alizadeh, Hamsa Bal- akrishnan, and Hari Balakrishnan. Toward marketplace for aerial computing. In Proceedings of the 7th Workshop on Micro Aerial Vehicle Networks, Systems, and Applications, 2021.
[50] Srinivasan Iyengar, Ravi Raj Saxena, Joydeep Pal, Bhawana Chhaglani, Anurag Ghosh, Venkata N Padmanabhan, and Prabhakar T Venkata. Holistic energy awareness for intelli- gent drones. In BuildSys, 2021.
13


Reviews:
[{"review_id": "snvTsMR2Wy", "reviewer": "Reviewer_Y14P", "summary": "The paper presents Chanakya, a learned framework for real-time perception that automatically balances accuracy and latency. Unlike previous fixed rule-based methods, Chanakya considers both intrinsic and extrinsic factors and is trained to make flexible decisions. evaluations show that it outperforms existing execution policies on both server GPUs and edge devices with low overhead.", "questions": "1. It would be good to have a diagram figure to show the high level overview of the whole framework.\n2. There exists some work that use elastic model setting for different scenarios such as [1]. More discussions and comparisons are needed.\n\n[1] Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. \"Scaled-yolov4: Scaling cross stage partial network.\" Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. 2021.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. It proposes a learning based approximate execution framework to learn runtime decisions such as the resolutions of input, which model to use, etc.\n2. The proposed framework considers multiple context both intrinsic and extrinsic to improve the reliability of the runtime decisions.", "weaknesses": "1. It is unclear how generalizable is the trained policy. More discussions on when train from scratch and when transfer learning and the corresponding cost when the system is in a different environment is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review!\n\n1. We train the controller from scratch depending on the conditions simulated. We leave robustness as future work, as large number of inferences from existing RL works can be drawn to improve this aspect (such as domain randomization). Cost of training the policy is a fraction of training the detection models themselves (a 4 layer MLP for 10 epochs).\n\n2. Thank you for this suggestion! We shall add a high-level overview diagram in the camera ready version of the paper.\n\n3. The paper (Scaled-YOLOv4) suggested is interesting, structure of the model is search for apart from network depth and width to obtain a family of models with good accuracy-latency characteristics. Critically, that paper optimizes decisions taken during training a family of detectors, which is not our focus. We focus only on runtime decisions and we can learn to select from a slew of already trained models (such as family in the mentioned paper) for a given hardware/environment. We shall mention it in related work."}, {"review_id": "6Sual9N7jB", "reviewer": "Reviewer_L1Mt", "summary": "The author designs Chanakya, a learned execution framework for streaming perception that jointly optimizes accuracy and latency. To achieve so, the framework captures both intrinsic and extrinsic contexts and utilizes a novel reward function to train a learned controller.", "questions": "How does your approach compare to hybrid solutions that employ handcrafted rule-based heuristics for extrinsic factors and learning-based decision algorithms for intrinsic factors?", "limitations": "In the subsection \"Limitations and Broader Impact\", the authors point out that their framework can be used to deploy unethical applications, and I think this is just a potential negative societal impact, not a limitation. Therefore, I encourage authors to talk about the limitations of your work and potential solutions.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clear and the discussion of related work is comprehensive.\n2. The description of experimental setting is detailed and results of ablation studies are provided.", "weaknesses": "1. Novelty is somewhat limited. The methodological innovation in this paper is limited to use both intrinsic and extrinsic contexts to better learn runtime decisions with reinforcement learning based methods. The fundamental methodology is mostly shared with prior work.\n2. Presentation can be improved. Table 1 is at the top of page 6, but no context related to it appears until page 7. Asynchronous Training and Training for edge devices in Section 3.5 seems unnecessary to discuss.\n3. Baselines are incomplete. I think the experiment should at least have vanilla RL-based baselines and then show the effectiveness of the proposed framework step by step.\n4. I'm not sure it's a good idea to incorporate all factors into a learning controller. For single GPU program, I believe rule-based decision algorithms can adapt extrinsic factors (e.g. software and hardware status) very well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review, it has helped us clarify aspects of our work. We believe the raised concerns are addressable in the manuscript.\n1. Please see the main comment for novelty concerns.\n2. We shall improve the presentation of the work and move the tables and add a high level overview diagram.\n3. **Vanilla RL:** Please see main comment.\n4. **All Factors:** This is a problem specifically on edge devices, where users have multiple applications running on device simultaneously. While rule based decisions for this consideration do appear to work [4], however, even methods like ApproxDet hypothesize using NNs to infer configurations and RL to learn scheduling due to its distinct advantages.\n5. **Hybrid Methods:**\n     - We compare with one such baseline for heuristics with a learned metric – AdaScale [2]. The rule is greedily accepting the scale value from the learned scale predictor for the next frame.\n     - No other entirely learning-based real-time decision algorithms or “handcrafted rule-based heuristics for extrinsic factors and learning-based decision algorithms for intrinsic” exist as far as we are aware. We present first such general framework.\n\n6. **Limitations:** (a) Our training and test environment is the same and we haven’t tested the robustness (and adverserial robustness) of the learnt policy. With the vast literature on RL in domain/environment randomization, we expect that this limitation will be resolved in future work. (b) We considered context to be instantaneous w.r.t to a frame, however, for some considerations like power and temperature, accounting for context over a longer time horizon is generally required [Chen2021].\n\n[Chen2021] Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimization, e-Energy, 2021\n"}, {"review_id": "jZKn4Z2n2J", "reviewer": "Reviewer_fYnV", "summary": " This work provides a novel learning-based approximate execution framework to learn runtime decisions for real-time perception. The learned controller proves to be efficient and performant, which appears to be useful for many real-time perception applications in the cloud and edge.", "questions": "How to evaluate and compare between the controllers on different hardware, which can have different features? ", "limitations": "As listed in weakness part, more experimental results on other perception task can be reported, and not sure whether the decision space is valid.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The focus of the work on real-time perception system's runtime execution decisions is important and timely.\n2. The proposed Chanakya proves to learn performant execution policies and can incorporate new decision dimensions and be ported to different hardwares.", "weaknesses": "The experimental results are only reported on detection tasks, and other perception task like segmentation might be reported.\nThe decision dimention seems to be discrete and limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review, we are glad to see that you agree that Chanakya has been proved to learn performant execution policies in a variety of scenarios.\n\n1. **Single Task Experiment:** We performed a thorough study using one task – detection, but across datasets, scenarios and edge and cloud devices. Critically, runtime characteristics of instance segmentation models (e.g. Mask R-CNN) are like detection models (e.g. Faster R-CNN) [6]. For experimenting with instance segmentation, we need to additionally define different set of context functions and a more complicated forecasting mechanism which further complicates experimental design. \n2. **Discrete Dimensions:** Most of the prior work also operate on discrete decision dimension(s), usually one such dimension.\n3. **Evaluation across hardware:** Could you please clarify the question regarding comparing controllers on different hardware? What is to be compared?\n"}, {"review_id": "GZw6u1dvNd", "reviewer": "Reviewer_Gsqp", "summary": "The paper looks at the problem of unpredictable compute requirements for real-time perception. It addresses this in term of a multi-objective optimisation problem (quality of results and latency). The authors propose using RL in order to optimise the selection of various characteristics in order to achieve an optimal result.", "questions": "- The authors spend a lot of the introduction naming and saying how good their approach is. This should be more analytical than what is presented.\n\n- Lines 89-94 give a formal definition of the perception problem. However, most of the introduced terminology does not get used further in the paper. This space could be used for other material.\n\n- Figures 2,3&4 present results, but it is not clear where these results came from and how they were produced.\n\n- \"In our experiments we observed that controller collapsed to predict a single configuration when rewards of this form were provided (depending on λ), and did not learn\" - which experiments?\n\n- ", "limitations": "Small amount of discussion of limitations in conclusion.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and provides a good introduction for the reader and guides them through the work.", "weaknesses": "The proposed approach does not come across as that novel. There is a lot of prior use of RL techniques for multi-objective optimisation which the authors don't appear to have looked at. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions in the review, they will definitely improve the quality of the manuscript.\n1. Novelty: Please see main comment. \n2. Thank you for this suggestion. We have added some key numerical highlights in main comment, which we will add in the introduction.\n3. Thank you for this suggestion, we shall add a connection to Section 3.4. The setup mentioned in Section 3.1 is necessary for understanding Section 3.4.\n4. We have mentioned experimental, implementation and a few algorithmic details in supplementary, we shall add more explanation there. 5. Please let us know what exactly is missing in the results section, we shall do our best to incorporate the suggestion.\n6. Apologies for the confusion, we trained the controller with reward defined in [3] and the learnt policy always picked a singleton configuration (like static policy) depending on the hyperparameter \\lambda. Previous work [3, 21] data is not public, so we could not compare directly on their data. Hence, we omitted the comparison in our results. \n"}, {"review_id": "rt8nKjQzi8", "reviewer": "Reviewer_7j3D", "summary": "This paper introduces Chanakya, a framework for computation planning in real-time (streaming) perception. Chanakya uses a novel reward to make run-time decisions based on content- and system-based characteristics, simultaneously optimizing for both accuracy and latency. The proposed controller is learned on individual frames and uses a normalized reward to reduce bias. Chanakya’s performance is evaluated on a real-time perception stack consisting of a detector, scheduler, forecaster and tracker; these individual components are made part of the search space (eg: choice of model), along with their associated configurations (scale, #proposals, etc.). Results are reported across edge devices, runtime contention levels, and for cases where the perception stack has been migrated from one GPU to another.\n", "questions": "The comparison in Figure 1(c) is to RetinaNet, which is a network from 2017. Do more modern networks (eg: ConvNeXt/Swin, CoAt-Net, MaxViT, etc.) fulfill the real-time/streaming constraint by being able to handle higher resolutions?", "limitations": "Limitations have been adequately addressed in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This is a well-written paper, and the core ideas have been explained clearly.\n* Impact & relevance: the paper targets real-time perception, which is an important component of modern AV and embodied systems.\n* Strong results: Chanakya appears to outperform SOTA static and dynamic approaches, and ones designed by domain experts.\n", "weaknesses": "* It is not clear how well Chanakya scales with increasing search space size. While the paper includes an evaluation of a reasonable number of decision dimensions, it would be interesting to get an understanding of Chanakya's scalability limits, especially since the authors claim in Section 2 that the search space could potentially be expanded by trying to account for varying input resolutions or via techniques such as pruning and/or quantization.\n* No comparisons to non-RL-based approaches, including rule-based and purely latency-focused systems.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review, we shall incorporate these points in the final version of the manuscript.\n1. **Increasing Search Space:** We agree that this is an important direction. However, adding even more decision dimensions – such as edge-cloud processing (should an image be sent to a cloud GPU or executed at edge GPU like Jetson) would be involve development of more complicated simulations (for network variability). We leave this for future work.\n2. **Prior work:** The execution model of prior work is different. It’s difficult to compare as every algorithm needs to be modified for a fair comparison. We did include one such baseline AdaScale [2] (and modified the scheduling algorithm) – which is not RL based, which reduces image scale and solely optimizes latency by proxy. Performance is worse by 35%.\n3. **New Models:** We are looking to develop execution policies that are agnostic to the model choice and hardware choice. Recent work has shown models with better accuracy-latency tradeoffs (like, StreamYOLO, YOLO-X), however, the fundamental tradeoff remains. Models like StreamYOLO do satisfy the real-time constraint on high end GPUs like V100, but are not hardware-agnostic, due to design limitations performance suffers on older and edge hardware [13]. \n"}, {"review_id": "tTywWPnzJ5", "reviewer": "Reviewer_6nGS", "summary": "This paper proposes a learned approximate execution framework named Chanakya. Chanakya considers intrinsic context like images and extrinsic context like latency and predicts runtime decisions. The reward function helps to learn a better trade-off online. Extensive experiments on the Argoverse-HD dataset show that Chanakya outperforms static policy and can be applied to new hardware and action spaces.", "questions": "1. Why did employing L_{streaming} not converge which is mentioned in Line 165?\n2. In table 1, how do you get sAP for offline upper bound?", "limitations": "The authors just mention that their method might be employed to improve unethical applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper jointly optimizes accuracy and latency for real-time tasks. The reward function proposed considers both the simultaneous optimization between accuracy and latency, and different characteristics of the video sequence.\n2. The detailed analysis of experiment results demonstrates the performance improvements of Chanakya.\n3. This paper is well-organized and well-written to read.", "weaknesses": "1. Some settings of experiments are not clarified, like offline upper bound and RL training method.\n2. Some details of learning the controller should be introduced in the main part of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thought provoking review, we shall incorporate the suggested point in the final manuscript!\n1. Thank you for the upper bound suggestion, please see main comment.\n2. Details of controller training method are provided in the supplementary along with the anonymous code link, please let us know if there are specific questions that need clarification. \n3. In both the datasets, the number of videos available for training is of the magnitude of hundreds (say N). L_{streaming} yields only one reward value per video execution (thus one epoch → N samples for training) and the model performance was worse than static policies after 5 epochs. To circumvent this, we essentially chunked every video into temporal segments (essentially R_{1}) yielding far more samples for training ((T/30)*N samples where T is the average length of the video).\n"}]

Author rebuttal:
We thank all the reviewers for their insightful comments. We are glad to see that reviewers noted that our paper is “well written” (R4, R5, R6) and “well-organized” (R6) as we tackled an “relevant and impactful problem” (R5) with a “clear motivation” (R3). It is heartening to note that our “learned execution framework for real-time perception” is “proved to learned performant execution policies” (R4) through “detailed experimental setting” (R2), showing “strong results” (R5) by “demonstrating the performance improvements” (R6) while considering “both intrinsic and extrinsic” (R1) environment contexts.

### Contribution
We propose a novel learned framework to learn execution decisions for real-time perception, optimizing accuracy and latency simultaneously in a variety of scenarios. Detector/tracker etc are **not retrained**, improvements are **only from learned execution policy**.
1. We obtain improvements of 17% on Argoverse-HD and 9% on Imagenet-VID for a pre-defined perception system. This improvement is complementary to the components and is better than the best static policies found by benchmarking and domain experts.
2. Our method can improve performance in a variety of hardware and scenarios. E.g., when environment has process contention, our execution policy degrades better than static policies. Our algorithms (like scheduler) are not modified to account for new considerations, unlike prior work.

### Novelty (R2/R4)
While finding good runtime decisions through execution frameworks, and improving the possible pareto frontier through better models for real-time perception has been studied, however, 
1. Prior work do not learn the execution framework and thus generally operate on **one** consideration or decision dimension – either intrinsic or extrinsic. In reality, real-time perception systems have many considerations. Thus, they need to modify algorithms, e.g. the scheduler to incorporate any new consideration (and model the interactions of every consideration with others; or come up with a new heuristic). We claim novelty and show advantages of our learned execution framework.
2. Prior work optimize suboptimal proxy metrics. For example, optimizing scale solely [2] (a proxy for latency itself) drastically reduced performance. We claim novelty of the objective to optimize and the design decision to situate our execution framework in the streaming perception problem [6] which led to this natural objective.

### RL Novelty/Vanilla RL (R2/R4)
Proposing new RL algorithms is not the focus of this work and there might be other RL algorithms (like Actor-Critic Methods) that are also applicable. Due to resource constrained nature of real-time perception, we formulate an RL setup that does not hamper performance over static policies.

### Offline Upper Bound (R6)
We take the best configuration of the static policy from [6] and simulate a detector with 0ms latency for every frame and obtain an upper bound. We shall mention this in the camera ready version.

Denoting R1 to 6 for Y14P, L1Mt, fYnV, Gsqp, 7j3D, 6nGS respectively.

Meta-review: The submission proposes a method for optimizing resource utilization when running neural networks for perception tasks. In particular, two considered tradeoff factors are accuracy and latency, which are important for real-world applications, such as AVs. The method uses an approximate execution framework to automatically learn decisions induced by these tradeoffs, which incorporates the so-called intrinsic (eg scene clutter) and extrinsic (eg hardware status) characteristics. The reviewers unanimously recommend the submission for acceptance. The weaknesses raised in the reviews were relatively on the insignificant side, and the clarifications provided in the rebuttal were found helpful. The authors are recommended to incorporate them and the other comments raised in the reviews in preparing their camera ready.
Decision: Accept (poster)