SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anurag Ghosh $^{1*}$  Vaibhav Balloli $^{2*}$  Akshay Nambi $^{3}$  Aditya Singh $^{3*}$  Tanuja Ganu $^{3}$ \
 $^{1}$ Carnegie Mellon University  $^{2}$ University of Michigan  $^{3}$ Microsoft Research India\
anuraggh@cs.cmu.edu, vballoli@umich.edu, aditya.singh.mat17@iitbhu.ac.in, {akshayn, taganu}@microsoft.com
Real-time perception requires planned resource utilization. Computational planning in real-time perception is governed by two considerations – accuracy and latency. There exist run-time decisions (e.g. choice of input resolution) that induce tradeoffs affecting performance on a given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics.
Earlier runtime execution frameworks employed rule-based decision algorithms and operated with a fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate execution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoffs instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-art static and dynamic execution policies on public datasets on both server GPUs and edge devices. Code can be viewed at https://github.com/microsoft/chanakya.
Real-time perception is an important precursor for developing intelligent embodied systems. These systems operate on sensory data on top of a hardware substrate, viz, a mix of resource-constrained, embedded and networked computers. Resource planning is a critical consideration in such constrained scenarios. Further, these systems need to be carefully designed to be latency sensitive and ensure safe behaviour (See Figure 2).
Many runtime execution decisions exist, e.g. the spatial resolution (scale) to operate at, temporal stride of model execution, choice of model architecture etc. The runtime decisions are influenced by (a) intrinsic context derived from sensor data (e.g. image content) (b) extrinsic context observed from system characteristics, e.g. contention from external processes (other applications). A execution framework is needed to use available context, jointly optimize accuracy and latency while taking these decisions.
Traditional execution frameworks [1, 2, 3, 4] operate in the following paradigm – optimize model to operate within a latency budget. Largely, they do not jointly optimize accuracy and latency for real-time tasks. Instead, they optimize proxies like latency budget [1], input resolution [2], latency from resource contention [4] or energy [5]. Their approach requires rule-based decision making – heuristic design for every characteristic, instead of learning the decision function. These frameworks explicitly handcraft rules to accommodate hardware capabilities. Moreover, traditional strategies (Figure 2) budgeting latency are suboptimal when operating on streaming data [6].
*Work done at Microsoft Research India
Misaligned Truck
Figure 1: Optimizing real-time perception inference. (a) A powerful detector (HTC [7]) has very good offline (non real-time)  $(38.2\text{mAP})$  performance. (b) However, by the time it finishes execution  $(750ms$  per frame), inferences are not aligned with objects  $(6.2\text{sAP})$ . (c) Solely optimizing for latency via a faster model  $(30ms$ , RetinaNet [8] at low resolution) leads to suboptimal and unsafe behavior  $(6.0\text{sAP})$ , like missing pedestrians. (d) Learning decisions with favourable inference tradeoffs with Chanakya  $(21.3\text{sAP})$  achieves better real-time performance.
Detected Pedestrian
Missed Pedestrian
Operating in the new streaming perception paradigm, we propose Chanakya $^2$  – a novel learning-based approximate execution framework to learn runtime decisions for real-time perception. Chanakya operates in the following manner – (a) Content and system characteristics are captured by intrinsic and extrinsic context which guide runtime decisions. (b) Decision classes are defined, such as the input resolution (scale) choices, the model choices etc. Chanakya handles these interacting decisions and their tradeoffs which combinatorially increase. (c) Accuracy and latency are jointly optimized using a novel reward function without approximating either objectives. (d) Execution policy is learnt from streaming data, and is dynamic, i.e. configuration changes during real-time execution of the perception system.
Chanakya learns performant execution policies and can incorporate new decision dimensions, stochastic system context and be ported to different hardware. Importantly, our improvements are in addition to, and complementary to models and the decision dimensions themselves - faster & more accurate models [9, 10], vision-guided runtime decisions [11, 12, 13] and system-focused decisions [14, 15] can be easily incorporated.
Flexible Real-Time Decision Making. Intelligent systems that are reactive [16] and flexible [17] are well studied. Our work is related to classical resource utilization and scheduling literature [18] and theoretical frameworks for real-time reinforcement learning in asynchronous environments [19, 20]. Contrasting with prior RL based execution frameworks [3, 21] which are operating in the suboptimal traditional latency budget paradigm [6], Chanakya is situated in the streaming perception paradigm. The key challenge for all these works has been the difficulty in designing rewards to jointly optimize accuracy and latency, and previous frameworks have learnability issues (See Section 3.4). We propose Chanakya with a novel reward and asynchronous training procedure for training approximate execution frameworks on large public datasets.
Real-time Perception and Video Analytics. In the vision community, accuracy-latency pareto decisions or "sweet spots" for vision models have been studied [22, 6]. Drawing from these inferences, proposed design decisions exploit spatial [11, 13] and temporal [10] redundancies and improve real-
$^{2}$ Chanakya was an ancient Indian polymath, known for good governance strategies balancing economic development and social justice.
Figure 2: High level overview. Chanakya is a learned execution framework for real-time perception that jointly optimizes accuracy and latency. Assuming a perception system operating on a real-time video stream, we describe meta-parameters that are modified during execution – they depend on the scene (intrinsic context) and hardware considerations (extrinsic context). A specific choice of meta-parameters corresponds to a runtime execution configuration decided by the learned controller. The controller emits a configuration given context so that real-time accuracy is maximized.
time perception. These works are complementary to our work, and their meta-parameters can be learned via Chanakya, and changed at runtime (static in their work). Improved models [23, 24, 9], say via found by architecture search [25, 26] during training, compression techniques like quantization [27] and pruning [28] are also complementary.
In the systems community, Chanakya is related to video analytics efforts for the edge and cloud, wherein works aim to optimize design decisions [15, 29, 14, 30, 31] during training and inference. Chanakya can incorporate their decision designs during inference (like AdaScale [2]). For example, LegoDNN [15] decides which NN blocks to use via a novel optimization formulation, which instead can be learned via Chanakya. Mistify and Ekya [14, 31] tailor models considering offline accuracy, latency and concept drift and these model choices can be selected by Chanakya at runtime.
Approximate Execution Frameworks. Many approximate execution frameworks have been designed for real-time perception. However, instead of maximizing real-time performance, they optimize proxies such as energy [5], latency from contention [4], or latency budget [1] and employ handcrafted rule-based heuristics. Previous works [4] have also highlighted the need for reinforcement learning (RL) based methods to learn runtime decisions. Chanakya instead learns runtime decisions to maximize streaming performance conditioned on intrinsic and extrinsic context, operating on a variety of decision dimensions. Many systems frameworks describe runtime decisions such as changing input resolution [2, 32], switching between models [1, 32], changing model configurations with latency [4], selecting frames to execute models vs trackers [33, 5], deciding between remote execution on the cloud and local execution on the edge [33, 34, 35]. These decisions can be learnt by Chanakya.
3 Chanakya: Learned Execution Framework for Real-Time Perception
3.1 Background on Streaming Perception
Traditionally, for real-time perception, model is executed at input video's frequency, say, 30 FPS. Thus the model is constrained to process a frame within a latency budget, in this case, 33 ms per frame. Streaming perception [6] instead places a constraint on the algorithm's output, no constraints are placed on the underlying algorithm. The constraint is - anytime the algorithm is queried, it must report the current world state estimate at that time instant.
Formal Definition: Input stream comprises of sensor observation  $x$ , ground truth world state  $y$  (if available) and timestamp  $t$ , denoted by  $\{O_i : (x_i, y_i, t_i)\}_{i=1}^T$ . Till time  $t$ , algorithm  $f$  is provided  $\{(x_i, t_i) | t_i < t\}$ . Algorithm  $f$  generates predictions  $\{(f(x_j), s_j)\}_{j=1}^N$ , where  $s_j$  is the timestamp when a particular prediction  $f(x_j)$  is produced. This output stream is not synchronised with input stream and is not necessarily operating on every input. Real-time constraint is enforced by comparing the most recent output  $f(x_{\varphi(t)})$  to ground truth  $y_i$ , where  $\varphi(t) = \operatorname{argmax}_j s_j \leq t$ .
Throughput vs Latency. Assuming two hardware setups, one with a single GPU available and another with infinite GPU's available (simulated), we employ a Faster R-CNN detector (no forecaster)
Figure 3: Configuration tradeoffs of a perception system. Many configurations (tracker scale, detector scale, temporal stride, number of proposals). Configs. in red ellipse satisfy real-time budget  $33ms$
Figure 5: Extrinsic Factors. Process contention is dynamic, and induces latency differences for a detector (Faster-RCNN) at a given scale. Notice that this relationship is non-linear. Other components follow similar trends.
Figure 4: Offline vs Real-time Perception. mAP vs sAP for different configurations of same system. An increase in offline performance (mAP) doesn't imply an increase in real-time performance (sAP).
on a P40 GPU. The sAP scores obtained are 13.9 sAP and 15.7 sAP respectively. Additional compute (GPU's) improves system throughput, but does not improve latency of real-time execution. Thus it is not a solution for improving real-time perception.
3.2 High Level Overview
We design a learned execution framework for real-time perception that jointly optimizes accuracy and latency. We consider object detection to be the instantiation of our perception problem, however, the described principles apply without loss of generality.
Consider real-time object detection. The real-timeness requirement of streaming perception is fulfilled by pairing our detector (say Faster R-CNN) with object tracker (Tracktor [36]) and state estimator (Kalman Filter). We can change meta-parameters like detector's and tracker's scale, temporal stride of tracker along with internal parameters (for instance, number of proposals in the detector) and generalize this to include the choice of components themselves (the detector/tracker etc). Consider these decisions to be configuration of the algorithm  $f$ , and a policy  $\pi$  that emits this configuration. We wish to learn a controller that governs the execution policy  $\pi^*$ , which is triggered periodically to decide the runtime configuration. In this work, we assume that all the other components (detector, tracker etc) have been trained already and we solely learn the controller.
We formulate learning the runtime decision as a deep contextual bandit problem, where a sequence of independent trials are conducted and in every trial the controller chooses a configuration (i.e. action)  $a$  from the set of possible decisions  $\mathbb{D}$  along with the provided context  $z$ . After a trial the controller receives a reward  $r$  as feedback for the chosen configuration and updates the execution policy. As we will see in Section 3.3, intrinsic and extrinsic factors affect policy  $\pi$ . Thus we define context  $z$  encapsulating these factors which is provided to the controller. We learn a  $Q$ -function  $Q_{\theta}(z,a)$ , for every action  $a$  given context  $z$ , where  $Q$ -values define the policy  $\pi$  to configure  $f$ .
3.3 Intrinsic and Extrinsic Context Guide Runtime Decisions
Numerous factors inform runtime decisions in real-time perception. Figure 3 depicts configurations for a predefined perception system. Randomly picking a "good" configuration (marked by red ellipse) can result in a sub-optimal decision, causing a drop of at least  $3\mathrm{mAP}$  in comparison to the "optimal" configuration. Optimal decision shifts with hardware change, on Argoverse-HD using a P40 GPU, the best scale turns out to be 600 instead of 900 on a 1080Ti GPU [6]. Selection of algorithm components (say, the detector) is itself a decision with trade-offs [22]. It should be noted that while recent models [10] satisfy the streaming perception constraint, they are optimized for specific hardware [13]. Detection models exhibit different kinds of errors [37, 38], and changing the scale affects their performance. Lastly, offline and real-time perception performance are not correlated (Figure 4).
(a)  $R$  for sequences  $A$  and  $B$ .
(b)  $R_{\text{fixed\_adv}}$  for sequences  $A$  and  $B$ .
Figure 6: Understanding  $R_{fixed\_adv}$ . In many video sequences, performance measure (single frame loss  $L$ ) is high for all configurations (e.g. all objects are large, highly visible), such as Sequence A. Compared to sequences (like B) where performance is lower and more heavily dependent on detector type and scale (e.g. objects are small or occluded).  $R_{fixed\_adv}$  corrects this bias.
Intrinsic factors like image content influence decisions – static configurations are sub-optimal. E.g., We can optimize scale at runtime [2] based on video content to improve throughput while maintaining accuracy. Extrinsic factors introduce additional challenges. Optimal decisions change with the stochastic nature of the external process, such as resource contention induced by other running processes. Figure 5 shows change in latency with contention levels and this relationship is non-linear. Other such extrinsic factors include network conditions and power considerations. Extrinsic context are non-differentiable, thus we do not use an end-to-end network to learn the overall context  $z$ .
We concatenate outputs of the following functions to form the overall context to the controller,
1. Switchability Classifier: Models exhibit different error characteristics, motivating the need to switch models at runtime [1]. Switchability classifier guides model switching at runtime.
2. Adaptive Scale Regressor: AdaScale [2] predicts the optimum scale an image should be inferred at, such that single frame latency is minimized. We use their regressor as an intrinsic context.
3. Contention Sensor: Process Contention affects model latency. We employ the GPU contention generator and sensor from [4], simulating discrete contention levels.
4. Scene aggregates: We also compute frame-level aggregates: (i) Confidence Aggregates: We compute the mean and the standard deviation of the confidence value associated with the predictions. The intuition is if the average confidence is low, which might indicate execution of an "expensive" configuration. (ii) Category Aggregates: We count of instances of each category in the predictions. This might capture rare objects, which may require additional resources to be localized. (iii) Object Size Aggregates: We categorize and aggregate predictions as small, medium and large detections (like COCO thresholds [39]). This is useful as small objects are considered difficult to detect.
3.4 Reward Jointly Optimizes Latency And Accuracy
Traditionally, output  $\hat{y}_i$  is emitted for observation  $O_i$  at time  $\hat{t}_i (> t_i)$ , and the reward [3, 21] is defined as
$$
R _ {t r a d} = \phi (y _ {i}, \hat {y} _ {i}) + \lambda \xi (t _ {i}, \hat {t} _ {i})
$$
considering accuracy via  $\phi$  and latency via  $\xi$  separately. However, hyperparameter  $\lambda$  for this additive reward is globally inconsistent (i.e. changes depending on intrinsic/extrinsic context) which doesn't work for large datasets. In our experiments we observed that controller collapsed to predict a single configuration when rewards of this form were provided (depending on  $\lambda$ ), and did not learn.
Now, consider the streaming perception setup described in Section 3.1. Deriving from this paradigm, our rewards optimize accuracy and latency simultaneously. Intuitively, reward  $R$  compares the model output with a ground truth stream of the actual world state. Assume algorithm  $f$  is governed by policy  $\pi$  receives observation  $O_{x}$ , controller performs action  $a_{x}$  at time  $t_{x}$  to configure  $f$  and another
Algorithm 1: Obtaining Observations From Streaming Perception System
Initialize policy  $\pi$    
Set change system configuration probability  $p$ $\tau = 0$    
for  $e = 0,1,\dots ,E - 1$  do Reset simulator;   
for every sequence do Initialize empty stream replay buffer S;  $i = 1$  while streaming do // System executes on streaming data value  $=$  Draw from Bernoulli(p); if value  $= =$  true then Observe  $x_{i}$  from the stream;  $z_{\tau} =$  context  $(x_{i})$  .  $a_{\tau} = \pi (z_{\tau})$  . Change system config. using  $a_{\tau}$  .  $t_{a\tau} =$  time-now(); Store  $(z_{\tau},a_{\tau},t_{a\tau})$  in stream replay buffer  $S$  ..  $\tau = \tau +1$  .  $i = i + 1$ $\pi =$  train_controller  $(\pi ,S)$
Algorithm 2: Training the Controller
Function train_controller( $\pi$ : Policy, S: Stream Replay Buffer): Unpack  $K$  length sequence  $\{(z^1,a^1,t^1)\dots (z^K,a^K,t^K)\}$  from stream replay buffer  $S$ ; for  $n = 1,\dots ,K$  do  $\begin{array}{r}r^n = R(t^n,t^{n + 1});\\ \mathrm{Store}(z^n,a^n,r^n)\mathrm{in~controller's~replay~buffer}\\ B;\\ \pi = update(\pi); \end{array}$  Update Policy Function update( $\pi$ : Policy): // Optimize the  $Q$  function Sample  $N$  tuples  $\{(z^1,a^1,r^1),\dots ,(z^N,a^N,r^N)\}$  uniformly from buffer  $B$ ; for  $n = 1,\dots ,N$  do // Set the targets  $y^{n} = r^{n}$  // Calculate the loss for the batch  $\mathcal{L} = \frac{1}{N}\sum_{n = 1}^{N}\frac{1}{|D|}\sum_{d\in a^n}[y^n -Q_\theta (z^n,d)]^2$  Use optimizer to optimize  $\theta$  to minimize  $\mathcal{L}$ ;
observation  $O_y$  with action  $a_y$  is taken at  $t_y$  ( $t_y > t_x$ ). We propose a reward,
$$
R \left(t _ {x}, t _ {y}\right) = L \left(\left\{y _ {t _ {k}}, \hat {y} _ {\varphi \left(t _ {k}\right)} \right\} _ {k = x} ^ {y}\right) \tag {1}
$$
where  $L$  is an arbitrary single frame loss. Intuitively, the reward for action  $a_{x}$  is streaming accuracy obtained for a stream sequence segment until a new action  $a_{y}$  is taken in the future and  $\varphi$  ensures only real-time results are accounted for. Reward  $R(t_{x}, t_{y})$  ensures learned controller implicitly considers real-time constraints and accuracy of the decision  $a_{x}$ , not assuming any properties of the algorithm  $f$  or loss  $L$ . Thus, it can be applied to other perception tasks.
While [6] defined  $L_{\text{streaming}} = L(\{y_{t_k}, \hat{y}_{\varphi(t_k)}\}_{k=0}^N)$  and corresponds to  $R(0, T)$  (Loss over N observations from a video stream of length T),  $L_{\text{streaming}}$  was employed to characterize performance of  $f$  on the sensor stream. Directly employing it as reward made training the controller sample-inefficient, and did not converge before we exhausted our training budget (of 10 epochs).
While reward proposed in Equation 1 is learnable, it is biased towards the inherent characteristics of the video sequence. Consider two video sequences  $A$  and  $B$ , Sequence  $A$  with one large object that can be easily detected would have much higher average rewards than Sequence  $B$  with many occluding and small objects. To normalize for this factor, we consider a fixed policy  $\pi_{fixed}$ , i.e., a fixed configuration of algorithm  $f$  and propose,
$$
R _ {f i x e d \_ a d v} \left(t _ {x}, t _ {y}\right) = R _ {\pi} \left(t _ {x}, t _ {y}\right) - R _ {\pi_ {f i x e d}} \left(t _ {x}, t _ {y}\right) \tag {2}
$$
where  $R_{\pi_{fixed}}$  is the reward obtained by executing algorithm  $f$  following a fixed policy. To implement  $R_{fixed\_adv}$ ,  $f$  with fixed policy  $\pi_{fixed}$  are executed and prefetch the predictions and the timestamps are stored. During training, we compute  $R_{\pi}$  and load  $\pi_{fixed}$  outputs to simulate  $R_{\pi_{fixed}}$ .
This eliminates the bias in the initial proposed reward, and provides the advantage of learned policy over fixed policy (See Figure 6). In Section 4.1, we show any fixed policy configuration  $(\pi_{fixed})$  suffices and the policy learned is resilient to variations of this selection. We refer to the reward in Equation 1 as  $R_{1}$  and Equation 2 as  $R_{2}$  in our experiments for simplicity.
3.5 Controller Design Decisions
Our controller design decisions (such as employing deep contextual bandits) are motivated by severe real-time constraints. For e.g., we considered individual frames as independent trials while learning our controller, i.e. we compute context from a single frame to incur minimal overhead (amortized overhead of around  $1\%$ ). Considering sequences as controller's input incurred significant overheads and impeded performance.
Table 1: Improvements on a predefined system. Chanakya outperforms competing execution policies for a predefined system. All the execution policies operate on top of a perception system employing the same components: Faster R-CNN and Kalman Filter.
Approach	sAP	sAP50	sAP75
1. Streamer (s=900) [6] (Static Policy)	18.2	35.3	16.8
2. Streamer (s=600) [6] (Static Policy)	20.4	35.6	20.8
3. Streamer (s=600, np=300) (Static-Expert Policy)	20.8	36.0	20.9
4. Streamer (s=600, np=500) (Static-Expert Policy)	20.9	35.9	20.9
5. Streamer + AdaScale [2] (Dynamic-Traditional Policy)	13.4	23.1	13.8
6. Streamer + AdaScale + Our Scheduler (Dynamic-Trad. Policy)	13.8	23.4	14.3
7. Chanakya (s1, np1, R=R1)	21.0	36.8	21.2
8. Chanakya (s1, np1, R=R2, πfixed=(s=480, np=300))	21.3	37.3	21.1
9. Offline Upper Bound (s=600, Latency Ignored)	24.3	38.9	26.1
Combinatorially Increasing Configurations. Consider decision space  $\mathbb{D} = \{D_1, D_2, \dots, D_m\}$ , with dimensionality  $M = |\mathbb{D}|$ , where each decision dimension  $D_i$  corresponds to a configuration parameter of the algorithm  $f$  discretized as  $d_k$ . Thus, an action  $a$  has  $M$  decision dimensions and each discrete sub-action has a fixed number of choices. For example, if we consider 2 decision dimensions,  $\mathbb{D} = \{D_{scale}, D_{model}\}$ , the potential configurations would be  $D_{scale} = \{720, 600, 480, 360, 240\}$ ,  $D_{model} = \{yolo, fcos, frcnn\}$ . Using conventional discrete-action algorithms,  $\prod_{d \in \mathbb{D}} |d|$  possible actions need to be considered. Efficiently exploring such large action spaces is difficult, rendering naive discrete-action algorithms like [3, 21] intractable.