DySR: Adaptive Super-Resolution via Algorithm and System Co-design
Authors: Syed Zawad
Cheng Li
Zhewei Yao
Elton Zheng
Yuxiong He
Feng Yan
Venue: ICLR 2023 poster (2023)
Keywords: super-resolution
quality of service
inference
deep learning
systems

Abstract: Super resolution (SR) is a promising approach for improving the quality of low resolution steaming services on mobile devices.
On mobile devices, the available computing and memory resources change dynamically depending on other running applications.
Due to the high computation and memory demands of SR models, it is essential to adapt the model according to available resources to harvest the best possible model performance while maintaining quality of service (QoS), such as meeting a minimum framerate and avoiding interruptions.  Nevertheless, there is no SR model or machine learning system that supports adaptive SR, and enabling adaptive SR model on mobile devices is challenging because adapting model can cause significant framerate drop or even service interruption. To address this challenge, we take an algorithm and system co-design approach and propose DySR that maintains QoS while maximizing the model performance.  During the training stage, DySR employs an adaption-aware one-shot Neural Architecture Search to produce sub-graphs that share kernel operation weights for low model adaption overhead while striking a balance between performance and framerate. During the inference stage, an incremental model adaption method is developed for further reducing the model adaption overhead. We evaluate on a diverse set of hardware and datasets to show that DySR can generate models close to the Pareto frontier while maintaining a steady framerate throughput with a memory footprint of around 40\% less compared to baseline methods.

Published as a conference paper at ICLR 2023
DySR: ADAPTIVE SUPER-RESOLUTION VIA ALGO- RITHM AND SYSTEM CO-DESIGN
Syed Zawad' Cheng Li", Zhewei Yao², Elton Zheng", Yuxiong He?, Feng Yan 'University of Nevada, Reno, "Microsoft Research, 'University of Houston szawad(nevada. unr. {chengli1, zheweiyao, elton. zheng, puxhe)@miccrosffe EyanSecentral. uh. edu
ABSTRACT
Super resolution (SR) is a promising approach for improving the quality of low resolution streaming services mobile devices. On mobile devices, the avail- able computing and memory resources change dynamically depending on other running applications. Due the high computation and memory demands of SR models, it is essential to adapt the model according to available resources to har- vest the best possible model performance while maintaining quality of service (QoS), such as meeting minimum frame rate and avoiding interruptions. Never- theless, there is no SR model or machine learning system that supports adaptive SR, and enabling adaptive SR model on mobile devices is challenging because adapting model can cause significant frame rate drop or even service interrup- tion. To address this challenge, take algorithm and system co-design ap- proach and propose a Dynamic Super Resolution framework called DySR that maintains QoS while maximizing the model performance. During the training stage, DySR employs an adaptation-aww produce sub- -ob-graphs that share kernel operation weights for low model adaptation overhead while striking a balance between performance and frame rate. During the inference stage, an incremental model adaptation method is developed for fur- ther reducing the model adaptation overhead. We evaluate on a diverse set of hardware and datasets show that DySR can generate models close to the Pareto frontier while maintaining a steady frame rate throughput with memory footprint of around 40% less compared the assembled baseline methods.
one-shot Neural Architecture Search to
INTRODUCTION
Deep super-resolution (SR) has been widely used in applications such as medical imaging (Li et al. (202T), satellite imaging (Shermeyer & Van Etten (2019)), and image restoration (Qiu al. 2019) )
SR has attracted lots of attentions in recent years due to the surging demands in mobile services such
video conference, content sharing, and video streaming, where it helps provide high-resolution visual content from low-resolution data source (Zhang et al. (2020); Li et al. (2020: 2021)) ) SR models are resource demanding (Li al. (2021): Lu & Hu (2022) and need to meet Quality of Service (QoS) standards to provide good user experience in visual services. Examples of QoS including meeting minimum frame rate and avoiding interruptions so that users perceive smooth motions. This, however, is challenging for mobile devices where computing and memory resources are limited and the availability which also depends on other running applications. To meet QoS for different mobile devices, existing works develop models for specific devices (Liu al. (2021b): Lee et al. (2019) Ayazoglu (202)) or use Neural Architecture Search (NAS) (Chu et (202 Guo al. (2020)
et al. (2021) to generate multiple serdware-tailoree models. However, none of these approaches considers the fluctuating resource environment of mobile devices and often leads to poor QoS. One potential way achieve good QoS is to dynamically adapt the model based on available resources. The challenges are two folds. First, how to design an adaptive model. Second, how to enable model adaptation live inference system.
Work done part as intern with the Microsoft DeepSpeed team.

---

Published as a conference paper at ICLR 2023
To enable adaptive model, employ NAS to generate set of models with different sizes so that the most profitable model is used under each resource availability situation to ensure a steady frame rate while maximizing the model performance. Unfortunately, none of existing machine learning frameworks supports live model adaptation. To enable model adaptation in actual system, we explore two ideas. The first idea is use an assemble method to keep all models loaded in the system at all times to avoid model switching overhead. However, such a method results in significantly larger memory footprint, which is unsuitable for mobile devices. The second idea is load single model at time, but the the model switching overhead is high as it interrupts the steaming for 1-3 seconds each time switches models, leading to even worse QoS.
To achieve low resource consumption while minimizing the model switching overhead, we propose DySR | an algorithm and system co-design approach for adaptive SR. To keep a small memory footprint and minimize adaptation overhead, DySR employs an adaptation-aware one-shot NAS ap- proach, where a large meta-graph trained in one-shot such that sub-graphs share kernel operation weights while exploring the best tradeoff's between performance and trames-per-second (FPS). Dur- ing inference, the meta-graph is fully loaded in the memory and operations are dynamically adapted according to the real-time resource availability in an incremental manner, i.e., only affected opera- tions are swapped or rerouted. Since we do not need to load new models from the hard disk, there is no data transfer overhead.
We evaluate DySR against baselines across a wide variety of hardware (from powerful GPUs to low- end mobile processors) using image and video SR datasets (e.g.. Urban (Huang al. (2015) and Vimeo90k (Xue et al. (2019)). Results show that DySR can generate models close to the Pareto frontier of the performance vs. FPS tradeoff's while maintaining steady frame rate throughput with low memory footprint (40% less compared to ensemble method).
2 RELATED WORKS
SR. DDng et al. 2014)) is among the first works that employs deep learning models for super-
resolution. Since then deeper and more complex models such as (Soh et al. (2019); Nazeri et al.
(2019) were proposed for better performance. Generative Adversarial Networks (GANs) (Creswell
cct al. (2018); Wang et al. (2019); Ahn et al. (2018) Wang et al. (2018a)) and its variations (Prajapati eet al. (2021) D: Guo et al. (2020); Shahsavari et al. (2021)) have been shown to be highly effective in tackling this task. Attention mechanisms were introduced to SR as well (Zhao et al. (2020); Mei et al. I) Chen et al. (2021)) Methods such as network pruning, knowledge distillation, and quantization have been applied to reduce computational overhead of existing SR deep learning models (Jiang al. (202) /Zhang al. (202Tb); [Hong al. (2022); Wang et al. (202T)) However, all the above efforts focus on building a single model for each hardware and do not consider the dynamic resource environment in mobile devices, and thus fall short in meeting streaming QoS for mobile devices.
NAS. Earlier neural architecture search methods rely on Reinforcement Learning (RL) Zoop & Le (2016)) and evolutionary algorithms (Lu et al.] (2018); van Wyk & Bosman (019)) for archi- tecture engineering. However, these methods are extremely resource demanding and often require thousands of GPU hours. Later works such as (Wen et al. 20 Jin et al.] (2019) introduce per- formance prediction, shared weight training, and proxy training to speed up the architecture engi- neering process. DARTS (Liu et al. (2018) and its followups (Chen et al. (2019); adopt a differentiable architecture search paradigm. The once-for-all work (Cai et al. (2019)) pro- poses the idea of generating a single model for multiple hardware deployments though pruning and model swapping is needed for each deployment. One-shot NAS (Bender et al. (018)) and its vari- ations (Huang & Chu (202I): Zhang et al. (202 Zhao et al.
y Wu et al. (219))
(202)) can generate models with few search iterations and have been explored for SR (Chu et al. (2021): : Guo et al. (2020); Liu et al. (2021a); Zhan et al. (202T)) but existing works only focus designing single model and do not consider QoS for streaming on mobile devices. In Section we compare DySR with existing SR models. The results show that our model achieves Pareto optimal performance while meeting QoS.
3 MOTIVATIONS AND CHALLENGES On mobile devices, SR tasks are often running along with other applications, e.g., users often watch streaming videos while browsing websites or doing online shopping; making video calls while play-
2

---

Published as a conference paper at ICLR 2023
100
40
40
40
75
30
30
50
20
20
EL 20
Utll
25
10
LL
10
10
0
10
20
10
20
10
20
10
20
Time (s)
Time (s)
Time (s)
Time (s)
(a) Utilization Trace models respectively due to being designed specifically for the corresponding hardware.
(b) Snapdragon 855
(c) Intel i5-560M
(d) 1080Ti
Figure 1: (a) Resource utilization cap over time for each device. (b-d) FPS drop due to reduced
resources. Uses FALSR-C, FALSR-B (Chu et al. (2021) I). and CARN (Ahn et al. (2018)
ing games or taking notes. Therefore, the available computing and memory resources for SR is constantly changing the already resource constraint mobile devices. We demonstrate the impact of changing available resources on frame rate for static models running on real mobile devices. We test set of state-of afrt SR models the mobile devices they were targeted for and manually limit the amount resource available over time, i.c., by limiting GPU power and adding background loads CPU. We workload trace show how the utilization changes over time, shown in Figure We observe that reducing the processing power available for the models over time results in significant wrames-per-secone (FPS) drop. This drop higher utilizations show that the QoS is impacted. Therefore, the slatc-of-the-an SR models do not perform as expected under practical cir- cumstances, and maintain QoS (i.e., minimum FPS), models need be adapted according the available resources. To fill this gap, however, is challenging. First, we need to design a set of mod- els that can achieve Pareto-optimal performance under different frame rates. However, no existing works can generate such a set of models. Second, we need to adapt models within milliseconds in real-time system meet QoS requirements. Unfortunately, no existing machine learning frame- works support real-time model adaptation.
One potential solution is to switch between a list of
40
40
40
Params (M)
21
FRS
20
20
FPS
20
14
7
0
0
5
10
15
20
15 20 Time (s)
10
5 10 15
20
10
Time (s)
Time (s)
# of Models
(a) Snapdragon 855
(b) Intel i5-560M
(c) 1080Ti
(d) Avg. reload time
Figure 2: (a-c) FPS over time when models reloaded every 5 seconds.
(d) Total number of parameters memory against the number of static models loaded as one assembled set.
static models according to the available resources. We prototype this idea and perform experiments on different devices, as shown in Fig. [2] Here, we unload and reload static models as soon as the utilizations change at the intervals 5, 10, and 15 seconds, as shown in (a), (b), and (c) respectively. Across all devices, we observe an interruption of service during each model swapping period. As pointed out in (Liao et al.) (2019), such interruption is the time it takes to read the new model from storage and initialize all the parameters in memory. Pre-loading multiple static models as one is potential solution to avoid such interruption. However, the assembled model has significantly higher memory consumption, as shown in Fig. 2d which is not practical for resource constraint mobile devices.
4 ALGORITHM AND SYSTEM CO-DESIGN: DySR
To address the aforementioned challenges. in this section, we adopt an algorithm and system co- design approach and propose DySR. DySR employs adaptive-aware one-shot NAS create set of models that achieve Pareto-optimal performance under different frame rate with minimum adapta- tion overhead during runtime. DySR also introduces an incremental adaptation approach to further reduce the model adaptation overhead to meet QoS.
4.1 ADAPTATION-AWAL AWARE ONE-SHOT NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) allows creating multiple models in an automated fashion, and thus choose it for model generation. However, for SR. existing NAS methods only target for a finding a single model with the highest performance while ignoring the frame rate and model adaptation overhand. To provide QoS mobile devices, we design an adaptive-aware one-shot
3

---

Published as a conference paper at ICLR 2023
NAS method for generating set of models that achieves Pareto-optimal performance with frame rate constraint and low model adaptation overhead under different hardware devices and resource availability. Following (Liu et al. (2018); Bender et all (018)), we define the architecture search space as S, which is represented as a single Directed Acyelic Graph (DAG) containing all possible path and operation combinations, called the meta-graph. The main idea here is to train meta- graph's is sub-graph models with Pareto-optimal performance across a range of frame rates. The sub- graphs are all part of the same meta-graph, that the full meta--gaphh is deployed and the sub-graphs can be adapted in real time during inference to maintain QoS using an adaptation policy.
Adaptation-awan aawaaettio--ware Sub-graph Selection. The generation of adaptation- aware sub-graphs requires two unique criteria to be fulfilled during the NAS search phase to be effective. First, in order to reduce resource consumption, we need to reduce total memory of the meta-graph to be significantly less than an assembled model. Second, we need to minimize the sub-graph adaptation time to ensure uninterrupted service. order to keep memory consumption a minimum, we design our meta-graph space share operations between sub-graphs wherever possible (e.g. two sub-graphs requiring a 3x3x64 convolutional operation at the same layer will use the same meta-graph operation, and thus have intersecting execution paths). To keep the switching time between models low, need to reduce the number of execution path re-routings between sub -obbggraps to a minimum. Luckily, find that both of these properties are related based on the observation that the number of total operations teetagrrphh and the number of path re-routes are inversely proportional to the number of shared operations. In other words, the more operations that are shared among sub-graphs. the less number of redundant operations there are and so meta-graph memory size is reduced. At the same time, more shared operations also means more common execution paths between sub- sub-graphs and so less number of paths need to be changed while adapting.
We use this observation to develop the custom search policy to generate our adaptive models. Thus the objective of our sub-graph selection policy can be formally defined as the constrained joint optimization problem
P arg max Eval(A(a, D)) + p(a, A a) s.t. fps(a, D) 2 F
(I)
min
Here, P is the HR quality metric (i.e. PSNR SSIM), Eval(A(a, D)) is the PSNR value of SR output of sub-graph a of meta-graph A on dataset D, is the number of operations a has shared with all other sub-graph A a. fps is the trames-per-secom allowed under available resource p. Note that p can change over time and can be generalizable to any type of computational resource even though in our case we mainly demonstrate using the available processing power. The implementation details are described in detail below.
and Fmin the minimum FPS
Meta-graph
Skippable Layers
Conv
1x1x32
RCAN
1x1x3
Input
15
16
50
Depth25pace
FC
Multibranch
RCAB
WDSR
....
R
LR
input
Bicubic
HR oulput
Searchable Cells
Upscaling
Output
(b)
Figure 3: Meta- graph with (a) search space and (b) adaptive sub-graph cell architectures. Each cell search-able and all layers after 15 are skippable. Four types of upscaling layers can be searched.
Meta-Graph Design. The first step in designing a NAS framework is the Meta-graph. This needs to be diverse enough to generate a wide range of efficient candidate architectures with competitive performance with ffate-of-hhearts In our case, we focus on designing gueta-graph with variety of inference FPS with a good set of PSNR values, rather than developing single novel cell architecture that beats all otate-of-the-art SR models. We design our meta-graph as a cell-based architecture (as shown in Figure The input layer and the last output layer are kept constant, and the architecture between them consists of layers of cells followed by an upsampling layer. During the search phase, we sample paths within each searchable cell such that only one type of cell block is chosen at any one time. Existing works have proposed a wide variety of cell blocks which were demonstrably efficient and have many operations in common, and use the cells from CARN (Ahn et al. (2018), FALSR (Chu et al. (202T)). RCAB (Zhang et al. 2018)), AutoGAN (Fu et al. (2020)), WDSR Yu al. (018)), ESRGAN (Wang et al. (2018a))
them for our search space. Specifically,
4

---

Published as a conference paper at ICLR 2023
and MoreMNAS (Chu al. ) since they well-known, efficient, self-contained blocks. Another large benefit is that all these cell blocks share many of the same operation types, making it possible for them to share weights. The most commonly shared operations between them are the convolutional and dense layers. As such, we set their parameter search spaces as: Convolutional Filters- 1x1, 3x3, 5x5, Convolutional Channels 3, 16, 32, 64, Dense 64, 128, 256. Cell blocks for RCAB, AutoGAN, ESGRAN and FALSR also have attention and residual connections, which we can enable or disable as paths since we find that while their inclusion result in increased performance, it is not significant at the higher end models when considering the FPS increases.
Layers. The number of layers one of the most important factors when determining the overall FPS and is therefore very important search parameter. Additionally, different layer types perform distinctly with different layer numbers (Wang et al. (2018)); Ahn et al. (2018); Chu et al. (202T) ) Therefore, we set the range of our number of layers of cells from between 15 to 50 with inter- vals/groups of for every single cell type. Lastly, for our upscaling layer, we keep the choices be- tween four types- RCAN (Zhang al. (2018)), Deconvolutional (Dong et al. (2016)) Depth2Space (Bhardwaj et al. (222)) and Multi-branch bicubic (Behjati et al. (202I)) We have a total of number of possible cell types of 12 when including blocks with and without attention and residual networks and a total number of parameter combinations of 3 4 * 3 = 36, giving us a total number of 12 36 432 cell combinations. With possible layer combinations of 3 to 10, have a permutation formula 432P10 + 432p, + +322ps +... +..232p, times 4 for the number of possible upscaling layers, which gives us total of 5. 5.8e1023 possible networks.
Sampling Paths Given such large number of possible graphs, we need an efficient method for traversal. Fortunately, our hard constraint of Eq. for FPS can play an important part in reducing the space here. We first start with setting the number of layers to the minimum possible. This is tunable hyperparameter, and we find that below 15 convolutional layer blocks results in all sub-graph with low PSNR value. Therefore, we set it to 15 and start random uniform sampling models without replacement. We profile them on the desired hardware to ensure the FPS is under the Fiarx If not, it is permanently discarded. Once we have N₂ models, we move to the next layer limit and iterate until we have reached the last layer limit I (50 in our current setting) which gives us a total of N₁ * models. Note that for our scenario, need multiple models across wide FPS spectrum. In other words, we need multiple choices of models for the same dataset deployable for different hardware and levels of resource availability. Therefore, before starting the training we first determine suitable models by binning them. We select how many bins of FPS we need as B with each bin having a window size of 200ms, and we bin our profiled models within each bin. We then train and evaluate each model within each bin for 5 epochs and evaluate them which gives us a sufficient estimate of final model performance. We now need use an evaluation criteria rank No models per bin. For this method, we set a desired PSNR threshold for bin b decided by the Pareto frontier. For example, from figure g we see that models on 1080Ti with 20 FPS needs to have at least 34 PSNR to be competitive, so we select that as the threshold. Then we take all models above that threshold and rank them based on the number of intersecting operations with others ju from [q We iterate this process until all bins B have N₂, models. This way, we have a total of N₂ * B models for exploration. The parameters N₁. N₁ and B are very important for determining the search and model efficiency for framework and their choice is discussed later in Section [5] We throw out all other models which are not within the bins and redeploy the meta-graph with only the selected model operations. This reduces the number of models in the search space drastically and reduces weight sharing which further alleviates the co-sharing issue.
Training. After reducing the search space to N₂ B models, then train them till convergence. As mentioned above, we perform GAN training. Our predictor models are the searched models and for our discriminator we use the pre-trained VGG-16 model. We use the Adam optimizer with loss function for the generator defined as
Lyen le? |hhr G(Z )1 + (egggG((zg (tr) + (5e mean((((((m))))
(2)
where The and Tir are the HR and LR images respectively, G is the generator network, logg is the VGG loss (Wang al. (2018b) and D the discriminator.
4.2 ADAPTIVE SUB-GRAPHS
Standard implementations employ static graphs with fixed inputs and outputs pre-set between oper- ations and are not changed during forward passes. As result, traditionally designed meta-graphs
5

---

Published as a conference paper at ICLR 2023
cannot change operation sequences in real time. Instead, they have reroute the batches among different model inputs for inference.
For case, have single large model in the form of veca-graph. It contains sub-graph architectures which share operations. In order to switch between sub-graphs, we can change the input-output paths for the operations. We take advantage of the non-static graph property to imple- ment this. We first define the sub-graphs as a network with the nodes representing the operations and the input-output variables the end-point for paths between them. We store each of the meta-graph operations as node objects with its input and output variable references within. During inference, we can change the input/output variables one operation node to point to and from another node. This effectively decides the execution path of the input batch, thereby defining the sub-graph architecture itself. Since we have all the operations with their trained weights loaded in memory, we can call every available operation with our desired input values at any time. Thus, when we want to exe- cute specific sub-graph path, we start by first selecting the required operations from the list of all operation nodes. We then create the sub-graph architecture by routing the output variables (which are pointers to Tensors memory) of one operation to the input of the next desired operation by changing the variable references. We can do this by applying the equal operation of the output vari- able of one operation node to the imput variable the next operation node, which essentially links them together into direct execution path. Since variables at any time between forward passes without disrupting the computation results, we can re-route paths per-batch granularity. Thus, by changing the pointer variables of input-outputs, we can select sub-graphs architectures as per our requirements. This allows us to adapt between sub- sub-graphs in real-time without degrading QoS.
set the output pointer variables to the input
4.3 MODEL ADAPTATION POLICY
Algorithm Model Selection Policy
Meta-graph after training A", CPU/GPU utilization % p time c, maximum allowed FPS Fmau. empty profile table T,, Osset is the list of sub-graphs that meet the selection criteria in descending order.
Inputs:
for each a; in A" do for each Pj do
Ti, = Pro=Profile(ann end for end for
while True do
Pe Currrent for each a; in A do
Q'sel
if Ti 2.c55 I Fmaz then
Otse += end if
end for
Osel = (ort(aael) based on max PSNR
Switch to model [0]
end while return None
We design an adaptation policy decide the model adaptation strategy in real time for given re- source availability. This is done in two steps. First, we generate a profiling table which contains the frame rate of each model under different resource availability (c.g.. utilization) for device. We pro- file each model using 20 image frames and use the average latency to infer frame rate. The profiling cost no. of models * resource availability granularity. This step is one-time offline effort and thus the profilng cost is not major overhead. During the real-time SR task, we keep track of the resource availability system via resource monitoring tools. Once we observe a change in resource availability, we adapt the model by using the search criteria Oseet where fps(a < Fmoz and sort the candidates based on their performance. We use the adaptive sub- sub-graph mechanism to select the top performing sub-gras uub-graph and switch to it in real time maintain QoS. The algorithm constantly running while the application is active. It does not output any specific variable, but constantly controls the switch between sub-graphs (see Alg. for details).

---

Published as a conference paper at ICLR 2023
5
EVALUATION
Hardware
FLOPS
Freq.
Model # of Params.
Chkpt. Size
Snapdragon 855
899G
2.8 GHz
Assembled
11.3M
88MB
Intel 15-560M
388G
2.9 GHz
DySR
8.44M
67MB
Nvidia 1080Ti
11.3T
1.4 GHz
Nvidia A100
312T
1.3 GHz
Figure 5:
Model and hardware device specification comparison. Checkpoint Size is the size of the file on disk of the trained weights stored by Pytorch via its checkpointing mechanism
5.1 TRAINING SETUP
We implement DySR using Py Torch and perform the search and training using 4 A GPUs, taking 21 GPU days complete per During the deployment, perform quick profiling (c.g., few steps of forward passes) on the target device (e.g. Snapdragon 855) to measure its inference latency and create profiled database. We then change
adaptation based the expected inference la- lency from this profile. For training the searched models till convergence, we use Div2K training dataset with 64x64 patch size, batch size of 16 and learning rate of 0.0043, For the search param- eters, we use the values 15, 2 and 5 for N₁. N, and B respectively. For the video super-ress resolutono dataset, we train with Vimeo90k (Xue al. (2019)) The LR images and frames were generated by bi-cubic downsampling them by factor of 2 and 4. To generate the Pareto Optimality curve, we use the the models as shown in Figure 6a)
5.2
BASELINES AND PARAMETERS
38
Memory (MB)
30
BURST-5R FALSRA
Fareto Fronter
36
CARN $68S8
BFSRGAN DASR
20
34
10
CARN-ML
ByteESR
32
FALSI-B
Asmbld Static
10
15
20
30
17.5
20.0
22.5 25.0 FPS
27.5 30.0
32.5
# of Models
(a)
(b)
Figure 6: (a) State-of-the-art model's PSNR FPS. Used generate the Pareto curve determining the optimal trade-offs. Shown here for Div2K dataset run with 1080Ti for 2x scale. (b) Comparison
of RAM consumption for Assembled, Statie (CARN) and DySR with different numbers of sub- graphs generated by the N₂ and B parameters. It is calculated as the sum of memory used for both model and activation (i.e., intermediate results).
We now describe our choice of baselines and search parameters. Since no other papers (to our knowledge) address this problem in the SR scope, create our baseline model by combining all the models from Figure 6a (referred to as Assembled). It contains high-performing models over a wide FPS spectrum including the NTIRE 2022 Efficient Super-Resolution Challenge (Li al. (2022) winner ByteESR (Kong et al. (2022)). Our DySR should ideally generate similar models and so makes for fair comparison. As mentioned above, the parameters N,, and B are very important in determining the quality of the models. Together they determine the total number of sub-gram sub-graphs generated which turn determines the amount of memory taken up by the -raph. It also indirectly determines how much weight is shared by the sub-graphs (more sub-graphs means more chances of operations being shared). In Figure Gh) we show the memory consumed when executing DySR, Assembled and the single static CARN model after applying dynamic 8-bit quantization for all of them (Pagliari et al. (2018)) ). As we tune the N, and B, we get 5, 10, 15 and 20 models (no. of models N4 * B). We that even with 15 models, DySR has almost equal consumption to Assembled due to weight sharing among sub- aab-graphs. With models, DySR consumes around twice much memory compared to the single model, illustrating the memory efficiency of weight- shared NAS system. Since Assembled contains 10 models, we select 10 models for our DySR as well for fair comparison for the rest of the experiments. Table 4 shows that with equal number of models, our framework has less number of parameters and size on disk.
5.3 PARETO OPTIMALITY
We now test our searched and fully trained sub- bub-graphs against state-of-the-an the- models. Figure 7 shows the PSNR FPS performance of each of the 10 sub-networks in DySR for different datasets.

---

Published as a conference paper at ICLR 2023
38
36
36
PSNR
SotA
Ours
PSNR
PSNR
34
32
32
2--
39.
20
25
30
28
15
20
25
30
28
15
20
25
30
FPS
FPS
FPS
30
PSNR
28
28
PSNPA
28
26
24
22
22
15
25
15
15
20
25
FPS
FPS
FPS
(a) Div2k
(b) Urban100
(c) Vimeo90k
Figure 7: PSNR FPS for DySR generated sub-graphs for different datasets. Top upscaling. Bottom row 4X upscaling. Legend is applicable for all subfigures.
2X
The sub- sub-graphs are evaluated by passing the input through the execution path of the specific sub- graphs and their corresponding inference time measured. We observe here that across both im- age super-resolution and video super-resolution datasets for 2X and 4X upscaling, our sub-gras sub-graphs achieve Pareto Optimality for the PSNR vs. FPS tradeoff created by the state-of-the-art models across the full FPS spectrum, demonstrating the efficiency our search space.
38
PSNR
SolA
Ours
34
34
39
30
36
FPS
FPS
(a) 1080Ti
(b) A100
38
38
PSNR
34
390
30
15
20
25
12
16
20
24
FPS
FPS
(c) Intel 15-560M
(d) Snapdragon 855
Figure 8: Pareto Optimality for different hardware. Generated by running DySR once and profiling FPS 1080Ti to generate models between 15 and 30 FPS. While the FPS spectrum shifts across datasets, the Pareto Optimality is maintained. Legend is applicable for all subfigures.
We next evaluate our generated models against a diverse set of hardware, and the results are shown in Figure We use mobile CPU Snapdragon 855, a laptop CPU Intel i5-560M, a desktop grade GPU 1080Ti and a server-grade GPU 100 (Table 5). This figure shows the results for 2X scaling for the Div2k dataset (other datasets show similar results, we eliminate them for lack of space). We search our models on the 1080Ti to be above 15 FPS. Therefore, the model's FPS shift for different hardware but still have models above 15 FPS (as these hardware, the Pareto Optimality our generated sub-graphs are maintained, demonstrating that even with search of models. This property allows for model adaptivity since this allows the selection of models with any desired FPS for any hardware, and ensures close to the optimal HR output quality. Along with the ability to change models on-the-fly, DySR allows for optimal PSNR vs. FPS tradeoff' for resource adaptivity as well, as will be demonstrated next.
search criteria). We see that across all
can generate efficient models for inference for such wide variety
5.4 DYNAMIC RESOURCE ADAPTIVITY
To demonstrate the resource adaptivity of our sub-graphs and model selection policy, we use three traces to generate real world resource fluctuation scenarios. We use a bursty trace to simulate ex- treme changes in resource availability, stable to illustrate a relatively small variation in resource availability and random. Since resource adaptivity has rarely been discussed in the SR context, create two possible baselines. The static baseline contains a single model deployed at any one time from the list of state-of-the-art models used above for generating the Pareto Optimality curve. The
8

---

Published as a conference paper at ICLR 2023
Util (%)
100
75
100
75
Util (%)
100
75
50
50
Ut1
10
20
30
20
30
20
Time
Time (s)
Time (s)
30
30
30
15
Ours
15
- 15
Assembled
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
p
35.0
35.0
35.0
PSNR
PSNR
PSNR
32.5
32.5
32.5
30.8.0
0.5
1.0
30.0.z
0.0
0.5
1.0
30.8.0
0.5
1.0
p
p
(a) Bursty
(b) Random
(c) Stable
Figure 9: CDF comparison between DySR and baselines for FPS and PSNR based on bursty, random and stable trace.
Assembled baseline contains all the models from the state-of-the-ee art list deployed at the same time to represent the scenario of keeping models pre-loaded. Based on available resources, we use our real-time swapping policy to choose the best model for both our DySR well as the two baselines. FigureG the results of our experiment using the Div2k dataset on 1080Ti for 2x scaling. The top row shows the trace used, the middle row shows the CDF of the resulting FPS fluctuation over time, and the bottom row shows the the Cumulative Distribution Frequency (CDF) of the PSNR/FPS. It is the probability of the PSNR/FPS being less than a certain value any given time. Higher y-axis values at lower values means better performance.
Here we observe that across all traces, DySR achieves overall higher values of PSNR compared to the baselines. The FPS for Assembled and DySR are somewhat similar due to both having little to no reloading overhead. However, Assembled performs worse for PSNR due to having less granularity of models available across the full FPS spectrum. For example, between 20 to 25 FPS in Figure 7 for Div2K with 2X upscaling, we see that our framework generates four models but the Assembled only has two. Therefore, Assembled has less options and tends to choose models with lower PSNR order to be able to maintain its FPS. The Static models perform the worst due to having many interruption of services expected. For the PSNR values, Static has significant portion of its CDF distribution at and therefore performs much worse than the others, especially at the lower p val- ues. Our framework outperforms all baselines by having the highest PSNR probabilities. Another observation that the largest improvement in PSNR and FPS gained from DySR is for the bursty workload. This is mainly due to there being more model switches required here compared to the other two workloads. This amplifies the benefits derived from the model selection policy and gran- ularity and thus greater difference is observed. As the workloads grow more stable, the numbers vary less, illustrating that the more unstable system the more it should benefit from our solutions. Based on all these observations, we conclude that DySR outperforms both baselines and can address the resource adaptivity issue
CONCLUSION
In this paper, we focus on providing QoS for super resolution used in steaming services of mo- bile devices. We propose an algorithm and system co-design approach DySR, which employs an adaptive-aware one-shot NAS to generate models with Pareto-optimal performance, low switching overhead, and frame rate constraint that can be adapted in real-time via our low cost incremental adaptation method. DySR outperforms existing -fate-of-the-art models well as the simple semble method we explored by achieving the best performance while providing a steady frame rate throughput with memory footprint that is 40% less compared the 10-model Assembled baseline.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
We are grateful to our anonymous reviewers for their valuable comments and suggestions that sig- nificantly improved the paper. This work is sponsored by National Science Foundation CAREER- 2048044.
REFERENCES
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. Proceedings of the European conference on com- puter vision (ECCV), pp. 252-268, 2018.
Mustafa Ayazoglu. Extremely lightweight quantization robust real-time single-image super resolu- tion for mobile devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472-242224 2279, 2021.
Parichehr Behjati, Pau Rodriguez, Armin Mehri, Isabelle Hupont, Carles Fernandez Tena, and Jordi Gonzalez. Overnet: Lightweight multi-scale super-resolution with overscaling network. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2694- 2703, 2021.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference on Machine Learn- ing, pp. 550-559. PMLR, 2018.
Kartikeya Bhardwaj. Milos Milosavljevic, Liam O' N'Neil, Dibakar Gope, Ramon Matas, Alex Chalfin, Naveen Suda, Lingchuan Meng. and Danny Loh. Collapsible linear blocks for super-efficient super resolution. arXiv preprini arXiv: 100.::200300 2021.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment. In International Conference on Learning Representations, 2019.
Xin Chen, Lingxi Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1294-1303, 2019.
Yuantao Chen, Linwu Liu, Volachith Phonevilay, Ke Gu, Runlong Xia, Jingbo Xie, Qian Zhang, and Kai Yang. Image super-resolution reconstruction based feature map attention mechanism. Applied Intelligence, 51(7):4367-4380, 2021.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Multi-objective reinforced evolution in mobile neural architecture search. In European Conference on Computer Vision, pp. 99-113. Springer, 2020.
Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. In 2020 25th International Con- ference on Pattern Recognition (ICPR), pp. 59-64. IEEE, 2021.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35 (1):53-65, 2018.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. European conference on computer vision, pp. 184-199. Springer, 2014.
Chao Dong, Chen Change Loy. and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European conference on computer vision, pp. 391-407. Springer, 2016.
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan- distiller: Searching to compress generative adversarial networks. In International Con- ference Machine Learning, pp. 3292-3303. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Yong Guo, Yongsheng Luo, Zhenhao He, Jin Huang, and Jian Chen. Hierarchical neural architecture search for single image super- reperrresolutio. IEEE Signal Processing Letters, 27:1255 77:1225-1259, 2020.
Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Lee. Daq: Channel- wise distribution-aware quantization for deep image super-resolution networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2675-2684, 2022.
Han Huang, Li Shen, Chaoyang He, Weisheng Dong, Haozhi Huang, and Guangming Shi. Lightweight image super-resolution with hierarchical and differentiable neural architecture 2021.
search. arXiv preprint arXiv:2105.085:
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197-5206, 2015.
Sian-Yas Huang and Wei-Ta Chu. Ponas: Progressive one-shot neural architecture search for very efficient deployment. In 202/ International Joint Conference on Neural Networks (IJCNN) pp- 1-9. IEEE, 2021.
Xinrui Jiang, Nannan Wang, Jingwei Xin, Xiaobo Xia, Xi Yang, and Xinbo Gao. Learning lightweight super-resolution networks with weight pruning. Neural Newworks, 144:21-32, 2021.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search sys- tem. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1946-1956, 2019.
Fangyuan Kong, Mingxi Li, Songwei Liu, Ding Liu, Jingwen He, Yang Bai, Fangmin Chen, and Lean Fu. Residual local feature network for efficient super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 766-776, 2022.
Royson Lee, Stylianos I Venieris, Lukasz Dudziak, Sourav Bhattacharya, and Nicholas D Lane. Mobisr: Efficient on-device super-resolution through heterogeneous mobile processors. In The 25th Annual International Conference on Mobile Computing and Networking, pp. 1-16, 2019.
Xiaofang Li, Yirui Wu, Wen Zhang, Ruichao Wang, and Feng Hou. Deep learning methods in real- time image super-resolution: a survey. Journal of Real-Time Image Processing, 2020.
Y Li, Bruno Sixou, and F Peyrin. A review of the deep learning methods for medical images super resolution problems. IRBM, 4202):120-133, 2021.
Yawei Li, Kai Zhang, Radu Timofte, Lue Van Gool, Fangyuan Kong, Mingxi Li, songwei Liu, Zongcai Du, Ding Liu, Chenhui Zhou, et al. Ntire 2022 challenge efficient super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Put- tern Recognition, pp. 1062-1102, 2022.
Zhuofan Liao, Ruiming Zhang, Shiming He, Daojian Zeng, Jin Wang, and Hye-Jin Kim. Deep learning-based data storage for low latency in data center networks. IEEE Access, 7:26411- 26417, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference on Learning Representations, 2018.
Shaoli Liu, Chengjian Zheng, Kaidi Lu, Si Gao, Ning Wang, Bofei Wang, Diankai Zhang, Xiaofeng Zhang, and Tianyu Xu. Evsrnet: Efficient video super-resolution with neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2480-2485, 202
Xin Liu, Yuang Li, Josh Fromm, Yuntao Wang. Ziheng Jiang, Alex Mariakakis, and Shwetak Patel. Splitsr: An end-to-end approach super-resolution mobile devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(1):1-20, 2021b.
Enmin Lu and Xiaoxiao Hu. Image super-resolution via channel attention and spatial attention.
Applied Intelligence, 52(2):2260-221) 2268, 2022.
11

---

Published as a conference paper at ICLR 2023
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nega-net: A multi-objective genetic algorithm for neural architecture search. 2018.
Yiqun Mei, Yuchen Fan, and Yuqian Zhou. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3517-3526, 2021.
Kamyar Nazeri, Harrish Thasarathan, and Mehran Ebrahimi. Edge-informed single image super- resolution. In Proceedings the shops, pp. 0-0, 2019.
the IEEF/CVF International Conference on Computer Vision Work-
Daniele Jahier Pagliari, Enrico Macii, and Massimo Poncino. Dynamic bit-width reconfiguration for energy-efficient deep learning hardware. In Proceedings of the International Symposium on Low Power Electronics and Design, pp. 2018.
Kalpesh Prajapati, Vishal Chudasama, Heena Patel, Kishor Upla, Kiran Raja, Raghavendra Ra- machandra, and Christoph Busch. Direct unsupervised super-resolution using generative ad- versarial network (dus-gan) for real-world data. IEEE Transactions on Image Processing. 30: 8251-8264, 2021.
Yajun Qiu, Ruxin Wang, Dapeng Tao, and Jun Cheng. Embedded block residual network: A re- cursive restoration model for single-image super-resolution. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4180-4189, 2019.
Ali Shahsavari, Sima Ranjbari, and Toktam Khatibi. Proposing a novel cascade ensemble super reso- lution generative adversarial network (cesr-gan) method for the reconstruction of super-resolution skin lesion images. Informatics in Medicine Unlocked, 24:100628, 2021.
Jacob Shermeyer and Adam Van Etten. The effects of super-resolution on object detection perfor- mance in satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Jae Woong Soh, Gu Yong Park, Junho Jo, and Nam Ik Cho. Natural and realistic single image super-resolution with explicit natural manifold discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8122-88 8131, 2019.
Gerard Jacques van Wyk and Anna Sergeevna Bosman. Evolutionary neural architecture search for image restoration. In 2019 International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2019.
Han Wang, Wei Wu, Yang Su, Yongsheng Duan, and Pengze Wang. Image super-resolution us- ing improved generative adversarial network. 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 312-315. IEEE, 2019.
Hu Wang, Peng Chen, Bohan Zhuang, and Chunhua Shen. Fully quantized image super-resolution networks. Proceedings of the 29th ACM International Conference on Multimedia, pp. 639-647, 2021.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 0-0, 2018a.
Zheng Wang, Mang Ye, Fan Yang. Xiang Bai, and Shin ichi Satoh. Cascaded sr-gan for scale- adaptive low resolution person re-identification. In IJCAI, volume 1, pp. 4, 2018b.
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pp. 660- 676. Springer, 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742 2019.
12

---

Published as a conference paper at ICLR 2023
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. volume 127, pp. 1106-1125 Springer, 2019.
Jiahui Yu, Yuchen Fan, Jianchao Yang, Ning Xu, Zhaowen Wang, Xinchao Wang, and Thomas Huang. Wide activation for efficient and accurate image super-resolution. arXiv preprint arXiv: 1808188888,888, 2018.
Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, Wei Niu, Yushu Wu, Tianyun Zhang, Malith Jayaweera, David Kaeli, Bin Ren, et al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 4821 1821-4831, 2021.
Anlan Zhang, Chendong Wang, Xing Liu, Bo Han, and Feng Qian. Mobile volumetric video stream- ing enhanced by super resolution. In Proceedings of the 18th International Conference Mobile Systems, Applications, and Services, pp. 462-463, 2020.
Jianwei Zhang, Dong Li, Lituan Wang, and Lei Zhang. One-shot neural architecture search by dynamically pruning supernet in hierarchical order. International journal of neural systems, (07):2150029, 2021a.
Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. Proceedings of the European conference computer vision (ECCV), pp. 286-301, 2018.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Learning efficient image super-resolution net- works via structure-regularized pruning. In International Conference on Learning Representa- tions, 2021b.
Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. Efficient image super- resolution using pixel attention. In European Conference on Computer Vision, pp. 56-72. Springer, 2020.
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In International Conference Machine Learning, pp. 12707-12718. PMLR, 2021.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arX1v:1611.01578. 2016.
13

---

Published as a conference paper at ICLR 2023
A
APPENDIX
FALSR-C
AutoGAN
CARN
DySR-5
DySR-M
DySR-L
FALSR-C
AutoGAN
CARN
DySR-S
DySR-M
DySR-L
Figure 10: Visual Comparison Comparing DySR's HR output to other state-of-the-and models of similar sizes. For 4x upscaling, with samples from Set5 (top) and Urban100 (below)
A.1 VISUAL COMPARISON
We compare our DySRoutputs against other models in current literature with similar sizes, DySR- L,M.S represent models with 1.6. 0.74 and 0.38M parameters respectively. CARN al. (2018), AutoGAN et al. (2020) and FALSR-C'S ChCCCCCCChueee et al. (202T) have 1.5, 0.71 and 0.41M parameters. We observe that across both datasets, our HR outputs are quite similar to comparable state-of-the-art.
14


Reviews:
[{"review_id": "ZGuFp_aeoj", "reviewer": "Reviewer_9DkJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a systematic solution for image super-resolution tasks. In order to provide good image/video quality under limited resources on mobile devices, the proposed method employs adaptive Neural Architecture Search (NAS) technique to produce model adaptation with minimum overhead. The proposed method aims to dynamically switch SR models that consistently adapts to the varying computer environments so as to balance the visual quality of super-resolution and the limited resource on the mobile devices. Comprehensive experiments show that the proposed method is able to provide steady throughput. \n", "summary_of_the_review": "In summary, this paper is creating a new area in image super-resolution domain, which is quite practical and engineering-centric. I would like to suggest the authors to address the problems in the weaknesses mentioned above. ", "strength_and_weaknesses": "Strengths\n1. This paper has clearly pointed out the strong motivation of the work and the necessity of getting a ad-hoc switchable super-resolution system that can automatically adjust the model based on the resources available in the system. I do believe the proposed method is useful in practical situations on mobile devices, especially for those low-end devices or embedded systems.\n2. This paper has compared the proposed method with state of the arts baseline methods and outperform them in terms of the processing speed as well as the accuracy using Pareto curve to indicate the validity of performance. \n\nWeaknesses\n1.  In the algorithm, one of the most important input factors to decide sub-graph is the current available resources in the devices. However, i could not find the descriptions and explanations about how to to quantify the current resources, for example, you need to balance a bunch of parameters including memory, traffic, load on CPU /GPU , etc.. This is, I believe, the important starting point to decide whether your solution is applicable to mobile systems. \n2. In this experiment, the training step is conducted on desktop-level system on A100 graphic cards. However, it is not quite clear about the deployment pipeline. If the model needs to be trained off-the-shelf and deployed on a mobile system or it should be trained on the target system otherwise the input hardware parameters are not available beforehand.\n\n\n\n1. There are quite a number of typos and grammar errors:\n(1). Abstract: Line2 steaming services -> streaming services\n(2). Section 3 Motivation and Challenge: Line 8:  ..., see Figure 1 -> as shown in Figure 1. Two verbs in one sentence. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CIH0BbRtgYv", "reviewer": "Reviewer_bx3i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a model for adaptive super resolution on mobile devices. Their approach employs a one-shot neural architecture search to generate alternatives sharing weights. For inference, they introduce an incremental adaptation method. Their model aims to mantain a steady frame rate with a low memory footprint relative to baseline methods.", "summary_of_the_review": "The article provides an interesting idea for adaptatively defining a super-resolution operation during streaming. However, the authors could improve the presentation. Also, verifying the source code for this method wiould be interesting.\n\nThe authors have addressed the reviewers comments. I raised my evaluation to \"marginally above the acceptance threshold\". Thanks", "strength_and_weaknesses": "This study is the first paper addressing the problem of dynamically adapting a network architecture to available resources while providing constant throughput. In their approach, the results seem close to the Pareto frontier in the tradeoff between FPS and PSNR. \n\nPlease clarify how your 40%footprint reduction is achieved. The source code is not provided. The article requires some Editing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7xPVgpkB9rs", "reviewer": "Reviewer_bgMe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a DySR method that maintains QoS while maximizing the model performance. The proposed method is mainly based on NAS. Experimental results show the effect of the DySR. \n", "summary_of_the_review": "The experimental evaluations are not enough. ", "strength_and_weaknesses": "Strength \n\nThis paper aims to develops an image SR method on the QoS. The main goal is to propose an efficient method that can work well on the limited computing resources. The provided results demonstrate the effect of the proposed method to some extent. \n\nA new meta-graph design is developed. In addition, introducing a NAS for the SR is interesting. \n\nWeakness:\n\nMy major concern is experimental evaluation. \n\nThere are lots of methods that focus on developing efficient SR for mobile devices [e.g., NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results, CVPRW 2022]. The paper does not compare with these methods. It is not clear whether the proposed method performs better or not. \n\nIn addition, no visual comparison is provided. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "03fx1MweTs", "reviewer": "Reviewer_6CHN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for adapting a single-image super-resolution [SR] deep network to different computational environments while maintaining a real-time quality of service. The method used is to identify sub-networks (graphs) within a larger network (graph) that are optimized to solving the SR problem but with varying levels of computational requirements. At run-time the whole network is loaded into memory, and then the computational paths used can be varied on the fly as computational resources change. Comparison with prior methods show the overall network and the subgraph network are competitive in their results with prior work with similar computational costs.", "summary_of_the_review": "The concepts in the paper are of broader potential interest than super-resolution.  Within that narrow domain, the novel contribution is maintaining the minimum quality of service while adapting to computational resources.  Otherwise, the absolute performance is similar to existing methods at varying levels of computational complexity.  The method of identifying the relevant sub-graphs is of potential interest across other domains.  The paper is clearly written, and the comparisons with SOTA methods are appropriate.", "strength_and_weaknesses": "The main strength of the paper is enabling real-time switching between different networks to maintain a certain quality of service under different computational environments. The optimization process used appears to identify useful sub-graphs of the overall network that are competitive with prior work of similar computational complexity.\n\nThe results are competitive with state of the art methods, with the addition of being able to swap models on the fly as computational resources change.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster