ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Authors: Lufei Liu
Tor M. Aamodt
Venue: ICML 2025 poster (2025)
Keywords: Computer Graphics
Neural Network Inference
Training-Free Optimization
Feature Caching
Inference Acceleration
Primary area: applications->computer_vision
TL;DR: Caching layer outputs in network inferences for graphics rendering.

Abstract: Graphics rendering applications increasingly leverage neural networks in tasks such as denoising, supersampling, and frame extrapolation to improve image quality while maintaining frame rates.
The temporal coherence inherent in these tasks presents an opportunity to reuse intermediate results from previous frames and avoid redundant computations.
Recent work has shown that caching intermediate features to be reused in subsequent inferences is an effective method to reduce latency in diffusion models.
We extend this idea to real-time rendering and present ReFrame, which explores different caching policies to optimize trade-offs between quality and performance in rendering workloads.
ReFrame can be applied to a variety of encoder-decoder style networks commonly found in rendering pipelines.
Experimental results show that we achieve 1.4$\times$ speedup on average with negligible quality loss in three real-time rendering tasks.
Code available: https://ubc-aamodt-group.github.io/reframe-layer-caching/

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Lufei Liu
Tor M. Aamodt
Abstract
applies neural networks to upscale images to 4x resolution and 4x frame rate, effectively rendering only one of every 16 pixels through traditional methods.
Graphics rendering applications increasingly leverage neural networks in tasks such as denois- ing, supersampling. and frame extrapolation to improve image quality while maintaining frame rates. The temporal coherence inherent in these tasks presents an opportunity reuse interme- diate results from previous frames and avoid re- dundant computations. Recent work has shown that caching intermediate features to be reused in subsequent inferences is an effective method to reduce latency in diffusion models. We extend this idea real-time rendering and present Re- Frame, which explores different caching policies to optimize trade-off's between quality and per- formance in rendering workloads. ReFrame can be applied to a variety of encoder-decoder style networks commonly found in rendering pipelines. Experimental results show that we achieve 1.4x speedup on average with negligible quality loss in three real-time rendering tasks. Code available: https: //bc aamodt- group. gi github. ht//eefraae--ayr--caeee
Real-time rendering applications are siency-seesitive es- tablishing an important trade-off between quality and per- formance. Neural network inferences now make up sig- nificant portion of the rendering pipeline, especially with ray tracing accelerated using dedicated hardware like the NVIDIA RT Core. Any reduction of the inference latency can be used to allocate more resources to traditional render- ing and improve the final image quality or directly improve the frame rate, both of which benefit user experience.
One approach to accelerate inference latency is by identi- fying sources of redundancy in the computation (LeCun et al., 1989). Rendering workloads naturally exhibit tempo- ral coherence, where the content of consecutive frames highly correlated (Scherzer et al., 2012). The inter-frame similarities particularly high with fast frame rates, which frame content changes slowly even with fast camera movements. These similarities also persist deep within the neural networks, exhibiting redundancy in the computation of intermediate layers. Figure I shows a sequence of 20 frames an Unreal Engine demo scene to visually illustrate similarities between frames and slow scene changes.
1. Introduction
A similar observation recently made in the context of diffusion models, where intermediate layer outputs were cached and reused in subsequent inferences of the model by injecting cached results in concatenations with skip connec- tions instead of computing deeper layers (Ma et al., 2024; Wimbauer et al., 2024). This eaching scheme was shown to be effective in reducing the number of FLOPs required for inference, leading faster inferences and lower computa- tional costs. We explore adapting this caching technique to neural networks used in real-time rendering applications.
Real-time rendering an important application that enables high-quality visualizations and interactivity across many industries such gaming, virtual reality, design, and health- care. In recent years, neural networks have become a critical component in real-time rendering workloads. To achieve high-quality renderings within a limited frame budget, only a small fraction of the image is rendered through tradi- tional techniques like ray tracing. The majority of the pixels are actually generated using upsampling methods such as frame extrapolation and supersampling neural networks. The newest version of NVIDIA's Deep Learning Super Sam- pling (DLSS) technology, DLSS 4.0 (Lin & Burnes, 2025),
Although the caching technique was originally designed for U-Nets in diffusion models (Ho et al., 2020; Rombach et al., 2022), we show that it can be applied to variety of encoder-decoder style networks commonly found in render- ing pipelines. Adapting the technique for rendering work- loads comes with several challenges. Unlike diffusion mod- els that rely on long sequences of inference iterations to produce a single output, every inference iteration must pro- duce a high-quality frame real-time rendering. Therefore,
Department of ECE, University of British Columbia, Vancouver, Canada. Correspondence to: Lufei Liu <liu- l student.ubc.cat cb>, Tor M. Aamodt <aamodt wew.ddd@ce.uuc.c>.....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Frame 1
Frame 4
Frame
vs. Frame 0
vs. Frame t-1
FUP
0.0
10
15
20
Frame (1)
Frame 10
Frame
Frame
Frame 19
Figure 1. Input similarity in a rendering workload showing every three frames. Upper right plot shows the similarity compared to Frame 0 (Frame t vs. Frame 0) and similarity between consecutive frames (Frame vs. Frame 1) measured as FLIP scores.
approximate features in the cache used by ReFrame must trade image quality for improved frame rates in way that does not impede the user experience. Furthermore, diffu- sion models rely on fixed number of forward passes with consistent pattern of change to guide the cache (Ma et al., 2024), while ReFrame must address the challenge of high variations and unpredictability in rendering workloads.
et al., 2022) and Event-NN (Dutson et al., 2022) exploit this similarity by processing only the differences (deltas) be- tween frames of video from still camera, effectively spar- sifying the computation. MotionDeltaCNN (Parger et al., 2023) extends this idea to accommodate camera motion, and Eventful Transformers (Dutson et al., 2023) adapts it to transformer architectures. However, despite high FLOPs reduction, these techniques struggle to generate significant speedups because the underlying hardware is not optimized for the scattered nature of the eliminated computations, which is a problem that ReFrame avoids.
Our contributions are as follows:
We extend existing methods of caching intermediate network features beyond U-Nets in diffusion models to a variety of encoder-decoder style networks that include feature concatenation.
Hardware accelerators like Cambricon-D (Kong et al., 2024) and Diffy (Mahmoud et al., 2018) have also been proposed to exploit frame similarity on dedicated hardware, but are not adopted in commodity hardware. A more practical ap- proach to exploiting frame similarity is through caching intermediate layer outputs of neural networks (Ma et al., 2024; Wimbauer et al., 2024; Xu et al., 2018), but has only been applied to diffusion models over long sequences of inferences. Caching approaches have also been used for neural rendering (Steiner et al., 2024), which is typically used for view synthesis rather than rendering synthetic vir- tual worlds. Inter-frame similarity is a well-known source of redundancy and ReFrame introduces technique that produces real benefits in real-tine rendering workloads.
We explore new caching configurations to optimize trade-offs between quality and inference time that take advantage of patterns in real-time rendering workloads.
We demonstrate empirically Ti GPU that using ReFrame reduces the inference time of three different real-time rendering networks (frame extrapolation, supersampling, and image composition) by 1.4x on average with negligible quality loss.
NVIDIA RTX 2080
2. Related Work
This section reviews related research exploiting frame similarity in rendering and video-processing, trading output accuracy for faster inference in neural networks, and the use of neural networks in real-time rendering applications.
2.2. Approximate Inference
Not all applications require exact inference results, and ap- proximate inference can be used to reduce the computational cost of neural networks (Sun et al., 2024). Approximate inference can be achieved through quantization (Hubara et al., 2016), pruning (Han et al., 2016), or low-rank fac- torization (Denton et al., 2014) and demonstrate potential of trading accuracy for performance. These approaches re- cover the accuracy loss by training the network to adapt
2.1. Inter-frame Similarity
Temporal coherence in rendering workloads and in video processing has long been explored to reduce the compu- tational cost of processing frames (Walter et al., 1999; Scherzer al., 2012; Arnau et al., 2013). DeltaCNN (Parger
2

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
the approximate inference, while ReFrame is training-free. However, ReFrame is orthogonal to other approximate infer- ence techniques and can potentially benefit from combined approach with these methods.
tional blocks (X*) that process outputs from previous blocks
concatenated (concat) with skip connections:
0 (concaall( (X==1
2.3. Real-time Rendering Networks
concal (X- (...) (x..),,x(xxx(
In the real-time rendering pipeline, low resolution image is usually generated through ray tracing, an algorithm capa- ble of creating highly photo-realistic images. However, ray tracing is an expensive and very noisy process, and cannot independently produce a clear image within the frame bud- get to meet 60-90 frames per second. Therefore, noisy low resolution ray-traced image is passed through series of neural networks for denoising (Choi et al., 2024; Scardigli et al., 2024; Chen et al., 2023), frame extrapolation (Guo et al., 2021; Wu et al., 2023; Yang et al., 2024), and neural supersampling or super resolution (Zhang et al., 2024; He et al., 2024; Zhong et al., 2023; Yang et al., 2023; Mercier et al., 2023) to generate the final high resolution image. Augmented and virtual reality (AR/VR) applications further require additional network inferences for image composi- tion (Watson et al., 2023; Yu et al., 2023). DLSS (Lin & Burnes, 2025) and Intel XeSS (Chowdhury et al., 2022) are popular examples of a neural networks that are widely used in the gaming industry for real-time rendering, but require high-end GPUs and can still benefit from dedicating more resources better input image.
x²(I))) (I)
In U-Net++, the network branches off to multiple skip connections at each downsampling block, creating a nested U-Net structure, as shown in Figure 2 (b). U-Net++ blocks are denoted as X where i is the depth of the block and indexes a convolution along the skip connection where E [0, 1] nz.
3.1. Layer Caching
In a U-Net, the final output from the network is the result of the last block (X"),, which takes in a concatenation of results from previous blocks as its input. At the first frame (at time 1), we compute the network inference end-to-end as normal, and store the inputs to the last block of the network in a cache (Ct). Specifically, we cache the portion that is produced by the previous encoder block, X=-1, which con- tains high-level features extracted from the input image that often change slowly between frames. Then, for subsequent frames, we reuse the cached results (C₂) from the main branch and concatenate them with the skip connection from the first block, X This process allows to compute only X and X", skipping all of the deeper layers in the network and replacing them with cached inferences, as described in DeepCache (Ma et al., 2024):
Real-time rendering networks commonly take advantage of U-Net (Ronneberger et al., 2015) and U-Net++ (Zhou et al., 2018) architectures for their ability to capture spatial information at different scales, which is useful for extracting features from images and managing different image reso- lutions. Although these networks are designed to fast, they still contribute a significant portion of the rendering pipeline latency, often exceeding capabilities of lower-end devices (Bhuyan et al., 2024; Yang et al., 2023). ReFrame aims to reduce FLOPs in these workloads to support better quality rendering on mobile devices.
C = X" (comcat(X* (...) (xx (x-"))x
(2)
0 X" (C,,
In a U-Net++ architecture, the final block (X...) concate- nates more skip connections than only the first block like in a U-Net. Each skip connections that feeds into a concatena- tion can be a potential candidate for caching. One possible way to extend the caching scheme used in DeepCache to U-Net++ is to cache the results from each branch, except the skip connection from the first block (X%.0):
3. ReFrame
We focus ReFrame networks that include encoder- decoder architecture with skip connections, such as U-Net or U-Net++, due to their ubiquity in state-of-the-ar real- time rendering pipelines. These networks are characterized by a series of convolutional layers that downsample the in- put image (/) to low-dimensional feature space, followed by series of convolutional layers that upsample the fea- ture space back to the original image resolution. At each downsampling block, the network branches off to skip con- nection that concatenates to the corresponding upsampling block, illustrated in Figure 2 (a). For a U-Net with n blocks, the output of the network (O) can be described as convolu-
0 = Xcomacat 1000-1 ; ,...
,CP.m-2....
(T. (4)
However, to better pass new information through the net- work, we instead cache the branches that lead into all the higher-level blocks (X..)). Figure shows the U-Net and U- Net++ architectures with layer caching for the full inference frame and subsequent cached frames.
3

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Skipped black
Save to cache
Upsample
Downsample
Skip connection
Frame t
Frame t +
Frame
out
out
X
X
X
X
X
X
X
X
x²
X
X
X
C
X
C
C
X
X
X
X
x²
X=
X
X²
X
X
X
X
(a) U-Net
(b) U-Net++
Figure 2. Diagrams of U-Net (a) and U-Net++ (b) architectures with layer caching. In frame the network computes the full inference end-to-end, saving intermediate outputs into a cache. In subsequent frames + 1, the network reuses the cached results instead of re-computing the intermediate layers.
In some cases, ReFrame can be applied on networks beyond U-Net and U-Net++ architectures if the network comprises concatenations of intermediate layer outputs. For example, some networks concatenate results from several feature traction blocks before passing them to the final block. In these cases, we can cache the results of any slow-changee channingggng feature extraction blocks to reuse in subsequent frames.
Frame
Frame
Frame
Level
Level Level 2
3.2. Cache Policies
Although the caching scheme is simple, the choice of which layers to cache and how to manage the cache can have a significant impact on the performance of the network. First, our caching scheme can be applied at any depth in the U- Net and U-Net+ architectures instead just the last block. Figure shows the features different depths (levels) over consecutive frames in rendering workload. Caching the last block is the most effective for reducing the number of FLOPs required, but also suffers the highest quality degra- dation because the changes in intermediate features (light green regions in Figure 3) at all levels are ignored, We find empirically that the quality degradation from caching at the last block is outweighed by the performance gains, which demonstrate in Section 4.5. More importantly, plore different policies to manage how the cache is refreshed. If the cache is updated every frame, the network will always compute the full inference, which defeats the purpose of the caching scheme. However, the cache is never updated, the ligh-level features in the cache will become stale and no longer be representative of the input image. Ideally, the cache should be refreshed just before its contents noticeably affect the output quality.
Very
Very Different
Simliar
Figure 3. Feature similarity between consecutive frames in ren- dering workload relative to Frame Levels 1-3 correspond to the depth of the U-Net architecture (i.c. outputs of X X'. and X
this policy simple and effective, it fails consider the vari- ations in camera motion and scene complexity of real-time rendering workloads. For example, fast-mmvvig camera or scene with many moving objects can quickly cause the cache to become stale. This behavior does not exist in diffusion models because the diffusion process known to be smooth over time and not prone to sudden changes. DeepCache also proposed a non-linear cache policy that dis- tributes the cache refreshes fit temporal similarity patterns observed in diffusion models. The proposed distribution prioritizes refreshing the cache at the beginning and end of the image generation process where the most significant changes occur, but there is no such pattern for sal- ren- dering workloads. We evaluate both policies in Section 4.5 and compare it to adaptive policies for ReFrame.
3.2.1. EVERY-N
DeepCache (Ma et al., 2024) proposes a simple policy to update the cache every N frames, where N can be a hyperpa- rameter tuned balance quality and performance. Although
4

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
3.2.2. FRAME DELTAS
the performance gains. A different approach for mitigating quality degradation is to allocate the saved inference time to rendering a better input image, such increasing the samples per pixel count in ray tracing or slightly increas- ing the resolution of the input image. Depending on the application, this approach can lead to higher quality images overall or higher frame rates with negligible quality loss. We explore this trade-off between allocating resources to traditional rendering versus network inferences empirically in Section 4.2.2
We propose an adaptive policy that updates the cache only when the imput image changes significantly from the cached frame. Instead of storing only Ct, we also store 4t which we use to compute the symmetric mean absolute percentage error (SMAPE) between the input image and the cached image, SMAPE(I,I) If the SMAPE exceeds a threshold T, we refresh the cache by computing the full inference and saving the new Cp and Iv. This approach incurs additional computation for the SMAPE and additional memory to store 4 but better adapts to the unpredictable nature of real-time rendering workloads, prevents sudden quality drops, and still reduces inference time overall as shown in Section 4.2. A secondary benefit is that the computed frame deltas can exploit techniques such as DeltaCNN (Parger et al., 2022) to further reduce the number of FLOPs required for inference.
4. Experiment Evaluation
We evaluate ReFrame on three common real-time rendering tasks: frame extrapolation (FB), supersampling (SS), and image composition (IC). In rendering pipeline, all of these tasks may be required, in addition to tasks including ray tracing, denoising, adaptive sampling, and other processing steps. For each task, we execute end-to-end inference on a sequence of frames, comparing the quality and performance of ReFrame against the baseline network. We modify cach network in PyTorch to add our caching scheme, follow- ing implementation details in Appendix A.2. We choose frame deltas with both high (Delta H) and low (Delta I.) sensitivities as the default policy to refresh the cache for all experiments except the ablation study in Section 4.5, where we compare different cache policies.
By applying an adaptive policy, we increase average com- putation savings during periods of low camera motion or low scene complexity because the cache can be refreshed less frequently than in the every-N approach. Also, this approach avoids sudden quality drops than can occur with every-N policies when content changes are not aligned with the fixed refresh intervals. This benefit is particularly impor- tant in real-time rendering workloads because the end user may not notice smooth quality differences, but will notice sudden changes in quality (Thakolsri et al., 201 1). We con- sider both high and low sensitivity thresholds (Delta and Delta L) for the SMAPE in our experiments, which can be tuned to balance quality and performance.
4.1. Evaluation Metries
We evaluate the performance of ReFrame using a combi- nation of image quality metrics and latency metrics. For image quality, we use FLIP (Andersson et al., 2020), learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018), structural similarity (SSIM), and peak signal to noise ratio (PSNR), which are all full-reference image quality as- sessments (FR-IQA) commonly used to evaluate rendering quality. The FLIP score is specifically designed to evalu- ate rendered images and measures the perceptual similarity between two images, which provides the most insight to how our technique affects the end-user experience. We use the results of the baseline network without ReFrame as the reference image for all metrics unless otherwise specified.
3.2.3. MOTION VECTOR THRESHOLD
Another possible policy for real-time rendering is the use of motion vectors to identify when the input image changes sig- nificantly. Motion vectors are already rendered as part of the G-buffers in many rendering pipelines and are often required as an input for supersampling and frame extrapolation tasks. Therefore, we propose to set a maximum motion threshold T, then check whether the average motion exceeds at each frame, and refresh the cache if the threshold is exceeded. This policy results in similar advantages of adaptability using frame deltas, but does not require any additional stor- age when motion vectors are already computed. However, motion is only one source of change in the input image, and frame deltas present a more comprehensive indicator for cache refreshes.
For latency, we compare relative speedup in inference la- tency measured on NVIDIA RTX 2080 Ti GPU, We also measure the number of floating point operations (FLOPs) required for inference within each encoder-decoder network (Enc-Dec FLOPS) to quantify the computational savings of our caching scheme. We also report the proportion of frames that use the cached features reduce computation (Skipped Frames). The remaining frames require full in- ference due to cache refreshes or sudden movements that invalidate the cache, and these frames might benefit from orthogonal techniques like DeltaCNN (Parger et al., 2022).
3.3. Quality Trade-offs
Our caching scheme introduces an additional trade-off be- tween quality and performance, where the quality of the inference result is diminished due to the reuse of cached features. Caching at deeper layers in the network can help recover some of the quality degradation, but also reduces
5

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Skipped Frames
Eliminated Enc-Dec FLOPs
Policy
Workload
Scene
Speedup
FLIP +
SSIM
PSNR f
LPIPS
MSE
Delta
Temple
50%
1.42
0.0169
0.994
41.41
0.0066
1.63
Cyberpunk
30%
16%
1.10
0.0207
0.981
31.08
0.0103
3.66
Asian Village
35%
19%
1.24
0.0241
0.985
33.19
0.0248
3.89
SS
Sun Temple
40%
29%
1.30
0.0490
0.970
53.13
0.0263
17.62
IC
Garden Chair
13%
6%
1.05
0.0006
1.000
47.93
0.0001
0.07
Delta L
FE
Sun Temple
80%
43%
1.72
0.0325
0.984
37.35
0,0198
4.71
Cyberpunk
60%
32%
1.49
0.0345
0.968
31.29
0.0181
6.07
Asian Village
60%
32%
1.55
0.0462
0.969
32.44
0.0508
7.81
SS
Sun Temple
80%
57%
1.85
0.1180
0.930
34.52
0.0725
41.40
IC
Garden Chair
79%
34%
1.20
0.0127
0.991
33.34
0.0109
1.79
Table 1. Performance and image quality results of our selected workloads (FE frame extrapolation, SS supersampling, IC image composition). We compare two sensitivity settings in the delta cache policy: high sensitivity (Delta_H) and low sensitivity (.)lta_)), detailed in Appendix A.3. Results relative to the baseline network without caching. Ene-Dee FI.OPs do not include additional inference operations outside the encoder-decoder network.
Baseline
ReFrame
FLIP
Figure 4. Frame extrapolation results with and without caching on the Asian Village scene using ExtraNet. The FLIP error map is shown on the right (pink indicates highly noticeable regions differences).
4.2. Performance and Quality Results
frames (ex. Te+0.5). The final result displayed to the user alternates between rendered frames and extrapolated frames, which doubles the effective frame rate. We evaluate our caching scheme ExtraNet (Guo et al., 2021), which uses U-Net architecture with skip connections ponent of the network. We generate our test set with the Un- real Engine build published by the authors and free scenes from the online Unreal Engine Marketplace (Fab), detailed in Appendix A.1. Although the scenes we evaluate are not identical the original dataset, which is not freely available, we are only interested in the relative performance between the baseline and cached networks this does not affect evaluation. Figure shows visual example of our frame extrapolation results with and without caching, including the FLIP error map.
Table 1 summarizes our key results, showing that ReFrame consistently reduces FLOPs and inference latency across all tasks. Our FLIP score results highlight that perceptual quality loss is negligible. All scores are generally below the acceptable losses observed in other neural rendering sys- tems, which report scores between 0.05 and 0.28 in their fi- nal results (Müller et al., 2021; Li et al., 2022; Vaidyanathan et al., 2023). Even with higher MSE in some workloads, user experience should not be affected, as demonstrated by low perceptual loss scores such as FLIP and LPIPS.
major
ReFrame applies 72% of inferences on average with low sensitivity setting (Delta.L) resulting in 40% reduction in FLOPs and 1.6x speedup in inference latency. With high sensitivity setting (Delta H), our caching scheme still reduces 19% FLOPs, with negligible quality loss. Even though the cache is only applied to a subset of inferences on portion of the network, we still achieve performance gains in all cases.
4.2.2. NEURAL SUPERSAMPH MPLLING
Neural supersampling is similar to image super resolu- tion and upsamples low resolution (LR) rendering in- put to high resolution (HR) image output, but supersam- pling is adapted to reflect aliasing properties in computer graphics (Xiao et al.. 2020). We evaluate ReFrame on a Fourier-based super resolution network (FBSR) for super- sampling (Zhang et al., 2024). The network concatenates
4.2. 1. FRAME EXTRAPOLATION
Frame extrapolation takes historical and current frames as in- puts (ex. Ig-2, I and I and predicts future in betweee
6

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Ground Truth
Base (d pereamplnn))
Refreme
Refrome (Detb
ReFrame supersamsing
Figure 5. Supersampling results on the Sun Temple scene using FBSR. Ground truth image is rendered at high resolution. Delta and Delta show results with caching enabled. 3x supersampling shows quality improvements from using a larger input image. Zoomed-in regions show detailed quality differences between each configuration.
Scaling Factor
Toral Time
Time
et al., 2023) test network, which estimates the depth of a real image and composites virtual rendering into the real scene with correct occlusions. Implicit Depth uses a U-
Config
FLIP L.
SSIM
PSNR
LPIPS
Savings
1x
Baseline
1,49
0.00
0.398
0.761
21.305
0.356
Delia.f
1.122
0.34
0.399
0.761
21.239
0.357
Deta
0.761
0.367
Net+ architecture for depth estimation and we evaluate the network using sample data released by the authors. Figure 6 shows a visual example our image composition workload, with the FLIP error map comparing the final composite results with and without caching.
3x
Baseline
1.506
-0.05
0.395
0.765
21.353
0.359
Della.
0.766
0.360
Deto,J
0.922
0.53
0.399
0.761
21.174
0.371
Table Supersampling results the Sun Temple scene. The scaling factor is the ratio of 1080p output resolution to 270p and 360p input resolutions. Time savings is measured relative the baseline inference time for 4x scaling. x scaling is slower than 4x scaling without ReFrame, Image quality is measured against ground truth 1080p rendered images.
Figure 7 shows the inference time of the decoder blocks in Implicit Depth over time. By applying our caching scheme, the average inference time is significantly reduced. Spikes in inference time are cache refreshes, which occur when the input image changes significantly from the cached frame and only add a small latency overhead. Using our Delta L policy, we can store the cache for longer periods of time when the input image is stable, as shown in the first 20 frames. Then, when the input image shows more changes, the cache is refreshed more often to prevent quality degradation.
three feature extraction blocks that feed into a feature fu- sion block. Similar to caching downsampling blocks in the U-Net architecture, we cache the temporal and HR feature extraction blocks in FBSR. We use the same Unreal Engine build and the Sun Temple scene from test set to eval- uate caching for FBSR. Figure 5 shows visual example of our supersampling results with different cache policies compared to the baseline and ground truth HR images.
4.3. Overheads
The performance overheads of storing and loading cached tensors are negligible compared to the performance gains from reducing computation. There is a small lag during cache refreshes to compute frame deltas as observed in Figure/ for Delta L., but this additional latency falls in the range of noise for the full inference time and can be eliminated with simpler policies such as every-N. Also, the memory overhead of our caching scheme is small, storing only one or few tensors in total for the cache and one copy of the network input for frame deltas. Details of the cache memory consumption are included in Appendix A.6.
We also evaluate the trade-off between quality and perfor- mance by comparing between a small LR input using no cache and larger LR input with cache. Table shows that by using cache, we improve the final image quality by allocating the saved inference time to rendering a larger low resolution image. Although inference time for 3x upscaling is slower (-0.05s) due to the larger input size, by applying our proposed caching scheme, we can exploit the quality benefits of the larger input without suffering performance penalties. For example, xx upscaling with a cache achieves higher FLIP score with the Delta H policy and results in 0.19s latency savings over 20 frames.
4.4. Comparison to DeltaCNN
DeltaCNN (Parger et al., 2022) is another method that ex- ploits frame similarity by processing only the differences (deltas) between frames. We evaluate DeltaCNN on the frame extrapolation task using the Asian Village scene from our test set and the public DeltaCNN library for Py PoTch. Ta-
4.2.3. IMAGE COMPOSITION
Image composition is useful for augmented reality (AR) applications that require multiple images to be combined into single frame. We choose Implicit Depth (Watson

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Real
Render
Composite
Composits w/ ReFrame
FLIP
Figure 6. Implicit Depth results combining Real and Render into Composite. The FLIP error map is shown on the right comparing results with and without using cache, where pink indicates highly noticeable regions of differences.
Inference Time (ms)
No Cache
N-5
Delta
Level
Level 2
Level I
U-Net++
U-Net++
FLIP
0.023
0.026
0.035
0.013
0.014
SSIM 1
0.982
0,979
0,968
0,991
0.989
20
PSNR
34.04
33.37
31.29
33.34
31.03
LPIPS
0.010
0,012
0.018
0.011
0,013
10
MSE
3.457
4.087
6.073
1.787
1,918
FLOPs
71.8%
62.9%
46.2%
56.6%
30.1%
0
20
40
60
80
100
Table 4. Results of ablation study ExtraNet and Implicit Depth comparing different cache levels and configurations. Image quality results are averaged over all frames in test set and FLOPs are relative to the baseline encoder-decoder network.
Frame
Figure 7. Inference time of decoder blocks in Implicit Depth over 100 frames. Both N-5 and Delta_L policies show signifieant reduc- tions in inference time compared to No Cache. Delta.L spreads cache refreshes more strategically enforce output quality.
ment versus using ReFrame only. Although the combined method reduces FLOPs, the hardware is not optimized for the scattered nature the eliminated computations, causing high overheads and low performance gains.
Prame
FLOPs
FLOPs
FLIP
SSIM 1
PSNR
LPIPS
MSE
1- Delta
44 124
48.561
0.130
0.854
28.413
0.148
37.502
2- Detma
46.449
0.184
0.826
0.200
4.5. Ablation Study
Table 3. Results of ExtraNet with DeltaCNN on Asian Village scene. Full inference is computed in the first frame (0- Ref), then only deltas for the subsequent frames (1 Delta and 2 - Delta). FLOPs are reported using the DeltaCNN performance metrics manager. Actual FLOPs are higher than theoretical savings due to the overhead of the DeltaCNN framework and tiling strategy.
We choose to cache the last block of the U-Net architec- ture for all experiments because it is the most effective for reducing the number of FLOPs required. We explore caching deeper blocks in the U-Net architecture, which re- duces fewer FLOPS but also suffers less quality degradation. Similarly, the U-Net++ architecture can be cached in vari- ety of ways, with each concatenation in the network offering potential caching point. We choose to cache at the last block (Config B), but also consider eaching all additional U- Net++ blocks while only evaluating the foundational U-Net blocks during cached inference (Config A). More details the configurations are provided in Appendix A.3. Ta- ble 4 presents the results of our ablation study, showing that all options result low quality degradation. Figure 8 visually shows that errors are nearly unnoticeable and in- distinguishable between levels according to the FLIP error map. Therefore, we choose the option that produces the most significant performance gains.
ble reports our results and show that our caching scheme is more effective in reducing FLOPs and improving inference latency than DeltaCNN. Techniques such as DeltaCNN re- quire very high sparsity achieve realized speedups unless dedicated sparse accelerator is used, which is not commer- cially available. Furthermore, DeltaCNN is designed for video processing, which typically take a single RGB frame as input, while our rendering workloads require several con- catenated G-buffers and time-warped frames as input. Un- fortunately, these input configurations are less suitable for computing deltas between frames, worsened by the channel- wise masking strategy used in DeltaCNN.
We also evaluate other cache policies on the frame extrap- olation task, including updating the cache every N = 2 frames (N-2), every = 5 frames (N-5), based on motion vectors, and based on the non-linear cache policy proposed by DeepCache (Ma al., 2024) (Non-Linear N-5). Table
Our caching scheme can be combined with DeltaCNN to fur- ther reduce FLOPs in the non-cached blocks of the network. We evaluate the combined method on the frame extrapola- tion task but find that there is little performance improve-
8

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Level
Level 2
Level
N-5
Delta
No Update
Output MAPE (%)
10
FLIP (0.029)
FLIP 10.018)
FLP (0.014)
0
100
120
140
160
180
200
Frame
Figure 8. Ablation study results for ExtraNet U-Net architecture. Config A caches all U-Net++ blocks, while Config B caches only the last block.
Figure 9. Mean absolute percentage error (MAPE) of outputs from Implicit Depth using N-5, Delta L, and No Update cache policies after 100-frame warmup. Both N-5 and Delta policies show low errors, but Delta_ avoids larger error spikes.
Motion Vector
Non- LineL
Pulicy
D
Deltu
N-5
Refresh
will require periodic refreshes. Since ReFrame focuses on the post-processing stage that enhances the rendered images, additional image quality improvements from the neural networks be included in a best-effort manner. average still reduces
FLIP 1.
0.035
0,021
0,040
0,025
0.030
0,040
SSIM
0.968
0
0,963
0.976
0.971
0.961
PSNR
31.29
LPIPS
0.018
0.010
0.020
0.013
0.016
0.022
MSE 1
6,073
3.655
7,046
4.653
5.471
7,259
Furthermore, reducing computation energy consumption, which is an especially important target in mobile-class devices such as VR headsets.
Table Results of ablation study ExtraNet comparing different cache refresh policy settings on a sequence of 10 frames. Re- fresh frames full inference computations to refresh the cache. Delta_L results in better image quality with fewer cache refreshes.
6. Conclusion
In this work, we propose novel cache mechanism for neural networks that exploits the temporal coherence of real- time rendering workloads. Our method caches intermediate layer outputs in encoder-decoder style networks and reuses them in subsequent frames to reduce the number of FLOPs required for inference. Our proposed approach limited to networks with skip connections and concatenations between intermediate layers, but this is a common architecture in real-time rendering pipelines. We explore different cache refresh policies and demonstrate reduced inference latency on three real-time rendering tasks. As future work, we plan to investigate more sophisticated cache policies with dynamic sensitivity settings and applying our method to other network architectures.
shows that the frame delta policy (Delta L) outperforms the other policies in balancing performance and quality. Motion vectors are also very effective, but are not reliably avail- able in all rendering workloads. Figure 9 confirms that the every-N policy suffers from spikes of low quality when movements do not match the cache refresh rate, while the maximum error from Delta L consistently lower. Figure also shows that the Delta L policy allocates cache refreshes more strategically according the input behavior, leading to results with more consistent quality.
5. Discussion and Limitations
ReFrame is designed for U-Net and encoder- decoder style networks only and does not support other architectures such transformers. Although there are many transformer- models used in real-time rendering applications, such DLSS 4.0 (Lin & Burnes, 2025), we believe U-Net-like convolutional networks are still heavily employed and more feasible to execute on lower-end devices. ReFrame partic- ularly useful on these lower-end devices where trading slight quality loss for lateney reduction valuable. The trade-off between quality and latency will differ for each network, depending on the cache configuration and the portion of the overall network architecture that can be skipped.
A Acknowledgements
We thank the reviewers for their feedback and the authors of our test workloads for their open-source implementations. This work was supported by the Natural Sciences and Engi- neering Research Council Canada (NSERC), Tor Aamodt recently served consultant for IBM and Cisco.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
Although ReFrame can effectively reduce inference time in many frames to lower average latency, our technique cannot maintain a consistently faster frame rate since the cache

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
References
Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffmmnn coding, Proc. Int'l Conf. on Learning Representations (ICLR), 2016.
Andersson, P., Nilsson, J., Akenine-Möller. T., Oskarsson, M., Aström, K., and Fairchild, M. D. FLIP: A difference evaluator for alternating images. Proc. Int'l I Conf. on Com- puter Graphics and Interactive Techniques (SIGGRAPH), 3(2), 2020.
He, R., Zhou, S., Sun, Y., Cheng, R., Tan, W., and Yan, В. Low-latency space-time supersampling for real-time rendering. In Proc. Conf. Artificial Intelligence (AAAI), volume 38, 2024.
Arnau, J.-M., Parcerisa, J.-M., and Xekalakis, P. Parallel frame rendering: Trading responsiveness for energy on a mobile gpu. In Proc. IEEE/ACM Conf. on Par. Arch. and Comp. Tech. (PACT), pp. 83-92, 2013.
Ho, J., Jain, A., and Abbeel, Denoising diffusion prob- abilistic models. Proc. Conf. and Workshop on Neural Information Processing Systems (NeurIPS), 33, 2020.
Barman, N., Zadtootaghaj, S., Schmidt, S., Martini, M. G., and Möller, S. VonningVieeoETT; a dataset for gaming video streaming applications. In Workshop on Network and Systems Support for Games (NetGames). IEEE, 2018.
Hubara, I., Courbariaux, M., Soudry, D., Yaniv,, R., and
Bengio, Y. Binarized neural networks. Proc. Conf. and Workshop on Neural Information Processing Systems (NeurIPS), 29, 2016.
Bhuyan, S., Ying, Z., Kandemir, M. T., Gowda, M., and Das, C. R. GteeStreamSR: Enabling neural-augmented game streaming on commodity mobile platforms. In Proc. IEEE/ACM Int'l Symp. on Computer Architecture (ISCA), 2024.
Kong. W., Hao, Y., Guo, Q., Zhao, Y., Song, X., Li, X., Zou, M., Du, Z.., Zhang, R., Liu, C., et al. Cambricon-DD Full- network differential acceleration for diffusion models. In Proc. IEEE/ACM Int'l Symp. on Computer Architecture (ISCA), 2024.
Chen, Y., Lu, Y., Zhang, X., and Xie, N. Interactive neural cascade denoising for 1-spp Monte Carlo images. The Visual Computer, 39(8), 2023.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 1989.
Choi, H., Hong. S., Ha, I., Kang, N., and Moon, B. Online neural denoising with cross-regression for interactive ren- dering. ACM Transactions on Graphics (TOG), 43(6), 2024.
Li, T., Slavcheva, M., Zollhoefer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S., Goesele, M., Newcombe, R., et al. Neural 3D video synthesis from multi-view video. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2022.
Chowdhury, H., Kawiak, R. R., de Boer, G. F., and Xavier, L. Intel XeSS-an AI based super sampling solution for real-time rendering. In Game Developers Conference (GDC), volume 1, pp. 7, 2022.
Lin, H. and Burnes, A. NVIDIA DLSS 4 Introduces Multi Frame Generation Enhancements For All DLSS Tech- nologies, 2025. URL htwp:: www. nvi di com /en http:///eeoorreeee///essseeeeeee e inneeratioon-aiinnvatttte
fram
Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R. Exploiting linear structure within convo- lutional networks for efficient evaluation. Proc. Conf. and Workshop on Neural Information Processing Systems (NeurIPS), 27, 2014.
Ma, X., Fang, G., and Wang, X. DeepCache: Accelerat- ing diffusion models for free. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024.
Mahmoud, M., Siu, K., and Moshovos, A. Diffy: A déjà vu-free differential deep neural network accelerator. In Proc. IEEE/ACM Symp. on Microarch. (MICRO), 2018.
Dutson, M., Li, Y., and Gupta, M. Event neural networks. In European Conf. on Computer Vision (ECCV), 2022.
Dutson, M., Li, Y., and Gupta, M. Eventful transformers:
Mercier, A., Erasmus, R., Savani, Y., Dhingra, M., Porikli, F., and Berger, G. Efficient neural supersampling on a novel gaming dataset. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 296-306, 2023.
leveraging temporal redundancy in vision transformers. In Proc. IEEE Int'l Conf. on Computer Vision (ICCV), 2023.
Guo, J., Fu, X., Lin, L., Ma, H., Guo, Y., Liu, S., and Yan, L.-Q. ExtraNet: real-time extrapolated rendering for low- latency temporal supersampling. ACM Transactions on Graphics (TOG), 40(6), 2021.
Müller, T., Rousselle, F., Novák, J., and Keller, A. Real-time neural radiance caching for path tracing. ACM Transac- tions on Graphics (TOG), 40(4), 2021.
10

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Parger, M., Tang, C., Twigg, C. D., Keskin, C., Wang, R., and Steinberger, M. DeltaCNN: End-to-end CNN in- ference of sparse frame differences in videos. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Walter, B., Drettakis, G., and Parker, S. Interactive rendering using the render cache. In Rendering Techniques' 99: Proc. of the Eurographics Workshop, pp. 19-30. Springer, 1999.
Watson, J., Sayed, M., Qureshi, Z., Brostow, G. J., Vicente, S., Mac Aodha, O., and Firman, M. Virtual occlusions through implicit depth. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), pp. 9053-9064, 2023.
Parger, M., Tang, C., Neff, T., Twigg, C. D., Keskin, C., Wang, R., and Steinberger, M. MotionDeltaCNN: Sparse CNN inference of frame differences in moving camera videos with spherical buffers and padded convolutions. In Proc. IEEE Int'l Conf. on Computer Vision (ICCV), 2023.
Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X., Hou, J., He, Z., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., et al. Cache me if you can: Accelerating diffusion models through block caching. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
Wu, S., Kim, S., Zeng, Z., Vembar, D., Jha. S., Kaplanyan, A., and Yan, L.-Q. ExtraSS: A framework for joint spatial super sampling and frame extrapolation.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In Medical image computing and computer-assisted assisted inter- vention (MICCAI), pp. 234-241, 2015.
Proc. Int I Conf. on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), 2023.
Sayed, M., Gibson, J., Watson, J., Prisacariu, V. Firman, M.,, and Godard, C. SimpleRecon: 3D reconstruction without 3D convolutions. In European Conf. on Computer Vision (ECCV), 2022.
Xiao, L., Nouri, S., Chapman, M., Fix, A., Lanman, D., and Kaplanyan, A. Neural supersampling for real-time rendering, ACM Transactions Graphics (TOG), 39(4), 2020.
Scardigli, A., Cavigelli, L., and Müller, L. K. RL-based stateful neural adaptive sampling and denoising for real- time path tracing. Proc. Conf. and Workshop Neural Information Processing Systems (NeurIPS), 36, 2024.
Xu, M., Zhu, M., Liu, Y., Lin, X., and Liu, X. Deepcache: Principled cache for mobile deep vision. In Proc. Ini' Conf. on mobile computing and networking (MobiCom), pp. 129-144, 2018
Scherzer, D., Yang, L., Mattausch, O., Nehab, D., Sander, P. V., Wimmer, M., and Eisemann, E. Temporal coherence methods in real-time rendering. In Computer Graphics Forum, volume 31, 2012.
Yang, S., Zhao, Y., Luo, Y., Wang, H., Sun, H., Li, C., Cai, B., and Jin, X. MNSS: Neural supersampling framework for real-time rendering on mobile devices. IEEE Transac- tions Visualization and Computer Graphics (TVCG), 2023.
Steiner, M., Köhler, T., Radl, L., and Steinberger, M. Frus- tum volume caching for accelerated NeRF rendering. Proc. Int'l Conf. on Computer Graphics and Interactive Techniques (SIGGRAPH), 7(3), 2024.
Yang. S., Zhu, Q., Thugeu J., Qiu, Q. Li, C.,, Yan, Y., Xu, H., Yan, L.-Q., and Jin, X. Mob-FGSR: Frame generation and super resolution for mobile real-time rendering. In Proc. Ins'l Conf. on Computer Graphics and Interactive Techniques (SIGGRAPH). 2024.
Sun, D., Tian, H., Lu, T., and Liu, S. FlexCache: Flexible approximate cache system for video diffusion. arXiv preprint AXXv:25001000022. 2024.
Yu, P., Guo, J., Huang, F,, Chen, Z., Wang, C., Zhang, Y., and Guo, Y. ShadowMover: Automatically projecting real shadows onto virtual object. IEEE Transactions Visualization and Computer Graphics (TVCG), 29(5). 2023.
Thakolsri, S., Kellerer, W., and Steinbach, E. QoE-based cross-layer optimization of wireless video with unper- ceivable temporal video quality fluctuation. In IEEE Int'l Conf. on communications (ICC), 2011.
Zadtootaghaj. S., Schmidt, S., Sabet, S. S., Möller, S., and Griwodz, C. Quality estimation models for gaming video streaming services using perceptual video quality dimen- sions. In Proceedings of the ACM multimedia systems conference, 2020.
Vaidyanathan, K., Salvi, M., Wronski, B., Akenine- T., Ebelin, P., and Lefohn, A. Random-acees neural compression of material textures. ACM Transactions on Graphics (TOG), 42(4), 2023.

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Zhang, H., Guo, J., Zhang, J., Qin, H., Feng, Z., Yang, M., and Guo, Y. Deep Fourier-based arbitrary-scale super- resolution for real-time rendering. In Proc. Int'l Conf. on Computer Graphics and Interactive Techniques (SIG- GRAPH), 2024.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E.,, and Wang. O. The unreasonable effectiveness of deep features perceptual metric. In Proc. IEEE Conf. on Computer Vision and Putlern Recognition (CVPR), pp. 586-595, 2018.
Zhong, Z., Zhu, J.. Dai, Y., Zheng, C., Chen, G., Huo, Y., Bao, H., and Wang. R. FuseSR: Super resolution for real-time rendering through efficient multi-resolution fusion. In Proc. Ins'l Conf. on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), 2023.
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., and Liang, J. UNet++: A nested U-net architecture for med- ical image segmentation. In Medical image computing and computer-assisted intervention (MICCAI). Springer, 2018.
12

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
A. Appendix
A.1. Test Set
We select free scenes from the online Unreal Engine Marketplace to generate our test set and render with a modified version of Unreal Engine 5.1 from the authors of ExtraNet (Guo al., 2021) to produce necessary G-buffer data, Table 6 summarizes the test scenes used for evaluation.
Scene
Sun Temple
Cyberpunk
Asian Village
Resolution
1920x 1080
1920x1080
1920x 1080
Test frames
20
10
20
Frame sequence
150
30
150
Scene from NVIDIA ORCA repository
Scene from video game Cyberpunk 2077
Free scene from Unreal Engine Marketplace
Description
Buffers
Base color, normal, metallic, roughness, scene depth, motion vector, HDR color, world position, NoV
Table 6. Test scenes used for evaluation.
Authors of Implicit Depth (Watson et al., 2023) released their sample test scenes, which we use to evaluate the performance of ReFrame on their network. The Garden Chair scene has 295 frames and rendered 720x5500 resolution.
A.2. Cache Implementation
In this section, we provide more details on how we adapted our test networks to include the caching scheme.
1. EXTRANET
ExtraNet (Guo et al., 2021) uses several history encoders to capture temporal information, which are concatenated to U-Net architecture. We cache the U-Net described in Section 3.1, which includes the inputs from the history encoders.
A.2.2. IMPLICIT DEPTH
Implicit Depth (Watson et al., 2023) uses SimpleRecon (Sayed et al., 2022) as the backbone network to estimate depth features. SimpleRecon is based on a U-Net++ architecture, using ResNet blocks for the encoder and a nested U-Net for the decoder. We cache the U-Net++ architecture as described Section 3.1 and keep the remaining network the same.
A.2.3. FBSR
Fourier-based super resolution (FBSR) (Zhang et al., 2024) does not use standard U-Net or U-Net++ architecture. However. the network still generally follows decoder-encoded structure, with feature extraction blocks concatenated before the final block. We apply similar principle and cache inputs to the concatenation to be reused in subsequent frames. In this case, we choose to cache the temporal and high resolution (HR) features since the low resolution (LR) features are most similar to the shallow layers a U-Net and provides the most contextual information of the current frame. We also consider caching only the temporal features, but find that while quality does not improve noticeably, the performance gains are not as significant as caching both temporal and HR features. Figure 10 shows the FBSR architecture with ReFrame.
13

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Frame
Frame ++1
Skipped block
Tempoocl
Save to cache
Fucon
Fucoo
Convolution
Concatenate
Figure 10. Diagrams of the FBSR architecture with ReFrame. In frame t, the network computes the full inference end-to-end, saving intermediate features from the temporal recurrent feature extractor and HR G-buffer feature extractor. subsequent frames + 1, the network reuses the cached features.
A.3. Ablation Configurations
Figure 11 shows the Level 1-3 cache configurations used in the ablation study for U-Net architectures. Figure 12 shows the U-Net++ A and B cache configurations used in the ablation study for U-Net++ architectures.
Full Inforence
Level
Level
Level 3
Skipped block
X
X5
X
X²3
Save to cache Upsample
X1
X
X
X5
C
X
X
X
X
Downsample
Skip connection
X2
X2
Z+
X3
X
X²
X3
X
X
X3
X"1
Figure 11. Ablation configurations for ExtraNet U-Net architecture.
Famee
Sepped bleck
Sawe cache
X30
XU
Dowrrample connection
XT
U-Nett+ A
U-Net++
Figure 12. Ablation configurations for Implicit Depth U-Net++ architecture.
Table 7 shows the detailed settings of the cache refresh policies used in the ablation study.
Configuration
Delta
Delta L
Motion Vector
Non-Linear
Parameters
0.20
0.25
110, 1.4
Table Detailed settings of cache refresh policies. 7 values are thresholds for SMAPE and average motion. c and p are parameters for the non-linear policy described in DeepCache (Ma et al., 2024).
A.4. Comparison to Video Games
We compare the amount of motion in our test set to those commonly observed videos of real-world game play from
Gaming VideoSET (Barman al., 2018) and CGVSD (Zadtootaghaj et al., 2020). These datasets are designed to evaluate
14

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
25th Percentile
Median Motion
75th Percentile
Temple
Cyberpunk
Village
Chair
Motion Magnitude
100
250
50
150
250
100
150
200
250
Fortnite
Dota 2
FIFA 17
CS:G0
10
0
20
60
80
20
40
60
80
20
40
60
20
40
60
80
Frame
Figure 13. Optical flow motion comparison matching our real-world game play from GamingVideoSE:: (Barman al., 2018) and CGVSD (Zadtootaghaj et al., 2020).
(Rel. to Ref Frame)
0.4
No Update
Zero
FLI Score
0.3
Random
Normal Dist.
0.2
Noisy
0.1
Very Noisy
Cache
0.0
Ref
2
3
4
5
6
8
9
10
11
12
13
14
15
16
17
18
19
Frame
Figure hypothesis test with Sun Temple on ExtraNet.
video compression algorithms and contain a representative variety video game content, We measure the per-pixel deltas between each subsequent frame, as we use this metric to trigger cache refreshes. We also measure optical flow between frames as secondary check that our test set exhibits similar range of motion to real gameplay scenarios. Table 8 shows that our demonstrates variety of motion patterns, within similar range to the gaminng video datasets.
A.5. Null Hypothesis
We test the null hypothesis that the contents of the cache are irrelevant to the final result, meaning that the cached features can be replaced with any arbitrary value without affecting the network output. Specifically, we try replacing the cache with all zero values (Zero), uniformly random values (Random), normally distributed random values (Normal Dist.), all of which generate significantly worse results according to the FLIP score. Notably, the results are worse than not running the network at all (No Update).
We also test variations that add noise to the cached features. small amount noise (one standard deviation) does not affect the results, implying that the cached features are not sensitive to small perturbations. However, larger amounts noise do degrade the final results. Figure 14 shows the results of the null hypothesis test for the Sun Temple scene on ExtraNet.
15

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Per-Pixel Delta
Per-Pixel Delta (25th percentile)
Per-Pixel Delta (Median)
Per-Pixel Delta (75th percentile)
Average Optical Flow Magnitude
Dataset
Scene
(Average)
GamingVideoSET
CS:GO
12.85
2.00
5.79
14.58
11.23
GamingVVdeoSFT
Diablo III
2.73
0.71
1.41
3.08
2.35
Ti
Dota 2
0.99
0.00
0.00
0.71
0.19
GamingVideoSSS
FIFA
5.51
0.71
1.50
3.24
0.61
Gaming VideoSET
HIZI
7.97
1.00
3.00
8.40
7.28
GamingVideoSHE VidcoSET
Hcarthstone
0.45
0.00
0.00
0.00
0.19
CGVDS
Overwatch
8.19
0.88
2.60
8.82
3.67
CGVDS
Fortnite
7.85
1.28
3.38
8.98
2.48
Ours
Sun Temple
0.00
2.55
10.20
1.81
Ours
Cyberpunk 10.73
0.99
2.37
7.45
0,36
Ours
Asian Village
17.85
2.55
7.65
22.95
5.80
Ours
Garden chair
40.89
9.71
23.07
51.47
4.53
Table Video motion analysis comparison of per-pixel deltas and optical flow between test sequences and real-world gaming video from GamingVideoSEEr VideoSET (Barman et al., 2018) 8) and CGVDS (Zadtootaghaj et al., 2020) datasets.
Saved input size (for frame deltas)
Peak memory usage (baselinc)
Peak memory usage (ReFrame)
Network
Cache size
ExtraNct
21MB x24x360
21MB warped image
787 MB
787 MB
float tensor)
(6x720x 1280 float32)
FBSR
158MB (16x540x960
1MB LR feature
5.1GB
5.3 GB
float32 temporal feature
(3x270x480 Flaat32) +
+ 64x5400960 float.32
24MB prev. result
HR feature)
(3 1x100xx1900 31oat32)
Implicit Depth
84MB (seven 64x 64x192x256
2MB imput image
396 MB
457 MB
float32 lensors)
(3x384x512 float32)
Table 9. Cache memory usage of our selected workloads. The cache size is determined by the size of the tensors stored between sequential
frames. Peak memory usage is measured for end-to-end network inference.
A.6. Memory Consumption
ReFrame has a relatively small memory footprint, requiring storage of only a few tensors. The storage size is dependent on network architecture and the resolution of the input image. Although high resolution images require a larger cache, the high resolution would also result in more FLOPs reduction with our technique, which helps justify the larger memory consumption. Table 9 measures the cache memory usage of our selected workloads. memory large concern, the cache also be stored in lower precision, such float with negligible changes in per-frame quality.
A.7. Additional Results
We provide additional results for longer frame sequences in Table 10. Our main test already contains several cache refreshes, which captures the latency and image quality effects of applying ReFrame. Longer sequences simply include more varied set of cache refreshes, which does not significantly change the results.
We also provide additional results reporting the latency of the network inferences. Table 11 compares the latency with DeltaCNN, ReFrame, and both techniques combined against the baseline network. Applying ReFrame in addition to DeltaCNN compounds the latency reduction.
Table 12 reports the average latency and the worst case latency represented by the 95th percentile. As explained in Section 5, ReFrame is only effective reducing the average latency.
16

---

ReFrame: Layer Caching for Accelerated Inference in Real-Time Rendering
Skipped Frames
Eliminated Ene-Dee FLOPs
Workload
Scene
Speedup f
FLIP
SSIM 1
PSNR
LPIPS 4
MSE
FE
Sun Temple (Long)
45%
24%
1.25
0.025
0.990
36.01
0.011
4.31
Asian Village (Long)
69%
37%
1,50
0.037
0.987
39.55
0.018
4.85
Asian Village (Long)
45%
32%
1.33
0.070
0.950
49.37
0.050
15.40
Table 10. Performance and image quality results with longer frame sequences. (FE frame extrapolation, SS supersampling)
Frame
Baseline Latency
Latency w/ DeltaCNN
Latency w/ ReFrame
Latency w/ DeltaCNN + ReFrame
0 Ref
Full Inf.
4.6 ms
Full Inf.
4.6 ms
Full Inf.
4.7 ms
Full Inf.
4.7 ms
1 Dellta
Full Inf.
4.6 ms
Delta Inf.
3.8 ms
Cached Inf. Full Inf.
2.0 ms
Delta + Cached Inf.
1.9 ms
2 Delta
Full Inf.
4.7 ms
Delta Inf.
3.7 ms
4.8 ms
Delta Inf.
3.7 ms
Table Latency breakdown three sequential frames of the Asian Village scene with ExtraNet comparing baseline network DeltaCNN, ReFrame, and the combined approach of DeltaCNN + ReFrame.
Workload
Scene
Baseline Avg.
Baseline 95pet.
Delta L Avg.
Delta L 95pet.
Delta H Avg.
Delta H 95pet.
FE
Sun Temple
4.60 ms
4.75 ms
2.67 ms
4.55 ms
3.24 ms
4.65 ms
Cyberpunk
3.56 ms
4.22 ms
2.39
3.46 ms
2.89 ms
3.80 ms
Asian Village
4.16 ms
4.72 ms
2.68 ms
4.64 ms
3,36 ms
4.64 ms
SS
Sun Temple
68.6 ms
69.0 ms
35.4 ms
68.0 ms
51.8 ms
68.9 ms
Garden Chair
109 ms
118 ms
91 ms
109 ms
104 ms
ms
Table 12. Latency results of our selected workloads, measured in milliscconds. Avg. is averaged over all frames in the test set; 95pct. the worst case latency the 95th percentile.
17


Reviews:
[{"review_id": "inzqYC2uec", "reviewer": "Reviewer_PRfu", "summary": "The authors propose a technique for caching intermediate features of U-net style networks to skip computation of hidden layers. These intermediate features are recomputed when the changes is above a certain threshold. Overall, this produces an average improvement of performance without significant drop in quality on three tasks (frame extrapolation, neural supersampling and image composition).\n\n## Update after rebuttal\nThe author's rebuttal arguments have addressed my concerns adequately. I've updated my rating for this review.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review and feedback on our paper.\n\nWe have added additional data on the 95th percentile as suggested, reported as runtime in milliseconds.\n\n| **Workload**                  | **95th Percentile Latency (with cache)** | **95th Percentile Latency (baseline, no cache)** |\n| ----------------------------- | ---------------------------------------- | ------------------------------------------------ |\n| ExtraNet - SunTemple          | 4.55 ms                                  | 4.75 ms                                          |\n| ExtraNet - CyberPunk          | 3.46 ms                                  | 4.22 ms                                          |\n| ExtraNet - Asian Village      | 4.64 ms                                  | 4.72 ms                                          |\n| Supersample - SunTemple       | 68 ms                                    | 69 ms                                            |\n| Implicit Depth - Garden Chair | 109.2 ms                                 | 118.2 ms                                         |\n\nHowever, as the reviewer points out, our method improves the latency ***on average*** and therefore has little effect on the slowest percentiles. Relating to question (1), we concede that our method cannot maintain a consistently faster frame rate. Although rendering requires a consistent frame rate, our cache technique focuses on post-processing networks that enhance the rendering, rather than rendering the base image itself. We believe our method is most suitable in low-end devices where additional image quality improvements from neural networks are included in a best-effort manner. Furthermore, reducing computation on average still reduces energy consumption, which is an especially important target in mobile-class devices such as VR headsets. \n\nOne possible solution to produce a more stable frame rate is to amortize the cache refresh over two (or more) frames. The first frame can continue utilizing the cache contents while starting a full inference to refresh the cache. The refresh completes in the second frame asynchronously without delaying the first frame and the second frame can use the updated cache. \n\nTo answer question (2), we do not leverage any CUDA specific memory or custom kernels in our cache implementation. The cache is relatively small, storing one or several tensors ranging from 20-160MB in total for our experiments. Also, loading these tensors from the cache is not different from loading the intermediate results from the previous network layer in a regular full inference, thus we find it unnecessary to optimize this operation. \n\nWe add DeltaCNN related runtime results to answer question (3), matching Table 3 in the paper. Our caching approach produces higher quality results in less time than DeltaCNN due to the nature of the workload as described in the paper (high channel dimension and lower sparsity). When combined, the runtime is not significantly improved, at detriment to the quality. \n\n| Baseline | Runtime (ms) | DeltaCNN        | Runtime (ms) | Cache            | Runtime (ms) | DeltaCNN + Cache         | Runtime (ms) |\n| -------- | ------------ | --------------- | ------------ | ---------------- | ------------ | ------------------------ | ------------ |\n| Ref      | 4.6          | Full Inference  | 4.6          | Full Inference   | 4.7          | Full Inference           | 4.7          |\n| Delta-1  | 4.6          | Delta Inference | 3.8          | Cached Inference | 2            | Delta + Cached Inference | 1.9          |\n| Delta-2  | 4.7          | Delta Inference | 3.7          | Full Inference   | 4.8          | Delta Inference          | 3.7          |", "claims_and_evidence": "The proposed method caches intermediate feature maps and leveraging temporal coherence in U-net style networks to gain performance in rendering applications without significant drop in quality. Various caching policies are compared.", "methods_and_evaluation_criteria": "The method evaluates the average latency of the networks and also the quality of the rendered image against the full network evaluation using various image quality metrics like FLIP, LPIPS, SSIM and PSNR on a variety of tasks such as frame extrapolation, neural supersampling and image composition.", "theoretical_claims": "No theoretical claims in the paper.", "experimental_designs_or_analyses": "The overall experimental design makes sense and accounts for the total number of full inferences, floating point operation decreases, average speedup and quality measurement. An additional metric that might be relevant for this article might be the 95th percentile of inference time.", "relation_to_broader_scientific_literature": "The key contributions of the paper are directed towards the high throughput inference line of work that leverages a variety of techniques such as sparsity, network architectures, precision, etc to gain performance while maintaining quality.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "The proposed technique is original and clearly explained. The technique while achieving an average speedup has slowdowns at irregular intervals to recompute the caches.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The method proposed provides an average improvement in throughput of the framerate but it is unclear how a consistent framerate could be achieved which is important for rendering applications. Do you have any thoughts or ideas on how it could be achieved?\n2. Do you leverage any CUDA specific memory or caching techniques using custom kernels or APIs?\n3. What runtime do you observe when using DeltaCNN on the Asian village scene both with and without the proposed caching scheme?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PONh4VJmSI", "reviewer": "Reviewer_7Mb7", "summary": "The paper proposes to speed up real-time rendering tasks by leveraging the feature caching technique proposed in DeepCache, with extension to UNet++ and adaptive cache polices that are more suitable in the rendering context. Results show 1.4x speed up on average with negligible quality loss.\n\n## update after rebuttal\n\nI keep my sore for the reasons explained in the rebuttal comment", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful review and feedback on our paper.\n\nTo clarify key points:\n\n1. *Testing sequence length:*\nWe tested short sequences because the cache is only valid for a few frames, and the test of 10-20 frames already includes several cache refreshes. However, we add additional data for both the Asian Village scene and Sun Temple scene that tests sequences of 250 frames. We find that the speedups and quality results for 250 frames are aligned with our original experiments, with a more varied pattern of cache refreshes under an adaptive policy. \n\n| **Workload** | **Scene**     | **Full Frames** | **Enc-Dec FLOPs** | **Speedup** | **FLIP** | **SSIM** | **PSNR** | **LPIPS** | **MSE** |\n| ------------ | ------------- | --------------- | ----------------- | ----------- | -------- | -------- | -------- | --------- | ------- |\n| ExtraNet     | Asian Village | 31%             | 63%               | 1.50        | 0.037    | 0.987    | 39.55    | 0.018     | 4.85    |\n|              | Sun Temple    | 55%             | 76%               | 1.25        | 0.025    | 0.990    | 36.01    | 0.011     | 4.31    |\n| Supersample  | Asian Village | 55%             | 68%               | 1.33        | 0.070    | 0.950    | 49.37    | 0.050     | 15.40   |\n\nFurthermore, we investigated the quality of our test sets by comparing to examples from GamingVideoSET [1] and CGVDS[2], which are datasets of real gameplay video recordings. We compare the distribution of our frame-to-frame pixel deltas to those observed in the gaming datasets to judge if our videos are a reasonable representation of motion that occurs in video games. We find that our distribution aligns well with GamingVideoSET [1] and CGVDS[2], matching better to first-person perspective games. \n| **Dataset** | **Scene**    | **Per-Pixel Delta (Average)** | **Per-Pixel Delta (25th percentile)** | **Per-Pixel Delta (Median)** | **Per-Pixel Delta (75th percentile)** |\n| ----------- | ------------ | ----------------------------- | ------------------------------------- | ---------------------------- | ------------------------------------- |\n| [1]         | CSGO         | 12.85                         | 2.00                                  | 5.79                         | 14.58                                 |\n| [1]         | Diablo III   | 2.73                          | 0.71                                  | 1.41                         | 3.08                                  |\n| [2]         | Overwatch    | 8.19                          | 0.88                                  | 2.60                         | 8.82                                  |\n| [2]         | Fortnite     | 7.85                          | 1.28                                  | 3.38                         | 8.98                                  |\n| Ours        | SunTemple    | 10.20                         | 0.00                                  | 2.55                         | 10.20                                 |\n| Ours        | CyberPunk    | 10.73                         | 0.99                                  | 2.37                         | 7.45                                  |\n| Ours        | AsianVillage | 17.85                         | 2.55                                  | 7.65                         | 22.95                                 |\n| Ours        | Garden chair | 40.89                         | 9.71                                  | 23.07                        | 51.47                                 |\n\n2. *Image quality metrics:*\n\nThe reported image quality metrics are reference-based and show the relative quality of our proposed method when compared to the baseline image without our method. For example, the baseline image would have MSE = 0 against itself. We are not concerned with the orthogonal measure in quality of the baseline network since we are not targeting quality improvements, only latency and computational reduction. \n\n3. *FLIP score:*\n\nUnfortunately, there is no generally agreed upon threshold of “acceptable” quality loss for FLIP. The FLIP score purely measures perceptual quality differences between two rendered images and is not influenced by the rendering method. Therefore, we included a range of scores from other neural rendering papers that implicitly claim their scores as “acceptable” in order to help readers interpret and judge our scores. \n\n4. *Supersampling:*\n\nYes, the baseline scaling factor is always 4x in our results.\n\n\n\n[1] N. Barman, S. Zadtootaghaj, S. Schmidt, M. G. Martini and S. Möller, \"GamingVideoSET: A Dataset for Gaming Video Streaming Applications,\" *2018 16th Annual Workshop on Network and Systems Support for Games (NetGames)*\n\n[2] S. Zadtootaghaj, S. Schmidt, S. S. Sabet, S. Möller, and C. Griwodz. 2020. \"Quality estimation models for gaming video streaming services using perceptual video quality dimensions,\" In Proc. of the 11th ACM Multimedia Systems Conference (MMSys '20)", "claims_and_evidence": "There are some claims that need further clarification, which I will discuss below under Experimental Designs Or Analyses.", "methods_and_evaluation_criteria": "They make sense. The method is very straightforward, and the evaluation criteria are well-known in the literature.", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "Here are three key points that need further clarification:\n\n1. Each scene contains only 10–20 frames for testing, which seems insufficient to capture the variety of motions and changes in adjacent frames. For example, the frames would be very similar to each other if there are only 10 frames, in which case the uniform cache policy may perform just as well as the proposed adaptive cache policy. ExtraNet, in comparison, uses 6000 frames for training and 1000 frames for testing. This is a significant difference, raising concerns about whether the proposed method has been evaluated on a sufficiently diverse test set. I expect the authors to provide an explanation for this choice and discuss its potential impact on the results.\n\n2. For the image quality metrics in table 1, the authors only show the results for two variants of the proposed method, but not the results for baselines for comparison. How do the baselines perform in terms of those image quality metrics?\n\n3. In section 4.2 the authors say \"all scores are generally below the acceptable losses observed in other neural rendering systems, which report scores between 0.05 and 0.28 in their final results\". Does the \"score\" here mean the FLIP score? Additionally, could the authors elaborate on why comparisons are made with neural rendering systems (specifically, the three cited papers), which seem to be addressing different tasks?\n\n4. In the supersampling task, what's the original scaling factor used in the baseline's experiments? Is it also 4 times?", "relation_to_broader_scientific_literature": "This paper has the potential to accelerate various components of the modern rendering pipeline, such as frame extrapolation, supersampling, and image composition, as demonstrated in the paper. The proposed method could be further extended to video processing tasks, such as video generation.", "essential_references_not_discussed": "To the best of my knowledge the essential related works are cited.", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "The proposed method is simple yet effective. However, I remain unconvinced due to the concerns raised under Experimental Designs or Analyses. I would appreciate it if the authors could provide a more detailed explanation and justification addressing these issues.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KetP5VD4Up", "reviewer": "Reviewer_p743", "summary": "The paper proposed a training-free intermediate features caching method to accelerate diffusion model inferencing in real-time rendering. Targeting encoder-decoder style networks, the proposed method caches intermediate network layer outputs to be reused in subsequent inferences in order to reduce frame rendering latency.  Different caching policies are explored in the paper, including different cache refresh referencing methods.  The paper claims to achieve a speedup of 40% on average with negligible quality loss in three real-time rendering workloads.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and will update Table 1 as suggested.\n\nAddressing reviewer questions:\n\n*Memory overhead:*\n\nScenes with a higher resolution will require more memory. However, the cache only stores values of one or a few tensors, which will not cause a huge occupation of system or GPU memory. Our test scenes of 1080p is a typical input resolution. A higher resolution would also result in more FLOPs reduction by our technique, which helps justify the larger memory consumption. \n\nWe add additional data measuring the memory footprint of the cache for each workload and peak memory usage during inference:\n\n| **Network**    | **Cache Size**                                               | **Peak Memory Usage (Baseline)** | **Peak Memory Usage (Caching)** |\n| -------------- | ------------------------------------------------------------ | -------------------------------- | ------------------------------- |\n| ExtraNet       | 21MB (24$\\times$360$\\times$640 `float32` tensor)             | 787 MB                           | 787 MB                          |\n| Supersample    | 158MB (16$\\times$540$\\times$960 `float32` temporal feature + 64$\\times$540$\\times$960 `float32` HR feature) | 5.1GB                            | 5.3 GB                          |\n| Implicit Depth | 84MB (seven 64$\\times$192$\\times$256 `float32` tensors)      | 396 MB                           | 457 MB                          |\n\nPeak memory usage is measured with `torch.cuda.max_memory_allocated()`. For ExtraNet, the peak memory usage occurs at a stage in the overall network that does not use our caching technique, thus the peak usage is unaffected by the caching scheme. \n\nIf memory is an important concern, the cache can also be stored in a lower precision, such as `float16`, with negligible changes in per-frame quality. \n\n*Dynamic Sensitivity*\n\nWe do not currently consider dynamic sensitivity. However, this is an interesting direction to potentially investigate. For our current work, we believe that dynamic sensitivity is not necessary. The sensitivity setting directly impacts the output quality, which should preferably stay consistent within one scene. The sensitivity settings can be tuned differently for each scene based on its contents.", "claims_and_evidence": "The claims are well supported by extensive experiment results.", "methods_and_evaluation_criteria": "The methods and benchmark baseline make sense for evaluating the performance of the proposed method.", "theoretical_claims": "The theoretical claims of layer caching for U-Net and U-Net++ in the paper are checked to be correct.", "experimental_designs_or_analyses": "The choices of workloads and cache refresh referencing methods are solid enough for designing extensive experiments.", "relation_to_broader_scientific_literature": "The key contributions of the paper extend caching methods, which are originally used diffusion models in to real-time rendering area.", "essential_references_not_discussed": "The related works listed the paper sufficiently covers the content for understanding the paper, including the tnter-frame similarity nature of real-time rendering and real-time rendering networks.", "other_strengths_and_weaknesses": "Strengths:\n- The results show a great improvement in the rendering latency with negligible quality degradation.\n- The paper thoroughly investigates the possibility of different caching policies.\n\t\nWeaknesses:\n- The test sets are slightly inadequate for evaluating different rendering scenes.\n- The overhead of caching is yet to be discussed.", "other_comments_or_suggestions": "In Table 1, it would be clearer to list the type of workloads rather than the name of them.", "questions_for_authors": "- How is the overhead (i.e., memory) of the proposed method? Will scenes with higher resolution cause a huge occupation of system or GPU memory?\n- Has a dynamic sensitivity been considered? The experiment results show very different performance levels; a fixed sensitivity may not be the best choice in practical usage in various scenes.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7JMNFwjScN", "reviewer": "Reviewer_axt7", "summary": "This paper incorporates traditional caching methods previously used in U-Net based diffusion models into modern real-time rendering applications. The authors propose a novel caching policy that leverages motion vectors in graphics rendering pipeline to adaptively perform cache updates when the difference between frames has exceeded a preset threshold. The experiment results demonstrate that the proposed layer caching technique can accelerate several real-time rendering workloads with a speedup of up to 1.85x.\n\n## update after rebuttal\n\nThank you for the rebuttal and the additional material that demonstrates the degree of motion across different evaluation datasets. However, I think the limitation to resource constrained devices and low-motion scenes considerably impacts the novelty of this work, as such environments are less sensitive to latency issues compared to dynamic scenes. In addition, one main contribution of the work is the adaptive policy, which distinguishes the work from prior uniform caching policies. However, I find the evaluation datasets not comprehensive and representative enough to demonstrate the effectiveness of the adaptive policy. There were only two videos provided during rebuttal, and the duration (number of frames) of the SunTemple scene is too short, which fails to demonstrate the superiority of an adaptive policy over a uniform policy. Furthermore, from the analysis of the optical flow measurements, the evaluation should incorporate more datasets exhibiting motion characteristics and temporal duration at least comparable to the AsianVillage scene. To this end, I would keep my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed and thoughtful review. We appreciate the advice for Table 1 and will update accordingly. The baseline is used as the ground truth reference and the metrics are computed compared to this reference (i.e. MSE = $\\frac{1}{N_{pixels}} \\sum_{i=0}^{N_{pixels}-1} (pixel^{baseline}_i - pixel^{test}_i)^2$) \n\nTo address reviewer concerns:\n\n*Cache Updates*\n\nWith an adaptive policy, the cache would not be active during periods of rapid camera movement or complex animations that take up large portions of the screen space. However, this type of high activity is not typically sustained for a long period of time. In a video game example, there is likely periods of small motion while the players scanning the scene or hiding, interleaved with periods of large motion when a fight occurs. Our caching technique is most effective during small motions and we verify on gameplay recordings from GamingVideoSET [1] and CGVDS[2] that this behaviour exists for many real games. \n\nFurthermore, we believe the \"lag\" to refresh the cache is very small compared to the full inference time and mostly falls within the range of noise fluctuations of the full inference time. The lag is also primary caused by computing deltas rather than refreshing the cache. If this lag is a concern, switching to the N-5 policy nearly eliminates it. Storing the cache only adds around 0.02-0.05ms in latency.\n\n*Limitation to U-Net*\n\nOur proposed method indeed is designed for U-Net-like networks. We appreciate the advice to bring more attention to our current limitations and will update our paper accordingly. \n\nAlthough there are many transformer-based models used in real-time rendering applications (notably in NVIDIA DLSS 4.0), we believe U-Net-like convolutional networks are still heavily employed and more feasible to execute on lower-end devices. For example, super resolution in the Meta Quest VR headsets still relies on traditional algorithmic methods. Our caching technique is particularly useful in these lower-end devices where the proposed trade-off of slight quality loss for performance gains is valuable. \n\nWe have not evaluated our technique on any transformer-based model. However, our method *can* be applied in a more general setting where there is a concatenation of extracted features, as demonstrated in the supersampling workload. If a network uses transformer-based feature extractors, then concatenate the features, our caching technique should still apply.\n\n*Highest Speedup*\n\nWe apologize for the misleading text, which we will fix. In general, we expect the caching scheme to be most effective when the skipped computations contribute to a large part of the overall network inference. For ExtraNet, the U-Net dominates inference time, resulting in good speedup. Supersample, although not U-Net based, uses a concatenation that combines three major components of the network, two of which we cache. Therefore, Supersample actually skips more of the overall network, resulting in its high speedup.  \n\n*Scene Motion*\n\nWe measure \"motion\" to the best of our understanding as the average optical flow magnitude over our test scene frame sequences. As a comparison point, we also include this metric measured on some GamingVideoSET [1] and CGVDS[2] recordings. \n\n| **Dataset** | **Scene**    | **Average Optical Flow Magnitude** |\n| ----------- | ------------ | ---------------------------------- |\n| [1]         | CSGO         | 11.23 |\n| [1]         | Diablo III   | 2.35 |\n| [2]         | Overwatch    | 3.67 |\n| [2]         | Fortnite     | 2.48 |\n| Ours        | SunTemple    | 1.81  |\n| Ours        | CyberPunk    | 0.36  |\n| Ours        | AsianVillage | 5.80    |\n| Ours        | Garden chair | 4.53 |\n\nWe have also uploaded plots of optical flow magnitude per frame over time as a profile for each of our test scenes here:\nhttps://drive.google.com/drive/folders/1hupA8P0ya11EFftxQAtL2RgMghG6XWrZ?usp=sharing\n\n*Supplementary Video*\n\nPlease find sample videos of our workloads with and without the cache applied here: https://drive.google.com/drive/folders/1pQHb7kgL4Dy6rMyTIM7g1JnNpFB9koI4?usp=sharing\n\n*ExtraNet_Baseline_AsianVillage.mp4* shows the output from the baseline network (reference). *ExtraNet_Cache_AsianVillage.mp4* shows the output with our caching technique applied. \n\n*Supersampling_Baseline_HR_SunTemple.mp4* shows the output from the baseline network (reference).  *Supersampling_Cache_HR_SunTemple.mp4* shows the output with our caching technique applied. \n\n\nReferences:\n\n[1] N. Barman, S. Zadtootaghaj, S. Schmidt, M. G. Martini and S. Möller. 2018. \"GamingVideoSET: A Dataset for Gaming Video Streaming Applications,\" 16th Annual Workshop on Network and Systems Support for Games (NetGames)\n\n[2] S. Zadtootaghaj, S. Schmidt, S. S. Sabet, S. Möller, and C. Griwodz. 2020. \"Quality estimation models for gaming video streaming services using perceptual video quality dimensions,\" In Proc. of the 11th ACM Multimedia Systems Conference (MMSys '20)", "claims_and_evidence": "Most claims made in the paper are supported by clear and convincing evidence. However, the authors claim in Section 2.3 that “Real-time rendering networks commonly take advantage of U-Net (Ronneberger et al., 2015) and U-Net++ (Zhou et al., 2018) architectures …”. To my knowledge, there are also recent works that propose transformer-based networks in real-time rendering applications, which the paper does not mention. It is essential to either mention the shortcomings of the transformer-based methods and provide relevant references or demonstrate that U-Net is indeed the mainstream SOTA method with more supportive materials.", "methods_and_evaluation_criteria": "The evaluation criteria mostly follow the evaluations from prior work. However, there is limited discussion and/or demonstration of the dynamics of the scenes selected for evaluation, given that the effectiveness of the work largely depends on the degree of camera/object motion present in the scene.", "theoretical_claims": "There’s no proof for theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound to me.", "relation_to_broader_scientific_literature": "The proposed adaptive layer caching technique for real-time rendering is relevant to layer caching in U-Net based diffusion models.", "essential_references_not_discussed": "References are sufficient.", "other_strengths_and_weaknesses": "Strengths:\n* The technique is training-free and does not require prior knowledge of the workload.\n* The proposed adaptive caching policy is novel and leverages existing information in the rendering pipeline (motion vector) and requires no additional storage overhead.\n\nWeaknesses:\n* The caching technique’s effectiveness largely depends on the dynamic/motion of the scene.\n* The work is limited to U-Net-like network architectures. There is also no mention of other types of networks (e.g. transformer-based models) in real-time rendering applications.", "other_comments_or_suggestions": "* The metrics in the evaluation tables are not intuitive to read. For example, Table 1 states that the results are relative to the baseline, but does not clearly state how the presented FLIP, SSIM, PSNR, LPIPS, MSE values are calculated (does the baseline and experiment have a 41.41 PSNR and 0.0169 FLIP difference).\n* (Minor) The topic of this work is better suited for computer graphics conferences, e.g., SIGGRAPH, EGSR.", "questions_for_authors": "* How does the technique perform in scenes with rapid camera movement and/or complex animations, which is common in many real-time latency-sensitive rendering applications such as gaming? In a highly dynamic scene where both the camera and the objects in the scene experience large degree of motion, it seems like the frequent cache updates may instead cause lag, as Figure 7 demonstrates that many frames with cache updates have longer inference time.\n* In Section 3.1, it is mentioned that the caching scheme can be applied to networks beyond U-Net and U-Net++ architectures. Can you potentially adapt it to work with transformer-based models as well?\n* In Section 4.2.1, it is claimed that the caching scheme is most effective on frame extrapolation tasks where the U-Net dominates inference time. Why is it that in Table 1, for Delta_L, Supersample instead achieved the highest speedup among the three workloads?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received four scores: two weak accepts and two weak rejects, placing it exactly on the borderline. On one hand, the reviewers praised the simple and efficient approach, which does not require training. The method achieves a 40% speed improvement with a negligible difference in quality. On the other hand, there were concerns about the evaluation setup (each scene contained only 10–20 frames for testing) and the use of a U-Net instead of Transformers. The authors provided some additional details and results addressing the former. Regarding the latter (U-Net vs. Transformers), while it is indeed important to consider recent architectures, the AC believes the authors’ response is adequate. We should adapt methods to new architectures, but we also need to start somewhere.

Given that even the negative reviews do not strongly recommend rejection and acknowledge the potential significance of the work, the AC has decided to accept the manuscript.
Decision: Accept (poster)