ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
Authors: Mingzhi Zhu
Ding Shang
Sai Qian Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Codec Avatar
PTQ
co-design
VR
Primary area: applications

Abstract: Photorealistic Codec Avatars (PCA), which generate high-fidelity human face renderings, are increasingly being used in Virtual Reality (VR) environments to enable immersive communication and interaction through deep learning–based generative models. However, these models impose significant computational demands, making real-time inference challenging on resource-constrained VR devices such as head-mounted displays (HMDs), where latency and power efficiency are critical.
To address this challenge, we propose an efficient post-training quantization (PTQ) method tailored for Codec Avatar models, enabling low-precision execution without compromising output quality. In addition, we design a custom hardware accelerator that can be integrated into the system-on-chip (SoC) of VR devices to further enhance processing efficiency.
Building on these components, we introduce ESCA, a full-stack optimization framework that accelerates PCA inference on edge VR platforms. Experimental results demonstrate that ESCA boosts FovVideoVDP quality scores by up to +0.39 over the best 4-bit baseline, delivers up to 3.36× latency reduction, and sustains a rendering rate of 100 frames per second in end-to-end tests, satisfying real-time VR requirements. These results demonstrate the feasibility of deploying high-fidelity codec avatars on resource-constrained devices, opening the door to more immersive and portable VR experiences.

ESCA: Enabling Seamless Codec Avatar Execution through Algorithm and Hardware Co-Optimization for Virtual Reality
Mingzhi Zhu 1.3
Ding Shang' Sai Qian Zhang
"Tandon School Engineering, New York University
Courant Institute of Mathematical Sciences, New York University 'Rensselaer Polytechnic Institute {mingzhi. e dingshang, sai. (aaaz)aaannnnaa. edu
Abstract
Photorcalistic Codec Avatars (PCA), which generate high-fidelity human face derings, are increasingly being used in Virtual Reality (VR) environments to enahle immersive communication and interaction through deep learning- based generative models. However, these models impose significant computational demands, mak- ing real-time inference challenging resource-constrained VR devices such head-mounted displays (HMDs), where latency and power efficiency are critical. To address this challenge, we propose an efficient post-training quantization (PTQ) method tailored for Codec Avatar models, enabling low- sowpreccision execution without compromising output quality. In addition, we design custom hardware accelerator that can be integrated into the system-on-chip on-chip (SoC) of VR devices to further enhance processing efficiency. Building on these components, we introduce ESCA, full-stack optimization framework that accelerates PCA inference on edge VR platforms. Experimental results demonstrate that ESCA boosts Fov Video VDP quality scores by up to +0.39 over the best 4-bit baseline, delivers up to 3.36 latency reduction, and sustains a rendering rate of 100 frames per second end- to-end tests, satisfying real-time VR requirements. These results demonstrate the feasibility of deploying high-fidelity codec avatars on resource-constrained devices, opening the door to more immersive and portable VR experiences. Paper website can be found at wtthttps://eee /ttpm///......//// gi github. io/ /SCCA/
Introduction
Photorealistic telepresence 29 41 in VR requires real-time transmission and rendering of facial expressions with lifelike fidelity. Photorealistic Codec Avatars (PCA) have emerged as promising solution by leveraging variational autoencoder (VAB) models to compress and reconstruct human faces for remote interactions (30]. In this framework, an inward-facing camera on the sender's VR device captures the user's 's facial expressions, and an encoder generates a compact latent representation. This latent code transmitted wirelessly the receiver, where a decoder reconstructs high-quality facial image and 3D avatar. Although the pipeline supports efficient streaming of photorcalistic facial data, achieving the combination of high visual fidelity and ultra-low latency on mobile VR hardware continues to pose significant technical challenge B3
A primary source of processing latency in Codec Avatars is the decoder network, which relies heavily on transposed convolution layers to synthesize high-resolution facial images. While effective for generating high-quality visuals, these layers computationally demanding and introduce significant
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

latency. This poses major obstacle to real-time performance, as delivering a seamless user experience typically requires sustaining 90 frames per second [14
To address this bottleneck, neural network quantization has been widely explored in prior work However, key challenges arise in the context of Codec Avatars: First, due to the large scale of these networks, applying quantization-wware training (QAT) across the entire model is train computationally impractical. Second, activation outliers greatly exacerbate quantization errors, especially in low- precision regimes, resulting in pronounced degradation of visual quality. This issue is most evident in transposed convolution layers, where stochastic latent codes and the absence of normalization induce long-tailed activation distributions with pronounced spikes, illustrated in Figure Because these outliers dominate the value range, they diminish quantization precision and cause caueeddpenden errors. This manifests as temporal artifacts in the reconstructed facial image, including flickering. checkerboard patterns, and unstable shading [88]
a means enable low-latency execution of deep models 9 4
Recent advances have proposed effective strategies for managing outliers in low-bit inference of large mod- els, such activation smoothing [52] and weight rota-
Max (Activation)
Frequency
100
tion [1) These approaches enable 4-bit precision in
large language models by rescaling channels or rotat- ing weight matrices to suppress extreme values while preserving accuracy. However, such strategies cannot be directly applied to the PCA model. The presence of transposed convolution layers and non-linearities prevents straightforward use of offline weight rotations or channel scaling, as these modifications would alter the model's outputs. Consequently, existing outlier- smoothing techniques are incompatible with the PCA decoder architecture, leaving efficient 4-bit quantization for high-fidelity facial generation as open challenge.
100
200
Channels
Activation Value
Figure 1: The left panel shows the maximum activation value of each channel of a sample input. The right part shows the aggregated activation distribution over all spatial loca- tions and channels.
Beyond algorithmic challenges, VR HMDs remain resource- con.urcc-ccnttrieeed which makes execution of the computationally intensive PCA decoder slow. While some devices incorporate GPUs and NPUs 34 these units must also support concurrent tasks such as rendering 32 image process- ing [3 and other Al workloads [48 26
16. Addressing this constraint calls for a dedicated plug-in module within the SoC to manage Codec Avatar infer-
hardware accelerator integrated ence. Yet, the decoder's reliance transposed convolution layers poses unique difficulty, their intrinsic structured sparsity severely limits accelerator utilization and efficiency.
To overcome these challenges, we propose a comprehensive quantization and acceleration framework for Codec Avatars, enabling real-time inference on resource-cons constraied VR devices. Our approach introduces several novel techniques to maintain visual quality under low-bit quantization and co-designed hardware solution to meet strict latency requirements. summary, our contributions are:
We introduce novel input channel-wise smoothing module inserted during training to alleviate extreme inter-channel activation dis- parities in the VAE decoder. By reducing outlier activations, ICAS diminishes quantization error and prevents aberrations when the model is later quantized to low bit-widths.
Input Channel-wise Activation Smoothing (ICAS):
Facial-Feature- -eaturee-Aware Smoothing (FFAS):
We develop region- smoothing strategy that uses facial masks to identify key areas like the eyes and mouth. Based on the activation variance in these regions, FFAS selectively skips smoothing for the channels most critical to fine details, preserving important textures while still smoothing less sensitive regions.
UV-weighted Hessian-Based Weight Quantization: We propose a weight quantization scheme guided by a UV-mapping weighted Hessian. This method computes second-order sensitivity and weights it by the UV importance of each face region, thereby concentrating on the precision of weights that most affect critical facial features.
Customized Hardware Accelerator: We co-design a specialized hardware accelerator to support our quantized Codec Avatar model with high-throughput 4-bit and 8-bit operations. The accelerator features an input-combining mechanism to exploit the structured sparsity of the activation matrix. Moreover, an optimized end-to-end pipeline is applied to deliver over 100 FPS inference on an VR headset ensuring smooth, real-time avatar rendering.

---

Cinteret
Conv ToanTranap
Rendering
Latent Code
2D Texture
Pre. tox
Transmitter
Pre_mesh
view
Encoder
Decoder
3D Mesh
Receiver
Figure 2: VAE Framework of Codec Avatar models.
Background and Related Works
2.1 Codec Avatar and Photorealistic Pipeline
PCA are neural face models that enable authentic telepresence in VR by rendering lifelike 3D avatars of users in real-time [2T 30). They are typically implemented as variational autoencoder (VAF) [29] framework that transmits facial expressions efficiently between users. Figure Saarreeillustaar illuustrates the overall
stucture of Codec Avatar models. This VAE-based approach has been demonstrated on VR headsets (e.g. Meta Quest Pro) as feasible solution for real-time face-to-face communication, achieving convincing sense of social presence while drastically reducing transmitted data compared to raw video 33
Figure (a) decomposes the end-to-end execution pipeline of the Codec Avatar upplication, which can be divided into five main stages: Sensoring, Encoding, Transmission, Decoding, and Rendering. As illustrated in Figure 3 (b), VR device comprises several hardware modules, primarily the CPU, GPU, front-facing camera, and memory subsystem. To gauge the limits of current commercial platforms, we profile the Snapdragon XR2 Gen 2 SoC [39] powering Meta's Quest 3. Running the full Codec Avatar model on Qualcomm AI Hub [40] yields median inference latency of 39.6 ms, only 25.25 FPS, even before accounting for Sensoring. Transmission, and Rendering overheads. Offloading to cloud servers is likewise untenable due to added latency and privacy concerns (33]. Thus, to enable truly immersive telepresence, Codec Avatar must execute on-device within the available compute budget 46 77. Moreover, running the PCA module continuously consumes additional VR hardware resources, leaving fewer computational resources for other applications to operate effectively. This motivates the development of custom hardware accelerator plug-in of the VR SoC handle Codec Avatar decoding. Prior works have shown that specialized architectures for generative models vastly improve efficiency [20] For instance, FPGA-based cansposed-convolttion engine achieved up to 3x better performance-per-vee
than GPU on similar tasks 4 In summary, full-stack approach that co-designs the Codec Avatar with hardware support is essential reach 90 FPS real-time photorealistic telepresence 14]
8
on power-comstrained VR devices.
2.2 Post-Training Quantization
converts a pre-trained foating-poin network to low-precision integers without retraining, making it attractive deployment strategy for resource-constralie VR hardware. Classic weight-only PTQ schemes such as AdaRound [35], GPTQ [1], and OmniQuant 42 minimize layer-wise reconstruction error by solving local opti- mization problems, while recent activation-aue aware methods further suppress outliers to unlock 4-bit inference for language models. SmoothQuant [52) migrates per-channel activation magnitude into the weights via learned scaling factors, where QuaRot Rot 3 DuQuant [22] and SpinQuant (28) apply orthog- onal rotations jointly smooth activations and weights in Hessian-aware manner. Given Y XW. they insert a matrix R where R R / and rewrite the product as Y XW = (XR) W). thereby smoothing X without changing the network output. The new weight R W is folded offline, while the rotated activation is absorbed into the preceding layer, inference cost is unchanged.
Post-training quantization (PTQ)
35 12 52
28
These techniques are effective because Transformer layers are dominated by matrix multi- plications, whose linear properties remain intact under such transformations. However, directly applying these methods to convolution-ban based codec-avatar decoders is non-trivial. First, convolu- tional generators utilize 4-D kernels and 3-D activations, violating the 2-D matrix assumptions that

---

Sensoring
VR Headset
Weight kernel size: 4x4
CPU
GPU
Encodiing
Inderpaiation Padsing
Transmission
Network Chip (NoC)
Decoding
Activation
Rendering
Cameras
SoC
DMA
VR Display
Input size: 2x2
7x7
Output
10-16
(a)
[b)
(c)
Figure (a) Execution pipeline the entire Codec Avatar system. (b) Architecture of normalized VR headset SoC. (c) Illustration of transposed convolution. Purple squares represent non-zero activation, and white squares represent zero activation.
underpin SmoothQuant's channel-wise scaling. Second, our decoder extensively employs transposed convolutions followed by non-linear activations (e.g.. LeakyReLU (53)), rendering any offline weight rotation invalid once activations pass through these non-linear transformations. In summary, existing PTQ methods ill-suited to the architectural and signal-processing peculiarities of codec-avatar decoders, leaving efficient 4-bit quantization of high-fidelity face generators an open problem.
2.3 Transposed Convolution and Im2col Transformation
Modern Codec Avatar decoders rely on transposed convolution (ConvTransppa Transposp) layers, also known as deconvolutions, upsample low-resolution feature maps into nigh-resolution images or textures [29 The transposed convolution expands the spatial dimensions by inserting zeros between and around input pixels before applying convolution kernel, effectively spreading the feature map. The output width W of the activation after this pre pre-proeessing can be computed as:
30 13].
W' + 2(K - P 1) + (W - -)(S-)
(I)
where W denotes the original width of the activation map. and K, P. and represent the kernel size, padding, and stride, respectively. The activation maps and kernels assumed to be square, meaning the width and height are equal. For example, considering the first layer of the decoder, the activation width is W 2 and K 4, S P According to the equation, the activation width after inserting zero becomes + 22x x (4 14-1- 1) (2 - 1) (2 - 1) = 7. As illustrated in Figure (c), one zero is inserted between every adjacent activation (interpolation), and two layers of zeros are padded around the boundary. Thus a 2 x 2 feature map turns into x 7 after applying zero-inserting.
To leverage the high throughput of systolic-array-bee based accelerator, the convolution operations are typically transformed into general matrix-matrix multiplication (GEMM) [I.. This requires flattening righ-dimensional activations and weights into wooddimeniional matrices using the im2col transformation 66] []. However, due to the zero-inserting introduced during transposed convolution, activation matrix becomes extremely sparse As shown in Figure 3 (c), this manifests as a checkerboard- sparsity pattern, with more than 85% of the elements being This high degree of sparsity significantly degrades the efficiency of the hardware accelerator (20), (MAC) operations become redundant (multiplication by zero), wasting both compute cycles and memory bandwidth (54).
the resulting P22col-transfommtt
as the majority of
Methods
Our pipeline tackles the twin challenges of low-bit quantization and real-time deployment of Codec Avatar decoders through four tightly-coupled components: (a) Channel-wise Activation Smooth- ing: (b) Facial-Feature-Awarr Smoothing: (c) UV-Weighted Post-Training Quantization; (d) Input- combined DNN hardware accelerator. Together, the techniques shown in Figure 4 and 5 deliver artifact-free 4/8-bit inference while boosting avatar-decoder throughput on our prototype accelerator.
3.1 Input Channel-wise Activation Smoothing
We introduce Input Channel-wise Activation Smoothing (ICAS) to equalize the scale of activation across channels in each transposed convolution which aims to reduce the quantization difficulty of activations. Our method is inspired by SmoothQuant [52), which migrates the quantization difficulty

---

High Bit
CONVERT
Low
Texture
Smooth
Sulo
Fold&ReLU
Texture
Ty
Deconv Block
Deconv Block
Quanlize
Fold&ReLU
Fold&ReLU
Hessian Matrik
SdB
Tasks
WS:
Count&Norm
Batt Deconv Block
Blas
DownSample
Update
S.WS
Quantize
Deconv Block
Weight
UnFold
UnFold
Mesh
UnFold
Linear
Bmooth
Encoder
Encoder
(0)
(c)
Figure 4: Convert the Codec Avatar decoder to quantized model. The pipeline consists of three main components: (a) The original Codec Avatar decoder, (b) the UV-Weighted Post-Training Quantization, and (c) the decoder layer after Input Channel-wise Activation Smoothing. The smooth operation is designed to reduce the difficulty of quantizing activations, while the UV-PTQ method uses a UV weight map to guide the quantization process. Together, these techniques enable efficient and accurate quantization the Codec Avatar decoder for real-time inference on VR headsets.
from activations to weights through a mathematically equivalent transformation transformers models. For a linear layer Y = XW = (Xdiag(s) |! ()) it introduces scaling factors s to smooth activations.
We adapt this principle to transposed convolutions. For Conv'Transposs layer with input tensor X E R x H.. Wir and weight W E CR. X let 8 ($1 , be set of positive smooth factors, one for each input channel. Specifically, let X denote the scaled input, where each channel is multiplied by its corresponding scale factor s.. In other words, ;, 8. x[c,;,] for every channel E C.n. effectively scaling the activations of each channel by 8c- To preserve the output activations, define W as the adjusted weight tensor for this layer, where the filter corresponding input channel c is scaled by 1/5e. Formally, for each input channel index c, [[[c, i. The proof of the equivalence between Y X W and Y X W is provided in Appendix A where denotes the convolution operation. This formulation introduces a pair of transformations 8, and - for each channel. These transformations offset each other with respect to the layer output. The scale values are carefully determined using the calibration dataset to yield activations that more suitable for quantization, consistent with observations from prior 23).
W : 1 W ee.
post-training smoothing methods 52
Importantly, ICAS incurs runtime overhead, the scale and its inverse are precomputed and fused into the network parameters using offline operation. Specifically, for a transposed convolution followed by nonlinear functions (e.g. LeakyReLU (53)), we fuse the scales into adjacent layers to avoid explicit multiplication at inference. For example, consider two consecutive layers L; and L with non-linear activation function a between layers.
((+1) a (W(), ( + B()),
x(++2) (w(++1) * ( + ((++1)
(2)
where (0) is the input to L,, ( and B(i) are its weights and bias, respectively. Specifically, we have the following observation:
((++))
x(++)) (8
x(() +(())
where @ represents the elementwise product. Continuing with the Codec Avatar model, we adopt the LeakyReLU activation function [53], defined as ((x) max (ax,, 2r), where denotes the negative slope. For any channel e and scaling factor 8e > follows that (88 X (l,,;;)) = S. aX (e, :)). Hence,
( = a(s & [.....))) (0) B())
(4)
where s X [....] denotes the fused weight of layer L.. The detailed proof is provided in Appendix B Based on this formulation, the scales 8 can be incorporated into the weight Wi(0) of the preceding layer by scaling each output channel of (i()) and the corresponding bias ( with the associated factor. We enforce > 0 for all channels to preserve the sign of activations under smoothing. This eliminates explicit scaling operations during inference, as the calibrated scales are pre-fused into the convolutional weights during the offline calibration stage.
5

---

Inspired by the migration strategy of SmoothQuam | we determine the smoothing factor for each channel by balancing the dynamic ranges of the original activations and corresponding weights.
(maxm, X(e, m, ni))
$c
(max....R.) W [e, Co. K )2
(5)
The exponent a E [0, serves as a migration-strennthh hyperparameter. In practice, we sweep over different values of a and select the optimal value of 0.8. Here, m and denote the spatial dimensions of the activation X; k and h represent the kernel dimensions of the weight filters; and c and Co correspond to the input and output channels of the activations and filters, respectively.
3.2 Facial-Feature-Awrrr Smoothing
While ICAS uniformly scales all channels, certain feature channels correspond to critical facial details that should be preserved. We propose Facial-FeatureAAward Smoothing (FFAS) targeted refinement to ICAS. In a Codec Avatar decoder, outputs are often mapped to a texture space repre- senting the face. We leverage predefined facial region masks to identify which feature maps carry fegu-fre details in those regions, Concretely, for each channel с in given layer / with input feature map of size H' W' in texture space, we compute the activation variance within the important facial regions. Let Ry S {1,... H x {l,..., W denote the set of texture pixels belonging to particular facial region of interest. We measure the pixel-wise variance of channel over that region,
a (R.)
=
R
Σ (m,))Rt.
(X' m, n] (o(R))))
(6)
where [',,,i,] is the activation value of channel c at spatial location (t.j) and (e((..) ) is the mean activation over region Ry A large (R.)) indicates that channel e exhibits significant variation within facial region. We rank all channels by o² and identify the top-&% channels with highest variance in critical regions. FFAS exempts these top-k% channels from ICAS smoothing, k is a peprpaarameter. For channels in this set, effectively set s, at full magnitude. The remaining channels still receive smooth facotr determined by ICAS. By skipping smoothing on the most detail-sensitive feature maps, FFAS preserves frequeryy facial details like eye wrinkles and lip creases that might otherwise be attenuated by ICAS. Notably, this selection is data-driven and region-specifie. In our experiments, integrating FFAS the top of ICAS framework effectively mitigated over- smotthit in critical facial regions such as the eyes and mouth. This approach preserved essential expression details, reduced global artifacts, and enhanced the overall visual quality of the generated avatars.
that their activations remain
3.3 Weighttt Post-Trasd Tes-Trainnng Quantization
To preserve perceptually critical facial details under low-bit weight quantization, propose UV-mapping Weighted Post- bost-Training Quantization (UV- Weighted PTQ) that uses view-dependent UV coordinates guide weig-weighted error minimisation. In computer graphics, UV coordinates denotes the 2-dimension texture domain onto which a 3-D surface mesh is parametrically unwrapped. Each vertex of the face mesh stores fixed (u, u) coordinates, allowing the decoder output flat texture map that is later sampled during rendering. Consequently, pixels in UV regions corresponding to salient facial areas (eyes, mouth, nose) are perceptually critical, whereas others may be invisible or less important in the final view [0]]
We compute per-feature UV coordinates by leveraging the existing rendering pipeline of the pretrained VAE decoder. The decoder outputs 3D mesh M VV3 where V is the number of vertices in the mesh. We then rasterize M onto the 2-D grid of feature map with H W to obtain barycentric coordinates for each grid location. Assume @u.p and Φu.n are the UV coordinates of pixel p. weight wp rasterize(M,, H x W), [ou.p: &c.p) - 02 Using these weights, we interpolate the per vertex UV coordinates t E [0, 1]2 to each pixel. Then, we map the normalized coordinates to integer texel indices on the 2-D texture map of resolution H x W, and accumulate a hit-count map A E N We normalize A, apply a upper bound Wmaw and broadcast across channels to form the
W5p -ny

---

soft importance weight Weo E [0, Wmaw Oin x HxW
Alm.n] mazp.a(Ap..).
Wmax
if A [m, n] [ if A(m, n] =
Weel; m. n)
where Afm,n| E ((1 the round function. For each layer in the decoder, we downsample UV weight to match layer's input size H1 x Wi.
Lou W = (m,n)), 1(-) is the indicator function and |- is
During quantization, our goal is to minimize the quantization error between the original and quantized weights. To achieve this, we utilize approximate second-order information by calibrating the weights using the Hessian matrix. When computing this matrix, the activations X are pre-multppiied by the downsampled UV importance weights W
S S Σ (2(WL. un X.) (W un x() + AI
H
(8)
where S is the number of samples, X is the s-th sample of the input layer 1, and À is small regularization term. Given the weighted Hessian H we follow the GPTQ [12] greedy quantization procedure. For each layer with unfold weight W column-by-column. For r-th column, we have
Ka)) we quantize weights
[[rr]] r| = quant(W"((.r1))
e, = W'], -
(9)
where quant(-) maps weights to the nearest quantized value. The quantization error e, is then compensated across remaining unquantized columns using the inverse Hessian.
H [r.J] Ey H Hhr
w'[: + w'
Vj>r
(10)
The UV-weigtting in H',, ensures that errors affecting critical facial regions are penalized heavily.
3.4 Input-Combining Mechanism for PCA Accelerator and Optimized Pipeline
As discussed in Section I VR head-mounted displays (HMDs) are typically resource-conse conttraicd.d, making the execution PCA computationally expensive and slow. Moreover, performing PCA on the GPU or NPU within the HMD can heavily consume hardware resources and degrade the performance other concurrently running VR applications. To address this issue, we design dedicated hardware accelerator for efficient PCA execution. As illustrated in Figure (a), the core of the accelerator 16 systolic array 19, a dataflow architecture consisting of a grid of interconneeted processing elements (PEs). Each PE performs and transmits intermediate results to neighboring PEs in a rhythmic, wave-like manner. This structure is highly efficient for matrix multiplication tasks. In our design, employ weight-stationar systolic array configuration, where weights are preloaded into the array and remain fixed during computation, while activations are streamed in from bottom top in staggered sequence. Partial sums flow horizontally from left to right and are collected from the rightmost column of PEs.
accumulate (MAC) operations
As detailed in Section 2.3 the transposed convolution and im2col transformation introduce extreme sparsity into the activation maps, leading to severe underutilization of the hardware accelerator. To mitigate this inefficiency, following the prior work [20]. we propose an optimization technique called input-comine combbicominnig to compress the activation input and enhance hardware utilization. As shown in ( the activation map is first partitioned into 4x4 tiles. These tiles then categorized into two types: (1) tiles with checkerboard-1 We discard the fully zero tiles and vertically stack the remaining tiles to form compact representation. This eliminates a significant portion of zero activations without loss of useful information.
sparsity patterns; and (2) tiles that are entirely zero.
To implement this combined input format, we modify the PE design as illustrated in Figure 5 (c). Every PE preloads two weights and accepts two activations per cycle, one of which would be zero Two multiplexers are used to select the non-zero activation and its corresponding weight, allowing each PE to perform single MAC operation per cycle. This lightweight enhancement enables the accelerator to bypass most zero activations, focusing computation only non-zero data with minimal
7

---

XanXia
Register
Worght Buffer
Y1
Ws
MUX
MAC
16x16 Systolic Arrey
Codec Avatar Accelerator
DMA
Indrel
PE
Input Buffer
(a)
(b)
(c)
Figure 5: (a) Architecture the proposed hardware accelcrator for Codec Avatar inference. (b) Input-combining tiling scheme applied to the activation matrix. Red lines partition the input activation into smaller tiles. Purple/white squares denote non-zero/zero activations, and yellow squares are zero but be assumed non-zero for simplicity. (c) Internal architecture of the proposed PE.
hardware overhead. the ideal case, this strategy can reduce the number of operations by up to 75%, delivering significant latency improvements, as shown in the experiment results in Section 4.5
As described in Section 2.1 and Figure (a), the complete pipeline of Codec Avatar consists of five stages. To further reduce end-to-end latency, we propose an optimized execution pipeline. As illustrated in Figure (b), Transmission and Decoding are performed in parallel, and multiple frames are processed in an overlapped fashion. Two scheduling constraints must be satisfied: (1) Decoding can only start after Encoding finishes, since both are executed on the same hardware accelerator; (2) Decoding must wait until the corresponding Transmission completes to receive the latent code from the remote user. This pipeline design fully exploits inter-frame parallelism and significantly improves overall throughput. The resulting end-to-end latency and frame rate are analyzed in Section 4.5
4 Experiments
4.1 Experimental Setup
We extensively evaluate the proposed quantization methods on PCA decoding, focusing visual quality and system performance under 4-bit and 8-bit settings. We evaluate our quantization method on the MultiFace dataset [50]. This dataset provides high-quality captures 65 scripted facial expressions, along with ground-trut textured 3D face meshes. For each expression, we use the provided 3D geometry and texture map (1024 1024 UV atlas) render the avatar from three camera viewpoints: one front and two approximately 45 side views.
Classical full-reference metrics such Peak Signal-to-Noise Ratio (PSNR) 45 and Structural Similarity (SSIM) [49] measure absolute pixel-wise or local structural differences. Hence, they correlate weakly with human perception when small mis-alignments high- high-frequency phase shifts are present, both of which are common in generative avatars [33]. We report the VovVideoVDP (VDP) metric 331, full-reference perceptual metric that accounts for spatio-temporal human visual sensitivity designed for wide field-of-view VR content. We also report LPIPS (56) metric which compares feature activations of a pretrained network. All methods evaluated an NVIDIA A100 GPU and the hyperparameter = 75 in our experiment.
As for model inference, we choose Snapdragon XR2 Gen 2 SoC as the baseline against our proposed hardware accelerator, because it is integrated within Meta Quest 3 headset, representing real-world deployment constraints. And we perform the rendering process on NVIDIA Tesla T4 16GB GPU (1590 Mhz clock, 2560 CUDA cores). The experiments are conducted under certain conditions to mimic the performance of an edge device GPU, specifically NVIDIA Jetson Orin NX 16GB [37] (918 MHz clock, 1024 CUDA cores), a mobile platform which has been frequently used in prior research to model rendering latency in VR headsets [5 55
4.2 Baseline Methods
We compare the proposed quantization approach with several state-of-the-art baselines. Full Codec Avator [50] is the original model with no quantization. AdaRound [35] is adaptive weight

---

Table VDP [1 and LPIPS [66 scores for different methods 4-bit and 8-bit quantization. Gray cells indicate our proposed methods. Best results are in bold.
Front
Left
Right
Method
Precision
VDPt
LPIPS
VDPt
LPIPS.
VDP
LPIPS.
Full Model
FP32
6.5364
0.21604
5.9480
0.21965
5.8625
0.20428
Adaround 35 9
4.2531
0.22612
3.6143
0.24000
3.5606
0.22031
POCA 33
5.2310
0.22200
4.3838
0.23643
4.3457
0.21347
2DQuant 25
5.2987
0.22186
4.3948
0.23243
4.3712
0.21209
GPTQ 12
5.4980
0.22048
4.5868
0.23085
4.5729
0.21256
W4A4
ICAS (Ours)
5,5901
0.21981
4.7317
0.227883
4,7536
0.21127
UV-W (Ours)
5.7559
0.21778
4.8130
0.22699
4,8187
0,21062
ICAS-UV (Ours)
5.6438
0.21941
4.9145
0.22650
4,9057
0,20840
FFAS-UV (Ours)
5,8541
0.21746
4.9795
0.22649
4.9605
0,20719
Adaround 35 9
6.2106
0.21667
5,5004
0,22135
5.4381
0.20601
POCA [33
6.4827
0.21612
5,8511
0.22048
5.7565
0.20408
2DQuant 25
6.4983
0.21645
5.8313
0.22088
5.7497
0.20436
GPTQ [2]
6,2359
0.21687
5.6188
0.22101
5.3613
0.20546
W8A8
ICAS (Ours)
5.6007
0.21748
5.3913
0.22973
5.0762
0.20840
UV-W (Ours)
6.5271
0.21610
5,9101
0.22036
5,7610
0.20543
ICAS-UV (Ours)
6.3690
0.21682
5.6615
0.22091
5.5989
0,20541
FFAS-UV (Ours)
6,5241
0.21605
5.8589
0.22068
5.8071
0.20441
rounding technique for post-training quantization. LSQ (Learned step size quantization) [9] is method that learns the quantization scaling size for each layer during training. POCA (Post-training Quantization with Temporal Alignment) is a recent method tailored for codec avatars [33]. We also adapt 2DQuant (25). two-stage PTQ method originally proposed for 4-bit image super-resolution models, to our avatar decoder. GPTQ 12) is a fessian--asses quantization method that uses layer-wise Hessian information to optimize weight updates. We also include a UV-only quantization method that applies quantization without any smoothing(UV-V) Channel-wist wise Activation Smoothing without UV guidance (ICAS) and smooth-UV method without Smoothing (ICAS-UV).
a smooth-only method that only applies
Each baseline is applied our pre-trained avatar decoder, and we evaluate both 8-bit (INT8) and 4-bit (INT4) quantization settings for all methods. For fair comparison, all models including baselines and our use the same pre-trained float32 decoder starting point and are calibrated on a small sample set of avatar frames (5 12 frames). VDP provide more faithful estimate of user- eser-perceived quality than PSNR/SSIM, and thus form the primary metries in our study.
4.3 Low-hit Quantization Results
Table I demonstrates that our complete method achieves superior perceptual quality across all three camera perspectives at 4-bit quantization, surpassing the best- best-performing baseline (GPTQ) with improvements of +0.36/+0.39/ 039+9..9 VDP for front/left/right views, respectively. These gains confirm that combining channel smoothing. UV-guided calibration, and facial-feature protection is crucial for perceptual realism. Ablations highlight complementary benefits: ICAS suppresses bursty channels, UV-W reallocates error to less-visible texels, and FFAS preserves fine eye-mouth details, together yielding the observed jump in temporal fidelity.
At 8-bit precision all methods converge to high quality, yet our methods still edges out the best baseline by up 0.06 VDP. The ICAS-only variants underperform compared other methods at 8-bit precision. This suggests that excessive smoothing can degrade important high- freuueey details, leading to reduction VDP by 0.9 on the frontal view. When sufficient quantization levels are available, aggressive rescaling is unnecessary and may even be detrimental to perceptual quality. Moreover, our proposed method significantly reduces the quality gap between frontal and side views The performance gap from front to left is 0.88 for ours vs. 1.21 for GPTQ, indicating improved view-consistency. This property partieularly important for immersive VR applications, where users frequently change their gaze direction and head pose.

---

Table 2: Fov VovVideoDDP scores for spontaneous facial expressions at 4-bit quantization (W4A4).
Shushing
Surprise
Frowning
Method
Front
Left
Right
Front
Left
Right
Front
Left
Right
GPTQ 12
5.2911
4.4226
4.3828
5,4066
4.5072
4.4783
5.2713
4.4064
4.3877
FFAS-UY UV (Ours)
5.7832
5.0065
5.0280
5,8486
5.0930
5.0989
5.8686
5.0210
5.0606
42.04
Latency (ms)
Encoder
enc
5
enc
%
Docoder
User
dec
tra
dec
tra
15
ren
ren
m8
18
12.51
8
eec
Lenc
ms
3.06
3.05
3.063.13
User B:
ms
tra
tra
dec
dec
Accelerator Accelerator keahocrdtinng appoccotiiing Bat
ren
ren
XR2 Gand
Biit
ms
Figure 6: (a) Inference latency of PCA different hardware platforms. (b) Optimized PCA pipeline.
4.4 Spontaneous Facial Expression Results
To assess how well ESCA generalizes spontaneous facial expressions, evaluate three diverse expressions from the MultiFace dataset's 65 expressions: shushing, surprise, and frowning. These expressions represent range of facial dynamics including subtle mouth movements, wide-eye surprise, and brow furrowing, Table truth across three viewpoints (front, left, and right) at 4-bit quantization. Our FFAS-UV method consistently outperforms the best baseline (GPTQ) across all expressions and viewpoints, with improvements ranging from +0.49 +0.66 VDP. The results demonstrate that ESCA maintains high visual fidelity for spontaneous expressions while preserving consistent performance across viewing angles, confirming its robustness for real-worke avatar applications.
the Vov scores computed against ground
4.5 Latency Results
We evaluate the latency improvement achieved by the proposed input-combining accelerator. The results are shown in Figure 6 ()). For the encoder, which consists solely of standard convolution layers, the inference latency is 3.05 ms under INT8 quantization and remains unchanged for baseline and input-combining accelerators. For the decoder, which is dominated by transposed convolution, the INT8 baseline accelerator achieves latency of 42.04 ms, while the input-combining design reduces this to 12.51 ms, representing a 3.36x speedup. Under INT4 quantization, our input-combining accelerator achieves a minimum latency of 3.13 ms.
We adopt latency references from prior VR research: camera sensor acquisition takes approximately I while Wi-Fi transmission requires around 5 under favorable conditions [57]. Our accelerator executes both Encoding and Decoding in approximately 3 ms each. And Rendering on GPU requires 9.5 ms under the configuration in Section 4.1 With the optimized pipeline introduced in Section 3.4. the effective per-frame latency, indicated by the interval between the two red dashed lines in 6 is determined by twice the Transmission delay, totaling 10 ms. Consequently, the effective frame rate reaches 1000/10 = 100 FPS, achieving significant throughput improvement and fully satisfying the real-time requirement for immersive Codec Avatar applications.
ms 2 24 44
Conclusion
We have presented ESCA, comprehensive framework that co-optimizes neural network algorithms and hardware design to enable real-time PCA inference on VR devices. ESCA combines two smoothing techniques (ICAS and FFAS) with a UV-weighted Hessian- based quantization strategy to achieve high fidelity at 4-bit precision, and it includes custom accelerator optimized for transposed convolutions that yields 3.36 reduction decoder latency. Despite these promising results, several limitations remain. It relies on accurate facial UV priors and only accelerates the decoding stage, leaving other parts of the pipeline unoptimized. Future work will aim to reduce dependence on UV maps and extend co-optimization to the full avatar pipeline for more seamless systems.
10

---

References
[1] torch.nn.unfold.
http:://ttttatoooooooo////s/s/as nn. Unfold
html Accessed: 2025-05-14.
[2] Anastasios N Angelopoulos, Julien NP Martel, Amit PS Kohli, Jorg Conradt, and Gordon Wet- zstein. Event based, near eye gaze tracking beyond 10,000 hz. arXiv preprint arXiv:2004.03577, 2020.
[3] Salch Ashkboos, Amirkeivan Mohtashami, Maximilian Croci. Bo Li. Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213- 100240, 2024.
[4] Hanning Chen, Yang Ni, Ali Zakeri, Zhuowen Zou, Sanggeon Yun, Fei Wen, Behnam Khaleghi, Narayan Srinivasa, Hugo Latapie, and Mohsen Imani. Hdreason: Algorithm-hardware codesign for hyperdimensional knowledge graph reasoning. arXiv preprint arXiv:2403. 055663033 2024.
[5] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3009-3018 IEEE, 2019.
[6] Felix Dangel. unfoldnd: (0=11.2.)) unfold (im2col) and fold (col2im) in pytorch. https://g http:///tttuu..... hatp://///dddagggll/nooollllll 2021. Accessed: 2025-05-14 14.
[7] Minh N. Do et al. Immersive telepresence for entertainment and meetings (item). IEEE Signal Processing Magazine, 116))118812,, 2014.
[8] Zhenyuan Dong and Sai Qian Zhang. Ditas: Quantizing diffusion transformers via enhanced activation smoothing. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 6606-4615.. IEEE, 2025.
[9] Steven K Esser, Jeffrey I. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint arXiv: 1902.08/53, 2019.
[10] Michael Floater and Kai Hormann. Surface parameterization: tutorial and survey. Advances in multiresolution for geometric modelling, pages 157-186, 2005.
[11] Jordi Fornt, Pau Fontova-Musté, Martí Caro, Jaume Abella, Francesc Moll, Josep Altet, and Christoph Studer. An energy-efficient gemm-based convolution accelerator with on-the-fly im2col. IEEE Transactions Very Large Scale Integration (VI.SI) Systems, 31(11):1874-1875 2023.
[12] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. /7323, 2022.
[13] Yonggan Fu, Yuecheng Li, Chenghui Li, Jason Saragih, Peizhao Zhang, Xiaoliang Dai, and Yingyan Celine Lin. Auto-card: Efficient and robust codec avatar driving for real-time mobile telepresence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21036-21045, 2023.
[14] Ali Geris, Baris Cukurbasi, Murat Kiline, and Orkun Teke. Balancing performance and comfort in virtual reality: A study of fps, latency, and batch values. Software: Practice and Experience, 54(12):2336-2333 -34488 2024.
[15] Antonin Gilles, Pierre Le Gargasson, Grégory Hocquet, and Patrick Gioia. Holographic near-eye display with real-time embedded rendering. SIGGRAPH Asia 2023 Conference Papers, pages 1-10, 2023.
[16] Shangchen Han, Beibei Liu, Randi Cabezas, Christopher Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, et al. Megatrack: monochrome egocentric articulated hand-tracking for virtual reality. ACM Transactions on Graphics (ToG), 39 2020.
11

---

[17] Tairan He. Zhengyi Luo, Xialin He. Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humee to-humnood whole-body teleoperation and learning. arXiv preprint arXiv:2406.088856 2024.
[18] IVRHA. Survey of motion sickness mitigation efforts in virtual reality, https: iirhaa.iaaaaaaa. org/ survey-of mmotiinnsiccennn wit.eeee--iiiiiaatiioooooooooo-- Accessed: 2025-05-14.
wwwpratiiiioeeoootteeeeeeeeeeoee efforts- in- an-virtual-rrealiyy
2024.
[19] Hsiang Tsung Kung and Charles E Leiserson. Systolic arrays (for vlsi). In Sparse Matrix Proceedings 1978, volume 1, pages 256-282. Society for industrial and applied mathematics Philadelphia, PA, USA, 1979.
[20] HT Kung, Bradley McDanel, and Sai Qian Zhang. Packing sparse convolutional neural networks for efficient systolic array implementations: Column combining under joint optimization. In Proceedings of the Twenty-Fourd Fourth International Conference Architectural Support for Programming Languages and Operating Systems, pages 821 921-834, 2019.
[21] John Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. Practice and theory of blendshape facial models. Eurographics (State of the Art Reports), 188):2, 2014.
[22] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized Ilms. Advances in Neural Information Processing Systems, 37:87766-87800, 2024.
[23] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[24] Chiao Liu, Lyle Bainbridge, Andrew Berkovich, Song Chen, Wei Gao, Tsung-Hsun Tsai, Kazuya Mori, Rimon Ikeno, Masayuki Uno, Toshiyuki Isozaki, et al. A 4.6 μm, 512x 512, ultra-low power stacked digital pixel sensor with triple quantization and 127db dynamic range. In 2020 IEEE International Electron Devices Meeting (IEDM), pages 16-1. IEEE, 2020.
[25] Kai Liu, Haotong Qin, Yong Guo, Xin Yuan, Linghe Kong, Guihai Chen, and Yulun Zhang. 2dquant: Low-bit post-training quantization for image super-resolution. Advances in Neural Information Processing Systems, 37:71068-71000 1068-71884 2024.
[26) Wenxuan Liu, Budmonde Duinkharjav, Qi Sun, and Sai Qian Zhang. Fovealnet: Advancing ai-driven gaze tracking solutions for efficient foveated rendering in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2025.
[27] Wenxuan Liu and Sai Qian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprint arXiv:2405. 19751, 2024.
[28] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantization with learned rotations. arXiv preprint wwXiv:2405.16400 2024.
[29] Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. Deep appearance models for face rendering. ACM Transactions on Graphics (ToG), 37(4):1-13, 2018.
[30] Shugao Ma, Tomas Simon, Jason Saragih, Dawei Wang, Yuecheng Li. Fernando De La Torre, and Yaser Sheikh. Pixel codec avatars. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 64-73, 2021.
[31] Rafal K Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney. Fovvideovdp: A visible difference predictor for wide feld-of-view video. ACM Transactions Graphics (TOG), 40(4):1-19, 2021.
12

---

[32] Sage L Matthews, Alvaro Uribe- Quevedo, and Alexander Theodorou. Rendering optimizations for virtual reality using eye-tracking. In 2020 22nd symposium on virtual and augmented reality (SVR), pages 398-405 IEFF, 2020.
[33] Jian Meng, Yuecheng Li, Chenghui Li, Syed Shakib Sarwar, Dilin Wang, and Jae-sun Seo. Poca: Post-training quantization with temporal alignment for codec avatars. In European Conference on Computer Vision, pages 230-246. Springer, 2024.
[34] Meta. Meta Quest Pro. https://www.cr meta. httptatecocccuuetuuutteeepeee 2022. Accessed: 2025-07-21.
[35] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International conference on machine learning, pages 7197-7206. PMLR, 2020.
[36] Divya Nimma. Image processing in augmented reality (ar) and virtual reality (vr). International Journal on Recent and Innovation Trends in Computing and Communication, 12(2):475 2024.
[37] NVIDIA. Jetson Orin. .tmm n.d.
900-
[38]
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 1(10):e3,001 2016.
[39] Qualcomm Technologies, Inc. hhapg//sssseeennngggoooa///o//oo
com/ 2023.
Accessed: 2025-05-16.
[40] Qualcomm Technologies, Inc. Qualcomm ai hub. https://aihub..ual.. com/ 2025. Accessed: 2025-05-15.
[41] Gabriel Schwartz, Shih-En Wei, Te-Li Wang, Stephen Lombardi, Tomas Simon, Jason Saragih, and Yaser Sheikh. The eyes have it: An integrated eye and face model for photorealistie facial animation. ACM Transactions on Graphics (TOG), 39()-911-, 2020.
[42] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprint 2.0.::2300.11177,, 2023.
[43] Sangeetha Siddegowda, Marios Fournarakis, Markus Nagel, Tijmen Blankevoort, Chirag Patel, and Abhijit Khobare. Neural network quantization with ai model efficiency toolkit (aimet). arXiv preprint arXiv:2201.08444 000442004422221 2022.
[44] Xiaoyu Sun, Xiaochen Peng.
Sai Qian Zhang, Jorge Gomez, Win-San Khwa, Syed Shakib Sarwar, Ziyun Li, Weidong Cao, Zhao Wang, Chiao Liu, et al. Estimating power, performance, and area for on-sensor deployment of ar/vr workloads using an analytical framework. ACM Transactions on Design Automation of Electronic Systems, 29(6):1227, 2024.
[45] Alexander Tanchenko. Visual-psnr measure of image quality. Journal of Visual Communication and Image Representation, 25(5):874 878, 2014.
[46] Hanzhang Tu, Ruizhi Shao, Xue Dong. Shunyuan Zheng, Hao Zhang, Lili Chen, Meili Wang, Wenyu Li, Siyan Ma, Shengping Zhang, Boyao Zhou, and Yebin Liu. Tele-aloha: A low-budget and high-authenticity telepresence system using sparse rgb cameras. In Proceedings of the ACM SIGGRAPHI Conference, 2024.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Haiyu Wang, Wenxuan Liu, Kenneth Chen, Qi Sun, and Sai Qian Zhang. Process only where you look: Hardware and algorithm co-optimization for efficient gaze-td foveated rendering in virtual reality. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, pages 344-358, 2025.
13

---

[49] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing. 13(4):600- 612, 2004.
[50]
Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson, Rohan Bali, Danielle Belko, Eric Brock- meyer, Lucas Evans, Timothy Godisart, Hyowon Ha, Xuhua Huang, Alexander Hypes, Taylor Koska, Steven Krenn, Stephen Lombardi, Xiaomin Luo. Kevyn McPhail, Laura Millerschoen, Michal Perdoch, Mark Pitts, Alexander Richard, Jason Saragih, Junko Saragih, Takaaki Shira- tori, Tomas Simon, Matt Stewart, Autumn Trimble, Xinshuo Weng, David Whitewolf, Chenglei Wu, Shoou-I Yu, and Yaser Sheikh. Multiface: A dataset for neural face rendering. arXiv, 2022.
[51] Jingyang Xiang and Sai Qian Zhang. Dfrot: Achieving outlier-free and massive activation-free for rotated 1ims with refined rotation. arXiv preprint arXiv:2412.00648, 2024.
[52] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[53] Jin Xu, Zishan Li, Bowen Du, Miaomiao Zhang, and Jing Liu. Reluplex made more practical: Leaky relu. In 2020 IEEE Synposium on Computers and communications (ISCC), pages 1-7. IEEE, 2020.
[54] Jianchao Yang, Mei Wen, Junzhong Shen, Yasong Cao, Minjin Tang, Renyu Yang, Jiawei Fei, and Chunyuan Zhang. Bp-im2col: Implicit im2col supporting ai backpropagation on systolic arrays. In 2022 IEEE 40th International Conference on Computer Design (ICCD), pages 415-418. IEEE, 2022.
[55] Baoheng Zhang, Yizhao Gao, Jingyuan Li, and Hayden Kwok-Hay So. Co-designing sub-millisecond latency event-based eye tracking system with submanifold sparse enn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5771-5779, 2024.
[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[57] ZTE Corporation. Wi-fi 6 technology and evolution white paper. https: www zte. com. cn/
www.::/ttttoo--------o--- book/Wi-Fi. 6_ oe
http://wwwww......ooooooooooo//a http:/ihlllep/////tteteeeeeeeeee wwite.
htrr:eaaanddddtoiieeeeeee Evolution Whtppa_WWiieee__ee htt.p/toooo..ttaaooooorr00022220 pdf 2020.
Accessed: 2025-05-16.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our main claim are summarized in Section Sectior detailed explainations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We have Appendix A and B on this
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yes
Justification:
We explained our settings in Section 4 and hyperparameters in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:[Yor [es]]
Justification:
GitHub link offered in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
coiiioonoolcyy lc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuiu wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are summarized in Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We attach the training log in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We offer these details in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp://horrrpp..ccccbbb///hiaa
Answer: [Yes]
Justification: conform.
We have read and understood the code of ethics; and have done our best to
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification; We include a dedicated Broader Impacts paragraph on Section that describes positive applications of Codec Avatars in telepresence and remote collaboration.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide an anonymized GitHub repository (URL included the supple- mental material) that contains all newly introduced code and scripts.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:The paper does not involve crowdsoureing nor research with human subjects.
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21

---

A Proof of Scaling Invariance
Let the convolution layer take an input tensor X RC.*H*W and weight tensor W Row x xks, x For clarity we fix single output channel and suppress the output-channel index; the argument is identical for every output filter. With the usual definition of discrete convolution (+), the pre-activation output is
Gin Wld * X(c,
(1I)
where W[c] E Bkaxk and X[c] E & denote the e-th input-channel kernel and feature map, respectively. Choose positive scalars $1,..., SCi.. Define the scaled activations and the compensated weights
X|c= = 80 X(e,,
Wld -[c.
W(d.
e= e= Cin
(12)
8c
For any scalar a E R and tensors A, B of compatible shape, convolution is bilinear:
(aA) = a (A* B), A (aB) = a (A*
(13)
Using Equation
13 with 8e and 1/80:
( W(c) * (s.X(c)) = 1 de
(Wic) X(c) Wld X[d]
(14)
Summing Equation 14 over all input channels reproduces Equation
Cm
Oi
Σ
Wld [ = Σ wld * =
(15)
Channel-wise scaling of activations, paired with the reciprocal scaling of the corresponding kernels, leaves the convolution output unchanged. Hence y Y, proving the scale invariance claim.
B Proof of Fusing Scaling into Previous Layer Weights Let
(R) w CyuNRRu B()) CR
(16)
and define the
x = (1)) * (1) + B(1)
(17)
Let E RS be pertone scale, and write T for broadcast Hadamard multiplication along all trailing dimensions of a tensor whose first index has size Cour-
Claim.
(2) = 8 ( = (s @ (.....))) * ((1) + B(1)
(18)
Fix an output channel {1...., , Cour and spatial index (i.j). By definition of convolution with bias,
C₁ ky
(2)
w()
(1)
m.j-n.j-o + B())
(19)
m=1
u=0
v=0
Multiply Equation 19 by the scalar 8c:
X C.
Σ (sc
W(1)
((1)
mi--ujj- 8c B(1
(20)
22

---

The summation term in Equation 20 is exactly the (c, i, of the convolution (s W()) x while the final term is the (c, i, -entry of @ B(1) (broadcast spatially). Since (2) holds for every e, i. we obtain
2 x (s W()) * x +sOB())
(21)
which proves the claim.
Scaling each output channel by s can equivalently implemented by scaling the corresponding output-channel kernels in W(1) before convolution. In quantization inference-time fusion, this lets us absorb channel-wise activation rescaling into the layer's weights, avoiding an extra runtime operation.
C Proof of scaling invariance before and after im2col
Let X e PC be the activation tensor feeding Conv Transpose2d layer. im22col.) convert its argument to 2-D matrix in which every column is the receptive-fiele field patch that contributes to one output location.
X im2 Ko.).
(22)
where N = Honr W is the number of spatial sites produced by the layer. W E RCaxCouxXxas be the kernel of the transposed convolution, reshaped into
W mat reshape(W) e R (C,,,, (CC)))
(23)
After im2col, transposed convolution is a plain matrix multiply followed by col2im accumulation
Yes WT Xeol
(24)
mat
then Y col2im((Yot).
Let the per-channel scale vector be = [$1..... SC T with se Define Sact diag(s) II, K..
where /x, K.. is the identity matrix of size K₁ x K The Kronecker product @ constructs a block- diagonal matrix Sact, and Sact E (C,, channel e multiplied by the same scalar Sc.
K, k..) ( KnKw), Every column block belonging to
We scale activations and invert-scale the weights X = Sact and Wmntt Sect Wo-dmman mmtt
Propagating the scaled quantities through the algebra 24
Yool = WT Whaat = (Seet (S-tWmalk (mar) (oactXoot) (
(25)
From Sact is diagonal, S-1 = 8-1T
Yeol WT W S S-1 Sact- TT
mattXxoo Y
(26)
23


Reviews:
[{"review_id": "UhHi4J5huz", "reviewer": "Reviewer_6fss", "summary": "The paper introduces a framework designed to optimize photorealistic codec avatars for real-time execution on resource-constrained AR/VR devices. It focuses on post-training quantization techniques and a custom hardware accelerator to reduce the computational burden and latency associated with decoding. Specifically, the proposd ESCA integrates several techniques, including input channel-wise activation smoothing (ICAS), facial-feature-aware smoothing (FFAS), UV-weighted hessian-based weight quantization (UV-Weighted PTQ), and a hardware accelerator tailored to the specific needs of AR/VR devices. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "1. What is reason for the flickering artifacts around the edges between the hair and face in the final rendering?\n2. How do the proposed quantization techniques and the custom hardware accelerator contribute to the overall latency reduction individually?\n3. How sensitive is the proposed method to variations in the quality of the UV map?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.  It is interesting to integrate the algorithmic optimizations with a custom hardware accelerator for reducing latency in codec avatar rendering.\n2. The proposed techniques, e.g., ICAS, FFAS, UV-PTQ, are meaningful and well-motivated. They together provide an efficient way to reduce computational demands while maintaining high fidelity in avatar reconstruction, especially for critical facial features.\n3. The proposed method demonstrates obvious improvements in both visual quality and latency. \n4. The end-to-end pipeline runs at over 100 FPS on resource-constrained platforms. This is valuable for real-time AR/VR applications.\n\nWeaknesses:\n1. I carefully check the video results, and find that there are noticeable flickering artifacts (around the edges between the hair and face) in the final rendering, which could negatively impact the user experience in immersive environments.\n2. The contribution of each part of the pipeline in reducing latency is not very clear to me. More explaination is expected. \n3. Only the MultiFace dataset and a small number of avatars are used for evaluation. More diverse datasets and real-world scenarios are expected. This would help demonstrate the robustness and adaptability of the proposed method.\n4. I am also curious about how sensitive the proposed method is to the quality of the UV map.\n\nOverall, while I am not an expert in this specific task, I find the proposed method both interesting and meaningful. The combination of algorithmic and hardware optimizations is promising. Further clarification and broader evaluation would strengthen the paper. I would like to raise my score based on the authors' feedback.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns well, and I would like to keep the postive rating.", "rebuttal": "Thank you for your insightful comments. We have summarized your questions below and provided detailed responses to each. \n\n### Weaknesses and Questions\n### Q1. What is the reason for the flickering artifacts around the edges between the hair and face in the final rendering?\n\n**Ans:** Thanks for pointing this out, as discussed in lines 38-42, flickering in low-bit quantized models stems from unstable quantization of activation outliers. Our decoder's transposed-convolution layers process stochastic latents without normalization, creating heavy-tailed activation distributions (Figure 1). Hair-face boundaries are particularly challenging due to alternating dark/bright UV texels, generating the largest activation spikes that dominate the quantization range and manifest as flicker. \n\nESCA explicitly addresses this through Input Channel-wise Activation Smoothing (ICAS) which suppresses outlier-driven quantization error, and Facial-Feature-Aware Smoothing & UV-Weighted Quantization which strategically allocates precision to perceptually critical regions (eyes, mouth)(Eq. 4). This design leverages psychovisual research showing viewers are sensitive to facial artifacts [1,2]. While minor artifacts may remain at the edge under aggressive 4-bit quantization, we substantially improve stability where it matters most. Additionally, our method achieves +0.39 FovVideoVDP improvement over the best 4-bit baseline (Table 1). This metric specifically captures spatio-temporal artifacts, providing strong quantitative evidence of reduced flickering. \n\nSupplementary video and Figure 7 show baseline has severe flickering in key facial regions. ESCA limits flickering mainly to hair-face boundaries while maintaining stable, high-fidelity details elsewhere. Future work could refine training algorithms to further reduce these artifacts.\n\n[1] You look blocky, is everything alright? Influence of video distortions on facial expression recognition & quality\n\n[2] InternVQA: Advancing Compressed Video Quality Assessment with Distilling Large Foundation Model\n\n ### Q2. Clarify latency reduction of each pipeline component.\n\n**Ans:** Figure 3(b) illustrates the end to end execution flow, where all five components, Sensing, Encoding, Transmission, Decoding, and Rendering, are executed sequentially to process each sample. As detailed in Section 4.4 (lines 335-339), the latencies for Sensoring, Transmission, and Rendering are 1 ms, 5 ms, and 9.5 ms, respectively. Meanwhile, as discussed in Section 2.1 (lines 107-110) and shown in Figure 6(a), the inference times for Encoding and Decoding are 13.8 ms and 25.8 ms on Snapdragon XR2 Gen 2 (XR2-2) SoC. Consequently, the total latency per frame would be 55.1 ms for the original pipeline, corresponding to a throughput of 18.15 frames per second (FPS).\n\nIn contrast, Figure 6(b) illustrates the optimized pipeline, which introduces two major changes. First, the pipelines of two users are processed in an overlapped fashion to maximize throughput. The introduction of the accelerator provides an additional hardware resource dedicated to executing Encoding and Decoding, while the rendering operations can be assigned to the GPU to achieve maximum parallelism. In addition, Transmission and Decoding of each user are executed concurrently. At the same time, the decoder waits for corresponding transmission completion since it requires the other user's encoder output as input. This reduces end-to-end latency to 10 ms (2 $\\times$ 5 ms transmission time), achieving 100 FPS with significant latency and throughput improvements.\n\n### Q3. Evaluation use only MultiFace dataset.\n**Ans:** To the best of our knowledge, MultiFace is the only public Codec-Avatar dataset with pretrained models, enabling our post-training quantization experiments. Alternative datasets (Goliath [3], Tele-ALOHA [4]) provide only raw data requiring extensive training unfeasible within rebuttal timeframes [13], We will include additional evaluations in the final version of the paper once a suitable dataset with pretrained models becomes available.\n\nAlthough our study uses a single dataset, MultiFace offers a rigorous evaluation setting: 65TB with 13,000+ 2K frames per subject across 65 expressions and multiple viewpoints. The comprehensive coverage of extreme poses, occlusions, and high-frequency details creates a demanding testbed validating low-precision decoder performance. Current MultiFace evaluation represents the most scientifically sound approach given practical constraints while comprehensively validating quantization robustness.\n\nWhile we did not evaluate ESCA on additional datasets, we demonstrate its applicability to image super-resolution tasks, which aim to enhance the resolution of input images. For this task, although our UV-weighted PTQ cannot be extended to this domain due to the absence of UV coordinates, our smoothing can be used to facilitate the quantization over these super-resolution tasks.\n\nWe compare our method against GPTQ, the best-performing baseline from Codec-Avatar quantization, across five commonly used super-resolution benchmarks [5, 6, 7, 8, 9]. Our quantization experiments used the SwinIR [10] super-resolution model, which was pretrained on the combined DIV2K [11] and Flickr2K [12] datasets. \n\n| Method   | Bit | Set5 PSNR | Set5 SSIM | Set14 PSNR | Set14 SSIM | B100 PSNR | B100 SSIM | Urban100 PSNR | Urban100 SSIM | Manga109 PSNR | Manga109 SSIM |\n|----------|-----|-----------|-----------|------------|------------|-----------|-----------|----------------|----------------|----------------|----------------|\n| SwinIR R | 32  | 31.13   | 0.8967    | 27.45    | 0.7867     | 26.37   | 0.7434    | 25.15        | 0.7993         | 29.60        | 0.9115         |\n| ESCA     | 4   | 28.63   | 0.8310    | 25.85    | 0.7277     | 25.29  | 0.6885    | 22.87        | 0.6852         | 26.79        | 0.8088         |\n| GPTQ     | 4   | 27.90   | 0.7729    | 25.41    | 0.6776     | 24.74  | 0.6338    | 22.68       | 0.6462         | 25.94        | 0.7555         |\n\nESCA achieves superior 4-bit quantization performance compared to GPTQ across all five super-resolution benchmarks, with PSNR improvements ranging from 0.19 dB (Urban100) to 0.85 dB (Manga109) and SSIM improvements ranging from 0.039 (Urban100) to 0.0581 (Set5).\n\n[3] Codec avatar studio: Paired human captures for complete, driveable, and generalizable avatars\n\n[4] Tele-Aloha: a low-budget and high-authenticity telepresence system using sparse RGB cameras\n\n[5] Low-complexity single-image super-resolution based on nonnegative neighbor embedding\n\n[6] On single image scale-up using sparse-representations\n\n[7] A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics\n\n[8] Single image super-resolution from transformed self-exemplars\n\n[9] Sketch-based manga retrieval using manga109 dataset\n\n[10] Swinir: Image restoration using swin transformer\n\n[11] Ntire 2017 challenge on single image super-resolution: Methods and results\n\n[12] Enhanced deep residual networks for single image super-resolution\n\n[13] High-quality joint image and video tokenization with causal VAE\n\n### Q4.  How sensitive is the proposed method to variations in the quality of the UV map?\n\n\n**Ans:** To quantitatively assess UV map quality sensitivity, we conducted an ablation study by randomly dropping UV importance weights from 10% to 50% before 4-bit PTQ processing. This simulates progressively degraded UV priors. The table below shows the FovVideoVDP results on the three views renderings as we degrade the UV map quality.\n| UV Map Dropout Ratio               | Front View | Left View  | Right View  |\n|-----------------------|------------|--------------|--------------|\n| 50%           | 5.7828     | 4.9034       | 4.9084       |\n| 40%           | 5.8366     | 4.9718       | 4.9540       |\n| 30%           | 5.8421     | 4.9739       | 4.9549       |\n| 20%           | 5.8401     | 4.9756       | 4.9560       |\n| 10%             | 5.8454     | 4.9754       | 4.9582       |\n| ESCA  | 5.8541     | 4.9795       | 4.9605       |\nThe ablation study shows that ESCA maintains robustness despite declining UV map quality.\n\n### Q5.  Latency contributions of the quantization and hardware accelerator?\n\n**Ans:** The impact of the custom hardware accelerator on overall latency reduction is discussed in Section 4.4 (lines 327–337) and illustrated in Figure 6(a). A summary of the results is provided below:\n\n| Device   | Precision         | Latency (ms) |\n|---------|----------------|--------------|\n| Baseline Accelerator | INT8   | 45.09         |\n| Input-combing Accelerator | INT8   | 15.56         |\n| Input-combing Accelerator | INT4   |  6.18      |\n\nFor the contribution of quantization techniques, we only report in Section 2.1 (lines 107-111) that the inference latency of the full-precision Codec Avatar model on XR2-2 is 39.6 ms. Due to space constraints, the performance of quantized models was not included in the paper. We provide the detailed results here:\n\n\n| Device   | Precision         | Latency (ms) |\n|---------|----------------|--------------|\n| XR2-2 | FT32   | 39.6         |\n| XR2-2 | INT8   | 18.5         |\n| XR2-2 | W4A8   |  17.0      |\n\nW4A8 reflects current Qualcomm AI Hub capability limits (4-bit weights, 8-bit activations only) [14]. Overall, the proposed quantization techniques and custom hardware accelerator each contribute to a multi-fold reduction in inference latency.\n\n[14] Qualcomm AI Hub. (n.d.). Documentation. Qualcomm. app.aihub.qualcomm.com/docs/"}, {"review_id": "2V51p7CXjx", "reviewer": "Reviewer_p8Jg", "summary": "This paper proposes a new hardware chip and several quantization methods tailored for encoder decoder blocks common among avatar codecs.", "questions": "1. Creating a new custom hardware is quite expensive, and it usually requires very strong motivations and business justifications. I think for proposing such a new hardware, the paper needs to first show the case for not being able to leverage existing support on AI accelerators for sparse operations. For example, similar sparse data pattern also show up in variants of attention heads. How does the proposed architecture compare with those baselines with all sparsity features leveraged?\n\n 2. Are the latency results in Figure 6.a comparing Snapdragon XR2 Gen2 with NVIDIA A100? In general, lines 287-292 need further clarification about the choices in the benchmarking hardware and more explanation of the reasoning behind it. \n\n3. What prevents your proposed hardware from further reducing the latency?\n\n4. Do you have any evaluation or back of the envelope calculations re the energy consumption comparisons with baselines?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The optimization techniques introduced for avatar generation are cleverly designed and their effectiveness is well-supported by informed evaluation metrics, successfully enabling 4-bit quantization. The problem that this paper is trying to address is timely and has established real-world applications. \n\nHowever, the paper still has some primary weaknesses too:\n\n    - Insufficient Justification for Custom Hardware: Proposing custom hardware is a significant undertaking that requires strong justification. The authors should first demonstrate why existing AI accelerators with support for sparse operations are inadequate. For instance, a comparison with baselines that leverage sparsity features is necessary to validate the need for a new architecture.\n\n    - Unclear Benchmarking: The latency results presented in Figure 6a are ambiguous. It is not clear whether the comparison between a Snapdragon XR2 Gen 2 and the NVIDIA GPU is fair in terms of energy consumption and chip technology price point. \n\nAdditionally, a minor point is that the paper's title, \"Algorithm and Hardware Co-Optimization,\" may be a misnomer. The contributions appear to be separate optimizations at the algorithm (quantization) and hardware levels, rather than a true co-optimization where the design of each is mutually influential. The proposed hardware does not seem specifically tailored to the novel quantization techniques.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We have summarized your questions below and provided detailed responses to each.\n\n### Weaknesses and Questions\n### Q1. Custom hardware needs stronger motivation and justification for why existing AI accelerators cannot be leveraged.\n\n**Ans:** Codec Avatar represents a transformative paradigm for next-generation social interaction, enabling users in distant locations to communicate in VR as naturally as if they were face-to-face. Leading companies such as Meta and Apple have invested heavily in this technology, recognizing its critical role in the future of spatial computing and virtual telepresence. Meta’s photorealistic avatar research and Apple’s Vision Pro platform both showcase Codec Avatars as foundational to immersive, high-fidelity communication. As these systems evolve into core components of digital life, Codec Avatars are expected to operate as _always-on_ applications, running continuously in the background to support persistent, real-time interaction in social, educational, and professional settings. Their ability to deliver low-latency, lifelike experiences positions them as essential infrastructure for the next era of computing.\n\nExecuting Codec Avatar on general-purpose hardware such as GPUs is resource- and power-constrained, resulting in high inference latency of 39.6 ms (lines 107–111), whereas a seamless user experience typically requires 90 FPS [1, 2]. Moreover, GPUs must concurrently support other demanding tasks such as rendering and AI workloads. Therefore, we argue that designing a dedicated hardware accelerator specifically for Codec Avatar applications is both beneficial and necessary to meet real-time performance and user experience requirements.\n\nFurthermore, Codec Avatar models require dedicated hardware due to their unique computational characteristics. The specialized computation flow illustrated in Figure 6(b) necessitates customized computation patterns and scheduling that existing AI accelerators cannot provide. Additionally, conventional AI accelerators are developed without considering VR system integration constraints and compatibility requirements. Most critically, as discussed in Section 2.3, the transposed convolution layers in the Codec Avatar decoder pose a significant challenge due to their inherent structured sparsity (lines 143–147), which substantially reduces the utilization rate and efficiency of the conventional AI accelerator. In contrast, our proposed input-combining accelerator effectively exploits the checkerboard-like structured sparsity in the input activations following the im2col transformation (lines 151–154), leading to a more efficient execution.\n\n[1] Balancing performance and comfort in virtual reality: A study of FPS, latency, and batch values\n\n[2] Survey of motion sickness mitigation efforts in virtual reality\n\n### Q2. Unclear Benchmarking about the comparison between Snapdragon SoC and NVIDIA GPU.\n\n**Ans:** We apologize for the ambiguity.  The A100 GPU (line 287) was used only for offline PTQ calibration, not runtime benchmarking.\n\nAs for model inference, Figure 6 (a) compares three configurations:\n\n| Device                          | Description                                                  |\n|---------------------------------|--------------------------------------------------------------|\n| XR2-2 NPU        | Current commercial baseline (42.04 ms)                       |\n| Baseline Accelerator            | Weight‑stationary dataflow design                            |\n| Our Accelerator                 | Proposed custom hardware accelerator with input‑combining     |\n\nWe chose Snapdragon XR2 Gen 2(XR2-2) as the baseline because it is integrated within Meta Quest 3 headset, representing real-world deployment constraints.\n\nThe NVIDIA Tesla T4 (lines 289-292) is used to model rendering performance by downclocking to match the capabilities of mobile platforms.\n\nWe will clarify these hardware roles in the revised manuscript.\n\n### Q3.  Reason for the “Co-Optimization” in title.\n\n**Ans:** Our proposed system is explicitly co-designed across both the algorithm and hardware levels, with tightly coupled innovations that are jointly optimized for maximum efficiency.\n\nOn the algorithm side, we develop novel smoothing schemes specifically tailored for the Codec Avatar decoder quantization, including both 8-bit and 4-bit. These smoothing strategies are carefully designed to maintain high reconstruction fidelity while significantly reducing computational cost. On the hardware side, we propose a systolic array architecture with a customized multiply-accumulate unit tailored specifically for the quantized decoder model. The design is reconfigurable to support both $8\\times 8$-bit and $4\\times 4$-bit multiplication. Additionally, each processing element (PE) is optimized to exploit the checkerboard-like activation sparsity that arises from transposed convolution operations after the im2col transformation. Our design is not a generic sparse accelerator, but one that is highly specialized for our quantized Codec Avatar model. It is specifically built to operate efficiently on low-precision (INT8/INT4) data and structured sparse activations.\n\nMoreover, Figure 6(b) illustrates the optimized execution pipeline of Codec Avatar application. Once our quantization algorithm is applied, we can evaluate the inference latency (encoding and decoding) on the proposed custom hardware accelerator. Based on this evaluation, the optimal Codec Avatar working flow can be derived, where the end-to-end latency per frame is bounded by the interval between the two red dashed lines in Figure 6(b). This tight coupling between model quantization and hardware specialization exemplifies our algorithm–hardware co-optimization approach. \n\nSimilar way of co-optimizations has been adopted by a lot of other works. For example, in [3, 4, 5], the authors propose FABNet, a hardware-friendly variant of the Transformer architecture, adopting a unified butterfly sparsity pattern for both attention and feed-forward network (FFN) layers. Then, they design a reconfigurable butterfly accelerator, capable of runtime configuration to handle different layers using a unified hardware engine. Our proposed Codec Avatar system follows a similar philosophy. The design of each side was guided and constrained by the other, rather than being independently optimized.\n\n[3] Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design\n\n[4] Packing sparse convolutional neural networks for efficient systolic array implementations: Column combining under joint optimization\n\n[5] SCNN: An accelerator for compressed-sparse convolutional neural networks\n\n### Q4. What prevents your proposed hardware from further reducing the latency?\n\n**Ans:** The current latency of our proposed hardware accelerator is primarily limited by the size of the systolic array and the capacity of the on-chip buffers. Specifically, we employ a $16\\times 16$ systolic array for matrix multiplication, supported by on-chip buffers of 4096 KB for weights, 2048 KB for activations, and 1024 KB for accumulators. While increasing the array size could further reduce latency, it would also result in greater chip area consumption. Under a fixed chip area constraint, this would limit the resources available for other hardware components. Therefore, the array size is carefully selected to achieve a balanced trade-off between performance and energy efficiency.\n\nAdditionally, it is important to note that even if the systolic array and memory size are further increased, the per-frame processing latency of ESCA will not be reduced, as the computation is no longer the primary bottleneck. As discussed in Section 4.4 (lines 333-348), the end-to-end frame latency of Codec Avatar application is constrained by Transmission latency. As illustrated in Figure 6(b), the time between the two red dashed lines represents the effective per-frame latency, which equals twice the Transmission latency (10 ms). Further, if the transmission latency gets reduced, the Rendering stage, currently taking 9.5 ms, would become the new bottleneck.\n\n### Q5. Evaluation regarding the energy consumption comparisons with baselines.\n\n**Ans:** For the custom hardware accelerator, we summarize the estimated energy consumption per frame of model inference in the following table:\n\n| Device   | Precision         | Energy (uJ) |\n|---------|----------------|--------------|\n| Baseline Accelerator | INT8   | 11010.09         |\n| Input-combining Accelerator | INT8   | 3549.64       |\n| Input-combining Accelerator | INT4   |  1653.09      |\n\nAs for the XR2-2, there is currently no official disclosure of the absolute power consumption of its on-chip NPU. However, based on available research report [6], typical AI inference power for mobile NPU is approximately 500 mW. Using 0.5 W as a conservative estimate, we perform a back-of-the-envelope calculation by multiplying the power with the measured inference latency to estimate energy consumption:\n\n| Device   | Precision         | Energy (uJ) |\n|---------|----------------|--------------|\n| XR2-2 NPU | FT32   | 19800         |\n| XR2-2 NPU | INT8   | 9250       |\n| XR2-2 NPU | W4A8   |  8500      |\n\nNotably, the W4A8 reflects the current capability of the Qualcomm AI Hub, which supports 4-bit weights and 8-bit activations [7]. Our ESCA accelerator achieves significantly lower energy consumption compared to both the baseline accelerator and the commercial XR2-2 SoC.\n\n[6] Deep Learning on Mobile Devices With Neural Processing Units\n\n[7] Qualcomm AI Hub. (n.d.). Documentation. Qualcomm. app.aihub.qualcomm.com/docs/"}, {"review_id": "TeZh5zQ4Le", "reviewer": "Reviewer_52i2", "summary": "ESCA presents a full-stack framework for running photorealistic Codec Avatars in real-time on resource-constrained AR/VR devices. Majorly targeting on the VAE decoder side, it aims to addresse the quantization and efficiency challenges of transposed convolution-heavy decoders by co-designing algorithmic and hardware solutions. Key components include ICAS, FFAS, UV-weighted perceptual loss, and a custom DNN accelerator optimized for sparse patterns. ESCA achieves state-of-the-art 4-bit inference quality and enables 100 FPS rendering with 3.36× latency reduction.", "questions": "Current experiments and ablation studies are clear and effective in illustrating the contribution of each module to the performance gains. \nAre there alternative design choices that could further accelerate each component? and possibly to discuss the theoretical upper bound of achievable latency reduction under the current framework. \n\nAdditionally, it would be helpful to understand potential directions for mitigating remaining artifacts, such as flickering in hair regions and loss of detail around the eyes. \n\nHow complex can the avatar subject be, e.g., can the system support full-body avatars with expressive dynamics? Providing more architectural details (e.g., latent space dimensionality, image resolution, and overall model size) would help better assessment.", "limitations": "As mentioned in previous weakness section, the scope of impact is of one major concern. Grounding each module design in broader conceptual principles, for example, by discussing how these components might migrate to emerging avatar decoder architectures with predictive capabilities would significantly enhance the generality of the work and help situate the proposed method within a wider design landscape. Expanding the discussion to include upstream and downstream components would provide readers a more complete picture of the system-level context.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is clearly written and well-structured and easy to follow. The experimental evaluation is thorough. I can see the approach will directly addresses many practical deployment bottlenecks for high-fidelity codec avatar decoding in AR/VR, if current architecture is fixed. \nOverall, ESCA presents a well-integrated co-design strategy for adapting quantization and accelerator design specifically to the decoder side of Codec Avatars - ICAS and DNN Hardware Accelator are reasonably adapted, while UV-weighted quantization and facial-feature-aware smoothing are novel in combining perceptual significance with quantization robustness. \n\nWhile the proposed framework is well-optimized for Codec Avatar decoders, my main concern is that its scope appears tightly coupled to this specific architecture and application. Any substantial change in the decoder structure may necessitate rethinking or redesigning key components of the system. The demonstrated rendering results (in supp) still exhibit noticeable flickering artifacts (e.g., hair regions). This suggests a degree of reliance on human priors when selecting important regions and likely not generalize well to other subject types, e.g., even for full-body avatars, where semantic saliency and deformation patterns differ. \nExperiments on larger-scale avart datasets would also be necessary.\n\nSome figures (e.g., Figure 1, 2) are overly compressed within the text, making them slightly difficult to read.", "was_revised": true, "final_justification": "After reading the rebuttal, I'll maintain my score.", "rebuttal": "Thank you for your insightful comments. Below, we have summarized your questions and provided detailed responses to each one.\n\n### Weaknesses and Questions\n\n### Q1. While the proposed framework is well-optimized for Codec Avatar decoders, my main concern is that its scope appears tightly coupled to this specific architecture and application. \n\n**Ans:** We agree that ESCA is specialized by design, this is essential for addressing the challenging task of real-time photorealistic avatar inference on mobile AR/VR hardware. The concept of hardware-algorithm co-design has been widely adopted to support other AR/VR applications as well, such as image rendering [1–4], demonstrating its effectiveness in meeting stringent performance and efficiency requirements.\n\nMoreover, our core techniques have broader applicability. Input Channel-wise Activation Smoothing addresses activation outliers in generative models using transposed convolutions followed by non-linearities, which is a typical pattern in VAEs and GANs. Our channel-wise scaling and weight fusion method (Sec. 3.1, Eq. 2) is architecture-agnostic. Input-combining hardware optimization exploits the inherent checkerboard sparsity pattern from transposed convolutions after im2col transformation (Fig. 3a). This sparsity is a fundamental property of the operation itself, making our accelerator efficient for any decoder heavily using transposed convolutions for upsampling. Perceptually Guided Quantization (Sec. 3.2/3.3) is the most domain-specific component, designed to leverage facial UV priors and concentrate precision on perceptually critical regions for low-bit quantization.\n\nSimilar to how ray tracing accelerators are optimized for specific traversal patterns [2] and 3D reconstruction systems codesign algorithms with hardware [1], codec avatars require specialized optimization due to their real-time requirements. General purpose solutions cannot achieve the 90+ FPS needed for immersive VR on resource constrained hardware.\nIn general, ESCA offers valuable insights that can be applied to the optimization of other AI models across a range of tasks.\n\n[1] Fusion-3d: Integrated acceleration for instant 3d reconstruction and real-time rendering\n\n[2] CoopRT: Accelerating BVH Traversal for Ray Tracing via Cooperative Threads\n\n[3] AQB8: Energy-Efficient Ray Tracing Accelerator through Multi-Level Quantization\n\n[4] Neural rendering and its hardware acceleration: A review\n\n### Q2. The demonstrated rendering results (in supp) still exhibit noticeable flickering artifacts. Experiments on larger-scale avatar datasets would also be necessary.\n\n**Ans:** We agree that our approach leverages semantic priors, which is an executive design choice to address the core challenge of 4-bit quantization for high-fidelity generative models. Under aggressive 4-bit quantization, perfect reconstruction everywhere is challenge. Our approach strategically allocates limited precision to perceptually critical facial regions (eyes, mouth) where users focus attention and convey emotion. This leverages psychovisual research showing viewer sensitivity to facial artifacts [5, 6].\n\nRegarding generalization: the principle of semantic-guided quantization is highly adaptable. For full-body avatars, the same methodology applies using full-body UV maps and prioritizing other salient regions (face, hands). Our work establishes this perceptually-guided PTQ framework for facial avatars, demonstrating a methodology extensible to other domains.\n\nRegarding dataset scale: MultiFace contains over 13,000 2k-resolution frames per subject across  65 diverse expressions, totaling 65TB, making it the largest available dataset with pretrained models. By comparison, FaceScape [7] supplies total 18760 textured meshes and requires approximately 350 GB of storage. 4DFAB [8] records 1.8 M high‑res meshes across repeated sessions and takes around 200 GB.\n\nDue to the characters restriction, more information is available at our response to Reviewer 6fss's Question 3 regarding additional evaluation concerns.\n\n[5] Influence of video distortions on facial expression recognition & quality\n\n[6] InternVQA: Advancing Compressed Video Quality Assessment with Distilling Large Foundation Model\n\n[7] Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction\n\n[8] 4dfab: A large scale 4d database for facial expression analysis and biometric applications\n\n### Q3. Are there alternative design choices that could further accelerate each component? \n\n**Ans:** Yes.  While our current framework represents a highly optimized design point, there are promising avenues for further acceleration, both at the algorithmic and hardware levels.\n\nFor the Algorithm: \nOne could explore more aggressive mixed-precision quantization, using our post components to automatically assign different bit widths to layers. Our quantization methods are largely orthogonal to and can be combined with other efficiency techniques like network pruning. One could first apply pruning and then apply our PTQ pipeline to the pruned model. While our current paper focuses specifically on advancing the state of the art in quantization for this domain, our method is designed to be a compatible component within a broader optimization toolkit.\n\nFor the Hardware: \nWithout any optimization, the latency is bottlenecked by the computation components of the framework (lines 108-111), namely, Encoding and Decoding. Therefore, our accelerator focuses on accelerating them where we use a systolic array, which is well known to be effective in operating matrix multiplication, and widely adopted in commercial AI accelerators such as Google’s TPU. Furthermore, we propose a novel input-combining mechanism to exploit the structured sparsity of the activation matrix, which significantly optimizes the performance of the accelerator.\n\nA further design can be applied on top of ours, because now we notice that after optimizing the computation, the bottleneck becomes the Transmission (lines 333-348). Therefore, further work can be done on minimizing the Transmission latency. \n\n### Q4. The theoretical upper bound of achievable latency reduction.\n\n**Ans:** Our decoder acceleration reduces processing latency from 25.68 ms to 3.13 ms. However, as discussed in Section 4.4 (lines 333–348), the end-to-end frame latency of the Codec Avatar application is no longer constrained by computation but by Transmission latency. As shown in Figure 6(b), the effective per-frame latency, indicated by the interval between the two red dashed lines, is determined by twice the Transmission latency, totaling 10 ms. Without optimizing the Transmission latency, this will remain the upper bound for ESCA’s overall latency, regardless of further improvements in computation speed.\n\n\n### Q5. Potential directions for mitigating remaining artifacts.\n\n\n**Ans:** Here are several promising future directions to build upon our work:\nOur current work uses uniform 4-bit or 8-bit quantization. A powerful extension would be to apply mixed precision, where different layers are assigned different bit-widths based on their sensitivity. For instance, layers responsible for fine facial features could maintain higher precision (8-bit) while background synthesis layers use aggressive 4-bit quantization, potentially reducing flickering in hair regions while preserving eye detail.\n\n### Q6. How complex can the avatar subject be, e.g., can the system support full-body avatars with expressive dynamics? Providing more architectural detail.\n\n**Ans:** Our work targets facial avatars which is critical for social VR. The framework can extend to full-body avatars: UV-Weighted PTQ works with any UV texture map, and our accelerator optimizes transposed convolutions used across generative models.\n| Output texture resolution                  | 1024×1024×3 (Sec. 4.1, line 278) |\n|--------------------------------------------|----------------------------------|\n| Latent space              | 256×1                          |\n| Image resolution after rendering           | 2048×1334×3                    |\n| Encoder size                               | 6.79 M                         |\n| Decoder size                               | 24.68 M                        |\n| Number of transposed convolution layers    | 8                              |\n| Kernel size of transposed convolution layers | 4×4 (stride: 2, padding: 1)    |\n| Total model size                           | 31.47 M                        |\n\nWe will add these details in the next version to improve clarity. The computational bottleneck is the decoder's transposed convolutions, not parameter count. \n\n### Q7. The scope of impact is of one major concern. Expanding the discussion to include upstream and downstream components would provide readers a more complete picture of the system-level context. \n**Ans:**  Our method's design patterns can extend to emerging predictive decoders like Auto-CARD's LATEX framework [paper citation 12] and the dual-path architecture of ReliaAvatar [9], enabling integrated Codec Avatar pipelines.\n\nUpstream Integration: Our quantized decoder efficiently processes compressed representations from multiple sensing modalities (3D Gaussian splatting, NeRF, ray tracing, eye/gaze/hand tracking, long-context video understanding) while maintaining quality and reducing computational bottlenecks in sophisticated input processing pipelines.\n\nDownstream Benefits: Quantized latent representations preserve detail for high-quality rendering across diverse display technologies. By reducing decoder latency from 42.04ms to sub-10ms, we enable >90 FPS real-time rendering required for immersive VR while preventing frame drops and maintaining visual quality across display hardware.\n\n[9] Reliaavatar: A robust real-time avatar animator with integrated motion prediction\n### Paper Format\nWe will carefully adjust the layout and sizing of all figures, particularly Figures 1 and 2."}, {"review_id": "0hlyLMZt4e", "reviewer": "Reviewer_CJWi", "summary": "The paper presents ESCA, a framework for accelerating Code Avatar inference on AR/VR devices. Authors concentrate on two challenges: preserving fidelity during 4-bit quantization of VAE  and inefficient inference of transposed convolutions on target devices. They propose a number of techniques, including two smoothing techniques, uv-weightd post-training quantization and designing a custom accelerator. The resulting pipeline runs under impressive 100 fps while preserving and even improving a bit the visual quality.", "questions": "1. what is offline time and memory footprint of the techniques you use\n2. how well does it perform on spontaneous facial exprresions?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strenghts:\n* achieves real-time 100 fps on device\n* new accelerator design that exploits structures sparsity\n* paper addresses both algorithmic and hardware problems\n* clear system diagrams, demonstrating the whole pipeline picture\n\n\nWeaknesses:\n* Little qualitative results. For example, it is especially intresting how facial feature-aware smoothing actually affects the visual quality of the important features (eyes, mouth) compared to others\n* limited discussion of training cost and calibration overhead for ICA/UV-PTQ\n* evaluation is limited to MultiFace dataset", "was_revised": true, "final_justification": "After reading the rebuttal, I raise my score. The authors show their smoothing does not change the convolution output and keeps key facial details; the eye/mouth metrics improve over baselines. The quantization is post-training and light with clear memory figures. The system reaches ~100 fps on device and keeps quality, including for spontaneous expressions.", "rebuttal": "Thank you for your insightful comments. We have summarized your questions below and provided detailed responses to each.\n\n### Weaknesses and Questions\n ### Q1. Little qualitative results. For example, it is especially interesting how facial feature-aware smoothing actually affects the visual quality of the important features (eyes, mouth) compared to others.\n**Ans:** Smoothing alone does not impact visual quality, as mathematically proven in our paper. It significantly reduces quantization noise by removing outliers, yet the process is effectively lossless—it does not alter the output of the convolution operation. Specifically, FFAS preserves critical facial details by identifying channels responsible for high-frequency features (eye wrinkles, lip creases) and exempting them from aggressive smoothing (Sec 3.2, lines 209-213). We analyzed reconstruction quality specifically within eye and mouth regions using facial masks. Results show FFAS-UV significantly outperforms both our ablation (ICAS-UV) and baseline methods:\n\n| Method             | Front PSNR | Front SSIM | Left PSNR | Left SSIM | Right PSNR | Right SSIM |\n|--------------------|------------|------------|-----------|-----------|------------|------------|\n| **FFAS‑UV (Ours)** | 35.501 | 0.920   | 34.255 | 0.919  | 34.040  | 0.920   |\n| **ICAS‑UV (w/o FFAS)** | 34.040 | 0.919 | 32.739 | 0.917  | 32.200021  | 0.918   |\n| **Baseline**       | 32.732  | 0.904   | 31.907 | 0.900  | 31.244  | 0.900   |\n\nThis demonstrates FFAS successfully preserves visual quality in the most perceptually important facial features. We'll add more qualitative results in the next version due to rebuttal figure restrictions.\n\n### Q2. Limited discussion of training cost and calibration overhead for ICA/UV-PTQ\n\n**Ans:** We would like to emphasize a key aspect of our framework: it is entirely post-training quantization (PTQ) and requires no expensive gradient-based training or backpropagation. The overhead is a one-time, offline calibration process. The calibration process is highly efficient. For each layer in the decoder, the overhead on an NVIDIA A100 GPU is as follows, using a calibration set of 512 avatar frames:\n\n| Stage                   | Description                                                 | Approx. Time per Layer     |\n|------------------------|-------------------------------------------------------------|----------------------------|\n| UV‑W Mask Generation   | Generate UV‑weighted importance mask from 512 samples.      | ~ 30 seconds (one‑time cost) |\n| ICAS Calibration       | Single forward pass over 512 samples (batch size 16) to find smoothing factors. | ~ 3 minutes               |\n| Weight Quantization    | Apply UV‑weighted Hessian and solve for the optimal quantized weights. | < 1 minute                 |\n| Total Calibration Cost | Offline, one‑time process                                     | ~ 4 minutes per layer      |\n\nWe will include a detailed breakdown of these calibration costs in the Appendix of the next version to enhance clarity. More discussion is at Question 4.\n\n### Q3. Evaluation is limited to MultiFace dataset\n\n**Ans:** To the best of our knowledge, MultiFace is the only public Codec-Avatar dataset with pretrained models, enabling post-training quantization experiments. Alternative datasets (Goliath [1], Tele-ALOHA [2]) provide only raw data requiring extensive training unfeasible within rebuttal timeframes [11]. \n\nDespite single dataset use, MultiFace provides rigorous evaluation: 65TB with 13,000+ 2K frames per subject across 65 expressions and multiple viewpoints. The comprehensive coverage of extreme poses, occlusions, and high-frequency details creates a demanding testbed validating low-precision decoder performance.\n\nWe plan broader evaluation once additional pretrained models become available. Current MultiFace evaluation represents the most scientifically sound approach given practical constraints while comprehensively validating quantization robustness.\n\nAlthough we did not evaluate ESCA on other codec avatar tasks, we show the applicability of ESCA to image super-resolution tasks, whose goal is to increase the resolution of an image. Although our UV-weighted PTQ cannot be extended to this domain due to the absence of UV coordinates, our smoothing can be used to facilitate the quantization over these super-resolution tasks.\n\n\nWe compare our method against GPTQ, the best-performing baseline from Codec-Avatar quantization, across five commonly used super-resolution benchmarks [3, 4, 5, 6, 7]. Our quantization experiments used the SwinIR [8] super-resolution model, which was pretrained on the combined DIV2K [9] and Flickr2K [10] datasets. \n\n| Method   | Bit | Set5 PSNR | Set5 SSIM | Set14 PSNR | Set14 SSIM | B100 PSNR | B100 SSIM | Urban100 PSNR | Urban100 SSIM | Manga109 PSNR | Manga109 SSIM |\n|----------|-----|-----------|-----------|------------|------------|-----------|-----------|----------------|----------------|----------------|----------------|\n| SwinIR R | 32  | 31.13 dB  | 0.8967    | 27.45 dB   | 0.7867     | 26.37 dB  | 0.7434    | 25.15 dB       | 0.7993         | 29.60 dB       | 0.9115         |\n| ESCA     | 4   | 28.63 dB  | 0.8310    | 25.85 dB   | 0.7277     | 25.29 dB  | 0.6885    | 22.87 dB       | 0.6852         | 26.79 dB       | 0.8088         |\n| GPTQ     | 4   | 27.90 dB  | 0.7729    | 25.41 dB   | 0.6776     | 24.74 dB  | 0.6338    | 22.68 dB       | 0.6462         | 25.94 dB       | 0.7555         |\n\nESCA achieves superior 4-bit quantization performance compared to GPTQ across all five super-resolution benchmarks, with PSNR improvements ranging from 0.19 dB (Urban100) to 0.85 dB (Manga109) and SSIM improvements ranging from 0.039 (Urban100) to 0.0581 (Set5).\n\n[1] Codec avatar studio: Paired human captures for complete, driveable, and generalizable avatars\n\n[2]  Tele-Aloha: a low-budget and high-authenticity telepresence system using sparse RGB cameras\n\n[3] Low-complexity single-image super-resolution based on nonnegative neighbor embedding\n\n[4] On single image scale-up using sparse-representations\n\n[5] A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics\n\n[6] Single image super-resolution from transformed self-exemplars\n\n[7] Sketch-based manga retrieval using manga109 dataset\n\n[8] Swinir: Image restoration using swin transformer\n\n[9] Ntire 2017 challenge on single image super-resolution: Methods and results\n\n[10] Enhanced deep residual networks for single image super-resolution\n\n[11] High-quality joint image and video tokenization with causal VAE\n\n### Q4. What is the offline time and memory footprint of the techniques you use?\n\n**Ans:** ESCA adopts a post-training quantization (PTQ) approach, which involves a one-time, offline calibration process that is significantly faster and more resource-efficient than full Quantization-Aware Training (QAT). To provide a clear summary, we have detailed the approximate calibration time and peak memory usage for our model, conducted on a single NVIDIA A100 GPU.\n| Resource Metric            | Description                                                              | Approximate Cost           |\n|---------------------------|--------------------------------------------------------------------------|----------------------------|\n| Calibration Time (per layer) | Time to calibrate one layer (ICAS + UV‑W PTQ) using 512 samples.       | < 4 minutes per layer      |\n| Calibration Time (total)     | Total one‑time, offline calibration time for the entire network.       | ~ 1 hour                   |\n| QAT Time                     | Total training time with A100×4 GPU setup.                             | > 1 day                    |\n| Calibration Memory Footprint | Peak GPU memory usage during the offline calibration process.           | ~ 60 GB                    |\n| QAT Memory Footprint         | Peak GPU memory usage during the QAT process.                          | ~ 200 GB                   |\n\nFor the inference of our model, we have also detailed the estimated peak memory usage.\n| Model                 | Device                   | Source               | Peak Memory Usage |\n|-----------------------|--------------------------|----------------------|--------------------|\n| Encoder               | Snapdragon XR2 Gen 2     | Qualcomm AI Hub      | ~ 39 MB            |\n| Decoder               | Snapdragon XR2 Gen 2     | Qualcomm AI Hub      | ~ 50 MB            |\n| Encoder & Decoder     | Hardware Accelerator     | Hardware Simulation  | ~ 7 MB             |\n\nPTQ delivers 24x faster calibration with 70% less memory than QAT, while custom hardware achieves 87% memory reduction over mobile processors.\n\n### Q5. How well does it perform on spontaneous facial expressions?\n\n**Ans:** We evaluated spontaneous facial expressions by selecting three expressions from the MultiFace dataset's 65 expressions and computing FovVideoVDP scores against ground truth across three viewpoints. \n\n| Method            | Shushing Front | Shushing Left | Shushing Right | Surprise Front | Surprise Left | Surprise Right | Frowning Front | Frowning Left | Frowning Right |\n|-------------------|----------------|---------------|----------------|----------------|----------------|-----------------|----------------|----------------|-----------------|\n| **FFAS‑UV**       | 5.7832         | 5.0065        | 5.0280         | 5.8486         | 5.0930         | 5.0989          | 5.8686         | 5.0210         | 5.0606          |\n| **Best Baseline** | 5.2911         | 4.4226        | 4.3828         | 5.4066         | 4.5072         | 4.4783          | 5.2713         | 4.4064         | 4.3877          |\n\nThe results demonstrate ESCA maintains high visual fidelity for spontaneous expressions, with consistent performance across viewing angles."}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field, with 1 Boarderline Reject, 2 Boarderline Accept and 1 strong Accept. After reviewing the rebuttal and discussion, most reviewers are inclined to acceptance. Reviewers appreciate that this paper proposes an interesting solution for light-weight Codec Avatar, with reasonable technical contributions like post-training quantization and a custom hardware accelerator. Thus, AC sided with the positive reviewers. AC highly urges the authors to go through the detailed comments carefully to polish the writing and tune down the vague illustration, and provide extra experimental details and  qualitative video results, so as to ensure the final acceptance.
Decision: Accept (poster)