DVI:A Derivative-based Vision Network for INR
Authors: Runzhao Yang
Xiaolong Wu
Zhihong Zhang
Fabian Zhang
Tingxiong Xiao
Zongren Li
Kunlun He
Jinli Suo
Venue: ICML 2025 poster (2025)
Keywords: Data Compression
Implicit Neural Representation
Primary area: applications->computer_vision
TL;DR: we propose DVI, a novel Derivative-based Vision network for INR, capable of handling a variety of vision tasks across various data modalities

Abstract: Recent advancements in computer vision have seen Implicit Neural Representations (INR) becoming a dominant representation form for data due to their compactness and expressive power. To solve various vision tasks with INR data, vision networks can either be purely INR-based, but are thereby limited by simplistic operations and performance constraints, or include raster-based methods, which then tend to lose crucial structural information of the INR during the conversion process. To address these issues, we propose DVI, a novel Derivative-based Vision network for INR, capable of handling a variety of vision tasks across various data modalities, while achieving the best performance among the existing methods by incorporating state of the art raster-based methods into a INR based architecture. DVI excels by extracting semantic information from the high order derivative map of the INR, then seamlessly fusing it into a pre-existing raster-based vision network, enhancing its performance with deeper, task-relevant semantic insights. Extensive experiments on five vision tasks across three data modalities demonstrate DVI's superiority over existing methods. Additionally, our study encompasses comprehensive ablation studies to affirm the efficacy of each element of DVI, the influence of different derivative computation techniques and the impact of derivative orders. Reproducible codes are provided in the supplementary materials.

DVI: A Derivative-based Vision Network for INR
Runzhao Yang" Xiaolong 2
Zhihong Zhang 13 Fabian Zhang' Tingxiong Xiao Zongren Li
Kunlun He
Jinli Suo 167
Abstract
1. Introduction
Recent advancements in computer vision have seen Implicit Neural Representations (INR) be- coming dominant representation form for data due to their compactness and expressive power. To solve various vision tasks with INR data, vi- sion networks can either be purely INR-based, but are thereby limited by simplistic operations and performance constraints, or include raster-based methods, which then tend to lose crucial structural features and important information of the INR during the conversion process. To address these issues, we propose DVI, novel Derivative- Vision network for INR, capable of handling a variety of vision tasks across various data modali- ties, while achieving the best performance among the existing methods by incorporating state of the art raster-based methods into a INR based archi- tecture. DVI excels by leveraging the valuable features captured in the high order derivative map of the INR, then seamlessly fusing them into pre-existing raster-based vision network, enhanc- ing its performance with additional, task-relevant structural information. Extensive experiments on five vision tasks across three data modalities demonstrate DVT's superiority over existing meth- ods. Additionally, our study encompasses com- prehensive ablation studies to affirm the efficacy of each element of DVI, the influence of different derivative computation techniques and the impact of derivative orders. Reproducible codes are pro- vided in the supplementary matcrials.
Implicit Neural Representation (INR) is novel form of data representation that models data through a mapping from coordinates to values. Unlike traditional raster repre- sentations, INR has the capability to model complex struc- tural patterns and relationships within the data (Xu et al., 2022; Costain et al., 2023; Zhou et al., 2023a; De Luigi et al., 2023; Ramirez et al., 2023; Navon et al.. 2023; Zhou et al.,, 2023b), making it extensively applicable in various vision data representations such images (Strümpler et al., 2022), 3D volumes (Wu et al., 2021), and videos (Sitzmann et al., 2020). This enhanced capacity to model complex structural features makes INR particularly suited for tasks where traditional pixel or voxel representations are limited by resolution and scalability.
For performing specific vision tasks the data in INR form, vision networks are required. These can be seperated into two categories, depending on the necessity to convert the data into raster form or not. Raster-based methods uti- lize pre-existing vision networks, on the other hand purely INR INR-based approaches operate solely the INR domain. Raster-based methods involve converting INR back raster form, subsequently employing a pre-existing vision net- work to execute the vision tasks. This approach effectively leverages the vast repository of existing algorithms. In con- trast, standalone INR-based methods extract the structural information directly from the INR for visual tasks with- out converting it back to the raster form, which can save memory and hard disk bandwidth.
Currently, both approaches have significant drawbacks. Raster-based methods tend to lose crucial structural infor- mation modeled in the INR during the conversion process, limiting their performance in vision tasks. On the other hand, -aRe methods also face challenges due to their relative novelty, resulting in fewer existing vision networks that can serve as references. This often confines INR approaches to simpler architectures, compared to the vast array of so- phisticated methods developed for raster-based processing, potentially hindering their adaptability and effectiveness for broader range of complex vision tasks. Consequently, this can lead to limitations such as applicability to primarily simpler vision tasks and the reliance on structure-speciff INR models that may not generalize well. These issues will
'eparrmennt of Automation, Tsinghua University, Beijing. China IInstittee of Advanced Technology, University of Science and Technology of China, Hefei, China 'Xiaomi Corporation, Shanghai, China 'Department of Computer Science, ETH, Zurich, Switzerland 'The People's Liberation Army General Hospital, Beijing, China IIsstitute of Brain and Cognitive Sciences, Ts- inghua University, Beijing, China 'Shanghai Artificial Intelligence Laboratory, Shanghai, China. Correspondence to: Jinli Suo <jl- suo weingguu.cddus edu.en>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DVI: A Derivative-based Vision Network for INR
be explored in the following sections.
2021a; Sitzmann et al., 2020; Chen al., 202 Saragadam
et at. 2022), 3D volumes (Wu et al., 2021; Saragadam et al.,
To address the issues of both approaches, we propose DVI, a servaaiivebbsedd Vision network for INR, capable of han-
2022; Peng et al., 2020; Yariv et al., 2021; Takikawa et al.,
2021), and video data (Sitzmann et al., 2020; Chen et al.,
dling variety of vision tasks across various data modalities, while achieving the best performance among existing meth- ods. Specifically, DVI firstly transforms INR data into a raster form, harnessing the strengths of pre-existing vision networks. Simultaneouslyy DVI extracts the structural in- formation from a high order derivative map of the INR. This information is then seamlessly fused into the vision network, enhancing its performance with additional, task- relevant structural features that improve task outcomes.
2021a; Saragadam et al., 2022; Chen et al., 2022; Mai & Liu, 2022). In those modalities, INRs perform various tasks, including rendering (Corona-Figuerge et al., 2022; Wang et al., 2022: Fang et al., 2022; Saragadam et al., 2022: Sitz- mann et al., 2020; Takikawa et al., 2021; Qiu et al., 2023), registration (Li et al., 2024b; Wolterink et al., 2022; Byra et al., 2023; Zimmer et al., 2023; Sideri-Laamreetsa et al., 2024; van Harten et al., 2024), and compression (Yang et al., 2023; Yang. 2023; Yang et al., 2024; Li et al., 2024a; Dupont et al., 2021a; Guo et al., 2024; Pistilli et al., 2022; Zhang et al., 2021c; Lee et al., 2021; Kwan al., 2024).
We evaluate our method on five different vision tasks across three data modalities, demonstrating its superiority over the existing methods through extensive experiments on various datasets. Additionally, our research includes comprehensive ablation studies to validate the effectiveness of each com- ponent of our proposed method and explores the impact of different derivative computation techniques and derivative orders in our approach.
2.3. Vision Tasks
Vision tasks can be divided into pixel-wise and image-wise categories. Pixel-wise vision tasks refer tasks with finer granularity, where each pixel or voxel corresponds cific outcome, such as super-resolution (Image SR) (Lim et al., 2017: Liang et al., 2021), denoising (Image DN) (Zhang et al., 2017), segmentation (Volume Seg.) (Milletari et al., 2016; Çięek et al., 2016), deblurring (Video DB) (Cao et al., 2023; Son et al., 2021), and optical flow estimation (Video FE) (Huang et al., 2022; Zhang et al., 2021a). For these tasks, numerous neural networks have been devel- oped that process data in raster form. Our proposed method targets these pixel-wise vision tasks, aiming to extract struc- tural information from the INR to enhance the performance of pre-existing raster-based networks. However, current INR- based vision networks are limited to basic operations such as interpolation and filtering (Xu et al., 2022; Nsampi et al., 2023), which often results in suboptimal performance in these detailed, pixel-wise vision tasks.
spe-
2. Related Work
2.1. Raster Representation
For vision data with shape $1 X x Sn and channels, we typically represent it as an n + 1 dimen- sional array D E R X-xxxxx in raster form. With this representation, we can obtain a coordinate map X := [x|a1 E (1..s))....,.; E (1..8,,),, corresponding to the data shape. The data values at any coordinate x can be fetched by indexing directly from the array as D( D(zI....,Z,,) Zn].
2.2. Implicit Neural Representation
In the realm of implicit neural representation (INR), we ap- proach data representation through a fundamentally different lens. Unlike raster form, INR employs neural network, de- noted as the function F= : X - RR, mapping coordinates to data values, offering a more dynamic and potentially richer data interpretation. For optimal representation accuracy, the best F can be found by solving the following optimization problem:
2.4. INR- Vision Network
Many standalone INR methods which are not utilizing raster based methods have been proposed, however they face numerous challenges. (Cardace et al., 2024) proposed novel network capable of directly processing structure- specific INR for segmentation or classification. Several works (Schürholt et al., 2021; Dupont et al., 202 2022; Berardi et al., 2022; Schürholt et al., 2022; You et al., 2023; Lee et al., 2023; Bauer et al., 2023) introduced genera- tive model for INR by modeling the latent space of INR parameters, which is capable of handling basic completion and classification tasks. Further works (Zhou et al., 2023a; De Luigi et al., 2023; Ramirez et al., 2023; Navon et al., 2023; Zhou et al., 2023b) designed an encoder that converts all parameters in an INR into a feature vector for subsequent vision tasks. The trained encoder only suitable for INR with fixed number of parameters, yet vision data often
min Σ [((F()))(xx). XEX
(1)
where C() measures the representation accuracy. With this representation, we can fetch data values at any coordinate x by inputting it into F as Fxx). So for any INR F, we can easily convert it back to raster structure as D = F(X). Currently, INRs have been extensively applied in a multi- tude of modalities, such as for the representation of images (Strümpler et al., 2022; Shen et al., 2022; Dupont et al.,
2

---

DVI: A Derivative-based Vision Network for INR
INR High Order Derivatives Computation
INR Feature Extraction
Extractor
Extrral
INR Feat. pinR
F
Vision Task Results
INR
INR Feature Fusion
Fusion
Fus. Feat.
Pusion
F
INR Convertion to Raster
Pre-existing Vision Network
Block
Int. Feat.
Block
Fe
Figure 1. The overall architecture of the proposed DVI. Abbreviations stand for: Feat.: Feature, Fus.: Fused, Int.: Intermediate.
necessitates INR with varying parameters to accommodate different resolutions or demands. Following methods (Xu et al., 2022; Nsampi et al., 2023) extract structural infor- mation from the high order derivative of INR, capable of handling various vision tasks without restrictions on INR structure. However, without the use of conventional raster based networks, it is difficult to achieve good performance. Our approach DVI draws on these methods for extracting structural information from high order derivatives, and pro- poses progressive fusion strategy fuse the structural information with pre-existing raster-based vision networks to achieve the best performance.
This strategy not only maintains the richness of the original INR data, but also augments the existing network's perfor- mance by infusing it with additional, task-relevant structural features that have been shown to improve task outcomes.
3.2. Pipeline
Please find symbols definition in the begining of the back- ground section. Initially, we select a pre-existing vision network H(-)) for given vision task that takes a raster rep- resentation as input and the task result as output. There are no restrictions on the network architectures, and we verified this by choosing several different networks in our experi- ments. At the beginning of the task, we will first transform the data from the INR F into raster structure:
3. Methodology
In this section, we delve into the methodologies for extract- ing structural information from Implicit Neural Represen- tations (INR) and fusing this information into pre-existing raster-based vision networks. We begin by introducing the architecture and pipeline of our model DVI, outlining its innovative feature extraction and fusion strategy. Finally, we discuss in detail the critical designs of DVI.
= FXX,,
(2)
which will be fed into the vision network later. Simultane- ously, DVI computes the high order derivative map of INR, encapsulating the structural information in
HD - (F):
3.1. Overall Architecture
where FHD RCHDXXIX represents CHD partial derivatives of F at each point in X, and (+) is a spe- cialized module that efficiently computes these derivatives, overcoming the limitations of traditional autograd methods (used (Xu et al., 2022)) terms of speed for higher order computations.
The overall architecture of our method DVI, as depicted in Figure 1, comprises four primary components: 1) Pre- existing Vision Network, 2) INR High Order Derivative Computation, 3) INR Feature Extraction, and 4) INR Fea- ture Fusion. As the first step, convert INR to raster form, enabling the utilization of pre-existing vision net- works. This ensures compatibility with established methods and harnesses their proven capabilities. Concurrently, we extract structural information from the high order derivative map of the INR. This information is then integrated into the vision network using the INR feature fusion module.
Next, DVI implements a progressive INR feature extrac- tion and fusion strategy, designed to extract and integrate multiple levels of features from the INR into the vision net- work. This process involves using a set of K INR feature extractors,
to sequentially derive K distinct features {FINR) k=1 from the derivative map FHD These
3

---

DVI: A Derivative-based Vision Network for INR
features represented
3.3. INR High Order Derivative Computation
Let the vector consisting of all pth order derivatives of J with respect point x E X be denoted by:
FINR GIN (F!ID); ); F FIRR gk+1 (F & F for k 1,...,k. K (5)
(4)
DFF(x)
8 f(x)1 du dx 022 0rn dun
0 f²f())
0 f²x)
8 f())
where FEUS is feature fused from the previous level. Using the fused features from the previous level as additional in- puts to the next feature extractor helps to extract features that are aligned with the target vision task. We aimed to align the INR's s structural features with the target network's 's fea- tures in multi-level, facilitating the extraction of structural information with varying densities relevant to the visual task.
P-11
dxh
(10)
which contains c derivative map we need to compute can be represented as a tensor consisting of the 1st to pth order derivatives at each point:
( ("-p-1) partial derivatives. Then, the
D2F(11 ;n)
Further, the outputs from K distinct layers of the vision network H())) are selected as intermediate features, denoted as ] k=1- For simplicity, we segment H into K + sequential blocks based on the positions of these features, denoted as = H4 a ... H. o HK-11
Fdrro
(11)
F( F(11. ,n))
X
Computing all these derivatives using autograd would be highly time consuming. We use the recursive formula for high order derivatives in (Xiao al., 2023) compute the derivative map at an accelerated speed.
Finally, the extracted INR features {FINR) K J==1 are fused with the corresponding intermediate features (Fx) k== of the vision network into the fused features F This is achieved through series of K INR feature fusion modules, ( (-))km> which are employed successively:
Due to the large difference in values between different order derivatives, we need to normalize them. We first counted the distribution of each order derivatives on the training set, (which was found to be approximated as a O-mean Gaussian distribution), computed the maximum value, and normalized each order derivatives by their corresponding maximum value.
(FINR F&). for I,..., K. FFLS 9
(6)
The fused features FFUS have the same shape as the interme- diate features Fg, and will replace them as the input to the subsequent vision network block. This progressive fusion not only aligns, but also enriches the network's 's intermediate features with the structural feature captured from the INR.
3.4. INR Feature Extraction
The INR feature extraction in DVI employs a series of K extractors, all sharing similar structure, with the exception of the first extractor, which lacks residual connection at its entrance. As illustrated in Figure 2(a) and (b), each INR feature extractor is comprised of residual connections, Swin Transformer layers (STL), and convolutional layers (CONV). The specific calculation process for the feature extraction is:
The overall pipeline becomes:
F HI(D):
F P++1 = H4++1 (1+11 (FFUS)
for 1,..., K
F INR INR k++ - C (F F. GCONV GSTL GSTLO
(8)
Task Results- = *H (F
(9)
(12)
GOONY (FINR FFUS + FINR
The progressive strategy adopted by DVI offers remark- able fiexibility, adapting seamlessly wide range of pre- existing vision networks. This approach not only facilitates the effective extraction of structural information from INR, but also ensures its precise integration into the network's 's processing flow. In the subsequent sections, we will delve
The feature extraction process involves residual connections at both the entrance and exit of the extractor. The entrance residual connection robustly incorporates the fused feature from the previous level, enhancing the stability of feature introduction, as seen in (He et al., 2016). The exit residual connection on the other hand, aggregates features from each level, contributing to a more coherent feature extraction
into the details of DVI.

---

DVI: A Derivative-based Vision Network for INR
LayorNorm
CONY
STL
STL
STL
STL.
CONY
MLP
(a) INR Feature Extractor
(b) Swin Transformer Layer (STL)
STCI. STCE STC
Feat, Int. Feat.
Fuux Feat.
iNR Feat,
JayorNorm
MLF
(c) INR Feature Fusion Module
(d) Swin Transformer Cross-attention Layer (STCL)
Figure 2. The architectures of (a) INR feature Extractor, (b) Swin Transformer Layer, (c) INR Feature Fusion Module, and (d) Swin
Transformer Cross-attention Layer. Abbreviations stand for: Feat.: Feature, Fus.: Fuxed, Int.: Intermediate.
(Liang et al., 2021). CONVs are positioned at the entrance and exit as well. The entrance CONV facilitates early vi- sual processing (Xiao et al., 2021), while the exit CONV enhances the extractor's translational equivariance (Liang al., 2021).
fusion process.
The fusion of FINR into F& accomplished by mapping & F& to the query and F to both key and value in cross-attention framework. The cross-attention computation follows the formula outlined Equation (14).
At the heart of the extractor lies the STL (Liu et al., 2021; Liang et al., 2021), which shares structural similarities with traditional Transformer architectures, comprising of Layer Normalization (LN), Multi-Head Self-Attention (MSA), and a Multi-Layer Perceptron (MLP). A distinctive feature of STL its approach to attention computation within the MSA. STL partitions input features into non-overlapping windows and computes standard self-attention separately within these windows as:
A distinguishing feature of all three STCLs is their use of regular window partitioning within the MSA, albeit with varying window sizes. This allows DVI to execute feature fusion across three distinct spatial scales. Once the fused features are computed at these varying scales, they are con- catenated along the channel dimension. Subsequently, their dimensionality is adjusted align with that of the interme- diate feature using x 1 CONV. The specific calculation process for the feature fusion is:
Q F WIN WWN. K = WWN WK. V = -FWINWW (13)
Attention Q, K, V) = (OEMaaXQKT/VELL (14)
FFUS = GE F&)
= GOONY CIGSSCC (F Fg), 1 (FINR FR). GSTCL (FINR FR)), (
(15)
where WIN denotes the features within each window, WQ. WK. and Wv are the shared projection matrices, is the feature dimension, and represents learnable relative positional encoding. To facilitate woos-window attention, the STL alternates between regular and shifted window partitioning in its MSA, proposed in (Liu et al., 2021).
where C(-) denotes the channel concatenation operator and tos represents the size of the regular window partitioning.
3.5. INR Feature Fusion
4. Experiments
The INR feature fusion in DVI consists of K identical fu- sion modules. Each module is characterized by three Swin Transformer Cross-attention layers (STCL) and a convo- lutional layer (CONV), as depicted in Figure 2(c) and (d). Each STCL closely mirrors the structure of the STL, with a notable distinction being the incorporation cross-attention within the MSA. This adaptation pivotal for the feature
We evaluate our proposed DVI on multiple vision tasks across three different types of data modalities: 1) Image Tasks; 2) 3D Volume Tasks and 3) Video Tasks. The meth- ods named 'DVI(ner) represents DVI with net as the respec- tive pre-existing network. All details of data preparation and training from this section can be found the supplementary materials.
5

---

DVI: A Derivative-based Vision Network for INR
4.1. Data
complex vision tasks, except for 3D volume segmentation. It should be noted that this comparison is influenced by fun- damental methodological differences: INSP tackles more challenging problem by processing INRs solely through their weights without materializing discrete signals. 3) For the image super-resolution, 3D volume segmentation and video flow estimation tasks, the improvement of DVI is most pronounced compared to the raster-based methods, indicating that the structural information encoded in the INR is more helpful for these specific tasks. 4) DVI under- performs on tasks requiring coarse structural information, such as video classification. To improve performance, we can reduce the spatio-temporal resolution (res.) to remove redundant information and increase the order of derivatives (rft) to expand the structural feature "receptive field." Test- ing on ViVIT model with the Cameehing-Somettinnn V2 dataset, as shown in Table 5, supports this approach.
For the image super-resolution task, we adopted the setup from works (Lim et al., 2017; Liang et al., 2021; Li et al., 2023), utilizing DIV2K (Agustsson & Timofte, 2017) as the training set, with Set5 (Bevilacqua et al., 2012). Set14 (Zeyde et al., 2012), BSD100 (Martin et al., 2001b), Ur- ban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017) serving as test sets. Similarly, for the image denoising task, the setup from works (Zhang et al., 2021b; Liang et al., 2021; Li et al., 2023) was followed, employing BSD500 (Martin et al., 2001a) and WED (Ma et al., 2016) as training sets, along with CBSD68 (Martin et al., 2001a), Kodak24 (Franzen, 1999), and McMaster (Zhang et al., 2011) as test sets. In the domain of 3D volume segmentation, the setup from works (Milletari et al., 2016; Çiçek et al., 2016), using Synapse (Landman et al., 2015), was followed. For video tasks, GoPro (Nah et al., 2017) was used as the benchmark for video deblurring, following the setup in works (Cao et al., 2023; Son et al., 2021), and Sintel (Butler et al., 2012) for video optical flow estimation, based on the methods described in (Huang et al., 2022; Zhang et al., 2021
Table 1. Quantitative results (PSNR) & SSIM for image super- resolution task.
ASUII
MADICI
All input data, including downsampled images for super- resolution, noise-added images for denoising, 3D volumes for segmentation, and videos for deblurring and optical flow estimation, were converted to Implicit Neural Representa- tion (INR) form to standardize the data processing pipeline across different tasks.
10:37
8:80
0.819
3.90
$.991
3.6
0.881
$4.11
06211
58.03
166
193
Svith
3:30
8.911
21.2
21.3
07915
27.0
03945
11
31.66
1.09
31.9
1877
35.4
0890
2518
0006
15
(AiN 300
2:0
1.111
2123
5.96
2.34
2318
0.788
56.81
10300
1359
148
Table 2. Quantitative results (PSNR1 & SSIM for image denois- ing task.
4.2. Baselines
KodseM
McMane
We distinguish between two types of approaches: Raster- based approaches, where we choose EDSR (Lim et al., 2017), SwinlR (Liang et al., 2021) and StableSR (Wang et al., 2024) as comparison algorithms for image super- resolution task, SwinIR (Liang et al., 2021) , DnCNN (Zhang et al., 2017) and DifIBIR (Lin et al., 2024) for im- age denoising task, VNET (Milletari et al., 2016), UNet3D (Çiçek et al., 2016) and MedSegDiff-V2 V2 (Wu et al., 2023) for 3D volume segmentation task, VDTR (Cao et al., 2023),
Meisal
PSNRT
85IM:
PSNRT
S5MM
PSNRT
SSIMI
MACIG)
INSP Ne st. 2022)
23:46
0.770)
22.45
077443
23.33
6,7035
1034
DeChN (Zhang el, 2017:
24:13
0.2444
38.57
0.7533
15.73
0.709
167
0.6
DVNIMCNNN)
31.95
03717
M.36
31.35
Swadk (Lazg sta.. 3021)
34.53
0.188
33.69
09242
34.0
B9247
590
12
35.85
BJ852
35.85
809655
36.36
89445
5871
16
DVNDMEBRR
Table 3. Quantitative results (DSC1) for 3D volume segmentation task.
PVDNet (Son et al., 2021) and VD-Diff (Rao et al., 2024)
Mocod
Mons
5pt
Koc
06
St
Ave
Phs
MAUGI
for video deblurring task. FlowFormer (Huang et al., 2022),
INSP (Ne et a. 2020
45.97
36.55
25:85
35.58
535
50.0
12.12
4575
41.5
45
02
SepFlow (Zhang et al., 2021a) and FlowDiffuser (Luo et al.,
90.99
83.51
03.55
66.19
(MElaat 2004)
72.02
16.27
15:42
55.54
49:38
51
2024) for video optical flow estimation task. For the INR- based approach, we use INSP (Xu et al., 2022) as the com- parison for all vision tasks.
Ke V (Wa 307N
06.35
1735
75.0
1860
196
4.3. Main Results
Table 4. Left: Quantitative results (PSNRT & SSIM1) for video deblurring task. Right: Quantitative results (EPFL) for video optical flow estimation task.
Quantitative results are shown in Tables to 4. Visual results are shown in Figures 3 and SI to S5. Following observations can be made: 1) Our approach DVI consistently outperforms the baster-based approaches and the INR-based on all vision tasks. 2) The INSP method is not suitable for performing the
Volua
Milbed
121g
12
13
2:38
1712
317
25
ii 3020
1.80
33.20
221
1837
20
12
1130
6

---

DVI: A Derivative-based Vision Network for INR
5.3. Contribution of Feature Extraction and Fusion
Table Quantitative results for video classification task. DVINIVVTT-nuu-eff
Mathod
viviT
DVI(VIVI))
DNT(VIVIT)+RA
Figure 5(a) shows the impact removing the feature extrac- tion and fusion modules from DVL In the "w/o E&F setting, removed the feature extraction and fusion modules and plainly fused the derivative map into the pre-existing net- work by concatenating them to the channel dimension of the input data, where the first layer of the pre-existing network was adjusted to fit the expanded channels. We find that even after removing the feature extraction and fusion mod- ules, there is still some performance improvement, due the structural information in the derivative maps. However, there is a significant decrease in performance compared to DVI. This fully demonstrates the importance of the fea- ture extraction and fusion modules to DVI. Please refer to Figure S7 for more details.
Topl-aconncyt
56.8
$7.0
61L2
64.9
Figure Visual comparisons. Please refer to Figures to for
more results.
Image SR
INSP
EDSR
DVIEDSSI
SwiniR
DV
GT
Image DN
SwintR
Volume Seg.
5.4. DVI is Better than INR-SR in Image Super-resolution
INSP
UNeGD
DVHUUNeSD
DVEIVNET)
Gr
Video DR
For the INR-SR approach, we achieve image super- resolution by supersampling the INR. Figure 5(b) demon- strates that method DVI surpasses the INR-SR in terms of performance improvement relative to the pre-existing network. Please refer to Figure S7 for more details.
DVEPYDNEe) VDTR
Video FE!
SepFlow
DVIiSepFlusBlow-
Forner
5.5. Derivative Computation Techniques
Figure 6 shows that the performance of our method is com- parable to autograd. However, as illustrated in the right panel, our method demonstrates notable speed advantage over autograd, particularly when dealing with higher order derivatives. Please refer to Figure S8 for more details.
5. Analysis
We further verify the validity various aspects of DVI and investigate the effect of different orders of derivatives on DVL All implementation details are deferred to supplemen- tary materials. Following distinct observations can be made:
5.6. Impact of the Highest Order of Derivative Map
5.1. DVI is Robust to Various Pre-existing Network Architectures
Figure shows the performance of DVI varies with the change in the highest order differently in the two tasks, which may suggest different role for the derivative map in the tasks. Also, in both tasks there was significant drop in performance when the highest order reached 5, which may be due to the excessive redundancy of the derivative map affecting the training of the neural network. Please refer to Figure S9 for more details.
Table 6 reveals significant (p < 0.05) improvement in performance with our method compared to the respective pre-existing network, irrespective of the network architec- tures employed. This consistency underscores the robust nature of our method diverse network architectures.
5.2. Derivative Map Contains Task-Relevant Structural Features
6. Discussions
Figure shows that employing appropriate derivative maps substantially elevates performance over the pre-existing net- work. In contrast, mismatched map can significantly re- duce performance, and maps with not yield significant improvements. These findings suggest that derivative maps are integral to enhancing task perfor- mance, presumably due to their encapsulation critical structural information. Please refer to Figure S6 for more details.
6.1. Computational Costs of DVI
During the training phase, our method requires additional computational overhead compared to pre-existing vision networks, primarily due to the need to train the INR fea- ture extractors and fusion modules. In the inference stage, additional computational load mainly stems from the com- putation derivative maps, the INR feature extraction and fusion network inference. For efficient computation
random values do

---

DVI: A Derivative-based Vision Network for INR
Table Statistical significance of performance differences between DVT(ner) and the respective pre-existing network different tasks.
Iruge
Imagee
Wormm Seg.
Video
SwittR
Swinkk
VNET
VDTR
FloeFoase
(Milletai 2016)
p-ater
3.8-M1
3.F-0
368-15 4001-04
1,00am
2:8-00
7,88-11
2 77:00
7 7.0000
2 2.3447
Figure 4. Assessing the impact of substituting DVI's derivative map with alternative maps zero-value (zero), random- (random), and mismatched derivative (mismatch) the super-resoluue super-resolution task for the Manga 109 dataset using SwinlR the pre-existing network. On the left are har plots for each alternative, with significant differences indicated (*8*#; < 0.0001). On the right are box plots showing the performance improvement of each alternative the pre-existing network. ****
PSNR (dB)
30
Improvement (dB)
2.5
28
PSNR
0.0
26
-2.5
DVI(SWinIR)) zero random Swin
DVI(SWinIR)
zero
random
mismatch
Figure 5. (a) The impact of removing the feature extraction and fusion modules from DVI dennoising task, Kodak24 dataset, with DnCNN as pre-existing network. (b) The comparison between DVI and the super-resolution sampling technique using INR (INR-SR) on
super-resolution task, Urban100 dataset, with SwinIR pre-existing network. Both subfigures have the same layouts Figure 4.
Improved rovementnt
PSNR (dB)
32
Improvement
PSNR (dB)
26
10-
PSNR
[dB)
PSNR
(dB)
DVlo
DVI(D.)
SNR
DVI(S.)
(a)
(b)
Figure 6. The performance comparison of DVI on 3D volume segmentation task employing two distinct derivative computation techniques with VNET as pre-existing network. Left: barplots each techniques. Right: curves of (network inference time derivative map calculation time) vs. the highest order of the derivative map for each technique in log scale.
DSC (%)
Time (second)
10a
autegrai
10a
VNET
60
DVI
autograd
VNET
4
The highest order of the derivative map
Figure 7. The curves of performance on video deblurring task (a) and vidco flow estimation task (b) versus the highest order of the derivative map employed DVI.
EPE (inverted)
PSNR
(dB)
27.5
5.5
27.0
2
5
The highest order of derivative map
The highest order of derivative map
(a)
(b)
of derivative maps, our method has already achieved signif- icant improvements over autograd. Further enhancements could potentially arise from more optimal choices of deriva- tive map orders (discussed in detail in the following section) or through CUDA code restructuring. For efficient com-
putation in the INR feature extraction and fusion network, future work could explore sparser feature fusion strategies (also discussed in the next section) the adoption of more efficient neural network architectures.
8

---

DVI: A Derivative-based Vision Network for INR
6.2. Exploring More Rational Orders of Derivative Maps
Table 7. Quantitative results for 3D volume segmentation task with three different INRs and different derivatives calculation methods.
The complexity of computing 1st to pth order derivatives in our method is ((P)) < (P) << O(n"). There- fore, reducing the order P can lower the complexity. We can identify the minimal order suitable for specific vision tasks through multiple experiments, thus ensuring efficient computation without compromising accuracy. Additionally, we can compute different orders of derivatives for different points. For example, we could estimate the error map of the pre-existing vision network (Selvaraju et al., 2017), and in areas with higher errors, compute higher order derivatives, while lower orders suffice in other regions. This approach could strike a better balance between performance and effi- ciency.
Method
SIKEN
Rel.U PE
PFN
Neumrical
Ours
VNET
Ours
VNET
Ours
VNET
DSCT
83.60
72.22
80.29
71.03
80.00
68.52
74.33
6.5. Adapting DVI to non-CNWWrransfrmmer networks
DVI can enhance performance by extracting structural infor- mation from INRs, applicable to the algorithm using INR, including NeRF-like models such as SPIn-NeRF (Figure 8). We calculated first-order derivatives of the logit and density with respect to all feature embeddings and fed them into a new fully connected layer to predict the logit. As shown in Figure 8(b), DVI improves the accuracy and continuity of the segmentation mask.
6.3. Exploring Sparser Feature Fusion Strategies
6.6. The Performance of DVI on Other INRs
Our method separately fuses two feature maps within mul- tiple non-overlapping windows. Reducing the number of windows can decrease complexity, as seen in (Liu et al., 2021). Thus, we could predict highly sparse mask before feature fusion, then conducting feature fusion only within the windows covered by this mask. This strategy can poten- tially reduce computational demand while maintaining the integrity and effectiveness of the feature fusion process.
DVI achieves similar results on other INRs as long as higher- order derivatives can be computed, as shown in Table
6.7. Using The Derivatives from The Raw Signal
Numerical derivatives from the raw signal are ineffective, as shown in the "Numerical" column of Table compared to the
column. The derivatives from INR are effective because they encode structural information during the fitting process.
6.4. Data Augmentation
For simple data augmentation such as flipping, rotation, and cropping, we can obtain the augmented paired data (raster form and INR) on-the-fly by performing the same operation on the INR. However, for complex data augmentation such as color jittering, adding noise and scaling, we need to further investigate how generate the corresponding INR on-the-fly. It is worth noting that although we removed these complex data augmentations in all experiments, we still achieved the best performance overall.
7. Conclusions
Our study presents DVI, a Derivative-based Vision Network for INR, addressing the limitations of existing methods in handling vision tasks for INR. DVI excels by extract- ing structural information from INR's ' high order derivative map, enhancing the performance of an array of different pre-existing vision networks with deeper, task-specific in- sights. Extensive testing across various vision tasks and data modalities confirms DVI's superior performance over exist- ing methods, proving the efficacy of our approach of fusing and harnessing strengths of both INR and raster-based meth- ods.
Figure 8. The performance comparison of DVI on segmentation task with Nike-like methods.
Input View
SPIn-NeRF
DVI(SPIn-N.)

---

DVI: A Derivative-based Vision Network for INR
Acknowledgements
on Circuits and Systems for Video Technology, 33(1): 160-171, 2023, doi: 10.109//CSSV.22022221118
We would like to express our sincere gratitude to review- ers, especially "Kpuk", for providing comprehensive and detailed suggestions that significantly improved this paper. We also extend our appreciation to Dr. Xia Li and Dr. Weijie Wang from ETH Zurich and Ph.D. candidate Qianni Cao from Tsinghua University for their valuable insights and constructive feedback throughout this research. This work is jointly supported by the National Key R&D Program of China (Grant No. 2024YFF0505703) Beijing Municipal Natural Science Foundation (Grant No. Z200021) and the National Natural Science Foundation of China (Grant No.
Cardace, A., Ramirez, P. Z., Ballerini, F,, Zhou, A., Salti, S., and Stefano, L. D. Neural processing of tri-plane hybrid neural fields, 2024.
Chen, H., He, B., Wang, H., Ren, Y., Lim, S. N., and Shri- vastava, A. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34: 21557-21568, 2021a.
Chen, Y., Liu, S., and Wang, X. Learning continuous image representation with local implicit image function. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8628-88688 0628-8638, 2021b.
62088102).
Impact Statement
Chen, Z., Chen, Y., Liu, J., Xu, X., Goel, V., Wang. Z., Shi, H., and Wang. Videoinr: Learning video implicit neural representation for continuous space-time super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2047- 2057, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
References
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted InterventionMMICAA. 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part 1l 19, pp. 424- 432. Springer, 2016.
Agustsson, E. and Timofte, R. Ntire 2017 challenge single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J. R., and Kim, H. Spatial functa: Scaling functa to imagenet classification and generation, 2023.
Corona-Figueree A., Frawley, J., Bond-Taylor, S., Betha- pudi, S., Shum, H. P., and Willcocks, C. G. Mednerf: Medical neural radiance fields for reconstructing 3d- aware et ct-pojeections from a single x-ray. In 2022 44th An- nual International Conference of the IEEE Engineering Medicine Biology Society (EMBC), pp. 3843-3848. IEEE, 2022.
Berardi, G., De Luigi, L., Salti, S., and Di Stefano, L. Learn- ing the space of deep models. In 2022 26th International Conference on Pattern Recognition (ICPR). pp. 2482- 2488, 2022. doi: 10.1 D.1109/1CPR663611000 2022999560....
Bevilacqua, M., Roumy, A.. Guillemot, C., and Morel, M.-
L. A. Low-complexity single-image imgge- super-resolution based on nonnegative neighbor embedding. In British Machine Vision Conference (BMVC), 2012.
Costain, T. W., Li, K., and Prisacariu, V. A. Contextualising implicit representations for semantic tasks, arXiv preprint trXiv:2305.13312 2023.
Butler, D. J., Wulff, J., Stanley, G. B., and Black, M. A naturalistic open source movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Confer- ence on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part 12, pp. 611-625. Springer, 2012.
De Luigi, L., Cardace, A., Spezialetti, R., Ramirez, P. Z., Salti, S., and Di Stefano, L. Deep learning im- plicit neural representations of shapes. arXiv preprint arXiv:2302.05438 2023.
Byra, M., Poon, C., Rachmadi, M. F., Schlachter, M., and Skibbe, H. Exploring the performance of implicit neural representations for brain image registration. Scientific Reports, 1731), 2023.
Dupont, E., Golinski, A., Alizadeh, M., Teh, Y. W., and Doucet, A. Coin: Compression with implicit neural rep- resentations, arXiv preprint arXiv:2103.03123 la.
Dupont, E., Teh, Y. W., and Doucet, A. Generative
Cao, M., Fan, Y., Zhang, Y., Wang, J., and Yang, Y. Vdtr: Video deblurring with transformer. IEEE Transactions
models as distributions of functions. arXiv preprint 102iv:2102.04776, 2021b.
10

---

DVI: A Derivative-based Vision Network for INR
Dupont, E., Kim, H., Eslami, S., Rezende, D., and Rosen- baum, D. From data functa: Your data point is function and you can treat it like one. arXiv preprint arXiv:2201. 12204, 2022.
Li, X., Zhang, F., Li, M., Weber, D., Lomax, A., Buh- mann, J., and Zhang, Y. Neural graphics primitives-based deformable image registration for on-the-fly motion ex- traction. arXiv preprint arXiv:2402.05555 2024b.
Fang, Y., Mei, L.., Li, C., Liu, Y., Wang, W., Cui, Z., and Shen, D. Snaf: Sparse-view cbet reconstruction with neu- ral attenuation fields. arXiv preprint arXiv:2211.17048, 2022.
Li, Y. Fan, Y., Xiang, X., Demandolx, D., Ranjan, R., Timofte, R., and Van Gool, I.. Efficient and explicit modelling of image hierarchies for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18278-18289, 2023,
Franzen, R. Kodak lossless true color image suite. http: //r0k. us/ http//gaaccsss////// /oo///ooaaa kodak/, 1999.
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, I.., and Timofte, R. Swinir: Image restoration using swin trans- former. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833-1844, 2021.
Guo, Z., Flamich, G., He, J., Chen, Z., and Hernández- Lobato, J. M. Compression with bayesian implicit neural representations. Advances in Neural Information Process- ing Systems, 36, 2024.
Lim, B.. Son, S., Kim, H.,, Nah, S., and Mu Lee. K. En- hanced deep residual networks for single image super- resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136-144,77 2017.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Huang. J.-B., Singh, A., and Ahuja, N. Single image super- resolution from transformed self self-exemplars. In Pruceed- ings of the IEEE conference on computer vision and pat- tern recognition, pp. 5197-5206, 2015.
Lin, X., He, J., Chen, Z., Lyu, Z.,, Dai, B., Yu, F., Ouyang, W., Qiao, Y., and Dong, C. Diffbir: Towards blind image restoration with generative diffusion prior, 2024.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Huang, Z., Shi, X., Zhang, C.,, Wang, Q. Cheung, K. C.,, Qin, H., Dai, J., and Li, H. Flowformer: A transformer architecture for optical flow. In European Conference on Computer Vision, pp. 668-685 Springer, 2022.
Kingma, D. P. and Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: arXiv:14/2.6988 14/2.6980 2014.
Luo, A., Li, X.. Yang. F., Liu, J., Fan, H., and Liu, S. Flowd- iffuser: Advancing optical flow estimation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19167- 19176, 2024.
Kwan, H. M., Gao, G., Zhang, F., Gower, A., and Bull, D. Hinerv: Video compression with hierarchical encoding- based neural representation. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Ma, K., Duanmu, Z., Wu, Q., Wang, Z., Yong, H., Li, H., and Zhang, L. Waterloo exploration database: New chal- lenges for image quality assessment models. IEEE Trans- actions on Image Processing,
Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., and Klein, A. Miccai multi-atlas labeling beyond the cranial woult-wo and challenge. In Proc. MICCAI Aulti-Atlas Labeling Beyond Cranial Vorls-Workshop Challenge, volume 5, pp. 12, 2015.
2016.
Mai, L. and Liu, Motion-adjustable neural implicit video representation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp. 10738-10747, 2022.
Lee, D., Kim, C., Cho, M., and Han, W.S. Locality- aware generalizable implicit neural representation. arXiv preprint ar Xiv: wtt.::3300000000 0.05624, 2023.
Martin, D., Fowlkes, C., Tal, D., and Malik, J. A database of human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pp. 416-423, July 2001a.
Lee, J., Tack, J., Lee, N., and Shin, J. Meta-learning sparse implicit neural representations. Advances in Neural In- formation Processing Systems, 34:1 11789 2021.
Li, R., Yang, R., Xiang, W., Cheng, Y., Xiao, T., and Suo. J. A Compact Implicit Neural Representation for Efficient Storage of Massive 4D Functional Magnetic Resonance Imaging, 2024a.
Martin, D., Fowlkes, C., Tal, D., and Malik, J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring

---

DVI: A Derivative-based Vision Network for INR
ecological statistics. In Proceedings Eighth IEEE Inter- national Conference Computer Vision. ICCV 2001, volume 2, pp. 416-423. IEEE, 2001b.
representation. In European Conference on Computer Vision, pp. 318-333. Springer, 2022.
Schürholt, K., Kostadinov, D., and Borth, D. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural In- formation Processing Systems, 34:16481-16493 2021.
Matsui, Y. Ito, K., Aramaki, Y., Fujimoto, A., Ogawa, T., Yamasaki, T., and Aizawa, K. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 76:21811-21838, 2017.
Schürholt, K., Taskiran, D., Knyazev, B., Giró-i Nieto, X., and Borth, D. Model zoos: A dataset of diverse popu- lations of neural network models. Advances in Neural Information Processing Systems, 35:38134-3811 2022.
Milletari, F., Navab, N., and Ahmadi, S.-A. V-net: Fully con- volutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pp. 565-571. Icce, 2016.
Nah, S., Hyun Kim, T., and Mu Lee, K. Deep multi-scale convolutional neural network for dynamic scene deblur- ring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3883 3, 2017.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626, 2017. doi: 10. 109/CCC.2017..4
Navon, A., Shamsian, A., Achituve, I., Fetaya, E.,, Chechik, G., and Maron, Equivariant architectures for learning in deep weight spaces. arXiv preprint 1rX17: 2023.
Shen, L.., Pauly, J., and Xing, L. Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Nsampi, N. E., Djeacoumar, A., Seidel, H.-P., Ritschel, T., and Leimkühler, Neural field convolutions by repeated differentiation. arXiv preprint (rXiix2334.0883. 2023,
Sideri-Lampretsa, V., McGinnis, J., Qiu, H., Paschali, M., Simson, W., and Rueckert, D. Sinr: Spline-enhaneee enhanced im- plicit neural representation for multi-modal registration. In Medical Imaging with Deep Learning, 2024.
Peng, S., Niemeyer, M.. Mescheder, L., Pollefeys. M., and Geiger, A. Convolutional occupancy networks. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020. Proceedings, Part III 16, pp. 523-540. Springer, 2020.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with peri- odic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020
Pistilli, F,, Valsesia, D., Fracastoro, G., and Magli, E. Sig- nal compression via neural implicit representations. In ICASSP
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3733-3737, IEEE, 2022.
Son, H., Lee, J., Lee, J., Cho, S., and Lee, S. Recur- rent video deblurring with blur-invariant motion estima- tion and pixel volumes. ACM Transactions on Graphics (TOG), 40(5):1-18, 2021.
Qiu, J., Yin, Z.-X., Cheng, M.-M., and Ren, B. Nerc: Ren- dering planar causties by learning implicit neural rep- resentations. IEEE Transactions on Visualization and Computer Graphics, 2023.
Strümpler, Y., Postels, J., Yang, R., Gool, L. V., and Tombari, F. Implicit neural representations for image compression. European Conference on Computer Vision, pp. 74-91. Springer, 2022.
Ramirez, P. Z., De Luigi, L., Sirocchi, D., Cardace, A., Spezialetti, R., Ballerini, F., Salti, S., and Di Stefano, L. Deep learning on 3d neural fields. arXiv preprint arXiv:2312. 2023.
Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., and Fidler, S. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11358 11558-11367... 2021.
Rao, C., Li, G., Lan, Z., Sun, J., Luan, J., Xing. W., Zhao, L., Lin, H., Dong, J., and Zhang, D. Rethinking video deblurring with wavelet-aware dynamic transformer and diffusion model, 2024. URL httttt:////xxivi abs/2408 13459.
van Harten, L., Van Herten, R. L. M., Stoker, J., and Isgum, I. Deformable image registration with geometry-in informed implicit neural representations. In Medical Imaging with Deep Learning, pp. 730-742, PMLR, 2024.
Saragadam, V., Tan, J., Balakrishnan, G., Baraniuk, R. G., and Veeraraghavan, A. Miner: Multiscale implicit neural
12

---

DVI: A Derivative-based Vision Network for INR
Wang, J., Yue, Z.,, Zhou, S., Chan, K. C. K., and Loy, C. C. Exploiting diffusion prior for real-world image super- resolution. International Journal of Computer Vision, 132((2):5929-5533 (9990599999..0 2024.
Lower Bandwidth Using Implicit Neural Function. Pro- ceedings of the National Academy of Sciences, 121() e2320870121, 2024.
Yariv, L.,, Gu, J., Kasten, Y., and Lipman, Y. Volume ren-
Wang, Y., Long, Y., Fan, S. H., and Dou, Q. Neural render- ing for stereo 3d reconstruction of deformable tissues in robotic surgery. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 431 151-441. Springer, 2022.
dering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
You, T., Kim, M., Kim, J., and Han, B. Generative neural fields by mixtures of neural implicit functions. arXiv preprint arXiv:2310. 19464, 2023.
Wolterink, J. M., Zwienenherg, J. C.,, and Brune, C. Implicit neural representations for deformable image registration. In International Conference on Medical Imaging with Deep Learning. pp. 1349-1359. PMLR, 2022.
Zeyde, R., Elad, M., and Protter, M. On single image scale-up using sparse rerresteppessnttiioon.. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pp. 711- 730. Springer, 2012.
J., Ji, W., Fu, H., Xu, M., Jin, Y.. and Xu, Y. Medsegdiff- v2: Diffusion based medical image segmentation with transformer. arXiv preprint arXiv: 11798, 2023.
Zhang, F., Woodford, O. J., Prisacariu, V. A., and Torr, P. H. Separable flow: Learning motion cost volumes for optical flow estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10807-10817, October 2021a.
Wu, Q., Li, Y., Xu, L.,, Feng, R., Wei, H., Yang, Q. Yu, B., Liu, X., Yu, J., and Zhang, Y. Irem: High-resolution magnetic resonance image reconstruction via implicit neural representation. In Medical Image Computing and Computer Assisted Intervention- ternational Conference, Strasbourg, France, September 27 27-October 1, 2021, Proceedings, Part VI 24, pp. 65-74- Springer, 2021.
Zhang, K., Zuo, W., Chen, Y., Meng, D., and Zhang, L. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26(7):3142 111131111515,,- 2017.
2021: 24th In-
Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., and Timofte, R. and- image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 44(10):6360-6377 2021b.
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., and Girshick, R. Early convolutions help transformers see better. Advances in neural information processing systems, 34:30392-30400, 2021.
Zhang. L., Wu, X., Buades, A., and Li, X. Color demo- saicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging, 20 (2)003016-023016, 2011.
Xiao, T., Zhang, W., Cheng, Y., and Suo, J. Hope: High- order polynomial expansion of black-box neural net- works, 2023.
Xu, D., Wang, P., Jiang, Y., Fan, Z., and Wang, Z. Signal processing for implicit neural representations. Advances in Neural Information Processing Systems, 35:13404- 13418, 2022.
Zhang, Y., van Rozendaal, T., Brehmer, J., Nagel, M., and Cohen, T. Implicit neural video compression. arXiv preprint arXiv:2112.1131 2021c
Zhou, A., Yang, K., Burns, K., Jiang, Y., Sokota, S., Kolter, J. Z., and Finn, C. Permutation equivariant neural func- tionals. arXiv preprint arXiv:2302. /4040, 2023a.
Yang, R. Tine: Tree-structured implicit neural compression. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 18517-18526, 2023.
Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J. Z., and Finn, C. Neural functional transformers. arXiv preprint arXiv:2305. 3346, 2023b.
Yang. R., Xiao, T., Cheng, Y., Cao, Q.. Qu, J., Suo, J., and Dai, Q. SCI: Spectrum Concentrated Implicit Neural Compression for Biomedical Data. In AAAI Conference on Artificial Intelligence (AAAI), pp. 4774-4782. AAAI Press, 2023.
Zimmer, V. A., Hammernik, K., Sideri-Lampretss V., Huang, W., Reithmeir, A., Rueckert, D., and Schnabel, J. A. Towards generalised neural implicit representations for image registration. In International Conference on Medical Image Computing and Computer-Assistee Inter- vention, pp. 45-55 Springer, 2023.
Yang, R., Xiao, T., Cheng, Y., Li, A., Qu, J., Liang, R., Bao. S., Wang, X., Wang, J., Suo, J., Luo, Q., and Dai, Q. Sharing Massive Biomedical Data at Magnitudes
13

---

DVI: A Derivative-based Vision Network for INR
A. Implementation Details
A.1. Image Super-resolution Task
A. DATA PREPARATION
We downloaded DIV2K dataset (Agustsson & Timofte, 2017) from this link, Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2012), BSD100 (Martin et al., 2001b), Urban (0annu et al., 2015), Manga (Matsui et al., 2017) from this link. Where DIV2K, Set5 and S datasets already contain images with x2 downsampled using the cubic method. We used the cv2.ITTER CUBIC to generate x2 downsampled images for BSD100, Urban 100, and Manga datasets. We convert each downsampled image to INR using SIREN (Sitzmann al., 2020) with Adamax (Kingma & Ba, 2014) as optimizer with learning rate of le-3 and 20,000 iterations. Specifically, to keep the INR representation accuracy of each INR consistent, we set the total number of parameters in the INR based on a percentage of the number of parameters in each image, and the percentage was set to 50%.
A.1.2. TRAINING
For INSP (Xu et al., 2022), we implemented it based on their open-source code. And we expand the number of layers to 10, and the number of neurons per layer to 1024, making its MAC comparable that of other methods. We use autograd to compute all 1st to 3rd order derivatives of INR at each points as described in (Xu et al., 2022). Then we set the input to be all 1st to 3rd order derivatives of INR at point, and the output to be the rgb of a high-resolution image at that point. We trained 100 epochs with Adam (Kingma & Ba, 2014) optimizer at 0.001 learning rate after random initialization. For EDSR (Lim et al., 2017) and SwinIR (Liang et al., 2021), we implemented them based on link and link, largely maintaining the original hyperparameters. We trained them after random initialization. For our approach DVI, we set in the INR High Order Derivatives Computation module 3. When using EDSR as the pre-existing network, we set the K to 2 and select the outputs of conv first layer and body layer in EDSR as intermediate features for fusion. When using SwinIR as the pre-existing network, we set the K to 2 and select the outputs of conv ffrstt layer and conv _boyy layer in SwinIR as intermediate features for fusion. We trained DVI with the pre-existing configuration (optimizer, learning rate, etc.). We use torchinfo to count the Trainable Parameters of all models and compute their MACs on an 156 x 240 image.
A.2. Image Denosing Task
1. DATA PREPARATION
We downloaded BSD500 dataset (Martin et al., 2001a) from this link, WED (Ma et al., 2016), from this link, CBSD68 (Martin et al., 2001a), Kodak24 (Franzen, 1999), and McMaster (Zhang et al., 2011) from this link. We used the cv2.add to add Gaussian noise with sigma of 15. We used only the first 1000 data in the WED dataset sorted by name. We convert each noisy image to INR using SIREN (Sitzmann et al., 2020) with Adamax (Kingma & Ba, 2014) as optimizer with a learning rate of 1e-3 and 20,000 iterations. Specifically, to keep the INR representation accuracy of each INR consistent, we set the total number of parameters in the INR same as the number of parameters in each image.
A.2.2. TRAINING
For INSP (Xu et al., 2022), we implemented it based on their open-source code. And we expand the number of layers to 10, and the number of neurons per layer to 640, making its MAC comparable to that of other methods. We use autograd to compute all 1et to 3rd order derivatives of INR at each points as described in (Xu et al., 2022). Then we set the input to be all 1s to 3rd order derivatives of INR at a point, and the output to be the rgb of a clear image at that point. We trained 100 epochs with Adam(Kingma & Ba, 2014) optimizer at 0.001 learning rate after random initialization. For DnCNN (Zhang al., 2017) and SwinIR (Liang et al., 2021), we implemented them based on link, largely maintaining the original hyperparameters. We trained them after random initialization. For our approach DVI, we set P in the INR High Order Derivatives Computation module to 3. When using DnCNN as the pre-existing network, we set the K to 2 and select the outputs of m head layer and body layer in DnCNN as intermediate features for fusion. When using SwinIR as the pre-existing network, we set the K to 2 and select the outputs of conv_fne _first layer and conv conv_after__a faww_afef__ody_y body layer in SwinIR as intermediate features for fusion. We trained DVI with the pre-existing configuration (optimizer, learning rate, etc.). We use torchinfo to count the Trainable Parameters of all models and compute their MACs on a 500 x 500 image.
14

---

DVI: A Derivative-based Vision Network for INR
A.3. 3D Volume Segmentation Task
A.3.1. DATA PREPARATION
We downloaded Synapse dataset (Landman et al., 2015) from this link. We used the siipy.ddimae..zooe to scale down each volume with the corresponding label to x.5 We used the first 18 of the volumes for training and the last 12 for testing. We convert each volume to INR using SIREN (Sitzmann et al., 2020) with Adamax (Kingma & Ba, 2014) as optimizer with a learning rate of le-3 and 20,000 iterations. Specifically, to keep the INR representation accuracy of each INR consistent, we set the total number of parameters in the INR based on a percentage of the number of parameters in each volume, and the percentage was set to 20%.
A.3.2. TRAINING
For INSP (Xu et al.. 2022), we implemented it based on their open-source code. And we expand the number of layers to 5, and the number of neurons per layer to 1 180, making its MAC comparable to that of other methods. We use autograd to compute all 1xt to 3rd order derivatives of INR at each points described in (Xu et al., 2022). Then we set the input to be all 1st to 3rd order derivatives of INR at point, and the output to be the segmentation label at that point. We trained 100 epochs with (Kingma & Ba, 2014) optimizer at 0.001 learning rate after random initialization. For VNET (Milletari et al., 2016) and UNet3D (Çiçek et al., 2016), we implemented them based on link, largely maintaining the original hyperparameters. We trained them after random initialization. For our approach DVI, we set in the INR High Order Derivatives Computation module to 3. When using VNET as the pre-existing network, we set the K to 2 and select the outputs of in _ut layer and up_tr33 up_tr32 layer in VNET intermediate features for fusion. When using UNet3D as the pre-existing network, set the K to 2 and select the outputs of conv3d_el 11 layer and norm wwmmmrrrluuuuue wtt.luuusclle.cooo _nrm Irouu_rellJI3 layer in UNet3D as intermediate features for fusion. We trained DVI with the pre-existing configuration (optimizer, learning rate, etc.). We use torchinfo to count the Trainable Parameters of all models and compute their MACs on a 64 x 64 x 64 volume.
A.4. Video Deblurring Task
1. DATA PREPARATION
We downloaded GoPro dataset (Nah et al., 2017) from this link. We used cv2.INTER_ LINEERR to resize each frame to 690 x 360. We used only the first 40 frames of each video. We convert each video to INR using SIREN (Sitzmann et al., 2020) with Adamax (Kingma & Ba, 2014) as optimizer with a learning rate of 1e-3 and 20,000 iterations. We allocated 12,000 KB parameters for each INR.
A.4.2. TRAINING
For INSP (Xu et al., 2022), we implemented it based on their open-source code. And we expand the number of layers to 6, and the number of neurons per layer to 512, making its MAC comparable to that of other methods. We use autograd to compute all 1s to 3rd order derivatives of INR at each points as described in (Xu et al., 2022). Then we set the input to be all 1st to 3rd order derivatives of INR at the same point in 5 consecutive frames, and the output to be the rgb at that point in the center frame. We trained 100 epochs with Adam (Kingma & Ba, 2014) optimizer at 0.001 learning rate after random initialization. For VDTR (Cao et al., 2023) and PVDNet (Son et al., 2021) we implemented them based on link and link, largely maintaining the original hyperparameters. Except for VDTR, we adjusted the patch size to 128 to ensure its compatibility with our graphics card. We trained them after random initialization. For our approach DVI, we set P in the INR High Order Derivatives Computation module to 3. When using VDTR as the pre-existing network, we set the K to 2 and select the outputs of img2feats layer and feature _encoder layer in VDTR as intermediate features for fusion. When using PVDNet as the pre-existing network, we set the K to 3 and select the outputs of d0 layer, d1 layer, and temp layer in PVDNet as intermediate features for fusion. We trained DVI with the pre-existing configuration (optimizer, learning rate, etc.). We use torchinfo to count the Trainable Parameters of all models and compute their MACs on the GoPro dataset.
A.5. Video Optical Flow Estimation Task
1. DATA PREPARATION
We downloaded Sintel dataset (Butler et al., 2012) from this link. We used L to resize each frame to 512 x 218. We divided the Sintel Training data into the training and testing sets required for this experiment in a ratio of
15

---

DVI: A Derivative-based Vision Network for INR
14:9. We convert each video INR using SIREN (Sitzmann et al., 2020) with Adamax (Kingma & Ba, 2014) as optimizer with learning rate of 1e-3 and 20,000 iterations. We allocated 160 KB parameters for each INR.
A.5.2. TRAINING
For INSP (Xu et al., 2022), we implemented it based on their open-source code. And we expand the number of layers to 6, and the number of neurons per layer to 512, making its MAC comparable to that of other methods. We use autograd to compute all 1s to 3rd order derivatives of INR at each points as described in (Xu et al., 2022). Then we set the input to be all 1s 3rd order derivatives of INR the same point in consecutive frames, and the output to be the optical flow at that point. We trained 100 epochs with Adam (Kingma & Ba, 2014) optimizer at 0.001 learning rate after random initialization. For FlowFormer (Huang et al., 2022) and SepFlow (Zhang et al., 2021a), implemented them based on link and link, largely maintaining the original hyperparameters. We adjusted image_size to [216, 480] for FlowFormer and image_size _size to [192, 448] for SepFlow to ensure the compatibility with our graphics card. We trained them after random initialization. For our approach DVI, we set P in the INR High Order Derivatives Computation module to 3. When using Flowerrmer as the pre-existing network, we used two sets of feature extraction fusion networks, one for the Cost Volume Encoder and the other for the Cost Memory Decoder. We set K layer in MemoryEncoder as an intermediate feature, and the latter uses the output of the context.encodee in FlowFormer as an intermediate feature. When using SepFlow as the pre-existing network, we used two of feature extraction fusion networks, one for fnet layer and the other for enet layer. We set K to I for both. We trained DVI with the pre-existing configuration (optimizer, learning rate, etc.). We use torchinfo to count the Trainable Parameters of all models and compute their MACs on the Sintel dataset.
for both. The former uses the output of the channel_convenn
A.6. DVI is Robust to Various Pre-existing Network Architectures
A. DATA ANALYSIS
We calculated the statistical significance of performance differences between our method and the respective pre-existing network by Two-Sample t-Test. For the image super-resolution task, we used the PSNR metric on BSD100 (Martin et al., 2001b). For the image denoising task, we used the PSNR metric on CBSD68 (Martin et al., 2001a). For the 3D volume segmentation task, we used the DSC metric on Synapse 'mean' (Landman al., 2015), and trim=0.2 for VNET. For the video deblurring task, we used the PSNR metric on GoPro (Nah et al., 2017). For the video optical flow estimation task, we used the EPE metric on Sintel ammauah 2' (Butler al., 2012).
A.7. Derivative Map Contains Task-Relevant Structural Features
A.7.1. TRAINING
In the 'zero' setting, we use torch.zeros like to replace the derivative map. the 'random' setting, we use torch.rand like replace the derivative map. We retrained DVI the 'zero' and 'random' settings. In the 'mismatched' setting, We used the trained DVI from the original setting.
A.8. Derivative Computation Techniques
1. DATA ANALYSIS
In the experiments on 3D volume segmentation task, we used DVI(VNET) and '0029' volume from Synapse (Landman et al., 2015) to calculate the total time (network inference time derivative map calculation time). In the experiments on image super-resolution task, we used DVI(SwinIR) and 'barbara' image from Set14 (Zeyde et al., 2012) to calculate the total time (network inference time + derivative map calculation time). These two experiments were conducted on one GPU RTX3090.
B. More Results
16

---

DVI: A Derivative-based Vision Network for INR
Figure SI. Visual comparisons for image task on images 'img_093" and 'img_089' from Urban 100 (Huang et al., 2015).
SwinIR (Liang al., 2021)
INSP (Xu al., 2022)
EDSR (Lim al., 2017)
DVI(EDSR)
DVI(SwinIR)
Figure Visual comparisons for image denoising task on images "kodim01' "kodim from Kodak24 (Zhang et al., 2011).
DnCNN (Zhang al.,
SwinIR (Liang al., 2021)
INSP (Xu al., 2022)
2017)
DVI(DCCNN)
DVI(SwinIR)
17

---

DVI: A Derivative-based Vision Network for INR
Figure S3. Visual comparisons for 3D volume segmentation task on data "0040" and '0034' from Synapse (Landman et al., 2015).
INSP (Xu al., 2022)
UNer3D (Çiçok et al., 2016)
DVI(UNe33)
VNET (Milletari et al., 2016)
DVI(VNET)
Figure S4. Visual comparisons for video deblurring task on videos 'GOPR0384_11.00 and 'GOPR0410.11.00 from GoPro (Nah et al., 2017).
INSP (Xu al., 2022)
PVDNet (Son al., 2021)
DVI(PVDNet)
VDTR (Cao al., 2023)
DVI(VDTR)
18

---

DVI: A Derivative-based Vision Network for INR
Figure Visual comparisons video optical flow estimation on videos 'bandage" and 'market" from Sintel (Butler et al., 2012).
2022)
SepFlow (Zhang DVI[SepHlow)
FlwFFrreer (Huang al., DVI(HlowFormee)
Figure S6. Assessing the impact of substituting DVT's derivative map with alternative maps zero-value (zero), random-value (random), and mismatched derivative (mismatch) the super-resolution task for the Urban 100 dataset using SwinIR the pre-existing network. On the left are bar plots for each alternative, with significant differences indicated (**; 0.01, ****; < 0.0001). On the right are box plots showing the performance improvement of cach altermative the pre-existing network.
PSNR (dB)
26
Improvement
5
24
PSNR
(dB)
0
22
-5-
DVI(SWinIR) zero
random
SwinIR mismatch
DVI(SwinIR)
zero
random
mismatch
Figure S7. (a) The impact of removing the feature extraction and fusion modules from DVI on denoising task, CBSD68 dataset, with DnCNN as pre-existing network. (b) The comparison between DVI and the super-resolution sampling technique using INR (INR-SR) on
super-resolution task, BSD100 dataset, with SwinIR as pre-existing network. Both subfigures have the same layouts as Figure 4.
PSNR (dB)
32
***
Improvement
PSNR (dB)
Improvement
10
28
30
PSNR
(dB)
PSNR
(dB)
26
28
-10
Dw/( F&F D.
DVI(D.)
w/o E&F
D S.
DVI(S.)
INR-SR
(a)
(b)
19

---

DVI: A Derivative-based Vision Network for INR
Figure The performance comparison DVI image super-resolution task employing two distinct derivative computation techniques with SwinlR as pre-existing network. Left: barplots of cach techniques. Right: curves of time (network inference time + derivative map calculation time) versus the highest order the derivative map for each techniques log scale.
PSNR (dB)
30
Time (second)
DVI aulograd
102
SwinIR
25
DVI
autograd
SwinIR
0
2
3
The highest order of the derivative map
Figure S9. The curves of performance video deblurring task (SSIM) (a) and video flow estimation task (Sintel clean dataset) (b) the highest order of the derivative map employed in DVI.
(inverted)
5.5
SSIM
0.825
EPE
6.0
0.800
3 The highest order of derivative map
2
2
3
The highest order of derivative map
(a)
(b)
20


Reviews:
[{"review_id": "SfFurYhgZz", "reviewer": "Reviewer_Ghp4", "summary": "This paper proposes a novel architecture DVI that integrates high-order derivative information from implicit neural representations (INRs) into raster-based vision networks. The authors address the limitations of existing types of networks for vision tasks: a) raster-based methods lack semantic information due to the conversion process, and b) INR-based methods show limited performance. To resolve it, they extract high-order derivative features from INR and progressively fuse them into existing vision networks.  As a result, DVI demonstrates superior performance for various vision tasks, compared to both raster-based and INR-based methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive feedback. We address your questions point by point below:\n## Q1: Include diffusion-based methods for comparison\nWe included **5 diffusion-based methods** for comparison across all vision tasks addressed in our paper. These **include the 2 algorithms you suggested** (StableSR [IJCV'24] and DiffBIR [ECCV'24]), as well as 3 additional algorithms (MedSegDiff-V2 [AAAI'24], VD-Diff [ECCV'24], and FlowDiffuser [CVPR'24]) to demonstrate the effectiveness of our method:\n\n**Table R1: Quantitative results (PSNR↑) for image super resolution task.**\n\n| Method | Set5 | Set14 | BSD100 | Urban100 | Manga109 | MAC(G) | Param(M) |\n|:---------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:----:|:------:|\n| StableSR | 30.09 | 27.25 | 25.34 | 23.18 | 26.81 | 12453 | 148 |\n| DVI(StableSR) | 31.58 | 31.12 | 27.06 | 25.12 | 28.97 | 12581 | 156 |\n\n**Table R2: Quantitative results (PSNR↑) for image denoising task.**\n\n| Method | Kodak24 | CBSD68 | McMaster | MAC(G) | Param(M) |\n|:--------------:|:-------:|:-------:|:-------:|:------:|:------:|\n| DiffBIR | 34.34 | 33.42 | 33.98 | 3596 | 379 |\n| DVI(DiffBIR) | 35.53 | 35.05 | 35.79 | 3690 | 385 |\n\n**Table R3: Quantitative results (DSC↑) for 3D volume segmentation task.**\n\n| Method | Mean | Spl | Rkid | Lkid | Gal | Liv | Sto | Aor | Pan | MAC(G) | Param(M) |\n|:--------------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:----:|:------:|\n| MedSegDiff-V2 | 75.79 | 86.35 | 85.31 | 87.25 | 48.39 | 89.55 | 73.40 | 75.36 | 60.67 | 1966 | 44 |\n| DVI(MedSegDiff-V2) | 85.46 | 77.63 | 87.03 | 94.23 | 75.36 | 82.31 | 82.53 | 95.02 | 89.58 | 2101 | 46 |\n\n**Table R4: Quantitative results (PSNR↑) for video deblurring task.**\n\n| Method | GoPro | MAC(G) | Param(M) |\n|:--------------:|:-------:|:------:|:------:|\n| VD-Diff | 28.23 | 236 | 12 |\n| DVI(VD-Diff) | 29.07 | 259 | 13 |\n\n**Table R5: Quantitative results (EPE↓) for video optical flow estimation task.**\n\n| Method | Sintel(final) | MAC(G) | Param(M) |\n|:-------------------:|:---------------:|:------:|:------:|\n| FlowDiffuser | 4.94 | 312 | 15 |\n| DVI(FlowDiffuser) | 3.92 | 341 | 16 |\n\nOur approach consistently **improves performance across all tasks with minimal overhead**: 1+ dB PSNR gain with only **1%** cost for super resolution and **3%** for denoising; nearly 10 percentage point DSC improvement for 3D segmentation with **7%** overhead; and significant performance gains for video deblurring and optical flow with just **9-10%** additional computation.\n\n## Q2: Provide the computational overhead\n\nWe provided the MAC (Multiply-Accumulate Operations) for each algorithm in Tables R1 to R5 in our previous response, as well as in Tables 1 to 4 in our initial submission. **For your convenience, we rearranged typical comparison results below**, showing both performance improvement and computational overhead for each comparison:\n\n**Table R6: Performance improvement and computational overhead on different vision tasks.**\n| Method/Dataset/Metric | Performance (Baseline → Ours [+Gain]) | Computational Cost (MAC in G) (Baseline → Ours [+Increase])|\n|:---------------------:|:--------------------------------------------:|:-------------------------------------------:|\n| StableSR/Manga109/PSNR↑ | 26.81 → 28.97 [**+2.16**] | 12453 → 12581 [**+128**] |\n| EDSR/Urban100/PSNR↑ | 23.49 → 24.71 [**+1.22**] | 1532 → 1681 [**+149**] |\n| DiffBIR/McMaster/PSNR↑ | 33.98 → 35.79 [**+1.81**] | 3596 → 3690 [**+94**] |\n| MedSegDiff-V2/Synapse/DSC↑ | 75.79 → 85.46 [**+9.67**] | 1966 → 2101 [**+135**] |\n| VNET/Synapse/DSC↑ | 72.62 → 83.60 [**+10.98**] | 31 → 43 [**+12**] |\n| VD-Diff/GoPro/PSNR↑ | 28.23 → 29.07 [**+0.84**] | 236 → 259 [**+23**] |\n| PVDNet/GoPro/PSNR↑ | 25.98 → 27.09 [**+1.11**] | 250 → 338 [**+88**] |\n| FlowDiffuser/Sintel/EPE↓ | 4.94 → 3.92 [**-1.02**] | 312 → 341 [**+29**] |\n| FlowFormer/Sintel/EPE↓ | 6.35 → 5.67 [**-0.68**] | 93 → 139 [**+46**] |\n\nThis demonstrates that our approach consistently **improves performance across all vision tasks with minimal computational overhead**, making it practical for real-world applications. We achieve this efficiency primarily by reducing the complexity of high-order derivative computations through our MLP-specific derivative computation paradigm. Furthermore, the overhead from our attention-based feature processing can be further reduced using PyTorch 2.2.2's FlashAttention-V2, making our method even more lightweight in future implementations.\n\nWe sincerely appreciate your thorough review and insightful questions. If our supplementary experiments and analyses have addressed your concerns, **we would be grateful if you could consider adjusting your evaluation**. If questions remain, we welcome further discussion to address any outstanding issues.", "claims_and_evidence": "The authors demonstrate sufficient experimental results to support their claims including:\n- superior performance for five vision tasks (image super-resolution, image-denoising, 3D volume segmentation, video deblurring, and video optical flow estimation)\n- ablation studies in Section 5 that validate the impact of derivative maps and the technique to acquire derivative maps.", "methods_and_evaluation_criteria": "The main idea of this method is the progressive fusion strategy to exploit semantic information from high-order derivatives for pre-existing raster-based networks to achieve better performance. Also, the authors introduce a recursive high-order derivative computation technique to reduce the computational cost compared to Autograd.\n\nAlso, they provide sufficient evaluations on various vision tasks with corresponding metrics.", "theoretical_claims": "The authors provide a detailed computation process for efficiently acquiring high-order derivatives with clear proofs provided in supplementary material. Also, experimental evaluations confirm the effectiveness of the proposed method in terms of efficiency.", "experimental_designs_or_analyses": "Experiments on various tasks and detailed ablations successfully validate the effectiveness of this paper.", "relation_to_broader_scientific_literature": "This paper clearly suggests the limitation of existing networks and claims the importance of fusing semantic information to raster-based vision networks, which has not been explored.", "essential_references_not_discussed": "This paper claims the importance of exploiting semantic information for various vision tasks to achieve better performance. Thus, it would be helpful to compare this paper with diffusion-based image-restoration methods, such as DiffBIR [ECCV'24] and StableSR [IJCV'24].", "other_strengths_and_weaknesses": "**Strengths**\n- It shows strong generalization ability for various vision tasks.\n- The authors provide extensive experiments for validating the effectiveness.\n\n\n**Weaknesses**\n- As mentioned in the paper, the burden of computation costs should be resolved.\n- Also, the lack of comparison with diffusion-based method need to be supplemented.", "other_comments_or_suggestions": "- Several tables and figures in the manuscript are too small. It would be better to adjust the size of tables and figures.", "questions_for_authors": "- Although a lot of diffusion-based models have been recently introduced, this paper does not mention any diffusion-based method. It could be helpful to include diffusion-based methods in the experiments due to their superior performance on various vision tasks.\n- Can you provide the amount of additional computational overhead of this method compared to baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BXWlXg9b8L", "reviewer": "Reviewer_kMWA", "summary": "This paper proposes a framework that combines implicit neural representations (INRs) with a traditional raster-based vision network, leveraging high-order derivatives to capture additional semantic or structural information. Experimental results demonstrate performance improvements across a variety of tasks, such as super-resolution, denoising, segmentation, and video processing.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive feedback. We address your questions point by point below:\n## Q1: Discuss the computational overhead \nWe provided the MAC (Multiply-Accumulate Operations) for all algorithms in Tables 1-4 of our submission, including tests on more complex generative networks. **For your convenience, we summarize key results below**:\n\n**Table R1: Performance improvement and computational overhead.**\n| Method/Dataset/Metric | Performance (Baseline → Ours [+Gain]) | MAC in G (Baseline → Ours [+%])|\n|:---------------------:|:--------------------------------------------:|:-------------------------------------------:|\n| StableSR/Manga109/PSNR↑ | 26.81 → 28.97 [**+2.16**] | 12453 → 12581 [**+1.0%**] |\n| EDSR/Urban100/PSNR↑ | 23.49 → 24.71 [**+1.22**] | 1532 → 1681 [**+9.7%**] |\n| DiffBIR/McMaster/PSNR↑ | 33.98 → 35.79 [**+1.81**] | 3596 → 3690 [**+2.6%**] |\n| VNET/Synapse/DSC↑ | 72.62 → 83.60 [**+10.98**] | 31 → 43 [**+38.7%**] |\n| PVDNet/GoPro/PSNR↑ | 25.98 → 27.09 [**+1.11**] | 250 → 338 [**+35.2%**] |\n| FlowFormer/Sintel/EPE↓ | 6.35 → 5.67 [**-0.68**] | 93 → 139 [**+49.5%**] |\n\nOur approach **improves performance with acceptable overhead across all tasks**. For large networks (StableSR, EDSR, DiffBIR), the overhead is minimal (1.0%-9.7%). Smaller networks show higher relative increases but maintain reasonable absolute MAC counts. We achieve this efficiency through our MLP-specific derivative computation paradigm, **ensuring scalability for larger networks** where relative overhead becomes increasingly negligible. Future implementations could further increase scalability using PyTorch's FlashAttention-V2.\n## Q2: Why INR's high-order derivations help capturing semantic information\n**INR's high-order derivatives contain richer spectral information than raster representations**, providing more comprehensive semantic information for visual tasks.**Due to space limitations, we provide concise theoretical proofs for three visual tasks**, with the remaining proofs being similarly derived.\n### Definition\nLet $I\\_d \\in \\mathbb{R}^{H \\times W \\times C}$ be a discrete image and $I\\_h = \\\\{I\\_{h\\_0}, I\\_{h\\_1}^{1,0}, ..., I\\_{h\\_n}^{i,j}, ...\\\\}$ be the higher-order derivative map, where $I\\_{h\\_n}^{i,j}$ represents the sampled partial derivative $\\frac{\\partial^n I}{\\partial x^i \\partial y^j}$ with $i+j=n$. Here, $I$ denotes the continuous function represented by the INR.\n### Analysis: Super-Resolution\n**Proposition:** In super-resolution, $I\\_h$ contains richer semantic information than $I\\_d$.\n\n**Justification:** Defining semantic richness as:\n$$\\mathcal{S}\\_{SR}(I) = \\int\\_{\\\\|\\omega\\\\| > \\omega\\_0} |\\hat{I}(\\omega)|^2 d\\omega,$$\nwhere $\\omega_0$ is the Nyquist frequency in $I\\_d$, we can derive:\n$$\\mathcal{S}\\_{SR}(I\\_h) = \\sum\\_{n=0}^{N} \\sum\\_{i+j=n} \\int\\_{\\omega\\_0 < \\\\|\\omega\\\\| \\leq \\omega\\_N} |\\omega\\_x|^{2i}|\\omega\\_y|^{2j}|\\hat{I}(\\omega)|^2 d\\omega.$$\nSince $|\\omega_x|^{2i}|\\omega_y|^{2j} > 1$ for $\\\\|\\omega\\\\| > \\omega_0$ and $(i,j) \\neq (0,0)$:\n$$\\mathcal{S}\\_{SR}(I_h) > \\int\\_{\\omega_0 < \\\\|\\omega\\\\| \\leq \\omega_N} |\\hat{I}\\_d(\\omega)|^2 d\\omega = \\mathcal{S}\\_{SR}(I_d).$$\n### Analysis: Denoising\n**Proposition:** In denoising, $I\\_h$ contains richer semantic information than $I\\_d$.\n\n**Justification:** Defining semantic richness as:\n$$\\mathcal{S}\\_{DN}(I) = \\frac{\\int\\_{\\Omega\\_S} |\\hat{I}(\\omega)|^2 d\\omega}{\\int\\_{\\Omega\\_N} |\\hat{I}(\\omega)|^2 d\\omega},$$\nwhere $\\Omega\\_S$ and $\\Omega\\_N$ represent signal and noise domains, and modeling noisy image as $I = I\\_{clean} + \\eta$.\n\nSignal components exhibit directional coherence while noise is isotropic. For appropriately chosen derivatives:\n$$\\frac{\\int\\_{\\Omega\\_S} |\\omega\\_x|^{2i}|\\omega\\_y|^{2j}|\\hat{I}\\_{clean}(\\omega)|^2 d\\omega}{\\int\\_{\\Omega\\_N} |\\omega\\_x|^{2i}|\\omega\\_y|^{2j}|\\hat{\\eta}(\\omega)|^2 d\\omega} > \\frac{\\int\\_{\\Omega\\_S} |\\hat{I}\\_{clean}(\\omega)|^2 d\\omega}{\\int\\_{\\Omega\\_N} |\\hat{\\eta}(\\omega)|^2 d\\omega}.$$\nTherefore, $\\mathcal{S}\\_{DN}(I\\_h) > \\mathcal{S}\\_{DN}(I\\_d)$.\n### Analysis: 3D Volume  Segmentation\n**Proposition:** In 3D volume segmentation, $I\\_h$ contains richer semantic information than $I\\_d$.\n\n**Justification:** Defining semantic richness as:\n$$\\mathcal{S}\\_{VS}(I) = \\sum\\_{k=1}^K \\alpha\\_k \\cdot \\int\\_{\\Omega\\_k} |\\hat{I}(\\omega)|^2 \\cdot H(\\hat{I}|\\_{\\Omega\\_k}) d\\omega,$$\nwhere $H$ is entropy.\n\nSince higher-order derivatives enhance:\n1. Boundary contrast: $|\\hat{I}\\_h^{i,j,l}(\\omega\\_{boundary})|^2 \\gg |\\hat{I}\\_d(\\omega\\_{boundary})|^2$,\n2. Directional information: $H(\\hat{I}\\_h^{i,j,l}|\\_{\\Omega\\_k}) > H(\\hat{I}\\_d|\\_{\\Omega\\_k})$,\n\ntherefore, $\\mathcal{S}\\_{VS}(I\\_h)>\\mathcal{S}\\_{VS}(I\\_d)$.\n\nWe sincerely appreciate your thorough review and insightful questions. If our analyses and theoretical justifications have addressed your concerns, **we would be grateful if you could consider adjusting your evaluation**. If questions remain, we welcome further discussion to address any issues.", "claims_and_evidence": "The claims regarding performance gains are supported by sufficient empirical results on multiple benchmarks. No other major claims appear to lack evidence.", "methods_and_evaluation_criteria": "The proposed method and its chosen benchmark tasks (e.g., image denoising, 3D volume segmentation) are appropriate to showcase the contribution of derivative-based features.", "theoretical_claims": "The paper relies on the premise that high-order derivatives from INRs encapsulate semantic information not captured by purely raster-based methods, but provides limited theoretical explanation for why derivatives specifically convey such semantics. A clearer theoretical underpinning would strengthen the argument that these derivative features genuinely reflect deeper semantic cues beyond standard raster data.", "experimental_designs_or_analyses": "The experiments appear well-structured, with ablation studies and comparisons to both baseline raster-based and INR-based approaches。 Additional discussion about potential biases in data pre-processing or hyperparameter tuning would be helpful.", "relation_to_broader_scientific_literature": "They extend prior implicit neural representation research by proposing a derivative-based approach to incorporate semantic information to bridge the gap between continuous function representations and traditional raster-based architectures.", "essential_references_not_discussed": "Related works are well discussed.", "other_strengths_and_weaknesses": "1. The idea of progressively fusing high-order derivative features into a standard vision backbone is novel and shows promise in multiple settings.\n\n2. The computational overhead is not discussed. Will the increased computational overhead for deriving and processing higher-order derivatives limit practical scalability?\n\n3. My major concern lies in the theoretical justification of the why high-order derivations help capturing semantic information that raster-based methods connot.", "other_comments_or_suggestions": "There should be a space between the references and the main text. For example: \"making it extensively applicable in various vision data representations such as images(Strumpler et al. ¨ , 2022)\", it should be \"making it extensively applicable in various vision data representations such as images (Strumpler et al. ¨ , 2022)\".\n\nInproper citations: \"Anonymous. Neural processing of tri-plane hybrid\nneural fields. In Submitted to The Twelfth International Conference on Learning Representations,\n2023. URL https://openreview.net/forum?\nid=zRkM6UcA22. under review\". It has an arxiv version with author names.", "questions_for_authors": "See comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vhae64AkO", "reviewer": "Reviewer_kpuk", "summary": "The paper presents DVI, a Derivate-based Vision network for INRs (Implicit Neural Representations). It consists of a neural network architecture which combines pre-existing, task-specific architectures working on raster data, like images or voxel maps, with INR feature extraction modules, that process the derivative maps obtained from the INR. The features computed by the conventional network and those computed out of the derivative maps are fused by feature fusion modules at several layers. The fused features replace the features from raster data in the task-specific network. The paper also claims to contribute a technique to reduce the computational cost to compute the derivative map from the INR compared to using autograd. The paper presents experiments on multiple pixel- or voxel-wise tasks, like image super-resolution, image denoising, 3D volume segmentation, video deblurring and video optical flow estimation. The paper also reports several ablation studies.\n\n## Update after rebuttal\nI carefully read all reviews and responses. The authors confirmed my doubt about the novelty with respect to the derivations presented in (Xiao et al., 2023) and will revise the paper accordingly. The other reviews do not uncover critical weaknesses and the responses to them include additional experiments, which confirm the empirical evidence already reported in the paper, and sketches of theoretical arguments, which seems valid and convincing to me. I'd actually suggest to include the latter in the revised appendix. For all these reasons, I confirm my overall recommendation", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and constructive feedback. We address your questions point by point below:\n\n## Q1: Clarify the relation with the derivations presented in (Xiao et al., 2023)\n\nWe confirm that we use the recursive formula for high order derivatives in (Xiao et al., 2023) to compute the derivative map at an accelerated speed. We will follow your suggestion to revise our claims throughout the paper as requested. We will also follow your suggestion to remove section 3.3 and use the space to increase the size of tables and figures.\n\n## Comments or suggestions:\n\n* We will follow your suggestion to reword the claims about INR containing semantic information into sentences that more plainly and clearly state that high order derivatives from INRs are shown to be useful in solving tasks more effectively.\n* We will follow your suggestion to tone down the claims regarding INSP's (Xu et al., 2022) inability to handle the tasks. We will also note in the main text that the comparison is greatly affected by this fundamental difference between the methodologies.\n* We will follow your suggestion to move Appendix B.1 about normalization of the derivative maps into the main paper to make it integral to the methodology for how derivative maps are normalized.\n* We will follow your suggestion to address the issues you identified in Line 064, Line 072, Lines 89-92, Line 153, as well as the problems in the Figures and tables.\n\nWe sincerely appreciate your thorough review and insightful questions, which will significantly improve the quality of our research!", "claims_and_evidence": "1. The paper claims to \"proposes a novel technique to reduce the complexity of the derivative computation\" (line 114). Yet, in section 3.3 the paper states \"We use the recursive formula for high order derivatives in (Xiao et al., 2023) to compute the derivative map at an accelerated speed.\" Indeed, as far as I can tell, the derivations in the appendix, bar a change of notation, are the same presented in (Xiao et al., 2023), which presents a general framework to compute derivative maps of generic neural network, therefore covering also the case of INRs (i.e. fully connected networks). If this is the case, the authors should remove the claim on the novelty of the technique, and just state that to compute derivate maps of INR they rely on the accelerated recursive operators defined in (Xiao et al., 2023). I'd suggest to remove also section 3.3, and use the space to increase the size of tables and figures. If this is not the case, the I'd suggest to the authors to point out clearly what are the main differences with respect to (Xiao et al., 2023).\n\n2. The method is based on the claim that \"high order derivative map of INR encapsulates the semantic information\" or, similarly \"The derivatives from INR are effective because they encode semantic information during the fitting process\". It is never clarified what \"semantic information\" are encoded in the INR.  The claim is said to be validated by the higher performance obtained by DVI with respect to conventional raster-based neural networks. The authors indeed control for the confounding factor given by the larger capacity of the model with the fusion modules by feeding into it zero or random derivatives and verifying that in this case the DVI add-on does not improve performance. Therefore, I believe that the authors have provided clear and convincing evidence that the derivative maps are useful to increase performance on several tasks. Yet, they haven't shown that these high-order derivative maps \"encapsulates\" semantic information. They haven't even clarified what they mean by \"semantic information\". I'd suggest to reword the claims on INR containing semantic information into sentences that more plainly and clearly state that high order derivatives from INRs are shown to be useful to solve tasks more effectively.", "methods_and_evaluation_criteria": "The proposed method is tested on several benchmarks for 5 different, pixel-level tasks. I'm not an expert in all the tasks, but the diversity of tasks makes me positive toward the soundness of the evaluation procedure.", "theoretical_claims": "I checked at a high-level the derivation of the fast derivative map operator in the appendix. They seem to me to resemble closely the derivations in (Xiao et al., 2023). If this is confirmed by the authors, I'd suggest to remove them from the appendix and point the reader to the original paper.", "experimental_designs_or_analyses": "I had a doubt about the comparisons against pre-existing networks, since the INR processing module and, more importantly, the fusion module, add complex operations like cross- and self-attention to the computational graph of the network, which may have invalidate the comparisons. Yet, the ablation studies in section 5.2 and 5.3 show that this is not the case: when feeding into these modules zero or random derivatives, performance regresses to that of the pre-existing network; at the same time, when concatenating the derivative in input to the original network, performance increases. This validates that the higher performance is due to the information provided by the derivative map of the INR, and makes the main comparisons sound.\n\nI'd suggest to tone down the claims on the inability of INSP (Xu et al., 2022) to handle the tasks. The method is trying to solve a much harder task, i.e. processing INRs by processing only its weights, i.e. without materializing the discrete signals. It is good to have it as a baseline to show that this approach, while intellectually more satisfying, gives at the moment inferior performance. But it should be noted in the main text that the comparison is greatly affected by this fundamental difference between the methodologies.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the idea of combining pre-existing networks with derivative maps of INRs to solve dense, pixel-wise tasks has never been explored before.", "essential_references_not_discussed": "All relevant references are cited or discussed. I just note here that the References section needs some cleaning. Three random examples: \"Neural processing of tri-plane hybrid neural fields.\" has been published, so it is not anonymous anymore; \"Neural functional transformers\" was published in NeurIPS 23; (Xiao et al., 2023) was published on TPAMI as of December 2024.", "other_strengths_and_weaknesses": "S1. The proposed method tackles the unexplored problem of solving dense tasks while processing INRs.\n\nS2. The proposed method achieves very good performance on a variety of tasks.\n\nS3. The experimental results and the thorough ablation studies provide empirical evidence for most of the claims.\n\nW1. The main weakness of the proposed method is the need to materialize the raster signal. Ideally, a method processing INRs because of their continuous nature should be able to perform on par with existing methods processing the original, discrete signals without the need to query the INR to reconstruct the underlying signal, which can be slow and requires to make arbitrary decisions like the resolution or the point of view of the rendering for NeRFs.", "other_comments_or_suggestions": "Line 064 -> \"Vision tasks can be divided into low-level and high-level categories.\" I don't think this is common terminology and it wasn't clear to me what the authors were referring to. A suggestion could be to use \"pixel-wise\" or \"pixel-level\" tasks, like denoising, as opposed to image-wise tasks like classification.\n\nLine 072 Video FE -> acronym undefined\n\nI'd suggest to change the description of \"Neural processing of tri-plane hybrid neural fields.\" at lines 89-92. Currently, it reads \"However, this INR is not suitable for other types of vision data beyond point clouds, and its representation accuracy is lower than SIREN(Sitzmann et al., 2020b).\" Both statements are false: in the paper they process several fields, obtained from voxel maps, meshes, and even NeRFs; and Tab. 1 shows that they are equivalent to SIREN, e.g. slightly worse on point clouds but slightly better on meshes. \n\nline 153: This information is then intricately integrated -> \"intricately\" does not seem to me the right word here\nline 155: two-pronged -> unclear, please reword\nline 185: we will delve into the intricate details of DVI. -> again \"intricate\" seems misplaced here\nline 306: caption of table 2 \"denosing\"\n\nAll the figures in Section 5 have wrong references. For instance, line 326 \"Figure S6 shows\" should be Figure 4. Same for the references to Fig. S7(a), S7(b), S8 and S9 at the beginning of the subsequent sections.\n\nTable 7 \"neumrical\". Moreover, it is not clear how numerical derivatives are used to obtain the reported results, nor to which other columns it should be compared. Please improve the organization of the table.", "questions_for_authors": "My main request is to clarify the relation with the derivations presented in (Xiao et al., 2023), and fix the claims throughtout the paper if they are confirmed to be the same presented in the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes DVI, a derivative-based vision framework that bridges implicit neural representations (INRs) and raster-based vision networks. By extracting high-order derivative maps from INRs and integrating them into task-specific vision architectures, DVI addresses the limitations of both pure INR-based and raster-based methods. The paper demonstrates strong empirical performance across five vision tasks and three data modalities, including comparisons with state-of-the-art diffusion models. The authors also introduce an efficient method for computing higher-order derivatives and provide comprehensive ablation studies.
The reviews were generally positive. Reviewers praised the breadth of experimental validation and the consistent performance improvements with minimal computational overhead. Some concerns were raised about the novelty of the derivative computation (notably overlap with Xiao et al., 2023) and the theoretical justification for semantic richness in derivatives. The authors responded thoroughly, acknowledged the overlap, and proposed revisions. They also provided additional theoretical sketches and clarified the rationale behind performance gains, which reviewers found satisfactory.
While some claims need to be toned down or reworded in the final version (as the authors promised), I believe the paper makes a valuable contribution to the vision community by opening up new ways of integrating INRs into practical vision pipelines. It is well-executed, thoughtfully evaluated, and addresses reviewer concerns constructively.

I recommend acceptance.
Decision: Accept (poster)