Meta-Learning Approach for Joint Multimodal Signals with Multimodal Iterative Adaptation
Authors: Sehun Lee
Wonkwang Lee
Gunhee Kim
Venue: Accepted by TMLR ()

Abstract: In the pursuit of effectively modeling real-world joint multimodal signals, learning to learn multiple Implicit Neural Representations (INRs) jointly has gained attention to overcome data scarcity and enhance fitting speed. However, predominant methods based on multi- modal encoders often underperform due to their reliance on direct data-to-parameter map- ping functions, bypassing the optimization steps necessary for capturing the complexities of real-world signals. To address this gap, we propose Multimodal Iterative Adaptation (MIA), a novel framework that combines the strengths of multimodal fusion with optimization-based meta-learning. The key idea is to enhance the learning of INRs by facilitating exchange of cross-modal knowledge among learners during the iterative optimization processes, improv- ing generalization and enabling a more nuanced adaptation to complex signals. To achieve this, we introduce State Fusion Transformers (SFTs), an attention-based meta-learner de- signed to operate in the backward pass of the learners, aggregating learning states, capturing cross-modal relationships, and predicting enhanced parameter updates for the learners. Our extensive evaluation in various real-world multimodal signal regression setups shows that MIA outperforms existing baselines in both generalization and memorization performances.
Our code is available at https://github.com/yhytoto12/MIA.

Published in Transactions on Machine Learning Research (08/2024)
Meta-Learning Approach for Joint Multimodal Signals with Multimodal Iterative Adaptation
Sehun Lee"
y
Deportment of Computer Science and Engineering Scoul National University
Wonkwang Lee"
workwangg.ce Ben ac.kr
Department Computer Science and Engineering Seoul National University
Gunhee Kim
gunhee@snu.oor oc.
Department of Computer Science and Engineering Scoul National University
Reviewed on OpenReview: / net/ forum? id- LVO4KB.IQ0
Abstract
In the pursuit of effectively modeling real-world joint multimodal signals, learning to learn multiple Implicit Neural Representations (INRs) jointly has gained attention to overcome data scarcity and enhance fitting speed. However, predominant methods based on multi- modal encoders often underperform due to their reliance on direct lata-to-parameter map- ping functions, bypassing the optimization steps necessary for capturing the complexities of real-world signals. To address this gap, propose Multimodal Iterative Adaptation (MIA), novel framework that combines the strengths of multimodal fusion with oplimization-bassed meta-learning. The key idea is to enhance the learning of INRs by facilitating exchange of cross-modal knowledge among learners during the itcrative optimization processes, improv- ing generalization and enabling a more nuanced adaptation to complex signals. To achieve this, we introduce State Fusion Transformers (SFTs), an attention-basod meta-learner de- signed operate in the backward pass of the learners, aggregating learning states, capturing cross-modal relationships, and predicting enhanced parameter updates for the learners. Our extensive evaluation in various real-world multimodal signal regression setups shows that MIA outperforms existing baselines in both gencralization and memorization performances. Our code is available at hhtps:/ httg//gttubb.oooooo httphhub..om/yyyyyoll/////////
1 Introduction
Implicit ncural representations (INRs) are a class of noural networks designed to represent signals or data as continuous coordinate-tos to-aattutt mapping functions (e.g. an image is a function [(x, y) = (r.g. mapping 2D coordinates to color intensity values). INRs offer several advantages over traditional data representations (e.g. discrete 2D arrays for images), including the ability to recover inherently continuous signals that are often sampled sparsely and stored discretely (e.g. videos the recordings of dynamic scenes that change continuously over space and time) and better scaling property with signal resolution (Dupont et al. 2022a
Since a variety of real-world signals can be represented as such continuous functions or they even inherently arise from such functions, numerous efforts have been made to formulate data as functions and represent them with INRs a wide variety of domains or modalities, including audios (Sitzmann et al. 2020), time se- ries (Fons et al. )022), images (Sitzmann et al. 2020), vidcos (Chen al. 2021a), 3D geometrics (Park al.
"Equal Contribution

---

Published in Transactions on Machine Learning Research (08/2024)
Observed
Prediction
Observed
Prediction
Observed
mustmmddi Iarsiie intrrction with lamen
Prediction
RGB Laimer
Leanur
ileaiive Sedbeck
Multimodal Encoder
intraciion rundiiiii
MIA
O
de
Skucch
Suurc Lutner
Skardh Lainet
Multimodal Enooder-Baseed
Unimodal Optimization Bassed
Our Approach
Figure 1: Left: Multimodal encoder- based methods predict INR. parameters directly from signals, omitting optimization steps for rapid adaptation, often failing to fully capture complexities of real-world signals (e.g. blurry RGB prodiction). Middle: Unimodal methods struggle with generalization in data-sparse scenarios due to their lack of cross-modal interactions (e.g. inaccurate Sketch prediction). Right: Our Multimodal Iterative Adaptation (MIA) enhances learner interaction and cross-modal knowledge exchange during the backward optimization pass, leading to improved fitting and generalization.
2019), 3D scenes (Mildenhall et al. 2020 and 3D motions (Pumarola et. al. 2021 They have shown promise in various practical applications that require the precise modeling of signals and their inherently continuous propertios, such as generation (Yu et al. 2022), compression (Kim et al. 2022b), super-rosolution (Chen
et al. 2021b), video super-slomo (Chen et al. 2022b), and novel-view synthesis [Takikawa et al. 2021
As the field of LNRs advance, learning to learn a group of multiple INRs jointly has gained attention recently enhance convergence and data efficiency when modeling joint multimodal signals (Kim al. 2022a
Shen et al., 2023). Such signals, often arising from correlated multimodal distributions with interdependent
information, crucial in diverse areas such embodied agents (Xia et al., 22018). climate analysis (Wang
et al.
2019
healthcare (Muhammad et al. 22221), neuroscience (Ulrich et al. 2015 drug discovery (Chan
ct al.
2023 and smart-grid systems Kamarthi et al. 2022).
In modeling such joint multimodal signals, the primary approach involves encoder-based meta-learning meth-
ods (Kim et al. 2022a Shen et al. 2023 that use multimodal encoders to predict INR parameters directly from data through ditentionn-bsse fusion. Concurrently, optimization-
methods (Tancik al.] 2021; Dupont et al. 2022a), though typically unimodal, focus on learning an effective parameter initialization speed up convergence and enhance generalization. These methods itcratively optimize INR, parameters from this initialization using gradient descent algorithms at test time.
Nevertheless, both approaches come with their own limitations, as illustrated in Figure Encoder methods primarily roly on direct data-to-parametor mappings without optimization stops for rapid adaptation, often lailing to capture complex signals (e.g., underfitting high-frequency details) (Kim et al. 2019). In contrast, unimodal methods, despite their rapid signal fitting capability thanks to iterative optimization processes from good initializations, lack mechanisms for leveraging interacting multimodal structures in data, limiting their generalization in data-scarce situations (Kim et al. 2022a.
To bridge these gaps, we present novel optimization- meta-learning framework called Multimodal Iterative Adaptation (MIA). Unlike the encoder-based methods, MIA builds upon the advantages of optimization- based meta-learning methods, í.e. meta-learning good initializations and adapting them to signals through iterative optimization steps for accurate fitting. In contrast to the unimodal counterparts, however, MIA further empowers each modality learner to interact and inform one another during adapta- tion. This fosters rich exchange of information and synergistic learning of INRs across different modalities, leading to enhanced generalization capability with limited observations.
The core of MIA is an attention-based meta-learner coined as State Fusion Transformers (SFTs), particularly designed to operate in the backward pass of the INR learners (i.e. within the space of learners' parameters and gradients). SF's are meta-learned to achieve three goals: (1) aggregating the current learning stotes (i.e. parameters and gradients) of the learners, (2) capturing modality-spccific and cross-modal relation- ships between those states through attention mechanisms, and (3) utilizing this knowledge to predict better parameter updates for each learner, facilitating moosl interactions at each iterative adaptation step.
2

---

Published in Transactions on Machine Learning Research (08/2024)
It's noteworthy to mention that attention mechanisms well-studied across diverse multimodal learning contexts and applications. However, their potential to uncover and utilize interdependencies within multiple optimization problems and their trajectories (often represented by states of the learners for each optimization step) remains relatively unexplored. Our work contributes uniquely to the literature by being the first to investigate the integration of those potential multimodal structures within the scope of jointly learning mul- tiple correlated INRs, showing how optimizing for one modality can inform and enhance solutions for others. While doing so, we detail recipes for effectively identifying and leveraging these multimodal relationships, including techniques for scaling gradients properly for each modality and integrating original, unimodal, and multimodal learner states. We believe these insights offer interesting new directions in the study of interacting optimization problems, especially useful when data scarcity could lead to bad optimization solutions.
In experiments, we apply our technique to prior unimodal INR meta-learning frameworks such as Functa and Composers, and evaluate them on a variety of multimodal signal regression scenarios, including modeling 1D synthetic functions, ERA5 global climate data, 2D CelebA visual images, and Audiovisual-MNIIS data. The results consistently demonstrate that, our MTA significantly enhances the learning capabilities of the unimodal baselines. Moreover, it outperforms the encoder-bace based baselines in terms of both generalization and memorization performance. We also conduct in-depth and widc-ranging analysis studies to validate the necessity of each component in SFTs for achieving these improvements.
2
Related Work
2.1 Meta-learning for INRs.
Fitting INRs to a function from randomly initialized weights has been found to be notoriously inefficient; it requires hundreds or thousands of optimization steps to converge (Sitzmann et al. 2020 Mildenhall et al. 2020), and learned representations do not gencralize well on novel input coordinates under the few-shot learning regimes (Jain et al. 2021). To overcome such limitations, there have been increasing studies to apply meta-learning to facilitate the learning of INRs. The methods typically include either (1) meta-
learning encoder to infer the parameters of INRs directly from observed data. (Chen & Wang 2022 Kim
ct al. 2023) or optimization-basee approach where a weight initialization of INRs is learned to enable rapid
adaptation to signals with a few optimization steps (Tancik et al. 2021
Bauer et al. 2023
2.2 Multimodal Learning.
Multimodal lcarning aims to integrate data or representations from multiple modalities to enhance the robustness and accuracy of learning systems, leveraging the complementary information available across different data types. Most of the work in the literature typically focus on learning multimodal encoders to infer enhanced representations from observed data, where attention mechanisms are utilized for multimodal fusion (Liang al., 2022 Bachmann et al.] 2022). Unlike this traditional focus on enhancing representations of models in their forward pass, we explore novel setup where utilize attention modules to enhance the learning of the models in their backward pass at each iterative optimization step.
2.3 Learning to Optimize (L20).
Our method also aligns with the works in L2O, another meta-learning domain that focuses on enhancing an optimization algorithm rather than weight initialization. Its idea is to empower optimizer with neural networks that meta-learn useful prior knowledge the optimization process (Andrychowicz et al. 2016). To achieve that idea, Metz et al. (2022a) utilize various state features of learners, including paramcters, gradicnts and their higher-order statistics. In addition, Kang el. al. (2023) meta-learn a gradient preconditioning method for improved convergence and Baik et al. (2023) introduce a meta-learned mechanism to estimate weight docaying and learning rate factors for improved generalization. Distinct from these methods, our work takes an intesting orthogonal direction by exploring the potential of leveraging motal interactions among state features of learners to address data scarcity issues.
3

---

Published in Transactions on Machine Learning Research (08/2024)
3 Preliminaries
3.1 Implicit neural representations (INRs)
INRs are a class of neural networks, often parameterized as a stack of MLPs, that are designed to approximate a function f F y mapping the coordinates x to the features y. We interchangeably use data, signals, and functions throughout the paper. Given a set of coordinate-feature pairs D [[)",")) of function, an INR fINR(:;) parameterized by is optimized to minimize a loss function as below:
((D;0)
E ( v'112. P
1
(1)
Despite the recent advances, optimizing individual INRs from randomly initialized weights are notorionsly inefficient. (Sitzmann et al., 2020; Mildenhall et. al. 2020 . Also, they often do not generalize well when observations sparse (Tancik ct al., 2021 JJin et al. 2021), í.c. when P is small. To address these problems, oplimizatiinnbassed mneaallearning lechniques have been proposed.
3.2
Meta-Learning Approach
Two notable examples Functa (Dupont et al. 2022b a Bauer et al. 2023 and Composers (Kim et al, 2023 These methods build upon CAVTA (Zintgraf et al. 2018) and implement two key ideas: (1) Meta- learming INR parameters 0 that capture data-agnostic priors, enabling faster adaptation when modeling each signal. (2) Introducing additional context parameters Φ E RSxx a set of D-dimensional features that are adapted to each signal. These features encapsulate data-specific variations and conditions of the mota-learned INRs f))R(=;0) via modulation schemes (Perez al. 2018 Schwarz et al., 2023) during the adaptation stage.
Formally, given dataset D {D,) of N functions, the objective in Eq. is extended to accommodate both 0 and on for each function D, as follows:
[(D (n))
P. Σ (r 0, -.
1
(2)
Then, INR weights A and initial context parameters o are optimized with the meta-objective as below:
0. d argmin E,,
where
(3)
0.0
(k+1) (b+)) a V
) ((
(4)
for 0....
1. 1 optimization problem: (1) In the inner loop (Eq. 4 each INR learner is initialized with weights and o
is a learning rate and t (0) 6. Intuitively, the objective can be cast a bi-level
and then adapted to each function Dn through on 1, K). (2) In the outer loop (Eq. 3), the meta learner updates the INR weights and initial context parameters so that each fitted INR. fINR(:: oh ((()) recovers the signal D, well within K steps.
4 Approach
We delve into our optimization--ased meta-learning framework for learning multiple INRs jointly when modeling multimodal signals. We first extend previous meta-learning frameworks to multimodal setups. Then, we introduce our MIA, its core components SFTs, and the meta-learning objectives. We present a schematic illustration in Figure and provide a meta-learning algorithm our MIA in Algorithm
We consider dataset D [ with N pairs of multimodal signals. Each pair Dn {Dum) m is composed of signals from M different modalities. Each signal D Dam {(r Fam ) Pam contains Pam coordinate-feature pairs which could vary with signals.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Multimodal Iterative Adaptation (MIA)
forward computation bod. update (CAVIA) bud. updase (MIA)
State Fusion Transformers (SFTs)
INR f01
Proj MLP
USFT1
Fusian MLP
(M)
(M) P.M
INR Se
(k)
Syukf
7.M
Praj MLP
USFT
Fusion
(8)
(R)
MLP
Figure An illustration on our proposed Multimodal Iterative Adaptation (MTA). Left: At each step k, each INR learner (fo..) calculates the loss (Chi) given the weights (o and data (Foms Fam) during the forward pass (black arrows), followed by computing gradients (ghm) to update the weights. Before updating the weights with the gradients (red arrows),
enhance them via State Fusion Transformers (SFTs) (blue arrows). Right: SFTs aggregate the states (føhm, (K) mmmm)) capture the unimodal and cross- modal dependencies via USFTs and MSFTs, and fuse this knowledge to compute enhanced parameter updates (19mm) )m.) via Fusion MLPs.
4.1 A Naive Framework for Multimodal Signals
We begin with a simple baseline that combines per-modality meta-learners logether while treating each of them separately. This framework consists a set of independent per-modality INR fiNu(: Qm) with parameters 0 = (0m Jm and initial context parameters Φ = [ [ Then, we modify the meta-objective promote both memorization and generalization performances during the adaptation. For this, we split each data into support Dirain and query Dyal sets, and use the following bi-level optimization algorithm to meta-learn the parameters for each modality:
mm Rm argmin En, [C(Diall mm ((K)) where
(5)
0f (k+1)
fm O'm g/b
(k)
V
Voas 00; L(Diraim Om d (b))
(6)
for & = 0...., K with @rom Øm. is a learning rate for (k) of cach modality.
4.2 Multimodal Iterative Adaptation (MIA)
We enable cross-modal interaction among the learners through our MIA for enhanced parameter updates. Our rationale is grounded by the fact that the parameters of one modality learner carry information on a comprehensive encoding of the signals within modality, whereas the gradients highlight both the adequacy of this encoding via magnitude and the direction for improvement. Thus, by allowing access to the states from hther modalities through our MTA, we enable each learner to benefit from broader, richer context for its own updates. Essentially, this process encourages learners to not only optimize their individual parameters but also to contribute to and bencfit from a collective improvement across modalitics. This process is described as:
(k)
Iam Ug (1 (0 )
where Ug denotes State Fusion Transformers (SFTs), a collection of meta-learned modules.
4.3 State Fusion Transformers (SFTs)
As shown Figure 2 SFTs cuhance the learning of INRs via threc-stop process: (1) aggregating the states of the unimodal learners, (2) capturing modality-specific and cross-modal information within those states to promote the knowledge exchange across them via attention mechanisms, (3) and utilizing this knowledge
5

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1 Meta-Learning Algorithm of MIA.
Require: Batch size B; Inner step K; Outer learning rates λe (0 = 1: Initialize θ = 10, (,
2: repeat
3: Sample a batch B joint multimodal signals {{D 1M Sample a sampling ratio Rum ~ ( RRmin Rmax) for Vn, m. Split pm into Darain Dval where D P Prm Bum and [Dxal
Im=1)
P nm-
for 1...., B do for k O,............... K 1-110 do
8:
for m I.....M
M do
frm
(k) V (h) L(Dirain On, Pre
(R)
Eq. 6
10:
Jam f exp (11)) grim
(k)
(k)
Eq. 12
11:
2mm
(k)
ProjectMLP m (Jim, Oums Em)
(k) (k)
12:
Anm USFT m (zmmi S.
Eq.
13:
end for =(k)
(b) MSFT (k)
(1k)
14:
21
AM
Paii Ex)
Eq. 9
15:
for
.....
.. M do
16:
Fram
(k)
FusionMLP
2nm
2rmm
2 Em
b Eq.
(b++1) (K)
17:
Prm
Fuo
Eq.
18:
end for
19:
end for
20:
end for
21:
Louter 2B
DB -== EM L(Dral mm (K)) θ Ne VeCouter, where θ {0, 0, E, m).
> Eq.
22:
Eq.
23:
until converged
to predict better parameter updates for the learners. SFTs consist of Unimodal State Fusion Transformers (USFTs), Multimodal State Fusion Transformers (MSFTs), and Fusion ML.Ps.
For each adaptation step k, we first compute per-modality state representations 2am E Sm from the gradients and parameters of each modality learner. We concatenate the context parameters ( E xg5mxD and gradients grm xDs along the feature dimension, followed by per-modality projection MLPs. Then, these state representations are fed into USFTss with per-modality L1 transformer blocks (Em):
Tame (k) = USFT m (k). Em), for m = =.....,,M
where s(h) are updated state representations by USFIs, which model dependencies among a set states
) within modality. Next, we combine these per-modality representations into a sequence a(*) = {2(th ) and
input to MSFIs with shared L2 transformer blocks (g.). These enhanced state representations z(k) further capture cross-modal relationships:
(k) n
(55FT((())55) where ((k) {2(k) 1 m==1
The final step is to integrate them all and estimate enhanced parameter updates for the learners. We concate- nate the computed per-modality state representations (k) z(k) s(R)
E 8 along the feature dimension,
followed by processing each feature with Fusion MLPs:
g(k) FusionMLP(s ( (skk) Em)) where
((k) "nm
(k)
s(k) Fum
(k) 2nm
(10)
Here, (1k) 1 M is the predicted weight updates for the unimodal learners. They are used to adapt (k) Pro for each inner step k 0,..., K 1, while the parameters of SFTs E. U [
IM along with those

---

Published in Transactions on Machine Learning Research (08/2024)
of INRs (9 and Φ) meta-learned during outer-optimization:
0, -. argmin En,m [C(Dinl: Pm: (nen) ]] where (l++1) = =0/11 -
(11)
0.0.5
4.4
Gradient Scaling
The direct use of gradients as inputs to neural networks could be problematic due to the presence of ex- tremely small values, leading to a bottleneck in learning process. Also, the gradient distributions could differ significantly across modalities, hindering their effective multimodal fusion. To remedy this, we meta-learn a scaling constant for each modality and rescale the gradients before feeding them into projection MLPs:
(k) + exp (mm) -gh
(12)
where we exponentiate the scaling constants {mmm m=1 ensure positive rescaling and maintain gradient directions.
5 Experiments
We demonstrate our proposed MIA improves performance of state-of-the-art meta-learned INR. models. We refer the reader to Appendix A for more qualitative samples.
Baselines. We adopt Functa Dupont et al., 2022a and Composers (Kim et al. 2023 as the base INR models, collectively referred to as CAVIA in subsequent sections. Then, we build our framework and other baselines upon this CAVIA. We include recent L20 methods such as MetaSGD (Li et al. 2017), GAP (Kang ct al., 2023 and ALFA (Baik ct al., 2023) as basclines integrated into CAVIA. These approaches aim enhance learner's convergence and generalization by modulating gradients, yet they do not account for cross- modal interactions. We also compare against the state-of- swwoooftthe-arte the-arts mccodrr-based multimodal meta-learning models, HNPs (Shen et al. 2023) and MTNPs (Kim et al.) 2022a),, which are based on Neural Processes (NPs) Garnelo
Garnelo et al., 20188. While HNPs are heavily demonstrated on multimodal classification, MTNPs are designed specifically for multimodal signal regression tasks like ours, using encoders with specialized attention and pooling instead of iterative optimization. We set K for optimization- based methods. See Appendix D.2 for detailed discussions on baselines.
Datasets and Metrics. Our method is evaluated across four datasets: (1) multimodal 1D synthetic
functions (Kim et al. 2022a), (2) multimodal 2D CelebA images (Xia et al. 2021 (3) ERA5 global climate dataset (Hersbach el al. 2019), and (4) Audiovisual-MNIIT (AV-MNIST) dataset (Vielzeuf et al. 2018). For training, we construct the support set Dirain by sampling Prom x Ram coordinate-featio
pairs from each full dataset Dum- The sampling ratio Ram varies within prodefined ranges [Rmin, Rmaa],, independently drawn for each signal. The full dataset serves as the query set Dial during validation to assess both memorization and generalization capabilities. We measure performance using mean squared errors (MSEs), averaged over 5 runs, across difffrent ranges of sampling ratios. More details, including the number of signals (N), modalities (M), and voordinato-value pairs (P) for each dataset, can be found in Appendix D.1
5.1 Multimoal 1D synthetic function regression
Setups. Following prior works (Finn et al. 2017 Guo al. 2020 Kim et al. 2022a we conduct experiments simple yet controlled 1D synthetic function regression setting. Specifically, we first define a canonical form of parametric multimodal functions as below:
Yum Gn A.Atttm (bn Fam + cn) + dn, where
Act, E (Sine, Gaussian, Tanh, ReLU).
(13)
Each function Dum is instantiated with the shared parameters (an, Cn. activation functions Ac),,()). We follow Kim et al. (2022a) construct the dataset: We define a uniform grid of Prm 200 coordinates within range of (5. We sample N 1000 function parameters
Cn. d,)) and modality-speeiffc non-lincar

---

Published in Transactions on Machine Learning Research (08/2024)
Table 1: Quantitative comparisons on the multimodal ID synthetic functions. We report the normalized MSEs (x10-2) computed over distinct ranges of sampling ratios, averaged over 5 random seeds. The multi- modal methods are marked with the dagger (t) symbol.
Modality
Sine
Gaussian
Thnh
RcLU
Ruiin
0.01
0.02
0.05
0.01
0.02
0.05
0.01
0.02
0.05
0.01
0.02
0.05
Rango
R
0.02
0.05
0.10
0.02
0.05
0.10
0.02
0.05
0.10
0.02
0.05
0.10
Functa
45.45
16.58
3.316
19.83
4.603
1.043
23.02
3.788
0.587
84.49
13.54
2.992
w/
MetaSGD
44.09
16.46
3.326
19.75
4.334
1.055
26.16
4.528
0.718
69.86
10:45
2.044
GAP
14.76
16.49
3.297
19.34
4.518
1.052
23.46
3.799
0.355
63.17
10.12
1.862
w/
ALFA
48.09
18.67
5.160
19.09
4.910
1.580
22.45
4,200
0.872
53.83
8.303
1.536
W/ HNPs
16.95
8.397
2.547
1.634
0.912
0.293
2.841
0.673
0.091
9.2117
0.772
0.095
w/ MTNPs
13.34
4.718
1.871
2.019
1.513
1.285
4.678
0.340
17.75
1.315
0.398
MIA
6.386
2.058
0.547
1.057
0.571
0.281
1.285
0.378
0.131
5.069
1.012
0.115
Composers
37.90
17.40
5.539
3.495
1.584
14.90
0.975
11.75
w/ MetaSGD
38.26
17.28
6.427
6.887
3.221
1.388
14.99
3.938
0.981
51.97
11.65
3.530
w/ GAP
37.53
17.52
5.397
6.630
3.409
1.526
14.40
3.828
0.978
50.90
10.85
3.128
ALFA
36.53
14.87
4.115
5.650
2.770
1.154
14.18
3.426
0.799
42.96
6.814
1.481
HNPS
20.38
9.480
2.413
2.091
1.075
0.342
4.497
0.991
0.111
19.13
2.698
0.129
w/ MTNPs
16.62
4.859
0.766
2.256
1.252
0.708
4.670
0.743
0.121
11.47
0.897
0.114
W/ MIA
5.564
1.844
0.627
0.975
0.528
0.237
1.257
0.343
0.128
4.715
0.943
0.156
(Ons dns Cn. dn) shared across modalities and add per-modality Gaussian noises 6xmm (0, 0.02) to control the cross-modal correlations. We use [Rmin, R_aa]] [0.01, 0.1] for all modalities.
Results. We present the quantitative results in Table 1 where we report normalized MSEs by the scale parameter anm per function, MSE = x E - 11@m yl following Kim et al. 2022a). The methods with no ability to handle multimodal signal jointly (CAVIA, MetaSGD, CAP, ALFA) fail to approximate the functions, showing high MSEs in all ranges of sampling ratios. In contrast, the multimodal methods (HNPs, MTNPs, and ours) are able to reconstruct each signal more precisely, even with extremely small number of support sets (e.g. R < 0.02). Moreover, while HNPs and MTNPs show strong fitting performances on smooth and low-curvature signals (í.e. Tanh and ReLU) when data sufficient, our method achieves the best performances overall among the multimodal methods, verifying its effectiveness in utilizing cross- mods- interactions to enhance generalization.
5.2 Multimodal 2D CelebA Dataset
Setups. We follow MTNP (Kim et al. 2022a) and conduct experiments on real-world 32 x 32 image function regression settings. In particular, we consider three different visual modalities of 2D facial data on CelebA (Liu et al. 2015). namely RGB images (Karras et al. 2018), surface normal maps, and sketches (Xia
ct al. 2021).
Results. Table 3 presents the quantitative results. The encoder baseline MTNPs outperforms unimodal baselines like CAVIA, MetaSGD, GAP, and ALFA in low-shot settings (R < 0.25). However, with more substantial support (R 2 0.50), MTNPs fall behind ALFA, likely due to the common underfitting issues seen in encoder-based meta- learning approaches, especially with complex target functions featuring high-frequency details (Kim et al. 2019 2022a Guo et al., 2023; Shen et al.) 2023). The poor performance of HNPs can be similarly explained, though they perform betler simpler 1D signal regressions. The original paper on HNPs (Shen al. (023)) might not have shown this degeneracy, focusing on multimodal classification and using pretrained image features (Simonyan & Zisscrman, 2014 He et al. 2016). In contrast, our MIA consistently out oulperforms all baselines across various support set sizes. F'iurre demonstrates that MIA not only excels in generalization in data-scarce conditions but also retains high-frequency details effectively when data is sufficient.
5.3 Multimodal Climate Data
Setups. ERA5 (Hessaahh et al. 2019 is global climate dataset that provides hourly estimates on wide range of atmospheric variables measured globally over grid of equally spaced latitudes and longitudes. Out
8

---

Published in Transactions on Machine Learning Research (08/2024)
Figure 3: Results on the multimodal 2D CelebA image function regression. We report the MSEs (x10
computed over distinct ranges sampling ratios. The multimodal methods are marked with the dagger (f) symbol.
Modality
RGBs
Normals
Sketches
Ruin
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Range
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
Functa.
13.32
4.008
2.900
2.408
5.079
2.401
2.028
1.859
13.57
6.966
5.368
4.756
w/ MetaSGD
13.02
3.830
2.685
2.182
4.923
2.268
1.864
1.682
12.72
6.278
4.532
3.839
w/ GAP
12.84
3.726
2.5413
2.021
4.805
2.181
1.762
1.570
12.
6.023
4.166
3.407
w/ ALPA
11.83
3.257
1.957
1.362
4.115
1.806
1.285
1.014
10.80
4,801
2.463
1.283
w/ HNPS
19.02
10.24
9.728
9.433
14.34
14.38
14.35
14.23
20.36
20.45
20.44
20.57
w/ MTNPs
4.807
4.105
3.644
3.983
2.552
2.339
2.221
9.680
6.568
5.395
w/ MIA
6.946
2.563
1.627
1.135
2.979
1.530
1.118
0.869
7.667
4.042
2.142
1.011
Composers
10.24
6.583
6.415
6.292
15.73
14.63
w/ MetaSGD
20.11
10.25
9.000
8.268
8.218
5.979
5.753
5.601
18.95
15.43
14.49
14.20
w/ GAP
20.07
10.05
8.785
8.039
8.149
5.847
5.616
5.461
18.58
15.24
14.39
14.15
ALFA
15.12
4.887
3.376
2.681
3.444
2.399
1.773
1.469
12.57
5.984
3.416
2.124
w/ HNPS
17.88
12.40
12.13
11.74
7.363
6.419
6.375
6.324
17.70
17.31
17.20
17.34
w/ MTNDS
9.902
4.957
4.269
3.813
4.184
2.747
2.545
2.437
9.791
6.425
5.163
4.544
w/ MIA
9.764
3.418
1.913
1.017
3.749
1.763
1.062
0.526
9.505
4.708
2.336
0.855
Supp.
ALEA
MTNP
Supp.
ALEA
MTNP
Supp.
ALFA
MTNP
Figure Three qualitative comparisons on CelebA. Compared to the baselines, MIA generalizes better in the extremely low-shot settings (Ist and 2nd panes) and preserves more rich high-frequency details nuances (2nd and 3rd panes) when sufficient data is observed.
of all the variables, we consider measuremenss on temperature, pressure, and humidity. Following Dupont ct al. (022c).. we resize each data to 46 x 90 resolution.
Resultss As indicated in Table some unimodal baselines like GAP and ALFA perform well, even sur- passing multimodal methods across various sampling ratios. This performance can be attributed the relatively stable nature of atmospheric variables over time and regions (Howells & Katz, 2019). where the fast memorizing capability is beneficial. Notably, our method surpasses all baselines, allirming its capability in handling real-world multimodal climate data effectively. Supporting evidence of its superior performance is also presented in Figure
5.4 Multimodal Visual AVMNIST Dataset
Setups. Unlike previous setups, modeling audiovisual signals presents unique challenges arising from the heterogencity in coordinate systems and lack of explicit spatiotemporal alignments between modalitios. Wo employ AVMNIST dataset (Vielzeuf et al. 2018), collection of greyscale digit images (LeCun al., 2010) and their pronounced audios [Jackson] 2016). We use the original images of 28 28 resolution, while we trim the audios with sampling rate of 2kHz and duration of 1 second.
Results. In Table both Functa-based CAVIA and MTNPs struggle with the audio modality, likely due to difficulties in handling high-frequeed content and noise. Furthermore, the encoder baseline HNPss fails in both audio and visual signals, possibly due to inadequate fitting of complex audio without the support of expressive pretrained features, which also affects performance on simpler image functions. In contrast, our method excels in both modalities across all sampling ratios, as confirmed by Figure 7 Our approach

---

Published in Transactions on Machine Learning Research (08/2024)
Figure 5: Results on ERA5 dataset in MSEs (x10-4) across different sampling ratios. The multimodal methods are marked in the dagger (t) sign.
Figure Results on AV-MNIST in MSES (x10 across different sampling ratios. The dagger (f) sign denotes multimodal methods.
Modality
Temperature
Pressure
Humnidit
Modality
Images
Audios
Ruiu
0.00
0.25
0.50
0.00
0.25
0.50
0.00
0.25
0.50
Ruii
0.00
0.25
0.50
0.25
0.50
0.75
Range
Ruax
0.25
0.50
1.00
0.25
0.50
1.00
0.25
0.50
1.00
Range R
0.25
0.50
1.00
0.50
0.75
1.00
Functa
8.56
4.04
3.79
2.56
1.39
1.32
33.7
24.2
22.6
Functa
29.7
7.98
3.84
2.98
3.03
3.00
w/ MetaSGD
6.89
3.29
3.10
1.93
0.96
33.1
21.6
19.5
w/ MetaSGD
29.8
7.94
3.71
0.95
0.50
0.31
w/ GAP
6.30
2.90
2.69
1.95
0.98
0.92
31.2
18.8
16.0
w/ GAP
29.8
7.97
3.76
0.93
0.47
0.29
w/ ALPA
4.03
1.64
1.30
0.61
0.19
0.15
28.3
15.1
11.2
W/ ALPA
28.1
6.84
3.12
0.79
0.36
0.20
w/ HNPs
35.0
35.2
34.6
2.85
2.82
2.82
19.6
19.6
19.6
W/ HNPs
50.3
16.2
10.5
2.99
3.03
3.00
w/ MTNPs
4.44
3.65
3.55
26.0
25.4
W/ MTNPs
2.37
2.30
2.23
MIA!
2.03
1.02
0.74
0.46
16.0
11.7
8.47
MIA
19.3
4.85
2.08
0.41
0.21
0.13
Compasers
8.36
7.51
43.2
Composers
0.47
w/ MetaSGD
9.65
8.47
8.38
2.41
2.29
2.28
37.5
48.9
48.4
MetaSGD
34.6
11:4
4.48
1.00
0.46
0.22
w/ GAP
8.88
7.74
7.65
2.77
2.43
2.43
46.0
42.1
41.5
W/ GAP
34.4
11.2
4.15
1.06
0.53
0.29
w/ ALFA
7.31
6.05
6.36
1.95
1.86
1.83
38.5
34.3
34.7
ALEA
31.9
8.27
2.46
1.09
0.48
0.22
w/ HNPS
39.6
40.0
39.3
4.18
4.16
4.16
49.4
49.5
49.5
HNPS
45.6
18.6
14.8
2.99
3.03
3.00
w/ M
4.52
3.61
3.49
1.38
1.32
1.31
28.5
25.3
24.4
W/ MTNPS
30.4
13.2
8.70
2.78
2.80
2.77
w/ MIA
3.93
2.48
1.40
1.13
0.68
0.11
24.1
16.6
7.77
W/ MIA
18.7
4.24
1.32
0.65
0.29
0.15
Image
Audio
GT
Support
ALFA
MTNP
MIA
GT 7
Humid. Press. Temp.
Support
MTNP
MIA
Figure 7: Qualitative comparisons in ERA5 and AV-MNIST. Left: Our method achieves the lowest errors
(shown in viridis color) in modcling real-world climate data. Right: Our method accurately predicts digit type with little image data and recovers the audio laithfully.
not only accurately reproduces audio signals but also elfectively predicts digit classes from minimal image support, showcasing robust fitting and moos-mmdall generalization capabilities.
6 Analysis
In this section, we delve into an in-depth analysis of our method.
6.1 Ablation Study
Impact of modules. We investigate the roles of the three modules (i.e. USFTs, MSFTT, and Fusion MLPs) of SFTs in terms of both memorization and generalization performances. We construct ablated methods by gradually augmenting Composers with USFTs, MSFTs, and Fusion MI.Ps one a time. We evaluate their fitting performances over the provided support the performances memorization and generalization, respectively. We report the relative performance improvement (Maninis et al., 2019 Kim et al. 2022a) achieved by each method compared to the vanilla Composers, averaged across all signals and modalities. As in the results of Table 2a USFTs specialize in enhancing modality-specific patterns that are useful for memorization, while MSFTs further emphasize shared information for generalization. Finally, Fusion MLPs effectively integrate the representations from USFTs and MSFTTs, achieving superior performances on both memorization and generalization. Note that the reported improvement here do not specifically account for the sampling ratios of each modality, whereas the efficacy of the MSFTss becomes more apparent in cases of large imbalance in these sampling ratios across
and unobserved non-support parts separately, indicating
10

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Ablation study on SFTs on generalization (G) and memorization (M) capability, We report relative
improvement (f) achieved by ablated methods over vanilla Composers.
Modules
CelobA
AVMNIST
States
ColebA
AVMNIST
USTTs
MSFTs
Pusion MLPs
G
M
G
M
Params
Grads
Scaling
G
G
M
x
X
N/A
N/A
N/A
N/A
x
X
58.3
92.1
60.7
84.1
x
x
54.3
68.7
12.6
13.0
92.5
63.9
60.8
87.3
61.4
93.0
65.6
88.7
61.4
93.0
65.6
88.7
(a) Impact of USFTs, MSFTs, and Fusion MLPs.
(b) Impacc of parameters, gradients, and scaling.
pearson correlation Cy
source:normal.sketchh
target:rgb
target:rgb
assigned attention weight Skotch
+1.00
improvenu (%6
0.01
R6BB
0.83
-0.84
-0.67
+0.75
30-
0.05
+0.50
25
0.25
Nornal
30-
-0.83
0.84
-0.53
+3.00
20
-0.25
-0.50
relatvie
-0.86
-0.83
0.93
-1.00
0.05
support set size
support size of source modalities
support set size of source modalities
(a) Pearson Correlation Cig-
(b) Multimodal Transfer Analysis.
Figure 8: Do our MSFTs identify and leverage cross-modal relationships as expected? Left: Pearson Corre- lation Cij between attention weights assigned to modality MSFTs and the support set size for modality j. Positive correlation indicates a tendency of modality attend more modality j when it has sufficient data. Right: Relative performance improvement (f) when multimodal support set sizes increase. Colors indicate support set sizes of the target modality, while the x-axis represents that of the source modalities.
modalities. For a detailed analysis of this aspect, we refer the readers to discussions on Figure 8b and Table Also, for comparisons and discussions on all possible ablated methods, please find Appendix E.5
Impact of learners' states. We also study the impact of states utilized by SFTs by ablating parameters, gradients, or the meta-learned gradient scaling from SFTs. Similar the previous analysis, we report their relative performance improvement compared Composers. Table 2b reports the results. We find the parameters-only method fails entirely (reported as N/A). In contrast, the gradients-one only method shows enhancements compared to vanilla Composers, and this improvement is significantly boosted with the meta- learned gradient scaling. This suggests that multimodal gradients indeed contain useful information for predicting better update directions of each INR learner, and a proper scaling technique is crucial for their effective utilization. Finally, the best performances are achieved when employing all of them. This implies that learners' current knowledge on signals (i.c. parameters) can be incorporated further to provide SFTs with more comprehensive view on the optimization process, leading better parameter adjustment.
6.2 Adaptive cross-modal utilization of MSFTs
We validate whether our MSFTs can (1) identify valuable cross-modal patterns in learners' states and (2) utilize them to enhance the learning of INRs. For the first aspect, examine how attention patterns of MSF'Ts relate to the quality of the support sets. We calculate the Pearson correlation coefficient Otj, which reflects the relationship between attention weights assigned to the learner's state reprosentations for modality and the support set size for modality j. Notably, as presented in Table [al we observe strong positive correlations along the diagonal. This reveals that, when observations within a specific modality suffice, MSFTs refrain from disrupting strong negative correlations off the diagonal imply that MSFTs tend compensate for suboptimal state representations driven by limited data in one modality by attending more to the other modalities.
state representations with those from other modalities. Conversely, the
11

---

Published in Transactions on Machine Learning Research (08/2024)
Table 3: Relative performance improvements (1), parameter counts (L), memory usage (4), and meta-training time per iteration (t) for methods considered in our paper. Notably, encoder baselines were excluded due out-of-memory issues. The analysis illustrates increased costs with SFT integration, mitigated by higher resolutions and paramcter alignment.
Resolution
Image
Method
Relative Performance
Improvement (%)
# Params
Memory (GB)
Training Time (ms/it)
CAVTA
3.00
60.0
MetaSGD
2.02
221K
3.01
60.2
GAP
-2.05
224K
3,01
66.8
ALFA
55.1
299K
3.01
67.2
Large
-16.5
2.08M
435.9
32 x 32
MetaSGD-Large
-10.3
2.08M
7.39
449.2
GAP-Large
-0.69
8.00
425.6
ALFg-Large
50.30
2.17M
7,69
154.5
HNP MTNP
-4.60
1.90M
58.47
8107.1
53.4
11.85M
44.35
2031.3
MIA
69.6
1.93M
3.61
136.2
CAVIA
0.00
298K
23.69
765.33
ALFA
8.86
644K
24.43
755.90
128 x 128
HNP
N/A
2.01M
OOM
N/A
MTNP
N/A
21.38M
OOM
N/A
MIA
2.04M
25.62
942.22
To examine the second aspect, assess the performances modal support sets from other modalities are increased. As in previous experiments, report the relative improvement by our method compared to the one obtained without the supports from the other modalities. Figure [b] visualizes the results that, in general, increased availability of multimodal support sets contributes to better performances in the target modality (i.e. increasing gains along the x-axis), while the benefit from multimodal observations becomes more significant. when the size of the target support sets smaller (í.e. blue lines). Based this analysis, we confirm MSFTss ability to correctly identify and leverage moosl interactions, which is particularly beneficial when modeling signals with scarce observations.
target modality while the sizes of multi-
6.3 Computational Overhead
Our SFIs add computational complexity over vanilla CAVIA during both meta-training and meta-testing phases. To justify this, we compare the computational resources required for both phases separately.
Meta-training Computational Overhead. Table 3 outlines the number of parameters, memory con- sumption, and required meta-training time per iteration for each method, measured on the 2D multimodal CelebA dataset. Additionally, we analyze larger baseline models (CAViA-Large, MetaSGID-Large, GAP- Large, and ALFA-Large) with parameter counts adjusted to match our MIA, along with models trained 128 x 128 image resolutions. Unfortunately, encoder-based multimodal baselines encountered out-of-memory (OOM) issues at this larger solution setting, even with a batch size of one, and were excluded from the comparison.
As shown the table, the integration SFTss into our MIA framework slightly increases memory overhead and doubles the training time compared unimodal frameworks. However, the additional resource require- ments become less significant. with higher resolution inputs and are offset when baseline parameters are aligned with those of MIA. This efficiency benefits from SFTs maintaining consistent computational costs relative to sample size, in contrast to the lincar complexity of INRs, and quadratic cost related to the number of INR context parameters, which scales more favorably with signal resolution as noted by Dupont et al. 2022a).
Meta-testing Computational Overhcad. Finally, we examine the time required by the optimization- based methods (CAVIA, GAP, and ALFA) to match the 3-step adaptation performance of our method during meta-testing phase, with respect to varying support set sizes. The results depicted in Figure g utilizing
12

---

Published in Transactions on Machine Learning Research (08/2024)
500
CAVIA
500
inference time (ms)
GAP
400
ALFA
400
MIA
300
300
200
200
100
100
0
0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
support set size (R)
support set size (R)
Figure Inference time comparisons for baselines to achieve 3-step adaptation performances of our MIA within 100-step optimization limit. Left: Cases where the basclines successfully match our performances. Right: For the cases where the baselines fail to match ours, we plot times for their best-achieved performances.
RGB modality in the 2D multimodal CclebA datasct. We showcase two distinct sccnarios separately: one the left where the baselines successfully match our performances, and another on the right where they fail to do so, all within the constraint of 100-step optimization limit. In cases where the baselines do not report the inference time for their early-stopped best-achieved MSEs. Based on
match our performance, the figures, we conclude: (1) Even with 100 optimization steps, matching our 3-step performance remains a challenge for the baselines. (2) The baselines exhibit a tendency towards early overfitting with smaller support sets or underfitting with more data, (3) Even when they successfully match our performance, they
often require more inference time (up to 421 ms) and optimization steps.
7 Discussion
In this paper, introduce new optimization-basedd meta-learning framework for INRs, substantially euhancing the performances of existing methods. This improvement is primarily attributed to our novel Multimodal Iterative Adaptation (MIA), which enables separate INR learners to share optimization processes through State Fusion Transformers (SFTSs. This unique multimodal adaptation scheme leverages cross-modal interactions among learner states, facilitating rapid adaptation and boosting performance.
Despite the improvement, we also acknowledge the fundamental challenges of scaling up meta-learning al- gorithms. In particular, the high memory requirement necessary for bi-level optimization, such as retaining all computational graphs in the inner loop to compute exact second-order derivatives for training the meta- learners in the outer loop, hinders their application to more complex high-defiition signals. To overcome such limitations, we recognize that recent advancements in the scalable meta-learning domain present viable paths to enhance the scalability of meta-learning frameworks, including ours. Notably, techniques like con- text pruming during meta-training (Tack et al.
2023 efficient estimation of second-order gradients (Chen
2023 Jain et al., 2023). and the design of more efficient et 2022a; Metz et al. 2022b Choe al.
brandsormer--ased meta-learnen (Jain et al. 2023) have demonstrated considerable promise; through these techniques, meta-learning algorithms have been effectively extended to address more complex real-world
scenarios, including the successful scaling of meta-learned INRs for 1024 x 1024 images or 256 x 256 x 32 videos Tack et al. 2023). and the optimization of medium to largo-scale models such as BERTs (Devlin el al. 2019). RoBERTas [Liu el al.) (2019), ViT-H (Dosovilskiy et al. 2021), ) or models even exceeding 11 11B+ parameters like T5-XXL RRafel et al.) (0220) Chen et al. 2022a Metz et al. 2022b Choe et al.) 2023 Jain ct al. 2023). Additionally, exploring more efficient (Choromanski et al. 2021) or local (Liu et al. 2021)
attention mochanisms, along with curriculum learning strategies such two-staged meta-learning scheme
where first. meta-learn independent. unimodal frameworks with USFTs for each modality, followed by
13

---

Published in Transactions on Machine Learning Research (08/2024)
joint learning of the entire multimodal framework augmented with MSFIs, could be promising avenue for reducing the computational overheads of our SFTs further. The integration of these techniques for scaling up our framework presents an intriguing direction for future research, which we intend to explore in our subsequent work.
Acknowledgment
This work was supported by the Institute of Information & Communications Technology Planning & Eval- uation (ITTP) grant funded by the Korea government (MSIT) (No. RS-2021-11211343, Artificial Intelligence Graduate School Program (Seoul National University)), the ITTP grant funded by the Korea government (MSIT) (No. RSS-002-INT 2022-11220156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation). IGG AT Research (Learning Robust and General- Purpose Multimodal Representation), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573.
References
Milad Abdollahzadeh, Touba Malekzadeh, and Ngai man Cheung. Revisit multimodal leeta-lernnig through the lens of multi-task learning. In NeurIPS, 2021.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. NeurIPS, 29, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-moda multi-tase masked autocncoders. In ECCV, 2022.
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lec. Learning to learn (ask-adaptive hyperparameters for few-shot learning. TPAMI, 2023.
Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Richard Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation, 2023.
Lucian Chan, Marcel Verdonk, and Carl Poelking. Embracing assay heterogeneity with neural processes for markedly improved bioactivity predictions. In NeurIPS 2023 Workshop on New Frontiers of AT for Drug Discovery and Development, 2023.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. In NewrIPS, 2021a.
Xuxi Chen, Tianlong Chen, Yu Cheng, Weizhu Chen, Ahmed Awadallah, and Zhangyang Wang. Scalable learning to optimize: A learned optimizer can train big models. In Computer Vision ECCV 2022, 2022a.
Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural reprosentations. ECCV, 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, 2021b.
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In CVPR, 2022b.
Sang Keun Choe, Sanket Vaibhav Mehta, Hwijeen Ahn, Willie Neiswanger, Pengtao Xie, Emma Strubell, and Eric Xing. Making scalable meta learning practical. In NewrIPS, 2023.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Loe, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapler of the Associalion for Compulational Linguisties: Human Language Technologies, Vol- ume 1 (Long and Short Papers), June 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x 16 words: Translormers for image recognition at scale. In ICLR, 2021.
Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In ICML, 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Tch, and Arnaud Doucet. Coin++: Neural compression across modalities. 2022b.
Emilien Dupont, Yee Whye Tch, and Arnaud Doucet. Generative models distributions of functions. In AISTATS, 2022c.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCML, 2017.
Elizabeth Fons, Alejandro Satrajman, Yousef El-Laham, Alexandros Iosifidis, and Svitlana Vyetrenko. Hy- pertime: Implicit neural representations for time scries. In NewrIPS Workshop, 2022.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Tch. Neural processos. In ICML Workshop, 2018.
Pengsheng Guo, Chen- Yu Lee, and Daniel Ulbricht. Learning to branch for multi-task learning. In ICML, 2020.
Zongyu Guo, Cuiling Lan, Zhizheng Zhang, Yan Lu, and Zhibo Chen. Versatile neural processes for learning implicit neural representations. In ICLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
H Hersbach, B Bell, P Berrisford, G Biavati, A Horányi, J Muñoz Sabater, J Nicolas, C Peubey, R Radu, Rozum, et al. Era5 monthly averaged data on single levels from 1979 to present. Copernicus Climate Change Service (C3S) Climate Data Store (CDS), 2019.
T. Alfred Howells and J. 1. Katz. Long term trends in atmospheric pressure and its variance. arXio: Atmospheric and Oceanic Physics, 2019.
Zohar Jackson. Spoken 2016.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021.
Deepali Jain, Krzysztof Marcin Choromanski, Kumar Avinava Dubey, Sumeet Singh, Vikas Sindhwani, Tingnan Zhang, and Jie Tan. Mnemosyne: Tearning to train transformers with transformers. In NeurIPS, 2023. URL
Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodriguez, Chao Zhang, and B Aditya Prakash. Camul: Calibrated and accurate multi-view time-series forecasting. In Proceedings of the ACM Web Conference 2022, 2022.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Suhyum Kang, Duhun Hwang, Moonjung Eo, Taesup Kim, and Wonjong Rhee. Meta-learning with
geometry-adaptive preconditioner. In CVPR, 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICLR, 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
Chiheon Kim, Doyup Lee, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattorn composers. In CVPR, 2023.
Donggyun Kim, Seongwoong Cho, Wonkwang Lee, and Seunghoon Hong. Multi-task processes. In ICLR, 2022a.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In ICLR, 2019.
Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. In NewrIPS, 2022b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2014.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit databasc. ATT Labs [Online]. Available: hllp://yamn, www.aawww.ww.....ooooooooo www./oooo/weoo//ooeoooooeoeeeeee 2010.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NewrIPS, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning, 2017.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis- Philippe Morency, and Russ Salakhutdinov. High-modality multimodal transformer: Quantifying modality interaction heterogeneity for high-modality representation learning. TMLR, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
Kovis-Kokitsi Maninis, Iijaa Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In CVPR, 2019.
Luke Metz, C Daniel Freeman, James Harrison, Niru Maheswaranathan, and Jascha Sohl-Dickstein, Practical tradeoffs between memory, compute, and performance in learned optimizers. In CoL.AAs, 2022a.
Luke Metz, James Harrison, C. Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, and Jascha Sohl-Dickstein. Velo: Training versatile Icarned optimizers by scaling up, 2022b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Ghulam Muhammad, Fatima. Alshchri, Fakhri Karray, Abdulmotalcb El Saddik, Mansour Alsulaiman, and Tiago H. Falk. A comprehensive survey on multimodal medical signals fusion for smart healthcare systems. Information Fusion, 2021.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Jonathan Richard Schwarz, Jihoon Tack, Yee Whye Teh, Jaeho Lee, and Jinwoo Shin. Modality-agnoslic variational compression of implicit neural representations. In ICML, 2023.
Jiayi Shen, Xiantong Zhen, Marcel Worring, et al. Episodic multi-task learning with heterogeneous neural processes. NewrIPS, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2014.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020.
Ya Sun, Sijie Mai, and Haileng Hu. Learning to learn better unimodal representations via adaptive multi- modal meta-learning. IEEE Transactions on Affective Computing, 2023.
Jihoon Tack, Subin Kim, Sihyun Yu, Jaeho Lee, Jinwoo Shin, and Jonathan Richard Schwarz. Learning large-scale neural fields via context pruned meta-learning. In NewrIPS, 2023.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In CVPR, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In CVPR, 2021.
Kyle Ulrich, David E Carlson, Kafui Dzirasa, and Lawrence Carin. Gp kernels for cross-spectrum analysis. In Advances in Neural Information Processing Systems, 2015.
Valentin Vielzeuf, Alexis Lechervy, Stephane Pateux, and Frederic Jurie. Centralnet: a multilayer approach for multimodal fusion. In ECCV Workshops, September 2018.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Toward multimodal model-agnostic meta- learning. In NewrlPS Workshop, 2018.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim, Multimodal model-agnostic meta- meta-learning via task-aware modulation. In NeurIPS, 2019.
Bin Wang, Jie Lu, Zheng Yan, Huaishao Luo, Tianrui Li, Yu Zheng, and Guangquan Zhang. Deep uncertainty quantification: A machine learning approach for weather forecasting. In SIGKDD, 2019.
Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real-world perception for embodied agents. In CVPR, 2018.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Weihao Xia, Yujiu Yang, Jing- Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image gener- ation and manipulation. In CVPR, 2021.
Sibyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In ICLR, 2022.
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Caml: Fast context adaptation via betaaleerning, In 1CML, 2018.
18

---

Published in Transactions on Machine Learning Research (08/2024)
Appendix
This section provides more results and details that could not be included in the main paper.
A
More Qualitative Results
Sine
Gaussian
Tanh
ReI.U
Sine
Gaussian
Tanh
ReI.U
MeaSGD CAVIA
M
M
GAP
ALEA
M
Funeta
M
HNP
M
MINP
M
MIA
Sine
Gaussian
Tanh
RcLU
Gaussian
Tanh
ReLU
GAP MenaGG CAVIA
MAME
M
M
M
ALFA TALT
M
Composess
M
HNP
M
MIA MINP
M
Figure 10: Qualitative comparisons on 1D synthetic functions. The black and blue lines represent the ground- truth and approximated signals recovered by each method, respectively, while red crossmarks pinpoint the locations of the provided support points. While all the mothods operate well in relatively smooth and low- curvature signals (e.g. Tanh and ReLU), the baselines either fail entirely in Sine and Gaussian modalities struggle to approximate them correctly (i.e. the multimodal baselines). Unlike the baselines, our MIA fits almost perfectly to all functions, verifying its capability in fusing multiple source of information to improve the performances.
(i.e. the unimodal approaches)
19

---

Published in Transactions on Machine Learning Research (08/2024)
GT
Support
CAVIA
MetaSGD
GAP
ALFA
HNP
MTNP
MIA
RGBs
Normals
Functa
Sketchs
RGBs
Composers
Normals
Skerchs
GT
Support
CAVIA
MetaSGD
GAP
ALFA
HNP
MTNP
MIA
RGBs
Normals
Funeta
Sketchs
RGBs
Composers
Sketchs Normals
Figure 11: Qualitative comparisons on CelebA 2D visual modalitios. Compared to the basclines, our MIA generalizes better in that it successfully discovers the sunglasses from the observed Normals and Sketches, followed by transferring this knowledge RGBs for generalization (see the upper pane). Similarly, as shown in the lower pane, MIA captures the eyeglasses in RGBs and Normals and then transfer this knowledge to recover Edges more accurately.
20

---

Published in Transactions on Machine Learning Research (08/2024)
GT
Support
CAVIA
MetaSGD
GAP
ALFA
HNP
MTNP
MIA
RGBs
Normals
Functa
Sketchs
RGBs
Composers
Normals
Skerchs
GT
Support CAVIA
MetaSGD
GAP
ALFA
HNP
MTNP
MIA
RGBs
Normals
Functa
Sketchs
RGBs
Composers
Normals
Sketchs
Figure 12: Qualitative comparisons on CelebA 2D visual modalities. Compared to the baselines, our MIA can capture sophisticated nuances and render high-frequency details contained in each modality signal thanks to its inheritance of iterative optimization processes from the unimodal methods. In contrast, the multimodal encoder-based methods produces overly smooth predictions suffer from underfitting problems, due to the direct data-to-parament mapping without optimization steps.
21

---

Published in Transactions on Machine Learning Research (08/2024)
Temparatures
Humidities
GT
Support
GT
Support
GT
Support
MeaSGD CAVIA
Funeta
ALFA
HNN
MINP
Prediction
Absolute
Prediction
Prediction
Temperatures
Pressures
Humidities
GT
Support
GT
Support
GT
Support
CAVIA
Composers
TNNP
Prediction
Absolute Error
Prediction
Absolute Eror
Prediction
Absolute Error
Figure 13: Qualitative comparisons on ERA5 global climate data. The first row of each pane indicates the grount-truth signals and provided support, points, while the represents the approximations and their absolute errors achieved by each method. Our MIA shows superior generalization capability than the existing basclines, demonstrating its versatility in applications for real-world climate estimation.
22

---

Published in Transactions on Machine Learning Research (08/2024)
Temparaturrs
Humidities
GT
Suppoet
GT
Support
GT
Support
MenSGD CAVIA
Funera
Prediction
Absolute
Prediction
Prediction
Temperatures
Pressures
Humidities
GT
Support
GT
Support
GT
Support
MetuGD
Composers
TNNP
Prediction
Absolute Error
Prediction
Absolute
Prediction
Absolutc
Figure Qualitative comparisons on ERA5 global climate data. The first row of cach pane indicates the grount-truth signals and provided support points, while the rest represents the approximations and their absolute errors achieved by each method. Our MIA shows superior convergence speed than the existing baselines, which struggle to approximate the climate data precisely and show high errors even when abundant obscrvations are available.
23

---

Published in Transactions on Machine Learning Research (08/2024)
Images
Audios
Images
Audios
Images
Audios
3
Support
CAVIA
MetnSGD
Funetaa
GAP
ALFAA
HNP
MTNP
MIA
5
Images
Audios
Images
Audios
Images
Audios
3
GT
Support
CAVIA
MetaSGD
Composers
GAP
ALFA
HNP
MTNP
MIA
5
3
Figure 15: Qualitative comparisons on AV-MNIST. Our method successfully infer the digit classes from audio signals when little data is available from images (left two columns), while enjoying superior fitting performances when sufficient data is available (rightmost column).
24

---

Published in Transactions on Machine Learning Research (08/2024)
B Details of the INR frameworks
In this work, we adopt the meta-learning frameworks for INRs proposed in Functa (Dupont ct al. Bauer et al., 2023) and Composer (Kim et al. 2023). They builds upon CAVIA Zinngraal et al. 2018 where two separate parameters are meta-learned: the INR parameters 0 and the context parameters Φ E SxDs, In particular, the parameters 0 of INRs are meta-learned to capture underlying structure of data or data- agnostic shared information across signals. In addition, the context parameters ø E Rx5x, a set of Dø- dimensional features, are adapted to each signal and encode per-signal variations or characteristics, which are then utilized to condition the parameters of INRs to recover the signals they are modeling, Besides the shared concepts, they differ in how they construct and utilize the context paramcters to condition the INR parameters, which we detail below.
2022a
Functa. In Functa, the context parameters o, Sx for each signal are constructed as a spatially- arranged grid of Dy-dimensional features, where each of them encodes local variations in a signal. This grid of features is then utilized to modulate the activations of each INR layer. To do so; the set of features first processed by an additional linear convolutional module thn. Scamr (#u; Oeonr), followed by bilincar/ Interp(a;e>) that scale and/or /oror shift the activations of each layer in INRs given an input coordinate z. In the original paper, the authors of Functa opt to adopt a shift-only modulation scheme since it shows better rate-distortion (comprossion vs performance) tradc-offs than using both. In contrast, we adopt a scalc-only modulation scheme since it is empirically shown to perform slightly better in our earlier experiments. Also, we do not modulate the activations of the first and last layers since we empirically find it stabilizes the training without hurting the performances.
interpolation to compute layer-wise affine transformation parameters (x)
Composers. Unlike Funcia that adopt local grid of features and FiLM-like modulation scheme (Perez el al. 2018) applied to multiple layers of INRs, Composers introduce set of non-local features Vn E RSXD that are used as a low-rank approximation W, UVn on parameters Wn, of a single layer in INRs. Here, U E SXDD, is another low-rank approximation matrix that is incorporated in the INR parameters O and captures global patterns shared across various signals, which are composed and modulated in a data-specific manner via Vn. Following the original work (Kim et al. 2023), we approximate the second layer of INRs using U and V..
Throughout all experiments, we use a 5-layer MLPs with 128 hidden dimensions and ReLU non-linearities in-between as a base INR architecture for both Functa and Composers. We also use random fourier fca- tures Tancik el al.] 2020) with a = 30 to encode positional information, except that we do not use these features for the experiments on the ID synthetic dataset.
C Details on State Fusion Transformers (SFTs)
We construct USFTs and MSFTs using the transformer block of VIT (Dosovitskiy et al. 2021 where each of them parameterizod by one transformer block (í.e. L, = L2 = 1) with width of 192, a MLP dimension of 192, and 3 attention heads. In addition, we set the dinension of state representations (R) Sm.DD D: to D2 = 192 for both USFTs and MSFTs in all experiments.
(k) To compute the state representations Znm for each modality, we first scale the gradients by Eq 12 and then apply LayerNorm (Ba et al.) 2016) to the parameters Φmn E R DD, and the scaled gradients 9mn E R xD, separately to stabilize the training. Then, we concatenate them along the feature dimension (Lee [0mn ll 9mm] E Sm. x2D)), followed by projecting them to the hidden space of USFTTs and obtain the state representations (k) via a two-layer MLPs with hidden dimension of 192 and ReLU activations, which is dedicated to each modality. In addition, we add positional embedding to the state representations before feeding them into USFTs, where we use sinusoidal positional encodings (Mildenhall et al. 2020) and learnable embedding for Functa and Composers, respectively. Finally, we parameterize Fusion MLPs with a 3-layer MLPs with a hidden dimension of 192 and ReLU non-lineat activations. Also, we embed LayerNorm into the first and the penultimate layers of Fusion MLPs to stabilize the training.
We also attach Py PyTocc-styyle pseudo code for MIA in Listing 1 and SFTs in Listing 2 respectively.
25

---

Published in Transactions on Machine Learning Research (08/2024)
def inner miciiir__ode_dict, aft_model, modes, etx_params_ _icct, x_spt_dict, y_spt_dict):
CCode for Equations (6) and (7).
Arguments:
inr_ llmoddl_ddctt:. A dictionary of INRs for each modality.
fft_model: State Pusion Transformers.
modes: list of modalities (e.g. ['rgb', 'normal', 'sketch']).
dictionary of INRs' the content parameters for each modality. spt dict: A dictionary of the provided support coordinates for each modality.
spt dict: A dictionary of the provided support features for each modality.
11
12
grad_dict dict dict()
13
for mode in modes:
# for each modality, obtain predicted features given support cooridnates. y-pred dnr_model_dict [mode] [mmp_eddcctmmmdee, ctx_params_dae [mode]] y_ept_dict(xodaa
15
16
loss mse_loss
17
# compute gradients w.r.t t content parameters.
grad_ [ad__dic[[mode]] = torch. autograd ( loss,
19
20
ctx_ _cx_params__m walpppaaaa_icttiiieee [mode])
31
create_graph True
# Set to "True" during meta-training, otherwise 'False". [o]
23
24
# Use SFTs to
25
# 1. consider cross-modal interactions among the learners
# 2. and enhance their weight udpates.
27
# Please refer Listing for the details. grad_dict fuse_ aas__sttaesssft_motte modes, grad_dict, ctx_parame_dict) for mode in modea:
38
29
# up date contezt parameters using the enhanced gradients for each modality. ctx_params_diae dict [mode] ctx_params_diie aiiaatssdiiiiiiieeeeeeeee dict [mode] grad_dict [mode]
31
32
33
return ctx_parana_dict
Listing 1: Torch style pseudo-code for inner optimization step via. MIA.
26

---

Published in Transactions on Machine Learning Research (08/2024)
def fuse_stte aase_statessste (sft (sft modes, grad_dict, aaapaaasaaiciii 1Code for Equations (8), (9), (10), and (12).
Arguments:
sft_model: State Fusion Transformers.
modes: list of modalities.
grad_dict. dict: dictionary of the gradients w.r.t the content parameters for each modality.
eta_params_ dict: A dictionary of the content parameters of INRs for each modality.
ori_state_dict dict()
10
umi_state_dict dict()
milti_stateddia dict()
12
states []
13
for mode in modes:
14
# scale gradients from each modality with the meta-learned scaling constant.
[mode[ *= sft_ sft_model, fttt_odel.grad_1 geed__aalinggmooo [mode]
16
# project gradients and content params to hidden space
17
ori tri_state_dice [mode] aft_model..res _mpp [mode] (grad_dictt [mode],
15
stx_params_dice [mode])
# add positional embedding and apply USFT for each modality sft_model, pos_pos__ee [mode]
20
state tri_state_dicee
21
umi_state____ dict [mode] eft_model. uaft [mode] (state) states. aww.ees..pppndddccc (uni_state_____ee dict [mode]]
22
# concatenate states for all modalities and apply MSFT states torch. cat (states, dim 1)
25
36
states sft_model..cm maft mafttsttee))))
for mode in modes:
28
# calculate enhanced gradients by applying FustonMLP for each modality multi_state_diee [mode] split(states, mode)
29
90
state = torch. cat [lorr__tate_dicttfaaaeee
[oodd)
32
multi ietti__tttt_dicttnooooi dim -1)
33
grad_dict[mode
sft_model. wwl.model.fusionn. fusion__llp [mode] (state)
34
return grad_dict
Listing 2: PyTorch style pseudo-code for State Fusion Transformers (SFTs).
27

---

Published in Transactions on Machine Learning Research (08/2024)
Table
List of common configurations for each dataset.
Hyperparameters
Syuthetic
CelebA
ERA5
AVMNIST
RGB
Temperature
Sine, Gaussian Tanh, ReLU (M 4)
Images Audios (M = 2)
Normal Sketch (M = 3)
Pressure Humidity
modalities
(M 3)
train
900
27143
11327
60000
number of signals (N)
Lest
100
2821
2328
10000
batch size
64
32
16
32
epoch
16,000
300
300
300
RGB (32, 32, 3)
Images (28, 28, 1)
resolution
(200, 1)
Normal (32, 32, 3)
(46, 90, 1)
Audios (2000,
Sketch (32, 32,
RGB 1024
Images 784
number of original function samples (P)
200
Normal 1024
4140
Audios 2000
Sketch 1024
sampling ratio
Images 00.001
[0.01, 0.1]
[0.001, 1]
[0.001, 1]
Audios [0.250.
outer learning rate momentum (2)) for Adam total inner step K for optimization-Biii based methods
10
(0.9, 0.999)
scale for uncertainty Ir width/depth INRs
0.1
10
128/5
Images (8, 8, 4)
dimension of context parameters
Functa
(8, 16)
(8, 8, 16)
(8, 16, 16)
Audios
(64, 32)
Images
(64, 128)
Composer
(8, 128)
(64, 128)
(128, 128)
Audios
(64, 128)
for fourier feature
None
30.0
30.0
30.0
bootstrapping factor for evaluation
10
4
4
More Experimental Details
D.1 Common Details
In all experiments, we interpret data as a set of cordinato-feature pairs and normalize the values of both coordinates and features. Specifically, for coordinates, the values are normalized within the range [-1, 1j4, where d varies with the dimensionality of the datasct. For fcatures, normalization varies by data type: values are normalized to the range [0, 1] for image and climate data, and to [-1, for audio signals. Normalization is performed by computing the minimum and maximum values across all data points for each dataset, which are then used to scale each data point accordingly. In the case of the 1D synthetic functions, we do not normalize the features; instead, we use the normalized MSE for evaluation.
We subsample coordinate-featur pairs from the full set during the meta-training phase to promote the gencralization, as well as memorization. For each signal D Dam, the sampling ratio Rym is independently and identically drawn from uniform distribution U(Rmiin Rmma) ) for each modality m to construct the support sets Dirsin For evaluation, employ bootstrapping technique the meta-test dataset. We iteratively
28

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 2 Meta-Learning Algorithm for Optimization based Baselines.
Require: Batch size B; Inner step K; Outer learning rates λe.
1: Initialize θ
2: repeat
Sample a batch of B joint multimodal signals [{Dum 1M Im= m==1/n=1 Sample Rum ~ [( Rmpin Rmp for Vn, m. Split Dum into Dirain Deal where [Dtrain| Pum Ram and [Dwa) Pam-
6: for 1, ,,.., B do for k = O.....
7:
......K.... K do
8:
for m = ......,. M do
(k) V L(Durain Om: Prm
Jnm
am
10:
Prm
(k+1)
update @rm: From)
(k)
Inner optimization
11:
end for
12:
end for
13:
end for
14:
Louter 1 DE ((Dal,
ami m, Pmm ).
B
15:
θ O - AeVe Lomter.
D. Outer optimization
16:
until converged
sample the support and query sets multiple times for each data to mitigate potential variances that may arise from the sampling process. For 1D synthetic experiments, apply bootstrapping factor of 10, while for other scenarios, the factor is set to 4.
In all experiments, we use Adam optimizer (Kingma & Ba 2014) for meta-optimization, with learning rate of 10-4 and the momentum paramcters are set as (81, 82) (0.9, 0.999).
Also, we apply uncertainty-aware loss weighting technique (Kendall et al. 2018) for all multimodal methods. Please find Table 4 for the clear list of common configurations for each dataset.
D.2 Baselines
In this section, we provide the more explanation of baselines and implementation details. To facilitate better comparison with MIA, we include a pseudo-code for optimization basod meta-learning basclines (CAVTA, MetaSGD, GAP, ALFA) in Algorithm 2 In this context, the meta-learnable parameters Q and the update(-) function differ according to each baseline's unique optimization procedure.
CAVIA. We use global fixed learning rate of a = 1.0 to adapt the context parameters of CAVIA-like methods (Functa and Composers) in the inner-loop for all experiments. Here, θ = {), and the update (k)
(k) (k) ((k) am.nmm rule is defined as update(dlan, (altee (rmm
MetaSGD. Li et al (017) propose inner-loop adaptation phase, which is optimized along with the leta-learner in the outer-loop optimization phase. We apply this technique to adapt the context parameters of CAVIA-based frameworks, where we initialize their initial values to 1.0 in all experiments. In this setup, θ = [0, Φ, a), and the update rule is defined update(dlmn ghi)
meta-learned per-parameter learning rate RXx in the
(k) (k) d am gon.. (k)
GAP. Kang et al. (0233 propose to acceeerate the optimization process via Geometry-Adaptive Precon- ditioner (GAP), which preconditions the gradients g(k) at inner step k by manipulating its singular values with meta-learned parameters M. The procedure can be written as:
g(6)_ ( (M (kk) (k)", where ((a) (kk)s(())
(14)
In the original paper KKng et al. 2023 gradient matrix unfolding technique introduced to facilitate SVD the gradients of convolutional weights. Different from the original setup, now the shape of the context
29

---

Published in Transactions on Machine Learning Research (08/2024)
parameters and their associated gradient matrices is R Therefore, we do not using this unfolding tech- nique and define the meta parameters as M = (Sing(S((M)).... ..... Sp 3p(Mmin(S.D.))) In addition, we experiment with Approzimate GAP as well and report the best performances achieved from the two methods. This ap- proximated version bypasses the need of SVD for calculating the proconditioned gradients by g(k) ~ Mg(().
Here, θ = (, M), and the update rule is defined as update( (k) (+)) (k) frm)
(k) -rm-
ALFA. ALFA is proposed to meta-learn the weight update procedure along with the weights to facilitate the learning. Unlike GAP, ALFA introduces an additional meta-learned neural network h(ol*), ( g) that dynamically predicts learning rates (*) and weight decaying terms (a()) in a data-specific manner and for each inner step k given the current parameters ((()) and their gradients (k)) of the learners. The resulting weight update rule can be described as follows:
=,(0) (0(k) (((k)-(()) (k) (R), gh)) where g = =0)== pick; ((K)
((k)
(15)
To construct the meta-learned network h, we follow the original setup (Baik el al. 2023) and parametrize it with 2-layer MLPs with ReLU activations. Also, we reduce the context parameters and gradients by averaging them along the feature dimensions (i.e. 9, Ø E RSDD, - g. ø E R25) and feed them into the meta- learned network h. Finally, we augment the predicted learning rates and decaying terms a ,(()) E R with additional meta-learned weights oo; 8o E IR:5, as suggested in the paper. We set the initial values of og. Po to 1. In summary, θ = (0, {), and the update rule is defined as update gran) g Drane (k) (k) where (8) fum h (6lhi, gami i Em).
,
(k) g())
giam
(4).
MTNPs. Multitask Neural Processes (MTNPs) (Kim et al. 2022a) is another class of meta-learning app- proach for INRs based on Neural Processes (Garnelo et al. 2018), aimed at modeling multimodal signal distributions, similar to ours. This method replaces iterative optimization steps with feed-forward encoder networks to directly predict the parameters of INRs from observed signals. For this, MTNPs adopts a dual-stream and hierarchical fusion approach to capture cross-modal relationships among signals. The first stream, driven by a latent encoder, is tasked to capture uncertainty in recovering the entire function given partial observations and to infer global latent variables shared by the observed data points for each signal in modality (Dmn). On the other hand, the second stream is guided by deterministic encoder and is responsible for extracting local per-coordinate representations specific to each data point (Nrmm Unm) that belongs to a signal in modality. In addition, to improve the expressive power of the model, cach stream is composed of a stack of specialized hierarchical multi-head attention blocks, where the earlier part captures the dependencies among data points within a modality and the latter discover their potential cross- modal relationships. For instance, in the first latent stream, cross-modal relationships among global latent variables for each modality are captured. Similarly, multimodal dependencies among local representations that belong to the same coordinate are considered in the second deterministic stream. We refer more interested readers to Section 3 and Figure 2 in the original paper of MTNPs (Kim et al. 2022a). To evaluate MTNPs in our setup, we apply two modifications to MTNPs to adapt it in our experiments: (1) We change the output dimension of the latent encoder so that it directly predicts the context parameters 6am E Smm that con- ditions the parameters of INRs. (2) For experiments on AVMNIST, we omit the module in the deterministic stream that captures cross-modal interactions among the axis-aligned local representations since there no one-to-one correspondence between the image and audio coordinates. We use the official implementation ! by the author to construct MTNPs.
HNPs. Heterogeneous Neural Processes (HNPs) (Shen et al. 2023) largets general multinodal meta- learning setups, which can be applicable to both multimodal signal regression and multimodal classification sconarios. Similar to MTNPs, HNPs has modality-specific and modality-agnosthi inference modulcs, where the former processes per-modality observations and produces their representations, whereas the latter fuses those representations to capture pross-modal relationships. Unlike MTNPs, however, HNPs introduces ad- ditional meta-learned modality-specific and modality-agnostt latent priors, w and 2,, which are respectively fed into the modality-speeific and modality-agnostie inference modules as well to induce the respective latent variables, z and w. Finally, the per-modality decoder takes w to approximate the target function. In our problem sctup, we slightly modify HNPs and formulate w to model the parameters of INR learners Φ, and
http:://gthhubbrmm//CiCCy////// fask_ neural processes
30

---

Published in Transactions on Machine Learning Research (08/2024)
use Functa and Composer for the decoder framework. We use the official implementation provided by the author to construct HNPs.
E
Additional Analysis
E.1
Correlation between attention pattern and observation quality
Table 5: Pearson Correlation Coefficient (Cty) between the attention scores of MSFTs assigned to the learner's
state representations for modality i and the support set size for modality j.
Functa
Composers
Sine
Gaussian
Tanh
ReLU
Sine
Gaussian
Tanh
ReLU
Sine 0.561
-0.393
-0.434
-0.357
0.620
-0.520
-0.455
-0.523
Gaussian
-0.089
0.290
-0.074
-0.020
-0. 0.406
0.473
-0.262
-0.222
Tanh
-0.163
-0.068
0.319
-0.036
-0.493
-0.439
0.597
-0.421
ReLU
-0.144
0.082
-0.165
0.270
-0.255
-0.226
0.117
0.353
(a) Coefficients on multimodal 1D synthetic functions.
Functa
Composers
RGBs
Normals
Sketchs
RGBs
Normals
Sketchs
RGBs
0.779
-0.645
0.579
0.826
-0.837
-0.671
Normals
00.994
0.843
0..81
-0.26
0.838
-..59
Sketchs
0.923
-0.91
0.973
-0.856
-0.899
0.933
(b) Coefficients on multimodal 2D CelebA images.
Figure Sa and Table 5 present when and how SFTs facilitate the interaction among the learners, quantified
by Pearson Correlation Coefficient (Cty). This coefficient measures the correlation between the attention scores of MSFTs assigned to a learner's state representations for modality í and the support set size for modality j. Here, we describe the detailed methodology used calculate this correlation.
Given joint multimodal signal Dr (Dro JM K-step adaptation of the context parameters {dum Jm= RSxDs with MSFTs of H multihcad attention produces the attention score map A, with shape of K x H x MS x MS. We reshape this attention score map to (K x x S x 5) x M x M, followed by reducing the leading dimensions to obtain an average attention score matrix A, with a shape of x M. This matrix quantifies the average interactions among the learners, where the (i, element this matrix amounts to the average attention scores directod from the learner of modality towards the learner of modality j. Finally, we compute the Pearson correlation coefficient C between these average attention score maps and the sizes of the support sets [Puml] Im across N sets of joint multimodal signals, where Cip amounts to the correlation between (i, and {Panj]),, which is the
E.2 Evolving attention maps through optimization steps
We also investigate the evolving interactions among the learners through optimization steps 1, ........ K, as presented in Figure 17. Here, instead of computing the Pearson Correlation Coefficient between attention maps and the support set sizes as above, we quantify this degree of interactions among the learners solely by analyzing how the attention patterns within MSFTs evolve over optimization steps. This procedure involves the following steps: (1) We first reshape the attention score map A, for signal defined in Section E.1 to
(xxSxx)) Kx M (2) Then, we average this map over the first dimension to obtain A, E B Mxxxxxx (3)
Finally, this map is aggregaled further for all signals and then averaged to À = | Σ. An = { We visualize and analyze how this A evolves with k. Note that this final attention map contains values in the
http:://ghhau.ccmmm//mmmmm2299/9
31

---

Published in Transactions on Machine Learning Research (08/2024)
goussian
gaussian
Smc
goussion
retu
0:244
0,250
0.239
0:272
sine
0.5.5
0.304
0.125
0.5
0.4
tanh
tanth
reu gauzsinn
0.308
0.254
0.187
0.251
reUy gausssan
0.294
0.278
0.152
0.275
reu gaussoar
0.282
0.206
0.311
0.201
0.254
0.297
0.171
0.265
0:217
0.235
0.248
0.374
0.1
Step
Step
Step
0.0
(a) Multimodal 1D synthetic functions.
rgb
normal
sketch
rgb
normal
sketch
rgb
normal
sketch
0.7
rgb
0.516
0.249
0.235
rgb
0.650
0.171
0.179
19b
0.714
0.163
0.123
0.6
horma
0.385
0.422
0.190
norma
0.171
0.691
0.137
norma
0.160
0.746
0.094
0.4
0.3
skeech
0.506
0.181
0.313
sketch
0.348
0.060
0.571
sketch
0.299
0.070
0.631
0.1
0.0
Step
Step
Step
(b) Multimodal CelebA images.
Figure 17: Evolving interactions at each optimization step = 2, Each (i,j) element in the matrices indicates the average attention score assigned by the learner of modality to the learner of modality j. Please refer to Appendix E.2 for the detailed method for computing these matrices. The patterns show that learners tend interact extensively with each other in the beginning (high off dif-diaonal attention scores in Step 1) and gradually attend more themselves towards the end (high diagonal attention scores in Step 3). This highlights MIA's is adaptability in ensuring balanced utilization of multimodal interactions, emphasizing the necessity of applying multimodal adaptation iteratively for each optimization step.
range of [o, 1], where higher values indicate significant. cross-modal information exchange between distinct modality learners at each step, while a value of zero indicates no interaction.
The results in the figure reveal that the interactions among the learners mostly occur in the beginning of the optimization (high off-diagonal attention scores step 1). Then, find that the learners gradually focus more their own states towards the end (high diagonal attention scores at step 2 and 3). This analysis shows that SFTs ensure balanced utilization of multimodal interactions.
32

---

Published in Transactions on Machine Learning Research (08/2024)
RGB ratio: 0.01
RGB ratio: 0.05
RGB ratio: 0.1
0.150
0.150
/B/8.
0.150
GD/5.74
0.0/7.22
0.0/5.47
0.125
0.125
0.025
0.125
0.025/4.49
0.1/
0.1/4.27
0.100
0.100
1.0
0.100
1
0.075
0.075
0,075
PC2
PC2
GD/48. 46.78
0.025
0.025
0.025
0.09
1..95/16.94
0.000
0.000
0.000
1,5/
-0.025
-0.15
-0.10
-0.05
0.00
-0.15
-0.10
-0.05
0.00
-0.15
-0.10
-0.05
0.00
PC1
PC1
PC1
(a) Functa
RGB ratio: 0.01
RGB ratio: 0.05
RGB ratio: 0.1
GD/48.11
3.09
0.025/22.45
0.1/
2-
C2/24.04
GD/ 18.10
0.0: 0..54
0.0/1
0.0.5/13.58
0.0.8
0.1/13.30
0.1/ 11.61
1.0/12.29
1.0/ 10.45
00.050 -0.025
0.000
0.025
0.050
0.050 -0.025
0.000
0.025
0.050
0.050
-0.025
0.000
0.025
0.050
PC1
PC1
PC1
(b) Composers
Figure 18: Learning trajectory of RGBs context parameters for the 3-step adaptation process with or without MIA, depending the support set size of the target RGB and the sources, Normals and Sketches. For each plot, the green dotted line represents the trajectory of CAVTA adapted with the gradient descent (GD). The solid lines represent the trajectory adapted with MIA, where the different colors reflect the support set sizes of the source modalities (Normals and Sketches). The scalar values in the left side of the legend refer to the support set ratio of the source modalities and the right side means the achieved reconstruction performance (MSEs).
E.3 Analysis study on learning trajectories via MIA
We investigate how MIA adjusts the optimization trajectories of context parameters during the adaptation process compared to CAVIA. To do this, we freeze the trained weights of CAVIA, followed by integrating it with our SFTs. Then, we meta-learn our SFTs only. This ensures that the trajectory space of the context parameters of both CAVIA and our method is aligned, thereby enabling the comparison of trajectories between the two methods. After that, we visualize the learning trajectories both methods by applying Principal Component Analysis (PCA) on the obtained context parameters described in Li et al. (0188).
Figure 18 illustrates that as the amount of observations from additional modalities increases, the learn- ing trajectory with MIA shifts towards more effective solutions (lower MSE loss), moving away from bad solutions. This improvement is attributed to the SF'ss ability to enhance the sub-optimal gradients de- rived from limited obsorvations by utilizing information from other modalitics' learners. Conversoly, when adapting through a gradient descent (GD), the learning trajectory of CAVIA remains constant regardless of observations in other modalities due to the absence of cross-modal interaction.
33

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Results on CelebA dataset (N En Am m)) of MSFTs when injecting noise to gradients of other modalities' learners with varying noise level y.
MSEs (x10-3) and averaged unimodal attention scores
Functa
Composers
7
RGBs
Normals
Sketchs
RGBs
Normals
Sketchs
0.00000
0.948 / 0.502
0.769
0.579
0.578 / 0.710
0.667 / 0.635
0.289
0.690
0.255
0.637
0.00001
0.976
0.525
0.774
0.644
0.581
0.720
0.664 / 0.630
0.288
0.696
0.255
0.643
0.00005
0.982 / 0.527
0.774
0,651
0.582 / 0.719
0.664 / 0.631
0.288
/ 0.709
0.253
0.651
0.00010
0.987
0.527
0.775
0.653
0.582
0.718
0.673 / 0.656
0.290
0.727
0.253
0.658
0.00050
1.056 / 0.539
0.786
0.664
0.583 / 0.726
0.708 / 0.720
0.293 / 0.748
0.256
0.681
0.00100
1.092 0.549
0.796
0.677
0.584
0.741
0.714 / 0.724
0.293
0.752
0.256
0.687
0.00500
1.122 / 0.560
0.819
0.705
0.587 / 0.772
0.718 / 0.727
0.294
0.755
0.256
0.690
0.01000
1.124 / 0.561
0.823
0.709
0.588
/ 0.775
0.717 / 0.727
0.294
0.755
0.256
0.690
0.10000
1.126 0.562
0.826
0.712
0.589
0.777
0.718
0.727
0.294
0.755
0.256
0.691
1.00000
1.127 0.562
0.826 0.713
0.589
0.778
0.718 / 0.727
0.294
0.755
0.256
0.691
E.4 Analysis study on negative transfer from other modalities
In this study, we explore the impact of noisy state information from source modality m' on target modality m. Our evaluation focuses on the robustness of SFTs against the potential negative transfer of incorrect information from source modalities to the target. To assess this, we inject varying levels of (k) Gaussian noise Cam' ~I) into the gradient 9am' of source modalitics before they are fed into SFTs. As the noise level 7 increases, the relevance of the slate information from the sources to the targel learner diminishes. To quantify the effect of noises, we measure the average MSEs and the average attention scores ( Σn An m)) within target Icarners' states, while varying the noise level from 10-5 to 1.0.
As shown in Table t while there is a marginal rise in MSEs upon the introduction of noise, no addit performance degradation is observed beyond a certain noise level. Interestingly, as the gradient noise from learners of other modalities increases, there is concurrent increase in unimodal attention scores for the learner of the target modality. This indicates that SFTs inherently have the ability to handle potential negative transfers across modalities.
E.5 More in-depth ablation study for components of SFTs
Table 7: Ablation study on the components in SFTs on generalization and memorization ability. We report relative performance improvement (f) achieved by ablated methods (2-8) over vanilla Composers (1) on multimodal ID synthetic function dataset.
Modules
Synthetic
USFTs
MSFT
FusionM..Ps
Generalization
Memorization
(1)
X
0.00
0.00
(2)
X
X
38.9
54.2
3
X
44.4
63.6
X
43.1
68.0
55)
X
/
86.8
71.6
X
/
86.8
76.7
V
X
86.7
75.5
(8)
88.7
81.6
This section provides more in-depth ablation study results that could not be included in Table 2a present the ablation study the synthetic dataset that encompasses all possible component combinations
We first
34

---

Published in Transactions on Machine Learning Research (08/2024)
to investigate how each of them contribute to performance. The results are shown in Table The table shows the substantial impact of FusionMI.Ps in enhancing the weight updates of vanilla Composers when used independently (1 vs 2) or in combination with USFTs, MSFTs, or both (3 vs 4, 5 vs 6, 7 vs 8). This demonstrates the general versatility of FusionMLPs in enhancing gradient directions or magnitudos. Moreover, we observe that incorporating USFTs significantly enhances memorization capabilities (1-2 vs 3-4). This emphasizes the advantagoous role of USFTs in capturing modality-spccific patterns within the learners' states. In contrast, MSFTs excel in leveraging cross-modal interactions among the learners' states, leading to substantial improvement in the generalization performances of Composers (3-4 vs 5-8). Lastly, the most optimal performance is achieved when utilizing combination of USFTs, MSFTs, and FLsionMLPss validating unique and indispensable roles of each component within SFTs.
Next, we extend the ablation study in Table 2a and present more in-depth analysis results on the multi- modal 2D CelebA dataset. We first compare the meta-training overheads of the ablated methods, namely the number of parameters, memory consumption, and meta-training time per iteration. The results are summarized in Table [al The table shows that the overheads of our full model are comparable to the other ablated methods.
Finally, we delve into the role of MSFTs and Fusion MLPs in enhancing the generalization capability of CAVTA and preventing negative transfer during the cross-modal interactions conducted within MSFTs. For this, we compare the relative performance improvement of the ablated methods for each target modality (either RGB, Normal, or Sketch) over the vanilla. CAVIA while varying the sampling ratios of observable support sets from both the target and source modalities separately. The sampling ratios for the target and sources are set to either 0.01 (limited) 0.25 (sufficient). The results are presented in Table 8b) and Table Sc.
The results demonstrate that, when the support for the target modality is highly limited (i.e., R = 0.01), CAVIA augmented with MSFTs siguificantly boosts the performance of their unimodal counterparts, CAVIA and CAVIA with USFTs, regardless of the sampling ratios of the source modalities. This suggests that MSFTs are indeed essential for enhancing generalization capabilities when observations are limited.
On the other hand, when observations for the target are sufficient (i.e., R = 0.25) and those from the sources limited (i.e., R 0.01), CAVIA with USFTs achieves better performance than CAVIA with MSFTs CAVIA with both USFTs and MSFTs. This could be attributed to potential negative transfer from insufficient sources to the target. Finally, our full model alleviates this issue and achieves the best performances overall, indicating that Fusion MLPs are necessary to mitigate potential negative transfer during cross-modal interactions.
E.6 Comparisons with other multimodal meta-learning studies
Multimodal meta-learning is not new in the domain of meta-learning. However, existing studies (Vuorio ltt al. 2018 2019 Abdollahzadeh et al., 2021 Sun et al. 2023) differ significantly from our work in terms of the notion of modality and problem sctups. This not only makes direct comparisons challenging but also hinders fair comparisons since evaluating their frameworks or ours to within each other's context requires substantial modifications in the methodologies. Nonetheless, in this section, we describe the problem setup and the method focused by these studies, followed by their comparative evaluation results in the context of our joint multimodal signal modeling scenarios.
MMAML & KML. Unlike the prevalent focus of meta-learning and meta-testing on single-domain prob- lems, exemplified by few-shot classification tasks on a single individual dataset like Omniglot, Imageen, or CUB datasets, works such as Vuorio et al. 2018 2019) and Abdollahzadeh et al. (202] | extend their scope to encompass multiple domains of datasets. For example, they explore simple regression problems across union of sinusoidal, linear, and tanh functions, few-shot classification tasks combining Omniglot, mini- Imagenet, and CUB datasets, or reinforcement learning scenarios across various enviromments such as Point Mass, Reacher, and Ant. In these studies, each domain, whether dataset or environment, is treated as a distinct modality, leading to their concept of multimodal meta-learning. Importantly, this concept of multi- modality differs from the traditional understanding related to data types, as explicitly clarified in Section 3 of Abdollahzadeh et al. 20021
35

---

Published in Transactions on Machine Learning Research (08/2024)
Table 8: Comparisons on computation overheads and performances across ablated methods. Overheads are measured by parameter counts (4), memory usage (L), and meta-training time per iteration (L). Performances are reported relative performance improvements (t).
Modules
Functa
Composers
Fusion MLPs
Memory (GB)
Training Time (ms/it)
Memory (GB)
Training Time
USPTs
MSPIs
Params
# Params
(ms/it)
x
373K
8.54
251.3
224K
3.00
60.00
1.22M
8.76
266.7
1.26M
3.27
124.8
772K
8.75
264.6
819K
3.30
115.1
1.4MM
8.92
274.7
1.49M
3.48
133 .
1.89M
9.07
279.3
1.93M
3.61
136.2
(a) Computational overhead
Modules
RGB: 0.01
Normal: 0.01
Sketch: 0.01
RGB: 0.25
Normal: 0.25
Sketch:
0.25
USFT:
MSFTs
Fusion MLPs
Source:
Souroe:
Source:
Source:
Source:
Source:
Source:
Source:
Souroe:
Source:
Source:
Source:
0.01
0.25
0.01
0.01
0.25
0.25
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
X
23.01
23.01
31.22
31.22
27.02
27.02
24.76
24.76
27.38
27.58
23.53
23.53
x
x
23.24
50.87
29.13
44.30
29.38
50.85
16.68
29.19
23.32
28.97
22.21
32.17
26.11
55.44
32.31
49.13
30.93
52.88
20.82
32.43
26.89
32.20
23.89
33.82
/
/
/
29.13
57.98
32.19
50.11
30.83
53.13
22.75
34.74
27.51
33.36
23.85
34.25
(b) Ablation study based on Funcla
Modules
RGB:
Normal:
Sketch:
RGB:
Normal:
Sketch:
USFIs MSFIs
Fusion MLPs
Source:
Source:
Source:
Source:
Source:
Source:
Source:
Source:
Source:
Source:
Source:
Source:
0.01
0.25
0.01
0.25
0.01
0.25
0.01
0.25
0.01
0.25
0.01
0.25
0.00
0.00
0.00
0.00
0.00
0.00
x
x
38.65
38.65
38.33
38.33
23.76
23.76
73.35
73.35
50.29
50.29
51.13
51.13
72.19
73.39
50.51
51.89
48.82
52.48
/
40.70
50.47
38.89
48.73
26.37
41.05
72.75
74.05
52.21
53.78
50.64
54.64
41.54
52.70
11.05
49.88
26.11
39.85
74.06
75.16
53.80
55.09
50.75
51.10
(c) Ablation study based on Composers
These works highlight that meta-learning single initialization may be suboptimal due to the multimodal nature of the problem. To address this, they introduce an additional meta-learned module known as task encoder. The role of this task encoder is to identify the latent modality of the observed data and predict modulation parameters. These parameters guide the Icarned single initialization towards modality-spocific initializations. Experimental results indicate that the lask encoder indeed learns identify the modality (or dataset domain), and consequently, adapting from modality-specific initializations yields better performance than relying modality-agnostic single initialization..
However, it's important to note that in these papers, each data point is assumed to be sampled iid from the union of datasets. As a result, the learner adapts independently to each data point, without explicitly leveraging cross-modal relationships among modalities or incorporating mechanisms for multimodal fusion. For these reasons, we categorize these approaches as inherently unimodal meta-learning methods.
To adapt their work in our problem setup, we parameterize their task encoder with the same transformer architecture backbone as our SFT,s which is shared across all modalities, and uses it directly predict the parameters of INRs. Then, adapt those predicted parameters for additional & optimization steps.
AMML. Unlike the previously mentioned studies, Sun et al. (2023) delve into a scenario where multi- modality relates specifically to data types. Their primary focus lies in addressing multimodal sentiment analysis problems, where the learner is tasked with classifying discrete sentiment scores ranging from 1 to or binary sentiment classes (positive or negative). Since this classification relics on diverse sources of information represented in different modalities, encompassing texts, images, and andios, the task is referred to multimodal inference in their work.
36

---

Published in Transactions on Machine Learning Research (08/2024)
Table 9: Quantitative comparisons on the multimodal ID synthetic functions. We report the normalized MSEs (x10-2) computed over distinct ranges of sampling ratios, averaged over random scods.
Modality
Sine
Gaussian
Tanh
ReLU
Ruiia
0.01
0.05
0.02
0.01
Range
Rmax
0.02
0.05
0.10
0.02
0.05
0.10
0.02
0.05
0.10
0.02
0.05
0.10
Functa
44.26
16.07
3.319
18.81
4.388
0.953
22.61
3.667
0.586
65.29
10:79
2.157
w/ MMAML
43.35
10.09
1.233
30.87
4.270
0.698
17.20
2.351
0.312
72.79
5.630
0.261
AMML
10.12
4.402
1.717
1.253
0.959
0.638
1.719
0.541
0.267
7.063
1.254
0.305
W/ MIA
6.386
2.058
0.5417
1.057
0.571
0.281
1.285
0.378
0,131
5.069
1.012
0.115
Composers
37.40
16.70
6.284
5.923
3.149
1.460
14.81
4.053
1.029
48.49
11.98
3.232
w/ MMAML
14.26
11.63
1.570
31.09
4.664
0.895
16.46
2.330
0.309
74.75
3.845
0.291
AMML.
14.57
9.549
7.706
1.699
1.402
1.086
2.411
1.021
0.725
10.19
2.340
0.907
w/ MIA
5.564
1.844
0.627
0.975
0.528
0.237
1.257
0.343
0.128
4.715
0.943
0.156
Table 10: Quantative comparisons on the multimodal 2D CelebA image function regression. We report MSEs (x10-3) computed over distinct ranges of sampling ratios, averaged over 5 random seeds.
Modality
RGBs
Normals
Sketches
Rmii
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Range
Rmaax
0.25
0.75
0.25
0.75
1.00
0.25
0.50
0.75
1.00
Functa
13.44
4.117
3.052
2.577
5. 5.067
2.448
2.092
1.928
13.29
7.065
5.704
5.209
MMAML
11.22
3.775
2.772
2.264
4.311
2.266
1.872
1.672
10.96
6.021
4.452
3.751
w/ AMML
8.110
2.909
1.853
1.334
3.341
1.689
1.243
1.000
8.192
4.332
2.278
1.124
MIA
6.946
2.563
1.627
2.979
1.530
1.118
0.869
7.667
4.042
2.142
1.011
Composers
26.40
15.83
14.30
13:21
6.979
4.830
4.630
4.517
17.99
14.36
13.27
12.86
MMAML
12.35
1.228
2.806
2. 2.029
4.902
2 2.567
2.028
1.725
11.29
5.757
3.646
2.521
w/ AMML
11.28
5.195
3.971
3.241
4.209
2.336
1.745
1.339
9.813
5.084
2.727
1.319
MIA
9.764
3.418
1.913
1.017
3.749
1.763
1.062
0.526
9.505
4.708
2.336
0.855
In their work, the authors underscore the limitations of existing methodologies, pointing out their inability to consider the heterogeneous convergence properties of each unimodal encoder network. Naively adapting these encoders using identical optimization algorithms (e.g. SGD with the same learning rate) results in suboptimal unimodal encoder networks. Consequently, the fusion of these suboptimal representations yields unsatisfactory outcomes for the final prediction.
To address this, Sun al. (2023) propose to include the independent adaptation of each mopality-specific encoder using the unimodal classification loss within the inner loop. Following this inner-loop phase, a multimodal fusion network integrates the extracted representations derived from these individually adapted unimodal encoders. Finally, the fused representations are utilized for predicting class labels for sentimental analysis. It's also worth mentioning that Transformer structures are employed for independent modality- specific encoder (such as the pre-trained BERT for the text encoder and Vanilla. Transformers for image and acoustic encoders), whereas they are not utilized for multimodal fusion.
Since their framework not directly applicable to our joint multimodal function regression scenarios, we investigate the effectiveness of their ioptaationi--the-innee by applying our method's MIA only in the final optimization step.
followed by fnsion-in-tth--outer-loo scheme
Results. Fach compared method is run 5 times with different random seeds on Synthetic and CelebA datasets and their results are averaged. The results are in Table and I0 From the tables, we conclude the following things: (1) MMAML greatly improves the memorization performances of CAVIA (Functa and Composer) thanks to the task encoder network, while fails to generalize better than CAVTA due to its inherently unimodal nature of MMAML. (2) AMML further improves the gencralization performances of -hwrolll--uuuii----------uuueeee
scheme. However, its performances still fall short than our MIA, demonstrating the effectiveness of joint multimodal iterative adaptation of the learners during the adaptation stages.
CAVIA thanks to its
37


Reviews:
[{"review_id": "AcR7KKj4o1", "reviewer": "Reviewer_P66A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* MIA is claimed to be the first to explore the integration of multimodal structures within the learning of multiple correlated INRs, offering a new perspective on optimizing across modalities.\n* By allowing modalities to inform each other, MIA significantly improves generalization capabilities, even with limited data, outperforming unimodal and encoder-based baselines.\n* The paper presents a thorough evaluation of MIA across various real-world multimodal signal regression scenarios, demonstrating its superior performance in both generalization and memorization.\n\n\nWeaknesses:\n\n* Although the paper is generally well-written, it is better for the authors to provide an application example of the INR setting where the data is scarce to make the audiences better understand the targeted tasks, especially for the ones that are not familiar with the INR research.\n\n* While the setting focuses on the optimization of INRs when the observations are sparse, it is unclear how sparse the four datasets used in experiments are. It would be better for the authors to provide some quantitative measurements regarding to $P$ of each dataset to further demonstrate their setting matches their motivation. Meanwhile, how many INR learners used (the $n$ and $m$) should also be specified for each dataset.\n\n* According to the results in Table 2 for the ablation study regarding the impact of modules, it seems like the USFTs is the most important module in SFTs as adding the other two modules only exhibits relative marginal improvements over USFTs-only variants. It would be beneficial for the authors to elaborate more on the rationale behind the design of both MSFTs and Fusion MLPs, especially considering together with the additional computational complexity brought by adding MSFTs and Fusion MLPs. This would be interesting for practitioners in employing the proposed method in industrial applications.\n\n* One minor suggestion is, given the designs of the MIA and SFTs are relatively intuitive, it would be interesting to see some theoretical groundings behind the designs with the current motivation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Multimodal Iterative Adaptation (MIA), a framework that advances the learning of Implicit Neural Representations (INRs) by integrating multimodal fusion with optimization-based meta-learning. MIA addresses the limitations of encoder-based methods by enabling cross-modal knowledge exchange during iterative optimization, which enhances the generalization of INRs to complex signals. The framework’s centerpiece, State Fusion Transformers (SFTs), is an attention-based meta-learner that operates during the backward pass, aggregating learning states and predicting improved parameter updates. This approach allows for a more nuanced adaptation to multimodal signals and overcomes data scarcity challenges.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please kindly refer to the Strengths And Weaknesses.", "broader_impact_concerns": "No broader impact is discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pRwhEps2sO", "reviewer": "Reviewer_NAX6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written in general.\n- Studying better methods for representation learning over multi-modal inputs is of great importance.\n- The experimental results of the proposed MIA approach look promising.\n\nWeaknesses:\n- After reading the entire paper, I'm still not fully convinced of the importance of INRs in practice. Why is it an important technique that deserves more research effort? And how can it improve our machine learning pipeline (e.g., by either improving learning efficiency or generalization)? I see it more as a writing/paper organization issue. One way to improve it is to add more motivating examples and discussions in the introduction.\n- Most of the multimodal experiments presented in this paper are actually still limited to visual input. I wonder if MIA can also be applied to more popular multimodal inputs like vision-language models, e.g., LLaVA [1]?\n- The overall procedure of the proposed MIA approach is not quite clear. It would be great to add an algorithm box.\n\n[1] https://arxiv.org/abs/2304.08485", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework called Multimodal Iterative Adaptation (MIA) to improve the learning of Implicit Neural Representations (INRs) by leveraging cross-modal knowledge exchange. Current encoder-based and unimodal optimization methods for learning INRs are insufficient for capturing the complexities of real-world multimodal signals, e.g., sketch prediction. The proposed MIA framework enhances INRs by facilitating cross-modal knowledge exchange during the optimization process, using State Fusion Transformers designed to operate in the backward pass of the learners. Extensive experiments across various multimodal datasets demonstrate that MIA significantly improves generalization and memorization performance compared to existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I summarize the requested changes below (Please see the Weaknesses section above for more details.):\n\n- Add more motivating examples and discussions for the proposed MIA approach in the introduction.\n- Explore the compatibility of MIA to vision-language models.\n- Add algorithm boxes for MIA and the major baseline methods.", "broader_impact_concerns": "I don't find any Broader Impact Concerns in the current manuscript.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1l8uuqlHtF", "reviewer": "Reviewer_B5Ai", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper makes a nice contribution in the multimodal INR literature. The approach is a clever combination of the two primary paradigms in multimodal INR (encoder-based and optimization-based), and it is well-motivated.\n- The SFT is clearly structured into three stages, where each stage corresponds to a different functionality (unimodal learning, cross-modal learning, and state fusion).\n- The experimental evaluation is quite extensive. The authors compare MIA to several baseline methods, and they try two different base INR models, namely, Functa or Composers. They also use 4 datasets, 3 of which are real. With few exceptions, MIA outperforms the other methods, with the gap being particularly pronounced in the more challenging low-sampling regime.\n- The authors include an interesting ablation study that discusses key aspects of the MIA framework, e.g., the relative impact of each module of the SFT, the cross-modal utilization of SFTs, and the computational overhead of MIA. The findings in the ablation study seem to confirm that the SFT can identify useful cross-modal patterns, and make use of them to improve performance in the target modallity.\n\nWeaknesses\n- The MIA framework seems hard to scale to more complex signals, e.g., images larger than 128x128. This is  fundamental challenge of meta-learning algorithms, as the authors acknowledge, but it still significantly limits he applicability of he proposed framework in complex real settings.\n- Table 2(a) suggests that the relative improvement of MIA mainly comes from the unimodal learning with USFT. The multimodal MSFT provides very small additional improvement, while MP Fusion also results in rather modest improvement. Given that this framework specifically addresses the multi-modality challenge, one may have expected that a much larger part of the relative improvement would be attributed to the cross-modal interactions, but it seems that unimodal learning can in fact primarily account for the improvement performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes Multimodal Iterative Adaptation (MIA), a framework for learning Implicit Neural Representations (INRs) for joint multimodal signals. The two primary existing approaches for this problem are (i) encoder-based meta-learning methods that leverage multimodality using attention-based fusion, or (ii) unimodal optimization-based methods that focus on learning a good initialization to speed up convergence. However, both approaches have limitations: the former suffers from slow adaptation and the inability to capture complex signals, while the latter is unable to leverage the multimodal data for better generalization in data-scarce settings. To tackle this, MIA proposes the best of the two worlds: an optimization-based meta-learning framework with an attention-mechanism, namely, the State Fusion Transformer (SFT). The SFT aggregates learning states and allows the different modalities to interact during adaptation, and results in better parameter updates during meta-learning. The authors conduct a comprehensive experimental study on various multimodal settings, and show that MIA outperforms other approaches in terms of both generalization and memorization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not have major concerns with respect to this work. Some questions to the authors:\n- Have they tried experiments with more complex datasets? If not, what was the main obstacle? If yes, how did they perform?\n- Can the authors comment on the fact that unimodal learning in USFT mainly accounts for the performance improvement? It seems that cross-modal interactions help, but to a much lesser degree.\n- In Figure 8(a) and Table 5, the Pearson correlation coefficient is negative in the off-diagonal entries, whereas in Figure 17(b) the off-diagonal entries are positive. Does Figure 17 depict something different from the Pearson correlation?", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a new meta-learning framework for multimodal modelling. Specifically, a Multimodal Iterative Adaptation (MIA) approach was proposed that combined multimodal fusion with optimization-based meta-learning. Extensive experimental evaluations on several real-world scenarios show the effectiveness of the proposed method. The paper is generally well-written and easy to follow.

Three expert reviewers were invited to review the paper, and both strengths and weaknesses were raised. With back-and-forth discussions and revision, most of the concerns were well addressed by the authors in the revised version, and all the reviewers are satisfied with the revision. In the end, a positive score (with two *Leaning Accept* and one *Accept*) was recommended.
On the other hand, both the reviewers and AE found that the interesting approach and the contributions made in this paper could be of interest to a group of audiences in TMLR. As a result, the AE is pleased to inform that the paper has been accepted to be published in TMLR.
Decision: Accept (poster)