TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modality
Authors: Yinsong Wang
Shahin Shahrampour
Venue: Accepted by TMLR ()

Abstract: This paper addresses a cross-modal learning framework, where the objective is to enhance the performance of supervised learning in the primary modality using an unlabeled, unpaired secondary modality. Taking a probabilistic approach for missing information estimation, we show that the extra information contained in the secondary modality can be estimated via Nadaraya-Watson (NW) kernel regression, which can further be expressed as a kernelized cross-attention module (under linear transformation). This expression lays the foundation for introducing The Attention Patch (TAP), a simple neural network add-on that can be trained to allow data-level knowledge transfer from the unlabeled modality. We provide extensive numerical simulations using real-world datasets to show that TAP can provide statistically significant improvement in generalization across different domains and different neural network architectures, making use of seemingly unusable unlabeled cross-modal data.

Published in Transactions on Machine Learning Research (06/2024)
TAP: The Attention Patch for Cross-Modal Knowledge Trans- fer from Unlabeled Modality
Yinsong Wang
wwng.yyissoohhoorheestsee.cmmm
Deportment of Mechanical and Industrial Engineering
Northeastern University
Shahin Shahrampour
8. s.llahrmmpprr@hhorttssttte....
Department of Mechanical and Industrial Engineering Northeastern University
Reviewed on OpenReview: https: // openreview. net/ pdf? id= 73yeras553
Abstract
This paper addresses a cross-modal learning framework, where the objective is to enhance the performance of supervised learning in the primary modality using an unlabeled, unpaired secondary modality. Taking a probabilistic approach for missing information estimation, we show that the extra information contained in the secondary modality can be estimated via Nadaraya-Wate Watson (NW) kernel regression, which can further be expressed as kernelized cross-attention module (under linear transformation). This expression lays the foundation for introducing The Attention Patch (TAP), simple neural network add-on that can be trained to allow data-level knowledge transfer from the unlabeled modality. We provide extensive numerical simulations using real-world datasets to show that TAP can provide statistically signíficant improvement in generalization across different domains and different neural network architectures, making of seemingly unusable unlabeled cross-modal data.
1 Introduction
Consider a cross-modal learning framework where there exist a labeled primary modality and an unlabeled, unpaired secondary modality. In this paper, we address the following research question: can we boost the performance of supervised learning the primary modality by exploiting the extra information in the secondary modality (that unlabeled and unpaired with the primary modality)? Our work naturally lies the intersection of cross-modal learning and semi-supervised learning. The cross-modal learning paradigm learns from data in different modalities (Li et al. 2018). From a probability theory perspective, cross-modal data often refers to data with different support dimensions and distribution curvatures. The semi-supervised learning paradigm uses unlabeled data to improve the model performance learned by limited labeled data (Zhu 2005 Van Engelen & Hoos 2020). While cross-modal learning and semi-supervised learning have been extensively studied independently, the intersection of the two has been less explored in the literature and remained elusive. To be specific, when we have limited amount of labeled data in the primary modality and large set of unlabeled and unpaired data the socondary modality available during training, there no principled learning paradigm that can make of the secondary modality to create a model that is better than the model learned with only the primary modality. Figure la presents visualization of our target problem to solve.
Our larget problem is frequently encountered in different research communities. For example, when building a battery failure prediction model using temporal current and voltage readings, one option to learn a supervised learning model with only current and voltage readings. However, there exist unlabeled videos of battery inner structure changes that are generated by different research labs, so a potentially more effective option is to incorporate the information of these videos into the model. These videos information-rich,

---

Published in Transactions on Machine Learning Research (06/2024)
X:A
2:
M2:
M1 O
Z
4b W
cb W
Φ2 W
NW Kernel Regression
y:
M22
(a) For primary modality X dx and the secondary modality Z S Rd= there exists a space Mi that contains the mutual information between X' and Z, where &n Z Mi and dg X' + M₁ transform data E 2 and E X to space MI, respectively. There also exists space M2 that contains the exclusive information present in but not in X. transformation Φ2 - M2 takes data E Z to in is available, but data Z are unlabeled. There also alignment between data in X and 2.
(b) The Attention Patch (TAP) parameterized by query transformation Φ Wg. key transformation ái Ws: and value transformation d2 W.. TAP takes one in- stance x E X (or a representation of x), uses a batch of reference data (z; E 2) generate the responding representation ma E M2, which contains information that not present space X. The rep- resentation my will be concatenated with input x for downstream tasks.
MM.. The label information for data
Figure
but labeling them requires repeating the collection process, which is expensive and polentially impossible due to equipment constraints (Davidson et al.) 2018 y. In general, reusing information-rich but hard-to-label datasots for different learning tasks calls for developing novel methods at the intersoction of cross-modal learning and semi-supervised learning.
To solve the target problem, start by formulating the missing information estimation problem from probability density estimation perspective. Using kernel density estimation (Rosenblatt, 1956 Wand & Jones)
1994 Wang et al. 2023), we show that this formulation leads to multivariate NW kernel regression, which can further be expressed as kernelized cross-attention module (under linear transformation). We name this module The Attention Patch (TAP), as shown in Figure lb TAP is a simple neural network plugin that can be attached between two consocutive layers in a neural network. The TAP integration requires minimal modification the original network (i.e., only moderate dimension change), and the parameters of mappings 02, 63 can be learned in parallel with the original network training.
1.1 Summary of Contributions
This is the first. work that investigates a learning paradigm in semi-supervised learning and cross- modal learning, where the "extra" information in Z not only unlabeled and unpaired but also comes from a different modality than the primary modality X. We propose framework enhance supervised learning of the primary modality X' using unpaired, unlabeled secondary modality Z.
In Sections 3 and we show that formulating our target problem as a missing information estimation problem leads to a multivariate Nadaraya-Watte Watson (NW) kernel regression, and further recovers a kernelized version of the popular cross-attention mechanism (Vaswani et al.) (017)) (under lincar transformation). Based on our observation, we propose The Attention Patch (TAP) neural network plugin for cross-modal knowledge transfer from unlabeled modality. We further propose batch
2

---

Published in Transactions on Machine Learning Research (06/2024)
training strategy to incorporate more unlabeled cross-modal data while maintaining moderate memory costs.
We provide detailed simulations on three real-world datasets in different domains to examine various aspects of TAP and demonstrate that the integration of TAP into a neural network can provide stalistically significant improvement in generalization using the unlabeled modality. We also provide detailed ablation studies to investigate the best configuration for TAP in practice, including the choice of kernel, the choice for latent space transformation, and compatibility with CNN and Transformer-basee backbone feature extractors with an additional text-image dataset.
2
Related Literature
In this section, we provide review of related literature that explains why existing techniques can not be directly applied to our target problem.
2.1 Cross-Moda Learning
Cross-mode learning focuses on learning with data from different modalities. The most representative topic application-wis is cross-modas retrieval. This topic focuses finding relevant samples in one modality given query in another modality (Wang et al. 2016). The most important step in cross-modal retricval tasks is learning a coupled space that can correctly describe the correlation between data points from different. modalities. Traditional techniques including canonical correlation analysis (Hardoon et al. 2004 Andrew
ct al. 2013; Wang & Shahrampour
2021p, partial least squares (Geladi & Kowalski 1986 Cha 1994). and Tenenbaum & Freeman 2000) find a simple projection of the matching
bilinear model (Sharma et al.) 2012
pairs that minimize certain pre-defined loss functions. This framework, though has seen several variants
and extensions (Ngiam et al. 2011
Hong et al. 2015 Sohn et al. 2014 Xu et al. 2015). remains the most
popular framework in cooss-modal correlation learning.
In addition to cross-modal retrieval, cross-modal supervised learning also follows the same principle. Examples
include but are not limited to Lin & Tang (2006); Evangelopoulos et al. (2013): Jing et al. (2014); Feichtenhofer
et al. (2 Peng et al.] (2017); Li et al. (2019) All of these models involve coupled space learning through either implicit or explicit learning loss with matching pairs of gross-modae inputs.
With the rise in popularity of natural language processing and time series analysis, a concept of weak alignment appears in modal learning. Weak aligument refers to the missing alignment of sub-components in instances from different modalities (Baltrušaitis et al. 2018). More specifically, this often means cross- modal input sequences with different sampling intervals or ordors. For example, in the case of vision and language models, a text description sequence of video will usually differ from the time and order of information that appeared in the video (Venugopalan et al. 2015 Tsai al. 2019), or a text description of an image will need to map a sequence of words a collection of objects without any specific orders (Mitchell al. 2012
Kulkarni et al. 2013 Chen et al. 2015 Karpathy & Fei-Fei
2015
There exists research direction that taps into instance-wise nonaligmment for cross-modal learning, which is called non-parallel co-learning (Baltrušaitis et al.) 2018). Non-parallel co-learning aims at improving the model learned on a single modality using another modality that is unaligned with the primary data. However, this is a concept that has only been studied with very specific applications, and it also requires the reference modality to be labeled during the training process. For example, cross-modal transfer learning (Frome et al. 2013 Kiela & Bottou| 2014 Mahasseni & Todorovic 2016)) mainly focuses on transferring supervised pre-trained embedding networks for improved cross- model prediction accuracy. Cross-modal meta-learning (Phoo & Hariharan] 2020 Islam et al. 2021) investigates improving few-shot learning performance of primary modality using. a labeled, yet. unaligned secondary modality.
In a nutshell, almost all cross cross-mode modal learning fromeworks focus on the case where there are known alignments between different modalities of data least during the learning phase. Our work considers the case where both alignment and labels do not exist during learning, and our proposed architecture can be applied to arbitrary modalities.
3

---

Published in Transactions on Machine Learning Research (06/2024)
2.2 Semi-Supervised Learning
Semi-supervised learning focuses addressing the challenge of limited labeled data availability in building learning algorithm (Zhu, 2005: Van Engelen & Hoos 2020). Among all kinds of semi-supervised learning methods, self-training is the closest class of methods that relates to our study. Self-training refers to the class of hots that Irain a supervised learning algorithm using labeled and unlabeled data logether [Trguuero et al. 2015). This approach is usually done by assigning pseudo labels to the unlabeled data and jointly refining the supervised learning model and the pscudo labels by iterative training (Yarowsky 1995: Rosenberg et al.) 2005 Dópido et al., 2013 Wu et al., 2012 Tanha al.) 2017). For most. traditional learners, this means re- training the learning algorithm many times as the pseudo labels are being updated. However, this pseudo-label training approach naturally works with incremental learning algorithmss like neural networks, where the model is gradually learned through optimizing an objective function (Lee et al., 2013 Berthelot et al.] 2019 Zoph et al.) 2020 Xie et al. 2020 Sohn et al., 2020). Strictly speaking, all methods focus on assigning labels in the prediction space to the data points in the primary space that is the same as the labeled data.
pseudo-labelingg seweeling/sllfttreeeee
There exists another line of research that studies semi-supervised learning in the context. of multi-view multi-modal learning. For example, multi-view co-training methods (Blum & Mitchell, 1998 Kiritchenko & Matwin 2001 Wan 2009
Du et al., 2010) propose to train different classifiers on different views of the same data. Multi-view co-training generally assumes the two views are independent of each other but can perform equally well in a single-view prediction task. The development of deep neural networks also introduced a lot of cross-modal semi-supervised learning works, which mainly focus on computer vision and natural language processing (Nie et al., 2017b a. 2019: Jīa et al. 2020 Again, all of these mentioned works rely on the availability of labels in all the modalities.
As can see, all eristing semt-supervised learning frameworks consider the case where the unlabeled data resides in the same space joint space the labeled data. Our work considers the case where the unlabeled data resides in a completely different space than the labeled data.
3 Estimating the Missing Information
In this section, using kernel density estimators, we show that the missing information estimation problem coincides with the Madaraya-Watson (NW) kernel regression. We further establish that this particular multivariate estimation scheme yields asymptotically vanishing crror.
3.1 Cross-Modal NW Kernel Regression
For clarity, we focus on estimating the missing information of one data point x, which can be obtained from a data point z in another mode. We denote the corresponding representation of x in M₁ as m, = (x).. Consider the missing information for x in space M2 as m = Φ2(2), which needs to be estimated with the
help of reference dataset Z {2. 2 y- Let write the conditional expectation of the missing information m2 given the representation mı follows
B(malm;) / f m2 P2 (2(m))
P1 (m2, mi
dmg,
(1)
where Pı (my, m1) is the joint density of m2 and m1, and p2 (mı) is the marginal density of m Using reference data Z: ya= we apply kernel density estimation (Rosenblatt) 1956 for both above densities in the following form
1
Pi (1 (m)
th A Pp (12(2)))
n
==1
(2)
(2(m1)
~
f2())(x)),
4

---

Published in Transactions on Machine Learning Research (06/2024)
for proper kernel functions l and k1, which results in the following proposition.
Proposition 1. The missing information estimalion formalalion in Equation can be ayprozimaled with kernel density estimators in Equation [ When the kernel function k1(,, in Equation (2) is density function for distribution with mean ,,, the opprorimation leads to
Rx. (())((x), Φ1 (Z.) (z))) (m
(3)
دزر (x) 01 (z,))
The proof can be found in the Appendix. The proposition implics that the conditional expectation of missing information (given reprosentation) leads to multivariate version of the well-known NW kernel regression estimator (Nadaraya) 1964 Watson 1964) that employs both modalities of data with the help of kernel we will see that the above formulation recovers the popular kernelized cross-attention when Φ1, 02, Ø3 are linear mappings.
density estimators. In Section 4.1
3.2 Estimation Error Guarantee
We now investigate the estimation error of Equation (3) under several assumptions, especially for its connection to the reference sample size and the choice of the kernel function. In particular, we consider the case that x and z replace my (i.e., Ø3 (x(x), (z)) and ma (i.e., d2(z)), respectively. The NW kernel regression formulation
in Equation now becomes
(x, x;) 1=1 Σ (x,,)
f(x) Σ
(4)
where kx x.) subscript of n, is dropped for convenience.
[HT k( xx is a kernel that satisfies the following mild technical assumption and the
Assumption 1. Wand & Jones 1994) The bandwidth matrix H h2 of the kernel function kH(-) with
H| h (the subscript shows the dependence of h to the number of data points) has the following
properties
lim hn O, n n-00
(5)
lim nhe co.
which implies that the bandwidth parameter hn decays slower than n 1/d and converges to The standard ihift-invariant kernel function k--) is bounded, symmetric probability density function with zero first moment and finite second moment. That is, the following properties hold
f 1(x)dx=1, S xk(x)dx 0,
[ xx xxTx(x)dxx= =2(()]
(6)
ldd
/ (x)xdxx== = R(k),
where (k)))) < oo and R(k) < oo are constants decided by the choice of kernel k.
It is easy to verify that several popular shift-invariant kernels k(x y) (e.g., Gaussian kernel) satisfy the above assumption with proper normalization. We also put mild assumptions on the density functions and true mapping.
The true density function p(x) is differentiable and the f2-norm of its gradient is bounded. The underlying true function f : X 2, (i.e., Mi - M222) has a bounded gradient and Hessian in the norm sense, and we have z = fxx) +e, where is an isotropic noise vector from (0,
Assumption 2.
Theorem 1. Under Assumption and Assumption g the NW kernel regression estimator f(xx in Equation 4 with isolropic shift-inwariant kernel of bandwridth H I yields estimation error e(x) f(x) f(x)
5

---

Published in Transactions on Machine Learning Research (06/2024)
that asymptotically converges in distribution
h (k) p(x)
R(k)o² 3, No p(x)
nhh e(x)
1);
(7)
where x E Rd, and the i-th entry of (x) is
L (x) p0(x) 2+ (2) (x)) Tr
(8)
where V2 (x) and (x) are the Hessian and gradient of i-th entry of the true function with respect to x, Vp(x) is the gradient of the true density function, and [r is the trace operator.
The proof is provided in the Appendix. The above theorem shows that with proper kernel function, the estimation error e(x) + 0 as n - under Assumption and Assumption 2 This suggests that under an ideal latent space transformation, more unlabeled data helps drive the estimation error to zero. We also observe that the error term e(x) converges to a sequence that scales with the factor p2(k), which implies kernel functions with lower (2(k) might contribute to a smaller e(x) the non-asymplotic regime.
4 The Attention Patch
In this section, we formally propose The Attention Patch (TAP) by showing that the NW kernel regression formulation in Equation 3 with linear latent space transformation results in the popular 'cross-attention'
module (Vaswani et al. 2017
We then propose a batch training strategy for scalability.
4.1
Module
In Equation (3), we showed how to estimate the missing information using the reference cross-modal data Z. However, implementing this mechanism requires learning the latent space transformations Φ1, d2,, Φ3- Fortunately, a neural network integration of such formulation will allow the latent space transformation to be learned in parallel with the main learning objective. To be more specific, we can define the latent space transformations 02; 62, Φ3 with learnable linear weight matrices WR, W., Wg: and we can see that the RHS (3) now becomes the kernelized version of the "cross-attention' module (Vaswani et al. 2017 Furthermore, shown in Figure 2 TAP can be inserted between any two consecutive layers in a deep neural network with minimal modification to the original network. The integration process is equivalent to applying a patch to an existing neural network, hence the name The Atlention Patch (TAP).
of Equation
Now, we formally propose The Attention Patch in the following corollary.
Corollary 1. For an output x of a layer in a deep nearal network and an unlabeled cross-modal reference dataset Z, the TAP integration means calculating the following
n2
k(W W kW,
m2
and concatenating and fyg for downstream tasks. The parameters W 9. WR, W. are learned in parallel with the original neural network. k(.,-) is shift- inift-invariant kernel of choice, and Gaussian kernel is recommended.
Note that Equation 9 allows for easy implementation of TAP by feeding the whole set of reference data Z as keys and values in cross-attention module during the training process. However, the latent space transformation can go beyond linear, we will discuss later in the simulation.
4.2 Batch Training
The attention formulation of cross-modal learning in Equation requires setting the keys and values to be the set of unlabeled reference data. points Z. Theorem 1 suggests that using more reference data for the model

---

Published in Transactions on Machine Learning Research (06/2024)
w/o TAP
with TAP
X
TAP
DNN
DNN
NW
y
y
Figure 2:
The Attention Patch (TAP) neural network integration visualization: TAP takes the output of layer to calculate the missing representation using reference data Z, and the output of TAP will be concatenated with TAP input and fed to the next layer. The only modification to the original deep neural network (DNN) is increasing the input dimension of the integration layer (blue layer).
will potentially result. in lower estimation error. However, the computation complexity of the cross-attention module scales linearly with respect to the sequence length of keys and values, which is equivalent to the number of reference points n, in Z E xd2, So, it is not practical to feed all the reference points at once, mainly due to memory limitations.
Therefore, we can break the reference dataset Z into m batches { and train each epoch by iterating over the set of reference batches together with input batches of primary data points. This is similar in vein training with stochastic gradient descent.
Each batch of reference points Z; will make the neural network yield output in space V Evaluating all batches [Z. will in turn yield m outputs, which can be used in different ways depending the application, like the ensemble approach (Deetterich et al. 2002 Sagi & Rokach 2018 in classification tasks.
In a nutshell, TAP intogration without batch training will incur an additional memory cost of O(dgnz), and batch training reduces the memory cost to ((===)) For reference, the additional memory required for a forward path in TAP integrated model (written in PyTorch) is around 1.3GB for one training data point and 100K reference data of dimension 98.
5 Numerical Experiments
5.1 Performance Evaluation
In this subsection, evaluate TAP by plugging it into neural network classifiers. We show the effectiveness of TAP by comparing the performance of TAP-integrated networks with other variants. The simulations are implemented on three real-world datasets in different areas. Full implementation details for all experiments in this section can be found in the Appendix for reproducibility.
Datascts: To ensure a comprehensive evaluation of the performance of TAP intogration, we select/ /cccat three real-world cross-modal datasets in three different areas. All datasets are open-access and can be found online. A detailed dataset and pre-processing description be found in the Appendix.

---

Published in Transactions on Machine Learning Research (06/2024)
We crop the upper half of all images the primary modality X for digit prediction and use the lower half of all images as the reference modality Z (without labels and without pairing). This creates two data modes that have guaranteed complementary information while having different distributions. In model inference (testing); the lest data points are upper-half images from the primary modality that are not present. in the training set.
Computer Vision:
We start with the MNIST dataset (MNIST) (Deng 2012
Healthcare: We use the Activity dataset (Activity) (Mohino-Herrame et al. 2019), where the Electrodermal Activity (EDA) signals are the primary modality X for predicting the subject activity. Thoracic Electrica Bioimpedance (TEB) signals are used as the reference dataset Z (without labels and without pairing). In model inference (testing), the test data points are EDA signals from the primary modality that not present in the training set.
Remote Sensing: We also choose the Crop dataset (Crop) (Khosravi et al., 2018 Khosravi & Alavipanah, 2019), where the optical features X are used to predict the crop type, and the radar readings used reference dataset Z (without labels and without pairing). In model inference (testing), the test data points optical features from the primary modality that not present in the training set.
There is overlapping instance among the training data (in space X), reference data (in space 2), and test data (in space X)..
Models: It is difficult to directly compare the effectiveness of TAP against existing state-of-the-ad neural network architectures since this work the first to propose cross-modal learning from a fully unlabeled data modality. However, we can show the effectiveness of TAP by carefully examining the performance difference across different variants of TAP.
Our Baseline competitor is single-modal neural network without TAP integration at all. However, TAP integration will bring three changes to the baseline model, including depth increase, the addition of cross- attention structure, and the reference data Z. To examine the contribution of these three aspects, we further propose three competitors as follows:
1. FFN: We replace TAP with a feedforward network to create the FFN variant. So, if FFN performs worse than TAP, we are able to factor out the impact of depth increase with TAP integration.
2. Control Group: We replace the real reference data Z in TAP with random noise of the same mean and variance. So, if Control Group performs worse than TAP, it suggests that the addition of a cross-altention structure is not the main reason for TAP to work, and using meaningful reference data (like Z) is important.
3. TAP w/o Batch:
We disregard batch training stratggy by incorporating the whole reference dataset. throughout the training process. So, if we observe TAP w/o Batch outperforms TAP, it would suggest the model is learning from the reference data, and more reforence data will help.
We now highlight some important details here: First, the normalization parameter in the attention module of TAP set to vd(n./m)-1// such that it follows the bandwidth Assumption [ while being close to the generally recommended normalization constant vd. Second, at each Monte-Carlo simulation, the set of training data, reference data, and evaluation data are shuffled while keeping the amount the same. The Control Group also generates a new set of random reference data at each Conte-Carlo simulation.
The simulation results are shown in Figure The error bars are caleulated over 20 Monte-Carke simulations to reflect the statistical significance of the results. As we can see, there is a consistent performance hierarchy among all the benchmark models throughout all datasets in different areas. First, we see that TAP integration always leads to a performance improvement compared to the baseline classifier. Second, we see that the FFN variant shows no performance advantage against the bascline model, which rules out the possibility that the depth increase in TAP the major factor causing performance improvement. Third, observe the worst generalization performance across all benchmark models for the Control Group. This
Results:
8

---

Published in Transactions on Machine Learning Research (06/2024)
MNIST prediction comparison
Activity prediction comparison
Crop prediction comparison
70.0220
74.12
70,265
36.065
Predicioon Accurecy
37,375
72,543
05.025
15.165
Control Group FFN
Baselire
TAP TAP we Batch
Control Group FFN
Baseline
TAP TAP we Batch
Control Group FFN
Baseline
TAP TAPP w/o Batch
Figure 3: Simulation results on three real-world datasets. TAP integration shows a consistent performance advantage compared to other variants.
Reference Batch Size Comparison for MNIIST
Reference Batch Size Comparison for Activity
Reference Batch Size Compartson for Crop
36.-
Prediction Accuracy
005
36.-
0.5
1.25
2.5
Unlabeled to labeled ratio
Unlabeled labeled rotio
Unlabeled to labeled ratio
Figure 4: Reference batch size comparison on three real-world datasets. The generalization accuracy increases as the reference batch size becomes larger.
shows that feoding irrelevant information will exacerbate the generalization performance. Finally, that batch training and evaluation for TAP results in slightly worse generalization performance compared to TAP w/o Batch, which trains directly with the whole reference dataset. Therefore, having more reference data indeed helps.
5.2 Ablation Study
In this subsection, further conduct an ablation study for TAP to investigate its best configuration in practice. We discuss the batch size comparison and the choice of kernel function for TAP. Next, we look at the impact of nonlinear latent space transformation on TAP. Then, we examine the compatibility of TAP with large backbone feature extractors (for vision and language). Finally, we investigate shared space Icarning and dummyy reference modality.
Reference Batch Size Comparison: To further evaluate the effect of batch training, we look the performance of different reference batch sizes. The total number of reference data n, is five times of labeled data n. Figure shows the prediction accuracy with respect to 'unlabeled to labeled ratio", which is defined as the reference batch size divided by Rz.
The performance of the baseline model is shown as the horizontal dotted line. The standard errors are calculated over 10 Monte-Carlo simulations. We observe that the generalization performance of TAP improves as the reference batch size incrcascs, even though the total number of reference data remains the same. This observation is consistent with the estimation error characterization in Theorem [ In practice, this suggests that one can increase the reference batch size much possible until the memory limit is reached.
Choice of Kernels: Theorem I suggests that the estimation error diminishes as the number of reference data points goes to infinity. However, we observe that the error term is also related to p2(k), which a constant determined by the choice of the kernel function. This connection is asymptotically negligible, but it might still be relevant in finito-sample cases. The choice of kernels in kernelized attention litcrature (Peng al.) 2020 Choromanski et al. 20200: Chen et al. 2021 Luo et al.) 2021 Xiong et al. (021)) has been empirically studied, but these works have not suggested the theoretical intuition behind the kernel choices. Here, we
9

---

Published in Transactions on Machine Learning Research (06/2024)
compare three kernels, namely the Gaussian kernel, Laplace kernel, and Inverse Multiquadric kernel. Notice that the (2( value is the smallest for the Gaussian kernel, larger for the Laplace kernel, and unbounded for the Inverse Multiquadric kernel.
Table 1:
The prediction accuracy of TAP/ / TAP w/o Batch on three datasets. Gaussian kernel moderately outperforms the other two, and the Laplace kernel is slightly better than the Inverse Multiquadric kernel within the margin of error.
Gaussian
Laplace
Inverse Multiquadric
MNIST
0.25 0.33
0.21 0.23
69.7 = 0.20 0.24
Activity
37.4 0.25/38.11 #
37.1 = 0.22/38.4 0.22
37.3 0.28 /3.5 0.28/88. 0.16
Crop
72.3 0.21 0.19
72.2 0.15/72.8 0.17
71.9 71.9 0.14 0.25
The results are shown in Table The standard errors are calculated over 20 Monte-Carle simulations, We observe that for TAP, the Gaussian kernel consistently oulperforms the other two kernels. The general performance hierarchy is also consistent with the order of (2(k) value for three kernels. So, we recommend using the Gaussian kernel for the TAP integration.
Nonlinear Latent Space Transformation: We further investigate whether nonlinear latent space transfor- mation will benefit TAP. We compare TAP with a variant where the latent space transformations Φ1, Φ2, Ø3 are modeled with multi-layer perceptrons (MLP). The results are shown in Table 2
Table 2: The prediction accuracy of TAP / TAP w/o Batch on three datasets, MLP variant shows little to no improvements compared to the linear version of TAP.
Linear
MLP 70.3 + 0.274
MNIST
70.3 + 0.25/70.9 0.22
Activity
37.4 0.25/88.. 0.19
37.8 0.22/8..2 0.16744444 0.26
Crop
72.3 7 0.2177...1
The standard errors calculated over 10 Monto-Carlo simulations. We observe that MLP variants in general provide accuracy that is no worse than the linear version of TAP. The only statistical advantage of MLP can be found in the Crop dataset, where the data are radar and optical images, which are information- rich and often require nonlincar mappings for fcature extraction. We suggest using nonlincar latent space transformation, including backbone feature extractors for vision or language data, but adhering to linear transformation for tabular data for computational purposes.
Backbone Compatibility: We note that TAP considers cross-modal learning from a probability theory standpoint, which is domain agnostic. However, given the popularity of computer vision and large language models, it is appealing to see if TAP is compatible with pre-trained feature extractors, such as Convolutional Neural Networks (CNNs) for images and Transformers for language.
Table 3: The prediction accuracy (%) and F1 score comparison between baseline model and TAP integrated model. TAP integration shows clear advantage in both accuracy and Fl score.
Accuracy(%)
F1 Score
Baseline TAP
44.56 = 1.07
0.453 + 0.005
47.68 + 0.48
0.470 0.006
We carry out the test on a fourth dataset, Memotion 7K dataset (Sharma et al. 2020 Specifically, we focus on sentiment classification tasks with images as the primary modality and text as the sccondary modality.
We investigate modified TAP integration as shown in Figure 5 We choose pre-trained NeficientNet-B00 (Tan & Le 2019) the image feature extractor and pre-trained distilled-RooFne (Sanh et al. 2019 the text.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Baseline
TAP
TAP
Efficient
Efficient
Net
Net
DNN
DNN
Classifier
Classifier
NW
Neutral
Neutral
Figure 5: TAP integration with pre-trained feature extractors: The primary modality prediction model takes meme image as input to predict the sentiment of the meme. A text set of batch size 100 is used as the reference secondary modality in TAP. The text data goes through pre-trained DistilledRRoBERRI before being used in TAP.
feature extractor. The implementation details can be found in the Appendix. The test accuracy and F1 score are tabulated in Table We observe that TAP integration improves both test accuracy and Fl score, which is the main objective for the imbalanced Memotion 7K dataset.
Shared Space Learning: The benefit of TAP relies on finding shared information space M1, such that similar samples in the primary modality and reference modality are close to each other in the space Mi (i.e., after proper transformations). Although we have observed clear generalization advantage, it is still important to examine whether TAP is able to find such suitable space M₁. To this end, we look at the kernel values in TAP after training for x and with the same/difee differenn labels. The results are shown in Table
Table 4: The kernel evaluations between data (one from the primary modality and one from the reference modality) that shares the same label against data with different labels.
Dataset
Same Label
Different Label
MNIST
0.4008 + 0.0001
0.3945 = 0.0001 0.7606 + 0.0003
Activity
0.7655 + 0.0004
Crop
0.6424 + 0.0001
0.6398 + 0.0001
The standard errors are calculated over all evaluation data. We observe that the kernel value k(W,,, x, Waz) is larger when and z share the same label, which means TAP is able find the appropriate shared space Mi without accessing label information for reference modality Z. This observation helps explain why TAP can provide better generalization performance.
To further verify the claim that the "extra information" in the reference modality improves learning, we carry out a simulation tricking TAP by creating dummy reference modalities for all previous three real-world datasets. To be specific, we create the reference modality in Section 5.1 by transforming the training data in the primary modality using randomly initialized feedforward neural networks (with the same output dimension as the reference data). For example, for Crop dataset, we randomly initialize neural network with input dimension 76 (for optical features) and output dimension 98 (for radar features) to transform the primary modality training data into the reference dataset. Using this have created
Dummy Reference Modality:
11

---

Published in Transactions on Machine Learning Research (06/2024)
'rfferencc-ikee dataset without any relevant cross-modal information. With 20 Mlone-Carle simulations, we get the performance results shown in Table 5
Table 5: The performance comparison of the bascline model compared to TAP with dummy reference modality.
Dataset
Baseline
TAP with dummy reference
MNIST
69.91 + 0.37
70.11 + 0.24
Activity
36.99 + 0.26
36.78 + 0.15
Crop
70.52 + 0.42
68.6 0.36
We observe that TAP with "dummy" reference modality has no statistically significant performance gain over the baseline model not being worse. This intuitive as the embedded cross-modas information does not carry any extra information.
6 Conclusion and Discussion of Future Directions
This paper investigates a novel perspective on cross-modal learning, where the objective is to transfer knowledge from an unlabeled secondary modality to improve generalization in the primary modality. We showed that the missing information in the primary modality can be estimated using an NW kernel regression formulation. This perspective leads to the design of cross-attention module in neural net work integration, which refer to The Attention Patch (TAP). We demonstrated the effectiveness of TAP integration four real-world cross-modal datasets in different domains, highlighting its performance advantage and domain robustness.
We hope that this work will inspire new perspectives on knowledge transfer from the data level for neural networks, showing how seerningly unusable data can leveraged for improved generalization. However, our work can still be improved in the future certain aspects. For example, having the secondary modality keys and values requires large computation memory in the forward pass for large datasets, and there is also guarantce that shared space and exclusive space exist between the two modalitics. Therefore, there are several promising directions for future research. For example, developing approaches that can efficiently store and reference unlabeled modalities could significantly benefit the technique, given the wide availability of massive datascts. Additionally, a principled framework for sclecting relevant reference datasets, as seen in cross-modal and semi-supervised learning, could help guide the data selection process.
Acknowledgments
The authors gratefully acknowledge the support of NSF Award #2038625 as part of the /ST/DDS/DOTT/NHHUUSDAA...... Cyber-Physical Systems Program.
References
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255. PMLR, 2013.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pottern Analysis and Machine Intelligence, 41(2):423 443, 2018.
David Berthelot, Nicholas Carlini, Tan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, 32, 2019.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Fleventh Annual Conference on Computational Learning Theory, pp. 92 100, 1998.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Jacsung Cha. Partial least squares. Advanced Methods of Marketing Research, 407:52-78, 1994.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint wrXi:1554.00922. 2015.
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nyström method. Advances in Newral Information Processing Systems,
2135, 2021.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyon Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations, 2020.
Rachel Davidson, Ankit Verma, David Santos, Feng Hao, Coleman Fincher, Sisi Xiang, Jonathan Van Buskirk, Kelvin Xie, Matt Pharr, Partha P Mukherjec, et al. Formation of magnosium dendrites during electrodepo- sition. ACS Eneryy Letlers, 4(2):375 376, 2018.
Li Deng. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
LP Devroye and TJ Wagner. The Lı convergence of kernel density estimates. The Annals of Statistics, 7(5): 1136- 1139, 1979.
Thomas Dietterich et al. Ensemble learning. The Handbook of Brain Theory and Neural Networks, 2(1): 110-125 125, 2002.
Inmaculada Dópido, Jun Li, Prashanth Reddy Marpu, Antonio Plaza, José M Bioucas Dias, and Jon Atli Benediktsson, Semisupervised self-learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing, (77)):033-44)) -44442444444 2013.
Jun Du, Charles X Ling, and Zhi-Hua Zhou. When does cotraining work in real data? IEEE Transoctions Knowledge and Data Engineering, 23(5)7788- 799, 2010.
Georgios Fvangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Ra- pantzikos, Georgios Skoumas, and Yannis Avrithis. Multimodal saliency and fusion for movie summarization based aural, visual, and textual attention. IFEE Transactions on Multimedia, 15(7):1553-1568 2013.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition, pp. 1933 1941, 2016.
Andrea Frome, Greg Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Advances in Neural Information Processing Systems, 26, 2013.
Paul Geladi and Bruce R Kowalski. Partial least-squares regression: tutorial. Analytico Chimica Acta, 185: 1-17, 1986.
David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-266) 2004.
Chaoqun Hong, Jun Yu, Jian Wan, Dacheng Tao, and Meng Wang. Multimodal deep autoencoder for human pose recovery. IEEE Transactions on Image Processing, 24(12):5659 5670, 2015.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domain few-shot recognition with unlabeled data. Advances in Neural Information Processing Systems, 34:3584 3595, 2021.
Xiaodong Jia, Xiao-Yuse Jing, Xiaoke Zhu, Songcan Chen, Bo Du, Ziyun Cai, Zhenyu Hc, and Dong Yuc. Semi-supe multi-view deep discriminant representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2496-255) -509,, 2020.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Xiao-Yuan Jing, Rui-Min Hu, Yang-Ping Zhu, Shan-Shan Wu, Chao Liang, and Jing- Yu Yang. Intra-view and inter-view supervised correlation analysis for multi-view feature learning. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 28, 2014.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 3128 3137, 2015.
Iman Khosravi and Seyed Kazem Alavipanah. A random forest-based framework for crop mapping using temporal, spectral, textural and polarimetric observations. International Journal of Remole Sensing, 40 (18):7221-7255) 2019.
Iman Khosravi, Abdolreza Safari, and Saeid Homayouni. MSMD: maximum separability and minimum dependency feature selection for cropland classification from optical and radar data. International Journal of Remote Sensing, 39(8):2155)) 2119922152 2018.
Douwe Kiela and Léon Bottou. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 36 45, 2014.
Svetlana Kiritchenko and Stan Matwin. Email classification with co-training, In Proceedings of the 2001 Conference of the Centre for Advanced Studies on Collaborative Research, pp. 8, 2001.
Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(12):2891-22) -22239999999,,,, 2013.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, 1CML, volume 3, pp. 896, 2013.
Chao Li, Shangqian Gao, Cheng Deng, De Xie, and Wei Liu. Cross-modal learning with adversarial samples. Advances in Neural Information Processing Systems, 32, 2019.
Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation lcarning. JEEE Transactions on Knowledge and Dala Engineering, 31(10):1863 1883, 2018.
Dahua Lin and Xiaoou Tang. Inter-modality face recognition. In European Conference on Computer Vision, pp. 13-26. Springer, 2006.
Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, and Tic-Yan Liu. Stable, fast and accurate: Kernelized attention with relative positional encoding. Advances in Neural Information Processing Systems, 34:22795 22807, 2021.
Behrooz Mahasseni and Sinisa Todorovic. Regularizing long short term memory with 3d human-skeleton sequences for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pullern Recognition, pp. 3054 3062, 2016.
Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alexander Berg, Tamara Berg, and Hal Daumé III. Midge: Generating image descriptions from computer vision detections. In Proceedings of the 13th Conference of the European Chapler of the Associalion for Computational Linguistics, pp. 747-756, 2012.
Inma Herran,, Roberto Gil-Pita, Manuel Rosa-Zurera and Fernando Scoane. Activity recognition using wearable physiological measurements: Selection of features from a comprehensive literature study. Sensors, 19(24), 2019.
Elizbar A Nadaraya. On estimating regression. Theory of Probabilily & Ils Applicalions, 9(1):111- 142, 1964.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multimodal deep learning. In International Conference on Machine Learning, 2011.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Feiping Nie, Guohao Cai, and Xuelong Li. Multi-view clustering and semi-supervised classification with adaptive neighbours. In Thirly-first AT Conference Artificial Intelligence, 2017a.
Feiping Nie, Jing Li, and Xuelong Li. Convex multiview semi-supervised classification, IEEE Transactions on Image Processing, 26(12)55788-5729) 2017b.
Feiping Nie, Lai Tian, Rong Wang, and Xuelong Li. Multiview semi-supervised learning model for image
classification. IEEE Transactions on Knowledgee and Dala Engineering, 32(12))2389 2400, 2019.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations, 2020.
Yuxin Peng, Jinwei Qi, Xin Huang, and Yuxin Yuan. CCL: Cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Transactions Mullimedia, 20(2):405 -4000-40,, 2017.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. International Conference on Learning Representations, 2020.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In Applications of Computer Vision and the IEEE Workshop on Motion and Video Computing, volume 1, pp. 29-32 36. IEEE Computer Society, 2005.
Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The Annals of Mathematical Statistics, pp. 832-837, 1956.
Omer Sagi and Lior Rokach. Ensemble learning: A survey, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiu preprint 1510:191000100 2019.
Abhishek Sharma, Abhishek Kumar, Hal Daume, and David W Jacobs. Generalized multiview analysis: A discriminative latent space. In 2012 IEEE Conference on Computer Vision and Pottern Recognition, pp. 2160 2167. IEEE, 2012.
Chhavi Sharma, William Paka, Scott, Deepesh Bhageria, Amitava Das, Soujanya Poria, Tanmoy Chakraborty, and Björn Gambäck. Task Report: Memotion Analysis 1.0 @SemEvval 2020: The Visuo-Lingua Metaphor! In Proceedings of the 14th International Workshop on Semantic Evaluation SSemEval-2020, Barcelona, Spain, Sep 2020. Association for Computational Linguistics.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. Advances in Neural Information Processing Systems, 27, 2014.
Kihyuk Sohn, David Berthclot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 61144 PMLR, 2019.
Jafar Tanha, Maarten Van Someren, and Hamideh Afsarmanesh. Semi-supervised seff-training for decision
tree classifiers. International Journal of Machine Learning and Cybernetics, 8(1):355-33) -370, 2017.
Joshua B Tenenbaum and William Freeman. Separating style and content with bilinear models. Neural Computation, 12(6):1247 2000.
Isaac Triguero, Salvador García, and Francisco Herrera. Selfllabeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):245-28) -244, 2015.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the Conference. Association for Computational Linguistics. Meeting, volume 2019, pp. 6558. NIH Public Access, 2019.
Jesper E Van Engelen and Holger H. Hoos. A survey semi-supervised learning. Machine Learning, 109(2):
373 440, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Milia Polosukhin. Attention is all you need. Advances in Newral Information Processing Systems, 30, 2017.
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond J Mooney, and Kate Sacnko. Translating vidcos natural language using deep recurrent neural networks. In Proceedings of the 2015 Conference of the North American Chapler of the Associalion for Compulational Linguistics: Human Language Technologies, 2015.
Xiaojun Wan. Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 235 243, 2009.
Matt P Wand and M Chris Jones. Kernel Smoothing. CRC press, 1994.
Kaiye Wang, Qiyue Yin, Wei Wang, Shu Wu, and Liang Wang. A comprehensive survey cross-modal retrieval. arXiv preprint XrXiv:1607.000 06215, 2016.
Yinsong Wang and Shahin Shahrampour. ORCCA: Optimal randomized canonical correlation analysis. JEEE Transactions on Newral Networks and Learning Systems, 2021.
Yinsong Wang, Yu Ding, and Shahin Shahrampour. TAKDE: temporal adaptive kernel density estimator for real-time dynamic density estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Geoffrey S Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, pp. 359-372 1964.
Zhiang Wu, Junjie Wu, Jie Cao, and Dacheng Tao. HySAD: A semi-supervised hybrid shilling attack detector for trustworthy product recommendation. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 085-9993, 2012.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10687-1000 10698, 2020.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-basee based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14138-14148, 2021.
Ran Xu, Caiming Xiong, Wei Chen, and Jason Corso. Jointly modeling deep vidco and compositional text to bridge vision and language a unified framework. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 29, 2015.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pp. 189-196, 1995.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. Technical Report TR 1530, 2005.
Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe Le. Rethinking pre-training and self-tree training. Advances in Neural Information Processing Systems, 33:3833 3845, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2024)
A
Appendix
The organization of this Appendix is as follows:
Subsection A.1
presents the experimental details, including dataset descriptions, preprocessing descriptions, and training details for reproducibility.
In Subscction
A.2 we provide the proof for all of the theoretical claims in the paper.
A.1 Experimental Details
A.1.1 Dataset
Dataset Description:
MNIST is well-known dataset with 784 pixel value features. We crop the upper half of images the primary modality, which corresponds to the first 392 features in the dataset, and the other 392 features form the lower half images. In this case, we know for sure that the two modalities are located in different spaces and carry complementary information.
Activity is an activity prediction dataset that predicts the status of a person through wearable physiological measurements. There are 151 features for Thoracic Electrical Bioimpedance (TEB) readings and 208 features for Electrodermal Activity (EDA) readings, including 104 for left readings and 104 for right readings. The objective is to predict the activity that the subject is going through, including neutral, mental, emotional, and physical activities.
Crop is a crop type prediction dataset that predicts the type of crop land grows using satellite and radar readings. There are 76 optical features collected with RapidEye satellites and 98 radar features collected with the Unmanned Aerial Vehicle Synthetic Aperture Radar system. The labels contain seven crop types, including Corn, Peas, Canola, Soybeans, Oats, Wheat, and Broadleaf.
Memotion 7K is a meme image and text caption dataset. This dataset has several prediction objectives, and we implement sentiment prediction in our simulation. For each meme image and its text caption, the instance can be labeled as very negative, negative, neutral, positive, and very positive. We treat the image dataset as the primary modality and the unlabeled text caption dataset as the secondary modality.
Data Pre-processing:
Similar to semi-supervised learning, the motivation behind utilizing unlabeled data points is the limited availability of labeled data. So, we randomly sample 200 data points in the primary modality to serve as the training data for each dataset. We further randomly sample 1000 data points in the secondary modality to serve as the cross-modal reference data. For MNIST, it mcans 200 upper half images as primary modality training data and 1000 lower half images as reference data. All the remaining data points will be the evaluation data.
The memotion 7K datasct requires more training data learn a model. So, we randomly sample 5000 data the training set, 1000 data the reference data, and the rest of data points the evaluation data.
A.1.2 Performance Evaluation
Parameter Setting:
In the performance evaluation, the reference batch size for TAP is chosen as 250, which is 1.25 times the training data. The training data batch size is set to 100. As shown in the reference batch size comparison, 250 is not the best-performing choice, but it always yields better generalization accuracy on first three datasets.
The backbone neural network structure for all three datascts two-hidden-layer neural network with 64 hidden neurons at each layer. The activation function is ReLU with a dropout rate of 0.5. Layer normalization is implemented after each hidden layer. TAP is integrated between the two hidden layers. For TAP, TAP
17

---

Published in Transactions on Machine Learning Research (06/2024)
w/o Batch, FFN, and Control Group, the second hidden layer still has 64 neurons, except the linear layer that takes input from R and outputs R61
The hidden dimension in TAP is set to 64, and the output dimension is also 64. FNN replaces the attention module in TAP with a linear layer from I6 to R Layer normalization implemented in TAP (inchuding TAP w/o Batch and Control Group) and FNN as well. The normalization parameter the attention module of TAP set to vd(nz/m)) such that it follows the bandwidth Assumption while being close to the generally recommended normalization constant Vd.
At each Monte- Simulation, the set of training data, reference data, and evaluation data are shuffled while keeping the amount the same. The Control Group also generates new set of random reference data at each Monle-Carlo simulation.
Network Training:
All benchmark variant models share similar backbone structure; therefore, the training process is also similar. All models are trained with the cross-entropy loss using the Adam optimizer with fixed learning rate of 0.0001. All models are trained for 1000 epochs (8000 in the Crop dataset) except for TAP. Since TAP updates the model several times within one epoch, the total training epochs for TAP is reduced to 1000/m, where m is the number of reference batches. For example, in the case of performance evaluation where the reference batch size is 250, there are four batches of reference data, and the total training epochs is 250 for TAP. The prediction accuracy is calculated as the lowest five-moving-averaae of validation accuracy throughout the training process, which is a common practice in neural network performance evaluation.
The batch evaluation of TAP is done by having cach batch of reference data generate prediction class and finding the majority vote among all batches just like ensemble methods.
A.1.3 Ablation Study
The experimental settings in this section of the simulation are very much identical to the previous section. The reference batch sizes are chosen 100, 200, 250, 500, 1000, which corresponds to the unlabeled to labeled ratios of 1/2, 1, 5/2, 5, respectively. The corresponding training epochs are 100, 200, 250, 500, 1000, respectively. For the Crop dataset, all training epochs are multiplied by 8 since it takes longer to train.
Reference Batch Size Comparison:
Choice of Kernels: This section adds two kernel variants of TAP integration. The only difference between standard TAP and these two variants is the kernel function in Equation 9 The Taplace kernel function is set to
x(x,)) =
(10)
where || 1/1 denotes the (--ormm, and the normalizing factor h is chosen the same as the Gaussian kernel bandwidth in TAP. The Inverse Multiquadric kernel is set to
1
y(x,y)
(11)
VIx - y12 +1
All the other training settings are kept the same as the performance evaluation section for first three datasets.
Nonlinear Latent Space Transformations: This section adds the MLP variant of TAP integration. In this variant, linear transformations are replaced with a wwo-hidden-layer neural network with a ReLU activation function. All the other training settings are kept the same as the performance evaluation section for first three datasets.
Backbone Compatibility: In this simulation, the classifier appended to feature extractors is two-hidden- layer neural network with 16 neurons at cach layer. The learning rate is set to 0.00002 for both baseline and TAP integration. Each model is trained for 20 epochs, where we observe the validation accuracy stabilizes. The reported accuracy is the average of the last five validation accuracy values.
Shared Space Learning: The model and learning process are the same as experiments in performance evaluation. We use the trained-mode with the highest accuracy in the last Monto-Carlo simulation evaluate the kernel values.
18

---

Published in Transactions on Machine Learning Research (06/2024)
A.2 Theory of TAP
A.2.1 Proof of Proposition 1
We rewrite the conditional expectation in Equation
as follows
(m2, m))
E(my ((m)\m)) = m2
dm2
Pa (m
(12)
/
P1 (mg, mi
d2(2)
ddg (z).
p2 p2(mi)
Next, plugging the density estimations in Equation
2 back into the above equation, have the following
pi (m2, mi) dz2(2))
E(m2 mi) 2
Φ2(2)
f2(
(2(22) x), Φ1 (1: (2)))
(13)
(x)u(x), 01 (z: 62
02 (oz22))
Σ/= 1(0s(()) (z;))
Σ (x)) @1 (o)))) Σj= (x)s((), 01 (2,))
where the last line simply follows from the fact that kernel function (.) has mean μa. The proof is complete.
A.2.2 Lemma
Lemma 1. Under Assumplions 7 and a kernel densily estimator
f(x)
kx xi);
(14)
is consistent, i.e., it converges in probability to the true density function p(x);
p(x) P. p(x).
(15)
This is a well-known result that can be found in the existing literature (Devroye & Wagner 1979
A.2.3 Proof of Theorem
Throughout the proof, all integrals are Riemann integral fge, and we omit the subscript for simplicity. Since we only consider shift-invariant kernels, we will use kH(x - x,) to represent an isotropic kernel function with bandwidth H = I,I, where (x(x - x;) = H k( x-Xi) for a standard inift-nnvarian kernel function k(-). The determinant of the bandwidth matrix by definition is H hd. We will further omit the subscripts of h for simplicity.
[
19

---

Published in Transactions on Machine Learning Research (06/2024)
Consider the j-th element in 2g, which we simply write as 20), Then, we have the following
(x) = 1 Ei=1 1i4( x,)=(0)
p(x)
1 Ei=1 1H -x,) (x)(xx) (x) + +0)
f(x)
(16)
= fx) + - 1H(x - (f)(((xxx)))) - (x))
1 E kH(x xx))e
f(x)
f(x)
- fx) + B(x)
x(x) + V(x)
p(x))
where theoretically characterize the properties of (xx) and V(x). We refer to the i-th term in the summation (xx) and ((x) as (x) and u, (x), and the subscripts will be omitted from now on for simplicity.
We first look at the term v(x), and it is easy to see that E[[(x)] = 0 due to our assumption on the noise term e. For the variance, we have the following asymptotic equality
H| H| (x)) [
1 s)ply)dy
a² / A2( x²(x 1P(x-N1p )p(x)dd
H
(17)
d²
12(2)p(x- HHzzdd
=0² / ((2)(((x) + o(1))dz
R(k))p()o'
where the last line follows from omitting the higher order term. Since (x) is the sample mean of random variables v(x), we can apply the central limit thcorom to show that
Vnh NO, R(k)p(x)o?)
(18)
Since p(x) 2g p(x) following Lemma
make the claim following Slutsky's theorem
VnhdV(x)) 4, NO,
R(k)o²
(19)
p(x)
p(x)
20

---

Published in Transactions on Machine Learning Research (06/2024)
Then, let us look at the term b(x). We have the following
E [bx(x)]
E
le
x
(x())(x) fx))
H
k("
(f(((()) (y)
H
k(z)
[x H|z) (x))xx) + Hz2
/ k(z)
(112) y (x) (p(x) ((H))) [p((x) dz
(20)
H f k(z)z ( "TY / A(2))(X)) V2 (x) dz
2
+ +1 / ((2) 2a Tde
[ (2 (k) p [ (x) + + (x)
H U2 D (5) (x) htdd m2()) (x).
Any term that is H ?) is omitted it asymptotiallly vanishes.
For the term [x), we use similar expansion by omitting terms of o o(H))) follows
[xb)
1
(x)) (x)
H
1
2 H
()()())) (x))x))pp())d)
H
/ (²(2(((())
H2z) - 1 (x)) + H2z)dz
(21)
H
H / (2( (v) ((x))
[HIV)(() (x) V (x)p(())))))
This suggests that the variance of b(x) of a higher order than the variance of VlH[(x). Thus, we know
vnld(B(x) 12dd ( & o.
(22)
Combining with the convergence in probability for density estimator p(xx, we have
Vnhd B(x)
h2d #2()) (1) (xx) 2 0.
(23)
x(x)
p(x)
Finally, we conclude that
h22d (1())) (x())
v Vnh (e(x)
p(x)
I), (24)
=V V d
B(x)
1/2d u2 (k) 1 (x)
V(x)
d No,
R(kk) p(x)
(x)
p(x)
((x)
where Bxx) and V(x) are the vectorization B(x) and (xx, respectively. The vectorization follows the isotropic assumption on noise
The proof is complete.
21


Reviews:
[{"review_id": "7CM4TCizOL", "reviewer": "Reviewer_XGTR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an interesting problem.\n- The paper is rather well-written.\n- The proposed methodology seems reasonable.\n\nWeaknesses:\n- I miss a complexity analysis. At the moment, it is hard for me to assess the difference between a DNN w/ TAP and a DNN w/o TAP in terms of, e.g., the number of weights.\n- I presume the authors will hate me for this comment but it would be interesting to consider a case when the \"extra\" information z comes from the prime modality and is paired with an object x from the primary modality. Then it would be interesting to see how TAP deals in such a scenario. In other words, this experiment would weaken or strengthen the paper's first contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper attacks the problem of cross-modal learning.\n- The authors propose a cross-modal mechanism based on the Nadaraya-Watson kernel regression.\n- In the experiments, they provide evidence in favor of their approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please provide some complexity analysis. It is very important for broader applicability to indicate the additional complexity of a DNN w/ TAP compared to a DNN w/o TAP, e.g., in terms of the big-O notation of extra calculations. Moreover, what is the number of parameters of the DNN w/ TAP compared to its version w/o TAP.\n- Figure 3: Please enlarge the font size.\n- Figure 4: Please remove the yellow background and enlarge the font.\n- Please consider adding a new experiment as described in the weaknesses.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wXJRgXDzNX", "reviewer": "Reviewer_6oEH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The manuscript is easy to read. Fig. 1 is clear and succinctly describes the problem setting.\n* The authors propose an important and practical setting where the reference dataset is unlabeled, is unpaired, and, can come from a different modality.\n* The paper introduces a theoretical argument to the success of the attention patch which helps the reader develop an intuition in this area.\n* The comparison of TAP with batch and without batch is quite interesting and shows promising generalization given unlabeled data at scale.\n\nWeaknesses:\n* The evaluation seems unconvincing to support the claims:\n1. Suppl. Sec. A.1.2 describes \"All benchmark variant models share a similar backbone structure\", whereas the tasks are quite different - \"MNIST\" classifies images whereas \"Activity\" classifies signals. The authors use a trivial architecture (MLP) and it is unclear if TAP would show performance gains when using a more suitable model architecture for each task. \n2. TAP is similar in nature to well-studied multi-head attention and therefore it is imperative that the authors evaluate a transformer backbone to demonstrate the gains of TAP over vanilla transformers, however such a comparison is missing. \n3. The kernel evaluations in Table 4 only show a weak difference (about 0.005) between the two cases which raises questions regarding the effectiveness of TAP (could this difference be negligible when using a more suitable architecture for the tasks?). Moreover, the ablations in Table 1 for TAP do not seem to be statistically significant (accuracy scores are within standard deviation).\n4. The most practical use-case (integrating TAP in pretrained feature extractors; Fig. 5) is under-studied -- Table 3 provides no insight into how TAP functions when using cross-modal data such as image and text. For instance, it is important to visualize what text caption is retrieved for a given image, to verify TAP indeed works in retrieving complementary information from the reference modality (similarly, for MNIST, given the upper half of the image, verify whether the lower half is retrieved from TAP).\n* Key details missing: Fig. 2 caption states \"the output of TAP will be fed to the next layer together with TAP input\" - is this a concatenation operation, or are the two inputs added? Further, could the authors clarify if backpropagation happens through the TAP module?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a plug-in module called The Attention Patch (TAP) to leverage complementary information from an unlabeled modality for semi-supervised cross-modal learning. Specifically, the authors propose TAP as a kernelized cross-attention module between the labeled and unlabeled modalities, and formulate the task as a NW Kernel Regression. The key insight is that TAP can provide meaningful signals as the data scales up. Empirical results on 4 datasets shows that TAP consistently improves the performance on classification tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please refer to the weaknesses above (for convincing results, use a more suited architecture for each task, evaluate using transformer backbones, and verify if TAP functions as expected using qualitative visualizations).\n\n* According to Assumption 2, \"the true density function p(x) is differentiable\". Could the authors clarify if this assumption imposes limitations on the kinds of datasets / tasks / input modalities?", "broader_impact_concerns": "No significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LHIVXbNpmL", "reviewer": "Reviewer_Yp98", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The writing is clear and easy to follow. Fig. 1 illustrates the studied task well.\n2. The idea of estimating the missing information using kernel density estimators is new to me and building cross-attention for better integrating the two different modalities sounds reasonable.\n3. Experiments demonstrate the effectiveness of the proposed method on its formed datasets.\n\nWeakness: \n1. Important related works are missing. The claim that the problem is the first exploration of combining cross-modal learning and semi-supervised learning is overstated. Cross-domain few-shot learning (CD-FSL) [1,2] explores a similar setting, learning from labeled source data (primary) and unlabeled, unpaired target data (secondary) of different domains. While specific settings may differ, the assertion of being the first is challenged.\n2. The testing data (T) should be defined and formulated, considering the supposed importance of domain distances between primary training data (X), secondary training data (Z), and testing data (T). The introduction of T in Sec. 5.1 might not be the most effective choice.\n3. The constructed datasets for experiments lack convincing modal differences. The paper emphasizes tackling datasets of different modalities, but the constructed datasets do not seem to meet this requirement. For example, the use of the upper half and lower half of MNIST images as X and Z, respectively, raises questions about modality differences.\n4. Discrepancies exist in the mention of datasets. The abstract and introduction state experiments on four datasets, while the experiments section includes only three.\n\n[1] Self-training for few-shot transfer across extreme task differences. ICLR. 2021.\n[2] Dynamic distillation network for cross-domain few-shot recognition with unlabeled data. NeurIPS. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies semi-supervised learning when dealing with labeled primary data and unlabeled secondary data of different modalities. To enhance supervised learning on the labeled data, the paper introduces a cross-modal learning framework. The proposed approach involves estimating missing information through kernel density estimation, and further developing a kernelized cross-attention module (TAP) that could be inserted into the existing models. The study includes experiments conducted on several real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see weakness", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All major concerns of the reviewers have been addressed. Two of the reviewers recommend leaning to accept, but neither is very excited with the overall work.
Decision: Accept (poster)