Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning
Authors: Eslam Abdelaleem
Ahmed Roman
K. Michael Martini
Ilya Nemenman
Venue: Accepted by TMLR ()

Abstract: Current experiments frequently produce high-dimensional, multimodal datasets—such as those combining neural activity and animal behavior or gene expression and phenotypic profiling—with the goal of extracting useful correlations between the modalities. Often, the first step in analyzing such datasets is dimensionality reduction. We explore two primary classes of approaches to dimensionality reduction (DR): Independent Dimensionality Reduction (IDR) and Simultaneous Dimensionality Reduction (SDR). In IDR methods, of which Principal Components Analysis is a paradigmatic example, each modality is compressed independently, striving to retain as much variation within each modality as possible. In contrast, in SDR, one simultaneously compresses the modalities to maximize the covariation between the reduced descriptions while paying less attention to how much individual variation is preserved. Paradigmatic examples include Partial Least Squares and Canonical Correlations Analysis. Even though these DR methods are a staple of statistics, their relative accuracy and data set size requirements are poorly understood. We use a generative linear model to synthesize multimodal data with known variance and covariance structures to examine these questions. We assess the accuracy of the reconstruction of the covariance structures as a function of the number of samples, signal-to-noise ratio, and the number of varying and covarying signals in the data. Using numerical experiments, we demonstrate that linear SDR methods consistently outperform linear IDR methods and yield higher-quality, more succinct reduced-dimensional representations with smaller datasets. Remarkably, regularized CCA can identify low-dimensional weak covarying structures even when the number of samples is much smaller than the dimensionality of the data, which is a regime challenging for all dimensionality reduction methods. Our work corroborates and explains previous observations in the literature that SDR can be more effective in detecting covariation patterns in data. These findings strengthen the intuition that SDR should be preferred to IDR in real-world data analysis when detecting covariation is more important than preserving variation.

Published in Transactions on Machine Learning Research (09/2024)
Simultaneous Dimensionality Reduction: A Data Efficient Approach for Multimodal Representations Learning
Eslam Abdelaleem
ealan.ddelaleemeemoryyy
Department of Physics
Emory University
Ahmed Romant
Demoryy
Deportment Physics
Emory University
K. Michael Martini Department of Physics Initiative in Theory and Modeling of Living Systems Emory University
kartmmiccaad..martineeeeenyy edu
Ily Nemenman
slyo. Saarmemman@emooy edu
Department Physics 8 Deportment of Biology & Initiative in Theory and Modeling of Living Systems
Emory Universily
Reviewed on OpenReview: https:
// openreview. net/ form?? id= N614fbbyTV
Abstract
Current experiments frequently produce high- high-dimensional, multimodal latascts-such as those combining neural activity and animal behavior or gene expression and phenotypic profiling-wii with the goal of extracting useful correlations between the modalities. Often, the first step analyzing such datasets is dimensionality reduction. We explore two primary classes of approaches dimensionality reduction (DR): Independent Dimensionality Reduc- tion (TDR) and Simultaneous Dimensionality Reduction (SDR). In IDR methods, of which Principal Components Analysis is paradigmatic example, cach modality is compressed independently, striving to retain much variation within each modality contrast, in SDR, one simultaneously compresses the modalities to maximize the covaria- tion between the reduced descriptions while paying less attention to how much individual variation is preserved. Paradigmatic examples include Partial Least Squares and Canonical Correlations Analysis. Even though these DR methods are a staple of statistics, their rel- ative accuracy and data set size requirements are poorly understood. We use a generative linear model to synthesize multimodal data with known variance and covariance structures to examine these questions. We assess the accuracy of the reconstruction of the covariance structures as function of the number of samples, signal-to-noise ratio, and the number of varying and covarying signals in the data. Using numerical experiments, we demonstrate that linear SDR methods consistently outperform linear IDR. methods and yield higher- quality, more succinct. aadnned-dimennionall representations with smaller datasets. Remark- ably, regularized CCA can identify low-dimensional weak covarying structures even when the number of samples is much smaller than the dimensionality of the data, which is regime challenging for all dimensionality reduction methods. Our work corroborates and explains previous obscrvations in the literature that SDR can be more effective in detecting covariation patlerns in data. These findings strengthen the intuition that SDR should be preferred to IDR in real-world data analysis when detecting covariation is more important than preserving variation.
possible. In
"Corresponding author Currently romandbroadinstitute. org Department Medical Oncology Dana-Farbed Cancer Institute & Broad Institute of MIT and Harvard Harvard Medical School

---

Published in Transactions on Machine Learning Research (09/2024)
1 Introduction
Many modern experiments across various fields generate massive multimodal data sets. For instance, in neuroscience, it is common to record the activity of a large number of neurons while simultancously recording the resulting animal behavior (Stringer et al. 2019 Steinmetz et al. 2021 Urai et al. 2022 Krakauer et al, 2017). Other examples include measuring gene expressions of thousands of cells and their corresponding phenotypic profiles, or integrating gene expression data from different experimental platforms, such as RNA- Sog and microarray data (Clark et al. 2013 Zhong ct al. 2017 Svensson ct al.) 2018 Huntley et al. 2015 Lorenzi et al., 2018). In economics, important variables such as inflation are often measured using combinations of macroeconomic indicators as well as indicators belonging to different economic sectors Goosselin & Tkacz, 2001 Baillie et al. 2002 Freyaldenhoven, 2022: Rudd 2020). In all of these examples,
important goal is to estimate statistical correlations among the diflerent modalities.
Analyses usually begin with dimensionality reduction (DR) into a smaller and more interpretable represen- tation of the data. We distinguish two types of DR: independent (IDR) and simultaneous (SDR) (Martini & Nemenman 2024). In the former, each modality is reduced independently, while aiming to preserve its varia- tion, which we call self signal. In the latter, the modalities are compressed simultaneously, while maximizing the covariation (or the shared signal) between the reduced descriptions and paying less attention preserv- ing the individual variation. It is not clear if IDR techniques, such as the Principal Components Analysis (PCA) (Hotelling 1933 are well-suited for extracting shared signals since they may overlook fealures of the data that happen to be of low variance, but of high covariance (Colwell et al. 2014 Borga et al. 1997 In particular, poorly sampled weak shared signals, common in high-dimensional datasets, can exacerbate this issue. SDR techniques, such as Partial Least Squares (PLS) (Wold et al.) 2001) and Canonical Correlations Analysis (CCA) (Hotelling) 1936), are sometimes mentioned as more accurate detecting weak shared sig- nal CChin & Newsted 1999 Hair al.) 2011 Pacharawongsakkda & Theeramunkong 2016). However, the relative accuracy and data set size requirements for detecting the shared signals in the presence of self signals and noise remain poorly understood for both classes of methods.
In this study, we aim to assess the strengths and limitations of linear IDR, represented by PCA, and linear SDR, exemplified by PLS and CCA, in detecting weak shared signals. For this, we use a generative linear model that captures key features of relevant examples, including noise, the self signal, and the shared signal components. Using this model, we analyze the performance of the methods in different conditions. We assess how well these techniques can (i) extract the relevant shared signal and (ii) identify the dimensionality of the shared and the self signals from noisy, undersampled data. We investigate how the signal-to-noise ratios, the dimensionality of the reduced variables, and the method of computing correlations combine with the sample size to determine the quality of the DR. We propose best practices for achieving high-quality reduced representations ions with small sample sizes using these linear methods.
Overall, the main contributions of this paper are:
We define a tractable, generative, linear model for producing multimodal datasets; the model al- lows us to tune the numbers and the strengths of the shared and the self signals in the generated modalities.
We characterize the accuracy and data set requirements for reconstructing the shared signals by SDR methods (CCA, rCCA, and PLS) and IDR methods (PCA) as a function of the parameters of the generative model.
We find that SDR methods generally outperform IDR methods detecting shared signals in mul- timodal data; this is true for both the synthetic generative linear model, as well for the non-linear data derived from the MNIST dataset.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2
Model
2.1
Relations to Previous Work
The extraction of signals from largo-dimensional data sets is a challenging task when the number of observa- tions comparable to or smaller than the dimensionality of the data. The undersampling problem introduces spurious correlations that may appear as signals, but are, fact, just statistical fluctuations. This poses a challenge for DR techniques, as they may retain unnecessary dimensions or identify noise dimensions as true signals. Here, we focus exclusively on linear DR methods. For these, the Sarchenko-Pastur (MP) distribu- tion of eigenvalues of the covariance matrix of pure noise derived using the Random Matrix Theory (RMT) methods (Marchenko & Pastur 1967) has been used to introduce cutoff between noise and true signal in real datasets. However, recent work (Fleig & Nemenman 2022a) has shown that, when observations are a linear combination of uncorrelated noise and latent low-dimensional self signals, then the self signals alter the distribution of eigenvalues of the sampling noise, questioning the validity of this naive approach.
Moving beyond single modality, [Buchhud et. al. (2007) calculated the singular value spectrum of cross- correlations between two nominally uncorrelated random signals. However, it remains unknown whether the linear mixing of self signals and shared signals affects the spectra of noise, and how all of these components combine to limit the ability to detect shared signals between two modalities from data sets of realistic sizes. Filling in this gap using numerical simulatiions is the main goal of this paper, and analytical treatments of this problem are left for the future.
The linear model and linear DR approaches studied here do not capture the full complexity of real-world data sets and state-of-the-art algorithms. However, if sampling issues and self' signals limit the ability of linear DR methods to extract shared signals, it would be surprising for nonlinear methods to succeed in similar scaling regimes on real data. Thus extending the previons work to explicitly study the effects of linear mixtures of self signals, shared signals, and noise on limitations of DR methods is likely to result in intuition transferable to more complex scenarios. Examples of such scenarios might include inference of dynamics of a system through a latent space (Creutzig et al. 2009 Chen et al.] 2022), where shared signals correspond to latent factors that are relevant for predicting the future of the system from its past, while self signals correspond to nonpredictive variation (Bialek et al., 2001]. In economies, shared and self signals correspond
to diverse macrocconomic indicators that are grouped into correlated distinct categories in structural factor
models (Forni & Gambetti 2010 Gosselin & Tkacs| 2001 Rudd 2020 Baillie et al. 2002). In neuroscience, shared signals can correspond to the latent space, by which neural activity affects behavior, while self signals encode neural activity that docs not manifest in behavior and behavior that is not controlled by the part of
the brain being recorded from (Sponberg et al.] 2015 Stringer el al., 2019 Natraj el al. 2022 Sani et al.
2021
Pang et al. 2016 Urai al.) 2022 Krakaner et al. 2017).
Interestingly, in the context of the neural control of behavior, it was noticed that SDR reconstructs the shared neuro-behavion latent space more efficiently and using a smaller number of samples than IDR (Sani et al.
2021
Similar observations have been made in more general statistical contexts (Chin & Newsted 1999: Hair
et al. 2011 Pachharaonngaakda & Theeramunkong 2016 Vogelstein et al. 2021), though the agreement is not uniform [Goodhue et al.) 2006] 2012 2013). Because of this, most practical recommendations for detecting shared signals are heuristic (Hair Jr al. 2021), with widely acknowledged limitations (Kock & Hadaya 2018). Our goal to ground such rules numerical simulations and scaling arguments.
2.2 Linear Model with Self and Shared Signals
We consider linear generative model of data, which includes noise, Maelf.. Maelty self signals that are relevant to each modality independently, as well as mshared shared signals that capture the interrelationships
3

---

Published in Transactions on Machine Learning Research (09/2024)
between modalities
It results in T observations of two standardized observables, X and Y:
[X E [ =
Rx
UxVx
+
PQx
Independent white noise
Self Signal for X
Shared-Signal
[ E T =
Ry
Uv
+
PQy
(1)
Independent white noise
Self-Signal for Y
Shared-Signal
x=
(2)
The observations of X and Y are linear combinations of the following: (a) independent white noise com- ponents Rx E RTxNN and Ry E RxxNy with variances Jy and Ry. : (b) self-signal components Ux and Uy residing lower-dimensional spaces Txxmaett and IR x respectively, with the corresponding variances ox and oty; (c) shared-signal components P in shared lower-dimensional space IR with variance pp. These lower- diver-dimensional components are projected into their respective high-dimensional spaces RTxNx and RTxNN using fixed quenched projection matrices Vx E _==xattxxxxxx Vy gm X. Qx E T Nx, and Qy E "R x XNy, with variances o oyy Tgx, and ogy, respectively. Entries in these matrices are drawn from Gaussian distributions with zero means and the corresponding variances. Further, division by x and oy standardizes each column of the data matrices by their empirical standard deviations. The total variance in the matrix X can be calculated as the sum of the variances of its individual components: xx fix + Maelf,X X oi oix + Mahared X ox and similarly for Y.
We define self and shared signal-to-noise ratios X/Y, Xsh as the relative strength of signals com- pared to background noise per component in each modality. These definitions allow us to examine how easily self or shared signals in each dimension can be distinguished from the noise.
ot txyy Exir
O2-âxIN
X/Y
X/y
R
Rx/Y
Our main goal is to evaluate the ability of linear SDR and IRR methods to reconstruc 2 the shared signal P, while overlooking the effects of the self signals Ux/y on the statistics of the shared ones. We formalize this goal in the next section by defining a shared signal reconstruction metrie RC'.
3 Methods
We apply DR techniques to X and Y to obtain their reduced dimensional forms Zx and Zy, respectively. Zx. Zy are of sizes that can range from T x to T Nx and T Ny, respectively. As an IDR method, we use PCA (Hotelling) 1933). As SDR methods, we apply PLS (Wold et al. 2001) and CCA (Hotelling) 1936; Vinod 1976: Arup Nielsen et al. 1998), including both normal and regularized versions of the latter. Each of these methods focuses on specific parts of the overall covariance matrix
Cxx Cxr Crx
L x IXTY 1 L X L
Cx.y
(4)
Crr
PCA aims to identify the most significant features that explain the majority of the variance in Cxx and Crv; independently. PLSS, on the other hand, focuses on singular values and vectors that explain the covariance component CxY. Along the same lines, CCA aims to find linear combinations of and Y that are responsible for the correlation (Cxy (Cxx/VCxxCvv) between X and Y (Borga et al. 1997.. See Appendix A.I for detailed description of these methods.
"This model is am extension of the model introduced by Fleig & Nemenman 2022a and its probabilistic form has been studied by Murphy (2022). In its turn, the latter is extension of work by Klami al. (2012), and Bach & Jordan (2006). However, within this model, we focus on the intensive limit, common in RMT (Potters & Bouchaud 2020), where the number of observations scales the number of observed variables. This scenario is common in many real-world applications. To knowledge, treatment of the extensive regime assess different DR methods fuuction of various parameters of the system does not exist.
2In this context, use "reconstruccton" and "detection" R interchangeably. Detection means that are able capture the shared signal, which is equivalent to reconstruction, as the signal is represented by the correlation.
4

---

Published in Transactions on Machine Learning Research (09/2024)
For every numerical experiment, we generate training and test data sets (Xurams Yuaiu) and (Xuett Yees) according to Egs. 12 We apply PCA, PTS,, CCA, and regularized CCA (rCCA) to the training to obtain the singular directions W Wxeram and WY..in for each method (see Appendix A.1 We then obtain the projections of the test data on these singular directions
Zx = Xtest WX Wxisin
Zy = Yost WY
Finally, we evaluate the reconstructed correlations metric RC', which measures how well these singular directions recover the shared signals the data, corrected by the expected positive bias due to the sampling noise, see Appendix A.2 for details. RC' = 0 corresponds to no overlap between the true and the recovered shared directions, and RC" 1 corresponds to perfect recovery.
4 Results
4.1 Results for The Generative Linear Model
We perform numerical experiments to explore the undersampled regime, T & Ny, We use T {100, 300, 1000, 3000} samples, Nx =
1000. We explore the case of one shared signal only, mahared = = =Ny and we mask this shared signal by a varying number of self signals and by noise. We vary the mumber of retained dimensions, [1 Zy | and explore how many of them are needed to recover the shared signal in the noise and the self signal background with different SNRs.
For brevity, we explore two cases: (1) one self-signal each in X and Y in addition to the shared signal (Maet 1); (2) many self-signals in X and Y. For both cases, we calculate the quality of reconstruction as the function of the shared and the self SNR, "shared and Yself- In all figures, we show RC' for severely undersampled (first row, T = 300) and relatively well sampled (second row, T= 3000) regimes. We also show the value of RCo, the bias that we removed from our reconstruction quality metric, for completeness, see Appendix A.2 for details. Experiments at different parameter values can be found in Appendix A.4
Figure 1 shows that, in Case 1, when one dimension is relained in DR of X and Y, PCA populates the compressed variable with the largest variance signals and hence struggles to retain the shared signal when Yeelf Yshard, regardless of the number of sample However, both PLS and rCCA excel in achioving nearly perfect reconstructions. When T & Nx, straightforward CCA cannot be applied (see A.1.3 A.1.4 4. but it too achieves a perfect reconstruction when T> Nx.
In Fig. 2 we allow two dimensions in the reduced variables. For PCA, we expect this to be sufficient to preserve both the self and the shared signals. Indeed, PCA now works for all ys and Ts, although with a slightly reduced accuracy for large shared signals compared to Fig. [ PI.S and rCCA continue to deliver highly accurate reconstructions. So does the CCA for T > Nx. Spurious correlations, as measured by RCo grow slightly with the increasing dimensionality of Zx. Zy compared to Fig. [ This is expected since more projections must now be inferred from the same amount of data.
We now turn to Mself > Msharod- We use mahared = 1, Msser = 30 for concreteness. We expect that the performance of SDR methods will degrade weakly, as they are designed to be less sensitive to the masking effects of the self signals. In contrast, we expect IDR to be more easily confused by the many strong self- signals, as IDR is designed to pick up any large variance components, which would include self signals if they are stronger than the shared signal, degrading the performance. Indeed, Fig. 3 shows that PCA now faces challenges in detecting shared signals, even when the self signals are weaker than in Fig, ! Increasing
WW fix of "xix"" Axyy 2 @x/y and allow o of to vary as equally spaced values between 0.05 and 1- when we choose X/Y "shared.X/Y The SNRs of X and Y are symmetric. We first generate the fixed projection matrices Qx/Y: and we vary Rxyy- Ux/Y, P for each trial.
WWe use the notation | to specify the dimensionnlity (or the number of dimensions are keeping after reduction) of variable. We reserve the usage of the notation ||. || to the matrix norm, as used defining the RC metric (c.f. A.2
Thee transition from being able to observe the shared signal to not being able to observe it in data as function of the sample size resembles the famous BBP transition (Baik et al. 2004), which can be explored analytically for random matrices of the form Eqs. I which we leave for future work.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Adding self-signal, keeping dimension after reduction
PCA
CCA
iCCA
RC
Noise
RCg
1.0
1.0
0.12
A1
B1
C1
D1
E1
T 300
0.7-
0.10
3.06
0.8
9.5
0.4
0.08
0.1
0.06
1.0
0.9
E22
T= 3000
0.8
30.6-
20.5
0.4
-0.02
0.3
02
07
033
0
0.01
07 0.3
Figure Performance of PCA, PLS, CCA, rCCA, and noise in recovery of the shared signal for [x Zy 1 maelf- The
undersampled (top) and relatively well-sampled (bottom) scenarios, respec- tively. PCA struggles to detect shared signals when they weaker than the self signals, with more samples. PLS and rCCA demonstrate nearly perfect reconstruction. CCA displays no reconstruction in the undersampled regime T' < Nx. and it is nearly perfect for large T.
Adding self-signal, keeping 2 dimensions after reduction
PCA
PLS
CCA
ICCA
RC
Noise
RC₂
A1
B1
D1
E1
300
0.7
0.10
2.06
0.55
0.2
-0.6
0.1
-0.06
1.0
86
B2
D2
004
E2
T= 3000)
0.8
0.04
10.6
2.55
0.1-
0.02
0.3
0.
1--
0.0
0.00
Figure Same as Fig. 1 but for |xxl = |2y = = Mself Mshared- Now there are enough compressed
variables for PCA detect the shared signal. Other methods perform similarly to Fig. I albeit the noise is larger.
T improves its performance only slightly. Somewhat surprisingly, PLS performance also degrades, with improvements at T > Nx. CCA again displays reconstruction when T & Nx, switching to near perfect reconstruction at large T. Crucially, rCCA again shines, maintaining its strong performance, consistently demonstrating nearly perfect reconstruction.
Since one retained dimension not sufficient for PCA to represent the shared signal when Tshared S Yeelf we increase the dimensionality of reduced variables, |Zxl |Zy| maelt > mabared- cf. Fig. PCA now detects shared signals even when they are weaker than the self-signals, Yshared < Yself but a cost of the reconstruction accuracy plateauing significantly below 1. In other words, when self and shared signals

---

Published in Transactions on Machine Learning Research (09/2024)
Adding 30 self-signals, keeping dimension after reduction
CCA
iCCA
RC'
Noise
RC
1.3
A1
B1
C1
D1
E1
300
0.8
0.7-
0.8
1.4
0.3
1.2
0.2
-0.6
0.1
1.0
1.3
A2
B2
C2
D2
0.4
E2
0.88
T= 3000
0.7
0.8-
0.6
30.5
50:5-
0.2
0.4
0.3-
0.2
0.2
10
0.
0.2
02
00
04
0.0
0.0
Figure 3: Reconstruction results for maelf 30, Mahared = 1, and |xx | PCA struggles to detoct any shared signals when they are even comparable to the self ones. PLS performance also degrades. CCA displays its usual impotence small T. Finally, rCCA demonstrates nearly perfect reconstruction for all parameter values.
Nx, PCA cannot are comparable, they mix, allowing for partial reconstruction. However, even reconstructing shared break into the phase diagram's lower right corner. Other methods perform similarly, signals over the same wider ranges of sampling and the SNR ratios than in Fig. 3 For all of them, the improvement comes at the cost of the decreased asymptotic performance. The most distinct feature of this regime the dramatic effect noise, where 30-dimensional compressed variables can accumulate enough sampling fluctuations to recover correlations that are supposedll nearly twice as high as the data actually has.
Figure now explores regime when the dimensionality of the compressed variables is enough store both the self and the shared interactions at the same time, |Zx |Zy| Mself Myhared 31. With just more dinension than Fig, 4 PCA abruptly transitions to being able to recover shared signals for all SNRs, albeit still saturating at a far from perfect performance at large T. PLS, CCA, rCCA, and noise show behavior remain similar to Fig.
Our analysis suggests that there are three relevant factors that determine the ability of DR to reconstruct shared signals. The first is the strength of the shared and the self signals compared to each other and represent
noise. For brevity, in the following analysis, we fix 7seff and define the ratio 7 "sshare//ssff this effect. The second factor affecting the performance is the ratio between the number of shared and self signals, denoted by ni = Mshared/ Mself- The third factor is the number of samples per dimension of the reduced variable, denoted by q == TT
In Fig. 6 we illustrate how these parameters influence the performance of DR, RC'. Each subplot varies q. while holding constant and changing x- We compare the results of PCA (representing IDR) and rCCA (representing SDR). Each curve averaged over 10 trials, with crror bars indicating standard deviation around the mean, using algorithmic parameters as described in Appendix A.3
We see that the relative strength of signals, as represented by 7, plays a significant role in determining, which method performs better. If the shared signals are larger (bottom) both approaches work. However, for weak shared signals (top), SDR is generally more effective. Further, the ratio between the number of shared and self signals, m, also plays an important role. When mn is large (left), IDR is more likely to detect the shared signal before the self signals, and approaches the performance of SDR. However, when in small, IDR is more likely capture the self signals before moving on to the shared signals, degrading performance (right). Finally, not surprisingly, the number of samples per dimension of the compressed variables, is also critical

---

Published in Transactions on Machine Learning Research (09/2024)
Adding 30 self-signals, keeping 30 dimensions after reduction
CCA
iCCA
RC'
Noise
RC
1.3
A1
B1
C1
D1
E1
T= 300
0.8
0.7
0.8
144
0.3
1.2
0.2
-0.6
0.1
1.0
1.8
A2
B2
C2
D2
0.4
E2
0.8
T= 3000
0.7
0.8-
0.6
30.5
30.5-
0.2
0.4
0.3
0.2
0.2
0.0
0.0
18
02
02
Figure 4: DR performance for |xx |Zy| Myelf > mehared). PCA now detects shared signals even when they weaker than the self signals. However, the quality of reconstruction rucoist is significantly lower than in Fig. 2 PLS detects signals in larger part of the phase space, but also with significant reduction in quality, which improves with sampling. CCA has its usual problem for T & Nx, and, like PLS, it has significantly lower reconstruction quality than in the regime in Fig. 3 rCCA is able to detect the signal in the whole phase space, but again with worse quality. Finally, spurions correlations are high, though they decrease with better sampling.
Adding 30 self-signals, keeping 31 dimensions after reduction
PCA
PLS
CCA
rCCA
RC
Noise
RC
1.3
1.0
1.8
0.9
C1
D1
E1
T= 300
1.6
0.7-
0.88
0.5
0.8
1.44
1.2
0.2
0.6
0.1
0.9
1.3
A2
D2
0.4
E2
0.8
I= 3000
0.7
0.6
0.55
0.55
0.2
0.4
0.4
0.2
0.2
0.1
0.0
0.0
04
Figure 5: PCA, PLS, CCA, rCCA, and noise results when 31 dimensions are kept after reduction (|2x |Zy| Mself Mshared). PCA now can detect more shared signals when they are weaker than the self signals
(A1), however, with significantly lower quality compared to Fig. 2 but suddenly explores the whole phase space, still with lower accuracy than Case PLS, CCA, rCCA, and noise show similar behavior to figure
the success. q small, the signal drowned the sampling noise, and adding more retained dimensions hurts the DR process. This expresses itself a peak for SDR performance around |xxl = mehared- For IDR, the peak is around |xx| Mself + Mshared; thus requiring more data to achieve performance similar to SDR.
8

---

Published in Transactions on Machine Learning Research (09/2024)
1000 total dimensions, 10 shared dimensions, 1000 samples
Xx Mehared
Xxl Madt
12x1 = Moelf + Mebared
PCA(IDR)
CCA(SDR)
m
10.0, 9-0.1 0.1
m = 1.0, 5 = 0.1
m
0.1, 9-
A1
B1
C1
RC
0.2
0.2
0.2
0.01
0.0
0.0-
10
10
10
10
10
10
10
10
10
1/9
1/9
1/9
77
10.0,
5 1.0
nñ m== 1.0, 5
1.0
m-= 0.1, 5
1.0
1.0
A2
1.0-
B2
1.0-
C2
0.8
0.8-
0.8-
RC
0.6
0.6
RC
0.6-
0.4
0.4
0.4-
0.2
0.2
0.2
0.0
0.0
0.0
10
10
10
10"
10
10
10
10
10"
1/4
1/@
1/9
m-
10.0, 5 10.0
nñ
1.0,
7
10.0
0.1, 7 10.0
1.0
A3
1.0-
B3
1.0-
C3
RC'
RC'
0.2
0.2
0.2
0.0
0.0
0.0-
10
10
10
10
10
10
10"
10
10
1/
1/@
1/0
Figure 6: Performance of PCA (IDR) and rCCA (SDR) for different values of the relevant parameters of the model: the number of samples per dimension of the compressed variable (ø), the strength of shared signals relative to the self ones (5), and the ratio of the number of shared to self signal components (m), while fixing the number of samples (T = 1000) and the number of shared dimensions (Mahared = 10). Note that increasing 1/q (left to right) corresponds to increasing the dimension of the latent space [Zx| at fixed number of samples T.
These results can be understood from a simple counting argument. In SDR, the objective function is basod on the cross-covariance or the cross-correlation matrix. In our linear model, the cross-covariance Cxy x QxQ (up to sampling effects) should depend on the quenched projection matrices Qx and Qy. The number of parameters defining these quenched projection matrices is Mchared Nx and Mghared X Ny respectively. To be able to reconstruct the cross-covariance the correlation matrix from DR method, we need a number of observations that scales with this number of parameters. The data matrices X and Y have T x Nx and T x Ny entries respectively. Thus, we need T 2 Mishared samples to have a chance of having enough samples to capture the shared signal. In contrast, in IDR, the objective function is built from the variance matrix Cxx & o T oZ V + x which jointly depends on the quenched projection matrices Vx and Qx, having Meelf, Nx and mehared X Nx parameters, respectively. Therefore, to get enough samples reconstruct large correlation dimensions in this matrix, we would need at least
9

---

Published in Transactions on Machine Learning Research (09/2024)
T 2 Maelf,X + Mehared measurements. Furthermore, not all parts of the variance matrix Cxx are relevant to the shared signal. Thus, the relative strength of the shared signal to the sell signal and the number of self and shared signals will determine which part of the signal is captured first. Finally, adding unnecessary dimensions will hurt performance, as there will be fewer samples per dimension.
We observe that the performance of rCCA (SDR) almost independent of changing mn or 3, indicating that it focuses on shared dimensions even if the latter is masked by self signals. The algorithm crucially depends where adding more dimensions (decreasing q) than needed hurts the reduction. This is because, for fixed number of samples, the reconstruction of each dimension then gets worse. In contrast, for PCA (IDR), the performance depends on all three relevant paramcters, q. m, and 5. At some parameter combinations, the performance of IRR in reconstructing shared signals approaches SDR. However, in all cases, SDR never performs worse than IDR on this task.
4.2
Beyond The Linear Model: Noisy MNIST
4.2.1
The Dataset
2
3
4
5
2
3
4
5
Cor(X, X)
Cor(Y.1)
ConX. r
8000
Froquency
0000
4000
2000
-0.4
0.2
0.4
0.0
-0.4
0.4
Correlation
Correlation
Correlation
Figure 7: Dataset containing paired MNIST digit samples sharing only the same identity (shared signal). The first row (X) shows MNIST digits randomly subjected to scaling, (0.5 - 1.5), and rotation with an angle of (0- - 7/2), while the second row (Y) shows MNIST digits with an added background Perlin noise (self signals). In the bottom row, histograms of self correlations for the X and datasets (left and middle, respectively) illustrate wide range of correlations, while the histogram of the cross correlation between X and Y (right) demonstrates smaller range.
To analyze lincar DR methods nonlincar data, followed the same procedure in Fig. 0 for dataset inspired by the noisy MNIST dataset (LeCun et al., 1998 Wang et al. 2015 2016 Abdelaleem et al. 2023
This dataset has two distinct views of data, each of dimensionality 28 x 28 pixels, examples of which are shown in Fig. 7 The first view is an image of the digit subjected to a random rotation within an angle uniformly sampled between and I 2: along with scaling by a factor uniformly distributed between 0.5 and 1.5. The second view consists of another image with the same digit identity with an additional background layer of Perlin noise (Perlin, 1985), ; with the noise factor uniformly distributed between and 1. Both views are normalized to an intensity range of [0, 1), then flattened to form an array of 784 dimensions.
To cast this dataset into our language, we shuffled the images within labels, retaining the shared label identity (that is the shared signal), but we still have the view-specific details (which is the self signal). This resulted in a total dataset size of ~ 56k images for training and ~ 7 images for testing. The correlation histogram of X (or Y) with itself shows a relatively wide spectrum when compared to the cross correlation between X and Y. highlighting that the self signal is stronger, and can lead to different DR methods overscoing the shared one. The complexity of the tasks makes it sufficiently challenging, serving as a good benchmark for evaluating the performance of the different. DR techniques.
10

---

Published in Transactions on Machine Learning Research (09/2024)
4.2.2 Results
Total Corrected Correlation vs 1/q for different
PCA
PLS
CCA
.....
1OCA
T= 1000
T= 10000
T=
[Cant (2x Sel2 [Car(aantbaall
0.6-
1.5
0.2
0.0
0.0
10
10
10
10
10
10
1/7
1/9
1/9
Figure Performance of PCA, PLS, CCA, rCCA applied to the modified Noisy MNIST dataset across varying sampling scenarios. Each panel represents different sample sizes (1000, 10, 000, and approximately 56, 000 samples). The x-axis denotes the inverse of the number of samples per retained dimensions (1/q), while the y-axis represents the total corrected correlation between the obtained low-dimensional representa- tions Zx and Zy.
Figure Shsows the performance of PCA, PLS, CCA, and rCCA applied to the modified Noisy MNIST dataset for varying sampling scenarios. The three panels are evaluated for different sample sizes (1000, 10, 000, and ~ 56, 000 samples), from undersampled to the full dataset.
In each scenario, the training samples are used for the DR methods. Subsequently, the learned projection matrices onto the singular directions are used transform separate test dataset of around 7, 000 samples into low-dimensional spaces, yiclding Zx and Zy. The correlation between these transformed spaces is computed using the Frobenius norm of the correlation matrix. As before, then subtracted from it the correlation value obtained from random matrix of the same size. This difference is then plotted against 1/Ø, which is the measure of how many dimensions are retained at each sampling ratio.
In the undersampled scenario (1000 samples), rCCA and PLS demonstrate an early delection (in terms of the number of kept dimensions after reduction) of shared signals, whereas PCA initially lags behind. As the number of dimensions increases, all methods exhibit a decline in correlation due to increased noise have fewer samples per dimension. CCA does not work in this scenario, since covariance matrices are degenerate.
Upon increasing the sample size (10, 000 samples), a similar pattern emerges initially, where all methods experience an increase in total correlation till a certain number of kept dinensions is reached, then a decline when adding more dimensions. The decline is because one needs to estimate more singular vectors from the same number of samples. However, beyond a ocrtain number of singular vectors, an increase correlation is observed. This is because the number of vectors is now sufficient learn both the shared and the self signals. We observe that rCCA maintains superior performance, while PCA reaches peak correlation higher number of kept dimensions, providing rough estimation of the number of true self and shared signals. With the full dataset (approximately 56,000 samples), a similar trend is seen. Yet CCA's performance approaches that of rCCA.
Notably, these analyses confirm that linear Simultancous Dimensionality Roduction (SDR) are consistently better at detecting shared signals than linear Independent Dimensionality Reduction (IDR), even in some nonlinear datasets.
11

---

Published in Transactions on Machine Learning Research (09/2024)
5 Discussion
We used generative linear model which captures multiple desired features of multimodal data with shared and non-shared signals. The model focused only on data with two measured modalities. However, while not a part of this study, the model can be readily extended to accommodate more than two modalities (e. g X, = Rg + U.V, + PQ; for i= 1,.... n, where n represents the number of modalities). Then, methods such as Tensor CCA, which can handle more than two modalities (Luo et al. 2015), can be used to get insight into DR on such data.
We analyzed different DR methods on data from this model in different parameter regimes. Linear SDR methods were clearly superior to their IDR counterparts for detecting shared signals. In particular, SDR performance peaked around |xx| = Mshared and IDR performance peaked around |7xl = Mself + Mshared- Thus, SDR required fewer samples for similar detection performance to WR. We observed similar results on a nonlinear dataset as well. We thus make a strong practical suggestion that, whenever the goal is to reconstruct& a low dimensional representation of covariation between two components of the data, IDR methods (PCA) should always be avoided in favor of SDR. Of the examined SDR approaches, rCCA is a clear winner in all parameter regimes and should always be preferred.
While we performed the analysis using specific examples of SDR, such as PLS, (r)CCA, and specific examples of IDR, like PCA, we anticipate similar results to hold for other SDR and IDR methods. For instance, meth- ods that optimize certain aspect within one modality (e. g., Independent Component Analysis Hyvärinen & Oja (0000), Nonnegative Matrix Factorization Lee & Seung (2000), Autoencoders Hinton & Salakhutdinov (2006)), ) or methods using multiple modalities but retaining only certain aspects within each modality (e. g., Multiview PCA Xia et al. (2021)) should be able to detect the shared signals only if they are stronger than the self signals, given the other detection criteria (of having enough number of kept dimensions and enough samples to sample them properly) are met. Otherwise, they will likely detect the self signals first, wasting some of their statistical power. Alternatively, methods that work across modalities (e. g., Cross-mode modal Fac- tor Analysis Li et al. (2003), Deep Variational Symmetric Information Bottleneck Abdelaleem et al. (2023) ) should identify the shared signal first. We leave the verification of this hypothesis across a broader range of methods for future work. To analyze the behavior of such more complicated methods, we will need gener- ative models of data, that have low-dimensional structure, which cannot, nonetheless, be approximated by methods that rely on singular value spectra and their nonlinear generalizations; first steps towards creating such generative models were recently taken Fleig& & Nemenman (2022)).
These findings explain the results of, for example, [Sani et al. (2021) h and others that SDR can recover joint neuro-behavioral latent spaces with fewer latent dimensions and using fewer samples than IDR methods. Further, our obscrvation that SDR is always superior to IDR in the context of our model corroborates the theoretical findings of Martini & Nemenman (2024),, who proved a similar result in the context of discrete data and different SDR algorithm, namely the Symmetric Information Bottleneck (Friedman et al., 2013). Vogelstein al. (2021) made similar conclusions using conditional covariance matrices for the reduction in the context of classification. More recent work of Abdelaleemm al. 2023) showed similar results using deep variational methods. Collectively, these diverse investigations, linear and nonlinear, theoretical, computational, and empirical, provide strong evidence that generic (not just linear) SDR methods are likely to be more efficient in extracting covariation than their IDR analogs.
Our study also answers an open question in the literature surrounding the effectiveness of SDR techniques. Specifically, there has been debate about whether PLS, an SDR method, effective at low sampling (Chin & Newsted 1999 Hair et al. 2011 Goodhue et al.] 2006 2012). Our results show that SDR is not necessarily effective in the undersampled regime. It works well when the number of samples per retained dimension is high (even if the number of samples per observed dimension is low), but only when the dimensionality of the reduced description is matched to the actual dimensionality of the shared signals.
Finally, our results can be used as a diagnostic test to determine the number of shared versus self signals in data. As demonstrated in Fig. 0 total correlations between Zx and Zy obtained by applying PCA and rCCA increase monotonically as the dimensionality of Zs increases, until this dimensionality becomes larger than the signal dimensionality. For PCA, the signal dimensionality is equal to the sum of the number of
12

---

Published in Transactions on Machine Learning Research (09/2024)
the shared and the self signals, mihaared Maelf. For rCCA, it is only the mumber of the shared signal. Thus increasing the dimensionality of the compressed variables and tracking the performance of rCCA and PCA until they diverge can be used to identify the number of self signals in the data, provided that the data, indeed, has a low-dimensional latent structure. This approach can be a valuable tool in various applications, where the characterization of shared and self signals in complex systems can provide insights into their structure and function.
In summary, highlight general principle that, when searching for shared signal between different modalities of data, SDR methods preferable to IDR methods. Additionally, the differences in performance between the two classes of methods can tell us a lot about the underlying structure of the data. Finally, for a limited number of samples, naive approaches, such as increasing the number of compressed dimensions indefinitely to overcome the masking of shared signals by self signals are infeasible. Thus, the use of SDR methods becomes even more essential in such cases.
6 Limitations and Future Work
While this work has provided useful insight, the assumptions made here may not fully capture the complexity of real-world data. Specifically, our data is generated by linear model with random Gaussian features. It is unlikely that real data have this exact structure. Therefore, there is a need for further exploration of the advantages and limitations of linear DR methods on data that have a low-dimensional, but nonlinear shared structure. This can be done using more complex nonlincar generative models, such as nonlincarly transforming the data generated by Eq. 112,, or random feature two-layered neural network models (Rocks & Mehta 2022). Alternatively, analyzing the model, Eq. (T) using various theoretical techniques (Borga eet al. 1997 Vogelstein et al. 2021 Potters & Bouchaud (2200) is likely to offer even more insights into its properties. Collectively, these diverse approaches would aid our understanding of different DR methods under diverse conditions.
A different possible future research direction is to explore the performance of nonlinear DR methods on data from generative models with a latent low-dimensional nonlinear structure. Autoencoders and their variational extensions are a natural extension of TDR to learn nonlinear reduced dimensional representations (Hinton & Salakhutdinov, 2006; Kingma & Welling 2014 Higgins et al. 2016). Meanwhile, Deep CCA and its variational extensions (Andrew et al. 20133 Wang al. 2015 Chandar al.] 2016; Wang al.) 2016) potentially capture more complex relationships between the modalities and improve the quality of the reduced representations,
should be explored nonlinear version of SDR. Both of these types of methods
and while recent work suggests that (Abdelaleem et al. 2023), it is not clear if the SDR class of methods is always more efficient than the IDR one.
Further, our analysis depends on the choice of metric used to quantify the performance of DR, and different choices should also be explored. For example, to capture nonlinear correlations, mutual information can be utilized to quantify the relationships between the reduced representations.
Despite the aforementioned limitations, we believe that our work provides compelling addition to the body of knowledge that SDR outperforms IDR in detecting shared signals quite generally.
Acknowledgments
We thank Sean Ridout for providing feedback on the manuscript, IN thanks Philipp Fleig for useful dis- cussions. This work was funded, in part, by NSF Grant No. 2010524 and 2014173 and by the Simons Investigator award to IN.
References
Eslam Abdelaleem, Nya Nemeemann, and K Michael Martini. Deep variational multivariate information bottleneck- a framework for variational losses. arXiv preprint arXio:2310.0333 2023.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine
13

---

Published in Transactions on Machine Learning Research (09/2024)
Learning, volume 28 of Proceedings of Machine Learning Reseorch, pp. 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMT.R
András Antos and Toannis Kontoyiannis. Convergence properties of functional estimates for discrete distri- butions. Random Structures & Algorithms, 19(3-4):163 193, 2001.
Francis R Bach and Michael I Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report 688, Department of Statistics, University of California, Berkeley, 2005.
Jinho Baik, Gerard Ben Arous, and Sandrine Peche. Phase transition of the largest cigenvalue for non-mall complex sample covariance matrices. Xiv preprint math/0403022, 2004.
Richard Baillic, G Geoffrey Booth, Yiuman Tsc, and Tatyana Zabotina. Price discovery and common factor models. Journal of financial markels, 5(3):309 321, 2002.
William Bialek, Ilya Nemenman, and Naftali Tishby. Predictability, complexity, and learning. Newral computation, 13(11):2409-2444 2463, 2001.
Magnus Borga, Tomas Landelius, and Hans Knutsson. unified approach pea, pls, mlr and coa. Linköping University, Department of Electrical Engineering, 1997.
Jean Philippe Bouchaud, Laurent Laloux, M Augusta Miceli, and Mare Potters. Large dimension forecasting
models and random singular value spectra. The European Physical Journal B, 55:201 207, 2007.
Joël Bun, Jean-Philippe Bouchaud, and Mare Potters. Cleaning large correlation matrices: Tools from
random matrix theory. Physics Reports, 666:1 109, 2017. ISSN 0370-1573. doi: https://do (ooi..og/11.10011111 2016.10.005. Cleaning large correlation matrices: tools from random matrix theory.
Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational neural networks. Newral Compulation, 28(2):257-285, 2016. doi: 10.1162/NECO 00801.
James Chapman and Hao-Ting Wang. Cca-zoo: A collection of regularized, deep learning based, kernel, and probabilistic cca methods in a scikit-learn style framework. Journal of Open Source Software, 6(68))38823, 2021.
Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du, and Hod Lipson. Au- tomated discovery of fundamental variables hidden in experimental data. Nature Computational Science, 2(7):433-444) 442, 2022.
Wynne Chin and P. Newsted. Structural equation modeling analysis with small samples using partial least square. Statistical Strategies for Small Sample Research, 01 1999.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (tcia): maintaining and operating public information repository. Journal of digital imaging, 26:1045 1111045-155,, 2013.
Lucy J Colwell, Yu Qin, Miriam Huntley, Alexander Manta, and Michael P Brenner. Feynman-hellie theorem and signal identification from sample covariance matrices. Physical Review X, 433)003,03,, 2014.
Felix Creutzig, Amir Globerson, and Naftali Tishby. Past-future information bottleneck in dynamical sys- tems. Physical Review E, 79(4):041925, 2009.
Philipp Fleig and 1lya Nemenman. Statistical properties large data sets with linear latent features. Physical Review E, 106(1):014100) 2022a.
Philipp Fleig and Ilya Nemenman. Generative probabilistic matrix model of data with different low- dimensional linear latent structures. arXiv preprint arXiu:2212.02987, 2022b.
Mario Forni and Luca Gambetti. The dynamic effects of monetary policy: A structural factor model ap- proach. Journal of Monelary Economics, 57(2):203 216, 2010.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Simon Freyaldenhoven. Factor models with local cettors-ddterminiigg the number of relevant factors. Journal of Bconometrics, 229(1 (0001)))8002222 2022.
Nir Friedman, Ori Mosenzon, Noam Slonim, and Naftali Tishby. Multivariate information bottleneck. aiv preprint arX1e:1301.2270,, 2013.
Dale L Goodhue, William Lewis, and Ron Thompson. Pls, small sample size, and statistical power in mis research. In Proceddings of the 39th Annual Howaii International Conference on System Sciences (HICSS'06), volume 8, pp. 202b-202b, 2006. doi: (.009/HICSS2000
Dale Goodhue, William Lewis, and Ron Thompson. Does pls have advantages for small sample size or non-normal data? MIS Quarterly, 36(3):981 1001, 2012. ISSN 02767783.
Dale L Goodhue, Ron Thompson, and William Lewis. Why you shouldn't use pls: Four reasons to be uneasy about using pls in analyzing path models. In 2013 46th Hawaii International Conference on System Sciences, pp. 4739 4748. IEEE, 2013.
Marc-André Gosselin and Greg Tkacz. Evaluating factor models: An application to forecasting inflation in canada. Technical report, Bank of Canada, 2001.
Joe F Hair, Christian M Ringle, and Marko Sarstedt. Pla-sem: Indeed a silver bullet. Journal of Markeling Theory and Practice, 19(2):139-185 152,, 2011. doi: (2773//MTP1003-667111000 90202.
Joe Hair Jr, Joseph F Hair Jr, G Tomas M Hult, Christian M Ringle, and Marko Sarstedt. A primer partial least squares structural equation modeling (PLS-SEM). Sage publications, 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2016.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(6)6)):504 507, 2006. doi:
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Harold Hotelling, Relations between two sets of variates. Biometrika, 1936. doi: 10.00779978-11-6124488844
14.
Rachael Huntley, Tony Sawford, Prudence Mutowo-Meullenet, Aleksandra Shypitsyna, Carlos Bonilla, Maria Martin, and Claire O'Donovan. The goa database: gene ontology annotation updates for 2015. Nucleic acids research, 43(D1):D1057 111111007-111,,,,, 2015.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Newral networks, 13(4-)):411 1100)30,,,11111- 2000.
Diederik P Kingma and Max Welling. auto-Encoding Variational Bayes. In 2nd International Confer- ence on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Arto Klami, Scppo Virtanen, and Samuel Kaski. Bayesian exponential family projections for coupled data sources. arXie preprint 1rx3::2003.3489 2012.
Ned Kock and Pierre Hadaya. Minimum sample size estimation in pls-sem: The inverse square root and gamma--xponential methods. Information systems journal, 28(1):2277205 2211 2018.
John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin, Malcolm A MacIver, and David Poeppel. Neuro- science needs behavior: correcting reductionist bias. Neuron, 93(3):480-499 2017.
Yann LeCun, Lćon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied docu- ment. recognition. Proceedings of the IEFE, 86(11):221 2324, 1998.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Daniel Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000.
Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K Sethi. Multimedia content processing through cross-modal association. In Proceedings of the eleventh ACM international conference on Multimedia, pp. 604-611, 2003.
Marco Lorenzi, Andre Altmann, Boris Gutman, Selina Wray, Charles Arber, Derrek P Hibar, Neda Jahan- shad, Jonathan M Schott, Daniel C Alexander, Paul M Thompson, et al. Susceptibility of brain atrophy to trib3 in alzheimer's disease, evidence from functional prioritization in imaging genetics. Proceedings of 3162233112216 2018.
the National Academy of Sciences,
Yong Luo, Dacheng Tao, Kotagiri Ramamohanarao, Chao Xu, and Yonggang Wen. Tensor canonical correla- tion analysis for multi-view dimension reduction. IEEE transactions on Knowledge and Dato Engineering. 27(11):111-3114, 2015.
Vladimir Alexandrovich Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets random matrices. Matematicheskii Sbornák, 114(4))507 536, 1967.
K Michael Martini and Tlya Nemenman. Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck. Newral Computation, 36(7):1353-137) 1379,, 2024.
Kevin P Murphy. Probabilislic machine learning: on introduction. MIT press, 2022.
Nikhilesh Natraj, Daniel Silversmith, Edward Chang, and Karunesh Ganguly. Compartmentalized dy- namics within common multi-area mesoscale le manifold represent repertoire of human hand movements. Newron, 110(1):154-174..2 2022. ISSN 0896-6273. doi: https://doi.org/ 1a.rr///11111///111 nttp:/.jnuoo..222222222 wwt..2221..00000
Eakasit Pacharawongsakda and Thanaruk Theeramunkong. A comparative study on single and dual space reduction in multi-label classification. In Andrzej M.J. Skulimowski and Jamusz Kacprzyk (eds.), Knowl- edge, Information and Crealivity Support Syslems: Recend Trends, Advances and Solutions, pp. 389 400, Cham, 2016. Springer International Publishing. ISBN 978-3-319-19090-7.
Rich Pang, Benjamin J Lansdell, and Adrienne L Fairhall. Dimensionality reduction in neuroscience. Current Biology, 26(14):6656-660, 2016.
Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15())11191-1253, 2003.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Ken Perlin. An image synthesizer. ACM Siggraph Computer Graphics, (9(3):287-996 1985.
Mare Potters and Jean-Philippe Bouchaud. A First Course in Random Matria Theory: For Physicists, Engineers and Data Scientists. Cambridge University Pross, 2020.
Jason W Rocks and Pankaj Mchta. Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models. Physical review research, 4(1):013201, 2022.
Jeremy B Rudd. Underlying inflation: Its measurement and significance. FEDS Notes. Washington: Boord of Governors of the Federal Reserve System, September 18, 2020, 2020.
Omid G Sani, Hamidreza Abbaspourazad, Yan T Wong, Bijan Pesaran, and Maryam M Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 24(1):140-149, 2021.
Claude Elwood Shannon. A mathematical theory of communication Bell Syst. Techn. J., (7: 423, 1948.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Simon Sponberg, Thomas L Daniel, and Adrienne L Fairhall. Dual dimensionality reduction reveals inde- pendent encoding of motor features in a muscle synergy for insect flight control. PLOS Compulational Biology, 11(4): 23, 04 2015. doi:
10041..
Nicholas A Steinmetz, Cagatay Aydin, Anna Lebedeva, Michael Okun, Marius Pachitariu, Marius Bauza, Maxime Beau, Jai Bhagat, Claudia Böhm, Martijn Broux, et al. Neuropixels 2.0: A miniaturized high- density probe for stable, long-term brain recordings. Science, 372(6539), 2021.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Ken- neth D. Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437); caav7893, 2019. doi: 10.1126/sciencc.aav7888.
Valentine Svensson, Roser Vento-Tormo, and Sarah A Teichmann. Exponential scaling of single-cell rna-seq in the past decade. Nature protocols, 13(4):599-604, 2018.
Anne Urai, Brent Doiron, Andrew M Leifor, and Anne K Churchland. Large-scale neural recordings call for new insights to link brain and behavior. Nature neuroscience, 25(1):11-11) 19,1119,,-1 2022.
Hrishikesh D Vinod. Canonical ridge and econometrics of joint production. Journal of Econometrics, 1976.
doi: 10. 10.1016/03044407700 (0304-4076677)99000---
Joshua T Vogelstein, Eric W Bridgeford, Minh Tang, Da Zheng, Christopher Douville, Randal Burns, and Mauro Maggioni. Supervisod dimensionality reduction for big data. Nature communications, 12(1):2872, 2021.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pp. 1083-1092. PMLR, 2015.
Weiran Wang, Xinchen Yan2 Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXie preprint 1rXi::6600.3344, 2016.
Svante Wold, Michael Sjöström, and Lennart Friksson. Pls-regression: basic tool of chemometrics.
Chemometrics and Intelligent Laboratory Systems, 58(2):109 130, 2001. ISSN 0169-7439. doi: https: //doi.org/10.100 10.1016//0169- 7439 PLS Methods.
Zhiming Xia, Yang Chen, and Chen Xu. Multiview pea: A methodology of feature extraction and dimension reduction for high-order data. IEEE Transactions on Cybernetics, 52(10):11068 11008-1110000,,, 2021.
Grace XY Zheng, Jessica M Terry, Phillip Belgrader, Paul Ryvkin, Zachary W Bent, Ryan Wilson, Solongo B Ziraldo, Tobias D Wheeler, Geoff P McDermott, Junjie Zhu, et al. Massively parallel digital transcriptional profiling of single cells. Nature communications, 8(1):14049, 2017.
Finn Arup Nielsen, Lars Kai Hansen, and Stephen Strother. Canonical ridge analysis with ridge parameter optimization. Neurolmage, 1998. doi: 10.1016/110033881119133311131111
A Appendix
A.1 Linear Dimensionality Reduction Methods
A.1.1 Principal Components Analysis (PCA)
PCA is widely used linear IDR method that aims to find the orthogonal principal directions, such that a few of them explain the largest possible fraction of the variance within the data. PCA decomposes the covariance matrix of the data matrix X, Cxx = x. X, into its cigenvectors and cigenvalues through singular value decomposition (SVD). The SVD yields orthogonal directions, represented by the vectors w that capture the most significant variability the data. In most numerical implementations (Pedregosa) et al. 2011) these directions are obtained consecutively, one by one, such that the dot product between
17

---

(8) are then stacked together to form the projection matrix Wx. The low-dimensional reprosentation Zx is then obtained by multiplying the original data matrix X with this projection matrix, Published in Transactions on Machine Learning Research (09/2024)
any two directions is zero wxx wxx 8iz- The eigenvectors wx are obtained the best solution to the optimization problem:
( T X(1
x
(t) w x = arg max
(6)
( w
w x
Here X is the ith deflated matrix where (1) is the original matrix, and for every subsequent + 1, the matrix is deflated by subtracting the projection of X on the obtained weights: X = X - X (())) The eigenvectors are sorted in decreasing order according to their corresponding eigenvalues, and the first k eigenvectors "' (im1:k) are selected to form the projection matrix Wx- The obtained vectors determine the size of the reduced form Zx, where |2x| = the number of vectors retained from the decomposition of X. The vectors _
resulting in the reduced data matrix Zx= = XWx. Similar treatment. is done for Y in order to obtain Sy=YWr
One of the main advantages of PCA is its simplicity and efficiency. However, one of the drawbacks of this method is that it performs DR for X and Y independently, and one then searches for relations bet Zx and Zy by regressing one on the other-the so-called Principal Components Regression. Thus obtained low-dimensional descriptions may capture variance but not the covariance between the two datasets.
A.1.2 Partial Least Squares (PLS)
PLS performs SDR by finding the shared signals that explain the maximum covariance between two sets of data (Wold et al. 2001 PLS performs the SVD of the covariance matrix Cxy X Y (or equivalently Cyx +Y X). The left and right singular vectors (wxx) =(f) my mi()) are obtained consecutively pair by pair ( such that x wy 8ig. They solutions of the optimization problem:
(0) T
Y w
(0) , () = arg max- (
Wx
(7)
(w w w
(( ()))
The matrices x(),y* (+) are deflated in similar manner to PCA A.1.1 The singular vectors are sorted in the decreasing order of their corresponding singular values, and the first k vectors are selected to form the projection matrices (Wx, Wy). The obtained vectors determine the size of the reduced form (Zx, Zy), where |xx| = |y| = k is the number of vectors retained. The vectors (w wf?") are then stacked together to form the projection matrices (Wx, x, Wy) respectively. The low-dinensional representations (Zx, Zy) are obtained by projecting the original data matrices (X, Y) onto these projection matrices: Zx XWX, and
Zy = YWr.
In summary, PLS performs simultaneous reduction on both datasets, maximizing the covariance between the reduced representations Zx and Zy. This property makes PLS a powerful tool for studying the relationships between two datasets and identifying the underlying factors that explain their joint variability. In practice, PLS can suffer from widely distinct variances along different singular directions, as well as covariances within X or Y. At the very least, in practical implementations, each component. of X and Y is typically standardized to unit variance before applying PLS to avoid some (but not all) of these issues.
A.1.3 Canonical Correlations Analysis (CCA)
CCA is another SDR method, which aims to find the directions that explain the maximum correlation between two datasets Hotelling (1366). However, unlike PLS, CCA obtains the shared signals by performing
SVD on the correlation matrix CxY
VCxxVCrr" The singular vectors (wxx) wy wx()) are obtained consecutively
=(1)
Thhis PLS Implementation with iterative deflation often referred PLSCanonical An alternttve simpler approach based on direct. SVD calculation the covariance matrix often known as PLSSVD Pedregosa al. 20111
18

---

Published in Transactions on Machine Learning Research (09/2024)
pair by pair such that wx "y 6z. CCA enforces the orthogonality of (0) my w
(6)
( independently as well,
such that "xx " wy: wy
w w w)
= 8ig. The singular vectors are obtained by solving the optimization
problem:
W ( T X T
yu (
(wx² "y ) arg max.
(()
(8)
wx a
V(ulg) X X x (u) YO (OTT(ufy)
(
Like in PLS A.1.2 the matrices (0),, ( are deflated in a similar manner. In addition, the first & singular vectors (xxx wy )) are stacked together to form the projection matrices (Wx, Wy), which then are used to obtain the reduced data matricos Zx= = XWx, and Zy- YWy.
One of the key differences between PLS and CCA is that while both perform SDR, CCA also simultaneously performs IRR implicitly. Indeed, it involves multiplication of Cxy by Cxx on the left and Cyy on the right, which, in turn, requires finding singular values of the X and the data matrices independently.
A.1.4 Regularized CCA rCCA
While CCA is a useful method for finding the maximum correlating features between two sets of data, it does have some limitations. Specifically, in the undersampled regime, where T S max(Nx,,Ny) the matrices Cxx and Cyy are singular and their inverses do not exist. Using the pseudoinverse to solve the problem can lead numerical instability and sensitivity to noise. Regularized CCA (rCCA) (Vinod) 1976;
Arup Niclsen et al. 1998) overcomes this problem by adding a small regularization term to the covariance matrices, allowing them to be invertible. Specifically, one tales
Čxx = Cxx +exxx,
(9)
čyy= + +cyly,
(10)
where Cxx, Cvy are the new regularized matrices, cx,ey > 0 are small regularization parameters and Ix, Ty are identity matrices with sizes Nx x Nx,Ny x Ny respectively.
This original implementation of rCCA resulted in correlation matrices with diagonals not equal to one. Thus, a better implementation uses different form of regularization (Arup Nielsen et al., 1998) by adding the regularization parameters cx and cy individually to the equations affine combination (i. c., Σ, c = 1) as the following:
1
Õxx = - wxx Xwx + ex, wXwx),
(11)
1
Ōrr = 7 (1r, W T + or, wp)))
(12)
This results in the regularized equations for X and Y:
Čxx ((1 -
(11-cx)w cx) X Xwx + ex Örr = ((1 - T Yuy + ey wx))
(13)
(14)
where cx and cy are the regularization parameters, with values between 0 and 1. Hence our new optimization problem becomes:
( xx x(O (roTytee y(
=(8) (wxxx my x1) arg max- Vixxčrr
(15)
Writing the regularization conditions in this form is, in fact, a convex interpolation problem between PLS and CCA, which is a more robust solution and does not suffer from shortening the length of correlations due to the added regularization. As a result, this implementation of rCCA achieves the best accuracy among all other methods
"Conceptually, the main difference between PLS and CCA is that PLS does not enforce orthogonality among the weights w and ul() that diagonalixe Cxx and Cyy, whereas CCA does. For instance, while two pairs of singular vectors (w w
( and
19

---

Published in Transactions on Machine Learning Research (09/2024)
A.2 Assessing Success and Sampling Noise Treatment
To assess the success of DR, we calculated the ratio between the total correlation between X and Zy defined as in Eq. ( and the total correlation between X and Y, which we input into the model. Specifically, where
we take the total correlation as the Frobenius norm of the correlation matrix, ||A|| = (AA) are the singular values of the matrix A. Therefore, the metric of the quality of the DR is
RC ||Corr(Zx,nn BrealllF ||Corr(Zx, Zrrea lll
(16)
I|Corr(P, P)l[F
Mshared
where Corr stands for the correlation matrix between its arguments, and we use ||Corr(P, P)ll = Mahared as the total shared correlation that one needs to recover. Statistical fluctuations aside, RC should vary between zero (bad reconstruction of the shared variables) and one (perfect reconstruction
3.0
1
[ 2
255
1/ 30
12 1 31
2.0
RCp
1.5
1.0-
0.5
0.0
10a
10
T
Figure 9: The resulting correlations are averages of all the points in the phasespace, then averaged over 10 different realizations of the matrices. The error bars are for two standard deviations around the mean
In many real-world applications, the number of available samples, T, is often limited compared to the dimensionality of the data, Nx and Ny. This undersampling can introduce spurious correlations. We are not aware of analytical results to calculate the effects of the sampling noise on estimating singular values
(w(²), ) are mutually orthogonal as pairs, ( and w(( are not orthogonal to each other. This partial overlap can result in signals not being fully expressed in these directions, reducing the total correlation compared CCA, where each direction " and w²" is orthogonal to every other direction, allowing each signal to fully occupy these directions and maximizing total correlation. Therefore, our goal is to make CCA work, but due to inversion issues, rCCA serves as a substitute. This is evident in Figures 1-5, where, under good sampling conditions, rCCA and CCA yield almost identical results.
We note that the choice of RC is not unique, and other choices could be valid as well. However, we believe that RC is suitable for our study. For example, one could use mutual information x(xx; Zy) [Shanoon 1948) to quantify how much information Zx and Zy carry about each other. Mutual information is fundamental statistic that is 0 if and only if the two variables are statistically independent. However, its estimation is challenging problem, as it depends on the probabilities p( xZX)), p(Z)); and pl (x Zy) Antos & Kontoyiannis (200T): Paninski] (2003). In this setup, while the individual matrices of X and Y are Gaussian with known probability deusity functions, their product is not. Consequently, closed analytical form of mutuua information
known. Even if assumed Zx and Zy be Gaussian and used the formula I(Zx; - ln|Cx,y where l.I denotes the determinant of the original correlation matrix Cxy Exy/VUXx- Cyy, we might end up with directions that do not carry information due to having more dimensions in Zx (or Zy) than needed, thus creating numerical instabilities. These instabilities are often addressed by employing threshold on the determinant (considering the determinant as the product of singular values, we threshold the small singular values). However, we chose not to such method, focusing instead RC for the following reasons: (i) With this choice, we do not need to employ thresholding, and by using metric that uses the singular values of the correlation matrix in additive fashion, rather than multiplicative as in mutual informstion, we avoid this issue. (ii) Correlation is intuitive metric, commonly used by practitioners, with clear meaning and bounds between -1 and +1. While one could the covariance matrix rather than the correlation matrix, such choice does not affect the optimization process, as X aud Y are standardized. Therefore, the covariance and correlation matrices of the original data are equivalent, and the difference would of PTS in such case. However, due the intuitiveness and utility of correlation, we use it in the RC metric.
the calculation of (on(Zx, Zy) vs Corr(Zx, Zy), potentially increasing the accuracy
20

---

Published in Transactions on Machine Learning Research (09/2024)
in the model in Eq. Bun al. 2017.. Thus, to estimate the effect of the sampling noise, adopt empirical approach. Specifically, we generate two random matrices, 7x iaudom and Zx yandom? of sizes T x xxx and T |, respectively. We then calculate the correlation between these matrices, denoted as RCo, for multiple such trials using the metric in Eq. (16). For random Zx.andom and Zy RC should be zcro. However, Fig- shows lhat, especially for large dimensionalities of the compressed variables and small T, the sampling noise results in a significant spurious RC, O, which may even be larger than Crucially, RCo does not fluctuate around its mean across trials, that the sampling bias is narrowly distributed.
To compensate for this sampling bias, subtract it from the reconstruction quality metric,
RC' RC RCo-
(17)
It is this RC' that plot in all Figures in this paper the ultimate metric of the reconstruction quality. While subtracting the bias is not the most rigorous mathematically, it provides practical approach for reducing the effects of the sampling noise.
A.3 Implementation
We used Python and the scikit-iieeee learn Pedregosa et al. 2011 library for performing PCA, PLS, and CCA, while the cca-zoo Chappaan & Wang 2021 library was used for rCCA. For PCA, SVD was performed with default parameters. For PLS, the PLS Canonical method was used with the NIPALS algorithm. For both PLS and CCA, the tolerance was set to 10-4 with a maximum convergence limit. of 5000 iterations. For rCCA, regularization parameters were set as C1 C2 = 0.1. All other parameters not explicitly here were set their default values.
All figures shown in this paper were averaged over 10 independent realizations of Rx, Ry, Ux, Uy P, while fixing the projection matrices Vx, Vy, Qx Qy We then performed an additional round of averaging every- thing over 10 realizations of the projection matrices themselves. The simulations were parallelized and run on Amazon Web Services (AWS) servers of various instance types.
A.4 Extended Figures
In this section, provide results of simulations similar to the main text Figs. range of T.
3
5
but with wider
21

---

Published in Transactions on Machine Learning Research (09/2024)
Adding self-signal, keeping dimension after reduction
PCA
PLS
CCA
rOCA
RC
Noise
RCa
0,200
1996
0.150
500
0.6
09-LAS LS-LS
DS
-0.100
1000
105-
03
00 A4
144
D4
-02
E4
T- 3000
105-
0.025
Figure 10: Performance of PCA, PI.S, CCA, and rCCA detecting shared signals with one self signal and one dimension kept after DR.
Adding self-signal, keeping two dimensions after reduction
PLS
rOCA
RRC"
Noise
RC,
DI
0,200
67-
044
61-
0.150
300
60
66
0.6
09
-0.100
T- (000
03
64-A4
402
41.000
T- 50000
86
03
Figure 11: Performance of PCA, PLS, CCA, and rCCA in detecting shared signals with self signal and two dimensions kept after DR.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Adding 30 self-signals, keeping dimension after reduction
PCA
PLS
CCA
rCCA
RC
Naise
R2a
0..5
A2
D2
822
3000
0..5
-15
1000
1.0
84
14
R4
0.2
3000
-0.5
0.0
Figure 12: Performance of PCA, PLS, CCA, and rCCA in detecting shared signals with 30 self signals and one dimension kept after DR.
Adding 30 self-signals, keeping 30 dimensions after reduction
PCA.A
PLS
CCA
rCCA
RCC"
Noise
RCC
0.5
A2
C2
D2
22
100
0.6
1155
1000
1.0
14
0.2 E4
3000
-0.5
0.0
Figure 13: Performance of PCA, PLS, CCA, and rCCA in detecting shared signals among 30 self signals and with 30 dimensions kept after DR.
23

---

Published in Transactions on Machine Learning Research (09/2024)
Adding self-signals, keeping 31 dimensions after reduction
RZ"
[RCy
1.0
DI
E1
3.0
000
2.55
3000
2.0
0.6
131
13
1000
-.4
T-
E4
T- 3000
0.9
Figure 14: Performance of PCA, PLS, CCA, and rCCA in detecting shared signals among 30 self signals and with 31 dimensions kept after DR.
24

---

Published in Transactions on Machine Learning Research (09/2024)
A.5 Additional Figures
Here we provide additional figures that show the singular value spectra of the matrices Cxx and Cxy (Eq. The original X and Y matrices are generated using the same parameters as in Fig, 6
1000 total dimensions, 10 shared dimensions, 100 samples
Mett
Matt + Mahazed
Cxx
Cxv
77
10.0, 5= 0.1
m
1.0, 5=0.- 0.1
m 0.1, 0-155=0.
1600
B1
50
C1
singular value
singular value
120-
singular value
40-
100-
30-
200
150-
80-
20
60-
100
10
50
20-
-10
10
10
10
10
10
10
10
10
10"
m-
10.0, 1.0
1.0,
f= 1.0
m 5- 1.0
160
A2
B2
C2
140-
100-
40-
singulur value
120-
singular value
singular value
100-
80
50-
80
20-
60-
40
10-
10
20-
20
0-
-10
10
10
10"
10
10'
10'
10
10'
10"
10.0,
4= 10.0
1.0, 5 10.0
m8
0.1, 5
10.0
A3
160-
B3
C3
140-
singular value
150-
singular value
120-
singular value
80
125
100-
60-
100-
80-
75-
60-
40
50
20
25
10"
10
10
10"
Figure 15: Singular values obtained from Cxx and Cxy (Eq, 4 for the parameters used Fig. Using two generated data matrices X and Y with Nx Ny = 1000, Miharedd = 10 while we change m to get the corresponding m, and "selr corresponding Cxx and CxY and plot their singular values versus their order for 100 samples. We observe similar behavior to Fig, 6 in the sense that if the shared signal is strong (Panels A3, B3, C3, where 10), then we see 10 distinct singular values that correspond to the shared signal for all the different values of m. This is a regime where the results of applying IDR and SDR are equivalent (we even see the first 10 singular values are on top of each other). In Panel A1, we see one singular value of Cxxx standing out, followed by another 10 values of the shared signals, while Cxy only identifies those 10 shared signals first. Panel A2 shows similar behavior, but the single self signal is now mixed among the others for Cxx. In Panel 12, while we see 10 distinct singular values of Cxx appearing first, these correspond to the self signals, and the next 10 correspond to the shared signals, meaning that if we stopped in an IDR application after retaining 10 dimensions, we would not capture any shared signals. OXY, on the other hand, shows a continuation of singular values without a gap, due to undersampling. Panel B2 shows similar behavior for CXY, and for Cxx, we cannot even see a gap between the self and shared singular values. In Panels C1 and C2, the shared signal is overwhelmed by the self signals, and we cannot see any gap for Cxx or Cxy due to undersampling.
while we change Yshared to get the corresponding 5. We calculate their
25

---

Published in Transactions on Machine Learning Research (09/2024)
1000 total dimensions, 10 shared dimensions, 10000 samples
m aboned
mell
Mhel + Mahaord
Cxx
Cxv
10.0, 7 0.1
m
1.0, 7= 0.1
n
0.1, 7 0.1
300
B1
100
singular valuue
250
singular value
singular value
15
10
1.50
40-
20-
-5
01
10"
10
10'
m7
10.0,
17
1.0, 7 1.0
m
0.1, 7- 1.0
B2
60-
15
singular value
singular value
singular value
80-
50-
10
60-
30-
40
20-
20
10-
-5
-10
10"
10
10
10
10
10'
10"
rñ
10.0, 4= 10.0
1.0,
== 10.0
m
0.1, 5
10.0
120
120
B3
60
C3
singular value
singular value
singular value
50
80-
40
30
60
20
20
10"
10
10
10
10
10
10
10 10'
Figure 16: Similar to Fig. 15 but for 10000 samples well-sampled regime. We observe similar behavior
to Fig. 15 in Panels A1, A2, A3, B3, and C3. In Panel B1, we now see a small gap in the spectrum of CXY that was not apparent in Fig. 15 due to undersampling. Panel B2 shows an even clearer gap for CXY. whereas for Cxx, we still cannot see a gap between the self and shared singular values. Panel C1 shows similar behavior for Cxy, but for Cxx, we can see gap after 100 singular values, followed by another small gap after an additional 10 singular values that correspond to the shared signal. However, if we are using IDR method, this means need to retain 110 dimensions, which is challenging to sample adequately. In Panel C2, can now see a clear gap for Cxy after 10 singular values corresponding to the shared signals, while we cannot see any gap for Cxx because all the singular values for self and shared signals have equal power.
26


Reviews:
[{"review_id": "ZPCqUOOI8T", "reviewer": "Reviewer_ngHm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the article is well-written, although a bit verbose. Furthermore, it addresses a significant question to the community. In particular, the article underscores a notable phenomenon in rCCA performance: the performance curve exhibits a peak as a function of the inverse of the number of samples per retained dimension. However, the lack of theoretical insights or geometrical intuitions made the content difficult to follow and hindered the ability to draw definitive conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts a series of numerical experiments comparing independent dimensionality reduction (IDR) and simultaneous dimensionality reduction (SDR) methods for analyzing high-dimensional, multimodal datasets.\n\nIn the first set of experiment, using a linear generative model, the authors demonstrate that SDR methods, like Partial Least Squares (PLS) and Canonical Correlations Analysis (CCA), outperform IDR methods such as Principal Components Analysis (PCA) in reconstructing covariation structures, especially in small sample regimes. Moreover, the authors investigated the performance of regularized CCA (rCCA), for which the definition involves a convex interpolation problem between PLS and CCA, and it was showed that rCCA excels particularly in undersampled scenarios by maintaining nearly perfect reconstruction of shared signals even when self signals and noise are present. \n\nThe second set of experiments explores the performance of linear dimensionality reduction (DR) methods on nonlinear data using a noisy MNIST-inspired dataset. The dataset consists of digit images with variations such as random rotations, scaling, and background noise. The study compares methods like PCA, PLS, and rCCA, focusing on their ability to detect shared signals amidst view-specific noise, which is evaluated by calculating the total corrected correlation between the obtained low-dimensional representations and the inverse of the number of samples per retained dimensions.\nResults show that, in the undersampled scenario with only 1000 samples, rCCA and PLS show an earlier detection of shared signals (measured by the number of retained dimensions after reduction) compared to PCA, which initially falls behind. However, as the number of dimensions grows, all methods experience a reduction in correlation, attributed to the higher noise levels caused by having fewer samples per dimension. While, in the oversampled scenario with 10,000 samples, all methods initially show increased correlation with more retained dimensions, then decline as more singular vectors are estimated from the same sample size, with rCCA consistently outperforming the other methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A summary of key takeaways would facilitate reading the paper.\n- Some theoretical insights or geometric intuitions would be helpful in making the paper easier to comprehend.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ol7bh6OB6q", "reviewer": "Reviewer_xNmU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper performs a careful evaluation of several linear dimensionality reduction methods. Given their prevalence in machine learning, for example in the analysis of representations of trained neural networks, or in the analysis of large-scale neural recordings, such a careful evaluation is welcome.\n- The paper carefully studies several regimes of interest, including over- and undersampled regimes.\n- The paper is clearly written and well-structured.\n\n## Weaknesses\n\nAs the authors acknowledge themselves, the restriction to a linear model of data is the main limitation of the present study. However, I believe that this study is an excellent starting point for an analysis of the algorithms on non-linear models of data, of which I would like to add models of the ICA flavour to the ones mentioned by the authors in their conclusions!). Furthermore, analysing the performance of various algorithms in extracting low-rank signals from data is a very active research area, and the present model might inspire a more mathematical analysis of the performance. \n\nI therefore believe that this study, while being undoubtedly technically sound, should attract the interest of  some individuals in TMLR's audience, and therefore be published in TMLR.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study several methods for linear dimensionality reduction on multi-modal data sets.\n- Principal component analysis (PCA) as an example for Independent Dimensionality Reduction (IDR), where each data modality is compressed independently;\n- Partial Least Squares (PLS) and Canonical Correlations Analysis (CCA) as examples of d Simultaneous Dimensionality Reduction (SDR), where one simultaneously compresses the modalities to maximise the covariation between the reduced descriptions, while paying less attention to how much individual variation is preserved.\nTheir goal is in particular to contrast independent with simultaneous dimensionality reduction techinques.\n\nTo carefully analyses the properties of the various methods, the authors study their performance on a synthetic model of multi-modal data. Their idea is to generate two data matrices, $\\tilde X$ and $\\tilde Y$, which have three parts: an independent white noise; a lower-rank signal that is specific to each matrix, $U_X V_X$ and likewise for $Y$; and finally a shared lower-rank signal that is partly shared across the two data matrices: $P Q_X$ vs. $P Q_Y$. They evaluate the methods by measuring how well the singular directions recovered by each method recovers the shared signals in the data. They broadly find that methods that do a simultaneous decomposition of two data sets perform better at recovering the shared structure than methods that decompose the two datasets independently. Qualitatively similar results are obtained on a non-linear example (noisy MNIST)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper is publishable as is.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nQ2Kt33oh9", "reviewer": "Reviewer_CC1Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Numerical results on a data model which is simplified and interpretable, but practically relevant. \n- Contribution to the development of best practices among practitioners of multi-view data analysis.\n\n## Weaknesses\n- Model and task definition are currently imprecise\n- Unclear motivation of the performance metric $\\mathcal{RC}^\\prime$, and the design of the IDR-SDR comparison\n- Limited mechanistic interpretation of the performance results for PLS and rCCA, and why rCCA performs best", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a numerical study of the relative performance of PCA compared to PLS, CCA and regularized CCA in extracting the shared variation from two noisy data views with partly shared and partly independent low-dimensional latent structure. Given the goal to denoise the cross-correlation matrix by dimensionality reduction, the results show that in the data model considered by the authors the shared dimensionality reduction methods PLS and (r)CCA outperform independent dimensionality reduction of the two views via PCA. The analysis shows that any independent variation of each view retained by PCA  leads to additional spurious correlations in the cross-correlation matrix for small sample size. This can be avoided by PLS and (r)CCA which directly select directions of shared variation. Of the methods compared, rCCA shows best overall performance. The authors also briefly discuss estimating the dimensionalities of shared and independent structure, and show an experiment with nonlinear data structure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Main requested changes\n1. **Clarification of model and task definition, scaling of low-rank terms**  \nThe goal of the IDR/SDR analysis did not seem clearly operationalized (only implicitly through the definition of $\\mathcal{RC}^\\prime$, see point 2. below). The authors use both the terms \"detection\" and \"reconstruction\" of shared signals in the text. At least if the goal to study when the DR methods are able to detect the signal, the scaling of the low-rank signal terms should be explicitly discussed in the introduction or limitation sections: In the current strong scaling of the low-rank spikes, the signal should always be detectable in the proportional $T, N_{X/Y} \\to \\infty$ limit (except by CCA for $T/N < 1$). However, if scaling the variance of the low-rank entries with $N^{-1}$, a for each of the methods a BBP-type transition should occur, where detection becomes impossible. (I am aware of one reference where such a transition was studied for CCA, arXiv.2306.16393).  \nConcerning the model definition, eqs. (1,2), the surrounding text and footnote 2 on p.4,  slightly confusingly suggest that $X/Y \\in \\mathbb{R}^{N_{X/Y}}$ are single sample vectors, and correspondingly one would have  $R_{X/Y} \\in \\mathbb{R}^{N_{X/Y}}$, $U_{X/Y} \\in \\mathbb{R}^{m_{\\textrm{self, }X/Y}}$, $P \\in \\mathbb{R}^{m_{\\textrm{shared}}}$ vectors. However later the canonical version with $X/Y \\in \\mathbb{R}^{T \\times N_{X/Y}}$ as data matrices is used. \n\n2. **Motivation of $\\mathcal{RC}^\\prime$ and the design of the IDR-SDR comparison**  \nI was missing a discussion why $\\mathcal{RC}^\\prime$ captures precisely the goal of the data analysis. In particular, it seems to be only a fair metric if the goal is to extract the shared correlation removing any other variability, and it is very close to the quantity which CCA is optimizing. Would the performance difference between PLS and (r)CCA change if $\\textrm{Cov}[Z_X, Z_Y]$ was used instead of $\\textrm{Corr}[Z_X, Z_Y]$ ?  Furthermore, in the presence of strong (self- and shared-) signals, it is by design that PLS and (r)CCA outperform PCA on this metric. However, if $\\geq m_{\\textrm{self}} + m_{\\textrm{shared}}$  dimensions are retained after denoising by PCA, the shared variability can still be extracted, for example by PLS, possibly with the same or better accuracy (or $Z_Y$ could be predicted from $Z_X$ using PLS-regression). In this sense it might be prudent to argue carefully in how far direct SDR is always preferable to using IDR as a pre-processing step.  \n\n3. **Regression step in the NIPALS algorithm**  \nIn Appendix A, PLS is described as just performing SVD on $X^T Y$. However this is only true for the first singular value, afterwards the NIPALS algorithm deviates from the SVD result due to the regression steps on $X$ and $Y$ followed by deflation of $X^T Y$ (e.g. Wegelin, 2000, Tech Report 371, Univ. Washington). \n\n## Additional changes\n\n4. **Interpretation of performance results for PLS and rCCA**  \nIt would strengthen the results of the paper to explain why PLS fails for small $T$ and $\\gamma_{\\textrm{shared}} > \\gamma_{\\textrm{shared}}$ (Fig. 3.B1), and why rCCA clearly outperforms both PLS and CCA in Fig.3 but also in Fig.8, even though as mentioned in Appendix A.1.4 it corresponds to interpolating between CCA and PLS when varying the regularization parameter.\n\n5. **Test to determine the number of self- and shared signals in the data**  \nOn p.11, varying $\\tilde{q}$ by varying $|Z|$ is discussed as a possibility to determine the number of shared and independent signal components. This is an interesting suggestion. Does this method work better than obtaining $m_{\\textrm{shared}}$ from thresholding the fraction of explained variance of $X^T Y$, e.g. from PLS or (r)CCA,  and $m_{\\textrm{self}} + m_{\\textrm{shared}}$ from thresholding the fraction of explained variance of $X, Y$ from PCA ?\n\n6. **Typos / small changes**  \n   - abstract: \"data set size\" -> dataset size\n   - In sect. 4.1 it is not mentioned that the SNR's of $X$ and $Y$ are symmetric (I assume?)\n   - p.5: \"we expect IDR to be more easily confused [...] by the self-signals\". Could be made more precise\n   - As a suggestion, it could be interesting to also show the singular value spectra of the underlying matrices for one or two relevant parameter choices", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The decision to accept the paper is based on its overall technical soundness, clear writing, and valuable contributions to understanding linear dimensionality reduction methods. While the reviewers note some areas for potential improvement, such as providing more theoretical insights and considering non-linear models, they generally find the paper well-prepared and relevant.
Decision: Accept (poster)