ATD: Augmenting CP Tensor Decomposition by Self Supervision
Authors: Chaoqi Yang
Cheng Qian
Navjot Singh
Cao Xiao
M Brandon Westover
Edgar Solomonik
Jimeng Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: Tensor Decomposition
Self-supervised Learning
Non-convex Optimization

Abstract: Tensor decompositions are powerful tools for dimensionality reduction and feature interpretation of multidimensional data such as signals. Existing tensor decomposition objectives (e.g., Frobenius norm) are designed for fitting raw data under statistical assumptions, which may not align with downstream classification tasks. In practice, raw input tensor can contain irrelevant information while data augmentation techniques may be used to smooth out class-irrelevant noise in samples. This paper addresses the above challenges by proposing augmented tensor decomposition (ATD), which effectively incorporates data augmentations and self-supervised learning (SSL) to boost downstream classification. To address the non-convexity of the new augmented objective, we develop an iterative method that enables the optimization to follow an alternating least squares (ALS) fashion. We evaluate our proposed ATD on multiple datasets. It can achieve 0.8%~2.5% accuracy gain over tensor-based baselines. Also, our ATD model shows comparable or better performance (e.g., up to 15% in accuracy) over self-supervised and autoencoder baselines while using less than 5% of learnable parameters of these baseline models.

ATD: Augmenting CP Tensor Decomposition by Self Supervision
CChaoqi Yang, CChnng Qian, 'Najjot Singh, CCao Xiao, 4.6M Brandon Westover,
EEdgar Solomonik, JJienng Sun
University of Illinois Urbana-Champaaggn, IQQIA, $Relativity,
Massachusetts General Hospital, 'Harvard Medical School
[chaoqiy soloooo.. jimeng] @illinois.ede edu., Palextoqe @ gmmai.ccom waw...aaa@relliii caa.xiio@rrlatiity..oom 4. m="westover ing.harvard.ede
Abstract
Tensor decompositions are powerful tools for dimensionality reduction and feature interpretation of multidimensional data such as signals. Existing tensor decom- position objectives (e.g., Frobenius norm) are designed for fitting raw data under statistical assumptions, which may not align with downstream classification tasks. In practice, raw input tensors can contain irrelevant information, while data augmen- tation techniques may be used to smooth out class-irrelevant noise in samples. This paper addresses the above challenges by proposing augmented tensor decomposi- tion (ATD), which effectively incorporates data augmentations and self-supervised learning (SSL) to boost downstream classification. To address the non-convexity of the new augmented objective, we develop iterative method that enables the optimization to follow an alternating least squares (ALS) fashion. We evaluate our proposed ATD multiple datasets. It can achieve 0.8% 2.5% accuracy gain over tensor-based baselines. Also, our ATD model shows comparable or better perfor- mance (e.g., up 15% in accuracy) over self-supervised and autoencoder baselines while using less than 5% of learnable parameters of these baseline models. We have released our data processing and codes in https//gtthbb.comm/yy//11144411
Introduction
Extracting unsupervised features from high-dimensional data is essential in various scenarios, such as physiological signals (Cong et al., 2015), hyperspectral images (Wang al. 2017) and fMRI (Hamdi et al.] 2018). Tensor decomposition models are often used for high-order feature extraction (Sidiropoulos et al. 2017). Among these, the CANDECOMPIPARAFAC (CP) decomposition is one of the most popular models. The low-rank CP tensor decompositions (Kolda and Bader 2009) assume that the input data is composited by a small set components, while the reduced features are the coefficients that quantify the importance of each basis, which provides a compact representation.
Under this low-rank assumption, existing tensor decomposition objectives aim to fir individual data samples with statistical eтror measures (Hong et al. 2020, Singh et al.) 2021; Yang et al. (022). e.g., Frobenius norm or Kll-divergence. Though fitness an essential principle for feature reduction, common objective functions do not account for downstream tasks, e.g., classification.
Contrastive self-supervised learning (SSL. (He et al. 2020) is recently popular for unsupervised feature learning, which utilizes the class-preserving data augmentations (Dao et al. 2019) and learns an encoder that can filter class-irrelevant information. The optimization goal to enforce alignments (Chen et al. 2020; Wang and Isola 2020), ensuring that the anchor sample is closer to the positive sample (which has the same latent class as the anchor sample) than the negative sample (which is in different latent class) in the embedding space. In an unsupervised setting, positive
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

samples are given by data augmentations, while negative samples are hard to acquire (Arora et al. 2019; He et 2020 Chen al. 2020). Also, previous models are mostly built on deep neural networks, which are often black-box models with tens of thousands of learnable parameters.
This paper aims to incorporate both the fitesss and alignment principles into CP tensor decomposition by augmenting the common fitness objective with a new self-supervised loss. The new self-supervised loss is based on the unbiased estimation of negative samples (Chuang et al. 2020), which effectively prevents the sampling bias issue (Arora et al., 2019). The purpose of our design (i.e., introducing learn class-aware tensor decomposition results for boosting
SSL into CP tensor decomposition) the downstream tensor sample classifications. To address the non-convex subproblems from the new objective, we formulate an iterative method, which solves least squares optimization in each step with a closed-form solution. The main contributions are summarized below.
We propose augmented tensor decomposition, named ATD, which learns an unsupervised CP structure decomposition by extending the original fitness objective with self-supervised loss on the contrastiveness of similar and dissimilar tensor samples.
We develop an iterative method to address the non-convex subproblem from the objective, which enables our algorithm to follow an alternative least squares fashion. Our algorithm has asymptotically the same complexity of each optimization sweep as the common CP-ALS.
We provide extensive evaluations on four real-world datasets and compare to recent tensor decomposition models, autoencoder models, and self-supervised models. Our method shows better or comparable prediction performance in various downstream classifications while only requiring much fewer (e.g., less than 5% of) parameters than that of deep learning baselines.
2 Background
Notation. We use plain letters for scalars, such as or X, boldface uppercase letters for matrices, e.g., X. boldface lowercase letters for vectors, e.g., x, and Euler script letters for tensors, random variables of tensors, and probability distributions, c.g., X. Tensors are multidimensional arrays indexed by three or more indices (modes). For example, an N-mode tensor X' is an --dimensional array of size I x ... x IN. where Zi...... is the element at the (f1,-.. ... )-t position. For matrix X, the r-th row and column are x() and x, respectively, while Fij is for the (i, element. ||X is the Frobenius norm. For vector x, the r-th element Zy, and we use ||x||2 to denote the vector 2-norm, (---) for the vector inner product, o for the outer product, and [-] for the Kruskal product. Indices in the paper start from 1, e.g., X is the first column of the matrix.
2.1 Tensor Modeling
This paper aims to learn tensor bases from unlabeled data and then use the bases to build feature extractor for downstream classification. Without loss of generality (w.rt. tensor order), we consider the fourth-order tensor, e.g., a collection of multi-channel Electreencephaloggapyy (EEG) signals,
1 T(N]] E NXIxJXK
where (()) RIXXX The first dimension of corresponds to data samples (e.g., one for each patient), while the other three are feature dimensions (e.g., channel by frequency by timestamp).
Data Model. To model the above tensor, previous works KKlla and Bader 2009 assume that
There are a set of rank-one tensor components E = {EI...., Ex}, which are learnable;
The tensor data sample/slice (x) admits a low-rank structure and can be represented as a weighted sum of these tensor components E, where (")) denotes its coefficient vector;
On top of the low-rank structure, each data sample (m) also contains additional element-wise i.i.d. Gaussian noise due to real-world distortion (e.g., physical noise in signal measurements).
OOu design may work for other tensor models, such Tucker decomposition. We leave it future work.

---

In the context of downstream classifications, we further assume that each sample (()) is semantically associated to one of the latent classes p (l...., P). and we let D be the sample distribution of class-p. Thus, the data sample can be formulated as, vn.
T
Σ
(n) F E, c( DDs p (I....,,pp
(I)
where
& a, o b, Cr, TE (E,..,,,
(() miid ((0, where a is generally small.
Here ay, by, Cy are the learnable parameters, which produces the rank-one component &,, and they the column vectors of (A IRX B E RJRRR C E RRRRR) referred "bases".
CANDECOMPIPARAPE
Decomposition (CPD). Given the above tensor model, standard CPD only captures the i.i.d. Gaussian noise by minimizing the negative log-likelihood (NLL), which results in the following standard fimesssreconnttructionn loss,
T(m)
CI - - -TA,
eepd
1=
Here, the Kruskal product [+]] outputs fourth-order reconstructed tensor from four input factor matrices. For consistency, if the first input is vector, the output is considered as third-order tensor.
2.2 Problem Formulation
CP decomposition seeks low-rank reconstruction, without special consideration for the downstream task. this paper, we are motivated to improve the CPD model by exploiting the latent classes (in an unsupervised way) and learn good bases provide better class-aware features for classification.
What Good Bases? This paper considers two design principles for good bases. The first principle is fitness, which requires a low-rank tensor reconstruction with the bases. Second, data samples associated with the same latent class should be decomposed into similar coefficient vectors, with the bases, while the vectors should be dissimilar if the samples are from different latent classes. This principle is called alignment, which is important for classification but not considered in the standard tensor decomposition. In this paper, we assess the quality of the learned bases by the performance of downstream classification, where the coefficient vectors (obtained using the bases via decomposition) are the feature inputs (into the downstream classifier).
Working Pipelines. To put it succinetly, the paper tackles an unsupervised learning problem while using downstream supervised classification for evaluation. The procedures are briefly outlined:
First, learn the bases (A, B, C) from large set of unlabeled data. The loss function is developed in consideration of the fitness and alignment (defined in the next section) principles.
Then, we construct the following feature extractor given {A, B, C}. The feature vector of new sample is obtained by the closed-form solution of the least squares problem (a > is hyperparameter),
(1 [x, A. A. C1 |
f f(T A. arg min XERIXR
a||x||
Note that, when f(.) is applied batch of samples, it outputs coefficient matrix.
Next, we evaluate the feature extractor with a set labeled data. Given f(-), we first apply on the labeled data to extract their features and then train an additional logistic regression model (as the downstream classifier) on top of the extracted features, so that the result of classifications will implicitly reflect how good the bases are.
3

---

decompase
loss
Lops + Lreg
T
B
Standard CPD
Our ATD Model
M
Mytwhher Nruteeiaal
decompose
Legd Lreg
E.s
data augmentation
Jalaate www
B
x
Figure 1:
Standard CPD vs Our ATD Model
3 Augmented Tensor Decomposition
We show our model in Figure The design is inspired by the recent popularity of SSL. To exploit the latent class information, we introduce class- preserrreg data augmentation into CPD model and design self-supervieed loss to constrain the learned low-rank features (i.e., the coefficient vectors).
Data Augmentatior In general, data augmentation methods are chosen to perturb the raw data while preserving class label information (He et al. 2020 Chen et al., 2020). Given a sample ((m), we assume that after data augmentation, its perturbation ((n) preserves the label and admits the component-ber representation as Eqn.
3.1 Self-supervised Loss
The design of our self supervised loss corresponds to the alignment principle, which based on pairwise feature similarity and dissimilarity. We call a pair of data samples from the same latent class as positive pair, a pair of samples from different latent classes as negative pair and a pair of independent samples (two random samples from the dataset) as random pair. Intuitively, an anchor plus a positive sample composes a positive pair, similarly for negative pairs and random pairs. In this work, our superviedd loss aims to maximize the feature similarity between positive pairs and minimizes that between negative pairs in an unsupervised way (no labels during optimization).
Formally, let X/. be discrete random variables (of tensor samples) distributed Dpr P {1...., P}. We want to minimize the following objective when no class labels are given,
Las
Lpps + XLnegy
where À > 1 hyperparameter and
Cpos - [sim (f ((4)) | p=
Lueg [sim (f ( | p
Here Cpos maximizes the feature similarity of positive pairs while fueg minimizes the feature similarity of negative pairs. f(.) is the feature extractor, defined in Eqn. @ and the similarity measure is given by cosine distance, parameterized by two random variables.
f(X,) f()g)
sim (f (x,),
(A,
(F())))
To this end, the key is implement the self-supervised loss, i.e., Eqn. 3 in unsupervised setting. Specifically, we want to construct the sampler of positive pairs and the sampler of negative pairs with unlabeled data only. The sampler of positive pairs (in short, positive sampler) can be easily approximated by data augmentation techniques, which provides "surrogate" positive pairs (given any
We provide further discussions and ablation studies data augmentation in Appendix C.3 and C.5

---

sample as the anchor, we apply data augmentation methods to generate a perturbed data as positive sample, and then the anchor plus the perturbed data is a positive pair). However, the negative sampler is infeasible, without labels. As compromise, previous works (He et al, 2020) Chen et al. 2020) consider using random sampler to replace the negative sampler given that the random sampler can be easily achieved by picking two independent samples from the dataset. However, this practice is shown to induce sampling bias and hurts the learned representation (Arora et al. 2019 Chuang et al. 2020) since random pair may be from the same latent class.
Construction of Negative Sampler. In this paper, we consider using the law of total probability to construct the negative sampler in statistical way. Formally, assume "p is the label rate of latent class-p (thus, we have Ep "p
= 1), we apply the law of total probability and the following holds,
P
E [sim (f (X)), ((())) 1pp g)
Ti Bp (sim (f (
Pp
p=1
q=p
P
Tp"' Epgg [sim (Xp) (f (Vr)) p = ql +
rp'g Bp. [sim (f (f
p=1 Tp
Tp
p=
9=1
E
Pp
sim (f
p
+
1
sim (f (X,) ((,)).(())
Tp
Tp
Here, the usage of E|. means that the expectation is taken over four interdependent random variables, i... p. q; X,, y/, while Ey,o(:] means that p. q is fixed and thus only taken over two random variables, i.e., Y... The result shows that the negative sampler can be equivalently replaced by weighted combination of the random sampler and positive sampler. Here we do not have access to
Tp: Vp with unlabeled data, this issue is dealt with later.
Self-supervised Loss.
Consequently, we can reformulate our self-supervised loss as,
Las pos + Afrnng
- E [sim (f ((g)) p-= q] + AE [sim (f xf)))(())))))))))))))))))) p ql
(5)
Arp [ +1)
= EE sim (f (Xt,), (D,))) - E
sim (f (xx, ff (v,)) 1p q
(6)
P
From Eqn.
to Eqn. we use the results in Eqn.
Two-sided Bound. The above form still requires label rate information, i.e., Tp. Vp, we therefore
consider using the following approximation to the above loss Lass
(7) = (7 + 1)E (sim (f (xx),, - E [sim (f ((²)) == d)
(7)
Here, 7 2 is hyperparameter, while Las be bounded (derivations in appendix E).
C ( A S Los SCC
C2 (-)) <
( A (7)
1-rp 12=1+ Arp Tp
Xrp
C1 C1= =
max P
min
(8)
C₂
The equivalence is established when C1 C2, i.e., the class labels are balanced. To simplify the derivation, we ignore in the following and let y be new hyperparameter. Also, the constants C and C2 can be absorbed into weight hyperparameter B, given in the next section. This bound implies that, an easy-to-compute 3C (7) is often a good approximation of L.s for some B. The next section specifies how to compute 3 (7)
(7) unsupervisedly as our empirical self-supervised loss.
3.2 The Objective of ATD
Empirical Estimator: We obtain an empirical estimator of C with Monte Carlo method. Suppose T and T are the input tensor and the augmented tensor respectively, and X f(T), X f(T) N xt are the coefficienn/feature matrices. We use the row vectors of X, X to estimate Eqn. 7
The first term E [sim (f (X₂), f (2))) essentially requires a random sampler, which is approximated by the average cosine similarity of all possible pairs of non-corresponding row vectors of X, X, while
5

---

the second term E [sim (f (xp). ((%))) p dl requires positive sampler, which is estimated by the average cosine similarity all pairs of corresponding row vectors,
(-) N(N 1)
Σ
NV.
x(m)
{(0)
1 N
x(x)
x(m)
||x"112
|x(
|
2
1|
=Tr x D(X)G())D((XX
(9)
where D (X) diag
is the row-wise scaling matrix and
|x|>0||2
7+1
N
N(N-I)
N(N
7+1
G(1)
N(W-1)
N(N
7+1
7+1
N(N-1)
N(N-1)
Note that, the form in Eqn. 9 significantly different from tensor discriminant analysis (Jia et al. 2014 Tao et al. 2007). which integrates the actual label information as regularizer and is also different from graph regularized tensor decomposition (Cai et al. 2010), which incorporates side information, such geometrical positions (Maki et al. 2018). Compared to standard noise contrastive estimation (NCE) (Gutmann and Hyvärinen 2010 Chen et al. 2020) in the area of contrastive SSL, our SSL form in Eqn. 4 considers a subtraction form instead of the softmax formulation, making amenable to quadratic optimization, as we will show in Sec. 3.3
Overall Objective. According Eqn. 8 the self supervised loss L.e is bounded by L (v), while the constants (i.e., C1, C2) can be absorbed into a weight hyperparameter 3. We let the empirical self- supervised loss, = (v). Our objective follows both the fimess (i.c., CPD reconstruction loss) and alignment (i.e., self supervised loss) principles, while also considering Tikhonov regularization (Gollbb and Von Matt 1997) to constrain the scale of all parameters.
L = Leed + foreg + Les,
(10)
where
Leed IIT [X, A. B C T [X A. B, CI
Creg = a ||x +
X + ||A
B +
E.s = (2) BTr x D(X))())
(11)
The objective has (i) three hyperparameters, a, B > 0: (ii) five factor matrices, A,B,C,,,
3.3 Alternating Least Squares Optimization
Ideally, we would like to use alternative least squares (ALS) algorithm (Sidiropoulos et al. 2017 for optimizing the objective, where each factor matrix is updated in sequence by solving least squares subproblems. With large scale tensors, we can also resort mini-batch stochastic ALS (Cao et al. 2020) to reduce memory footprint of the decomposition. However, the objective in Eqn.
is non non-convex to X and X, respectively. For addressing the non-convex subproblem, we design an iterative method in this section, which only involves solving least square problems.
Addressing Non-convex Subproblem.
We use the subproblem of X as an example. Given A, B, C, X fixed, we want to minimize Eqn. (Iop by finding the optimal solution, denoted as X*,
X + arg min (IT (IT--XXA
a al 3Tr (x D(X)))))))
(12)
First, we are interested to find that the matrix-formed problem in Eqn. 12 can decomposed into row-wise subproblems. To obtain the solution of Eqn. (12 is suffice to solve each subproblem

---

independently. Let consider the subproblem of the k-th row, which is
arg min
[x, Ax, -x.
+a||x|/ 3Tr
x
(4)
D(X)))
(13)
|x||2
where ((k) is the k-th slice of T, and g(() is the k-th row of G(7).
Here, Eqn. (T3) is still non-convex with respect to x. We let the derivative of Equ. be zero and arrange the terms, which yields.
objective to
8v2 2||x||2
(I
x= x=
V3.
(14)
where
V1 = T (A
B C), V2 = g D(X)
V3 V (A=AAAABBBBCCCCaaa
Here x, V1, V2 are row vectors and Va a matrix. ((x) the 1-mode unfolding 7((), O the
Khatri- -ao product and is the Hadamard product (i.e., element-wise product).
We consider the following iterative rule and the fixed point is a solution for Eqn. stationary point Eqn.
14 which is
8v2 2 Xinit
X Xinit Xinit
Ximpe VI V3
Va-
(15)
Xinit 12
We use an initial guess Xiii (obtained by solving Equ. with while B O, which is least squares problem) to start. Then, we repeat Eqn. (15) for each row (i.c., each k) independently and let the improved guess be the initial guess. Xinit - Ximpe- to iteratively improve the result.
Theorem ! (proof in Appendix A ensures that Eqn. [15 converges linearly
chosen to be sufficiently small. In Appendix we verify the liner convergence and also empirically show that one round of Eqn. (5 is sufficient in our experiment, where = 2. The non-convex subproblem of X can be solved in the same way.
Theorem 1. Given non-zero row vectors VI, V2, u Rdd, non-singular matrix Va E Rddd and 8ra
B > 0. The sequence {u'), generated by u V1 V3
(I- u V3. satisfies,
2/[u*|12
1
1u+1-uulg 3(2m + M)Il Valla ||u u 11
3
where u² is the fixed point and m min, || M = max; ||u'||2 are the bound of the sequence. With good uD and sufficiently small B, we can safely assume < m M 00.
Optimization Procedures. To minimize Eqn. (T)) we alternatively update A, C. and X. where each subproblem involves only solving least squares problems with closed-form solutions (summarized in Algorithm D. With large-scale tensors (as in the experiments), we optimizes the factors mini-batches. Between mini-batches, the basis factors A. B, C are shared and updated incrementally. We show the batch algortihm in Appendix D The computation head of the algorithm matricized tensor times Khatri-Rao product (MTTKRP). The complexity of our optimization algorithm is asymptotically the same as applying CP-ALS on the original tensor T with the same rank R, which costs O(NIJK to sweep over the whole tensor once.
Experiments
This section presents the experimental evaluations. Due to space limitation, additional details, including data augmentations and baseline implementation, are presented in Appendix
4.1 Experimental Setup
Data Preparation. We use four real-world datasets: (i) Sleep-EDF Kemp et al. 2000), which contains EOG, EMG and EEG Polysomnography recordings; (i) human activity recognition (HAR)
7

---

Algorithm 1: Alternating Least Squares
Input: Data tensor T E R NxIxxxX, initialized A, B, C, X, X; other hyperparameters a. 9,
Obtain the augmented tensor T:
repeat
Use A, B, C,X to update X by our iterative rules (one iteration) in Eqn.
15
Use A, B. C,X to update X by our iterative rules (one iteration) in Eqn. 15
Use B, C, X,XXX to update A by solving least squares problem;
Use A, C, X,X to update by solving least squares problem:
Use X,X update by solving least squares problem:
until max sweep exceeds change of loss < 0.1% within 3 consecutive sweeps; Output: the learned bases {A, B, C).
Table I: Dataset Statistics
Name
Duta Sumple Format
Angmentations
# Unlabeled (N)
Truining
# Test
Task
8 Class
Sleep Sleep-EDD EDF
Ix x K: 14 x 129 x 86
(a), (b)
331,208
42,803
41,078
Sleep Siuging
HAR
1x K: 18 33 33
(a), (b), (c)
7,352
1,473
1,474
Activity Recognition
PTB-XL MGH
x
(a), (b)
17,469
2,183
2,185
Gender Idntiiiiation
J x K: 12
257 x 43
(4), (b)
4,377,170
238,312
248.041
Sleep Staging
We report the data format after sheet time Fourter transform (STFFT Appendix C22
which is used to obtain the spectrogram.
Anguita et al. 2013) with smartphone accelerometer and gyroscope data; (iii) Physikalisch Technis- che Bundesanstalt large scale cardiology database (PTB-XL) (Alday et 2020) with 12-lead ECG signals; (iv) Massachusetts General Hospital (MGH) (Biswal et al. 2018) datasets with multi-channel EEG waves. All datasets are split into three disjoint (i.e., unlabeled, training and test) by subjects, while training and test sets have labels. Basic statistics are shown in Table 1. All models (baselines and our ATD) use the same augmentation techniques: (a) jittering, (b) bandpass filtering, and (c) 3D position rotation. We provide an ablation study on the augmentation methods in Appendix C.5
Baseline Methods. We include the following comparison models from different perspectives:
Tensor based models: ATD..- is our variant, which removes the self-supervised loss from the objective Eqn. {0); Stochastic alternating least squares (SALS) applies on the the CPD objective
with Tikhonov regularizer. which works on large tensors; Graph regularized SALS (GR-SALSS
augments the objective of SALS with a graph regularizer Maki al.) 2018 define as Tr [XTGX GX).
Cai et al. 2010).
Self-supervised models: SimCLR-r (Chen et al. 2020) and BYOL-r (rill et al. 2020 are two popular SSL models with their own objective functions, where indicates the size of the output representation.
Auto-encoder models: AE-r denotes a CNN based autoencoder with mean square error (MSE) reconstruction loss, and AEss-r denotes the same autoencoder model with standard NCE loss in the bottleneck layer, where is the representation size.
All models use the unlabeled set to train a feature encoder and use training and test sets to evaluate. Note that, deep neural network models use the same CNN backbone. Appendix C.8 we have also compared with two recent supervised tensor learning models, which shows the usefulness of our ATD and the large unlabeled set, especially in low-label rate scenarios.
Evaluation and Environments. We evaluate model performance mainly based on classification accurucy, where we train an additional logistic classifier (He al. 2020) on top the feature encoder. Also, for different models, we compare their number of learnable parameters. The experiments are implemented by Python 3.8.5, Torch 1.8.0 I on a Linux workstation with 256 GB memory, 32 core CPUs (3.70 GHz, 128 MB cache), two RTX 3090 GPUs (24 GB memory each). All training performed on the GPU. For tensor based models, we use R = 32 and implement the pipeline in CUDA manually, instead of using torch-autograch
4.2 Experimental Results
This section shows the experimental results on downstream classification. We use all the unlabeled data to train the encoder feature extractor, and use training data (since Sleep-EDF and MGH

---

Table 2: Result of Downstream Classification (%). The table shows that our ATD can provide comparable or better performance over all baselines with fewer parameters, especially deep learning models. It also shows the usefulness of considering both fitmess and alignment as part of the objective.
Sleep-EDi EDF (5,000)
HAR (1.47))
PIB-XL (2,183)
MGH (5,000)
Accuracy
Parans.
Accuracy
of Params.
Accuracy
of Params.
Accuracy
Params.
Self-aup models:
SimCLR-32
81.98 1 0.358
210,384
74.75 1: 0.723
53,286
69.25 1: 0.355
200,960
67.34 +: 0.970
212,624
SimCLR-128
85,19
0.358
222,768
76.69
0.697
65,670
68.19 0.793
237.920
66.98
1.331
246.608
BYOL-32
84.29
0,405
211,440
73.71 2.832
65.08
1.535
202,016
68.83 1.168
214,736
BYOL-128
83.26 2: 0.337
239,280
71.79
1,866
82,182
63.49 +: 0.612
254,432
68.55 + 1.339
279,632
Auto-encuders:
AB-32
74.78 0.723
217,216
63.13
0.775
62,940
59.01 0.896
224,578
68. 68.58 0,427
220,088
AE-128
75.17 1: 0.897
241,885
60.52 =: 1660
$7,612
58.29 1: 0.412
298.352
67.05 :: 1.375
257 047
-3,
80.92 0.345
217.216
71.70
2.135
62,940
68.47 0.231
224,528
71.46 0.386
220,088
122
81.84
0.259
241,888
72.43
1.370
87,612
68.88 0.604
298,352
19 0.617
257 0048
Tensor models:
SALS
84.27 0.488
7,328
91.86
0.295
2,688
69.15 0.483
7,296
71.93 + 0.379
9,984
GR-SALS
84.33 0.356
7.328
92.33
0.282
2,688
69.02 0.477
7,296
72.35
0.228
9,984
AID
81.19 :: 0.221
7,328
92:41 = 0.391
2,688
69.38 +: 0.612
7.296
72.78 +: 0.522
9,981
ATD
85.01
0.224
7,328
93.35
0.357
2,688
70,26
0.523
7,296
74.15 0.431
9,984
Parentheuss shows the mumher of training somples. Our improvements are statistically significunt with yi <. 0:00 (detnits appendix D.7)).
datasets have enough training samples, we randomly selected a subset of them) for learning a downstream classifier and use all test data. Each experiment is conducted with five different random seeds and the mean and standard deviations are reported. The metrics are the accuracy and the number of learnable parameters. All models have 32-dim features in the end, except that for two self-supervised baselines and autoencoder variants, which have 128-dim options.
4.2.1 Better Classification Accuracy with Fewer Parameters
From Table ATD shows comparable or better performance over the baselines. We have also reported the running time per epoch/sweep in Appendix C.7 for all models. Compared to the variant ATD our ATD can improve the accuracy by 1.0% ~ 1.9%. which shows the benefit of the inclusion of self-supervised loss. SALS and ATD have similar performance, while their objectives differ in that ATD..- considers the Frobenius norm of the augmented data. Thus, their accuracy gap is caused by the use of data augmentation. Also, the experiments show that the fitness and alignment principles are both important. We observe that with self-supervised loss (i.e., alignment), AE.s can give significant improvements over AE, while ATD shows ~ 8% accuracy gain over the self-supervised models on MGH dataset, since we can better preserve the data with reconstruction loss fitmes).
Moreover, the table shows that tensor based models require fewer parameters, i.e., less than 5% of parameters compared to deep learning models. On HAR, the deep unsupervised models show poor performance due to (i) they may not optimize a large number of parameters on middle-scale dataset; (ii) movement signals in HAR might have few degrees of freedom, which matches well with the low-rank assumption of tensor methods. On large-scale Sleep-EDF, self-supervised models outperforms ATD marginally since they have more parameters thus can capture more information.
4.2.2 Better Performance in Low-label Rate Scenarios
On the MGH dataset, also show the effect of varying the amount of training data in Figure We include an end-to-end convolutional neural network (CNN) model based on (Biswal et al. 2018) 8) called Reference CNN, which is supervised model and only uses the training and test sets. To be more readable, we separate the comparison figure into two sub-figures: the left compares our ATD model with self-supervised and auto-encoder baselines and the right one compares ATD with tensor baselines and the reference model, and the scale of y-axis on two sub-figures are the same.
We find that all unsupervised models outperform the supervised reference CNN model in scenarios with fewer training samples. With more training data, the performance all models get improved, especially the reference CNN model, which can optimize the encoder and predictive layers in an end-to-end way and finally outperforms our ATD when more training samples is available.
4.2.3 Stable Results with Hyperparameter Variation
For a comprehensive evaluation, we also conduct ablation studies on the effect of the data augmenta- tion methods and on hyperparameters. Due to space limitation, move the experimental settings and

---

0.75
Classftication (36)
Cisgiiication Accaracy
0.65
0.65
0.60
0.00
SiMALA
6
Reference CMN
SALS
GRLSALS
0.50
0.50
0.45
0.45
64
128
256 512
2048 # of training samples
1024
4096
6192
16384
64
128
256
4056 9192 16384 trainingg samples
512
1024
2045
Figure 2: Varying the of Training Data
results to Appendix C.6 while summarizing the general conclusions here: (i) with more diverse data augmentation methods, final results are relatively better; (ii) with a larger rank R, the performance will be better generally; (iii) our ATD is not sensitive hyperparameters a and 7, and 3 A 0 can be chosen from a large range (e.g., 3 = 2 in the experiments) for decent performances.
Related Work
Data augmentation and Self-supervised Learning. Data augmentation exploits class-preserving
perturbations to smooth out noise and encode task-invariances (Dao et al. 2019). It has been widely
used in various data formats, such images (Cireşan al. 2010), text (Lu al. 2006). audio
(Uhlich et al. 2017). and time series (Wen et al. 2020 Ya Vang et al. 2021b). Data augmentation also benefits the recent development contrastive SSL (e al. 2020 Chen al. 2020). which extracts class-relevant features by optimizing deep neural network encoder to achieve agreements between semantically similar samples and disagreements on dissimilar samples. However, in contrastive SSL, a recent work (Arora et al., 2019) highlighted that the common practice of replacing negative samples with random samples leads to sampling bias, which may hurt the learned representation significantly (Chuang et al., 2020). In this paper, we introduce unbiased self-supervised objective into CP tensor decomposition model and shows that the new design can helpful in producing class-aware outputs.
Stochastic Algorithms for Tensors. With the rapid growth in data volume, efficient stochastic tensor methods become increasingly important for higher-order data structures boost scalability. These methods are largely based on sampling (Ma and Solomonik] [02T] Yang et al. 202 la Kolda and Hong 2020). which accelerates the computation of over-determined least square problems (Battaglino et al.
2018 Larsen and Kolda 2020) in ALS for dense (Ailon and Chazelle 2006) and sparse (Eshragh
al. 2019) tensors by effective strategies, such as Fast Johnson- Lindnsson- Transform (Ailon and Chazelle, 2006), leverage- evveaage-bsed sampling (Eshragh et al. 2019),, and sketching. However, these algorithms only focus on making ALS steps less costly and require to load the full data into memory. Thus, we do not consider them in our setting. This paper integrates augmentation techniques and self-supervised loss into tensor decomposition, and later we adopt an effective stochastic alternating optimization to handle large scale optimization with less memory consumption.
6 Conclusion
This paper introduces the concept of self-supervised learning for tensors and proposes Augmented Tensor Decomposition (ATD) and the ALS-based optimization. We show that by explicitly contrasting positive and negative samples, the decomposition results are more aligned with downstream classifi- cation. On four real-world datasets, we show the advantages of our model over various unsupervised models and in low-label rate scenarios, our model even outperforms the reference supervised models.
Compared deep learning methods, tensor based models and diverse inputs, such as natural images. However, applying tensor decomposition on the outputs of earlier layers of pre-trained deep neural networks may be feasible way address the weaknesses. This direction would be interesting for future work.
flexible in processing multimodal
Acknowledgements
This work was supported by NSF award SCH-2205289, SCH-2014438, ISS-183004,, NIH award RO1 IRO1NS
10

---

References
Ailon, N. and Chazelle, B. (2006). Approximate nearest neighbors and the fast Johnsoon--Lnnddessaasss transform. In STOC, pages 557-563.
Alday, E. P., Gu, A., Shah. A. J., Robichaux, C., Wong, A.-K. I., Liu, C., Liu, F. Rad, B., Elola,
A., Seyedi, S., et al. (2020). Classification of 12-lead ecgs: the physionet/computing in cardiology challenge 2020. Physiological measurement, 41(12):124003.
Anguita, D., Ghio, A., Oneto, L.., Parra, X., and Reyes-Ortiz, L.. (2013). A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, page 3.
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. (2019). theoretical analysis of contrastive unsupervised representation learning. arXiv preprini urXiv:/902.0922)
Battaglino, C., Ballard, G., and Kolda, T. G. (2018). practical randomized CP tensor decomposition. SIAM Journal on Matrix Analysis and Applications, 39(2):876-901.
Biswal, S., Sun, H., Goparaju, B., Westover, M. B., Sun, J., and Bianchi, M. T. (2018). Expert-level sleep scoring with deep neural networks. Journal of the American Medical Informatics Association, 25(12):1643-1650
Cai, D., He, X., Han, J., and Huang, T. S. (2010). Graph regularized nonnegative matrix factor- ization for data representation. IEEE transactions on pattern analysis and machine intelligence, 3388)1154-1550.
Cao, Y., Das, S., Ocding. L., and van Wyk, H.-W. (2020). Analysis of the stochastic alternating least squares method for the decomposition of random tensors. arXiv preprint 2iti::2004.123300
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597- 1607. PMLR.
Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O., and Azemi, E. (2020). Subject-aware contrastive learning for biosignals. arXiv preprint arXiv:2007.04877.
Chuang. C.,Y., Robinson, J., Yen-Chen, L., Torralba, A., and Jegelka, S. (2020). Debiased contrastive learning, 1XXiv2007.000244
Cireşan, D. C., Meier, U., Gambardella, L. M., and Schmidhuber, J. (2010). Deep. big, simple neural nets for handwritten digit recognition. Neural computation, 22(12):3207-3220
Cong, F., Lin, Q.- Kuang, L.-D., Gong, X.-F., Astikainen, P., and Ristaniemi, T. (2015). Tensor decomposition of EEG signals: brief review. Journal of neuroscience methods, 248:59-69.
Dao, T., Gu, A., Ratner, A., Smith, V., De Sa, C., and Ré, C. (2019). kernel theory of modern data augmentation. In International Conference Machine Learning, pages
PMLR.
Eshragh, A., Roosta, F., Nazari, A., and Mahoney, M. (2019). Lsar: Efficient leverage score sampling algorithm for the analysis of big time series data. arXiv.
Golub, G. H. and Von Matt, U. (1997). Tikhonov regularization for large scale problems. Citeseer.
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H.. Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020). Bootstrap your own latent: A new approach to self-supervised learning. arXiv:2006.07733
Gutmann, M. and Hyvärinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings.
Hamdi, S. M., Wu, Y., Boubrahimi, S. F., Angryk, R., Krishnamurthy, L. C., and Morris, R. (201 8). Tensor decomposition for neurodevelopmental disorder prediction. International Conference on Brain Informatics, pages 339-348. Springer.

---

He, K., Fan, H., Wu, Y., Xie. S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Hong, D., Kolda, T. G., and Duersch, J. A. (2020). Generalized canonical polyadic tensor decomposi- tion, SIAM Review, 62(1):133-16)
Jia, C., Zhong, G., and Fu, Y. (2014). Low-rank tensor learning with discriminant analysis for action classification and image recovery. In Twenty-eighth AAAI conference on artificial intelligence.
Kemp, B.. Zwinderman, A. H., Tuk, B., Kamphuisen, H. A.. and Oberye, J. J. (2000). Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. IEEE Transactions Biomedical Engineering, 1719):11851119.
Kolda, T. G. and Bader, B. W. (2009). Tensor decompositions and applications. SIAM review, 5155):55-500.
Kolda, T. and Hong, D. (2020). Stochastic gradients for large-scale tensor decomposition. SIAM Journal on Mathematics of Data Science, 2(4):1066-1095.
Larsen, B. W. and Kolda, T. G. (2020). Practical leverage-based sampling for low-rank tensor decomposition. arXiv preprint arXiv:2006. 16438
Lu, H., Plataniotis, K. N., and Venetsanopoulos, A. (2008). Uncorrelated multilinear discriminant analysis with regularization and aggregation for tensor object recognition. IEEE Transactions on Neural Networks,
Lu, X., Zheng, B., Velivelli, A., and Zhai, C. (2006). Enhancing text categorization with semantic- enriched representation and training data augmentation. JAMIA, 13(5):526-53 5355.
Ma, L. and Solomonik, E. (2021). Fast and accurate randomized algorithms for low-rank tensor decompositions. 1rXiv:2104.01101.
Maki, H., Tanaka, H., Sakti. S., and Nakamura, S. (2018). Graph regularized tensor factorization for single-trial eeg analysis. In ICASSP, pages 846-850. IEFF.
Sidiropoulos, N. D., De Lathauwer, L., Fu, X., Huang, K., Papalexakis, E. E., and Faloutsos, C. (2017). Tensor decomposition for signal processing and machine learning. JEEE Transactions on Signal Processing, 65(13):3551-358)
Singh, N., Zhang, Z., Wu, X., Zhang, N., Zhang, S., and Solomonik, E. (2021). Distributed-memon tensor completion for generalized loss functions in python using new sparse tensor kernelsate randomized algorithms for low-rank tensor decompositions. arXiv preprint arXiv:1910.02371.
Tao, D., Li, X., Hu, W., Maybank, S., and Wu, X. (2005). Supervised tensor learning. In Fifth IEEE International Conference on Data Mining (ICDM '05), pages 8 pp. IEEE.
Tao, D., Li, X., Wu, X., and Maybank, S. (2007). General tensor diseriminant analysis and gabor features for gait recognition. IEEE transactions on pattern analysis and machine intelligence, 770005:1
Uhlich, S., Poreu, M., Giron, F., Enenkl, M., Kemp, T., Takahashi, N., and Mitsufuji. Y. (2017). Improving music source separation based on deep neural networks through data augmentation and network blending. ICASSP, pages 261-265. IEEE
Wang, T. and Isola, P. (2020). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR.
Wang, Y., Peng, J., Zhao, Q., Leung, Y., Zhao, X.-L., and Meng, D. (2017). Hyperspectral image restoration via total variation regularized low-rank tensor decomposition. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 124):1227-1
Wen, Q. Sun, I.., Song, X., Gao, J., Wang, X., and Xu, (2020). Time series data augmentation for deep learning: A survey, arXiv preprint 27X4v:2002.12478
12

---

Yang, C., Qian, C., and Sun, J. (2022). Gocpt: Generalized online canonical polyadic tensor factorization and completion. In IJCAI.
Yang. C., Singh, N., Xiao, C., Qian, C., Solomonik, E., and Sun, (2021a). MTC: Multiresolution tensor completion from partial and coarse observations.
Yang. C., Xiao, D., Westover, M. B., and Sun, J. (2021b). Self-suuervised eeg representation learning for automatic sleep staging. arXiv preprint arXiv:21
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No] or [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 1.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] see Theorem 1
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material URL)? [Yes] See supplementary
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [Yes] See Appendix C.I
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sQy-K26XhXo", "reviewer": "Reviewer_rtia", "summary": "This paper proposes a tensor decomposition leveraging contrastive self-supervised learning based on the data augmentation for positive samples and the unbiased estimation of negative samples, which can align with downstream classification tasks. The proposed method optimizes the objective by using an alternating least squares algorithm. The experimental results using four real-world tensor datasets show that the proposed method achieves better accuracy on the downstream classification tasks using a much smaller number of parameters than several comparison models including tensor-based models, self-supervised models, and auto-encoder models. ", "questions": "Have you compared the proposed method with some supervised tensor learning by using the training/test dataset?", "limitations": "As mentioned above, the classification accuracy is not significantly different with or without the empirical self-supervised loss in the experimental results. It should be made more clear the situation the proposed method is strongly needed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThe proposition of contrastive learning using data augmentation for positive samples will greatly contribute to the research community of tensor decomposition. Although the proposed method is a fundamentally unsupervised method and able to utilize a large number of unlabeled data, it should be compared with several works of supervised tensor learning such as [Tao et al., 2005].\n\n[Tao et al., 2005] Dacheng Tao, Xuelong Li, Weiming Hu, Stephen J. Maybank, Xindong Wu: Supervised Tensor Learning. ICDM 2005: 450-457\n\nQuality:\nThe problem setting makes sense. The proposed method is convincing along with theoretical discussion.\n\nClarity:\nThis paper is well-organized and clearly written.\n\nSignificance:\nThe classification accuracy shown in Table 2 is not significantly different with or without the empirical self-supervised loss. Moreover, as described in the appendix, the proposed method is not sensitive to beta. There is room to make the proposed method more significant.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dphfe9BzKF4", "reviewer": "Reviewer_8z4A", "summary": "This paper propose a novel self-supervised framework for CP Tensor Decomposition, in which an iterative method is developed to enable  an alternative least squares fashion. The experiment results on various datasets show its efficiency.", "questions": " could the authors provide more explaination about the derivation of Eq.(8)?", "limitations": "no", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n1. incorporating self-supervised learning into CP Tensor Decomposition seems to new to me.\n2. the experiments are extensive and promising\n3. this paper is well-motivated\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "IfQanMj4NO", "reviewer": "Reviewer_qLDm", "summary": "In the paper, the authors investigated a new variant of CP decomposition (CPD) for the classification task. Compared to the classic CPD, the new one imposes the self-supervised loss, so better embedding spaces, modeled by latent factors, can be learned by augmenting the training data. ", "questions": "1. What is the running time for training the all methods implemented in the experiment?\n2. What happens if the augmented data get unreliable, e.g., low-quality?\n3. How other tensor methods are influenced by varying the rank in the experiment?", "limitations": "The limitation compared with deep learning methods are briefly mentioned, but there is no discussion compared with other tensor-based methods. For example, does the additional loss reduces the interpretability or the expressive power of tensor models? ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n\n- (clarity) The paper is well written. The background, formulation, and algorithm are clearly introduced.\n- (originality) Considering the self-supervision loss is new for tensor decomposition, and the experimental results verified the effectiveness of the method.\n\n**Weakness:**\n\n- (quality, clarity) Part of the technical details are not well discussed\n    1. Although Thm. 1 proved the convergence for the sub-problem of learning $\\mathbf{X}$, only *one* iteration is employed in the whole algorithm as given in the appendix (page 14). It might work well in practice (mentioned in the appendix), but such modification makes me suspect if the whole algorithm enables to converge to fixed points. Careful discussions lack on this point.\n    2. It also lacks the discussion (apology if I miss it) regarding how the data augmentation quality impacts the new method's performance.\n    3. In the ablation study, the comparison with other tensor baselines should be added. Otherwise, it is hard to fairly evaluate if the proposed method is non-sensitive to the parameters, e.g., the CP rank.\n    4. The proposed method's advantage of the number of learnable parameters is not clearly discussed. For example, the deep learning models may run faster than tensor-based ones on GPU even though more parameters are required. If the aim is to evaluate the expressive power of the model, the number of parameters also seems not a good metric, especially when the model architectures are different.\n- (significance) The proposed method might be useful for specific tasks such as those in the experiment section. But there are few new ideas or contributions to the tensor community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DEFrBTkmMZM", "reviewer": "Reviewer_cxKw", "summary": "The authors present  \"augmented\" tensor decomposition, which utilizes an objective that is designed for downstream tasks (by way of \"alignment\") in addition to data reconstruction (\"fitness\"). Hence, the quality of the decomposition is determined not only by reconstruction error, but by correct clustering on features. The authors contribute an algorithm for implementing this approach and motivate the significance with a comparison to existing CP decomposition approaches. The authors' algorithm builds on Alternating Least Squares (ALS), a typical method in CP decomposition algorithms.", "questions": "- Is there a concern about the validity of using classification accuracy to measure model performance, in cases where classes are highly imbalanced?\n- Appendix D.6 addresses that ATD's performance improves with larger R. What about its performance relative to the other models in Table 2?\n", "limitations": "- Good discussion of limitations. Please emphasize whether the computational complexity compared to existing methods is a limitation or strength. One obvious limitation (given that medical data is heavily used in the experiments on real data) that may be useful to mention is that this method is developed for real-valued data (as opposed to bounded data). ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach is well motivated. The novelty and potential significance of the authors' contributions is made immediately clear in Section 1.\n- The contributions are clearly defined and include: a method to generate dissimilar data samples in the absence of labels.\n- Presentation is clear and easy to follow.\n- The experiments on real data use a convincing variety of datasets: there is sufficient variation in dataset size, balance of dimensions, and downstream tasks.\n\nWeaknesses:\n- The discussion of classification appears to be wholly motivated by classification of samples using features (eg. classifying patients based on EEG-related features). The nature of the classification task is heavily dependent on the application -- eg. decomposition of DNA methylation data (where it is desirable to do dimensionality reduction on the # of features, which is very large, in addition to clustering patients) is often motivated by simultaneous clustering/classification of features in addition to patients. Would be helpful to clarify whether more flexible classification is feasible/reasonable under this approach.\n- The perennial question in CP/Tucker decomposition is how to choose the dimension of latent features (ie. the decomposition rank). Would like to see more discussion on this topic, eg. the stability of this method's performance to perturbations in this regard (ex. is ATD still superior if R is varied within a reasonable range?) \n- The datasets in Section 4 (Experiments) all have a relatively similar # of classes. The results would be more compelling if there was more variation in this dimension.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept