Nonlinear Sequence Embedding by Monotone Variational Inequality
Authors: Jonathan Yuyang Zhou
Yao Xie
Venue: ICLR 2025 Spotlight (2025)
Keywords: Monotone Variational Inequality
Convex Optimization
Sequence Data
Time Series
Representation Learning
Primary area: learning on time series and dynamical systems
TL;DR: We introduce a convex method to learn low dimensional representations of nonlinear time-series and sequences with recovery guarantees.

Abstract: In the wild, we often encounter collections of sequential data such as electrocardiograms, motion capture, genomes, and natural language, and sequences may be multichannel or symbolic with nonlinear dynamics. We introduce a method to learn low-dimensional representations of nonlinear sequence and time-series data without supervision which has provable recovery guarantees. The learned representation can be used for downstream machine-learning tasks such as clustering and classification. The method assumes that the observed sequences arise from a common domain, with each sequence following its own autoregressive model, and these models are related through low-rank regularization. We cast the problem as a convex matrix parameter recovery problem using monotone variational inequalities (VIs) and encode the common domain assumption via low-rank constraint across the learned representations, which can learn a subspace approximately spanning the entire domain as well as faithful representations for the dynamics of each individual sequence incorporating the domain information in totality. We show the competitive performance of our method on real-world time-series data with baselines and demonstrate its effectiveness for symbolic text modeling and RNA sequence clustering.

Published as a conference paper at ICLR 2025
NONLINEAR SEQUENCE DATA EMBEDDING BY MONO- TONE VARIATIONAL INEQUALITY
Jonathan Y. Zhou, Yao Xie
School of Industrial & Systems Engineering
Georgia Institute of Technology
Atlanta, GA 30332
jyz Jyz@gatech.ent edu, yao. xiexi weeeissyeggttcheeee...
ABSTRACT
In the wild, we often encounter collections of sequential data such as electrocar- diograms, motion capture, genomes, and natural language, and sequences may be multichannel or symbolic with nonlinear dynamics. We introduce method to learn oow-dimensional representations of nonlinear sequence and time-series data without supervision which has provable recovery guarantees. The learned repre- sentation can be used for downstream machine-learning tasks such as clustering and classification. The method assumes that the observed sequences arise from common domain, with each sequence following its own autoregressive model, and these models are related through low-rank regularization. We cast the problem as a convex matrix parameter recovery problem using monotone Variational Inequalities (VIs) and encode the common domain assumption via low-rank constraint across the learned representations, which can learn a subspace approximately spanning the entire domain as well as faithful representations for the dynamics of each individual sequence incorporating the domain information in totality. We show the competitive performance of our method on real-world time-serics data with baselines and demonstrate its effectiivneess for symbolic text modeling and RNA sequence clustering.
INTRODUCTION
Collections of time-series data, where each sequence is represented by series of points indexed over time, are ubiquitous and increasingly prevalent. Notable examples include physiological signals (Cohen 2014 Alday et al. 2020), power systems (Van Wijk & Van Sclow (999), financial data (Taa 2005 Min et al. (2021) computer networks (Basu et al.) 1996), and electronic health records (Reyna et al. 2020 Rasmy al. (02T). In addition to traditional time series data, other sequential
data like gene and protein sequences (Argelaguet et al.) 2020 Jumper et al.] [202] as well as natural
language have garnered significant attention, particularly with the advent of large language models
Brown et al. 1992 Peters et al. 2018 Reimers & Gurevych, 2019:; Cer et al. 2018).
Learning high-quality representations of sequences and time series (Mikolov et al. 2013 is an essential building block for understanding the dynamics underlying observed sequences, enabling informed decision making and downstream machine learning tasks (Trirat al.] 2024). key paradigm underlying the unsupervised representation learning has been that of self-supervised learning (Shwartz Ziv & LeCun (024)) where we first solve some auxiliary task (e.g., autoregression or reconstruction), which leads implicitly to compressed representation of the input data (Murphy) 2023; Kingma & Welling 2022). The development of self-supervised methods for natural language has been, in turn, paralleled by embedding methods for other types of sequential data, with there now being a burgeoning literature on time series and sequence representation (Lafabregue et al. 2022
Krishnan et al. 2022).
lasting challenge in bringing representation learning to time series is how to learn the information common to the entire domain in conjunction with faithful individual representations of each sequence. Indeed, when learning a model for time series or sequence data a common assumption to make is that the sequence observations arise from repeated realizations of single random source. While

---

Published as a conference paper at ICLR 2025
this is effective in the natural language setting where there exists a shared "universal" embedding space language (Yang et al. 2020) backed up by a large amount of available training data, many time series data are often highly domain-specttn (e.g., Flectrocardiogram (ECG) vs power systems). Sequences may be also highly distinct from one another (e.g., healthy vs sick patients) and our observations of each individual be partial or limited. To this end, recent empirical evidence indicates (Tan et al., (024) that augmenting time-series prediction with large language models results in performance no better than models trained from scratch, and removing the LLM components. For many settings, the individual processes which we receive observations for may fact also be substantially different among themselves (e.g.. differences between sick and healthy patients). There thus is challenge in balancing learning the common dynamics of a set of observed sequences in addition to faithfully representing the dynamics of each sequence. This is especially the case when observations of all sequences individually are limited or otherwise partially observed. To this end, we take inspiration from an area where the above challenges are common and well known low-rank matrix recovery (Davenport & Romberg 2016
in the sense that each domain is distinct from another
Candes & Tao 2010 Candes& & Plan 2010 Ahmeed & Romberg 2013) Juditsky & Nemirovski 2020)
previously applied to collaborative filtering problems and develop towards the general sequential and time series representation learning setting, enabling us to bring provable recovery guarantees to the modeling of broad class of sequences with autoregressive character.
To this end, we introduce an approach for unsupervised learning of low-dimensional representations for collections of nonlinear sequences, time-series, and dynamical systems based on the assumption that each sequence behaves according to its own autoregressive model but that the sequences are related each other through low-rank regularization. We cast the problem as computationally efficient convex matrix parameter recovery problem using monotone VIs. This formulation maintains problem convexity and recovery guarantees, while allowing for a broad range of autoregressive sequence dynamics through arbitrary monotone link function. By enforcing low-rank assumption across the learned representations we efficiently learn a subspace to capture entire domain. We apply method to real-world time-series data and demonstrate its effectiveness in symbolic text modeling and RNA sequence clustering. On many datasets, our method performs comparably to neural-network based deep representation models.
1.1 RELATED WORK
We review related work in time-series representation learning, clustering, and classification. Simple methods include feature extraction (Ye & Keogh 2009) or defining distance metric between time- series (Cormen et al., 2001; Müller] 2007| Bagnall et al. 2017). Another approach to model each series, which aligns with our model--asee representation approach (Smyth) 1996 Kalpakis cet al., 2001). Recent time-series representation learning methods often use contrastive learning to distinguish sequences, employing deep networks to treat sub-samples of the same sequence as positives and different sequences as negatives (Yang & Hong 2022 Yue et al., 2022; Xiao et al.
Fraikin et al.) 2024 Wang et al. (023). These approaches focus neural architecture, data
augmentation for robustness, and contrastive learning strategies (Ma et al. 2019 Fortuin et al. 2020
Devlin et al. 2019).
In our work, we adopt autoregression as the auxiliary task. Unlike techniques which use contrastive learning to indirectly learn an encoder for the latent space, we do not assume inherent similarities or differences across sequences. Instead, we explicitly constrain the representations to lie in a low-rank space. We motivate our work from the perspective of low-rank matrix recovery (Davenport & Romberg 2016). common in other areas of machine learning and serving as the foundation for principal component analysis (Hotelling, 1933). classical methods in natural language processing (topic modeling) (Blei et al.] 2003; Blei 2012) and collaborative filtering (Koren et al.) 2009). Problems in this area typically admit convex (Juditsky& & Nemirovski] 2020). Most recently, a line of work has signal recovery by convex optimization has loosened the structural assumptions needed for signal (time-scries) recovery in an autoregressive context while still maintaining problem convexity, using VIs with monotone operators the main tool (Juditsky et al. 2023 2020 Juditsky & Nemirovski 2019).
comveexformuleee formulations and come with provable recovery guarantees
The basic idea that sequences series can be represented in a -iw-dimensional space (c.g.. by latent factors) has a long history, such as hierarchical time-series models (Laird & Ware 1982] Gamerman & Migon 1993). More recently, Kirchmeyer et al. (2022) and Kostic et al. (2024) address
2

---

Published as a conference paper at ICLR 2025
dynamical systems learning, where either context latent vector aids in governing each sequence's dynamics, in contrast to directly learning an encoder from observations into latent space.
PROBLEM SETUP
We aim to represent observations into N vector-valued time series of length T each of the form (x,,)- where x E RR E [7]. and E [M] Here, we introduce the algorithm under the assumption that all sequences have the same length for notational clarity. However, the proposed results and algorithms can be extended to accommodate sequences that are partially observed or of varying lengths.
The sequences are sampled from common domain independently of each other across i, but have temporal dependence across We refer to the history of events for sequence up time-point as Hi. := (xi,* < 1). We expect the behavior at event Xi, to be function of past observations. Namely, at each time-point we suppose X 's dependence on its past values Hi, sufficiently captured by nonlinear vector autoregressive model of order d with C' channels. In particular, we package the preceding d observations with a bias term as a vector
E., vec(1, {Xi,==)=m1) E RC++1
where vec(.) arranges its arguments into single column vector so that E[x,,1 H4,ul = (R,E,.,), vi [N].
Note that we allow each sequence E [N] have its own own model. In particular, aim to learn matrices R. E CCC((1++1) which serve model parameters for prediction of the focal observation Xi,t- For each R,, we write b, = vec(R.) E TRm m == C2d + C as the corresponding parameter vector sufficient to capture the dynamics to the 8th sequence.
The structure of our model is inspired by Generalized Linear Models (GLMs) (Nelder & Wedderburn 1972). with a monotone link function é RC - RC that captures sequence dynamies. The choice of naturally corresponds to variety of different models and phenomena, for example:
Vector auto-regression: (xx) = x; x E C e.g. motion capture, Electrocardiogram (ECG) signals.
Symbolic sequences: o(x) exp(x)) ( Σ exp(x;): [Σ] e.g. natural language, genes.
Count processes: d(x) = exp(x):; x): x E C
20. egg. traffic intensity, call center arrival rates.
Bernoulli Processes: (xx) exp(x) (/)x))((11)) exp(x)); E BCC e.g. wildfire presence, neuron firing.
We do not restrict the mechanics of the link function e beyond the fact is monotonically increasing. We remark also that each vector b, corresponding to each sequence may itself be high dimensional.
The key aspect of our method for low dimensional representation learning lies the common domain assumption, which should limit how the sequences are similar (different) through their individual model. We leverage this information by a low rank assumption on the space of parameters by which each sequence is described. In this way, constrain the individual b, lie approximately on low dimensional linear subspace of the possible full parameter space R" The representation of each sequence's parameter within this subspace may be taken as a low-dimensional embedding and used for downstream tasks such as clustering, classification, and anomaly detection.
In particular, consider the autoregressive sequence model introduced in (T. allowing b, to be those parameters unique to the hd sequence. Allow the matrix = [b_
bx] E RMXN denote the parameters across all the sequences. We aim recover a good choice of the matrix B without supervision and balancing two goals: (I) we desire each b, to be as faithful to the generating dynamics of their respective observed data as possible; (2) we hope leverage the common domain assumption about the sequences and use information from the other sequences to inform the prediction of the focal sequence. To express the corresponding low-rank constraint, consider the rank r Singular Value Decomposition (SVD) of B
B= UEV* Σ VR
(2)
where Σ diag(ox) (ox) corresponds the singular values, columns of U jun]-_1 E R form orthobasis in I and columns of V [væ]k=1 E R form orthobasis in R.N The
3

---

Published as a conference paper at ICLR 2025
RR give an r-dimensional representation for each of the sequences. Likewise, the subspace sp(u+) describes the common domain from which the generating processes of the sequences arise. We consider the low dimensional representation c; for the 7th sequence b, = Uc, as an embedding of the dynamics for the 7th sequence.
recovered columns C ΣV-
[c
Because rank- optimization is in general NP-hard problem (Natarajan 1995 to enforce the low-rank requirement on B, we instead constrain our setup to nuclear norm ball. The nuclear norm given by | Σj=1 (X) where o; the 7th singular value of the matrix X. The nuclear norm is the tightest convex relaxation to matrix rank (Recht et al. 2010) leading tractable parameter recovery and allows us to leverage a long line of work from convex optimization and matrix recovery (Cai al. 20T0 Davenport Romberg 2 Nesterov & Nemirovski] 2013)).
We model each sequence as originating from an individual stochastic source, evolving according the parametric observation model of [ This model, detailed in Juditsky & Nemirovski (2020), balances the need for learned dynamics to closely resemble the original time-series observations (by using the most general convex model) with the requirement for efficient and identifiable parameter recovery via first-order methods (Facchinei & Pang 2003). The choice to make the observation model convex is motivated not only by parameter recovery guarantees regularity of the parameter space for low-rank estimation of sequence parameters in aggregate. By framing time-serics and sequence representation learning as a convex low-rank matrix parameter estimation task, our method is particularly well-suited for representation learning in contexts with limited, partially observed, and highly heterogeneous sequence data.
also by the need to ensure
3
METHOD
In the following, we present our method, first linear auto-regressive models and then for general non-linear auto-regressive models, including categorical sequences.
3.1 Low RANK TIME-SERIES EMBEDDING FOR LINEAR AUTO REGRESSIVE MODELS
First, suppose events X, E RC obey a linear autoregressive model corresponding to ① with ((x) = x. To recover the parameter matrix B = (e E R a natural choice is to take least squares loss and write
min BERdEN
Σ N
Σ t=d-1
X - (R,E,,
s.t.
BB|, S X.
(3)
where À 2 0 is a regularization parameter governing the rank of B.
Low-rank recovery and nuclear norm regularization. We now discuss Program (3) in the context of low-rank matrix recovery (Davenport & Romberg 2016). We aim to recover matrix B, but instead of observing it directly, we receive through indirect observations through successive linear measurement operators
A,B) == vec([R R + CCN
which at each time-step t, provide for the prediction of the states x,, across all observed sequences i [.]. We likewise define y; == vec([x,,,]\,,) E RRN to be the true values for the focal time-point across all N sequences. We then use all of the observed temporal slices of size d running up to time T, to and consider the least squares loss via the program
min {(B) N(T Σ t=1+1
[|a
yell3
S.I.
B S X
(5)
B
which is a Lipschitz smooth convex program the nuclear ball of radius À (Shapiro et al. 2021 Program (5) is exactly the same as Program 3 except placed in a matrix recovery context. We aim to recover the optimal B from the samples while accounting for the global structure. When À is arbitrarily large. there is no constraint on B and Program (5) corresponds to fitting each sequence individually with no global information. On the other extreme, forcing B to be rank constrains
4

---

Published as a conference paper at ICLR 2025
the models of each sequence to be multiples of each other. Intermediate values of À correspond to various trade-offs between learning common global structure and personalization to individual sequences.
Program (5) can be readily cast as a Semidefinite Program (SDP) solvable using standard interior point methods (aln-Tal & Nemirovski 2001). However, the size of B may reach into the hundreds of thousands of decision variables, we turn our discussion to efficient first-order algorithms (Combettes)
& Pesquet 2011
Parikh & Boyd 2014) analogous those for linear inverse problems (Beck & Teboulle 2009). Indeed, consider the following Proximal Gradient (PG) procedure
B = P (7h VB, (((x))),),,
Bo E {X
||||- S A},
(6)
consisting of a gradient step followed by proximal projection for an appropriately chosen sequence of steps } In this case, the prox-mapping associated with the indicator of the nuclear ball with radius À is the projection
Prox yx 6x (X) =
arg min
JX - 1
(E(YI
We can compute this projection mapping using Singular Value Thresholding (SVT), eliminating the small singular values (Cai et al. 2010 Nesterov & Nemirovski 2013). Namely, with X R=x"
first compute its SVD = U diag os
We then project the singular values onto a simplex
t = min mm
Σ (a.
Σ 1G=At20
The matrix projection can then be constructed by reapplying the singular vectors
Prox,16, (X) U diag(t)V".
In the worst case, Program requires that find ((m) singular values. However, it is typically the case that the parameter matrix has a decaying spectrum of singular values, and we take a small choice of X in (6) as ensure the representation is low rank. Thus, we typically need only to calculate a few largest singular values of B need, for instance via the processes (Gollb & Van Loan 1996). We compute singular values and vectors in incremental fashion, up to the yd singular value that still satisfies or > (- Σラ= Fÿ) (Duchi et al. 2008).
Saob--Kahan-Laaan
bidiagonalization
Then, the optimal solution to (7 is given in closed form by
o 1 (->+EZ=1))
i isr
0
i>r
In our real-data experiments, the iteration cost is typically dominated by the cost of computing the gradient, which can be mitigated by stochastic approximation.
3.2 NONLINEAR SERIES EMBEDDING BY MONOTONE VI
We now extend our discussion to the nonlinear case. Our goal is likewise form rank-constrained stochastic estimate to B when we recieve observations according to ① However, with arbitrary monotone link C + ICC the Least Squares (LS) approach outlined in 3 and (5) loses convexity and computational tractability in general. Likewise, parameter estimation by Maximum Likelihood Estimation (MLE) also becomes computationally difficult (Juditsky & Nemirovski 2020p. By contrast, we shall cast the parameter recovery problem into a monotone VI formulation, the most general type of convex program with known methods to efficiently find high accuracy solutions (Juditsky et al. 2023 Juditsky & Nemirovski, 2019: Juditsky et al. 2020).
Preliminaries on monotone VI. A monotone vector field on R. with modulus of convexity / is a vector field G Rm + R. such that
(G(x) G(x'), 20
B||x x'Il
Vx,x's E X
when 3 > 0, G is strongly monotone. For some convex compact set X C RM point x* is weak solution to the VI associated with (G, X) if for all x E X' we have (G(x), x - x*) 2 0. If strongly monotone and weak solution exists, then the solution is unique. When for all x E X, we term x" a strong solution to the VI. When G is continuous on X, all strong solutions weak solutions and vice versa (Facchinei & Pang 2003
0
5

---

Published as a conference paper at ICLR 2025
Monotone VI for nonlinear parameter recovery. We turn attention now the construction of a Monotone VI which has as its root optimal parameters corresponding to Model ( We will use the same operator A, from the linear case defined in 4 together with its corresponding adjoint A RCN Yt vec([x;, ) and maps them back to the parameter space. Note that A can be computed
+ I m which takes the pre-image of the multichannel predictions (observations)
using the below formula
(y) [voo[[ (eee, ee)))=))
RCN RMXN
where c2 is eth standard basis. The adjoint, for each entry and channel, multiplies the parameters by the value of the observation corresponding to the channel.
Consider the vector field the space of matrices
1 ( y)] RICU+CIXNN R(C4+0)
V (B)
Σ
(B)) AT
N(T - d)
(8)
t=++1
Σ AT)o(A(()) yv)-
N(T d)
t=d+1
where we extend the link function ei to act sample wise. Notice that the matrix B of true generating parameters a zero of V,
(B = E, (LAA 47
=E((A(((AA())) - (T( 0.
We note that both A, and A may be computed in time O(NC'd) In most of computations, we average across all available time steps giving a cost of O(TNC We also illustrate averaging instead using smaller random aab-windows of the data in Section 4.3 Analogous Program (5). the VI associated with (8) likewise admits solutions by PG. To illustrate, consider the recurrence
B Prox 5, (7h (B+)),
Bo E (X
X S A}
with step sizes {.x). Note d Id, the identity function, then vector field associated with the VI corresponds exactly to the gradient field of Program 5 In this case, (X) xx[[(X) and the PG procedures for VI and LS are the same.
First order methods for monotone To concretely solve the monotone VI outlined in (8. we detail extragradient scheme with backtracking for nuclear norm constrained VI in Algorithm of Appendix A which addresses the following general problem
BB).BBBB)
VB B
Bl| S {}}
where T an (unbiased estimator of a) k-lipschitz continuous monotone vector field as detailed in [usem al. (2019), and which addresses the difficulty that is in most cases is unknown to us beforehand. The convergence results for this class of algorithm are typical and presented in
(Tusem al. 2019 Gorbunov al. 2022 Korpelevich 1976). Namely, for e error as measured by
B, Proxx (B, requires iteration complexity on order of Ó outer iterations of the extragradient Algorithm The convergence of the algorithm applied to the vector fields given (8) particular may be established similarly to Juditsky & Nemirovski (2019), and when the data follow the true model (I) parameter recovery guarantees can be established similarly to Juditsky et al. (2020). In addition to the ordinary projection, proximal (Nesterov & Nemirovski] 2013)
Chen et al. 2017 Nemirovski 2004; Juditsky al.) 2011), and extragradient schemes, the program induced by & can be solved using projection-free methods like such conditional gradient (Frank- Wolfe) scheme, which requires only computing singular value/vector per iteration but converges at a slower rate as compared to projection/proximal based schemes and is less stable with respect to stochastic gradients (Hammond 1985).

---

Published as a conference paper at ICLR 2025
Parameter recovery for symbolic sequences. As special case of ( consider now that each channel represents the probability of emitting a token from syllabary {sc c=1 of size C. Then each X,,t represents a probability vector Ee Litt, = where Ef Hi,e = P( sc). We take the softmax activation function (y) = vec(Il eI
()) where y( corresponds to values from the 7th sequence. This problem corresponds to learning representations for different sequences. We illustrate Section 4.3 the above as applied to learning dynamics for genomics data and natural language, for which autoregressive models have become increasingly popular.
4 EXPERIMENTS
We first illustrate parameter recovery using synthetic univariate time-series in Section 4.1 We
investigate the choice of nuclear penalty À and the rank of the recovered parameter matrix relates to reconstruction quality. Section 4.2 describes benchmarks using real-world time-series data from the UCR Time Series Classification Archive (Dau et 2018). We report classification and runtime performance against number of baselines. Section 4.3 provides two illustrations on embedding of real-world sequence data. We first consider a language representation task where we embed without supervision series of excerpts taken either from the works of Lewis Carroll or abstracts scraped from arXiv (Carroll 1865 1871, Kaggle Team 2020). In the second illustration, we apply our method to the clustering of gene sequences for strains of Influenza A and Dengue viruses (Sayers)
et al. 2022). Appendix A provides implementations for solving the programs Sections 3.11 and and 3.2 pseudocode and the Julia programming language. The full experimental setup and results
detailed in Appendix
B
In Appendix A we discuss and provide implementations solve the programs described in Sections
and 3.2 and describe the experimental setup and results in detail in Appendix B
4.1 PARAMETER RECOVERY WITH SYNTHETIC AUTOREGRESSIVE SEQUENCES
To illustrate parameter recovery across autoregressive sequences, synthetically generated set of ten baseline parameters for linear autoregressive sequences of order d = 15. Within each class, we then created N 300 sequences of each type, perturbing the baseline coefficients by adding small amount of noise according to a fixed rule for the set of parameters. We then generated 7' = 250 observations for each sequence according to the autoregressive model in I with linear link function (xx) = x. We formed all 120 combinations of & 3 type of sequences from the ten classes and recovered the underlying parameter matrix by solving Program (5) to optimality. We report the data-generating procedure and experimental details in Appendix B.1 To further illustrate parameter recovery with a nonlinear link function, we provide an additional illusration using synthetic symbolic sequences in Appendix B.2
Table reports averages and standard deviations for the relative reconstruction error B B ||
the least squares error of the objective function given in (3) the Adjusted Rand Index (ARI) using k-means clustering with A: clusters in the embedding space across the runs, and the number of large singular values (singular values within 10-2 of the principal singular value). We first learn representations for sequences without nuclear norm constraint.
To illustrate the low-rank matrix recovery, we search across the values of nuclear regularization via Brent search (Brent 2002) and report the performance for close to the optimal value of with respect to the reconstruction eтror. When the underlying dynamics share a common low-rank structure. the nuclear constraint effectively leverages the common information shared across the sequences to recover the true parameters more faithfully using the common domain information with comparable error the least squares sense. Furthermore, we observe the nuclear regularization procedure driving the singular value spectrum to sparsity, with the number of large singular values being much smaller than in the unconstrained case.
To further illustrate, Figure I depicts parameter recovery for collection of sequences with autore- gressive order d = 15. In the leftmost pane, we report the relative reconstruction error and the number of large singular values across different values of nuclear constraint X. In the central pane, we depict the singular value spectra of the true parameter matrix (which has approximately low-rank structure plus noise) and recovered matrices with differing numbers of large singular values. In the third
7

---

Published as a conference paper at ICLR 2025
Table 1: Time series parameter recovery for synthetic autoregressive time-scries.
Relative Err.
LS Err.
Cluster
ARI
Rank
À Selection
Avg.
Std.
Avg.
Std.
Avg.
Std.
Avg.
Std.
Unconstrained
0.341
(0.016)
79.333
(0.169)
0.967
(0.049)
14.442
(1.203)
arg min,
[B
B
0.158
(0.020)
80.709
(0.233)
0.997
(0.008)
7.392
(4.255)
Second Phincipal Component: 6,
0.15
13
0.30
12
0.10
10
Rank
Specitumoof
0.08
0.25
0.00
0.20
2"
-0.10
17.5
20.0
22.5
25.0
1.0
15
0.2
0.1
0.0
0.1
0.2
=2 (Rank(8a)
First Principal Component: B3..
Figure 1: Simulation results: Parameter recovery for collection of univariate time series drawn
from & = 3 classes. Left: Relative reconstruction eπror and approximate rank of recovered parameter matrices across levels of nuclear constraint A. Center: Singular values of the true parameter matrix B and the singular values of the recovered solutions of varying dimensions. Right: First two principal components of the recovered matrix with the smallest reconstruction error and original class labels.
pane, we show the first two principal components (total explained variance = 0.868) of the sequence embedding with the smallest reconstruction error along with the original three generating classes of the data. With the introduction of sufficient nuclear regularization, we drastically reduced the reconstruction error our recovered solution and observed that the solutions with low reconstruction error were of approximately low rank. In the plots of the singular value spectra and the projection of the learned sequence embeddings, we observe that those consttriid recoveries effectively recover those large singular values in the spectrum of the true parameter matrix. By contrast, the parameter recoveries performed without or insufficient nuclear constraint fit the noise component of the data, evident in the distribution of singular values.
4.2 REAL TIME-SERIES CLASSIFCATION
Following (Middlehurst et al. 2024 Ma al. 2019 Yue et al., 2022 Zakaria et al. 2012), conduct experiments on 36 UCR time series datasets (Dau et al.) 201 2018). Dataset statisties are provided in Appendix [B.3.1] with each dataset using its default train/test split. Each time series is re-encoded as a multichannel signal comprising the original signal and its first finite difference. We embed the data without supervision by solving (8) using the extragradient scheme given in Agooithmm]or Appendix A link function. The value of is selected via a two-step process: first, bisection identifies when the solution becomes rank-one, and then grid search refines the choice for aankcoontrriied parameters. We report results for the À value with the best training performance. Evaluation metrics include ARI (Hubert & Arabie 1985). Normalized Mutual Information (NMI) (Vinh et al.) (2009). macro-FI score (Fawcett (206)). accuracy on the test set, and average runtime, including the full grid search and SVD at each step. Runtime improvements using partial SVD computed up to required threshold À are discussed Appendix B.8
with look-back length of = 20. running the algorithm for 256 steps using linear
We compare our method with five represennative time series embedding and classification methods: K-nearest neighbors (KNN) using Euclidean (f2) distance (Cover & Hart 1967), KNN with Dynamic Time Warping (DTW) the distance metric (Müller) (007). DTa (another method based on DTW but with additional features) (Zakaria et al. | [0122] Ye & Keogh 2009). a dictionary- based method MultiROCKET-HE Hydra (Dempster et al. 2023),
deep representation method based on contrastive learning (TS2Vec) ( (Yue et al. 2022) and one based on masked modeling (Ti-MAE) (Cheng
et al. 2023). In line with Yue al. (2022) Franceschi al. (2019), evaluate the classification
8

---

Published as a conference paper at ICLR 2025
Table 2: Time series classification performance on UCR time series data of our method vs a number of baselines (higher is better, except for runtime). We outperform simple approaches and perform close to classification using the embeddings from the neural network based Vee but use only 37% of the runtime. The best performing method, MR-Hydra, is a ensemble based on handpicked features tuned specially for time series classification and does not produce latent embeddings.
ART
NMI
F1
Accuracy
Runtime (Sec)
Method
Avg.
Std.
Avg.
Std.
Avg.
Stu.
Avg.
Std.
Avg.
Std.
E2KKNN
0.422
(0.294)
(0.290)
0.752
(0.144)
0.725
(0.166)
0.128
(U.413)
DIW+KNN
0.447
(0.303)
0.435
(0.300)
0.766
(0.150)
0.738
(0.170)
42.988
(134.320)
shapeDDW
0.470
(0.307)
(0.299)
0.773
(0.152)
0.746
(0.178)
21.871
(71.655)
TiMAE
0.461
(0,275)
0.457
(0.274)
0,763
(0.135)
0.723
(0.170)
1275,712
(1161,375)
TS2Vee
0.606
(0.282)
0.580
(0.287)
0.840
(0.138)
0.814
(0.178)
1,085,092
(1,408.458)
Ours
0.602
(0.282)
0.562
(0.293)
0.817
(0.180)
0.788
(0.193)
400.031
(677.486)
MR-Hydra
0.682
(0.273)
0.656
(0.285)
0.877
(0.121)
0.851
(0.162)
10.197
(16.459)
performance on test set for methods which produce embeddings (TS2Vec, Ti-MAE, and our method), we perform vross-validated grid search across KNNs with &: = (2 i E [0. 4]} neighbors or SVMs with RBF kernels with penalty values c {2' E 1-10, 15]} U co. We defer all further details of our experimental setup to Appendix B To further compare the quality of representations, we provide in Appendix B.5 projections of the learned latent space for various UCR datasets. Table displays the mean and standard deviation across the metrics across the datasets. We provide the detailed results per dataset in Tables and Appendix B.3 [.33 We observe superior performance baseline methods based on distance metrics, such as Euclidean distance or DTW, and observe performance between that of TiMAE and TS2Vec. We note that for this class of sequence, the heuristic dictionary-basee ensemble (MR-HYDRA) outperforms both our approach and the seee-learninggbaee -baned approaches. However, this method has been tuned specifically for this type of classification problem. By contrast, similar to Yue et al. (2022); Cheng et al. (2023), we consider classification only as one potential downstream task.
4.3 SYMBOLIC SEQUENCES: LANGUAGE AND GENOMICS
Symbolic sequences and language: arx abstracts or " "Alice in Wonderland"" To illustrate the capability of our method to learn meaningful representations for sequences with nonlinear dynamics, we first consider an autoregressive language modeling task, drawing textual sequences from three sources: two works by the same author Lewis Caroll Alice's 's Adventures in Wonderland (n 228)
(Carroll, 1865) and Through the Looking Glass (n 316) (Carroll [871)- - and machine learning related abstracts scraped from arXiv (n 600) (Kaggle Team 2020) (details in Appendix (B6). We embed the sequences without supervision with a lookback of = 75, and in order to reduce the number of symbols in our alphabet and avoid the blowup in the number of channels, we converted each of the sequences into a c= = 4 symbol code via Huffman coding, hased on the overall frequencies of letters in the English language (Cover & Thomas 2005). ) We then solve Program induced by (8) using the multichannel measurement operator given in 44h to optimality and using the softmax activation discussed in Section 3.2 We show in Figure 2a the space learned when was chosen to be sufficiently small as to give a rank three representation of the data. We then project the learned representation via Uniform Manifold Approximation and Projection (UMAP) (Melnnes et al. 2018). Two distinct clusters form corresponding to the two different genres of writing, however, whereas the paper abstracts are clearly separable from the works of Lewis Caroll, the two books written by him are not as clearly disambiguable as they are from the same author.
Virus strain identification from genome sequences For the final illustration, in line with Mil- Ian Arias et al.) (2022), the problem of classifying genetic sequences, which allows for the plac- ing of species/variants in the evolutionary context of others (phylogenetic classification). We consider gene sequence data from segment six of the Influenza A virus genome (n 949, average sequence length = 1409) (Bao et al. 2008) and the complete genome the Dengue virus (n 1633, average sequence length = 10559) (Hatcher et al. 2017). We consider gene sequences from five strains of Influenza and the four strains of Dengue. Likewise, we provide a detailed overview of the data and learning procedure in Appendix B.7 We encode the genomes in a similar manner
9

---

Published as a conference paper at ICLR 2025
Alci's Adrmrann in Wanderiand
Soore UnDU Gonoreet i
Thrugh the usaking Glam
Secon UIWO Companmen
Sectrd UWAP Comoonee
70
DE
DEw-
H1N33
ID14 +LML
1262
DEW-3
Hio
DEW?
UMAP
Firt USAP Componart.
Firtt UMAP Component
(a) Lewis Caroll or ArXiv abstracts? (b) Clustering strains of Influenza A (c) Clustering using full genome for the four strains of Dengue Virus
Genres form clusters.
virus genome data (segment 6)
Figure 2: Learned embeddings for symbolic sequences collections using our method- visualized by UMAP projections shows clear groupings based on sequences with similar underlying dynamics.
as for the natural language illustration, assigning one channel to each nucleotide (A, C, G), and encode the presence/absence of each nucleotide at each position via one-hot encoding.
To recover the embedding. we adopt the same softmax activation scheme as described in Section 3.2 Since the genomes are of variable length, we consider stochastic approximation to the monotone field V (8) by taking the sample average of randomly selected length 800 sub-windows from each of sequences at each training step. We consider clustering the Influenza and Dengue genome segments individually and report UMAP projections of the learned representations in Figure 2b and 2c] respectively. The dimensions of the learned embeddings these subspaces, we note the clear grouping viral strains obtained via solving the stochastic approximation the VI (8)
and 20, respectively. In
5 DISCUSSION
We propose method to learn embeddings for sequences and time series by framing it low-rank matrix recovery problem cast into a VI form. This approach is particularly amenable to settings with partial or limited observations, allowing similar sequences inform the representation of focal sequence. Each sequence is modeled with its own autoregressive process through a monotone link function. Our observation model both a strength and limitation: on one hand is as general as possible while still maintaining convexity, and thus flexible enough to handle number of diverse scenarios notably probabilistie modeling symbolic data and demonstrates empirical performance comparable to methods based on contrastive learning and masked modeling. On the other hand, reliance on convexity to ensure regularity and identifiability limits its ability to capture highly non-convex structures and provide universal approximation guarantees. Our method performs well under low-rank and monotonicity assumptions, is sample-efficient and is faster in limited-data settings, shown in most cases. However, its performance declines when these assumptions violated, seen in certain UCR datasets, where may be outperformed by energy- based approaches in data-rich scenarios. Future work could explore alternative objectives within the VI framework and non-convex extensions to address these limitations.
ACKNOWLEDGMENTS
This work is partially supported by NSF
REFERENCES
Ali Ahmed and Justin Romberg. Compressive multiplexing of correlated signals. IEEE Transactions on Information' Theory, 61(1):479 498, 2015. doi: 10. 00.1009/11.200142366535
Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Qiao Li, Ashish Sharma, Gari D Clifford, and Matthew A Reyna. Classification of 12-lead ecgs: the physionet/comppting in
10

---

Published as a conference paper at ICLR 2025
cardiology challenge 2020. Physiological Measurement, (112:/mmmmmmmm4111100000000000 12 2020. doi: 10.1088/ 1361-6579/abc96..
R. Argelaguet, D. Amol, D. Bredikhin, Y. Deloro, B. Velten, J.C. Marioni, and O. Stegle. Mofa+: statistical framework for comprehensive integration of multi-modal single-cell data. Cenome Biology, 2111):111, 2020.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 31(3):606--6 660, 2017. doi: 10.1007/s10618. 1-0006681----------
URL |tt.L/////////...oo/////////0000
Yiming Bao, Pavel Bolotov, Dmitry Dernovoy, Boris Kiryutin, Leonid Zaslavsky, Tatiana Tatusova, Jim Ostell, and David Lipman. The influenza virus resource at the national center for biotechnology information. J. Virol., (2(2):596-601) January 2008.
Basu, A. Mukherjee, and S. Klivansky. Time series models for internet traffic. In Proceedings of IEEE INFOCOM '96. Conference on Computer Communications, volume 2, pp. 1996. doi: 10. 109/NFCOM. 190/INFFCM..999.49335555
vol.2,
Amir Beck and Mare Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal Imaging Sciences, 2(1):183-202, 2009. doi: 10.1137/0807 16542.
URL httph:ttp///iii wtt.:/o.org111111///1111100
Aharon Bel and Arkadi Nemirovski. Lectures Modern Convex Optimization. MOS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics, 2001.
David M. Blei. Probabilistic topic models. Commun. ACM, 55(4):77-84, 4 2012. ISSN 0001-0782. doi: 10.1 2111145//133880.213333333
URL http:tts///io..r//////// wttp://///...1//13333333333 211.33333666.182222222 2133826
David M. Blei, Andrew Y. Ng, and Michael Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993 1022, 2003. ISSN 1532-4435.
R. P. Brent. Algorithms for minimization without derivatives /. Dover Publications, Mineola. N.Y. 2002. URL
2001 047449..ttmm
Peter F. Brown, Vincent Della Pietra, Peter V. deSouza, Jenifer C. Lai, and Robert I.. Mercer. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467-480 1992. URL httpt::t/tcaaattttttt
Jian-Feng Cai, Emmanuel J. Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20(4):1956-198) 2010. doi: 10.1137/ 080738970. httptttts///....... http://i..oorggg//// 01tp//000..11770773887777777
Emmanuel J. Candes and Yaniv Plan. Matrix completion with noise. Proceedings of the IEEE, 98(6):
925-936, 2010. doi: 10. 00.100//PRCC..200990037777
Emmanuel J. Candes and Terence Tao. The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory, 56(5):2053-208)) 2010. doi: 10. 109/TTT. 2010.2044061.
Lewis Carroll. Alice's Adventures in Wonderland. Macmillan Publishers, 1865.
Lewis Carroll. Through the Looking-Glass and What Alice Found There. Macmillan Publishers, 1871.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedeer Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. Universal sentence encoder for English. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 169-174, Brussels, Belgium, November 2018, Association for Computational Linguistics. doi: 11L155//1/D11-2222 URL wttpt:tt:///.t. cattps://acll antho ont.aaooooooooooo//1---0-00000
11

---

Published as a conference paper at ICLR 2025
Yunmei Chen, Guanghui Lan, and Yuyuan Ouyang. Accelerated schemes for class of variational inequalities. Mathematical Progranmingg 165(1):113-44, 149, doi: 10.100775101007-111111111 4. 00t.a//tss///eaa..oooo//////0//0
Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv preprini arXiv:2503.00:320, 2023.
Mike X Cohen. Analyzing Neural Time Series Data: Theory and Practice. The MIT Press, 01 2014. ISBN 9780262319553,, doi: 10.755 1 http///mmeeee/99...////// 00..0000.000... URL https://doi org/10
7557/mitpress//660 001, 00000001100001110000000000000010
Patrick L. Combettes and Lean-Christophe Pesquet. Proximal Splitting Methods in Signal Processing.
Springer New York. New York, NY, 2011. ISBN 978- 978-1-441999956999 doi: 10. 10.1007/978- 111199788--41199 1-4419- 9569-8 10. URL https: doi wttp://o./.......111 /t0p://////.....or00...010/1//11
9569-
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms. The MIT Press, edition, 2001.
Athel Cornish-Bowden. Nomenclature Committee of the International Union of Biochemistry (NC- IUB). Nomenclature for incompletely specified bases in nucleic acid sequences. Recommendations
1984. Biochemical Jourmal, 229 286, 07 1985. ISSN 0264-6021. doi: 10.1042/bj2290281.
URL https: /tteps://ooi.e .rrg 100.11042/bbb22
Cover and Hart. Nearest neighbor pattern classification. IEEE Trunsactions on Information Theory, 13(1):21-27, 1967. doi: 10.1 111.///TT11966.13366666
Thomas M. Cover and Joy Thomas. Data Compression, chapter 5, pp. 103-158. John Wiley Sons, Ltd, 2005. ISBN 9780471748827 doi: http::///ooi.org111000000000002/ https://on inel werirrrrr... wwraar..wwii.i.. http://ly..cc.///////////1/1111/
URL
ch
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi. Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML The ucr time series classification archive, 10
2018. https://wwwt ww..s...................
Mark A. Davenport and Justin Romberg, An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing, 10(4):608 June 2016. ISSN 1941-0484. doi: 10.1 00.1109/sssp222112555000000 URL http:/ dx dow. 1109/
USTSP 2016 02116.253999 2539100
Angus Dempster, Daniel F. Schmidt, and Geoffrey I. Webb. Hydra: competing convolutional kernels for fast and accurate time series classification. Data Mining and Knowledge Discovery, 37(5): 1779-1805, 2023. doi: 11100///1011--02300000000 s10618- 023- 00939-0033333
URL ht..htts:: http://o.oorg/..11111111
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguisties. doi: (0.18653/v//N11-44402 URL/https: http:/::///aaaathhllcccc www.ahooooooogg/ooo....
John Duchi, Shai Shalev-Shwarde Yoram Singer, and Tushar Chandra. Efficient projections onto the 11-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, ICML, '08, pp. 272-279, New York, NY. USA, 2008. Association for doi: 10.1145/1390156 1390191. URL https
Computing Machinery. ISBN 9781605582054,
Francisco Facchinei and Jong-Sin Shi Pang. Finite-dimensional variational inequalities and complemen- tarity problems, volume of Springer series in operations research. Springer, New York, 2003.
ISBN 9780387955803.
12

---

Published as a conference paper at ICLR 2025
Tom Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861-8744 2006. ISSN 0167-8655. doi: http://koa.ar////ooooooprrrrroo0 URL https:/ /ttps://ww/ sciencedirect. http:////...........a//////////.
ROC Analy-
sis in Pattern Recognition.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilistic time series imputation. In International conference artificial intelligence and statistics, pp. 1651-1661. PMLR, 2020.
Archibald Felix Fraikin, Adrien Bennetot, and Stephanie Allassonniere. T-rep: Representation learn- ing for time series using time-embeddings, In The Twelfth International Conference Learning Representations, 2024. URL
meee:/oorummdd=32215... 966N
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. "'Ach--BBcc
E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: http:s:://ppoocediiggseeeeee neur wttp:/iipppcccpppaaaae wwwpaper.pppppppee files/
Dani Gamerman and Hélio S. Migon. Dynamic hierarchical models. Journal of the Royal Statistical Sociery. Series B (Methodological), (5(3):629-642) 1993. ISSN 00359246. URL //wt/ www
Gene H. Golub and Charles F. Van Loan. Matrix Computations. The Johns Hopkins University Press, third edition, 1996.
Eduard Gorbunov, Nicolas Loizou, and Gauthier Gidel. Extragradient method: O(1/k) last-iterate convergence for monotone variational inequalities and connections with cocoercivity. In Gustau Camps- Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 366-402. PMLR, 28-30 Mar 2022. U
mlr. wttp::///11.//ggooooo2222222
Janice Hammond. Solving asymmetric variational inequality problems and systems of equations with generalized nonlinear programming algorithms. PhD thesis, Massachusetts Institute of Technology, Aug 1985.
Eneida L Hatcher, Sergey A Zhdanov, Yiming Bao, Olga Blinkova, Eric Nawrocki, Yuri Ostapchuck, Alejandro A Schaeffer, and J Rodney Brister. Virus variation resource improved response to emergent viral outbreaks. Nucleic Acids Res., 454D1):1482--499, January 2017.
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology, 24(6):417, 1933.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classificatiio, 193):193-218 1985. doi: 10. 10.100//F01900075. URL fattps://doo www.oo.goo....o.oooooooooo.....o 1007//P019080755
Alfredo N. Jusem, Alejandro Jofré, Roberto I. Oliveira, and Philip Thompson. Variance-based extragradient methods with line search for stochastic variational inequalities. SIAM Journal on Optimization, 29(1):155-266, 2019, doi: N0.1137/1MM1144999 URL / 1137/17M11423 44799
A. B. Juditsky and A. S. Nemirovski. Signal recovery by stochastic optimization. Automation and Remote Control, 80(10):188--189 2019. doi: 00.113//0000111199990000 | https
//doi.org//// 1ttp://0...113/00011111111111111
Anatoli Juditsky and Arkadi Nemirovski. Solving variational incqualities with monotone operators
domains given by linear minimization oracles. Mathematical Programming, 156(1):221-255))
2016. doi: 10. 10,10077//00177000--0077333 URL.
Anatoli Juditsky and Arkadi Nemirovski. Statistical Inference via Convex Optimization. Princeton University Press, 2020. ISBN 9780691197296. URL hwl.hhtpp//// www. jst.rsstooo... westor.orrile wtt/stable//
ctvasdxgd
13

---

Published as a conference paper at ICLR 2025
Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1(1):17-58, 2011. doi: 10.1287/10-SYY011 URL JtRppss:s///o..ooo http://66...011111112//////00000
Anatoli Juditsky, Arkadi Nemirovski, Liyan Xie, and Yao Xie. Convex parameter recovery for interacting marked processes. IEEE Journal on Selected Areas in Information Theory, 1(3): 799-813, 11 2020. ISSN 2641-8770. doi: 10.1100/SAAT.220000000000
Anatoli Juditsky, Arkadi Nemirovski, Yao Xie, and Chen Xu. Generalized generalized linear models: Convex estimation and online bounds, 2023.
John Jumper; Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with doi: 10. 10.103844415--0--0033---- URL https: //d::///oo...o www.ioorg/11..0 1038/ /455//4115555 021-03819-2
alphafold. Nature, 596(7873):583 589, 2021.
Kaggle Team. 2020. URL http:://wwoo.aaaar....oo////////
Leveraging MI. to Fuel New Discoveries with the arXiv Dataset, machine
learning- to- fuel new
new discoveries with the wxxii-ddtttser
K. Kalpakis, D. Gada, and V. Puttagunta. Distance measures for effective clustering of arima time- series. In Proceedings 2001 IEEE International Conference on Data Mining, pp. 273-280, 2001. doi: 10.1109/CDM..2001.8888
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2022.
Matthieu Kirchmeyer, Yuan Yin, Jeremie Dona, Nicolas Baskiotis, Alain Rakotomamonjy, and Patrick Gallinari. Generalizing to physical systems via context-informed dynamics model. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1283-11301, PMLR, 17-23 Jul 2022. URL https: pttps://proceeccingg mlr press 62 wttp///kiinneeeeeeaeeaaaaaaaaa html
Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009. doi: 10. 109/MMC2009.225
G.M. Korpelevich. An extragradient method for finding saddle points and for other problems. Ekonomika Matematicheskie Metody, 12(4)7747-756, 1976.
Vladimir R Kostic, Pictro Novelli, Riccardo Grazzi, Karim Lounici, and massimiliano pontil. Learning invariant representations time- haee--hoogeneouss stochastic dynamical systems. In The Twelfth International Conference Learning Representations, 2024. URL htRL:httts/:/peeeee
net/ wwt::ttttttwwi...m.mm
Rayan Krishnan, Pranav Rajpurkar, and Eric Topol. Self-supervised learning in medicine and healthcare. Nature Biomedical Engineering, 6(12):1346-13522 2022, doi: 10. 1533//4555---02-- 00914-1.
Baptiste Lafabregue, Jonathan Weber, Pierre Gançarski, and Germain Forestier. End-to-end deep representation learning for time series clustering: comparative study. Data Mining and Knowledge Discovery. 2022. doi: 10. 0000///011010211007777007
URL https:/ stttss://do1 org
10.1007/310618-022- 0C7-66
Nan M. Laird and James H. Ware. Random-effect models for longitudinal data. Biometrics, 38 (4):963 974, 1982. ISSN 0006341X, 15410420. URL http: /wwww jstor. atg/stable/ 2829
14

---

Published as a conference paper at ICLR 2025
Qianli Ma, Jiawei Zheng, Sen Li, and Gary W Cottrell. Learning representations for time ries clustering. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL http:httpps//prrrceeennnsseeeeee 201 htt.:///..........000..022222210
fi Seteerr__ilssapappeeee les/paper)
9aa
Leland McInnes, John Healy. Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. Joumnal of Open Source Software. 3(29):861, 2018. doi: 10.21 105/
joss.00861. htt.htttps///oi.ooo. http:://ooooorgo/..ooo 20100.0111111111 21105/ joss. 00861
Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. Bake off redux: a review and experimen- tal evaluation of recent time series classification algorithms. Data Mining and Knowledge Discov- ery, 2024. doi: 10. 024 01022-1
URL httpt:tts///att http://tt..rgg.....
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C.J. Burges, L.. Bottou, M. Welling, Z. Ghahra- mani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL
neurips wwwpppppp.pppppp.pppppppppppppep wwtpse00l---aprr....ee pdf
Ei miles/paper/20133 ftt///il//////ee 9aa 233
Pablo Millan Arias, Fatemeh Alipour, Kathleen A. Hill, and Lila Kari. Delucs: Deep learning for unsupervised clustering of dna sequences. PLOS ONE, 17(1):1-25, 01 2022. doi: 10. 1.jurnlljuunal..
pone.0261531. URthttss: wrg.:oi...rrglol 11711 wtwp///ouoaaalaaa otrp:////oooooooooo//////////rar pone. 0261531
Wei Min, Weiming Liang, Hang Yin, Zhurong Wang, Mei Li, and Alok Lal. Explainable deep behavioral sequence clustering for transaction fraud detection. CoRR, abs/2101.04285, 2021. URL
https://axxils httpgggab///11011 /22p:p/////xxx.orrr////////....0 04285
Meinard Müller. Dynamic Time Warping, pp. 69-84. Springer Berlin Heidelberg, Berlin, Heidelberg.
2007. ISBN 978-3-540-74048-33 doi: 10, 10.1007/978-3-540-444444
URL WRLLhttps:////
org 000.1007/50
3_4
Kevin Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
http://probnn: http://.......uu.....ooooooo ic/book2
B. K. Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24 (2):227-234, 1995. doi: 10.11177500997539999220000
SDC97539792240400
J. A. Nelder and R. W. M. Wedderbumn. Generalized linear models. Journal of the Royal Statistical Sociery. Series (General), 135(3):370 384, 1972. ISSN 00359238, 23972327. URL http:
jator wwt.ooogg///ooo. http//ttbbee224441111 2344614
Arkadi Nemirovski. Prox-method with rate of convergence o(1/t) for variational inequalities with lipschitz continuous monotone operators and smooth convex-concave saddle point problems.
SIAM Jourmal on Optimization,, 15(1):229 251, 2004. doi: 10. 10.11377510522224044466222 URL
ottps::/doi.oog
Yurii Nesterov and Arkadi Nemirovski. On first-order alg algorithms for 11/nuclear norm minimization. Acta Numerica, 22:509-575, 2013. doi: 2..1077/096224921100000
Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim., 1(3):127-239, 1 2014. ISSN 2167-3888. doi: 10.1561/24000000003 URL hattps://dol.one 24C0000003
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). pp. 2227-2237 New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10. 2011886511/// 1/N18 1202. URL wwt.hhttt///// http://o/.caaatooogoo.oooooo www.anoogoooogg//..ooo org 82222
15

---

Published as a conference paper at ICLR 2025
Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. upj Digital Medicine, 4(1):86, 2021. doi: 10. 10.1038//414660021-00000 1038/s 1746 021 000455-
URL https://66...rg/////
Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3)471-501, 2010. doi:
10.1137/0706978355. URL
Nils Reimers and Iryna Gurevych. Sentence-BERT. Sentence embeddings using Siamese BERT- networks, In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP) pp. 3992-3992, Hong Kong, China, 11 2019, Association for Computational Linguistics. doi: 10. (0.18653/11/D19--1
URL
https:
org/D19 1410
Matthew A Reyna, Christopher S Josef, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Shamim Nemati, Gari D Clifford, and Ashish Sharma. Early prediction of sepsis from clinical data: The PhysioNeu/ompputng in cardiology challenge 2019. Crit. Care Med., 48(2):210-217, February 2020.
Eric W Sayers, Evan E Bolton, Rodney Brister, Kathi Canese, Jessica Chan, Donald C Comeau, Ryan Connor, Kathryn Funk, Chris Kelly, Sunghwan Kim, Tom Madej, Aron Marchler-Band Bauer, Christopher Lanezycki, Stacy Lathrop. Zhiyong Lu, Francoise Thibaud-Nissen, Terence Murphy, Lon Phan, Yuri Skripchenko, Tony Tse, Jiyao Wang, Rebecca Williams, Barton W Trawick, Kim D Pruitt, and Stephen T Sherry. Database resources of the national center for biotechnology information. Nucleic Acids Res., 50(D1):D20-DD January 2022.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures Stochastic Program- ming: Modeling and Theory, Third Edition. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2021. doi: 10.113771.97716111 178116197619955 BRL.httss://pplle
si org/
1137/1
Ravid Shwartz Ziv and Yann LeCun. To compress or not to semppress--sff--uueeeeeeeee learning and information theory: A review. Entropy, 26(3), 2024. ISSN 1099-4300. doi: 10.3390/e22030225.. URL https:://wwtt http://wwwwwppp...co wwwppmipmmmmmmmmmm.mmm.... ctc.pp..cer com. /ttp://9999939233//////333
Padhraic Smyth. Clustering sequences with hidden markov models. In M.C. Mozer, M. Jor- dan, and T. Petsche (eds.), Advances in Neural Information Processing Systems, volume 9, MIT Press, 1996. URL htt_iitpee//ppocceengss.co
http://pcpppppppppeeeeeaeeaaaaaa files/paper/
Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I. Webb. Multirocket: multiple pooling operators and transformations for fast and effective time series classification. Data Mining
and Knowledge Discovery, 36(5):1623-16444 164666663666666666 2022. doi: 10.1007/s10611--2220064444 1ttp://///..../11..111011//////0 1111://0010001111111111111110111 022 02000022200002222
URL
https: / /oi. org/ wttp://...111/111111
Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. Are language models actually useful for time series forecasting?, 2024. URL httttttt//aaxii.....m org abs/
2406. 16964
Patara Trirat, Yooju Shin, Junhyeok Kang, Youngeun Nam, Jihye Na, Minyoung Bae, Joeun Kim, Byunghyun Kim, and Jae-Gil Lee. Universal time-series representation learning: A survey, 2024.
Ruey S. Tsay. Analysis of financial time series. Wiley series in probability and statis- tics. Wilee-Interscience, Hoboken, NJ, 2. ed. edition, 2005. ISBN 978-0-471-69074--
0. URL http://gso. gbv.
www.ggpww...ee... de/DB=e C/C TAATAAAATTSCCCAAAET--O00-------- ottp://pw..................aaaaa
TRM
htwp://wwwaaaaaaaooo.oo..oo.ooo. bibsonomy.
J.J. Van Wijk and E.R. Van Selow. Cluster and calendar based visualization of time series data. In Proceedings 1999 IEEE Symposium on Information Visualization (InfoVis '99), pp. 4-9, 1999. doi: 10.1 109/NEVIS. 1999,801899
16

---

Published as a conference paper at ICLR 2025
Nguyen Xuan Vinh, Julien Epps, and James Bailey, Information theoretic measures for clusterings
comparison: a correction for chance necessary? In Proceedings of the 26th Annual International
Conference Machine Learning, ICML '09, pp. 1073-1080, New York. NY, USA, 2009. Associ- ation for Computing Machinery. ISBN 978 605585161 doi: 10.114451553337415555511 URL
111111///533333 https://doi: /t.p.//.....11/1333334..1.1 1553511
Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang. Contrast everything: A hierarchical contrastive framework for medical time-series. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
net/ forum wer.m s Cmzp)
Zhiwen Xiao, Huanlai Xing, Bowen Zhao, Rong Qu, Shouxi Luo, Penglin Dai, Ke Li, and Zonghai Zhu. Deep contrastive representation learning with self-distillation. IEEE Transactions Emerging Topics in Computational Intelligence, 8(1):3-15, 2024. doi: 10.1 109TETTCL2223. 3304948.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear -emporal-spectral fusion. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25038-25054 PMLR, 7 2022. URL wtRL:tttss///prceed.....
mlr prwss/ 62 yang:/ann222eeemm html
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo. Jax Law, Noah Constant, Gustavo Hernan- dez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual universal sentence encoder for semantic retrieval. In Asli Celikyilmaz and T'sung-Hsien Wen (eds.), Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics: System Demonstrations, pp. 87-94, Online, July 2020. Association for Computational Linguistics. doi: 10.18655/v1/2020:cc--dmmm 12. URL httss://aclamthhooooy.ooooo// www.ttttww.oooyyogggo0ooooooo
demos.
Lexiang Ye and Eamonn Keogh. Time series shapelets: a new primitive for data mining. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09, pp. 947-956, New York, NY, USA, 2009, Association for Computing Machinery. ISBN 9781605584959,
doi: 10.1145/155001915555222 URL httpttt////ddo org/
01.14451155019.1155 1557122
Zhihan Yue, Yujing Wang, Juanyong Duan, Tiammeng Yang. Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. Proceedings the AAAI
Conference on Artificial Intelligence, 36(8))8980 6 2022. doi: 10.1609/aaii. v36.8 20881.
Utt.tttts::///o. aaai wawaai.orrggoooo ww.. wttp////r/....gg////ea.. httpa//...........ppaa//aaaaaaaa
Jesin Zakaria, Abdullah Mueen, and Eamonn Keogh. Clustering time series using unsupervised shapelets. In IEEE 12th International Conference on Data Mining, pp. 785-794. IEEE, 2012.
Jaaping Zhao and Laurent Itti. shapedtw: Shape dynamic time warping. Pattern Recognition, 74: 171-184, 2018. ISSN 0031-3203. doi: htt.:///ww...neeee.....///////// 7/www www.sccience.. rect www
URLhttpa
A IMPLEMENTATION DETAILS
A.1 FIRST ORDER METHODS FOR MONOTONE VI WITH NUCLEAR BALL SETUP
In Algorithm [ we present concrete extragradient method with backtracking for nuclear norm constrained VI based primarily on the Algorithms given in (Tusem et al. 2019) and (Duchi et al.) 2008).
The monotonicity of V itself when the link function á is monotone is readily established by two facts about the calculus of monotone vector fields:
Affine substitution of argument If ((-) is monotone vector field on m and A E Raxm is a matrix, the vector field
g(x) AΦ(A a
is monotone R
17

---

Published as a conference paper at ICLR 2025
Summation If S is Polish space, Φ( R m 5 - R is Borel vector-valued function which is monotone in æ for every 8 E S and u(ds) is a Borel probability measure on S such that the vector field
F(z) / p s)u(ds)
is well defined for all then F(-) is monotone.
for more detailed discussion, see Juditsky & Nemirovski (2016). Since the vector field given by the link function f()) is continuous and monotone, and the expectation is well-defined, the vector field
V (B) E, (o - -r)]
monotone and well defined since (f,)(A((B) yt) is monotone for all linear operators A and vectors Yt by affine substitution, and since expectation can be expressed by definition as an integration (summation in the empirical approximation) with respect to Borel measure.
Algorithm Extragradient Method with Backtracking for Nuclear Norm constrained VI
1: procedure
N, X, ki, 0, 1)) V (stochastic Estimate to) monotone VI, N > 0 number of steps, À > radius of nuclear ball. Ko E (0, 1] initial step size, (0, 1] step size decay parameter, (0, /2 line search parameter
2:
B1 := 0
3:
for t 1,N do
4:
for j = do
D. Line Search
5:
0 00
6:
R = PROXNUC(B, - -ROXNU)(B:-K((BBB)))) xB(((B)))
Extragradient step
7:
K: (RF) (B,)) S R B,ll then
8:
break
end if 10: end for
9:
B++1 == PBO 12: end for
11:
13: return BN
14: end procedure
15: procedure PROXNUC(A.))
> Computes Proxs, (A), A E mxn
16:
r: min miinmmm1
17:
U = [ V = :: =
18:
cs
Cumulative sum of singular values
19:
for = ,,...,r do
Compute u, J.. Vj
> From Lanczos iteration on A
21:
o * < (cs + rT)) - À then
break
23:
end if
24:
es == cs + oj
25:
v V
vil, ja 0,), U: U un
26:
end for
27:
= (cs A)/j
28:
if j < r then
29:
return U diag(o V*
30:
else
return end if end procedure
32:
A.2 IMPLEMENTATION
We implement Algorithm and associated subroutines (evaluation of the monotone field V, as defined in Equation 8 incremental simplex/nuclear ball projection), using the Julia program- ming language. The implementation is available at hhttps: //tt:///tttbbb..// com com//Spaace2133 LowRankTimeSer iet.coverrr
18

---

Published as a conference paper at ICLR 2025
Table 3: Five classes of sequence generating procedure
Baseline Coefficients
Perturbation Pattern
Exponentially Time Decaying
Gaussian
Exponentially Time Decaying Exponentially Time Decaying Uniform Uniform
d/3 Most Recent Uniform Fixed Vector
Gaussian
Uniform Fixed Vector
B DETAILED EXPERIMENTAL SETUP AND RESULTS
We evaluated all experiments and illustrations using a cluster with 24 core Intel Xeon Gold 6226 CPU (2.7 GHZ) processors, and NVIDIA Tesla V100 Graphics coprocessors (16 GB VRAM), and 384 GB of RAM. However, it is also possible to reproduce the results on a standard personal computer.
B.1 SYNTHETIC TIME SERIES
B.1.1 DATA GENERATION
For the synthetic sequence recovery experiment, we adopt the following data-generating procedure: We take the order of the sequences to be d = 15, and generate data according the following procedure within each of the five generated classes of observations
Pick a baseline set of coefficients according to a given random distribution
2. For each of the N 300 sequences to generate, perturb the coefficients according to the pre-specified rule
3. Generate the data matrix of size consisting of T = 250 of the N = 300 sequences according to the perturbed coefficients such that the data obeys ( with linear link function o(x) = x. To do so, we seat the first 15 observations using random noise such that Fi,t ~ NN =
0, 0² = 1), Vt E [1, d]. E [N]. Then each successive entry Ti,t, 1 E [d+ 1, is then given by taking F4, Σ Di,s Gi Gi,t Ni 0, o² 0.02).
We draw the ten generated classes of data from the following five generation procedures given in 3 We use each procedure twice to generate the classes of data. We denote the coefficients common the sequences (for some class) as Dommmon and the coefficients for the 7th sequence in said class as b;.
The baseline coefficients generation methods are given as:
Exponentially Time Decaying: beommoons 77 / 7
Z Uniform([O, 1)), Vs e [d]
Uniform: beommon,, = Z Z ~ Uniform((0, Vs [d]
and the perturbation methods are given as:
Gaussian:
b, Dcommon + Z
Z, ~ N(p 0, 0.02)
[N(p 0, = 0.02) j [d/3]
d/3 Most Recent: b, b Drommon + Z
Zg 10
Wi [N]
Uniform Fixed Vector: b; Doommon + fv chosen uniformly on unit hypersphere, and is the same for all sequences generated in the class)
0 Uniform([-1, 1). ||| 1, [N] Sv
B.1.2 PARAMETER RECOVERY
For the parameter recovery experiment, we take all (10) = 120 combinations of l sequences from the 10 classes and concatenate the generated sequences to form matrix of 900 observations.
19

---

Published as a conference paper at ICLR 2025
A 65,86 Rani==10
108,70
Ranio33
Rank=35
Ranko 47
A 264.82
349.36 Ranio-4B
Rainko
Figure 3: Learned representations for symbolic sequences generated from Hidden Markov Models (HMMs), recovered by solving Program (5) with varying nuclear constraints visualized using UMAP projection.
5.216
Predicion Entropy)
5,214
5,212
5.210
5.206
Figure 4: Cross entropy as function of À across for symbolic sequence generated by HMMs.
We then recover the baseline coefficient matrix E R 15x900 To recover the parameters for each sequence, solve the program given in (5) to optimality for differing levels of À (using a standard convex solver by contrast to Algorithm I]. To find which levels of to solve for, we first solve the unconstrained version of the problem. We then compute the nuclear norm of recovered || We then successively search for the optimal X the interval [0, [[|]]] using the relative reconstruction error |B Bl| | F as the objective until we achieve an absolute tolerance of 10-3 We report the results in Table using the matrices and B,. across the 120 runs. For Figure we use data drawn from the classes uniform baseline coefficients with Gaussian perturbation, exponentially time-decaying baseline coefficients with Gaussian perturbation; and exponentially time-decaying baseline coefficients with uniform*fixed vector perturbation. We find the X* via Brent search, and
sweep 40 values of E [16.3, 25.2] (the right bound corresponding to the value of |[[]|]) to produce Figure [ When reporting the spectra of singular values, we pick solutions that correspond to the largest value we have for some fixed number large singular values. Finally, we depict the principal components according to the recovered matrix B which is optimal in the sense of the reconstruction error (found via Brent search).
B.2
SYNTHETIC SEQUENCE DATA
B.2.1 DATA GENERATION
To illustrate representation learning and parameter recovery in the nonlinear simulate HMMs with four hidden and four observed states. Transitions between states are random, and the emission matrix assigns 0.9 probability to emitting the symbol corresponding the hidden state,
20

---

Published as a conference paper at ICLR 2025
Table 4: Basic statistics for the UCR time-series classification benchmark data
Training Samples
Testing Samples
Length
Classes
AnooHead
175
25]
Reef
50
4.00
BeetleFly
20
512
BirdChickkm
20
ChlorineCunc.
467
3840
166
3
Coffee
28
28
286
2
DinommiieRotiction
506
Dist Pha Outln, AgeGrp
139
80
Dest Pho. Ouili Concct
600
276
80
ECG200
100
95
Days
23
CanPaiint
Hom
109
105
431
2
Herring
64
512
2
Lightning?
60
Meaa
60
Phd.Pha Ounln.uuGnnnnn AgoCirp
400
154
80
3
Mid Pha Outln Outle,Comect Conect
600
201
80
2
PhallamTTT
MoneStrai
20
1252
OSULeaf
200
242
427
6
Plane
108
AgeOrp
205
Prox. Phalanc
208
SlnyARBORobod
20
601
70
Sony ABm
27
951
65
SwedishLeaf
Symbuls
25
995
I
40
228
277
2
TosSegmentziun2
36
130
TwoPaitems
IwoLewEEC
23
1139
82
Water
1000
6161
152
Wine
57
234
2
WordSynanyys
with equal probability among other symbols. The models are initialized randomly and simulated for 100 time steps, generating 150 trajectories for each random HMM.
B.2.2 PARAMETER RECOVERY WITH SEQUENCE DATA
For parameter recovery demonstrating nonlinear sequence dynamics, sample three random HMMs and aggregate their realizations (450 sequences in total). We solve Program induced by using these realizations, sweeping across different values of X.
Figure depicts the learned low-rank space projected into two dimensions using UMAP, showing the choice of À and the approximate rank of the space. Figure 4 presents the relationship between the nuclear ball size (A), and the loss as measured cross entropy of the trained parametric model (.) with the data across the entire observation timeframe.
The results largely align with the discussion in Section 4.1 Small amounts of nuclear regularization improve the quality of sequence representations and improve the prediction performance, as the information from other sequences becomes incorporated to aid in prediction. The sequences sequences also cluster within constrained representational subspace. The data remains well-separated even in low-dimensional spaces (e.g., rank 3).
B.3
UCR TIME SERIES
B.3.1 DATA OVERVIEW
compare our method with the following representative time series clustering methods. In line with Ma et al. (2019) we selected 36 of the univariate UCR time series classification datasets [ We report basic statistics (training samples, testing samples, length, and number of classes) of the datasets in T
https:/ fttps://www.c. uer edu httaaa////eeememmmeemeeeeeeeeeee 2018
21

---

Published as a conference paper at ICLR 2025
B.3.2 OVERVIEW OF EVALUATION METHODS
We evaluate the classification performance using the following methods:
ARI: Similarity of learned and ground truth assignments (Vinh et al. 2009). For matched clustering partitions
N 2
RI R
RI = (a + /)/
ARI
max RI) - E[Rr]
NMI: The mutual information between the true class labels and the cluster assignments, normalized by the entropy of the true labels and the cluster assignments (Vinh et al. 2009).
21(X; Y)
NMI H(X) + HYY
where X, Y are the true and assigned labels, (X) is the entropy of X, and ((;Y) the mutual information between X and Y.
Proportion of correct predictions to a total number of predictions. Fl: Harmonic mean of the precision and recall. the multiclass case, we take the macro average by calculating the metric for each label and computing their unweighted mean.
Accuracy:
2 x TP
F
TP FP + FN
Runtime: Runtime of the algorithm in terms the user CPU time in the computational setting de- scribed in 4 If the method GPU accelerated we report the user CPU/GPU time spent in the routine.
B.3.3 OVERVIEW OF METHODS
In addition to our method, we evaluate the performance of the following baseline methods
k2+ K-Nearest Neighbors Classification with the distance metric as the Euclidean distance between two time-series treating the entire observation sequence as a high dimensional vector (Cover & Hart 1967).
DTW+KNN K-Nearest Neighbors Classification with the distance metric calculated according to DTW (Müller 2007), which aims to align the two given sequences by solving the following program
DTW (x,x') x')
min (An, (x // =EA(xxx)
The A(x, x') is the set of all admissible paths as represented by boolean matrices. Non- zero entries correspond to matching time series elements in the path. A path is admissible the beginning and end of the time series are matched together, the sequence monotone in both i and j, and all entries appear at least once. We take q = 2 as the Euclidean metric.
shapeDTW Extension DTW scheme by incorporating point-wise local structures into the match- ing procedure (Zhao & Itti] 2018). Examples of such shape descriptors include data itself, rolling average of, discrete wavelet transform, and a finite differencee/derivative. Finally, the encoded sequences are then aligned by DTW and used for nearest neighbor classification.
MR-Hydra Combination of dictionary- based Multirocket and Hydra algorithms for time series classification, extracts and counts symbolic patterns using competing convolutional kernels (Dempster et al.) 2023 Tan et al.] 2022).
TS2Vec Construct an encoder network for time series embedding based on hierarchical contrastive learning (uue et al. 2022). The discrimination is done both between sequences and within the sequences themselves. The encoder network consists of an input projection layer, timestamp masking module, and a dilated convolutional module, and is optimized jointly with temporal and sesueee contrastive loss.
Ti-MAE Like all auto-encoding models, encoder network maps time series signal into latent representational space, and then a decoder aims to reconstruct the original sequence from the representational space. Once the input has been tokenized, random sample of tokens are masked, and then the decoder attempts to reconstruct the time series optimizing the on self supervised reconstruction loss (Cheng al. 2023
22

---

Published as a conference paper at ICLR 2025
B.3.4 DETAILED EXPERIMENTAL PROCEDURE
We split our data into testing and training splits according to those given by the UCR repository. For the methods that directly perform classification (KNN, shapeDTW, Inception Time), train on the test set and then report the performance on the training set. In line with Yue et al. (0022); Franceschi al. (2019) to evaluate the classification performance test set for methods which produce embeddings (TS2 Ti-MAE and our method), we perform cross- cross-validated grid search (based on k 5 folds) across KNNs with & {2 i E [0, 4]} neighbors or SVMs with RBF kernels with penalty values E (2' i 10, 15] U co. For the KNN-based methods, we do the same grid search as outlined above. For our own method, we also grid search across parameters of À and report the performance for the best choice under rank constraint. To find the embedding, we run Algorithm for 256 iterations.
B.4 DETAILED NUMERICAL. RESULTS
In Tables and and we present the classification performance for the discussed metrics for the evaluated methods for each of the tested UCR datasets.
B.5 REPRESENTATION VISUALIZATION
In Figure 5 provide comparison of the embedding quality for some of the recovered real-time series for our method (left) as compared two recent and popular neural-network based time-series representation learning methods TS2Vec (Yue et al.] 2022 (center) and Ti-MAE (Cheng et al.. 2023) (right). We plot the category the data in color, though this information is not provided to the models during training. We note that our method produces similar quality embeddings these approaches, with better separation of the data according to category in the low sample cases (e.g.. BeetleFly, BirdChicken). For all three models, there exist cases, where the separation is worse for certain datasets, for example Ti-MAE on TwoLeadECG, our method on TwoPatterns.
B.6 NATURAL LANGUAGE EMBEDDING
To acquire the data, we retrieved the raw text of Alice's 's Adventures in Wonderland and Through the Looking Glass from Project Gutenberg R of the MLL-AzXiv-Papee
For the paper abstracts, we used the training portion dataset 3 For each dataset, we stripped all non ASCII characters and uncommon punctuation (s,>, ",=, 1, ?, &, [. *, #, G, and ").
After acquiring the data, we then encoded using Huffman tree with n. = 4 symbols derived from the frequency of letters in our corpora. We treated each abstract as document and considered 500-character chunks of the two books. We rejected abstracts containing less than 500 words. After encoding the sequences using the Huffman code, we cut off each sequence at 1000 coded symbols and rejected all sequences less than this length after coding. This left with n 228 samples from "Alice's Adventures in Wonderland", n = 316 samples from "Through the Looking Glass", and 600 machine ceaming-rellttd ArXiv abstracts.
To learn the embedding, we use the method described in Appendix A. and grid searched across values of X for 512 steps using the softmax link function described in Section 3.2
B.7 GENE SEQUENCE EMBEDDING
Data acquisition and processing In line with Millan Arias et (2022). downloaded viral genome sequences for two different kinds of human viruses: Influenza A virus and Dengue virus. We consider different strains of each virus in addition to the species whole. We provide a textual description below. In Table we provide summary statistics, including the number of sequences in the strain, the average and standard deviation of the sequence lengths, and the length of the shortest and longest sequences in the strains.
https:://// www::///wwwww....m http://wwkkkgggg kaggle hotea//eeeeaaaattt/r/eeeee1--11e verrs
23

---

Published as a conference paper at ICLR 2025
Arrowllend
Dure
TS2
TI-M
TI-MA
12+
Beetlefly
DistomSiieRRcducion
Ours
T52
MAE
Ours
"Savec
TIMAE
13
23
Car
EC0200
Ours
TSeWee
THMAT
Ours
TI-MA
255
5.2
15
100
25
10
12.5
75
100
125
OSULeea T52Vec
TosSogmentation T52Vcc
Ours
Ti MAE
Ours
TIMAE
TwoPatterns 15.0icc
Ours
152Wcc
TLMAE
13
23
23 5.
38.0
ProxinolPhaloneTWW
TwoLeadECo TS2Ve:
T52
TH-ME
TMM
5.
Figure 5: UMAP projections of learned embeddings for UCR datasets by method, TS2Vec Yuue ct al. 2022). and Ti-MAE (Cheng et al. 2023)
24

---

Published as a conference paper at ICLR 2025
Table 5: Detailed results per UCR dataset (Part I)
Dataset
Mathod
ARI
NMI
Acc.
RT
Method
ARE
NMI
Acc.
RT
Anod
42
0.482
0.453
0.800
0,800
0.007
MR Hydra
0.629
0,585
0.883
01863
3.183
Reef
KNN
0.667
0.672
0.001
0.454
0.767
0.768
RestieFly
0.219
0.344
0.750
0.733
0.001
0.621
0.619
0,900
0.822
1.532
BlrdChicken
-0.044
0.007
0.550
0.549
0.001
0.621
0.619
0.500
0,899
1.536
Car
0.403
0.477
0.731
0.737
0.003
0.830
0.860
0.933
0.931
4033
ChiorineCone.
0,231
0.650
0,60
0.633
0.472
0.780
0.753
57.36
Coffee
1,000
1,000
1.000
1.000
0.001
1.000
1,000
1,000
1,000
1411
0.872
0.830
0.935
0.833
0.004
0.921
0.896
0.964
0.947
6.537
Dist Auett.
0.190
0.224
0.626
0.613
0.019
0.383
0,404
0.770
0.775
4.754
Phu Oulln Correct
0.717
0,684
0.054
0.790
8,056
ECC300
0.571
0.445
0.580
0868
0.004
0.667
0.542
0,910
0.902
1.765
Days
0.352
0.304
0.797
0.794
0.011
1.000
1.000
1,000
1,000
7914
GunPoint
0
0.578
0.913
0.913
0.004
1,000
1,000
1,000
1,000
2,064
0,400
0,600
0.006
0.229
0.742
5055
Herring
-0.015
0.003
0.516
0.916
0.003
0.207
0,155
0.734
0.726
3
T Lightiing
0,246
0.191
0.754
0.750
0.003
0.104
0.084
0.672
0.665
52088
Most
0.799
0.797
0.911
0,935
0.003
0.810
0.808
0.933
0.933
3,0%6
Mid. Pho Agerr.
0.055
0.443
0.021
0.092
Mid Phu Outln Cornect
0,280
0.308
0.766
0.766
0.057
0:475
0.372
0.845
0.843
8.154
Mid PhalanXTW
0.379
0.367
0.513
0.382
0.020
0.383
0.433
0.513
0.339
5.382
MoteStrain
0.573
0.467
0.879
0.877
0.015
0.794
0.699
0,946
0.945
7,062
OSUL
0,298
0.521
0.925
0.023
0.921
0.956
10.3
Plane
0.19
0.62
0963
0.005
1.000
1.8001
1,000
1,000
2.505
Prus. Pho. AuuGn.ggGrrpp
0.492
0.422
0.785
0.693
0.027
0.662
0.564
0.868
0.797
5.489
Pros. Phalans TW
0.584
0. 0.566
0.707
0.444
0.027
0.718
0.671
0.805
0.400
4.922
Sony Surr
0.148
0.688
0.007
0.598
0.557
3.396
Sony AuBORobwssu
0,514
0.395
0.839
0.849
0.013
0.782
0.682
0.942
0,940
4.828
Swalishl zaf
0.629
0.761
0.789
0.782
0.109
0.950
0.965
0.976
0.977
10.78
Symbols
0,791
0.843
0.899
0.398
0.017
0.955
0.954
0,981
0.988
21.89
TouSegmentation
0.126
0.680
0.675
0.006
0.832
ToeSegmentation2
0.340
0.344
0,908
0.44
0.003
0.640
0.464
0.915
0.566
3,608
TwoPatteras
0,770
0.726
0.907
0.906
1.328
1.000
1.000
1,000
1,000
50.14
TwoLewlEC
0.244
0,217
0.747
0,741
0.015
0.993
0.983
0.998
0.998
6.182
Wafer
0.923
0.995
0.988
2.088
0.999
76.27
Wine
0.031
0.006
0.611
0611
0.002
0.720
0.687
0.926
0.926
1.718
WonlSynonyms
0.537
0.571
0.618
0.465
0.069
0.725
0,753
0.777
0.658
15.76
Anro Head
DTW+
0,312
0.282
0.903
0.700
2.139
TS2
0.480
0.462
0.794
0.794
$1.08
Deef
KNN
0.276
0.190
0.633
0.629
L.158
Vec
0.284
0.194
0.667
0.670
109.0
HectieHly
0.131
0.700
0.670
0.618
0,800
0,950
0.950
BirdChicken
0,212
0.750
0.244
0.606
0.621
0.619
0.899
90.00
Car
0.446
0.501
0.733
0.728
7.565
0.209
0.787
0.867
0.867
208.9
ChiorineCone.
0:231
0.154
0.648
0.007
247.313
0.452
0,333
0.764
0.730
2439
Coffee
1.000
1,000
1000
1.000
0,336
0.857
0.811
0,951
0.961
129.1
0.938
0.921
0.967
0942
3:272
0.968
0.952
0.984
0.973
87.44
Pha Agela.ggetepp
0,389
0.308
0.710
0.763
1.804
0.272
0.277
0.705
0,099
2046
Phat Outin.Crrret
0,183
0.132
0.717
0.690
5.382
0.246
0,176
0.750
0.737
2582
ECG200
0.280
0.770
0.749
0.468
0.540
0.358
463.5
0.286
0.252
0.768
0263
1.848
0.991
0.979
0.988
0.988
9084
GunPoint
0.659
0.557
0.07
0.907
0.847
0.973
0.969
0.993
0.993
234.7
Ham
-0.005
0.008
0.467
0.467
12.116
0.210
0.168
0.733
0.733
526.1
Herring
-0.012
0.320
6.556
0.064
0.047
0.641
0.625
326.8
Lightning2
0.537
0.569
0.864
8.898
0.318
0.252
0.787
0.783
301
Meat
0,799
0.797
0.933
0.935
4.497
0.687
0.714
0.883
03383
300.
Mid Www..Pa..Ottn.cee Outtn.geeeapppp
0.024
0.022
0.500
0.411
2,066
0.038
0,019
0,319
0.426
2012
Pho Outln. Correct
0.153
0.098
0.691
5.663
0.385
5055
PhalannTTW
0
0.368
0.506
0.774
1.9%
0.420
0.404
0.396
1997
MoteStrain
0
0.351
0.835
0.834
0.900
0.528
0.424
0,803
0.862
86.43
OSULear
0.309
0.392
0.591
0.588
50.818
0.644
0.671
0.823
0.799
1071
Plane
1.000
1.000
1.000
1.172
1.000
1.000)
1.00
S41.7
Outin Agolirp.
0.504
0.400
0.905
0.16
2.696
0.506
0.437
0.780
0.689
2061
Prox. PhalaneTW
0.644
0.587
0.756
0.511
2.675
0.674
0.628
0.771
0.562
2052
Sony A
0.200
0.316
0.725
0.721
0.309
0,588
0,571
0.884
0.383
84.73
Sony Sur
0.435
0.817
0.571
0.671
0.584
125.5
Swodiahl.ca
0.639
0.792
0.787
25.957
0.875
0.936
0.937
2573
Symbols
0.891
0,913
0.950
0.999
21.459
0.928
0.936
0.989
01969
152.7
ToeSegmentution
0.293
0.360
0.772
0.762
3,704
0.816
0.723
0.952
0.952
213.5
TeeSegmeattiion2
0.398
0.764
2.904
0.714
0.899
170.8
TwoPateerss
1.000
1,000
1.000
1.000
330.31
0.973
0.964
0.990
0.990
5216
Twal edFECG
0.654
0.564
0.004
0.904
0.901
0.958
0.918
0.989
0.989
87.20
Wafer
0.867
0.748
0.580
0.944
720.459
0.942
0.868
0.991
0.576
5386
Wine
0.003
0.574
0.574
0.860
0.339
0.796
301.5
WondSynonyms
0.575
0.64)
0.533
66.754
0.349
0.421
0.523
0.309
1407
25

---

Published as a conference paper at ICLR 2025
Table 6: Detailed results per UCR dataset (Part II)
Datoset
Method
ARI
NMI
Asc.
RT
Method Qurr
NMI
ArrowHead
shape DTW
0,521
0.492
0.117
0.818
D.672
0,356
0.306
0.720
0,230
136.7
Reaf
0.518
0.672
0.214
0:453
0.654
BortleFly
0.750
0.735
0:107
1,300
1.000
1000
51,50
BlrdChickes
.0044
01007
0.550
0.549
0.106
1,300
1.000
1000
1000
51,92
Cur
0.560
0.585
0317
0.855
1,077
0.372
0.57
0.667
173.1
ColorineCone.
1201
0.557
0.414
Cuffee
1,000
1.000
1.000
1,000
0.105
1,000
1.000
1.000
10000
39,92
Ditomsizeeelaation
0.921
0,800
0958
0.921
0.840
0.818
0.865
0.882
0.704
331.7
Diet Phu Outtn.Aeefipp
0.209
0.251
0.633
0.615
1.584
0.323
0.402
0.741
0.748
163.9
Outle.Commet
0.316
0.233
RCG200
0.541
0.200
0.600
0.397
0.668
0.554
0.910
0.04
43,80
BCGFiieDDays
0.705
0.665
0.200
0.921
1.279
0.765
0.666
0.937
0.937
306.1
GunPoint
0.845
0.761
0.960
0.960
0.517
0.845
0.761
0.960
0,960
70.03
Ham
2.714
0.160
0.134
365.6
Heriing
-0.012
0.006
0.531
0,531
1.129
0.176
0.128
0.719
0.688
162.0
Lig Lightning?
0.358
0.299
0.803
0.797
1.345
0.399
0.320
0.820
0.817
230.8
Meat
0.799
0.797
0.931
0.935
0,850
0.8%
0.541
0.950
0,950
133.3
1.722
0.143
Midl Ouilin Correct
0.281
0.307
0.766
0.759
5.110
0.410
0.374
0.821
0813
177.9
Mid PhalaanTT
0.357
0.360
0.487
0.361
1.740
0.362
01436
0.591
0.334
107.5
MoteStmin
0.575
0.467
0.879
0.877
0.804
0.212
0.175
0.731
0.731
243.3
OSUT
0,600
0.625
584.7
0937
0.061
0.71
0.972
0.644
1,000
1.000
74.50
Prot. Phu OudIn AgeGir
0-482
0.399
0.780
0.688
2.244
0,681
0.554
0.583
0.808
119.3
Pros PlaloneTW
0.585
0.565
0,702
0:426
2.240
0.752
0.738
0.834
0.476
118.7
0.282
0.619
0.572
Sony 2
0.589
0.468
0.885
0.876
0.560
0.767
0.655
0.938
0.34
139,3
Swedish zuf
0.697
0.806
0830
0.827
16.027
0.805
0.864
0.809
0,807
441.6
Symbuls
0.825
0,8641
0,918
0.917
1,965
0.903
0.917
0.956
0,956
1127
TueSegmentation]
1.136
0.678
ToeSegmentation?
0.486
0.379
0.809
0.780
0.685
0.411
158.2
TwoPatlems
0
0.870
0.965
0.964
20:0000
0.879
0.145
0.143
0.514
1719
0:484
0.438
0.448
0.846
0,742
0.993
0.981
0.998
0.998
240.5
373.891
0.939
0.858
Wine
0.016
0.025
0.593
0,593
0,300
0.150
0.128
0.704
0.702
63.29
WordSynongms
0,578
0.600
0.639
0.487
20.963
0.346
0.325
0.395
0.220
764.7
Acrowliedd
Ti-
0.374
0.334
0.237
0.735
465.061
Beef
MAE
0,179
0.415
0.533
0.551
163.791
BecticHly
0,344
196
Car
BirdChicken
0.324
0.260
0.278
0.382
0.650
945-30
390-536
Chlorieeoonc
0,376
0.273
0.226
0690
2065 996
Coffee
1,000
1.000
1000
1,000
DinonsizeRetaction
0.71
0.90
310
Dust Pha Auerp.
0.357
0.395
0.763
1957 101
Pist Cuttn.Currei
0.267
0.199
0.761
878
BUAG200
0.635
0.167
0:22
PuniPoint
0.575
0.505
0.800
Ham
0.130
0.102
0.686
01685
0952.93 9224
0,044
0.382
Lightning2
0.140
0.656
10000
Meat
0.622
0.646
0867
Mid.Pha WwthoOtttt....... Outln.AgeGrpp.
0
0.032
0.571
01115
0014
Pha Carte.Conect
Mid PhalanXTW
0.409
0.604
1631 140
1022009
MoneStrain
0
0.380
0.839
OSULcaa
0.216
0.290
0.483
2750-500 507
Prox Plu Ourtn Agotiep
0.023
0.90
1407
Pron PhalaneTW
0.682
0.648
0.740
1407 9006
Scny I.t.abbotto......
0.650
0.614
0.256
0.184
239.141
274
SwedishL.cf
0,722
2093.867
Symbols
0.617
0.709
0.297
0,799
456.173
I
0.557
0.992
0,696
0.660
446.146
ToeSegmentation2
451.744
TwoPatiems
0.401
0.380
0.730
0.720
4361
Tuol cdROG
0.359
0.356
0,696
0.698
302.257
Water
0.925
0.826
0.98
0.968
5112.925
WordSynonyns
0.509
0.503
0.394
2173.234
26

---

Published as a conference paper at ICLR 2025
Table 7: Statistics for selected viral genomes.
Virus
Strain
Count
Sequence Length
Avg.
Std.
Min.
Max.
Influenza-A
H5N1
188
1368.521
(21.682)
1350
1457
HINI
191
1421.0
(15.25)
1350
1468
H7N9
190
1403.521
(12.048)
1389
1444
H2N2
187
1430.053
(17.87)
1376
1467
H7N3
193
1423.15
(21.537)
1345
1468
Total
949
1409.326
(28.512)
1345
1468
Dengue
DENV-1
409
10577.812
(194.4)
10176
10821
DENV-2
409
10592.504
(196,308)
10173
10991
DENV-3
408
10614.137
(132.911)
10173
10810
DENV-4
407
10452.469
(205.208)
10161
10772
Total
1633
10559.328
(194.74)
10161
10991
Influenza A The Influenza A virus genome data (n 949) is acquired from the NCBI Influenza Virus Resource (Bao et al. 2008). We consider the genome of segment 6, which encodes the neuraminidase protein, and include sequence samples belonging subtypes HIN1, H2N2, H5N1, H7N3, and H7N9.
Denuge We consider n 1562 full Dengue virus genomes downloaded from the NCBI Virus Variation Resource (Hatcher al. (017). We consider all four subtypes of the virus DENV-1, DENV-2, DENV-3, and DENV-4.
We encoded all the sequences as four-channel signals via one-hot encoding, with each nucleotide (A,C,T,G) corresponding one of the channels. In the encounter incompletely specified bases in the nucleic acid sequences (Cornish-Bowdar 1985), we give equiprobable weights to the possible corresponding nucleotides.
Learning procedure Same as the natural language case, we represent the data four channel signal and adopt the softmax activation scheme described Section 3.2 Since the sequences
are of considerable and variable length (> 1000 nucleotides, see Table Z we adopt stochastic
estimation to (8) by randomly sampling length = 800 sub-windowe from each of sequences. We
take the sample average for each of the sub-window observations similar to (8). We run Algorithm for N 1024 iterations, using the stochastic approximation described above and grid searching
across values of A. To produce Figures 2bland and 2c we took the learned representations and projected them into two dimensions via UMAP.
B.8 RUNTIME WITH PARTIAL SVD
we report runtime figures where each iteration computes the Prox step via full SVD. In practice, the recovered B₃ matrices often exhibit an exponentially decaying spectrum, allowing singular values to be computed only up to a threshold. our recovery experiments in Section 4.2 we use all time slices to form the stochastic approximation for Equation (8) and thus, the runtime is primarily determined by the time spent evaluating the VI field.
In Section 4.2
Table presents the time spent evaluating the VI outer iteration (each iteration also includes line search) the PG Algorithm [ (in seconds) for four representative UCR datasets and their accompanying statistics. We consider window sizes d 20 and d 60.
Table we report the spectra of singular values across different values X, and in Table 10 we present the time spent per outer iteration when using either full or partial SVD, described in the PROINUC subroutine of Algorithm [ We note that for small matrices, there is no computational benefit to using the partial SVD. However, as the matrix size increases and for smaller values of X, early termination of the singular value search can improve the runtime of the projection step and especially the case when the structure of the data is indeed well approximated by low rank subspace.
27

---

Published as a conference paper at ICLR 2025
d Dataset
B dimension
Seq. Len.
VT Runtime per iteration (avg. sec)
20
Meat
82 x 120
448
4.00
ECG200
82 x 200
96
1.12
82 x 539
80
2.83
Wafer
82 7164
152
78.31
60
Meat
242 120
448
5.57
ECG200
242 200
96
0.99
DistalPPPlammOOntiieAAgGooou
242 x 539
80
1,90
Wafer
242 x 7164
152
103.53
Table 8: Statisties for four representative datasets. The time to evaluate the VI field given in Equation g is given in seconds per outer iteration of Algorithm across all choices of À given Table 10
28

---

Published as a conference paper at ICLR 2025
Dataset
d
20
40
Meat
030
1.0
1.5
20
25
35
ECG200
10--
10
20
10
DisPhaOutAgeGrp
10-1
16"4
20
14
20
36.
31-
1.:
117
Wafer
10-7
10"1
3200
Table 9: Spectrum of recovered singular values for B, with differing levels of X for different real world datasets from UCR data as described in Table 8 When À is smaller, it is generally sufficient to compute a fewer number of singular values.
29

---

Published as a conference paper at ICLR 2025
Dataset
d
20
40
Auriol
Rutl 90p
0.08
vil threal
5.01
Meat
8.02
PAR
-2E5VD
0.23
WelThele
ECG200
5.19
0.12
35
6.6
Buttere 24
DisPhaOutAgeGrp
0.2
60
10
20
26
10
0.0
255
5.0
7.5
10.0
12.5
150
175
Torkal
PertelSVD
Wafer
120
20
50
Table 10: Average outer iteration runtime (seconds) of Algorithm I using PROXNUC for the Prox step (computing incremental singular values) versus using the full SVD across different À values for datasets Table
30


Reviews:
[{"review_id": "bluDiq8Gkl", "reviewer": "Reviewer_wK5W", "summary": "This work aims to learn the low dimensional representation of sequential data with nonlinear dynamics. It combines low-rank matrix recovery and monotone VI to learn low-rank representation of nonlinear signals, while maintaining the signal recovery problem’s convexity for acceleration. The low-rank constraint is enforced through nuclear norm regularization with proximal algorithms, while the modeling of the nonlinear dynamics and the monotone VI both rely on the monotone nonlinear link function.", "questions": "1. Does the link function $\\eta$ have to be continuous? \n    - Just curious, where does the name “link” come from originally? Why is it called \"link\"?\n2. Because the $\\eta$ is monotone and fixed, does the model in (1) in general have enough complexity to capture the nonlinear dynamics of a complicated sequence, e.g., a sequence with many high frequency components with rather large amplitude? \n    - Which hyperparameters of it can affect the complexity? \n        - If my sequential data have very low $C$ and complicated dynamics, is increasing $d$ basically the only way to increase your model's complexity? Can increasing $d$ always improve the signal recovery effectively?\n    - I guess that if the link function is not good enough, $\\mathrm{rank}~\\mathbf{B}$ cannot be sufficiently reduced, right? How to choose or design one?\n3. It seems like the model can be used for denoising by limiting $\\mathrm{rank}~\\mathbf{B}$. What if some very important signal components have very low SNR, and I don’t care much about the principal signal components with large SNR? Can the model potentially differentiate between them and only remove the noise? \n4. Why does the relative reconstruction error have a quadratic relation w.r.t. $\\lambda$ in Fig 1.a? Underfitting and overfitting?\n5. How long does it take to search for $\\lambda$? Does it scale with the problem’s dimensionality?\n6. What is the speed difference between recurrence (8) and (15)?\n7. How widely applicable is the assumption \"we suppose...\" on Line 114 in reality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well written work, with comprehensive introduction, clear problem statement, detailed methodology presentation and thorough experiments. Pseudocode is also given.\n2. Dimension reduction or representation of sequential data is indeed a crucial problem.\n3. The method looks novel and sound.\n4. The experiments can clearly reflect this method’s efficacy.", "weaknesses": "## Major\n\n1. I think the performances of linear and nonlinear monotone autoregressive models (both with low-rank regularization) might need to be compared with further experiments. It seems to me that the choice of the link function $\\eta$ is still kinda restrictive and problem specific, as it needs to be monotone and remains fixed after being selected. Intuitively if a nonlinear $\\eta$ is very similar to $\\eta(\\mathbf{x}) = \\mathbf{x}$, the models using them may not be significantly different in terms of performance. Not sure if spending so much effort making the autoregressive model a tad nonlinear is worth it, especially considering that the nonlinear one needs the additional evaluation of $\\mathcal{A}$ and $\\mathcal{A}^*$. \n    - However, l like this paper overall, so this is not fatal.\n    - See Question 2 for more detail.\n2. The authors had better mention their method’s limitations and suggests some future research directions. For instance, a few things mentioned in the my questions. \n\n## Minor\n\n1. You should briefly explain why the *monotone property* of the link function is important when you first introduce it on Line 122 and 132 (like simply referring to Sec. 3.2), or cite related works given that you have so many examples. \nIn addition, you should explicitly refer the readers to Line 263 for the accurate definition of  “monotone”. A better option is to turn Line 262 to 269 into a formal Definition block with number (given how crucial monotonicity is to your method), then \\ref it on Line 122.\n2. Line 136, is the subspace linear? If so, you had better mention it explicitly here, or claim that all subspaces in this work mean linear subspaces.\n3. “Geometry” in phrases like “geometry of the subspace” “geometry for the entire domain” “geometry of the representations” is a very awkward term for your work. The subspaces in this work are just flat planes with no special “geometry”. It’s not like you have to deal with Riemannian metric, angle, geodesic, curvature and so forth on a nonlinear submanifold. \n    1. For instance, on Line 154, you can basically just say “the subspace…captures…’’\n    2. If you insist on using “geometry”, be clear about what you mean by it. \n\n## Trivial:\n\n1. Use the correct citation format: \\citep and \\citet. For example, Line 089. \n2. Typos. For instance:\n    1. Line 053: duplicate “indicates”\n    2. Line 106 “thewfirst”; Line 285 “spacewof”\n    3. Line 115, missing “by”\n3. Clarify your special notation when it’s introduced. What is “vec()”? Vectorize? Flatten? \n4. In Table 2, make the best Avg. in each column stand out with bold font.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWcLXMeH1m", "reviewer": "Reviewer_9iKd", "summary": "This work introduces an unsupervised method for learning low-dimensional representations of sequences. Specifically, it assumes that each sequence relates through a low-rank latent autoregressive matrix, and, by casting the problem as matrix recovery, the authors derive an efficient algorithm to obtain the latent matrix. The authors evaluated the resulting representations obtained by the proposed method in the UCR time-series classification task and clustering on language and genome sequences.", "questions": "Typo\n- l:285 the spacewof -> the space of ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. \n- Bringing the matrix recovery technique in sequential learning enables us to efficiently compute latent autoregressive parameters with some theoretical guarantees such as computational complexity and parameter recovery guarantees, which look a good contribution to the representation learning of sequential data.", "weaknesses": "- Regarding the clustering experiments; it is hard to tell the model's significance since the authors do not include any other model's results. I see the obtained representations by the proposed method are well-clustered in every dataset, but how do they become when using different models (e.g. models shown in table 2.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4KsvH4ZmRr", "reviewer": "Reviewer_7R4f", "summary": "The paper introduces a novel approach for unsupervised learning of low-dimensional representations for multiple related time series using a monotone variational inequality framework. The approach combines traditional autoregressive modeling with a low-rank regularization, which leads to efficient matrix recovery and nonlinear sequence embedding while maintaining convexity. The authors test their approach through model recovery experiments on synthetic linear AR sequences and applications to real world time series classification and more complex symbolic datasets, such as language and genomics.", "questions": "- Related Work: this is a broad field, so it’s challenging to cover every related approach. A class of approaches that goes in a similar direction which could be mentioned as well are hierarchical (Bayesian) time series models, e.g. Random Effects Models for Longitudinal Data (Laird and Ware, 1982) or dynamical hierarchical models (Zoeter and Heskes, 2003). Likewise, in the field of dynamical systems modeling, there are some approaches going in a similar direction, extracting low-rank representations across multiple observed time series (e.g. CoDA, Kirchmayer, ICML 2022)\n\nTypos:\n- indicates twice around page break between pages 1 to 2\n- Page 3: sufficiently captured an order (missing “by”)\n- Page 6: spacewof matrices", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is neatly written, and I overall found the presentation of the figures and clarity of writing nice.\n- The paper looks mathematically rigorous and well-rooted in related literature and approaches.", "weaknesses": "- Method Section: I am admittedly not familiar with much of the literature cited in this section. I appreciate the mathematical rigor but found some details in this section hard to follow. This also makes it harder to assess which parts of this section are novel contributions. A summary section at the end of the introduction or the method section that explicitly states what is novel about this approach could make the main contributions of the paper clearer.\n\n- Results: Linear and Nonlinear Models: The shift between linear and nonlinear models was somewhat confusing on the first read. \nSection 4.3 mentions learning representations for sequences with nonlinear dynamics, but the nonlinear approach seems to be applied earlier in Section 4.2 as well. \n\n- The model recovery experiments are specifically for the linear case. Clarifying why only the linear model was used for parameter recovery while not applied to real data would help. Is the idea to test the nuclear ball regularization separately in a simpler setting?\n\n- On the other hand, including a direct comparison between the linear and nonlinear models in terms of performance on the real-world data might also be interesting. Vice versa, it could also be beneficial to add parameter recovery experiments for the nonlinear case as well.\n\n- The role of  $\\lambda$: what role does the low-rank constraint play on performance? You mentioned you tuned lambda on the training set. Can we learn something about the complexity of the datasets depending on which lambda is optimal? How much better does performance get using $\\lambda$, compared to the unconstrained case?\n\n- Choice of comparisons: None of the evaluated comparisons incorporate low-rank constraints. While as the authors mentioned, time series classification is just one example downstream task (that is easy to quantify), some more direct comparison to other low-rank approaches could make the specific advantages of your method more clear.\n\n- Limitations:\nThere is no discussion of limitations. Besides descriptions of the approach/metrics, it is hard for me to assess what potential shortcomings/challenges the approach faces if I were to use it in practice. Tuning for $\\lambda$ is discussed in a lot of detail, but not so much for other aspects of the algorithm. This part could also include something on potential extensions, such as handling higher-dimensional examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9wgTBH4X9", "reviewer": "Reviewer_iV1B", "summary": "The paper introduces an unsupervised representation learning framework for sequential data, formulated as a low-rank matrix recovery problem using a convex optimization approach with monotone VIs.\nThe framework assumes a shared underlying domain among sequences and enforces this assumption through a low-rank constraint on the learned representations, implemented as a constraint of a nuclear norm ball.\nThis method is theoretically grounded and shows competitive empirical performance, while achieving provable recovery guarantees.", "questions": "1. Could the authors elaborate on their choice not to include benchmarks against popular representation learning methods?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Theoretical Rigor**: The proposed framework is strongly backed by theoretical foundations, leveraging low-rank matrix recovery and convex optimization over monotone VIs.\nThe approach is both well-formulated and analytically justified, with provable recovery guarantees under the low-rank assumption, which strengthens the theoretical contribution.\n2. **Empirical Performance**: Experimental results support the framework’s ability to recover parameter matrix and demonstrate competitive performance across real-world data.", "weaknesses": "1. **Benchmark Comparisons**: Given the objective of learning representations, it would strengthen the work to benchmark against established representation learning techniques like contrastive learning and masked modeling.\n2. **Minor Issues**\n    - **Clarity**: Consistency in notation, particularly in equations and indexing conventions such as the program notation in eqs. (6-7) and the loss function in eq. (14), would enhance readability.\n    - **Notation**: The notation should be updated to $\\mathcal{H}_{i,t}$ for clarity and consistency.\n    - **Errata**: There are minor typos, such as repeated words on pages 1–2 and “spacewof” instead of “space of” on p6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)