SOM-CPC: Unsupervised Contrastive Learning with Self-Organizing Maps for Structured Representations of High-Rate Time Series
Authors: Iris A.M. Huijben
Arthur Andreas Nijdam
Sebastiaan Overeem
Merel M Van Gilst
Ruud Van Sloun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contrastive Predictive Coding
Self-Organizing Maps
Time series
Dimensionality Reduction

Abstract: Continuous monitoring with an ever-increasing number of sensors has become ubiquitous across many application domains. Acquired data are typically high-dimensional and difficult to interpret, but they are also hypothesized to lie on a low-dimensional manifold. Dimensionality reduction techniques have, therefore, been sought for. Popular linear methods like Principle Component Analysis (PCA) have been extended to non-linear techniques such as Self-Organizing Maps (SOMs) or deep learning (DL) models. DL models have the ability to act on raw data, preventing heuristic feature selection, but the resulting latent space is often unstructured and still multi-dimensional. PCA and SOMs, on the other hand, need to be preceded with a feature-extraction step, but can then map high-dimensional features to 2D space. In this work we propose SOM-CPC, a model that jointly optimizes Contrastive Predictive Coding and a SOM to find an organized 2D manifold. We address a largely unexplored and challenging set of scenarios comprising high-rate time series, and show on both synthetic and real-life data (medical sleep data and audio recordings) that SOM-CPC outperforms both DL-based feature extraction, followed by PCA, K-means or a SOM, and strong deep-SOM baselines that jointly optimize a DL model and a SOM. SOM-CPC has great potential to expose latent patterns in high-rate data streams and may therefore contribute to a better understanding of many different processes and systems. 

Under review as a conference paper at ICLR 2023
SOM-CPC: UNSUPERVISED CONTRASTIVE LEARNING WITH SELF-ORGANIZING MAPS FOR STRUCTURED REPRESENTATIONS OF HIGH-RATE TIME SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
Continuous monitoring with an ever-increasing number of sensors has become ubiquitous across many application domains. Acquired data are typically high- dimensional and difficult interpret, but they are also hypothesized to lie on a low- dimensional manifold. Dimensionality reduction techniques have, therefore, been sought for. Recently, expressive non-linear deep learning (DL) models have gained popularity over more conventional methods like Principle Component Analysis (PCA) and Self-Organizing Maps (SOMs). However, the resulting latent space of a DL model often remains difficult to interpret. In this work we propose SOM-CPC, a model that jointly optimizes Contrastive Predictive Coding and a SOM to find organized 2D manifold, while preserving higher-dimensional information. We address largely unexplored and challenging set of scenarios comprising high- rate time series, and show on both synthetic and real-life data (medical sleep data and audio recordings) that SOM-CPC outperforms both DL-based feature extraction, followed by PCA, K-means or SOM, and strong deep-SOM baselines that jointly optimize DL model and a SOM. SOM-CPC has great potential to expose latent patterns in high-rate data streams and may therefore contribute to a better understanding of many different processes and systems.
INTRODUCTION
The improvement and abundance of sensor technology has led to large amounts of high-dimensional, information-rich continuous data streams. However, gaining actionable insights from these data is challenging due to their low interpretability. The main objective of this study is, therefore, to develop an algorithm for acquiring a structured and interpretable representation of (high-rate) time series. We define such an interpretable representation as one that has the ability to be informative and to facilitate exploration of the underlying structure (Lipton 2018
According to the manifold hypothesis, high nigh-dimensional real-world data lies on a low-dimensional manifold, comprising disentangled latent factors of variation. The area of unsupervised representation learning is concerned with models that learn this manifold from a set of training data, without the bias of human annotations. Dimensionality reduction techniques like Principle Component Analysis (PCA), possibly in combination with clustering methods like K-means clustering, have conventionally been used for this purpose. Acquiring an interpretable representation with PCA requires omitting many principle components in order to achieve an interpretable number of components. This, however, may discard important information that can not linearly be projected on these few dimensions. A Self-Organizing Map [Kohonen] 1990.. on the other hand, is extension of K-means clustering that creates a low-dimensional interpretable visualization, while still representing the data in multiple dimensions. However, SOMs typically act on features, which need to be selected heuristically and may, therefore, strongly depend on the use case and/or data modality.
Deep learning (DL) models have become popular alternatives for non- lonerr dimensionality reduction that can be applied directly on raw data. Such models have been combined with joint clustering
objectives in the latent space [Xie et al.] 2016 Yang et al. 2017 Madiraju 2018 Lee & Schaar
2020). These methods, however, do typically not create a (visually) interpretable representation, and sometimes make use of label information during training (Lee & Schaar 2020
To enhance

---

Under review as a conference paper at ICLR 2023
interpretability, latent space representations of DL models are often visualized using t-distributed stochastic neighbor embedding (t-SNE) (Hinton & Roweis 2002 Albeit its frequent use, t-SNE does not allow a direct deployment on unseen data as it does not learn reusable mapping between the multi-dimensional and the low-dimensional space.
To acquire visually interpretable data representations from raw data, without assuming that data must live in two or three dimensions only, non- -inelinearrrr DL encoders have been combined with SOMs Ferles
al.
2018
Pesteie et al. 2018 Fortuin al. 2019 Forest et al. 2019 Manduchi et al. 202 Forest
ct al.
In the resulting joint training strategy of these deep-SOM models, the SOM objective can seen as regularizer on the encoding procedure, as promotes cluster-friendly feature
202
space. Most of these models have focused on autoencoders as feature extractors. However, similar to Mrabah et al. 2020), we hypothesize that their reconstruction objective may hamper the clustering or structured representation learning objective: while within-cluster similarities should remain preserved for latent clustering, reconstruction demands a preservation of all factors of similarity. Moreover, in the context of time series representation learning, other self-supervised models that take the temporal nature of the data into account during training might be more suitable.
Contrastive self-supervised learning approaches have quickly become popular thanks to their superior representation learning performance in many domains (see Lsce|Le-Khae et al. (2020) for a review). While many of these models rely on data augmentations during training in order to construct pairs of similar data points, Contrastive Predictive Coding (CPC) (Oord et al. 2019 leverages the temporal dimension for this purposes, making it a natural choice for self-supervised representation learning of time series. In CPC, the temporal dimension not only serves as a pretext task, but simultancously enforces latent smoothness over time. The contributions of this work are as follows:
We propose a new model in the deep-SOM family: SOM-CPC, which is suitable for learning structured and interpretable 2D representations of (high-rate) time series by encoding subsequent data windows to a topologically ordered set of quantization vectors.
Using regression and classification probing tasks, we show that SOM-CPC preserves more information in its 2D representation than CPC that is followed by PCA, and a linear classifier or K-means, or directly encoding CPC's latent space to two dimensions. SOM-CPC' joint optimization, moreover, facilitates smooth temporal trajectory through 2D space.
We show that SOM-CPC quantitatively and qualitatively outperforms deep-SOM models with reconstruction objective in terms of both clustering and topological ordering. It, moreover, requires less auxiliary loss functions (and associated hyperparameter tuning) thanks to its natural tendency to incorporate temporal smoothness. Lastly, SOM-CPC's training behavior shows that the SOM clustering objective better aligns with the CPC objective than with reconstruction loss.
2 PRELIMINARIES
2.1 KOHONEN SELF-ORGANIZIN MAPS
Kohonen's 's Self-Organizing Map (SOM) (Kohonen) 1990) is an algorithm find a visually interpretable topological data representation. It has been found useful to reveal intricate patterns and structure in a plethora of applications. The algorithm's output, the low-dimensional visualization. is often referred to as a SOM as well. We choose to use use a 2D visualization to enhance interpretability.
We define set of data points 2, and quantized counterparts (z) E T for z E Z. The set } is a trainable quantization codebook containing vectors or prototypes Φi E R h. The jmh prototype (n) = do (z) is the 'winning vector' for data point at iteration of the training procedure. The learned codebook vectors are placed on a pre-defined 2D grid by assigning an xy-coordinate to each vector at initialization. Note that this creates 2D representation, while each data point z still lives in R. with F >> 2. This is conceptually different than the way in which PCA achieves dimensionality reduction to 2D, where all information in the 3rd and higher principle components is strictly omitted. During training of a SOM, each di is updated as follows (Kohonen 1990). with E Z:
(n+1) (n) +n (n) Si (1 ()))) (! 6
2

---

Under review as a conference paper at ICLR 2023
where 17(*) is time-decreasing learning rate. Topological neighborhood structure is promoted via neighbourhood kernel that weighs nodes inversely proportional to their distance with the winning node. A Gaussian kernel is often used which weighs node according to:
( exp (- 2(0(m))))
with and
/ | P(o: ((oo 112
((1) = o exp(((n/))
(3)
where P projects a codebook vector to its corresponding coordinate on the grid, (00) denotes the initial standard deviation, and À the decay factor. Setting À = -Reax log(o(((m)) (o(0) sweeps o between (0)) and a in 7mmx steps. The dependence of S; on the distance dj,i, implies a weighing of I for the winning node (i.e. distance equals zero), and lower than 1 for neighbour nodes. Note that other neighbourhood structures have been proposed as well, for example using the four closest neighbours on the grid, which results in a kernel with plus-shape (Fortuin et al.) 2019).
2.2
DREP-SON MODELS
All deep-SOM research has focused on combining autoencoders (Ferles et al. 2018 Pesteie
2018 Fortuin et al.) 2019 Forest et al.] 2019) Manduchi et al. 2021; Forest broadly be summarized (VQ) VAE van den Oord et al.) (017). with topological organization of the vectors in the quantization codebook: the SOM. The models are trained end-to-end using error backpropagation of both a reconstruction task loss Lusk and loss Liopo that encourages topological ordering the SOM. In general, deep-SOM training objective takes the following form:
eet al.]
al.] 2021 with SOM. These models
vector-quantized
Leeup-SN = Linc + CoCuoo
(4)
with Lupo
R.g Σ (6
1 (5)
Hyperparameter a controls the trade-off. The topological loss thus replaces the original update rule of the SOM algorithm (see eq. (T). The features z E Z are jointly optimized, and thus also depend on n now. To prevent clutter we will, however, omit the (n)-superseript in the following.
Fortuin et al. (2019) propose the SOM-VAE model. As opposed to VQ-VAE, SOM-VAE has two decoders, as also decodes the continuous latents. Topological organization of the codebook vectors is enforced by using a plus-shaped neighbourhood kernel, which affects the codebook vectors the direct neighbours of the winning node (i.e. up, down, left, and right on the grid). The encoder parameters are, however, unaffected by the quantization error of these neighbour nodes. To facilitate the latter, the topological loss was split in a commitment loss (committing the winning codebook vector and vice versa) and SOM loss
(pulling the codebook vectors of the neighbours to z): Lusoo Leommitment + LsoM Formally:
Coommimmen = Ez E2|11= 211
(6)
and
LsOM = Eg
Σ (=1
(7)
with sg[-] gradient blocker that impedes gradient updates the encoder. Note that for 0 B/aa the proposed neighbourhood plus-kernel is a coarse approximation of the Gaussian kernel. The sum of the reconstruction losses Creconccont and Erecondise from the continuous and discrete decoder, respectively, yield the total task loss Lisk, which is combined with the topological loss to create the training objective of the SOM- VAE model.
SOM-VAE- (Fortuin et al. 2019) and (T)-DPSOM (Manduchi et al. 2021 are extensions of SOM-VA -AE. SOM-VAE VAE-prob enforces smoothness over time by adding a transition loss (multiplied by 7) to optimize a first-order Markov model to learn the node transition probabilities, and smoothness loss (multiplied by t in there work, we will use C here) to minimize the quantization error of highly probable transitions. DPSOM is probabilistic model, based a variational autoencoder with non-degenerate approximate posterior (Kingma& & Welling 2013) with soft cluster assignment and cluster assignment hardening (CAH) loss (Xie et al. 2016). T-DPSOM additionally incorporates a temporal smoothness loss, and an LSTM, which aims to predict the future latent space. This latter functionality is similar the future-prediction task that is already naturally embedded in the
3

---

Under review as a conference paper at ICLR 2023
SOM-VAE
b) DESOM
d) SOM-CPC
.(1)
R10
RCO
S0M
10
RCF
Suserssss 2(1)
200
2B P0
f+
x(O)
Forward pass
Badipropagation parameters
SON
Figure 1: Architectures of different deep-SON SOM models including the gradient paths in green. a) SOM-VA VAE (Fortuin al. (2019) h) DESOM (Forest et al. 2021) SOM-CPC (ours). The two decoders in the SOM-VAE model independent and have their own trainable parameters, while the visualized encoders in the SOM- CPC model are all the same (i.c. parameters are shared). The ge block in the SOM-CPC model indicates an autoregressive component (e.g. a GRU), and 2,, refers to a set of drawn negative embeddings.
CPC objective that we propose as a task loss (see section 3.2).
The probabilistic additions in the (T-)DPSOM model, with respect to SOM-VAE, are orthogonal to the developments in this work.
Forest et al. (22TT) propose Deep embedded SOM (DESOM), Compared to SOM-VAL the decoder on the discrete space is omitted (therewith also Lrecon.dis), gradients from LsOm the encoder not
being blocked (i.e. sg[-] is removed from eq. (7 ). and the topological loss Liopo is given by eq. (2) i.e.
with Gaussian neighbourhood function with decaying variance. In short work, |Forest al. (2019)
speculate about adding an LSTM the latent space to train a SOM on sequential data, and refer to this model as LSTM-DESOM. Figure Th-b visualizes the SOM-VAE and DESOM architecture.
SOM-CPC
3.1 MOTIVATION
In this work, we propose the SOM-CPC model, representation learning model that learns to map windows of time series data to structured 2D grid. The model jointly optimizes a temporal contrastive learning objective extract features, and topological loss that organizes the SOM space.
In order to learn features that are both suitable for SOM organization and accurately reflect the data, the model should ideally invert the original data generating process (which is in general unknown and implicit). Assuming that this generative process has been highly non-linear, feature learning can be formulated as a non-linee non-linear independent component analysis (ICA) problem, which has proven to be non-identifiable (Hyvärinen & Pajunen 1999). However, recent advances showed that the problem becomes identifiable under the assumed presence of an auxiliary variable (Hyvärinen et al. 2018) Such auxiliary variable (e.g. temporal component) contrastive learning paradigm was shown to conform to this assumption (Hyvärinen et al. 2018
present in plain autoencoders, but the
Zimmermann et 202I). This theory is in line with the hypothesis stated by Mrabah et 2020 that a reconstruction objective may hamper clustering performance in the latent space.
3.2 ALGORITHMIC DETAILS
We introduce X' = (..., (t), xt + .)....).a. set of non-overlapping data windows xt) E Rexxx with ch the number of channels, and T'' the number of samples the window. For brevity we omit the time index when possible. An encoder, parameterized by 0, maps each data window æ latent representation z = fo E R², with F' the number of features. The set Z includes the embeddings of all windows in X. A causal auto-regressive (AR) module Io parameterized by v. e.g. a gated-recurrent unit (GRU). subsequently aggregates the current and previous embeddings. to generate (current) context vector e(t) IR. Given this context, the pretext task in SOM-CPC
4

---

Under review as a conference paper at ICLR 2023
model aims to minimize the prediction error for P future (or 'positive") embeddings z(t + p). for E (1....,Pl P), compared to this error for N 'negative' embeddings. These negatives may be sampled across the dataset, within the same time-series, and are the fly encoded to their latent representation during training. The task objective, being the InfoNCR loss (Oord et al. 2019). is defined
exp x + Wp((t)
1 P E with cp= -E log E (z exp ( =/W,e(1))
Cusk == LIntONCE
(8)
with 2, Z set of embeddings of drawn negative samples ( (2)) N), and W, E R trainable mapping between the current context vector and the thh future embedding.
The context vector is not only used to predict future embeddings, it is also the input to the SOM module that selects the winning node. The SOM is optimized using the topological loss Lugos as defined in eq. 5 with Gaussian neighbourhood kernel S, as defined in eq. (2. Depending on the use case, was found to not always be necessary, or even beneficial (due to higher risk of overfitting), to use AR module by to aggregate causal context into the current embedding. If the AR module is not used, the future predictions are made directly from the current (continuous) latent space z (t) instead of c(t). Likewise, (t) rather than c(t) is being quantized by the SOM module. Depending on the presence of this AR module, both Lueo and Lusk are thus computed on either xt)) or c(t).
All model elements optimized jointly, with the training objective being: LSOM-CPC Lusk auupo which adheres to the general objective of a deep-SOM model as formulated in eq. (4p. Figure [e provides an overview of the SOM-CPC model, and its gradient paths in green. The initial standard deviation σ()) of the Gaussian kernel (from eq. (2 was set to half the squared-root of the number of SOM nodes k. Given the square topology of the SOM grid, this setting of dg ensures that the full grid is captured by the neighbourhood kernel at the start of training. Algorithm I in appendix A.
provides pseudocode of the full SOM-CPC algorithm.
3.3 PERFORMANCE EVALUATION
Forest et al. (2020) provide a taxology of SOM metrics that distinguishes external vs internal and topological vs clustering metrics. External metrics are related to labels (which are not used during unsupervised training), while internal metries do not depend on such information. Topological metrics assess the topological ordering (i.e. neighbourhood relations) of the SOM, while clustering metrics are more related to, for example, pureness of nodes.
To evaluate clustering performance, linked to external labels, we leverage purity and the normalized mutual information (NMI). The latter corrects for a high number of clusters (i.e. nodes), which could easily lead to high pureness, but leaves the NMI more conservative.
Even though scoring high on external metrics is not the main goal of a representation learning model like SOM-CPC, we do report it as it provides indication of how well information was preserved. To compute aggression/lassifiiaan occurring (for discrete labels) or median (for continuous labels) label from the training set. The test set predictions are then converted from node indices to label predictions by using these colorings. Regression performance is expressed as the average squared regression error with the target: SEunget- Classification performance is reported with Cohen's 's kappa (Cohen 1960), commonly used metric that corrects for correctness by chance.
performance, we first 'color' (or label) each node with the most
Topographic performance is measured using the (internal) topographic error (TE) Kiiiiluoto 1996), which reports the fraction of windows (between 0 and 1) for which the winning and second-best winning node are not neighbours in the SOM (lower is better). Finally, to measure whether a time series conveys a smooth trajectory through SOM space, we measure the average Euclidean distance (denoted l2, smooth) between all subsequent windows in each time series. The lower this value, the less frequently large jumps in the 2D map occur. Note that in extreme cases where many windows collapsed to the same node, both the TE and the average &g &2,mootth metric are artificially pushed down. We can thus only interpret these metrics in conjunction with mentionde clustering and classification metrics.
5

---

Under review as a conference paper at ICLR 2023
Table 1:
Mean and one std. dev. across test set series synthetic data. Results varying values of a, 7 and ( can be found in table (appeeiiai AA33..) SOM-CPC clearly outperforms all baselines. Models below the dashed line are ablations. Regression plots and SOMs of models with a are visualized in fig. t-c.
Model
SEuee
smooth
TE
CPC
262:22.37
+ K-msins
CPC 2) Linear classifier
25.01
(PC 2)
26-h-1.31
lineur
42.81-4-88112
CPC a PCA K-means
4.42:29.01
SOM-VAE
Plus
8.02::::35 58
2.412.68
.2:20070
VAE
1.921.34
SOM-WV PaE--robb
Plus
/
20.10:2-48.88
3.154.73
63:2.000
DESOM
Canveian
K
10:77:110.85
2.202:444
061 019
SOM-CPC
Ganvsita
724
1,372,37
.022::0011
Attaton
SOM-CT CPC
le-2
Caussian
.7:1,48
:9::24
069:2.000
SOM CPC
1e-4
Plus
1.71
2.96
,30:-00062
SOM- CPC
le-2
Plus
116:1:61
L8545.26
.12:2.037
(disjoin))
1.11..14
1.4.2.50
028:2.01
3]
SOMAVAE
b)
DESOM
SOM-CPC
d] Training task topological loss
(valication set)
50
SOM-VAE (Gaussian leernel)
Epoch
1.0
and
20
10
DESOM
0.5
Target frequencc
50
0.0
30
SOM-CPC
0.5
1.0
10
30
50
1013
10
10
107
frequency [Ha]
Frope
Figure 2: SOMs and regression plots for SAE (a). DESOM (b) and SOM-CPC (c). Both DESOM and SOM-CPC show a gradual change of frequency over the grid, but the regression error SEnages lower for SOM-CPC, which can also be seen from the regression plot, where the predicted window frequencies are plotted against the target frequencics (i.c. training set median label) for the node on which the window was mapped. d) Task loss versus the topological loss for SOM-VAL (with Gaussian neighbourhood), DESOM and SOM-CPC (both with and without sgl-)). The different curves display various values of for which the DESOM model seems most sensitive. The SOM-CPC models follow a smooth optimization curve, minimizing both task and topological loss, while these losses seem to be more conflicting in SOM- VAE and DESOM training.
4 EXPERIMENTS
We compare SOM-CPC to several other 2D representation learning methods. First, deep-SOM models with a reconstruction task loss (i.e. SOM-VAE, COM-VAE-prob. and (GRU-)DESOM), Second, vanilla CPC with multi-dimensional latent space (F >> 2) (Oord et al. 2019), followed by PCA for additional dimensionality reduction to 2D. Third, CPC with a 2D latent space (F 2). For the latter two CPC-based models, linear and non-linear read-out is, respectively, tested using linear neural classifier, and K-means clustering with the same number of clusters as the number of nodes used in SOM-CPC. nigh-dimensional vanilla CPC (F > 2) without additional dimensionality reduction is, moreover, tested as well as it sets a baseline for the amount of information that can be preserved given the encoder architecture, while not providing an interpretable 2D representation. The same encoder architecture is used for all models that are compared in single application domain, and all models are run with the same seed for randomization. Details on model architectures and training settings for the different applications can be found in appendix 1 A.4.2 and A.5.I
4.1 SYNTHETIC DATA
Data generation: A synthetic dataset was created, consisting of sinusoids with an initial frequency sampled from uniform distribution between 20 and 40 Hz. The frequency of the signals altered over time according to a random walk process with a step size of 0.1 Hz. As such, at each time step (i.e. sample), the signal's frequency either increased with 0.1 Hz (with probability Pup 0.1),

---

Under review as a conference paper at ICLR 2023
Gaussian kemel $
Plus kernel
Init
200
400
600
800
1000
Epoch
Figure 3: Progression of SOM training (nodes indicated in black) in the SOM-CPC model, using either Gaussian (top) or plus neighbourhood (bottom) kemel. The PCA projection of the test set latent space is plotted behind the nodes. It can clearly be seen that the Gaussian kernel enforces more strict organization of SOM nodes, where nodes are non-uniformly quantizing the latent space, placing more nodes at higher density areas.
decreased with 0.1 Hz (Piown 0.1), remained constant (p. = 0.8). In case the random walk crossed either or 60 Hz, the probabilities were (temporarily) altered to [Pup- Pe- Poowal = [0.5, 0.5, 0] or PPp- Pc: Paown] = [0.0, 0.5, 0.5], respectively. All series were finally corrupted with additive white Gaussian noise vector ~ (o 0.01). Formally, each generated signal took the form: a: [r] sin 2 [m-1)++in)) + e, with I/m 0] ~ U[20, 40). Aʃ ~ Categorical Pe: Pdown)). and fa = 128 Hz the sampling frequency. A total of 200 of such time-series, each of 5 minutes, were generated, and labels were defined per 1-second window by taking the median frequency. The set was randomly divided into training (n 100), validation (n 50), and test split (n 50).
Results: Table I shows that SOM-CPC outperforms all deep-SOM baselines on all metrics. Figure appendix A.3.2 shows the PCA projections of the (continuous) latent spaces of the three deep-SOM models indicated with a * in table It reveals that the latent space disentanglement of the SOM-CPC model is much better than that the SOM-VAE and DESOM models. Figure - displays the resulting SOMs (colored with the median test set labels) for the same three models. Uncolored nodes in the SOM were not assigned in the test set. Interestingly, although the SOM for the DESOM and SOM-CPC model look similar, the SEaager is higher for the DESOM model, which can also be seen from the regression plots below the SOMs.
The addition of two temporal losses in the SOM- VAE-prob model, as compared to SOM-VAR, did deteriorate the given metrics, even though a range values for multipliers a, 7 and ( was tested (see table in appendix A.3.2 for the full sweep). The deterioration of the results can be explained by the difficulty of finding the correct scaling factors for these additional losses. Note that the SOM-CPC model automatically incorporates smoothness over time thanks the nature of the CPC task loss, therewith preventing additional hyperparameter tuning.
Additionally, study the optimization behavior of SOM-VAE, DESOM, and SOM-CPC by plotting the progression of the task versus the topological loss during training (see fig. ( To make a fair comparison, we plot the SOM-VAE models that are trained with Gaussian neighbourhood kernel. Different curves in the graphs indicate runs with varying values for a, and the line color's gradient denotes the training iteration. The SOM-CPC graphs include the models run with and without gradient detachment of LsOM to the encoder. It can be seen that both losses jointly minimize in SOM-CP4 training, while there is counteracting effect visible for SOM- VAE, and a high influence of the value of a for DESOM training.
Comparing to non non-deep-SOM baselines, it can be seen from table that CPC (with P= 2), and CPC followed by PCA, resulted in much higher regression error SEungee than SOM-CPC when using linear read-out. Non-linear K-means clustering improved performance for both cases, but only for CPC with F 2, performance nearly reached SOM-CPC performance. Later we will see that optimizing CPC with F = can hamper optimization for more intricate data spaces (see section 4.3) slightly better than that of Interestingly, regression performance of SOM-CPC was found to be classification and K-means. the vanilla multi- diieensionnl CPC model (with F = 128), both for linear This could be explained by the additional regularization that the SOM provides in SOM-CPC training.
7

---

Under review as a conference paper at ICLR 2023
We perform several ablation experiments on SOM-CPC, which are reported below the dashed line in table ! Blocking the gradients of the neighbour nodes with respect to the encoder during training (CsOm sg|.] column) slightly improved the regression eπror and temporal smoothness, but decreased the topographic error. Looking at the models with various values for a (reported in table 3 appendix A..22). the effect of gradient blocking can be considered small and ambiguous when considering different metrics. Disjoint training (CPC + SOM) resulted in a less smooth trajectory over time through the 2D SOM space, seen from the higher l2 smooth. Using a plus neighbourhood kernel instead of a Gaussian kernel decreased performance on all three metrics. The increase in TE, is well explainable by the fact that a plus kernel takes into account fewer neighbours (at least at the start of training) and therefore has more difficulty to find a good topological mapping. Figure 3 shows the development of the SOM node spread (projected on top of a PCA projection of the continuous test set latents), during training for SOM-CPC with the two type of kernels. It can indeed be seen that the Gaussian kernel enforces a more strict topological organization. Interestingly, the kernel does not only influence the codebook vectors, but also seems to influence the organization of the latent space, seen from the differently-shaped PCA projections in the background.
4.2 SLEEP
We analyse SOM-CPC on subset of the Montreal Archive of Sleep Studies (MASS) database
OO'keilly et al. 2014). consisting of whole-night polysomnography recordings, for which every 30-second window is labelled with a sleep stage label from {NI, N2, N3, REM, Wake]. The 62
recordings (from 62 unique subjects) were randomly split into a training (n = 48), validation (n = 8)
and hold-out test set (n 7). Details on the data preprocessing can be found in appendix A.4.I
Table 5 in appendix A.4.3] shows that SOM-CPC again clearly outperformed SOM-VAE and DESOM on all metrics. Whether or not the gradients of the SOM loss were stopped towards the encoder did not greatly influence SOM-CPC performance. Topological ordering, measured by TE, and temporal smoothness (42,smoth) deteriorated when changing the Gaussian kernel to a plus kernel, or training high-dimensional CPC and SOM disjointly. SOM- CPC's 's classification performance was higher than that of CPC with F 2, and CPC followed by PCA.
CPC K-means
SOM-CPC
ATM
N8
N2
Figure 4: Deep sleep N3 is isolated from light sleep NI, Wake and REM sleep with cluster of oodium-deep sleep N2.
Figure 4 shows the test set PCA projection of the latent space of CPC (F = 128). with the K-means nodes as black stars (left), and the SOM (right) trained by the SOM-CPC model (nodes are colored with the most-occurring label in the test set). Both visualizations show similar clustering patterns: deep sleep N3 is isolated from lighter forms of sleep (i.e. N1, Wake and REM sleep) by thick cluster of medium-deep sleep N2. However, the higher performance of SOM-CPC (see table 5) indicates that more information is preserved in the 2D space resulting from the SOM-CPC model. The size of the nodes in the SOM map of SOM-CPC indicates the average time in the night of windows on that node. A difference is visible in node sizes within the Wake, N2 and N3 clusters, suggesting possible existence of different sub-categories of sleep within the pre-defined sleep stages.
4.3 AUDIO
For the audio experiments, we use a subset of the publicly available LibriSpeech dataset (Panayotov) et al. 2013). The dataset contains multiple minute-long English voice recordings of 251 different speakers, sampled at 16 KHz. We used the publicly available train-test split, as provided by Oord et al. (2019), and created an additional validation set by randomly selecting 25% of the training set. Recordings of the ten speakers with the longest recording time were selected to alleviate computational burden. This resulted in a total of 150.9, 54.6, and 46.5 minutes in the training, validation, respectively test set. The full model and training details can be found in appendix A.5.
Table 7 in appendix A.5.2 shows the results of SOM-CPC (which includes a GRU for this dataset), compared to different variants of the DESOM model. SOM-CPC outperforms all DESOM variants by a wide margin and for all choices of the a parameter. The difference in performance between DESOM and SOM-CPC is also visible in fig. 5 SOM-CPC has clustered the SOM nodes belonging
8

---

Under review as a conference paper at ICLR 2023
to the same speaker, and seems to group male and female speakers (denoted with the node's shape), while these effects are not present in the SOM of the GRU-DESOM model. Minimizing the InfoNCE training objective of CPC with F 2 was found challenging for this dataset, which resulted in non-competitive performance of the linear classifier and K-means clustering trained on the resulted 2D latent space. Using PCA for dimensionality reduction of the high-dimemnsional CPC latent space (with F = 512) performed better, but still inferior to SOM-CPC.
The SOM of the SOM-CPC model (fig. Ftright) reveals two separate clusters both for speaker 2 (green) and 3 (red), The LibriSpeech corpus contains multiple recordings from each speaker, grouped by (book) chapters from which the speaker was reading. Interestingly, additional analyses revealed that the red and green sub-clusters represented recordings belonging to different chapters: 99.99% of the test-set windows mapped to the upper red sub-cluster belong to the same chapter, while 99.97% of the windows in the lower red sub-cluster belong to another chapter read by this speaker. Similarly, 100.0% of the test set windows in the right green sub-cluster belong to two chapters read by speaker while 98.91% of the windows in the left green sub-cluster belong to another chapter. An auditory inspection revealed that the room acousties of the recordings belonging to the chapters in different clusters were different, causing changes in the signals which the SOM-CPC model has picked upon. This division between recordings of the same speaker is not visible in the 2D PCA projection of the CPC (with F 128) features, as seen from fig. g in appendix
GRU-DESOM
SOM-CPC
Figure 5: SOM-CPC is able to better cluster different speakers than GRU-DESOM. Stars denote women and dots are men.
A.5.2
5 DISCUSSION
We proposed a new member of the deep-SOM family: SOM-CPC, suitable for interpretable 2D representation learning of high-rate data streams. Earlier proposed deep-SOM models mainly used reconstruction objectives. In general, SOM-CPC outperformed these models with a wide gap on variety of metrics. Moreover, it implicitly enforces temporal smoothness, while autoencoder-based models require additional losses and hyperparameter tuning to achieve this. SOM-CPC's task loss was found to align better with the topological SOM objective than a reconstruction loss, as already hypothesized by Mrabah et al. 2020). While for some applications CPC could succesfully be trained with a 2D latent space directly, optimization was found to be hampered in case of more intricate data spaces. Compared to vanilla CPC with multi-dimennional latent space, SOM-CPC enables pattern recognition and knowledge discovery. The SOM objective did not hamper CPC optimization. Even better, in the synthetic setup it had a regularizing effect, resulting in lower regression error than vanilla CPC. The use of a Gaussian neighbourhood kernel, as opposed to a plus kernel, was found to improve the topological ordering in the SOM. No decisive conclusions could be made regarding gradient blocking from the SOM loss towards the encoder parameters. Allowing these gradients to flow did not hurt performance, so for coding simplicity, we would advice to not detach the SOM loss.
Setting an appropriate stopping criterion for self-supervised (SSL) models is debatable. In the SSL literature models with the best test set performance are sometimes reported (He et al. 2019 Fortuin et al. 2019). This is, however, questionable as it may artificially boost reported performance. As such, we created validation set to apply early stopping in all experiments. Another challenge arises when dealing with aggregated loss functions, since not all losses may smoothly decay and the weighted summation of losses may result in a different optimal epoch than the sub-losses separately. Besides, classification performance (often used as proxy for information preservation) does not necessarily align with SOM performance or information preservation (see fig. 7 in appendix AA.43)).
We believe that SOM-CPC will facilitate knowledge discovery in real-life time series and opens up new research directions for representation learning of time series. Directions include investigation to whether additions like the soft-cluster assignment, cluster hardening loss Gaussian latent prior which have shown to improve the SOM-VAE model (Manduchi et al.) 2021 improve SOM-CPC performance as well. Moreover, the CPC objective assumes slowly (or nonnchanging) data characteristics within the time frame in which positive samples are drawn. mudti-modal variational future prediction could possibly improve performance for data that do not meet this assumption.
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All code used to train and evaluate the models as presented in this paper can be found at https //anonymous 4open. Becieee//////MCCP The details regarding model architectures and training settings for each of the application domains are also presented in appendix [A.3.1 A.4.2 and Pseudocode of the proposed SOM-CPC algorithm given algorithm in appendix A.
A.5.I
REFERENCES
Richard B. Berry, Rita Brooks, Charlene E. Gamaldo, Susan M. Harding, Robin M. Lloyd, Carole L. Marcus, and Bradley V. Vaughn. The AASM manual for the scoring of sleep and associated events. Rules, Terminology and Technical Specifications. Darien, Illinois, American Academy of Sleep Medicine, 176:2012, 2012.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, pp. 1597-1607, 2020. URL
http://arxivy wtt.://gggabbbb/aobobbbb
2002 05709
Jacob Cohen. coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46, 1960.
Christos Ferles, Yannis Papanikolaou, and Kevin J. Naidoo. Denoising Autoencoder Self- Seff-Organizing Map (DASOM). Neural Networks, 1351112-131, 9 2018. ISSN 18792782. doi: 10. 10.1016 n/..net 2018.04.016.
Florent Forest, Mustapha Lebbah, Hanane Azzag, and Jérôme Lacaille. Deep Architectures for Joint Clustering and Visualization with Self-organizing Maps. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, volume 11607 LNAL, pp. 105-116. Springer Verlag, 2019. ISBN 9783030261412. doi: 10.1007/978-3--030 20000-2614--91100
Florent Forest, Mustapha Lebbah, Hanane Azzag, and Jérôme Lacaille. A survey and implementation of performance metrics for self-organized maps. arXiv preprint arXiv:2011.05847 2020.
Florent Forest, Mustapha Lebbah, Hanene Azzag, and Jérôme Lacaille. Deep embedded self- organizing maps for joint representation learning and opology-preserving clustering. Neural Computing and Applications, 33(24):17439-1744 12 2021. ISSN 14333058. doi: 10.1007/ s00521-021-06331
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- Vae: Interpretable discrete representation learning on time series. 7th International Conference on Learning Representationss ICLR 2019, pp. 1-18. 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 2019.
Momentum contrast for
Geoffrey Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural information processing systems, 15, 2002.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks. 12(3):429 439, 1999.
Aapo Hyvärinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. In The 22nd International Conference Artificial Intelligence and Statistics, volume 89, pp. 859-868, 2018. htl.ttttp///axiiee /bs/
1805. 08651
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141iv:1412.6900 2014.
Diederik P. Kingma and Max Welling, 1312 2013.
Auto-encoding variational bayes. arXiv preprint
10

---

Under review as a conference paper at ICLR 2023
Kimmo Kiviluoto. Topology preservation in self-organizing maps. In Proceedings of International Conference on Neural Networks (CCNN'6)), volume 1, pp. 294 299. IEEE, 1996.
Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464 110444444800,,, 1990.
Phuc H. Le-Khae, Graham Healy, and Alan F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
Changhee Lee and Mihaela Vander Schaar. Temporal phenotyping using deep predictive clustering of discase progression. 37th International Conference on Machine Learning, ICML 2020, PartF1681A
5723-5733, 2020.
Zachary C. Lipton. The mythos of model interpretability. Quene, 16(3)311-57,00 2018.
Naveen Sai Madiraju. Deep temporal clustering: Fully unsupervised learning of time-domaie features. PhD thesis, Arizona State University, 2018.
Laura Manduchi, Matthias Hüser, Martin Faltys, Julia Vogt, Gunnar Rätsch, and Vincent Fortuin. T-DPSOM: An interpretable clustering method for unsupervised learning of patient health states, volume Association for Computing Machinery, 2021. ISBN 9781450383592. doi: 10.1 1445/ 3450439.345188
Nairouz Mrabah, Mohamed Bouguessa, and Riadh Ksantini. Adversarial deep embedded clustering: on a better trade-off between feature randomness and feature drift. IEEE Transactions on Knowledge and Data Engineering, 2020.
Aaron van den Oord, Yazhe Li, and Vinyals Oriol. Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv: 2019. URL http:://aaa/t /abs/
1807.03748 03748
Christian O' O'Reilly, Nadia Gosselin, Julie Carrier, and Tore Nielsen. Montreal Archive of Sleep Studies: open-access resource for instrument benchmarking and exploratory research. Journal of Sleep Research, 23(6):628 635, 12 2014. ISSN 09621105. doi: 10.r 12169. URL
http:// wi
ey. com 10. 1 111 f 12169
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 122285528830 2011.
Mehran Pesteie, Purang Abolmaesumi, and Robert Rohling. Deep neural maps. arXiv preprint arXiv:1810.07291, 2018.
Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 216-220, 2017.
Aaron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural Discrete Representation Learning. Conference on Neural Information Processing System, 2017.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive Learning of Disentangled Teaeo Representations for Time Series Forecasting. In International Conference on Learning Representations, 2 2022. URL http:://///axxiiorrg
abss2202. 01575
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference machine learning, pp. 478 487. PMLR, 2016.

---

Under review as a conference paper at ICLR 2023
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means-frient friendly spaces: Simultaneous deep learning and clustering. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3861-3870. PMLR, 06-11 Aug 2017.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. arxiv, 2021. URL http::/aax:: org/ brq/abs/Dee 2abb/21022 08850
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
This appendix contains all information for full reproducability of the experiments. Domain- independent details SOM-CPC and its evaluation are provided in appendix [..]] while benchmark implementations are discussed in appendix A.2 Domain-specific settings for the synthetic, sleep and audio experiments are discussed in sections
A.4 and A.5 respectively.
A.1 GENERAL DETAILS
Algorithm I provides pseudocode of SOM-CPC, when considering the presence of an AR module. For 4open. science/
reproducibility. the full code base can be found at
I/SOM- CPC
Algorithm 1 SOM-CPC
Input: Dataset X, model comprising fo, Gv. Y= number of past windows L, # positive samples P, # negative samples N, # SOM nodes k. Gaussian neighbourhood function S with o (fhma), loss trade-off'
parameter a.
Output: A trained SOM: topologically ordered codebook & that represents data X in 2D. Initialize the Gaussian kemee according eq. with: g0)) V and -mmxx/ log( (o (mama) (0(0))
for Ylmux do
Sample a sequence datapoints: [a=( L),... (()).,),,(.,)))))
[1)] ~
Define P positive samples: (ait /==1
Sample negative samples X" X, with |x P
Encode
data sequence: e(I) 96 (Jo((e((1 D) =).....(((1))
positive samples: (=(t + p) = fo ((a((t + p)
negatiie samples: 2' fo (X))
Predict future: 2(( + p) W (()()) with 1 S P
QQaantize: &i=j = SOM, (e(l)
Updltte (n) according to eq.
Compute Lmpp((((1) and LInOONER according to eq. and eq. (8)
Update trainable parameters x aLuu and LisNNCE
end for
Several metrics were used to quantify SOM performance, as explained in section 3.3 The purity implementation is taken from the Github implementation of Fortuin al. (011)) while NMI was computed using the sklearn library (Pedregosa et al. (01I). The topographic error implementation comes from the SOMperf python library (Forest et al.) 2020). Finally, the E2 distance is computed using the norm function in the linalg library of numpy.
Section 3.3 already shortly elucidated upon the way in which Cohen's 's kappa and SE were computed. For both metrics, each SOM node was labeled/colored with the most-occuring (in case of Cohen's 's kappa) or median (in case SEurget) label in the training set. Note that it can questioned whether this node labelling should be done training validation set, directly the test set for which performance is reported. In earlier times when more conventional clustering approaches (e.g. K-means) were used, Raining/validationl/ was directly performed on the one and only (test) set, that was also used to label the clusters/nodes. Moving towards deep learning based approaches where more hyperparameters need to be set and overfitting can become a larger problem, we found it necessary to report in this work on a test set that not used for labelling the nodes and/or setting hyperparameters. It should thus be taken into account, that direct comparison to results in other deep-clustering works may need a critical eye to see whether similar procedures were used or not.
split typically not made. As result, clustering
A.2 BENCHMARKS AND ABLATIONS
To benchmark our implementation of the SOM-VAL model (and the very similar DESOM model), replicated the results on MNIST. MNIST was not used for further experimentation with SOM-CPC in the main body of this paper since this work focuses high-rate time series. Using k = 16 SOM
13

---

Under review as a conference paper at ICLR 2023
nodes, Fortuin et al. (2019) report purity 0.731 = 0.004 and NMI = 0.594 = 0.004 in table 1, and purity = 0.721 = 0.006 and NMI = 0.587 = 0.003 in table S1. All numbers are averages and standard errors over 10 runs.
Settings that were not provided in the paper. were taken from the hard-coded settings that we found in the provided code base. Instead of splitting the standard MNIST training set in training and test split, as done by |oortin et al. (2019), we used the available train/test split that comes with the standard MNIST dataloader from Pytorch. From the first ten runs, one run fully collapsed and resulted in extremely poor performance. Considering this run as an outlier, we run an 11th run and here report the average and standard errors of the 10 collapsllp runs: purity 0.705 = 0.002 and NMI 0.584 + 0.001. Our performance does reasonably well match the reported performance by Fortuin et al. 2019), given the fact that different test set of MNIST was used for our experiments.
To restrict the search space of hyperparameters in the (OMM-VA((prro) model, which has multiple loss multipliers, we fixed some of the these multipliers for further experiments in this paper. Multiplier 3 scales the SOM loss that sums the quantization error of the four neighbour nodes in the plus kemell. is, therefore, expected to be at least 4 times larger than the commitment loss, which only reflects the quantization error of the winning node. Choosing 3 = a/4 4 would imply that the weighing of the summed quantization error of the four neighbours is equal to the weighing of this error for the winning node. To give the winning node slightly higher importance, we set 8 a/5 0.2g. The authors of SOM-VAE (Fortuin et al.) 2019) used, instead, a search strategy to find the optimal setting. Their code base shows that the SOM loss was multiplied with 0.9, while a 1. Taking into account that their implementation of the commitment loss averaged the quantization error of the four neighbour nodes, while we summed the contribution, their effective setting was thus set to 3 0.9 a = 0.225a, which is close to what we used in our experiments.
We compared SOM-CPC also against vanilla CPC training followed by linear classifier or K- means clustering, while freezing the encoder parameters. The supervised linear classifier took in all experiments the form of one fully-connected layer, including biases, that was followed by a log-softmax activation for the sleep and audio cases. It was trained using the mean squared error for the synthetic data set, and cross-entropy loss for sleep and audio experiments. K-means clustering was run from the sklearn library, with the default settings. The number of clusters was chosen to be equal to the number of nodes in the SOM-CPC models against which the performance was compared. Also the disjointly-trann trained SOM had exactly the same settings as the SOM in the SOM-CPC models with which was compared.
Several ablation are performed on the SOM-CPC model. We test the effect of propagating gradients of SsOM to the encoder parameters, the difference between using a Gaussian neighbourhoood kernel versus plus kernel, and the effect of jointly training CPC and the SOM. Moreover, the effect of certain settings that are typically used in the CPC objective are investigated. CPC's InfoNCE objective for one window p, given eq. can follows be generalized more general contrastive learning objective:
exp (sim(zo, 2p)/T)
4. - log X Σ Extezgu(e)) exp (iim(za, #///T)
where za is the latent space of the current (or anchor) window, sim(-) similarity metric, and the
other symbols are equivalent to eq. ( CPC typically uses 7 = 1, and the dot product as the similarity metric. However, other related contrastive learning objectives, e.g. in SimCLR (Chen all 2020). use a temperature value that is often set to 0.07 (Chen et al. 2020 Woo et al. 2022] and a cosine
similarity instead of the (unnormalized) dot product. As such, we add ablations where we set 7 at or 0.07, and use either the dot-product or the cosine similarity as the similarity metric.
A.3 SYNTHETIC EXPERIMENTS
A.3.1 TRAINING DETAILS
Table 2 summarizes the encoder and decoder architectures used in this experiment. The output size column in the table uses channels-fistt notation. The SOM-VAE and DESOM models were
https: //g thub. http://hhh..a..oaeeooeeoee//eeee
py
line 79.
14

---

Under review as a conference paper at ICLR 2023
Table 2: Model details synthetic data experiments in section
Layer type
Output size
Channels
Activation
Kernel size
Strides
Ditation
Padding
Eneoder for SOM-CCC
and CPC
Conv ID
he x 16
128
16
Lenky Rel.I (001)
MaxPoolID
x31
Dropout (0.1)
bs x116 x 32
ID
Leaky ReLU (0.01)
MasPoolID
hi x32 x
4
Dropout (0.1)
Conv11
bs X64 x
Leaky KeLU (0.01)
1
Saune
MixPoolLD
bs x66
Drapout (0.1)
x64
Con/ID
hi x:128 x 2
128
Lesky Rel.U (0.01)
MaxPoolLD
x:28 x
Encodler SOM-VAL
and DESOM
ConviD
x.16 1225
Losky RcLU (0.1)
MaxPoolLD
bs x6 x :32
Drspout (0.1)
Con
hi x32 x :22
Leaky Rel.I (0.0))
(0.0))
7
Same
MaxPoolLD
Drupout (0.1)
bs x32 x
Com ID
bx x61
Leoky ReLU (0.01)
Flatien
x612
Fully Connected
bi x 128
128
Lesky Rel.U (0.01)
Decoder for SOM- VAE and DESOM
Fully Connected
bi x612
512
Laaky Rel.U (0.01)
Unffatten
Com ID
bs x33 x 8
32
Leaky ReLU (0.01)
same
Tranepose
None
Com ID
hi 16 x 322
16
Leaky Rel.I (0.1)
same
Con'Transoose1>>
1228
None
Com- ID
bs x1 128
Tash
SAIN
found benefit from a convolutional part of the encoder that did not fully reduce the temporal dimension to size 1. As such, the last convolutional layer of the SOM-CPC encoder was changed for a fully connected layer preceded by a flattening operation for the nutoencoder-basse models. The SOM-CPC and CPC model are run without an AR module, to make the fairest comparison to the (OM-o))(prob) and DESOM models, which also do not incorporate such component.
For SOM-CPC, P = 3 future predictions (i.e. positive samples) were used, and N 3 negative samples were drawn for each positive sample. The latter were drawn randomly from the entire training set. The standard deviation of the Gaussian neighbourhood kernel was exponentially decayed until o (Phma) = 2. Choosing a lower value at the end of training induced instable optimization behavior.
All models (including the benchmarks) were trained using the Adam optimizer (Kingma Ba 2014 with a learning rate of 0.001 and batch size of 128. Each model was trained for maximally 1000 epochs. The best model was selected based on the lowest task loss on the validation set, being Ceeonn for SOM-VAE and DESOM and LInSNNCE for SOM-CPC and CPC. We did not use the full training objective Laeep-SOM as model selection criterion, as both the commitment and SOM loss showed to be low initially (possibly due to low values of the random initialization of the model), while both increased and reached steady-state later in training. The lincar classifier and the disjointly-trainn SOM on the CPC embeddings were trained until convergence, for maximally 1000 epochs.
A.3.2 EXTENDED RESULTS
Table 3 extends table with additional sweeps of hyperparameters o, and (, and ablations with different settings of the temperature value and the used similarity metric. Compared to SOM-VAE. SOM-VAE-prob is expected to show more smooth trajectories over time, captured in the Eg smooth distance metric, thanks to the additional transition and smoothness loss (multiplied by 7 and C, respectively). It can be seen that tuning these hyperparameters is a complex process, and sweep did not result in one SOM- VAE-prob run that performed better than the best SOM- VAE model. In contrary, the addition of the extra losses possibly interfered with the optimization process, and only very delicate settings of 7 and ( might improve model performance eventually. A sweep over the topological loss multiplier a, revealed a low sensitivity of SOM-CPC to this value. It can be seen that changing the temperature value and/or the similarity metric did not significantly alter the performance consistently all reported metrics.
15

---

Under review as a conference paper at ICLR 2023
Table This is the extended version table on synthetic data results, including sweep of hyperparameters a, y and Ç. Bold values indicate the best performance per column (excluding the upper bound of the vanilla CPC model, which does not result in a 2D representation). The models indicated with a were used to depict trained SOMs in fig- and PCA projections in fig.
Model
ZoOM mgl-
Seerp
smsoch
linerr classifier
2.6222.37
CPC K-meins
1.09
CFC U = 2]+ liner classilier
15.0119291
2) K-means
CPC PCA- lincar classifeer
42.81
CPC + PCA- K-means
4/42:99011
SOM-VAE
11.50:E13.0
4600:466
14.57444
2.84..84
Plus
8.02
2/1
28
Plus
134343366
4775
31
SOMEVAE
16-5
Guisecan
11.1347708
4705
le-4
Gonsezan
1.22#26611
1994
le.3
Gosseaan
11.50+22543
1.22
086 023
Gonssian
18.134
2.09-:42
(y = Se-3. = 1e:3)
le-3
Plus
21.261:5000
3
( = 4e.5. C 1e:3)
1e:3
Plus
21.30
5:23
$8.00
3.3e5.
le:3)
le:3
Plus
22.52:1:48.97
3811.51
98110015
(
14.65:199
$61. 083
(1 = 4e-4 1e:2)
le:2
Plus
27.25
3.541-67
941.022
( = 3.3e4, < le-2)
le-2
Pus
21.62:138.75
3.711.76
97.1 014
(1 Se-5,
1e-7)
Plus
26.87:168 84
3.23
68 067
( 4. 1e:21
(7 = 3.ee5. < le-2)
Plus
2.10
3.134:771
K3
( = Se-4, C 1)
Plus
7589
3.061.55
84 050
4e-4
Plus
29.9511-66.6
2.111.31
82.1.13
(y
3.951.81
DESOM
le-5
Gunssian
14,00
1.99-.36
114 069
1o4
Grasean
19.09444004
1.95
11 046
le33
Gusssaan
12.58:277.10
19244.31
Gussian
43.66224477
1.944:11
Gonssian
10.77:
2.20
061
Gunsian
22.86
2.26
092 034
(OurS)
Canesan
1.211:31
le-4
Ganssan
.222 1.08
1.371.37
022
le.3
Gosssaan
81 81:5.75
1.06+1255
028 + 012
le2
Gonzsian
.62
1.08.1.25
1: 005
1.184.30
Te-5
Gunssian
642 71
1.042:26
099 000
1o-4
Gonseian
68 67
1.19::43
057 033
1.121:30
991:24
ANaiion
_I
Gonssan
.9 1,50
1.16.1.32
069 00
le-4
Plus
1,71
2.46:00511
30:0:002
1.551.76
12:4
SOM-CDC (r 0.07, sim cosine sim.)
le-4
Gansian
1.47
1.511.34
.14:1.015
SOM-CPC (r Lii cosine sim.
le-4
Gomsian
4.26124911
96:2.13
0631 000
SOM-CPC ( 0.07, simo dot prod.)
To-4
Guasiin
1.2222408
1.15
066-2 046
CPC
1.7.1.50
SOM-VAE
DESOM
SOM-CPC
[Hz]
50
40
10
Figure 6: PCA projections of the continuous latent spaces of the full test set for the SOM- VAF, DESOM, and SOM-CPC models of which the SOMs were visualized in fig. Disentanglement of the signal frequencies much better in the SOM-CPC model.
Figure 6 shows PCA projections of the latent space of the SOM-VAE, DESOM, and SOM-CPC models that are indicated with a * in tables and and for which the SOMs were visualized in fig. 2 Disentanglement of the signal frequencies is much better for the SOM-CPC model, providing explanation for the better (i.e. lower) SEaaget of this model, as compared to SOM- VAE and DESOM.
16

---

Under review as a conference paper at ICLR 2023
A.4 SLEEP EXPERIMENTS
A.4.1 DATA PROCESSING
For the experiments on sleep data, we used subset 3 of the publicly available Montreal Archive of Sleep Studies (MASS) database (O'Reilly et al. 2014), consisting of 62 whole-night polysomnography recordings. Each recording contains, among others, electroencephalographe (EEG), chin electromyography (EMG), and electrooculography (EOG) data. We refer the reader to O'Reilly et al. (2014) for more details regarding this dataset. We selected the channels that are typically used in clinical practice, comprising three EEG channels (F4, C4, O2), the two EOG channels, and one chin EMG derivation, and downsampled the data to 128 Hz. Sleep stage labels that follow the guidelines of the American Academy of Sleep Medicine (AASM) Berry et al. (2012) (being wakefulness (Wake), rapid-eye movement (REM) sleep, or non-REMI till non-REM3 (NI, N2, N3)) were available for every non-overlapping 30-second window, the common label resolution in clinical practice.
Some processing had already been done by the distributors of the MASS dataset (O'Reilly al. 2014). The 60 Hz powerline interference was, however, not fully suppressed, and we wanted to down sample each signal to 128 Hz to reduce computational complexity. As such, before downsampling, all derivations were additionally filtered with zero-phase (i.e. two-directional) 5th order Butterworth band-pass filter (0.3- 59 Hz), followed by another zero-phase 5th order Butterworth notch filter (59 - 61 Hz). Channels were normalized within within-patient and per channel, yielding mean subtraction, followed by normalization such that amplitudes of 95% of the samples were mapped between -1 and +1. The 62 recordings (numbered 1 - 64, with number 43 and 49 missing) were split into training set including patients 48 (n = 47), a validation set including patients 50 - 57 (n = 8), and hold-out test set that included patients 58 64 (n = 7).
A.4.2 TRAINING DETAILS
Dimensionality reduction of polysomnography data was done by encoding all selected channels in each non-overlapping 30-second window using standard convolutional encoder. Table 4 summarizes the used encoder and decoder (for SOM-VAE and DESOM) architectures. The latent space for decoding in the SOM-VAN and DESOM benchmark models was not fully reduced to a 1D vector to enhance training. Nevertheless, the last adaptive average pooling layer that was used in the encoder of SOM-CPC, was applied in the bottleneck of SOM-VAE and DESOM before SOM quantization took place. As result the feature vectors in all models were of size F 128. The decoder architecture (see 4 was used both for the continuous and discrete decoding in the SOM- VAE model (without weight tying). No AR-component was used in the SOM-CPC model to make a fair comparison the SOM-VAE and DESOM model that also did not include such a component.
For the SOM-CPC model, P future predictions (i.e. positive samples) were used, and N negative samples were drawn for each positive sample. The latter were drawn from the same subject as the positive sample. The a of the Gaussian neighbourhood kernel was exponentially annealed to (Prm 0.5 during training.
All models were trained with the Adam optimizer (Kingma & Ba 2014), with a learning rate of 1c-4 and batch size of 128. Each model was trained for maximally 500 epochs, and the best model was selected based on the lowest Ereom (for SOM-VAN DOM- and DESOM) or LIatNNE (for SOM-CPC and CPC) on the validation set.
A.4.3 EXTENDED RESULTS
Table shows the quantitative results on sleep data, comparing SOM-CPC against deep-SOM models (SOM-VAL and DESOM) and disjoint training of CPC, followed by either a supervised linear
classifier, K-means or SOM. Discussion of the main results in this table can be found in section 4.2
The ablation experiments in which the value of r and/or the similarity metric was altered show that classification and clustering performance slightly dropped when using the cosine similarity with temperature value of 1, while the topographic organization slightly improved (i.e. lower TE). These effects vanished when using a temperature of 7 = 0.07. Both runs showed worse temporal smoothness (i.e. higher f.,ot))t) As expected, only changing the temperature value to 0.07 did almost not affect results, suggesting that the linear projector heads were able to adjust for this scaling factor.
17

---

Under review as a conference paper at ICLR 2023
Table 4: Model details for the sleep experiments in section 4.2
Layer type
Output size
Channels
Activation
Kernel size
Strides
Mdation
Pndding
Knender for SOM-CCC and CPC
Conv ID
bu 128
16
Lony (001)
15
MaxPoollID
bs x16 x 32
5
5
Dropout (U.1)
bt bs x32 x 32
Con-ID
32
Leaky ReLU (0.01)
9
MaxPoolID
be x32 x 8
5
Dropoct (0.1)
he x32 X 8
ConvID
bi X64 x 8 bs by x64 x 2
64
Leaky ReLU (0.01)
D
MaxPoolLD
2
Dropout (0.1)
ConclD)
Is 11
178
1 Emak Rel.U (001)
AdapiieeAogoollI
bi x8 x
Enoxder for SOM-VAE
Com(1D
bs x x 128
16
Leoky ReI.U (0.0))
15
(18,17)
MaxPoolLD
bs x
5
Dropoct (0.1)
bs x16 x 32
ConvID
be x.3 x 32
32
Leaky ReL.U (1)01)
9
MixPoolID
he x32 x
Dropoct (0.1)
bi xx22 x 8
CD
bs bu X64 x 2
Leaky ReLU (0.01)
MasPoolDD
5
Dropout (0.1)
Is bi x1 x 2
ComclD
128
Leaky Kel.U (0.01)
3
Dernder For SOM-VAF DESOM
ComvlD
bs XG4 x 2 bs x.64 x 2
64
Leaky ReL.U (0.01)
8
None
5
Com-ID ID
bs x32 x 2
32
Leaky ReLU (0.01)
5
Con Tr
hi xx 2
32
Nome
5
ConcID
be X16 2
Leaky ReL. (0.01)
9
Com-Transposelll
bs X6 x
16
None
5
Cun-ID
X6
Nome
15
We also tested the performance when using the SimCLR (Chen al. instead of the CPC objective. SimCLR is also a contrastive learning framework, but instead of drawing positive samples from the future latent space, these samples are created by applying augmentations on the anchor window. Inspired by Um et al. 2017 we used the following augmentations: independent and identically distributed Gaussian noise NO, 0.05) was added (called jitter in their implementation), each channel was scaled with a value drawn from (0, 0.1), windows were split in 4 sub- windows of minimal 2 seconds and randomly permuted, and lastly time series were both time warped and magnitude warped. The latter two augmentations make use of smooth curves that smoothly vary the positions of time stamps or magnitude values, respectively.
2020 objective for the task loss,
Besides the difference on how to create positive samples, the originally proposed SimCLR model has some other slight differences with respect to the CPC model:
The SimCLR loss uses the cosine similarity, while CPC uses the (unnormalized) dot product
as the similarity metric (see eq.
SimCLR uses an additional temperature in its loss function (see eq. g ), for which the
value is often set to 0.07 (Chen et all 2020 Woo et al.) 2022 CPC does not incorporate such temperature, which effectively means that uses value of 1.
SimCLR uses non-linear MLP projection head, while CPC uses linear projection heads.
SimCLR uses negative samples from within the batch, while this is not specified in the CPC paper. This specified design choice makes SimCLR typically very sensitive to the batch size.
SimCLR was not proposed to include an auto-regressive component, and can not straightforwardly be extended to do so, while CPC can be implemented with or without such a module.
For the most fair comparison, the procedure for drawing negative samples in SimCLR is done equivalently as for SOM-CPC, i.e. within the recording, instead of within the batch. However, in the C0M-SimCLR model (i.e. the joint training of SOM with SimCLR), each drawn negative sample is added to the set of negative samples both in its raw form, and with a random augmentation, which effectively doubles the number of negative samples. Table 3 reports the performance of the baseline SOM-SinCLR model (i.e. with settings 7 = 0.07 and the cosine similarity), and variants using a temperature value of and/or the dot product as the similarity metric. All settings regarding training procedure and the SOM were set equivalently as in the SOM-CPC training. Table 5 shows that SOM-SimCLR results for 7 = 0.07 are better than those with T = 1, which is in line with findings from Chen et al. 2020 Woo et al. 2022p. However, even with 7 = 0.07, performance of SOM-SimCLR is lower on all metrics compared to the SOM-CPC model with the same value for lower
o. The higher f2 metric of SOM-SimCL. R indicates on average larger jumps over the SOM
18

---

Under review as a conference paper at ICLR 2023
Table 5: Test set performance of various models trained on sleep recordings. SOMs of models with a are visualized in fig. Bold values indicate the best performance per columa (excluding the upper bound of the vanilla CPC model, which does not result in 2D representation).
Model
Cura
Purily
NMT
Coben's
smeech TE
CPC chesalier
Komeans
CPC 2) linzar classifer
524..10 10
CPC(F K-meuns
.55.
CPC-4 PCA K-menns
.77
266
SOM-VAE
lo-3
Plus
771
23
512,004
2361 26
24 031
Plus
.22
52.2.03
2.601:34
.2822.0
Plus
.11
23
53 03
3.8.1 30
.1 054
Ic-6
Goassian
53
019:2
le.5
2.104-26
11=:008
lo-4
Gunssan
.71
51
2.854-74
172
lo-3
Gonssan
2.40 16
.224 0085
Gozssan
SOM SIIC (T = 0.07)
le:
Gomssan
.13
53
1
.293
SOM SimCLR (7 1)
le-3
Gomesan
.70
20
481.11
1.871 30
.0.1: 068
SOM-CPC
le-5
Gansian
To-4
Goassian
le:3
Gosssian
.78
11
1.02
J832 DM96
Te-2
Ganssian
77
2N
802 11
00
1922 042
Goassian
109
Gonssan
T.02-111
06771
le
Gonssian
.78
27
50
1.074.09
.178.04
Ganssian
.78
27
821.10
1.061.111
0884
Goassian
079-20099
Ie:3
Plus
$11.11
1.351:24
.265..07
1o.3
Plus
.79
102.10
1.88
.27 080
SOM CPC cosine
le.3
Gonssian
136.-11
.000-005
SOM.CPC (r 0.07, sim doe prod.) CBC++ SOM (disjoint)
le:3
Gonssian
.79
28
524
1006
0059:: 020
Gozssian
.79
.28
824.11
121-4.11
32440042
InfoNCE
Commitment loss
SOM loss
Purity
NMI
Cohen's kappa
0.10
0.00
0.70
0.5
0.70
0.06
3
0.85
0.20
0.6
0.15
0.4
0.02
0.55
Thaft
0.50
Figure 7: Training curves of SOM-CPC (the model indicated with in table 5) for both the training and validation set. The green dashed line indicates the epoch of the used model, i.c. the with the lowest validation InfoNCE loss. can seen that epoch with the best clustering classification performance does not necessarily align with the epoch
the lowest loss commitment and/or SOM loss.
map through time, which might be caused by the fact that the SimCLR task objective does not incorporate temporal information, while InfoNCE does exploit this. Training time of COM-SimCLP was, moverover, considerably longer than SOM-CPC with the same settings due to the additional augmentations that need to be computed for every data window and its negative samples.
Figure 7 shows training curves of the training and validation set for the SOM-CPC model that is indicated with a in table The green line indicates the epoch with the lowest InfoNCE validation loss. These graphs show that the performance of InfoNCE, the commitment and SOM loss, and classification metrics do not necessarily align, making it dependent on your final goal with the SOM-CPC model what the most appropriate stopping-criten criterion.
19

---

Under review as a conference paper at ICLR 2023
A.5
AUDIO EXPERIMENTS
A.5.1 TRAINING DETAILS
Audio streams were encoded in windows of 0.01 seconds (=160 samples). For CPC, GRU-DESOM, and SOM-CPC the contextual information of L 127 previous windows was aggregated using GRU, equivalent proposed by Oord et al. 2019) The InfoNCE objective for (50M-JCPC was computed on top of the last context vector
GRU. To test different settings for the GRU-desom model, distinguished GRU-DESOM that decodes only the last window, given the last context vector, and GRU-DESOM that decodes the full sequence of 128 windows from the respective context vectors.
Table 6 provides the model details of the encoder, and the decoder for the DESOM benchmarks. The reconstruction loss of the DESOM model was found to be hampered in its optimization when using the encoder architecture, as adopted for the SOM-CPC model. As such, the downsampling factor of the encoder reduced for the DESOM model enable minimization of the task loss during training.
Table 6: Model details for the audio experiments in section 4.3
Layer type
Output size
Channels
Activation
Kernel size
Strides
Dilation
Padding
Eacoder for SOM-CPC und CPC
Coev
512
CoevlD
bs x 512 x
512
ReI.1
4
CoavID
512
ReLU
ConvID
bs x 512 x
512
ReLU
2
ConvID
bs x 512
512
ReLU
2
512
Encuder for SOM-VAE and DESOM
CreviD)
x 32
Rel.I
CoavlD
bs x 512
512
ReLU
4
CoaviD
ConvID
bs x 512 x
512
ReLU
2
ReL.
Flatten
bs x 1024
1024
Decodler for SOM-VAR and DRSOM
Unilutten
Conv
x 512 x
512
ReLU
same
Coev DramTrpposeeI
bs x 512
512
ReLU
2
DnvTTrasppoeID
Rel.1
Con/TransposeDD
bs x 512 32
512
ReLU
Con/TransoseeID
ReLU
cutput pad
For training of SOM-CPC, followed the settings from Oord al. (2019) and P 12. The number of negative samples was set to N = 10, which were drawn randomly from the entire training set. All deep-SOM. models were trained for maximally 3000 epochs, using the Adam optimizer Kingma & Ba (2014) with a learning rate of 1e-4 and batch size of 8. One epoch was defined as push through of one sequence 128 windows (or window for DESOM) from each recording. The best model was selected based on the lowest validation task loss.
The supervised linear classifier and disjoint SOM training on top of the frozen CPC embeddings were trained with batch size of 128, and learning rate of 1e-4 and 1e-2, respectively. Both models were stopped upon convergence the validation loss (which was after 200 and 250 epochs, respectively).
A.5.2 EXTENDED RESULTS
Quantitative results of the audio experiments can be found in table 7 For this application, Cohen's 's kappa is computed as the average over all data windows in the test set, which is different from the synthetic and sleep case, where was computed as the average recordings. In these audio experiments, all windows from recording contain the same speaker id label. Computing Cohen's 's kappa per-recording, i.c. with having the same label for all windows in that recording, is therefore inappropriate the computation can not correct for correctness by chance. Table that SOM-CPC clearly outperformed all variants the (GRU-IDESOM model, and feature extraction using CPC, followed by PCA and linear non-linear classification.
one standard deviation across
Ablations with respect to the temperature T and the similarity metric the loss function indicated, similarly as in the sleep case, that simply changing the temperature value did hardly affect SOM-CPC
20

---

Under review as a conference paper at ICLR 2023
performance. However, changing the similarity metric to be the cosine similarity caused drop in clustering and classification performance, only when using a temperature value of 1. This result overlaps with the experimental findings in the sleep case (see appendix [A.4.3). and suggests that
low temperature value which was found beneficial the SimCLR objective (Chen et al. 2020 Woo
et al. 2022 does not equivalently improve SOM-CPC performance.
Figure compares the test set projection on the 2D PCA space, created on the CPC features (with F 128), the SOM from the SOM-CPC model that is denoted with in table (and also visualized in fig. rrght)..
Table 7: Test set performance of various models trained on audio recordings. SOMs of models with a are visualized in fig. Bold values indicate the best performance per column (excluding the upper bound of the vanilla CPC model, which does result 2D representation).
Model
Lsns dR
Parity
NMI
Cohen's s kappa
CPC lincur clavsifier
1.00
CPC K-meens
1.00
.60
1,00
2) classilier
CPC (F' 2) K-meuns
.13
.013
.055
CPC PCA + lincar classiierr
.66
CPC PCA K means
144
le-4
Gonssion
X
.33
.08
.11
34:2. 045
1o-3
Canviia
31
.13
.00
6682 050
Ganesina
1.0:2.00
GRL DESOM teconsttuucnng last window)
Ie-5
Guusiia
-19
.0S
.08
.13-.002
le-4
Gaussian
X
26
.09
.15
.20.5.033 0.36
le-3
Caussian
311
-21
(423. 076
GRU-DESOM reconstructing full sequence
1e-5
Guussins
.19
.05
08
34 0488
le4
Cousiio
.20
59.h 072
Guussian
SUM-UP. (OuS) (oun))
16.5
Gaussius
-59
.73
.99
114::
Gaussian
1.00
1.00
.24:8.12
.33-2
1e-2
Guussia
1.00
.61
.99
3336.099
le-3
Caussian
1.00
28:11 087
.5.10.10
ANHEOE
Caussia
1.00
.61
1.00
38:2.11
SOM-CPC 0.07, sim cosine sim)
le-3
Caussion
02:10.12
(OM-CPC 1,iin cosine sim.)
SOM-CPC (r 0.07, sim dol prod.)
le-3
Caussan
1.00
.3811.00 098
CPC SOM (disjoin))
Gaussian
1.00
62
1.00
28:2.
CPC PCA
SOM-CPC
Speaker id
9
8
6
5
4
3
2
Figure 8: Projecting the test set on the 2D PCA space shows no division of the green and the red clusters in two sub-clusters, something that is visible in the SOM of the SOM-CPC model. These sub-clusters were found to relate recordings that made with different acoustics.
21


Reviews:
[{"review_id": "iElpnG6AA9B", "reviewer": "Reviewer_FiNp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission proposes an algorithm to learn representations of high-rate time series.They focus on a family of models called deep Self Organizing Maps (deep-SOM), models that combine the original SOM objective with (1) neural networks and (2) a task reconstruction loss. The authors introduce a novel variant, SOM-CPC, which follows the deep-SOM formulation, but replaces the task reconstruction loss with an info-NCE objective. Empirical evaluation of their model showcases the strenghts of their proposed approach.\n", "summary_of_the_review": "A well-written, clear paper with good empirical contributions. The authors are clearly proposing a working solution to a real problem, hence my rating of at least 6. On the other hand, the proposed approach can be read as marginally derivative, applying an algorithm that intuitively should work for the problem being considered. This in my opinion justifies not giving a higher score.", "strength_and_weaknesses": "Strenghts:\n- The paper is clear, highlighting the necesary related work to understand the proposed algorithm. \n- The empirical evaluation demonstrates good performance across a variety of settings.\n- The proposed method is reasonably novel in that it differs from the other instances of SOM-type models.\n- Code is released, making reproducing the results more feasible.\n\nWeaknesses:\n- The proposed approach reads as a semi-direct application of CPC. CPC (and other objectives similar to InfoNCE) are shown to work across a variety of domains. It is therefore not that surprising that incorporating CPC in a representation task (as one of the two objectives works well).\n- I feel a comparative evaluation of different contrastive objectives might be warranted: does adding temperature bring benefits as in [1]? What about MoCo, SimCLR-type objectives (do they work well in this context)?\n\nReferences:\n[1] CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aMNOl_QkvW-", "reviewer": "Reviewer_eMcU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for representation learning of multivariate time-series, which consists in the combination of Contrastive learning (for encoding the data in a compact latent space) and Self Organizing Maps (for better interpretability of the latent space). The use of interpretable unsupervised techniques is of the highest importance for biomedical applications, which makes the proposed technique appealing.", "summary_of_the_review": "This paper introduced an interesting technique for interpretable representation learning, which could be highly interesting for biomedical applications. However biomedical data can be highly non-stationary which might prevent the use of temporal contrastive learning. ", "strength_and_weaknesses": "Strengths:\n1.\tThe paper is well written and clearly organized.\n2.\tThe authors have tested the proposed approach on several applications (real and simulated datasets). Although what is lacking is for example the evaluation of classification performance when using the proposed unsupervised technique (for example in the sleep staging application)\n3.\tThe authors test their approach with state-of-the-art techniques, from which the proposed technique is inspired.\n4.\tThe authors provide well structured source code, which will help greatly for reproducibility of research\n\nWeaknesses:\n\n1.\tThe authors introduce the concept of aggregate causal context $c(t)$, but explain that the use of this $c(t)$ is not mandatory. The authors do not clearly explain (or I failed to understand it) whether the results in tehri experiment did use the aggregate causal context or not (and did not integrate in the ablation study)\n2.\tBiomedical time series are by essence non-stationary, therefore the use of contrastive learning may not be optimal for such data. Could the authors expand a bit on this possible drawback?\n3.\tThe explanation of multiple cluster in the audio dataset seem to be a bit overstretched. Explaining that each cluster (for same speaker correspond to a different chapter might be plausible. But did the authors listen carefully to the audio time-series, and did they notice a difference?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DP6mAZGRoaz", "reviewer": "Reviewer_hhoQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main technical contribution of this paper is in proposing a variant of deep SOM (self organizing map) model that uses contrastive learning. Experimental results show that the proposed method outperforms various baselines on synthetic and real data.", "summary_of_the_review": "I think this paper proposes an interesting although fairly straightforward extension to existing deep SOM models that incorporates contrastive learning. More extensive evaluation of \"interpretability\" and considering additional baselines would really help improve the work although, from what I can tell, neither of these improvements would resolve the issue of the technical novelty being limited.", "strength_and_weaknesses": "Strengths:\n1. I found the paper fairly straightforward to follow.\n2. The proposed method is easy to understand.\n\nWeaknesses:\n1. The proposed method is rather incremental. How contrastive learning is incorporated here is really straightforward and a bit obvious.\n2. The introduction partially emphasizes interpretability. I would have liked to see much more of a discussion on interpretability in the main paper regarding the real data experiments. Specifically, how does the interpretability of the resulting 2D SOM maps relate to existing domain knowledge for the applications of interest? How does this interpretability compare to those of baseline models that are used (i.e., does the proposed SOM-CPC model somehow yield more interpretable 2D maps than best-performing baselines?)? Moreover, I think that in answering such questions, it would be helpful to be more precise about what you mean by \"interpretability\" (maybe using terminology from, say, Lipton [2018]).\n3. I would argue that more baselines are needed. Right now the baselines are selected to be ones that use 2D embeddings, which seems like an artificially imposed constraint that really isn't required for many real applications (e.g., I'd imagine that with some real high-dimensional datasets, the low-dimensional manifold that the data approximately reside on simply can't be very accurately represented in 2D and could require more dimensions; forcing the use of a 2D embedding in this case seems like a bad idea). There have been a number of temporal clustering methods proposed in recent years, many of which use deep neural networks to help learn representations (such as but not limited to the work by Lee and Van Der Schaar [2020]). Adding some of these methods as baselines would improve the paper as we would have a better sense of how the proposed method compares to other approaches that aren't restricted to 2D embeddings.\n4. There are some parts where the notation and exposition are a little clunky and could use more polish. For instance, in the second paragraph of Section 2.1 when notation is first introduced, it is not made immediately clear what the index $j$ is for or what it ranges over, and then suddenly the index $j$ gets switched to $i$ in the next sentence. Minor: The phrase \"a given point in training\" could also perhaps be more clearly phrased as \"a given iteration of the training procedure\" where $n$ indexing the iteration number (\"a given point\" makes it sound like it might be referring to a data point).\n\nReferences:\n- Changhee Lee, Mihaela van der Schaar. Temporal Phenotyping using Deep Predictive Clustering of Disease Progression. ICML 2020.\n- Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject