Sequence Metric Learning as Synchronization of Recurrent Neural Networks
Authors: Paul Compagnon
Grégoire Lefebvre
Stefan Duffner
Christophe Garcia
Venue:  (2021)
Keywords: Metric learning
sequence processing
siamese recurrent neural network
dynamical systems

Abstract: Sequence metric learning is becoming a widely adopted approach for various applications dealing with sequential multi-variate data such as activity recognition or natural language processing and is most of the time tackled with sequence alignment approaches or representation learning. 
In this paper, we propose to study this subject from the point of view of dynamical system theory by drawing the analogy between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese recurrent neural network. 
Indeed, a siamese recurrent network comprises two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. 
We therefore propose a new neural network model that implements this coupling with a new gate integrated into the classical Gated Recurrent Unit architecture. This model is thus able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way.
Our experiments show that introducing such a coupling improves the performance of the siamese Gated Recurrent Unit architecture on an activity recognition dataset. 

Under review as a conference paper at ICLR 2021
SEQUENCE METRIC LEARNING AS S SYNCHRONIZATION OF RECURRENT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Sequence metric learning is becoming widely adopted approach for various applications dealing with sequential multi-variate data such as activity recognition or natural language processing and most of the time tackled with sequence alignment approaches representation learning. In this paper, we propose to study this subject from the point of view of dynamical system theory by drawing the analogy between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by siamese recurrent neural network. Indeed, siamese recurrent network comprises two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if coupling is introduced between them. We therefore propose a new neural network model that implements this coupling with a new gate integrated into the classical Gated Recurrent Unit architecture. This model is thus able to simultancously learn a similarity metric and the synchronization of unaligned multi-variate sequences in weakly supervised way. Our experiments show that introducing such a coupling improves the performance of the siamese Gated Recurrent Unit architecture on an activity recognition dataset.
INTRODUCTION
Metric learning aims at learning an essential component for numerous machine learning algorithms used for classification or clustering: a similarity. has the benefit be usable weakly supervised settings where only equivalence constraints between samples are known (Xing et al. (2003)), which allows for a large number of applications on various data types: from person re- ce-ideniifictton (Yang et al. (2088)), object tracking (Bertinetto et al. (0166) and gesture recognition (Berlemont) et al. (2018) to sentence similarity computation (Mueller & Thyagarajan (2016)). Among those applications, less attention has been given to design specific sequence metric learning algorithms, specifically with neural networks despite the simplicity of the siamese architecture (Bromley et al. (1994)).
One easy way adapt existing approaches to sequential data is learn representations through Sequence-to-Sequene models (Sutskever et al. (2014)) or Transformers (Vaswani et al. (201)). How- ever, these models would be difficult to learn weakly supervised way for providing similarity metric and further lose temporal dependency information inside the sequence and alignment informa- tion between sequences. On the contrary, Dynamic Time Warping (DTW) (Sakoe & Chiba (978)) is classical approach to measure distance between sequences and relies on aligning sequences. Its integration inside learning algorithms has been rendered difficult by con-differentiabiliyy and its theoretical quadratic time complexity which badly suits the equivalence constraint framework and some associated more complex losses (Oh Song et al. 2016 Sohn 2016)] Yang et al., 2018). Recent works mitigate these drawbacks notably with virtual metric learning (Perrot & Habrard 2015 Su &
Wu 2019) and soft versions of DTW (Cai et al. 2019 Abid & Zoul (0188).
Therefore, aim at designing neural network architecture specifically adapted to sequence metric
learning. Recurrent neural networks (RNN) have a temporal dynamic behavior which allows to study them as dynamical systems. We propose in this paper new framework for sequence metric

---

Under review as a conference paper at ICLR 2021
learning based on dynamical system synchronization theory. We propose to replace the concept of metric in a vector space by the concept of synchronization of trajectories in a state space. Instead of computing distances input representations, we propose to measure how two dynamical systems, and precisely two RNNs, respond to input pairs in term of synchronization. The notion of coupling is crucial when trying synchronize dynamical systems. We introduce coupled version of the Gated Recurrent Unit (GRU) (Cho et al. (2014)) to implement coupling inside a siamese architecture. Our experimental evaluation shows that this modification provides an improvement over a classical Siamese GRU implementation.
The paper is organized as follows: Section outlines the state-of seee--ff-th-eaee art approaches in sequence metric learning, Section describes our framework and our new siamese architecture, Section 4 shows our experimental results to assess the performances of our approach compared to the state of the art, and Section 5 presents our conclusions and perspectives.
RELATED WORK
Recurrent neural networks and dynamical system theory. A main property of RNN is to exhibit a dynamic behavior which enables them to learn temporal sequence correlations. This behavior can therefore be studied using dynamical system theory: an important result being that RNN can approximate any finite-time trajectory of a dynamical system (Funahashi & Nakamura (993).. Other early works analyzed the RNN convergence stability (Hirsch (Hirsch((1999) (1989)) and helped to understand the problem of long term dependencies (Bengio et al. (1994)). Laurent & von Brecht (2016) studied the dynamics of Long-Short Term Memory (LSTM) Neural Networks and GRU and observed that is chaotic in the absence of input data. They designed Chaos-Free RNN architecture having more predictable behavior. In another recent publication, Chang et al. (2019) studied RNN trainability and established connection with discretized Ordinary Differential Equations stability. They identified a criterion to guarantee that the system can preserve long-term dependencies and proposed a new version of RNN based on those observations. Both papers demonstrate that dynamical system theory is a fertile soil to study and conceive new RNN models. Finally, we would like to mention works on the definition of metrics to compare non-linear dynamical systems (Martin) (2000); Ishikawal et al. ) although our objective is not exactly the same, as we propose to use dynamical system synchronization theory to improve metric learning on any type of sequential data, whereas these methods have been conceived to work more specifically with structural data.
Sequence metric learning. DTW is a classical approach for measuring distances between se- quences (Sakoe & Chiba 19788) Numerous improvements have been brought to the original formulation notably to improve the k-nearest neighbor performance (Xi et al. (2006)) [Abid & Zou (2018) proposed to learn the DTW parameters that allow to reproduce the Euclidean distances between sequence representations learned with Sequence-to-Sequence model (Sutskever et al. (2014)) ). In
contrast, Su & Hua (0017) proposed an alternative to DTW, the delle-Preeerving Wasserstein (OPW) distance, by viewing the problem of metric learning between sequences as an optimal transport problem regularized to preserve the temporal relationships between the samples, and they solved with the matrix scaling algorithm. Later (Su & Wu (2019)), the authors reformulated the DTW and OPW distances as parameterized meta-metrics of single ground metric and proposed an optimization process learn the metric and the latent alignment with virtual metric learning (Perrot & Habrard (2015)), which reduces the number of constraints. Not only this approach speeds up training but it also outperforms several other metric learning approaches, notably approaches conceived for points generalized to sequences. In comparison, we propose pure RNN approach similar to Mueller & Thyagarajan 2016) who presented siamese neural network approach learn sentence similarities as alo-norm In their method, the LSTM network combines the embeddings of the words of the sentence to learn a distance between representations of sentences. Finally, Varior et al.) (2016) proposed a siamese convolutional architecture for person re-identification from video data with gates linking parallel layers allowing to accentuate common patterns between both representations. This leads to representations that more suited to distinguish some pairs of similar dissimilar images.
In this paper, we introduce an alternative to DTW: a pure neural network approach to sequence metric learning based on the siamese RNN architecture. We propose to enhance the classical Siamese RNN by studying this model from dynamical system point of view, as it has been already done for standard RNN. The disadvantage of DTW-based approaches compared to ours is that it can be
2

---

Under review as a conference paper at ICLR 2021
computationally inefficient and non-differentiable, which prevents their possible combination with other gradient-bart based models.
3 SYNCHRONIZING GRU SIAMESE NETWORKS
In this section, we first draw parallel between the concept of synchronization for dynamical systems and the task of sequence metric learning with siamese RNN. We then justify from theoretical point of view the introduction of coupling inside the siamese architecture. We finally introduce our main contribution in Section 3.3 modified Siamese GRU model implementing this coupling.
3.1 SYNCHRONIZATION OF CHAOS AND METRIC LEARNING
The concept of synchronization is generally well-understood for time- iime-periodic dynamical systems: this phenomenon is called phase synchronization. However, it is cess-nnown that it can also occur for chaotic dynamical systems (Pecora & Carroll (1990)) that is, systems for which resulting trajectories exponentially diverge for infinitesimally close initial conditions. This practically means that the behavior of such systems can become rapidly umpredictable solely due small variations of the initial conditions. Common examples of such systems are the double pendulum or the n-body problem. To formalize the concept of synchronization for chaotic systems, Brown & Kocarev 20000) proposed general definition of it:
Definition Let Z be a dynamical system composed of two subsystems X and Y such that:
X dæ (1,,1;11)
dl
(1)
Y dy dl = f2(y,2;t),
where x E Rdi and y E RRd2 with di, d2 E N. Let 6(z0) be a trajectory of Z with initial conditions z0 [To, yo] Rdi x Rd2, Finally let g X (resp. Y) x TR* with k E N, be measurable property of the subsystems. They are synchronized on the trajectory 6(20) with respect to the property g if there is time independent function R IR I2t such that:
[1b(g(z). =0,
where
is norm.
From this definition, is possible to derive several ways to measure synchronization between two trajectories. Brown & Kocarev (2000) report several slightly different formulations of the synchronization error with the following being the most used. According to them, for identical systems:
h(g(x). g())) lim (g(x) - g(v)),
t
where h and g are the same as in Definition I
We rewrite this synchronization eтrоr for discrete systems in a continuous form by replacing the limit by a comparison of the last element of each trajectory X and Y of length T::
M(g(X),g())) = Xx Yr
with y being here a function returning the coordinates of the points. We define d distance on discrete dynamical system trajectorics derived from the synchronization error by replacing the difference with the Euclidean norm (in definition [ any norm can be used) to get only positive values:
d(x, Y) ||XT Yrl12-
However, RNNs dynamical systems, and the output sequences trajectories. Thus, learning Euclidean distance with a siamese RNN is equivalent to trying to synchronize the output sequences of the two sub-networks of the siamese network for similar pairs
While being intuitive and suitable for metric learning, the metric of Equation 3 measures synchroniza- tion only one point in time, which forces the system achieve synchronization at this precise point.
3

---

Under review as a conference paper at ICLR 2021
This is called dead-beat synchronization, synchronization in a finite number of steps (De Angeli jet al.] (1995)). Even at first sight, this seems not really different from computing distance on input sequence representations, synchronization could actually be assessed at several samples of the sequence and even continuously. We will now specify what type of synchronization siamese RNNs able achieve and under which conditions.
3.2 COMPLETE SYNCHRONIZATION OF COUPLED IDENTICAL SYSTEMS
A special case of Definition is when fi and f2 are the same function, the dynamical systems share the same parameters, and they are said to be identical. Analogously, the two (or more case of triplet inputs) sub-networks of siamese network also share the same parameters; only input sequences differ (Bromley et al. (1944). To simplify, we will first study the case where the dynamics of the RNNs are solely driven by its initial condition (the initial hidden state) and where no input sequence is given (i.e. a sequence of Os). We obtain what is called the dynamical system induced by the RNN. In this case, only the initial conditions differ and the sub-networks are identical dynamical systems. According to experiments conducted by Laurent& & von Brecht (2016), dynamical systems induced by RNN exhibit chaotic behavior. Under what conditions identical systems can synchronize? Consider now the following two identical systems:
X dx
[x; () + xv))
(6)
Y dy de f(vit) + C(x - --²
where 2, E R" and C is coupling matrix in RTx" This type of coupling is called diffusive because it will dissipate the dynamics of each sub-svstem (Boccaletti et al. (2002). X and Y are bidirectionally coupled systems Fujisaka & Yamada (1983) showed that the system described by Equation g can achieve complete synchronization if C' is a multiple of the Identity matrix and constant e which verifies the following condition:
e> 2 AL
where d is the largest Liapunov exponent of the system. The Liapunov exponents quantify the sensibility of system initial conditions: if it has one positive Liapunov exponent, predictability of its behavior becomes impossible beyond a certain time horizon; it is therefore chaotic (Strogatz (2018))
But the trajectories of RNNs are most of the time also influenced by external inputs: the input sequence. In this case, the dynamics of the RNNs are mostly driven by these external inputs (Laurent & von Brecht (0166) and RNN starting from different initial conditions but given identical input sequence will see their trajectories synchronize, i.e. the hidden states become the same after a few steps. Coupling is in this case not necessary achieve synchronization: regarding metric learning. siamese LSTM actually works without coupling (Mueller & Thyagarajan (2016)). However, coupling could allow to enforce lower distances with sequences that have similar dynamics but are composed of quite different data, i.c. so-called hard positive samples, and even to force the synchronization regardless of the input pair. Hard positive samples are samples that are factually belonging to one class, by the label, but lie close to the decision boundary even beyond in terms of distance (conversely for hard negative samples). They are particularly studied by the metric learning community to improve the convergence and performances of metric learning models Schroff et al. (2015).
We showed in this section the motivation and aim of implementing coupling into the siamese RNN architecture. Indeed, the induced dynamics of GRU and LSTM are chaotic and, in this case, coupling allows their complete synchronization as the sub-networks of a siamese network share the same weights and represent thus identical dynamical systems. When given input sequences, the dynamics of GRU and LSTM are mostly driven by these external inputs. In this case, while not being critical to achieve synchronization (and therefore low distances between similar elements), coupling could facilitate bringing similar inputs closer, particularly for hard positive pairs.
"ooupingg can also he directional, only one system influences the other: they are called in this case drive- response systems.
4

---

Under review as a conference paper at ICLR 2021
h-1
h+
h
hg
4
Zt
Figure 1: Schema of the CGRU architecture. Blue arrows represent information coming from the input at time red ones are for the hidden state and green ones for transmissions between the gates.
3.3 COUPLED GRU
We present neural network model that directly implements coupling within siamese RNN architecture. From a machine learning perspective, this coupling needs to be trainable such that the network learns to achieve synchronization for similar inputs and stay desynchronized for different ones. We propose to apply the coupling by means of two new gates inside the GRU architecture which call in the following CGRU (Coupled Gated Recurrent Unit). We chose LSTM (Hochreiter & Schmidhuber (1997)) because the operation of a GRU is simpler (i.e. defined by fewer equations) while showing comparable performance general (Chung et al. (2014)). The following equations describe the modifications brought to the architecture. Update, Reset and New gates are not modified. Let us notate N/1 and hç the states coming from the second sub-network (see Figure
GRU and
Hidden state: he = (1 zz) 2t)ht + zeh1-1
(8)
Update: 2 σ (W + bis + Whohe- + bnz))
(9)
Coupled New State: h4 (1 (q) c4) + cgti
(10)
Coupling: et a (Wroo 1111) bnc)
(11)
New State: h4 = tanh(W ii + b,g. + re re AiW 1--1 + bnū))
(12)
Reset: rt oWar.: dyr + Whrhe-1 + bar).
(13)
The Coupling gate ot (see Equation
and vt, controlling the information flow and is thus computed in similar fashion, but only from the hidden states. This forces the model to apply the coupling on the new content be added at time solely based on the previous inputs. Then, Equation 10 hy and fig are combined similarly as h, and h4 are combined in the original GRU architecture. This prevents he and subsequently h2 from exploding and saturate the gates. Finally. in Equation h replaces h,: the New state has been replaced by a coupled version of both New states of the siamese GRU. Several possibilities exist to implement this coupling. The idea behind this proposal is to alter as little as possible the GRU architecture and to stay close to the original purpose each equation. Indeed, the addition of the coupling already greatly modifies the information flow inside the GRU and the gradient flow during training (see the supplementary material for the complete differentiation of CGRU), and RNNs are known to be difficult to train. Therefore, by staying relatively close the original model, rigorous comparison is simpler, and the impact the actual coupling can be studied more reliably. In fact, if c, is a vector of norm equal to is exactly GRU. This suggests initialize the coupling weights with very
11 serves the same purpose
zero, each
5

---

Under review as a conference paper at ICLR 2021
small values and accentuate the decay. In this way, increase of the norm of Whe during training would signify that coupling is useful. Another interesting configuration of the coupling weights is when they are all equal to 0.5: in this configuration, the Coupled New States are the same, and the distance between the outputs will become null. That means, theoretically, this approach can make close any pair of input sequences, especially hard-positive samples.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We experiment the CGRU architecture on the dataset UCI HAR AAnguita et al. (2013) ). a dataset of 6 SniivitteeCcotaannniii 9 features: total acceleration, body acceleration and angular velocity on 3 axes. The sequences have a length of 128. We chose this dataset because it has been extensively used by the activity recognition community and provides at the same time real data and a simple and well defined benchmark to study the behavior of CGRU and Siamese GRU (SGRU). Moreover, several activities should look very similar (e.g. three variants of walking or standing and sitting), and it should make the dataset harder to process for metric learning algorithms. Finally, walking or running exhibits dynamic components which could be differently processed by CGRU and SGRU. No further preprocessing has been applied. The features are globally centered and the standard deviations oscillate between 0.1 and 0.4. We kept the train-test split proposed by the authors of the dataset: there are 21 users in the training set and we therefore performed a 7-fold validation, leaving each time different users out. Finally, the training set comprises 7352 sequences and the testing set 2947.
We compared SGRU and CGRU on learning the metric describes in Equation using the same hyperparameters for both models. We used the same architecture for both networks, a one-layer network with 20 neurons and initial learning rate of 0.001 which we decreased by a factor of 0.5 after 10 epochs if training loss does not improve. The training is stopped based on the accuracy on the validation set (early stopping). We chose use structural loss (Yang et al. (2018) to train the model since it is a recent loss working on distances and not embeddings. It combines a local term similarly to the n- n-pair loss (Sohn) (2016)) but emphasizes the weights on the hard positive samples, and a global term to improve the generalization. We used the same hyperparameters for the loss as in the original paper. The full equations of the loss are available the second section of the supplementary material. Regarding the batch size, each batch contains 6 elements of each class (total 36) to form a valid batch for structural loss. We applied a general weight decay with a factor of 10 4, A stronger weight decay was applied on the coupling by adding 1% of the coupling weight norm to the loss, which seems to slightly improve the performances. The gradient was clipped according to (Pascanu et al. (203)) to a norm of 6. The coupling weights were initialized with normal distribution having mean of 0 and standard deviation of 0.1. The purpose of this initialization is to make the model start its training close the behavior of SGRU, with very weak coupling and to let it increase during training. Our implementation was done in Python and CUDA with Pytorch (Paszke et al. (2019).
4.2 RESULTS
4.2. 1
STUDY OF THE COUPLING WEIGHIT NORM
We first analyze the evolution of the coupling weight norm during training. The coupling is initialized very low which theoretically makes it behave nearly as an SGRU at the beginning of training. We can observe in Figure 2a that the norm increases quickly during the first 20 epochs and more than doubles. Red points indicate the iterations where validation accuracy increased. This correlation thus indicates that the overall generalization is improving with the increase of coupling strength. In Figure 2b] we present the evolution of the loss on the train dataset in terms of the coupling weights which we compute a linear regression of the first part. We can observe an almost linear relationship between the increase of the norm and the decrease of the loss suggesting again that the coupling is helpful to the model. Those figures also show that the convergence during training is rather smooth.
hhttps://arch ve. wus..ui..... httpd//mmmm/aattstts//maaaaa http://sses.eeooooooooooooooooon
US using+ wennnngsmaatt smartthness walking, walking upstairs, walking downstairs, sitting, standing, laying

---

Under review as a conference paper at ICLR 2021
16
check point
1400
empirical
1200
lincar regression
14
coupling norm
epochh
600
400
50
10 coupling norm
(b)
In figure (a), evolution of the norm of the coupling gate weights during training on UCI HAR dataset, a red point indicates an increased accuracy for the validation set. On figure (b), Epoch loss in terms of coupling norm during training on UCI HAR dataset. The first part (rapid decrease) has been approximated with a linear regression. The correlation coefficient is -0.984.
Figure 2:
Algorithms
Accuracy
Fl score Macro
MAP 0.701 0.900
Siamese GRU
0.805
0.074 0.016
Coupled GRU
0.913
(a) Validation results (21 fold average). An asterisk means a significant result with threshold of 1%
Algorithms
Accuracy
F1 score Macro
MAP
RVSML (OPW) (Su & Wu (2019)
0.597
0.568
0.438
RVSML (DTW) (Su & Wu (019))
0.698
0.687
0.437
Siamese GRU
0.782-10.041
0.781
0.633:5.1122
Coupled GRU
0.885:0.0144
0.891
(b) Test results, average of run neural networks approaches.
Table 1: Results on UCI HAR
4.2.2 CLASSIFICATION PERFORMANCE ON UCI HAR
We now present classification results on UCI HAR using 3 metrics : accuracy, Fl score Macro averaged and Mean Average Precision (MAP), similarly to Su & Wu (2019). The first two are computed by classifying the test samples using 1 l-nearest neighbor from training samples. The MAP is computed by querying the training set with a test sequence to retrieve all training samples of the same class. The value averaged for all test sequences. This metric shows the ability the algorithm to bring close every sequence of each class and not just few references to be used as nearest neighbors. The validation results are presented in Table Ia We observe an improvement of CGRU over SGRU of about 8% points for accuracy and Fl-score, and an improvement of 19% points for the MAP. On the test set, we compared our approach with Regressive Virtual Sequence Metric Learning (RVSML) (Su & Wu (019))) with OPW and DTW distances. We chose to compare with this approach because recently outperformed several other metric learning approaches (e.g. Large Margin Nearest Neighbors (Weinberger & Saul (2009). Regressive Virtual Metric Learning (Perrot & Habrard (2015) etc.) although not all were specifically adapted for sequences, it is not based on neural networks and uses alignment-bascd distances. We keep the hyperparameter values recommended by the authors. The test results are presented in Table [b] Here again we observe that CGRU outperformed SGRU with a notable improvement of 10% points of the accuracy and Fl-score. Both neural network approaches clearly outperformed RYSML, especially the OPW variant. This can be, among other factors, attributed to weak capacity to distinguish similar activities such as standing and sitting. On the other hand, standing was recognized perfectly by both RVSML variants. We also remark that they reach MAP values the same order as in the original paper (around 0.4 0.45) despite the fact that the data of completely different nature (signal instead of images). This could suggest that these approaches reached some kind of saturation whereas CGRU and SGRU are able to achieve much higher values.
7

---

Under review as a conference paper at ICLR 2021
Average distance between positive samples
0.7
SGRU positives
CGRU positives
0.6
negatives
CGRU negatives
0.5
0.2
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Standard deviation white noise
Figure 3: Evolution of the average distance between the positive/negative samples for SGRU and CGRU on more and more noisy test sets.
4.2.3 PERFORMANCES HARD POSITIVE SAMPLES
We made the hypothesis that CGRU could perform better on hard positive samples due to coupling theoretically being able to bring close any input pair of sequences: with enough coupling, both networks can output the same sequence whatever of the input pair. We propose to verify this by observing the evolution of the average distance between the positive samples and between the negative samples when white noise is gradually added to the feature sequences of the testing set. The standard deviation of the noise goes from 0 2. We also note that both models were trained up comparable validation accuracy. The results are presented on Figure 3 We observe that the curves for both models evolve similarly with positive distances gradually increasing with the noise. The negative and positive curves join the moment too much noise is added and the sequences become indistinguishable. CGRU produces slightly higher distance average than SGRU but able to maintain higher margins more longer: up 1.75 units of standard deviation compared to about 1.25 for SGRU. This shows that, as theoretically possible with the coupling, CGRU better discriminate the hard samples. The third section of the supplementary material extends this argumentation by analyzing average distance to nearest neighbors.
CONCLUSION AND PERSPECTIVES
We presented new framework for sequence metric learning based on dynamical system synchroniza- tion theory. We drew parallel between synchronized trajectories and output sequences of siamese recurrent neural networks produced from similar input pairs. After characterizing Siamese GRU as identical chaotic systems, showed the contribution of introducing coupling inside the siamese architecture to achieve synchronization more easily and to increase the capacity of the network to bring closer the embedding of some similar input pairs, especially hard positive samples. This coupling was implemented through a new gate inside the Siamese GRU architecture which allows the network to mix the new content of both sides of the siamese network. Our experiments showed that the Siamese GRU architecture benefits from the coupling, can be smoothly trained with it and fits well with recent complex metric learning losses such structural loss. CGRU proved to be outperforming SGRU with the same architecture on an activity recognition benchmark and was able maintain higher margin for hard samples.
The study of sequence metric learning with synchronization opens several perspectives especially to design new forms of metries by taking inspiration from the literature of synchronization criteria (Schiff et al. (1996); Kreuz et al. (2007)) though non-differentiabiiity notably when those metrics use mutual neighbors which are computed using explicitly the time steps, has to overcome in many cases thanks to an attention mechanism, for example. The coupling itself could be tuned with theoretical contributions (Brown & Rulkov (1997)). One drawback of the proposed architecture is that each pair has to be passed through the network inseead of just computing once each representation and then the distance for each pair. This could balanced by the use virtual metric learning during training. Finally the coupling allows to bring any pair of inputs close another if sufficiently
8

---

Under review as a conference paper at ICLR 2021
strong and could be indicator in weakly supervised settings to invert the equivalence constraint of some pairs dynamically if the network is forcing the synchronization too much.
REFERENCES
Abubakar Abid and James Zou. Autowarp: learning a warping distance from unlabeled time series using sequence autoencoders. In Proceedings the 32nd International Conference Neural Information Processing Systems, pp. 10568-10578. Curran Associates Inc., 2018.
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In ESANN, 2013.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Samuel Berlemont, Grégoire Lefebvre, Stefan Duffner, and Christophe Garcia. siamese neural networks. Neurocomputing, 273:47-56, 2018
Luca Bertinetto, Jack Valmadre, Joao Henriques, Andrea Vedaldi, and Philip Torr. Fully- convolutional siamese networks for object tracking. In European conference on computer vision, pp. 850-865. Springer, 2016.
Stefano Boccaletti, Jürgen Kurths, Grigory Osipov, DL Valladares, and CS Zhou. The synchronization of chaotic systems. Physics reports, 366(1-22 12011100,, 2002.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. In Advances in Neural Information Processing Systems, pp. 737-744, 1994.
Reggie Brown and Ljupčo Kocarev. unifying definition of synchronization for dynamical systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 10(2):344-349, 2000.
Reggie Brown and Nikolai F Rulkov. Designing coupling that guarantees synchronization between identical chaotic systems. Physical review letters, 78(22):4189, 1997.
Xingyu Cai, Tingyang Xu, Jinfeng Yi, Junzhou Huang, and Sanguthevar Rajasekaran. Dtwnet: dynamic time warping network. In Advances in Neural Information Processing Systems, pp. 11636-1166,, 2019.
Bo Chang, Minmin Chen, Eldad Haber, and Ed Chi. Antisymmetricrnn: A dynamical system view recurrent neural networks. arXiv preprint ar arXiv:1902.09659 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint / 1078, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
A De Angeli, R Genesio, and A Tesi, Dead-beat chaos synchronization in discrete-ting systems. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 42(1):54-56. 1995.
Hirokazu Fujisaka and Tomoji Yamada. Stability theory of synchronized motion in coupled-oseill oscillator systems. Progress of theoretical physics, 69(1):32-47, 1983.
Ken-ichi Funahashi and Yuichi Nakamura. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks, 6(6):801-806, 1993.
Morris W Hirsch. Convergent activation dynamics in continuous time networks. Neural Networks, (5):331-349, 1989.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.

---

Under review as a conference paper at ICLR 2021
Isao Ishikawa, Keisuke Fujii, Masahiro Ikeda, Yuka Hashimoto, and Yoshinobu Kawahara. Metric on nonlinear dynamical systems with perron-frobenius operators. In Advances in Neural Information Processing Systems, pp. 2856-2866. 2018.
Thomas Kreuz, Florian Mormann, Ralph G Andrzejak, Alexander Kraskov, Klaus Lehnertz, and Peter Grassberger. Measuring synchronization in coupled model systems: A comparison of different approaches. Physica D: Nonlinear Phenomena, 225(1):29 42, 2007.
Thomas Laurent and James von Brecht. recurrent neural network without chaos. arXiv preprint arXiv:1612.06212, 2016.
Richard Martin. A metric for arma processes. IEEE transactions Signal Processing, 48(4): 1164-1170, 2000.
Jonas Mueller and Aditya Thyagarajan. Siamese recurrent architectures for learning sentence similarity. In AAAI, pp. 2786-2792 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4004-4012, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. 1310-1318, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pp. 4024-8035, 2019.
Louis M Pecora and Thomas L Carroll. Synchronization in chaotic systems. Physical review letters, 64(8):821, 1990.
Michaël Perrot and Amaury Habrard. Regressive virtual metric learning. In Advances in neural information processing systems, pp. 1810-1818, 2015.
Hiroaki Sakoe and Scibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions acoustics, speech, and signal processing, 26(1):43 1978.
Steven J Schiff, Paul So, Taeun Chang, Robert E Burke, and Tim Sauer. Detecting dynamical interdependence and generalized synchrony through mutual prediction in neural ensemble. Physical Review E, 54(6):6708, 1996.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, pp. 1857-1865, 2016.
Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology. Chemistry, and Engineering. CRC Press, 2018.
Bing Su and Gang Hua. Order-preservin wasserstein distance for sequence matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1049-1057, 2017.
Bing Su and Ying Wu. Learning distance for sequences by learning a ground metric. In International Conference on Machine Learning, pp. 6015-6025, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3114-3112, 2014.
Rahul Rama Varior, Mrinal Haloi, and Gang Wang. Gated siamese convolutional neural network architecture for human re-identification. In European conference on computer vision, pp. 791-808. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008. 2017.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10(Feb):207-244, 2009,
Xiaopeng Xi, Eamonn Keogh, Christian Shelton, Li Wei, and Chotirat Ann Ratanamahatana. Fast time series classification using numerosity reduction. Proceedings of the 23rd international conference on Machine learning, pp. 1033-1040. ACM, 2006.
Eric P Xing, Michael Jordan, Stuart Russell, and Andrew Y Ng. Distance metric learning with application to clustering with side-information. In Advances in neural information processing systems, pp. 521-528, 2003.
Xun Yang. Peicheng Zhou, and Meng Wang. Person reidentification via structural deep metric learning. IEEE Transactions on Neural Networks and Learning Systems, 30(10))2987-2998, 2018.


Reviews:
[{"review_id": "oNIC-qS5Ig0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nDrawing inspiration from dynamic systems, the paper proposes a novel architecture that couple sequences. Such a system has easiness to bring two instances arbitrarily close and authors have shown the superiority of the approach over an action recognition dataset; but the results seem to far from state of the art on the dataset (see questions section). The authors also recognize that currently such systems need to calculate each pairs (can't be cached due to coupling) at inference time, which is slow. \n\nThe main issue I found in this paper is its presentation. The authors claim to draw inspirations from dynamic systems (e.g. the important notion of  synchronized trajectories) in the abstract/introduction, cite related work and introduce a formal definition (with some errors, see section minor issues). However, the metric that ends up being used in eq(5) is a metric function of two RNN end states, which is common and wouldn’t help to highlight authors’ contributions.\n\nAnother concern is that the authors have tested on only one dataset where the claimed results are quite below state of the art on the dataset (see questions section). There are two baselines implemented (one very recent approach), however no simple baseline or other datasets to further support the approach.\n\n\nFinally, after reading the paper, I think the paper proposes a new neural architecture for similarity learning rather than focusing on metric learning. In this aspect, the paper misses references to papers in the area.\n\n\n#### Pros: \nDrawing inspiration from dynamic systems, the paper proposes a novel architecture that is not only dynamically involved with timesteps but is also coupled (between two instances), which has the capability of bringing two arbitrary sequences close. The authors have shown the superiority of the proposed approach over several metric learning baselines including the recently proposed RVSML (Su and Wu, 2019) over an action recognition dataset.\n\n#### Questions:\nClarification question: Given the architecture, the methods can only be applied to sequences that have the same number of timesteps, is that correct?\n\nThe approach seems to be far from the state of art: In Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope [Wong and Kolter, 2018], the baseline network investigated achieves 5% error rate; for papers more focused on performance, Human activity recognition with smartphone sensors using deep learning neural networks [Ronao and Cho 2016] archives accuracy 95.75%. Why is there such a discrepancy between the results in the paper and state of the art results please (as it is more convincing to build upon state of the art results when possible). \n\n#### Minor Issues:\nIn Definition 1, the function two subsystems are synchronized if there exists a time-independent g (instead of a time dependent g as written). In fact, one can always find a trivial time dependent g satisfying the equation (by the way, it is written correctly with time independence in Brown & Kocarev (2000))\n\n#### Minor suggestions:\n\nModifying neural architecture to capture better similarity has numerous works, one notable example in NLP is for example: A Decomposable Attention Model for Natural Language Inference [Parikh et al. 2016]. I would recommend including similar references in the paper to better situate the paper’s contribution.\n\nThe loss function is missing in the paper, so leading some missing details for training. \n\nFinally, I want to detail my argument to change the paper’s highlight on synchronization, which ends being a distance between two RNN end states. First, this is very far from the formal definition in eq(1) with a quite weak link; secondly, it is one of the most commonly used metrics in NLP/CV applications (and very probably other areas that I am much less familiar with). Given that an ICLR main paper is only about 8 pages, I strongly recommend authors to reconsider the paper structure to highlight its main contributions.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "khg73n5Sk0W", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I liked the formulation and motivation of the paper, explaining the sequence metric learning problem  and drawing parallel between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese style recurrent neural network. The authors propose modification the siamese recurrent network setting called classical Gated Recurrent Unit architecture (CGRU). The premise being two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. The authors describe how this model is able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way with the coupling demonstrating performance of the siamese Gated Recurrent Unit (SGRU) architecture on UCI activity recognition dataset (mobile data).\n\nThe increase in norm with the epochs shows overall generalization is improving with the increase of coupling strength and an almost linear relationship between the increase of the norm and the decrease of the loss suggesting again that the coupling is helpful. Computing accuracy, F1 score Macro averaged and Mean Average Precision (MAP),similarly to Su & Wu (2019) also shows improvement against SGRU and RVSML.  The authors mention a drawback of the proposed architecture is that each pair has to be passed through the network instead of just computing once each representation and then the distance for each pair and that this could be balanced by the use of virtual metric learning during training. \n\nThe improvements points in the paper are\n- Comparing against more recent work on the activity recognition such as  Learning Discriminative Virtual Sequences for Time Series Classification\n- using more comprehensive evaluation (rather than a single data set)\n- will also be good to expand on the virtual  metric learning during training as passing each pair can increase the training complexity for large datasets\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VhXwbFpTaMz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis work concerns the metric learning between sequences using RNNs. The paper notices the similarity between a dynamical system and an RNN. Then it demonstrates that learning a pair of siamese RNNs is similar to learning synchronization between two subsystems of a dynamical system. Finally, the paper proposes to introduce coupling between the two RNNs in order to improve synchronization.\n\nThe paper conducts experiments comparing the siamese GRUs with the proposed couples siamese GRUs.\n\n\n# Quality\n\nWhile this is an interesting idea to explore, I am not entirely convinced about its value. The main claim of the paper seems to be that the baseline siamese GRU exhibits chaotic behavior while the proposed architecture does not. The former has to be tested experimentally. I would like to see an experiment that shows this chaotic behevior at least qualitatively. Secondly, the proposed architecture intermixes two sequences. Therefore, it is not possible to learn embeddings.\n\nThe experimental section is very scarce. The only dataset used in the paper, UCI HAR, is very small. Furthermore, there are not enough benchmarks for the metric learning on this dataset. The results reported here are not comparable with the supervised methods (~90-95%). This work needs more experiments on datasets with several published results.\n\nThe section 4.2.1 seems to contain a logical error. The linear relation between the norm and the loss cannot show that the coupling helps or hurts the model. Instead, a model with coupling should be compared to a model without coupling. Would it be better to plot separately the norm for positive samples and the negative samples? If I understand correctly, the first is supposed to decrease and the second is supposed to increase. There is more chance to see a negative pair, therefore you observe that the norm increases.\n\nThe section 4.2.3 aims to test the model for the hard positive samples. I can see several problems with the proposed approach to test this. Firstly, the absolute distance is meaningless here. I propose to normalize both curves. Secondly and most importantly, the average metrics cannot test the performance for the hard positives. Such metrics would be dominated by the majority of \"easy positives\". Therefore, I request a more formal description of what is a \"hard positive\" and an experiment  that better tests the claim.\n\n# Clarity\n\nIn general, the paper is hard to read. The logical flow of the paper is hard to follow. Some important aspects are only referenced in other papers or skipped through. Many typos hinder the understanding sometimes.\n\nThe introduction seems to claim \"an improvement over a classical GRU\". In fact, the paper proposes an improvement to the *siamese* GRUs.\n\nDespite the fact that the proposed idea is quite simple, it was hard to follow the paper. Sections 3.1 and 3.2 are too general. This creates discrepancy between Sections 3.1-2 and 3.3. I don't see the point of introducing so general description of the synchronization. It would be clearer to define this concept for the specific case at hand.\n\nThe abuse of notation made Definition 1 look nearly incorrect. I recommend either carefully introduce the specifics here (like in the original paper by Brown & Kocarev) or stick to the common mathematical notation. More specifically:\n- Definitions for \"X\", \"Y\", and \"X (resp. Y)\"\n- Definition for g(x)\n\nIt is easy to loose a thread when reading the section 3.1. I would recommend to start with the siamese RNNs and then demonstrate the connection to the dynamical system, not other way around.\n\nThe loss used in the paper has to be explicitly spelled out (perhaps, in the Appendix).\n\n# Originality\n\nThe paper is sufficiently original.\n\n# Significance\n\nThe paper has a potential to be have high significance. Unfortunately, the experimentation is too limited. The claims made in the paper are not sufficiently tested. Then, the performance is tested only on one dataset against a weak baseline. The presentation needs to be improved too (as noted above).\n\n# Conclusion\n\nThe idea is interesting, but the experimentation is very weak.\n\nSpecific requests for improvement:\n\n- Experiments on more datasets\n- Compare to the published metric learning literature\n- Test the claim that the baseline is chaotic\n- Define hard positive samples and test the model on them\n- Improve Definition 1\n- Conduct ablation experiments\n\n# Some of typos\n\n- Fix \\citep vs \\citet\n- \"further lose temporal dependency\"\n- \"in term of synchronization\"\n- \"system theory, an important result being ...\"\n-  \"recent approach <...> studied\"\n- \"further introduce\"\n- \"h and g are the same as in Equation 1\" -- there are no h and g in Equation 1\n- \"RNN are dynamical systems\" -> \"RNN is..\"\n- \"RNN are known\" -> \"RNNs are\" or \"RNN is\"\n-  \"3 axis\" -> \"3 axes\"\n- \"size of 128\" -> \"length of 128\"\n- \"a initial learning rate\" -> \"an initial\"\n- \"a SGRU\" -> \"an SGRU\"\n\n# Update\n\nThank you for the rebuttal. The writing was improved and it is easier to read the paper. Nevertheless, the experimentation remains weak. I increase the rating by one point.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed to learn a sequence metric by sharing a memory cell between two LSTMs that run on pairs of sequences. I found the idea quite interesting, but I (and the reviewers) found the inspiration and the analogy from a dynamical systems perspective a unclear, unconvincing and maybe not even necessary to the core essence of the method that was proposed.

 The reviewers appreciated the improvement in clarity over the course of the review, but felt that there was still some more distance to cover. In addition, the results were not of a high enough quality to lend support to the success of the method and the experimental section needs more work making it not ready for ICLR acceptance at this time. 
Decision: Reject