Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: I liked the formulation and motivation of the paper, explaining the sequence metric learning problem  and drawing parallel between synchronized trajectories produced by dynamical systems and the distance between similar sequences processed by a siamese style recurrent neural network. The authors propose modification the siamese recurrent network setting called classical Gated Recurrent Unit architecture (CGRU). The premise being two identical sub-networks, two identical dynamical systems which can theoretically achieve complete synchronization if a coupling is introduced between them. The authors describe how this model is able to simultaneously learn a similarity metric and the synchronization of unaligned multi-variate sequences in a weakly supervised way with the coupling demonstrating performance of the siamese Gated Recurrent Unit (SGRU) architecture on UCI activity recognition dataset (mobile data).

The increase in norm with the epochs shows overall generalization is improving with the increase of coupling strength and an almost linear relationship between the increase of the norm and the decrease of the loss suggesting again that the coupling is helpful. Computing accuracy, F1 score Macro averaged and Mean Average Precision (MAP),similarly to Su & Wu (2019) also shows improvement against SGRU and RVSML.  The authors mention a drawback of the proposed architecture is that each pair has to be passed through the network instead of just computing once each representation and then the distance for each pair and that this could be balanced by the use of virtual metric learning during training. 

The improvements points in the paper are
- Comparing against more recent work on the activity recognition such as  Learning Discriminative Virtual Sequences for Time Series Classification
- using more comprehensive evaluation (rather than a single data set)
- will also be good to expand on the virtual  metric learning during training as passing each pair can increase the training complexity for large datasets

review_title: The authors describe coupling view of the temporal LSTM/GRU style networks, suggesting modifications to the GRU called SGRU that offers improved results on UCI activity recognition against SGRU and RVSML style approaches.

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Summary

This work concerns the metric learning between sequences using RNNs. The paper notices the similarity between a dynamical system and an RNN. Then it demonstrates that learning a pair of siamese RNNs is similar to learning synchronization between two subsystems of a dynamical system. Finally, the paper proposes to introduce coupling between the two RNNs in order to improve synchronization.

The paper conducts experiments comparing the siamese GRUs with the proposed couples siamese GRUs.


# Quality

While this is an interesting idea to explore, I am not entirely convinced about its value. The main claim of the paper seems to be that the baseline siamese GRU exhibits chaotic behavior while the proposed architecture does not. The former has to be tested experimentally. I would like to see an experiment that shows this chaotic behevior at least qualitatively. Secondly, the proposed architecture intermixes two sequences. Therefore, it is not possible to learn embeddings.

The experimental section is very scarce. The only dataset used in the paper, UCI HAR, is very small. Furthermore, there are not enough benchmarks for the metric learning on this dataset. The results reported here are not comparable with the supervised methods (~90-95%). This work needs more experiments on datasets with several published results.

The section 4.2.1 seems to contain a logical error. The linear relation between the norm and the loss cannot show that the coupling helps or hurts the model. Instead, a model with coupling should be compared to a model without coupling. Would it be better to plot separately the norm for positive samples and the negative samples? If I understand correctly, the first is supposed to decrease and the second is supposed to increase. There is more chance to see a negative pair, therefore you observe that the norm increases.

The section 4.2.3 aims to test the model for the hard positive samples. I can see several problems with the proposed approach to test this. Firstly, the absolute distance is meaningless here. I propose to normalize both curves. Secondly and most importantly, the average metrics cannot test the performance for the hard positives. Such metrics would be dominated by the majority of "easy positives". Therefore, I request a more formal description of what is a "hard positive" and an experiment  that better tests the claim.

# Clarity

In general, the paper is hard to read. The logical flow of the paper is hard to follow. Some important aspects are only referenced in other papers or skipped through. Many typos hinder the understanding sometimes.

The introduction seems to claim "an improvement over a classical GRU". In fact, the paper proposes an improvement to the *siamese* GRUs.

Despite the fact that the proposed idea is quite simple, it was hard to follow the paper. Sections 3.1 and 3.2 are too general. This creates discrepancy between Sections 3.1-2 and 3.3. I don't see the point of introducing so general description of the synchronization. It would be clearer to define this concept for the specific case at hand.

The abuse of notation made Definition 1 look nearly incorrect. I recommend either carefully introduce the specifics here (like in the original paper by Brown & Kocarev) or stick to the common mathematical notation. More specifically:
- Definitions for "X", "Y", and "X (resp. Y)"
- Definition for g(x)

It is easy to loose a thread when reading the section 3.1. I would recommend to start with the siamese RNNs and then demonstrate the connection to the dynamical system, not other way around.

The loss used in the paper has to be explicitly spelled out (perhaps, in the Appendix).

# Originality

The paper is sufficiently original.

# Significance

The paper has a potential to be have high significance. Unfortunately, the experimentation is too limited. The claims made in the paper are not sufficiently tested. Then, the performance is tested only on one dataset against a weak baseline. The presentation needs to be improved too (as noted above).

# Conclusion

The idea is interesting, but the experimentation is very weak.

Specific requests for improvement:

- Experiments on more datasets
- Compare to the published metric learning literature
- Test the claim that the baseline is chaotic
- Define hard positive samples and test the model on them
- Improve Definition 1
- Conduct ablation experiments

# Some of typos

- Fix \citep vs \citet
- "further lose temporal dependency"
- "in term of synchronization"
- "system theory, an important result being ..."
-  "recent approach <...> studied"
- "further introduce"
- "h and g are the same as in Equation 1" -- there are no h and g in Equation 1
- "RNN are dynamical systems" -> "RNN is.."
- "RNN are known" -> "RNNs are" or "RNN is"
-  "3 axis" -> "3 axes"
- "size of 128" -> "length of 128"
- "a initial learning rate" -> "an initial"
- "a SGRU" -> "an SGRU"

# Update

Thank you for the rebuttal. The writing was improved and it is easier to read the paper. Nevertheless, the experimentation remains weak. I increase the rating by one point.
review_title: Good idea, but experimentation is weak

final_score: rating: 3: Clear rejection
review_content: #### Summary:
Drawing inspiration from dynamic systems, the paper proposes a novel architecture that couple sequences. Such a system has easiness to bring two instances arbitrarily close and authors have shown the superiority of the approach over an action recognition dataset; but the results seem to far from state of the art on the dataset (see questions section). The authors also recognize that currently such systems need to calculate each pairs (can't be cached due to coupling) at inference time, which is slow. 

The main issue I found in this paper is its presentation. The authors claim to draw inspirations from dynamic systems (e.g. the important notion of  synchronized trajectories) in the abstract/introduction, cite related work and introduce a formal definition (with some errors, see section minor issues). However, the metric that ends up being used in eq(5) is a metric function of two RNN end states, which is common and wouldn’t help to highlight authors’ contributions.

Another concern is that the authors have tested on only one dataset where the claimed results are quite below state of the art on the dataset (see questions section). There are two baselines implemented (one very recent approach), however no simple baseline or other datasets to further support the approach.


Finally, after reading the paper, I think the paper proposes a new neural architecture for similarity learning rather than focusing on metric learning. In this aspect, the paper misses references to papers in the area.


#### Pros: 
Drawing inspiration from dynamic systems, the paper proposes a novel architecture that is not only dynamically involved with timesteps but is also coupled (between two instances), which has the capability of bringing two arbitrary sequences close. The authors have shown the superiority of the proposed approach over several metric learning baselines including the recently proposed RVSML (Su and Wu, 2019) over an action recognition dataset.

#### Questions:
Clarification question: Given the architecture, the methods can only be applied to sequences that have the same number of timesteps, is that correct?

The approach seems to be far from the state of art: In Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope [Wong and Kolter, 2018], the baseline network investigated achieves 5% error rate; for papers more focused on performance, Human activity recognition with smartphone sensors using deep learning neural networks [Ronao and Cho 2016] archives accuracy 95.75%. Why is there such a discrepancy between the results in the paper and state of the art results please (as it is more convincing to build upon state of the art results when possible). 

#### Minor Issues:
In Definition 1, the function two subsystems are synchronized if there exists a time-independent g (instead of a time dependent g as written). In fact, one can always find a trivial time dependent g satisfying the equation (by the way, it is written correctly with time independence in Brown & Kocarev (2000))

#### Minor suggestions:

Modifying neural architecture to capture better similarity has numerous works, one notable example in NLP is for example: A Decomposable Attention Model for Natural Language Inference [Parikh et al. 2016]. I would recommend including similar references in the paper to better situate the paper’s contribution.

The loss function is missing in the paper, so leading some missing details for training. 

Finally, I want to detail my argument to change the paper’s highlight on synchronization, which ends being a distance between two RNN end states. First, this is very far from the formal definition in eq(1) with a quite weak link; secondly, it is one of the most commonly used metrics in NLP/CV applications (and very probably other areas that I am much less familiar with). Given that an ICLR main paper is only about 8 pages, I strongly recommend authors to reconsider the paper structure to highlight its main contributions.

review_title: Not strong contribution with seemingly wrong paper highlight

Meta-review: The paper proposed to learn a sequence metric by sharing a memory cell between two LSTMs that run on pairs of sequences. I found the idea quite interesting, but I (and the reviewers) found the inspiration and the analogy from a dynamical systems perspective a unclear, unconvincing and maybe not even necessary to the core essence of the method that was proposed.

 The reviewers appreciated the improvement in clarity over the course of the review, but felt that there was still some more distance to cover. In addition, the results were not of a high enough quality to lend support to the success of the method and the experimental section needs more work making it not ready for ICLR acceptance at this time. 
Decision: Reject