Venue: ICLR 2021 Conference

Reviews:

final_score: rating: null
review_content: Dear colleagues,

I feel I should open the discussion phase on this paper since my review score is the most divergent from all reviewers

I can see the concerns of R3 and R1 about the limited novelty of the work regarding the algorithmic proposal, and that’s also what I stated as the main flaw of the manuscript. However, in my opinion this paper is still a clear accept.

The siamese architecture employed is indeed not novel at all, but I do think the value of this work comes from that. For me the main contribution of this paper is to show how an specific feature representation for biometric data can work in a real world user authentication use case. And for me it’s not just an incremental contribution (term which I do not really like). The combination of feature proposal, practical experimentation and actual real world insights can be of great interest for the audience. Just the dataset collected is a very significant contribution to the community.

I could accept that the contribution of the work is not enough if there would be not a significant advance from the current state of the art, but to the best of my knowledge (and after checking bibliography myself) I’ve not seen any other work that could make this paper as merely incremental. 

Finally, I do not consider the paper is out of the scope of the conference. ICLR is indeed a proper venue to present advances in neural architectures but it is not its sole focus. For the case of this paper, whose main goal is to study the feasibility of its approach in real world scenarios, I could even see the use of a “simplistic” architecture as a pro rather than a con

review_title: About the novelty of the work

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Authors propose an approach to perform authentication based on users' behaviour (behavioral biometrics) on two different domains (web and mobile) by means of a siamese NN. The proposal is evaluated on three different dataset, collected ad-hoc for this work, showing a performance up to >0.9 acc. In the experimentation setup authors include different data splits, sequences lengths and training methods (pairs and triplets). The work shows the feasibility of using this type of approach for few-shot classification under real industry constraints.

------------------
Contributions

My recommendation for this manuscript is a strong accept. I believe this paper makes a relevant contribution to the field since it addresses a topic which is quite interesting for the audience, it is technically sound, covers different hypotheses and provides very useful insight to practitioners about the use of these approaches in real industry applications. The last point is the most interesting in my opinion, honestly. 
The paper is well structured, with a clean format, very well written and easy to read.
Probably its weakest point is the novelty, since it addresses a well-known problem using a long-established NN architecture. However the combination of one-shot learning, siamese architecture and behavioral biometrics is something I personally have not seen before, so I consider it robust enough.

------------------
Recommendations

Although it is covered in the section 3.1, given the scope of the conference and its relevance for the reproducibility of the results, I’d ask authors to include a clear and detailed explanation of the feature vector employed. Some chart or graph where the reader can see easily how many features come from which source (sensor, keypad, gesture), how the derived features are computed and how movements are translated to features. Even better if the authors can include some examples.  Maybe as an appendix, but I really believe it’s important and a clear improvement for the manuscript.

I think it’d be interesting to add further details about the data splitting mechanism. I believe the paper needs at least to state clearly whether the train/test split is done by users (where a group of users are employed to perform inference and the rest as control test) or by sequences (where the data from each user is split into train and test). This is a relevant topic since the later has been demonstrated in previous HAR works to offer less robust and generalizable models. Please expand this point.

Figure 4 - Please specify clearly which sequence length is the one used for this experiment. It seems taken for granted it is 10 sec , since it’s the configuration offering best results, but please make it more clear to the reader.

From a personal perspective I think it’d be interesting to expand the work adding feature extraction mechanism using the state-of-art in time series modelling, this is, recurrent neural units or attention based mechanism. I’m really curious to see how that compares with the current engineered features.

------------------
Minor

Figure 2b - I’d recommend to scale the Accuracy axis since the series are hard to analyse (they are too close together)

Typo: “ Figure 2 shows the ROC curve and the FAR and FRR for the best setup in web environment” -> “Figure 3 shows the ROC curve and the FAR and FRR for the best setup in web environment”

Typo: “Figure 2 shows the ROC curve and the FAR and FRR for the best setup in mobile environment.” -> “ “Figure 3 shows the ROC curve and the FAR and FRR for the best setup in mobile environment.”

review_title: Review of Paper 1323 (ICLR 2021)

final_score: rating: 4: Ok but not good enough - rejection
review_content: 

Summary:

 
The paper proposes a siamese neural network to perform few-shot verification of user behaviour.

##########################################################################

Reasons for score: 

 
Overall, I vote for rejection. I like the idea of applying the siamese paradigm to authentication however the paper offers limited novelty for a representation learning conference. There are zero baselines (no siamese) compared.

 
##########################################################################Pros: 

 

 
1. The proposed architecture seems to be very accurate, as reported in the experimental section. 

2. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results across different datasets, to show the effectiveness of the proposed framework.

 
##########################################################################

Cons: 

 
1. My major concern is the novelty of the siamese network. There seems to be no new component proposed that is tailored to biometric data.

 
2. I would suggest thinking about the contribution to the ML community. If the vanilla siamese network works like a charm, is the authentication problem solved? 
 

 
##########################################################################

Questions during the rebuttal period: 

 
Please acknowledge the contribution of the paper for an ML venue.

review_title: Straightforward approach but limited ML contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes the use of siamese neural networks for the task of user verification by learning to identify their behavioral patterns of interaction with mobile phones or web browsers. Notably, the behavioral characteristics of users when engaging with these devices, measured in the form of mouse movements or keyboard typing patterns on browsers or touch patterns on mobile phones, serve as a proxy for more direct biomarkers such as face recognition, eye scans, etc. The authors argue that the latter requires more specialised hardware and is thus not easily scalable. Instead, they architect the first NN based solution that serves both modalities, web and mobile phones, while trying to be more accurate and responsive.

Review:
(a) The study is of good quality. The language is clear and concise. Related work is well covered to the best of my knowledge, making reference to one of the oldest instantiations of such model for signature verification.

(b) The originality of the method viz-a-viz the domain 'of biometric verification' seems to be high. In particular, existing solutions to the problem of behavioral biometric verification have either used different models (SVMs or RDTs) or different features or different formulations (classic supervised recognition setting using one model per user). This disposes them to problems such as limited scalability, lower performance and high latency. Herein, their solution of using a single siamese network, for both modalities, and across different users, adds generality, performance and speed. They have curated experiments with different number of users at training time, and using different metric losses (pairwise and triplet) for representation learning via similarity matching, shedding light on the effectiveness of their approach.

(c) That said, originality in terms of research contribution is limited. Siamese NNs have long been in use and recently revisited for variety of vision tasks such as face recognition[1, 2], object tracking [3], etc. The loss formulation used here is also standard, not going beyond the staple pairwise and triplet. Papers where newer losses or network architectures or training paradigms have led to better analysis and results have made sound research contributions and exhibited suitability for the main track e.g. [1, 2]. In other cases, where tasks are key, and results are good but the approach is largely borrowed, it serves good to go for a workshop track e.g. [3].

(d) Comparisons with existing methods in the domain have not been included to the main paper (they are supposed to be in the appendix). The aim perhaps of this paper is to present a baseline NN-based approach for user verification/ biometrics. I believe this would be a good fit for presentation at a more focused workshop on biometric verification. As a main track ICLR submission I find it lacking in novelty, reach and research contribution.

[1] Triplet Loss in Siamese Network for Object Tracking, Dong and Shen (ECCV 2018)
[2] Occlusion Robust Face Recognition Based on Mask Learning With Pairwise Differential Siamese Network, Song et.al. (ICCV 2019)
[3] Fully-Convolutional Siamese Networks for Object Tracking, Bertinetto et. al. (ECCV 2016 workshop paper)

review_title: Good article, but most suitable as a workshop submission

Meta-review: This paper received 3 reviews with mixed initial ratings: 9,4,5. The main concerns of R1 and R3, who gave unfavorable scores, included lack of novelty and hence limited value of this work for the ML community. At the same time, R5 strongly advocates for acceptance and mentions meaningful contributions in the context of the specific application, including the new dataset. In response to that, the authors submitted a new revision and provided detailed responses to each of the reviews separately, which did not change the position of the reviewers.
The AC agrees with R1 and R3 that, even though the biometrics-related contributions are relevant, the scope of this work is too narrow and application-driven for presentation in the main track of ICLR. As a result, the final recommendation is reject.
Decision: Reject