Title: Learning Invariant Representations Of Planar Curves 
Authors: Gautam Pai
Aaron Wetzler
Ron Kimmel
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Supervised Learning
Applications

Abstract: We propose a metric learning framework for the construction of invariant geometric
functions of planar curves for the Euclidean and Similarity group of transformations.
We leverage on the representational power of convolutional neural
networks to compute these geometric quantities. In comparison with axiomatic
constructions, we show that the invariants approximated by the learning architectures
have better numerical qualities such as robustness to noise, resiliency to
sampling, as well as the ability to adapt to occlusion and partiality. Finally, we develop
a novel multi-scale representation in a similarity metric learning paradigm.


Review discussion:
First, I have to say I really did enjoy the result of the paper.
Is the number N of points fixed? If yes, this does not seem to be however a limitation thanks to Section 5, second paragraph, could you clarify this? How did you chose the red points? (via the support of the max pooling?)
Have you tried to check the stability with respect to small deformations: when applying the same deformation to different curves, are there any linearisation properties?
Why did not you use circular convolution, since this symmetry is present in your data?(the first and final points of the input of size N should not change the final result, if I'm correct?) Does the normalization of the signature makes it independent from the first point used to process the curve?
On the Figure 5, I do not understand why there is still a "spike" at the end of the curvature, for the 4th image?
Thank you.
Thank you for your comments. Below are the responses to the issues you raised.
(1.) As you rightly pointed out, the number of points in the curve is not a limitation.
In fact our goal was to ensure that we train for the invariant which is robust to sampling issues of the curve and the experiment in Figure 9 demonstrates this. The red points were chosen arbitrarily.
(2.) Our experience has shown that the method produces results which are stable to small deformations of the input curves but we do not provide a study of this direction yet as it requires a more comprehensive investigation which we are currently undertaking.
In fact our current work can be seen as the foundation for providing a basis to exploring the method when applied to families of affine and other more complex deformations, both small and large. For these cases the inherent differential geometry is entirely different. The reason for this is that, for example, affine deformations are associated with "affine curvatures" which are different from euclidian curvature (both are local geometric functions of the same curve, but invariant to different types of transformations).
(3.) We chose not to use circular convolution so that the network would compute the invariant specifically for open curves therefore not be constrained to input closed contours. This makes sense when one considers that we are more interested in the local structure of a curve than its global structure. This locality is an advantage  as we can then use the invariants to process even partial shapes (effects of occlusion and partiality). Unfortunately enforcing locality also means that the signature does depend on the first point.
However, as demonstrated in the shape retrieval experiment in Figure 8, Section 5, this is mitigated by using shape distances like the symmetric Hausdorff distance which are independent of the starting point of the curve.
(4.) Thank you for bringing our attention to this. This is in fact an error in the code we used to generate the graph and we have submitted an updated version.
Thank you very much for your answers. A few comments, then I'll do my final review:
Maybe it would be nice not to show only closed curves, on the Figures.
Besides, I think it would be a great analysis to study the nature of the filters of the first layer.
Indeed, you perform link between band pass filters and differential invariants that I found great.
However, without going in micro-details and analysing each operator, I believe an analysis of the first layer would make a lot of sens.
Also, when you mention the scale-space, I think you might directly refer to the equations (2) and (3) of https://dspace.mit.edu/bitstream/handle/1721.1/3330/P-2200-29057294.pdf for instance.
Indeed, in computer vision it can have a different meaning.
I would have a very simple question which is for my curiosity. Are there any works related to wavelets?
- Why there is no rotation for the negative samples?
- Are the negative samples always the same contour, as visualized in the table 1, only with a different scale?
- Why is the contour retrieval not computed on the 350 testing and validation contours? The precision seems to be remarkably low even for such small dataset.
- Is the method able to achieve invariance to translation and uniform scaling or does it depend on the contour pre-processing?
(1 & 2. ) A positive example is formed by pairing the curve and its transformed version. A negative example is constructed by pairing a curve and the smoothed AND transformed version. This format of negative pairs enforces descriptiveness in the signatures locally i.e. for every point on the curve. During construction of the table shown in the paper we did not include examples of the negative transformations and we are submitting a revised table with the transformations to enhance clarity.
(3.)  The work presented is focused specifically on describing a novel learning based technique for extracting
a locally learnt descriptor associated to each point on a curve. We have not yet fully explored the behavior of our new descriptor
in the context of different applications. Our main aim has been to show that this new learning based signature has interesting and desriable numerical properties. Our use of Shape Retrieval in the paper is as a toy demonstration that the signature can in fact be applied. It does not represent a thorough exploration of the behavior of numerical results in this context as this will be the content of later work.
(4.) Invariance to translation and uniform scaling is done in the pre-processing stage. Every curve input to the network is mean centered and uniformly scaled to unity.
Why not use a rasterized representation of the shapes rather than the parametric curve representations?
In other words, you could pursue the same Siamese network based metric learning approach on pixel-domain representations of the MPEG-7 shapes. Parametric curves are of course elegant from a mathematical perspective, but when mixed with the machinery of CNNs, one wonders why not go all-in on image based representations, using spatial transformer networks, etc. I'm asking this since the sight of MPEG-7 shapes, the MATLAB bwboundaries() function and Mokhtarian citations seem oddly anachronistic in a submission to ICLR 2017.
Thank you for your question and comments. Our response is as follows:
The choice between a parametric 1D form or a 2D implicit representation of a shape fundamentally changes the approach of the underlying research.  For this work we made the specific choice of using a parametric curve representation since our main purpose was to connect the theory of differential geometry of curves with the computational engine of a convolutional neural network. This choice led us to relate powerful mathematical theorems of curves and curve evolution with the contrastive loss-function of metric learning, thereby providing a geometric insight into the action of a convolutional neural network in the learning process.
Even though, the differential geometric analysis of curves is indeed an old topic, we feel that it's connection to modern deep learning systems is worthy of highlighting, thereby making a stronger case for using learning methods in different areas. The choice of using MPEG7 and extracting curves with bwboundaries() was as a result of the desire to quickly find a toy dataset enabling us to progress with our research. We trust the reviewer can appreciate that this choice was practically oriented and has little bearing on the content of our submission. We chose ICLR as the best option to submit this research based on the idea that the LR in the title of the conference (Learning Representations) would perfectly match the spirit of our work: investigating the properties of a 1D convolutional neural network to learn an invariant representation of intrinsic properties of a curve undergoing transformations.
Pros :
- New representation with nice properties that are derived and compared with a mathematical baseline and background
- A simple algorithm to obtain the representation
Cons :
- The paper sounds like an applied maths paper, but further analysis on the nature of the representation could be done, for instance, by understanding the nature of each layer, or at least, the first.
We thank you  for your positive reviews and appreciate the constructive suggestions.
About your queries:
We have updated the paper and incorporated your suggestion of analysing the first layer. We have put in a figure in the appendix section showing some of the filters from the first layer and compared their shapes to standard gaussian filters and their derivatives to obtain a rough perspective on their action. We can see that barring random sign flips (which are possibly corrected for in the later stages of the network), some of the filters reasonably co-relate with standard 1D derivatives.
About your question on Wavelets, I think you might be interested in the following:
Tieng, Quang Minh, and W. W. Boles. "Recognition of 2D object contours using the wavelet transform zero-crossing representation." IEEE Transactions on Pattern Analysis and Machine Intelligence 19.8 (1997): 910-916.
Authors show that a contrastive loss for a Siamese architecture can be used for learning representations for planar curves. With the proposed framework, authors are able to learn a representation which is comparable to traditional differential or integral invariants, as evaluated on few toy examples.
The paper is generally well written and shows an interesting application of the Siamese architecture.
However, the experimental evaluation and the results show that these are rather preliminary results as not many of the choices are validated. My biggest concern is in the choice of the negative samples, as the network basically learns only to distinguish between shapes at different scales, instead of recognizing different shapes. It is well known fact that in order to achieve a good performance with the contrastive loss, one has to be careful about the hard negative sampling, as using too easy negatives may lead to inferior results.
Thus, this may be the underlying reason for such choice of the negatives? Unfortunately, this is not discussed in the paper.
Furthermore the paper misses a more thorough quantitative evaluation and concentrates more on showing particular examples, instead of measuring more robust statistics over multiple curves (invariance to noise and sampling artifacts).
In general, the paper shows interesting first steps in this direction, however it is not clear whether the experimental section is strong and thorough enough for the ICLR conference.
Also the novelty of the proposed idea is limited as Siamese networks are used for many years and this work only shows that they can be applied to a different task.
Thank you for your review and detailed comments. Our response is as follows:
About the negative examples: We like to highlight that analogous to Euclidian curvature, our goal in this paper is to train for a local descriptor of the curve (a feature for every point of the contour) and NOT a global shape descriptor (a feature for the entire shape as a whole) usually used for classifying shapes. This implies that a negative example is something that has to be different locally, for all points on the curve.
A negative example is obtained by pairing a curve with its much smoothed or evolved counterpart (after euclidian transformations of course), because, if we observe the curve locally at any point,  the analogous point in the smooth curve would have destroyed important features of the contour and this is NOT something we want our learned descriptor to have.
As you have rightly pointed out: “ using too easy negatives leads to inferior results “. What we wish to highlight is that the theorems by Hamilton and Grayson (Section 4, second paragraph) define a geometric notion of this “easiness". It says that any closed curve can be abstracted to various levels using mean curvature smoothing. The highest level is a circle, which has a constant curvature for every point. This is exactly in sync with the contrastive loss function of the siamese network. If we do not provide any negative example (the easiest training possible), we expect a constant function as the network output, analogous to a constant curvature. As we begin to provide negative examples with increasing richness profiles as in Table 1, we make it less and less easier for training, thereby discovering a scale-space of signatures which is “learned" rather than axiomatically constructed as demonstrated in Figure 6.
The novelty in our paper is first, to show this connection between classical theorems in differential geometry and modern deep-learning systems, and second, to highlight that these learned signatures have better numerical properties as compared to their axiomatic counterparts.
Although we agree that a larger evaluation set would be useful, we do not believe it would change the core results presented here. Different datasets are currently under investigation for future work.
Thank you for the clarification with regards to the negative samples used. I must admit that my comment comes from misunderstanding of the fact that it is only a local shape descriptor.
But then I have also overstated usefulness of the tackled task.
I still miss another novelty other than applying a simple deep learning architecture to a task where it was not used before. And in this case, it is questionable whether this task is still relevant.
I'm torn on this one. Seeing the MPEG-7 dataset and references to curvature scale space brought to mind the old saying that "if it's not worth doing, it's not worth doing well. " There is no question that the MPEG-7 dataset/benchmark got saturated long ago, and it's quite surprising to see it in a submission to a modern ML conference. I brought up the question of "why use this representation" with the authors and they said their "main purpose was to connect the theory of differential geometry of curves with the computational engine of a convolutional neural network. " Fair enough. I agree these are seemingly different fields, and the authors deserve some credit for connecting them.
If we give them the benefit of the doubt that this was worth doing, then the approach they pursue using a Siamese configuration makes sense, and their adaptation of deep convnet frameworks to 1D signals is reasonable. To the extent that the old invariant based methods made use of smoothed/filtered representations coupled with nonlinearities, it's sensible to revisit this problem using convnets. I wouldn't mind seeing this paper accepted, since it's different from the mainstream, but I worry about there being too narrow an audience at ICLR that still cares about this type of shape representation.
Thank you for your review and assessment. We appreciate your comments and agree with most parts of it.
However, we do feel that it is definitely interesting and worthwhile to use learnable architectures to model geometric representations, thereby re-discovering traditional representations from a learning viewpoint.  We believe that the marriage of modern ML tools with classical geometric frameworks and approaches is in fact becoming popular. See for example [Bronstein, et al. Geometric deep learning: going beyond Euclidean data]. We believe our work plays a role in supporting this direction and therefore our approach in this paper was to ask:  “ Can it be done ? “ and “ Can it be done better ? “. We are happy to present positive outcomes on both counts.
Based on the constructive suggestions of the reviewers, we have updated the paper with two minor changes:
(1.) We have added a figure in the appendix section, showing the learned filters from the first layer of the network.
(2.) We have added an additional line in Section 4:  "Since we are training for a local descriptor of a curve, that is, a function whose value at a point depends only on its local neighborhood, a negative example must pair curves such that corresponding points on each curve must have different local neighborhoods",  in order to highlight the locality property of our framework.

Meta-review:
role: PC
data: comment: This work proposes learning of local representations of planar curves using convolutional neural networks.
 Invariance to rigid transformations and discriminability are enforced with a metric learning framework using a siamese architecture. Preliminary experiments on toy datasets compare favorably with predefined geometric invariants (both differential and integral). 
 
 The reviewers found value on the problem set-up and the proposed model, and were generally satisfied with the author's response. They also expressed concern that the experimental section is currently a bit weak and does not include any real data. Also, the paper does not offer theoretical insights that inform us about the design of the representation or about the provable invariance guarantees. All things considered, the AC recommends acceptance in the form of a poster, but strongly encourages the authors to strengthen the work both in the experimental and the theoretical aspects.

Decision: Accept (Poster)