Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper is very well written and easy to understand. They focus no domains of data where there exists some controllable parameter(s) for data generation, using this parameter in a way that resembles self-supervised learning losses. The main contribution I think is in the use of correlations of changes in the scoring space (d) with changes in the generative parameters.

The main weakness of the paper though is the novelty / proper connection to self-supervised learning work. There is a large body of research that uses techniques similar to those outlined in this work, particularly in the use of a siamese-like network to ensure that representations follow some characteristic. Besides being very close to [1] (cited in this work), there are a number of works that try to make sure that the output (loosely, and not directly) corresponds to a binary variable indicating whether the two inputs come from the same image / sequence / graph / etc [2, 3]. Closer to this work, there are self-supervised methods that learn by "ego motion" [4] and video representation comparisons [5], to name a few.

Full disclosure: I didn't read the appendix, and there seems to be interesting / useful results there. Perhaps some of the architecture description can be scaled down in the main text and some of the useful stuff from the appendix could be added.

[1] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
[2] Data-Efficient Image Recognition with Contrastive Predictive Coding
[3] Learning Representations by Maximizing Mutual Information Across Views
[4] Learning to See by Moving
[5] Unsupervised Learning of Visual Representations using Videos

Update:
I have read the responses and generally I am happy with them. I'd like to wait on R2 and see how their concerns over the metric play out, but I'm still leaning towards acceptance.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This is a well-written paper which looks into options for learning similarity metrics on data derived from PDE models common in the sciences. In comparison to other metric learning settings, here a type of ground-truth distance information is available (rather than, say, triplets), and it is possible to attempt to directly target an objective function which aims to match the learned distance to the ground-truth distance. The model architecture follows a fairly standard siamese-network setup.

Quite a bit of space is devoted to ensuring that the learned metric actually satisfies pseudo-metric axioms. This is all very clearly presented, with justifications for different modeling choices and how they preserve the axioms; my only criticism here is that many aspects of this are fairly obvious (i.e. an architecture which shares weights in computing the embeddings of both data points, followed by computing a squared L2 distance, will quite clearly get us in the ballpark of a pseudometric), but in my opinion "excess" clarity is much better than the opposite.

I think the more important contribution of the paper is in sections 4 and 5, which outlines a specific data generation process, including means of injecting noise, and compares options for loss functions. I would have expected pearson correlation to work quite well in this context, and it is interesting to note that performance notably improved by also adding an MSE term. I am curious about the "distance" prediction, as described just before equation 5, where it is stated that d \in R^n — is this really R^n? The target distance c is in [0,1]^n, and it seems like a simple modification to the distance prediction network would be capable of ensuring that the predicted values d also fall in this range. Such normalization could reduce the need for the MSE loss term, which presumably helps keep the overall relative scales of the two distances in check.

The empirical testing is also thorough, and I particularly appreciate the use of the random-weight networks as a baseline — I think it is good to note that these are actually fairly competitive on many of the test data sets (in fact, I believe it should be in bold for "TID" in table 1). 

I think the main weakness of this paper is that it falls slightly short of actually presenting the real use cases and needs for a similarity metric on PDE outputs — in my opinion, this comes to play when matching the output of a PDE-based model with real data. It would be nice to see a discussion of how this could be useful for parameter inference in PDE models. If there are other important applications of a distance learned in this way, I think the paper could benefit *greatly* by pointing them out. Otherwise, this risks being perceived as adding little value, since for individual PDE runs with known parameters, there is a ground-truth distance available — in which case, why bother using deep learning to estimate the distance, if the parameters are known? I think relevance to applications should to be clearly addressed.

The supplemental material is long, but complete and clearly presented.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use siamese convolutional network to generate a metric for images. The authors use both PDE generated simulated data and some real-world set to evaluate the proposed metric. 

While it seems to be a legit metric I have some concerns listed below:

1. Need a proof why the cnn-based evaluation metric is a metric. In particular, the triangle inequality.

2. Instead of using traditional metrics, I can first do a feature transform and then try a metric on top of the transformed feature space. Intuitively I do not quite see how the proposed metric is better than this naive feature+distance design.

3. The paper spend one and a half page describing the data generation process. The authors claim using PDEs to generate data can have some special control of the data de-similarity, on the other hand, the performance on the simulated data does not indicate how practical the designed “metric” is for real-world data which is possibly the main interest of most of the audience. 

4. The evaluation on the real-world data set is very limited. How does the proposed metric perform on image datasets such as CIFAR, ImageNet, MNIST? For example we may design experiment comparing inter/intra-class metric comparisons. And how that metric can be used to improve the sota results on the data sets?

5. The metric construction section is not quite self-contained. For example, could the authors states in details how the feature map normalization and aggregations are actually done in the algorithm instead of just citing some related works?

6. The loss function used in the training seems weird. I am worried how those two terms balance. Also it is a batch-dependent loss given the \bar{c} and \bar{d}. As the batch size gets changed the estimation accuracy of the Pearson coefficient may change in a different way as your first squared loss.

7. Pearson coefficient only captures the linear correlation. I would suggest looking into something like mutual information instead.

review_title: Official Blind Review #2

Meta-review: The authors present a Siamese neural net architecture for learning similarities among field data generated by numerical simulations of partial differential equations. The goal would be to find which two field data are more similar to each. One use case mentioned is the debugging of new numerical simulators, by comparing them with existing ones. 

The reviewers had mixed opinions on the paper. I agree with a negative comment of all three reviewers that the paper lacks a bit on the originality of the technique and the justification of the new loss proposed, as well as the fact that no strong explicit real world use case was given. I find this problematic especially given that similarities of solutions to PDEs is not a mainstream topic of the conference. Hence a good real world example use of the method would be more convincing.
Decision: Reject