Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors proposed a template-based interpretable representation that works based on the earth mover's distance of each class to a number of "environments", which could be taken as union of a few random classes. To achieve this, they train several critics based on Fisher GAN. The method is evaluated based on classification and retrieval tasks. 
The representation, by construction, is aimed towards interpretation and is specially useful in multi-class classification tasks. 
Here are my concerns:
- Since the environments are taken randomly in the experiments, it is not investigated how sensitive the method is with respect to the choices of environments. Also, does it make any sense to design environments to include related (and not random) classes?
- It seems necessary to include some experiments to assess sensitivity of the interpretation with regard to the small perturbations that are not changing the class label. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper defines a representation learning strategy based upon estimation
of a matrix of Wasserstein distances.

The idea is excellent.  The ability to "solve" IPMs reliably is a recent
development in deep learning whose ramifications are still being explored.
Intuitively this line of research could plausibly result in general
methods which are theoretically intelligible and broadly applicable.
Indexing at least one side of the matrix of estimated WDs with events
(rather than classes) has interpretability properties useful for
information retrieval and also conveys benefits reminiscent of learning
with privileged information.

However, the exposition could be greatly improved by using the
standard language of probability theory.  The discussion in 3.1
was particularly painful to read.  What is the difference between
"existing in an environment" and "conditioning on a measurable event"?
Phrases like "belonging to any random subset of the dataset" suggest
a non-deterministic method of selecting an element of the power set of
the training data, but it is unclear what to do if more training data
arrives in this case.  

Throughout the entire paper the word "random" is apparently used in the 
colloquial sense of "arbitrary".  *Correct every instance of this.*
If you actually are referring to generating samples from a distribution,
be explicit about the generative process.

Section 3.5 was more confusing than enlightening.  In general I understand 
that environments can be leveraged for intelligibility and admit manipulation 
for information retrieval.  The exact strategy remains somewhat opaque.  If 
you are under space constraints refer to an appendix with more explicit details.

In the experiments section phrases like "environments consist of random 
combinations of classes" is also not helpful.  Do you mean something like 
"uniformly selected from the set of all class pairs?"  Or something like 
"uniformly selected from the power set of all classes?"  How volatile
are the experimental results with respect to the non-deterministic choice 
of environments? 

I want to accept this paper if the exposition is improved, which I think
is possible during the response period.

My other comments are not blocking issues, but would either improve the
current paper or inform future directions of research.

The technique bears some resemblance to Wasserstein Discriminant
Analysis.[1]  That paper seeks a projection that maximizes the ratio of 
Wasserstein distance between classes vs. within classes.  Here, 
although the common representation is a nonlinear mapping 
analogous to a projection, we merely try to estimate all the
Wasserstein distances rather than maximize them, so it is not trained
to be discriminative per se.   That is ok since the representation is
designed to be used for a variety of tasks (modulo section 4.2), but it 
does leave open the question "what if the matrix of estimated 
Wasserstein distances isn't informative, e.g., due to poor choice of 
environments?"  There is no attempt to assess the representation 
except via utility in downstream tasks.

The common representation was justified computationally, but I suspect
is beneficial statistically.  It might facilitate safely including a
large number of environments and then spectrally compressing (i.e., SVD)
the resulting matrix without overfitting the data.  However clearly if
the capacity of this layer is too small, then all estimated WDs will
be close to zero.  If we posit a low Bayes error classifier for the
multi-class problem associated with the dataset, that might imply there
is some conditioning of the input under which the matrix of (actual) WDs
has rank equal to the number of classes, which would in turn provide a
useful diagnostic to guard against an insufficiently discriminative choice
of environments or insufficient capacity in the common representation. If
the matrix is full rank with a flat spectrum, however, that might indicate
the choice of environments is too granular and overfitting has occurred,
it's not immediately obvious to me how to guard against this.  

I am curious what the results in appendix A.1. look like relative to the spectral 
norm or the smallest eigenvalue of the estimated WD matrix (smallest 
eigenvalue assuming number of environments < number of classes,
otherwise the k-th eigenvalue where k = number of classes).

[1] https://arxiv.org/abs/1608.08063

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Paper contributions
=================
- This paper proposes a method for constructing representations using a matrix of Wasserstein distances. These distances measure the discrepancy between each class and each environment, that is a random combination of some classes.
- The paper evaluates this approach on a task of image retrieval.

General notes
============
The general idea of measuring the distribution divergence for a set of classes is interesting and seems to be novel. But I argue that this representation can be limiting:
- A set of divergences doesn't contain any pixel-level information, only divergences to some predefined classes
- As a consequence, this representation will not be able to discover information that is not covered by the labels
Because of these limitations, it seems that this particular representation may be less useful for some applications than others.

I don't follow why the paper proposes to use 'environments'  -- random combinations of classes. It seems that a square matrix (n_c x n_c) with all classes should do the same job.

The experimentation is very weak and does very little to support the claims. The paper considers only one substantial task to test the representation. This task is image retrieval by image query. The paper doesn't provide any comparison to existing methods or simple baselines.

The second contribution that the representations are interpretable and composable is not addressed.  I seems that it should be hard to interpret a large vector of distances to randomly chosen subsets of classes. There is no experiment demonstrating interpretability of the proposed approach. The compositionality is not addressed either. The samples provided in the appendix are not convincing.

The paper is generally well written and it is easy to follow. The literature review can be improved by providing prior work where "approaches use hidden state vector of LSTM" and "features extracted from CNNs" instead of generic references. 

Some of the claims are vague and excessively broad:
- The proposed technique can be used with any task, but the paper is clearly limited to the retrieval task
- The environments are too vaguely described and can be misinterpreted in the introduction

Conclusion
=========

I recommend to reject on the basis that 
- the approach is more limited than the paper advocates
- the experimentation is weak
- some claims are not addressed

Other notes
==========
I recommend using term divergence instead of distance when it is not symmetrical.
review_title: Official Blind Review #2

Meta-review: The paper proposes an approach for learning class-level and individual-level (token-level) representations based on Wasserstein distances between data subsets.  The idea is appealing and seems to have applicability to multiple tasks.  The reviewers voiced significant concerns with the unclear writing of the paper and with the limited experiments.  The authors have improved the paper, but to my mind it still needs a good amount of work on both of these aspects.  The choice of wording in many places is imprecise.  The tasks are non-standard ones so they don't have existing published numbers to compare against; in such a situation I would expect to see more baselines, such as alternative class/instance representations that would show the benefit specifically of the Wasserstein distance-based approach.  I cannot tell from the paper in its current form whether or when I would want to use the proposed approach.  In short, despite a very interesting initial idea, I believe the paper is too preliminary for publication.
Decision: Reject