Distance-based Composable Representations with Neural Networks
Authors: Graham Spinks
Marie-Francine Moens
Venue:  (2020)
Keywords: Representation learning
Wasserstein distance
Composability
Templates

Abstract: We introduce a new deep learning technique that builds individual and class representations based on distance estimates to randomly generated contextual dimensions for different modalities. Recent works have demonstrated advantages to creating representations from probability distributions over their contexts rather than single points in a low-dimensional Euclidean vector space. These methods, however, rely on pre-existing features and are limited to textual information. In this work, we obtain generic template representations that are vectors containing the average distance of a class to randomly generated contextual information. These representations have the benefit of being both interpretable and composable. They are initially learned by estimating the Wasserstein distance for different data subsets with deep neural networks. Individual samples or instances can then be compared to the generic class representations, which we call templates, to determine their similarity and thus class membership. We show that this technique, which we call WDVec, delivers good results for multi-label image classification. Additionally, we illustrate the benefit of templates and their composability by performing retrieval with complex queries where we modify the information content in the representations. Our method can be used in conjunction with any existing neural network and create theoretically infinitely large feature maps.

Under review as a conference paper at ICLR 2020
DISTANCE-BASED COMPOSABLE REPRESENTATIONS WITH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a new deep learning technique that builds individual and class rep- resentations based on distance estimates to contextual dimensions for different modalities. Recent works have demonstrated advantages to creating representa- tions from probability distributions over their contexts rather than single points low-dimensional Euclidean vector space. These methods, however, rely on pre- existing features and are limited to textual information. this work, we obtain generic template representations that vectors containing the average distance between the distribution of a class and that of contextual information, the latter which consists of selection of thresholded number of classes from the power set of classes. These representations have the benefit of being both interpretable and composable. They are initially learned by estimating the Wasserstein distance for different data subsets with deep neural networks. Individual samples or in- stances can then be compared to the generic class representations, which call templates, to determine their similarity and thus class membership. We show that this technique, which we call WDVec, delivers good results for multi-label im- age classification. Additionally, composability by performing retrieval with complex queries where we modify the information content in the representations. Our method can be used in conjunction with any existing neural network and create theoretically infinitely large feature maps.
illustrate the benefit of templates and their
INTRODUCTION
We introduce new deep learning technique to create interpretable and composable representations both for generic classes as well as for individual samples based on distance estimates with respect to contextual information. The generic class representations, which refer to 'templates' express how samples from a class relate to other classes on average and can used to efficiently determine class membership.
Most neural heworrk-bssed approaches to representation learning, focus on learning locations of entities in low dimensional Fuclidean vector space. Word2Vec, for example, extracts meaning from the learned location of words in a vector space (Mikolov et al.] 2013a b). Other text-based approaches use the hidden state vector of LSTM networks (Hochreiter & Schmidhuber, (197) that learn useful information while performing sequence-to-sequence learning tasks (Sutskever et al.) 2014), such machine translation (Wu et al. 2016), surrounding sentence reconstruction (Kiros) et al. 2015). For images, relevant features are typically extracted from convolutional neural net- works (CNNs) (LeCun al., 1998). When they are pre-trained on a classification task, they are con- sidered to contain abstract knowledge that can be used as input to perform further tasks (Vinyals al. 2015).
al. 2015 Karpathy Fei-Fei 2015
Such point-based representations have achieved great results for many tasks, but lack flexibility. Word2Vec representations don't change depending on the context. CNN-based representations fail to accurately relay all relevant features for interacting objects in scenes. Recent works in Natural Language Processing (NLP) have attempted to make point estimations dependent on the context, for example ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018). yet such representations still lack interpretability. Other approaches have started looking at representations that are essentially probability distributions that are built from the co-occurrence of particular contexts. [Singh et al.

---

Under review as a conference paper at ICLR 2020
2019) create sentence embeddings by defining Context Mover's Distance words occurring in different contexts and Wu et al. (2018) create text document embeddings with feature maps by estimating the distance of document to range of arbitrarily created documents. In this work
Estimated distance to environment "bat'
Template for Football
Football
Baseball
Template Baseball
Representations for individual samples are interpretable along contextual dimensions and
can be compared to generic class representations.
Representations created for input samples.
Figure 1: Our method creates representations for individual samples that can be interpreted along contextual dimensions, which we call environments. Those subvectors can then be compared to generic class representations, which we refer to as 'templates'.
will build on the latter approaches by creating a method that is not restricted by modality type. Our approach relies neural networks estimate Wasserstein distances between the sample spaces for different classes and the sample spaces for events that we call environments. These environments defined by the occurrence of any element in subset of attributes where each subset, limited to a certain size, is selected uniformly from the power set of all attributes. While in this work we limit broader
the definition of attributes to simply refer 'class labels', future work might build interpretation. In the simplest case, when the size of the subsets is limited to 1, each environment is given by the occurrence of a singular class label. The goal is to capture some common sense knowl- edge about how classes relate to a range of differentiating contexts. Such environments are bound to have distinctive features relation any given class as they are made up of combination of arbi- trarily chosen factors. This mechanism loosely resembles the associative nature of human memory. Long-term memory storage believed to rely on semantic encoding that performs better if can be associated with existing contextual knowledge (Lepage et al. 2000 Murdock 1982). Additionally, relating classes to arbitrarily chosen environments increases the probability of encountering distinc- tive diagnostic features, which should help retrieval, as might be the case with human long-term memory (Naime 2002).
We also propose method generate template representations of classes such that individual input samples can be interpreted reference the template representations. template representation is thus vector containing the average distance estimates between the sample distribution of a partieu- lar class and those of the available environments. This approach, illustrated in figure addresses the limitation of Euclidean point based representations where no distinction is made between generic class representations and specific instance representations. With the use of such templates, class membership of individual samples can then easily be determined by computing the similarity be- tween the representation of an individual sample and all template representations. We will show that, for the same neural network, this method performs well on lulti-label classification tasks for images, where any image can be assigned several classes (for example, 'baseball' and "bat").
Both the template and individual sample representations are interpretable in the sense that they contain distance estimates along contextual dimensions. Samples that belong to a particular class can be analyzed as to how and to what extent they differ from the generic template representation. A sample that is classified that that particular cat looks more like a bear than the average cat. Another advantage is that we can alter combine representations in order modify the information content. The usefulness of these characteristics becomes apparent in retrieval applications. We will demonstrate that allows one to retrieve image similar particular image but with altered information content.
'cat' might have smaller distance to the 'bear' class, suggesting
2

---

Under review as a conference paper at ICLR 2020
CONTRIBUTIONS
We introduce a technique to build representations based on probability distribution esti- mates that can be used in downstream tasks. The method. which we call WDVec (Wasser- stein Distance Vector), is general that it can be combined with any neural network.
The obtained representations are interpretable as distance estimates between class and con- textual information. They are composable in the sense that they can be modified to reflect different class membership or contextual information. Similar inputs lay near each other in the representation space.
We define template representations for concepts to which individual samples can be com- pared. This allows efficient class maasermmmbrsshi computation and easy manipulation in the representation space.
BACKGROUND
We will shortly explain some useful concepts, mostly in relation to distance estimates and how they can be found with neural networks. Such distance estimates will form the basis for our representation learning method further on. We will also refer to recent work that uses similar tactics.
Useful concepts Central to approach is the concept of the 'Earth Mover's Distance' (EMD) (Rubner et al. 2000), also known as the Wasserstein distance. It can be understood as the minimal amount of effort that is required to move the mass from one probability distribution another.
Finding the EMD between two distributions is traditionally done by solving the 'optimal transport problem' (Hitchoock) 1941 Altschuler al. 2017: Singh et 2019).
While some approaches Kusner et al. 2015 Wu et al. 20188) use some variation of Sinkhorn itera- tions (Sinkhorn 1964 Altschuler al., [001 ) to solve the transport problem, neural network-based approximations of the EMD have also been developed in recent years. This is usually in connection Generative Adversarial Networks (GANs) (Goodfellow al. 2014) where two neural networks, generator and discriminator, are staged against each other. Most recent GAN formulations make use of an Integral Probability Metric (IPM) that essentially defines the loss function for neural network (referred to the critic f instead of the discriminator) that allows it to maximally dis- criminate between two distributions. Most famously, in the Wasserstein GAN (WGAN) formulation (Arjovsky al., 2017). the IPM of the critic attempts estimate the Wasserstein distance between the distribution of real samples versus the distribution of generated samples. As the generator im- proves, the distance estimated by the critic should approach zero. The loss function for the critic of the Wasserstein GAN can be defined as follows:
dy(RR, G) sup
Bf((z) E 2 f())
fEr
where represents the distribution of real samples a dataset and represents the distribution of fake (or generated) samples. F is set of measurable, symmetric and bounded real valued functions. F is unbounded, equation will scale without bounds.
In order for this approximation to work, JF thus needs to be bounded. Usually, the critic network restricted to be I-Lipschitz function. Several methods have been proposed to enforce this con- straint, such as weight-clipping (Arjovsky et al.] (017). gradient penalties (Gulrajani et al., (017)) spectral normalization (Miyato et 20188). The Fisher GAN formulation takes a different approach and bounds J by construction, that is, by defining data dependent constraint on its second order The IPM becomes:
moments (Mroueh & Sercu 2017).
[[())) [f+)) R~R
E
dFisher (R, G) sup /EF 1122
RNG
(2) 1/2B (x)
Equation can be interpreted as the search for a critic function f that maximizes the average discrep- ancy between two distributions IR and G (thus maximizing inter-class variance) whilst minimizing the second order discrepancy (i.c., limiting intra-class variance) (Mrouch & Sercu 2017). Upon completing training. the numerator thus gives good Wasserstein distance estimate when inter-class
3

---

Under review as a conference paper at ICLR 2020
variance is small in comparison to intra-class variance. When this is not the case, the numerator will be large, yet not an exact approximation of the Wasserstein distance. In practice, the Fisher GAN IPM can be estimated with neural network training where the numerator in equation i maximized while the denominator is expressed as a constraint that is enforced with a Lagrange multiplier.
Recent work The EMD has been successfully applied to NLP problems. Kusner et al. (2015), for example, define the "Word Mover's Distance' (WMD), which measures the minimal amount of effort to move Word2Vec based word embeddings from one document to another. Their method is interpretable and outperforms other document distance metrics in text-based classification tasks. Wu et al. (2018) improve upon their solution by defining a Word Mover's Embedding, an unsupervised feature representation for documents, created by concatenating Word Mover's Distance estimates to arbitrarily chosen feature maps. They then calculate an approximation of the distance between a pair of documents with the use of kernel over the feature map. The building blocks of the fea- ture maps are thus documents built from an arbitrary combination of words. This idea is based on the Random Features approximation (Rahimi & Recht 2008) that suggests that randomized feature maps are useful for approximating shift-invariant kernels. [Singh et al. (2019) build on this idea to create unsupervised sentence representations where each entity is a probability distribution based on co-occurrence of words. They embed the distributions in a low-dimensional representation space for text and demonstrate state-of-the -hrt performance on tasks such as sentence similarity and word en- tailment. They note that their approach captures uncertainty and allows to interpret the outcome over different contexts, Also relevant is the concept of "classemes' in computer vision, where rich features are created out of combination of visual concepts (Torresani et al.. 2010). Finally, 'Wasserstein Discriminant Analysis' (Flamary et al. 2018) obtains features through dimensionality reduction by maximizing Wasserstein distances between classes while minimizing intra-class discrepancy.
Our method borrows some concepts from these works, such as the creation of possibly infinite- dimensional feature maps and representations that are built on (randomized) contextual information.
WDVec can be used with any modality though and features can be found with neural network based EMD estimations between different sample spaces. Additionally, we will create generic template representations well as individual sample representations, which allows efficient membership tests and complex retrieval queries. Our representations will be both interpretable and composable.
3 WDVEC: METHOD
We first define some notions that are of use to understand the WDVec method.
3.1 NOTATIONS
Given dataset with distribution P., we denote Ig as the sample space for the event E. In this paper, we compare the sample spaces Fc, of classes c; with the sample spaces Pe, of environments ey. Let ne denote the amount of classes and The the amount of environments for the representation spaces that are built in this work. Each environment e is composed of the union of rý attributes ag. where ry is an integer uniformly chosen from the range [1, R| with R the maximum amount of attributes per environment. In turn, each attribute 0g is uniformly selected without replacement from the set of all possible attributes. As a concrete example, given R = 5, one might obtain ri = 2 and r2 = 3. Environments might then become e (as U (13), e2 = (a1 U og U as1) and so on. Pe, would thus refer to the sample space for which attribute ag or @13 occurs. Note that 'attribute' in principle can be interpreted in a general manner, as it can refer to any attribute, class, feature, and so on. We limit the scope the experiments in section 4 where attributes will simply be classes, i.e., environments will be built from combinations of classes. We leave it to future work to explore other possibilities.
3.2 CONTEXTUAL DISTANCE
We propose to represent each sample as feature map that is constructed with distance estimates be- tween the distributions of classes and environments. Therefore, we calculate the estimated Wasser- stein distance W,, W.y(Pe,,
between the sample spaces of all classes c; in C' and of all environments ej. Intuitively, one can understand why such co-occurence contains useful information. A subset of image samples containing 'cats", for example, will have relatively small distance the subset
4

---

Under review as a conference paper at ICLR 2020
containing "dogs', and larger distance to the subset containing 'fork'. If one of the environments is defined by the occurrence of 'cats' as well, the estimated distance should be closer to O. For a con- crete example, see table [5 in the appendix. In cases where the estimated distance is not small, the IPM optimization essentially y splits samples from the classes and the environments by maximizing the outputs relating distinctive features of both groups. In the next section we explain how such class representations can be built with neural networks, as well as how representations for individual samples can be built.
3.3 IMPLEMENTATION WITH NEURAL NETWORKS
To get estimates of the EMD, we will train critic ff to maximize the Fisher IPM in equation 2 The advantage of the Fisher IPM over other Wasserstein distance estimation methods, is that any neural network can be used also leads to more stable and accurate distance estimates. The distance W (FF.,, Pe,) can then be found by maximizing equation 2 over the distributions P.,, and Pe,- As there are n. classes and ne environments, this would require training n. "n, critics which is not feasible in practice. Therefore pass samples through common neural network N for which the output layer has dimension of nt-- These n features are then passed to ne ",, single layer neural networks, the outputs of which constitute the estimates for all (Py(Pez c2 P Pez))
as long as the last layer is linear, dense layer. Empirically, the Fisher IPM
During training, any given mini-batch will contain samples for many different e and e. For any sample s, backpropagation is then performed efficiently by multiplying all ne nne outputs fúy with a mask that is 1 if s E PP,, or 8 E le,, 0 otherwise. The average for a critic, E [fr( can xniee
be calculated over all samples (weighted), weighted). Depending on the application, the first can improve performance for unbalanced datasets by implicitly including estimate of the probability of occurrence of particular class. Algorithm Al in appendix A.4.exppaain the algorithm detail.
only over the samples where the mask is 1 (non-
For typical GAN training, the loss function receives separate batches of real and fake samples. In our case, to improve efficiency the same batch is used for both Pe;; and Pej as the multiplication with the mask guides backpropagation over different contexts. While the last layers impose a slightly larger memory footprint, computing time is barely impacted compared to similar neural network-based methods. Also, for applications where only the compact output representation needs to be stored in a database, such as image retrieval, our method is very efficient. The general approach is illustrated in figure As we will discuss in the next section, to determine class membership, one only needs to perform a similarity calculation between a compact representation and the template representations.
3.4 TEMPLATE REPRESENTATIONS
Let's denote 0ij Z~Pey [f,(,)]] E [fag())]. This is effectively an estimate of the distance z~Pes
between the distributions of e; and eg. This value can found by saving the average value of this output as all training batches pass through the network during the last training epoch. Thus, for each class c, the class template representation T(;;) is then defined as:
T(c)) = (o....0..
i..., vector containing distance estimates between the sample spaces of class c and all n. environ- ments ey. Additionally, the representation for any individual sample 8 can be defined as the matrix D(s) with n. rows and ne columns containing the distance estimates diy(s):
dkg (5) = Z-Fee
[f((z)]-fat)
The result is that for an imput with class label c; (s) is correlated to To, as its distance estimates with respect to all different environments should be similar, whereas this is not the case for sample without class label G. Therefore, the cosine similarity between vector (s. and the template T(c:) will be large for input samples from class i.
(s)
Such templates can then be used in a variety of ways, for example in multi-label classification tasks (see section 4.2 Finding the classes to which sample belongs can simply be calculated
5

---

Under review as a conference paper at ICLR 2020
nooutput features
2R
nc
Neural Network
Input batch
ne
Figure 2: Overview of the WDVec approach. A neural network obtains ny features from inputs. These features serve the common input for n. ne critic functions Jij. ne is the amount of classes and n, is the amount of environments. These critics are trained to estimate the EMD between sample spaces defined by classes e; and environments Ej. ne template representations Te, are created by computing [fi((x)] over all training samples of class ç. Each template has a dimension Re-
by computing the cosine similarity between (s. and T(;: over all classes If they are more similar than threshold (the level of which is determined during training), the sample is assumed to belong to class Being able to check class membership for all classes is particularly handy for tasks where multiple labels are connected to each sample. Similar to this method, one can also define templates for each column (i.e., environment) dimension and infer which attributes are relevant for the current sample. The template representations thus offer information about how different classes and attributes relate to each other average. This is shown in table A5
Another application can found in image retrieval (see section 4.3) As the representations are interpretable over contextual environments, complex queries can be formulated such as: Retrieve image that is similar to a particular sample and that contains a particular class C; yet doesn' contain class Cj. Taking this step further, will discuss the ability to compose representations from existing ones order to modify the content.
3.5 COMPOSING NEW REPRESENTATIONS
Given sample Pe., and E P.... with representation D(s) (see section 3.4b. The goal is to modify D(s) such that it represents a sample for which & & P.,, and E Pe_ while preserving the contextual information of s, which in this case reflects how the sample relates all other classes.
To achieve this, a Singular Value Decomposition of D()): (ss) USV is performed, such that the of U and the columns of V can be interpreted the factors leading to the distance estimates as contributed by the corresponding classes ç and environments e, respectively. By making relatively straightforward changes U and and subsequently reconstructing the information, one is able to compose new representation D(3). To control the extent to which we modify the representation we introduce a factor 0 < 9 <= where g weighted combination of the original and modified representation.
a fully modified representation and q < 1 is
Concretely, by increasing the value of Uc..: one can increase the distance estimate of a sample with respect to class C+. The opposite can be done for class c.. As we know that 8 E PP,, and 8 & IP.... it is thus sufficient to simply swap the values of Uzq.: and Ua..: order to reverse class membership.
In this study, the attributes (from which the environments are constructed) are simply object classes. In order to change class membership for the sample we also need to modify all the columns V.+ and V._ that correspond to environments that are constructed from Ct and c_ respectively. Intuitively. for any sample s, the value of the j-th column V.,, that corresponds to environment é; influenced by two factors. The first factor the amount classes in éj. This is easily understood larger means dissimilar samples and thus larger distance estimate. The second factor is

---

Under review as a conference paper at ICLR 2020
whether the current sample belongs to any of the classes from which e, is constructed, for example, e1 = (cs U 07) and s E Peyy If that is the case, the values of D. are mainly determined by the values of U rather than V.j-
A simple strategy is then to set each element of V proportional to the amount of attributes r its environment has: V.++ (1 V) + V X+*+ +++++(V,mma //..iin/rra //rrr))) where Tmaz is the maximum amount of attributes in any environment. As mentioned before, g is a factor that allows intermediate solutions (thus not fully changing class membership if g < 1). Next, we set the value of each element of V.,- to be close to the mean value of each row of V as follows: (1 V. + (V..mmn + x--- -V.--V..mee
V.mean. V.,mi and .maa are the mean, minimum and maximum as calculated over the rows of V respectively. X+ and X- are independent parameters between 0 and I and can be tuned on the validation set. By having X+ < 1, one avoids mapping to outliers in V. By setting X- relatively small, V will be close to V.mean and thus have less importance for the reconstruction of the representation than the values of U.
"..mnn/rrw
A valid representation can then be reconstructed by calculating the outer product D; LoxU, x &
VT. where σj are the eigenvalues of ((s)- k,
Interestingly, the spectrum of the singular values appears to be non-flat and the eigenvectors be- longing to the largest eigenvalue can be visually inspected to determine to which classes the current sample has the smallest distance. In section 4.3 we illustrate this methodology by modifying the in- formation in existing representations and retrieving similar images to the modified representations.
4 EXPERIMENTS
To illustrate the approach we perform two types of experiments. First, in section 4.2 we show how compares to a (binary) cross-entropy baseline for multi-label image classification. section 4.3 the unique benefits of the representations and their composability are illustrated in retrieval setting.
4.1
SETUP
The experiments are performed on the COCO dataset (Lin et al. 2014) which contains multiple labels for each image. We use all available 91 class labels (which includes 11 supercategories that contain other labels, c.g. 'animal' is the supercategory for 'zebra' and 'cat'). One image can contain more than one class label. We use the 2014 train/val splits where we split the validation set into two equal, arbitrary parts to have validation and test set for the classification task [1] Unless noted otherwise, the model a ResNet-18 with weighted calculation of averages. n. is 300 and R is 40. ny is chosen to be 1024. All images are randomly cropped and rescaled to 224 224 pixels. For all runs, an Adam optimizer was used with learning rate 5.e - 3. for the Fisher GAN loss was set to 1e-6 In all experiments in this work, the attributes from which the environments are constructed are class labels, e.g. e1 = Cperson U Cear. Parameters are found empirically based on performance on the validation set.
4.2 CLASSIFICATION
In this experiment, we compare WDVec an approach that uses cross-entropy for multi-label image classification. The multi-label image classification task is of interest as multiple labels per image need to identified. The typical approach uses binary cross-entropy loss to determine whether each label independently should be applied or not. With WDVec, classification is performed by comparing classes to similar contextual environments, that in aggregate contain information about all classes. This is done efficiently through the use of the templates; the cosine similarity is com- puted between sample representation and template representation and compared to a threshold that is determined during training.
We use some recent state-of-the of-the--rt classification models to compare performance: ResNet-18, ResNet- 101 (He et al., 2016) and Inception-v3 (Szegedy et al., 2016). For each experiment actly the same neural networks are used in both approaches where only the last layers are modified.
Dataset splits will be published upon acceptance

---

Under review as a conference paper at ICLR 2020
Table 1: Fl scores, precision (PREC) and recall (REC) for different models for the multi-label classification task. a is the standard deviation of the FI score over three runs. BXENT refers to binary cross-entropy loss. All results are the average of three runs.
MODEL
METHOD
F1
PREC
REC
ResNet-18
BXENT
0.517
0.677
0.418
6.3e
ResNet-18
WDVec
0.529
0.600
0.473
3.5e
ResNet-101
BXENT
0.505
0.663
0.409
1.39e
ResNet-101
WDVec
0.538
0.595
0.494
2.6e
Inception-v3
BXENT
0.562
0.707
0.4667
9.4e
Inception-v3
WDVec
0.554
0.550
0.559
1.0e=
In [ it is shown that WDVec performs better in terms of the F1 score when combined with the ResNet models, and yields similar results with the Inception-v3 model. The performance of WDVec does depend on the choice of the parameters ne and f. Increasing Tie the amount of environments, leads in general to better performance, although tends to plateau after a certain level. For R, the maximum amount of concepts per environment, value of roughly ne/2 leads relatively good results. This can be understood in the sense that combining a large amount of attributes creates a unique subset to compare samples with. When R is too large, however, subsets with unique features are no longer created and performance deteriorates. The influence of ne and R is illustrated in the appendix in figure [ We also find that, even when n, and R are small, the outcome not particu- larly sensitive with regard to the choice of environments. This suggests that the amount and diversity of environments more important than the composition of the environments. This is illustrated in the appendix in tables Al and
4.3 RETRIEVAL
This experiment is designed to show the interpretability and composability of the representations. We formulate some retrieval queries that seek to retrieve samples with modified class membership while retaining contextual information as follows: "Given a sample that belongs to class c+ but not c-, retrieve the sample in the dataset that is most similar to that belongs to e- and not et C and c- are uniformly chosen in each case from the remaining class labels. For this experiment, we uniformly select 100 samples from the validation set as reference images. Let cos(x, y) be the cosine similarity between two flattened representations and y and let mean_cos(z; y) be the mean cosine similarity between x and y with the mean calculated over all class dimensions. We denote the representation for sample retrieve from the remaining samples the nearest sample s according to the following methods:
(ss) as defined in section 3.4 For each reference sample s,
Nearest Neighbor (NN): cos(D(s,), D(s))
2. SIM: cos (Ds,))),) ((s)) subject to cos( D+.. (4), < l4 and where 4,,4,T and L_ are the template representations and thresholds for classes C.. and
c_ respectively.
3. COMP slight: mean_cos(D(E))) (s)) over all classes e for which cos (s). TT) >
0.9 x te where D(5,) is modified version of D(s,) with factor q < 1 (here: q 0.6).
4. COMP heavy: mean_cos(((s.) D(s))
all classes for which cos (s), 0.9 X te where D(5,) is a modified version of D(s,) with a factor q =
Remark that for methods 3 and 4, the similarity is calculated over class dimensions where classes with low relevance, i.e., those that have a low similarity with the templates, are not taken into account. The templates are thus essential to methods 2,3 and 4. Note also that methods and 4 rely on the methodology of section 3.5 to compose new representations. For these experiments we set X+ and X- to 0.2 and 0.07 respectively. The distinction between methods 3 and 4 reflects how the representations have acquired some type of common-sense knowledge: it is not necessarily reasonable to retrieve an image that replaces a train with the category 'orange', as such a request could be interpreted in many ways. Moving more dissimilar images thus reasonable outcome
8

---

Under review as a conference paper at ICLR 2020
Table 2: Precision and similarity scores for retrieved images. The baseline consists of CNN features from the last pooling layer of ResNet-18 architecture. For comparison, the 'NN' method for unaltered queries added. 'COMP heavy' achieves the highest precision for altered class queries. Our method outperforms CNN features on all accounts in terms of precision. Note: the value of the similarity between CNN and WDVec should be compared directly as the features differ size.
NN
SIM
COMP slight
COMP heavy
CNN features
Precision
0.80
0.14
0.06
0.23
Avg sim
0.85
0.78
0.74
0.70
WDVec
Precision
0.85 0.97
0.31
0.47
0,62
Avg sim
0.81
0.64
0.60
in such cases. As a method illustrate the sensitivity of the factor g. we measure the average q that is needed to modify the class label from the retrieved sample in the COMP method. It turns out that the expected q 0.505 with a a of 0.026, illustrating both that the composition method works as expected and that class membership isn't sensitive to small perturbations.
As baseline we retain CNN features of size 512 from the last average pooling layer of the ResNet- 18 model. To compare them to WDVec, we define templates for the CNN as the average feature vector for a particular class. The 'SIM' method can be directly applied. For the 'COMP' methods, modify the features of a sample by subtracting the template of ct and adding the one of c...
The advantages the composability of the representations become obvious table The precision is shown, which is determined by whether the retrieved sample belongs to the desired class(es), as well as the average similarity between retrieved image and queried image. The 'SIM' method was often able find very similar samples that were misclassified however, thus leading to relatively low precision score. With the 'COMP' methods a better balance between similarity and precision be found. Some examples of retrieval results are presented in appendix A..] The obtained representations can thus be interpreted, composed, and capture useful information such that similar instances are near each other in the representation space. The templates are useful to interpret class membership efficiently and manipulate the instance representations as demonstrated here for a retrieval task. We see potential additional uses for the templates in future work, for example as reference representation that retains knowledge in a continual learning setting. Additionally, further research might explore how the method can applied to other tasks and modalities with alternative building blocks for the environments.
5 CONCLUSION
Our main contributions are firstly the introduction of a technique to build representations that rely on distance estimates that can be combined with any neural network. This is demonstrated by per- forming multi-label image classification with different state-of-the-art models and achieving good results. Secondly, the representations are interpretable and composable in the sense that class mem- bership and contextual information can be observed and modified by means of a singular value decomposition. This shown to be useful in a retrieval task where the class membership of samples is modified while contextual information is maintained. Samples that are altered achieve better trade-off between precision and similarity than unaltered samples. Finally, we introduce the con- cept of template representations which are generic class representations. We show how they lead to efficient and accurate class membership calculation a multi-label classification experiment. Ad- ditionally, they help achieve good precision in the retrieval task when representations are modified along class dimensions. The distance estimates in the templates also provide an overview of how different classes relate to each other.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jason Altschuler, Jonathan Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Advances in Neural Information Processing Sys- tems, pp. 1964-199999 1974, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein gan.
arXiv preprint
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Rémi Flamary, Marco Cuturi, Nicolas Courty, and Alain Rakotomamonjy. Wasserstein discriminant analysis. Machine Learning, 107(12):1923-1445 2.18.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp- 5769-5779, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Frank L. Hitchcock. The distribution of a product from several sources to numerous localities. Jour- nal of Mathematics and Physics, 20(1-4):224 2200, 1941.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Andrej Karpathy and Li Fei-Fei. Deep sesual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137, 2015.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems, pp. 3294-3302, 2015.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In International Conference on Machine Learning, pp. 957-966, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232 1998.
Martin Lepage, Reza Habib, Holly Cormier, Sylvain Houle, and Anthony Randal McIntosh. Neural correlates of semantic associative encoding in episodic memory. Cognitive Brain Research, 9(3); 271-280, 2000.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3788 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111-3119, 2013b.
Takeru Miyato, Toshiki Kataoka, Masamori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, arXiv preprint /802 2018.
10

---

Under review as a conference paper at ICLR 2020
Youssef Mrouch and Tom Sercu. Fisher gan. In Advances in Neural Information Processing Systems, Pp. 2513-2523, 2017.
Bennet B Murdock. A theory for the storage and retrieval of item and associative information. Psychological Review, 89(6):609, 1982.
James S Nairne. The myth of the encoding-retrieval match. Memory, 10(5-6):38--395, 2002.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). pp. 2227-2231 2018.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, pp. 1177-1184, 2008,
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as a metric for image retrieval. International Journal of Computer Vision, 40(2):99-121, 2000.
Sidak Pal Singh, Andreas Hug, Aymeric Dieuleveut, and Martin Jaggi. Context mover's distance & barycenters: Optimal transport of contexts for building representations. In Proceedings of ICLR Workshop on Deep Generative Models, 2019.
Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matri- ces. The Annals of Mathematical Statistics, 35(2):876-87) 879, 1964.
llya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104-3112, 2014.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pullern Recognition, pp. 2818-2826, 2016.
Lorenzo Torresani, Martin Szummer, and Andrew Fitzgibbon. Efficient object category recognition using classemes. European Conference on Computer Vision, pp. 776-789. Springer, 2010.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pp. 3156-3164. IEEE, 2015.
Lingfei Wu, Ian EH Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Raviku- mar, and Michael J Witbrock. Word mover's 's embedding: From word2vec to document embed- ding. arXiv preprint 1rXiv:1811.01713. 2018.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 1rXiv:1609.08144 2016.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pp. 2048-2057, 2015.
A APPENDIX
A.1 HYPERPARAMETER SELECTION FOR ENVIRONMENTS
In image Al the influence of and n. on performance shown for multi-label image classification. In table Al and A2 the standard deviations of the Fl scores are given for different values of n., and R respectively. The standard deviation is smaller than or equal to 1% indicating low sensitivity to different choices of environments. Also shown in tables A3 and A4 are the average values of the spectral norms over 100 arbitrarily selected representations, for varying hyperparameters n, and R.
11

---

Under review as a conference paper at ICLR 2020
nhe (log)
10
100
0.55
0.5
F1 71500
0.45
F1 for different (top)
different (bottom)
10
100
R (log)
Figure Al: Influence of R and ne on the Fl score for multi-label image classification using the WDVec approach with ResNet-18, When modifying Rt ne is fixed to 300. When modifying The R fixed to 40. All datapoints are the average three runs.
Table A1: Standard deviations of FI scores of the classification experiment for different values of Ne. Each value is computed on the basis of three runs.
T1e 2
T1 4
16 nh
8
R ==1 3.6e
9.1e 3
9.5e 3 7.5e
A.2 DISTANCES BETWEEN DIFFERENT CLASSES
Table A5 shows some rows and columns of template representation where R is 1, i.e., every environment consists of I particular class. The results show that related concepts have smaller estimated distances than those that are less related. Note that the class 'mouse" only appears in the dataset in relation to computers, rather than animals, which is reflected in the estimates. Some interesting things happen as well, as presumably some of the bananas the dataset appear in images with computers or similar contexts, thus leading to a smaller estimated distance between the classes 'keyboard' and 'banana". The distance between the class 'vehicle" and itself is accurately estimated as exactly O.
A.3 RETRIEVAL EXAMPLES
In figure A2 some results of retrieval of images are shown. The goal of the 'SIM' and 'COMP' methods retrieve an image where the class of the retrieved image reflects the content change, while maintaining the context of the original sample. The original classes in this case were respec- tively: sheep, zebra, toilet, train. They were modified into the following classes respectively: bear, giraffe, airplane and orange. For the SIM method, similar images are often obtained as only results are returned when the class membership, as determined by the cosine similarity with the templates, allows for it. The method performs badly when the classification fails, as for example in the second row of figure A2 It often leads to very good results though, as in the third row where the 'SIM' method retrieves image of a bathroom stall from airplane or in the fourth row where orange truck is retrieved that not train. The 'COMP slight" method reflects an intermediate trade-off between similarity and modified class membership. 'COMP heavy' modifies class membership cor- rectly more often, at the cost similarity.
A.4 ALGORITHM
In algorithm AT we shortly explain the exact algorithm to calculate the backpropagation during the training phase.
12

---

Under review as a conference paper at ICLR 2020
Table A2: Standard deviations of Fl scores of the classification experiment for different values of R. Each value is computed on the basis three runs.
R=
R
R
16
=
3.6e 4.2e
5.1e-
1.3c
1.0e -2
Table A3: The average of the spectral norm taken 100 arbitrarily selected representations for different values of ng-
The
Th
16
The
64
The = 256
R = 40
9.1c
2.8c*
1.5c
6.7e
Table A4: The average of the spectral norm was taken over 100 arbitrarily selected representations for different values of R.
R =
1
R =
4
R =
16
R = 32
n2
300
2.6
6.6e
7
5.3e
Table A5: Example of a template representation with R Classes that are related have smaller estimated distances. Rows are classes whereas columns are environments that are made up of attribute (in this case an attribute a class).
VEHICLE
SHEEP
BANANA
TRAIN
MOUSE
SPORTS BALL
PERSON
5.09
5.11
4.99
4.60
5.06
4.32
VEHICLE
0.0
5.27
5.23
5.24
5.30
5.26
ANIMAL
5.21
3.92
5.23
5.12
5.15
5.15
BASEBALL BAT
5.28
5.26
5.19
5.18
5.18
3.80
KEYBOARD
5.27
5.20
3.86
5.17
2.99
5.13
13

---

Under review as a conference paper at ICLR 2020
reference image
nearest neighbor
content change
COMP slight
COMP heavy
SIM
not: sheep but: bear
not: zebra
but: giraffe
not: toilet
but: airplane
not: train but: orange
Figure A2: Some results for image retrieval with the different methods. The column 'nearest neigh- bor' shows the most similar image without modification of the content. The column 'content change' refers to how the class of the representations was altered. The nearest neighbors to the altered rep- resentations then retrieved with the 'SIM', 'COMP slight' and 'COMP heavy' methods respec- tively.
14

---

Under review as a conference paper at ICLR 2020
Algorithm A1 Algorithm of the training process. Note that in this work the attributes are class labels. For matrices and tensors, refers to matrix multiplication and refers element-wise multiplication.
For each environment uniformly sample r from the range [1, RJ and uniformly select attributes from the set of all attributes. Create random feature vector v with shape [nc. nel which has value of for each uniformly selected attribute. Each column thus has maximum R non-zero entries.
Initialize À as tensor of zeros with shape [ne, me].
while Training do
Sample a mini-batch b, with batch size ny, containing samples s and one-hot labels 1, with shape (n6, nel.
Create masks
Expand to create a mask me with shape [m,, ne: nel. such that mek.i,; = if the k-th sample, 8k. belongs class 0 otherwise.
Multiply and v. then expand the result, to create a mask m, with shape [ns, nes ne), such that mrk..jj a where is the sum of all the attributes of the k-th sample that are present in environment j.
Calculate the FISHER GAN loss Propagate 5 through the neural network to obtain outiogits with shape (ne, ne].
Calculate out oultogits my and outq outiogits me-
Epj sum( p, dim 0)/ summmm dim 0)
Epfz suut put dim = 0)/ 00/sum((mm) (,y, dim 0)
Eqs sum(outq dim 0)/ sum(me. dim 0)
Eqs= = sum(outq outq, dim 0)/ sum(me: dim = 0) constraint (0.5 Epfs + 0.5 Eqjs)
Minimize the loss loss torch. surch.sum((F). Epy Fof + X constraint p/2 constraint end while
15


Reviews:
[{"review_id": "rkxEKlP0tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a template-based interpretable representation that works based on the earth mover's distance of each class to a number of \"environments\", which could be taken as union of a few random classes. To achieve this, they train several critics based on Fisher GAN. The method is evaluated based on classification and retrieval tasks. \nThe representation, by construction, is aimed towards interpretation and is specially useful in multi-class classification tasks. \nHere are my concerns:\n- Since the environments are taken randomly in the experiments, it is not investigated how sensitive the method is with respect to the choices of environments. Also, does it make any sense to design environments to include related (and not random) classes?\n- It seems necessary to include some experiments to assess sensitivity of the interpretation with regard to the small perturbations that are not changing the class label. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx0fuGctH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper contributions\n=================\n- This paper proposes a method for constructing representations using a matrix of Wasserstein distances. These distances measure the discrepancy between each class and each environment, that is a random combination of some classes.\n- The paper evaluates this approach on a task of image retrieval.\n\nGeneral notes\n============\nThe general idea of measuring the distribution divergence for a set of classes is interesting and seems to be novel. But I argue that this representation can be limiting:\n- A set of divergences doesn't contain any pixel-level information, only divergences to some predefined classes\n- As a consequence, this representation will not be able to discover information that is not covered by the labels\nBecause of these limitations, it seems that this particular representation may be less useful for some applications than others.\n\nI don't follow why the paper proposes to use 'environments'  -- random combinations of classes. It seems that a square matrix (n_c x n_c) with all classes should do the same job.\n\nThe experimentation is very weak and does very little to support the claims. The paper considers only one substantial task to test the representation. This task is image retrieval by image query. The paper doesn't provide any comparison to existing methods or simple baselines.\n\nThe second contribution that the representations are interpretable and composable is not addressed.  I seems that it should be hard to interpret a large vector of distances to randomly chosen subsets of classes. There is no experiment demonstrating interpretability of the proposed approach. The compositionality is not addressed either. The samples provided in the appendix are not convincing.\n\nThe paper is generally well written and it is easy to follow. The literature review can be improved by providing prior work where \"approaches use hidden state vector of LSTM\" and \"features extracted from CNNs\" instead of generic references. \n\nSome of the claims are vague and excessively broad:\n- The proposed technique can be used with any task, but the paper is clearly limited to the retrieval task\n- The environments are too vaguely described and can be misinterpreted in the introduction\n\nConclusion\n=========\n\nI recommend to reject on the basis that \n- the approach is more limited than the paper advocates\n- the experimentation is weak\n- some claims are not addressed\n\nOther notes\n==========\nI recommend using term divergence instead of distance when it is not symmetrical.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HygNqe2ntB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper defines a representation learning strategy based upon estimation\nof a matrix of Wasserstein distances.\n\nThe idea is excellent.  The ability to \"solve\" IPMs reliably is a recent\ndevelopment in deep learning whose ramifications are still being explored.\nIntuitively this line of research could plausibly result in general\nmethods which are theoretically intelligible and broadly applicable.\nIndexing at least one side of the matrix of estimated WDs with events\n(rather than classes) has interpretability properties useful for\ninformation retrieval and also conveys benefits reminiscent of learning\nwith privileged information.\n\nHowever, the exposition could be greatly improved by using the\nstandard language of probability theory.  The discussion in 3.1\nwas particularly painful to read.  What is the difference between\n\"existing in an environment\" and \"conditioning on a measurable event\"?\nPhrases like \"belonging to any random subset of the dataset\" suggest\na non-deterministic method of selecting an element of the power set of\nthe training data, but it is unclear what to do if more training data\narrives in this case.  \n\nThroughout the entire paper the word \"random\" is apparently used in the \ncolloquial sense of \"arbitrary\".  *Correct every instance of this.*\nIf you actually are referring to generating samples from a distribution,\nbe explicit about the generative process.\n\nSection 3.5 was more confusing than enlightening.  In general I understand \nthat environments can be leveraged for intelligibility and admit manipulation \nfor information retrieval.  The exact strategy remains somewhat opaque.  If \nyou are under space constraints refer to an appendix with more explicit details.\n\nIn the experiments section phrases like \"environments consist of random \ncombinations of classes\" is also not helpful.  Do you mean something like \n\"uniformly selected from the set of all class pairs?\"  Or something like \n\"uniformly selected from the power set of all classes?\"  How volatile\nare the experimental results with respect to the non-deterministic choice \nof environments? \n\nI want to accept this paper if the exposition is improved, which I think\nis possible during the response period.\n\nMy other comments are not blocking issues, but would either improve the\ncurrent paper or inform future directions of research.\n\nThe technique bears some resemblance to Wasserstein Discriminant\nAnalysis.[1]  That paper seeks a projection that maximizes the ratio of \nWasserstein distance between classes vs. within classes.  Here, \nalthough the common representation is a nonlinear mapping \nanalogous to a projection, we merely try to estimate all the\nWasserstein distances rather than maximize them, so it is not trained\nto be discriminative per se.   That is ok since the representation is\ndesigned to be used for a variety of tasks (modulo section 4.2), but it \ndoes leave open the question \"what if the matrix of estimated \nWasserstein distances isn't informative, e.g., due to poor choice of \nenvironments?\"  There is no attempt to assess the representation \nexcept via utility in downstream tasks.\n\nThe common representation was justified computationally, but I suspect\nis beneficial statistically.  It might facilitate safely including a\nlarge number of environments and then spectrally compressing (i.e., SVD)\nthe resulting matrix without overfitting the data.  However clearly if\nthe capacity of this layer is too small, then all estimated WDs will\nbe close to zero.  If we posit a low Bayes error classifier for the\nmulti-class problem associated with the dataset, that might imply there\nis some conditioning of the input under which the matrix of (actual) WDs\nhas rank equal to the number of classes, which would in turn provide a\nuseful diagnostic to guard against an insufficiently discriminative choice\nof environments or insufficient capacity in the common representation. If\nthe matrix is full rank with a flat spectrum, however, that might indicate\nthe choice of environments is too granular and overfitting has occurred,\nit's not immediately obvious to me how to guard against this.  \n\nI am curious what the results in appendix A.1. look like relative to the spectral \nnorm or the smallest eigenvalue of the estimated WD matrix (smallest \neigenvalue assuming number of environments < number of classes,\notherwise the k-th eigenvalue where k = number of classes).\n\n[1] https://arxiv.org/abs/1608.08063\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for learning class-level and individual-level (token-level) representations based on Wasserstein distances between data subsets.  The idea is appealing and seems to have applicability to multiple tasks.  The reviewers voiced significant concerns with the unclear writing of the paper and with the limited experiments.  The authors have improved the paper, but to my mind it still needs a good amount of work on both of these aspects.  The choice of wording in many places is imprecise.  The tasks are non-standard ones so they don't have existing published numbers to compare against; in such a situation I would expect to see more baselines, such as alternative class/instance representations that would show the benefit specifically of the Wasserstein distance-based approach.  I cannot tell from the paper in its current form whether or when I would want to use the proposed approach.  In short, despite a very interesting initial idea, I believe the paper is too preliminary for publication.
Decision: Reject