Title: REPRESENTATION LEARNING WITH MULTISETS

Abstract: We study the problem of learning permutation invariant representations that can capture "flexible" notions of containment. We formalize this problem via a measure theoretic definition of multisets, and obtain a theoretically-motivated learning model. We propose training this model on a novel task: predicting the size of the symmetric difference (or intersection) between pairs of multisets. We demonstrate that our model not only performs very well on predicting containment relations (and more effectively predict the sizes of symmetric differences than DeepSetsbased approaches with unconstrained object representations), but that it also learns meaningful representations.

1. INTRODUCTION:

Tasks for which the input is an unordered collection, i.e. a set, are ubiquitous and include multipleinstance learning Ilse et al. (2018) , point-cloud classification Zaheer et al. (2017) ; Qi et al. (2017) , estimating cosmological parameters Zaheer et al. (2017) ; Ravanbakhsh et al. (2016) , collaborative filtering Hartford et al. (2018) , and relation extraction Verga et al. (2017) ; Rossiello et al. (2019) . Recent work has demonstrated the benefits of permutation invariant models that have inductive biases well aligned with the set-based input of the tasks (Ilse et al., 2018; Qi et al., 2017; Zaheer et al., 2017; Lee et al., 2019) .

The containment relationship between sets -and intersection more generally -is often considered as a measure of relatedness. For instance, when comparing the keywords for two documents, we may wish to model that {currency, equilibrium} describes a more specific set of topics than (i.e. is "contained" in) {money, balance, economics}. The containment order is a natural partial order on sets. However, we are often interested not in sets, but multisets, which may contain multiple copies of the same object; examples include bags-of-words, geo-location data over a time period, and data in any multiple-instance learning setting (Ilse et al., 2018) . The containment order can be extended to multisets. Learning to represent multisets in a way that respects this partial order is a core representation learning challenge. Note that this may require modeling not just exact containment, but relations that consider the relatedness of individual objects. We may want to learn representations of the multisets' elements which induce the desired multiset relations. In the aforementioned example, we may want money ≈ currency and balance ≈ equilibrium.

Previous work has considered modeling hierarchical relationships or orderings between pairs of individual items (Ganea et al., 2018; Lai and Hockenmaier, 2017; Nickel and Kiela, 2017; Suzuki et al., 2019; Vendrov et al., 2015; Vilnis et al., 2018; Vilnis and McCallum, 2015; Li et al., 2019; Athiwaratkun and Wilson, 2018) . However, this work does not naturally extend from representing individual items to modeling relations between multisets via the elements' learned representations. Furthermore, we may want to consider richer information about the relationship between two multisets beyond containment, such as the size of their intersection.

In this paper, we present a measure-theoretic definition of multisets, which lets us formally define the "flexible containment" notion exemplified above. The theory lets us derive method for learning representations of multisets and their elements, given the relationships between pairs of multisets -in particular, we propose to use the sizes of their symmetric differences or of their intersections. We learn these representations with the goal of predicting the relationships between unseen pairs of multisets (whose elements may themselves have been unseen during training). We prove that this allows us to predict containment relations between unseen pairs of multisets. We show empirically that the theoretical basis of our model is important for being able to capture these relations, comparing our approach to DeepSets-based approaches (Zaheer et al., 2017) with unconstrained item representations. Furthermore, we demonstrate that our model learns "meaningful" representations.

2. RELATED WORK:

2.1 SET REPRESENTATION Qi et al. (2017) and Zaheer et al. (2017) both explore learning functions on sets. Importantly, they arrive at similar theoretical statements about the approximation of such functions, which rely on permutation invariant pooling functions. In particular, Zaheer et al. (2017) show that any set function f (A) can be approximated by a model of the form ρ a∈A φ(a) for some learned ρ and φ, which they call DeepSets. They note that the sum can be replaced by a max-pool (which is essentially the formulation of Qi et al. ( 2017)), and observe empirically that this leads to better performance.1 More recently, there has been some very interesting work on leveraging the relationship between sets. Probst (2018) proposes a set autoencoder, while Skianis et al. (2019) learn set representations with a network that compares the input set to trainable "hidden sets." However, both these approaches require solving computationally expensive matching problems at each iteration. Vendrov et al. (2015) and Ganea et al. (2018) seek to model partial orders on objects via geometric relationships between their embeddings -namely, using cones in Euclidean space and hyperbolic space, respectively. Nickel and Kiela (2017) use a similar idea to embed hierarchical network structures in hyperbolic space, simply using the hyperbolic distance between embeddings. These approaches are unified under the framework of "disk embeddings" by Suzuki et al. (2019) . The idea is to map each object to the product space X × R, where X is a (pseudo-)metric space. This mapping can be expressed as A → (f (A), r(A)), and it is trained with the objective that A B if and only if d X (f (A), f (B)) ≤ r(B) -r(A). An equivalent statement can be made for multisets (see Proposition 3.2.4).

2.2. ORDERS AND HIERARCHIES:

Other work has taken a probabilistic approach to the problem of representing hierarchical relationships. Lai and Hockenmaier (2017) attempt to formulate the Order Embeddings of Vendrov et al. (2015) probabilistically, modeling joint probabilities as the volumes of cone intersections. Vilnis et al. (2018) represent entities as "box embeddings," or rectangular volumes, where containment of one box inside another models order relationships between the objects. (Marginal and conditional probabilities can be computed from intersections of boxes.) Vilnis and McCallum (2015) propose modeling words as Gaussian distributions in order to capture notions of entailment and generality, and this work has been extended to mixtures of Gaussians by Athiwaratkun and Wilson (2017) .

2.3. FUZZY-AND MULTI-SETS:

The theory of fuzzy sets can be traced back to Zadeh (1965) . A fuzzy set A of objects from a universe U is defined via its membership function µ A : U → [0, 1]. Fuzzy set operations -such as intersection -are then defined in terms of this function. In modern fuzzy set theory, intersection is usually defined via a t-norm, which is a function T : [0, 1] 2 → [0, 1] satisfying certain properties. The intersection of two fuzzy sets A and B is defined via the membership function µ A∩B (x) = T (µ A (x), µ B (x)). (More in-depth background, including the defining properties of t-norms, is provided in Appendix B.) There is also more recent literature on extending fuzzy set theory to multisets (Casasnovas and Mayor, 2008; Miyamoto, 2000) , using a membership function of the form µ A : U × [0, 1] → N, where µ A (x, α) is the number of appearances in A of an object x with membership α.

3. PROBLEM FORMULATION:

Our goal is to learn to represent and predict a notion of containment between multisets. We begin with a brief motivating example, and then move on to provide the formalization of the problem. Recall our example, where for A = {currency, equilibrium} and B = {money, balance, economics}, we have a sense in which A is "contained" in B. After seeing many such example pairs of multisets, we want to be able to deduce that for A = {currency, food} and B = {money, food}, the relation A ⊆ B holds.

3.1. MULTISETS:

In general, there exists a universe Ω of objects (in our above example, words). We let Ω * denote the set of all multisets of objects from Ω, formally defined as follows.

Definition 3.1.1 A multiset A is defined by a its membership function m A : Ω → M , where M ⊆ R + is a subset of the non-negative reals, and m A maps each object to the "number of times" it occurs in A.

The choice of M dictates the kind of multiset A is. In particular, M = {0, 1} gives classical sets, and M = N gives the traditional notion of multiset -a set which may contain multiple copies of the same object. If M = R + , then we call A a "fuzzy multiset." 2The cardinality (or size) of a multiset is defined with respect to a measure λ on Ω.

Definition 3.1.2 The cardinality of a multiset A is |A| = Ω m A (x)dλ(x).

We will always fix some measure λ on Ω (which may be called the dominating measure) and take all cardinalities with respect to λ. Note that we can always view m A as a density (i.e. the Radon-Nikodym derivative) of some measure µ A on Ω with respect to λ. We can thus identify a multiset A with the measure µ A on Ω, and write |A| = µ A (Ω).

In the case that the universe Ω is countable, we simply let λ be the counting measure, in which case the cardinality of any multiset A is |A| = x∈Ω m a (x).

All the usual operations on pairs of multisets are defined in terms of their membership functions.

Definition 3.1.3 For two multisets A and B, their intersection A ∩ B, union A ∪ B, sum A + B, difference A -B, and symmetric difference A B, are multisets given by the following memberships functions, respectively:

• m A∩B (x) = min{m A (x), m B (x)} • m A∪B (x) = max{m A (x), m B (x)} • m A+B (x) = m A (x) + m B (x) • m A\B (x) = max{m A (x)-m B (x), 0} • m A B (x) = |m A (x) -m B (x)|

These definitions are standard for multisets with both whole-number and real-valued memberships (Casasnovas and Mayor, 2008; Miyamoto, 2000; Blizard, 1989) . To those familiar with fuzzy set theory, it should immediately stand out the intersection and union are given by the standard T-norm and T-conorm (functions used to define these operations on fuzzy sets; see Appendix B). This means that our definition of fuzzy multiset contains a copy of fuzzy set theory. Unfortunately, there is no intuitive way to use other T-norms in order to define multiset operations. (For intuition on the above operations and why this is the case, see Appendix A.) Finally, for multisets, containment is formally defined as follows.

2 Note that this is not the same formulation of "fuzzy multisets" usually given in literature (Casasnovas and Mayor, 2008; Miyamoto, 2000) . However, this formulation will be much more easily amenable to the machine-learning setting. Our notion here is also more closely related to the "real-valued multisets" of Blizard (1989) , although the author approaches the subject from a standpoint of formal logic and axiomatic set theory. Definition 3.1.4 For two multisets A ∈ Ω * and B ∈ Ω * , we say that B contains A, or A ⊆ B, if and only if m A (x) ≤ m B (x) for all x ∈ Ω.

Note however that as demonstrated by our motivating example above, for two multisets A and B, we want to have a more "flexible" notion than A ⊆ B. (It is not actually the case that {currency, equilibrium} ⊆ {money, balance, economics}.) We will now formally provide a structure allowing for this flexibility.

3.2. A "FLEXIBLE" NOTION OF CONTAINMENT:

Let us first make two observations. Firstly, the desired flexibility will depend on some notion of "similairty" between the objects in Ω. Secondly, this similarity must be externally provided by our observations of these "subset" relations. In our example above, we had a sense that money ≈ currency and balance ≈ equilibrium, because we observed that A is "contained" in B (perhaps along with many other similar examples). We now formalize this idea. Definition 3.2.1 For two universes Ω and U, a map T : Ω * → U * is a called a multiset transformation from Ω to U.

The idea is that there exists some multiset transformation T from Ω to U, but we may not observe the structure of T or this new universe U. However, we indirectly observe this structure, because our notion of subsets will be taken in U * rather than in Ω * . In particular, in our example above, the sense in which

A is "contained" in B is that T (A) ⊆ T (B).

The simplest example of such a setting, on which we focus, is when U = {1, . . . , k} = [k]. That is, T maps each multiset in Ω * to a multiset of numbers from 1 to k. These numbers can be thought of as "tags," "classes," or "labels," meaning that each multiset has associated to it some tags, each of which may occur more than once. Say for our running example, T (A) = {1, 2} and T (B) = {1, 2, 3}. We then obtain that T (A) ⊆ T (B), as desired.

The example mapping T above is suggestive. It suggests a category of such T functions that are commonly useful: when each object in Ω is itself associated with a "tag" in U. Here, money and currency both are associated to tag 1, balance and equilibrium are both associated to tag 2, and economics to tag 3. Our map T is induced by this element-wise mapping. We formalize this notion as follows.

Definition 3.2.2 We call a function t : Ω → U a universe transformation. Each universe transformation t induces a pushforward multiset transformation T , where the membership function of

T (A) is m T (A) (y) = t -1 (y) m A (x)dλ(x).

Proposition 3.2.3 Every pushforward multiset transformation T preserves cardinalities; that is, for any A, we have |A| = |T (A)|.

Note that if we view A as the measure µ A on Ω, then the measure µ T (A) on U is the honest-togoodness pushforward measure µ A • t -1 . The above result follows easily.

Let's summarize where we are so far. We observe some relations between pairs of multisets over Ω. We also assume there is a multiset transformation T from Ω to some "latent" universe U, and that our observed relations are explained by relations that hold in U * . In general, we wish to understand the structure of T , in order to predict similar relations for unobserved pairs of multisets. We may also hope that in the process we learn something about the structure of Ω, and that in general this learning process is feasible due to U being much smaller or simpler than Ω. For example, we might assume that T is in fact a pushforward transformation induced by an unobserved labeling of the elements in

Ω by elements in U = [k]. 3

However, there is a problem with this setup: it is unlikely that for two multisets A ∈ Ω * and B ∈ Ω * , we have either T (A) ⊆ T (B) or T (B) ⊆ T (A). However, there are richer relations that can exist between two multisets than just containment, and which can be observed for any such pair. For example, regardless of whether either of T (A) or T (B) is a subset of the other, we can ask about how much they overlap -i.e. the size of either their intersection |T (A) ∩ T (B)|, or of their symmetric difference, |T (A) T (B)|. Note that if we also know their sizes |T (A)| and |T (B)|, then we can know whether one contains the other (Theorem 3.2.6). This follows directly from the following:

(1) that the size of the symmetric difference gives rise to a (pseudo-)metric on multisets, which can then be used to express a "disk embedding" inequality (Suzuki et al., 2019) relating containment to cardinalities (Proposition 3.2.4); and (2), the sizes of the symmetric difference and intersection are related via the sizes of the multisets themselves (Lemma 3.2.5). See Appendices C and D for proofs; note also that for each of the following three statements, we assume A and B are multisets over the same universe. We therefore see that we can use a training signal more readily available than binary yes-no containment -measurements of overlap between multisets -to still learn to predict containment relations. Thus, the problem we will be solving here is learning to predict either

|T (A) T (B)| or |T (A) ∩ T (B)| from examples.

Importantly, the error on these predictions will indicate how well we learned to capture our "flexible" notion of containment.

3.3. THE LEARNING TASK:

Formally, our learning task will therefore be as follows.

There exists a universe Ω, which we assume for practical purposes can be embedded in R d for some known d. There is also some latent universe U = [k] together with an unknown multiset transformation T : Ω * → U * . We will assume that T preserves cardinalities -in practice this means either that T is a pushfoward transformation induced by some t : Ω → U, or an "expectation transformation," which we define in Section 4.2. We then observe samples (A, B) from a training distribution D over pairs of multisets in Ω * . For practical reasons, our sampled multisets will wholenumber multiplicities. For each such pair, we also observe the overlap via either

|T (A) T (B)| or |T (A) ∩ T (B)|.

Which of these is used is fixed beforehand for the entire task, and we assume this choice is known. (We test both choices in our experiments.) Our assumption that T preserves cardinality is important, because together with these observations, it allows us to conclude whether T (A) is a subset of T (B). We then pick a hypothesis target universe Û = [ k], and we let k = k if k is known. (Experimentally, we examine the cases k < k and k = k.) Finally, our goal is to learn a model -i.e. a map T : Ω * → Û * -that minimizes squared error in the predicted overlaps. That is, we learn T in order to minimize the appropriate choice of the following two losses:

L = E A,B∼D T (A) T (B) -|T (A) T (B)| 2 L ∩ = E A,B∼D T (A) ∩ T (B) -|T (A) ∩ T (B)| 2 .

4. MODEL DEFINITION:

Having formulated our learning task, we now define our learnable model T : Ω * → Û * . In order to do so, we want our model to give us "representations" of the multisets in Û * in the most common machine-learning sense -i.e. vectors in some Euclidean space. We begin this section by defining how we obtain and use such representations, and then conclude by defining our model T itself that gives us these representations. In general, we want our representations of multisets to be "useful," in the sense that we can use them to perform common operations -such as those in Definition 3.1.3. More importantly for our task, we need to be able to calculate the size of either the symmetric difference or the intersection of two multisets. Our choice of target universe Û = [ k] gives us such a representation function.

Definition 4.1.2 Let Û be the finite universe [ k] . The natural representation function

Ψ k : Û * → R k is the map S → [m S (1), . . . , m S ( k)].

This should be an intuitive concept. For example, the natural representation function for classical sets gives the familiar indicator vector representation Ψ k(S) = [1 1∈S , . . . , 1 k∈S ]. Furthermore, we get "usefullness" of these representations for free, since all operations defined via membership functions (e.g. those in Definition 3.1.3) can be performed coordinate-wise. Furthermore, the cardinality of a multiset S ∈ Û * is given by thus sum of the entries in Ψ k(S). Together with the non-negativity of membership functions, this gives us the following. (As these two results are essentially immediate, we omit their proofs.) We thus use the natural representation function on Û to train our model. We note that Proposition 4.1.4 could provide a reason to prefer the size of the symmetric difference over the size of the intersection as the training signal. The reasoning is that 1 -distance has a gradient which depends on both the representations Ψ k(R) and Ψ k(S) in each coordinate (except at 0), while the coordinate-wise minimum can only depend on one of the representations in each coordinate. We test this idea in our experiments.

4.2. THE LEARNABLE MODEL:

Recall that the unobserved multiset transformation T : Ω * → U * preserves cardinalities. In particular, suppose for the purpose of exposition that T is the pushforward transformation induced by some labeling t : Ω → U. We both want our hypothesis class of models T to contain all such pushforward multiset transformations, and to potentially be restricted to those T which preserve cardinalities. Unfortunately, we cannot directly learn over the set of all pushforward transformations, as this is equivalent to learning the correct discrete labeling t : Ω → [k], which is both a hard and nondifferentiable problem. 4 Instead, we take a probabilistic approach.

Definition 4.2.1 For two universes Ω and Û, a probabilistic universe transformation is a map : Ω → ∆( Û) , where ∆( Û) is the space of probability measures on Û.

As we will see, probabilistic universe transformations to [ k] have the advantage of being smoothly parametrizable. In analogy to the pushforward multiset transformation induced by a t : Ω → U, we leverage our probabilistic transformation above to define a different kind of induced multiset transformation.

Definition 4.2.2 Let : Ω → ∆( Û) be a probabilistic universe transformation. The expectation multiset transformation L : Ω * → Û * is defined to be the map

A → E P ∼(µ A • -1 ) [P ].

We first note that L is well defined, in the sense that L(A) is always a valid measure on Û. This can easily be seen by re-writing the expression as follows:

E P ∼(µ A • -1 ) [P ] = ∆( Û ) P d µ A • -1 (P ) = Ω (x) dµ A (x).

L(A) has a natural interpretation, as the expected multiset obtained by sampling an element of Û for each x ∈ A according to its corresponding distribution (x) (with contribution weighted by m A (x)).

Additionally, we have the desirable property that L preserves cardinalities (see Appendix E for proof). We will thus let our learned model T : Ω * → Û * be an expectation transformation L induced by some probabilistic universe transformation . Note that we expect to be able to learn not only in the case where the unknown T is a pushforward transformation, but in fact when T itself is some expectation transformation (although we do not test the latter experimentally.)

4.3. PUTTING IT ALL TOGETHER:

The outstanding question is: how do we parametrize our representations Ψ k(L(A)) for any given A ∈ Ω * ? By the definition of the natural representation function Ψ k, we can write the i-th component of the representation of L(A) as

m L(A) (i) = Ω ( (x))({i}) dµ A (x) = Ω m (x) (i) dµ A (x) = Ω Ψ k( (x)) i dµ A (x),

where the last two equalities come from viewing (x) as a general measure and thus multiset on Û. Now, assuming that A is in fact one of the multisets sampled from our training distribution D, we know that A has whole-number multiplicities. (This will also be the case during evaluation, and thus in fact for any multiset we are trying to represent.) The above can then by simply written as Ψ k(L(A)) i = x∈A Ψ k( (x)) i , where each x occurs in the sum m A (x) times. More simply, we have

Ψ k(L(A)) = x∈A Ψ k( (x)).

Since each (x) is just a distribution over k elements, it suffices to learn a map from Ω to the probability simplex in R k -the non-negative vectors whose components sum to 1. Recalling that we assumed our input universe Ω consists of vectors in R d , we pick a favorite object-featurization network φ : R d → R k. We then guarantee than we obtain a point in the probability simplex by taking φ(a) to f (φ(a)) ||f (φ(a))||1 , where f : R → R + is a function applied component-wise. For differentiability, we choose the softplus function f (x) = log(1 + e x ). Our complete model is thus the representation

Ψ(A) = x∈A f (φ(a))

||f (φ(a))||1 . Our losses, in terms of these representations, are

L = E A,B∼D (||Ψ(A) -Ψ(B)|| 1 -|T (A) T (B)|) 2 L ∩ = E A,B∼D (min{Ψ(A), Ψ(B)} -|T (A) ∩ T (B)|) 2 .

5. EXPERIMENTS:

We begin here with an overview what we want to test about our model. In Section 5.1 we move on to describe our training and evaluation procedures. The experimental results themselves follow.

A clear question to seek the answer to empirically is whether the size of the symmetric difference or of the intersection works better in practice. (Recall that the symmetric difference may have more informative gradients, possibly leading to better learning and performance.) We thus compare these two approaches, both in terms of the error on the respective tasks themselves, and in terms of the error on predicting containment. More generally, the theory motivating our model suggest that there is a delicate balance in the properties that make the model well-posed. 5 We tackle this idea from two directions.

First, we ask how important is the precise definition of our model,

Ψ(A) = x∈A f (φ(a))

||f (φ(a))||1 . An obvious baseline to compare against is Ψ(A) = x∈A φ(x), which should help us answer the question of how important it is that each object is mapped to a point in the probability simplex. Looking at this formulation, an immediate connection one might make is to the DeepSets model of Zaheer et al. (2017) : Ψ(A) = ρ 1 ( x∈A φ(x)) for some learnable function ρ 1 . The authors prove this model can learn any permutation invariant function -e.g. the size of the intersection or symmetric difference of two multisets. We thus use both the models above as baselines in all our experiments, calling the former "unrestricted multisets" and the latter "DeepSets"

The second category of question we ask here is whether we gain anything from our construction of the multiset operations on representations. We tackle this question replacing the terms ||Ψ(A) -Ψ(B)|| 1 and min{Ψ(A)lΨ(B)} in our losses with ρ 2 (Ψ(A)+Ψ(B)) for a learnable function ρ 2 . The intuition here is that this new prediction is in fact a second DeepSets model trying to learn of our prediction functions -where we choose DeepSets because both intersection and symmetric difference are permutation invariant (i.e. commutative). This setting will be called the "learned operation" setting. We further test whether our parametrizations of the multisets operations are somehow intrisically good via a scheme of "cross-wiring" them -using one for a task where we should use the otheron which we elaborate in Section 5.4.

Finally, we will also examine the learned representations φ(x) of elements x ∈ Ω.

5.1. TRAINING AND EVALUATION PROCEDURES:

We use MNIST (LeCun, 1998) as our dataset. The training set consists of 60,000 handwritten images of digits, and the test set of 10,000.

We train all the models on 3×10 5 training pairs of multisets (A, B) ∈ Ω. At each iteration of training, both A and B are generated randomly, as follows. First a whole-number cardinality is uniformly sampled in some chosen range -in our experiments we use [2, 5] . (We exclude singleton sets to ensure that the models aren't just learning from comparing pairs of singletons.) Once the cardinality is chosen, then that number of images x ∈ Ω is then chosen uniformly at random (with replacement) from the training set. The multiset representation is calculated as usual, via one of the representation functions Ψ defined above, and the predicted cardinality of the symmetric difference or intersection is then calculated using these representations. The value to be predicted is calculated directly from the labels of the images in the multisets -e.g. if A is two images of ones, and B is a one and a three, the target value will be 1 for intersection, and 2 for symmetric difference. The squared error is minimized using Adam (Kingma and Ba, 2015) , with the default parameters β 1 = 0.9 and β 2 = 0.999, and a learning rate of 5 × 10 -5 . The learning rate was chosen by logarithmic grid search from 1 down to 5 × 10 -6 , training on up to 10 4 pairs during the search. (All models performed best with the chosen learning rate -or at least no worse than any of the other learning rates.) Given this learning rate, we chose to train the models for 3 × 10 5 iterations, finding that almost all of the models converged by this point.

Evaluation is performed similarly to training, with the addition of multiset sizes uniform on [2, 20], and with images sampled from the test set. Importantly, this means that the none of the images seen during training appear during evaluation. Each model is evaluated on 3 × 10 4 such multiset pairs, and unless otherwise stated we let φ : Ω → R k (that is, ĥ = k = 10).

For the object featurizing function φ, we use a variant of the LeNet-5 neural network (LeCun, 1998) . Specifically, we adopt the same architecture as used by Ilse et al. (Ilse et al., 2018) . (See Appendix F for network architectures, including those used for ρ 1 and ρ 2 .)

5.2. CARDINALITY PREDICTION:

We examine the performance of six kinds of models -multisets, unrestricted multisets, and DeepSets, each with or without learned multiset operations -on the tasks of predicting cardinality, either of the symmetric difference of the intersection. We will refer here to Tables 1 (symmetric difference) and 2 (intersection), which report the mean absolute errors of the predictions. Within each of the tables, two patterns are immediately clear. First, a amount portion of the prediction error may be explained by whether the multiset operations are learned, or taken to be the theoretically-motivated parametrizations; the models with learned operations exhibit more than twice the prediction error. While this shows there is a benefit to using our theoretically-motivated definitions, it does not necessarily mean that our definitions are intrinsically or uniquely well-suited for the task. We will revisit this point later.

The second salient pattern is that as we move away from the expectation-transformation model (which we simply call "multisets" in our tables), first to the unrestricted multiset model, and then to DeepSets, there is a rapid decrease in performance (in some cases almost ten-fold). This suggests that the theory behind our model is indeed useful.

Finally, when we compare across the two tables -that is, compare cardinality prediction for symmetric difference and for intersection -we observe a surprisingly large gap in error. The error on intersection size prediction is consistently about twice as small as on the other task. It is worth noting that if anything, we expected an opposite effect. This gap is intriguing, and we believe that it should be explored further.

5.3. CONTAINMENT PREDICTION:

We now compare the same models above on what is perhaps the more important task: prediction whether there exists a containment relation between T (A) and T (B) for some A and B. In particular, for any such pair, we predict whether

T (A) = T (B), T (A) T (B), T (B) T (A)

, or there is containment relation (i.e. we treat this as a classification problem). We perform this prediction by relying of Theorem 3.2.6 (and the assumptions on our representations). In particular, for any pair A and B, we predict the containment relation implied by Proposition 3.2.4, where we take the cardinality of the symmetric difference predicted by our model. (If the model predicts intersection, we just use Lemma 3.2.5 to go from one to the other.) Note that in this experiment we sample pairs A and B such that the probability of each kind of containment is essentially uniform. Referring to Tables 3 and 4 , we observe almost exactly the same patters as above -with over 96% accuracy achieved by both multiset models. The one difference is that for the unnormalized model and the DeepSets model (with non-learned operations), the version of the models trained on the intersection task perform noticeably worse than the corresponding models trained on symmetric difference. Furthermore, on all other models, the performances are comparable. This further complicates the picture from above, as it suggests that while intersection may somehow be easier to learn to predict the cardinality of, perhaps the task itself is a worse way to capture containment relations.

5.4. "CROSS-WIRING" OPERATIONS:

Motivated by our observation that models less closely aligned with our theory seems to perform worse, we devise a small test to see whether our symmetric difference and intersection cardinality operations are somehow intrinsic. To do so, we perform two experiments. First, we train our regular expectationtransformation-based multiset model to predict symmetric difference cardinality, but where it's prediction function is given by min{Ψ(A), Ψ(B)}. Similarly, in the second experiment, we train the model to predict intersection cardinality, but where the prediction function is ||Ψ(A) -Ψ(B)|| 1 . We observe that the former model achieves and mean-absolute error of 2.3710 on the test set, while the second achieves ones of 0.9929. There values are significantly higher than the errors achieved with the "correct" prediction functions, suggesting that there indeed a sense in which these are the "right" functions.

5.5. EXAMINING LEARNED REPRESENTATIONS:

We finally turn to examining the object-representations learned by our model. As one would expect, the learned representations of objects are approximate the standard basis vectors (as shown in Figure 1 for n = 3). This suggests our expectation-transformation model is learning appropriate point-mass probabilities corresponding to each object's label in U.

We also examine the case k < k, which may occur when we don't know the true size of U. Here, the "pinched" nature of the restricted representations may be undesirable (Figure 2a ). This problem, of course, gets worse with the discrepancy between number of objects and dimension (Figure 2b ). On the other hand, the unrestricted multiset model is able to learn more balanced-looking clusters. However, the clusters for d = n appear slightly less well-separated (Figures 3 and 4 ). The DeepSets model didn't learn interpretable representations (Figure 4c ). Furthermore, when we measure the accuracy of the regular multiset model on the containment prediction task above of each of the models, we obtain good results even when k is "too small." In particular, fixing k = 5: k = 5 we obtain an accuracy of 0.9586, and k = 3 gives 0.9045. This suggest that the representations learned are in fact robust to small discrepancies in dimension. 

6. CONCLUSION:

We propose a novel task: predicting the size of either the symmetric difference of the intersection between pairs of multisets. We motivate this construction via a measure-theoretic notion of "flexible containment." We demonstrate the utility of this idea, developing a theoretically-motivated model that given only the sizes of symmetric differences between pairs of multisets, learns representations of such multisets and their elements. These representations allow us to predict containment relations with extremely high accuracy. Our model learns to map each type of object to a standard basis vector, thus essentially performing semi-supervised clustering. One interesting area for future theoretical work is understanding a related problem: clustering n objects given multiset difference sizes. As a first step, we show in Appendix H that n -1 specific multiset comparisons are sufficient to recover the clusters. We would also be curious to see if one can learn the latent multiset space U.

(A \ B) ∪ (B \ A) -addition and union both work because (A \ B) and (B \ A) are necessarily disjoint. This gives us:

m A B (x) = max{m A (x) -m B (x), 0} + max{m B (x) -m A (x), 0} = |m A (x) -m B (x)|.

(The equation still holds if we replace the addition with a maximum.) B FUZZY SETS A fuzzy set A over a universe Ω is given by a function m A : Ω → [0, 1]. Intuitively, m A maps each x ∈ Ω to "how much of a member" x is of A, on a scale from 0 to 1. With this simple idea, fuzzy set operations can be defined. This is traditionally done by leveraging element-wise fuzzy logical operations, which we define below. • 1 is the identity: T (a, 1) = a T-norms generalize the notion of conjunction. Note that the above conditions imply that for any a, T (a, 0) = 0, and that T (1, 1) = 1. These two observations show that t-norms are "compatible" with classical, non-fuzzy logic -where we identify 0 with "false" and 1 with "true." The standard t-norm is T (a, b) = min{a, b}.

Definition B.0.2 A strong negator is a strictly monotonic, decreasing function n :

[0, 1] → [0, 1] such that n(0) = 1, n(1) = 0 and n(n(x)) = x.

Unsurprisingly, strong negators generalize logical negation. The standard strong negator is n(x) = 1 -x.

Definition B.0.3 An S-norm (also called a t-conorm) is a function with the same properties as a t-norm, except that the identity element is 0.

S-norms generalize disjunction. For every t-norm (and a given negator), we can define a complementary s-norm: S(a, b) = n(T (n(a), n(b))). This is a generalization of De Morgan's laws. The standard s-norm, complementary to the min t-norm, is S(a, b) = max{a, b}.

The membership function for the intersection of two fuzzy sets A and B is naturally defined as µ A∩B (x) = T (µ A (x), µ B (x)) for a t-norm T . Similarly, the complement of a fuzzy set is given by µ A (x) = n(µ A (x)) for a strong negator n, and the union of two fuzzy sets is given by µ A∪B (x) = S(µ A (x), µ B (x)) for an s-norm S. Usually, we want T and S to be complementary with respect to n. Then, we can generalize all the usual set operations to fuzzy sets by combining the three basic operations above.

C PROOF OF PROPOSITION 3.2.4 We show that for any two multisets A and B over the same universe Ω, A ⊆ B if and only if |A B| ≤ |B| -|A|. In fact, noting that it is always the case that |A B| ≥ |B| -|A| (which we will not prove but is easy to show), the following proof shows this holds with equality.

Proof. Let λ be the dominating measure with respect to which the cardinalities are taken.

We first show the forward direction. Suppose A ⊆ B, that is, for every x ∈ Ω, we have m A (x) ≤ m B (x). The result follows directly (with equality):

Ω m A B (x) dλ(x) = Ω |m B (x) -m A (x)| dλ(x) = Ω m B (x) dλ(x) - Ω m A (x) dλ(x).

For the converse direction, suppose on the other hand that |A B| ≤ |B| -|A|. Now suppose for the sake of contradiction that for some x * ∈ Ω, we have m

A (x * ) > m B (x * ). Then m B (x * )-m A (x * ) < |m B (x * ) -m A (x * )|. But this implies that |B| -|A| = Ω m B (x) -m A (x) dλ(x) < Ω m B (x) -m A (x) dλ(x) = |A B|,

which is a contradiction. D PROOF OF LEMMA 3.2.5

We show that for any two multisets A and B over the same universe Ω, |A B| = |A|+|B|-2|A∩B|.

Proof. Let λ be the dominating measure with respect to which the cardinalities are taken. Let Ω A ⊆ Ω be the elements x ∈ Ω on which m A (x) > m B (x), and similarly let Ω B ⊆ Ω be the elements x ∈ Ω on which m B (x) > m A (x). Finally let Ω 0 be those elements x ∈ Ω for which m A (x) = m B (x). Note that Ω A , Ω B , and Ω 0 are disjoint, and that their union is the entire universe Ω. We can then write the cardinality of the intersection of A and B as

Ω m A∩B (x) dλ(x) = Ω min{m A (x), m B (x)} dλ(x) = Ω A m B (x) dλ(x) + Ω B m A (x) dλ(x) + Ω0 m A (x) dλ(x). We then observe that |A| + |B| -2|A ∩ B| is Ω A m A (x) dλ(x) + Ω B m A (x) dλ(x) + Ω0 m A (x) dλ(x) + Ω A m B (x) dλ(x) + Ω B m B (x) dλ(x) + Ω0 m B (x) dλ(x) -2 Ω A m B (x) dλ(x) -2 Ω B m A (x) dλ(x) -2 Ω0 m A (x) dλ(x).

Simplifying, we obtain:

Ω A m A (x) -m B (x) dλ(x) + Ω B m B (x) -m A (x) dλ(x) = Ω A |m A (x) -m B (x)| dλ(x).

But this is exactly the cardinality of the symmetric difference |A B|.

E PROOF OF THEOREM 4.2.3:

We will show that for any probabilistic universe transformation : Ω → ∆(U), the induced expectation transformation L : Ω * → U * preserves cardinalities. That is, |A| = |L(A)|. (For easy of notation, we write U here rather than the Û used in statement of the result in the main text.)

Proof. Recall that since L(A) is a multiset and thus a measure, we have |L(A)| = (L(A))(U). Expanding L(A), we obtain

Ω ( (x)) dµ a (x) (U) = Ω ( (x))(U) dµ a (x) = Ω dµ a (x) = µ A (Ω) = |A|,

where the second equality holds because (x) is a probability measure over U.

F NETWORK ARCHITECTURES:

Given input images with c channels, and an output dimension d, the function φ is parametrized by the network: C i are disjoint in terms of represented clusters. We proceed recursively, each query "splitting" the sets in half (in terms of which clusters they contain). The number of such steps required is k -1 (which is the number of internal nodes in a balanced binary search tree for k objects). We'll call the resulting disjoint sets C1 , . . . , Ck (since we technically don't yet know they correspond to the true clusters).

For the second step, we must verify that the objects in each of our sets resulting from step one all belong to the same cluster. This can be done by ordering the objects within each set, and comparing each consecutive pair as singletons. For each of our sets Ci , we thus make | Ci | -1 such queries. Across all such sets, we thus make

k i=1 | Ci | -1 = n -k queries.

So, the total number of queries made is (k -1) + (n -k) = n -1.

:

We believe there is an interesting theoretical distinction worth noting here, which may help explain this observation. Namely, max-pooling is idempotent, meaning that repeatedly pooling a representation with itself does not change the result. On the other hand, summation does not have this property, and so repeated copies of an element are reflected in the result. In this way, DeepSets (with the sum rather than max-pool) is in fact modeling multisets rather than sets, which depending on the application may be undesirable.

It is worth noting that there do exist examples where each object in Ω does not obviously have an associated tag in[k], such as the RCV1 dataset ofLewis et al. (2004), which consists of documents -each of which can be thought of as a multiset of words -together with collections of categories for each document.

In fact, we only care about learning such a labeling up to permutation (as permutations of labels cannot affect containment relations), but this is still a hard problem.

In particular, the restriction of the object representation to the probability simplex is important for our expectation-transformation-based model to be a well-defined and cardinality-preserving map between multisets; furthermore, the guarantee that the learned representations Ψ(A) and Ψ(B) for any pair (A, B) preserve cardinalities is theoretically key both for our being able to use |A| and |B| (in addition to our observed symmetric differences or intersections) in order to deduce containment relations, as is the way in which we parametrize operations on multisets via operations on their representation. There are many possible directions of inquiry; we break apart the empirical question into two rough categories.

Figures and Tables:

Proposition 3.2.4 For any two multisets A and B, A ⊆ B if and only if |A B| ≤ |B| -|A|. Lemma 3.2.5 For any two multisets A and B, |A B| = |A| + |B| -2|A ∩ B|. Theorem 3.2.6 Given A and B (whose cardinalities we can calculate), it suffices to know either |A ∩ B| or |A B| to conclude whether A ⊆ B and whether B ⊆ A. Theorem 3.2.6 thus motivates the use of either |T (A) ∩ T (B)| or |T (A) T (B)| to learn about containment, in the case where T preserves cardinalities. In particular, we then know the cardinalities |T (A)| = |A| and |T (B)| = |B|, and thus either |T (A) ∩ T (B)| or |T (A) T (B)| is sufficient to deduce the containment relations between T (A) and T (B).

4.1 REPRESENTATION OF MULTISETSDefinition 4.1.1 For any universe Û, a d-dimensional representation function is a map Ψ : Û * → R d . For a multiset S ∈ Û * , we call Ψ(S) the representation of S.

Lemma 4.1.3 For any multiset S ∈ [ k] * , |S| = ||Ψ k(S)|| 1 . Proposition 4.1.4 For any two multisets R and S over [ k], we have |R S| = ||Ψ k(R) -Ψ k(S)|| 1 and |R ∩ S| = min{Ψ k(R), Ψ k(S)}, where the minimum is applied coordinate-wise.

Any expectation multiset transformation L : Ω * → Û * preserves cardinalities, i.e. for any A ∈ Ω * , |A| = |L(A)|.

Figure 1: Three-dimensional representations of test-set MNIST images generated by the restricted multiset model trained on multisets of sizes ∈ [2, 5]; the model is trained on images of zeros, ones, twos.

Definition B.0.1 A t-norm is a function T : [0, 1] 2 → [0, 1], satisfying the following properties: • Commutativity: T (a, b) = T (b, a) • Monotonicity: If a ≤ c and b ≤ d, then T (a, b) ≤ T (c, d) • Associativity: T (a, T (b, c)) = T (T (a, b), c)

(a) Model trained on zeros, ones, twos, and threes. (b) Model trained on zeros, ones, twos, threes, and fours.

Figure 2: Three-dimensional representations of test-set MNIST images generated by the restricted multiset model trained on multisets of sizes ∈ [2, 5]. Note that in (b) the representations of twos and threes are essentially inseparable.

(a) Model trained on zeros, ones, and twos. (b) Model trained on zeros, ones, twos, and threes. (c) Model trained on zeros, ones, twos, threes, and fours.

Figure 3: Three-dimensional representations of test-set MNIST images generated by the unrestricted multiset model trained on multisets of sizes ∈ [2, 5]. Note that in (c), the clusters essentially form a tetrahedron, with one of the vertices being the combination of twos and threes.

Figure 4: Two dimensional TSNE (van der Maaten and Hinton, 2008) "projections" of the tendimensional representations of test-set MNIST images generated by the models; the models were trained on multisets of sizes ∈ [2, 5].

Mean absolute errors in symmetric difference size prediction on MNIST sizes ∈ [2, 5] sizes ∈ [2, 10] sizes ∈[2, 20]

Table content: <table><tr><td>Multisets</td><td>0.0722</td><td>0.1264</td><td>0.2061</td></tr><tr><td>Multisets, learned op.</td><td>1.1748</td><td>1.7159</td><td>4.0468</td></tr><tr><td>Unrest. multisets</td><td>0.5614</td><td>0.7693</td><td>1.1813</td></tr><tr><td colspan="2">Unrest. multisets, learned op, 1.1402</td><td>1.7317</td><td>4.4426</td></tr><tr><td>DeepSets</td><td>0.6630</td><td>1.4097</td><td>4.0330</td></tr><tr><td>DeepSets, learned op.</td><td>1.2152</td><td>1.9386</td><td>5.2831</td></tr><tr><td colspan="4">Table 2: Mean absolute errors in intersection size prediction on MNIST</td></tr><tr><td/><td colspan="3">sizes ∈ [2, 5] sizes ∈ [2, 10] sizes ∈ [2, 20]</td></tr><tr><td>Multisets</td><td>0.0375</td><td>0.0648</td><td>0.1092</td></tr><tr><td>Multisets, learned op.</td><td>0.6021</td><td>0.8583</td><td>2.3487</td></tr><tr><td>Unrest. multisets</td><td>0.30644</td><td>0.4211</td><td>0.5708</td></tr><tr><td colspan="2">Unrest. multisets, learned op, 0.6122</td><td>0.8750</td><td>2.5015</td></tr><tr><td>DeepSets</td><td>0.3583</td><td>0.6763</td><td>2.4277</td></tr><tr><td>DeepSets, learned op.</td><td>0.5258</td><td>0.8420</td><td>2.6967</td></tr><tr><td colspan="4">1. A two-dimensional convolution layer with c input channels, 20 output channels, kernel size</td></tr><tr><td>5, and stride 1 (no padding)</td><td/><td/><td/></tr><tr><td>2. A ReLU activation</td><td/><td/><td/></tr><tr><td colspan="4">3. A two-dimensional max-pooling layer with kernel size 2 and stride 2</td></tr><tr><td colspan="4">4. A two-dimensional convolution layer with 20 input channels, 50 output channels, kernel</td></tr><tr><td>size 5, and stride 1 (no padding)</td><td/><td/><td/></tr><tr><td>5. A ReLU activation</td><td/><td/><td/></tr><tr><td colspan="4">6. A two-dimensional max-pooling layer with kernel size 2 and stride 2</td></tr><tr><td colspan="4">7. A fully-connected linear layer with output size d (the input size is determined by c)</td></tr><tr><td colspan="2">For the DeepSets model, we used for ρ 1 the architecture:</td><td/><td/></tr><tr><td colspan="4">1. A fully-connected linear layer with input size d and output size 100</td></tr><tr><td>2. A hyperbolic tangent activation</td><td/><td/><td/></tr><tr><td colspan="3">3. A fully-connected linear layer with input and output size 100</td><td/></tr><tr><td>For ρ 2 we used:</td><td/><td/><td/></tr><tr><td colspan="3">1. A fully-connected linear layer with input and output size 100</td><td/></tr><tr><td>2. A hyperbolic tangent activation</td><td/><td/><td/></tr><tr><td colspan="4">3. A fully-connected linear layer with input size 100 and output size 1</td></tr><tr><td>G ADDITIONAL FIGURES</td><td/><td/><td/></tr></table>

Accuracy on containment prediction for models trained on symmetric difference on MNIST sizes ∈[2, 5]

Table content: <table><tr><td>Multisets</td><td>0.9653</td></tr><tr><td>Multisets, learned op.</td><td>0.2576</td></tr><tr><td>Unrest. multisets</td><td>0.6332</td></tr><tr><td colspan="2">Unrest. multisets, learned op, 0.2679</td></tr><tr><td>DeepSets</td><td>0.6299</td></tr><tr><td>DeepSets, learned op.</td><td>0.2487</td></tr></table>

Accuracy on containment prediction for models trained on intersection on MNIST sizes ∈ [2, 5]

Table content: <table><tr><td>Multisets</td><td>0.9633</td></tr><tr><td>Multisets, learned op.</td><td>0.2494</td></tr><tr><td>Unrest. multisets</td><td>0.4916</td></tr><tr><td colspan="2">Unrest. multisets, learned op, 0.2819</td></tr><tr><td>DeepSets</td><td>0.5018</td></tr><tr><td>DeepSets, learned op.</td><td>0.3389</td></tr></table>

Peer review (iclr2020):
This paper presents a framework for learning representations of multisets. The approach is built on top of the DeepSets (Zaheer et al., 2017) model, which already is capable of learning representations of multisets, with a new objective of predicting the size of symmetric differences between two multisets, plus a small change in the implementation of the DeepSets model by restricting the per-element module to output a point in the probability simplex. I liked the background on fuzzy sets and the development of fuzzy multisets and different operations on these multisets. The definitions and arguments are quite clear and helpful. One small suggestion for page 4 is that I can understand why the formulation is the only sensible choice for multisets with desired properties, but a claim like this deserves a proper proof. Model-wise the paper made two contributions for learning representations for multisets as mentioned above: (1) proposed the symmetric difference prediction as a task for learning representations, the argument for this task is that predicting symmetric difference implicitly encourages the model to learn about containment; (2) a slight change in the DeepSets model architecture where the outer rho function is identity and the inner phi function has to output a point in the simplex. I found these technical contributions to be a bit small. In addition to this, the paper only presents results on MNIST in a toyish setting, this makes me feel the paper may be more suited for publication in a workshop (idea is interesting, small scale experiments to illustrate the insights, but not complete enough to be published at a conference). Regarding contribution (1), I can see why predicting symmetric difference makes sense as argued in the paper, but I’m not convinced that this is better than other alternatives. In order to show that this is a reasonable approach for learning representations, some results that compare this with other possible learning objectives would be necessary. But I don’t see any such results in this paper. Regarding contribution (2), I feel the restriction of the phi function to output points in simplex is not very well motivated and confusing in the first read. Again I can understand why we may want to do this but don’t see why we need to do this. I’m also concerned that such an architecture may only be good for the task of predicting symmetric difference as it is customized for this task. Figure 3 shows that an unrestricted model seems to learn better representations despite a worse symmetric difference prediction error, which again confirms the concern. Another thing about the experiment setup: the second baseline, labeled “DeepSets” in Table 1 actually changed two things compared to the proposed approach: (1) changing the psi function and (2) also changed the symmetric difference function. It would be good to isolate the contribution of the two. Overall I feel this paper is not yet ready to be published at ICLR.