Title: No title found

Abstract: Good representations facilitate transfer learning and few-shot learning. Motivated by theories of language and communication that explain why communities with large number of speakers have, on average, simpler languages with more regularity, we cast the representation learning problem in terms of learning to communicate. Our starting point sees traditional autoencoders as a single encoder with a fixed decoder partner that must learn to communicate. Generalizing from there, we introduce community-based autoencoders in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations. Our experiments show that increasing community sizes reduce idiosyncrasies in the learned codes, resulting in more invariant representations with increased reusability and structure.

1. INTRODUCTION:

The importance of representation learning lies in two dimensions. First and foremost, representation learning is a crucial building block of a neural model being trained to perform well on a particular task, i.e., representation learning that induces the "right" manifold structure can lead to models that generalize better, and even extrapolate. Another property of representation learning, and arguably the most important one, is that it can facilitate transfer of knowledge across different tasks , essential for transfer learning and few-shot learning among others (Bengio et al., 2013) . With this second point in mind, we can define good representations as the ones that are reusable, induce the abstractions that capture the "right" type of invariances and can allow for generalizing very quickly to a new task. Significant efforts have been made to learn representations with these properties; one frequently explored direction involves trying to learn disentangled representations (Schmidhuber, 1992; Kingma & Welling, 2013; Higgins et al., 2016; van den Oord et al., 2017) ), while others focus on general regularization methods (Srivastava et al., 2014; Vincent et al., 2010) . In this work, we take a different approach to representation learning, inspired by successful abstraction mechanisms found in nature, to wit human language and communication.

Human languages and their properties are greatly affected by the size of their linguistic community (Reali et al., 2018; Wray & Grace, 2007; Trudgill, 2011; Lupyan & Dale, 2010) . Small linguistic communities of speakers tend to develop more structurally complex languages, while larger communities give rise to simpler languages (Dryer & Haspelmath, 2013) . Moreover, we even observe structural simplification as the effective number of speakers grows, as in the example of English language (McWhorter, 2002) . A similar relation between number of speakers and linguistic complexity can also be observed during linguistic communication. Speakers, aiming at maximizing communication effectiveness, adapt and shape their conceptualizations to account for the needs of their specific partners, a phenomenon often termed in dialogue research as partner specificity (Brennan & Hanna, 2009) . As such, speakers form conceptual pacts with their listeners (Brennan & Clark, 1996) , and in some extreme cases, these pacts are so ad-hoc and idiosyncratic that overhearers cannot follow the discussion (Schober & Clark, 1989 )! But how are all these linguistic situations related to representation learning? We start by drawing an analogy between language and representations induced by the traditional and extensively used framework of autonencoders (AE). In the traditional AE set-up, there is a fixed pair of a single encoder and a single decoder that are trained to maximize a reconstruction loss. However, encoders and decoders co-adapt to one another, yielding idiosyncratic representations. The encoders spend repre-sentational capacity modeling any kind of information about the data that could allow the decoder to successfully reconstruct the input; as long as the encoder and the decoder agree on a representation protocol, this information need not be abstract or systematic. This has a negative impact on the reusability of the representations, something that afterall is a key objective of representation learning. Evidence of this co-adaption is found in the above-mentioned efforts targeting generalization. The human language analogy of the traditional AE setup would be an extreme version of the conceptual pact experiments from Schober & Clark (1989) , where two people never communicate with anybody else: the resulting language would be very hard to understand for any outsider.

In this work we test whether removing this co-adaptation between encoders and decoders can yield better generalization, much as dropout removes co-adaptation between activations and thereby yields better generalization in general neural networks. We hypothesize that machines that communicate not with a specific partner but with a multitude of partners, will shape the representations they communicate to be simpler in nature. We introduce a simple framework that we term communitybased autoencoders (CbAEs), in which there exist multiple encoders and decoders, and at every training iteration one of each is randomly sampled to perform a traditional autoencoder (AE) training step. Given that the identity of the decoder is not revealed to the encoder during the encoding of the input, the induced representation should be such that all decoders can use it to successfully reconstruct the input. A similar argument holds for the decoder, which at reconstruction time does not have access to the identity of the encoder. We conjecture that this process will reduce the level of idiosyncrasy, resulting in representations that are invariant to the diverse encoders and decoders.

We apply CbAEs to two standard computer vision datasets and probe their representations along two axes; their reusability and their structural properties. We find that in contrast to representations induced within a traditional AE framework 1) the CbAE-induced representations encode abstract information that is more easily extracted and re-used for a different task 2) CbAE representations provide an interface that is easier to learn for new users 3) and the underlying topology of the CbAE representations is more aligned to human perceptual data that are disentangled and structured.

2. COMMUNITY-BASED AUTOENCODERS:

Background One of the simplest and most widely used ways to do representation learning is to train an autoencoder, i.e., encode the input x, usually in a lower-dimensional representation, z = e(x, θ) using some parameters θ, then use the z representation to decode back the input x = d(z, φ) through another set of parameters φ. θ and φ are trained by minimizing a reconstruction loss, e.g.,:

L(x, x ) = ||x -x || 2 = ||x -d(e(x, θ), φ)|| 2

(1)

The resulting latent vector z is then treated as the induced representation of the input data, and is often re-used for other problems, such as supervised learning or reinforcement learning. Because this approach is very general and can be applied to any data set, it holds the promise of being able to leverage existing unlabelled data, in order to then quickly solve other problems, using much less data and/or computation. However, the loss in Eq. 1 has an important flaw: it does not directly incentivize the formation of latents that have all the properties of good representations, such as appropriate abstraction and reusability. As a result, significant amounts of research effort have been dedicated to finding a better loss (Vincent et al. (2010 ), Kingma et al. (2014) , inter alia).

Our method The CbAE framework (see Figure 1 ) is inspired by the hypothesis that the size of a linguistic community has a causal effect on the structural properties of its language. Unlike the traditional autoencoder framework, which uses a single encoder paired with a single decoder, the CbAE set-up involves a community of K enc encoders and K dec decoders. 1 As such, we are not dealing with a single autoencoder, but rather a collection of K enc × K dec autoencoders. No single encoder and decoder are associated with one another, but rather the community of encoders are associated with the community of decoders and all combinations may be used together. Importantly, while the network architectures can be (and in fact in this work are) identical across a community (i.e., all encoders and decoders have the same number and organization of units and weights) there is no weight-sharing among members of the community. Training procedure At each training step, given a data point x, we form an autoencoder by randomly sampling an encoder and a decoder from the respective communities. Then, we perform a traditional autoencoding step where we minimize the mean-squared (L 2 ) loss between the input x and its decoding (see Eq. 1 and Algorithm 1). Trivially, the traditional autoencoder training protocol can be recovered by setting

K enc = K dec = 1. Algorithm 1 Community-based autoencoders initialize encoders E = {e 0 , ..., e Kenc } initialize decoders D = {e 0 , ..., e K dec } for each iteration i do sample input data x i sample encoder e i from E z i ← e i (x i ) sample decoder d i from D x i ← d i (z i ) L i ← L(x i , x i )

see Eq. 1 optimize e i and d i with respect to L i end for There are two main reasons why we think this will have a positive effect on the quality of the representations. First, given that the chosen encoder e i for iteration i does not have a priori information about the identity of the chosen decoder d i , and given that there are a number of decoders all with different weights, the encoder should produce a latent z i that is potentially decodable by all different decoders. Similarly, given that each decoder d i receives over its training lifetime latents from a number of different encoders, the decoder should learn to decode representations produced by all encoders. We hypothesize that this training regime will produce latents that are less prone to have idiosyncrasies rooted in the co-adaptation between a particular pair of encoder and decoder.

Relation to dropout The CbAE setup is reminiscent of dropout (Srivastava et al., 2014) : The entire community can be viewed as one much larger and highly parallel model, from which at each iteration a selection of weights (corresponding to one specific community member) is chosen. However, a crucial difference is that here, the choice of weights happens in a very correlated way; it is not a random set of weights, but one of K enc or K dec non-overlapping subsets that is selected at each training step. As a consequence, the weights in one community member (e.g. an encoder) will be much slower to adjust (if at all) to those in the rest of the community, and a higher degree of diversity is maintained. There is some mutual adjustment, of course, but it is a second-order effect: encoder e i and encoder e j will only get information about each other's encoding through the gradients of decoders that have learned to decode them.

The curse of co-adaptation:

The goal of our method is to avoid co-adaptation between the encoder and decoder. However, due to their flexibility, neural networks are in principle capable of co-adapting to several partner modules at once. As a consequence, the encoders can avoid convergence and still learn to produce latents from which the decoders can successfully reconstruct the input by capitalizing on encoder-specific information. Intuitively, we can think of this as the encoder essentially "signing" the latents with their unique ID. We test whether this indeed manifests in the setup by training a linear classifier whose task is to identify the encoder from the latent representation: p e (z) = exp(w T e z)/ e exp(w T e z). 1 shows, the encoder classifiers perform significantly better than chance in spite of having to keep up with shifting representations, indicating that pairwise co-adaptation does indeed happen to some extent for all community sizes. The nonmonotonous behaviour seen on the row labelled without entropy loss is due to the two competing effects. As the community size grows, the encoder identification task becomes harder (hence the lower chance), and the error rate naturally increases. However, larger communities also lead to slower rates of representation shift for every individual encoder, making it easier for the encoder classifier to keep up with their changing representation.

Community size:

A similar phenomenon of co-adaptation is often encountered in domain-adaption neural frameworks. To alleviate this, adversarial losses or gradient reversal layers (Ganin et al., 2016) are introduced to penalize representations from retaining domain-specific information. Here, in order to counteract the all-to-all pairwise co-adaptation effect, we add a simple adversarial loss forcing the encoders to be indistinguishable for the encoder classifier while keeping the encoder classifier itself fixed. In particular, the extra loss term is the negative entropy of the classifier, L entropy (z) = e p e (z) log p e (z).

Training of CbAE:

We use MNIST and CIFAR-100, with community sizes of 1, 2, 4, 8 and 16. The batch size is fixed at 128 throughout all experiments. We use the Adam optimizer with a learning rate of 10 -4 . The encoders are straightforward convolutional neural networks of VGG-flavour, with depths of 6 (MNIST) and 10 (CIFAR-100) layers respectively. For the details we refer the reader to the Appendix. The decoders implement the corresponding transpose convolutions. Having to respond to more communication partners makes the job of the individual encoders and decoders harder. This effect is seen in the reconstruction loss (see Figure 2 , where an increase in community size leads to a penalty in the reconstruction error even when correcting for the amount of training data seen by each community member. Note that this is not necessarily limiting when considering the desired properties of the representation, since the pixel-loss is merely a self-supervision signal: some pixel-level information is lost, but ultimately pixel-level information is not the true goal of the representation learning exercise, as discussed in Section 1. The interesting question is however: given the capacity of the latents, are we trading-off reconstruction performance for other more relevant properties such us reusability or structure? The experiments presented in section 3 aim at answering precisely this question.

3. ASSESSING THE QUALITY OF REPRESENTATIONS:

In the previous section we introduced the CbAE set-up and found that the reconstruction loss increases as the community size grows. However, the reconstruction loss in this setup is just a learning signal for representation learning, rather than the end goal. Ultimately, we are interested in good representations that could allow for generalization, knowledge transfer and reusability. In this section, having trained the CbAEs, we devise a number of parametric (Section 3.1) and non-parametric (Section 3.2) evaluation methods that probe the representations for exactly these properties.

3.1. PARAMETRIC PROBE TASKS:



3.1.1. TASKS AND METRIC:

Training new encoders and decoders Human languages have the property that the more regular and systematic they are, the easier they are for learners to acquire. We examine whether the latent interface found by the CbAE setup has the same property, i.e., is easier to learn for new users. To do so, we train newly initialized encoders and decoders. The hypothesis is that if the CbAE-trained encoders and decoders have learned to encode information in the representation in a systematic way, rather than in an ad-hoc and idiosyncratic way, this would result in the new, untrained, encoders and decoders being able to learn the representation with less effort, which we define operationally as better sample complexity (see below). This evaluation task is illustrated in the two leftmost panels of Figure 3 . Experimental setup In these probe tasks, only the newly initialized evaluation modules (new encoders, new decoders, and linear classifiers) are trained, and the encoders and decoders trained in the CbAE setup are frozen. The tasks share the same basic set-up: all frozen members of the community of encoders or decoders are coupled individually to an evaluation module, which is trained to perform best-response to their pre-trained partner.

For the new encoders and decoders, we use the same architecture as for the CbAE-trained ones. We use the Adam optimizer with a learning rate of 10 -4 . For the linear classifiers, we fit a linear layer, followed by a softmax, on the latents of each CbAE-trained encoder. We use the Adam optimizer with a learning rate of 10 -3 and optimize the cross-entropy between the predicted label ŷ and the actual label y. We use a minibatch size of 128 throughout all experiments.

Sample complexity gain For every parametric probe task, we first record the average performance achieved by all modules trained with a given community after a given number of training iterations. We then obtain the number of training iterations needed for the traditional AE (i.e., community size 1) to reach the same performance, and compute the ratio of these two training durations as the sample complexity gain.

The above formulation takes the following form in more mathematical notation: Given a learning curve L(i) which maps an iteration i to an obtained result L(i), we define the inverse learning curve:

L inv (L ) = min s.t.L(i)≤L i (2)

The inverse learning curve returns the first iteration at which the result dropped below the argument L . Equipped with this function, the sample complexity gain of curve L at iteration i relative to curve L baseline is straightforward to compute:

SCG(L, i, L baseline ) = i L inv baseline (L(i)) -1 (3)

If the two curves L and L baseline are identical, L inv baseline (L(i)) = i, and SCG(L, i, L baseline ) = 0 as expected. A negative sample complexity gain indicates that L reaches the value L(i) at a later iteration than L baseline , i.e. L inv baseline (L(i)) < i.

3.1.2. PARAMETRIC PROBE RESULTS:

Transferring representations to a new task After training the image classifier, we evaluate its performance on the test set. Figure 4 shows the sample complexity gain relative to the traditional AE case Overall, we find that classifiers trained on the latents learned by larger communities learn faster. For example, the leftmost bar on the MNIST plot shows that a classifier trained on the latents from community of size 2 needs 8 iterations to reach a performance that takes almost twice (1.6 times) as many iterations for a classifier trained on the latents from a classical AE. Moreover, the MNIST plot clearly shows that larger communities lead to faster classifier learning. As the classifier training progresses, the gains relative to the classical AE become smaller; this suggests that there is might still be some co-adaption, which however is significantly delayed by the introduction of the community. This positive effect of the community is also clear since the largest community is still speeding up relative to the classical AE after 32 iterations. In the case of CIFAR-100, we observe similar sample complexity gains, but the community size effect appears to be reversed. We attribute this to the larger model used (and needed) for this data set, which presents a significantly more complex task both for autoencoders and for classifiers. This larger model in turn requires more CbAE iterations for the communities to learn to represent the data at all, with each community member only seeing 1/K of the CbAE iterations (K being the community size). Figure 6 shows the results for training new encoders on MNIST and CIFAR-100. The new encoders learn faster with decoders trained in larger communities, and display roughly the same sample complexity gain pattern as the new decoders, albeit with slightly better absolute numbers. To understand this, we note that there is an asymmetry between encoders and decoders, in that the decoders can learn to decode a large hypervolume in latent space into roughly the same image, encompassing the encodings of all individual encoders. A new encoder then only has to learn to encode an image somewhere into that hypervolume to get a reasonable reconstruction error. A new decoder, however, has to learn best-response to a specific encoder, which essentially involves more adaptation to residual idiosyncrasies and can therefore be a more difficult task. Comparison to other regularization mechanisms We have performed the same probe analyses presented in sections 3.1.1 and 3.1.2 on the representations learned by a traditional AE setup enhanced with (neuron-level) dropout, and found no gains relative to the dropout-free setting. Exploring variations on dropout that interpolate between large expected overlaps between subsets (the traditional implementation) and fully mutually exclusive subsets (our method) is an interesting direction of future work. Moreover, all our models are trained with batch normalization, indicating that the gains we find are orthogonal to the specific regularization advantages it provides.

3.2. NON-PARAMETRIC PROBE TASK: LATENT SPACE STRUCTURAL ANALYSIS:

Finally, we ask the question of to what degree the speaker-invariance bias imposed by the CbAE framework induces abstract representations that share the same underlying structure with human perceptual data.2 As a proxy of human perceptual data, we use the Visual Attributes for Concepts Dataset (VisA) of Silberer et al. (2013) , which contains human-generated per-concept attribute annotations for concrete concepts (e.g., cat, chair, cat) spanning across different categories (e.g., mammals, furniture, vehicles), annotated with general visual attributes (e.g., has whiskers, has seat). Table 2 presents some examples of the conceptual representations found in VisA. As we can see, concepts representations are structured and disentangled. Therefore, achieving high similarity with these would indicate that the CbAE-induced representations encode similar conceptual abstract information. Most importantly, this is an independent task and requires no additional training of parameters.

Representation Similarity Analysis For measuring the similarities between the human perceptual data and the CbAE-induced representations, we perform Representational Similarity Analysis (RSA) in the two topologies, a method popular in neuroscience (Kriegeskorte et al., 2008) . For each community configuration, we sample 5,000 images and encode them with all encoders. Following that, for each encoder-specific set of latents, we apply concept-based late fusion, meaning that we average in a single latent all latents belonging to the same concept, to arrive to 68 concept-based representations. We then compute two sets of pairwise similarities of the 68 concepts, i.e., one set using their concept-based CbAE-induced latent representations and one set the concept-based VisA attribute representations. With these two lists of cosine similarities in hand, the RSA between the two topologies is taken as the Spearman correlation of these two lists of similarities. Given that RSA is a second-order similarity, we are not asking the question of how similar (say in terms of cosine) the two spaces are, but rather how similar their topology is, i.e., whether points that are nearby in the latent space are also nearby in the VisA space.

Results Table 3 summarizes our results. For each community configuration we report the mean RSA performance (obtained by averaging the RSA scores produced by the different encoders) and the maximum performance. To account for the potential confounder caused by the different initializations in the CbAE, we compare the results with the best result found from same number of independent AE. We observe that the mean similarity increases with the size of the population, confirming the hypothesis that CbAE produce on average abstract representations that to some degree reflect the topology of the highly structured and disentangled human data. Moreover, we observe even higher gains when looking at the best RSA value within a CbAE; training an encoder within a community of diverse partners can lead to more abstract and structured representations than training a diverse set of independent encoders each with a fixed decoder partner. This result rejects an alternative hypothesis; the gains cannot by explained just by increasing the diversity of the initializations, it is the community training of these diverse models that leads to increases performance.

Finally, while the largest CbAE (i.e., community size 16) has higher RSA similarity than the baseline, it shows the smallest gains, a pattern consistent with the the rest of the parametric probe results of CIFAR-100. We attribute this to the fact that this community had the smallest number of iterations per member, and had therefore not had the opportunity to learn to represent the data well yet. 

4. DISCUSSION:

We have presented Community-based AutoEncoders, a framework in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations, encouraging a similar lack of co-adaptation that dropout does at the activation level, at model level. Analogous to the structural simplicity found in languages with many speakers, we find that the latent representations induced in this scheme are easier to use and more structured. This result is philosophically interesting in that it suggests that the community size effects found in human languages are general properties of any representation learning system, opening avenues to potential synergies between representation learning linguistics.

The price for obtaining these representations is the increase in computational requirements, which is linear in the community size. Due to the reusability of the resulting representations, this cost may be amortized over a number of applications trained on top of the encoders. Furthermore, the community-based training procedure is highly parallelizable, since only the latents and corresponding backpropagated errors need to be sent between the encoders and decoders.

:

For simplicity, in our experiments, we use Kenc = K dec .

We note that this experiment is only conducted for CIFAR-100 as this dataset contains categories for common nouns (e.g., cat, dog, chair) for which we can meaningfully probe humans for perceptual similarity (as opposed to the numerical categories found in MNIST).

Figures and Tables:

Figure 1: Left: for each iteration, a randomly selected encoder-decoder pair is used. Right: in expectation, all encoders are trained with all decoders, and vice versa.

Figure 2: The reconstruction error per pixel on MNIST (left) and CIFAR-100 (right).

Transferring representations to a new task Next, we investigate the transfer capabilities of the representations to a different task; we freeze the CbAE encoders, perform supervised learning on image classification by training linear classifiers and evaluate their sample complexity. The hypothesis is that the CbAE framework induces abstract representations of the input data that would allow a simple linear classifier to achieve a better sample complexity.

Figure 3: Set-up for assessing the quality of the representations. Purple boxes represent trainable evaluation modules; blue boxes represent trained encoders and decoders that are kept fixed during the evaluation training. Left: linear classifiers; centre: new encoders; right: new decoders.

Figure 4: The sample complexity gain relative to the traditional AE setup when training a linear classifier, on MNIST (left) and CIFAR-100 (right).

Figure 5: The sample complexity gain relative to the traditional AE setup when training new decoders, on MNIST (left) and CIFAR-100 (right), while training new encoders.

Figure 6: The sample complexity gain relative to the traditional AE setup when training new encoders, on MNIST (left) and CIFAR-100 (right), while training new encoders.

Encoder identification error rate on MNIST.

Table content: <table><tr><td/><td>2</td><td>4</td><td>8</td><td>16</td></tr><tr><td>Chance</td><td>0.5</td><td colspan="3">0.75 0.875 0.938</td></tr><tr><td>No entropy loss</td><td>0.3</td><td>0.48</td><td>0.47</td><td>0.28</td></tr><tr><td colspan="5">With entropy loss 0.52 0.764 0.875 0.937</td></tr></table>

Examples of conceptual representations in the VisA dataset.

Table content: <table><tr><td colspan="3">Community size mean ρ max ρ</td></tr><tr><td>1</td><td>0.341</td><td>0.314</td></tr><tr><td>2</td><td>0.355</td><td>0.382</td></tr><tr><td>4</td><td>0.372</td><td>0.389</td></tr><tr><td>8</td><td>0.401</td><td>0.423</td></tr><tr><td>16</td><td>0.352</td><td>0.369</td></tr></table>

Perceptual similarity between CbAE-induced and VisA representations.

Table content: <table/>

Peer review (iclr2019):
 It is very refreshing to see a submission that investigates the fundamental question of representation: what is considered as a good representation. This works argues that is it the representation learned through a community of autoencoders. The idea is very well motivated, taking inspiration from linguistic development. The idea is quite intuitive to understand, and makes sense. The narrative is fluent, with some accidental syntax errors. I believe this line of thinking is original to some extent, although bears some similarity with ideas in Dropout, DropConnect, however, motivated differently. It is, as pointed out by the authors, indeed a generalization of the co-adaption idea. Here is a list of questions that I have in mind: 1. The use of the cross-entropy regularization deserves more attention. It is not clear how important this regularization is experimentally. Is there a coefficient to control its contribution? 2. Experiments on MNIST and CIFAR require some more clarification. First of all, the use of SCG as a metric is OK, but experiments seem to only focus on early iteration comparisons. I would like to see, for example, the entire classification error (or reconstruction error) curve on the test set, epoch by epoch, with standard AEs and your proposed CbAEs side by side. 3. Secondly, it is not clear from the text of how your test error is obtained from the community as it contains many models. It is important to factor out the effect of model averaging in comparison with standard AEs (or a community of size 1). 4. Lastly, the negative effects of increasing community size is not clearly explained. I would like to gain more explanation of its source, especially on CIFAR100. Overall, I think the premise of this work could be more substantiated by some semi-supervised (for transfer learning) tasks with a more realistic setup and public benchmarks.