SlowGuess/ACL_2020_OCR huggingface dataset:
We demonstrated in the main paper that agent's generalization correlates with input size. In fact, agents can successfully reconstruct new attribute combinations if trained on large input spaces. This could be due to agents overfitting when presented with few training samples. To test this hypothesis, we repeat the training/evaluation experiments with GRU agents of different capacities in the following settings:  $(i_{att} = 2, i_{val} = 10)$ , a small input space where agents do not generalize; and  $(i_{att} = 2, i_{val} = 100)$ , a large input space where agents generalize.7 Fig. 3 shows that, even for small-capacity agents (one-layer GRU with hidden state of size 100), test accuracy is 0 for  $(i_{att} = 2, i_{val} = 10)$ . Moreover, agents do not overfit when trained on  $(i_{att} = 2, i_{val} = 100)$  even with two-layer GRUs with hidden state of size 500.
8.4 Input space density
We showed in the main paper that generalization positively correlates with  $|I|$ . We further investigate here whether it is simply the increasing abso
${}^{6}$  Only Lang3 uses the whole available  ${c}_{voc}$
We only report experiments with GRUs, but the same results were replicated with differently-sized LSTMs.
\( c_{voc} \)\( (i_{val}, i_{att}) \)	5	10	50	100
2	3	4	{6,8}	2	3	4	{6,8}	2	3	4	{6,8}	2	3	4	{6,8}
(4,4)	X	X	X	X	X	X	X	X	X
(5,2)	X	X	X	X	X	X	X	X	X	X	X	X
(5,3)	X	X	X	X	X	X	X	X	X	X
(5,4)	X	X	X	X	X	X	X	X	X
(10,2)	-	X	X	X	X	X	X	X	X	X	X
(10,3)	X	-	X	X	X	X	X	X	X	X
(10,4)	{-, X}	-	X	X	X	X	-	X	X	X
(16,2)	-	X	X	X	X	X	X	X	X
(25,2)	-	X	-	X	X	X	X	X	X
(50,2)	X	-	X	-	X	X	X	X	X	X	X
(100,2)	{-, X}	-	X	X	X	X	-	X	X	X
Table 3: Grid search. 'X' indicates tested settings with at least one successful run. '-' indicates tested settings without any successful run. Finally, blank cells correspond to settings that were not explored for the reasons indicated in the text.
Input	Lang1	Lang2	Lang3
0,0	0,0,0	0,0,0	0,0,4
0,1	0,0,1	0,0,1	0,0,5
0,2	0,0,2	0,0,2	0,0,6
0,3	0,0,3	0,0,3	0,0,7
1,0	0,1,0	1,2,0	1,4,1
1,1	0,1,1	1,2,1	1,5,1
1,2	0,1,2	1,2,2	1,6,1
1,3	0,1,3	1,2,3	1,7,1
2,0	2,0,0	2,3,0	2,4,2
2,1	2,0,1	2,3,1	2,5,2
2,2	2,0,2	2,3,2	2,6,2
2,3	2,0,3	2,3,3	2,7,2
3,0	2,1,0	3,1,0	3,4,3
3,1	2,1,1	3,1,1	3,3,5
3,2	2,1,2	3,2,1	3,3,6
3,3	2,1,3	3,3,1	3,3,7
topsim	0.82	0.75	0.75
posdis	1	0.79	0.43
bosdis	0.42	0.13	1
Table 4: Input-message mappings and compositionality measures for the miniature languages.
Figure 3: Average accuracy on unseen combinations as a function of agents capacity ((hidden size, number of layers)) for input sizes  $(i_{att} = 2, i_{val} = 10)$  and  $(i_{att} = 2, i_{val} = 100)$ . Vertical bars represent SEM.
lute number of distinct training samples that is at the root of this phenomenon, or whether the variety of seen inputs also plays a role, independently of absolute input size.
To verify this, we design an experiment where we keep the absolute number of distinct input samples constant, but we change their density, defined as the proportion of sampled items over the size of the space they are sampled from. When sampling points from a small space, on average each value of an attribute will occur with a larger range of values from other attributes, compared to a larger space, which might provide more evidence about the combinatorial nature of the underlying space.
In practice, we fix  $(c_{len} = 3, c_{voc} = 100, i_{att} = 2)$  and sample 10000 points from spaces with  $i_{val} = 100$  (density  $= 1$ ),  $i_{val} = 140$  (density  $= 0.51$ ) and  $i_{val} = 200$  (density  $= 0.25$ ), respectively. As usual, we use  $90\%$  of the data for training,  $10\%$  for testing. In all cases, we make sure that all values are seen at least once during training (as visually illustrated in Fig. 4).
We obtain test accuracies of  $92.7\%$ ,  $66.7\%$  and  $22.8\%$  for densities 1, 0.51 and 0.25 respectively. That is, the high generalization observed in the main paper is (also) a consequence of density, and hence combinatorial variety, of the inputs the agents are trained on, and not (only) of the number of training examples.
8.5 Impact of channel capacity on generalization
We showed in the main paper that generalization is very sensitive to input size. In this section, we focus on the relation between channel capacity  $|C|$  and generalization.
First, when we aggregate across input sizes, Fig. 5 shows that  $|C|$  has a just small effect on generalization, with a low Spearman correlation  $\rho = 0.14$ . Next, if we study this relation for specific large  $|I|$  (where we observe generalization), we notice in Fig. 6 that agents need to be endowed with a  $|C|$  above a certain threshold, with  $\frac{|C|}{|I|} > 1$ , in order to achieve almost perfect generalization. Moreover, contradicting previous claims (e.g., Kottur et al., 2017), having  $|C| >> |I|$  does not harm generalization.
8.6 Impact of channel capacity on the compositionality measures
A good compositionality measure should describe the structure of the language independently of the used channel, so the corresponding score should not be greatly affected by  $|C|$ . However, Fig. 7 shows clear negative correlations of both topsim and bosdis with  $|C|$ .
8.7 Analysis of example medium- and low-posdis languages
We present more data about the medium-posdis language analyzed in the main article, and we provide comparable evidence for a language with similarly excellent generalization  $(>99\%)$  but very low posdis (0.05), that we will call here low-posdis. The latter language is depicted in black in Fig. 2 of the main text. Both languages come from the training
configuration with 2 100-valued input attributes and 3 100-symbol positions.
Mutual information profiles. Table 5 reports mutual information for the two languages. Note how the highly entangled low-posdis is almost uniform across the table cells.
medium-posdis	low-posdis
att1	att2	att1	att2
pos1	1.10	2.01	1.72	1.95
pos2	0.19	4.16	1.74	1.71
pos3	4.44	0.13	2.16	1.77
Table 5: Mutual information of each position with each attribute for the studied languages.
Vocabulary usage. Considering all messages produced after training for the full training and test set inputs, effective vocabulary usage for pos1, pos2 and pos3 are as follows (recall that 100 symbols are maximally available):
medium-posdis: 91, 96, 98
- low-posdis: 99, 99, 100
Although vocabulary usage is high in both cases, medium-posdis is slightly more parsimonious than low-posdis.
Ablation studies. Table 6 reports ablation experiments with both languages. The results for medium-posdis are discussed in the main text. We observe here how virtually any ablation strongly impacts accuracy in denoting either attribute by the highly entangled low-posdis language. This points to another possible advantage of (partially) disentangled languages such as medium-posdis: since pos2 and pos3 are referring to att2 and att1 independently, in ablations in which they are untouched, Receiver can still retrieve partial information, by often successfully guessing the attribute they each refer to. We also report in the table the effect of shuffling across the positions of each message. This is very damaging not only for medium-posdis, but for low-posdis as well, showing that even the latter is exploiting positional information, albeit in an inscrutable, highly entangled way. Note in Fig. 2 of the main article that neither language has high bos.
(a) density=1
(b) density=0.51
(c) density=0.25
Figure 4: Sampling the same number of input instances  $(= 10000)$  with different densities. The axes of the shown matrices represent the values of two attributes, with the dark-red cells standing for inputs that were sampled. We ensure that each value of each attribute is picked at least once by always sampling the full diagonal.
Figure 5: Average accuracy on unseen combinations as a function of channel capacity of the successful runs. The x-axis is ordered by increasing channel capacity. In the brackets we note  $(c_{len}, c_{voc})$ . Vertical bars represent SEM.
8.8 Effect of channel capacity on ease of transmission
Table 7 replicates the ease-of-transmission analysis presented in the main text across various channel capacities. We observe in most cases a significantly positive correlation, that is even higher (1) for larger Receivers and (2) for emergent languages with shorter messages (smaller  $c_{len}$ ).
medium-posdis	low-posdis
att1	att2	both	att1	att2	both
fixing pos1	1	3	0	4	5	0
1 position pos2	1	68	0	4	4	0
pos3	89	1	1	8	5	0
shuffling pos1	89	69	61	31	18	6
1 position pos2	100	3	3	30	25	8
pos3	1	100	1	15	20	3
shuffling msg	1	2	0	2	4	0
Table 6: Feeding shuffled messages from the medium-posdis and low-posdis languages to the corresponding trained Receivers. Mean percentage accuracy across 10 random shufflings (standard deviation is always  $\approx 0$ ) when: top: symbols in all positions but one are shuffled across the data-set; middle: symbols in a single position are shuffled across the data-set; bottom: shuffling the symbols within each message (ensuring all symbols move). The data-set includes all training and test messages produced by the trained Sender and correctly decoded in their original form by Receiver ( $>99\%$  of total messages).
(a)  $(i_{att} = 2, i_{val} = 50)$
(b)  $(i_{att} = 2, i_{val} = 100)$
Figure 6: Average accuracy on unseen combinations as a function of channel capacity of the successful runs for two different  $(i_{att}, i_{val})$ . The x-axis is ordered by increasing channel capacity. In the brackets we note  $(c_{len}, c_{voc})$ . Vertical bars represent SEM.
Figure 7: Average of different compositionality measures in function of channel capacity  $(c_{\text{voc}}, c_{\text{len}})$ . Vertical bars represent SEM.
(clen, c VOC)	measure	posdis	bosdis	topsim
GRU(500)	GRU(50)	FFN(500)	GRU(500)	GRU(50)	FFN(500)	GRU(500)	GRU(50)	FFN(500)
(3,50)	Learning Speed	0.82	0.78	0.74	0.71	0.67	0.62	0.72	0.74	0.66
Generalization	0.77	0.77	0.75	0.61	0.62	0.66	0.75	0.76	0.74
(4,50)	Learning Speed	0.79	0.44	0.48	0.76	0.51	0.47	0.89	0.59	0.61
Generalization	0.73	-	0.50	0.77	0.27	0.54	0.84	0.41	0.61
(6,50)	Learning Speed	0.82	0.77	0.79	0.79	0.76	0.77	0.89	0.85	0.87
Generalization	0.78	0.56	0.69	0.76	0.55	0.67	0.85	0.65	0.77
(8,50)	Learning Speed	0.75	0.56	0.78	0.80	0.68	0.78	0.75	0.55	0.71
Generalization	0.67	0.27	0.68	0.78	0.41	0.70	0.53	-	0.54
(10,50)	Learning Speed	0.51	0.29	0.60	0.42	0.31	0.48	0.72	0.49	0.73
Generalization	0.39	-	0.44	0.47	-	0.36	0.41	0.27	0.57
(12,50)	Learning Speed	-	-	-	0.33	-	-	0.49	-	0.35
Generalization	-	-0.28	-	-	-	-	-	-	-
(3,100)	Learning Speed	0.87	0.71	0.35	0.85	0.68	0.33	0.87	0.71	0.35
Generalization	0.80	0.55	0.50	0.81	0.55	0.51	0.79	0.54	0.48
(4,100)	Learning Speed	0.84	0.54	0.43	0.82	0.54	0.46	0.86	0.57	0.49
Generalization	0.82	0.38	0.47	0.80	0.39	0.47	0.82	0.41	0.48
(6,100)	Learning Speed	0.88	0.83	0.80	0.89	0.78	0.78	0.94	0.87	0.83
Generalization	0.87	0.68	0.68	0.90	0.69	0.67	0.90	0.70	0.68
(10,100)	Learning Speed	0.85	0.58	0.62	0.82	0.59	0.64	0.72	0.74	0.66
Generalization	0.86	0.39	0.47	0.81	0.50	0.37	0.72	0.35	0.46
(8,100)	Learning Speed	0.73	0.58	0.65	0.79	0.59	0.65	0.70	0.57	0.66
Generalization	0.69	0.39	0.37	0.67	0.37	0.37	0.49	0.34	0.46
(12,100)	Learning Speed	0.39	-	0.27	0.69	-	0.40	0.67	-	0.51
Generalization	0.38	-	0.34	0.52	-	0.38	0.36	-	-
Average	Learning Speed	0.75	0.62	0.61	0.72	0.63	0.59	0.79	0.67	0.62
Generalization	0.71	0.42	0.54	0.72	0.49	0.51	0.70	0.51	0.63
Table 7: Statistically significant  $(p < 0.01)$  Spearman correlations between retraining performance (measured by new Receiver Learning Speed and Generalization) and compositionality measures (posdis, bosdis and topsim) for  $(i_{att} = 2, i_{val} = 100)$  and different channel capacity.  $-$  indicates no significant correlations.