SlowGuess/NAACL_2022_OCR huggingface dataset:
Q4 (VGG-16 & DenseNet-161). Under the constraints of Section 4, we repeat the experiment using encoders based on VGG-16 (Figure 9) and DenseNet-161 (Figure 10). Notably, variants based on VGG-16 exhibit the same trend from Section 4 for more complex  $k = 1000$  agents to perform poorly as receivers (orange lines in Figure 9). The results for DenseNet-161-based variants introduce nuance to this trend, since complex receivers in this setup perform within the random variance of other receivers (i.e., orange lines in Figure 10 being within blue and pink shading). Referencing the upstream performance of each variant in Tables 13, 15, and 17 offers some insight. First, we observe that DenseNet-161 variants are the best performing variants of ProtoPNet (up to  $97\%$  accuracy compared to  $90\%$  and  $95\%$  of VGG-16 and
Hyperparameter	Main paper choice(s)	Remarks
Sender learning rate (LR)	1-3
Receiver LR	1-3
Lewis Signaling Game Epochs	30
Number of distractors	5
Message length L	10
LSTM Hidden Dim.	256
ConvNet fine-tuning LR	1-2	Standard setting for CNNs
ProtoPNet Dense Rep. LR	1-5	Same as Chen et al. (2019)
ProtoPNet Interface LR	3-3	Same as Chen et al. (2019)
ProtoPNet Prototypes LR	3-3	Same as Chen et al. (2019)
ProtoPNet Classifier LR	1-4	Same as Chen et al. (2019)
ProtoPNet Epochs	20
CW LR	1-2	Same as Chen et al. (2020)
Table 5: Hyperparameter choices based on grid-search.
(b)
Figure 7: Comparison of communication success rate on CUB10 testing data for permutations of a) ConvNet senders, b) ProtoPNet senders, and c) CW senders, using image encoders based on VGG-16. Shaded regions denote standard deviation over five random seeds (each swapping in a new CUB10 test split, sender encoder checkpoint, and receiver encoder checkpoint), with  $L = 10$ .
ResNet-50, respectively), and second, DenseNet-161 requires more model parameters (on the order of 26M compared to 15M and 24M of VGG-16 and ResNet-50 variants). Given this result, it may be possible to mitigate the complexity penalty observed in Section 4 by leveraging parameter-heavy densely-connected neural networks, although the impact on smaller models is notable. The effect of model parameterization on the language evolution offers an interesting direction for future work.
A.4.2 Upstream task metrics
In Table 12 we provide the accuracy score for each ResNet-50 encoder's upstream classification task on CUB10 data. We use the training code open-sourced by the respective paper authors (Chen et al., 2019, 2020). Table 13 gives the accuracy scores for ProtoPNet encoders used in Section 4. The same scores are provided for supplemental variants corresponding to VGG-16 (Tables 14 and 15) and
DenseNet-161 (Tables 16 and 17).
A.5 Supplemental Analysis
A.5.1 LSA vocabulary usage
In Figure 11 we repeat the same experiment from Section 4, this time for ProtoPNet variants. Although the ProtoPNet+LSA variant (bottom row) does not score the highest cumulative vocabulary usage compared to the ConvNet variant (0.445 and 0.645, respectively), it encourages more exploration of the vocabulary compared to the regular ProtoPNet variant (0.327). We observe the same trend with CW in Figure 5, which suggests the LSA mechanism can enable better vocabulary diversity on multiple types of disentangled representations.
A.5.2 LSA concept alignment
Our proposed LSA module requires the disentangled sender agent to weight their utterances by the concept weight matrix  $M^{\prime}$ , which is obtained by
(b)
Figure 8: Comparison of communication success rate on CUB10 testing data for permutations of a) ConvNet senders, b) ProtoPNet senders, and c) CW senders, using image encoders based on DenseNet-161. Shaded regions denote standard deviation over five random seeds (each swapping in a new CUB10 test split, sender encoder checkpoint, and receiver encoder checkpoint), with  $L = 100$ .
(a)
Figure 9: Modulating the sender's categorization ability over realized concepts  $k$  for a)  $S_{k} = 10$ , b)  $S_{k} = 100$ , and c)  $S_{k} = 1000$ , when paired with receivers of varying categorization ability and using VGG-16 as the base encoder architecture.
the product of the learned  $L$ -head matrix  $M$  and initial disentangled representation  $z'$  (as discussed in Section 3.1). This design enables a qualitative analysis of the agent's concept activation at each time step, i.e., at some time  $t$ , we can plot the image that corresponds to the highest-weighted dimension in that time-step,  $\max(M'[t])$ . This is done for six images across three data classes for the best performing LSA+ProtoPNet model (Figure 12) and LSA+CW model (Figure 13) from Section 4. We observe that during the decoding of utterances, the most active concept does not always correspond to visual semantic features of the input data class, e.g., brown and black textures are not always present in the top two rows. Instead, the sender agent may rely on an arbitrary positional encoding of known concepts to describe certain visual features. This is evident across both ProtoPNet and CW models. A
notable effect of the LSA module on CW variants is an increased exploration of the agent vocabulary, as discussed in Section 4, which can be observed in Figure 13 by the lack of repetition between top categories, whereas in Figure 12 we note the same concept is often repeated several times within the same message. Given the qualitative results, we posit that agents rely on an encoding preference that can be considered incongruent with realized concepts of the disentangled models. For example, agents might rely on the position of a concept in the utterance, rather than the concept itself, to capture information that leads to better decisions by the receiver. The LSA module enables a first look into the connection between these grounded concepts and their recall during agent utterances, the enforcement of which offers an exciting direction for future work.
Figure 10: Modulating the sender's categorization ability over realized concepts  $k$  for a)  $S_{k} = 10$ , b)  $S_{k} = 100$ , and c)  $S_{k} = 1000$ , when paired with receivers of varying categorization ability and using DenseNet-161 as the base encoder architecture.
Figure 11: Heat-maps of the best-performing agent's vocabulary usage across time and data classes over the entire CUB10 test set for ProtoPNet, corresponding to experiments based on ResNet-50 in Section 4, where  $|V| = k = 10$ , and agents communicate for ten time-steps (with end-of-sentence symbol '0' at the final time-step). The final column shows the cumulative sum over all time steps, subtitled by the normalized area of non-zero cells.
Sender Arch.	Recv. Arch.	Top. Sim.	Dis. Sim.
ConvNet	ConvNet	0.305 ±0.111	-
ConvNet	CW	0.312 ±0.068	-
ConvNet	ProtoPNet	0.305 ±0.066	-
ProtoPNet	CW	0.299 ±0.143	0.299 ±0.143
ProtoPNet	ProtoPNet	0.380 ±0.101	0.380 ±0.101
ProtoPNet	ConvNet	0.255 ±0.063	0.255 ±0.063
CW	CW	0.330 ±0.042	0.401 ±0.079
CW	ProtoPNet	0.186 ±0.039	0.261 ±0.050
CW	ConvNet	0.297 ±0.032	0.435 ±0.089
Table 6: Language compositionality under different VGG-16 encoder variants  $f$  for sender and receiver on CUB10 test set (higher is better). All measures were statistically significant with  $p \approx 0$ .
Sender Arch.	Recv. Arch.	Top. Sim.	Dis. Sim.
ConvNet	ConvNet	0.437 ±0.078	-
ConvNet	CW	0.296 ±0.168	-
ConvNet	ProtoPNet	0.304 ±0.029	-
ProtoPNet	CW	0.208 ±0.115	0.208 ±0.115
ProtoPNet	ProtoPNet	0.369 ±0.054	0.369 ±0.054
ProtoPNet	ConvNet	0.288 ±0.118	0.288 ±0.118
CW	CW	0.391 ±0.112	0.265 ±0.068
CW	ProtoPNet	0.367 ±0.033	0.186 ±0.100
CW	ConvNet	0.390 ±0.013	0.235 ±0.056
Table 7: Language compositionality under different DenseNet-161 encoder variants  $f$  for sender and receiver on CUB10 test set (higher is better). All measures were statistically significant with  $p \approx 0$
Figure 12: Visualization of the most activated prototypical image at each time-step in an LSA+ProtoPNet agent's concept weight matrix  $(\mathbf{M}^{\prime})$ , given an input image from three classes in CUB10. When decoding utterances, the most activated concept at a given time-step does not always correspond to the class of the observation.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
ProtoPNet	0.263	0.263	58.110
Utterance	±0.039	±0.039	±14.306
RNN	0.468	0.468	76.737
±0.064	±0.064	±1.995
RNN+LSA	0.424	0.424	76.619
±0.109	±0.109	±1.863
RNN
(ConvNet
Baseline)	0.422	-	92.590
±0.080	±2.945
Table 8: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using ProtoPNet disentangled structure to weight the vocabulary  $(|V| = 10)$  and VGG-16 as the base image encoder.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
CW	0.354	0.496	77.928
Utterance	±0.032	±0.018	±2.208
RNN	0.313	0.385	83.167
±0.048	±0.058	±2.032
RNN+LSA	0.294	0.520	86.495
±0.035	±0.047	±0.622
RNN
(ConvNet
Baseline)	0.367	-	92.653
±0.033	±1.894
Table 9: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using CW disentangled representation to weight the vocabulary  $\left( {\left| V\right|  = {10}}\right)$  and VGG-16 as the base image encoder.
Figure 13: Visualization of the most activated image category at each time-step in an LSA+CW agent's concept weight matrix  $(\mathbf{M}^{\prime})$ , given an input image from three classes in CUB10. When decoding utterances, the most activated concept at a given time-step does not always correspond to the class of the observation.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
ProtoPNet	0.319	0.319	80.276
Utterance	±0.027	±0.027	±2.416
RNN	0.325	0.325	79.793
±0.037	±0.037	±1.136
RNN+LSA	0.376	0.376	79.612
±0.072	±0.072	±3.409
RNN
(ConvNet
Baseline)	0.407	-	88.929
±0.092	±1.253
Table 10: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using ProtoPNet disentangled structure to weight the vocabulary  $(|V| = 10)$  and DenseNet-161 as the base image encoder.
Sender Arch.	Top. Sim.	Dis. Sim.	Recv. Acc.
CW	0.210	0.771	40.547
Utterance	±0.061	±0.188	±9.303
RNN	0.377	0.135	44.182
±0.073	±0.085	±29.066
RNN+LSA	0.203	0.785	45.487
±0.070	±0.183	±15.481
RNN
(ConvNet
Baseline)	0.431	-	89.454
±0.048	±3.403
Table 11: Comparison of LSA module, 1-Length utterance baseline, and normal RNNs on CUB10 testing data using CW disentangled representation to weight the vocabulary  $(|V| = 10)$  and DenseNet-161 as the base image encoder.
fS Acc. (# Param.)	fR Acc. (# Param.)
ConvNet	95.280 ± 1.143 (23.52M)	94.951 ± 1.744 (23.52M)
ProtoPNet	91.559 ± 3.751 (23.81M)	93.183 ± 2.731 (23.81M)
CW	72.899 ± 8.264 (23.52M)	78.593 ± 8.293 (23.52M)
Table 12: Average classification accuracy of all Q1 CUB10 sender and receiver encoders based on ResNet-50 (i.e., the variants used for main paper results). Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total). Parameter counts are applicable to Q2 experiments, due to shared model architectures.
k	fS Acc. (# Param.)	fR Acc. (# Param.)
10	95.884 ± 0.980 (23.78M)	94.335 ± 2.457 (23.78M)
100	91.559 ± 3.751 (23.81M)	93.183 ± 2.731 (23.81M)
1000	92.891 ± 1.017 (24.05M)	93.371 ± 1.209 (24.05M)
Table 13: Average classification accuracy of sender and receiver encoders based on ResNet-50 used in Section 4 (Q4). Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total).
fS Acc. (# Param.)	fR Acc. (# Param.)
ConvNet	93.327 ± 1.860 (134.30M)	94.191 ± 1.822 (134.30M)
ProtoPNet	86.558 ± 3.036 (14.83M)	88.799 ± 2.449 (14.83M)
CW	87.419 ± 3.736 (134.30M)	88.127 ± 2.352 (134.30M)
Table 14: Average classification accuracy of all Q1 CUB10 sender and receiver encoders based on VGG-16 (from supplemental results). Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total).
k	fS Acc. (# Param.)	fR Acc. (# Param.)
10	90.001 ± 4.486	89.523 ± 3.719
(14.80M)	(14.80M)
100	86.558 ± 3.036	88.799 ± 2.449
(14.83M)	(14.83M)
1000	90.884 ± 3.125	89.482 ± 3.464
(15.07M)	(15.07M)
Table 15: Average classification accuracy of sender and receiver encoders based on VGG-16 for Q4 supplemental results. Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total).
fSAcc. (# Param.)	fRAcc. (# Param.)
ConvNet	97.974 ± 0.472	98.175 ± 0.759
(26.49M)	(26.49M)
ProtoPNet	96.485 ± 1.063	94.793 ± 2.188
(26.79M)	(26.79M)
CW	25.690 ± 24.975	25.545 ± 30.644
(26.49M)	(26.49M)
Table 16: Average classification accuracy of all Q1 CUB10 sender and receiver encoders based on DenseNet-161 (from supplemental results). Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total).
k	fS Acc. (# Param.)	fR Acc. (# Param.)
10	97.085 ± 0.679 (26.77M)	96.756 ± 0.666 (26.77M)
100	96.485 ± 1.063 (26.79M)	94.793 ± 2.188 (26.79M)
1000	95.767 ± 2.818 (27.03M)	95.398 ± 0.966 (27.03M)
Table 17: Average classification accuracy of sender and receiver encoders based on DenseNet-161 for Q4 supplemental results. Each entry represents the average over 5 random seeds (each row consisting of 10 seeds total).