SlowGuess/NAACL_2022_OCR huggingface dataset:
Washington Garcia
University of Florida Gainesville, FL, USA
w.garcia@ufl.edu
Hamilton Scott Clouse
ACT3, Air Force Research Laboratory  
Wright-Patterson AFB, OH, USA
hamilton.clouse.1@afrl.af.mil
Kevin R.B. Butler
butler@ufl.edu
The emergence of language between artificial agents is a recent focus of computational linguistics, as it offers a synthetic substrate for reasoning about human language evolution. From the perspective of cognitive science, sophisticated categorization in humans is thought to enable reasoning about novel observations, and thus compose old information to describe new phenomena. Unfortunately, the literature to date has not managed to isolate the effect of categorization power in artificial agents on their inter-communication ability, particularly on novel, unseen objects. In this work, we propose the use of disentangled representations from representation learning to quantify the categorization power of agents, enabling a differential analysis between combinations of heterogeneous systems, e.g., pairs of agents which learn to communicate despite mismatched concept realization. Through this approach, we observe that agent heterogeneity can cut signaling accuracy by up to  $40\%$ , despite encouraging compositionality in the artificial language. We conclude that the reasoning process of agents plays a key role in their communication, with unexpected benefits arising from their mixing, such as better language compositionality.
A recent interest in the design of multi-agent systems is the unsupervised emergence of complex behaviors (Havylov and Titov, 2017). Rather than completely supervising agents to perform a mapping between inputs and outputs, it is observed that with enough capacity and bandwidth, agents eventually learn to produce emergent properties for their own benefit, e.g., the creation of artificial language to solve a task more efficiently (Resnick et al., 2020; Lee et al., 2018; Chaabouni et al., 2019). With carefully designed constraints, the agents may eventually produce a proto-language that mimics the complex properties of human lan
Figure 1: Approximating categorization ability in agents through disentangled representations (denoted  $\mathbf{z}^{\prime}$ ), generated by either (a) prototypical parts network (ProtoP-Net) (Chen et al., 2019), where each dimension is the input's activation of a prototypical part previously seen by the agent, or (b) concept whitening (CW) (Chen et al., 2020), where each dimension corresponds to a previously-observed data category.
guages, such as compositionality and basic interpretability (Kottur et al., 2017).
Despite recent progress, previous works have not investigated inter-agent communication in the context of abstract categorization, an ability that allows humans to represent new observations by recalling previous experiential knowledge of abstract concepts or classes (Hofstadter, 2007; Gentner, 2010). Abstract categorization in the human brain can be attributed to a combination of recognition and recall of episodic memory, although identifying the exact mechanism remains an open problem (Tulving, 2002). Due to the power and flexibility of categorization, theories of human intelligence often place categorical reasoning at the center of human cognition, e.g., the notion of analogical symbolic categorization by Gentner (2010), or categorization through situated templates, i.e., frames (and corresponding frame-systems) by Minsky (1988). In the study of artificial multi-agent systems, we propose
Human Language Technologies, pages 4523 - 4540
to operationalize the notion of categorization, and subsequent ability to reason over new observations, through the use of disentangled neural networks as alternate image encoders in agents. Disentangled neural networks are a suitable choice since they can leverage a fixed number  $(k)$  of pre-defined or self-learned concepts to build representations of the observable environment. As illustrated in Figure 1, disentangled agents extract meaning from an observation by leveraging their past experiential knowledge, which is analogous to categorical reasoning in theories of human intelligence (Minsky, 1988; Gentner, 2010).
Due to the subjectivity of experiential knowledge, a natural consequence of the categorical reasoning theory is that two human minds may be heterogeneous, i.e., they do not share the same internal categorization of a situation or object. Natural language serves to mediate these cognitive differences, thus enabling the communication of ideas (Chandler, 2007). Under our synthetic analogue of communicating multi-agent systems, how well can heterogeneous agents mediate their differences in categorization ability through the use of emergent language? In this paper, we offer a differential analysis of agents leveraging heterogeneous techniques for abstract categorization, taking full advantage of disentangled representations as a computational analogue for categorical reasoning. Contrary to the scenario of different internal representations (Chaabouni et al., 2020), our agents can possess either different or identical means of categorical abstraction, which influences the course of their language evolution, e.g., improving compositionality or signaling success of the final proto-language. Our analysis offers a substrate to answer four main research questions about heterogeneous multi-agent communication, revealing subsequent findings as a result:
Q1. What is gained or lost with agent heterogeneity? A1: We find that heterogeneity is at odds with signaling performance. Despite this, heterogeneous systems exhibit better potential survival through higher compositionality.
Q2. Does categorization power and heterogeneity affect signaling ability on novel objects? A2: Simpler disentangled agents with poor performance in supervised tasks can outperform all other agents on novel objects by up to  $9\%$  on signaling accuracy. In some cases,
mismatched agents manage to improve compositionality score.
Q3. Can disentangled representations directly inform agent utterances? A3: Through our proposed Latent Self-attention (LSA) module, incorporating the disentangled representation into agent utterances can encourage inputmessage alignment and improve signaling accuracy by up to  $27\%$ .
Q4. Does increased categorization ability imply better communication? A4: Agents with complex concept realization are potentially poor receivers, despite a successful upstream classification task. Signaling can be performed up to  $33\%$  better with "simpler", i.e., smaller  $k$ , agents.
To encourage reproducibility, we share the code for our experiments online.
2.1 Multi-agent Referential Games
We model the commonly-used Lewis signaling game from cognitive science (Lewis, 1969; Skyrms and Press, 2010). A sender agent  $S$  is shown a target object, and must describe that object to a receiver agent  $R$  using a combination of discrete tokens from a fixed vocabulary  $\mathcal{V}$ .  $R$  then signals the object out of a set of candidate objects (called the candidate set). Our setup is based on that of Lazaridou et al. (2018), who investigated emergent language with the same game. The sender  $S$  and receiver  $R$  each use their own encoder (e.g., image CNNs)  $f^{S}$  and  $f^{R}$  to encode any pre-linguistic object  $\mathbf{o}$  into an initial dense representation  $\mathbf{z}_0$ . The encoder is parameterized by agent-specific weights  $\theta_{f}$ , e.g.,  $f^{S}(\mathbf{o},\theta_{f}^{S}) = \mathbf{z}_{0}$ , where  $S$  denotes the sender. Since the receiver processes the candidate set, which consists of multiple pre-linguistic objects, the receiver's dense representations form a matrix denoted  $Z$ . Given  $\mathcal{V}$  and  $\mathbf{z}_0$ , the message is obtained by sampling tokens at each time step  $t$  from a recurrent policy defined as  $g^{S}(\mathbf{z}_{t - 1},\theta_{g}^{S}) = \mathbf{m}_{t}$ . The decoding process ends as soon as an end-of-sequence token or maximum length  $L$  is reached. In practice, the recurrent decoder is implemented as a recurrent neural network (RNN). The receiver's only input from the sender
https://github.com/FICS/disentangling_category
Figure 2: Whole-system perspective of our approach. (Top - Original) Traditional Sender-Receiver architecture for playing the Lewis Signaling game, with entangled deep networks as encoders  $f^{S}$  and  $f^{R}$ . (Bottom - Ours) Proposed approach which leverages disentangled networks to instantiate  $f^{S}$  or  $f^{R}$  (denoted by latent dimension colors matching candidate set colors), and our new Latent Self-Attention (LSA) module for senders, which enforces alignment between utterances (yellow rectangles) and the disentangled structure over time steps.
is message  $\mathbf{m}$ , which is encoded using encoder  $h^R$  at each time step as  $h^R (\mathbf{m}_t,\theta_h^R) = \mathbf{h}_t$ . Similar to Lazaridou et al. (2018), the receiver predicts the position of the described object within the candidate set as  $\mathrm{argmax}(Z\mathbf{h}_t)$ . The agents are successful during a round of the signaling game if the receiver correctly predicts the index of the target object at the final time-step. The typical whole-system perspective of this game is shown in the top method (Original) of Figure 2.
2.2 Learning
We must jointly optimize sender and receiver policy weights  $\Theta \coloneqq \{\theta_g^S,\theta_h^R\}$ , such that they minimize a game cost function  $\mathcal{L}$ , i.e.,  $\min_{\Theta}\mathcal{L}$ . However, no weights are shared between sender and receiver during the game, since each agent has a uniquely instantiated encoder, and the sender can only transmit discrete signals to the receiver during the game. As a result the gradient of the game cost function is not defined, so it is common to rely on gradient estimation algorithms such as REINFORCE (Schulman et al., 2015) or continuous relaxations of the discrete channel, such as the Gumbel-Softmax estimator (Havrylov and Titov, 2017). We use the straight-through Gumbel-Softmax estimator due to its functional flexibility and performance improvement compared to REINFORCE at test time.
2.3 Disentangled Representations
A common technique is to employ intermediate feature representations of CNNs as the encoder function  $f$ . The purpose of the encoder function is to approximate a mapping function from raw image space to a dense representation  $\mathbf{z} \in \mathbb{R}^d$ . In general, this representation is a point in lower-dimensional latent space that adequately describes useful conceptual priors of the data. From the perspective of model interpretability, it is expected that points closer together in latent space correspond to image samples which are conceptually similar, and likewise, dimensions in the latent space correspond to human understandable concepts (Rudin et al., 2022). However, it was shown that traditional CNN architectures learn latent representations which may not achieve concept separation (Chen et al., 2020), and in fact can activate dimensions for concepts that are completely unrelated (Zhou et al., 2015, 2018). Thus several methods have been proposed in order to disentangle the latent representation of deep networks into  $k$  categories or concepts. In this work, we propose to leverage two such methods, shown in Figure 1, in order to analyze the effect of categorization power in multi-agent communication: (a) Prototypical Parts Network (ProtoPNet) by Chen et al. (2019), which recalls prototypical parts of previously-observed images to describe the current one, and (b) Con
cept Whitening (CW) by Chen et al. (2020), which allocates an observed data class to each of the  $k$  dimensions in  $\mathbf{z}^{\prime}$ .
In order to answer the research questions in Section 1, we implement agents which can play a Lewis Signaling game (described in Section 2.1) using disentangled neural networks, e.g., Prototypical Parts Network (ProtoPNet) by Chen et al. (2019) and Concept Whitening (CW) by Chen et al. (2020). ProtoPNet is trained through unsupervised disentanglement, where given a pre-defined number of concepts per class (totaling  $k$  over all classes), it extracts prototypical image patches representing each concept. Concept Whitening instead aligns orthogonal directions in the encoded input's latent space with  $k$  pre-defined (supervised) concepts through training. In practice, CW can only align a subset of the pre-existing latent space dimensions, leaving the rest of the dimensions to handle residual information. To avoid ambiguity, we denote a sample from this aligned subspace as  $\mathbf{z}' \in \mathbb{R}^k$ . Since ProtoPNet learns a new latent space, it can be said to align the entire latent space to extracted concepts, thus  $\mathbf{z}' = \mathbf{z}$ . In either case, we use the latent vector  $\mathbf{z}$  as the initial input to the sender agent's decoder,  $g^S$ . The whole-system perspective of this approach is shown in Figure 2, where the bottom case (Ours) corresponds to the use of disentangled networks, compared to a traditional approach above (Original). We can manipulate which agents are paired (e.g., ProtoPNet talking to CW) in order to simulate scenarios where two agents have different categorical reasoning. Likewise, we can model categorization power by increasing or decreasing the number of realized concepts  $k$  in disentangled agents. This enables the study of incongruous agents to answer Q1, Q2, and Q4.
3.1 Latent Self-attention (LSA)
Unfortunately, in order to synthesize the discrete message and eventually communicate with the receiver, the sender's disentangled representation must recurrently pass through the decoder  $g^{S}$ , which defines a different (entangled) latent space. This could mean the grounding to the original disentangled space is lost, since traditional RNN decoding is not aware of the encoder's concepts, thus we propose a module named Latent Self-Attention (LSA), inspired by the design of multi-head attenuated RNNs.
tion in transformer networks (Vaswani et al., 2017). We treat the aligned latent dimensions from  $\mathbf{z}'$  as units that align one-to-one with the vocabulary logits of each time step, which can be weighted by a combination of aligned concepts. This mechanism is illustrated in the bottom method (Ours) of Figure 2, distinguished by the dashed rectangle. In order to satisfy the one-to-one alignment, we assume that (1) for the aligned representation  $\mathbf{z}' \in \mathbb{R}^k$ ,  $|\mathcal{V}| = k$ , and (2) agents communicate with fixed message length  $L$  (due to discarding end-of-sequence token).
More formally, consider the recurrent output  $\mathbf{z}_t\in \mathbb{R}^d$  from the sender decoder at time  $t$ . At each time step, the recurrent output is projected to dimension  $|\mathcal{V}| = k$  using a fully-connected network, of which the output is treated as logits over vocabulary space (yellow rectangles), and is simply denoted  $\mathbf{v}_t$ . During the training process, the sender agent learns an  $L$ -head matrix  $M\in \mathbb{R}^{k\times (L\cdot k)}$  (purple rectangle). which is softmaxed row-wise to obtain a weighting function over  $k$  concepts for each time step. The weighting of concepts is applied as  $\mathbf{z}'^T M$  and reshaped to form matrix  $M'\in \mathbb{R}^{L\times k}$ . During decoding, the  $t$ -th row of  $M'$  is elementwise multiplied with  $\mathbf{v}_t$ . Through this mechanism, we can experimentally validate Q3 from Section 1. In our experiments, the receiver is implemented the same as previous works (shown as Original in Figure 2) (Lazaridou et al., 2017). To experimentally isolate the effect of the LSA module, we leave the receiver as-is, treating the disentangled networks as a drop-in selection for  $f^R$ . Incorporating the aligned vector  $\mathbf{z}'$  into the pointing module is left for future work.
We adopt standard metrics from the literature to quantify our study of Q1-Q4, and propose one new variant named disentangled similarity.
Receiver Accuracy (Recv. Acc.). A standard metric in the study of referential games is the communication success rate, i.e., the receiver's effective signaling accuracy on objects from a held-out test set (Skyrms and Press, 2010; Lazaridou et al., 2017). The metric is defined as Recv. Acc. = #correct target objects / #total target objects shown.
Compositionality (Top.Sim.) A common concern in the study of language emergence is compositionality of the proto-language, which can be
described informally as the tendency to re-use vocabulary to describe similar objects (Brighton and Kirby, 2006; Chaabouni et al., 2020). Brighton and Kirby (2006) operationalized the measure of language compositionality with topographic similarity (Top.Sim.), the Spearman correlation between pairwise input distances and their corresponding pairwise message distances. We compute pairwise cosine distances between latent vectors  $\mathbf{z}$  (inputs), and use edit distance to compute pairwise distance between messages. The reported Top.Sim. value is the Spearman correlation of distances (and where available, the associated  $p$ -value is reported).
Disentangled Similarity (Dis.Sim.) Since CW only aligns a subset of the latent space dimensions, it is possible that Top.Sim. will describe correlation to unaligned dimensions. Thus we complement the calculation of Top.Sim. with a new metric, disentangled similarity (Dis.Sim.), which only uses the aligned dimensions to calculate the pairwise input distances. In practice, this is implemented by calculating pairwise cosine distances over  $\mathbf{z}'$  instead of  $\mathbf{z}$ . Consequently, the reported values for ProtoPNet will be equal to Top.Sim., since in that case  $\mathbf{z} = \mathbf{z}'$ .
Preliminaries. To address Q1-Q4, we investigate mixed/same-encoder variants of multi-agent systems. Pairs of agents play the Lewis signaling game over five random seeds. Each agent in a two-player game is instantiated with distinct image encoder checkpoints, thus requiring ten image encoder checkpoints total for each variant (with each checkpoint using a different seed). In all reported results of the main text, we compare against finetuned ResNet-50 encoders (denoted ConvNet) as baselines (He et al., 2016), and subsequently use them as the base models for training ProtoPNet and CW variants. We offer supplemental results for VGG-16 (Simonyan and Zisserman, 2015) and DenseNet-161 (Huang et al., 2017) variants in Appendix A.4.1. With ResNet-50 as the base architecture, each disentangled variant has a similar parameter count on the order of 24M (reported in Appendix Table 12), thus any performance difference due to parameter count is negligible. We source prelinguistic objects from two benchmark image data sources: 10-class subsets of Caltech-UCSD CUB200 (Wah et al., 2011) (denoted CUB10) and the entire few-shot mini-ImageNet dataset (Vinyals et al., 2016). For mini-ImageNet experiments, we train
agents using the 64-class meta-training split, and report communication success rate (Recv. Acc.) on the meta-test set. For CUB10, we uniform randomly sample ten classes for each random seed, ensuring sender and receiver encoders are trained using the same subset. Our communicating agents are derived from those implemented in the publicly-available EGG framework (Kharitonov et al., 2019). We only employ our proposed LSA module for Q3 results, as it is not salient to Q1, Q2, and Q4. Details of training and hyper-parameters are given in Appendices A.1 and A.2, respectively.
Heterogeneous multi-agent systems (Q1): To examine the effect of heterogeneous multi-agent systems, we train nine permutations of mixed/same-encoder systems over five random seeds on CUB10 data (45 permutations total), setting  $k = 100$  for ProtoPNet (default value from Chen et al. (2019)) and  $k = 10$  for CW (using CUB10 classes as concept classes). In Figure 3, we initially compare the communication success rate (y-axis) over training epochs (x-axis), with each sub-figure corresponding to a fixed sender encoder architecture. The results exhibit a tendency for same-encoder systems to outperform their heterogeneous counterparts. This trend is evidenced by ConvNet-ConvNet systems in Figure 3a (red line) and ProtoPNet-ProtoPNet systems in Figure 3b (red line). CW models generally under-performed as receivers in Figure 3a-b, scoring  $47.978 \pm 14.325$  and  $42.001 \pm 5.203$  receiver accuracy, respectively. In terms of disentangled categorization power, ProtoPNet is the strongest with  $k = 100$  and up to  $93\%$  accuracy on the CUB10 classification task (compared to  $78\%$  of CW).
In Table 1 we report Top. Sim. and Dis. Sim. measures for systems from Figure 3 at the best epoch on the test set (according to Recv. Acc.). Although CW models generally under-performed in Figure 3, they score the best compositionality (0.405) paired with ConvNet receivers. This is followed by ConvNet senders (top three rows), with 0.316-0.363 Top.Sim. score. CW-ConvNet scored the highest Dis. Sim. score despite having a Recv. Acc. of  $48.939 \pm 22.226$ , aligning with Chaabouni et al. (2020), who found that compositionality does not necessarily emerge from generalization pressure. Based on Figure 3 and Table 3, we conclude:
2We provide encoder accuracy scores from the upstream classification task in Appendix Table 12.
(b)
Figure 3: Comparison of communication success rate on CUB10 testing data for permutations of a) ConvNet senders, b) ProtoPNet senders, and c) CW senders. Shaded regions denote standard deviation over five random seeds (each swapping in a new CUB10 test split, sender encoder checkpoint, and receiver encoder checkpoint), with  $L = 10$ .
Sf	Rf	Top. Sim.	Dis. Sim.
ConvNet	ConvNet	0.360±0.104	-
ConvNet	CW	0.363±0.130	-
ConvNet	ProtoPNet	0.316±0.056	-
ProtoPNet	ProtoPNet	0.349±0.114	0.349±0.114
ProtoPNet	CW	0.224±0.092	0.224±0.092
ProtoPNet	ConvNet	0.209±0.043	0.209±0.043
CW	CW	0.238±0.047	0.305±0.010
CW	ConvNet	0.405±0.057	0.415±0.052
CW	ProtoPNet	0.287±0.064	0.298±0.092
Table 1: Language compositionality under different ResNet-50 encoder variants  $f$  for sender and receiver on CUB10 test set (higher is better). All measures were statistically significant with  $p \leq  {1}^{-{97}}$  .
Sf	Rf	Top. Sim.	Dis. Sim.
ConvNet	ConvNet	0.254±0.054	-
ConvNet	ProtoPNet	0.323±0.039	-
ConvNet	CW	0.181±0.028	-
ProtoPNet	ProtoPNet	0.395±0.097	0.395±0.097
ProtoPNet	ConvNet	0.357±0.215	0.357±0.215
ProtoPNet	CW	0.450±0.012	0.450±0.012
CW	CW	0.380±0.071	0.162±0.134
CW	ProtoPNet	0.335±0.056	0.052±0.035
CW	ConvNet	0.259±0.078	0.131±0.012
Table 2: Language compositionality under different encoder functions  $f$  for sender and receiver on miniImageNet test set (higher is better). All measures were statistically significant with  $p \approx 0$
A1: Heterogeneous systems appear to underperform compared to same-encoder (e.g., homogeneous) systems, although the former can exhibit higher compositionality in the emerged language.
Categorization power on novel objects (Q2): To better understand the description of novel objects, in Figure 4 we repeat the previous Q1 experiment by swapping CUB10 for miniImageNet, whose test set consists of object classes not seen during training. We observe a similar trend that aligns with A1, except CW-CW outperforms other methods, at best scoring  $85.585 \pm 5.476$  signaling accuracy compared to ConvNet-ConvNet  $(76.586 \pm 6.400)$  and ProtoPNet-ProtoPNet  $(67.542 \pm 3.001)$  despite realizing the least amount of concepts  $(k = 10)$  compared to ProtoPNet  $(k = 100)$ . In some instances, e.g., ConvNet-ProtoPNet, the mismatched system can match the ConvNet-ConvNet performance within  $3\%$  accuracy.
In Table 2, we capture Top.Sim. and Dis.Sim.