SimSiam Naming Game: A Unified Approach for Representation Learning and Emergent Communication

## Abstract

Abstract Emergent communication, driven by generative models, enables agents to develop a shared language for describing their individual views of the same objects through interactions. Meanwhile, self-supervised learning (SSL), particularly SimSiam, uses discriminative representation learning to make representations of augmented views of the same data point closer in the representation space. Building on the prior work of VI-SimSiam, which incorporates a generative and Bayesian perspective into the SimSiam framework via variational inference (VI) interpretation, we propose SimSiam+VAE, a unified approach for both representation learning and emergent communication. SimSiam+VAE integrates a variational autoencoder (VAE) into the predictor of the SimSiam network to enhance representation learning and capture uncertainty. Experimental results show that SimSiam+VAE outperforms both SimSiam and VI-SimSiam. We further extend this model into a communication framework called the SimSiam Naming Game (SSNG), which applies the generative and Bayesian approach based on VI to develop internal representations and emergent language, while utilizing the discriminative process of SimSiam to facilitate mutual understanding between agents. In experiments with established models, despite the dynamic alternation of agent roles during interactions, SSNG demonstrates comparable performance to the referential game and slightly outperforms the Metropolis-Hastings naming game.

## 1 Introduction

Emergent communication (EmCom) studies how multiple agents, through interaction, can develop a shared language, known as a symbol emergence system (Cangelosi & Parisi, ; Taniguchi et al., , ; Lazaridou & Baroni, ; Rita et al., ; Peters et al., ). Many studies in EmCom, based on Shannon-Weaver-like communication models (Shannon & Weaver, ), such as the Lewis signaling game (Lewis, ) or the referential game (Lazaridou et al., ), primarily focus on how agents can discriminate target objects or analyze the compositionality of the emergent signals (Havrylov & Titov, ; Denamganaï et al., ; Lipinski et al., ), often without considering internal representations. In contrast, collective predictive coding (CPC)-based EmCom (Taniguchi, ), such as the Metropolis-Hastings naming game (MHNG) (Hagiwara et al., ; Taniguchi et al., ), views EmCom as a form of decentralized Bayesian inference. This approach focuses on both the representations learned within individual agents and the emergence of symbols at a societal level, referred to as social representation learning.

Representation learning, on the other hand, has been a fundamental aspect of machine learning (Bengio et al., ; LeCun et al., ), particularly in tasks like image classification, where the objective is to extract meaningful features from raw data (Bishop, ). Within this domain, self-supervised learning (SSL) has attracted significant attention by enabling models to learn representations without relying on labeled data (Liu et al., ; Uelwer et al., ). One important approach in SSL is contrastive learning, which focuses on learning by comparing different augmented views of the same data point (Le-Khac et al., ). Notable models in this area, such as SimCLR (Chen et al., ), DINO (Caron et al., ), and SimSiam (Chen & He, ), have shown that this approach can align representations and improve feature extraction.

Both CPC-based EmCom and contrastive-based SSL follow a similar process. In CPC-based EmCom, agents observe the same object from different viewpoints and iteratively develop a common language by aligning their internal representations through generative modeling (Taniguchi, ). In contrast, contrastive-based SSL models, particularly SimSiam, align representations of augmented views of the same data point closer through a discriminative process, relying only on positive pairs (Chen & He, ). Furthermore, recent research (Nakamura et al., ) has applied variational inference (VI) to SSL models, providing a generative interpretation of traditionally discriminative methods, such as SimSiam, and capturing uncertainty in learned representations.

Building on the VI-based interpretation of SSL models, we propose a unified approach that connects discriminative SSL-based representation learning with generative CPC-based EmCom. We introduce SimSiam+VAE, which integrates a Variational Autoencoder (VAE) (Kingma & Welling, ) into the predictor of the SimSiam network. This integration enhances latent representations with uncertainty by combining two processes: aligning positive pairs through contrastive comparison and refining representations via the VAE’s encoding-decoding process, all without relying on negative samples.

We further extend SimSiam+VAE into a structured communication framework called the SimSiam Naming Game (SSNG), designed to facilitate EmCom between agents. In SSNG, each agent operates a separate SimSiam+VAE network, where the backbone and projector function as a perception module to transform observations into internal representations. The VAE predictor acts as a language coder, responsible for generating and decoding messages. Agents perceive different viewpoints of the same object and use a Bayesian approach to form internal representations and develop an emergent language. Through iterative exchanges, they interact similarly to the SimSiam+VAE, using its discriminative process to align their representations and achieve mutual understanding.

For evaluation, we conduct two experiments. First, we assess the performance of SimSiam+VAE in representation learning by measuring classification accuracy on the image datasets FashionMNIST and CIFAR-10. Second, we evaluate the SSNG’s capability in emergent communication (EmCom) using the dSprites dataset, measuring the compositional generalization of the emergent language by applying TopSim (Brighton & Kirby, ) to unseen data (Chaabouni et al., ; Baroni, )

Our contributions are summarized as follows:

- •
We formulate SimSiam+VAE, a unified model that bridges representation learning and EmCom through a generative and discriminative framework. By integrating a VAE into the SimSiam architecture, we enhance latent representation learning and uncertainty modeling, using only positive pairs.
- •
We introduce the SimSiam Naming Game (SSNG), a novel communication game grounded in the principles of CPC. SSNG utilized the combined generative-discriminative approach of SimSiam+VAE to iteratively align internal representations and develop a shared emergent language.

## 2 Preliminaries

Figure: Figure 1: Illustrations of the SSL interpreted as a form of VI. (a): The PGM representation of the inference process in SSL. Observations $x_{A}$ and $x_{B}$ represent two augmented views (considered as multimodal observations) of the same data sample, derived from a dataset $D$. The arrows point from $x_{A}$ and $x_{B}$ to the latent variable $z$, indicating that the augmented views share a common latent representation $z$, which is inferred from these observations. (b): The SimSiam framework (Chen & He, ). Two augmented views, $x_{A}$ and $x_{B}$, are processed through a shared backbone and projector network $f$ to produce latent representations $z_{A}$ and $z_{B}$. A predictor network $h$ generates a transformed representation $z^{\prime}_{A}$, which is compared to $z_{B}$ using a similarity measure. A stop-gradient operation is applied to $z_{B}$ to prevent gradient flow from $z^{\prime}_{A}$, ensuring stable training and avoiding model collapse. (c): The proposed VI-SimSiam framework (Nakamura et al., ) extends SimSiam by modeling representation uncertainty. Latent representations $z_{A}$ and $z_{B}$ are produced similarly, but two predictors output the mean direction $\mu$ and concentration parameter $\kappa$ of the power spherical distribution, enabling both the representation and its uncertainty to be modeled.
Refer to caption: extracted/5960865/imgSSVI.jpg

Self-Supervised Learning (SSL) as Variational Inference (VI):
Recent work (Nakamura et al., ) suggests that SSL can be interpreted through the lens of VI, a probabilistic framework for learning latent variable models (Blei et al., ). In SSL, representations are typically learned by minimizing a contrastive loss between different augmented views of the same data, with the aim of bringing these views closer in their latent space representation. This process is analogous to VI, where augmented views are treated as ”observations” that contribute to learning a shared latent variable. The augmentations in SSL function similarly to distinct modalities within a multimodal generative model in VI.

Denote ${\mathbb{X}}=\{x_{A},x_{B}\}$, where $x_{A}$ and $x_{B}$ are two augmented views of the same data point. Fig. 1(a) illustrates the probabilistic graphical model (PGM), where the latent variable $z$ represents a shared representation of the augmented data. The objective of SSL, when viewed through VI, is to find parameters $\theta$ that maximize the log marginal likelihood of the observations given $z$. However, computing the true posterior $p_{\theta}(z|{\mathbb{X}})$ directly is intractable, leading to the use of a variational distribution $q_{\phi}(z|{\mathbb{X}})$ to approximate the posterior. This formulation leads to the objective function given by:

$$ $\mathbb{E}_{p(z|{\mathbb{X}})}[\log p_{\theta}({\mathbb{X}}|z)]\geq\mathcal{J} _{\text{SSL}}:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}[\log p_{\theta}({\mathbb{ X}}|z)]-D_{\text{KL}}[q_{\phi}(z|{\mathbb{X}})\|p(z)]$ (1) $$

The SSL objective function is then decomposed as:

$$ $\mathcal{J}_{\text{SSL}}:=\mathcal{J}_{\text{align}}+\mathcal{J}_{\text{ uniform}}+\mathcal{J}_{\text{KL}}$ (2) $$

where $\mathcal{J}_{\text{align}}$ encourages the alignment of representations from different views of the same data point, bringing them closer in the latent space. This aligns with the goal of SSL to learn invariant representations across augmented views. $\mathcal{J}_{\text{uniform}}$ promotes a well-distributed representation over the latent space to avoid collapse. Finally, $\mathcal{J}_{\text{KL}}$, introducing a Kullback-Leibler (KL) divergence, regularizes the approximate posterior distribution $q(z|{\mathbb{X}},\phi)$ to be close to the prior $p(z)$.

The paper further demonstrates that specific SSL methods, such as SimSiam, SimCLR, and DINO, can be viewed under this VI framework by appropriately defining how they address alignment, uniformity, and regularization of latent variables. The inference process for these models operates as follows:

$$ $\displaystyle z$ $\displaystyle\sim q_{\phi}(z|{\mathbb{X}})=q_{\phi}(z|x_{A},x_{B})\quad$ ( $z$ is inferred from both $x_{A}$ and $x_{B}$ ) (3) $$

## 3 SimSiam+VAE for Representation Learning

Figure: Figure 2: Illustrations of the SimSiam+VAE. (a) The PGM representation of the generative and inference process in SimSiam+VAE. From the observations $x_{A}$ and $x_{B}$, the representation $z$ is inferred, which is subsequently used to infer latent variable $z$. Solid lines indicate the generative process (from $w$ to $z$), while dashed lines indicate the inference process (from $x_{A}$ and $x_{B}$ to $z$ and then to $w$). (b) Architecture of the SimSiam+VAE framework. Two augmented views, $x_{A}$ and $x_{B}$, are processed through a shared backbone and projector network $f$ to produce representations $z_{A}$ and $z_{B}$. The predictor $h$ incorporates VAE components: the predictor encoder outputs the mean $\mu_{A}$ and covariance $\Sigma_{A}$ of the distribution over the latent variable $w_{A}$. The predictor decoder reconstructs the representation $z^{\prime}_{A}$ from $w_{A}$. The similarity between $z^{\prime}_{A}$ and $z_{B}$ is measured, and a stop-gradient operation is applied to $z_{B}$ to prevent collapse.
Refer to caption: extracted/5960865/imgSSVAE.jpg

### 3.1 Model Description

The proposed SimSiam+VAE model (Fig. 2) extends SimSiam by integrating a VAE into the predictor. The backbone and projector network, denoted as $f$, serves as a feature extractor, mapping the augmented data $x_{i}$, for $i\in\{A,B\}$, to the latent representation $z_{i}$. The predictor, $h$, includes an encoder $h^{\text{(enc)}}$ that maps $z_{i}$ to the parameters of a Gaussian distribution over a latent variable $w_{i}$, from which $w_{i}$ is sampled. The decoder $h^{\text{(dec)}}$ then reconstructs $w_{i}$ to $z^{\prime}_{i}$. The overall model, $g$, represents the composition of the backbone-projector network $f$ followed by the predictor $h$, such that $g=h\circ f$.

The inference process of the SimSiam+VAE operates as follows:

$$ $\displaystyle z$ $\displaystyle\sim q_{\phi}(z|{\mathbb{X}})=q_{\phi}(z|x_{A},x_{B})\quad$ ( $z$ is inferred from both $x_{A}$ and $x_{B}$ ) (4) $\displaystyle w$ $\displaystyle\sim q_{\phi}(w|z)\quad$ ( $w$ is inferred from $z$ ) (5) $$

Similar to SimSiam, the proposed SimSiam+VAE model uses the stop-gradient mechanism to block gradients from being backpropagated through one of the branches. This mechanism treats the second latent representation as a constant, avoiding collapse to trivial solutions. Additionally, the VAE introduces a regularization term via the KL divergence, further preventing collapse through its encoding-decoding process. We conducted experiments to compare the model’s performance with and without the stop-gradient mechanism, as discussed in Section 5.1.

### 3.2 Objective Functions

Fig. 2(a) shows the PGM of SimSiam+VAE, showing the inference process from augmented data $x_{A}$ and $x_{B}$ to representation $z$, and subsequently to the latent variable $w$. The objective of this model, under VI, is to find a parameter $\theta^{*}$ that maximizes the log marginal likelihood of the data. However, since the true posterior $p_{\theta}(z,w|{\mathbb{X}})$ is intractable, we approximate it using the variational distribution $q_{\phi}(z,w|{\mathbb{X}})$. The resulting optimization problem is to maximize the objective function $\mathcal{L}_{\text{SSL}}$, which is defined as:

$$ $\theta^{*},\phi^{*}=\operatorname*{arg\,max}_{\theta,\phi}\mathbb{E}_{q_{\phi} (z,w|{\mathbb{X}})}\left[\log\frac{p_{\theta}({\mathbb{X}},z,w)}{q_{\phi}(z,w| {\mathbb{X}})}\right]$ (6) $$

This optimization leads to the objective function:

$$ $\displaystyle\mathcal{J}_{\text{SSL}}\approx\mathcal{J}_{\text{align}}+ \mathcal{J}_{\text{recon}}+\mathcal{J}_{\text{uniform}}+\mathcal{J}_{\text{KL}}$ (7) $$

where

$$ $\displaystyle\mathcal{J}_{\text{align}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\log p_{\theta}(z|{ \mathbb{X}})\right]-\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\log q_{\phi}(z |{\mathbb{X}})\right]$ (8) $\displaystyle\mathcal{J}_{\text{recon}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\mathbb{E}_{q_{\phi} (w|z)}\left[\log p_{\theta}(z|w)\right]\right]$ (9) $\displaystyle\mathcal{J}_{\text{uniform}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[-\log p_{D}(z)\right]$ (10) $\displaystyle\mathcal{J}_{\text{KL}}$ $\displaystyle:=-\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[D_{\text{KL}}\left( q_{\phi}(w|z,{\mathbb{X}})\|p(w)\right)\right]$ (11) $\displaystyle p_{D}(z)$ $\displaystyle:=\mathbb{E}_{p_{D}({\mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]$ (12) $$

The alignment loss (Eq. 8) encourages the latent representations from different augmented views of the same data point to align in the representation space. The reconstruction loss (Eq. 9) encourages the VAE to accurately reconstruct the representation from the latent variable $w$. The uniform loss (Eq. 10) promotes a uniform distribution of representations in the representation space to avoid collapse. The KL-divergence term (Eq. 11) regularizes the distribution of the latent variable $w$, keeping it close to the prior. Lastly, (Eq. 12) defines the empirical distribution of the latent variables derived from the data.

In SimSiam+VAE, the prior $p(w)$ is a standard Gaussian distribution, while the prior $p(z)$ is uniform on the hypersphere $S^{d-1}$. The distribution $q_{\phi}(w|z)$ is modeled as a multivariate Gaussian distribution conditioned on $z$. Meanwhile, $p_{\theta}(z|w)$ is defined as a Dirac delta function, indicating a deterministic mapping from $w$ to $z$. The distribution $q_{\phi}(z|{\mathbb{X}})$ is modeled as a mixture of experts, where each expert corresponds to the contribution of an augmented view. The distribution $p_{\theta}(z|{\mathbb{X}})$ is represented as a product of experts, capturing the joint distribution across all augmented views:

$$ $\displaystyle p(w)$ $\displaystyle\sim\mathcal{N}(0,I)$ (13) $\displaystyle p(z)$ $\displaystyle:=\mathcal{U}(S^{d-1})$ (14) $\displaystyle q_{\phi}(w|z)$ $\displaystyle:=\mathcal{N}(w;\mu_{w}=h^{\text{(enc)}}_{(\mu)}(z),\Sigma_{w}=h^ {\text{(enc)}}_{(\Sigma)}(z))$ (15) $\displaystyle p_{\theta}(z|w)$ $\displaystyle:=\delta(z-h^{\text{(dec)}}(w))$ (16) $\displaystyle q_{\phi}(z|{\mathbb{X}})$ $\displaystyle:=\frac{1}{M}\sum_{i=1}^{M}\delta(z-f_{\phi}(x_{i}))$ (17) $\displaystyle p_{\theta}(z|{\mathbb{X}})$ $\displaystyle:=\eta_{\theta}\prod_{j=1}^{M}\text{vMF}(z;\mu_{z}=g_{\theta}(x_{ j}),\kappa_{z})$ (18) $$

where

- •
$h^{\text{(enc)}}_{(\mu)}$ and $h^{\text{(enc)}}_{(\Sigma)}$ are the components of encoder network $h^{\text{(enc)}}$ that generate the mean vector $\mu_{w}$ and covariance matrix $\Sigma_{w}$ of the Gaussian distribution from which $w$ is sampled.
- •
$h^{\text{(dec)}}$ is are decoder of language coder $h$, providing a deterministic mapping from $w$ to $z$
- •
$\delta(z-f_{\phi}(x_{i}))$ is a Dirac delta function centered at $f_{\phi}(x_{i})$.
- •
$\eta_{\theta}^{-1}$ is a normalization constant.
- •
$\text{vMF}(z;\mu_{z},\kappa_{z}):=C_{\text{vMF}}(\kappa_{z})\exp(\kappa_{z}\mu
_{z}^{\top}z)$ is the von-Mises-Fisher distribution with mean direction $\mu_{z}$ and concentration parameter $\kappa_{z}\in\mathbb{R}^{+}$. The term $C_{\text{vMF}}(\kappa_{z})$ is a normalization constant defined using the modified Bessel function. $\kappa_{z}$ is also constant.

The objective function of SimSiam+VAE is then given as:

$$ $\displaystyle\mathcal{J}_{\text{SSL}}\approx\sum_{i,j}\left(g_{\theta}(x_{i})^ {\top}f_{\phi}(x_{j})\right)-\beta\sum_{i}D_{\text{KL}}\left(q_{\phi}(w|z,x_{i })\|p(w)\right)$ (19) $$

###### Proof.

See Appendix B.
∎

In Eq. 19, the first term encourages the alignment of representations from different augmentations of the same input, similar to the reconstruction loss in a VAE. The second term is a regularization, ensuring that the latent variable $w$ remains close to the prior distribution. The hyperparameter $\beta$ controls the balance between the alignment and regularization terms, similar to the $\beta$-VAE introduced by Higgins et al. (). Pseudocode for the SimSiam+VAE model is provided in Appendix D.

###### Proof.

## 4 SimSiam Naming Game for Emergent Communication

### 4.1 Model Description

The objective of SimSiam+VAE is to bring different views (augmentations) of the same data point closer in the representation space without relying on negative pairs. This aligns with the CPC-based EmCom, where two agents observe the same object from different viewpoints and develop shared representations without explicit labels. In this section, we extend SimSiam+VAE to facilitate EmCom between two agents, $A$ and $B$, through a communication game called the SimSiam Naming Game (SSNG). Each agent $*\in\{A,B\}$ operates as a branch of the SimSiam+VAE, processing its observation $x_{*}\in\{x_{A},x_{B}\}$, which is derived from a distinct viewpoint of the original object $x$.

Unlike the original SimSiam+VAE, which processes two augmentations of $x$ through a shared network to produce a single latent representation $z$ and a corresponding latent variable $w$, the SSNG introduces two separate latent representations, $z_{A}$ and $z_{B}$, one for each agent. Each branch of the network independently maps its observation $x_{*}$ to its internal representation $z_{*}$. These representations are then combined to form a shared message $w$, which acts as the emergent language for communication. The message $w$ enables the agents to align their internal representations, fostering mutual understanding. Through this structure, SSNG allows each agent to retain its unique perspective while contributing to a shared language. This approach aligns with Peirce’s semiotics theory (Chandler, ), establishing a triadic relationship among the symbol (observation $x_{*}$), the interpretant (internal representation $z_{*}$), and the sign (message $w$) (Fig. 3).

Each agent $*\in\{A,B\}$ in this communication game has the two components: perception and language coder. The perception ($f_{*}$), consisting of the backbone and projector, transforms the observation $x_{*}$ into the internal representations $z_{*}$. The language coder ($h_{*}$) includes the predictor, which consists of an encoder ($h_{*}^{\text{(enc)}}$) and a decoder ($h_{*}^{\text{(dec)}}$). The encoder maps the internal representation $z_{*}$ to a shared message $w$ while the decoder receives and decodes the message into an internal representation $z^{\prime}_{*}$.

The model components in the SSNG are identical to those in SimSiam+VAE. The key difference is that the latent variable $w$ now follows a categorical distribution over $K$, where $K$ represents the vocabulary or dictionary size. In the SSNG, the prior $p(w)$ is a uniform categorical distribution defined on the simplex $\Delta^{K-1}$ and $w$ is modeled as:

$$ $\displaystyle p(w)$ $\displaystyle:=\mathcal{U}(\Delta^{K-1})$ (20) $\displaystyle q_{\phi}(w|z)$ $\displaystyle:=\text{Cat}(w;\text{GS}(h_{*}^{\text{(enc)}}(z)))$ (21) $$

where $h_{*}^{\text{(enc)}}(z)$ represents the logits produced from the internal representation $z$ via the encoder of language coder. These logits are converted into a categorical distribution, $\text{Cat}(w)$, using the Gumbel-Softmax (GS) distribution (Jang et al., ). The Straight-Through (ST) estimator is then applied to obtain one-hot vectors, enabling gradient-based training while maintaining discrete message representations (Bengio et al., ).

Figure: Figure 3: The EmCom between two agents, A and B, based on the SimSiam Naming Game. (a) Two agents observe the same object from different perspectives. Each agent maps its observations to internal representations and uses them to infer and predict emergent language symbols, enabling them to communicate their perceptions and develop a shared emergent language. (b) The PGM of SSNG: Denote agent $*\in\{A,B\}$. Solid lines represent the generative process, which starts from the shared latent variable $w$ to the representation $z_{*}$. Dashed lines represent the inference process, where each agent infers its representation $z_{*}$ from its observation $x_{*}$, and the shared message $w$ is inferred jointly from both agents’ internal representations $z_{A}$ and $z_{B}$. (c) The structure of agents: Both agents $*$ have the same model architecture with a backbone and projector $f_{*}$ and the predictor $h_{*}$ acts as the language coder, consisting of an encoder $h_{*}^{\text{(enc)}}$ and a decoder $h_{*}^{\text{(dec)}}$. In the example shown, agent A (depicted as the speaker) generates and transmits a message $w_{A}$ to agent B (as the listener), who processes it through a predictor decoder, producing an internal representation $z^{\prime}_{A}$, which is then compared to $z_{B}$ to measure their similarity.
Refer to caption: extracted/5960865/imgSSNG.jpg

### 4.2 Loss Function

In this communication game, agents A and B alternately take on the roles of speaker $(Sp)$ and listener $(Li)$, with possible role pairs $(Sp,Li)\in\{(A,B),(B,A)\}$. Given the listener $(Li)$ and the message $w_{Sp}$ received from the speaker $(Sp)$, the objective function of the listener is given by:

$$ $\displaystyle\mathcal{J}_{Li}\approx[h_{Li}^{\text{(dec)}}(w_{Sp})]^{\top}f_{ Li}(x_{Li})-\beta D_{\text{KL}}\left(q_{Li}(w_{Li}|z_{Li},x_{Li})\|p(w_{Li})\right)$ (22) $$

###### Proof.

See Appendix C.
∎

This objective function is applied similarly for both agents A and B when either agent acts as the listener. In Eq. (22), the first term calculates the similarity loss between the decoded representation $z^{\prime}_{Sp}$ (obtained from the received message $w_{Sp}$ through the decoder of listener’s language coder $h_{Li}^{\text{(dec)}}$) and the listener’s internal representation $z_{Li}$ (generated by listener’s perception $f_{Li}$). The second term serves as a regularization component that regularizes the listener’s latent space $w_{Li}$.

The inference process via the SSNG builds on the SimSiam+VAE with the parameters $\theta$ and $\phi$ spanning both agents $\theta_{A}$, $\phi_{A}$ of agent A and $\theta_{B}$, $\phi_{B}$ of agent B operates as follows:

$$ $\displaystyle z_{A}$ $\displaystyle\sim q_{\phi}(z_{A}|x_{A})\quad$ (Agent A infers $z_{A}$ from $x_{A}$ ) (23) $\displaystyle z_{B}$ $\displaystyle\sim q_{\phi}(z_{B}|x_{B})\quad$ (Agent B infers $z_{B}$ from $x_{B}$ ) (24) $\displaystyle w$ $\displaystyle\sim q_{\phi}(w|z_{A},z_{B})\quad$ (The shared latent variable $w$ is inferred from both $z_{A}$ and $z_{B}$ ) (25) $$

###### Proof.

### 4.3 The SimSiam Naming Game (SSNG)

The SSNG facilitates communication and mutual understanding between agents through the following sequence of interactions:

- i)
Perception: The speaker ($Sp$) observes the input $x_{Sp}$ related to object $x$ to form an internal representation $z_{Sp}$ using its perception module $f_{Sp}$.
- ii)
Naming: The speaker ($Sp$) generates a message $w_{Sp}$ using the encoder $h_{Sp}^{\text{(enc)}}$ of the language coder and sends this message to the listener ($Li$).
- iii)
Communication: Upon receiving the message $w_{Sp}$, the listener ($Li$) decodes it into $z^{\prime}_{Sp}$ using the decoder $h_{Li}^{\text{(dec)}}$ of language coder.
- iv)
Learning: The listener ($Li$) calculates the loss using Eq. 22 by comparing $z^{\prime}_{Sp}$ with its own $z_{Li}$ (generated by $f_{Li}$), then updates its model parameters to refine its understanding.
- v)
Turn-taking: After the interaction, the roles of $Sp$ and $Li$ are swapped, and the process repeats from step i).

This communication game, aligning with the principle of CPC, enables each agent to iteratively update its understanding based on the shared symbols through encoding, sharing, decoding, and learning. A comparison among referential games (Lazaridou et al., ), Metropolis-Hastings naming game (Taniguchi et al., ) and our SimSiam naming game is presented in Appendix A. The pseudocode for the SSNG is provided in Appendix E.

## 5 Experiments and Discussions

This section presents two experiments to evaluate the proposed SimSiam+VAE model and SimSiam naming game. The source code for these experiments is available on GitHub(^1^11…).

### 5.1 Experiment 1: SimSiam+VAE in representation learning

Datasets: We use the FashionMNIST (Xiao et al., ) and CIFAR-10 (Krizhevsky, ) datasets.

Model architecture: A Convolutional Neural Network (CNN) backbone is used for FashionMNIST, while ResNet18 (He et al., ) is used for CIFAR-10. In both cases, the projector and predictor utilize a multi-layer perceptron (MLP) architecture. (More details in Appendix F)

Linear evaluation: All models are trained for 500 epochs. Then, a classifier is trained on the frozen representations obtained from the model using the training set labels and then evaluated on the test set. For FashionMNIST, Top-1 accuracy is reported, while for CIFAR-10, Top-2 accuracy is used.

Comparison Models: We compare our SimSiam+VAE model against SimSiam, VI-SimSiam.

Results and Discussion: (Table 1), the stop-gradient mechanism is essential for the proposed SimSiam+VAE framework. Without it, the model collapses to a trivial solution and fails to capture representation features. Our results show that SimSiam+VAE outperforms both SimSiam and VI-SimSiam, highlighting the advantage of integrating a VAE into the SimSiam. This integration enhances the model’s ability to capture diverse features, leading to improved representation learning.

**Table 1: Classification performance of different models on FashionMNIST and CIFAR-10.**
| Model | FashionMNIST (Top-1) | CIFAR-10 (Top-2) |
| --- | --- | --- |
| SimSiam | 82.95 | 59.24 |
| VI-SimSiam | 81.87 | 62.80 |
| SimSiam+VAE (no stop-grad) | 10.00 | 20.00 |
| SimSiam+VAE (ours) | 84.27 | 67.98 |

### 5.2 Experiment 2: SimSiam Naming Game in Emergent Communication

Datasets: We use the dSprites (Matthey et al., ) dataset, which consists of images of 2D shapes varying across different generative factors. Both agents are provided with the same data point but from different perspectives. The agents are then tested using a set of unseen data points to assess their generalization capabilities.

Model Architecture: The vocabulary size $|V|$ is 100, and the message length is 10. An MLP is used for the perception (backbone and projector). For the language coder’s encoder, we use an LSTM to generate a discrete distribution over $V$ at each time step ($t$). This process is performed autoregressively, using the embedding of the discrete token sampled at the previous step ($t-1$) as input. The decoder consists of another LSTM that processes the sequence of embeddings recurrently, with the final hidden state as the output. (More details in Appendix G).

Evaluation: All models are trained for 1000 epochs. We use Topographical Similarity (TopSim) to evaluate how well the emergent language disentangles and aligns with the generative factors.

Comparison Models: We compare the emergent language from our SSNG with those of the referential (Xu et al., ) and Metropolis-Hastings naming games (Hoang et al., ), all of which use the same LSTM-based models for generating and decoding messages.

Results and Discussion: (Table 2)
Compared to the referential game, where agents are fixed as either message generators or interpreters, SSNG demonstrates comparable performance. However, compared to MHNG, where agents can both create and interpret messages, SSNG achieves slightly better results. These suggest that SSNG is a potential alternative approach for facilitating EmCom.

**Table 2: TopSim of different communication games on the dSprites. The referential game produces a single TopSim value, while the other games produce separate values for each agent (A and B).**
| Model | TopSim (A) | TopSim (B) |
| --- | --- | --- |
| Referential Game | 0.22 |  |
| Metropolis-Hastings Naming Game | 0.19 | 0.18 |
| SimSiam Naming Game (ours) | 0.22 | 0.18 |

## 6 Related Work

Emergent Communication (EmCom) examines how agents develop a shared language through interactions, drawing inspiration from cognitive science theories (Wagner et al., ; Steels, ). Research in multi-agent reinforcement learning (Foerster et al., ) demonstrated how agents could develop communication to optimize collective rewards. Comprehensive surveys of this field include (Galke et al., ; Brandizzi, ; Boldt & Mortensen, ). Recent studies have focused on CPC-based, which emphasizes joint attention in human communication (Okumura et al., ). The MHNG (Taniguchi et al., ) utilizes decentralized Bayesian inference to achieve a consensus on shared symbols, aligning with predictive coding and world model (Hohwy, ; Friston et al., ; Taniguchi et al., ). The MHNG has been applied in multimodal datasets using methods like Inter-MDM (Hagiwara et al., ) and Inter-GMM+MVAE (Hoang et al., ). Moreover, MHNG has been extended to recursive multi-agent communication systems (Inukai et al., ) and integrated into multi-agent reinforcement learning (Ebara et al., ).

Representation learning is essential in machine learning tasks like image classification, allowing models to extract features from raw data (Goodfellow et al., ). SSL has become a popular method for learning representations without labels (Jing & Tian, ). A key SSL approach is contrastive learning, which aligns representations by comparing different augmented views of the same data point (Cole et al., ). MoCo (He et al., ) introduces a momentum encoder to maintain a queue of negative samples, while BYOL (Grill et al., ) eliminates the need for negative pairs, using a stop-gradient mechanism to avoid collapse. Recent research has combined VAE and contrastive learning to improve representation learning. CR-VAE adds contrastive regularization to the VAE objective (Lygerakis & Rueckert, ), while ContrastVAE employs a two-view approach with ContrastELBO for sequential recommendations (Wang et al., ). Noise contrastive estimation is used in (Aneja et al., ) to reweight the prior distribution. Contrastive VAEs (cVAE) focus on isolating salient features in datasets to refine latent space representation (Abid & Zou, ).

## 7 Conclusions

This research introduces the SimSiam Naming Game (SSNG) and SimSiam+VAE, a unified model that bridges discriminative contrastive SSL-based representation learning with generative CPC-based EmCom through the perspective of VI. Although originating from distinct domains, both SSL and EmCom share the goal of aligning representations—either by learning invariant representations from augmented data views in SSL or by developing a shared language between agents observing the same object from different perspectives. By bridging these objectives, our model demonstrates applicability to both representation learning and EmCom.

Our experiments show that SimSiam+VAE outperforms both SimSiam and VI-SimSiam in representation learning without requiring negative pairs. In EmCom, SSNG leverages the discriminative properties of SimSiam and the generative Bayesian perspective of the VI interpretation to align agents’ internal representations, fostering mutual understanding and enabling the development of an emergent language. This work, therefore, provides an alternative communication framework for EmCom systems.

#### Acknowledgments

## Appendix A Comparison among referential game, Metropolis-Hastings naming game and SimSiam naming game

**Table 3: Comparison between referential game (Lazaridou et al., ), Metropolis-Hastings naming game (MHNG) (Taniguchi et al., ), and SimSiam naming game (SSNG).**
| Aspect | Referential Game | Metropolis-Hastings (MH) Naming Game | SimSiam Naming Game (SSNG) |
| --- | --- | --- | --- |
| Objective | Develop emergent language (EmLang) to refer to shared objects or concepts, focusing on communication accuracy. | Develop EmLang through probabilistic updates, optimizing mutual understanding using MH algorithm. | Develop EmLang through self-supervised learning (SSL), focusing on similarity between representations of agents. |
| Communication method | Speaker sends a message to refer to a target object among distractors. | Agents exchange messages and update beliefs through acceptance rate based on MH algorithm. | Agents exchange messages to align and convey information based on representation similarity. |
| Learning Mechanism | Grounded in shared perception, where agents learn communication through feedback based on correct or incorrect reference selection. | Probabilistic updates of beliefs and message proposals using MH algorithm, incorporating joint attention. | Contrastive learning SSL via variational inference to align representations of agents, incorporating joint attention. |
| Agent Roles | A fixed speaker and a listener with distinct roles (describing and selecting objects). | Both agents are capable of proposing and evaluating messages iteratively to align their beliefs. | Both agents are capable of proposing and evaluating messages iteratively to align their latent representations. |
| Observations | Both agents refer to a single viewpoint of each object in the context. | Agents have different viewpoints or observations of the same object. | Agents have different viewpoints of observations of the same object. |
| Representation Space | Not a primary focus. | Continuous internal representation space, updated probabilistically through message exchanges. | Continuous internal representation space, aligned through maximizing similarity between different viewpoints. |
| Information Exchange | Messages are shared to refer to specific target objects. | Messages are exchanged and evaluated based on the MH acceptance rate. | Messages are exchanged and evaluated based on an SSL objective function. |
| Interaction Mode | One-way interaction: speaker sends a message, and listener interprets it to select the target object. | Iterative, bidirectional interaction: both agents propose and receive messages. | Iterative, bidirectional interaction: both agents propose and receive messages. |

## Appendix B SimSiam+VAE - Objective function

The objective function of SimSiam+VAE is derived as follows:

$$ $\displaystyle\mathcal{L}_{\text{SSL}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log\frac{p_{ \theta}({\mathbb{X}},z,w)}{q_{\phi}(z,w|{\mathbb{X}})}\right]$ (26) $\displaystyle:=\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log\frac{p_{ \theta}({\mathbb{X}}\mid z)p_{\theta}(z\mid w)p(w)}{q_{\phi}(w|z,{\mathbb{X}}) q_{\phi}(z|{\mathbb{X}})}\right]$ (27) $\displaystyle:=\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{\theta}({ \mathbb{X}}|z)+\log p_{\theta}(z|w)+\log p(w)-\log q_{\phi}(w|z,{\mathbb{X}})- \log q_{\phi}(z|{\mathbb{X}})\right]$ (28) $\displaystyle:=\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{\theta}({ \mathbb{X}}|z)\right]-\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log q_{ \phi}(z|{\mathbb{X}})\right]+$ $\displaystyle+\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{\theta}(z|w )\right]+\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p(w)-\log q_{\phi}( w|z,{\mathbb{X}})\right]$ (29) $$

Since $p_{\theta}({\mathbb{X}})$ is intractable, we approximate it with empirical data distribution $p_{D}({\mathbb{X}})$. Using Bayes’ theorem:

$$ $p_{\theta}({\mathbb{X}}|z)=\frac{p_{\theta}(z|{\mathbb{X}})p_{\theta}({\mathbb {X}})}{\mathbb{E}_{p_{\theta}({\mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]} \approx\frac{p_{\theta}(z|{\mathbb{X}})p_{D}({\mathbb{X}})}{\mathbb{E}_{p_{D}( {\mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]}$ (30) $$

then

$$ $\displaystyle\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{\theta}({ \mathbb{X}}|z)\right]$ (31) $\displaystyle\approx\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log\frac{p_{ \theta}(z|{\mathbb{X}})p_{D}({\mathbb{X}})}{\mathbb{E}_{p_{D}({\mathbb{X}})}[p _{\theta}(z|{\mathbb{X}})]}\right]$ (32) $\displaystyle\approx\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{ \theta}(z|{\mathbb{X}})+\log p_{D}({\mathbb{X}})-\log\mathbb{E}_{p_{D}({ \mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]\right]$ (33) $\displaystyle\approx\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{ \theta}(z|{\mathbb{X}})\right]-\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[ \log\mathbb{E}_{p_{D}({\mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]\right]+\log p _{D}({\mathbb{X}})$ (34) $$

Besides,

$$ $\displaystyle\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p(w)-\log q_{ \phi}(w|z,{\mathbb{X}})\right]$ (35) $\displaystyle=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\mathbb{E}_{q_{\phi}( w|z,{\mathbb{X}})}\left[\log p(w)-\log q_{\phi}(w|z,{\mathbb{X}})\right]\right]$ (36) $\displaystyle=-\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[D_{\text{KL}}\left(q _{\phi}(w|z,{\mathbb{X}})\parallel p(w)\right)\right]$ (37) $$

Substituting Eqs. (34) and (37) to Eq. (29), the objective function is:

$$ $\displaystyle\mathcal{J}_{\text{SSL}}$ $\displaystyle\approx\mathcal{J}_{\text{align}}+\mathcal{J}_{\text{recon}}+ \mathcal{J}_{\text{uniform}}+\mathcal{J}_{\text{KL}}+\log p_{D}(X)$ (38) $\displaystyle\approx\mathcal{J}_{\text{align}}+\mathcal{J}_{\text{recon}}+ \mathcal{J}_{\text{uniform}}+\mathcal{J}_{\text{KL}}$ (39) $$

where

$$ $\displaystyle\mathcal{J}_{\text{align}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\log p_{\theta}(z|{ \mathbb{X}})\right]-\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\log q_{\phi}(z |{\mathbb{X}})\right]$ (40) $\displaystyle\mathcal{J}_{\text{recon}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z,w|{\mathbb{X}})}\left[\log p_{\theta}(z| w)\right]$ (41) $\displaystyle\mathcal{J}_{\text{uniform}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[-\log p_{D}(z)\right]$ (42) $\displaystyle\mathcal{J}_{\text{KL}}$ $\displaystyle:=-\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[D_{\text{KL}}\left( q_{\phi}(w|z,{\mathbb{X}})\|p(w)\right)\right]$ (43) $\displaystyle p_{D}(z)$ $\displaystyle:=\mathbb{E}_{p_{D}({\mathbb{X}})}[p_{\theta}(z|{\mathbb{X}})]$ (44) $$

In SimSiam+VAE, we define $p(w)$, $p(z)$, $q_{\phi}(w|z)$, $p_{\theta}(z|w)$, $q_{\phi}(z|{\mathbb{X}})$, and $p_{\theta}(z|{\mathbb{X}})$ as mentioned in Eqs. (13), (14), (15), (16), (17), (18), respectively.

#### Alignment loss

$$ $\displaystyle\mathcal{J}_{\text{align}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\log p_{\theta}(z|{ \mathbb{X}})-\log q_{\phi}(z|{\mathbb{X}})\right]$ (45) $\displaystyle:=\frac{1}{M}\sum_{j=1}^{M}\left[\log p_{\theta}(f_{\phi}(x_{j})| {\mathbb{X}})-\log q_{\phi}(f_{\phi}(x_{j})|{\mathbb{X}})\right]$ (46) $\displaystyle:=\frac{1}{M}\sum_{j=1}^{M}\left[\log\left(\eta_{\theta}\prod_{i= 1}^{M}\text{vMF}(f_{\phi}(x_{j});\mu_{z}=g_{\theta}(x_{i}),\kappa_{z})\right)- \log\frac{1}{M}\right]$ (47) $\displaystyle:=\frac{1}{M}\sum_{j=1}^{M}\left[\log\eta_{\theta}+\log M+\sum_{i =1}^{M}\log\text{vMF}(f_{\phi}(x_{j});\mu_{z}=g_{\theta}(x_{i}),\kappa_{z})\right]$ (48) $\displaystyle\mathcal{J}_{\text{align}}$ $\displaystyle\approx\sum_{i,j}\left(g_{\theta}(x_{i})^{\top}f_{\phi}(x_{j})\right)$ (49) $$

#### Reconstruction loss

$$ $\displaystyle\mathcal{J}_{\text{recon}}$ $\displaystyle:=\mathbb{E}_{q_{\phi}(z|{\mathbb{X}})}\left[\mathbb{E}_{q_{\phi} (w|z)}\left[\log p_{\theta}(z|w)\right]\right]$ (50) $$

The inner term $\mathbb{E}_{q_{\phi}(w|z)}\left[\log p_{\theta}(z|w)\right]$ represents the reconstruction loss in the VAE component. In representation learning, this loss can be approximated by:

$$ $\displaystyle\mathbb{E}_{q_{\phi}(w|z)}\left[\log p_{\theta}(z|w)\right] \approx(z^{\prime})^{\top}z=g_{\theta}(x)^{\top}f_{\phi}(x)$ (51) $$

where $z^{\prime}$ denotes the reconstructed representation obtained from the latent variable $w$. This approximation captures the alignment between the original and reconstructed representations in the representation space. Thus,

$$ $\displaystyle\mathcal{J}_{\text{recon}}\approx\sum_{i}\left(g_{\theta}(x_{i})^ {\top}f_{\phi}(x_{i})\right)$ (52) $$

The reconstruction loss $\mathcal{J}_{\text{recon}}$ measures the alignment between the reconstructed representation $g_{\theta}(x_{i})$ and the original one $f_{\phi}(x_{i})$. This alignment is already captured by the $\mathcal{J}_{\text{align}}$. Hence, $\mathcal{J}_{\text{recon}}$ is omitted from the total loss.

#### Uniform loss

The role of $\mathcal{J}_{\text{uniform}}$ is to ensure that the marginal distribution $p_{D}(z)$ is uniform over the hypersphere, i.e., $p_{D}(z)=\mathcal{U}(S^{d-1})$. However, the predictor $h$, defined as a DirectPred (Tian et al., ), ensures that the latent representations $z$ are uniformly spread over the hypersphere. It achieves this by making the distribution of $z$ approximately isotropic, with each dimension being independent and having equal variance. Consequently, $h$ implicitly maximizes $\mathcal{J}_{\text{uniform}}$ (Nakamura et al., ).

Since the predictor already encourages a uniform distribution of the representations, explicitly including $\mathcal{J}_{\text{uniform}}$ in the total loss is redundant. Therefore, it can be omitted without losing the intended effect on the representation distribution.

#### KL divergence

Since each representation $z$ is derived from the same network with a stop-gradient operation, the KL divergence can be simplified as:

$$ $\displaystyle\mathcal{J}_{\text{KL}}$ $\displaystyle\approx-\sum_{i}D_{\text{KL}}\left(q_{\phi}(w|z,x_{i})\|p(w)\right)$ (53) $$

#### Total loss

Combining these components, the objective function of SimSiam+VAE is given by:

$$ $\displaystyle\mathcal{J}_{\text{SSL}}\approx\sum_{i,j}\left(g_{\theta}(x_{i})^ {\top}f_{\phi}(x_{j})\right)-\beta\sum_{i}D_{\text{KL}}\left(q_{\phi}(w|z,x_{i })\|p(w)\right)$ (54) $$

## Appendix C SimSiam Naming Game - Objective function

In the SimSiam naming game with two agents, A and B, the total loss function $\mathcal{J}_{\text{SSL}}$, derived from the objective function of SimSiam+VAE, is adapted to account for each agent’s individual observations and representations. Unlike the original SimSiam+VAE, the SSNG separates $z$ into two latent representations, $z_{A}$ and $z_{B}$, one for each agent. Each agent $*$ receives a unique observation $x_{*}$, which is encoded into a representation $z_{*}$, and subsequently mapped to a shared latent variable $w$. The total loss is reformulated as:

$$ $\displaystyle\mathcal{J}_{\text{SSNG}}\approx\sum_{i,j}\left(g_{\theta}(x_{i}) ^{\top}f_{\phi}(x_{j})\right)-\beta\sum_{i}D_{\text{KL}}\left(q_{\phi}(w|z_{A} ,z_{B},x_{i})\|p(w)\right)$ (55) $$

This loss consists of the optimization process for both agent A and B. Therefore, the total loss can be decomposed into contributions for each agent:

$$ $\displaystyle\mathcal{J}_{\text{SSNG}}=\mathcal{J}_{A}+\mathcal{J}_{B}$ (56) $$

where $\mathcal{J}_{A}$ and $\mathcal{J}_{B}$ represent the loss functions for agent A and agent B, respectively:

$$ $\displaystyle\mathcal{J}_{A}$ $\displaystyle\approx g_{B}(x_{B})^{\top}f_{A}(x_{A})-\beta D_{\text{KL}}\left( q_{A}(w|z_{A},z_{B},x_{A})\|p(w)\right)$ (57) $\displaystyle\mathcal{J}_{B}$ $\displaystyle\approx g_{A}(x_{A})^{\top}f_{B}(x_{B})-\beta D_{\text{KL}}\left( q_{B}(w|z_{A},z_{B},x_{B})\|p(w)\right)$ (58) $$

In the objective function of SimSiam+VAE, the parameters $\theta$ and $\phi$ are shared across all observations. When this objective is split into agent-specific losses, these parameters become agent-specific versions: $\theta_{A}$, $\phi_{A}$ for agent A and $\theta_{B}$, $\phi_{B}$ for agent B. For simplicity, we denote the functions with these parameters as $f_{A}$, $f_{B}$, etc., where the subscript “A” or “B” indicates the respective agent.

In this communication game, agents A and B alternately take on the roles of speaker $(Sp)$ and listener $(Li)$, with possible role pairs $(Sp,Li)\in\{(A,B),(B,A)\}$. Given the listener $(Li)$ and the message $w_{Sp}$ received from the speaker $(Sp)$, the objective function of the listener is given by:

$$ $\displaystyle\mathcal{J}_{Li}\approx g_{Sp}(x_{Sp})^{\top}f_{Li}(x_{Li})-\beta D _{\text{KL}}\left(q_{Li}(w_{Li}|z_{Li},z_{Sp},x_{Li})\|p(w_{Li})\right)$ (59) $$

In EmCom, agents are unable to observe each other’s internal concepts, much like humans cannot directly access one another’s thoughts. Therefore, the listener cannot access the speaker’s function $g_{Sp}$. Instead, the listener interprets the message received from the speaker using its own decoder. To do this, we start from each agent’s function $g_{*}$ which is composed as follows:

$$ $g_{*}=h_{*}^{\text{(dec)}}\circ h_{*}^{\text{(enc)}}\circ f_{*}$ (60) $$

where:

- •
$f_{*}$ is the perception, consisting of a backbone and projector, processing the observation $x_{*}$ to obtain the internal representation $z_{*}$.
- •
$h_{*}^{\text{(enc)}}$ is the encoder of the language coder $h_{*}$, mapping the representation $z_{*}$ to the latent variable $w_{*}$.
- •
$h_{*}^{\text{(dec)}}$ is the decoder of the language coder $h_{*}$, reconstructing a representation $z^{\prime}_{*}$ from the received message $w_{Sp}$.

As described in Section 4.3, the SSNG is follows these steps:

- •
The speaker generates a message $w_{Sp}$ from its observation $x_{Sp}$
$w_{Sp}=h_{Sp}^{\text{(enc)}}(f_{Sp}(x_{Sp}))$
(61)
- •
The message $w_{Sp}$ is then transmitted to the listener, who decodes it to produce a reconstructed representation $z^{\prime}_{Sp}$:
$z^{\prime}_{Sp}=h_{Li}^{\text{(dec)}}(w_{Sp})$
(62)

Since the listener cannot access the speaker’s component $g_{Sp}$, it uses the reconstructed representation $z^{\prime}_{Sp}$ to interpret the speaker’s intent. Thus, the function $g_{Li}$, which reflects the listener’s interpretation, is composed as:

$$ $g_{Li}=h_{Li}^{\text{(dec)}}\circ h_{Sp}^{\text{(enc)}}\circ f_{Sp}$ (63) $$

Besides, since the listener cannot access the speaker’s internal representation $z_{Sp}$, the $D_{\text{KL}}$ will be calculated based on its own $z_{Li}$. As a result, the loss function for the listener is reformulated as:

$$ $\displaystyle\mathcal{J}_{Li}\approx[h_{Li}^{\text{(dec)}}(w_{Sp})]^{\top}f_{ Li}(x_{Li})-\beta D_{\text{KL}}\left(q_{Li}(w_{Li}|z_{Li},x_{Li})\|p(w_{Li})\right)$ (64) $$

By this formulation, the listener’s loss emphasizes how well it can decode the speaker’s shared message $w_{Sp}$ using its own representations, as well as regularizing its own latent space via the KL divergence. This captures the partial observability and the need for the listener to independently infer and interpret the shared emergent language.

## Appendix D Pseudocodes of SimSiam+VAE

Figure: Algorithm 1 Pseudocode of SimSiam+VAE, PyTorch-like

## Appendix E Pseudocodes of SimSiam naming game

Figure: Algorithm 2 Pseudocode of SimSiam naming game, PyTorch-like

## Appendix F Experiment 1 - SimSiam+VAE in Representation Learning

### Datasets:

- •
FashionMNIST (Xiao et al., ) contains 70,000 grayscale images, each of size 28x28, representing 10 classes of objects with 60,000 training and 10,000 testing images.
- •
CIFAR-10 (Krizhevsky, ) is a collection of 60,000 color images, each of size 32x32 and belonging to one of 10 different classes with 50,000 training and 10,000 testing images.

### Model Architecture:

- •
Backbone network:
–
FashionMNIST Backbone: A custom CNN with two convolutional layers: the first outputs 16 channels (kernel size 4, stride 2, padding 1), and the second doubles the channels. A fully connected layer maps the features to 512 dimensions.
–
CIFAR-10 Backbone: ResNet18 He et al. ().
- •
Projector: A three-layer MLP with batch normalization projects the backbone features to a latent space (128 for FashionMNIST, 256 for CIFAR-10).
- •
Predictor: An encoder-decoder MLP pair:
–
Encoder: Reduces latent dimensions to 64 (FashionMNIST) or 128 (CIFAR-10).
–
Decoder: Reconstructs the latent dimension with sigmoid activation.

### Training Setup:

The model is trained with a batch size of 128, learning rate of 1e-3, and Adam optimizer (weight decay 1e-5). A StepLR halves the learning rate every 10 epochs over 500 epochs. A linear classifier is trained on the frozen representations, and performance is evaluated using Top-1 accuracy for FashionMNIST and Top-2 accuracy for CIFAR-10.

### Comparison Models:

We compare SimSiam+VAE against SimSiam and VI-SimSiam, all using the same backbone and projector for fair comparison, with consistent seeding. The predictor encoder of SimSiam+VAE is similar to SimSiam and VI-SimSiam. VI-SimSiam adds a predictor encoder for kappa estimation, while SimSiam+VAE includes a predictor decoder to reconstruct $z^{\prime}$ from the variable $w$.

## Appendix G Experiment 2: SimSiam Naming Game in EmCom

### Datasets:

We use the dSprites dataset (Matthey et al., ), which contains 2D shape images that vary across different generative factors (e.g., shape, scale, orientation).

### Model Architecture:

- •
Backbone: is an MLP with three linear layers: the input dimension of 4096 is reduced to 512, then 256, and finally 128. Each layer, except the last, is followed by batch normalization and ReLU activation.
- •
Projector: A multi-layer perceptron (MLP) projects the feature dimension (128) to a latent dimension of 256 through three fully connected layers. Each layer is followed by batch normalization and ReLU, except for the final layer.
- •
Predictor: (Language Coder) handles message generation and reconstruction (Fig. 4):
–
Encoder: An LSTM autoregressively generates a sequence of tokens, each sampled from a discrete distribution over a vocabulary of size. At each step, the LSTM produces logits, one-hot vectors, and messages by taking the embedding of the previously generated token as input.
–
Decoder: Another LSTM takes these one-hot token embeddings and processes the sequence to produce a final hidden state that reconstructs the original representation.

Figure: Figure 4: Architecture of the language coder comprising an LSTM-based encoder-decoder structure. The LSTM Encoder generates a sequence of discrete tokens to form the message $w=(m1,m2,...,mN)$ from a representation $z$ in an autoregressive manner, where each token is sampled from a discrete distribution over the vocabulary $V$. The LSTM Decoder then reconstructs a new representation $z^{\prime}$ from the sequence of token embeddings, with its final hidden state serving as the output.
Refer to caption: extracted/5960865/imgExpLSTM.jpg

### Training Setup:

The vocabulary size $|V|$ is 100, and the message length is 10. The model is trained with a batch size of 256 and a learning rate of 1e-5, using an Adam optimizer. The learning rate is scheduled to decay by half every 10 epochs using a StepLR scheduler. The model is trained for 1000 epochs.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Can you run on ImageNet and compare to known SimSiam numbers?
What benefits of the VAE can you *quantitatively* show?
soundness: 2
strengths: Integrating discretization and other inductive biases in SSL is an interesting research direction and the authors somewhat approach this.
summary: SimSiam is a simple SSL objective for images. VI-SimSian (Nakamura et al, 2023) is an extension that replaces the single predictor network with predictor of mean and variance to account for uncertainty. This paper proposes SimSiam+VAE, which extends VI-SimSiam by sampling from a gaussian with the predicted mean and variance. On FashionMNIST and CIFAR10, they show slightly improved performance over SimSiam and SimSiam-VI 


The authors also propose an emergent communication setup SimSiam Naming Game. This uses a discrete latent variable and is trained using the Gumbel-Softmax Straight-Through (which maybe doesn't make it a VAE?). They show comparable performance to a regular emergent communication setup on a toy task, dsprites.
weaknesses: This paper never really explains any motivation for adding a VAE into SimSiam and conducts experiments on toy tasks that are not sufficient to empirically demonstrate performance. Furthermore, the paper fails to cite very related work while being blatant in unrelated self-citation, in a way that arguably de-anonymizes the authors. 

The only two SSL datasets are CIFAR10 and FashionMNIST and the only baselines are SimSiam, and VI-SimSiam. Work in image SSL should *at minimum* demonstrate results on ImageNet, as done in VI-SimSiam. 

Emergent communication results are on little-used dSprites dataset with LSTMs for encoding/decoding. This is very dated and the authors don't even report communication success, the most basic metric for this task. They only show topographic similarity of the message space and find equivalent results to the most simple baseline. 

The connection of using SSL methods for emergent communication methods (and vice-versa) was notably done by [Dessi et al, 2021](https://arxiv.org/abs/2106.04258) and feels like a key citation left out. Not as important, but related, is [Lo et al, 2024](https://arxiv.org/abs/2307.01403) that showed an equivalence between contrastive SSL and learning emergent communication in situated MARL. On top of Jang et al (2017) Gumbel-Softmax, you should also cite Concrete Distribution [(Madison et al, 2017)](https://arxiv.org/abs/1611.00712) as it was simultaneous discovery. 

(Very probable) self-citation is blatant in the related work section on Emergent Communication. Barely related work on MHNG is cited and it is never explained how it relates to the paper at hand e.g. Citing Inukai et al on recursive multi-agent systems using MHNG. This isn't a huge issue but does look bad.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Representation learning results: how does the model perform re representation learning on more challenging datasets / classification tasks like ImageNet. How does the model perform compared to non-SimSiam approaches?

Uncertainty estimation: Are the uncertainty estimates learned from the model accurate and/or reflect something useful or interesting (e.g. relating to epistemic or aleatoric uncertainty)?

How does the model work: given that the stop gradient helps prevent collapse in SimSiam, and the authors speculate that use of a VAE will also help with this, what is the effect of removing the stop gradient in SimSiam+VAE? What is the effect of modifying the weightings on the loss terms, and what can this tell us about how adding a VAE is benefiting the approach. Why did the authors choose different latent distributions in for the representation learning versus emergent communication experiments? Are there other latent distributions that would make sense to use? Other priors that would make sense for the KL divergence term?
soundness: 2
strengths: Originality and significance: integrating a VAE into SimSiam is novel and builds on previous work expressing SimSiam and similar methods from a Bayesian and variational perspectives. It seems like a well motivated direction: Firstly, the goal of estimating uncertainty with representations is a useful one when thinking about actual applications of the learned representations. Secondly, the particular learning regime in SimSiam – learning to predict what the representation of a similar data point might be – seems less well-posed without explicitly casting that prediction in terms of uncertainty. Finally, the application of this method in an emergent communication framework is an interesting direction and a less explored application.

Quality and clarity: the paper is well written and presented, with a nice flow from motivation through methodology to experiments and places the contributions into the context of prior and related works. Diagrams and schematics are clear and help the reader visualise and understand what's being discussed. And there’s a good level of detail, with all the important elements described for understanding and reproducibility as well as concise supporting code.
summary: The authors introduce a model for a contrastive self-supervised learning called SimSiam+VAE that builds on previous work, SimSiam and VI-SimSiam. In common with SimSiam approaches, the goal is to learn a representation where similar datapoints are close together (by learning from positive examples alone).

The authors extend previous approaches by integrating a Variational Autoencoder (VAE) into the predictor, enabling the model to better capture uncertainty. Furthermore, by including a VAE, and its KL divergence term, the approach may aid representation learning by regularising the learned representation, which should help prevent collapse and encourage a better spread of representations across the latent space. Building on the model, the authors develop a framework for emergent communication called the SimSiam Naming Game (SSNG), where agents iteratively align their internal representations and develop a shared language

They show results where SimSiam+VAE outperforms SimSiam and VI-SimSiam in experiments on representation learning and emergent communication tasks.
weaknesses: 1) Focus on limited and less challenging datasets: given the motivation and promise of the approach, it’s surprising that the experiments are limited to simpler datasets/tasks like FashionMNIST and CIFAR-10. The original SimSiam work was tested on ImageNet so this would be a great candidate for expanding on the experimental results. If the approach was found to struggle on such data & tasks, it would be useful to discuss and explore this. Similarly, comparing the results to other non-SimSiam representation learning approaches on the same datasets would help put the work into a wider context.
2) Rationale: the paper introduces a VAE to model representation uncertainty and improve the learned representations. It could provide a clearer rationale for why a VAE as opposed to other probabilistic models is particularly suited for this in SimSiam. Similarly, the work discusses uncertainty but doesn’t fully explore how this aspect could be applied in downstream tasks where uncertainty estimation would be beneficial, such as anomaly detection or active learning. The work also does not explore the actual performance of the model's uncertainty outputs. 
2) Lack of ablations and variations that explore if and how the model benefits from the VAE: some ablation or other experimental variants that help explore and dig into the workings of the model would add to the understanding of the model (also see questions).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Is there any specific improvement to VAE that makes it suitable for the proposed method SimSiam+VAE?
2. How would you conclude the core contributions at a theoretical level?
3. Could you provide more explanations for Table 2?
4. In line 81, there should be a space before "O" and a hyphen after it.
soundness: 2
strengths: 1. It is insightful to link representation learning, such as VI-SimSiam, with emergent communication issues. This approach highlights the connection between these fields and opens new research avenues for training agents to develop shared languages. The idea is both novel and inspiring.
2. The presentation is clear, and the proofs are thorough. The authors have effectively organized the content, making the methodology easy to follow.
summary: The paper adopts self-supervised learning to learn emergent communication, enabling agents to develop a shared language. It first enhances self-supervised representation learning VI-SimSiam to propose SimSiam+VAE. This method is then applied to a communication framework called the SimSiam Naming Game (SSNG) to learn emergent communication. The experiments demonstrate that both components achieve comparable or improved performance compared to the baselines.
weaknesses: 1. With the development of LLMs, communication between agents can now be conducted in natural language, which may offer new pathways beyond traditional methods. However, the authors do not seem to mention this in the paper, which may undermine the significance of the study.
2. The motivation for and effectiveness of VAE are not clearly articulated. In the current version, while it may be natural for the audience to assume that VAE could provide a latent representation for emergent communication, the advantages over VI in the representation learning aspect are not specified. Additionally, there is no experimental analysis comparing the performance of the two methods.
3. Although I do not consider state-of-the-art performance as the sole criterion for evaluating the paper, it is concerning that SSNG achieves comparable performance on dSprites without a detailed explanation.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - Why use top-2 accuracy instead of top-1 for CIFAR-10? I believe the standard is to report top-1 accuracy.
- In alg. 2 (line 16), what would be the second argument to the KL term?
soundness: 2
strengths: The idea of incorporating uncertainty into representation learning with SimSiam is interesting and holds many potential advantages. For example, a probabilistic formulation can be useful in handling stochastic environments. Current SSL methods are not readily able to handle any stochasticity in the data.
summary: The paper proposes SimSiam+VAE, a model that integrates a Variational Autoencoder (VAE) with the SimSiam framework to incorporate uncertainty in representation learning. This model is extended into the SimSiam Naming Game (SSNG), a framework for emergent communication where agents align their representations and develop a shared language through iterative communication. The approach demonstrates improved performance in both representation learning (on CIFAR10 and FashionMNIST) and emergent communication tasks, outperforming related methods in experiments.
weaknesses: - The paper seems to be exploring two directions: 1) Representation Learning and 2) Emergent Communication. However, it seems that both these directions are not concretely explored or discussed. 
- In representations learning, only CIFAR-10 and FashionMNIST are used which are relatively simple and small dataset and its hard to see why SSL would be applicable to such small datasets. Also most standard works use imagenet for comparing SSL algorithms including the VI-Simsiam paper on which this paper builds.
- It would be useful to add a more thorough discussion of emergent communication literature and relevant baselines. Currently, the paper only uses dsprites which is a relatively simple environment. It would be useful to explore more complex environments such as clevr. Moreover, I find that the SimSiam naming game model is much different from the SimSiam VAE model. In the naming game model, it seems that the predictor outputs a distribution over discrete tokens which is different from what SimSiam + VAE does where the predictor outputs the mean and variance of a gaussian. Hence, I do not see how these two model can be considered as a single approach or how SSNG can be considered as an extension of SimSiam + VAE.
- It would be useful to provide a brief description of the TopSim metric in the paper itself.