SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Second, since cost matrices in a relative entropy regularized Optimal Transport problem can be freely "shifted" without affecting the optimal plan (Nutz, 2022), we can replace  $s_T$  with any  $|D|$ -length vector and  $s_L$  with any  $|H|$ -length vector. In particular, we could consider  $-\log \varphi$  and  $-\log \psi$  as
4 In autoencoding, the space to be encoded  $X$  is called the data space and the space to be decoded  $Z$  is called the latent space. In cooperative communication, the space to be encoded  $H$  is called the hypothesis space and the space to be decoded  $D$  is called the data space. Thus, the name data is given to the opposite objects in these two worlds. We point this out explicitly to hopefully avoid any confusion that may arise from these conventions.
replacements for  $s_T$  and  $s_L$  respectively, and replace the cost matrices  $c_i$  with the single cost matrix  $-\log M$ . Similarly, by adding  $\log \varphi$  to  $-\log M$ , the cost matrix  $-\log M$  defines the same entropy regularized Optimal Transport problem as the cost matrix  $c(d,h) := -\log \rho(h|d)$ .
Remark 2.7. In turn, cooperative communication under strong common ground, while formulated as two (relative) entropy regularized Optimal Transport problems, is actually just one. This is a rigorous manner in which to see that previous models of cooperative communication work under the strongest possible assumption of common ground wherein there is effectively only one agent.
In our formulation, if we take  $D$  and  $H$  to be discrete,  $\mathcal{P}_{\mu}^{L} = \{\rho (h|d)\mu (d)\}$ ,  $\mathcal{P}_T^\nu = \Pi_\mu^\nu$ , and  $\delta = 0$ , we recover the a posteriori single (from the omniscient teacher's perspective) entropy regularized Optimal Transport problem that governs cooperative communication in prior work. These restrictions provide us with one manner in which to recover previous models of cooperative communication from our model. Another way is to pass to unbalanced entropy regularized Optimal Transport (Séjouré et al., 2023), by taking  $\mathcal{P}_T^\nu = \Pi^\nu$  and  $\delta = k\in \mathbb{N}$ , and then take the limit as  $k$  tends to infinity.
Theorem 2.8. Let  $D$  and  $H$  be discrete spaces, fix  $\epsilon \geq 0$ , and consider the common ground pair  $(\Pi^{\nu}, \{\rho(h|d)\mu(d)\})$  for some  $(|D| \times |H|)$ -row stochastic matrix  $\rho$  with positive entries. Let  $(P_k, \rho(h|d)\mu(d))$  be an optimal pair for (2.3) with  $\delta = k \in \mathbb{N}$ . Then,  $P_k$  converges to  $P_{\infty}$ , and  $P_{\infty}$  minimizes (2.4), for  $i = T$  with  $c_T = -\log \rho$ .
This manner of recovering previous models of cooperative communication takes advantage of the ability to decompose a communication plan into a conditional plan and a prior (cf. Definitions 2.4 and 2.5), a major practical benefit when working with the variational model Equation 2.3.
In this section, we implement a further approximation scheme to study Equation 2.3 empirically. We sample initializations for a gradient descent based optimization scheme of  $\mathrm{L}_{\epsilon,\delta}$ , in PyTorch via Adam (Kingma and Ba, 2015), over  $(\mathcal{P}_\theta^g, \mathcal{P}_f^\lambda)$  from a pair of probability distributions  $\Theta$  and  $\Lambda$  on the parameters  $\theta$  and  $\lambda$ . We treat every parameter independently and assume that  $\Theta$  and  $\Lambda$  are the product of independent uniform distributions over an interval  $[-p,p]$ , one for each parameter. Sample initializations empirically explore common ground and the entire space of admissible communication plans. Averaging over the limiting pairs of communication plans, one for each initialization, yields an approximation to a minimizing pair of communication plans of our loss functional over the given common ground pair. In particular, we consider a multilayer perceptron-based form of common ground in which  $\theta$  and  $\lambda$  are neural network parameters. (We provide the full details regarding experimental setup and implementation in the Supplementary Materials.) We conduct experiments under various priors  $f$  and  $g$  and compare different initializations. We also analyze our model through variations on the coefficients  $\epsilon$  and  $\delta$ . These experiments are carried out in fully discrete and semi-continuous settings. As is typical, we represent discrete spaces as collections of one hot vector.
3.1 The Fully Discrete Setting
Here we assume that  $m = |D|$  and  $n = |H|$ , and we fix our common ground pair as follows:  $\mathcal{P}_{\theta}^{g} = \{g^{\theta}(d|h)g(h)\}$  with  $g^{\theta}(d|h) = \mathrm{Cat}(d|\alpha_{\theta}(h))$  and  $g(h) = \mathrm{Cat}(d|\alpha)$  and  $\mathcal{P}_f^\lambda = \{f^\lambda (h|d)f(d)\}$  with  $f^{\lambda}(h|d) = \mathrm{Cat}(d|\beta_{\lambda}(h))$  and  $f(d) = \mathrm{Cat}(d|\beta)$ . As mentioned previously,  $\theta$  and  $\lambda$  are parameters of neural networks. For simplicity, we assume that  $m = n$ . So  $\alpha = \beta = (1 / m,\dots ,1 / m)$ .
First, let us recall the cooperative index of two communication plans (Yang et al., 2018):  $\mathrm{CI}(P_{\theta}, P_{\lambda}) := \frac{1}{n} \sum_{i,j} g^{\theta}(d_i \mid h_j) f^{\lambda}(h_j \mid d_i)$ . It ranges between 0 and 1; the higher the better. The cooperative index is an established measure of the effectiveness of communication in fully discrete cooperative communication, and so, it is an important metric to consider. While we have already demonstrated that the cooperative communication as studied by Yang et al. (2018) is contained within our framework, we elaborate on this further in the Supplementary Materials.
In Figure 3.1, we study the effects of various initialization with respect to values of  $p$ , metrics ( $\ell_1$  and cooperative index), and values of  $\epsilon$ . Under the common ground pair fixed in this section, we find a single unique minimizing pair for all  $\epsilon > 1$  and  $\delta \geq 0$ . Hence, we do not consider any  $\epsilon > 1$ .
The two values of  $p$  considered here are 0.0625 and 0.5. The first choice is referred to as a uniform initialization in Figure 3.1 and corresponds to the lower row; the second choice is referred to as a sparse initialization in Figure 3.1 and corresponds to the upper row.<sup>5</sup>
Columns (a) and (b) consider variability with respect to the size of our spaces  $m$  with  $\epsilon = 1$  and  $\delta = 0$ . With this choice of coefficient pair and our fixed common ground pair, we find that minimizing pairs are formed by a single matrix paired with itself. Hence,  $\delta$  plays no role, and taking  $\delta = 0$  is equivalent to taking  $\delta > 0$ . Column (a) plots changes in the pairwise  $\ell_1$ -distances between these single matrix minimizers as a function of  $m$ . Column (b) plots changes in the cooperative index of these single matrices paired with themselves, again, as a function of  $m$ . We sample each of the two distributions (uniformish and sparse) 50 times to obtain 50 initializations of the conditional teaching and learning plans. In particular, for each  $m \in \{5, 10, 20, 40, 80\}$ , the error bar plots the sample standard deviation of the resulting  $\ell_1$ -distances/cooperative indices and the central dot the mean of these resulting  $\ell_1$ -distances/cooperative indices, for that  $m$ .
Columns (c) and (d) consider variability with respect to values of  $\epsilon$ , which ranges between 0 and 1 with a step size of 0.1 and  $m = 20$ . The two distributions (uniformish and sparse) here are sampled 100 times to obtain 100 initializations of the conditional teaching and learning plans. Here our gradient descent limits are pairs of distinct matrices, whose  $\ell_1$ -difference is plotted in column (c). Column (d) plots their cooperative indices. The error bars here are the standard errors of the means and the central dots are the means as determined by our 100 initializations.
Figure 3.1: Columns from left to right: (a) pairwise  $\ell_1$ -difference with  $\epsilon = 1$  as function of  $m$ ; (b) CI with  $\epsilon = 1$  as function of  $m$ ; (c) pairwise  $\ell_1$ -norm as a function of  $\epsilon$ ; (d) CI as a function of  $\epsilon$ . We take (a) as a verification of the existence of multiple minimizers when  $\epsilon = 1$  even under rather a structured form common ground (if not the pairwise  $\ell_1$ -differences would all be 0). From (b), we see that sparse initializations yield higher cooperative indices. From (c) and (d), display trends of between the  $\ell_1$ -distances between the teacher's and learner's optimal plans as  $\epsilon$  increases. Since  $\epsilon = 1$  suggests a regime changes, this trend is roughly decreasing for  $\ell_1$ -distances and increasing for cooperative index when  $\epsilon < 1$ .
3.2 The Semi-Continuous Setting
Here we assume that  $|D| = m$  and  $H = \mathbb{R}^n$ , and we fix our common ground pair to be  $\mathcal{P}_{\theta}^{g} = \{g^{\theta}(d|h)g(h)\}$  with  $g^{\theta}(d|h) = \operatorname{Cat}(d|\alpha_{\theta}(h))$  and  $g(h) = \sum_{i}\phi_{i}\mathrm{N}(h|\mu_{i},\Sigma_{i}^{2})$  and  $\mathcal{P}_f^\lambda = \{f^\lambda (h|d)f(d)\}$  with  $f^{\lambda}(h|d) = \mathrm{N}(u_{\lambda}(d),\sigma_{\lambda}^{2}(d))$  and  $f(d) = \operatorname{Cat}(d|\alpha)$ ;  $\theta$  and  $\lambda$  are parameters of neural networks. More specifically,  $g$  is a mixture of  $l$  Gaussians, for  $l = 1,3,5$ , with
5 When  $p = 0.0625$ , for every  $h$ ,  $g^{\theta}(\cdot |h)$  at initialization will be close to a uniform distribution because outputs of shallow neural networks are small given small weights and biases. But if  $p = 0.5$ , these initial conditional probabilities will become sparse.
mean  $\mu_{i}$  and variance  $\Sigma_i^2$  sampled from the standard Gaussian, for each  $i\in \{1,\ldots ,l\}$ . We assume that  $m = 100$  and  $\alpha = (0.01,0.01,\dots,0.01)$ . We begin with two definitions.
Definition 3.1 (Reconstruction). For  $h \in H$ , we let  $\hat{h} \coloneqq \frac{1}{m}\sum_{i}g^{\theta}(d_i|h)\mu_{\lambda}(d_i)$ , where  $\mu_{\lambda}(d_i)$  is the mean of  $f^{\lambda}(h|d_i)$ , denote its reconstruction. The average reconstruction loss of a set of samples  $\{h_i\}_{i \in I}$  is the sample mean of the  $\ell_1$ -difference between  $h_i$  and its reconstruction:  $\frac{1}{|I|}\sum_{i}\| h_i - \hat{h}_i\|_{\ell_1}$
Definition 3.2 (Data Activation). A data  $d$  is activated by a hypothesis  $h$  if  $d = \operatorname{argmax}_D g^\theta(\cdot|h)$ . The data activation percentage is the ratio of activated data (for some  $h$ ) over all data. $^6$
Remark 3.3. These two definitions provide one solution to the problem raised in Remark 2.3; this solution comes from variational autoencoding.
In Figure 3.2, we restrict our attention to case where  $g$  is a mixture of 3 Gaussians,  $\epsilon = 1$ , and  $\delta = 10$ . We sample 5 different initializing weights and biases for both the conditional teaching and learning plans. The value of  $p$  starts at 0.01 and increases by 0.01 until 0.8. We plot changes in average reconstruction loss and activation percentage as a function of  $p$ . The error bars correspond to the initialization sample standard deviation and central dots correspond to the initialization sample means. As  $p$  increases, standard deviation also increases, consistent with reduced common ground.
In Figure 3.3, we fix  $p = 0.0625$  and sample 10 initializations of our neural network parameters. We consider three cases corresponding to the number of Gaussians which determine  $g$ . Error bars and central dots display the standard deviation and mean of these sampled initializations with
respect to either average reconstruction loss or activation percentage.
Figure 3.2: Variation in initialization analysis. As  $p$  increases, average reconstruction loss increases and data activation percentage decreases.
Figure 3.4 plots reconstructions and data activation. The coefficient  $\epsilon$  and  $\delta$  are picked by the best reconstruction effectiveness, and an initialization is sampled at random with  $p = 0.0625$  (cf. Figure 3.3). The size of the sample set  $I$  is 6000. Column (a) plots samples in blue and their reconstructions in orange. Column (b) again plots samples in blue. The orange dots now are the means of  $f^{\lambda}(\cdot |d)\mathrm{s}$  for activated data  $d$ ; the green dots are the means of  $f^{\lambda}(\cdot |d)\mathrm{s}$  for non-activated data  $d$ . Column (c) partitions each sampled hypothesis according to the mean associated with the data it activates. These means, colored in orange in (b), are colored in orange and outlined in black here. These plots suggest that after optimizing in  $\epsilon$  and  $\delta$ , successful cooperative communication, defined through reconstruction coverage and continuity, is possible.
Our framework generalizes a broad class of models that have been proposed in linguistics (Frank and Goodman, 2012), cognitive development (Jara-Ettinger et al., 2015), robotics (Hadfield-Menell et al., 2016), education (Gweon et al., 2010), cognitive science (Shafto and Goodman, 2008; Shafto et al., 2014), and machine learning (Zhu, 2015). In linguistics, the original Rational Speech Act theory papers (Frank and Goodman, 2012; Goodman and Frank, 2016) describe applications of pragmatic inference in human language. These papers are widely cited and have been applied to problems in emergent language, vision and language navigation, cultural evolution, multi-agent RL, generating and following instructions, image captioning, and machine translation. In cognitive development, Naive Utility Calculus (Jara-Ettinger et al., 2015) is also widely cited and has applications to inverse reinforcement learning and scientific models of children's behavior. In robotics, Cooperative Inverse Reinforcement Learning (Hadfield-Menell et al., 2016) and Pedagogic-Pragmatic Inference (Fisac et al., 2017) have been proposed to explain value alignment and have been applied to deep reinforcement learning for aligning with human preferences, multi-agent systems, and learning from demonstration. In education and cognitive science, the Pedagogical reasoning model explains learning
<sup>6</sup> Even if the set  $\operatorname{argmax}_D g^\theta(\cdot|h)$  contains more than one element, PyTorch will only return one. Any plot that considers data activation suffers from this artifact. Hence, the equality.
1 Gaussian
3 Gaussians
5 Gaussians
Figure 3.3: Rows from top to bottom: (a) V-shaped relationship between  $\epsilon$  and average reconstruction loss across all  $\delta$  with vertex shifting to the right as  $g$  grows in complexity; (b) Near independence of data activation percentage with respect to  $\delta$ ; (c) Stratification of average reconstruction loss by  $\epsilon$  with a phase transition around  $\epsilon = 1$ , roughly independent of  $\delta$ ; (d) Stratification of data activation percentage by  $\epsilon$ . From (c) and (d), for  $\epsilon$  under a threshold that increases with the complexity of  $g$ , we see a roughly negative linear association between activation percentage and reconstruction loss.
from a teacher (Shafto and Goodman, 2008; Shafto et al., 2014), has been widely cited, and applied to understanding a broad array of experimental findings, informal human learning, and automated tutoring. In machine learning, the machine teaching approach is primarily a theoretical object, but has been highly influential in spawning iterative machine teaching and data poisoning approaches (Zhu, 2013, 2015). All of these models have been shown to be instances of entropic Optimal Transport (Wang et al., 2020), which our model generalizes, and so, they fail to consider imperfect common ground; using our theory, each of these works has an avenue to expand into a more realistic realm (see Section 2.2).
Figure 3.4: Columns from left to right: (a) Sample hypotheses versus reconstructions; (b) Sample hypotheses versus mean of activated data; (c) Sample hypotheses partitioned by mean of activated data. From (a), we see that hypotheses are well-reconstructed for all  $g$ . From (b), we see that not all data are activated. From (c), we see continuity of data selection with respect to hypotheses.
Recent proposals have used neural networks to offer an algorithmic perspective on cooperative communication. Early work by Andreas and Klein (2016) on neural pragmatics assumes, rather than derives, the encoder-decoder connection. Liu et al. (2017) formulate a teaching problem for learners who are unaware of the teacher's intent, and, thus, not cooperative, and Liu et al. (2018) assume teachers query learners to learn their beliefs. Fan et al. (2018); Wu et al. (2018) formulate teaching as a (reinforcement) learning problem. Yuan et al. (2021) consider teacher-aware learning in continuous spaces via neural networks, but do not formalize or study the common ground problem, and limit recursive reasoning between the teacher and learner to a single step.
Variational autoencoding (Kingma and Welling, 2014) is a generative probabilistic model enhanced by deep learning. Like variational autoencoding, we utilize neural networks to model conditional distributions. However, there is an important perspective shift when considering cooperative communication. Since cooperative communication is a model of two agents working together, both families of conditional probabilities are equally important, whereas in variational autoencoding the decoding process carries more emphasis, being generative.
We formalize common ground in cooperative communication and, thus, build a robust and accurate mathematical description of a fundamental aspect of the cooperative communication problem in two-party interaction. Our approach is an advance over prior theories which, through strong common ground, effectively assume away the most significant challenge for cooperative communication. Rooted in viewing the data selection and hypotheses inference processes that define communication as probabilistic encoding and decoding and drawing a connection between cooperative communication and variational autoencoding, this approach offers new theoretical and computational gains in the study of human-human and human-machine cooperation.
This research was supported in part by DARPA awards HR0011-23-9-0050 and W912CG22C0001 and NSF grants MRI-2117429 to P.S. and DMS-1954363 to Y.J.
Andreas, J. and Klein, D. (2016). Reasoning about pragmatics with neural listeners and speakers. arXiv preprint arXiv:1604.00562.
Bonawitz, E., Shafto, P., Gweon, H., Goodman, N. D., Spelke, E., and Schulz, L. (2011). The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery. Cognition, 120(3):322-330.
Clark, H. H., Brennan, S. E., et al. (1991). Grounding in communication. Perspectives on socially shared cognition, 13(1991):127-149.
Fan, Y., Tian, F., Qin, T., Li, X.-Y., and Liu, T.-Y. (2018). Learning to teach. arXiv preprint arXiv:1805.03643.
Fisac, J. F., Gates, M. A., Hamrick, J. B., Liu, C., Hadfield-Menell, D., Palaniappan, M., Malik, D., Sastry, S. S., Griffiths, T. L., and Dragan, A. D. (2017). Pragmatic-pedagogic value alignment. arXiv preprint arXiv:1707.06354.
Frank, M. C. and Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science, 336(6084):998-998.
Frank, M. C., Goodman, N. D., and Tenenbaum, J. B. (2009). Using speakers' referential intentions to model early cross-situational word learning. *Psychological science*, 20(5):578-585.
Goodman, N. D. and Frank, M. C. (2016). Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818-829.
Goodman, N. D. and Stuhlmüller, A. (2013). Knowledge and implicature: Modeling language understanding as social cognition. *Topics in cognitive science*, 5(1):173-184.
Gweon, H., Tenenbaum, J. B., and Schulz, L. E. (2010). Infants consider both the sample and the sampling process in inductive generalization. Proceedings of the National Academy of Sciences, 107(20):9066-9071.
Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. (2016). Cooperative inverse reinforcement learning. In Advances in neural information processing systems, pages 3909-3917.
Hao, X. and Shafto, P. (2023). Coupled variational autoencoder. In International Conference on Machine Learning. PMLR.
Ho, M. K., Littman, M., MacGlashan, J., Cushman, F., and Austerweil, J. L. (2016). Showing versus doing: Teaching by demonstration. In Advances in Neural Information Processing Systems, pages 3027-3035.
Jara-Ettinger, J., Gweon, H., Schulz, L. E., and Tenenbaum, J. B. (2016). The naive utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8):589-604.
Jara-Ettinger, J., Lydic, E., Tenenbaum, J. B., and Schulz, L. (2015). Beliefs about desires: Children's understanding of how knowledge and preference influence choice. In CogSci.
Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Kingma, D. P. and Welling, M. (2014). Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Laskey, M., Chuck, C., Lee, J., Mahler, J., Krishnan, S., Jamieson, K., Dragan, A., and Goldberg, K. (2017). Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 358-365. IEEE.
Liu, W., Dai, B., Humayun, A., Tay, C., Yu, C., Smith, L. B., Rehg, J. M., and Song, L. (2017). Iterative machine teaching. In International Conference on Machine Learning, pages 2149-2158. PMLR.
Liu, W., Dai, B., Li, X., Liu, Z., Rehg, J., and Song, L. (2018). Towards black-box iterative machine teaching. In International Conference on Machine Learning, pages 3141-3149. PMLR.
Nutz, M. (2022). Introduction to entropic optimal transport. https://www.math.columbia.edu/~mnutz/docs/EOT_lecture_notes.pdf.
Séjouré, T., Peyé, G., and Vialard, F.-X. (2023). Unbalanced optimal transport, from theory to numerics. arXiv preprint arXiv:2211.08775.
Shafto, P. and Goodman, N. (2008). Teaching games: Statistical sampling assumptions for learning in pedagogical situations. In Proceedings of the 30th annual conference of the Cognitive Science Society, pages 1632-1637. Cognitive Science Society Austin, TX.
Shafto, P., Goodman, N. D., and Frank, M. C. (2012). Learning from others: The consequences of psychological reasoning for human learning. Perspectives on Psychological Science, 7(4):341-351.
Shafto, P., Goodman, N. D., and Griffiths, T. L. (2014). A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive Psychology, 71:55-89.
Tomasello, M. (1999). The cultural origins of human cognition. Harvard University Press, Cambridge, MA.
Wang, P., Wang, J., Paranamana, P., and Shafto, P. (2020). A mathematical theory of cooperative communication. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20.
Wu, L., Tian, F., Xia, Y., Fan, Y., Qin, T., Jian-Huang, L., and Liu, T.-Y. (2018). Learning to teach with dynamic loss functions. Advances in Neural Information Processing Systems, 31.
Yang, S. C., Yu, Y., Givchi, A., Wang, P., Vong, W. K., and Shafto, P. (2018). Optimal cooperative inference. In AISTATS, volume 84 of Proceedings of Machine Learning Research, pages 376-385. PMLR.
Yuan, L., Zhou, D., Shen, J., Gao, J., Chen, J. L., Gu, Q., Wu, Y. N., and Zhu, S.-C. (2021). Iterative teacher-aware learning. Advances in Neural Information Processing Systems, 34:29231-29245.
Zhu, X. (2013). Machine teaching for bayesian learners in the exponential family. In Advances in Neural Information Processing Systems, pages 1905-1913.
Zhu, X. (2015). Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Twenty-Ninth AAAI Conference on Artificial Intelligence.