SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The sequential GBT captures the asymptotic behavior of a learning problem  $(C, \theta_0, h^*)$ . Static world where there exists a fixed true hypotheses  $h^*$  is considered in this section. Data is sampled from
$\eta = \mathbb{P}(d|h^{*})$  (not necessarily related to some  $h\in \mathcal{H}$ ). Then the learner follows GBT with cost  $C$ , and parameter  $\epsilon$ , starts with a prior  $\theta_0$ , then in each round  $k$ , applies GBT with  $\eta_{k - 1}$  and  $\theta_{k - 1}$  to generate  $\theta_{k}$ .
We investigate two cases. In the Preliminary sequential model (PS), we assume  $\eta_{k} = \eta$  for all  $k$ . In practice, often a learner does not have access  $\eta$ . Instead, in each round the learner may choose to use the current observed data distribution  $\eta_{k}(d)$  as an estimation of  $\eta$ . Thus we study the Real sequential model (RS) where  $\eta_{k} \xrightarrow{a.s.} \eta$ .
In statistics, a model is said to be consistent when, for every fixed hypothesis  $h \in \mathcal{H}$ , the model's belief  $\theta$  over the hypotheses set  $\mathcal{H}$  converges to  $\delta_h$  in probability as more data are sampled from  $\eta = \mathbb{P}(d|h)$ . Such consistency has been well studied for Bayesian Inference since Bernstein and von Mises and Doob [Doob, 1949], and recently demonstrated for Cooperative Communication [Wang et al., 2020a]. However, the challenge arises when one tries to learn a  $h^*$  that is not contained in the pre-selected hypothesis space  $\mathcal{H}$ . It is not clear which  $h \in \mathcal{H}$  is the 'correct' target to converge to.
In this section, we demonstrate GBT's ability of learning new hypothesis. Analogize to consistency, the properties are stated directly in the language of posterior sequence  $(\Theta_k)_{k=1}^{\infty}$  as random variables, focusing on whether the sequence converges (and in which sense), and how conclusive (how likely to a stable new hypothesis is learned) the sequence is.
For a learning problem  $(C, \theta_0, h^*)$ , results in this section are organized based on different  $\epsilon_{\theta}$  values.
Conclusive and Bayesian-style:  $\epsilon_{\theta} = \infty$ . These learners are located on the right side of Cube Fig. 1. Many well-studied learners are in this class: Bayesian, Cooperative, Discriminative, Row Greedy etc. According to Prop 8, learners in this class perform "Bayesian" style learning.
When  $\epsilon_{\eta} = 0$ , i.e. the learners who update their belief without considering data distribution, (PS) and (RS) are essentially the same. The following holds:
Theorem 9 ([Doob, 1949],[Wang et al., 2020a]). In GBT sequential model (both  $(\mathbf{PS})$  and  $(\mathbf{RS})$ ) with  $\epsilon = (\epsilon_{P},0,\infty)$  where  $\epsilon_{P}\in (0,\infty)$ , the sequence  $\Theta_{k}$  converges to some  $\delta_h$  almost surely,  $h$  is the closest column of  $e^{-C / \epsilon_P}$  to  $\eta$  in the sense of KL-divergence.
When  $\epsilon_{\eta} = \infty$ , the models (PS) and (RS) present slightly different behaviors.
Theorem 10 (PS). When  $\epsilon_{\eta} = \epsilon_{\theta} = \infty$ , for the PS problem belief random variables of a GBT learner  $(\Theta_k)_{k \in \mathbb{N}}$  converge to the random variable  $Y$  in probability, where  $Y = \sum_{h \in \mathcal{H}} \theta_0(h) \delta_h$  and  $Y$  is supported on  $\{\delta_h\}_{h \in \mathcal{H}}$  with  $\mathbb{P}(Y = \delta_h) = \theta_0(h)$  for  $\epsilon_{\eta} = \epsilon_{\theta} = \infty$  and  $\epsilon_P \in (0, \infty)$ .
Corollary 11. Given a fixed data sequence  $d_{i}$  sampled from  $\eta$ , if  $\theta_{k}$  converges to  $\delta_{h^j}$ , then the  $j$ -th column of  $M_{k}$  converges to  $\eta$ .
Thus a GBT learner, with access to the data distribution and using strict marginal constraints, converges to the true hypothesis mapping  $\eta$  with probability 1. Moreover, the probability of which  $h\in \mathcal{H}$  is shaped into  $\eta$  is determined by their prior  $\theta_0$ . That is, GBT learners converge to the truth by reforming one of their original hypotheses into the true hypothesis.
Proposition 12. When  $\epsilon_{\eta} = \epsilon_{\theta} = \infty$ , for the  $(RS)$  problem, the belief random variables of a GBT learner  $(\Theta_k)_{k \in \mathbb{N}}$  satisfies that for any  $s > 0$ ,  $\lim_{k \to \infty} \sum_{h \in \mathcal{H}} \mathbb{P}(\Theta(h) > 1 - s) = 1$ . As a consequence,  $M_k$  as the transport plan has a dominant column  $(h^j)$  with total weights  $>1 - s$ , and  $|(M_k)_{ij} - \eta_k(i)| < s$ .
In fact, as long as the sequence of  $\eta_{k}$  as random variables converges to  $\eta$  in probability, the above proposition holds. The limit  $\lim_{k\to \infty}\sum_{h\in \mathcal{H}}\mathbb{P}(\Theta (h) > 1 - s)$  measures how conclusive the model is.
In contrast with standard Bayesian or other inductive learners, Proposition 12 shows that a GBT learner is able to learn any hypothesis mapping  $\eta = \mathbb{P}(d|h^{*})$  up to a given threshold  $s$  with probability 1. In addition to unifying disparate models of learning, GBT enables a fundamentally more powerful approach to learning by empirically monitoring the data marginal.
Fig. 2 illustrates convergence over learning problems and episodes. In each bar, we sample 100 learning problems  $(C,\theta_0,h^*)$  from Dirichlet distribution with hyperparameters the vector  $\mathbf{1}$ . Then we sample 1000 data sequences (episodes) of maximal length  $N = 10000$ . The learner learns with Algo. 2 where the stopping condition  $\omega$  is set to be  $\max_{h\in \mathcal{H}}\theta (h) > 1 - s$  with  $s = 0.001$ . The
Figure 2: Evidence of general consistency: we plot the percentage of episodes that reaches a threshold (0.999) by round number (in colors of the bars). Each bar represents a size of matrix, for each bar 100 matrices were randomly sampled, and 1000 rounds were simulated per matrix. "exact" means learner uses  $\eta_{k} = \eta$ , (PS), "update" means learner uses statistics on current data in the episode (RS). "uot" takes  $\epsilon = (1,40,40)$  and "ot" comes with exact and  $\epsilon = (1,\infty ,\infty)$ .
$y$ -axis in the plots represents the percentage of total episode converged. The color indicates in how many rounds the episode converges. For instance, in the bar corresponding to  $10 \times 10$ -update_uot', with 10 data points (yellow portion), about  $50\%$  episodes satisfy the stopping condition.
The first plot shows results for  $10 \times 10$  and  $5 \times 3$  matrices. The second plot shows results for rectangular matrices of dimension  $m \times 10$  with  $m$  ranges in [5, 10, 25, 50, 100]. The third plot shows results for square matrices of dimension  $m \times m$  with  $m$  ranges in [10, 25, 50, 100]. Here 'exact' and 'update' indicate the problem is (PS) or (RS), respectively. For parameters,  $uot$  represents the parameter choice ( $\epsilon_{P} = 1$ ,  $\epsilon_{\theta} = \epsilon_{\eta} = 40$ ) vs.  $ot$  represents the parameter choice ( $\epsilon_{P} = 1$ ,  $\epsilon_{\theta} = \epsilon_{\eta} = \infty$ ).
The first plots demonstrate that learners that do not have access to the true hypothesis (empirically builds estimation of  $\eta$ ) learn faster than learners who have full access. The second plot indicates with a fixed number of hypotheses, learning is faster when the dimension of  $\mathcal{D}$  increases. The third plot shows that the GBT learner scales well with the dimension of the problem.
Figure 3: Left: Behavior of models spanning the line segment between BI and CI. With  $\epsilon_{P} = 1$  and  $\epsilon_{\theta} = \infty$ , when  $\epsilon_{\eta}$  varies from 0 to  $\infty$ , the theory changes from BI to CI. Each bar graphs the Monte-Carlo result of 400,000 teaching sequences, we empirically observe that the coefficients  $a(h)$  of the limit in terms of  $\sum_{h\in \mathcal{H}}a(h)\delta_h$  changes from BI to CI continuously from  $\delta (h^3)$  by Bernstein-von Mises to  $\theta_0(h)$  by Theorem 10. Right: the Euclidean distances of each coefficient  $a(h)$  to BI result (blue crosses), and to CI result (orange dots).
Then we study the learners that interpolate between Bayesian and Cooperative learners (located on the line connecting CI and BI in Fig 1). Consider a fixed learning problem  $(C, \theta_0, h^*)$ . Consistency of Bayesian inference states that asymptotically, the learner Bayesian converges to a particular hypothesis  $h_b \in \mathcal{H}$  almost surely where  $h_b$  is the hypothesis closest to  $h^*$  under KL divergence. Theorem 10 indicates that a GBT cooperative learner modifies one of the hypotheses into  $h^*$  in probability 1. The probability of  $h^j$  converges to  $h^*$  is determined by  $\theta_0(h^j)$ .
In Fig. 3, we study the asymptotic behavior of the learners corresponding to  $\epsilon = (1,\epsilon_{\eta},\infty)$ , with  $\epsilon_{\eta}\in \{0,0.02,0.2,0.5,1,2,5,50,\infty \}$ . We sample a learning problem with a dimension  $5\times 5$  from Dirichlet distribution with hyperparameters the vector 1. Each learner  $\epsilon = (1,\epsilon_{\eta},\infty)$  is equipped with a fixed  $C$ ,  $\theta_0$  and  $\eta_{k} = \eta$  for all  $k$ . We run 400,000 learning episodes per learner, and plot their convergence summary in the bar graph. A continuous transition from a Bayesian learner to a cooperative learner can be empirically observed: the coefficients  $a(h)$  of the limit in terms of  $\sum_{h\in \mathcal{H}}a(h)\delta_h$  changes from  $\delta (h^3)$  by Bernstein-von Mises to  $\theta_0(h)$  by Theorem 10.
From the previous empirical results, we conclude the following conjecture:
Conjecture 13. When  $\epsilon = (\epsilon_{P},\epsilon_{\eta},\infty)$ , where  $\epsilon_P\in (0,\infty)$ , the sequence of posteriors  $\Theta_{k}$  from generic  $C$ ,  $\eta$ ,  $\theta$  and  $\epsilon$  as random variables satisfy  $\lim_{k\to \infty}\sum_{h\in \mathcal{H}}\mathbb{P}(|\Theta_k(h) - 1| < e) = 1$  for any  $e > 0$ .
Further, we pick out those episodes with  $\theta_N(h) > 0.95$ , plot the values  $\mathbb{E}_{\theta_N(h) > 0.95}[\ln \theta_k(h) - \ln (1 - \theta_k(h))]$  for each  $h$  against  $k$  in Fig. 4. Near linear relations are observed away from the first several rounds and before the values reach the precision threshold. These are empirical estimates of the rate of convergence.
Figure 4: Top: For a learning problem  $C$ , behaviors of 9 different learners with  $\epsilon_{P} = 1$ ,  $\epsilon_{\theta} = \infty$  and various  $\epsilon_{\eta}$  (denoted in figure) on conclusion distributions,  $a(h)$  in bar graph, plots below bars are estimated convergence rates  $\mathbb{E}\ln (\theta_k(h) / (1 - \theta_k(h)))$  averaged on episodes converging to  $h$ , one curve per hypothesis.
Inconclusive and independent:  $\epsilon_{\theta} = 0$ . The following holds for both (PS) and (RS):
Proposition 14. For  $\epsilon = (\epsilon_{P},\epsilon_{\eta},0)$  with  $\epsilon_P\in (0,\infty)$ , as  $\eta_k\rightarrow \eta$  almost surely, the sequence  $\Theta_{k}$  of posteriors as a sequence of random variables converges in probability to variable  $\Theta$ , where  $\mathbb{P}(\Theta = \mathbf{v}^i) = \eta (i)$  and  $\mathbf{v}^i = P_{(i, - )} / \left(\sum_{j = 1}^{m}P_{ij}\right)$  and  $P = P^{\epsilon}(C,\eta ,\theta)$ . Therefore, for any  $s > 0$ ,  $\lim_{k\to \infty}\sum_{h\in \mathcal{H}}\mathbb{P}(|\Theta_k(h) - 1| < s) = 0$  for generic (for all but in a closed subset) cost  $C$  and  $\eta, \theta$ .
With  $\epsilon_{\theta} = 0$ , the constraint on column-sum  $(\epsilon_{\eta}$ -term) fails to affect the transport plan, thus the  $\Theta_k$ 's in the sequence are independent from each other, in contrast that in all other cases the adjacent ones are correlated via a nondegenerate transition distribution. The independence makes the sequence of posterior-samples in one episode behave totally random, thus rarely converge as points in  $\mathcal{P}(\mathcal{H})$ . Furthermore, when consider the natural coupling  $(\Theta_{k-1}, \Theta_k)$  from Markov transition measure for  $\epsilon_{\theta} = 0$  (which is independent),  $\mathbb{E}\left(|\Theta_{k-1} - \Theta_k|^2\right)$  converges to the variance  $Var(\eta)$ . In contrast, for  $\epsilon_{\theta} = \infty$ ,  $\mathbb{E}\left(|\Theta_{k-1} - \Theta_k|^2\right)$  converges to 0 if Conj. 13 holds.
$\epsilon_{\theta} \in (0, \infty)$ : partially conclusive. From Conj. 13 and Prop. 14, together with the continuity of the transition distribution on  $\epsilon$ , we conjecture the following continuity on conclusiveness:
Conjecture 15. For both  $(PS)$  and  $(RS)$  models, when  $\epsilon = (\epsilon_{P},\epsilon_{\eta},\epsilon_{\theta})$  with  $\epsilon_P,\epsilon_\theta \in (0,\infty)$ , the posterior sequence  $\Theta_{k}$  from generated from generic  $C$ ,  $\eta$ ,  $\theta$  and  $\epsilon$  satisfy that  $\lim_{k\to \infty}\sum_{h\in \mathcal{H}}\mathbb{P}(|\Theta_k(h) - 1| < s) = L$  exists, and  $L\in (0,1)$ , for any  $s > 0$ .
An Exploration on Interpolation It is popular in the state of art machine learning models that an agent learns probabilistically, but makes decisions greedily. This heuristic represents a path where a big leap on the cube was taken at the last step. An interesting question is under what circumstances this is optimal, what are the trade-offs, and under what conditions smoother trajectories are preferable. Instead of the giant leap, small steps along two paths are explored in the cube.
Our exploration takes a slightly different situation where the last step is replaced by a discriminative learning strategy  $\epsilon = (0, \infty, \infty)$ . We choose a matrix randomly of shape  $4 \times 4$  (see Supplementary for detail), set total steps or total data points taught  $N = 10$ , and uniform prior  $\theta \in \mathcal{P}(\mathcal{H})$  on hypotheses. Three learners are postulated: Blue performs Bayesian in first 9 steps and Discriminative in the last. Orange and Red follows two different interpolation curves with the same endpoints on the cube drawn in Fig. 5a. The curves are line and parabola segments on the cube sides.
Results of the three learners are shown in Fig. 5b. We sample  $h^*$  uniformly from the 4 columns of (the column-normalized)  $M$ , 40000 repeats for each learner. Conclusiveness (minimal  $l^1$  distance between posterior and a 1-hot vector) and posterior entropy are plotted as histograms. The results show that the smoother path may lead to a more conclusive posterior. Numerical results: Conclusiveness of Blue: mean 0.9406, standard deviation 0.1300. Conclusiveness of Orange: mean 0.9964, standard deviation
0.0327. Conclusiveness of Red: mean 0.9834, standard deviation 0.0676. Furthermore, compared with a sudden jump, gradual interpolations have lower entropy. Numerical results: entropy of Blue: mean 0.1261, standard deviation 0.2435, entropy of Orange: mean 0.0079, standard deviation 0.0629; entropy of Red: mean 0.0388, standard deviation 0.1336.
(a) Three learners
Figure 5: (a): Baseline Blue and two learners Orange and Red following corresponding interpolation paths. (b): Results of the three learners over 40000 repeats. Top: conclusiveness, the frequency distribution of maximal posterior component. Bottom: entropy distribution.
(b) Posteriors and Entropy distributions
From this experiment, in the 10-sample learning, two smoother learners behave more conclusive in their posterior with smaller posterior entropy. Meanwhile, we still know very little about the interpolation behavior, such as which path works better, how to distribute vertices on the curve, etc.
4 Sequential GBT - Dynamic
While static models are frequently studied, in many cases world changes dynamically. In this section, we take a first step in this direction by exploring the sequential behaviors of GBT learners assuming the world changes periodically. Demonstrate that unlike existing learners, GBT learner is capable of detecting the non-static property of a given problem.
Let integer  $p > 0$  be the period, given a set of true hypotheses distributions  $\vec{\eta} = \{\eta_0, \eta_1, \dots, \eta_{p-1}\} \subseteq \mathcal{P}(\mathcal{D})$ , datum  $d_t$  is sampled from  $d_{k\% p}$  where  $k\% p$  represents the remainder of  $k$  under division by  $p$ . Proposition 16. For a Bayesian learner, the posterior sequence  $\{\Theta_i\}$  converges almost surely to the average of true hypotheses  $\overline{\eta} = \frac{1}{p} \sum_{k=0}^{p-1} \eta_k$ .
For random variables of learner's posterior sequence  $(\Theta_k)_{k=1}^{\infty}$ , group them by period, we denote  $\vec{\Theta}_t := (\Theta_{tp}, \Theta_{tp+1}, \ldots, \Theta_{(t+1)p-1})$ . Here  $k$  represents the time step,  $t$  denotes the period index.
Proposition 17. For  $\epsilon$  in the interior of the cube, for  $(PS)$  problem, the sequence  $\{\vec{\Theta}_t\}$  (random variables over  $\mathcal{P}(\mathcal{H})^p$ ) form a time-homogeneous Markov chain. For  $(RS)$  problem,  $\{(\vec{\Theta}_t, \frac{1}{pt} \sum_{k=0}^{p-1} t \eta_k)\}$ , the random variable sequence producing samples  $\{(\vec{\theta}_t, \frac{1}{pt} \sum_{k=0}^{pt-1} \delta_{d_k})\}$ , forms a Markov chain.
Next we compare different learners' behavior empirically for (RS) problem. For visualization,  $M$  is taken of shape  $3 \times 3$ , thus  $\mathcal{P}(\mathcal{D})$  and  $\mathcal{P}(\mathcal{H})$  are both of dimension 2. If the Markov chain defined in Prop. 17 stabilizes,  $\mathbb{E}[\Theta_k]$  will be periodic, matching the pattern of  $\vec{\Theta}_t$ . In fact, the period could be  $p$ , or a factor of  $p$ , or stabilizes where the period can be considered as 1. Thus we analyze  $\vec{\eta}$  in  $\mathcal{P}(\mathcal{D})$  and  $\mathbb{E}[\Theta_k]$  in  $\mathcal{P}(\mathcal{H})$ , obtained from Monte-Carlo sampling along certain amount of episodes.
Fig. 6 (a) assume the true hypothesis travel along the triangular path connecting the 3 columns of  $M$  (shown in blue crosses). We found that GBT learners with  $\epsilon$  in the interior of the cube (general GBT) produce a posterior path of period  $p$ , while the posteriors of Bayesian and SCBI learners tend to converge (Fig. 6 (b-e)). Thus a general GBT learner can naturally detect the periodicity of the world. We simulate up to  $k = 400$  steps and 10240 repeats for each learner.
Moreover, we discovered that a general GBT learner's posteriors converge to a curve whose area is proportional to the area of the path of true hypotheses. In Fig. 7, as the path of true hypotheses varies by radius and shapes, the ratio (shown as slope) between both areas tends to be the same, it is 0.1620 in (a) and 0.1616 in (d), which suggests that this ratio is independent from the path of  $\vec{\eta}$ .
Figure 6: (a). Setup: each dot represents an  $\eta_{k}$ ; data are sampled along the dots from red to yellow of period 18,  $M$  has 3 columns represented by the blue crosses. (b-d). Bayesian, general GBT, SCBI learners, resp., blue curve shows  $\mathbb{E}[\Theta_k]$  and red crosses are  $\mathbb{E}[\vec{\Theta}_t]$  (mean of 18 consecutive  $\mathbb{E}[\Theta_k]$ 's). (e). for 6 different learners (shown in colors), plot (1) the averaged distance between  $\mathbb{E}[\Theta_k]$  and its center v.s. number of periods, in solid lines and left y-ticks, and (2) the step-length of  $\mathbb{E}[\vec{\Theta}_t]$  between consecutive periods in dash-dots and right y-ticks.
Figure 7: Behavior of a GBT learner with  $\epsilon = (1,10,10)$  on two different paths of  $\vec{\eta}$  with  $p = 20$ , tested in 300 steps and 10240 episodes.  $M$  is fixed and represented by the red dots. Learner's posteriors form roughly periodic paths, small panels on corners of (a, d), plot path of  $\vec{\eta}$  and posterior paths, ratio between their enclosed areas are shown in yellow to blue dots. (b, c) shows the 20 concentric similar paths that  $\vec{\eta}$  follow. Colors are matched between paths and corresponding area ratios.
Related Work. Prior work defines and outlines basic properties of Unbalanced Optimal Transport [Liero et al., 2018, Chizat et al., 2018, Pham et al., 2020]. Bayesian approaches are prominent in machine learning [Murphy, 2012] and beyond [Jaynes, 2003, Gelman et al., 1995]. There is also research on cooperative learning [Wang et al., 2019, 2020b,a] see also [Liu et al., 2021, Yuan et al., 2021, Zhu, 2015, Liu et al., 2017, Shafto and Goodman, 2008, Shafto et al., 2014, Frank and Goodman, 2012, Goodman and Frank, 2016, Fisac et al., 2017, Ho et al., 2018, Laskey et al., 2017]. Discriminative learning is the reciprocal problem in which one sees data and asks which hypothesis best explains it [Ng and Jordan, 2001, Mandler, 1980]. We are unaware of any work that attempts to unify and analyze the general problem of learning in which each of these are instances.
We have introduced Generalized Belief Transport (GBT), which unifies and parameterizes classic instances of learning including Bayesian inference, Cooperative Inference, and Discrimination, as Unbalanced Optimal Transport (UOT). We show that each instance is a point in a continuous, differentiable on the interior, 3-dimensional space defined by the regularization parameters of UOT. Moreover, to demonstrate general GBT's capacity of supporting generalized learning, we prove and illustrate asymptotic consistency and estimate rates of convergence, including convergence to hypotheses with zero prior support, and ability of gripping dynamic of the world. In summary, GBT unifies very different modes of learning, yielding a powerful, general framework for modeling learning agents.
This research was supported in part by DARPA awards HR0011-23-9-0050 and W912CG22C0001 and NSF MRI 2117429 to P.S.