Generalized Belief Transport
Authors: Junqi Wang
PEI WANG
Patrick Shafto
Venue: NeurIPS 2023 poster (2023)
Keywords: Bayesian theory
Belief transport
Unbalanced optimal transport
parametrization
asymptotic behavior
environment drift detection

Abstract: Human learners have ability to adopt appropriate learning approaches depending on constraints such as prior on the hypothesis, urgency of decision, and drift of the environment. However, existing learning models are typically considered individually rather than in relation to one and other. To build agents that have the ability to move between different modes of learning over time, it is important to understand how learning models are related as points in a broader space of possibilities. We introduce a mathematical framework, Generalized Belief Transport (GBT), that unifies and generalizes prior models, including Bayesian inference, cooperative communication and classification, as parameterizations of three learning constraints within Unbalanced Optimal Transport (UOT). We visualize the space of learning models encoded by GBT as a cube which includes classic learning models as special points. We derive critical properties of this parameterized space including proving continuity and differentiability which is the basis for model interpolation, and study limiting behavior of the parameters, which allows attaching learning models on the boundaries. Moreover, we investigate the long-run behavior of GBT, explore convergence properties of models in GBT mathematical and computationally, document the ability to learn in the presence of distribution drift, and formulate conjectures about general behavior. We conclude with open questions and implications for more unified models of learning.

Generalized Belief Transport
Junqi Wang
Pei Wang Department of Math & CS Rutgers University Newark, NJ, 07102 peiwangerutgers. edu
Patrick Shafto
Department of Math & CS Rutgers University Newark, NJ, 07102
Department of Math & CS Rutgers University Newark, NJ, 07102 shaf thafto@rugeess edu
junqi. wangrrienggrutggrr... edu
Abstract
Human learners have ability to adopt appropriate learning approaches depending constraints such prior on the hypothesis, urgency of decision, and drift of the environment. However, existing learning models are typically considered individu- ally rather than in relation to and other. To build agents that have the ability move between different modes learning over time, is important to understand how learning models are related as points in a broader space of possibilities. We introduce a mathematical framework, Generalized Belief Transport (GBT), that unifies and generalizes prior models, including Bayesian inference, cooperative communication and classification, as parameterizations of three learning constraints within Unbalanced Optimal Transport (UOT). We visualize the space of learning models encoded by GBT as a cube which includes classic learning models as special points. We derive critical properties of this parameterized space including proving continuity and differentiability which is the basis for model interpolation, and study limiting behavior of the parameters, which allows attaching learning models on the boundaries. Moreover, we investigate the long-run behavior of GBT, explore convergence properties of models in GBT mathematical and compu- tationally, document the ability to learn in the presence of distribution drift, and formulate conjectures about general behavior. We conclude with open questions and implications for more unified models of learning.
Learning and inference are subject to internal and external constraints. Internal constraints include the availability of relevant prior knowledge. External constraints include the availability of time accumulate evidence versus the need make the best decision now or environmental non-stationarity. Standard models of machine learning tend view different constraints as different problems, which impedes development of unified learning agents.
These internal and external constraints map onto classic dichotomies in machine learning. Availability of prior knowledge maps onto the Frequentist-Bayesian dichotomy in which the latter uses prior knowledge constraint posterior beliefs, while the former does not. Within Bayesian theory, classic debate pertains to uninformative, or minimally informative, settings priors [Jeffreys] 1946) Robert et al. 2009). Availability of time accumulate evidence informs the use of generative versus discriminative approaches [Ng and Jordan 2001]. and static or drift/dynamic models [Dagum et al.) 1992 Murphy 22002]. Combining constraints on probability of beliefs and costs of data models cooperative communication [Wang et al. 2020b].
Learning agents must interpolate between modes of reasoning necessary given the constraints of the moment. Imagine observing an agent behaving in an environment. As an observer, one may wish to learn about the environment from the agent's actions. However, any inferences depend on one't 'ss model the agent and their constraints. How is the agent updating their beliefs? Do they have stable goals, they changing time? Perhaps the agent is selecting actions communicate what they know? In order to draw inferences over these possibilities, one must parameterize the space,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

ideally in such a way one could optimize over the possibilities. Indeed, in order to implement these possibiliities, the agent themself must parameterize the space order interpolate between classic dichotomies such as Bayesian and frequentist, static and dynamic environments, and helpful versus neutral agent, given constraints.
We introduce Generalized Belief Transport (GBT), based on Unbalanced Optimal Transport (Sec. which paramterizes and interpolates between known reasoning modes (Sec. 2.2). with four major contributions. First, we prove continuity in the parameterization and differentaaility on the interior of the parameter space (Sec. 2.1] Second, we analyze the behavior under variations in the parameter space (Sec. 2.3 Third, we study sequential learning, where learners may (not) track the empiri- cally observed data frequencies in (Sec. 3). Fourth, we investigate predictive performance under environmental drift (Sec. 4
R20 denotes the non-negative reals. Vector (1,..., 1). The i-th component of vector v is v(i). P(A) is the set of probability distributions over A. For a matrix M, My represents its (i, entry, M(s__) denotes its i-th row, and M(.)) denotes its j-th column. Probability is IP(
Notations.
1 Learning as problem of unbalanced optimal transport
Consider general learning setting: an agent, which we call learner, updates their belief about the world based on observed data subject to constraints. There is finite set D =d' ..... dd) of all possible data, that defines the interface between the learner and the world. The world is defined by a true hypothesis h², whose meaning is captured by probability mapping P(d[/") onto observable data. For instance, the world can either be the environment in classic Bayesian inference [Murphy] 2012] or a teacher in cooperative communication (Wang et al. 2020b].
A learner is equipped with a set of hypotheses H = {h' ....,, . which may NOT contain h"; an initial belief on the hypotheses set, denoted by On P(H); and non-negative cost matrix C m where C.j measures the underlying cost of mapping d' into h The cost matrix can be derived from other matrices that record the relation between 'D and H. such as likelihood matrices in classic Bayesian inference or consistency matrices in cooperative communication (see details in Section 2.2) This setting reflects agent's 's learning constraints: pre-selected hypotheses, and the relations between them and the communication interface (data set).
A learner observes data in sequence. At round k the learner observes data d, that is sampled from DD by the world according to P(d]h"). Then the learner updates their beliefs over H from fx-1 to 9g through learning scheme, where - 0g E P(H). For instance, in Bayesian inference, the learning scheme is defined by Bayes rule; while in discriminative models, the learning scheme prescribed by code book.
The learner transforms the observed data into a belief on hypotheses h E with a minimal cost, subject to appropriate constraints, with the goal of learning the exact map IPd[ht). We can naturally cast this learning problem as Unbalanced Optimal Transport.
1.1 Unbalanced Optimal Transport
Unbalanced Optimal Transport (UOT), introduced by [iero et al. (2018). is a generalization of (entropic) Optimal Transport [Villani 2008 Cuturi 2013 Peyré and Cuturi 2019). that relaxes the marginal constraints. Formally, for non- non-negative scalar parameters e (EP, En, Eq). the UOT plan is,
arg min pe(Rzo)nxm
((C,P)) +
(I)
Here, P) = Ei,, Cij P.j is the inner product between C and P, H(P)= Paç(log P.g 1)
is the entropy P. and KL(a := E, log (a: /b:) - a + b,) is the Kullback-Leibee divergence between vectors. It is shown in Chizat et al. 2018 that UOT plans
be solved efficiently via Algorithm Given a cost C, P can be obtained by applying (1), 6, c) -unbalaane Sinkhorn scaling on K =
(e o Ci.) M with convergence rate Ö mm
Pham et al. 2020
'To guarantee the hypotheses are distinguishable, we assume that C' does not contain two columns that are only differ by additive scalar.

---

Proposition 1. The UOT problem with cost matrix C. marginals 8, 8) and parameters = (cp, Enpy Eq) generates the same UOT plan as the UOT problem with tC, 8, 1, (tep, tea) for any t E (0, oo). Therefore, the analysis on and le are the same for general cost C.
Thus a positive common factor on C, Ep, En, Eg does not affect the solution of Eq. Therefore, for the later analysis, we fix ep 1 unless otherwise stated.
Framework: Generalized Belief Transport (GBT). Learning. efficiently transport one's belief with constraints, is naturally a UOT problem, i.e. a Generalized Belief Transport. Each round, learner, defined by a choice of c Ccp,Cy, cg). updates their beliefs as follows. Let 1,s-1, 8 1 be the learner's estimations of the data distribution and the belief over hypotheses H. after round & 1, respectively. At round k, the learner first improves their estimation of the mapping between DD and H. denoted by Ms. through solving the UOT plan Eq. ① with (C, 7/--1, (..00-1.... i.e. M&- P*(C, 1/s- 0s-00 1). Then with data observation dy. the learner updates their beliefs over H. using corresponding row of Ms. i.c. suppose dę d' for some d' E D, the learner's 's belief 0g is defined to be the row normalization of the i-th row of Mg. Finally, the learner updates their data distribution to T/ by increment of the i-th element of 1]--1, Algorithm 2
Algorithm Unbalanced Sinkhorn Scaling
Algorithm 2 Generalized Belief Transport
input: C, 0, 17, < (cp, Gy. co), N stopping
input: C. 0o. 1/0, h", N, data sampler based on P(dlh"), ) stopping condition w
condition w
output: Pt 1)
output: M, 0
initialize: K = exp( ppp v(0) 1m
initialize: & 4 1
while k < N and not w do
while K < N and not (0) do
(*) +
17
M + PC,C, (T-1, 02-1)
1v
get data d' sampled from T
n] t (pdale(/a-11, d²) via update rule
((()=
0
rptop
v + M(r.)
KTT (R)
Bg t v hen v(h)
end while (C n, 0) = diag(u) (diag())
1 k +
end while
2 Generalized Belief Transport
Many learning models-including Bayesian inference, Frequentist inference, Cooperative learning, and Discriminative learning-are unified under our GBT framework under choice of In this section, we focus on the single-round behavior of the GBT model, i.e., given a pair of marginals (0, 1)), how different learners update beliefss with single data observation. We first visualize the entire learner set as a cube (in terms of parameters), see Figure []] Then, we study the topological properties of the learner set through continuous deformations of parameters In particular, we show that existing models including Bayesian inference, cooperative inference and discriminative learning are learners with parameters (1, 0, oo), (1, 00, 00) and (0, 0o, 20) respectively in our UOT framework.
2.1
The parameter space of GBT model
The space of learners in GBT are parameterized by three regularizers for the underlying UOT problem ( 6p, 6y and eg, each ranges in [0, oo). Therefore, the constraint space for GBT is R3 with the standard topology. When C, 0 and r are fixed (assume 1) E Rm))) the map (CP, Ey. co) + (Pr) bears continuous properties:
Proposition 3 The UOT plan P in Equation ① as a function of is continuous in (0, 00) x o0)2 Furthermore, P is differentiable with respect to in the interior of its domain 220
Continuity on e provides the basis for interpolation between different learning agents. The proof of Proposition also alsoo implics the continuity on 1 and 0. Further, towards the boundaries of the parameter space (where theories like Bayesian, Cooperative Communication live in), we show:
Prroof' of all claims are included in the Supplementary Materials.
3

---

Proposition 3. Let sp, Sn, 2 0 be arbitrary finite numbers, the following holds:
(1) The limit of Pe exists as approaches (83,89988) In fact, (o0.-o0 Pij for all i,
(2) As (0, so), P converges to the solution to min(C, P) Sp + sgKL(P" with constraint P1 (3) Similarly, as - Sy. 00), P converges to the solution min P) (P)(P) + s, with constraint pº -
(4) And when <+> (sp, 0c, oo), the matrix P converges to the EOT solution:
min (C, Spl((P,, with constraints pT and P1 17.
s0)s,) (0o, Syy oo) or 00, oc, oo), the limit does not exist, but the directional (5) When e + (oo, 00, so), limits can be calculated.
The parameter space for GBT with its bound-
aries can be visualized in Fig. [ Propo-
sition 3 implies that the parameter space is
S U. oc (ff(o,, 20,
[0, col} U
Brann
{(00, x. 00): : E [0, oo])). 1). In Fig. segment [0, oe) is mapped to [0, 1) by sigmoid(log(z)).
Ea Trequenthe
Then boundaries are added to the image cube [0, 1)a The dashed lines top of the cube indicates limits that do not exist.
Boyeston
Tayessio
(S mi
Column Grendy
2.2 Special points in the parameter space
Descr
Conperativn Communication
Bayesian Inference. Given observed data, Bayesian learner (BI) (Murphy 2012] derives posterior belief P(h)d) ([ based on prior belief P(h) and likelihood matrix P(d[h), according to the Bayes rule. Intuitively, due to soft time constraint (ep 1), Bayesian learner is generative agent who puts a hard internal con- straint on their prior belief (ep oo), and omits the estimated data distribution 1) in the learning process, (C,, 1 0).
Figure 1: The parameter space S of GBT. Parameters (ep, Exp ea) can take the value cc, rendering the corresponding regularization to strict constraint. The two dashed edges with ep = oo are not generally well- defined since the limits do not exist. The vertices corre- sponding 9 & 7). Frequentist (17 & 1) and 1 0 are the limits taken along the vertical edges. Given (C, 8, n) as shown in left corner, each colored map plots each GBT learner (differ by constraints)' 's estimation of the mapping between hypotheses and data (UOT plan).
Corollary 4. Consider a UOT problem with cost = log P(dh)) marginals P(h), E P(D). The optimal UOT plan p(l,s,,5)) converges to the posterior P(hld) En - and eg + Thus, Bayesian inference is special case of GBT with c = (1.0, 0, oc).
Frequentist Inference. A frequentist updates their belief from data observations by increasing the corresponding frequencies of datum. Intuitively, a frequentist is an agent who puts hard constraint on the data distribution 17 (e,, = oc), and omits prior knowledge 0 (Eg = 0) in a learning process without time constraint (Ep = oc). Formally we show:
Corollary 5. Consider UOT problem with 0 E P(H), n P(d). The optimal UOT plan coe converges to 1) & 1 as Cp - co. Frequentist Inference is a special case of GBT with c = (co, co, 0).
Cooperative Communication. Two cooperative agents, a teacher and a learner, are considered in Yang et al. (2018), Wang et al. 22200)] [Saato et al. [202T]. Cooperative learners (CT) draw inferences about hypotheses based on which data would be most effective for the teacher to choose Given data observation, a cooperative learner derives an optimal plan L P(H. D) based on prior belief P(h), a shared data distribution P(d) and matrix M specifies the consistency between data and hypotheses (such as Mij records the co-occureence of d' and h²). Intuitively, a cooperative learner is also generative agent who puts hard constraints on both data and hypotheses (6,, = 00, cg oo), and aims to align with the true belief asymptotically, (Ep 1). Thus show:

---

Corollary 6. Let cost C = - log M. marginals 0 = Ph) and 1j = P(d). The optimal UOT plan p(l,c,))) converges to the optimal plan L as En + oc and Eg + oc. Cooperative Inference is special case of GBT with (1, 20), which is exactly entropic Optimal Transport (Cuturi 2013
Discriminative learning. A discriminative learner decodes an uncertain, possibly noise corrupted, encoded message, which is a natural bridge to information theory [Cover] 1999. Wang et al. 2020b] A discriminative learner builds optimal map hypotheses H. conditioned on observed data D. The map is perfect when, for all messages, encodings are uniquely and correctly decoded. Intuitively, a discriminative learner aims to quickly build deterministic code book (implies Ep = 0) that matches the marginals on H and D. We show that discriminative learner is GBT with = (0, oo, x)):
Corollary 7. Consider a UOT problem with cost - (P(,,)) m n, and marginals 0 = are uniform. The optimal UOT plan ((t approaches to a diagonal matrix as 6nz 60 - oc and cp O. In particular, discriminative learner is a special case of GBT with 00, 00, 0c), which is exactly classical Optimal Transport [Villani 2008/.
Many other interesting models are unified under GBT framework as well. GBT with = (0, oo, 0) denotes Row Greedy learner which is widely used in Reinforcement learning community Suttonn and Barto 2018]; = (oo, oG, 00) yields @ 0 which is independent coupling used in x² FFiebberg
et al. 1970|;
(ep, ég; 00) is used for adaptive color transfer studied in RRbinn et al. 2014 and
== [0, ca, 6,) is UOT without entropy regularizer developed in Chapel et al. 2021
Other points in
the GBT parameter space are also of likely interest, past or future.
2.3 General properties on the transportation plans
The general GBT framework builds a connection between the above theories, and the behavior of theory varies according to the change of parameters. particular, each factor of 6 = (Ep, <ny Eg) expresses different constraints of the learner. Given (C, (,,))) as shown in the top-left corner of Fig. 1 we plot each learner's UOT plan with darker color representing larger elements.
EP controls a learner's ' learning horizon. When Ep + 0, a learner's 's UOT plan is concentrated on a clear leading diagonal which allows them to make fast decisions. This corresponding to agents who are under the time pressure of making immediate decision, i.e. discriminative learner, or row greedy learner on the bottom of the cube (Fig. D. Most of the time, one datum is enough to identify the true hypothesis and convergence is achieved within every data observation. When Ep - oc, GBT converges to a reticent learner, such as learners on the top of the cube. Data do not constrain the true hypothesis, and learners draw their conclusions independent of the data. In between, GBT provides a generative (probabilistic) learner. When 6p 1, we have Bayesian learner and Cooperative learner, for whom data accumulate to identify the true hypothesis in manner broadly consistent with probabilistic inference, and consistency is asymptotic.
en controls a learner's knowledge on the data distribution 7). When E, + co, GBT converges to a learner who is aware of the data distribution and reasons about the observed data according to the probabilities/costs of possible outcomes. Examples include the Discriminative and Cooperative learners on the front of the cube. When 6, + 0, GBT converges to a learner who updates their belief without taking into considcration, such as Bayesian learners on the back of the cube, and the Tyrant who does not care about data nor cost and is impossible to be changed by anybody.
co controls the strength of the learner's prior knowledge. When 69 - 0, GBT converges to learners who utilizes no prior knowledge. Hence, they do NOT maintain belief's over H. and draws their conclusions purely on the data distribution, such as a Frequentist learner on the left of the cube. When eg + oo, GBT converges to a learner who enforces a strict prior such as Bayesian, Cooperative and Discriminative learners on the right of the cube. In particular, we show that:
Proposition 8. In GBT with eg = oc, cost C and current belief 0. The learner updates 0 with UOT plan in the same way as applying Bayes rule with likelihood from Pe Pt v), 0), and prior 0.
3 Sequential GBT Static
The sequential GBT captures the asymptotic behavior of learning problem (C, 0o, lt). Statie world where there exists a fixed true hypotheses ht is considered in this section. Data is sampled from
5

---

P(d)h") (not necessarily related to some h E H). Then the learner follows GBT with cost C, and parameter e, starts with prior do, then in each round k, applies GBT with 7)- and 0 to generate 0z.
We investigate two cases. In the Preliminary sequential model (PS), we assume T/s- = 1) for all k. In practice, often a learner does not have access Instead, in each round the learner may choose to use the current observed data distribution 7g (d) as estimation of 17. Thus we study the Real sequential model (RS) where T/k "."" 1)-
In statistics, a model is said to be consistent when, for every fixed hypothesis h E H. the model's s belief 9 over the hypotheses set H. converges to 6 in probability as more data are sampled from 1= P(d Such consistency has been well studied for Bayesian Inference since Bernstein and von Mises and Doob [Dobb] 1949]. and recently demonstrated for Cooperative Communication [Wang] et al. 2020a]. However, the challenge arises when one tries to learn a ht" that is not contained in the pre-selected hypothesis space H. It is not clear which h E is the 'correct' target to converge to.
In this section, we demonstrate GBT's ability of learning new hypothesis. Analogize to consistency, the properties are stated directly in the language of posterior sequence (0x) as random variables, focusing on whether the sequence converges (and in which sense), and how conclusive (how likely to a stable new hypothesis is learned) the sequence is.
For a learning problem (C, fo h"), results in this section are organized based on different 6g values.
Conclusive and Bayesian-style: 60 = co. These learners are located on the right side of Cube Fig. D Many well-studied learners are in this class: Bayesian, Cooperative, Discriminative, Row Greedy etc. According to Prop 8 learners in this class perform "Bayesian" style learning.
When E, 0, i.e. the learners who update their belief without considering data distribution, (PS) and (RS) are essentially the same. The following holds:
Theorem 9 (Doob) 1949]. Wang et al. 2020al). In GBT sequential model (both (PS) and (RS)) with e = (ep, 0, 20) where Ep E (0, 00), the sequence On converges to some on almost surely, h is the closest column of eC/er 1j in the sense of Kl-divergence.
When 6, = co, the models (PS) and (RS) present slightly different behaviors.
Theorem 10 (PS). When En = Ce = oc, for the PS problem belief random variables of a GBT learner (9x) converge to the random variable Y in probability, where Y Enen (o(h)nn and Y is supported on {δr tee with TP(Y 5x) (h) for Cn Cc co and cp (0)
Corollary 11. Given fixed data sequence d, sampled from n, if 0g converges to ōná. then the j-th column of Mj converges to
Thus a GBT learner, with access to the data distribution and using strict marginal constraints, converges to the true hypothesis mapping 7 with probability 1. Moreover, the probability of which h H is shaped into 7) determined by their prior lg. That is, GBT learners converge to the truth by reforming one of their original hypotheses into the true hypothesis.
When en ta for the (RS) problem, the belief random variables of GBT learner (eN) satisfies that for any >
Proposition 12.
lim Σ [PO((h)) 1- s) 1. As consequence, Mg
k-y00 hEH
as the transport plan has a dominant column (h²) with total weights > 1 s, and [MM))s -m (f) < s.
In fact, as long as the sequence of nh as random variables converges to 7) in probability, the above proposition holds. The limit lim Σ P(O(h) > 1 s) measures how conclusive the model is.
1-000 DEH
In contrast with standard Bayesian or other inductive learners, Proposition 12 shows that GBT learner is able to learn any hypothesis mapping IP(dhh)) up given threshold s with probability 1. In addition to unifying disparate models of learning, GBT enables a fundamentally more powerful approach to learning by empirically monitoring the data marginal.
Fig. illustrates convergence over learning problems and episodes. In each bar, we sample 100 learning problems (C, 0o h²") from Dirichlet distribution with hyperparameters the vector 1. Then we sample 1000 data sequences (episodes) of maximal length N 10000. The learner learns with Algo. where the stopping condition w is set to be maxse ((h) > 1 - with 0.001. The

---

3333
20
youte cenate thate inaia arent
ser opcond
2018,
Figure 2: Evidence general consistency: we plot the percentage of episodes that reaches a threshold (0.999) by round number (in colors of the bars). Each bar represents a size of matrix, for each bar 100 matrices were randomly sampled, and 1000 rounds were simulated per matrix. "exact" means learner uses 7 = (PS), "update" means learner uses statistics current data in the episode (RS). "uot" takes (1, 40, 40) and "ot" comes with exact and (1, 00, 00).
y-axis in the plots represents the percentage of total episode converged. The color indicates in how many rounds the episode converges. For instance, in the bar corresponding '10 x 10 update__oot _ut', with 10 data points (yellow portion), about 50% episodes satisfy the stopping condition.
The first plot shows results for 10 10 and 5 matrices. The second plot shows results for rectangular matrices dimension m 10 with m ranges in [5, 10, 25, 50, 100]. The third plot shows results for square matrices of dimension m m with m ranges in [10, 25, 50, 100]. Here 'exact' and 'update' indicate the problem is (PS) or (RS), respectively. For parameters, uol represents the parameter choice (ep eo en 40) vs. of represents the parameter choice (ep e, en = oo).
The first plots demonstrates that learners that do not have access to the true hypothesis (empirically builds estimation of r)) learn faster than learners who have full access. The second plot indicates with a fixed number of hypotheses, learning is faster when the dimension of D increases. The third plot shows that the GBT learner scales well with the dimension of the problem.
1.0
6K
0.6
68 41
Bar 6
0.4
02
I [
0U
Figure 3: Left: Behavior models spanning the line segment between BI and CL. With ep 1 and cg = oc, when 6, varies from O to oc, the theory changes from Bl to CI. Each har graphs the Monte-Carlo result of 400,000 teaching sequences,
empirically observe that the coefficients a(h) of the limit in terms of Σ >bew a (h)),,
changes from BI to CI continuously from 8(h²) by Bernstein-vee von Mises to (o(h) by Theorem 10 Right: the Fuclidean distances of each coefficient a(h) to BI result (blue crosses), and to CI result (orange dots).
Then we study the learners that interpolate between Bayesian and Cooperative learners (located on the line connecting CI and Bl in Fig Consider fixed learning problem (C, 0, h²). Consistency of Bayesian inference states that asymptotically, the learner Bayesian converges to a particular hypothesis las E HH almost surely where hn is the hypothesis closest h" under KI divergence. Theorem 10 indicates that a GBT cooperative learner modifies one of the hypotheses into h² in probability The probability of h² converges to h is determined by (o())).
In Fig. 3. we study the asymptotic behavior of the learners corresponding to e = (1, oo), with e, 0.00, 0.2, 0.5, 2, 5, 50, oo). We sample learning problem with dimension 5 x 5 from Dirichlet distribution with hyperparameters the vector 1. Each learner c = (1, c,, oo) is equipped with fixed C, Ou and 77 17 for all k. We run 400, 000 learning episodes per learner, and plot their convergence summary in the bar graph. A continuous transition from a Bayesian learner to cooperative learner be empirically observed: the coefficients a(h) of the limit in terms of Ene h a(h)o, o changes from 5(h²) by Bernstein-von Mises to (h) by Theorem
From the previous empirical results, conclude the following conjecture:

---

Conjecture 13. When e = (Ep, Cn: oc), where ep E c0), ) the sequence of posteriors Q₂ from generic C, d, 0 and e as random variables satisfy lim Σ P([O, [|< for any e >> k-oonEH
Further, we pick out those episodes with 8 > 0.95, plot the values E (1)s0005 [In 0g (h) 1n(1 - 0 (h))] for each against & in Fig. Near linear relations are observed away from the first several rounds and before the values reaches the precision threshold. These are empirical estimates of the rate of convergence.
0.5
0-
30-
23
100
0.02
Figure 4: Top: For a learning prohlem C, hehaviors 9 different learners with EP = 1, 6g = 00 and various en (denoted in figure) on conclusion distributions, a(h) in bar graph, plots below bars are estimated convergence rates E l0(0, (h)/(1 (z())) averaged on episodes converging to ht one curve per hypothesis.
Inconclusive and independent: 6g = O. The following holds for both (PS) and (RS):
Proposition 14. For c = (cp, cy, 0) with cp E (0, oc), as mæ - 1 almost surely, the sequence sequence of random variables converges in probability variable O where
O₂ of posteriors P(O v²) = =n) and v² = -/cs./ 1 and P = p n) 0). Therefore, for any > 0, lim Σnen (h) - < = 0 for generic (for all but in a closed subset) cost C and r). 0.
With Eø the constraint column-sum (e,,-term) fails to affect the transport plan, thus the θ₂ in the sequence are independent from each other, in contrast that in all other cases the adjacent ones are correlated via a nondegenerate transition distribution. The independence makes the sequence of sasplei in one episode behave totally random, thus rarely converge as points in P(H). Furthermore, when consider the natural coupling (Qx-1: 1: θ₂) from Markov transition measure for Eg (which independent), (l0,-1 -O converges to the variance V Var(n). In contrast, for Eg = oc, (10x-1 - O (2) converges to if Conj. holds.
Eg (0, oo): partially conclusive. From Conj. 13 and Prop. 14 together with the continuity of the transition distribution on e, we conjecture the following continuity on conclusiveness:
Conjecture 15. For both (PS) and (RS) models, when e &n t9) with e te (0, oc), the pos- terior sequence Oş from generated from generic C. m, 9 and satisfy that limg EnER (h) 1 < 8) = L. exists, and L E (0,1) 1), for any 8 >> 0.
An Exploration on Interpolation is popular in the state art machine learning models that an agent learns probabilistically. but makes decisions greedily. This heuristic represents path where big leap on the cube was taken at the last step. An interesting question is under what circumstances this is optimal, what the trade-offs, and under what conditions smoother trajectories preferable. Instead of the giant leap, small steps along two paths are explored in the cube.
Our exploration takes slightly different situation where the last step is replaced by discriminative learning strategy e = (0, 20, oo). We choose a matrix randomly of shape 4 X 4 (see Supplementary for detail), set total steps total data points taught N = 10, and uniform prior 0 P(H) on hypotheses. Three learners are postulated: Blue performs Bayesian in first steps and Discriminative in the last. Orange and Red follows two different interpolation curves with the same endpoints on the cube drawn in Fig. 5a The curves are line and parabola segments on the cube sides.
Results of the three learners are shown in Fig. 5b We sample h² uniformly from the 4 columns of (the nolmaienormalized) M, 40000 repeats for each learner. Conclusiveness (minimal 1 distance between posterior and 1-hot vector) and posterior entropy are plotted as histograms. The results show that the smoother path may lead more conclusive posterior. Numerical results: Conclusiveness of Blue: mean 0.9406, standard deviation 0.1300. Conclusiveness of Orange: mean 0.9964, standard deviation

---

0.0327. Conclusiveness of Red: mean 0.9834, standard deviation 0.0676. Furthermore, compared with sudden jump, gradual interpolations have lower entropy. Numerical results: entropy of Blue: mean 0.1 0.1261, standard deviation 0.2435, entropy of Orange: mean 0.0079, standard deviation 0.0629; entropy Red: mean 0.0388, standard deviation 1336.
10-
10"
10%
107
10"
Parth
Seyesher
24
06
666
1.
04
5.0
04
26
26
11
Deserminntiive
(b) Posteriors and Entropy distributions Figure 5: (a): Baseline Blue and two learners Orange and Red following corresponding interpolation paths. (b): Results of the three learners 40000 repeats. Top: conclusiveness, the frequency distribution of maximal posterior component. Bottom: entropy distribution.
(a) Three learners
From this experiment, in the 10-sample learning, smoother learners behave more conclusive in their posterior with smaller posterior entropy. Meanwhile, we still know very little about the interpolation behavior, such as which path works better, how to distribute vertices on the curve, etc.
4 Sequential GBT Dynamic
While static models are frequently studied, in many cases world changes dynamically. this section, we take first step in this direction by exploring the sequential behaviors of GBT learners assuming the world changes periodically. Demonstrate that unlike existing learners, GBT learner capable of detecting the non-static property of given problem.
Let integer p be the period, given a of hypotheses distributions {11o- Tl,...... ........ 7p--1) } S P(D), datum de is sampled from d4%p where k%op represents the remainder of k under division by Proposition 16. For Bayesian learner, the posterior sequence } converges almost surely the average of true hypotheses
y Eh= Tlk-
For random variables of learner's posterior sequence (0x) group them by period, we denote 6, = (Otp. O1pt1:..., O(t+))p-)) Here k represents the time step, t denotes the period index. Proposition 17. For e in the interior of the cube, for (PS) problem, the sequence {G:) (random vari- ables over P(H))) form time-homogeneow Markov chain. For (RS) problem, {(θ,, p7 Et= bmk) the random variable sequence producing samples {(0,, 52 EP- ). forms Markov chain.
Next we compare different learners' behavior empirically for (RS) problem. For visualization, M is taken of shape
thus PD) and PHH) are both of dimension 2. the Markov chain defined in Prop. 17 stabilizes, E E[Ox] will be periodic, matching the pattern of 6,. In fact, the period could be factor of or stabilizes where the period can be considered 1. Thus analyze 7j in P(D) and [[O2] in P(H), obtained from Monte-Carlo sampling along certain amount of episodes.
Fig. 6 (a) assume the true hypothesis travel along the triangular path connecting the 3 columns of M (shown in blue crosses). We found that GBT learners with in the interior of the cube (general GBT) produce posterior path of period p. while the posteriors of Bayesian and SCBI learners tend to converge (Fig. (b-e)). Thus a general GET learner can naturally detect the periodicity of the 400 steps and 10240 repeats for each learner.
world. We simulate up
Moreover, we discovered that general GBT learner's posteriors converge to curve whose area proportional to the area of the path of true hypotheses. In Fig. 7 as the path of true hypotheses vary by radius and shapes, the ratio (shown as slope) between both areas tends be the same, it is 0. 0620 in (a) and O. 1616 in (d), which suggests that this ratio is independent from the path of

---

(e). 200)
0.75
12. D m)
Bayeslar
0.20
0.15,18
11 110 desh dots
0.16.
0:25
0.1.88
0.0.0 m1
013
mean (1:00p bestel
0.11
(c) stro (1.10 101
(d), KE1O Scai
0:59
0.03
11:161
0225
12 15
(e). number of perfoes (0)
Figure 6: (a). Setup: each dot represents Tı: data sampled along the dots from red yellow of period 18, M columns represented by the blue crosses. (b-d). Bayesian, general GBT, SCBI learners, resp., blue shows E[O+] and red crosses E[6,] (mean of 18 consecutive B[Ox]'s). (e). for different learners (shown colors), plot (1) the averaged distance between E[O,] and its center v.s. number of periods, in solid lines and left y-ticks, and (2) the step-length of E(,,) between consecutive periods in dash-dots and right y-ticks.
1AB
peesang wito
Ares postertor cycle (-110
(b).
MD: end teachers
KDl. and teschers
posterfars
PEHI: posteriors
0.5
(a).
(x10
Teachers (d).
Area of teaching cycle (X20-2;
Figure 7: Behavior of a GBT learner with = (1, 10, 10) on two different paths of 17 with p 20, tested in 300 steps and 10240 episodes. M is fixed and represented by the red dots. Learner's posteriors form roughly periodic paths, small panels corners of (a, d), plot path of and posterior paths, ratio between their enclosed areas are shown in yellow to matched hetween paths and corresponding area ratios.
(b, e) shows the concentric similar paths that 77 follow. Colors are
Related Work.
Prior work defines and outlines basic properties of Unbalanced Optimal Transport
[Liero et al.
2018 Chizat al., 2018 Pham et al., 2020]] Bayesian approaches prominent
in machine learning Murphy 2012] and beyond Jaynes 2003 Gelman et al. 1995]. There is
also research cooperative learning [Wng al. 2019 2020b see also [iiu al. 2021 Yuan
et al. [2021] /hu 2015 Liu et al., 2017 Shafto and Goodman 2008 Shafto et al. 2014 Frank and
Goodman 2012 Goodman and Frank 2016 Fisac al.) 20017 Ho et al. 2018 Laskey et al. 2017].
Discriminative learning is the reciprocal problem in which one sees data and asks which hypothesis
best explains [Ng and Jordan| 200T Mandler 1980]. We are unaware of any work that attempts to unify and analyze the general problem of learning in which each of these are instances.
Conclusions
We have introduced Generalized Belief Transport (GBT), which unifies and parameterizes classic instances of learning including Bayesian inference, Cooperative Inference, and Discrimination. as Unbalanced Optimal Transport (UOT). We show that each instance is point in a continuous, differentiable on the interior, 3-dimensional space defined by the regularization parameters of UOT. Moreover, to demonstrate general GBT's capacity of supporting generalized learning, we prove and illustrate asymptotic consistency and estimate rates convergence, including convergence to hypotheses with zero prior support, and ability of gripping dynamic of the world. In summary, GBT unifies very different modes of learning, yielding a powerful, general framework for modeling learning agents.
10

---

Acknowledgments and Disclosure of Funding
This research was supported in part by DARPA awards HR001 -22-99-000 and W912CG22C000 and NSF MRI 21 21122 to P.S.
References
Lactitia Chapel, Rémi Flamary, Haoran Wu, Cédric Févotte, and Gilles Gasso. Unbalanced optimal transport through non-negative penalized linear regression. Advances in Neural Information Processing Systems, 34:23270- 23282, 2021.
Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xarie Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):2563-222 2018.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pages 2292-239222222 2300, 2013.
Paul Dagum, Adam Galper, and Eric Horvitz. Dynamic network models for forecasting. In Uncer- tainty in artificial intelligence, pages 41-48. Elsevier, 1992.
Joseph L Doob. Application of the theory of martingales. Le calcul des probabilites et ses applications, pages 23-27, 1949.
Stephen E Fienberg al. An iterative procedure for estimation in contingency tables. The Annals of Mathematical Statistics, 4 41(3):907-917, 1970.
Jaime Fisac, Monica Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menelle Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan. Pragmatic- pedagogic value alignment. arXiv preprint
2017.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998-99)) 2012.
Andrew Gelman, John Carlin, Hal S Ster,, and Donald B Rubin. Bayesian data analysis. Chapman and Hall/CRC, 1995.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818822) 2016.
Mark K Ho. Michael L Littman, Fiery Cushman, and Joseph L Austerweil. Effectively learning from pedagogical demonstrations. In Proceedings of the Annual Conference of the Cognitive Science Society, 2018.
Edwin Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453444 1946.
Michael Laskey, Caleb Chuck, Jonathan Lee, Jeffrey Mahler, Sanjay Krishnan, Kevin Jamieson. Anca Dragan, and Ken Goldberg. Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 358-365. IEEE, 2017.
Matthias I. Liero, Alexander Mielke, and Giuseppe Savaré. Optimal entropy-transport problems and new hellinger- kantorovich distance between positive measures. Inventiones mathematicae, 21 (3)( 969-1117, 2018.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rebg. and Le Song. Iterative machine teaching. In International Conference on Machine Learning, pages 2149-2156 PMLR. 2017.

---

Weiyang Liu, Zhen Liu, Hanchen Wang, Liam Paull, Bernhard Schölkopf, and Adrian Weller. Iterative teaching by label synthesis. Advances Neural Information Processing Systems, 34, 2021.
George Mandler. Recognizing: The judgment of previous occurrence. Psychological review, 87(3):
252, 1980.
Kevin P Murphy, Machine learning: probabilistic perspective. MIT press, 2012.
Kevin Patrick Murphy. Dynamic bayesian networks: represenlation, inference and learning. Univer- sity of California, Berkeley, 2002.
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of
logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 1 1(5-6))355-6077 2019.
Khiem Pham, Khang Le, Nhat Ho. Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of sinkhorn algorithm. In International Conference on Machine Learning, pages 7673 -682.. PMLR, 2020.
Julien Rabin, Sira Ferradans, and Nicolas Papadakis. Adaptive color transfer with relaxed optimal transport. In 2014 IEEE international conference on image processing (ICIP), pages 4852-4856. IEEE, 2014.
Christian Robert, Nicolas Chopin, and Judith Rousseau. Harold jeffreys's theory of probability revisited. Statistical Science, 24(2):141- (412)):411177) 2009.
Patrick Shafto and Noah D. Goodman. Teaching games: Statistical sampling assumptions for learning in pedagogical situations. In Pruceedings of the 30th annual conference of the Cognitive Science Society, Austin, TX, 2008. Cognitive Science Society.
Patrick Shafto, Noah D Goodman, and Thomas L Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive Psychology. 71:55-89, 2014.
Patrick Shafto, Junqi Wang, and Pei Wang. Cooperative communication as belief transport. Trends in cognitive sciences, 25(10):826-828, 2021.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
Junqi Wang, Pei Wang, and Patrick Shafto. Sequential cooperative bayesian inference. In Internationa Conference on Machine Learning, pages 10039-10049, PMLR, 2020a.
Pei Wang, Pushpi Paranamana, and Patrick Shafto. Generalizing the theory of cooperative inference. AlStats, 2019.
Pei Wang. Junqi Wang. Pushpi Paranamana, and Patrick Shafto. A mathematical theory of cooperative communication. Advances in Neural Information Processing Systems, 33, 2020b.
Scott Cheng-Hsin Yang, Yue Yu, Arash Givchi, Pei Wang, Wai Keen Vong, and Patrick Shafto. Optimal cooperative inference. In AISTATS, volume 84 of Proceedings of Muchine Learning Research, pages 376-385. PMLR, 2018.
Luyao Yuan, Dongruo Zhou, Junhong Shen, Jingdong Gao, Jeffrey L Chen, Quanquan Gu, Ying Nian Wu, and Song-Chin Chun Zhu. Iterative teacher-aware learning. Advances in Neural Information Processing Systems, 34:29231-29245 2021.
Xiaojin Zhu. Machine teaching: An inverse problem machine learning and an approach toward optimal education. In Twent-Nimth AAA/ Conference on Artificial Intelligence, 2015.
12


Reviews:
[{"review_id": "Yd0Sx5EIKr", "reviewer": "Reviewer_vUNg", "summary": "This paper proposes a framework called Generalized Belief Transport that unifies different types of machine learning models. The authors have proven a number of properties about their proposed framework.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The problem proposed seems relevant given the different number of approaches to machine learning. If successful the proposed framework can greatly unify prior work.", "weaknesses": "* I do not have any prior knowledge about the problem studied in this paper at all, so I found the paper to be difficult to understand. That might not necessarily be the fault of the authors, but I will defer to the experts in this area to comment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, please see further clarification on the practical implications in the general response. We are hopeful that you will appreciate the importance of this work!"}, {"review_id": "0optdxrAGb", "reviewer": "Reviewer_7Pt6", "summary": "The authors introduce the concept of Generalized Belief Transport to unify and parameterize 3 different axes of learning from within the formalism of Unbalanced Optimal Transport.  Corresponding limits points on the \"cube\" of these 3 axes recapitulate many common learning paradigms from the literature (e.g., bayesian inference, frequentist inference, cooperative communication, etc.).  The authors further demonstrate an algorithm for solving some synthetic instances of these various problems, and demonstrate various tradeoffs as one moves from one type of learning to another.", "questions": "see above", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:  The connections between these formalisms through the lens of UoT (and the proofs) are certainly novel.\n\nQuality: The overall quality of presentation is high, and I particularly like the \"cube\" device for visualizing the different limit points.\n\nClarity: The presentation is incredibly straightforward, and the proofs are quite clear.\n\nSignificance: This is less clear, but a more general formalism that can automatically facilitate many different, familiar forms of learning seems instrumentally valuable for further methods development.\n", "weaknesses": "How useful this formalism will _actually_ be for practitioners is my primary point of uncertainty.  Typically, any given practitioner doesn't really have any doubt about where they are on the \"cube\" for their learning problem, and thus they throw the most powerful technique available to them at that particular point of learning-agent-space.  This work might be better served by a less synthetic example, i.e., an example problem that _requires_ moving between points in learning-agent-space, to which this formalism would actually be uniquely well suited.  But a good (better yet, a *convincing*) example of such a thing is not obvious to me.  tldr: how do I actually use this formalism, if I know exactly where I am in learning agent space, and why wouldn't I use one of the more well-known tools?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your generous comments and suggestions.\nWe are grateful that you find our utilization of UOT is novel and our presentation is clear and straightforward.\n\nMost importantly, your request for a *convincing* example hit home with us. The proposed addition to the introduction is, we believe, convincing. We would very much appreciate your thoughts on that example, and hope you agree!\n\nRegarding your concerns on `how useful this formalism will actually be for practitioners', \nplease see detailed clarification in the general response.\nOn the high level, we believe (1) models lie in the interior of the cube provide capabilities beyond existing models, which worth systematic exploration.\n(2) more importantly, rather than create a new model for the modeler, GBT is developed from agents point of view (thank you for pointing this out, we will add a clarification in revision). \nIn practice, agents do not know the situation well enough to identify an optimal model in the cube at the beginning of a mission.\nThey need to find their way towards the optimal model as data come in. The cube represents a space of possible ways agent may learn about the world.  Such interpolation can be done by gradually changing the agent's $\\epsilon$ as shown on Page11 Fig 1 of supplementary material. Algorithms that effectively (and ideally efficiently) do this inference based on observed data is an important direction for future work. "}, {"review_id": "9gjxhj7koL", "reviewer": "Reviewer_NpBr", "summary": "Standard models of machine learning treat different internal constraints (e.g., prior knowledge) and external constraints (e.g., time availability, environmental non-stationarity) as separate problems, and thus hinder the development of unified learning agents. This paper proposes a framework called Generalized Belief Transport (GBT), which builds upon Unbalanced Optimal Transport, to unify existing learning models. GBT offers a parameterization that allows for interpolation between different modes such as Bayesian inference, Frequentist inference, cooperative learning, and discriminative learning. The authors provide theoretical analyses and empirical investigations to support their claims.\n\nIn short, the paper makes the following contributions:\n- It proposes a parameterization that unifies existing learning approaches and shows its continuity and differentiability. \n- It analyzes the behavior of GBT under variations in the parameter space. \n- It studies sequential GPT for both static and non-static settings.\n", "questions": "- How is the area for the GBT learner’s posteriors measured for Figure 6 (the curves do not seem to be closed)?\n\n- Are there any existing efforts to combine discriminative learning methods with Bayesian approaches (class priors) to tackle a learning problem? How does the proposed framework handle such integration?\n", "limitations": "Given the lack of previous work attempting to unify and analyze the general problem of learning under constraints, what are the potential research directions and open questions in this area? How can the field benefit from a unified framework for learning and inference?", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality & Significance**\n- The paper presents an interesting point of view that unifies various learning models with different internal and external constraints. The proposed GBT framework allows flexible combinations of $(\\epsilon_P, \\epsilon_\\eta, \\epsilon_\\theta)$ and helps the development of unified learning agents.\n\n**Quality & Clarity**\n- The paper is well-structured and provides a clear explanation of the challenges and the proposed framework. The theoretical analyses conducted to establish continuity, differentiability, and behavior under parameter variations help improve the understanding of the framework. The empirical investigations on sequential learning and predictive performance under environmental drift also demonstrate the practical applicability of GBT. The authors provide the proofs for the key theorems and the code for reproducing experiments in the supplementary material.\n", "weaknesses": "**Empirical significance**\n- It is very interesting to unify seemingly different learning models like the Frequentist and the Bayesian. However, the paper touches vaguely on the practical implications of such a unified framework. For instance, are there trade-offs for taking different points in the parameter space? How does the proposed framework extrapolate? Does it lead to new learning methods that are previously under-explored? What are some practical future steps for improving/utilizing the proposed framework?\n\n\n**Presentation**\n- The paper assumes a very high level of familiarity with concepts such as the Frequentist-Bayesian dichotomy and generative vs. discriminative approaches. Readers without extensive prior knowledge in these areas may find it challenging to fully grasp the nuances of this work. Including additional explanations or references to relevant background material would enhance the accessibility of the paper.\n\n**Other details**\n- Section 1: missing definition of $\\eta$ (estimation of the data distribution) in the first 4 paragraphs.\n- The arrangement of the color bars in Figure 2 (left) can be more organized. Currently it’s hard to read and distinguish the setting names.\n- Color legend is missing for Figure 3.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive suggestions and insightful comments.\nWe are excited that you find our proposed 'framework allows flexible combinations of and helps the development of unified\nlearning agent'. Please see our answers to your questions below.\n\n* Empirical significance and limitations: thank you so much for the insightful comment on how we shall improve our presentation on these points. We have organized our thoughts in the general response, please see details there.\n\n* Presentation:  as suggested, we will add a preliminary section on existing learning models prior introduce GBT framework (around line 63, prior section 1.1). Thank you for the detail comments, they will also be addressed in revision.\n\n* Question: 'How is the area for the GBT learner’s posteriors measured for Figure 6 (the curves do not seem to be closed)?'\n\n* Answer: You are right, the entire curve is not closed. To calculate the area, we plotted the mean posterior data from round 1 to round 300 on $\\mathcal{P}(\\mathcal{D})$ (the unilateral triangle), \nand observed that these points become periodic of size 20 (the period of the true hypothesis) as data increase. Thus we divided the mean posterior data from round 1 to round 300 into 15 full periods of length 20.  For each period, consider the polygon spanned by the 20 points, its shape stabilized after initial periods. Hence, we used average areas of last a few periods as the area of GBT learner’s posteriors.\n"}, {"review_id": "neRjkUItIn", "reviewer": "Reviewer_1jqk", "summary": "This work aims to unify learning approaches by specifying a 3D space where the dimensions represent modalities of learning that can be combined to specify various learning approaches, e.g., Bayesian and Frequentist approaches. The 3D space is defined by three learning constraints within Unbalanced Optimal transport (UOT).", "questions": "Does this unification hold with infinite hypotheses or datasets? How might one use this unification to design new algorithms? ", "limitations": "This work is not presented in a digestible manner; the impact of this work is not effectively communicated or demonstrated.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This work's conceptual aims are quite interesting and potentially illuminating and instructive.", "weaknesses": "Many details are glossed over in the proofs, making them hard to read and understand. This work is limited to finite datasets and finite hypotheses, which significantly limits the generality of the work. Minimally, the hypothesis class' finiteness limits this unification's informativeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your generous comments and suggestions.\n\n* We are surprised, given the other reviewers' assessments, and disappointed that you find our paper hard to read.\nIt would be helpful if you provided specifics regarding what details were glossed over. Without that information it is not possible to respond to your critique or improve the paper.  \n\n* We respectfully disagree with your assessment on how finiteness limits our contributions.\n\nRather than being a limitation, we would like to point out that our approach in fact allows generalization to continuous setting.\nGBT captures existing learning models as unbalanced optimal transport (UOT) problems.   \nTheory and algorithms for continuous UOT were developed in [Chizat et al. 2018],\nwhich provide necessary machinery to go for continuity. We echo the reviewer that a good learner should not be limited by a pre-selected 'hypothesis class' (finite or infinite), which is a drawback of many learning model now. As detailed in general response (1a), Proposition 12 demonstrated that there are GBT learners who are able to learn hypothesis which was not in their initial hypothesis set. This potentially provides a more fundamental approach to overcome such limitation. \n\nOn the other hand, for practice and implementation purposes, most if not all machine learning models are done for finite samples. \nWe strongly disagree that finiteness (or not) limits the merit of this submission. \n\n* Towards question 'How might one use this unification to design new algorithms?', because the entire space of learning models is parameterized (can be viewed as a cube in Fig.1), new learning models can be explored naturally by varying the parameters. \nExisting models are mainly located on vertices of the cube, in this paper we explored two new types of learner in section 3 and section 4, please see details in the general response. For algorithm, as each model on the cube is an UOT problem, hence can be solved by any algorithm developed for UOT. The most popular approach is Sinkhorn iteration, which we included as Algorithm 1 on line 88.\n"}, {"review_id": "jgSJ0ezIA5", "reviewer": "Reviewer_mBxm", "summary": "This paper describes a unifying view at various learning settings in machine learning, such as Bayesian inference, optimal communication, supervised classification and frequentist inference. All three learning settings are shown as special cases of an objective function, which can be interpreted from the lens of optimal transport.", "questions": "* Does this work relate to Generalised Belief Propagation [1][2]? Or fit in this [3] hierarchy of inference methods?\n\n[1] Yedidia, Jonathan S., William Freeman, and Yair Weiss. \"Generalized belief propagation.\" NeurIPS 2000\n\n[2] Teh, Yee, and Max Welling. \"The unified propagation and scaling algorithm.\" NeurIPS 2001\n\n[3] Rosen-Zvi, Michal, Michael I. Jordan, and Alan Yuille. \"The dlr hierarchy of approximate inference.\" arXiv (2012).\n", "limitations": "-", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed objective function generalises such learning concepts in a general equation. This generalisation sheds new light on the various methods. ", "weaknesses": "The current paper generalises the learning paradigms, but what new insight can be gained from this? Does this generalisation give rise to a new learning method or a more efficient version of an existing model that would solve an existing problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your generous comments and suggestions.\nWe are grateful that you find our proposed framework generalises learning concepts in a general form.\n\n* Regarding your comment on 'what new insight can be gained from this': thank you for the guiding questions on how we shall emphasize implications of GBT.\nWe have organized our thoughts in the general response, please see details there.\nOn the high level, our generalisation give rise to new learning model (models lie in the interior of the cube) that provide capabilities beyond existing models, which worth to be explored, and lay out a platform where interpolations between these models can be properly investigated. \n\n* Question: 'Does this work relate to Generalised Belief Propagation'? \nThe provided citations are from a different perspective. Our emphasis is not on approximate inference (though there is inference, i.e. learning algorithms), but rather on formalization of the problems themselves.\n"}]

Author rebuttal:
We would like to thank all reviewers for their generous comments and suggestions. 
Here we clarify the practical implications of proposed framework.
Specific comments are addressed for each reviewer separately.

We are encouraged that all reviewers agree that Generalized Belief Transport (GBT) establishes a uniform mathematical foundation for a broad class of learning models, upon which, basic questions in learning can be answered rigorously.

* We agree with Reviewer 7Pt6 "This work might be better
served by a less synthetic example, i.e., an example problem that requires moving
between points in learning-agent-space, to which this formalism would actually be uniquely well suited".
Here is a concrete example:

``Suppose a learner observing an *agent* behaving in an environment. As an observer, one may wish to learn about the environment from the agent's actions. However, any inferences one draws depend on beliefs about the agent. How is the agent updating their beliefs? Do they have stable goals, or are they changing over time? Perhaps the agent is selecting actions to communicate what they know? In order to draw inferences over these possibilities, one must parameterize the space, ideally in such a way one could optimize over the possibilities. Indeed, with such a framework, one would be able to naturally interpolate between classic dichotomies such as Bayesian and frequentist, static and dynamic environments, and helpful versus neutral agents. We propose such a framework. ''

We will insert this text in the introduction between the second and third paragraphs, to help the reader understand the need for our generalized framework.

* Towards answering "what are the potential research directions and open questions in this area? 
How can the field benefit from a unified framework for learning and inference? what
new insight can be gained from this?" (Reviewer NpBr and Reviewer mBxm),
here are our thoughts:


(1) Because the entire space of learning models is now parameterized (can be viewed as a cube in Fig.1), 
new learning models can be explored naturally, 
or even optimized with respect to particular tasks in the feature. 
Existing models are mainly located on vertices of the cube, in this paper we take a first probe into the cube with two cases below. 
Full exploration of the cube and an effective optimization routine are left to future work, to avoid over complicating an already technically detailed paper. 

(a) In section 3,  we proved that there are learners in the cube can learn a new hypothesis naturally.
A drawback of Bayesian inference is that only hypotheses in the original hypothesis set can be learned. 
Proposition 12 showed that GBT learners with $\epsilon_{\eta} = \epsilon_{\theta} = \infty, \epsilon_P\in (0,\infty)$ are able to learn *any* hypothesis, which begins to approximate the flexibility of human learning.

(b) In section 4, we demonstrated learners in the cube who are capable of learning in dynamic environments (Fig.5 and Fig.6).
To be more concrete (as suggested by Reviewer 7Pt6), consider the situation where a learner observes data from a ground truth that is dynamic.
For example the weather gradually changes, the climate slowly drifts over time, learners learn with experience, etc. 
Figure 5 shows when the ground truth travel along a triangular path (Fig.5a), 
Bayesian learner converges to a fixed hypothesis on a vertex (Fig.5b) whereas GBT learner with parameters $(1, 10, 10)$ was able to detect there is a cyclic pattern (Fig.5c).


(2) The differentiability of GBT paves the path for online interpolation between learning models.
Here are several cases where movement in the space yields interesting, novel theory:

(a) It is popular in the state of art machine learning models that an agent learns probabilistically, 
but makes decisions greedily.
This heuristic represents a path where a big leap on the cube was taken at the last step. 
An interesting question is under what circumstances this is optimal, what are the trade-offs, and under what conditions smoother trajectories are preferable. 
 
(b) When we communicate with others, human learners can move from Bayesian inference to cooperative communication gradually, 
which involves recognizing that the people select data purposefully, rather than sampling at random, conditional on the hypothesis they wish to convey. 
Such smooth interpolations can be achieved by modifying a learner's location on the cube as demonstrated in Section 3.1 (page 11 Fig.1) in the supplement materiel.
As Reviewer 7Pt6 pointed out most of existing research focus on the case where "practitioner doesn't really have any doubt about where they are on the cube".
However, in reality, both environment and other agents are dynamically changing, 
we believe GBT can facilitate research in the direction of building a learner who is able to adopt appropriate learning models based on incoming data, 
rather than learning in a fixed model.


Meta-review: This paper proposes a framework called Generalized Belief Transport that unifies different types of machine learning models and allows for interpolation between different modes such as Bayesian inference, Frequentist inference, cooperative learning, and discriminative learning.  

The authors further demonstrate an algorithm for solving some synthetic instances of these various problems and demonstrate various tradeoffs as one moves from one type of learning to another.

Some reviewers found to paper difficult to understand, while others commented that the “presentation is incredibly straightforward, and the proofs are quite clear” and the paper “provides a clear explanation of the challenges and the proposed framework”

A weakness of the paper is that it touches only vaguely on the practical implications of the proposed model.
Decision: Accept (poster)