SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second part links  $C'$  to  $\tilde{C}$  in  $CTF_{k+1}$  such that  $C' \subseteq \tilde{C}$ . This is always possible since  $CTF_{k+1}$  is obtained after incrementally modifying  $CTF_{k,a}$  to add new factors. Thus, each clique in  $CTF_{k,a}$  is contained in some clique in  $CTF_{k+1}$ . For example,  $C_1' \subset \tilde{C}_2$  and the link is  $(C_1, C_1', \tilde{C}_2)$ .
Let  $L_{k}$  denote the set of links between  $CTF_{k}$  and  $CTF_{k+1}$ . We refer to the modified data structure consisting of the sequence of calibrated CTFs,  $SCTF = \{CTF_{k}\}$  and a list of links between all adjacent CTFs,  $SL = \{L_{k}\}$ , as the sequence of linked CTFs (SLCTF). Once the SLCTF is created, starting from the final CTF in the SLCTF, we successively back-propagate beliefs to the preceding CTFs via links between adjacent CTFs. To back-propagate beliefs from  $CTF_{k+1}$  to  $CTF_{k}$ , we choose a subset of links in  $L_{k}$  based on heuristics, which will be discussed later. Then, for each selected link  $(C, C', \tilde{C})$ , we update the belief associated with clique  $C$  as follows.
$$
\beta_ {\text {u p d a t e d}} (C) = \left(\frac {\beta (C)}{\sum_ {C \backslash V _ {i}} \beta (C)}\right) \sum_ {\tilde {C} \backslash V _ {i}} \beta (\tilde {C}) \tag {4}
$$
where,  $V_{l} = C \cap C'$ . This is followed by one round of message passing from  $C$  to all other cliques in the CT containing  $C$ . Once all CTFs are re-calibrated, we infer the marginal distribution of a variable using Equation 2 from the last CTF in which it is present.
For BNs, if incremental build is performed by adding variables in the topological order, then as shown in Theorem 1, the singleton marginals are consistent in CTFs  $\{CTF_{k\geq I_E}\}$ , where  $I_{E}$  is the index of the last CTF to which the factor corresponding to an evidence variable is added. Therefore, in this case, back-propagation of beliefs can be performed starting from  $CTF_{I_E}$  instead of starting from the last CTF. This reduces the effort required for belief update.
Heuristics for choice of links: To ensure that beliefs of all CTs in  $CTF_{k}$  are updated, at least one link must be chosen for each CT. It is also clear that for any CT in  $CTF_{k}$  more than one link may be required since variables that have low correlations in  $CTF_{k}$  could become tightly correlated when new factors are added in  $CTF_{k+1}$ . However, belief update via all links is expensive since a round of message passing is required for each link. Based on results over many benchmarks, we propose the following greedy heuristic to choose and schedule links for backward belief update.
(a) To minimize the number of selected links, we first choose a subset of link variables for which the difference in posterior margins in adjacent CTFs,  $CTF_{k}$  and  $CTF_{k + 1}$ , is greater than a threshold. Next, for belief update, we select a minimal set of links that cover these variables.
(b) The updated beliefs depend on the order in which the links are used for update. Based on the difference in marginals, we form a priority queue with the cliques containing link variables that have the lowest change in marginals having the highest priority. This is to make sure that large belief updates do not get over-written by smaller ones. This could happen for example, if two variables,  $v_{1}$  and  $v_{2}$ , that are highly correlated in  $CTF_{k}$  become relatively uncorrelated in  $CTF_{k+1}$  due to the approximation. Assume that new factors added to  $CTF_{k+1}$  affect  $v_{1}$  but not  $v_{2}$ . A belief update via the link containing  $v_{1}$  will make sure that its belief is consistent in  $CTF_{k}$  and  $CTF_{k+1}$ . Later, if we perform a belief update using a link containing  $v_{2}$ , the previous larger belief update of  $v_{1}$  will be overwritten by something smaller since the belief of  $v_{2}$  is not very different in the two CTFs.
**Complexity:** Let  $N_{CTF}$  be the number of CTFs in the sequence, and  $N_l$  be the maximum number of selected links between any two adjacent CTFs. IBIA requires  $2N_{CTF}$  rounds of message passing for calibrating CTFs and  $(N_{CTF} - 1)N_l$  rounds for belief update. Therefore, the runtime complexity of IBIA is  $O(N_{CTF}N_l2^{mcs_p})$ . To perform backpropagation of beliefs, all CTFs need to be stored and the memory complexity is  $O(N_{CTF}2^{mcs_p})$ .
All experiments were carried out on an Intel i9-12900 Linux system running Ubuntu 22.04.
Error metrics: For each non-evidence variable  $X_{i}$ , we measure error in terms of Hellinger distance between the exact marginal distribution  $P(X_{i})$  and the approximate marginal distribution  $Q(X_{i})$ . It is computed as follows.
$$
H D = \frac {1}{\sqrt {2}} \sqrt {\sum_ {s \in D o m a i n (X _ {i})} \left\{\sqrt {P \left(X _ {i} = s\right)} - \sqrt {Q \left(X _ {i} = s\right)} \right\} ^ {2}} \tag {5}
$$
We use two metrics namely, the average Hellinger distance (denoted as  $HD_{avg}$ ) and the maximum Hellinger distance (denoted as  $HD_{max}$ ) over all non-evidence variables in the network.
Table 1: Comparison of average  $HD_{avg}$  and average  $HD_{max}$  (shown in gray background) obtained using various inference methods with two runtime constraints, 2 min and 20 min. The minimum error obtained for each time limit is highlighted in bold. Entries are marked as '-' where all instances could not be solved within the set time limit. The total number of instances solved by each method is shown in the last row.  $ev_{a}$ : average number of evidence variables,  $v_{a}$ : average number of variables,  $f_{a}$ : average number of factors,  $w_{a}$ : average induced width and  $dm_{a}$ : average of the maximum variable domain size.
Total #Inst	Average stats (eva,va,fa,wa,dma)	2 min	20 min
LBP	WMB	IJGP	ISSwc	IBIA20	LBP	WMB	IJGP	ISSwc	IBIA23
BN	97	(76,637,637,28,10)	-	0.037	-	-	2E-4	0.023	0.025	-	-	6E-5
-	0.228	-	-	9E-3	0.230	0.170	-	-	2E-3
GridBN	29	(0,595,595,37,2)	0.075	0.066	0.011	0.003	5E-6	0.075	0.048	0.010	0.001	2E-7
0.478	0.416	0.111	0.051	7E-4	0.478	0.381	0.094	0.015	1E-4
Bnlearn	26	(0,256,256,7,16)	0.010	0.005	0.011	0.012	7E-5	0.010	5E-6	0.008	0.006	7E-6
0.089	0.021	0.050	0.064	0.002	0.089	1E-4	0.025	0.028	2E-4
Pedigree	24	(154,853,853,24,5)	0.075	0.018	0.035	0.033	0.009	0.075	0.015	0.033	0.021	0.008
0.555	0.253	0.470	0.292	0.204	0.555	0.194	0.446	0.234	0.198
Promedes	64	(7,618,618,21,2)	0.032	0.055	0.124	0.030	0.013	0.032	0.043	0.120	0.021	0.010
0.168	0.295	0.504	0.139	0.086	0.168	0.245	0.487	0.096	0.072
DBN	36	(653,719,14205,29,2)	-	0.069	0.081	0.016	0.020	-	0.018	0.060	2E-6	0.003
-	0.883	0.919	0.766	0.261	-	0.544	0.879	2E-4	0.098
ObjDetect	79	(0,60,210,6,16)	0.022	0.001	0.004	0.018	0.002	0.022	2E-4	3E-5	0.009	4E-4
0.130	0.010	0.037	0.189	0.020	0.130	0.003	3E-4	0.061	0.006
Grids	8	(0,250,728,22,2)	0.433	0.146	0.247	-	0.088	0.433	0.089	0.123	0.056	0.002
0.905	0.343	0.713	-	0.300	0.905	0.221	0.423	0.209	0.099
CSP	12	(0,73,369,12,4)	0.019	0.033	0.026	-	0.002	0.019	0.022	0.017	0.054	2E-4
0.066	0.101	0.134	-	0.011	0.066	0.057	0.073	0.093	0.003
Segment	50	(0,229,851,17,2)	0.035	1E-4	5E-6	5E-6	6E-5	0.035	5E-6	5E-6	5E-6	1E-7
0.258	0.002	7E-5	7E-5	0.001	0.258	7E-5	7E-5	7E-5	4E-7
Protein	68	(0,59,176,6,77)	5E-4	0.005	0.003	0.003	6E-4	5E-4	0.003	0.007	0.001	3E-5
0.007	0.102	0.094	0.049	0.039	0.007	0.066	0.230	0.015	0.002
#Inst	493	481	493	487	485	493	485	493	491	487	493
Benchmarks: We used the benchmark sets included in UAI repository [Ihler, 2006] and the Bayesian network repository [Scutari, 2007]. We classify instances for which exact solutions are present in the repository as 'small' and others as 'large'.
Methods used for comparison: In the UAI 2022 inference competition [UAI, 2022], the uai14-mar solver had the highest score for the MAR task. It is an amalgam of solvers that dumps solutions with different methods based on the given time and memory constraints. It uses loopy BP (LBP), generalized BP on loopy graphs where outer regions are selected using mini-bucket heuristics, and cutset conditioning of GBP approximations. The implementation of this solver is not publicly available. Therefore, we have compared our results individually with methods that belong to categories of methods used in uai14-mar. This includes LBP [Murphy et al., 1999], IJGP [Mateescu et al., 2010] and sample search [Gogate and Dechter, 2011] which is an importance sampling based technique that uses an IJGP based proposal and cutset sampling (referred to as 'ISSwc' in this paper). We also compare our results with weighted mini-bucket elimination (WMB) [Liu and Ihler, 2011]. Additional results showing a comparison with results published in Kelly et al. [2019] are included in the supplementary material.
Evaluation setup: The implementation of LBP and WMB were taken from LibDAI [Mooij, 2010, 2012] and Merlin [Marinescu, 2016] respectively. For IJGP and ISSwc, we used implementations [Gogate, 2010, 2020] provided by the authors of these methods. LBP, IJGP, ISSwc and WMB are implemented in  $\mathrm{C + + }$ . IBIA on the other hand has been implemented in Python3 and is thus, at a disadvantage in terms of runtime. We report results with runtime limits of  $2\mathrm{min}$  and  $20\mathrm{min}$  for small instances. In all cases, the memory limit was set to 8GB, which is the same as that used in UAI 2022 competition [UAI, 2022]. For IBIA, we set the maximum clique size bound  $mcs_{p}$  to 20 (referred to as 'IBIA20') when the time limit is  $2\mathrm{min}$  and we set it to 23 (referred to as 'IBIA23') when the time limit is  $20\mathrm{min}$ .  $mcs_{im}$  is empirically chosen as 5 less than  $mcs_{p}$ . The evaluation setup used for other methods is included in the supplementary material.
Results: Table 1 has the results for the small benchmark sets. It reports the average of  $HD_{avg}$  and  $HD_{max}$  over all instances in each set. We compare results obtained using LBP, WMB, IJGP, IBIA20, IBIA23 and ISSwc for both time constraints. The minimum error obtained for each time limit is marked in bold. IBIA20 and IBIA23 solve all small instances within 2 min and 20 min respectively. In 2 min, the accuracy obtained with IBIA20 is better than all other solvers for most benchmarks. For ObjDetect, Segment and Protein, it is comparable to WMB, IJGP/ISSwc and LBP respectively, which give the least errors for these testcases. In 20 min, IBIA23 gives lower or comparable errors in all testcases except DBN and ObjDetect. For DBNs, ISSwc reduces to exact inference in most instances and hence error obtained is close to zero. For ObjDetect, IJGP gives the least error closely followed by IBIA23. Note that for many benchmarks the accuracy obtained with IBIA20 in 2 min is either better than or comparable to the accuracy obtained with other solvers in 20 min.
A comparison of IBIA with results published in Kelly et al. [2019] for Gibbs sampling with Rao-blackwellisation (ARB) and IJGP is included in the supplementary material. It is seen that error obtained with IBIA is lower than both methods in majority of the testcases.
For BN instances, Table 2 compares the results obtained using IBIA20 when CTFs are constructed by adding factors in the topological order (columns marked as 'TP') with that obtained using a non-topological order (columns marked as 'NTP'). We compare the maximum error in partition function (PR) and the average  $HD_{max}$  over all instances in each benchmark set.
We observe that the topological ordering gives better accuracy for both PR and marginals in all testcases except Pedigree. The advantage of this ordering is that once all the evidence variables are added, no belief-update is needed is needed for the subsequent CTFs (using Theorem 1). So, the number of belief update steps is lower, resulting in lower errors. However, a drawback of this ordering is that it is rigid and it sometimes results in a larger number of CTFs in the sequence which could lead to larger errors if all the evidence variables are added in later CTFs. When no evidence variables are present (e.g.
Table 2: Comparison of maximum error in PR and average  $HD_{max}$  obtained using IBIA20 with CTFs constructed by adding factors in topological order (shown in columns marked 'TP') and that obtained using a non-topological order (shown in columns marked 'NTP').  $ev_{a}$ : Average number of evidence variables,  $\Delta_{PR} = |\log_{10}PR - \log_{10}PR^{*}|$  where  $PR$  and  $PR^{*}$  are estimated and exact values.
#Inst	eva	Max ΔPR	Avg HDmax
NTP	TP	NTP	TP
Bnlearn	26	0	0.02	0	0.023	0.002
GridBN	29	0	0.09	0	0.231	0.001
Promedes	64	7	1.5	0.4	0.322	0.086
BN	97	76	0.07	0.02	0.116	0.009
Pedigree	24	159	0.4	0.7	0.098	0.204
GridBN, Bnlearn), both runtime and memory complexity is lower with topological ordering since marginals are consistent in all CTFs (using Corollary 2) and belief update is not needed. The average runtime with and without topological ordering was 1s and 146s respectively for GridBN instances and 0.3s and 1.3s for Bnlearn testcases.
To evaluate the scalability of the proposed algorithm, we ran it for large networks where the exact solutions are not known. Table 3 tabulates the percentage of large instances in each benchmark set that could be solved using IBIA within  $2\mathrm{min}$ $20\mathrm{min}$  and  $60\mathrm{min}$ . For this experiment, we start with  $mcs_{p} = 20$  and allow it to increase if incremental build results in a CTF with larger clique sizes. IBIA could solve all large instances in benchmark sets BN, Promedes, ObjDetect and Segmentation and most instances in Protein within  $20\mathrm{min}$ . For other benchmarks, additional
Table 3: Percentage of large instances in each benchmark set solved by IBIA within 2, 20 and 60 minutes.  $ev_{a}$ : average number of evidence variables,  $v_{a}$ : average number of variables,  $f_{a}$ : average number of factors,  $w_{a}$ : average induced width and  $dm_{a}$ : average of the maximum domain-size.
Total #Inst	Average stats (eva, vα, fa, wα, dmα)	Instances solved (%)
2 min	20 min	60 min
BN	22	(188,1272,1272,51,17)	64	100	100
Promedes	171	(15,1207,1207,71,2)	77	100	100
ObjDetect	37	(0,60,1830,59,17)	27	100	100
Segment	50	(0,229,851,19,21)	100	100	100
Protein	395	(0,306,1192,21,81)	75	97	98
DBN	78	(784,944,47206,60,2)	38	77	77
Grids	19	(0,3432,10244,117,2)	16	37	58
CSP	54	(0,294,11725,175,41)	31	54	59
Type4b	82	(4272,10822,10822,24,5)	0	9	29
instances could be solved when the runtime was increased to  $60\mathrm{min}$ . The memory required for the remaining Grids, DBN, CSP and Protein instances is more than 8 GB. The increased memory usage is due to the following reasons. Firstly, all calibrated CTFs in the SLCTF need to be stored in order to allow for back-propagation of beliefs and the memory required increases with the number of CTFs. The average number of CTFs in the remaining Grid, DBN and CSP benchmarks is 22, 58 and 80 respectively. Secondly, for benchmarks with large variable domain sizes, the number of variables present in each clique in a CTF is small. Therefore, approximation using exact and local marginalization becomes infeasible and the subsequent CTFs have clique sizes greater than  $mcs_{p}$ , which results in increased memory usage. This is seen in 9 out of 395 Protein instances and 12 out of 54 CSP instances. In addition to memory, the remaining Type4b instances also require additional runtime. This is because during belief update of each CTF, we perform one round of message passing for each selected link and the number of links is large in these instances.
Limitations: While the belief update algorithm performs well for most benchmarks, it has some limitations. It is sequential and is performed link by link for each CTF that needs to be updated. The time and space complexity depends on the number of CTFs in the sequence and the number of selected links, which is large in some testcases. Also, after belief-update of all CTFs is completed, beliefs of variables present in multiple CTFs need not be consistent. However, good accuracies are obtained when beliefs are inferred from the last CTF containing the variable. For BNs, we found that building CTFs in the topological order gives larger errors in some cases. A possible extension would be to have an efficient build strategy where the ordering is decided dynamically based on the properties of the graph structure.
Comparison with related work: IBIA is similar to mini-bucket schemes in the sense that the accuracy-complexity tradeoff is controlled using a user-defined maximum clique size bound. While mini-bucket based schemes like IJGP [Dechter et al., 2002] and join graph linear programming [Ihler et al., 2012] use iterative message passing in loopy graphs, others like mini-bucket elimination (MBE), WMB [Liu and Ihler, 2011] and mini-clustering [Mateescu et al., 2010] are non-iterative approaches that approximate the messages by migrating the sum operator inside the product term. In contrast, IBIA constructs a sequence of clique trees. It performs belief propagation on approximate clique trees so that messages are exact and there are no issues of convergence.
Unlike sampling based techniques, there is no inherent randomness in IBIA that is each run gives the same results. There could be a variation if the order in which factors are added is changed. However, this variation is minimal since subsets of factors are added together in the incremental build step. In that sense, it is like mini-bucket based methods where results are the same if the variable elimination order and the partitioning technique used to generate the initial choice of mini-backets are the same.
In order to construct the sequence, IBIA requires a fast and accurate method to approximate clique trees by reducing clique sizes. The aim is to preserve as much as possible, the joint distribution of the interface variables. This is achieved by marginalizing out variables from large-sized cliques and a minimal set of neighbors without disconnecting clique trees. IBIA gives good accuracy since variables that are removed are chosen based on a mutual information (MI) based metric and a sufficient number of non-interface variables are retained so that a CT is never disconnected. In contrast, the Boyen Koller (BK) [Boyen and Koller, 1998, Murphy, 2002] and factored frontier (FF) approximations, although fast, retain only the interface variables which can disconnect the CT resulting in larger errors due to the underlying independence approximation. The thin junction tree approximations proposed in Kjærulff [1994] and Hutter et al. [2004] split cliques that contain a pair of variables that is not present in any of its sepsets. However, if there are large cliques that have only sepset variables (which is typical in most benchmarks), then the split has to be done iteratively starting from leaf nodes of multiple branches, until the large-sized clique can be split. When such cliques are centrally located in the CT, this process is both time-consuming and would result in approximation of a much larger set of cliques. Similar to BK and FF, this method can also disconnect the CT.
The other thin junction tree methods [Bach and Jordan, 2001, Elidan and Gould, 2008, Dafna and Guestrin, 2009] choose an optimum set of features based on either KL distance or a MI based metric. They cannot be directly used since IBIA requires a good approximation of the joint distribution of only the interface variables. Also, these methods are typically iterative and not very fast.
Yasaman Razeghi, Kalev Kask, Yadong Lu, Pierre Baldi, Sakshi Agarwal, and Rina Dechter. Deep bucket elimination. In International Joint Conference on Artificial Intelligence, pages 4235-4242, 2021.
Dan Roth. On the hardness of approximate reasoning. Artificial Intelligence, 82(1-2):273-302, 1996.
Marco Scutari. Bayesian network repository. https://www.bnlearn.com/bnrepository/, 2007. Accessed: 2021-10-15.
UAI. UAI 2022 - Competition. https://www.auai.org/uai2022/uai2022_competition, 2022.
Jonathan S Yedidia, William T Freeman, Yair Weiss, et al. Generalized belief propagation. In Advances in Neural Information Processing Systems, volume 13, pages 689-695, 2000.