Bounding and Approximating the Loss of Likelihood
In this section, we theoretically quantify the impact of edge pruning on model performance. In particular, we establish an upper bound on the log-likelihood drop \Delta\mathcal{LL} on a given dataset \mathcal{D} by comparing (i) the original PC \mathcal{C} and (ii) the pruned PC \mathcal{C}_{\backslash\mathcal{E}} caused by pruning away edges \mathcal{E} :

\Delta\mathcal{LL}(\mathcal{D},\mathcal{C},\mathcal{E})\!=\!\mathcal{LL}(\mathcal{D},\mathcal{C})-\mathcal{LL}(\mathcal{D},\mathcal{C}_{\backslash\mathcal{E}})

We start from the case of pruning one edge (i.e., |\mathcal{E}|\!=\!1 in Equation 2 ). In this case, the loss of likelihood can be quantified exactly using flows and edge parameters:

For a PC \mathcal{C} and a dataset \mathcal{D} , the loss of log-likelihood by pruning away edge (n,c) is

\Delta\mathcal{LL}(\mathcal{D},\mathcal{C},\{(n,c)\})\!=\!\frac{1}{|\mathcal{D}|}\!\!\sum_{\bm{x}\in\mathcal{D}}\!\log\!\left(\frac{1-\theta_{c|n}}{1\!-\!\theta_{c|n}\!+\!\theta_{c|n}\operatorname{F}_{n}(\bm{x})\!-\!\operatorname{F}_{n,c}(\bm{x})}\!\right)\!\leq\!\frac{-1}{|\mathcal{D}|}\!\sum_{\bm{x}\in\mathcal{D}}\log(1\!-\!\operatorname{F}_{n,c}(\bm{x})).

See proof in Appendix B.1 .
By computing the second term in Theorem 1 , we can pick the edge with the smallest log-likelihood drop.
Additionally, the third term characterizes the log-likelihood drop without re-normalizing parameters of \theta_{\cdot|n} . It suggests pruning the edge with the smallest edge flow.
A key insight from Theorem 1 is that the log-likelihood drop
depends explicitly on the edge flow \operatorname{F}_{n,c}(\bm{x}) and unit flow \operatorname{F}_{n}(\bm{x}) .
This matches the intuition from Section 3 and suggests that the circuit flow heuristic proposed in the previous section is a good approximation of the derived upper bound.

Next, we bound the log-likelihood drop of pruning multiple edges.

Let \mathcal{C} be a PC and \mathcal{D} be a dataset.
For any set of edges \mathcal{E} in \mathcal{C} ,
if \forall\bm{x}\!\in\!\mathcal{D},\sum_{(n,c)\in\mathcal{E}}\operatorname{F}_{n,c}(\bm{x})<1 , the log-likelihood drop by pruning away \mathcal{E} is bounded and approximated by

\begin{split}\Delta\mathcal{LL}(\mathcal{D},\mathcal{C},\mathcal{E})\leq-\frac{1}{|\mathcal{D}|}\sum_{\bm{x}}\log(1-\sum_{(n,c)\in\mathcal{E}}\operatorname{F}_{n,c}(\bm{x}))\approx\frac{1}{|\mathcal{D}|}\sum_{(n,c)\in\mathcal{E}}\operatorname{F}_{n,c}(\mathcal{D}).\end{split} (3)

Proof of this theorem is provided in Appendix B.2 . We first look at the second term of Equation 3 . Although it provides an upper bound to the performance drop, it cannot be used as a pruning heuristic since the bound does not decompose over edges. And hence finding the set of edges with the lowest score requires evaluating the bound exponentially many times with respect to the number of pruned edges.
Therefore, we do an additional approximation step of the bound via Taylor expansion, which leads to the third term of Equation 3 . This approximation matches the eFlow heuristic by a constant factor 1/|\mathcal{D}| , which theoretically justifies the effectiveness of the heuristic. Figure 6 empirically compares the actual log-likelihood drop and the quantity computed from the circuit flow heuristic (that is, the approximate upper bound) for different percentages of pruned parameters. We see that the approximate bound matches closely to the actual log-likelihood drop.