B.1 Pruning One Edge over One Example
For a PC \mathcal{C} and a sample \bm{x} , the loss of log-likelihood by pruning away edge (n,c) is

\Delta\mathcal{LL}(\{\bm{x}\},\mathcal{C},\{(n,c)\})=\log\left(\frac{1-\theta_{c|n}}{1\!-\!\theta_{c|n}\!+\!\theta_{c|n}\operatorname{F}_{n}(\bm{x})\!-\!\operatorname{F}_{n,c}(\bm{x})}\right)\leq-\log(1\!-\!\operatorname{F}_{n,c}(\bm{x})).

For notation simplicit, denote the probability of units m (resp. n ) in the original (resp. pruned) PC given sample \bm{x} as {p}_{m}(\bm{x}) (resp. {p}^{\prime}_{n}(\bm{x}) ). As a slight extension of Definition 4 , we define F_{n}(\bm{x};m) as the flow of unit n w.r.t. the PC rooted at m .

The proof proceeds by induction over the PC’s root unit. That is, we first consider pruning (n,c) w.r.t. the PC rooted at n . Then, in the induction step, we prove that if the lemma holds for PC rooted at m , then it also holds for PC rooted at any parent unit of m . Instead of directly proving the statement in Lemma 1 , we first prove that for any root node m , the following holds:

{p}_{m}(\bm{x})-{p}^{\prime}_{m}(\bm{x})=F_{n}(\bm{x};m)\cdot p_{m}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};m)}{\operatorname{F}_{n}(\bm{x};m)}-\frac{\theta}{1-\theta}\bigg{)}

Base case: pruning an edge of the root unit. That is, the root unit of the PC is n . In this case, we have

{p}_{n}(\bm{x})-{p}^{\prime}_{n}(\bm{x}) =\sum_{c^{\prime}\in\mathsf{in}(n)}\theta_{c^{\prime}|n}\cdot p_{c}(\bm{x})-\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}^{\prime}\cdot p_{c}^{\prime}(\bm{x}) =\theta_{c|n}\cdot p_{c}(\bm{x})+\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}\cdot p_{c}(\bm{x})-\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}^{\prime}\cdot p_{c}(\bm{x}), (5)

where \theta_{c|n}^{\prime} denotes the normalized parameter corresponding to edge (n,c) in the pruned PC.
Specifically, we have

\forall m\in\mathsf{in}(n)\backslash c,\quad\theta^{\prime}_{m|n}=\frac{\theta_{m|n}}{\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}}=\frac{\theta_{m|n}}{1-\theta_{c|n}}.

For notation simplicity, denote \theta:=\theta_{c|n} . Plug in the above definition into Equation 5 , we have

{p}_{n}(\bm{x})-{p}^{\prime}_{n}(\bm{x}) =\theta_{c|n}\cdot p_{c}(\bm{x})+\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}\cdot p_{c}(\bm{x})-\frac{1}{1-\theta}\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}\cdot p_{c}(\bm{x}) =\theta_{c|n}\cdot p_{c}(\bm{x})-\frac{\theta}{1-\theta}\sum_{c^{\prime}\in\mathsf{in}(n)\backslash c}\theta_{c^{\prime}|n}\cdot p_{c}(\bm{x}) =\theta_{c|n}\cdot p_{c}(\bm{x})-\frac{\theta}{1-\theta}(p_{n}(\bm{x})-\theta_{c|n}p_{c}(\bm{x})) =\frac{1}{1-\theta}\cdot\theta_{c|n}\cdot p_{c}(\bm{x})-\frac{\theta}{1-\theta}\cdot p_{n}(\bm{x}) \overset{(a)}{=}\frac{1}{1-\theta}\cdot p_{n}(\bm{x})\cdot\frac{\operatorname{F}_{n,c}(\bm{x};n)}{\operatorname{F}_{n}(\bm{x};n)}-\frac{\theta}{1-\theta}\cdot p_{n}(\bm{x}) =F_{n}(\bm{x};n)\cdot p_{n}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};n)}{\operatorname{F}_{n}(\bm{x};n)}-\frac{\theta}{1-\theta}\bigg{)}, (6)

where (a) follows from the fact that F_{n}(\bm{x};n)=1 and F_{n,c}(\bm{x};n)=\theta_{c|n}{p}_{c}(\bm{x})/{p}_{n}(\bm{x}) .

Inductive case #1: suppose Equation 4 holds for m . If product unit d is a parent of m , we show that Equation 4 also holds for d :

 p_{d}(\bm{x})-p^{\prime}_{d}(\bm{x}) =\prod_{n^{\prime}\in\mathsf{in}(d)}p_{n^{\prime}}(\bm{x})-\prod_{n^{\prime}\in\mathsf{in}(d)}p^{\prime}_{n^{\prime}}(\bm{x}) =(p_{m}(\bm{x})-p^{\prime}_{m}(\bm{x}))\prod_{n^{\prime}\in\mathsf{in}(d)\backslash m}p_{n^{\prime}}(\bm{x}) \overset{(a)}{=}F_{n}(\bm{x};m)\cdot p_{m}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};m)}{\operatorname{F}_{n}(\bm{x};m)}-\frac{\theta}{1-\theta}\bigg{)}\cdot\prod_{n^{\prime}\in\mathsf{in}(d)\backslash m}p_{n^{\prime}}(\bm{x}) \overset{(b)}{=}F_{n}(\bm{x};d)\cdot p_{d}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};d)}{\operatorname{F}_{n}(\bm{x};d)}-\frac{\theta}{1-\theta}\bigg{)},

where (a) is the inductive step that applies Equation 6 ; (b) follows from the fact that (note that d is a product unit) F_{n}(\bm{x};m)=F_{n}(\bm{x};d) and F_{n,c}(\bm{x};m)=F_{n,c}(\bm{x};d) .

Inductive case #2: for sum unit d , suppose Equation 4 holds for m , where m\in\mathcal{A} iff m\in\mathsf{in}(d) and m is an ancester of n and c . Assume all other children of d are not ancestoer of n , we show that Equation 4 also holds for d :

 p_{d}(\bm{x})-p^{\prime}_{d}(\bm{x}) =\theta_{m|d}\cdot(p_{m}(\bm{x})-p^{\prime}_{m}(\bm{x})) =\theta_{m|d}\cdot F_{n}(\bm{x};m)\cdot p_{m}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};m)}{\operatorname{F}_{n}(\bm{x};m)}-\frac{\theta}{1-\theta}\bigg{)} =\theta_{m|d}\cdot F_{n}(\bm{x};m)\cdot p_{m}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};d)}{\operatorname{F}_{n}(\bm{x};d)}-\frac{\theta}{1-\theta}\bigg{)} =\theta_{m|d}\cdot F_{n}(\bm{x};d)\cdot\frac{\sum_{m^{\prime}\in\mathsf{in}(d)}\theta_{m^{\prime}|d}{p}_{m^{\prime}}(\bm{x})}{\theta_{m|d}{p}_{m}(\bm{x})}\cdot p_{m}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};d)}{\operatorname{F}_{n}(\bm{x};d)}-\frac{\theta}{1-\theta}\bigg{)} =F_{n}(\bm{x};d)\cdot\bigg{(}\sum_{m^{\prime}\in\mathsf{in}(d)}\theta_{m^{\prime}|d}{p}_{m^{\prime}}(\bm{x})\bigg{)}\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};d)}{\operatorname{F}_{n}(\bm{x};d)}-\frac{\theta}{1-\theta}\bigg{)} =F_{n}(\bm{x};d)\cdot{p}_{d}(\bm{x})\cdot\bigg{(}\frac{1}{1-\theta}\frac{\operatorname{F}_{n,c}(\bm{x};d)}{\operatorname{F}_{n}(\bm{x};d)}-\frac{\theta}{1-\theta}\bigg{)}.

Therefore, following Equation 4 for root r , we have

\frac{{p}_{r}(\bm{x})-{p}^{\prime}_{r}(\bm{x})}{p_{r}(\bm{x})}=\frac{1}{1-\theta}\operatorname{F}_{n,c}(\bm{x};r)-\frac{\theta}{1-\theta}F_{n}(\bm{x};r) \Leftrightarrow\quad \frac{{p}^{\prime}_{r}(\bm{x})}{p_{r}(\bm{x})}=1+\frac{\theta}{1-\theta}F_{n}(\bm{x};r)-\frac{1}{1-\theta}\operatorname{F}_{n,c}(\bm{x};r)

Therefore, we have

\Delta\mathcal{LL}(\{\bm{x}\},\mathcal{C},\{(n,c)\}) =\log{p}_{r}(\bm{x})-\log{p}^{\prime}_{r}(\bm{x}) =\frac{1}{|\mathcal{D}|}\sum_{\bm{x}\in\mathcal{D}}\log\left(\frac{1-\theta_{c|n}}{1\!-\!\theta_{c|n}\!+\!\theta_{c|n}\operatorname{F}_{n}(\bm{x};r)\!-\!\operatorname{F}_{n,c}(\bm{x};r)}\right) \overset{(a)}{\leq}-\log(1-F_{n,c}(\bm{x})),

where (a) follows from the fact that F_{n,c}(\bm{x})\leq F_{n}(\bm{x}) .
∎

Theorem 1 follows directly from Lemma 1 by noting that for any dataset \mathcal{D} , \Delta\mathcal{LL}(\mathcal{D},\mathcal{C},\{(n,c)\})=\frac{1}{|\mathcal{D}|}\Delta\mathcal{LL}(\{\bm{x}\},\mathcal{C},\{(n,c)\}) .