SlowGuess/NAACL_2021_OCR huggingface dataset:
Experimental details for Section 5.2 We convert the annotated constituency parses using the Stanford CoreNLP package (Manning et al., 2014). We compute conditional MI and conditional PMI using the bert-base-cased model and run Gibbs sampling for 2000 steps. BERT's tokenization may split a word into multiple word pieces. We aggregate the dependencies between a word and multiple word pieces by taking the maximum value. We compute the PMI statistics and train the K&M model (Klein and Manning, 2004) on sentences of length  $\leq 10$  in the WSJ train split (section 2-21). For DMV, we train with the annotated POS tags using a public implementation released by (He et al., 2018). Results are averaged over three runs when applicable.
B.1 Additional Results in Section 5.2
We conduct an additional experiment on the English Penn Treebank to verify that conditional MI can extract parses for sentences longer than ten words. To expedite experimentation, we subsample 200 out of 2416 sentences from the test split of English Penn Treebank and the average sentence length of our subsampled dataset is 24.1 words. When applicable, we average over three random seeds and report standard deviations. Table 4 presents the UUAS of conditional MI and other methods. We draw similar conclusions as in Section 5.2, observing that the parses drawn by conditional MI have higher quality than those of other baselines.
Table 5 presents a comprehensive list of relations on which Conditional MI disagrees with LINEARCHINA under a log odds ratio test with  $p = 0.05$ .
Method	UUAS
RANDOM	9.14 ± 0.42
LINEARCHAIN	47.69
Klein and Manning (2004)	48.76 ± 0.24
PMI	28.05
CONDITIONAL PMI	44.75 ± 0.09
CONDITIONAL MI	50.62 ± 0.38
Table 4: Unlabeled Undirected Attachment Score on subsampled WSJ test split (section 23). Error bars show standard deviation across three random seeds.
Relation	Conditional MI	Linear Chain
xcomp	48.18	9.93
conj	43.36	7.58
nsubjpass	33.81	0.47
dobj	58.96	30.33
mark	30.71	9.45
poss	58.63	40.96
ccomp	20.92	4.18
vmod	55.32	41.84
tmod	39.25	27.68
dep	50.15	40.03
pobj	48.68	40.79
nsub	55.87	48.69
number	50.55	92.62
possessive	72.00	97.78
pcomp	60.00	77.00
quantmod	56.82	72.73
appos	55.56	70.59
num	65.11	76.49
cc	31.39	41.10
prep	56.41	66.12
auxpass	75.00	83.26
nn	72.97	77.88
aux	55.49	59.66
Table 5: All relations on which Conditional MI disagree with LINEARCHINA under a log odds ratio test with  $p = 0.05$ .
Proof of Proposition 2 We first recall our statement.
$$
\frac {\sqrt {2} \left\| \Sigma_ {\mathbf {X X}} \right\| _ {o p}}{\lambda_ {k} - \lambda_ {\mathbf {X X} , k + 1}} (\left\| A \mathbf {Z} \right\| _ {2} + \sqrt {\operatorname {t r} (\Sigma_ {\mathbf {X X}})}) + \left\| A A ^ {\top} \right\| _ {o p} \sqrt {\operatorname {t r} (\Sigma_ {\mathbf {X X}})},
$$
We will use the Davis-Kahan Theorem for our proof.
Theorem (Davis-Kahan (Stewart and Sun, 1990)). Let  $\sigma$  be the eigengap between the  $k$ th and the  $k + 1$ th eigenvalue of two positive semidefinite symmetric matrices  $\Sigma$  and  $\Sigma'$ . Also, let  $V$  and  $V'$  be the first  $k$  eigenvectors of  $\Sigma$  and  $\Sigma'$  respectively. Then we have,
$$
\frac {1}{\sqrt {2}} \left\| V V ^ {\top} - V ^ {\prime} V ^ {\prime \top} \right\| _ {o p} \leq \frac {\| \Sigma - \Sigma \| _ {o p}}{\sigma}.
$$
That is, we can bound the error in the subspace projection in terms of the matrix perturbation.
In our setting, we choose  $\Sigma = A\Sigma_{\mathbf{Z}\mathbf{Z}}A^{\top} + \Sigma_{\mathbf{X}\mathbf{X}}$  and  $\Sigma' = A\Sigma_{\mathbf{Z}\mathbf{Z}}A^{\top}$ . We know the eigengap of  $\Sigma'$  is  $\lambda_k$  because  $\Sigma'$  only has  $k$  nonzero eigenvalues. By Weyl's inequality, the  $k$ th eigenvalue is at most perturbed by  $\lambda_{\mathbf{X}\mathbf{X},k+1}$ , which is the  $k+1$  eigenvalue of  $\Sigma_{\mathbf{X}\mathbf{X}}$ . Let  $V$  be the top  $k$  eigenvectors of  $\Sigma'$  and assuming  $\lambda_k > \lambda_{\mathbf{X}\mathbf{X},k+1}$ , we have,
$$
\begin{array}{l} \frac {1}{\sqrt {2}} \left\| A A ^ {\top} - V V ^ {\top} \right\| _ {o p} \leq \frac {\| \Sigma - \Sigma^ {\prime} \| _ {o p}}{\lambda_ {k} - \lambda_ {\mathbf {X X} , k + 1}} \\ = \frac {\left\| \Sigma_ {\mathbf {X X}} \right\| _ {o p}}{\lambda_ {k} - \lambda_ {\mathbf {X X} , k + 1}}. \\ \end{array}
$$
Turning this operator norm bound into approximation bound, we have
$$
\begin{array}{l} \left. \mathbb {E} _ {\mathbf {X}} \left\| A \mathbf {Z} - \mathbf {X} _ {\mathrm {P C A}} \right\| _ {2} = \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} A \mathbf {Z} - V V ^ {\top} \mathbf {X} \right\| _ {2} \right. \\ = \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} A \mathbf {Z} - V V ^ {\top} A \mathbf {Z} + V V ^ {\top} A \mathbf {Z} - V V ^ {\top} \mathbf {X} \right\| _ {2} \\ \leq \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} A \mathbf {Z} - V V ^ {\top} A \mathbf {Z} \right\| _ {2} + \left\| V V ^ {\top} A \mathbf {Z} - V V ^ {\top} \mathbf {X} \right\| _ {2} \\ \leq \mathbb {E} _ {\mathbf {X}} \left. \left\| A A ^ {\top} A \mathbf {Z} - V V ^ {\top} A \mathbf {Z} \right\| _ {2} + \left\| V V ^ {\top} (A \mathbf {Z} - \mathbf {X}) \right\| _ {2} \right. \\ \leq \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} - V V ^ {\top} \right\| _ {o p} \cdot \| A \mathbf {Z} \| _ {2} + \left\| V V ^ {\top} \right\| _ {o p} \| A \mathbf {Z} - \mathbf {X} \| _ {2}. \\ = \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} - V V ^ {\top} \right\| _ {o p} \cdot \| A \mathbf {Z} \| _ {2} + \left\| A A ^ {\top} + V V ^ {\top} - A A \right\| _ {o p} \| A \mathbf {Z} - \mathbf {X} \| _ {2} \\ \leq \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} - V V ^ {\top} \right\| _ {o p} \cdot \| A \mathbf {Z} \| _ {2} + \left(\left\| A A ^ {\top} \right\| _ {o p} + \left\| V V ^ {\top} - A A \right\| _ {o p}\right) \| A \mathbf {Z} - \mathbf {X} \| _ {2} \\ = \mathbb {E} _ {\mathbf {X}} \left\| A A ^ {\top} - V V ^ {\top} \right\| _ {o p} \cdot (\| A \mathbf {Z} \| _ {2} + \| A \mathbf {Z} - \mathbf {X} \| _ {2}) + \left\| A A ^ {\top} \right\| _ {o p} \| A \mathbf {Z} - \mathbf {X} \| _ {2}. \\ \end{array}
$$
We use the fact that  $\mathbb{E}_{\mathbf{X},\mathbf{Z}}\| A\mathbf{Z} - \mathbf{X}\| _2^2 = \mathrm{tr}(\Sigma_{\mathbf{XX}})$  and Jensen's inequality to bound,
$$
\mathbb {E} _ {\mathbf {X}} \left\| A \mathbf {Z} - \mathbf {X} \right\| _ {2} \leq \sqrt {\operatorname {t r} \left(\Sigma_ {\mathbf {X X}}\right)}.
$$
Combining these inequalities, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {X}} \left\| A \mathbf {Z} - \mathbf {X} _ {\mathrm {P C A}} \right\| _ {2} \\ \leq \frac {\sqrt {2} \| \Sigma_ {\mathbf {X X}} \| _ {o p}}{\lambda_ {k} - \lambda_ {\mathbf {X X} , k + 1}} \cdot (\| A \mathbf {Z} \| _ {2} + \sqrt {\operatorname {t r} (\Sigma_ {\mathbf {X X}})}) + \left\| A A ^ {\top} \right\| _ {o p} \sqrt {\operatorname {t r} (\Sigma_ {\mathbf {X X}})} \\ \end{array}
$$
Proof of Proposition 1 We first recall our statement.
$$
\mathbf {x} _ {m a s k, i} = \beta_ {2 S L S, i} \mathbf {X} _ {\backslash i} + O (\left\| \Sigma_ {\mathbf {X X}, \backslash i, i} \right\| _ {2}),
$$
Proof Let  $A \backslash i \in \mathbb{R}^{d - 1 \times k}$  be the matrix where we omit the  $i$ th row of  $A$  and  $A_i \in \mathbb{R}^k$  be the  $i$ th row of  $A$ . Let  $\Sigma_{\mathbf{X}\mathbf{X}, \backslash i, \backslash i} \in \mathbb{R}^{d - 1 \times d - 1}$  be the matrix where we omit the  $i$ th row and  $i$ th column of  $\Sigma_{\mathbf{X}\mathbf{X}}$ , and  $\Sigma_{\mathbf{X}\mathbf{X}, \backslash i, i} \in \mathbb{R}^{d - 1}$  be the vector formed by dropping the  $i$ th row and taking the  $i$ th column of  $\Sigma_{\mathbf{X}\mathbf{X}}$ . Similarly, denote  $\mathbf{X} \backslash i \in \mathbb{R}^{d - 1}$  be the vector where we omit the  $i$  coordinate of  $X$ .
We start by writing down the expression of  $\beta_{2\mathrm{SLS},i}$ . Recall that the Least Squares regression between two zero-mean Gaussian variables  $\mathbf{X}$  and  $\mathbf{Y}$  can be written as
$$
\beta = C o v (\mathbf {X}, \mathbf {Y}) C o v (\mathbf {X}, \mathbf {X}) ^ {- 1},
$$
where  $Cov(\mathbf{X}, \mathbf{X})$  is the covariance matrix of  $X$  and we assume it is full rank. Since  $Cov(\mathbf{X}_{\backslash i}, \mathbf{Z})$  is  $A_{\backslash i} \Sigma_{\mathbf{Z}\mathbf{Z}}$ , we can write the coefficient of regression from  $\mathbf{X}_{\backslash i}$  to  $\mathbf{Z}$  as
$$
\beta_ {\mathbf {X} \backslash i \rightarrow \mathbf {Z}} = \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} (A _ {\backslash i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, \backslash i}) ^ {- 1}
$$
and by assumption we have  $\beta_{\mathbf{Z}\to \mathbf{x}_i} = A_i$  . So we can write down
$$
\beta_ {2 \mathrm {S L S}, i} = A _ {i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} (A _ {\backslash i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, \backslash i}) ^ {- 1}.
$$
Now we consider masked regression for the  $i$ th coordinate,  $\mathbf{x}_i$ ,
$$
\beta_ {\mathbf {X} \backslash i \rightarrow \mathbf {x} _ {i}} = (A _ {i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, i}) (A _ {\backslash i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, i}) ^ {- 1}.
$$
Comparing  $\beta_{2\mathrm{SLS}}$  and  $\beta_{\mathbf{X}\backslash i\to \mathbf{x}_i}$ , we observe that the second term is the same and the key is to bound the first term. Consider the error term between the coefficients,
$$
\begin{array}{l} \left\| \Sigma_ {\mathbf {X X}, \backslash i, i} (A _ {\backslash i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, \backslash i}) ^ {- 1} \right\| _ {2} \\ \leq \left\| \Sigma_ {\mathbf {X X}, \backslash i, i} \right\| _ {2} \left\| (A _ {\backslash i} \Sigma_ {\mathbf {Z Z}} A _ {\backslash i} ^ {\top} + \Sigma_ {\mathbf {X X}, \backslash i, \backslash i}) ^ {- 1} \right\| _ {o p} \\ \leq \left\| \Sigma_ {\mathbf {X X}, \backslash i, i} \right\| _ {2} \left\| \left(A \Sigma_ {\mathbf {Z Z}} A ^ {\top} + \Sigma_ {\mathbf {X X}}\right) ^ {- 1} \right\| _ {o p}. \\ \end{array}
$$
That is, the error term scales with the off-diagonal terms  $\left\| \Sigma_{\mathbf{X}\mathbf{X},\backslash i,i}\right\| _2$
Converting our bound on the error term into an approximation bound, we have
$$
\mathbf {x} _ {\mathrm {m a s k}, i} = \beta_ {\mathrm {2 S L S}, i} \mathbf {X} + O (\left\| \Sigma_ {\mathbf {X X}, \backslash i, i} \right\| _ {2}).
$$
Proof for Proposition 3.
Proposition 3. The gap between conditional MI with and without latent variables is bounded by the conditional entropy  $H(\mathbf{Z}|X_{\{\hat{i},j\}})$ ,
Proof The proof follows from the definition of conditional mutual information. Denote  $H(\cdot)$  as the entropy function.
We start by observing that
$$
I (x _ {i}; x _ {j} | \mathbf {Z}, X _ {\backslash \{i, j \}}) = I (x _ {i}; x _ {j} | X _ {\backslash \{i, j \}}) - I (x _ {i}; \mathbf {Z} | X _ {\backslash \{i, j \}}) + I (x _ {i}; \mathbf {Z} | x _ {j}, X _ {\backslash \{i, j \}})
$$
(Through chain rule of mutual information.)
$$
\begin{array}{l} = I \left(x _ {i}; x _ {j} \mid X _ {\backslash \{i, j \}}\right) + H \left(\mathbf {Z} \mid x _ {i}, X _ {\backslash \{i, j \}}\right) - H \left(\mathbf {Z} \mid X _ {\backslash \{i, j \}}\right) \\ + H (\mathbf {Z} | x _ {j}, X _ {\backslash \{i, j \}}) - H (\mathbf {Z} | x _ {i}, x _ {j}, X _ {\backslash \{i, j \}}). \\ \end{array}
$$
$$
\begin{array}{l} I \left(x _ {i}; x _ {j} \mid X _ {\backslash \{i, j \}}\right) - I \left(x _ {i}; x _ {j} \mid \mathbf {Z}, X _ {\backslash \{i, j \}}\right) \\ = - H (\mathbf {Z} | x _ {i}, X _ {\backslash \{i, j \}}) + H (\mathbf {Z} | X _ {\backslash \{i, j \}}) - H (\mathbf {Z} | x _ {j}, X _ {\backslash \{i, j \}}) + H (\mathbf {Z} | x _ {i}, x _ {j}, X _ {\backslash \{i, j \}}) \\ \leq H (\mathbf {Z} | X _ {\backslash \{i, j \}}) + H (\mathbf {Z} | x _ {i}, x _ {j}, X _ {\backslash \{i, j \}}) \\ \leq 2 \cdot H (\mathbf {Z} | X _ {\backslash \{i, j \}}). \\ \end{array}
$$
$$
\hat {I} _ {p _ {\theta}} = \underset {x _ {i}, x _ {j}} {\mathbb {E}} \left[ \log p _ {\theta} \left(x _ {i} \mid X _ {\backslash i} (j, x _ {j})\right) - \log \underset {x _ {j} \mid x _ {i}} {\mathbb {E}} p _ {\theta} \left(x _ {i} \mid X _ {\backslash i} (j, x _ {j})\right) \right]
$$
$$
\left. \left| \hat {I} _ {p _ {\theta}} - I _ {p} \right| \leq \underset {x _ {j}} {\mathbb {E}} D _ {\mathrm {k l}} \left(p \left(x _ {i} \mid X _ {\backslash i} (j, x _ {j})\right) \| p _ {\theta} \left(x _ {i} \mid X _ {\backslash i} (j, x _ {j})\right)\right), \right.
$$
Proof Expanding the definition of mutual information, we write
$$
\begin{array}{l} I \left(x _ {i}; x _ {j} \mid X _ {\backslash \{i, j \}}\right) - \hat {I} _ {\theta} \left(x _ {i}; x _ {j} \mid X _ {\backslash \{i, j \}}\right) = \mathbb {E} _ {x _ {j}} \left[ D _ {\mathrm {k l}} \left(p \left(x _ {i} \mid x _ {j}, X _ {\backslash \{i, j \}}\right) \| p _ {\theta} \left(x _ {i} \mid x _ {j}, X _ {\backslash \{i, j \}}\right)\right) \right] - \\ D _ {\mathrm {k l}} \left(\mathbb {E} _ {x _ {j}} p (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}}) \| \mathbb {E} _ {x _ {j}} p _ {\theta} (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}})\right). \\ \end{array}
$$
Dropping the the second term, we have
$$
\hat {I} _ {\theta} (x _ {i}; x _ {j} | X _ {\backslash \{i, j \}}) - I (x _ {i}; x _ {j} | X _ {\backslash \{i, j \}}) \geq - \mathbb {E} _ {x _ {j}} [ D _ {\mathrm {k l}} \left(p (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}}) \| p _ {\theta} (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}})\right) ].
$$
Dropping the the first term, we have
$$
\begin{array}{l} I (x _ {i}; x _ {j} | X _ {\backslash \{i, j \}}) - \hat {I} _ {\theta} (x _ {i}; x _ {j} | X _ {\backslash \{i, j \}}) \\ \leq D _ {\mathrm {k l}} \left(\mathbb {E} _ {x _ {j}} p (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}}) \| \mathbb {E} _ {x _ {j}} p _ {\theta} (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}})\right) \\ \leq \mathbb {E} _ {x _ {j}} D _ {\mathrm {k l}} \left(p (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}}) \| p _ {\theta} (x _ {i} | x _ {j}, X _ {\backslash \{i, j \}})\right), \\ \end{array}
$$
which uses the convexity of KL-divergence and Jensen's inequality.