SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. From Assumption 1 it follows that an output embedding, in the deepest attention layer, for each symbol is trained to predict the input symbol. It follows that the output embeddings of the deepest layer are one-to-one (injective) representations of the input symbols. Thus, the causal relations among the input symbols can be cast on their corresponding output embeddings. That is, a causal graph over the output embeddings is similar to causal graph over the input symbols  $\mathcal{G}_{\mathbf{Z}} = \mathcal{G}_s$ . In Section 4.1 we showed that self-attention in the Transformer architecture can be viewed as an SCM over the output embedding, and the covariance matrix,  $\mathbf{C_Z}$ , can be computed from the attention matrix (Equation 9). A conditional independence test between output embeddings can then be defined to calculate partial correlations from  $\mathbf{C_Z}$ . Finally, a sound constraint-based algorithm (35) that uses  $\mathbf{C_Z}$  for partial correlation CI-testing, recovers an equivalence class of  $\mathcal{G}_{\mathbf{Z}}$ .
(a)
Figure 8: An example for (a) a causal graph (PAG), and (b) the nodes that influence the prediction  $\tilde{I}_5$  having a PI-tree structure, depicted on coordinates where the radius axis is the distance from the prediction. In Algorithm 1-line 12, the algorithm iterates over radius values, starting with 1. In the first iteration, the sets  $\{I_2\}$  and  $\{I_3\}$  are tested if any of them qualifies as an explanation. If both sets fail, the search radius increases to 2 and the sets  $\{I_2, I_3\}$  and  $\{I_1, I_3\}$  are tested (set  $\{I_2, I_3\}$  is tested first as it members have smaller average distance from  $\tilde{I}_5$ ). If both fail,  $r = 3$  and the last tested set is  $\{I_1, I_2, I_3\}$ .
Figure 7: An example for a PI-set on  $\tilde{I}_5$ . Latent nodes are represented by squares. Nodes included in the PI-set are circled with a thick red line. Indirect dependency is marked with a dashed red line. (a) The smallest PI-set for a fully observed DAG is the Markov blanket  $\{I_2, H_2\}$ . However,  $H_2$  is latent. (b) Nodes  $I_2, I_3$  are included in the PI-set as they have direct effect on  $\tilde{I}_5$ , where  $I_5$  depends on  $I_3$  due to the latent confounder  $H_2$ . However,  $I_3$  is a collider, hence including it in the PI-set results in explaining-away and the PI-set is not complete. (c) A complete PI-set  $\{I_1, I_2, I_3\}$ . Including  $I_3$  creates an indirect dependence between  $I_5$  and  $I_1$  (a PI-path), which requires including  $I_1$  as well. Note that the distance from  $I_5$  to  $I_1$  is two edges in the graph over observed variables.
(b)
Corollary 2. Nodes  $X_{i}$  and  $X_{j}$  are  $d$ -separated in  $\mathcal{G}_{Z}$  conditioned on  $X' \subset X$  if and only if symbols  $s_{i}$  and  $s_{j}$  are  $d$ -separated in  $\mathcal{G}_{s}$  conditioned on  $s'$ , a subset containing symbols corresponding to  $X'$ .
Proof. This result follows directly from Theorem 1 and faithfulness assumption (Definition 2).