SlowGuess/NAACL_2021_OCR huggingface dataset:
Xiao Liu $^{1*}$ , Da Yin $^{2*}$ , Yansong Feng $^{1,3\dagger}$ , Yuting Wu $^{1}$  and Dongyan Zhao $^{1,3}$
$^{2}$ Computer Science Department, University of California, Los Angeles
<sup>3</sup>The MOE Key Laboratory of Computational Linguistics, Peking University, China
{lxlisa,fengyansong,wyting,zhaody}@pku.edu.cn
da.yin@cs.ucla.edu
Causal inference is the process of capturing cause-effect relationship among variables. Most existing works focus on dealing with structured data, while mining causal relationship among factors from unstructured data, like text, has been less examined, but is of great importance, especially in the legal domain. In this paper, we propose a novel Graph-based Causal Inference (GCI) framework, which builds causal graphs from fact descriptions without much human involvement and enables causal inference to facilitate legal practitioners to make proper decisions. We evaluate the framework on a challenging similar charge disambiguation task. Experimental results show that GCI can capture the nuance from fact descriptions among multiple confusing charges and provide explainable discrimination, especially in few-shot settings. We also observe that the causal knowledge contained in GCI can be effectively injected into powerful neural networks for better performance and interpretability. Code and data are available at https://github.com/xxxiaol/GCI/.
Causal inference is the process of exploring how changes on variable  $T$  affect another variable  $Y$ . Here we call  $T$  and  $Y$  as treatment and outcome, respectively, and the changes on  $T$  are called intervention. In other words, the process of drawing a conclusion about whether and how  $Y$  changes when intervening on  $T$  is called causal inference.
Most research in causal inference is devoted to analyzing structured data. Take the research question how smoking causes lung cancer (Pearl and Mackenzie, 2018) as an example. Smoking, lung cancer, together with distractors like age are extracted from structured data, like electronic health records, and considered as factors. Usually, such
Figure 1: An example of generated causal graph for the charge fraud. Colored words are matched between the exemplified fact description and the graph.
studies properly organize those factors into human-designed structures, e.g., a causal directed acyclic graph (Wright, 1921) with factors  $\{smoking, age, lung cancer\}$  as nodes and causal relations  $\{smoking \to lung cancer, age \to lung cancer\}$  as edges, and perform inference on such structures.
Recent works attempt to integrate text information into causal inference (Egami et al., 2018; Veitch et al., 2019; Yao et al., 2019; Keith et al., 2020), but they mainly treat the text as a single node in the causal graph, which is relatively coarse-grained. For instance, Yao et al. (2019) investigate how the complaints from consumers affect the company's responses (admission or denial). They regard the entire text of complaint as a treatment, without looking into different aspects of the text, like the events that consumers complained about and the compensation that consumers requested.
Actually, discovering causal relationship inside unstructured and high-dimensional data, like text, is also beneficial or even crucial for scenarios involving reading comprehensive text and making decisions accordingly. For instance, when a legal AI system assists judges to deal with complicated cases that involve multiple parties and complex events, causal inference could help to figure out the exact distinguishable elements that are crucial for fair and impartial judgements. As shown in Figure 1, if the system can automatically spot two
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1928-1941 June 6-11, 2021. ©2021 Association for Computational Linguistics
essential key points, 1) the deceitful acts of the defendant, and 2) obtaining properties from the victim, from the unstructured fact descriptions, then the prediction fraud can be more convincing and helpful, rather than a label from a black box. In practice, we would expect a legal AI system to provide human-readable and sound explanations to help the court make the right decisions. It is worthwhile especially for underdeveloped areas, where such techniques could help the judges of rural areas with more trustworthy references from previous judgements. This would further help to maintain the principle of treating like cases alike in many continental law systems. A main challenge in judicial practice is to distinguish between similar charges, we thus propose a new task similar charge disambiguation as a testbed. Cases of similar charges often share similar context, and a system is expected to mine the nuance of reasoning process from the text.
However, performing causal inference on fact descriptions of criminal cases is not trivial at all. It poses the following challenges. 1) Without expert involvement, it is not easy to extract factors that are key to prediction, and organize them in a reasonable form that can both facilitate the inference process and tolerate noise. For example, automatically extracted elements may not cover all the key points in the textual descriptions, and automatically built graphs may contain unreliable edges. 2) It is not easy to benefit from both traditional causal inference models and modern neural architectures.
In this paper, we propose a novel Graph-based Causal Inference (GCI) framework, which could effectively apply causal inference to legal text analysis. GCI first recognizes key factors by extracting keywords from the fact descriptions and clustering similar ones into groups as individual nodes. Then we build causal graphs on these nodes with a causal discovery algorithm that can tolerate unobserved variables, reducing the impact of missing elements. We further estimate the causal strength of each edge, to weaken unreliable edges as much as possible, and apply the refined graph to help decision making. Experimental results show that our GCI framework can induce reasonable causal graphs and capture the nuance from plain text for legal applications, especially with few training data.
We also explore the potential of  $GCI$  by integrating the captured causal knowledge into neu
ral network (NN) models. We propose two approaches: 1) imposing the causal strength constraints to the NN's attention weights; 2) applying recurrent neural networks on the causal chains extracted from our causal graphs. Experiments indicate that our methods can successfully inject the extracted causal knowledge and thus enhance the NN models. We also show that integrating GCI helps to mitigate the underlying bias in data towards the final prediction.
Our main contributions are as follows: 1) We propose a novel graph-based causal inference (GCI) framework to apply causal inference to unstructured text, without much human involvement. 2) We explore to equip popular neural network models with our GCI, by encouraging neural models to learn from causal knowledge derived from GCI. 3) We evaluate our methods on a legal text analysis task, similar charge disambiguation, and experimental results show that our GCI can capture the nuance from plain fact descriptions, and further help improve neural models with interpretability.
Two types of questions are typically related to causality. The first is whether there is causal relationship between a set of variables, and the second question is when two variables  $T$  and  $Y$  are causally related, how much would  $Y$  change if we change the value of  $T$ . Both of them are discussed in our GCI framework, and we first briefly introduce the key concepts.
2.1 Causal Discovery
Causal discovery corresponds to the first type of questions. From the view of graph, causal discovery requires models to infer causal graphs from observational data. In our GCI framework, we leverage Greedy Fast Causal Inference (GFCI) algorithm (Ogarrio et al., 2016) to implement causal discovery. GFCI combines score-based and constraint-based algorithms. It merges the best of both worlds, performing well like score-based methods, and not making too unrealistic assumptions like constraint-based ones. Specifically, GFCI does not rely on the assumption of no latent confounders, thus is suitable in our situation. More details of GFCI algorithm and its advantage is provided in Appendix A.
The output of GFCI is a graphical object called Partial Ancestral Graph (PAG). PAG is a mixed graph containing the features common to all Di
Edge	Meaning
A → B	A causes B.
A ↔ B	There is an unobserved confounder of A and B.
A o→ B	Either A causes B, or unobserved confounder.
A o-o B	Either A causes B, or B causes A, or unobserved confounder.
Table 1: Summary of edge types in PAG.
rected Acyclic Graphs (DAGs) that represent the same conditional independence relationship over the measured variables. In other words, PAG entails all the possibilities of valid DAGs concerning the original data. In causal inference settings (Zhang, 2008), four types of edges are provided in PAG, as listed in Table 1. With PAG, we are able to consider unobserved confounders and the uncertainty in causal inference.
2.2 Causal Strength Estimation
Causal strength estimation deals with the second type of questions. It is the very task to quantify the causal strength of each learned relation, i.e., whether the relation is strong or weak. To precisely estimate causal strength, confounders need to be kept the same. Confounder is a variable causally influencing both treatment  $T$  and outcome  $Y$ . Take the example of smoking, lung cancer and age in Section 1. Here we study if there is causal relationship between smoking  $(T)$  and lung cancer  $(Y)$ , and age is a confounder  $C$ . It is straightforward to compare the proportion of lung cancer among smokers and non-smokers. However, age influences both smoking and lung cancer. Older people are more likely to smoke. They also have a much higher risk of suffering from cancer. If we do not consider the value of age, its influence to lung cancer will be regarded as smoking's influence, thus wrongly amplify the causal effect of smoking to lung cancer.
In our  $GCI$  framework, we apply Average Treatment Effect (ATE) (Holland, 1986) as a measure of causal strength. All variables are binary in our work. So given an edge  $T \to Y$ , we quantify how the outcome  $Y$  is expected to change if we modify the treatment  $T$  from 0 to 1:
$$
\psi_ {T, Y} = E [ Y \mid d o (T = 1) ] - E [ Y \mid d o (T = 0) ], \tag {1}
$$
where  $E$  means expectation, and the do-calculus  $\text{do}(T = 1)$  indicates intervention on  $T$ , setting its value to 1.
We utilize the Propensity Score Matching (PSM) method to estimate ATE. PSM finds the
pairs of comparable samples with the most similar propensity scores, where each pair consists of one sample in the treated group and one in the untreated. Given the great similarity between the two samples, we could make a direct comparison between them. Specifically, propensity score  $L(z) = P(T = 1 \mid Z = z)$  is the probability of treatment being assigned to 1 given a set of observed confounders  $z$ . As  $T$  is binary, we have  $T \perp Z \mid L$  (  $\mathbb{L}$  means independence). So matching on propensity scores equals matching on the full set of confounders.
3 Graph-based Causal Inference Framework
Our graph-based causal inference (GCI) framework consists of three parts, constructing the causal graph, estimating causal strength on it, and making decisions. Figure 2 shows the overall architecture.
We first define the similar charge disambiguation task. Given the fact descriptions of criminal cases  $\mathbb{D} = \{d_1,d_2,\ldots ,d_N\}$ , a system is expected to classify each case into one charge from the similar charge set  $\mathbb{C} = \{c_{1},c_{2},\dots ,c_{M}\}$ .
3.2 Causal Graph Construction
Extracting Factors. To prepare nodes for the causal graph, we calculate the importance of word  $w_{j}$  for charge  $c_{i}$  using YAKE (Campos et al., 2020). We enhance YAKE with inverse document frequency (Jones, 1972) to extract more discriminative words of each charge.
To discriminate the similar charges, we select  $p$  words with the highest importance scores for each charge, cluster them into  $q$  classes to merge similar keywords. The  $q$  classes together with the  $M$  charges form the nodes of the causal graph. All these factors are binary. When the graph is applied to a case, each factor is of value 1 if it exists in this case, and 0 if not. Unlike factors extracted by experts, automatically extracted keywords may be incomplete, resulting in unobserved confounders in causal discovery.
Learning Causal Relationship. The next step is to build edges for the graph, in other words, discover the causal relationship between different factors. To learn causal relations and tackle the unobserved confounder problem, we use GFCI (Ogarrio et al., 2016), which does not rely on the assumption
Figure 2: Overall architecture of  $GCI$ . In the Extract Factors phase, solid circles indicate that these factors exist in this case, while hollow circles mean the opposite. For the nodes of causal graphs,  $A$ ,  $B$ ,  $C$ , and  $D$  denote the key points contributing to prediction, while  $Y_{1}$  and  $Y_{2}$  indicate the charges to discriminate. Four types of edges  $\rightarrow$ ,  $\leftrightarrow$ ,  $\circ \rightarrow$ , and  $\circ - \circ$  exist in the PAG in the Learn Causal Relationship phase, and they are converted to  $\rightarrow$  edges in the sampled graphs. In the rightmost phase, shaded factors are matched between fact descriptions of cases and the graph.
of no unobserved confounders. As mentioned in Section 2, the output of GFCI is a PAG. Appendix C gives an example of a generated PAG in detail.
We further introduce constraints to filter noisy edges. First, as the judgement is made based on the fact description, we do not allow edges from charge nodes to other ones, e.g., an edge from fraud to lie is prohibited. Second, given that causes usually appear before effects in time (Black, 1956), and fact descriptions in legal text are often written in the temporal order of events, we thus consider the chronological order of descriptions as temporal constraints to filter noisy edges. If factor  $A$  appears after  $B$  in most cases, we will not allow the edge from  $A$  to  $B$ . Note that this constraint does not imply there is an edge from  $B$  to  $A$ , as chronological order is not a sufficient condition of causality.
Sampling Causal Graphs. PAG contains uncertain relations shown in Table 1, which leaves challenges for quantification and further application. So we sample  $Q$  causal graphs from PAG. Among the four edge types,  $\rightarrow$  and  $\leftrightarrow$  are clear: in each sampled graph,  $\rightarrow$  edges are retained and  $\leftrightarrow$  edges are removed (because they do not indicate causal relations between the two nodes). For  $\circ \rightarrow$  edges, they have two possible choices: being kept (cause) and being removed (unobserved confounder). In the absence of true possibility, we simply keep an edge with  $1/2$  probability, and remove it with another  $1/2$ . And for  $\circ \circ$  edges, we give  $1/3$  probability for  $\rightarrow$ ,  $\leftarrow$ , and no edge, respectively. The quality of each sampled graph  $G_{q}$  is measured by its fitness with data  $\mathbf{X}$ , where we use the Bayesian information criterion  $\mathrm{BIC}(G_{q}, \mathbf{X})$  to estimate (Schwarz
et al., 1978).
3.3 Strength Estimation on Causal Graphs
As the resulting graphs are noisy in nature, we estimate the strength of the learned causal relations to refine a sampled causal graph. We assign high strength to edges with strong causal effect, and near-zero strength to edges that do not indicate causal relations or with weak effect. We regard the Average Treatment Effect  $\psi_{T,Y}^{G}$  (ATE, Section 2.2) as the strength of  $T\rightarrow Y$  in graph  $G$ , and utilize the Propensity Score Matching (PSM, Section 2.2) to measure it:
$$
\hat {\psi} _ {T, Y} ^ {G} = \left[ \sum_ {i: t _ {i} = 1} \left(y _ {i} - y _ {j}\right) + \sum_ {i: t _ {i} = 0} \left(y _ {j} - y _ {i}\right) \right] / N, \tag {2}
$$
where  $j = \underset {k:t_k\neq t_i}{\mathrm{argmin}}|L(z_i) - L(z_k)|$  means the most similar instance in the opposite group of  $i$  , and  $t_i,y_i,z_i$  are the value of treatment, outcome and confounders of instance  $i$  , respectively.
3.4 Making Decisions
When applying the sampled causal graphs to the similar charge disambiguation task, we simply extract factors and map the case description with the graph accordingly, and decide which charge in  $\mathbb{C}$  is more appropriate to this case. Firstly, we compute the overall causal strength of each factor  $T_{j}$  to  $Y_{i}$  among the  $Q$  sampled causal graphs, where  $Y_{i}$  represents whether charge  $c_{i}$  is committed:
$$
\tilde {\psi} _ {T _ {j}, Y _ {i}} = \sum_ {q = 1} ^ {Q} \operatorname {B I C} \left(G _ {q}, \mathbf {X}\right) \times \hat {\psi} _ {T _ {j}, Y _ {i}} ^ {G _ {q}}, \tag {3}
$$
where  $\hat{\psi}_{T_j,Y_i}^{G_q}$  is the measured causal strength in  $G_{q}$  and is 0 if edge  $T_{j}\rightarrow Y_{i}$  does not exist in  $G_{q}$ .
For each case, we then map the text with the graphs, and calculate scores for each charge:
$$
S \left(Y _ {i}\right) = \sum_ {T _ {j} \in T r \left(Y _ {i}\right)} \tilde {\psi} _ {T _ {j}, Y _ {i}} \times \tau \left(T _ {j}\right), i \in \{1, \dots , M \}, \tag {4}
$$
where  $\tau(T_j)$  is a dummy variable indicating the presence of  $T_j$  in this case, and  $Tr(Y_i)$  is the set of treatments of  $Y_i$  (from the view of graph, the nodes pointing to  $Y_i$ ). The calculated scores are fed into a random forest classifier (Ho, 1995) to learn thresholds between the charges. More advanced classifiers can also be used.
4 Integration of Causal Analysis and Neural Networks
Neural networks (NN) are considered to be good at exploring large volumes of textual data. This motivates us to integrate the causal framework with NN, to benefit each other. Here we propose two integration methods as shown in Figure 3.
4.1 Imposing Strength Constraint
First, we inject the estimated causal strength to constrain the attention weights of a Bi-LSTM with attention model (Zhou et al., 2016). A Bi-LSTM layer is first applied to the fact descriptions to obtain contextual embeddings  $\mathbf{H} = \{\mathbf{h}_1,\mathbf{h}_2,\dots ,\mathbf{h}_n\} ,\mathbf{h}_i\in \mathbb{R}^{b_0}$ , where  $b_{0}$  is the dimension of embeddings. Then, an attention layer assigns different weights  $\{a_{1},a_{2},\ldots ,a_{n}\}$  to each word, and sums the words up according to the weights to build a text embedding  $\mathbf{v}$ :
$$
a _ {i} = \frac {\exp \left(\mathbf {q} ^ {T} \cdot \mathbf {h} _ {i}\right)}{\sum_ {k = 1} ^ {n} \exp \left(\mathbf {q} ^ {T} \cdot \mathbf {h} _ {k}\right)}, \mathbf {v} = \sum_ {i = 1} ^ {n} a _ {i} \times \mathbf {h} _ {i}, \tag {5}
$$
where  $\mathbf{q} \in \mathbb{R}^{b_0}$  is a learnable query vector. Finally, we apply two fully connected layers to the text embedding  $\mathbf{v}$ , and form the prediction vector  $\mathbf{r}_{\text{cons}}$ .
Besides a cross-entropy loss  $L_{cross}$  on  $\mathbf{r}_{cons}$ , we introduce an auxiliary loss  $L_{cons}$  to guide the attention module with the causal strength learned from GCI. Given the golden label  $c_{j}$ , for each word  $w_{i}$  which belongs to the factor  $f$ ,  $\tilde{\psi}_{T_f,Y_j}$  is the corresponding causal strength, and  $g_{i}$  is the normalized strength over the whole sequence.  $L_{cons}$  is set to make the attention weights close to the normalized
Figure 3: Two ways of integrating causal analysis and neural networks.
$$
L _ {\text {c o n s}} = \sum_ {i = 1} ^ {n} \left(a _ {i} - g _ {i}\right) ^ {2}, \tag {6}
$$
$$
L = L _ {c r o s s} + \alpha L _ {c o n s}.
$$
Note that in the validation and testing stages, the inputs do not contain any strength constraint and golden charge information. Therefore, we select the epoch with the least cross-entropy loss in the validation stage to evaluate on the test set.
4.2 Leveraging Causal Chains
Causal chains are another type of knowledge that can be captured from causal graphs. In the legal scenario, causal chains depict the process of committing crimes. They can also be treated as the summarization of cases or behavioural patterns for each charge. Therefore, the second approach is to leverage the causal chains directly, as the chains may contain valuable information for judgement. For a given text, we extract factors and traverse all causal chains composed by the factors from the sampled causal graphs,  $Chains = \{chain_1, chain_2, \ldots, chain_m\}$ . In this task, we only consider chains ending up with treatments of charges, as they are more relevant with the judgement. An LSTM layer is applied to each chain, and all the chains are pooled to build case representation  $\mathbf{c} \in \mathbb{R}^{b_0}$ :
$$
\mathbf {c h} _ {i} = \sum_ {j = 1} ^ {l _ {i}} (\mathrm {L S T M} (c h a i n _ {i}) _ {j}), \tag {7}
$$
$$
\mathbf {c} = \operatorname {M a x P o o l i n g} \left(\operatorname {B I C} \left(G _ {q}, \mathbf {X}\right) \times \mathbf {c h} _ {i}\right),
$$
$$
1 \leq i \leq m, \operatorname {c h a i n} _ {i} \in G _ {q},
$$
where  $l_{i}$  indicates the length of chain.  $i$  .The case representation  $\mathbf{c}$  is then fed to two fully connected layers to make the prediction  $\mathbf{r}_{chain}$  , and a crossentropy loss is used to optimize the model.
Charge Sets	Charges	#Cases
Personal Injury	Intentional Injury & Murder & Involuntary Manslaughter	6377 / 2282 / 1989
Violent Acquisition	Robbery & Seizure & Kidnapping	5020 / 2113 / 622
F&E	Fraud & Extortion	3536 / 2149
E&MPF	Embezzlement & Misappropriation of Public Funds	2391 / 1998
AP&DD	Abuse of Power & Dereliction of Duty	1950 / 1938
Table 2: Summary of the similar charge sets.
Dataset. For the similar charge disambiguation task, we pick five similar charge sets from the Criminal Law of the People's Republic of China (Congress, 2017), which are hard to discriminate in practice (Ouyang et al., 1999), and select the corresponding fact descriptions from the Chinese AI and Law Challenge (CAIL2018) (Xiao et al., 2018). Detailed statistics of the charge sets are given in Table 2. Note we filter out the cases whose judgements include multiple charges from one charge set. The fact descriptions in our dataset are in Chinese.
Our Models. We evaluate our graph-based causal inference (GCI) framework as described in Section 3, and two models integrating GCI with NN (Bi-LSTM+Att+Cons and CausalChain) as described in Section 4.
Comparison Models. To study the effect of causal relationship captured by  $GCI$ , we implement a variant called  $GCI-co$ , which is built upon a correlation-based graph rather than our discovered causal graph. In detail, we compute the Pearson correlation coefficient  $\phi$  for every two factors, and draw an edge if  $\phi > 0.5$ . The direction of the edge is from the factor that appears earlier in the text more often, to the other. Then we compare  $GCI$  and two integration methods with NN baselines, including  $LSTM$ ,  $Bi-LSTM$  and  $Bi-LSTM+Att$ .  $Bi-LSTM+Att$  is a common backbone of legal judgement prediction models, while we do not add multitask learning (Luo et al., 2017) and expert knowledge (Xu et al., 2020) for simplicity. Since the prior knowledge learned from pre-trained models may result in unfair comparison, we do not choose the models such as BERT (Devlin et al., 2018) as baselines and backbones to eliminate the influence.
Previous works integrating text into causal inference are not able to find causal relationships inside text, so we do not take them into comparison.
We select a set of training ratios,  $1\%$ ,  $5\%$ ,  $10\%$ ,  $30\%$ , and  $50\%$ , to study how the performance gap changes along with different training data available. For each setting, we run the experiments on three random seeds and report the average accuracy (Acc) and macro-F1 (F1). More details about baselines, parameter selection, and training process are in Appendix B.