A Multi-level Neural Network for Implicit Causality Detection in Web Texts	Mining causality from text is a complex and crucial natural language understanding task. Most of the early attempts at its solution can group into two categories: 1) utilizing co-occurrence frequency and world knowledge for causality detection; 2) extracting cause-effect pairs by using connectives and syntax patterns directly. However, because causality has various linguistic expressions, the noisy data and ignoring implicit expressions problems induced by these methods cannot be avoided. In this paper, we present a neural causality detection model, namely Multi-level Causality Detection Network (MCDN), to address this problem. Specifically, we adopt multi-head self-attention to acquire semantic feature at word level and integrate a novel Relation Network to infer causality at segment level. To the best of our knowledge, in touch with the causality tasks, this is the first time that the Relation Network is applied. The experimental results on the AltLex dataset, demonstrate that: a) MCDN is highly effective for the ambiguous and implicit causality inference; b) comparing with the regular text classification task, causality detection requires stronger inference capability; c) the proposed approach achieved state-of-the-art performance.

Methods ::: Segment Level	We propose a novel approach to infer causality within sentences at the segment level. The model is named as Self Causal Relation Network (SCRN) due to it focuses on the causal relation intra-sentence compared with previous studies of RNs. Dealing with segments. The core idea of Relation Networks is operating on objects. In our task, the sentence is split into three segments $BL$, $L$, and $AL$ according to the position of AltLex. Then the input representations of these segments can be formulated as $X_{BL} \in \mathbb {R}^{T_{BL} \times d}$, $X_{L} \in \mathbb {R}^{T_{L} \times d}$ and $X_{AL} \in \mathbb {R}^{T_{AL} \times d}$ where $T_{BL}$, $T_{L}$, and $T_{AL}$ are the length of tokens in each segment.

Our intuition is if the sentence can be expressed as "B-AltLex-A", we could inference these segments in pairs to identify: 1) the semantic relation of "B-AltLex" and "AltLex-A"; 2) the cause-effect relation between "B-A" or "A-B". Then the segment-level representation is inferred by two non-linear layers. Finally, we combine word-level with segment-level representations to obtain the detection result. In general, our model MCDN has a simple architecture but effective reasoning potential for causality detection. The contributions can be summarized as three-fold: We introduce the task of mining causality from web text that is conducted into detection and extraction step. Utilizing detection instead of specific templates is a new direction and can provide a rich diversity of causality text with low-noise data for the subsequent extraction step and upstream applications. We propose a neural model MCDN to tackle the problem at the word and segment levels without any feature engineering. MCDN contains a relational reasoning module named Self Causal Relation Network (SCRN) to infer the causal relations within sentences. To evaluate the effectiveness of the proposed framework, we have conducted extensive experiments on a publicly available dataset.

In our research, the Transformer could pay attention to cause and effect entities, and capture long-distance dependency across connectives in the meantime. With this end-to-end module, we combine local context and long-distance dependency to acquire a semantic representation at the word level. Thus, we can relax the first limitation (i.e. feature engineering and accumulated errors). At the segment level, to inference the case and effect near the AltLex, we split the sentence into three segments on the ground of "segment before AltLex", "AltLex" and "segment after AltLex". To solve the second limitation, we propose a novel causality inference module, namely Self Causal Relation Network (SCRN). Due to the characteristics of the dataset, the input of SCRN is a single sentence. This is different from Relation Networks in other areas. The feature maps of segments are constructed into four pair-wise groups that are concatenated with a sentence representation respectively.

As seen in Table TABREF4, "consequently" is observed in either causal examples or non-causal examples. Luo et al.  leveraged causal pairs extracted by a set of pre-defined patterns to form CausalNet where the weight of a causality pair is a frequency of causality co-occurrence. Unfortunately, due to the volume of their corpus, there was no further analysis of sentences syntactic dependency. To some extent, this restricts the performance of causal pairs detection. To address the above problems, we propose a Multi-level Causality Detection Network (MCDN) for causality detection based on the following observations: 1) methods based on end-to-end deep neural networks could reduce the labor cost on feature engineering and relief errors propagation from the existing toolkits; 2) causality is a complicated relation, which calls for multi-level analysis, including parsing each word with its context firstly and inferring causality via the segments on both sides of the connectives secondly. Therefore, at the word level, we integrate word, position, and segment embeddings to encode the input sentence, followed by feeding it into stacked Transformer blocks, which have been widely used in various NLP tasks , .

Besides, F1-score of both BERT and MCDN is beyond 80 on the Bootstrapped dataset. All the results above suggest that the neural network is more powerful than the traditional co-occurrence and world knowledge-based methods on this task, as we expected. MCDN has learned various semantic representations of causal relations from word level and been able to inference causality from segment level supported by concise and effective SCRN. Furthermore, the deep classification methods we employed don't perform as well as MCDN which demonstrates causality detection is a much complex task that requires considerable relational reasoning capacity compared with text classification, although both can be generalized to classification problems.

Methods	In this section, we elaborate the MCDN, a multi-level neural network-based approach with Transformer blocks at the word level and SCRN at the segment level for causality detection, which is primarily targeted at ambiguous and implicit relations.

Consequently, we modify the SCRN architecture in a mathematical formulation and obtain the final output $sl\_{rep} \in \mathbb {R}^{4d_g}$ at the segment level: In general, the model transforms the segments into object-pairs by the TC-CNN and passes sentence through bi-GRU to obtain the global representation. Then we integrate object-pairs with global representation and make a pair-wise inference to detect the relationship among the segments. Ablation studies show that the proposed SCRN at the segment level has the capacity for relational reasoning and promotes the result significantly.

Our work is similar to previous works on detecting causalities , . The difference is we do not incorporate knowledge bases they used. We propose a neural-based multi-level model to tackle the problem without any feature engineering. Oh et al.  proposed a multi-column convolutional neural network with causality-attention (CA-MCNN) to enhance MCNNs with the causality-attention based question and answer passage, which is not in coincidence with our task. In compared with CA-MCNN, the multi-head self-attention within the Transformer block we used at the word level is more effective, and the SCRN at the segment level augments the reasoning ability of our model.

Methods ::: Causality Detection	Our model MCDN identifies causality of each sentence based on the output $wl\_{rep}$ at the word level and $sl\_{rep}$ at the segment level. The two outputs are concatenated as a unified representation $uni\_{rep} = [wl\_{rep}; sl\_{rep}] \in \mathbb {R}^{d + 4d_g}$. In this task, we use a 2-layer FFN consisting of $d_g$ units which have a ReLU activation and is followed by a softmax function to make the prediction: In the AltLexes dataset, the number of non-causal examples is over seven times the number of causal examples, and this leads to an extremely sample imbalance problem. If we adopt cross-entropy (CE) as model loss function, the performance would be unsatisfactory. Moreover, the difficulty in detecting each sample is different. For example, the sentence contains an ambiguous AltLex such as "make" is harder to infer than that contains "cause".

Preliminary Statement ::: Notations and Definitions	For a given Wikipedia sentence $S$, it is assumed that it has $n$ tokens. $S = \lbrace s_1, s_2, ... , s_{n-1}, s_n \rbrace $ where $s_i$ is a filtered token at position $i$. We use $L$ refers to the AltLex, $BL$ refers to the segment before the AltLex and $AL$ refers to the segment after the AltLex. Our objective is to generate a sentence-level prediction $\hat{y}$ of which the label is $y$ as Equation DISPLAY_FORM12. The proposed model MCDN is shown in Figure FIGREF15. We will detail each component in the rest of this section. It's worth noting that Hidey and McKeown  utilized English Wikipedia and Simple Wikipedia sentence pair to create a parallel corpus feature but still took one sentence as input each time. Unlike this approach, MCDN only leverages the input sentence for causal inference.

Implicit commonsense causality can be expressed by a simple sentence structure without any connectives: for example, "got wet" is the cause of "fever" in Example 1 has no connectives assisting detect causality, while there are explicit connectives (i.e. "since" and "result") in Example 2 to benefit complex causality detection. Example 1 I got wet during the day and came home with a fever at night. Example 2 Since computers merely execute the instructions they are given, bugs are nearly always the result of programmer error or an oversight made in the program's design. Normally, causality mining is divided into two sequential tasks: causality detection and cause-effect pair extraction. When dealing with large-scale web text, detecting causalities by specific classifiers with relational reasoning capacity is a pre-step of extracting cause-effect pairs. The performance of causality mining largely depends on how well the detection is performed. In this paper, we mainly focus on the detection step. This procedure can overcome the weakness of manual templates that hardly cover the linguistic variations of explicit causality expressions. It could help build a causality dataset with various expressions for extraction, which results in much less model complexity.

Related Work ::: Relation Networks	Relation Networks (RNs) is initially a simple plug-and-play module to solve Visual-QA problems that fundamentally hinge on relational reasoning . RNs can effectively couple with convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and multi-layer perceptrons (MLPs) to reduce overall network complexity. We gain a general ability to reason about the relations between entities and their properties. Original RNs can only perform single step inference such as $A \rightarrow B$ rather than $A \rightarrow B \rightarrow C$. For tasks that require complex multi-step of relational reasoning, Palm et al.  introduced the recurrent relational network that operates on a graph representation of objects. Pavez et al.  added complex reasoning ability to Memory Networks with RNs, which reduced its computational complexity from quadratic to linear. However, their tasks remain text QA and visual QA. In this paper, it's the first time that RNs is applied to relation extraction as proposed SCRN.

Experiment	In this section, we are interested in investigating the performance of MCDN that integrates Transformer blocks with SCRN and whether it is essential to incorporate inference ability in the sentence-level causality detection task.

Methods ::: Input Representation	Our input representation is able to incorporate multi-source information into one token sequence. Inspired by , the representation of each token in the input sentence is constructed by summing the corresponding word, position, and segment embeddings. Unlike the previous work, BERT, the segment embeddings here indicate the $BL$, $L$ and $AL$ segment in each sentence. As shown in Fig. FIGREF10, first, we adopt a word2vec toolkit to pretrain word embeddings with $d_{word}$ dimension on the English Wikipedia dump. Next, we utilize positional embeddings to map the positional information because our model has no recurrent architecture at the word level. Similarly, we use segment embeddings to involve more linguistic details. $d_{pos}$ and $d_{seg}$ is the dimension of positional embeddings and segment embeddings, respectively. By sum the three embeddings, finally, we get a new representation $x_i \in \mathbb {R}^d$ for token $s_i$ where $d = d_{word} = d_{pos} = d_{seg}$. The representation $x_i$ could provide basic features for high-level modules.

Related Work ::: Causality Relation	Causality mining is a fundamental task with abundant upstream applications. Early works utilize Bayesian network , , syntactic constraint , and dependency structure  to extract cause-effect pairs. Nevertheless, they could hardly summarize moderate patterns and rules to avoid overfitting. Further studies incorporate world knowledge that provides a supplement to lexico-syntax analysis. Generalizing nouns to their hypernyms in WordNet and each verb to its class in VerbNet ,  eliminates the negative effect of lexical variations and discover frequent patterns of cause-effect pairs. As is well known, the implicit expressions of causality are more frequent. J.-H. Oh et al.  exploited cue words and sequence labeling by CRFs and selected the most relevant causality expressions as complements to implicitly expressed causality. However, the method requires retrieval and ranking from enormous web texts. From natural properties perspective, causality describes relations between regularly correlated events or phenomena. Constructing a cause-effect network or graph could help discover co-occurrence patterns and evolution rules of causation , . Therefore, Zhao et al.  conducted causality reasoning on the heterogeneous network to extract implicit relations cross sentences and find new causal relations.

Preliminary Statement ::: Linguistic Background	This section describes the linguistic background of causal relation and the AltLexes dataset, which we used. It's a commonly held belief that causality can be expressed explicitly and implicitly using various propositions. In the Penn Discourse Treebank (PDTB) , over $12\%$ of explicit discourse connectives are marked as causal such as "hence", "as a result" and "consequently", as are nearly $26\%$ of implicit discourse relationships. In addition to these, there exists a type of implicit connectives in PDTB named AltLex (Alternative lexicalization) has been capable of indicating causal relations, which is an open class of markers and potentially infinite. The definition of AltLex was extended with an open class of markers that occur within a sentence in . The following are examples widespread in the new AltLexes set but are not contained in PDTB explicit connectives. The word "made" with many meanings here is used to express causality. Moreover, the expression of causality in the second example is somewhat obscure. Ambiguous causal verbs, e.g. The flood made many houses to collapse.

The input representation $X$ of the sentence pass through a bidirectional-GRU (bi-GRU) with $d_g$-dimension hidden units, and the final state $\gamma \in \mathbb {R}^{2d_g}$ of the bi-GRU is concatenated to each object-pair. SCRN. We construct four object-pairs concatenated with $\gamma $. Let $\#$ denote the pair-wise operation. For causality candidates, $BL\#L$ and $L\#AL$ indicate the relation between cause-effect and AltLex, while $BL\#AL$ and $AL\#BL$ inference the direction of causality. The object-pairs matrix $OP \in \mathbb {R}^{4 \times (2k + 2d_g)}$ is shown as follows: Here ";" is a concatenation operation for the object vectors.

Most existing works on causality detection have two common limitations. First, utilizing linguistic methods, such as part-of-speech (POS) tagging and syntax dependency parsing, to get handcrafted features is labor-intensive and takes ample time. Zhao et al.  divided causal connectives into different classes as a new category feature based on the similarity of the syntactic dependency structure within causality sentences. Also, the proposed model copes with the interaction between the category feature and other frequently-used features such as contextual features, syntactic features, and position features. However, these extracted features hardly capture a wide variety of causality expressions. The algorithms that used the NLP toolkits to extract the features can pass on the errors caused by the toolkits. Hidey and McKeown  incorporated world knowledge, such as FrameNet, WordNet, and VerbNet, to measure the correlations between words and segments while the method barely handles those words which never appear in the learning phase. Second, the quality of extracting co-occurrence by pre-defined patterns is influenced by ambiguous connectives, such as "subsequently" and "force."

The detailed information about these baselines is listed as follows: TextCNN used here has a convolution layer, the window sizes of which are 2, 3, 4 and each have 50 kernels. Then we apply max-overtime-pooling and 2-layer FFN with ReLU activation. The dropout rate is 0.5 and $L-2$ regularization coefficient is $3e^{-4}$. TextRNN uses a bidirectional GRU the same as sentence encoder in SCRN and use max pooling across all GRU hidden states to get the sentence embedding vector, then use a 2-layer FFN to output the result. Dropout rate and $L_2$ regularization coefficient is the same as TextCNN. SASE  uses a 2-D matrix to represent the sentence embedding with a self-attention mechanism and a particular regularization term for the model. It's an effective sentence level embedding method. DPCNN  is a low-complexity word-level deep CNN model for sentiment classification and topic categorization. It can make down-sampling without increasing the number of features maps which enables the efficient representation of long-range associations.

Due to the difference of segment lengths, we use a three-column CNN (TC-CNN) to parse $X_{BL}$, $X_{L}$, and $X_{AL}$ into a set of objects. Particularly the representations here only employ word embeddings and segment embeddings because the TC-CNN could capture the position information. Unlike [25], TC-CNN convolves them through a 1D convolutional layer to $k$ feature maps of size $T_{BL} \times 1$, $T_{L} \times 1$, and $T_{AL} \times 1$, where $k$ is the sum of kernels. The model exploits multi-scale kernels (with varying window size) to obtain multi-scale features. As seen in Fig. FIGREF15, the feature maps of each segment are rescaled into a k-dimension vector by the max pooling layer after convolution. Finally, we produce a set of objects for SCRN: Dealing with the sentence.

Introduction	Automatic text causality mining is a critical but difficult task because causality is thought to play an essential role in human cognition when making decisions . Thus, automatic text causality has been studied extensively in a wide range of areas, such as industry , physics  and healthcare , etc. A tool to automatically scour the plethora of textual content on the web and extract meaningful causal relations could help us construct causal chains to unveil previously unknown relationships between events  and accelerates the discovery of the intrinsic logic of the events . Many research efforts have been made to mine causality from text corpus with complex sentence structures in the books or newspapers. In Causal-TimeBank  authors introduced "CLINK" and "C-SIGNAL" tag to mark events causal relation and causal signals respectively based on specific templates (e.g., "A happened because of B"). Q. Do et al.  collected 25 newswire articles from CNN in 2010 and released event causality dataset that provides relatively dense causal annotations. Recently, Q. Do et al. improved the annotation method and implemented joint reasoning for causal and temporal relations .

In self-attention module, we set the stack time of Transformer blocks $N=4$ and the number of attention heads $h=4$. In SCRN, the window sizes of TC-CNN kernels are 2, 3, 4 while the sum of kernel $k=150$. We use a 2-layer bi-GRU with 64 units in each direction. As for the focal loss, we set $\alpha =0.75, \beta =4$. Evaluation Metrics. Different evaluation metrics including accuracy, precision, recall, and F1-score are adapted to compare MCDN with the baseline methods. To understand our model comprehensively, we employ both Area under Receiver Operator Curve (AUROC) and Area under Precision-Recall Curve (AUPRC) to evaluate its sensitivity and specificity, especially under the situation that causality is relatively sparse in the web text.

BERT  presented state-of-the-art results in a wide variety of NLP tasks, which is a pre-trained deep language representation model based on Transformer and Masked Language Model. BERT is inspired by transfer learning in the computer vision field, pre-training a neural network model on a known task, for instance, ImageNet, and then performing fine-tuning on a new purpose-specific task. It's worth noting that due to data imbalance and for comparison in the same situation, we also used focal loss in the above methods to acquire the best performance.

We apply feed-forward networks after the self-attention sub-layer. It consists of two linear layers and a ReLU activation between them. Note that $x$ is the output of the previous layer: where $W_1 \in \mathbb {R}^{d \times d_f}$ and $W_2 \in \mathbb {R}^{d \times d_f}$. We set $d_f = 4d$ in our experiments. The Transformer block is stacked $N$ times, of which the final output $wl\_{rep}$ is regarded as the representation of the sentence at the word level. We aim to deal the word with its fine-grained local context and coarse-grained global long-distance dependency information. Thus, our word-level module could acquire not only lexico-syntax knowledge that manual patterns hardly cover but also lexical semantics among the words.

Experiment ::: Experiment Settings	Dataset. We use the AltLexes dataset to evaluate the proposed approach. The detailed statistical information about the dataset is listed in Table TABREF30. The Bootstrapped set is generated using new AltLexes to identify additional ones based on the Training set, which increased causal examples by about 65 percent. In our experiment, we train MCDN on the Training set and Bootstrapped set separately and finetune hyperparameters on the validation set. The golden annotated set is used as the test set. Hyperparameters. We set the initial learning rate to $1e^{-4}$ then decreased half when the F1-score has stopped increasing more than two epochs. The batch size in this experiment is 32, and the epoch size is 20. To avoid overfitting, we employ two types of regularization during training: 1) dropout for the sums of the embeddings, the outputs of each bi-GRU layer except the last, each layer in FFN and residual dropout for Transformer blocks ; 2) $L_2$ regularization for all trainable parameters. The dropout rate is set to 0.5 and the regularization coefficient is $3e^{-4}$.

Partial prepositional phrases, e.g. They have made l4 self-driving car with the idea of a new neural network. According to our statistics in the parallel data constructed in , there are 1144 AltLexes indicate causal, and 7647 AltLexes indicates non-causal. Meanwhile, their intersection has 144 AltLexes, which is $12.6\%$ of causal sets and $1.8\%$ of non-causal sets. In conclusion, ambiguous connectives and implicit expressions are frequently observed in the AltLexes dataset. Methods based on statistical learning with manual patterns have demerits to build a reliable model in such contexts. However, with the abstraction and reasoning capacity, our model MCDN can be well adapted to these situations.

Consequently, we need to assign a soft weight to a causal and non-causal loss to make the model pay more attention to those examples which are difficult to identify. Motivated by the works , we introduce the focal loss to improve normal cross entropy loss function. The focal loss $L_{fl}$ is formulated as the objective function with the balance weight hyperparameter $\alpha $ and the tunable focusing hyperparameter $\beta \ge 0$. For optimization, we use the Adam optimizer  with $\beta _1 = 0.9$, $\beta _2 = 0.999$, $\epsilon = 1e^{-8}$ and clip the gradients norm.

However, the volume of textual data in the wild, e.g., on the web, is much larger than that in books and newspapers. With the help of mobile technologies, people tend to express personal opinions and record memorable moments on the web, which have become a rich source of causality, consequently. There is a huge demand to investigate an approach for mining both explicit and implicit causality from web text. Despite the success of existing studies on extracting explicit causality, there are few reasons why most existing works cannot be directly applied into causality mining on the web text where a large number of implicit causality cases exist. First, most public datasets for causality mining are collected from books and newspaper where the language expressions are more formal and less diverse than the textual data on the web. Second, it would make the perception of causality incomplete because the existing works mainly focus on explicit causal relations that are expressed by intra-sentence or inter-sentence connectives, without considering ambiguous and implicit cases.

The experimental results show that our model achieves significant improvement over baseline methods, including many state-of-the-art text classification models, which illustrates detecting causality is a complex task. The detection not only requires multi-level information but also needs more reasoning capability than the text classification.

Methods ::: Word Level	There are two sub-layers in the Transformer block: self-attention and feed-forward networks. For stability and superior performance, we add a layer normalization after the residual connection for each of the sub-layers. Self-Attention. In this paper, we employ scaled multi-head self-attention, which has many merits compared with RNN and CNN. Firstly, the "receptive field" of each token can be extended to the whole sequence without long distance dependency diffusion. And any significant token would be assigned a high weight. Secondly, dot-product and multi-head can be optimized for parallelism separately, which is more efficient than RNN. Finally, multi-head model aggregates information from different representation sub-spaces.

Experiment ::: Baseline Methods	In this section, we elaborate on 10 baseline methods. The first five methods are the most common class (MCC), $KLD$, $LS \cup KLD$, $LS \cup KLD \cup CONN$, and $KLD \cup LS \cup LS_{inter}$. $KLD$, $LS$, and $CONN$ represent KL-divergence score, lexical semantic feature, and categorical feature respectively. These methods are used as baselines in Hidey et al.'s work. $KLD$ and $LS \cup KLD$ acquire the best accuracy and precision on the Training set. $LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$ are the best systems with the highest recall and F1-score respectively. The next five are the most commonly used methods in text classification. They are TextCNN, TextRNN, SASE, DPCNN, and BERT. In our experiment, we reproduced all of them except BERT. For BERT, we use the public released pre-trained language model (base). and fine-tuned it on each dataset.

For scaled self-attention, given the input matrix of $n$ query vectors $Q \in \mathbb {R}^{n \times d}$, keys $K \in \mathbb {R}^{n \times d}$ and values $V \in \mathbb {R}^{n \times d}$, computing the output attention score as: We take the input vector matrix $X \in \mathbb {R}^{n \times d}$ as queries, keys, and values matrix and linearly project them $h$ times respectively. Formally, for $i$-$th$ head ${\rm H_i}$ it is formulated as below: Where the learned projections are matrices $W_i^Q \in \mathbb {R}^{d \times d /h}$, $W_i^K \in \mathbb {R}^{d \times d /h}$, $W_i^V \in \mathbb {R}^{d \times d /h}$. Finally, we concatenate each head and map them to the output space with $W_{MH} \in \mathbb {R}^{d \times d}$: Feed-forward Networks.

Experiment ::: Results	Table TABREF30 shows the detection results from the two datasets of our model and competing methods. Firstly, we can see that MCDN remarkably outperforms all other models when trained on both datasets. Although MCDN doesn't obtain the highest precision, it increases F1-score by 10.2% and 3% compared with the existing best systems $LS \cup KLD \cup CONN$ and $KLD \cup LS \cup LS_{inter}$. Furthermore, $KLD$ feature based SVM yields the highest precision on the Training set, though poor recall and F1-score, because it focuses on the substitutability of connectives while the parallel examples usually have the same connective that would be estimated as false negatives. It is remarkable that MCDN is more robust on the original Training set and Bootstrapped set while the feature-based linear SVM and neural network-based approaches presented a considerable difference and got gain even more than 20 on F1-score. Secondly, deep methods tend to acquire balanced precision and recall score except for BERT and MCDN whose recall is significantly higher than precision on Bootstrapped set.

Question: How is Relation network used to infer causality at segment level?
Answer: we integrate object-pairs with global representation and make a pair-wise inference to detect the relationship among the segments