# Causality Mining in Natural Languages Using Machine and Deep Learning Techniques: A Survey
## 3 Deep Neural Models, Frameworks, and Techniques
### Deep Learning Techniques for Causality Mining
They extract causality from each dataset as positive labeled data and extract a random mix of other relations as negative data. Ref. [174] propose a novel technique using multi-column convolutional neural networks (MCNNs) and source background knowledge (BK) for CM. It is a variant of CNN [175] with several independent columns. The inspiration for this work was [12]. They used short binary patterns to connect pairs of nouns like \"A causes B\" and \"A prevents B\" to increase the performance of event causality recognition. They focused on such event causalities, \"smoke cigarettes\" \(\rightarrow\) \"die of lung cancer\" by taking an original sentence from which the candidate of causalities is extracted with the addition of related BK taken from the web texts. Three distinct methods are used to get related texts for a given causality candidate from 4 billion web pages as a source of BK, including (1) Why-question answering, (2) Using Binary Pattern (BP), and (3) Clues Terms. These techniques identify useful BK scattered in the web archives and feed into MCNNs for CM. In Figure 7, the architecture of MCNN is presented, which consists of 8 columns, where five columns are used to process event causality candidates and their nearby contexts in the original sentence. The other three columns deal with web archives. Then the output of all columns based on their layers combination is combined into the last layer for final prediction. Using all types of BK (Base + BP + WH + CL), the top achieved average precision is 55.13%, which is 7.6% higher than the best of [12] methods (47.52%). Note that by extending single CNN's to multi-column CNN's (CNN-SENT vs. Base), the proposed work obtained a 5.6% improvement, and further gave 5.8% improvement by adding with external BK. Figure 6: Processing level of DL techniques. Ref. [176] enhanced MCNN by adding causality attention (CA), which results in the CA-MCNN model. This model is based on two notions that enhanced why-QA, which includes expressing implicitly expressed causality in one text by explicit cues from other text and describing the causes of similar events by using a set of similar words. In [177], a novel set of event semantics and position features are used to train a Feed-Forward Network (FFN) for implicit causality. This work aims to improve ANN with features that take assistance from linguistic and associated works. It captures knowledge about the position and content of events contained in the relation. They used Penn Discourse Treebank (PDTB) and CST News (CST-NC) corpus. The whole objective function of the proposed algorithm is shown in Equation (11). \[J=-\sum_{n=1}^{\infty}\sigma\bigg{(} \begin{array}{c}W_{2}||\max(\tanh(W_{1 }.(Xi.E\oplus Xe+b_{1})\\ \oplus Xp)||+b_{2})\log P(y)+\ell||\theta||2\end{array} \bigg{)} \tag{11}\]  
where the set of parameters is \(\Theta=\{\)E, \(W_{1}\), b\({}_{1}\), W\({}_{2}\), b\({}_{2}\)}, cross-entropy function is used for the loss function, which is regularized by the squared norm of parameters and scaled by hyperparameter (\(\ell\)), positional features (Xp), input indices array (Xi), the true class label (y), and event-related features (Xe). Table 4 lists the most popular DL approaches for CM based on their targets, architecture, datasets, and references. A neural encoder-decoder approach predicts causally related events in stories through standard evaluation framework choice of plausible alternatives (COPA) [178]. This was the first approach to evaluate a neural-based model for such kinds of tasks, which learns to predict relations between adjacent sequences in stories as a means of modeling causality. The bi-LSTM [179] is a linguistically informed architecture for automatic CM using word linguistics features and word-level embedding. It contains three modules: linguistic preprocessor and feature extractor, resource creation, and prediction background for cause/effect. A causal graph is created after grouping and proper generalization of the extracted events and their relations.