# Causality Mining in Natural Languages Using Machine and Deep Learning Techniques: A Survey
## 3 Deep Neural Models, Frameworks, and Techniques
### Motivation for Causality Mining
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**SNo** & **References** & **Deep Neural Networks** & **Applications and Structure** \\ \hline \multirow{5}{*}{5.} & & & RNN is a type of RvNN, comprehensively used in many NLP tasks. Since NLP is dependent on the sequence of words such as sentences /phonemes, it is beneficial to have a memory of the preceding elements when processing new ones. Sometimes, backward dependencies exist that correct processing of certain words/tokens may depend on words that follow it. Hence, it is crucial for RNN to look at the sentences in the forward and backward direction and integrate their outputs. This organization of RNN’s is known as a bidirectional RNN. This design may allow the effect of input to longer than a single RNN layer and letting for longer-term effects. This sequential design of RNN cells is known RNN stack [150]. RNN is applicable for sequential NLP tasks, and as well as for speech processing. \\ \hline
6. & [151, 152] & Generative & \\  & Adversarial Network (GAN) & Applicable for unsupervised learning and using game-theoretical context. \\ \hline
7. & [153] & Variational Autoencoder (VAE) & Applicable for unsupervised learning and based on the Probabilistic Graphical model. \\ \hline
8. & [131, 154] & GRU & GRU is an extended version of RNN and a simpler variant of the LSTM, usually perform better than standard LSTMs in several NLP sequential tasks. \\ \hline \multirow{9}{*}{9.} & & & LSTM is one of the prominently enhanced forms of RNN. In LSTMs, the recursive neurons are consist of many different neurons linked in a sequential structure to preserve, expose, or forget some precise information. While standard RNN’s of the single node seeking back to then and have some memory of long passed outcomes, these outcomes are merged in each consecutive iteration. Usually, it is significant to remember data from the distant past, however and at the same time, other very latest data may not be vital. LSTM can remember important data much longer, while inappropriate data can be forgotten. It plays a very important in sequential computation. \\ \hline
10. & [133] & bi-LSTM & Bi-LSTM is an enhanced form of LSTM that works in both left and right directions to deal with the problem. It is applicable for sequential NLP tasks and uses derived features from lexical resources such as NLP and WordNet systems. \\ \hline \multirow{11}{*}{11.} & & & Encoder-Decoder pair is typically used for text summarization, machine translation, or captioning, results is in textual form. An encoding ANN is used to yield a vector of a specific length and a decoding ANN is used to return variable size text based on the vector. Issue in this system: RNN is enforced to encode the whole sequence to a finite length vector without affecting to whether or not any of the inputs are more significant than others. A strong solution to this issue: Using the attention mechanism. The first prominent use of an attention mechanism is the condensed layer for an annotated parameter of RNN hidden state, letting the network obtain what to pay attention in accordance with the annotation and current hidden state [156]. \\ \cline{1-1} \cline{2-2}
12. & [135] & ELMo & They used a feature-based approach and task-specific architectures that contain pre-trained representations as additional features. \\ \hline
13. & [136] & Generative Pre-trained Transformer (OpenAI GPT) & Applicable for unsupervised learning by Improving language understanding. \\ \hline
14. & [137] & BERT-base & BERT-base is the enhance form of Transformer, which deal the source sentence in both direction. It uses a bi-directional encoder-decoder along with attention mechanism. It is conceptually very simple and empirically influential. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Cont. to overcome the dimensionality cures by learning a distributed illustration of tokens and giving a likelihood function for structures.