# Toward a Multi-Column Knowledge-Oriented Neural Network for Web Corpus Causality Mining
###### Abstract
We used a 2-layer FFN involving a \"dg\" unit with a ReLU function followed by SoftMax for prediction, which is expressed mathematically in Equation (9). \[\mathrm{FFN}(\mathrm{Final\_rep})=\mathrm{SoftMax}(\mathrm{ReLU}(\mathrm{ Final\_rep}\left(W_{1}+b_{1}\right)W_{2}+b_{2}) \tag{9}\]

There is rich discrimination between causal and non-causal samples in the AltLexes dataset. By using a Cross-Entropy (CE) loss function, the apparent inequality of causal and non-causal examples in the source dataset can lead to unsatisfactory outcomes. Since each connective and segments in the target sentence contain an ambiguous and heterogeneous connective (make, made, create, construct, etc. ), effect keyword (disable, lost, miss, destruction, company, died, etc. ), and causal keywords (lack, accident, fire, tupan, tsunami, flood, earth quick, blast, etc. ), it is hard to detect in each sentence. \[Lfl= \begin{cases}-a(1-\hat{g})\beta&y=1\\ -a(1-\hat{g})\beta&y=0\end{cases}  \tag{10}\]

As a result, it is required to give causal and non-causal losses a soft weight, enabling the model to focus more on ambiguous, implicit, and heterogeneous samples. Inspired by [54; 64], we consider the focal loss into a progress loss function [65], by adding a modulating factor to the CE loss \((1-\hat{g})\beta\), with a tunable hyperparameter \(\beta\geq 0\). In Equation (10), the focal loss \(L_{n}\) is formulated as the objective function, with '\(\alpha\)' denoting the balance weight hyperparameter. ## 4 Experimental Settings

In this part, we explore the MCKN model at the sentence level, which combines three knowledge-oriented channels for causality mining. ### Datasets

We used the \"AltLexes\" web corpus [22] in our proposed technique, which consisted of 86,896 training samples, of which about 7606 are causal, and 79,290 are non-causal samples, a bootstrapped set of 100,744 samples, of which about 12,534 are causal and 88,240 are non-causal. The bootstrapped dataset is produced using new AltLexes to bootstrap to find more examples, which increases the causal samples by about 70 percent. The Dev set of 488 samples, of which about 181 are causal and 307 are non-causal, and the test set contains 611 samples, of which about 315 are causal and 296 are non-causal. We train MCKN on the bootstrapped and training sets separately and fine-tune them on the Dev set. Finally, the model is tested using the test set. ### Hyperparameters and Evaluation Metrics

Hyperparameter: In the implementations, we set the initial learning rate of the proposed model as \(1\times 10^{-2.5}\), and gradually compress after the F1 score has stopped growing for more than 6 epochs. During training, we set the batch size to 32, the epoch size is 15, and apply L2 regularization to deal with the over-fitting issues with a 0.5 dropout rate. We set the regularization coefficient to \(3\times 10^{-5}\). For focal loss, we used \(\alpha=0.80\) and \(\beta=4.5\). For optimization purposes, Adam optimizer [66] is used with \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\in=1\times 10^{-8}\) hyperparameter and clipped gradients norm. We used \(k=130\) for the number of kernel/wf of various window sizes ranging from 1 to 8 at the \"L\" level and 1 to 64 at each of the \"AL\" and \"BL\" levels. In Table 2, we summarize all hyperparameters, which provides a more convenient approach for the reader. \begin{table}
\begin{tabular}{l l l} \hline \hline
**S.No.** & **Hyper Parameters** & **Values** \\ \hline
1. & Learning Rate & \(1\times 10^{-2.5}\) \\ \hline
2. & Batch Size & 32 \\ \hline
3. & Epoch Size & 15 \\ \hline
4. & Regulation function & L\({}_{2}\) \\ \hline
5. & Dropout Rate & 0.5 \\ \hline
6. & Regularization Coefficient & \(3\times 10^{-5}\) \\ \hline
7. & Focal loss & \(\alpha=0.80\), \(\beta=4.5\) \\ \hline
8. & Adam Optimizer & \(\beta_{1}=0.9\), \(\beta_{1}=0.999\), \(\epsilon=1\times 10^{-8}\) \\ \hline
9. & Kernel/Word filter Size & K = 130 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyperparameters and their values used by proposed work.