# Toward a Multi-Column Knowledge-Oriented Neural Network for Web Corpus Causality Mining
###### Abstract
Likewise, the early work segments embedded here indicate segments L, BL, and AL in each sentence. In Figure 2, first of all, the \"word2 vec toolkit\" is used for pre-training of word embedding with dimension \(d_{word}\), positional embedding with dimension \(d_{pos}\), and segment embedding with dimension \(d_{seg}\) for linguistic information. Lastly, summing all three embeddings results in new representation \(\hat{N}=\{z_{1},z_{2},z_{3}\ldots,z_{n-1},z_{n}\}\), where \(z_{n}\in R_{d}\) for token \(n_{i}\), and keep equal \(d=d_{word}=d_{pos}=d_{seg}\) dimensions of the word embedding, position embedding, and segment embedding. Therefore, the \(\hat{N}\) representation of input sentences could bring fundamental features to complicated networks. ### Relation Network

In visual question answering (V-QA) [59], the relation network (RN) plays a very significant role. It can be efficiently integrated with DL approaches including CNNs (DeepCNN, knowledge CNN, and MCNN) and RNN (GRU, bi-GRU, LSTM, bi-LSTM) for performance enhancement. The original RN, however, only performs single-step inference, such as \(A\to B\) rather than \(A\to B\to C\). For those tasks which need multistep relational reasoning, [PERSON]. [60] introduced RNNs that work on graph representations of entities.. Furthermore, [PERSON]. [61] made memory networks with RNs capable of complicated reasoning, which changed the computational complexity from nonlinear to linear. Though, most jobs are only used for text and V-QA. Similarly, we consider RN in the proposed model, which takes input object pairs from KCs and makes effective relational reasoning. ### About Knowledge-Oriented Channel

Encouraged by [26], we have applied three different KCs to recognize keywords, cue words, and cue phrases of causality in connective and segment levels of the input sentence. For convolution operation, we used \"wf\" in each channel, which is a variant of the convolutional filter. It is automatically generated from knowledge bases (WordNet, FrameNet) using the linguistic knowledge of causality. Compared with the CNN convolutional filters, the \"wf\" more precisely represents causal relationships. The weights of \"wf\" are the pre-trained word embedding, which can be used without additional training. Using the \"wf\" approach will significantly drop the number of pre-parameters of the model and reduce the over-fitting issue in the small data corpus. Figure 1 depicts the proposed network's architecture, which consists of three channels. Each channel has its specific input segment format including segments BL, AL, and connective L. The \"L\" is usually used to represent the cue phrases, cue words, and keywords for cause-effect relations. Figure 2: Input representation for MCKN. The word, segment, and position embedding are added to create the input embedding. Examples of such connectives include because, as result, lead to, resulted, due to, and trigger. These words in the connectives part away from the BL and AL segments and may affect the performance of a network. In the past, KCs only paid attention to the \"L\" part of the input sentence (between event e1 and event e2) because it usually represents causality signals. However, in the proposed model, each segment and connective has its own KCs. To decrease the morphological variations of tokens in each segment, we used WordNet tokens to make it consistent and mark each word in its lowercase by using the lemmatizer function, and further, every word is converted into a precise input format as shown in Figure 2. The single knowledge-oriented channel of the proposed model is shown in Figure 3. In the input format, we set the maximum size of L as 8 words, and each BL and AL to be 64 words. Sentences with fewer than 8 words in the \"L\" level and fewer than 64 words in the segments level are padded with padding characters with zero embedding because CNN works with a fixed input size.