# Unlocking Everyday Wisdom: Enhancing Machine Comprehension with Script Knowledge Integration
## 3 Method  
### Baseline Model  
The baseline model we re-implemented is called Three-Way Attentive Networks (Tri-ANs) [2]. It models the interactions between the passage and the question, the question and the answer, and the passage and the answer by using a three-way attention mechanism. An overview of the model framework is shown in Figure 3. It consists of an input layer, an attention layer, and an output layer. Input Layer: A training example consists of a passage \(\{P_{i}\}_{i=1}^{|P|}\), a question \(\{Q_{i}\}_{i=1}^{|Q|}\), an answer \(\{A_{i}\}_{i=1}^{|A|}\), and a label \(y^{*}\in\{0,1\}\) as input. \(P_{i}\) is the representation of the \(i\)-th word in the passage, which is the same for the question and the answer. For the representation, we used the Glove word vector [18] for the passage, the question, and the answer (\(\textbf{E}_{P_{i}}^{GLOVE}\), \(\textbf{E}_{Q_{i}}^{GLOVE}\), and \(\textbf{E}_{A_{i}}^{GLOVE}\)), as well as used the following features:  
* POS Embeddings (\(\textbf{E}_{P_{i}}^{POS}\) and \(\textbf{E}_{Q_{i}}^{POS}\)): Randomly initialized 12-dimensional vectors trained to encode pre-labeled part-of-speech tags (whether the word is a noun, verb, etc.) for passage and question texts. * NER Embeddings (\(\textbf{E}_{P_{i}}^{NER}\)): Randomly initialized 8-dimensional vectors trained to encode pre-labeled name-entity recognition tags (whether the word belongs to categories such as people, company, date, etc) for the passage texts. * REL Embeddings (\(\mathbf{E}_{P_{i}}^{REL}\)): Randomly initialized 10-dimensional vectors trained to encode a relationship between a word in the passage and any word in the question/answer. Such relationship comes from querying ConceptNet for an edge between the words. If multiple relationships exist, a random one is chosen. * Handcrafted Features: Handcrafted features include logarithmic term frequency (\(\mathbf{E}_{P_{i}}^{TF}\)) and co-occurrence features (\(\mathbf{E}_{P_{i}}^{CO}\)). Logarithmic term frequency features come from English Wikipedia. Co-occurrence features are binary, thus being true if a passage word is found in the question/answer. \[\mathbf{W}_{P_{i}}=[\mathbf{E}_{P_{i}}^{GLOVE};\mathbf{E}_{P_{i}}^{POS};\mathbf{E }_{P_{i}}^{NER};\mathbf{E}_{P_{i}}^{REL};\mathbf{E}_{P_{i}}^{CO};\mathbf{E}_{P_ {i}}^{TF}]\] (1)  
\[\mathbf{W}_{Q_{i}}=[\mathbf{E}_{Q_{i}}^{GLOVE};\mathbf{E}_{Q_{i}}^{POS}] \tag{2}\]  
\[\mathbf{W}_{A_{i}}=[\mathbf{E}_{A_{i}}^{GLOVE}] \tag{3}\]  
Attention Layer: We used word-level attention to model interactions between the passage, the question, and the answer. The model first calculates context vectors for the passage by attending to the question. Then, it calculates context vectors for the answer by attending to the question and the passage. The attention function is represented in (4) and (5). \(f\) is a non-linear activation function and is set to \(ReLU\). From the attention function, we can compute question-aware passage representations \(\mathbf{W}_{P_{i}}^{\vartheta}\), passage-aware answer representations \(\mathbf{W}_{A_{i}}^{p}\), and question-aware answer representations \(\mathbf{W}_{A_{i}}^{q}\) in (6)-(8). \[Att_{seq}(\mathbf{u},\{\mathbf{v}_{i}\}_{i=1}^{n})=\sum_{i=1}^{n}\alpha_{i} \mathbf{v}_{i} \tag{4}\]  
\[\alpha_{i}=softmax_{i}(f(\mathbf{W}_{1}\mathbf{u})^{T}f(\mathbf{W}_{1}\mathbf{ v}_{i})) \tag{5}\]  
\[\mathbf{W}_{P_{i}}^{q}=Att_{seq}(\mathbf{W}_{P_{i}},\{\mathbf{W}_{Q_{i}}\}_{i=1 }^{|Q|}) \tag{6}\]  
\[\mathbf{W}_{A_{i}}^{p}=Att_{seq}(\mathbf{W}_{A_{i}},\{\mathbf{W}_{P_{i}}\}_{i=1 }^{|P|}) \tag{7}\]  
\[\mathbf{W}_{A_{i}}^{q}=Att_{seq}(\mathbf{W}_{A_{i}},\{\mathbf{W}_{Q_{i}}\}_{i= 1}^{|Q|}) \tag{8}\]  
After that process, three groups of context embeddings are appended to the original embeddings to form the final embeddings. Then, three BiLSTMs are applied to the concatenated embeddings to model the temporal dependency, as shown in (9)-(11).