## 3 Methodology
### The Model Framework  
The overall architecture of the model in this paper is shown in Figure 3. It mainly consists of three parts: the input layer, feature extraction layer, and table-generation layer. **Input Layer. ** We use a pre-trained BERT-based model as the encoder for sentences. The concatenated sentence \(Z\) is input to obtain the token-level sentence representation \(H\in\mathbb{R}^{(n+m)\times d_{n}}\):  
\[H=\{t_{1},t_{2}\dots t_{n},t_{n+1}\dots t_{n+m}\}=BERT(\{x_{1},x_{2}\dots x_{n}, r_{1}\dots r_{m}\}) \tag{1}\]  
where \(x_{i}\) is a word in the text sequence \(Z\). After passing through the BERT encoder, it is converted into a word vector and then input into the BERT model to calculate the word embedding. \(t_{i}\) represents the encoded word embedding, and \(d_{n}\) represents the dimension of the word embedding (768 at the base level). We also extract the scores from the self-attention mechanism in the BERT encoder, originating from the multi-head self-attention computation in the encoder of the 12-layer transformer. The output of each BERT encoder layer is based on the output of the previous layer. The specific formula [18] is as follows:  
\[Attention(Q,K,V)_{i}=softmax(S_{i})V=softmax\left(\frac{\left(H_{i-1}W^{Q} \right)\left(H_{i-1}W^{K}\right)^{T}}{\sqrt{d_{k}}}\right)\left(H_{i-1}W^{V}\right) \tag{2}\]  
where \(S_{i}\) represents the scores of the multi-head self-attention mechanism in the \(i\)-th layer of the BERT encoder. \(W^{Q}\), \(W^{K}\), and \(W^{V}\), respectively, represent the learnable mapping matrices for the query matrix \(Q\), key matrix \(K\), and value matrix \(V\) in the attention mechanism. \(H_{i-1}\) represents the hidden layer output of the previous encoder layer, and \(d_{k}\) is the embedding dimension in the attention computation. Here, we focus on the attention scores of the final layer, denoted as \(S_{12}\in\mathbb{R}^{(n+m)(n+m)d_{h}}\), where \(d_{h}\) represents the number of heads. Note that the attention scores taken here are the unnormalized results, which means they have not undergone the operation of softmax normalization. **Feature Extraction Layer. ** We use the representation \(H\) from BERT's final layer output as input to the BiGRU [17] model, which enables capturing more semantic information, enhancing token-level representations, and obtaining the vector representation Y:  
\[Y=BiGRU(H) \tag{3}\]  
Figure 3: The overall architecture of the model, which is divided into three parts: the input layer, feature extraction layer, and table-generation layer. After the BiGRU update, we normalize \(Y\) horizontally, apply a tanh activation layer, and apply dropout, resulting in the input \(U\) for the multi-head tensor dot-product operation. The specific formula is as follows:  
\[U=\tanh(LayerNorm(Y)) \tag{4}\]  
Through our experiments, we found that in our model, the multi-head tensor dot-product and multi-head self-attention mechanisms exhibit similar performance when extracting table features. However, the computational cost of the multi-head tensor dot product is lower. Its computation is similar to that of self-attention but omits the operations involving the value matrix and the multiplication of feature vectors after linear transformation. Firstly, we map \(U\) to the \(Q\) and \(K\) vectors through two linear transformations. Then, we transpose the \(K\) vector and perform matrix multiplication. Finally, we divide the result by the square root of \(\sqrt{d_{k}}\) to obtain the table feature \(R\in\mathbb{R}^{(n+m)^{2}\times d_{h}}\). The specific formulas are as follows:  
\[Q_{i}=U_{i}W_{q}+b_{q} \tag{5}\]  
\[K_{i}=U_{i}W_{k}+b_{k} \tag{6}\]  
\[P_{i}=\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}} \tag{7}\]  
\[\mu_{i}(x,y)=q_{i,x}k_{i,y}^{T} \tag{8}\]  
Formulas (9) and (10) represent the linear transformation operations. \(W_{q}\) and \(W_{k}\) are learnable parameter matrices, \(b_{q}\) and \(b_{k}\) are bias values, and \(Q\), \(K\in\mathbb{R}^{(n+m)\times d_{h}\times d_{h}}\) are the query matrix and key matrix, respectively.