## 2 Methodologies
### Input Embedding Module  
When modeling text-related data, the first step is to vectorize it. In machine learning, common methods for text representation include one-hot encoding, bag-of-words model, and TF-IDF. However, in deep learning, the most common approach is to map individual words (or characters) to a low-dimensional dense vector space using an embedding layer known as Token Embedding. Unlike CNN models with n-gram or RNN models with temporal structures, these models' network structures already possess the capability to capture sequential features. However, this is not the case for models based on attention mechanisms. This is because attention mechanisms, in their computational process, involve linear transformations of a few matrices without considering the positional information of words in the text. In self-attention mechanisms, the representation of each word is computed based on the representations of all other words in the text, without explicit indication of which words are closer or farther from the current word. Therefore, to enable attention-based models to utilize the sequential information in the text, the addition of positional information becomes crucial. The introduction of positional vectors addresses this issue. By adding a positional vector to each word, the model can better differentiate words at different positions in the text and capture the relationships and dependencies between words more sensitively during the attention computation process. As shown in Figure 1, the CAC model consists of two key components in the Input Embedding module: Token Embedding and Positional Embedding. Token Embedding assigns an index to each word based on a constructed vocabulary (Word Sequence), and then uses an Embedding layer to represent each word as a word vector. On the other hand, Positional Embedding encodes the positional information of each word (Position sequence), and also utilizes an Embedding layer to convert the positional encoding into positional vectors. The presence of positional vectors enables the model to better understand the temporal information within the text and accurately capture important contextual relationships. In our model, we employ learnable embeddings to represent the positional encoding. As shown in Figure 2, for a given word, its input representation \(E=[e_{1},e_{2},\ldots,e_{n}]\) is obtained by element-wise summation of the word embeddings \(T=[t_{1},t_{2},\ldots,t_{n}]\) and position embeddings \(P=[p_{1},p_{2},\ldots,p_{n}]\) transformed by the Embedding layer. Each element of the positional embeddings \(P\) is learnable, effectively preserving the position information. The calculation formula is as follows:  
\[E=LayerNorm\Big{(}Dropout\Big{(}T*\sqrt{d}+P\Big{)}\Big{)}, \tag{1}\]  
where \(E\), \(T\), and \(P\) are all \(n\times d\) matrices, where \(n\) represents the length of the sentence sequence, and \(d\) represents the dimensionality of the vectors. In the calculation process, a series of operations are performed on the input word vectors. First, we scale the word vector matrix \(T\) by a factor of \(\sqrt{d}\) to adapt the range of values to the training process of the model, ensuring better convergence. Next, we perform an element-wise addition of the scaled word vector matrix \(T\) and the position vector matrix \(P\) to introduce positional information of the words in the sentence. With the addition of positional vectors, the model can better distinguish words at different positions in the sentence and more sensitively capture the relationships and dependencies between words in the subsequent feature extraction process. Figure 2: Computational process of input embedding. To address the issues of overfitting and vanishing gradients, we have incorporated Dropout and LayerNorm techniques. Dropout randomly sets a fraction of the neuron outputs to zero, reducing overfitting in neural networks.