## 3 Approach
### Sentence-Scale Local Features  
Document-scale essay scoring treats an essay as a sequence of words; during scoring with document-scale features, the influence of the sentence as a whole on the scoring of essays is ignored. In previous work, researchers have been focusing on the utilization of sentence-scale features. For example, [PERSON] et al. [37] proposed that combining document-scale features and sentence-scale features could achieve better performance, where the document-scale version is based on the features of the embedding of the input vocabulary and the other module is based on input sentence-scale features, combining the two for training to obtain excellent performance. [PERSON] et al. (2018) proposed a sentence-embedding model SBERT utilizing the pre-training method on the Semantic Textual Similarity (STS) task; its aim is to quantify the degree of semantic similarity or relatedness between pairs of text snippets. In this paper, we extract sentence vectorized representations of essays by SBERT and propose a new sentence-scale feature of the AES task, as shown in Figure 3.  
**Sentence vectorization layer. ** Let the words in the essay sentences be defined as \(\{t_{1},t_{2},\dots,t_{m}\}\), where m denotes the word number of one sentence. We use the pre-trained SBERT model to extract the deep semantic information of the words and obtain the vectorized representation of the sentences in the essay by mean pooling. The separate BERT model cannot be directly used for the calculation of sentence vectors without the pooling layer. The mean pooling applies the same weight to each word in the sentence and it can reduce over-reliance on specific words while preserving the overall semantic information. **Recurrent layer. ** After converting essays into sentence vectors, there is no sequential relationship between the vectors. In order to obtain the original temporal order relationship of the sentences in the essay, we utilize the LSTM model to mine the temporal order features. Specifically, the sequence of sentence vectors \(\{s_{1},s_{2},\dots,s_{N}\}\) obtained from the output of the sentence vectorization layer is added to the LSTM model to obtain the sequence of output vectors \(\{h_{1},h_{2},\dots,h_{N}\}\) to capture the dependencies of the sentences, where \(N\) is the number of sentences in the essay. **Pooling Layer. ** This layer generally uses attention-pooling to compute each time output of the LSTM model at each time step, converting it to a fixed-length vector. [PERSON] et al. (2018) compared MoT pooling and attention-pooling in a global essay scoring task, where MoT pooling can equally treat each word in an essay and can better extract the words' semantic information. During the process of sentence-level semantic extraction, attention-pooling is employed to emphasize the impact of important sentences on essay scoring. Additionally, the sentence sequences are much smaller in comparison to the extra-long sequences of global features. This causes attention to be diminished during attention-pooling for extra-long sequences. [PERSON] et al. (2018) focused on words on a single-layer LSTM but did not exceed the MoT baseline model. Attention pooling obtains the weight of the contribution of the words in the score to the final score in the essay, as shown in the following equation. \[m_{i}=\tanh(W_{m}\cdot h_{i}+b_{m}) \tag{9}\]  
Figure 3: A hybrid neural network model for extracting Sentence-Scale Local Features. \[u_{i}=\frac{e^{W_{m}\cdot h_{i}}}{\sum e^{W_{m}\cdot h_{j}}} \tag{10}\]  
\[x=\sum u_{i}h_{i} \tag{11}\]  
where \(W_{m}\) and \(W_{u}\) are the weight parameters, \(b_{m}\) is the bias vector, \(h_{i}\) is the LSTM intermediate hidden layer state at different time steps, \(m_{i}\) and \(u_{i}\) are the attention vector and attention weight at different time steps, and \(x\) is the final text representation. **Linear layer. ** This layer applies a linear transformation to the output of the pooling layer and normalizes the output using the sigmoid function.