## 2 Related Works  
This section starts by introducing the BERT network model, the baseline work in the area of ASAG, and describing how dataset augmentation influences the performance of ASAG. In some various NLP tasks, BERT set a new state-of-the-art performance. BERT is a pre-trained transformer network [18]. [PERSON] and [PERSON] recommended Sentence-BERT (SBERT), which uses Siamese BERT-Networks [14] to overcome some deficiencies in BERT. SBERT is a modification of the pre-trained BERT network that uses Siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity. The SentenceTransformers framework provides various pre-trained models for NLP tasks. The model size and performance are different to each other. All these models have been evaluated for performance sentence embeddings and performance semantic search [14; 19]. Table 1 shows some existing models related to sentence classification and question-answering. Several studies with varied models have achieved a good performance. [PERSON] et al. proposed automatic short answer grading using paragraph vectors and transfer learning embedding [12]. In this work, they utilized the Texas dataset by [PERSON] to evaluate the models. The input for the ASAG model is the vector that represents the student answer (SA), along with the vector that represents the reference answer (RA). In each experiment, the vectors are inferred using two models; the paragraph vector (PV) model and the transfer learning model. Then, the similarity between SA and RA is measured using the cosine similarity. After that, the computed cosine similarity is used as a feature for a regression model to predict a particular answer score. They evaluate the models by comparing the actual score provided in the dataset, along with the predicted score using two evaluation metrics, the Pearson correlation coefficient and root mean square error (RMSE). For the PV vectors, it achieved 0.401 for the Pearson correlation and 0.893 for the RMSE. For the transfer learning, they applied the Roberta-large and Scibert models. The best accuracy achieved by fine-tuning the Roberta-large model on the domain-specific corpus was 0.620 for the Pearson correlation and 0.777 for the RMSE. This superiority is reasonable, because transformers can learn the context of the words from both directions. On the contrary, the pre-trained paragraph vectors perform better than the trained paragraph vectors on a domain-specific corpus. This indicates that paragraph vectors increase the model's generalizability. The second approach for improving the performance of ASAG is to use transfer learning and augmentation described by the authors of [13]. They fine-tuned three-sentence transformer models on the SPRAG (Short Programming Related Answer Grading Dataset) corpus with five different augmentation techniques: viz., random deletion, synonym replacement, random swap, back translation, and NLPAug. The SPRAG corpus contains student responses involving keywords and special symbols. The dataset size is 4039 records, and it is a binary classification problem. They experimented with four different data sizes (25%, 50%, 75%, and 100%) with the augmented data to determine the impact of training data on the fine-tuned sentence transformer model. An SBERT architecture with a pretrained language model (PLM) was used for training. The experimentation used the stsb-disilbert-base, paraphrase-albertsmall-v2, and quora-distilbert-base pre-trained sentence transformer models. This paper provides an exhaustive analysis of fine-tuning pretrained sentence transformer models with varying sizes of data by applying text augmentation techniques. They found that applying random swap and synonym replacement techniques together while fine-tuning gave a significant improvement, with a 4.91% increase in accuracy (84.21%) and a 3.36% increase in the F1-score (88.11%).