# Uncertainty-Aware Active Meta-Learning for Few-Shot Text Classification
## 2 Background Literature
### LLMs for Few-Shot Text Classification  
In recent times, the field of natural language processing has seen the rise of language models trained through self-supervised learning, which is based on linguistic patterns and applied to vast amounts of text data [13; 14]. Self-supervised learning involves creating a task that allows the learning model to discern patterns in a dataset that lacks answer labels. This method enables the model to learn patterns in the data without constructing supervised learning data on large amounts of text data.  
The advent of such large language models can be traced back to the transformer model [15]. The transformer was introduced to enhance the computational speed of the Seq2 Seq model, which is an existing translation model that applies attention based on Recurrent Neural Networks (RNNs) [16]. The transformer model is composed of a combination of encoder and decoder modules. Since the RNN model that processes the input data sequence has been removed, the transformer uses a positional encoding method that artificially provides order information of input data. As a result, the sum of the token vector arbitrarily assigned to each token and positional encodings, which is a vector expressing the relative positions of words in a sentence, is input to the encoder. The corresponding input value extracts various features through a self-attention layer, which encodes the meaning of each token considering context information in the currently input sentence [17]. There are various methods of self-attention depending on the method of calculating the weight between the query and the key-value pair, but in the transformer, the multi-head attention that performs the scaled dot-product attention was used.  
Thereafter, the same linear transformation is performed for each token in a feed-forward layer and output representations. The output value goes into the input of the next encoder, and after repeating this process N times, it is input to the decoder. In performing each output, the decoder applies the attention mechanism using the information summarized in the encoder to perform the output for each sequence.  
The effectiveness of the transformer has been demonstrated in both natural language understanding and generation, and various large language models have been proposed based on it. BERT is a language model built using only the encoder part of a transformer. It has been established as one of the transfer learning in the field of natural language processing by updating the best performance in the task of NLU [14]. The process of creating a model that performs a target task using BERT is divided into two steps. The first step is pre-training, a self-supervised learning process for a large amount of text data, and the second step is fine-tuning, a supervised learning process for the downstream task.  
Despite the accomplishments NLU has so far, one of the biggest limitations, like other types of machine learning paradigms, is that they need large amounts of data. The performance in the different areas of NLP of such high-resource languages improved over time, but languages that lack these resources, i.e., low-resource languages, suffer performance loss [18]. Low-resource NLU is a task that requires an adequate understanding of a given natural language based on a small number of label examples. Meta-learning, discussed in the next section, gives an alternative solution to the problem of having a limited number of resources when training language models. It allows NLU models to understand low-resource languages better; hence, we utilize this method in combination with others to solve the problem of low-resource languages.