# Exploring Prompts in Few-Shot Cross-Linguistic Topic Classification Scenarios
## 4 Theoretical Analysis  
Before the experiment, this paper first compares the advantages and disadvantages of finetuning and prompts in theory, followed by a theoretical analysis of discrete prompt, continuous prompt, and hybrid prompt architectures. Taking text topic classification as an example, in the standard finetuning process, researchers typically add a fully connected linear layer to transform dimensions as a classifier after the model. Compared to the original pretrained model, the new layer introduces additional parameters, which increases time and space consumption during training. For example, a binary classification model based on RoBERTa-large [27] introduces at least 2048 additional parameters, and the model requires a large amount of annotated data for training to learn about downstream tasks. However, if there are only 100 training samples in a small sample scenario, finetuning becomes very difficult, which causes overfitting. Unlike finetuning, the prompt method directly converts downstream tasks into masked language modeling tasks with limited output space [28]. It only finetunes the pretrained model parameters without introducing new parameters, narrowing the gap between the  
Figure 3: Hybrid Prompt Model Structure. pretrained and downstream tasks. Therefore, the prompt method is more effective in small sample scenarios than in finetuning. For a discrete prompt, given a pretrained model \(M\), a discrete sequence of input tokens \(x=\{x_{1},x_{2},\ldots,x_{m}\}\) is used by the pretrained model \(M\) of the word-embedding layer \(e\) mapped as an embedding vector \(\{e(x_{0}),e(x_{1}),\ldots,e(x_{n})\}\). The role of the prompt is to map the input \(x\), the tag \(y\), and itself into a template T. For example, if we want to predict the capital of a country, we can construct the prompt: \"The capital of the UK is [MASK]\", where the UK is the input and [MASK] is the location of the label output. The construction method of the discrete prompt mentioned above is straightforward, but in practical use, it faces a discreteness challenge. During the gradient descent process, the word embedding e only changes in a small range for the dense vector generated by any discrete input [29]. In other words, the optimizer is prone to falling into local minima. In order to avoid such discreteness issues, continuous prompts have emerged. The structure of a continuous prompt is shown in Figure 2. Specifically, it includes an embedding layer for encoding virtual tokens into vectors. [PERSON] et al. [30] believed that the suggested embedding vectors should be interdependent rather than independent so that structures such as LSTM [31] or TextCNN [32] can be used to embed virtual tokens for correlation and finally a two-layer neural network connected using a ReLU activation function. If the virtual input token is \(v_{i}\), we can obtain its vector representation using the following equation \(h_{i}\) ([[https://huggingface.co/hfl/cino-large-v2/](https://huggingface.co/hfl/cino-large-v2/)]([https://huggingface.co/hfl/cino-large-v2/](https://huggingface.co/hfl/cino-large-v2/)) (accessed on 15 March 2023)):  
\[ \begin{array}{l}h_{i}=\text{MLP}(\overrightarrow{[h_{i}];\overleftarrow{ h_{i}}]})\\ =\text{MLP}([\text{LSTM}(h_{0i});\text{ LSTM}(h_{mi})])\\ =\text{MLP}([\text{LSTM}(e(v_{0}),e(v_{1}),\ldots,e(v_{i}));\text{LSTM}(e(v _{m}),e(v_{m-1}),\ldots,e(v_{i}))])\end{array}  \tag{2}\]  
Continuous prompts solve the instability of discrete prompts and avoid the problem of manual construction of hints, but they still have some shortcomings. For example, it is not clear what the virtual token really represents, so there is a real possibility that the virtual token will not work. To address this situation, this paper proposes a hybrid prompt that introduces a discrete prompt within a continuous prompt. The original intention of the hybrid prompt's design is to avoid a situation where virtual tokens do not work.