MutantPrompt: Prompt Optimization via Mutation Under a Budget on Modest-sized LMs
Authors: Arijit Nag, Soumen Chakrabarti, Animesh Mukherjee, Niloy Ganguly
Accepted at: acl25

MutantPrompt: Prompt Optimization via Mutation Under a Budget on Modest-sized LMs

Arijit Nag
IIT Kharagpur
arijitnag@iitkgp.ac.in

Animesh Mukherjee
IIT Kharagpur
animeshm@cse.iitkgp.ac.in

Niloy Ganguly
IIT Kharagpur
niloy@cse.iitkgp.ac.in

Soumen Chakrabarti
IIT Bombay
soumen@cse.iitb.ac.in

Abstract

Prompts serve as a critical instruction interface to unlock the diverse capabilities of Large Language Models (LLMs), thus directly influencing the quality of their outputs. While prompt engineering has shown great promise, identifying optimal prompts remains a significant challenge, particularly for low-resource languages, which often face higher computational costs due to increased token generation and limited gold standard task data. In response, we propose MutantPrompt: a framework that leverages multi-armed bandit algorithms to efficiently identify optimal prompts tailored to low-resource languages. By framing prompt selection as an exploration-exploitation problem under a fixed computational budget, the framework dynamically balances exploring new prompts with exploiting known high-performing ones. We demonstrate the framework’s effectiveness across multiple low-resource Indic language tasks, including classification, question-answering and causal reasoning using three small parameter-size LLMs. The results highlight the cost efficiency of the search method in finding optimal prompts and resulting performance improvements. Our codes are publicly available at https://github.com/NLPatCNERG/MutantPrompt.

1 Introduction

Large Language Models (LLMs) (OpenAI et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Chowdhery et al., 2022) have revolutionized natural language processing (NLP), enabling state-of-the-art performance across various tasks, from text generation to complex reasoning. However, LLMs’ effectiveness heavily depends on the quality of the prompts used to interact with them. Prompt engineering—the practice of crafting effective input instructions to elicit high-quality outputs—has gained considerable attention as a means to optimize LLM performance. Despite its potential, identifying the most effective prompts remains a challenging problem, particularly in low-resource language (LRL) settings, perhaps widening the performance gap between LRLs and HRLs (high-resource languages) (Hendy et al., 2023; Jiao et al., 2023; Bang et al., 2023; Toraman et al., 2023; Fujii et al., 2023).

Low-resource languages face unique obstacles when interacting with LLMs. These languages often have limited annotated datasets, making it difficult to fine-tune models effectively. In addition, due to differences in tokenization (Muller et al., 2021; Rust et al., 2021; Ahia et al., 2023; Petrov et al., 2023) and linguistic structures, they frequently incur higher computational costs (Ahia et al., 2023; Petrov et al., 2023; Nag et al., 2024) as LLMs generate more tokens to process input and produce output with slow inference (Petrov et al., 2023; Hofmann et al., 2022). A naïve approach of finding optimal prompts using a trial-and-error method can be costly in such settings, as the higher number of token generations results in a significantly increased budget requirement. Furthermore, compared to HRL tasks, LRL tasks have less available gold standard data (Ulmer et al., 2022; Ko et al., 2021; Zhou et al., 2022), necessitating prompt optimization methods that rely less on such data to be effective.

This paper introduces MutantPrompt, a novel framework that addresses these challenges by leveraging multi-armed bandit (MAB) algorithms to optimize prompt selection for LRLs. By framing prompt selection as an exploration-exploitation problem, the framework strategically balances the discovery of new prompts with the utilization of previously identified high-performing ones under a fixed computational budget. This approach enables efficient search and selection of effective prompts without exhaustive manual tuning or expensive computational resources.

We evaluate MutantPrompt across multiple
NLP tasks in 11 low-resource Indic languages, including classification, question-answering, and causal reasoning using three recent small-size (2b-7b parameters) LLMs, namely Llama-3.2 (Dubey et al., 2024), Mistral-3 (Jiang et al., 2023) and Gemma-2 (Gemma Team et al., 2024). Our experimental results demonstrate the cost efficiency of the search methodology in identifying optimal prompts while improving task performance. The findings underscore the importance of intelligent prompt selection mechanisms in promoting equitable and efficient LLM applications, ensuring that LRL users benefit from the advancement of NLP.

2 Related work

Continuous prompt-based methods (Li and Liang, 2021; Liu et al., 2023; Zhang et al., 2022a), which tune the parameters of specific input tokens, have demonstrated effectiveness but is costly and less interpretable (Lester et al., 2021), while discrete prompt optimization, using task-specific instructions or keywords, is more interpretable and user-friendly (Schick and Schütze, 2021; Liu et al., 2021). Various methods for automatic discrete prompt search and generation have been explored (Zhang et al., 2022b; Shin et al., 2020; Shi et al., 2022; Wallace et al., 2021; Deng et al., 2022; Chen et al., 2023; Yang et al., 2024; Lin et al., 2024). However, many rely on model gradients or token probabilities, which may not suit all models. Others select the best prompts through enumeration, re-sampling, or iterative edits (Zhou et al., 2023; Jiang et al., 2020; Zhang et al., 2022b; Prasad et al., 2023; Sclar et al., 2024). While effective, these methods can be computationally expensive or converge to suboptimal solutions. Recently, EvoPrompt (Guo et al., 2024) demonstrates the effectiveness of genetic and evolutionary algorithms to give optimal prompts without the need for model parameter tuning and is doing better than recent pseudo-gradient (Pryzant et al., 2023) and model token probability-based methods (Zhou et al., 2023). While their method provides valuable insights, it relies on extensive inference budgets due to longer instructions and full development data for scoring prompts, which can be costly. In contrast, MutantPrompt uses a budget-constrained MAB approach, exploiting and mutating the best-performing prompt using simple instructions and a dynamic mutation rate while using a small subset of data for scoring. We compare MutantPrompt with EvoPrompt to demonstrate performance and cost improvements.

Algorithm 1 MutantPrompt.

Inputs:
• Seed prompt \( p_{seed} \), Instruction \( p_{instr} \), Budget \( B \)
• Max mutation rate \( R_{max} \), Min mutation rate \( R_{min} \)
• Decay rate \( R_{decay} \), Decay step \( D_{step} \)
• Dev dataset \( D_{dev} \), Batch size \( b \)

1: \( \mathcal{P} \leftarrow \{p_{seed}\} \) /* Prompt pool */
2: \( \mathcal{T} \leftarrow \frac{B}{D_{step}} \) /* Total iterations */
3: \( Count_{succ} \leftarrow \{ \} \) /* Success counts, empty dictionary */
4: \( Count_{fail} \leftarrow \{ \} \) /* Failure counts, empty dictionary */
5: for \( i \in [0, T) \) do
6:   for \( p_i \in \mathcal{P} \) do
7:     \( \Theta[i] \leftarrow \beta_i(Count_{succ}[i] + 1, Count_{fail}[i] + 1) \)
      /* Sample prompt's success probability from Beta distribution */
8:     \( p_{curr} \leftarrow \arg\max_{p \in \mathcal{P}} \Theta[p] \)
9:     batch_succ, batch_fail = evaluate_prompt(\( p_{curr}, D_{dev}, b \)) /* Evaluate the prompt on a batch of dev set and return the success and failure count */
10:    \( Count_{succ}[p_{curr}] += batch\_succ \)
11:    \( Count_{fail}[p_{curr}] += batch\_fail \)
12:    \( R_{curr} = \max(R_{min}, R_{max} \cdot R_{decay}^{\frac{i}{D_{step}}}) \) /* Update mutation rate */
13:    \( Prob_{mut} \sim U(0, 1) \) /* Generate random probability of mutation from uniform distribution */
14:    if \( Prob_{mut} \leq R_{curr} \) then
15:      \( p_{best} = \arg\max_{p \in \mathcal{P}} \frac{Count_{succ}[p]}{Count_{succ}[p]+Count_{fail}[p]} \)
16:      \( p_{new} = refine\_prompt(p_{instr}, p_{best}) \) /* Generate new prompt from \( p_{best} */ 
17:      \( \mathcal{P} \leftarrow \mathcal{P} \cup p_{new} \)
18:      \( Count_{succ}[p_{new}] = 0 \)
19:      \( Count_{fail}[p_{new}] = 0 \)
20:      \( p_{final} \leftarrow \arg\max_{p \in \mathcal{P}} \frac{Count_{succ}[p]}{Count_{succ}[p]+Count_{fail}[p]} \)
21:      return \( p_{final} \) /* Best performing prompt */

3 Proposed method: MutantPrompt

In this work, given a task and a budget, we frame the search for an optimal prompt as a Multi-Armed Bandit (MAB) problem where each arm represents a prompt (we use Thompson Sampling to implement MAB). In addition, we want to extend the prompt pool by mutating the best-performing prompt after each iteration. We describe MutantPrompt in Algorithm 1. At the start of each iteration, we sample the probability of success of each prompt \( p_i \) denoted by \( \Theta_i \), from a beta distribution with the parameter \( \alpha_i \) and \( \beta_i \) as the number of successes and a number of failures correspondingly till now for the prompt (line 7). Initially, they are set to zero, making it a uniform distribution. Next, we take the prompt with the highest success probability (line 8) and explore it by evaluating it on a small test set, and update the success and failure count of the prompt (lines 9–11). Subsequently, we probabilistically mutate the prompt
<table>
  <tr>
    <th>Task\| Lang→</th>
    <th>Method</th>
    <th>as</th>
    <th>bn</th>
    <th>gu</th>
    <th>hi</th>
    <th>kk</th>
    <th>ml</th>
    <th>mr</th>
    <th>or</th>
    <th>pa</th>
    <th>ta</th>
    <th>te</th>
    <th>avg.</th>
  </tr>
  <tr>
    <td rowspan="6">IndicSentiment</td>
    <td><i>pseed</>(Llama-3.2)</i></td>
    <td>57.60</td>
    <td>55.60</td>
    <td>71.80</td>
    <td>54.40</td>
    <td>39.50</td>
    <td>53.80</td>
    <td>38.30</td>
    <td>50.30</td>
    <td>59.80</td>
    <td>34.70</td>
    <td>29.10</td>
    <td>49.54</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Llama-3.2)</i></td>
    <td>85.10</td>
    <td>80.60</td>
    <td>89.40</td>
    <td>92.00</td>
    <td>87.20</td>
    <td>87.20</td>
    <td>90.20</td>
    <td>76.20</td>
    <td>90.40</td>
    <td>92.30</td>
    <td>91.90</td>
    <td>88.32</td>
  </tr>
  <tr>
    <td><i>pseed</>(Gemma-2)</i></td>
    <td>89.00</td>
    <td>95.30</td>
    <td>92.50</td>
    <td>96.70</td>
    <td>92.10</td>
    <td>96.70</td>
    <td>96.50</td>
    <td>90.30</td>
    <td>93.50</td>
    <td>92.50</td>
    <td>90.70</td>
    <td>90.79</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Gemma-2)</i></td>
    <td>89.80</td>
    <td>95.50</td>
    <td>94.80</td>
    <td>96.70</td>
    <td>92.50</td>
    <td>93.30</td>
    <td>95.20</td>
    <td>86.40</td>
    <td>91.10</td>
    <td>93.50</td>
    <td>92.30</td>
    <td>90.79</td>
  </tr>
  <tr>
    <td><i>pseed</>(Mistral-3)</i></td>
    <td>64.70</td>
    <td>80.80</td>
    <td>61.50</td>
    <td>86.70</td>
    <td>61.50</td>
    <td>58.80</td>
    <td>55.10</td>
    <td>62.50</td>
    <td>63.50</td>
    <td>56.00</td>
    <td>53.50</td>
    <td>64.19</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Mistral-3)</i></td>
    <td>75.90</td>
    <td>86.40</td>
    <td>73.00</td>
    <td>91.80</td>
    <td>77.50</td>
    <td>64.60</td>
    <td>78.10</td>
    <td>67.30</td>
    <td>62.90</td>
    <td>73.30</td>
    <td>65.20</td>
    <td>74.18</td>
  </tr>
  <tr>
    <td rowspan="6">IndicXNLI</td>
    <td><i>pseed</>(Llama-3.2)</i></td>
    <td>33.90</td>
    <td>34.60</td>
    <td>31.55</td>
    <td>46.80</td>
    <td>11.35</td>
    <td>34.85</td>
    <td>42.45</td>
    <td>33.55</td>
    <td>28.55</td>
    <td>27.90</td>
    <td>30.60</td>
    <td>32.37</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Llama-3.2)</i></td>
    <td>36.35</td>
    <td>41.50</td>
    <td>36.75</td>
    <td>46.80</td>
    <td>43.10</td>
    <td>45.30</td>
    <td>45.55</td>
    <td>37.65</td>
    <td>43.40</td>
    <td>41.20</td>
    <td>37.40</td>
    <td>41.91</td>
  </tr>
  <tr>
    <td><i>pseed</>(Gemma-2)</i></td>
    <td>45.35</td>
    <td>52.35</td>
    <td>49.45</td>
    <td>58.85</td>
    <td>49.50</td>
    <td>49.50</td>
    <td>51.35</td>
    <td>38.10</td>
    <td>48.55</td>
    <td>52.05</td>
    <td>50.40</td>
    <td>49.59</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Gemma-2)</i></td>
    <td>52.40</td>
    <td>58.80</td>
    <td>58.35</td>
    <td>64.85</td>
    <td>57.00</td>
    <td>56.25</td>
    <td>57.75</td>
    <td>41.65</td>
    <td>56.65</td>
    <td>53.95</td>
    <td>57.70</td>
    <td>56.43</td>
  </tr>
  <tr>
    <td><i>pseed</>(Mistral-3)</i></td>
    <td>35.05</td>
    <td>40.50</td>
    <td>36.20</td>
    <td>45.45</td>
    <td>36.15</td>
    <td>35.05</td>
    <td>36.85</td>
    <td>33.45</td>
    <td>35.80</td>
    <td>38.55</td>
    <td>34.75</td>
    <td>37.07</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Mistral-3)</i></td>
    <td>43.00</td>
    <td>49.95</td>
    <td>46.10</td>
    <td>55.30</td>
    <td>45.65</td>
    <td>45.25</td>
    <td>44.70</td>
    <td>44.05</td>
    <td>49.30</td>
    <td>44.85</td>
    <td>39.25</td>
    <td>46.13</td>
  </tr>
  <tr>
    <td rowspan="6">IndicXParaphrase</td>
    <td><i>pseed</>(Llama-3.2)</i></td>
    <td>33.45</td>
    <td>31.00</td>
    <td>28.45</td>
    <td>36.95</td>
    <td>8.50</td>
    <td>26.10</td>
    <td>30.45</td>
    <td>49.95</td>
    <td>46.40</td>
    <td>-</td>
    <td>23.20</td>
    <td>31.45</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Llama-3.2)</i></td>
    <td>61.85</td>
    <td>73.60</td>
    <td>52.55</td>
    <td>49.95</td>
    <td>60.20</td>
    <td>67.20</td>
    <td>66.35</td>
    <td>49.30</td>
    <td>59.81</td>
    <td>-</td>
    <td>63.45</td>
    <td>60.43</td>
  </tr>
  <tr>
    <td><i>pseed</>(Gemma-2)</i></td>
    <td>60.00</td>
    <td>80.50</td>
    <td>52.80</td>
    <td>83.70</td>
    <td>65.45</td>
    <td>68.85</td>
    <td>72.05</td>
    <td>49.75</td>
    <td>53.40</td>
    <td>-</td>
    <td>69.80</td>
    <td>65.87</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Gemma-2)</i></td>
    <td>69.45</td>
    <td>91.95</td>
    <td>71.55</td>
    <td>94.50</td>
    <td>73.60</td>
    <td>76.30</td>
    <td>84.40</td>
    <td>50.15</td>
    <td>56.16</td>
    <td>-</td>
    <td>75.10</td>
    <td>74.31</td>
  </tr>
  <tr>
    <td><i>pseed</>(Mistral-3)</i></td>
    <td>63.40</td>
    <td>81.90</td>
    <td>51.35</td>
    <td>92.50</td>
    <td>65.90</td>
    <td>63.05</td>
    <td>72.50</td>
    <td>50.60</td>
    <td>52.25</td>
    <td>-</td>
    <td>54.70</td>
    <td>64.82</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Mistral-3)</i></td>
    <td>62.55</td>
    <td>86.10</td>
    <td>53.55</td>
    <td>93.40</td>
    <td>67.80</td>
    <td>66.75</td>
    <td>74.55</td>
    <td>50.70</td>
    <td>53.35</td>
    <td>-</td>
    <td>56.10</td>
    <td>66.49</td>
  </tr>
  <tr>
    <td rowspan="6">IndicCOPA</td>
    <td><i>pseed</>(Llama-3.2)</i></td>
    <td>54.20</td>
    <td>65.20</td>
    <td>55.36</td>
    <td>72.16</td>
    <td>57.80</td>
    <td>55.60</td>
    <td>57.02</td>
    <td>49.00</td>
    <td>58.40</td>
    <td>60.80</td>
    <td>61.20</td>
    <td>58.79</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Llama-3.2)</i></td>
    <td>57.20</td>
    <td>67.80</td>
    <td>59.60</td>
    <td>72.16</td>
    <td>62.20</td>
    <td>57.40</td>
    <td>59.91</td>
    <td>49.00</td>
    <td>65.00</td>
    <td>62.80</td>
    <td>59.40</td>
    <td>61.13</td>
  </tr>
  <tr>
    <td><i>pseed</>(Gemma-2)</i></td>
    <td>58.20</td>
    <td>62.00</td>
    <td>58.71</td>
    <td>70.60</td>
    <td>58.00</td>
    <td>57.80</td>
    <td>57.46</td>
    <td>49.20</td>
    <td>58.60</td>
    <td>59.60</td>
    <td>60.00</td>
    <td>59.11</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Gemma-2)</i></td>
    <td>60.60</td>
    <td>65.00</td>
    <td>62.50</td>
    <td>69.27</td>
    <td>58.00</td>
    <td>60.80</td>
    <td>60.36</td>
    <td>51.20</td>
    <td>60.00</td>
    <td>62.80</td>
    <td>64.00</td>
    <td>61.32</td>
  </tr>
  <tr>
    <td><i>pseed</>(Mistral-3)</i></td>
    <td>55.00</td>
    <td>64.60</td>
    <td>54.02</td>
    <td>73.05</td>
    <td>57.60</td>
    <td>58.60</td>
    <td>57.02</td>
    <td>49.40</td>
    <td>57.60</td>
    <td>61.80</td>
    <td>58.80</td>
    <td>58.86</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Mistral-3)</i></td>
    <td>55.00</td>
    <td>63.40</td>
    <td>57.37</td>
    <td>73.27</td>
    <td>59.60</td>
    <td>61.00</td>
    <td>63.03</td>
    <td>53.00</td>
    <td>61.40</td>
    <td>60.80</td>
    <td>67.60</td>
    <td>61.41</td>
  </tr>
  <tr>
    <td rowspan="6">CSQA</td>
    <td><i>pseed</>(Llama-3.2)</i></td>
    <td>29.66</td>
    <td>33.05</td>
    <td>13.99</td>
    <td>38.57</td>
    <td>32.68</td>
    <td>26.65</td>
    <td>33.50</td>
    <td>19.13</td>
    <td>32.48</td>
    <td>31.55</td>
    <td>31.75</td>
    <td>29.36</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Llama-3.2)</i></td>
    <td>35.50</td>
    <td>36.50</td>
    <td>32.30</td>
    <td>47.52</td>
    <td>41.04</td>
    <td>33.20</td>
    <td>41.10</td>
    <td>31.68</td>
    <td>39.49</td>
    <td>33.35</td>
    <td>33.15</td>
    <td>36.80</td>
  </tr>
  <tr>
    <td><i>pseed</>(Gemma-2)</i></td>
    <td>32.15</td>
    <td>34.95</td>
    <td>19.70</td>
    <td>38.72</td>
    <td>37.29</td>
    <td>27.50</td>
    <td>33.00</td>
    <td>26.55</td>
    <td>21.91</td>
    <td>31.40</td>
    <td>31.90</td>
    <td>30.47</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Gemma-2)</i></td>
    <td>34.28</td>
    <td>38.30</td>
    <td>38.33</td>
    <td>42.47</td>
    <td>37.37</td>
    <td>32.25</td>
    <td>37.00</td>
    <td>28.39</td>
    <td>26.23</td>
    <td>33.15</td>
    <td>31.15</td>
    <td>36.46</td>
  </tr>
  <tr>
    <td><i>pseed</>(Mistral-3)</i></td>
    <td>26.46</td>
    <td>24.35</td>
    <td>14.90</td>
    <td>19.76</td>
    <td>21.72</td>
    <td>23.05</td>
    <td>21.05</td>
    <td>25.48</td>
    <td>22.22</td>
    <td>20.50</td>
    <td>23.15</td>
    <td>22.44</td>
  </tr>
  <tr>
    <td><i>pfinal</>(Mistral-3)</i></td>
    <td>26.16</td>
    <td>24.35</td>
    <td>36.50</td>
    <td>22.06</td>
    <td>27.83</td>
    <td>26.30</td>
    <td>23.00</td>
    <td>28.69</td>
    <td>26.93</td>
    <td>25.80</td>
    <td>24.15</td>
    <td>26.52</td>
  </tr>
</table>

Table 1: Performance comparison of seed prompt (<i>pseed</i>) and the optimal prompt (<i>pfinal</i>) generated by MutantPrompt for different tasks, languages and LLMs. On the right, we report the average performance across languages for a particular task and LLM. Best performances are marked in Bold and underlined.

with the best performance and add it to the prompt pool (line 14-19) (Appendix A Figure 1 shows the instruction <i>pinstr</i> used for this). By mutating the best prompt, we expect another reasonably effective prompt to be added to the pool, making the prompt pool rich. We also exponentially reduce the mutation rate (line 12) as we move forward before it saturates to the minimum mutation rate. There are two motivations for doing this – first, as we start with a single prompt, it is essential to mutate more frequently in the beginning to increase the prompt pool. Second, if we keep mutating at a higher rate throughout iterations, the prompt pool will explode, not converging the beta distributions as exploration outweighs exploitation. After the entire budget is consumed, we select the prompt with the best success ratio as the optimal prompt (line 20).

In summary, starting with a single seed prompt, the algorithm iteratively evaluates its performance and refines the prompt pool. At each iteration, Thompson Sampling is used to estimate the success probabilities of all prompts in the pool, selecting the most promising one for evaluation. The success and failure counts of the selected prompt are updated based on its performance. To expand the pool, the algorithm probabilistically mutates the best-performing prompt into a new one using a dynamic mutation rate that decreases over time, ensuring a balance between exploration (discovering new prompts) and exploitation (exploring existing prompts to build confidence). This iterative process continues until the computational budget is exhausted, after which the prompt with the highest success rate is selected as the optimal solution. Details of algorithm parameters are in Appendix B (Table 5).

4 Experiment and results

To check the effectiveness of MutantPrompt, we experiment with five datasets, of which four are from IndicXTREME (Doddapaneni et al., 2023) – IndicSentiment (sentiment classification), IndicXNLI (natural language inference), IndicCOPA (commonsense causal reasoning), IndicX-Paraphrase (paraphrase detection) and one from IndicGLUE (Kakwani et al., 2020) – CSQA (multiple-choice QA) covering 11 Indian languages. We use three recent small parameter size LLMs (2B-7B parameters) to show the technique’s effectiveness in resource-constrained setup. The LLMs are Mistral-3(7B), Llama-3.2(3B) and Gemma-2(2B). For performance metrics, we use exact match accuracy for all the tasks. Details
<table>
  <tr>
    <th>Task</th>
    <th>Lang→</th>
    <th>Method</th>
    <th>as</th>
    <th>bn</th>
    <th>gu</th>
    <th>hi</th>
    <th>kn</th>
    <th>ml</th>
    <th>mr</th>
    <th>or</th>
    <th>pa</th>
    <th>ta</th>
    <th>te</th>
    <th>avg.</th>
  </tr>
  <tr>
    <td rowspan="2">IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>84.50</td>
    <td>83.80</td>
    <td>87.84</td>
    <td>80.88</td>
    <td>84.30</td>
    <td>83.60</td>
    <td>86.20</td>
    <td>45.65</td>
    <td>84.10</td>
    <td>81.50</td>
    <td>89.10</td>
    <td>81.04</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>85.10</td>
    <td>89.60</td>
    <td>89.40</td>
    <td>92.00</td>
    <td>87.20</td>
    <td>87.20</td>
    <td>90.20</td>
    <td>76.20</td>
    <td>90.40</td>
    <td>92.30</td>
    <td>91.90</td>
    <td>88.32</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>35.10</td>
    <td>35.70</td>
    <td>33.60</td>
    <td>47.60</td>
    <td>37.05</td>
    <td>39.15</td>
    <td>42.60</td>
    <td>36.70</td>
    <td>35.85</td>
    <td>30.25</td>
    <td>32.25</td>
    <td>36.90</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>36.35</td>
    <td>41.50</td>
    <td>36.75</td>
    <td>46.80</td>
    <td>43.10</td>
    <td>45.30</td>
    <td>45.55</td>
    <td>37.65</td>
    <td>43.40</td>
    <td>47.20</td>
    <td>37.40</td>
    <td>41.91</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>56.80</td>
    <td>69.55</td>
    <td>52.40</td>
    <td>75.20</td>
    <td>46.35</td>
    <td>51.25</td>
    <td>59.70</td>
    <td>57.95</td>
    <td>54.00</td>
    <td>-</td>
    <td>57.00</td>
    <td>57.66</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>61.85</td>
    <td>73.60</td>
    <td>52.55</td>
    <td>49.95</td>
    <td>60.20</td>
    <td>67.20</td>
    <td>66.35</td>
    <td>49.30</td>
    <td>59.81</td>
    <td>-</td>
    <td>63.45</td>
    <td>60.43</td>
  </tr>
  <tr>
    <td rowspan="2">IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>52.40</td>
    <td>66.40</td>
    <td>60.49</td>
    <td>72.61</td>
    <td>56.00</td>
    <td>59.20</td>
    <td>59.69</td>
    <td>47.60</td>
    <td>60.80</td>
    <td>62.40</td>
    <td>57.60</td>
    <td>59.56</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>57.20</td>
    <td>67.80</td>
    <td>59.60</td>
    <td>72.16</td>
    <td>62.20</td>
    <td>57.40</td>
    <td>59.91</td>
    <td>49.00</td>
    <td>63.00</td>
    <td>62.80</td>
    <td>59.40</td>
    <td>61.13</td>
  </tr>
  <tr>
    <td rowspan="2">CSQA</td>
    <td>EvoPrompt</td>
    <td>33.72</td>
    <td>35.75</td>
    <td>24.20</td>
    <td>38.87</td>
    <td>33.13</td>
    <td>35.55</td>
    <td>32.25</td>
    <td>30.76</td>
    <td>33.63</td>
    <td>36.30</td>
    <td>33.75</td>
    <td>33.45</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>35.50</td>
    <td>36.50</td>
    <td>32.30</td>
    <td>47.52</td>
    <td>41.04</td>
    <td>33.20</td>
    <td>41.10</td>
    <td>31.68</td>
    <td>39.49</td>
    <td>33.35</td>
    <td>33.15</td>
    <td>36.80</td>
  </tr>
</table>

Table 2: Performance comparison of MutantPrompt with EvoPrompt for the Llama-3.2(3B) model. On the right, we report the average performance across languages for the tasks. Best performances are marked in Bold and underlined.

of model parameters are in Appendix B (Table 6).

<table>
  <tr>
    <th>Lang</th>
    <th>\( p_{seed} \rightarrow p_{final} \)</th>
    <th>\( p_{seed} \rightarrow p_{final} \)</th>
    <th>\( p_{seed} \rightarrow p_{final} \)</th>
  </tr>
  <tr>
    <td>as</td>
    <td>35.35→35.09</td>
    <td>32.50→36.82</td>
    <td>29.56→35.50</td>
  </tr>
  <tr>
    <td>bn</td>
    <td>36.70→36.45</td>
    <td>34.55→37.25</td>
    <td>33.05→36.50</td>
  </tr>
  <tr>
    <td>gu</td>
    <td>21.50→33.60</td>
    <td>17.45→25.35</td>
    <td>13.90→32.30</td>
  </tr>
  <tr>
    <td>hi</td>
    <td>44.42→42.42</td>
    <td>40.82→42.02</td>
    <td>38.57→47.52</td>
  </tr>
  <tr>
    <td>kn</td>
    <td>37.57→37.12</td>
    <td>35.37→38.62</td>
    <td>32.68→41.04</td>
  </tr>
  <tr>
    <td>ml</td>
    <td>32.35→31.00</td>
    <td>29.10→29.65</td>
    <td>26.65→33.20</td>
  </tr>
  <tr>
    <td>mr</td>
    <td>37.90→36.10</td>
    <td>35.25→36.70</td>
    <td>33.50→41.10</td>
  </tr>
  <tr>
    <td>or</td>
    <td>26.86→27.77</td>
    <td>22.11→28.23</td>
    <td>19.13→31.68</td>
  </tr>
  <tr>
    <td>pa</td>
    <td>37.82→39.37</td>
    <td>34.82→34.72</td>
    <td>32.48→39.49</td>
  </tr>
  <tr>
    <td>ta</td>
    <td>35.95→36.30</td>
    <td>33.60→34.20</td>
    <td>31.55→33.35</td>
  </tr>
  <tr>
    <td>te</td>
    <td>35.00→33.20</td>
    <td>33.90→35.15</td>
    <td>31.75→33.15</td>
  </tr>
  <tr>
    <td>avg.</td>
    <td>34.67→35.31</td>
    <td>31.77→34.43</td>
    <td>29.35→36.80</td>
  </tr>
</table>

Table 3: Performance comparison between different seed prompt (\( p_{seed}^i \)) to their final optimal prompt (\( p_{final}^i \)) for the CSQA dataset using Llama-3.2(3B). The last row shows the average performance across languages for \( p_{seed}^i \) and \( p_{final}^i \). Here, the performances of \( p_{seed}^i \) degrade as \( i \) increases, but \( p_{final}^i \) remains in a similar range, showing the robustness of MutantPrompt.

4.1 Generated prompt is better than seed

In Table 1, we compare the performance of \( p_{seed} \) with that of \( p_{final} \) for different tasks and languages. We observe that \( p_{final} \) always performs better than \( p_{seed} \), irrespective of the LLMs used. The performance gain is as high as 90% for Llama-3.2 and 25% for Gemma-2 and Mistral-3, which shows the effectiveness of our approach. The huge gain for Llama-3.2 is possibly due to its ability to respond better to optimally designed prompts. Overall, the generation of a more elaborate prompt can be one of the reasons behind the performance improvement (details are in Figure 2 of the Appendix).

4.2 Our method beats recent baseline

In Table 2, we compare the performance of the final optimal prompt generated (for our case it is \( p_{final} \)) by MutantPrompt and the recent prompt generation technique EvoPrompt starting from same seed prompt (\( p_{seed} \)) for all the tasks using the Llama-3.2 model (results with other LLMs are in Appendix Table 8 & 7). Here, we see that most of the time, MutantPrompt shows better performance than EvoPrompt. Although there are cases where MutantPrompt lags behind or gives similar results compared to EvoPrompt, MutantPrompt proves to be more cost-effective (detailed discussion in Section 4.4).

<table>
  <tr>
    <th>Task</th>
    <th>Method</th>
    <th>Token<sub>instr</sub></th>
    <th>|\( p_{new} \)|</th>
  </tr>
  <tr>
    <td rowspan="2">IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>991</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>263</td>
    <td>129</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>1011</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>293</td>
    <td>130</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>983</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>275</td>
    <td>116</td>
  </tr>
  <tr>
    <td rowspan="2">IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>1039</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>332</td>
    <td>114</td>
  </tr>
  <tr>
    <td rowspan="2">CSQA</td>
    <td>EvoPrompt</td>
    <td>1019</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>303</td>
    <td>142</td>
  </tr>
</table>

Table 4: Here, we compare the token length of the instruction prompts (\( Token_{instr} \)) between EvoPrompt and MutantPrompt for different tasks using Llama-3.2(3B) (other LLM results are in Appendix Table 9 & 10). We also compare the average number of new prompts (\( |p_{new}| \)) explored by both techniques.

4.3 Robustness of generated prompt

In Table 3, we check the robustness of the generated prompt \( p_{final} \) for MutantPrompt. Here, we take the task as CSQA, Llama-3.2 as the LLM, and use three different seed prompts denoted by \( p_{seed}^1 \), \( p_{seed}^2 \) and \( p_{seed}^3 \). We see the seed prompts perform differently and, on average, follow the \( p_{seed}^1 > p_{seed}^2 > p_{seed}^3 \). But if we check the final prompt performances generated from each of these three seed prompts, they all give performance improvement but, more importantly, show similar performance. It shows MutantPrompt is not very dependent on the starting seed prompts, which makes it very practical in real-world use cases where the user does not need to worry much about designing the seed prompt.

4.4 Cost-effective and explores more

In Table 4, we analyze the token count (a proxy for incurred cost) of the instruction prompt that di-
rects the LLM to generate a new prompt for MutantPrompt alongside EvoPrompt. We find that the EvoPrompt instruction prompt results in 3X-4X more tokens than MutantPrompt, making it more cost-effective. Also, we compare the total number of new prompts (\( p_{new} \)) explored between the two techniques, given the same budget (here, we use 4000 as the total budget). It shows MutantPrompt explores 15%-40% more prompts throughout its journey, making it more probable to find the optimal prompt.

5 Conclusion

This paper presents a framework leveraging multi-armed bandit algorithms to generate and search for optimal prompts for low-resource language tasks. By framing the prompt selection process as an exploration-exploitation problem under a fixed computational budget, the proposed approach efficiently balances the discovery of new prompts with the exploitation of known high-performing ones. This method addresses the unique challenges faced by low-resource language tasks, including limited gold standard data, high computational costs due to increased token generation, and the need to optimize prompts without overwhelming smaller LLMs with overly complex instructions. Our experimental evaluations on various NLP tasks in low-resource Indic languages demonstrate the effectiveness and cost-efficiency of this approach. Moving forward, we want to explore the proposed framework for more complex tasks and other LRLs, fostering more inclusive and efficient LLM querying.

6 Limitations

While our proposed framework using the multi-armed bandit for prompt optimization demonstrates effectiveness in low-resource language settings, there are several limitations that we want to work on in future.

Generalization across languages: While the framework shows promise for low-resource Indic languages, its generalization to other low-resource languages with vastly different linguistic structures remains untested. Further evaluations are needed to understand its effectiveness across diverse language families.

Dependency on LLM quality: The effectiveness of the optimized prompts is inherently tied to the capabilities and limitations of the underlying LLM.

In low-resource settings, smaller or less powerful models may still struggle with nuanced tasks, even with optimized prompts.

Scalability to complex tasks: The framework has been evaluated on classification, question-answering, and causal reasoning tasks. However, its scalability and effectiveness on more complex NLP tasks, such as generative tasks or tasks requiring multi-step reasoning, remain to be explored.

Exploration-exploitation trade-off: Although the framework dynamically balances exploration and exploitation using simple heuristics, finding the optimal balance is challenging, especially in low-resource settings where computational budgets are limited. An inappropriate balance may lead to either premature convergence to suboptimal prompts or excessive exploration, wasting computational resources. We need to devise a concrete approach that can decide the balance based on task quality and the available development dataset.

References

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models.

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wengliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Zhiwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.

Lichang Chen, Jiahai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. 2023. Instructzero: Efficient instruction optimization for black-box large language models.

Aakanksha Chowdhery et al. 2022. Palm: Scaling language modeling with pathways.

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learning.

Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages.

Abhimanyu Dubey et al. 2024. The llama 3 herd of models.
Takuro Fujii, Koki Shibata, Atsuki Yamaguchi, Terufumi Morishita, and Yasuhiro Sogawa. 2023. How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 39–49, Toronto, Canada. Association for Computational Linguistics.

Gemma Team et al. 2024. Gemma 2: Improving open language models at a practical size.

Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2024. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers.

Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation.

Valentin Hofmann, Hinrich Schütze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b.

Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know?

Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine.

Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. In Findings of EMNLP.

Wei-Jen Ko, Ahmed El-Kishky, Adithya Renduchintala, Vishrav Chaudhary, Naman Goyal, Francisco Guzmán, Pascale Fung, Philipp Koehn, and Mona Diab. 2021. Adapting high-resource NMT models to translate low-resource related languages without parallel data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 802–812, Online. Association for Computational Linguistics.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, Online. Association for Computational Linguistics.

Xiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu, Yao Shu, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Use your instinct: Instruction optimization for llms using neural bandits coupled with transformers.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhihlin Yang, and Jie Tang. 2023. Gpt understands, too.

Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.

Anjiti Nag, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. 2024. Cost-performance optimization for processing low-resource language tasks using commercial llms.

OpenAI et al. 2023. Gpt-4 technical report.

Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages.

Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023. Grips: Gradient-free, edit-based instruction search for prompting large language models.

Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chen-guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search.

Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the
59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.

Timo Schick and Hinrich Schütze. 2021. Exploiting cloze questions for few shot text classification and natural language inference.

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.

Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2022. Toward human readable prompt tuning: Kubrick’s the shining is a good movie, and a good prompt too?

Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts.

Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 22(4).

Hugo Touvron et al. 2023. LLaMA 2: Open foundation and fine-tuned chat models.

Dennis Ulmer, Jes Frellsen, and Christian Hardmeier. 2022. Exploring predictive uncertainty and calibration in NLP: A study on the impact of method & data scarcity. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2707–2735, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2021. Universal adversarial triggers for attacking and analyzing nlp.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers.

Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022a. Differentiable prompt makes pre-trained language models better few-shot learners.

Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022b. Tempera: Test-time prompting via reinforcement learning.

Ran Zhou, Xin Li, Ruidan He, Lidong Bing, Erik Cambria, Luo Si, and Chunyan Miao. 2022. MELM: Data augmentation with masked entity language modeling for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2251–2262, Dublin, Ireland. Association for Computational Linguistics.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers.
MutantPrompt: Prompt Optimization via Mutation Under a Budget on Modest-sized LMs (Appendix)

A Supplementary results

Below is a description of a task and a sample prompt to query a LLM system to perform the tasks. You need to generate a new prompt to solve the task. Please use the given sample prompt as a reference for designing the new prompt, keeping the underlying theme intact try to improve it by adding variation in phrasing, structure, or style. Like the given sample prompt that assumes inputs are already available in the prompt, the new prompt also should not ask for user inputs/avoid word/phrase like as e.g., 'provide a', 'provide the') or keep placeholders for user inputs and should be phrased in the second person (e.g., 'you', 'your') when asking for correct alternative. The prompt should explicitly mention that only the correct option should be the output. Write the prompt in such a way that a user will use this generated prompt to query a LLM system to solve the task. Output the new prompt directly without starting with phrases like 'Here's a new prompt...'.

Task description: "A paragraph contains a word denoted by '<MASK>'. The task is to guess the correct alternatives to replace the '<MASK>' word out of the four words given as options."

Sample prompt: "In the given text, you'll find a blank represented by '<MASK>'. From the provided choices, select the word that, when inserted, would render the sentence grammatically correct and coherent within the given context. Your response should solely contain the appropriate word.

Example:
Text: I went to the <MASK> and bought a new book.
Options: store, library, market, park
Correct Answer: store
Paragraph: {paragraph}
Options: {option_a}, {option_b}, {option_c}, {option_d}"

New prompt:

Figure 1: Template for generating new prompt (\( p_{new} \)) in the *Refine\_prompt* function described in Algorithm 1.

B Experimental settings

We run all the experiments on a single RTX A6000 GPU with 48GB RAM. To save GPU hour and cost, we run all the inference and prompt generation experiments one time (except for Table 3, where we do it with 3 different seeds), and to make them reproducible, we fix the LLM seed value to 42. To run MutantPrompt, it took around 2 hr for a task and language with a budget of 4000 on a single 48GB RTXA6000 GPU. For zero-shot inference, for each task, we select 2000 random instances for each language (For IndicSentiment and IndicXParaphrase, we use 500 data as data is limited). For EvoPrompt, we use a development dataset size of 50 for prompt optimization.
<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>B</td>
    <td>[400–4000]</td>
  </tr>
  <tr>
    <td>R<sub>max</sub></td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>R<sub>min</sub></td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>R<sub>decay</sub></td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>D<sub>step</sub></td>
    <td>[1,5]</td>
  </tr>
  <tr>
    <td>b</td>
    <td>[3,5]</td>
  </tr>
</table>
Table 5: Algorithm 1 parameter details.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>LLM</td>
    <td>LLama-3.2(3B), Gemma-2(2B), Mistral-3(7B)</td>
  </tr>
  <tr>
    <td>LLM parameter size</td>
    <td>2-7 Billion</td>
  </tr>
  <tr>
    <td>LLM model type</td>
    <td>Instruction tuned</td>
  </tr>
  <tr>
    <td>LLM temperature</td>
    <td>0.5 (for prompt generation), 0.01(for evaluation)</td>
  </tr>
  <tr>
    <td>LLM top p</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>LLM max token length</td>
    <td>512(for prompt generation), 64(for prediction)</td>
  </tr>
  <tr>
    <td>Seed</td>
    <td>42</td>
  </tr>
</table>
Table 6: LLM parameter details.

<table>
  <tr>
    <th>Task | Lang →</th>
    <th>Method</th>
    <th>as</th>
    <th>bn</th>
    <th>gu</th>
    <th>hi</th>
    <th>kn</th>
    <th>ml</th>
    <th>mr</th>
    <th>or</th>
    <th>pa</th>
    <th>ta</th>
    <th>te</th>
    <th>avg.</th>
  </tr>
  <tr>
    <td rowspan="2">IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>88.90</td>
    <td>95.70</td>
    <td>92.80</td>
    <td>93.60</td>
    <td>92.50</td>
    <td>92.80</td>
    <td>93.90</td>
    <td>72.10</td>
    <td>90.20</td>
    <td>92.50</td>
    <td>92.60</td>
    <td>91.19</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>89.80</td>
    <td>95.50</td>
    <td>94.80</td>
    <td>96.90</td>
    <td>92.50</td>
    <td>92.30</td>
    <td>92.20</td>
    <td>65.40</td>
    <td>91.10</td>
    <td>95.50</td>
    <td>93.20</td>
    <td>91.20</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>54.75</td>
    <td>58.10</td>
    <td>56.45</td>
    <td>57.45</td>
    <td>47.20</td>
    <td>43.10</td>
    <td>47.25</td>
    <td>40.40</td>
    <td>38.00</td>
    <td>21.05</td>
    <td>10.07</td>
    <td>38.25</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>52.40</td>
    <td>58.80</td>
    <td>58.35</td>
    <td>64.85</td>
    <td>57.00</td>
    <td>56.25</td>
    <td>57.75</td>
    <td>41.65</td>
    <td>56.65</td>
    <td>59.35</td>
    <td>57.70</td>
    <td>56.43</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>68.40</td>
    <td>80.50</td>
    <td>72.95</td>
    <td>79.65</td>
    <td>81.45</td>
    <td>78.95</td>
    <td>53.20</td>
    <td>53.10</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>73.85</td>
    <td>72.51</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>69.45</td>
    <td>91.95</td>
    <td>71.50</td>
    <td>74.50</td>
    <td>73.60</td>
    <td>76.30</td>
    <td>84.40</td>
    <td>50.15</td>
    <td>56.16</td>
    <td>-</td>
    <td>-</td>
    <td>75.10</td>
    <td>74.31</td>
  </tr>
  <tr>
    <td rowspan="2">IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>58.00</td>
    <td>61.80</td>
    <td>53.35</td>
    <td>65.92</td>
    <td>52.00</td>
    <td>59.80</td>
    <td>60.58</td>
    <td>47.60</td>
    <td>55.40</td>
    <td>58.60</td>
    <td>59.40</td>
    <td>57.50</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>60.66</td>
    <td>65.00</td>
    <td>62.50</td>
    <td>69.27</td>
    <td>58.00</td>
    <td>60.80</td>
    <td>60.36</td>
    <td>51.20</td>
    <td>60.00</td>
    <td>62.80</td>
    <td>64.00</td>
    <td>61.32</td>
  </tr>
  <tr>
    <td rowspan="2">CSQA</td>
    <td>EvoPrompt</td>
    <td>31.89</td>
    <td>32.00</td>
    <td>31.70</td>
    <td>38.12</td>
    <td>35.84</td>
    <td>29.00</td>
    <td>31.15</td>
    <td>27.47</td>
    <td>25.58</td>
    <td>33.35</td>
    <td>31.95</td>
    <td>31.64</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>34.28</td>
    <td>38.30</td>
    <td>38.50</td>
    <td>42.47</td>
    <td>37.39</td>
    <td>32.25</td>
    <td>37.05</td>
    <td>28.39</td>
    <td>26.23</td>
    <td>35.15</td>
    <td>31.30</td>
    <td>36.46</td>
  </tr>
</table>
Table 7: Performance comparison between MutantPrompt technique with EvoPrompt for the generated optimal prompt (\( p_{final} \)) on Gemma-2(2B) model. On the right, we report the average performance across languages for the tasks. Best performances are marked in Bold and underlined.

<table>
  <tr>
    <th>Task | Lang →</th>
    <th>Method</th>
    <th>as</th>
    <th>bn</th>
    <th>gu</th>
    <th>hi</th>
    <th>kn</th>
    <th>ml</th>
    <th>mr</th>
    <th>or</th>
    <th>pa</th>
    <th>ta</th>
    <th>te</th>
    <th>avg.</th>
  </tr>
  <tr>
    <td rowspan="2">IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>74.90</td>
    <td>86.40</td>
    <td>72.20</td>
    <td>91.90</td>
    <td>77.10</td>
    <td>63.10</td>
    <td>77.70</td>
    <td>68.90</td>
    <td>62.50</td>
    <td>72.90</td>
    <td>63.60</td>
    <td>73.75</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>75.90</td>
    <td>86.40</td>
    <td>73.00</td>
    <td>91.80</td>
    <td>77.50</td>
    <td>64.60</td>
    <td>78.10</td>
    <td>67.30</td>
    <td>62.90</td>
    <td>73.30</td>
    <td>65.20</td>
    <td>74.18</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>42.10</td>
    <td>30.15</td>
    <td>37.15</td>
    <td>52.55</td>
    <td>42.80</td>
    <td>41.85</td>
    <td>48.20</td>
    <td>43.90</td>
    <td>42.75</td>
    <td>47.00</td>
    <td>40.30</td>
    <td>44.43</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>43.00</td>
    <td>49.95</td>
    <td>46.10</td>
    <td>55.30</td>
    <td>45.45</td>
    <td>45.25</td>
    <td>47.40</td>
    <td>44.00</td>
    <td>49.40</td>
    <td>44.85</td>
    <td>39.25</td>
    <td>46.13</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>68.05</td>
    <td>88.50</td>
    <td>52.50</td>
    <td>92.10</td>
    <td>64.50</td>
    <td>66.75</td>
    <td>72.45</td>
    <td>58.70</td>
    <td>57.75</td>
    <td>-</td>
    <td>-</td>
    <td>66.98</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>68.20</td>
    <td>91.10</td>
    <td>57.75</td>
    <td>92.75</td>
    <td>64.70</td>
    <td>66.75</td>
    <td>58.70</td>
    <td>57.75</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>66.98</td>
  </tr>
  <tr>
    <td rowspan="2">IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>53.40</td>
    <td>50.20</td>
    <td>49.11</td>
    <td>55.68</td>
    <td>49.60</td>
    <td>48.80</td>
    <td>53.45</td>
    <td>49.60</td>
    <td>40.40</td>
    <td>51.40</td>
    <td>49.60</td>
    <td>49.39</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>55.00</td>
    <td>63.40</td>
    <td>57.37</td>
    <td>57.27</td>
    <td>59.60</td>
    <td>61.00</td>
    <td>63.05</td>
    <td>53.00</td>
    <td>61.40</td>
    <td>60.80</td>
    <td>67.60</td>
    <td>61.41</td>
  </tr>
  <tr>
    <td rowspan="2">CSQA</td>
    <td>EvoPrompt</td>
    <td>29.91</td>
    <td>26.75</td>
    <td>23.55</td>
    <td>32.67</td>
    <td>25.63</td>
    <td>31.70</td>
    <td>26.35</td>
    <td>30.07</td>
    <td>20.32</td>
    <td>29.25</td>
    <td>23.35</td>
    <td>27.23</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>26.16</td>
    <td>24.35</td>
    <td>36.50</td>
    <td>22.06</td>
    <td>27.83</td>
    <td>26.30</td>
    <td>23.00</td>
    <td>28.69</td>
    <td>26.93</td>
    <td>25.80</td>
    <td>24.15</td>
    <td>26.52</td>
  </tr>
</table>
Table 8: Performance comparison between MutantPrompt technique with EvoPrompt for the generated optimal prompt (\( p_{final} \)) on Mistral-3(7B) model. On the right, we report the average performance across languages for the tasks. Best performances are marked in Bold and underlined.

<table>
  <tr>
    <th>Task</th>
    <th>Method</th>
    <th>Token<sub>instr</sub></th>
    <th>p<sub>new</sub></th>
  </tr>
  <tr>
    <td>IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>1185</td>
    <td>100</td>
  </tr>
  <tr>
    <td></td>
    <td>MutantPrompt</td>
    <td>275</td>
    <td>129</td>
  </tr>
  <tr>
    <td>IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>1138</td>
    <td>100</td>
  </tr>
  <tr>
    <td></td>
    <td>MutantPrompt</td>
    <td>296</td>
    <td>130</td>
  </tr>
  <tr>
    <td>IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>1091</td>
    <td>100</td>
  </tr>
  <tr>
    <td></td>
    <td>MutantPrompt</td>
    <td>278</td>
    <td>116</td>
  </tr>
  <tr>
    <td>IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>1234</td>
    <td>100</td>
  </tr>
  <tr>
    <td></td>
    <td>MutantPrompt</td>
    <td>345</td>
    <td>114</td>
  </tr>
  <tr>
    <td>CSQA</td>
    <td>EvoPrompt</td>
    <td>1406</td>
    <td>100</td>
  </tr>
  <tr>
    <td></td>
    <td>MutantPrompt</td>
    <td>316</td>
    <td>142</td>
  </tr>
</table>
Table 9: Here, we compare the token length of the instruction prompts (\( Token_{instr} \)) between EvoPrompt and MutantPrompt for different tasks using Gemma-2(2B). We also compare the average number of new prompts (\( p_{new} \)) explored by both techniques.
Task: CSQA

Seed prompt: "The paragraph below includes a gap indicated by <MASK>. From the given options, choose the word that accurately fills the gap. Answer with only the correct word:
Paragraph (paragraph)
Options: (option_a), (option_b), (option_c), (option_d)"

Generated prompt: "In the given text, there is a word signified by '<MASK>', and the objective is to select the appropriate word from the provided options that, when inserted, will result in a sentence that is both grammatically correct and logically sensible. Please identify the word that fits seamlessly into the sentence without any additional information:

Example:
Text: The child <MASK> the toy.
Options: broke, played, laughed, held
Correct answer: played

In this case, you are required to find the word that, when inserted into the text, creates a grammatically sound and logically consistent sentence. Provide the suitable word without any extra details:

Example:
Text: The bird <MASK> the nest.
Options: built, sang, flew, hatched
Correct answer: built
Paragraph: (paragraph)
Options: (option_a), (option_b), (option_c), (option_d)"

Task: IndicXNLi

Seed prompt: "Identify the relation('neutral'/contradiction'/entailment') between Premise and Hypothesis. Output only the correct relation.
Premise: (premise)
Hypothesis: (hypothesis)"

Generated prompt: "Determine the nature (Neutral, 'Contradiction', or 'Entailment') of the relationship that exists between the presented statement (Premise) and the following statement (Hypothesis). Your response should clearly illustrate the connection between the two statements.

For example, if the Premise is 'The cat is on the mat' and the Hypothesis is 'The cat is not on the mat', the relationship is 'Contradiction'.

Please provide the exact relationship for the given Premise and Hypothesis.
Premise: (premise)
Hypothesis: (hypothesis)"

Task: IndicXParaphrase

Seed prompt: "Identify if the Sentence2 is a paraphrase of Sentence1. Output only either 'yes' or 'no'.
Sentence1: (sentence1)
Sentence2: (sentence2)"

Generated prompt: "Analyze each pair of sentences and determine if Sentence2 is a paraphrase of Sentence1. For each pair, you should provide a clear 'yes' or 'no' response indicating whether Sentence2 is a paraphrase of Sentence1. Your response should only include the correct answer.
Sentence1: (sentence1)
Sentence2: (sentence2)"

Task: IndicCOPA

Seed prompt: "Given the Premise, Question and two choices(Choice1/Choice2), return the correct choice that answer the question for the premise. Output only 'Choice1' or 'Choice2', no need to output the whole sentence.
Premise: (premise)
Choice1: (choice1)
Choice2: (choice2)
Question: (question)"

Generated prompt: "Given a scenario described by a premise and a question about either 'cause' or 'effect', and presented with two possible answers ('Choice1'/'Choice2'), kindly determine the response that accurately answers the question within the context of the premise. Simply provide 'Choice1' or 'Choice2' as your answer, without including the question or the premise in your response.
Premise: (premise)
Choice1: (choice1)
Choice2: (choice2)
Question: (question)"

Task: IndicSentiment

Seed prompt: "Given the Sentence, predict the correct sentiment('positive'/negative') of the sentence. Output only the correct sentiment class.
Sentence: (sentence)"

Generated prompt: "Determine the sentiment of the given statement, categorizing it as 'positive' or 'negative'. Kindly provide the sentiment category as your response.

Example:
Statement: I absolutely love this movie!
Sentiment: positive
Sentence: (sentence)"
<table>
  <tr>
    <th>Task</th>
    <th>Method</th>
    <th>\( Token_{instr} \)</th>
    <th>\( p_{new} \)</th>
  </tr>
  <tr>
    <td rowspan="2">IndicSentiment</td>
    <td>EvoPrompt</td>
    <td>1187</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>288</td>
    <td>129</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXNLI</td>
    <td>EvoPrompt</td>
    <td>1277</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>303</td>
    <td>130</td>
  </tr>
  <tr>
    <td rowspan="2">IndicXParaphrase</td>
    <td>EvoPrompt</td>
    <td>1217</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>299</td>
    <td>116</td>
  </tr>
  <tr>
    <td rowspan="2">IndicCOPA</td>
    <td>EvoPrompt</td>
    <td>1334</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>370</td>
    <td>114</td>
  </tr>
  <tr>
    <td rowspan="2">CSQA</td>
    <td>EvoPrompt</td>
    <td>1298</td>
    <td>100</td>
  </tr>
  <tr>
    <td>MutantPrompt</td>
    <td>322</td>
    <td>142</td>
  </tr>
</table>

Table 10: Here, we compare the token length of the instruction prompts (\( Token_{instr} \)) between **EvoPrompt** and **MutantPrompt** for different tasks using Mistral-3(7B). We also compare the average number of new prompts (\( p_{new} \)) explored by both techniques.

Reviewer paper summaries: This paper proposes an automatic prompt learning method called MutantPrompt for low-resource Indic language tasks. MutantPrompt leverages the Thompson Sampling algorithm with beta distribution prior for prompt selection, where prompts are treated as bandits and their metrics on a large score set are the rewards. Each time a prompt is selected via the algorithm from the pool and evaluated in a only small batch from the large score set. Meanwhile, new prompts are generated randomly by mutating from the prompt with the best estimated reward so far. The introduction of multi-armed bandit into prompt learning avoids evaluation over the full large score set for each prompt and improves the search efficiency of the learning process.

Experimental results in five different types of Indic language tasks with Mistral-3-7B, Llama-3.2-3B, and Gemma-2-2B show consistent improvements of learned prompts over base prompts. MutantPrompt also demonstrates steady improvements over the base prompt on other 11 low-resource languages in CSQA with Llama-3.2-3B.

Moreover, MutantPrompt outperforms the baseline of EvoPrompt in Llama-3.2-3B on the same set of Indic language tasks. The incorporation of MutantPrompt in EvoPrompt leads to more explored prompts with the same budget, proving its cost-effectiveness.

This paper addresses the prompt optimization problem under low-resource scenarios. This paper proposes MutantPrompt: a framework that leverages multi-armed bandit algorithms to efficiently identify optimal prompts tailored to low-resource languages. By framing prompt selection as an exploration-exploitation problem under a fixed computational budget, the framework dynamically balances exploring new prompts with exploiting known high-performing ones.
Summary of strengths: 1. This paper is well-written and easy to follow, with only minor clarity problems.
2. The combination of prompt learning and multi-armed bandit algorithm is novel and interesting.
3. The superiority of the learned prompt in low-resource languages can lead to better downstream applications in those languages, bringing impact to multiple communities.
4. Extensive experiments have been conducted to demonstrate its practical value, superiority over baseline, and cost-effectiveness.

1. This paper addresses an interesting problem, namely prompt optimization, which is crucial in real application.

2. This paper proposes MutantPrompt, a novel algorithm to try to figure out prompt optimization in low-resource scenarios. Prompts optimized by MutantPrompt yield better performance than the original ones.

3. This paper conduct extensive experiments on numerous datasets and languages across different language models.
Summary of weaknesses: The quality of the paper can benefit from the theoretical guarantees of the Thompson Sampling algorithm. Also, more baselines are encouraged to be included in empirical comparisons, but not necessary.

1. **Writing**: This paper is quite hard to follow, especially on section 3.

2. **Lack of baselines**: As far as I know, there're many previous research focusing on prompt optimization but this paper does not compare MutantPrompt with those baselines, which makes it less convincing for the superiority of MutantPrompt.

3. **Lack of ablation studies**: This paper introduces many hyper-parameters but fails to conduct enough ablation studies to discuss how these hyper-params affect the final performance of MutantPrompt.

4. **Unfair comparison**: This paper compares MutantPrompt with EvoPrompt on their performance and token costs but fails to consider the comparison from the scope of additional costs (i.e., the additional costs caused by generating $p_{final}$ from $p_{seed}$). This might lead to an unfair comparison.
Comments, suggestions and typos: * Some missing citations in the related works: [1][2][3].
* line 17 of Algorithm 1 should be $\mathcal{P} \gets \mathcal{P} \cup \{p_{new}\}$.
* line 180: the fluency of the sentence here can be improved.
* Question: I am wondering what LLM is used for the prompt refinement operation in line 16 of Algorithm 1. If GPT-4, which version?

### References

[1]: Sun, Tianxiang, et al. "BBTv2: Towards a gradient-free future with large language models." EMNLP 2022.

[2]: Pan, Rui, et al. "Plum: Prompt learning using metaheuristic." Findings of the Association for Computational Linguistics ACL 2024.

[3]: Diao, Shizhe, et al. "Active prompting with chain-of-thought for large language models." ACL 2024.

See weakness.
Overall assessment scores: 4

3