DATA-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning (2024)

Abstract: Recently, multi-task instruction tuning has been utilized to improve sentence representation learning (SRL). It enables SRL models to generate task-specific representations with the guidance of task instruction, thus exhibiting strong generalization ability on unseen tasks. However, these methods mostly neglect the potential interference problems across different tasks and instances, which may affect the training of the model. To address this issue, we propose a data curriculum method, namely Data-CUBE, that arranges the order of all the multi-task data for training, to minimize the interference risks from two aspects. At the task level, we aim to find the optimal task order to minimize the total cross-task interference risk and formulate this problem as the traveling salesman problem, which is further solved by a specially designed simulated annealing algorithm. At the instance level, we propose a measurement method to quantify the difficulty of all instances per task, and then arrange instances in an easyto-difficult order for training. Experimental results show that our approach can boost the performance of state-of-the-art methods. Our code and data will be publicly released.

heading: 1 Introduction
text: Sentence representation learning (SRL) (Reimers and Gurevych, 2019; Gao et al., 2021) is a fundamental task in the NLP field, which focuses on encoding the semantic information of sentences into low-dimensional vectors. Typically, existing work (Karpukhin et al., 2020; Zhou et al., 2022) collects a set of sentence pairs (or augmented in an unsupervised way), and then learns the model parameters by maximizing and minimizing the similarity scores of relevant and irrelevant sentences, respectively. Previous SRL methods based on advanced language models and learning objectives (Reimers
*Equal contribution. †Corresponding author.
and Gurevych, 2019; Ni et al., 2022a), are capable of producing high-quality representations that perform well on various downstream tasks.
Despite the success, recent studies (Neelakantan et al., 2022) have revealed that it is challenging to directly transfer the learned sentence representations into new tasks, even causing significant performance degradation. To alleviate this problem, instruction tuning (Wei et al., 2022; Wang et al., 2022b) has been applied to sentence representation learning, which collects a diverse set of sentencepair datasets with task-specific natural language instructions (Su et al., 2023; Xiao et al., 2023). Before training, each of the collected datasets is generally divided into multiple mini-batches at random (Su et al., 2023), and the SRL model will be trained over the mini-batches of all datasets in a random order. After multi-task training on the dataset collection, the model would become capable of generating task-specific sentence representations with the guidance of the task instruction, exhibiting improved generalization ability on unseen tasks.
However, as the collected datasets vary in data distributions, a random order for instruction data
13748
scheduling would lead to potential cross-task interference risk for model optimization. As depicted in Figure 1 (left), when the neighboring mini-batches are from very different tasks, the successive learning of them would lead to conflict in the optimization objective, affecting the final performance of both tasks. In addition, the instances for a given task might be with varied difficulty levels. Randomly assigning them into mini-batches may result in potential cross-instance interference, which is also likely to cause the performance degradation. As shown in Figure 1 (right), the competitive model INSTRUCTOR (Su et al., 2023) struggles with distinguishing more than 80% positive and negative examples in almost half of the training datasets, indicating the severe underfitting problem.
To address these issues, in this paper, we propose DATA-CUBE, a Data CUrriculum method for instruction-Based sentencE representation learning. The core idea of our approach is to design a proper data curriculum that arranges the orders of all tasks and instances for minimizing the potential interference risks. Concretely, for the crosstask interference, we focus on finding the optimal task order where all the neighboring two tasks are as similar as possible, to minimize the total interference risks derived from task divergence. We formulate this task as the the travelling salesman problem (TSP) (Hoffman et al., 2013): all the tasks are regarded as the nodes in a fully connected graph with task similarity as the edge weights, and the optimal order search problem is essentially to find the longest route that visits all nodes. To solve the TSP, we employ the widely-studied simulated annealing algorithm to efficiently find its suboptimal solution. For the cross-instance interference, we measure the discriminability of positive and negative, as the estimated difficulty for sorting all the instances. Then, we divide them into easy-to-difficult minibatches for training, to minimize the interference risks caused by varied instance difficulty.
To integrate the two strategies, we first find the optimal task order with the simulated annealing algorithm, then sort the instances within each task according to their difficulty to obtain the instance order for mini-batch arrangement, and finally divide the sorted instances from all the tasks into a sequence of mini-batches for training SRL models.
Our contributions are summarized as follows:
(1) To our knowledge, the proposed Data-CUBE is the first attempt of data curriculum in instruction-
based SRL. It is a model- and data-agnostic approach for improving the training of SRL models.
(2) We reveal the interference problem in training instruction-based SRL, and propose to address cross-task interference by formulating it as a TSP and address cross-instance interference by employing an easy-to-difficult data curriculum.
(3) Extensive experiments on downstream tasks show the effectiveness of our approach, outperforming a number of competitive SRL models.

heading: 2 Related Work
text: Sentence Representation Learning. A robust sentence representation plays a pivotal role in diverse downstream tasks. Previously, most sentence representation models concentrate on a singular task or domain, weak in transferring to other downstream tasks without further fine-tuning. For instance, SimCSE (Gao et al., 2021), SBERT (Reimers and Gurevych, 2019), and DCLR (Zhou et al., 2022) are trained to address sentence similarity and classification tasks, while models like DPR (Karpukhin et al., 2020), Contriever (Izacard et al., 2022a), Master (Zhou et al., 2023), and GTR (Ni et al., 2022b) are applied to information retrieval. In response to this challenge, recent efforts have emerged to develop instruction-based sentence representation models through multi-task contrastive learning. Exemplars include INSTRUCTOR (Su et al., 2023), BGE (Xiao et al., 2023), and GTE (Li et al., 2023), which aim to enhance the adaptability and generalization capabilities of sentence representations across diverse tasks and domains. Existing studies primarily focus on aspects such as the training objective, model architecture, or training scale, while paying limited attention to the challenges posed by interference during the multi-task training process.
Instruction Tuning. Instruction tuning (Ouyang et al., 2022; Zhao et al., 2023) involves supervised fine-tuning pre-trained language models by integrating well-formatted natural language instructions into the input. This process is closely connected to multi-task learning and is believed to enhance the generalization capability of language models across a range of tasks (Wei et al., 2022). Previous studies have demonstrated that increasing the number and diversity of tasks associated with instructions can improve performance. Considering the effectiveness of instruction tuning, it has
been applied to various NLP tasks, such as sentence representation learning (Su et al., 2023; Xiao et al., 2023). However, with the increasing diversity of tasks, there is a potential for interference across different tasks, which may lead to performance degradation (Mueller et al., 2022). Hence, we propose to leverage data curriculum to alleviate the interference in the multi-task instruction tuning.
Traveling Salesman Problem (TSP). Traveling salesman problem is a classic combinatorial optimization problem in computer science and operations research. It is to find the shortest route for a salesman to visit a given set of cities exactly once and return to the start (Hoffman et al., 2013; Cheikhrouhou and Khoufi, 2021). Recognized as an NP-hard problem, TSP is costly to solve for large numbers of cities. Consequently, numerous heuristic methods have been developed to find near-optimal solutions efficiently (Helsgaun, 2006; Matai et al., 2010). A notable method is Simulated Annealing (SA) (Bertsimas and Tsitsiklis, 1993), an algorithm inspired by metallurgy’s annealing process. SA is effectively used in various combinatorial problems like TSP, Job Shop Scheduling Problem (Chakraborty and Bhowmik, 2015), and Graph Coloring Problem (Pal et al., 2012).

heading: 3 Preliminary

heading: 3.1 Task Definition
text: Sentence representation learning (SRL) is to train a capable text encoder that can map a sentence into a latent vector for downstream tasks. To enhance the generalization ability on unseen tasks of the SRL model, instruction based SRL models (Su et al., 2023) take as input the sentence s with a natural language instruction I , to obtain the task-aware sentence representation v. To train the SRL model, we are given m instruction formatted sentence-pair datasets D = {di}mi=1, where di denotes the i-th dataset and corresponds to the task oi. Each dataset typically consists of n queries {qj}nj=1 and their relevant sentences {s (+) j }nj=1 and irrelevant sentences {s(−)j }nj=1, with specific instructions ⟨I(q), I(+), I(−)⟩ for the three text types.
During training, the model follows a certain task order O = {oi} and instance order, typically random orders, to learn the model parameters. However, random data training would potentially lead to potential learning interference issues as discussed in Section 1. In this work, we aim to devise a
data curriculum approach to improve the multi-task training for SRL, to reduce the interference risk.

heading: 3.2 Travelling Salesman Problem
text: Considering the interference problem across the m datasets, we first estimate the mutual interference risks between every two tasks r(i, j), and then find the optimal order for all the tasks O = {oi}mi=1, to minimize the accumulated interference risk as:
argmin
m−1∑
i=1
r(oi, oi+1) + r(om, o1) (1)
where oi is the corresponding task of the dataset di from D. Such an optimal order search problem can be converted as the traveling salesman problem (TSP) that finds the shortest route to visit all cities (i.e., tasks) exactly once.
As TSP is proved to be an NP-hard problem, heuristic algorithms (Helsgaun, 2006; Matai et al., 2010) have been widely studied to find suboptimal solutions in a reasonable time. Simulated annealing (SA) (Bertsimas and Tsitsiklis, 1993) is a commonly used algorithm for TSP, and its basic idea is to start with an initial solution and then search by randomly perturbing the solution. In each iteration, the algorithm evaluates the quality of the new solution by computing the change in the objective function. If the new solution is better, it will replace the current one. Otherwise, the update would occur according to a probability calculated based on the temperature and the change in the objective function. As the iterations progress, the temperature decreases, gradually reducing the likelihood of accepting worse solutions and guiding the algorithm towards converging.

heading: 4 Approach
text: In this section, we present the proposed DATACUBE, a Data CUrriculum method for instructionBased sentencE representation learning. Following previous work, we develop the approach based on a multi-task contrastive learning framework, and while introduce novel data curriculum methods for SRL, considering both task-level and instance-level arrangement, which can significantly reduce the learning interference issue.

heading: 4.1 Multi-task Contrastive Learning
text: We employ multi-task contrastive learning to train a pre-trained language model (e.g., T5 (Raffel et al., 2020)) for producing sentence representations. In
general, it maximizes the similarity of positive pairs ⟨q, s(+)⟩ and minimizes the one of negative pairs ⟨q, s(−)⟩, based on specific task instructions ⟨I(q), I(+), I(−)⟩. Concretely, we first preprocess the collected datasets into multiple mini-batches with specific instructions and then optimize the model parameters via a multi-task learning loss.
For each dataset, we concatenate its contained queries, positive and negative sentences with corresponding instructions, to compose new instances:
q̃ = [I(q); q], s̃(+) = [I(+); s(+)], s̃(−) = [I(−); s(−)], (2)
where the instruction contains the description that specifies the task, e.g., “Represent the example for the following task: Given a scientific question, generate a correct answer to it”. Next, we perform the mini-batch splitting, and guarantee that all the in-batch instances come from the same task. Such a way avoids the possible cross-task interference when using in-batch negatives for contrastive learning. Thus, we leverage the following loss function:
L = m∑
i=1
∑
B∈di
|B|∑
j=1
e sim(vq̃j ,v (+) s̃j )/τ
∑|B| k=1 e sim(vq̃j ,vs̃k )/τ (3)
where B = {⟨q̃j , s̃(+)j , s̃ (−) j ⟩} |B| j=1 denotes the minibatch of |B| instances from dataset di, vq̃j and v (+) s̃j
refer to the representations of the j-th query q̃j and positive sentence s̃(+)j respectively, m denotes the number of datasets and τ denotes the temperature, and sim(·, ·) is the cosine similarity function.
Here, we adopt a similar setting in loss function (Eq. (3)) as previous study (Su et al., 2023), while our focus is to design a suitable data curriculum approach for scheduling the mini-batches from all the task datasets. In what follows, we will introduce the proposed task-level (Section 4.2) and instancelevel (Section 4.3) curriculum methods in detail.

heading: 4.2 Task-level Curriculum Arrangement: From Similar to Different
text: As the divergence of data distributions between neighboring tasks may affect the learning of both (Ding et al., 2023), we expect that the training order can be a “smooth” transition across tasks, to minimize the accumulated cross-task interference risk. Thus, we estimate the cross-task interference risk based on task similarity, and then search the optimal order by the simulated annealing algorithm.

heading: 4.2.1 Cross-task Interference Risk Estimation
text: To estimate the cross-task interference risk, we adopt the similarity of text representations for measuring the divergence in data distribution. Con-
cretely, we randomly sample nt queries per dataset to compose a representative subset, then compute the mean representation as the task representation:
v(t) = 1
nt
nt∑
j=1
vq̃j . (4)
Here, we use a pre-learned model (i.e., Instructor (Su et al., 2023)) to produce the query representation. Based on it, the task similarity can be measured using the cosine similarity of task representations. As similar tasks typically have lower interference risk (Mueller et al., 2022), we can roughly estimate the cross-task interference risk as:
r(i, j) ∝ −sim(v(t)i , v (t) j ), (5)
where sim(v(t)i , v (t) j ) is the cosine similarity between the representations of the i-th and j-th tasks. With these estimated interference scores, we next study how to schedule different tasks to reduce the entire risk across the data curriculum.

heading: 4.2.2 Optimal Order Search
text: As discussed in Section 3.2, we can formulate optimal order search as TSP over in a fully connected undirected graph, in which tasks are considered as nodes and the estimated interference risk (Eq. 5) between two linked tasks are considered as edge weight. In this way, our goal becomes how to find the shortest route that visits each node exactly once, with the objective function as Eq. 1.
According to the negative correlation between interference risk and task similarity as Eq. 5, the risk minimization objective is equivalent to maximizing the sum of neighboring task similarity:
argmax m∑
i=1
sim(v(t)i , v (t) i+1)+ sim(v (t) m , v (t) 1 ). (6)
Therefore, our goal is to find the most smooth transition path for all the tasks, to avoid the drastic distribution shift of the neighboring tasks.
To solve TSP, we adopt the simulated annealing algorithm to find a suboptimal solution within a reasonable amount of time. Specially, simulated annealing iteratively perturbs the current solution to explore the solution space, and accepts the new solution based on the objective in Eq. 6 and a gradually decaying temperature τs. Concretely, we first initialize a task orderO′ by random shuffling. Next, we repeat the perturb-then-check process until convergence. In each iteration, we randomly choose
a pair of tasks in the current order O′ , swap their positions to obtain the new order Õ′ , and check whether the total neighboring task similarity will increase. If increased, the new order will replace the current one. Otherwise, the new order will be accepted in a probability as:
p(O′ , Õ′ , τs) = exp(− ∆(O′ , Õ′)
τs ) (7)
where ∆(O′ , Õ′) denotes the difference of the total neighboring task similarity between the current and new orders using Eq. 6. Such a way prevents the solution from being stuck at a local minimum, and incorporating “∆” also reduces the likelihood of accepting worse solutions. it also reduces the instability close to the converged suboptimal point.

heading: 4.3 Instance-level Curriculum Arrangement: From Easy to Difficult
text: In addition to the task-level curriculum, we also devise the instance-level curriculum, to reduce the cross-instance interference risk. The basic idea is to first estimate the varying difficulty of instances and then reorder the instances in each task from easy to difficult. Next, we detail the two steps.

heading: 4.3.1 Instance Difficulty Estimation
text: As the tasks for SRL mainly focus on distinguishing the relevant sentence s̃(+) and irrelevant sentence s̃(−) according to the query q̃, we leverage the discriminability of s̃(+) and s̃(−) to measure the instance difficulty.
Specially, the positive and negative sentences of easy instances would be clearly distinguished by an SRL model trained on the data, while the ones of difficult instances would pose more challenges for successful discrimination. We employ a pre-learned model (i.e., Instructor) to encode the representations of the positive and negative pairs, then estimate the instance difficulty by computing the similarity difference as:
ϕ(q̃, s̃(+), s̃(−)) = sim(vq̃, vs̃(+))− sim(vq̃, vs̃(−)). (8) The smaller the difference is, the more likely the model struggles with distinguishing the positive and negative, which indicates a difficult instance.

heading: 4.3.2 Instance Curriculum Arrangement
text: According to the difficulty measurement in Eq. 8, we can assign the estimated scores to the instances for a given task. Then, we sort all the instances
in each task descendingly, and further divide them into multiple mini-batches B. In this way, we can obtain the easy-to-difficult mini-batches per task. Compared with randomly sampling, our method can alleviate the interference caused by varying instance difficulty within each mini-batch and the difficulty divergence in neighboring mini-batches.
In addition, we find that too difficult instances may not always be useful, as they could potentially introduce noise into data. Therefore, following existing work (Zhou et al., 2022), we design a binary mask αi using a threshold δ to reduce its influence as:
αi =
{ 0, ϕ(q̃i, s̃ (+) i , s̃ (−) i ) ≥ δ
1, ϕ(q̃i, s̃ (+) i , s̃ (−) i ) < δ
. (9)
Then, we apply the mask to the contrastive loss of each instance in Eq. 3. This ensures that noisy instances cannot be directly learned but serve as in-batch negatives for other instances.

heading: 5 Experiments
text: In this section, we train with DATA-CUBE based on INSTRUCTOR and conduct evaluations on four task categories within MTEB (Muennighoff et al., 2023), encompassing a total of 28 downstream tasks. Furthermore, we continue to investigate the effectiveness and robustness of DATA-CUBE. For detailed settings, see Appendix B and C.

heading: 5.1 Main Results
text: We present the main experiment results in Table 1. Based on the results, it is evident that instructionbased sentence representation models generally perform better than no-instruction models, although some no-instruction models are much larger than instruction-based models in terms of scale (e.g., Sentence-T5 XXL has 11B parameters while the largest version of BGE is about 300M). A potential reason is that instruction tuning enhances the models’ understanding of tasks by integrating natural language instructions. This integration assists
the models in encoding sentences into task-aware representations, thereby providing a significant advantage in downstream tasks.
Among all the compared models, our method achieves the highest average performance across STS tasks. Although it may not always rank first in certain tasks like STS-12, STS-13, STS-16, STS22, and STSBenchmark, it maintains competitive results. Notably, models achieving the best results in these tasks tend to excel in only one task but underperform in others. In contrast, our approach consistently demonstrates effectiveness across all tasks. When compared to our backbone model (i.e., INSTRUCTOR), our method significantly improves the performance on all STS tasks. In addition, we also evaluate our method on other task categories (e.g., Reranking, Clustering, and PairClassifiacation). As Table 2 shows, our method achieves average performance gains on these diverse task categories. This implies that our approach plays a significant role in reducing the interference among
tasks, which enhances not only the performance in specific tasks but also the overall versatility.
It is noteworthy that both the volume of training data and the mini-batch size utilized in our approach are considerably smaller compared to other robust instruction-based sentence representation models. Specifically, our model is trained with only 1 million sentence pairs and a batch size of 64, in stark contrast to models like BGE which use 300 million sentence pairs and a batch size of 32768. This indicates that our approach has the potential to enhance the model’s ability to learn more effectively from multi-task data in the context of data interference and achieve comparable or even superior performance despite limited data and computational resources.

heading: 5.2 Further Analysis
text: Next, we continue to investigate the effectiveness and robustness of Data-Cube. This involves conducting ablation studies on the two-level curriculums, assessing the impact of iterations in Simulated Annealing, analyzing the influence of minibatch size during training, and thoroughly exam-
ining the training convergence. To gain a better understanding of the variations between different settings, we carefully select several tasks in STS, such as BIOSSES (Sogancioglu et al., 2017), STS12, 14, and 22 (Agirre et al., 2012, 2014; Chen et al., 2022), where our method demonstrates more noticeable improvements.
Ablations of Two-level Curriculum. To explore the influence of task-level curriculum, we exclusively implement the instance-level curriculum to reorganize the training data, sorting instances within each task by difficulty using Eq. 8 while randomly shuffling task orders. Conversely, we employ the task orders generated by the task-level curriculum but randomly shuffle the instances within each task to assess the effectiveness of the instancelevel curriculum. Furthermore, we train our backbone model without extra operations as the vanilla baseline. As Table 3 shows, both two-level curriculums contribute to alleviating the interference of multi-task data and enhancing the performance of the sentence representation model.
Different Task-level Curriculum Methods. In the task-level curriculum, we calculate the similarity between tasks by meaning the representations of the sampled queries per task using Eq. 4 and Eq. 5 and utilize the SA algorithm to search the suboptimal results. To validate the effectiveness of our methods, we conduct experiments using other variations. Concretely, we change the way of task-level similarity measurement and optimization objectives. The detailed settings are as follows: (1) Mean-Q-ArgMax (MQMax, Ours): We use the task vector with Eq. 4 and estimate the crosstask interference risk using Eq. 5. When arranging the task order, we maximize the total task similarity. (2) Tf-idf-Argmax (TIMax): We use the Tf-idf (Salton and Buckley, 1988) vector of each
task and calculate the cosine similarity to estimate the cross-task interference risk. When arranging the task order, we maximize the total task similarity. (3) Mean-Q-Argmin (MQMin): We use the task vector with Eq. 4 and estimate the cross-task interference risk using Eq. 5. When arranging the task order, we minimize the total task similarity. (4) Mean-Q-Random (MQRand): We use the task vector with Eq. 4 and estimate the cross-task interference risk using Eq. 5. When arranging the task order, we shuffle the task order at random. As shown in Table 6, our approach can also perform better than all the other variations, indicating the effectiveness of using mean query representation for measuring the task similarity and maximizing the total task similarity.
Different Instance-level Curriculum Methods. In the instance-level curriculum, we leverage the Eq. 8 for instance difficulty estimation and reorder instances per task from easy to difficult along the thought of curriculum learning. To validate the effectiveness of our ordering method, we rearrange the order of instances to create two variations of our methods for comparison. The detailed settings are as follows: (1) Easy-to-Difficult (E2D, Ours): We arrange instances per task in an easy-to-difficult order. (2) Difficult-to-Easy (D2E): We arrange instances per task in a difficult-to-easy order. (3) Random: We randomly shuffle instances per task using seed 42. As shown in Table 7, our approach consistently outperforms all the variations, indicating the effectiveness of our designed Easy-toDifficult order.
Iterations of Simulated Annealing. In the tasklevel curriculum, we employ Simulated Annealing
algorithm that gradually obtains an approximate solution as the iterations progress. In broad terms, a higher number of iterations typically leads to a more optimal solution. Consequently, we undertook experiments on task orders generated through varying iteration counts to illustrate the adequacy of the specific iteration count we employed. We opt to compare the results of using task orders generated by SA at 800K, 2M (Ours), 3M, and 5M iterations. As Table 3 shows, these performances are comparable, indicating that the chosen iteration step of 2M is adequate for alleviating interference.
Size of Mini-batch. During multi-task contrastive learning, we leverage the in-batch negatives to extend the positive-negative ratio, which has been shown to enhance the uniformity of the sentence representation model and thereby improve overall performance (Karpukhin et al., 2020) In the instance-level curriculum, we propose to alleviate the interference between instances with different difficulty. However, when using a larger mini-batch, the variability in difficulty within the batch increases, seemingly conflicting with our curriculum design. To address this, we conduct experiments to evaluate how the instance-level curriculum affects model performance across different mini-batch sizes. As shown in Table 5, while larger mini-batches generally lead to better performance, our Data-CUBE results remain comparable even with smaller batch sizes. Notably, the model performs better with a batch size of 16 compared to 32, indicating that our approach is particularly effective in resource-constrained scenarios.
Analysis of Training Convergence. To validate the efficacy of our approach in mitigating data interference, we assess the model’s performance on STS tasks throughout the training process (See Figure 3). In comparison to directly continuing training, employing Data-CUBE leads to significantly improved performance in fewer training steps. Furthermore, we compare the ratio of underfitting instances within tasks before and after training with Data-CUBE (See Figure 4). It is evident that the ratio of underfitting instances consistently decreases across various tasks. It indicates the effectiveness of our method in alleviating interference for multitask contrastive learning and results in a more powerful and robust sentence representation model.

heading: 6 Conclusion
text: In this work, we proposed Data-CUBE, a data curriculum method for multi-task instruction based sentence representation learning. The key idea of our approach is to reduce the cross-task and crossinstance interference risks by using a more suitable data curriculum of instances for training. To achieve this, we employed a simulated annealing algorithm to find the optimal task order to minimize the cross-task interference, and assigned all instances per task into easy-to-difficult mini-batches to reduce the cross-instance interference. Experimental results on MTEB sentence representation evaluation tasks have shown that our approach can boost the performance of state-of-the-art baselines.
Limitations
Although Data-CUBE is a model-agnostic and dataagnostic approach to enhancing instruction-based sentence representation models, due to the lack of experimental details, we have yet not employed it on the state-of-the-art models. Actually, the improvement of integrating our approach on Instructor is able to indicate its effectiveness. We will conduct more experiments using our approach on other strong baselines in the future. Furthermore, due to our limited computational resources, we have not explored our method on larger models, e.g., 3B and 7B LLaMA. We also leave it in our future work, and explore more efficient and effective data curriculum methodology for large-scale datasets and models.

heading: Acknowledgement
text: This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. 4222027 and L233008. Xin Zhao is the corresponding author.

heading: A The process of training with DATA-CUBE
text: Algorithm 1: Training with Data-CUBE Input :Original Data D = {di}, the
temperature τ , cooling rate α, and max iterations N of SA, and the backbone model
Output :Instruction-tuned model M ’
1 // Task Curriculum 2 Initialize a random order O′ = {oi} 3 for i in range(N) do 4 Õ′ ← Swap a random pair of tasks inO′ 5 Calculate ∆(O,O′) using Eq. 6 6 if ∆(O′, Õ′) > 0 then 7 O′ ← Õ′ 8 else if rand() < p(O′ , Õ′ , τs) then 9 O′ ← Õ′
10 end 11 τs ← α · τs 12 end
13 // Instance Curriculum 14 for di in D do 15 Calculate ϕ(q̃, s̃(+), s̃(−)) of each instance in di using Eq. 8 16 Arrange di to d′i in descending order based on ϕ 17 end
18 // Combine Two-level Curriculum 19 Initialize an empty D′ 20 for oi in O do 21 Choose dataset d′i corresponding to oi 22 Select the first batch B of d′i 23 D′.append(B) 24 d′i.remove(B) 25 end
26 // Multi-task Contrastive training 27 Use D′ to train the backbone model with
Eq. 3 28 Get the final instruction-tuned model

heading: B Training Settings
text: Training Dataset. We opt for the multi-task sentence-pair dataset MEDI (Su et al., 2023), comprising 330 sub-datasets spanning various tasks and domains, with a total of 1.4 million instances for
training. Each sub-task is accompanied by corresponding natural language instructions elucidating its detailed goal or description.
Training Details To arrange the task-level curriculum, we utilize the simulated annealing algorithm and early stop at approximately 2 million steps to obtain a suboptimal task order O. To rearrange the instance-level curriculum, we calculate the difficulty of instances (ϕ in Eq. 8) in advance and sort all the instances in each task by descending. After pre-reassigning the training data following Data-Cube, we start training from the checkpoint of INSTRUCTOR-large (335M parameters) (Su et al., 2023) with a batch size of 64. We use a softmax temperature τ of 0.01 and optimize the model with the AdamW optimizer. The warmup ratio is set to 0.1 and the learning rate is 2× 10−5.

heading: C Evaluation Settings
text: Evaluation Dataset We conduct evaluations on four task categories within the MTEB dataset (Muennighoff et al., 2023), encompassing a total of 28 downstream tasks. For STS tasks, we choose 10 datasets (e.g., BIOSSES (Sogancioglu et al., 2017), STS12-77 (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017), and SICK-Relatedness (Marelli et al., 2014)), to assess the performance. We employ Spearman’s correlation of the English test set as the evaluation metric. Reranking tasks include AskUbuntuDupQuestions (Lei et al., 2016), MindSmall (Wu et al., 2020), SciDocsRR (Cohan et al., 2020), and StackOverflowDupQuestions (Liu et al., 2018). Mean Average Precision (MAP) of the test set is utilized to measure performance. Clustering tasks encompass 11 datasets, such as ArxivClusteringS2S, ArxivClusteringP2P, BiorxivClusteringS2S, BiorxivClusteringP2P, MedrxivClusteringP2P, MedrxivClusteringS2S (Muennighoff et al., 2023), RedditClustering, RedditClusteringP2P, StackExchangeClustering, StackExchangeClusteringP2P (Geigle et al., 2021), and TwentyNewsgroupsClustering*. In these clustering tasks, we use v-measure of the test set as the evaluation metric. Pair Classification tasks consist of SprintDuplicateQuestions (Shah et al., 2018), TwitterSemEval2015 (Xu et al., 2015), and TwitterURLCorpus (Lan et al., 2017). Performance is assessed using accuracy on the test set.
*https://scikit-learn.org/0.19/datasets/twentynewsgroups.html
Baseline Models We select several sentence representation methods that have achieved state-of-theart performance on STS tasks, including publicly available models scaling from 100M to 11B parameters, and APIs without exact parameter amounts. Concretely, we select traditional no-instruction sentence representation models (e.g., GloVe (Pennington et al., 2014), USE (Cer et al., 2018), Contriever (Izacard et al., 2022b), GTR (Ni et al., 2022b), SimCSE (Gao et al., 2021), SGPT (Muennighoff, 2022), E5 (Wang et al., 2022a), and SentenceT5 (Ni et al., 2022a)), instruction-based models (e.g., Jina (Günther et al., 2023), Udever (Zhang et al., 2023), Stella †, BGE (Xiao et al., 2023), GTE (Li et al., 2023), and INSTRUCTOR), and APIs (e.g., OpenAI Text Embedding (Neelakantan et al., 2022), Voyage ‡, Cohere §, and Ember ¶).
†https://huggingface.co/infgrad/stella-base-en-v2 ‡https://docs.voyageai.com/ §https://txt.cohere.com/introducing-embed-v3/ ¶https://docs.llmrails.com/embedding/embed-text

heading: Future Work
text: We will conduct more experiments using our approach on other strong baselines in the future. Furthermore, due to our limited computational resources, we have not explored our method on larger models, e. g. , 3B and 7B LLaMA. We also leave it in our future work, and explore more efficient and effective data curriculum methodology for large-scale datasets and models