Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge
Authors: Fang Dong
Mengyi Chen
Jixian Zhou
Yubin Shi
Yixuan Chen
Mingzhi Dong
Yujiang Wang
Dongsheng Li
Xiaochen Yang
Rui Zhu
Robert P. Dick
Qin Lv
Fan Yang
Tun Lu
Ning Gu
Li Shang
Venue: NeurIPS 2024 poster (2024)
Keywords: language model
long-tail
clustering
Primary area: natural_language_processing
TL;DR: The long-tail data in language model suffer from its gradient inconsistency with overall data, causing model struggle to capture domain knowledge during pretrain. We use a clustering-based sparse expert network, yields better performance.

Abstract: Language models (LMs) only pretrained on a general and massive corpus usually cannot attain satisfying performance on domain-specific downstream tasks, and hence, applying domain-specific pretraining to LMs is a common and indispensable practice.
However, domain-specific pretraining can be costly and time-consuming, hindering LMs' deployment in real-world applications.
In this work, we consider the incapability to memorize domain-specific knowledge embedded in the general corpus with rare occurrences and long-tail distributions as the leading cause for pretrained LMs' inferior downstream performance. 
Analysis of Neural Tangent Kernels (NTKs) reveals that those long-tail data are commonly overlooked in the model's gradient updates and, consequently, are not effectively memorized, leading to poor domain-specific downstream performance.
Based on the intuition that data with similar semantic meaning are closer in the embedding space, we devise a Cluster-guided Sparse Expert (CSE) layer to actively learn long-tail domain knowledge typically neglected in previous pretrained LMs.
During pretraining, a CSE layer efficiently clusters domain knowledge together and assigns long-tail knowledge to designate extra experts. CSE is also a lightweight structure that only needs to be incorporated in several deep layers.
With our training strategy, we found that during pretraining, data of long-tail knowledge gradually formulate isolated, outlier clusters in an LM's representation spaces, especially in deeper layers. Our experimental results show that only pretraining CSE-based LMs is enough to achieve superior performance than regularly pretrained-finetuned LMs on various downstream tasks, implying the prospects of domain-specific-pretraining-free language models.

Once Read is Enough: Domain-specific Pretraining-free Language Models with Cluster-guided Sparse Experts for Long-tail Domain Knowledge
Fang Dong'
Mengyi Chen 1.+
Jixian Zho
Yubin Shi
Yixuan Chen
Mingzhi Dong?.
Yujiang Wang2.t
Dongsheng Li
Xiaochen Yang*
Rui Zhu
Robert Dick?
Qin Lv Fan Yang" Tun Lu'
Ning Gu
Si
1 China and Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University
Oxford Suzhou Centre for Advanced Research, Suzhou, China
University of Bath
Microsooft Research Asia, Shanghai, China
5 School of Mathematics & Statistics, University of Glasgow Bayes Business School, City St George's, ss University London
7 Department of Electrical Engineering and Computer Science, University of Michigan
8 Department of Computer Science, University of Colorado Boulder School of Microelectronics, Fudan University
Abstract
Language models (LMs) only pretrained on a general and massive corpus usually cannot attain satisfying performance on domain-specific downstream tasks, and hence, applying domain-specific pretraining to LMs is a common and indispensable practice. However, domain-specific pretraining can costly and time-consuming, hindering LMs' deployment in real-world applications. In this work, we consider the incapability to memorize comain-specific knowledge embedded the general corpus with rare occurrences and fooggtaill" distributions as the leading cause for pretrained LMs' inferior downstream performance. Analysis of Neural Tangent Kernels (NTKs) reveals that those long-tail data are commonly overlooked in the model's gradient updates and, consequently, are not effectively memorized, leading poor domain-specific downstream performance. Based on the intuition that data with similar semantic meaning are closer in the embedding space, we devise Cluster-guided Sparse Expert (CSE) layer to actively learn long-tail domain knowledge typically neglected previous pretrained LMs. During pretraining, a CSE layer efficiently clusters domain knowledge together and assigns long-tail knowledge to designate extra experts. CSE also a lightweight structure that only needs to be incorporated in several deep layers. With training strategy, found that during pretraining, data of long-tail knowledge gradually formulate isolated, "outlier" clusters in LM's representation spaces, especially in deeper layers. Our experimental results show that only pretraining CSE-based LMs is enough to achieve superior performance than regularly pretrained-finetune LMs on various downstream tasks, implying the prospects of Nommain-peeciffc--eerrannii language models.
"Co-first authors. (Coresponding authors: syujiang. wangRoscar ax.ac.uko. <1 ilishng@fuuann.. edu, cn>.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
In natural language processing, it is a prevalent paradigm to pretrain language models (LMs) on large-scale unlabeled corpus covering plethora of knowledge, and those pretrained LMs have exhibited impressive performance in language tasks in the general domain [41]. When it comes to downstream tasks requiring specialized domain knowledge, e.g., legal search or medical question answering [23 7.. those models usually fail to expertise in such knowledge and cannot acquire desirable performance. As such, domain-specific pretraining on domain-specific datasets is deemed essential to fulfill pretrained LMs' potential in various downstream tasks [2] 14] 42 36]. However, applying domain-specifie pretraining instance, the involvement of a doctor for healthcare tasks [31], which can be costly and laborious. The associated catastrophic forgetting issue (26] could further complicate the domain-specift pretraining process.
LM could require domain expertise from humans, for
Top 20 Subredcit Count of Reddit Comments Datoset
400
5-
perplasity
perplestty
200
2-
150
100
50
AoARestor
politccs
rifl
riba
Danald
newe
paridnewss
garning
movites
Bitcom
pics
leapunttiegeen aaoee
Sargary
furtey
Destiny Trecuuee
foday rodayiettn redrreeeeeed videos
0.6
0.5
0.4
0.3
0.2
0.1
0.0
socter
seriience frequency score
(a)
(b)
Figure 1: a) The top 20 subreddits with the highest amount of data in the Reddit Comments Dataset, where a typical long-tail -ail distribution can be observed. b) Language Models struggle to memorize long-tail domain knowledge during pretraining. The less frequently a sentence appears in the training corpus, the higher its perplexity, indicating that it not effectively memorized.
In this work, re- visii the pretraining- domain-specific- pretrainnn paradigm and raise the follow- ing question: is domain-specific preiraining indispensable to LMs? Notably, the domain-specific knowledge necessary for various downstream tasks is usually embedded in the pretraining corpus of extensive information sources. Those pieces of domain-specific information may only appear a few times in the massive corpus, significantly less frequently than other ubiquitous and general knowledge. and there can be numerous pieces of such rare information, a distribution usually defined as "long- tail". In Fig I(a) we plot the frequency of the top-20 subreddits count on Reddit Comments Dataset (2]. and a typical long-tail distribution can be observed. Previous works have verified that LMs are not good learners of long-tail knowledge in the pretraining dataset with -Anseeing as the downstream task 20). Our experiments, as shown in Figure [(b) further illustrate that pretrained LMs do not adequately retain Nomain--peecif knowledge in loig-tail sequences which evidenced by surge in perplexity corresponding to decreased frequency score. This could result in inferior performance on downstream tasks. Domain-specific pretraining improves LMs' domain performance by providing second lesson, which could be avoided if the first (pretraining) is appropriately delivered.
To unveil the hidden mechanisms under LM's incapability to learn long-tail domain-specific knowl- edge, we investigate the behaviors of a GPT on the Wikipedia dataset. We examine LMs' learning capabilities on long-tail data by analyzing the Neural Tangent Kernels (NTKs) of long-tail data and all data. Recent research [5] 44) has indicated that the updating of deep networks can be governed by the gradient direction corresponding to the principle eigenvector of an NTK matrix, which reflects the most common -deeeendn- direction across the entire input space. Following those works, we consider an NTK's 's principle eigenvector (PF) gradient direction
primary indicator of an LM's overall gradient-updating direction over a data space. Our analysis has revealed that the PE gradient direction of toig-tail data, indicating the gradient-descending direction from long-tail tail knowledge, is

---

generally diverged from that of overall data, which rules the overall updating of network parameters. The observation that long- tail data cannot substantially impact LMs' parametric updates under regular pretraining settings explains pretrained LMs" incompetence on domain-specifie knowledge of rare occurrences, necessitating an effective solution.
To this end, we propose the Cluster-guided Sparse Expert (CSE) layer, an effective, efficient, and cosy-to-mmplement approach improve LMs' long-tail knowledge awareness. CSE layer, with intuition such that data with similar semantic meaning are closer in the embedding space, we perform efficient clustering on the embeddings to group data from different domains, and additional experts will be assigned to explicitly and appropriately memorize the information within those clusters. Models trained with CSE show pronounced cluster structure in the embedding space, where long-tail data forms small, outlier clusters. We empirically demonstrate that converting several deep layers into CSE ones can enough to achieve satisfying results, such as the last two layers of GPT (28) or BERTIO0, and the incurred computational costs are comparatively small and arguably acceptable. We have verified that pretrained CSE-based LMs have outperformed regularly pretrained domain-specific pretrained LMs downstream tasks from various domains, which implies that domain-specific pretraining may not be essential long-tail knowledge can sufficiently learned.
Our contributions are summarized as follows:
We have presented that datasets show long-tail distribution, with domain-specifi data in the long-tail, and revealed that long-tail data cannot substantially affect LMs' training, which is a leading cause of LMs' incompetence in learning rare,
knowledge.
We have devised a Cluster-guided Sparse Expert (CSE) architecture to better pretrain LMs memorize the long-tail domain knowledge. With such a training strategy, LMs can effectively capture long-tail tail domain data in the representation space as outlier clusters, thereby enhancing their ability handle less frequent contexts efficiently.
Promising performance downstream tasks has verified the effectiveness of the proposed method, indicating that domain-speciff pretraining may not be indispensable to LMs.
2 Analysis of Long-Tail -ail Domain Data
In this section, we first elucidate the challenges associated with learning from long-tail data through gradient analysis. We then explore the embedding space using the pouster-guided Sparse Expert (CSE) layer, which effectively captures the structural nuances of long-tail data. Furthermore, we examine the dynamics of these clustering structures, offering insights into how the learning processes of long-tail clusters adapt and evolve across various training stages and model layers.
2.1 Challenges in Learning Long-Tail Tail Domain Data
This subsection explores the significant challenges posed by long-tail domain data within language models (LMs). The primary issue stems from the divergence in gradient directions between long-tail data and the general gradient-updating trajectory of these models, which critically hampers effective learning.
2.1.1 Preliminaries and Definitions
Informed by seminal works [1] we utilize Neural Tangent Kernels (NTKs) scrutinize the gradient behavior of neural networks under a gradient descent training regime. The NTK represented (xX, X²), is defined
the outer product of the gradients of network outputs relative to its parameters O X, X) Jo(X))oo(Z) where Jo function f at the data points X.
Vef (X:; 0) denotes the Jacobian matrix of the
To determine the predominant gradient-descending direction across the input space, which is influ- enced by the gradient direction associated with the principal eigenvector of the NTK matrix, we first perform an eigenvalue decomposition of the NTK matrix. Recognized as a positive semi-definite real symmetric matrix, the NTK decomposes into the total number of training instances. The principal eigenvector umar is identified as the vector corresponding to the maximum eigenvalue. Then the primary gradient direction for a given input set X' is ge( go(X) Jo(X), Building upon the above preliminaries, we introduce the metric of
UAUT Σn A,u.u," Here, n represents
3

---

Gradient Consistency (GC) to evaluate the alignment between gradient directions for specific data subsets and the overall dataset.
Definition (Gradient Consistency (GC)). Let X' be specific subset of the training set X. The gradient consistency of X" is evaluated by computing the cosine similarity between the most prevalent gradient direction of X' and that of the entire dataset X:
go(X) 80 (2)) l|go x) go (X')
GC,(X')
(I)
higher GC value indicates that the model's optimization updates are well-aligned with the needs of the specific subset X', suggesting focused and effective learning of this data. Conversely, lower value indicates suboptimal learning of these data, pointing to potential areas for improvement in model training strategies.
2.1.2 Gradient Consistency (GC) Analysis
We assess the sentences from Wikipedia on standard GPT model using a sentence frequency score to gauge how frequently each sentence appears in the corpus. This score is calculated by averaging the frequency of its constituent tokens. ( displays the relationship between GC and sentence frequency score. Additionally, the figure includes histogram that details how many percentages of sentences across the whole dataset fall into each frequency bin.
There is a significant correlation between gradient consistency and the frequency with which sentences appear in the corpus. Notably, for sentences less frequently encountered the dataset, the model demonstrates substantial ineffectiveness in learning. As demonstrated, the GC value sharply declines from 0.8 0.4 as the sentence frequency score decreases from 0.3 0.2. Furthermore, the GC value continues to diminish as the sentence frequency score decreases further, indicating that the model's s gradient descent direction struggles to align with the requirements of these rare sentences.
Our analysis indicates that the optimization requirements for long-tail sentences are significantly overlooked during standard pretraining, largely due gradient conflicts between long-tail and common data. Research [38, 30) has demonstrated that these conflicts lead to suboptimal learning outcomes for the affected data. In typical pretraining, the gradient descent direction dominated by common data, which prevents the model from effectively capturing the unique characteristics of long-tail domain data. This oversight significantly impairs the performance of LMs learning domain-specttn knowledge, particularly when dealing with rare occurrences, highlighting the need for a more effective solution.
avg freq: 0.363
Soccer Ma agement
freq: 0.326 Religion & Ideaology
0.6
avg 0:41
0.4
0.0
Cluster 0
avg freq: 0.332
Cluster avg freq: 0.294
Java Network Debug
0.2
Public Health on
Document
0.1
Cluster
CHD
0.6
0.5
0.4
0.3
0.2
0.1
0.0
senterde frequenry
(a)
(b)
Figure 2: a) The correlation between sentence frequency score and gradient consistency in the baseline model. histogram is also included showing how many percentage of sentences across the whole dataset fall into each frequency bin. For further analysis using 2/3-gram averages, please refer to the appendix D b) sampled embedding space containing long-tail clusters, taken from our CSE layers. For more information on the detailed cluster contents, please refer to the appendix D.2

---

2.2 Embedding Space Analysis With Cluster-guided Sparse Expert (CSE) layer
Prior research 2 has shown that extensive domain-specific data reside within the long-tail distribution of general pretraining corpus, as illustrated in Figure[[(a) These data, often semantically similar, are likely to cluster closely within the embedding space, facilitating potential aggregation for dedicated learning. However,
analysis in Section 2. unddrrscorsss significant challenges in learning from long-tail data. Specifically, the model's I's gradient updates frequently fail to align with the optimization needs of these data, leading to their under-representatin obscures the inherent group structures that these domain data form based on their semantic similarities, thereby impeding dedicated learning efforts.
in the embedding space. Such misalignment
To address the issues outlined above and to facilitate a more effective examination of long-tal -til domain data in the embedding space, we propose the Cluster-guided Sparse Expert (CSE) layer. This layer groups proximate long-tail data points into clusters and directs them to specialized experts for dedicated learning. As demonstrated in Figure 3(a] the GC value of long-tail data in the baseline model initially increases at the beginning of the training stage but rapidly declines thereafter, indicating that the model's inability to capture the learning dynamics of long-tai tail data begins early in the training process. Our CSE layer capitalizes on the clustering structure at the point where the GC value peaks, subsequently taking effect to channel domain-specific clusters into dedicated learning pathways. Further details about this approach are provided in Section B
The clustering results from the CSE-based LM, shown in Figure 2(b) reveal four smaller clusters alongside a predominant one. Detailed analysis shows high domain coherence within the smaller clusters, each comprising sentences closely related to specific domains. The average sentence frequency score of these domain clusters falls into the long tail of the sentence frequency distribution, as shown in Figure 2(a) In contrast, the predominant cluster, colored in purple, contains diverse mix of more common data and exhibits a higher average sentence frequency compared to the smaller clusters. Further analysis of sentences with frequency scores below 0.2 shows their random distribution across clusters, suggesting these extremely infrequent sentences may serve as noise in the learning process.
This analysis demonstrates that our proposed CSE-based architecture effectively groups long- data from the same domains for dedicated learning, fostering domain-specifie clustering structure within the embedding space. The long-tail domain clusters, distinct from clusters containing common data, show higher degree of compactness and are clearly separated, highlighting the unique features embodied by these clusters.
2.3 Dynamic of Long-Tai Tail Domain Clusters
In this subsection, we explore the learning dynamics of long-tail domain data by tracking how clusters evolve across different training stages and model layers. We utilize DBSCAN clustering 29 to determine the number of clusters.
Long-tail clusters can be seen early in the training stage. As shown in Figure (() and Figure 3(c) the number of clusters quickly peaks early in the training stage, along with peak in the ratio of cluster distances cluster radii. This indicates that our CSE-based architecture effectively promotes the formation of a clustering structure early on.
The swift emergence of these clusters signifies substantial model adaptation to global features at the start of training, allowing for effective differentiation between clusters. As training progresses, inter-cluster distances gradually decrease, suggesting stabilization in the learning dynamics and potential shift in focus toward refining intra-cluster nuances.
Long-tail clusters become more pronounced with increasing network depth. Figures 3(b) and demonstrate that the number of clusters is consistently higher in the deeper layers compared to the lower layers, with the ratio of cluster distances to cluster radii escalating significantly in the last two layers and reaching its maximum in the final layer. This pattern indicates that clusters become increasingly distinct and better separated as they progress through the network's layers. The enhanced separation of clusters in deeper layers can be attributed to the hierarchical feature extraction inherent in deep neural networks. As data moves through successive layers, the network abstracts and compiles more complex features, transitioning from general to more specific attributes. This
5

---

gradilet
cluster number by DBSCAN
Cluster distarcc cloceer easli catoo
0.3
number
1.3
14
Leen 22
6,07
02
10
triiing stop
Step(10k)
10
traliing Mep
(a)
(c)
Figure 3: a) Evolution of the Gradient Consistency (GC) of long-tai onig-tail data in the baseline model over the first 8000 training steps. GC scores beyond this range are omitted, as they consistently remain below 0.2. For details on the method used select long-tail data, please refer the appendix [D3 b) Evolution of number of clusters over training steps. c) The ratio of cluster distances to cluster radii over training steps, providing measure of cluster structure clarity independent of norm values.
add & norm
Moture of FFNs
stop 10k
expert
expert
expert 3
dispatcher
routes
Stop
add
Gaussian
dispatcher
matrix
Mettue Attentions
low-dimensional embedding space
step 20.
dispatcher
slep ETh.
(a)
(b)
Figure a) Overview of the Cluster-guided Sparse Expert (CSE) layer. b) The cluster number fluctuation is mainly caused by the big common cluster. These four figures arranged sequentially from top to bottom, were sampled at every 10,000 steps throughout the process from the FFN of the 10-th layer in a GPT model.
hierarchical processing allows the final layers capture and enhance subtle distinctions between different data groups, leading to more defined and isolated clusters. This process not only underscores the capability of deep layers to refine and emphasize key features but also illustrates the network's efficiency in encoding progressively finer- greiner information as layer depth increases.
3 Cluster-guided Sparse Expert (CSE)
To avoid the troublesome and costly domain-specific pretraining, design a novel strategy, named Essuter-guided Sparse Expert (CSE), to help the model capture the taig domain knowledge during pretraining. Since long-tail tail domain data show poor gradient consistency with overall data, we employ a sparse expert architecture within the Transformer model to assign data to different parameters, thereby avoiding the gradient conflict in each parameter group. This strategy can be applied to either attention or FFN. To dispatch data, with a straightforward and generally accepted intuition such that data with similar semantic meaning closer in the embedding space, design a very simple, efficient but effective online clustering algorithm operating concurrently with the language model pretraining, separate embeddings into different clusters, and use the outcome of this algorithm to instruct the dispatching of embeddings. The proposed algorithm is outlined in Algorithm

---

Dimension Reduction In high- nigh-dimensional vector clustering, computational efficiency poses significant challenge due to the (df)) complexity of computing vector distance where denotes the dimensionality. So, we employ the same way of dimension reduction as is discussed in Section 2 before applying clustering on the embeddings, using a Gaussian random initialized matrix to project embeddings to a low-dimensional space[22]. This process, grounded in the Aaanson-LLideestrann Lemma, effectively preserves the pairwise distances between embeddings while reducing their dimensionalityy, thereby enhancing the efficiency of our clustering algorithm.
Initialization We commence by training baseline dense model devoid of any expert structure.
Algorithm 1 Cluster-guided Sparse Expert
Require: w: Warm-up step count
Our findings in Section [ illu- minate an initial rise in gradient consistency between long-tail do- main data and the general dataset at the onset of training, subse- quently followed by a downturn. Consequently, we adopt warm- up stage, letting the model learn the common features of long-tail tail and non-long-tall tail data. In our experiments, this process typi- cally accounts for no more than 1 1% of the overall training. We then sample N instances from the dataset and use its cluster- ing result to initialize the cluster structure. We utilized DBSCAN (29 in our experiments, clustering algorithm that does not explicitly require the number of clusters. For every identified cluster, we document its centroid and define its radius as the average distance of all constituent data points from this central point
Require: N: Initialization data count
Require: M: Gaussian random matrix E Rddxd for reducing
dimension
Require: S: Incoming embedding stream
Require: a: center update factor 1: Wait w steps till the warm-up end.
2: Sample N data and run a clustering algorithm. Initialize cluster structure with the outcome by recording the cluster center e; and radius r; for each cluster.
3:
for in S do
4:
o' = MM
5:
= arg min l1 - c,ll/T,
6: Dispatch e to parameter group
7:
C; ac; + (1 - a)y'
8:
end for
After this warm-up period, we fix the number of clusters and copy the module into cluster number copies. The module selection is introduced in the next paragraph. In our experiments, we noticed that the variations in the number of clusters were primarily driven by the splitting and merging of larger clusters, as illustrated in Figure 4(b) the smaller, long-tail clusters, however, remained largely unchanged. Consequently, adopting the initial clustering configuration directly, without further adjustments during training, was found to have no detrimental effect on model performance or the distribution of data handling. This approach capitalizes on the stability of the long-taii clusters and the dynamics of the larger ones, ensuring efficient data processing without compromising accuracy.
Select Layer Our motivation for performing clustering is rooted in the premise that semantically similar data tends to be closer. However, it is important to note that models learn the semantics of data progressively through layers; we delve deeper into the model layers, the semantic information becomes increasingly rich, which may in turn amplify distinctions between data points. To quantify this variation, we apply our strategy only layers with larger inter-cluster distances. Since the last layers show a significant increase in inter-cluster distance, we apply our strategy in the last 2 layers, which is also the empirical best practice observed in existing moe-related works. 25
Dispatch Embeddings For each coming embedding, we decide the index of the expert it is dispatched to with i arg minj=1 /|v/ c/l//r,, where сj denotes the center of cluster j, and r denotes the radius of cluster j. Note that the of here is the sequence embedding rather than token embedding and is defined as the mean of all token embeddings in the sequence and the dispatching also happens on the sequence level. The rationale for defining and analyzing long-tail at the sequence level stems from the fact that when we discuss long-tails, we are essentially referring to semantics rather than individual words tokens. Such semantics can only be observed within the context provided by sequences as a whole.
Update cluster center The model's parameter space undergoes gradual updates throughout training, causing a slow drift in the embedding space as the parameters evolve. To tackle this, we incorporate a dynamic mechanism to update the cluster centers concurrently with the assignment of clusters. For
7

---

given cluster me,, let its center at time be denoted as c.. When new embedding arrives and is assigned to mcg we update of with: c++1 a of + (1-a) v', where, a [0, 1] is a center update factor that determines the influence of the new embedding v' on the existing center d. This adaptive updating scheme ensures that cluster centers remain representative of the current state of the embedding space, even as evolves through the training process.
4 Related Works
Long-Tail Prior research addressing the issue of long-tail learning has predominantly been con- ducted within the domain of computer vision. The objective is accurately recognize and classify rare or infrequently occurring classes in given dataset together with frequently occurring classes 44]
There are several approaches address the problem, including re-weighting 8 logit adjustment
4 45], robust distributional matching [8 33], and knowledge transfer [39 34]. [7 declare that as
the number of samples increases, the diminishing phenomenon suggests that there is a decreasing
marginal benefit for a model to extract additional information from the data due to the presence of information overlap. Research in natural language processing has identified significant limitations in language models' capacity to learn long-tail knowledge [27 3 Furthermore, [46] suggests that attempting to address this issue during the domain-specific pretraining stage is often too late.
DSpeif-Specific Pretraining Domain-specific pretraining, also known as domain-specific pre- training, is highly advantageous assist language models in requiring specialized domain knowledge. In one approach, contextualized embeddings are adapted to text from the target domain using masked language modeling, as detailed by Han and Eisenstein [16]. The concept of multi-phase pretraining involves secondary-stage unsupervised pretraining. exemplified by broad-coverie
domain speciifc BERT variants like BioBERT (24). Research by Gururangan et al. [5] extends this by proposing domain-adaptive pretraining (DAPT) from a broader corpus and task-specific pretraining (TAPT) which uses unlabeled data increasingly aligned with the task distribution. These studies underscore the importance of domain-relevant data for pretraining in both high and low-resource scenarios
16 15).
5
Experiments
This section presents the experimental results of our model and other methods. In the experiments, our model only undergoes pretrained phase, reading domain-speeific data once. Other methods are pretrained on the same dataset and then continue- continue-peetrained on domain- suauni--peeffic datasets. Subse- quently, all models are used as embedding models with all parameters frozen generate embeddings for downstream tasks.
Dataset and Evaluation We employ Wikipedia 13 our pretraining dataset, which is also widely accepted in other works (24 10). We adopt some legal and medical domain-specific downstream tasks to show the effectiveness of model. To ensure that the pretraining data do contain domain knowledge required by the downstream tasks, we mixed a relatively small amount (less than 8%) of gggalddomain-speciff data ( and medical-domain-speiife data [9] into the pretraining data to simulate a long-tail distribution. The datasets selected are listed in Table 4 in Appendix A Concurrently, we report the test perplexity of each model after the pretraining phase, serving as evidence of model convergence. Task performances are reported by accuracy. Although we use the method of mixing small-scale comain-speciffe datasets into pretraining data to simulate the long-tail distribution in those huge corpora, we cannot fully simulate the extremely rich pretraining data used on L.LMs due to the limited training resources.
Baselines Since our strategy is not restricted to a specific model structure, we adopt both BERT 10) and GPT (28) the base models and compare all the strategies on these base models respectively. We also compare with a Switch-MoE [1I version of them to show the effectiveness of our routing strategy. More Detailed implementation setting is listed in Appendix
5.1 Main Result
Table 9 and Table 2 show the performance of all models/strategies under our experiment setting with trainable lincar classifier for downstream tasks. */med means a model continue-pretrann pretrained
8

---

on medical-domaiee domain-specific data, and */egal means a model continue- -rettnuie-e on legal-domain, specific data. We tested cluster-guided Sparse Expert on Attention and FFN respectively, denoted as MoA and MoF.
Table
Results of strategies applied on BERT
Models
Pretrain ppl
Overruling
Casehold
GAD
EUADR
SST2
average
BERT/med
37.00
86.67
50.51
67.09
84.23
66.86
71.07 0.22
BERT/legal
37.00
86.67
50.93
66.83
84.79
65.14
70.87 + 0.23
MoF/med
31.00
85.00
50.49
64.52
83.10
64.79
69.58 0.20
MoF/legal
31.00
85.83
50.30
64.32
84.79
63.88
69.82 + 0.19
Ours/MoA
28.25
86.62
50.94
72.90
90.09
66.60
73.43
0.18
Ours/MoF 34.64
89.10
50.82
71.65
91.23
67.98
74.16 + 0.20
BERT/med exhibited severe forgetting issue and details will be discussed in the Appendix
Table 2: Results of strategies applied on GPT
Models
Pretrain ppl
Overruling
Caschold
GAD
EUADR
SST2
average
GPT/med
55.59
88.33
49.82
71.56
84.23
73.90
73.57 0.17
GPT/legal
55.59
89.17
50.58
71.69
81.69
74.50
73.53 = 0.23
MoE/med
40.69
91.25
50.11
72.77
83.38
72.03
73.91 0.12
MoE/legal
40.69
91.60
49.68
72.66
83.38
71.97
73.86 = 0.23
Ours/MoA
42.99
91.68
50.70
71.75
85.91
74.61
74.93
0.08
Ours/MoF 43.38
93.33
51.26
73.30
85.63
76.00
75.90
0.19
Our method outperforms other models/strategies almost all tasks, with average improvement of around 3%, showing an ability to learn long-tail data from the pretraining dataset. Our method be applied either the Attention module or the FFN module, and both ways will yield better result compared with the domain-specific pretrained baselines, showing potential for eliminating the need for domain-specific pretraining. While in certain scenarios, ceamain-specifcc pretraining remains indispensable due to the privacy concerns associated with proprietary data, we argue that when pretraining datasets encompass domains similar to the proprietary one, our approach can still facilitate an enhanced domain-specife pretraining performance. We also present the result of larger scale model, in Table We used larger models (330M GPT-sttyee models trained with 20B tokens) and domain-spectie tasks from the academic, environmental, and financial domains to demonstrate the generalization capability of our method. The results indicate that our method, even without fine-tuning, consistently outperforms baselines by average of 3-5% in accuracy on domain-specific tasks while maintaining comparable performance on general tasks. also notable that domain- specific pretraining leads to overfitting and even catastrophic forgetting, resulting in a decrease in performance on tasks from non-related domains. More details are shown in Appendix [A] Further experiments show that our approach does not cause reduction in general knowledge acquisition, and more details are shown Appendix
Table 3: Results of strategies applied on 330M GPT
Domain
Task
GPT//uned
MoE/tuned
CSE/ w/o tune
academic
chem-prot
36.25
36.25
36.25
academic
MAG
63.22
64.91
65.47
academic
ret-20k
76.95
78.28
80.15
environment
clim. det.
78.94
79.90
80.26
environment
clim. sent.
66.81
68.31
69.98
financial
FPB
16.83
25.00
40.11

---

5.2 Analysis
Expert analysis We analyze our model's embedding space to determine if our method dispatches embeddings correctly. We sample data and perform a forward inference pass through the model, visualizing the dispatching path of our model. As shown Figure [ our distribution strategy correctly and effectively dispatches data from different long-tail clusters to different experts. We further visualize the NTK each expert of our model, and can be observed that by dispatching long-tail data separately. the NTK in each expert becomes more consistent. Whereas in a baseline model, NTK matrix shows a poor consistency the batch data, since long-tail and non-long-tail data are not separated.
47
(8)
(b)
(c)
Figure 5: a) The embedding space and routing result of our model. b) The NTK in each expert in our model. e) The NTK in baseline. b) and c) sampled from the FFN in the 10th layer.
For time overhead, the cluster initialization step is the additional computation introduced by our method throughout the training phase, relative to both the baseline and MoE. The time complexity of this operation will be On nd") and bounded by ('n2d') for the worst case, where n represents the number of sampled points. Notably, during subsequent inference stages, the quantum of parameters activated per iteration by our algorithm aligns identically with that of both MoE and the baseline. Since MoE and the baseline necessitate an additional fine-tuning phase, it becomes evident that the time cost of this phase far surpasses the On'2d') complexity cluster initialization. To provide more straightforward data, the wall-clock time of training & domain-specitn pretraining of 330M models for baseline GPT, MoE, and our CSE method and are 177 hours, 205 hours, and 160 hours respectively.
6 Conclusion
In this paper, we seek to elucidate why language models require domain-specific pretraining despite the presence of domain knowledge in their pretraining data. Our investigation uncovers that Sen- tences with lower frequency scores show diminished gradient consistency, resulting in increased test perplexity. This misalignment, particularly pronounced in low-frequency sentences, culminates in elevated test perplexity, suggesting a deficiency in effectively leveraging domain-specifie information. To address this challenge, we introduce Cluster-guided Sparse Experts (CSE), grouping diverse long-tail domain data and dispatching them to different experts to enhance gradient consistency within each expert, thereby enabling the model to incorporate long-tail domain knowledge during pretraining. Experiments suggest that our approach has the potential to supplant the need for dedicated domain- domain-specific pretraining stage. Through this approach, long-tai domain instances promote the formation of small, outlier clusters in the representation space, exhibiting a characteristic signature across varying stages of training and architectural depths.
Acknowledgements
This work supported by the National Natural Science Foundation of China under Grant 62090025. and the Natural Science Foundation of Jiangsu Province (BK20240414).
10

---

References
[1] Caselaw access project. https: /aae:://caaee.. law/ 2024.
[2] Reddit comments dataset.
hatpppeeeedttssecseeddd-----ooom 2024.
[3] Mallen A., Asai A., Zhong V, Das R., Khashabi D., and Hajishirzi H. When not to trust language models: Investigating effect. in Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
[4] Krishna Menon Aditya, Jayasumana Sadeep, Singh Rawat Ankit, Jain Himanshu, Veit Andreas, and Kumar. Sanjiv. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.0755 2020.
[5] Benjamin Bowman and Guido Montúfar. Spectral bias outside the training set for deep networks the kernel regime, ArXiv, abs/2206.02927, 2022. URL httpss/s: www.aesssaaaanslll.....
Ttgg/CrpusID::: :49431447
[6] Alex Bravo, Janet Piñero,, Núria Quenalt-Rosinach, Michael Rautschka, and Laura Furlong. Extraction of relations between genes and diseases from text and large-scale data analysis: implications for translational research. BMC Bioinformatics, 16(1), February 2015. doi: URL httptt//s///oi. http::://doooorr/1.... 0111886/122859-1--0077777
[7] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion An- droutsopoulos. Legal-bert: "preparing the muppets for court"", ArXiv, 2ts/22000.2259,2222
URL http::tt://set www.eteiiscoollleeeeee org/CorpusID : 222141043
[8] Huang Chen, Li Yining, Change Loy Chen, and Tang Xiaoou. Learning deep representation for imbalanced classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[9] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Scokhwan Kim, Walter Chang, and Nazli Goharian. A discourse- aware attention model for abstractive summarization long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Pupers), pp. 615-621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 18655//1/N1182000 URL htt.::ttt///.. acccl anthology aaattooo// wttphooooooyor///1-2200000
[10] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /800.0085 2018.
[11] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 2022.
[12] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution the neural tangent kernel. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020. NewrIPS 2020, December 6- 12, 2020, virtual, 2020. URL /tRp:https:///roccddingg neurips.com cc. wttpssccccppp//pppp/peee http:/pppprr/00 2200//ass/ html
[13] Wikimedia Foundation. Wikimedia downloads. L httph::t////tt http://ddmmpp.o http:://ddmmss.wiimiiiiaaa wikimedia.or
[14] Zhen Guo and Yining Hua. Continuous training and fine-tuning for domain-specific language models in medical question answering. ArXiv, abs/2311.0020. 2023. URL https://api
semanticscholar www//orrgusseeeeoooooo :2:4::329::222525555555 264832958
[15] Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004.10964, 2020. URL. httpp:://aat httppp//aap..smmmticccooooooooo
semanticscholar. htaprrttccccolaaaoooror/rrppspss Ir
216080466

---

[16] Xiaochuang Han and Jacob Eisenstein. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. Conference on Empirical Methods in Natural Language Processing, 2019.
http://ppp.emmmiiccccccoooe www.ssetimcccooooooooooo org/ wwgg/CorpusID:200 202541481
[17] Junjie Huang, Duyu Tang, Wanjun Zhong. Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. Whiteningbert: An easy unsupervised sentence embedding approach. arXiv preprint arXiv:2 2021.
[18] Zheng Huangjie, Chen Xu, Yao Jiangehao, Yang Hongxia, Li Chunyuan. Zhang Ya. Zhang Hao, Tsang Ivor, Zhou Jingren, and Zhou. Mingyuan. Contrastive attraction and contrastive repulsion for representation learning. Transactions on Machine Learning Research, 2023.
[19] Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Conver- gence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018. URL abbt:44att25aaaattattt...... www.riinnnnnnnnnnn..eoemm.
neurips nwwrips.cc/paaaae ccc/paperr2018/haaaeeee
[20] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle learn long-tail knowledge. In International Conference on Machine Learning. pp. 15696-15555 15707. PMLR, 2023.
[21] Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu, Adapting a language model while preserving its general knowledge. arXiv preprint (xXXx:2301.08986 2023.
[22] Kasper Green Larsen and Jelani Nelson. The johnson-linddeestraus lemma is optimal for linear dimensionality reduction. arXiv preprint arXiv:1411.2400 2014.
[23] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36: 1234 1240, 2019. URL httph::tts//ppp http//pp..smaancsccchhol.aaaaa
org/Corpus ID: 59291975
[24] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1234-1123411200 2020.
[25] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. Interna- tional Conference on Learning Representations, 2021.
[26] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empiri- cal study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv, abs/2308.08747, 2023. URL. attpps///ppl wwwanticscholeee..m. org. png/CorpusID
261031244
[27] Kandpal N., Deng H., Roberts A., Wallace E., and Raffel C. Large language models struggle to learn long-tail tail knowledge. In International Conference on Machine Learning. 2023.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188)9 2019.
[29] Erich Schubert, Jörg Sander, Martin Ester, Hans-Peter Kriegel, and Xiaowei Xu. Dbscan revisited, revisited. ACM Transactions on Database Systems (TODS), 42:1 - 21, 2017. URL https://api.. semanti sttts://ap...mmattcccchoooo
cachooaattcccclaa...ccmmmoo org wrw.oaarr.rrg/rouuuDD...... 5156876
[30] Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. Recon: Reducing conflicting gradients from the root for multi-task learning, 2023. URL htRpstts:: http::tttt://arttttt ://arxiv././/////aa www.aaarivorrrr.... abs/ 0bs/2302.rg 11289
org/
12

---

[31] K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mahdavi. Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani. Heather J. Cole-Lewis, Stephen J. Pfohl, A Payne, Mar- tin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, A. Mansfield, Blaise Agiera y Arcas, Dale R. Webster, Greg S. Corrado, Yossi Ma- tias, Katherine Hui-Ling Chou, Juraj Gottweis, Nenad Tomaev, Yun Liu, Alvin Rajko- mar, Joëlle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620:172 180, 2022. URL https:
semanti .or Pttp:/CoopusI1::524459555 255124952
[32] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. (631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL htt.t:htpp//eee.e http://wwwwwwwwe..rggg..../ wwwwwwwwww.b........ hwtpp//.....g..aooooo/////..e... Dttp:lllooyyyy111111111
[33] Erik M. van Mulligen, Annie Fourrier-Regatt David Gurwitz, Mariam Molokhia, Ainhoa Nieto, Gianluca Trifiro, Jan A. Kors, and Laura I. Furlong. The eu-adr corpus: Annotated drugs, diseases, targets, and their relationships. Journal of Biomedical Informatics, 45(5): 879-884, 2012. ISSN 0464. doi: Ccc.Be/c/aaaraaaa////////1110111 //www. www.enediieeer...c.............. com/ http:/ccccnccccceeeeeeeee http://ciccccccceceeeceee and Natural Language Processing in Pharmacogenomics.
URLhttps Text Mining
[34] Chen Xu, Chen Siheng, Yao Jiangehao, Zheng Huangjie, Zhang Ya, and W. Tsang. Ivor. Learning on attribute-missing graphs. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2022.
[35] Chen Xu, Pan Yuangang, Tsang Ivor, and Zhang. Ya. Learning node representations against perturbations. Pattern Recognition, 145: 2024.
[36] Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng Ann Heng, and Wai Lam. Unveiling the generalization power of fine-tuned large language models. ArXiv, abs/2403.09162, 2024. URL BRRLhttps::/ //ap:: http://ppp.ssmmaaccccccosooooooe orgg
org/ Corpus 2683885476
[37] Cui Yin, Jia Menglin, Lin Tsung- Yi, Song Yang, and Belongie. Serge. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[38] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020. URL htttttttt//xxiiii /wr.vv.xvvvvv. orgg/bs/////eeeo/oo/...o.ee wrrg/abs/2001.. 2001. 06782
[39] Wang Yu-Xiong, Ramanan Deva, and Hebert. Martial. Learning to model the tail. Advances in Neural Information Prucessing Systems, 2017
[40] Shi Yubin, Chen Yixuan, Dong Mingzhi, Yang Xiaochen, Li Dongsheng, Wang Yujiang, P. Dick Robert, Lv Qin, Zhao Yingying, Yang Fan, Lu Tun, Gu Ning, and Shang Li. Train faster, perform better: Modular adaptive training in over parammetteee models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[41] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXiv, abs/2303. 182223 2023. URL https: https///ap..emmatiicccoooo www.ssimicaooola.....m...
org/CorpusIDD : 257900969
[42] Jiawei Zheng, Hanghai Hong, Xiaoli Wang, Jingsong Su, Yonggui Liang, and Shikai Wu. Fine- tuning large language models for domain-speciffe machine translation. ArXiv, 150611 2024. htt./httt///aapppt hattps::/apl.semaaiiaehetee www.:esseaaaaaiooooooeeeei wtt.aacccc......rrrrrrrrssllllaa wwwp/rrrr//oooooo :2D8:26777771
[43] Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset. arXiv e-prints, art. arXiv:2104.08611, 1, April 2021. doi: 10.485500arXxi.21144.0666
13

---

[44] Zhou Zhihan, Yao Jiangchao, Wang Yan-Feng, Han Bo, and Zhang Ya. Contrastive learning with boosted memorization. In International Conference on Machine Learning. 2022.
[45] Zhou Zhihan, Yao Jiangchao, Hong Feng, Zhang Ya, Han Bo, and Wang. Yanfeng. Combating representation learning disparity with geometric harmonization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[46] Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, arXiv preprint arXiv:2309.14316, 2023.
14

---

Appendix
A Experiments
Sable the datasets used in our experiments. Table shows the hyperparameters used in our implementations. We use machine with NVIDIA GeForce RTX 3090 GPUs with 24GB GPU memory as our experiment platform. Pretraining costs about 24 hours.
Table 4: Datasets used for experiments
Pretraining dataset
Description
Wikipedia (13)
Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump with one split per language. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections.
legal(
In collaboration with Ravel Law, Harvard Law Library digi- tized over 40 million U.S. court decisions consisting of 6.7 million cases from last 360 years into dataset that widely accessible to use.
PubMed(
PubMed comprises more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books.
Finetuning task
Description
Overruling ([43)
A law dataset corresponds to the task of determining when a sentence is overruling prior decision.
Caschold 4
Case Holdings On Legal Decisions, comprising over 53,000+ multiple choice questions to identify the relevant holding of cited case.
GAD((6)
A relation extraction dataset, decide if gene is related to specific disease.
EUADR 33
Another relation extraction dataset, decide if gene is related to specific disease.
SST2 32
The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment.
Table 5: Hyperparameters of Models
Hyperparameters
BERT-based
GPT-based
FFN modules
Attention modules
attention heads
12
our-strategy-beet based layers
transformer layers
12
12
Hidden dimension size
768
768
Droupt
0.1
0.1
Attention dropout
0.1
0.1
Sequence length
128
256
Batch size
100
64
Max steps
36k
300k
Learning rate decay random seed used
Cosine
Cosine
14, 24
22, 80
By monitoring the validation loss of the pretraining dataset(Figure (6. we show the Catastrophic
Forgetting problem of the BERT model and its MOE method the domain-speciffe finetuning phase. Despite our attempts various combinations of generic data and domain-specinf data during domain
15

---

finetuning, the best outcome among these still resulted in a decline in model performance on domains unrelated its fine-tuning, indicating limitation the generalizability of the adapted model. As domain-specifee finetuning proceeds, the validation loss of pretraining dataset has significant rise and stays well above the convergence position of pretraining.
4.2
Validation Loss
4.0
3.9
BERT/legal
McE/egal
McE/med
3.7
BERT/med
200
400
600
800
1000
Steps
Figure 6: The validation loss of the pretraining dataset during the domain-specific finetuning phase.
Our training process strictly followed the basic framework of pretraining and domain-speciie pretraining, and we didn't control the occurrence of forgetting, focusing However as shown the appendix, we monitored the forgetting phenomenon using the test loss on pretraining data. Specifically, BERT//med(frrther trained in medical), exhibited more severe forgetting issuc, with test perplexity on pretraining data increasing by 24.33, compared to smaller increase of 8.41 for (ERr/eegll(further trained in legal). After correcting this imbalance by utilizing early-stop to select checkpoints where each model showed similar degrees of forgetting but not the best performance, the results came out that each domain-fineture finetuned model outperformed on its respective domain tasks. This further underscores the challenges posed by the fine-tuning stage, affirming the value of our approach in not requiring domain-speciff pretraining.
the best performance.
Table 6: Checkpoints selected with early-stop where two models show the same degree of forgetting.
Models
Casehold
Overruling
GAD
EUADR
BERT/legal
50.26 49.15
85.83 85.00
62.01
79.15
BERT/med
64.58
81.97
B Limitations Discussions
Although we use the method mixing small-scale domain-specific datasets into pretraining data to simulate the long-tail distribution in those huge corpora, we cannot fully simulate the extremely rich pretraining data used on LLMS due to the limited training resources.
C Discussions
Large Scale The maximum model scale presented the experiment section this paper is 330M GPT. So, discuss what if our proposed CSE method is applied to larger scale.
16

---

Model Size: Larger models have greater learning capacities, which can enhance the GC for long-tail knowledge given a fixed training data size.
Window Size: A larger window size allows the model to capture longer-range relationships within each training sequence, potentially highlighting more subtle distinctions and leading to a more long-tail distribution.
Training Data Size: Larger data sizes and larger window sizes might result in higher proportion of common data relative long-tail clusters. The increased presence of com- mon data and the potential subdivision of long-tail clusters into smaller groups could, paradoxically, reduce the GC for long-tail knowledge.
Performance Common Datasets Additionally, we have included results on general knowledge tasks to confirm that our dedicated long-tail learning method does not compromise general knowledge acquisition.
Table Results general tasks on BERT with the same small-scale setting the paper
Task
Domain
Freq. Score
Baseline(tuned)
MoE(tuned)
Ours(w/o tune)
COLA QNLI
general
0.389
69.22
69.32
69.42
general
0.325
65.92
65.07
66.04
MRPC
general
0.343
72.06
70.83
72.06
QQP
general
0.380
70.64
69.85
70.18
SST2
general
0.327
66.86
64.79
67.98
average
general
68.94(-0.20)
67.97(-1.26)
69.14
Table 8: Results of general tasks tested on GPT 330M trained with 20B tokens
Task
Domain
Freq. Score
Baseline(tuned)
MoE(tuned)
Ours(w/o tune)
COLA
general
0.389
69.10
69.10
69.20
QNLI
general
0.325
60.17
60.06
59.72
MRPC
general
0.343
70.18
71.75
71.98
QQP
general
0.380
73.28
74.47
75.95
SST2
general
0.327
74.50
72.03
76.00
average
general
69.45(-1.12)
69.48(-1.09)
70.57
Experiments on Pre-trained Model We also conducted experiments on a pre- pre-trained 110M scale model, wherein all methods continue training from single pre- teairraind checkpoint. The outcomes are presented the following table. Results show that directly applying our method to pre-trained model still yields superior performance compared to the baseline and MoE models.
Table 9: Results of strategies applied pre-trained model
Models
Overruling
Casehold
GAD
EUADR
SST2
average
BERT/med
85.00
51.11
70.84
89.86
64.11
72.18 + 0.21
BERT/legal
85.83
51.21
66.08
88.45
61.58
70.63 + 0.17
MoE/med
85,83
49.91
72.18
90.42
63.42
72.35
0.18
MoE/legal
86.67
50.83
69.44
89.86
62.16
71.88
+ 0.17
Ours/MoA
86.67
51.11
73.17
92.96
63.99
73.58 + 0.19
Ours/MoF
86.67
50.77
73.25
91.83
63.65
73.23 + 0.18
Using both MoA and MoF We tried this architecture using both MoA and MoF and tested it in the same small-scale setting reported the paper, and found that MoA+MoF yields improvements the overall performance compared to MoA/MoF only.
17

---

Table 10: Updated results of strategies applied on BERT
Models
Pretrain ppl
Overruling
Caschold
GAD
EUADR
SST2
average
MoA MoF
28.25
86.62
50.94
72.90
90.09
66.60
73,43
34.64
89.10
50.82
71.65
91.23
67.98
74.16
MoA+MoF
28.25
87.50
50.83
79.87
93.80
67.09
75.82
Analysis of Sentence Frequency Distribution and Cluster Details
D.1 Sentence Frequency Distribution and Gradient Consistency
We analyze the sentence frequency distribution using 2/3-gram averages, comparing gradient consistency across different training stages. Figure 7(a) and Figure 7(b) illustrate the frequency distribution of sentences based on their 2-gram and 3-gram patterns, respectively. The results are consistent with the I-gram analysis, confirming that despite its simplicity, the 1-gram method remains effective in capturing sentence frequency trends. The gradient consistency results align with these findings, reinforcing the robustness of this approach.
percentage of sentences
gradient consistency
percentage of sentences
gradient consistency
0.30
0.8
0.4
0.8
gradient consistency
gradient consistency
0.6
0.3
0.6
0.20
0.4
0.2
3
0.10
0.1
0
0.08
0.06 0.04
0.02
0.00
0.020 0.015 0.010 0.005 0.00
Average 2-gram frequency
Average 3-gram frequency
(a)
(b)
Figure 7: a) 2-gram sentence frequency distribution. b) 3-gram sentence frequency distribution.
D.2 Cluster Details and Sentence Content
Figure 8 shows the detailed content of sentences within each small cluster. For example, in the 'Java Network Debug' cluster, terms such as HTML, Javascript, and <link> frequently appear, indicating strong domain coherence. This example highlights how specific topics tend to cluster together, demonstrating the effectiveness of the clustering process.
Furthermore, we identified low- owwffreuency sentences that are scattered across clusters, lacking clear semantic correlation. These sentences often include irregular content such as misprinted formulas or non-Engee EnnlEnglish text, which prevents them from forming coherent clusters. Figure ( also illustrates the distribution and content of these low-frequency sentences, providing insight into their irregular placement across clusters.
D.3 Methodology for Long-Tail Data Selection
We applied the Elbow method determine threshold of 9.37% the curve of domain proportions, classifying any domain with data proportion at or below this level as a long-tail domain. By plotting the curve of Wikipedia domains and their corresponding proportions, we identified the point (9.37%) where the slope changes significantly, marking the transition from the head of the distribution to the tail. Following this, randomly selected sentences from these long-tail domains for further analysis.
18

---

EL cooraquiig oerer fevesiii vizie Eimeet 114
Intira tillors por wil vesppe
3.. Ciebo Choeeege planig Kaaee la Cleve EllA. Aemaaaaaienn boogl
levo Mil wiiie
inenctans,, Lalber..... tupilig
nevi no metiaiions wedded
Malliggeri aogeeppllie popeess
Rud metts
Eaunlily wid
thegh
Arg. buspt/36
of
ox high darrp-nt mi anly fartii anulle
Rosuni sadee Candlne
rmpard
Arg A 0.333
Capetta Lionns
unit bus usdician la slsmard
Diviidaa Cleju Bustioci iiDi ditaat
Mika pondle caaapi -lile denaeo -ladd decactioon guu
peospam sasclable IOHMASiaaa oduar 3M0%
rog kr supor aiChpar's as Pau ICSC...
bcoakiet
Rib
Figure 8: Detailed content of sentences within clusters, showing high domain coherence (e.g., 'Java Network Debug' cluster) and the distribution low-frequency, irregular sentences.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: claim clearly in Abstract and Introduction I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We write them in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution
19

---

is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We achieved this part in the Analysis 2 and the relevant assumptions are indicated in the Reference.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The Table the Appendix A shows the random seeds used in our experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
20

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release our source code and detailed instructions for reproducing our results upon acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp cc/
public/ Saciic/ggiies/CCe (CodeSSummssiooooooolo.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips. cc/
httpcccpppppcccccecccc.ce.l.cclc http:ppllllcceele wwwpc//ggieeeeeeeee Sodeee/issssSSSiiiiiiiaaaaaa ssioiioliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We show our experimental setting/details both in our Experiments 5 and the Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
21

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We show the standard deviation of accuracy in Table and Table 2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We show these in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have checked it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
22

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
23

---

Answer: [Yes]
Justification:
We cite the creators and introduce assets in the Appendix A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's ' creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "0P8DgplT3Y", "reviewer": "Reviewer_bZcS", "summary": "The paper focuses on the problem of modeling long-tail domain knowledge in language models, which can be missed during the general-purpose pretraining. While most approaches capture long-tail domains as a second domain-specific pretraining step, the paper proposes a cluster-guided method that encourages the model to learn long-tail domain knowledge during the initial pretraining stage. Specifically, the paper first conducts analysis to show the challenges in learning long-tail domain data and then uses these findings to propose a cluster-guided Sparse Expert (CSE) architecture, which routes sequences from long-tail clusters into separate expert modules (either attention or FFN modules of the last model layer). The proposed method is evaluated on a simulated setup with common domains drawn from wikipedia data and long-tail domains drawn from legal and medical datasets.", "questions": "Why is k-means clustering used for the analysis but DBSCAN is used for the main method? What is the value of K in k-means?\n\nCan the authors clarify the reasoning behind routing at sequence-level instead of token-level? \n\nAre there any recommendations for adapting to more frequent (instead of long tail) domains?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and easy to follow. The authors did a great job presenting a coherent analysis to motivates the design choices behind their proposed CSE method. CSE is promising as it alleviates the need to resort into a second training stage, which would require careful data-replay or regularization. CSE is also empirically shown to be effective for modeling legal and medical domain knowledge.", "weaknesses": "The experimental results raise questions about the effectiveness of CSE and baselines. \n1. The baseline results in Tables 1 and 2 are surprising as, for example, BERT/med (further trained on medical) would be expected to outperform BERT/legal (further trained on legal) on medical datasets, but the two models are instead performing very similarly to each other, which suggests potential issues with baseline tuning. \n1. It is not clear if the proposed method retains its general knowledge as there are not sufficient benchmarks to judge general knowledge (e.g., WebQS, TriviaQA). \n\nThe question \"is finetuning indispensable to LMs?\" and argument that \"Finetuning improves LMs’ domain performance by providing a second lesson, which could be avoided if the first (pretraining) is appropriately delivered.\" could be misleading as finetuning often serves multiple roles, including instruction tuning (as in SFT). To avoid misunderstanding, I would suggest being more precise, e.g., by changing \"finetuning\" into \"domain-specific pre-training\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and expertise you have invested in reviewing our submission. Below, we outline our responses to the specific points raised in your reviews.\n\n> Q1 The baseline results in Tables 1 and 2 are surprising as, for example, BERT/med (further trained on medical) would be expected to outperform BERT/legal (further trained on legal) on medical datasets, but the two models are instead performing very similarly to each other, which suggests potential issues with baseline tuning.\n\n**A1** Thank you for your comment. Upon investigation, we discovered that the issue stemmed from catastrophic forgetting during the fine-tuning stage. Our training process strictly followed the basic framework of pretraining and domain-specific pretraining, and we didn't control the occurrence of forgetting, focusing more on the best performance. However, we monitored the forgetting phenomenon using the test loss on pretraining data. Specifically, BERT/med(further trained in medical), exhibited a more severe forgetting issue, with test perplexity on pretraining data increasing by 24.33, compared to a smaller increase of 8.41 for BERT/legal(further trained in legal). After correcting this imbalance by utilizing early-stop to select checkpoints where each model showed similar degrees of forgetting but not the best performance, the results came out that each domain-finetuned model outperformed on its respective domain tasks. This further underscores the challenges posed by the fine-tuning stage, affirming the value of our approach in not requiring domain-specific pretraining.\n\n| Models     | Casehold | Overruling | GAD | EUADR |\n|------------|-----------|------------|-----|-------|\n| BERT/legal | **50.26** | **85.83** | 62.01 | 79.15 |\n| BERT/med   | 49.15     | 85.00     | **64.58** | **81.97** |\n\n> Q2 It is not clear if the proposed method retains its general knowledge as there are not sufficient benchmarks to judge general knowledge (e.g., WebQS, TriviaQA).\n\n**A2** Thank you for your comment. To address this, we've conducted experiments on general knowledge tasks, the results of which are detailed in the global rebuttal. These results demonstrate that our methods not only match but sometimes even surpass the performance of baseline models on a variety of general tasks. This is because our method enhances the model's capability to learn long-tail data by segregating the learning process between long-tail and non-long-tail data, allowing each category to be learned effectively without compromising one at the expense of improving the other.\n\n> Q3 Why is k-means clustering used for the analysis but DBSCAN is used for the main method? What is the value of K in k-means?\n\n**A3** Thank you for your comments. Initially, we used the K-means clustering algorithm with the elbow method to identify the optimal number of clusters (K). However, the elbow method's reliance on subjective judgment is a limitation. We then tested DBSCAN, which automatically determines the number of clusters based on density and is easier to set parameters compared to K-means. Consequently, we opted to employ DBSCAN in our solution design. In the global rebuttal Figure 1, we also presented cluster number and cluster distance calculated with DBSCAN, demonstrating that the choice of clustering algorithm does not impact the statements presented in Section 2.\n\n> Q4 Can the authors clarify the reasoning behind routing at sequence-level instead of token-level?\n\n**A4** Thank you for your comments. The definitions, analyses, and conclusions regarding long-tail data presented in Section 2 are all at the sequence level; consequently, routing is also performed at this same level. The rationale for defining and analyzing long-tail at the sequence level stems from the fact that when we discuss long-tails, we are essentially referring to semantics rather than individual words or tokens. Such semantics can only be observed within the context provided by sequences as a whole.\n\n> Q5 Are there any recommendations for adapting to more frequent (instead of long tail) domains?\n\n**A5** Thank you for your question. As discussed above, since data from frequent domains dominate the gradient direction of model optimization, our method separates the parameter of learning long-tail and frequent domains to improve the model's ability to capture long-tail domain knowledge. If we expect to improve the ability on frequent domains, we may need to design methods to analyze the gradient consistency within frequent clusters, and conduct for finer distinctions within the frequent domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OajYKnf5iu", "reviewer": "Reviewer_1S4s", "summary": "The paper presents an innovative approach to address the challenge of finetuning language models (LMs) for domain-specific tasks. The authors argue that the traditional pretraining-finetuning paradigm is suboptimal due to the high cost and time consumption of finetuning. To tackle this, the authors propose the Cluster-guided Sparse Expert (CSE) layer, which is designed to enhance the model's awareness of long-tail knowledge during the pretraining phase, eliminating the need for finetuning.\nThe CSE layer leverages the intuition that semantically similar data points are closer in the embedding space. It employs an efficient clustering mechanism during pretraining to group long-tail domain data into distinct clusters and assigns additional experts to memorize this knowledge. The paper demonstrates that with this strategy, the language models can achieve superior performance on various downstream tasks compared to traditionally pretrained-finetuned LMs.", "questions": "1. In Figure 3(c), the legend is lines 1-12. Maybe the author wants to show layers?\n2. Can you conduct computation time and memory analysis of the proposed methods as you claim that fine-tuning is time-costing.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I'm not that familiar with the moe and pretraining field, but I like the motivation and idea of this method despite the difficulty of verifying the idea in larger models in academic institutions. I agree with the author's claim and the methods proposed a new way for the future moe training.", "weaknesses": "1. The improvement in the GPT style model is subtle compared to the baseline moe method, which may undermine the advantage of the proposed methods.\n2. The experiments are not sufficient, maybe more tasks should be tested. Is it possible to conduct the experiments on an pre-trained model, which would make the contribution stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and expertise you have invested in reviewing our submission. Below, we outline our responses to the specific points raised in your reviews.\n\n> Q1 The improvement in the GPT style model is subtle compared to the baseline MoE method, which may undermine the advantage of the proposed methods.\n\n**A1** Thank you for your comments. The experiments presented in the paper are conducted with 130M models, and we've conducted larger-scale experiments with 330M GPT-style models and detailed the results in the global rebuttal. Results indicate that our approaches match and outperform the baseline MoE model, achieving an average accuracy improvement of 3.26%.\n\n> Q2 The experiments are not sufficient, maybe more tasks should be tested. Is it possible to conduct the experiments on a pre-trained model, which would make the contribution stronger?\n\n**A2** Thank you for your comments. To further substantiate the efficacy of our method, we've conducted experiments on a wider array of tasks drawn from diverse domains using the 330M GPT-style model. As detailed in the global rebuttal, we added domain-specific tasks from the academic, environmental, and financial sectors to demonstrate the generalization capability of our method across multiple domains. Additionally, we have included results on general knowledge tasks to confirm that our dedicated long-tail learning method does not compromise general knowledge acquisition.\n\nWe also conducted experiments on a pre-trained 110M scale model as you suggested, wherein all methods continue training from a single pre-trained checkpoint. The outcomes are presented in the following table. Results show that directly applying our method to a pre-trained model still yields superior performance compared to the baseline and MoE models.\n\n| Models     | Overruling |  Casehold |    GAD    |   EUADR   |   SST2    |  average  |\n|------------|------------|-----------|-----------|-----------|-----------|-----------|\n| BERT/med   |    85.00   |   51.11   |   70.84   |   89.86   | **64.11** |   72.18   |\n| BERT/legal |    85.83   | **51.21** |   66.08   |   88.45   |   61.58   |   70.63   |\n| MoE/med    |    85.83   |   49.91   |   72.18   |   90.42   |   63.42   |   72.35   |\n| MoE/legal  |  **86.67** |   50.83   |   69.44   |   89.86   |   62.16   |   71.88   |\n| Ours/MoA   |  **86.67** |   51.11   |   73.17   | **92.96** |   63.99   | **73.58** |\n| Ours/MoF   |  **86.67** |   50.77   | **73.25** |   91.83   |   63.65   |   73.23   |\n\n> Q3 In Figure 3(c), the legend is lines 1-12. Maybe the author wants to show layers?\n\n**A3** Thank you for your comments. We'll correct it to \"layers 1-12\" instead of \"lines\" in our revised document.\n\n> Q4 Can you conduct computation time and memory analysis of the proposed methods as you claim that fine-tuning is time-costing?\n\n**A4** Our method introduces negligible overhead in time and memory.\n\n- **Memory** The additional memory consumption incurred by our method compared with an MoE who has an equivalent number of experts is attributable to the storage requirements for $k$ distinct cluster centers, each endowed with a dimension of $d'$, which is $O(kd')$ in total. In most practical scenarios, the $k$ is lower than a hundred, and $d'$ tends to be around a hundred. Thus $O(kd')$ is negligible when compared with a neural network with often millions of parameters. Moreover, during a forward pass, our model mirrors both the Mixture of Experts and baseline architectures in the number of parameters activated, thus not introducing a memory burden.\n\n- **Time** Throughout the training phase, the additional computation introduced by our method, relative to both the baseline and MoE, is the cluster initialization step. The time complexity of this operation will be $O(n\\log n d')$ and is bounded by $O(n^2d')$ for the worst case, where $n$ represents the number of sampled points. Notably, during subsequent inference stages, the quantum of parameters activated per iteration by our algorithm aligns identically with that of both MoE and the baseline.\n\nSince MoE and the baseline necessitate an additional fine-tuning phase, it becomes evident that the time cost of this phase far surpasses the $O(n^2d')$ complexity of cluster initialization. To provide more straightforward data, we list the wall-clock time of training & finetuning 330M GPT on 20B tokens with 2 NVIDIA GeForce RTX 3090 GPUs each in the table below.\n\n| Model       | Baseline GPT   | MoE   | Ours |\n|-------------|---------------|-------|------|\n| Time(hrs)   | 177            | 205   | 160  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qay4qIsc2b", "reviewer": "Reviewer_YnWw", "summary": "This paper proposes a novel approach called Cluster-guided Sparse Experts (CSE) to improve language models' ability to learn long-tail domain knowledge during pretraining, potentially eliminating the need for domain-specific finetuning. This study introduced CSE layers that cluster semantically similar long-tail data and assign it to specialized experts for dedicated learning. Empirical  experiments demonstrate that CSE-based models outperform regularly pre-trained and finetuned models on various downstream tasks without domain-specific finetuning.", "questions": "No", "limitations": "No", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The CSE method offers an innovative solution to the challenge of learning long-tail domain knowledge in language models. The paper provides an in-depth analysis of gradient behaviors, embedding spaces, and cluster dynamics to support its claims. This approach could significantly reduce the need for costly and time-consuming domain-specific finetuning.", "weaknesses": "Typo: line 189/300 should be Cluster rather than Clsuter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and expertise you have invested in reviewing our submission. Below, we outline our responses to the specific points raised in your reviews.\n\n> Q1 Typo: line 189/300 should be Cluster rather than Clsuter.\n\n**A1** Thank you for your comments. It has been corrected to be _clusters_ in our revised document.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DC7prp3nP7", "reviewer": "Reviewer_4NSY", "summary": "The authors propose to add a simple and efficient Cluster-guided Sparse Expert (CSE) layer to Language Models to improve their capability on long-tailed knowledge. The authors demonstrate that pretraining LMs using CSE leads to better performance on domain specific tasks than vanilla pretraining and suggest that the resource intensive finetuning step may not be necessary if the CSE layer can learn long-tailed knowledge properly.", "questions": "1. I understand that doing this experiment for bigger models on academic resources is a tough ask. But I still ask if the authors have any intuition on what might happen if this method is applied to larger models? My second question is in a larger model, do the authors think the GC for long-tailed knowledge will be even lower? \n\n2. Can this method be applied to both FFN and Attention modules. I am curious since in some cases MoA is better and in some cases MoF is better and was wondering whether using both would lead to the best results.", "limitations": "The authors have address limitations but not in the main paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and concise while providing key insights from a research perspective. The results are also quite strong which demonstrate in all cases using MoA or MoF is better than domain specific finetuning.\n\n2. section 2 is especially quite well written and provides unique perspective into the working of language models. I especially enjoyed reading the gradient consistency analysis which is quite intuitive and easy to understand.\n\n3. I believe if the big players are able to use this method and this works for larger models, it could benefit the GPU poor a lot as they dont have to finetune their models and just need to worry about inference. This method has the potential to ensure LLMs can reach a wider audience.", "weaknesses": "Small nitpicks\n\nTypo in line 189: Should be \"Cluster\" and not \"Clsuter\"\n\nTypo in Table 1, 4th row: Should be \"legal\" and not \"lgeal\"\n\nIts better if the limitations are in the main paper and not in appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1 Typo in line 189: Should be \"Cluster\" and not \"Clsuter\"; Typo in Table 1, 4th row: Should be \"legal\" and not \"lgeal\".\n\n**A1** Thank you for your comments. In our revised version, the error on line 189 will be corrected to _Cluster_, and the typo in Table 1, fourth row, will be updated to _legal_.\n\n> Q2 Its better if the limitations are in the main paper and not in the appendix.\n\n**A2** Thanks for your suggestion. We agree that including the limitations in the main paper rather than the appendix will enhance clarity for readers. We will make this adjustment in our revised document.\n\n> Q3 I understand that doing this experiment for bigger models on academic resources is a tough ask. But I still ask if the authors have any intuition on what might happen if this method is applied to larger models? My second question is in a larger model, do the authors think the GC for long-tailed knowledge will be even lower?\n\n**A3** Thank you for your comments. Our latest experiments, which included 330M GPT-style models and tasks from more domains as detailed in the global rebuttal, demonstrate that our method continues to outperform baseline methods at a larger scale. \nYour question about the gradient consistency (GC) of long-tailed knowledge in larger models is crucial, as it directly relates to how our method would adapt to larger scales. We've identified three key factors influencing this:\n\n- **Model Size** Larger models have greater learning capacities, which can enhance the GC for long-tail knowledge given a fixed training data size.\n\n- **Window Size** A larger window size allows the model to capture \nlonger-range relationships within each training sequence, potentially highlighting more subtle distinctions and leading to a more long-tail distribution.\n\n- **Training Data Size** Larger data sizes and larger window sizes might result in a higher proportion of common data relative to long-tail clusters. The increased presence of common data and the potential subdivision of long-tail clusters into smaller groups could, paradoxically, reduce the GC for long-tail knowledge.\n\n> Q4 Can this method be applied to both FFN and Attention modules? I am curious since in some cases MoA is better and in some cases, MoF is better, and was wondering whether using both would lead to the best results.\n\n**A4** Thank you for your suggestion. We tried this architecture and tested it in the same small-scale setting reported in the paper, and found that MoA+MoF yields improvements in the overall performance compared to MoA/MoF only.\n\n| Models | Overruling | Casehold | GAD      | EUADR     | SST2      | average  |\n|--------|------------|----------|----------|-----------|-----------|----------|\n| MoA    | 86.62      | **50.94**| 72.90    | 90.09     | 66.60     | 73.43    |\n| MoF    |  **89.10** | 50.82    | 71.65    | 91.23     | **67.98** | 74.16    |\n| MoA+MoF| 87.50      | 50.83    | **79.87**| **93.80** |  67.09    | **75.82**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xl1PCdszJo", "reviewer": "Reviewer_P2qX", "summary": "This paper first shows that infrequent sentences tend to have gradients dissimilar to the gradient of the majority data points. This paper thus proposes a variant of mixture-of-expert (MoE) module,  Cluster-guided Sparse Expert (CSE), where inputs are routed to the k-mean clusters they are mostly similar to. They experiment with BERT and GPT and show that their proposed CSE moduled outperform Switch-MoE.", "questions": "Do you add a classification head and fine-tune the whole model?", "limitations": "In Appendix B.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "1. Albeit the simplicity of the way they compute sentence frequency, they are still able to show that less frequent sentences have gradients/Jacobian less similar to the gradient of majority.\n2. The proposed CSE is simple and does not incur much extra computational overhead.", "weaknesses": "**The Experiment Setup**\n\n1. First, the setup is not clear. I am not sure if this work follows the pretrain-and-fine-tune paradigm for both BERT and GPT. \n2. The more common use case of MoE is for language modeling, which is not covered by this work.\n3. The baseline should include an MoE model pretrained with Wikipedia.\n\n**Analyses**\n\n1. The sentence frequency score is just the average of 1-gram frequency, which may be too simple.\n2. Compared with Figure 2a, the consistency in Figure 3a, which I suppose is the result of using CSE, does not seem to be better. This contradicts their motivation for using CSE.\n3. The increased cluster distance shown in Figure 3c may not be very meaningful, as the distance can be larger simply because the norm of the representations is larger.\n2. Some arguments are not substantiated. For example:\n    1. line 152:  “Detailed analysis shows high domain coherence within the smaller clusters each comprising sentences closely related to specific domains.” The detailed analysis is not specified.\n    2. line 157: “Further analysis of sentences with frequency scores below 0.2 shows their random distribution across clusters, suggesting these extremely infrequent sentences may serve as noise in the learning process.” The meaning of “random distribution” is unclear, and the details about the analyses are also absent. \n3. It is not that surprising that samples in the same cluster have similar gradients, because if $x \\approx y$ then as long as $f$ is smooth in a certain way, we have $f’(x) \\approx f’(y)$.\n\n**Presentation/Writing**\n\n1. Section 2.2 analyzes their CSE module, but the CSE module is formally introduced in Section 3, which makes the paper hard to follow.\n2. What are line 1 to line 12 in Figure 3c?\n3. The font size of the text in the figures is too small.\n4. The authors should discuss why they choose the testing datasets and the domain of those datasets in the main text.\n5. There are many unsubstantiated or unclear sentences. For example:\n    1. line 128: “the optimization requirements for long-tail sentences are significantly overlooked under standard pretraining condition” not substantiated\n    2. line 146:  the model’s inability to capture the learning dynamics of long-tail data begins early in the training process\n    3. line 148: “Our CSE layer capitalizes on the clustering structure at the point where the GC value peaks” I can’t understand.\n\n**Misc**\n\nThe title “Once Read is Enough” seems to be over-claiming.\n\nThe authors may want to read/cite this paper\n- On the Benefits of Learning to Route in Mixture-of-Experts Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review of our paper. Our key contribution addresses the challenge of learning long-tail domain data during the pretraining stage, substantially reducing the need for an expensive, labor-intensive second domain-specific pretraining stage in practical applications.\n> Q1 ..setup is not clear\n\nBoth BERT and GPT undergo both pretraining and domain-specific pretraining, while our method only pretraining. All models then serve as embedding generators with all params frozen for downstream tasks.\n> Q2 ..language modeling not covered\n\nOur BERT-based MoE uses MLM for pretraining, while the GPT-based MoE employs causal language modeling. All downstream tasks are NLP tasks. We are unclear about the specific meaning of 'language modeling' you mentioned and appreciate further clarification.\n> Q3 .. MoE pre-trained with Wikipedia.\n\nWe compared MoE with 1) only pretraining 2) additional domain-specific pretraining and 3) our method. Results show that while domain-specific pretraining boosts baseline performance, our single-stage pretraining method surpasses both, efficiently learning long-tail domain.\n\n|Models|Overruling|Casehold|GAD|EUADR|SST2|avg|\n|-|-|-|-|-|-|-|\n|MoE/med|91.25|50.11|72.77|83.38|72.03|73.91|\n|MoE/legal|91.60|49.68|72.66|83.38|71.97|73.86|\n|MoE|91.33|49.10|71.70|83.10|72.28|73.50|\n|Ours/MoA|91.68|50.70|71.75|**85.91**|74.61|74.93|\n|Ours/MoF|**93.33**|**51.26**|**73.30**|85.63|**76.00**|**75.90**|\n> Q4 .. 1-gram frequency too simple.\n\nGlobal rebuttal Figure3 displays sentence frequency distribution using 2/3-gram averages, alongside gradient consistency. Results align with 1-gram analysis, confirming its effectiveness despite simplicity.\n> Q5 Compared with Figure2a, consistency in Figure3a..not..better..\n\nBoth figures were obtained on a baseline model. We'll clarify this in the main text.\n> Q6 cluster distance not meaningful..norm..larger.\n\nIn global rebuttal Figure2, we introduce a new metric—the ratio of cluster distances to cluster radii—to measure cluster structure clarity independent of norm values. This aligns with our observation that clusters become more distinct in deeper layers.\n> Q7 line 152:\"..high domain coherence within the smaller clusters..\" not specified.\n\nIn global rebuttal Figure4, we detail the content of sentences within each small cluster. For example, 'Java Network Debug' cluster frequently includes HTML, Javascript, and <link>, demonstrating high domain coherence.\n> Q8 line 157:\"..frequency scores below 0.2 shows random distribution..\"..random distribution unclear.\n\nThe term \"random distribution\" describes how extremely low-frequency sentences are irregularly scattered across clusters without clear semantic correlation. In Global rebuttal Figure4, we detailed the distribution and contents of these sentences, which often contains misprinted formulas or non-English.\n> Q9 not that surprising that samples in the same cluster have similar gradients.\n\nOur main contribution is that we are the first to apply gradient similarity to identify and tackle the challenges of learning long-tail data during pretraining, significantly reducing the need for subsequent domain-specific pretraining.\n> Q10 Section 2.2 analyzes their CSE module .. formally introduced in Section 3..hard to follow.\n\nOur goal was to ensure readers first understand the fundamental challenges of long-tail data learning, which sets the stage for appreciating the effectiveness of our method. We'll include a more detailed illustration of CSE in analysis section of revised version.\n> Q11 ..line 1 to line 12 in Figure3c?\n\nWe'll correct \"line\" to \"layer\".\n> Q12 The font size..in the figures is too small.\n\nWe'll increase the font size in figures.\n> Q13 ..discuss why..choose the testing datasets..domain of those datasets in the main text.\n\nOur research identified long-tail distribution as a prevalent phenomenon in many common knowledge datasets, posing a challenge for pretraining. We chose Wikipedia for pretraining because it mirrors this distribution pattern. We also chose specific domains within the long-tail distribution with publicly available training corpora and downstream tasks for further testing. A detailed explanation of dataset choices will be included in the main text in revised version.\n> Q14 ..line 128:\"optimization requirements for long-tail..overlooked ..\" not substantiated.\n\nExisting researches [A, B] has shown that gradient conflicts lead to suboptimal learning outcomes for the affected data. In standard pretraining, since the gradient descent direction is dominated by common data, the differences in gradient direction between long-tail and common data result in the optimization of long-tail data being overlooked.\n> Q15 ..line 146: model's inability..long-tail..begins early..\n\nGradient consistency indicates how much a model focuses on learning specific data. Figure3a in the paper reveals that the GC of long-tail data declines sharply at the beginning of the pretraining stage and remains low, suggesting that the model's inability to effectively learn from long-tail data starts early in the training process.\n> Q16 ..line 148:\"CSE layer..at..GC value peaks\"\n\nThe peak GC value of a baseline model signals when it struggles to learn long-tail data. We then introduce CSE layer at this point to improve the model’s ability to learn long-tail data in later phases.\n> Q17 title..over-claiming\n\nWe'll replace 'finetuning' with 'domain-specific pretraining' to narrow down the specific training stage we aim to reduce.\n> Q18 read/cite this paper ..\n\nWe find this paper relevant and will cite it.\n> Q19 Do you add a classification head and fine-tune the whole model?\n\nFor each classification task, we train a classification head with all other params frozen.\n\n**References**\n\n[A] T. Yu, et al. Gradient Surgery for Multi-Task Learning. NeurIPS 2020.\n\n[B] G. Shi, et al. RECON: REDUCING CONFLICTING GRADIENTS FROM THE ROOT FOR MULTI-TASK LEARNING. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

Dear AC and Reviewers,

We sincerely appreciate the time and expertise you devoted to reviewing our submission.

Given that experiments involving larger scales or a greater number of tasks is a shared concern among the reviewers, we have detailed the outcomes of such experiments in the global rebuttal. 

As listed below, we used larger models (330M GPT-style models trained with 20B tokens) and added more domain-specific tasks from the academic, environmental, and financial domains to demonstrate the generalization capability of our method. Additionally, we provided results on general knowledge tasks to show that our method does not impact general knowledge acquisition. The results indicate that our method, even without fine-tuning, consistently outperforms baselines by an average of 3-5\% in accuracy on domain-specific tasks while maintaining comparable performance on general tasks. This highlights the effectiveness of our approach.

In the pdf file, Figure 1 is for Reviewer bZcS and Figures 2,3,4 are for reviewer P2qX.

---
**Results of new tasks tested on GPT 330M trained with 20B tokens**

| Task           | Domain       | Freq. Score | Baseline(tuned) | MoE(tuned) | Ours(w/o tune)  |
|----------------|--------------|-------------|-----------------|------------|-----------------|
| chem-prot      | academic     | 0.207       | **36.25**       | **36.25**  | **36.25**       |
| MAG            | academic     | 0.324       | 63.22           | 64.91      | **65.47**       |
| rct-20k        | academic     | 0.261       | 76.95           | 78.28      | **80.15**       |
| climate detect | environment  | 0.376       | 78.94           | 79.90      | **80.26**       |
| climate sent   | environment  | 0.317       | 66.81           | 68.31      | **69.98**       |
| FPB            | financial    | 0.243       | 16.83           | 25.00      | **40.11**       |
| **average**    |**domain-spec**| -          | 56.50(-5.54)    | 58.77(-3.26) | **62.04**     |
| COLA           | general      | 0.389       | 69.10           | 69.10      | **69.20**       |
| QNLI           | general      | 0.325       | **60.17**       | 60.06      | 59.72           |
| MRPC           | general      | 0.343       | 70.18           | 71.75      | **71.98**       |
| QQP            | general      | 0.380       | 73.28           | 74.47      | **75.95**       |
| SST2           | general      | 0.327       | 74.50           | 72.03      | **76.00**       |
| **average**    | **general**  | -           | 69.45(-1.12)    | 69.48(-1.09) | **70.57**     |

---
**Results of general tasks on BERT with the same small-scale setting in the paper**

| Task        | Domain   | Freq. Score | Baseline(tuned) | MoE(tuned) | Ours(w/o tune) |
|-------------|----------|-------------|-----------------|------------|-----------------|
| COLA        | general  | 0.389       | 69.22           | 69.32      | **69.42**       |
| QNLI        | general  | 0.325       | 65.92           | 65.07      | **66.04**       |
| MRPC        | general  | 0.343       | **72.06**       | 70.83      | **72.06**       |
| QQP         | general  | 0.380       | **70.64**       | 69.85      | 70.18           |
| SST2        | general  | 0.327       | 66.86           | 64.79      | **67.98**       |
| **average** | general  | -           | 68.94(-0.20)    | 67.97(-1.26) | **69.14**       |

Meta-review: The paper introduces a novel Cluster-guided Sparse Expert (CSE) layer for language models that improves the capture of long-tail domain knowledge during pretraining, potentially reducing the need for domain-specific fine-tuning.  The idea of CSE layers is novel and the effectiveness of the idea in learning long-tail knowledge has been demostrated in the experiments.  In-depth analysis has been conducted.  Despite some concerns about presentation quality, the overall clarity of the paper and the coherence of the analysis are praised, making the paper's contributions understandable to the reader.  The authors address the reviewers' concerns in details in the rebuttal.
Decision: Accept (poster)