Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
Authors: Siyang Liu
Naihao Deng
Sahand Sabour
Yilin Jia
Minlie Huang
Rada Mihalcea
Venue: EMNLP 2023 Main (2023)
Keywords: task-adaptive tokenization
text generation
long-form generation
text segmentation
TL;DR: We design a task-adptive tokenizationer to boost text generation performance in downstream tasks --- taking mental health as an example field

Abstract: We propose task-adaptive tokenization\footnote{Our work will be publicly available upon acceptance.} as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. 
Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60\% fewer tokens. 
Preliminary experiments point to promising results when using our tokenization approach with very large language models.

Task-Adaptive Tokenization:
Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond
Siyang Liu
Naíhao Deng
Sahand Sabour
Yilin Jia
Minlie Huang
Rada Mihalcea
LLanuuage and Information Technologies Lab (LIT),
Department of Computer Science and Engineering, University of Michigan, Ann Arbor The CoAI group, Tsinghua University, Beijing (lsiyang, mihalcea)@@umich. eduu liusyang641 @ae44111mmal...mmmm com
Abstract
E.g.. "Social isolation and not having sense of purpose in life have been linked to mood disorders"
We propose task-adaptive tokenization! way to adapt the generation pipeline to the specifies of downstream task and enhance long-form generation in mental health. In- spired by insights from cognitive science, our task-adaptive tokenizer samples variable seg- mentations from multiple outcomes, with sam- pling probabilities optimized based on task- specific data. We introduce strategy for building a specialized vocabulary and intro- duce a vocabulary merging protocol that al- lows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments psycholog- ical question-answering tasks in both Chinese and English, we find that task-adaptive to- kenization approach brings a significant im- provement in generation performance while using up to 60% fewer tokens. Preliminary ex- periments point to promising results when us- ing our tokenization approach with very large language models.
Traditional tokenizer
Unique Seq (19 tokens)
look renn-eigcclicc vocabulary
sociall | iso I [a | lion | and | not
sutword segmentation
[ha] [ [ing [ [aree]] [f [pu [pu][ro]
[ow [in] [ill]]
represento tolkons long sequence
Task-adaptive tokenizer
Seq. tokens)
[socia]] ]| leolation ]| and [[not] [hs]
book apaciiiiiificcccc vocalbulary variable segmentattio
ving sense of purpose life
Seq 2 tokeree)
social isclation [ad I not having a sense of purpois |in life)
1. task spec-spectic takens
2. reduced sequenoe longth
Swe
sampling
unique
Token Seq
Impul lds
sense of//
social isolation
sense purpossis>
icclation
having
Decader
social
nol
having
sernee of
Figure I: A brief comparison between aaskaddapivve and traditional tokenizer. With task-adaptive tokeniza- tion, the same phrase appearing in different training in- stances may result in different tokenizations. For in- stance, Seq I and 2 illustrate two different token se- quences of the same input using a task-adaptive tok- enizer.
1 Introduction
During a time when mental health support is quickly growing (Hock et al., 2012; WHO, 2020), text generation techniques have been identified as potentially useful tools to assist mental health pro- fessionals (MHP) and provide mental health sup- port to those who share their struggles online (De- masi et al., 2020; Liu et al., 2021a; Sabour et al., 2022). With the help of such generation tools, counselors and social workers alike can increase their work efficiency and offer timely and expert feedback to those in need, especially for those in under-resourced areas where access to mental health services is challenging (Patel et al., 201 1; Brenman et al., 2014; Shen et al., 2022a).
help-seeking post, with an important requirement that the language used in the response aligns with the style used by MHPs (Sun et al.. 2021; Welch et al., 2020). The task poses significant challenges due to its domain-specific terminology, text struc- ture, and language style, which are often underrep- resented in the pre-training data. Recent research has demonstrated that the linguistic patterns of MHPs align with established counseling principles and behaviors (Lahnala et al., 2021). thus exacer- bating the difficulty of the task as pre-training data often soureed from laypeople (Kalla and Smith, 2023). Despite recent progress in large-scale lan- guage models (LLMs) (Touvron et al., 2023; Cui et al., 2023; Hu et al., 2022), we found that few-
The task of Psychological Question- Question-Answering Answering (PsyQA) is to generate a supportive response to a
'Our work is available at caw.ub.coo/Micchgantti P/t ask- eek-aadartiee..oeeiiaaio. tokenization.

---

shot abilities of these models lag behind in mental health applications due this misalignment with the MHP linguistic behaviors and style (see Appx A). This requires us to tune models to the down- stream domain. However, much of the current work focused on fine-tuning (Shen et al., 2020; Ji et al., 2022) still yields unsatisfying results due to the scarcity of professional data, which is often insufficient to capture all the task-specific intrica- cies.
1977; Wells, 1947), we advocate for using task- specific data and the developing variable segmen- tation for a downstream vocabulary as a pre-step for creating a task-adaptive tokenizer. (2) We con- struct a protocol for merging task-specific and pre- trained vocabularies, allowing for fine-tuning in- puts to be sampled from multiple tokenization re- sults. (3) We propose simple yet effective ini- tialization mechanism to alleviate the difficulty of learning representations for new tokens unseen during pre-training. Through thorough experi- ments the PsyQA task in both Chinese and English, demonstrate the significant improve- ments achieved by our task-adaptive tokenization approach. Notably, we achieve these enhance- ments while utiliziig 60% fewer tokens compared to expressing equivalent content length. addi- tion, we show that our tokenization brings signif- icant improvement in 7B LLaMA models, which suggests that our method is effective regardless of the model size and can unlock additional perfor- mance despite the booming era of LLMs.
In this paper, we introduce aask-adaptive tok- enization as a strategy to adapt a model to a down- stream task. Rather than focusing the adaptation effort on post-training fine-tuning, which has been the typical approach used in recent work (Shen et al., 2020, 2022b; Sun et al., 2021; Lee et al., 2021), we propose to make task-specific adjust- ments in the way the language is segmented during the tokenization stage (see motivation clabo- rated in Sec 3). Thus, we propose task-adaptive to- kenization, which is built specifically from the task data and equipped with variable text segmentation, yet can be seamlessly integrated into off off-the-shelf language models. To illustrate the potential of ada-tivee tokenization, we show an example in Fig 1. As seen in this example, our tokeniza- tion process allows for the inclusion of domain- specific terminology such as and las__ense_oo__urre____e_ model trained with task-specific tokenization is now able to generate these tokens through learned preference, which show can lead to significant performance improvements.
2 The PsyQA Task
The goal of the PsyQA task is to generate sup- portive response to the help-seeker via responding to their post, where an essential requirement is to imitate the use of language that is characterized professional by previous work (Sun et al., 2021; Welch et al., 2020). Figure 3 shows an example of question-answer pair in this dataset. Posts and responses are often extensive, with help-seekers providing detailed accounts of their experiences, and those offering assistance providing compre- hensive views, including emotional comfort, in- depth analysis, and various suggestions.
in the vocabulary. A
We confront three major challenges when de- signing tailored tokenization strategies. First, The creation of a task-specific tokenization vocabulary must be performed through automatic process due to the labor-intent inbor-intensive and time-consuming na- ture of manual selection. Integrating this task- specific vocabulary seamlessly with pre-trained models poses is challenging, and it requires tech- niques fuse the task-specific vocabulary with the pre-trained vocabulary and fine-tune the result- ing model accordingly. Lastly, we need to address the poor representation of newly added tokens in task-specific vocabularies that were not learned during the pre-training phase.
The formal definition of the task is as follows: given question Q, description D, and keywords K, let context C denote an integral description of Q D. and K; c (c1, C2, ..... Gm) is the sequence by segmenting the context C. We aim generate response R (a sequence of r (r1,72, .., Fn)) corresponding to the context C.
(21,02,
Motivation
Our motivation builds on arguments stemming from cognitive science, where (1) a clear distinc- tion being made between the vocabulary used interpret language versus the vocabulary used for language production; and (2) there evidence for increased efficiency in speech behavior stemming
In this paper, we propose task-adaptive tok- enization as a method for enhaneing text genera- tion in specialized tasks such as PsyQA. This pa- per makes three main contributions. (1) Building on insights from cognitive linguistics (Thomdyke,

---

from individual segmentation granularities. These arguments are further expanded by optimization and efficiency goals, which are better achieved in the presence of flexible segmentation.
As seen in Equation (2), different segmentations of a text can influence the entropy of the training corpus and thus can influence the model's perfor- mance.
In practice, researchers often blindly adopt a pre-existing vocabulary without considering the potential distribution inconsistency between the train data (typically used to generate the vo- cabulary) and the data of the downstream task, which can hinder the downstream performance (Liu et al., 2021c). For example, the data which the word-piece model is trained to obtain the BERT vocabulary originates from Google's Neural Machine Translation benchmark (Wu et al., 2016). The composition of this vocabulary de- signed for machine translation, which may not be ideal for PsyQA or for other tasks, according to Equation (2). This additionally supports our ar- gument that a custom vocabulary informed by the task at hand is needed to achieve the best optimiza- tion potential.
3.1 Receptive Productive Vocabulary
Within cognitive linguistic research, clear dis- tinction is being made between "receptive" and "productive" vocabulary (Bogaards and Laufer, 2004; Stwhr, 2008) - the former referring to words comprehended while reading, and the latter to words actively used in writing or speaking. A strong productive vocabulary has a direct impact on writing quality (Engber, 1995; Fajrina et al., 2021) and is essential for effective communication and precise articulation, particularly in technical fields where specialized terminology is common (Maskor al., 2016; Faraj. 2015). We, therefore, hypothesize that while creating large-scale vo- cabulary essential for training language models (i.e., the "receptive" vocabulary), generation tasks require more emphasis on designing and leverag- ing task-related vocabulary (i.e., the "productive" vocabulary).
3.2 Efficient Speaker Behavior
Empirical studies have demonstrated that during language production, individuals tend pause various syntactic constituents, such sentence boundaries or between clauses (Abney, 1992: Gee and Grosjean, 1984; Torrance et al., 2007). This phenomenon, referred to as "pause behavior," has been popular research topic in cognitive linguis- tics (Thorndyee, 1977; Wells, 1947). A possible explanation for this behavior is the fact that differ- ent individuals produce texts at various granulari- ties, from single letters and words to phrases and even entire sentences. When certain expressions are regularly used, they are stored as whole units in memories, thereby reducing the cognitive load for future usage.
To illustrate this gap in practice, considering the PsyQA task as described earlier, typical opti- mization objective used for fine-tuning would be
MLEE argmax I()),
(1)
where
D L(0) Σ logP P (1))))
Here, c and are sequences of tokens, i.e., the seg- mentations of the input context C' and the response R. The input of the function starts from and instead of the original texts C and R, due to the common practice of using a vocabulary that deter- mines the mapping relationship between texts and tokens. Thus, vocabulary construction is not nec- essarily considered the process of optimization. However, if we do not assume the vocabulary in this process, we obtain the log-likeline ligellliihod:
Building on this argument, we hypothesize that implementing a similar strategy in text generation can equally lead more efficient behavior. Simi- lar to human speakers, with variable segmentation, we can accommodate tokens at the sub-word level to address the rare word problem (Sennrich et al., 2015; Wu et al., 2016) while also including larger granularity units such phrases and clauses.
IDI
L(0) = Σ
E pmppri /io)t/ogoooooo PogP(r)e;))
(2)
This argument further supported by previ- ous work that has demonstrated that ine-grained segmentation, despite its flexibility, can lead to in- creased computational cost and degradation in to- ken representation (Zouhar et al., 2023; Liu et al., 2021e; Yu et al., 2021; Demeter et al., 2020; Wang
c)
where
{< C(+), R(0) }) takenizor (< (c((),,(0)) (((6)}]]) })!2

---

al., 2023). For instance, recent large language models such as GPT3 may require roughly two to- kens for the representation of an eight-letter length word?. This fragmentation also leads to tokens such as soci I, al or |_sso I, which are often shared by many words and lead to under- representation, preventing the fine-tuned model to better learning the compositionality of generation (Liu et al., 2021e; Dou et al., 2021; Yu et al., 2021). Instead, if we could allow for more cohe- sive information to be represented in token, in- cluding for instance task-specific tokens such as 1, we could potentially reduce the computational cost and achieve stronger token representation.
split sentences into words, as we want to include segmentations of various granularities.
To illustrate, consider the sentence "a sense of purpose in life" as an example. The original model segments it into subwords as follows:
-[_a 1 [_en 11 se 1f [o 1f _urppss 1 [[_i [ife
[[a]] [essse) ] [_of] p pose ] [in [lfef]]
With our modification, the model is also able to produce the following additional segmentations:
a_ J [[of [ [urposs [ [_in life]
[ a sense of purpose in life
Alg 1 shows the task vocabulary construction process, where underline text indicates our modi- fications to the original algorithm.
4 Task-adaptive Tokenization
Our approach to task-adeetive tokenization con- sists of three main steps:
Algorithm 1: Task Vocabulary
Construction
1. Task Vocabulary Construction: First, compile task-specific vocabulary (Sec by leveraging subword regularization algorithm.
Data: D = {As : task dataset)
Result: V=
- (1,,,),, ..., (LN, (x)].
vocabulary with size N, where t; is the ith token and & is the corresponding score.
2. Vocabulary Merging: Next, we merge the task-specific vocabulary with the original vo- cabulary from pre-trained models (Sec 4.2).
- P= {Ap: E
3. Token Mapping: Finally, we create new to-
pieces cut at any length]:
ken embeddings by mapping the new token to the sub-words in the pre- treitrinned vocabulary and
P Vhig {ti, ... twr), where
M >>>>>> N:
averaging the sub-word embeddings (Sec 4.3).
Unigram-model optimizes
L - (og(Ee))(e)) P(t)). where
4.1 Task Vocabulary Construction
S(t) is all possible segmentation for s;
To construct a task-specific vocabulary that al- lows for variable segmentation, as described in Section 3, we use subword regularization (Kudo, 2018). Subword regularization optimizes the like- lihood of the training corpus on all possible text segmentations and produces vocabulary that consists of set of tokens and their corresponding log-likelihood scores. Specifically, this algorithm leverages regularization coefficient to increase the sampling probability of low-score segmenta- tions during training to learn the representations of various segmentations. This allows for sampling a certain segmentation among various possible segmentations based on the score of the text being segmented. To adapt the original algorithm to our setting, we use task-specific data (i.c., all the re- sponse sentences from the QA pairs in the PsyQA task) to train a unigram language model. In addi- tion, contrary to the original algorithm, we do not
Compute the loss; of each 4 in Vaig:
Sort (41, (41,1081), ... (4a,, loss2s;;
(os,,,))),,.... (I,, loss
fruneate, V.
The process followed in each step in the algo- rithm is as follows:
1. Divide all sentences & E D into various gran- ularity pieces;
2. Choose the most frequent pieces and the union of characters up to big seed vocab- ulary Voig {f,,..., ..., tas where M >> N;
3. Build unigram model D. The proba- bility of possible segmentation (k.,..,(k) on is P(t) II p(t)). The most pos- sible segmentation is optimized by t* argmax P(t), where S(t) is all possi- ble segmentations for s. Apply EM algorithm to unigram model with the objective function = Σ. log(Enes(e) P(t));
"platform. opwmaa.mpmmmcccmmmmmmm wttp:iii.oooo/lag///////a

---

4. Compute the loss; of each in Voig where loss; represents how the likelihood has re- duced when the piece i, removed from the current vocabulary;
Merged Specisliedd Vacabulary
Driginal Vocabutary
Speezffe
[PACE
-100
Ixdex
Takwn
Vocatubrry
Feee
400
Pil)
Toben
Score
for
99
5. Sort the tuples (t1, lossi), (tM, loss in the descending order;
leex
you
+8.52
for
for
- 20
(OUR
-100
6. Keep the union of characters and the pieces with the highest loss score until it satisfies the target vocabulary size N; Get the final vo- cabulary V 41), (fv, IN)), 4 is the log probability of lú in the unigram model.
45000
[OUE
I'r aomy
+D:41
50032
110
50001
you
50000
JSNz]
X0002
cony
-0.4
}) where
Rulle
Specal tricrr meciie the Ioweet scom (-100)
calculated ove werlapping
original
Overtspping troons necaiun the soom carnted the rent-specilcc vorabulory
4.2 Vocabulary Merging
Non owwrraapingg takunaa from the lask-spuctii: vocabularyy appended to the and
After creating a task-specific vocabulary, an im- portant challenge is how to incorporate this "alien" vocabulary in the pre-trained models, as such models already have their own vocabulary and cor- responding embeddings. To this end, we build protocol for merging task-specific vocabulary with pre-existing vocabulary (referred as orig- inal vocabulary), as shown in Fig 2.
Figure 2: An overview of the merging protocol.
embeddings by using the primary pre-trainee tok- enizer to segment the new token into a set of sub- words. The initial embedding of new tokens is set as the average of its subwords' embeddings from the original embedding matrix. For overlapping tokens, we leverage the existing embedding repre- sentations.
The design of such a merging protocol consid- ers two aspects. First, to inherit the embedding matrix of the pre-trained models, the order of all tokens from the original vocabulary is maintained in the merged vocabulary. In this way, new repre- sentations could be added to the embedding matrix by extending the rows of the original one (Rule 4 in Fig 2). Second, special tokens and the tokens from the original vocabulary that are constructed based on unique segmentation algorithms (e.g., WordPiece and BPE) do not have a score for sam- pling. Thus, we have to assign them an appropri- ate score based on our guidelines (Rules 1, 2, and 3 in Fig 2). We assign - -bigScore ten(token) Ten((oken) to those tokens qualifying for Rule 2, where the bigScore ought be lower than the lowest score among task-specific tokens, ensure task- specific tokens have higher priority to be sam- pled; meanwhile, longer tokens will receive big- ger score than their sub-string tokens, following the BPE/WordPiece design that prioritizes longer segments. Please see token statistics for a merged vocabulary in Appendix B.
Experiments
We assess the effectiveness and efficiency of our adsk-ite tokenization the PsyQA task through several automatic and manual evaluations.
5.1 Datasets
CN PsyQA is a Chinese dataset of psychological QA support, where the answers are provided by well-trained volunteers and counselors (Sun et al., 2021). This dataset contains 22,346 questions and 56,063 answers, where each question relates to one more ground-truth responses.
MHP Reddit is an English dataset of Post- Response pairs crawled from Reddit, where re- sponders self-identify as mental health profession- als in their profiles (Lahnala et al., 2021). Com- pared to CN PsyQA, this dataset is smaller in size (9,501 QA pairs), and the responses are relatively shorter.
5.2 Compared Methods
4.3 Token Mapping
We compare our task-adaptive tokenizer with the tokenizers from off-the-shelf pre-trained models. We acknowledge that the implementation details of a tokenizer are bound to the pre-trained model type and corpora. We provide details of the to- kenizers we compare against in Tab 1, includ-
To address the possible difficulties of representa- tion learning for new tokens that are never seen during pre-training, we propose a simple but effec- tive initialization mechanism to alleviate the prob- lem. For each new token, we acquire its subword

---

ing the associated model type, vocabulary size, and the segmentation algorithm. For brevity, we use the notations [Lse and [LLM un- der PsyQa or MHP datasets to represent the cor- responding base and -aaative tokenizers for a model named LLM, respectively. Accordingly, we append the wo mapping to the notations to indi- cate whether the mapping mechanism is used.
CN PsyQA
Setting
Bleu
+pet
B-1
B-3
R-L.
+pct
EP ts
20.1
GPT2hase
18.2
55.5
2.5
15.5
GPT2
24.8
+35,9%
65.7
6.4
27.1
+74.8%
mapppnn
25.01
+37.1%
66.31
6.6
22.1
+42.1%
Barlbare
21.6
62.3
4.0
21.8
BartaT
26.2
+21.3%
69.2
6.71
27.2
+24.8%
+mapping
26.1
+20.8%
68.8
6.71
27.2
+24.8%
Dataset
Model source
Tokenizer Vocab size
MHP Reddit
spt2 chinn-hieese- cluecorpussmali bart cest-buue--hiieess
WordPiece
21128
Setting
Bleu
+pet
B-1
B-2
R-L.
+pet
CN PsyQA
WordPiece -T. 51271
GPT2
3.7
14.0
0.6
5.7
chinese-LLaMee LLaMA-
GPT2
3.6
-2.7%
13.0
1.3
8.1 +42.1%
BrE-duopout
49953
plus-7B
+mapping
4.5
+16.4%
16.3t
1.6
9.0
+57.9%
gpt2 bart-base?
BPE
50257
Barlpase 6.7
23.5
2.6
10.8
MHP Reddit
BPE
50265
Bartar 7.6
+13.4%
27.9
2.5
10.1
-6.5%
Table I: Tokenizers used in our comparative evalua- tions
+mupping
6.7
+0.0%
22.9
3.0
10.9
+0.9%
Table 2: Generation effectiveness. Bleu is calculated by averaging B-1,2,3,4, where B-n denotes the Bleu n- gram precision. R-L is Rougel. score. +pet denotes the percentage of improving scores corresponding to Bleu and RougeL over the base. indicates the sota results reported by Sun et al. (2021), who fine-tuned GPT2 with auxiliary support strategy information. indicates a significant improvement over the base (p- value < 0.05).
5.3 Experiment Details
As our backbone models, we adopt the small ver- sions of GPT2 (Radford et al., 2019) and BART (Lewis et al., 2019) to analyze the generative im- provements in both decoder-only and encoder- decoder models. Additionally, to study the effec- tiveness of our approach on larger language mod- els, we include 7B LLaMA model in our experi- ments (Touvron et al., 2023). However, due to lim- were only able to conduct ex-
2020), we report these results as reference since they are commonly used by previous research (Gu et al., 2023; Cao and Wang, 2022; Yue et al., 2021). In addition, we believe that the multi- ple gold responses provided for each question the CN PsyQa dataset alleviate the shortcoming of reference-based metries to a degree. We lever- age character-level comparisons for Chinese and word-level comparisons for English. To estab- lish fair comparison, we apply an NLTK word tokenizer (Bird et al., 2009) to all generated re- sponses.
ited resources, periments the Chinese version of LLaMA (Cui al., 2023). For training. we create 8:1:1 splits of the datasets (section 5.1) and fine-tune our small backbone models (i.e., GPT2 and BART) while only training the LoRA adapter (Hu et al., 2022) and the input & output embeddings of LLaMA. Additional details for training and generation are included in Appx C.
5.4 Automatic Evaluation
From Tab 2, the task-adaptive tokenizer consis- tently outperforms the baseline tokenizers, with a maximum increase 37.1% % on Bleu and 74.8% on RougeL. The results demonstrate two impor- tant insights. First, task-adaptive tokenization shows larger increase on Blcu-3 than Bleu- 1, indicating that variable segmentation can en- hance the expression of task-specific phrases. Sec- ond, the increase in RougeL suggests success- ful retrieval of task-specific expressions from the dataset.
Effectiveness. We use the HuggingFace evalua- tion tools (Wolf et al., 2019) and report Bleu-I (B- 1), Bleu-3 (B-3), average score of Bleu-1, 2, and 4 (Bleu) (Papineni et al., 2002) and RougeL (Lin, 2004) for the generation on the test set. Although such reference-basse metrics may not be sufficient for open-ended generation tasks (Zhang* et al.,
mall srrgginifaee..e//////rt------sse
"hwggingface..o/ wtg.gigggw....ooooggggggggeeeeee s-lora-7b
However, since the results from the automatic evaluation do not indicate large improvements for the mapping mechanism on CN PsyQA, we turn
Mnuggingface.. c
"huggingface sagagenggacecccom wwlaaaac.cofaaaaaaaaeae co/faceeooo//bart-baas

---

human evaluations in Sec 5.5, and the results demonstrate that the mapping mechanism im- portant for the generation quality in human per- ception. To also gain insights into the weaker ef- fectiveness of task-adaptive tokenization on Red- dit MHP, in addition the human evaluation con- ducted to validate its effectiveness, in Sec we extend our experimentation by creating a parallel English corpus, translated from CN PsyQA. This addition allows us to probe the underlying factors contributing to the limited improvement observed in Reddit MHP whether the disparity in perfor- mance be attributed linguistic differences (English versus Chinese) or to disparities in data quality observed within the Reddit MHP dataset. Efficiency. In order assess generation effi- ciency, we employ various metrics on the test set, including the average number of centiseconds per generation (#cSec), the average number tokens per generation (#Tok), response length (Len), generation length per token (Len/#Tok) and gen- eration length per centisecond (Len/#cSec). To calculate response length, we consider token- agnostic measures, such as the number of char- acters for Chinese and the number of words after whitespace-spliting for English. The token uti- lization rate is then derived by dividing the number of tokens by the response length.
Len/ #Tok
Len/ #eSec
Setting
#cSec
#Tok
Len
1
CN PsyQA
GPT2
5.7
440.2
365.9
0.8
64.2
GPT2
3.6
190.3
382.9
106.4
mapping
MHP Reddit
GPT2
1.6
117.1
86.9
0.7
54.3
GP
2.4
118.8
123.5
1.0
51.5
+ mupping
Table 3: Efficiency of generation. #cSec and #Tok de- note the average number of centiseconds and tokens per generation on the test respectively, Length denotes the average length of generated responses.
M vs.
NM vs. B Lose
M vs. NM Lose
Metric
Win
Lose
Win
Win
CN PsyQA
31
15
361
C
37
9
19
19
361
10
PE
23
20
18
22
32f
13
MHP Reddit
F
20
44
C
28
20
4
38
48f
1
PE
30
18
6
39
451
3
Table 4: Human Evaluation. An explanation for ab- breviations: M for GPT22T +mapping, B for GPT2 and NM for GPT2mm w/o mapping; F for fluency, C for coherence, and PE for professional expression. Ties are shown. denotes significant win (one sample sign test, p-value < 0.05).
Tab 3 indicates that by using task-adaptive tok- enization, models trained on Chinese and English datasets use significantly fewer tokens to repre- sent more content. This enhancement is more ap- parent in Chinese, as its generation length per to- ken is increased from 0.79 to 2.00, indicating a more than double time increase in token utilization rate. However, the results only show a significant improvement in generation speed for the Chinese dataset. We believe this occurs as responses in the MHP Reddit dataset are rather short and thus ben- efit less from using fewer tokens for generation, which is compensated by the increased time con- sumption from the expanded embedding layer. Vocabulary Size. We also investigate the influ- ence of different task-specific vocabulary sizes on generation quality (Appx D). The results indicate that an optimal size of vocabulary for merging may exist, but the difference may be subtle.
were provided with ten QA pairs that con- sidered the "gold standard." " They were instructed to familiarize themselves with the wording, struc- ture, and language style of these gold responses, serving as calibration exercise to assess the pro- fessional expression a response and how closely aligns with the standard responses. Each partic- ipant underwent ten rounds of evaluation under a guideline (see Fig 4). In each round, they were presented with set of posts and corresponding response triplets, comprising the responses from GPT2 GPT2 w/o mapping. and GPT2 with mapping. The participants were then tasked with ranking the three responses based three aspects: (1) Fluency: the response's fluency and readability, (2) Coherence: the responsiveness of the response to the post's content and its logical consistency, and (3) Professional expression: the proximity the generated response to the stan- dard responses in terms of wording, structure, and language style.
5.5 Human Evaluation
We recruit five native professionals for the human evaluation of the Chinese and English results, re- spectively. Prior to the evaluation, participants

---

From the findings presented in Tab 4, the inclu- sion of mapping mechanism is crucial for ensur- ing robust token representation system, partic- ularly when dealing with small-scale data (MHP Reddit). Without this mechanism, the generated responses exhibit a significant decline across three aspects, despite an increase in automatic eval- uation scores. Moreover, our lokenization ap- proach with the mapping mechanism outperforms the baseline on CN PsyQA in human evaluation, even though this improvement is not reflected in the results of automatic evaluation.
emerges in the contribution of non-overlapped to- kens when comparing the Reddit MHP and CN PsyQA datasets. Our hypothesis posits that the generation of newly-introduced task-specific to- kens may play a pivotal role explaining the less pronounced performance gains observed the Reddit MHP dataset, in contrast to the CN PsyQA datasct.
overlap
non-overlap
original
CN PsyQA 12.0%
84.3%
3.7%
MHP Reddit
62.4%
37.5%
0.1%
5.6 Performance on Large Language Models
Table 6: Length contribution of three types of tokens generated on both datasets.
We investigate the benefits of our task-adaptive to- kenization the effectiveness and efficiency gen- eration for the recent LLMs. Tab shows the results when using the 7B LLaMa model, de- scribed in Sec 5.3. The RougeL score increases by 15.0% when applying our tokenization, which in- dicates that our aask-adaetive tokenization brings about additional and model-agnosee performance benefits.
Further, we investigate whether the disparity in constitution analysis between the two datasets arises from linguistic distinctions or data quality concerns. As highlighted in the description of the datasets in See 5.1, participants in the MHP Reddit dataset primarily self-identify sionals in their profiles. Additionally, we observe that many QA pairs in Reddit MHP resemble gen- eral chitchat, where task-specific terminology may be less prevalent. To address this, we translate 5,118 QA pairs from the CN PsyQA dataset into a parallel corpus, split into an 8:1:1 ratio (train- ing, development, test). With this, we aim to re- assess the effectiveness of our proposed tokeniza- tion techniques within an English data context. As illustrated in Tab 7, task-adaptive tokeniza- tion markedly enhances generation effectiveness across all metries. Based on these results, con- clude that our proposed tokenization method per- forms effectively in tasks involving frequent of coomain-specing expressions, compared open domain communication.
profes-
Setting
Bleu
+pet
B-1
B-3
R-I.
+pet
CN PsyQA
ILaMMAuse
27.9
64.5
12.1
30.1
LLaMART
29.8
+6.8%
69.61
12.5
34.0
+13.0%
*mapping 29.5
+5.7%
69.6f
12.3
34.6i
+15.0%
Table Generation effectiveness on Chinese LLaMA. Bleu is calculated by averaging B-,,,,3,44 where B- n denotes the Bleu n-gram precision. R-L is RougeL score. +pet denotes the percentage of improving scores corresponding to Bleu and Rougel. over the base. indicates significant improvement over the base (p- value < 0.05)
6 Further Analysis and Supplementary Experiment
We conduct supplementary analyses and experi- ments to gain deeper insights into the observed performance disparities between the Chinese and English datasets. Specifically, we analyze the constituents within the generated responses, and classify them into three categories to quantify their contribution to the overall generation length. These categories are as follows: (1) overlap, de- noting tokens shared between the original and spec-ip vocabularies; (2) non-overlap. rep- resenting tokens exclusive the task-specific vocabulary; and (3) original, signifying tokens found solely within the original vocabulary. As exemplified Tab 6, a noticeable discrepancy
Setting
Bleu
+pet
B-1
B-3
R-I.
+pet
Translated PsyQA
GPTbee base
12.6
44.5
0.6
8.5
GPT
19.5f
+54.8%
58.11
2.41
15.2f +78.8
+mapping
19.9
+57.9%
56.3)
4.0
14.71
+72.9%
Table 7: Generation effectiveness Translated PsyQA. See Tab 2 for column and notation definition.
7 Related Work
Segmentation Algorithms and Vocabulary De- velopment. Mapping text into tokens, as a key step in the NLP pipeline, has a rich history of algorithms and vocabulary development. Early

---

proposals of segmenting text varied from utiliz- ing linguistic cues at different levels (c.g.. mor- pheme or syntax) to statistical methods (Creutz and Lagus, 2006; Luong et al., 2013; Zouhar al., 2023). Notably, during the cra of statistical ma- chine translation, phrase-level translation, which shares a similar idea with variable segmentation, was one of the most promising translation meth- ods at that time (Koehn et al., 2007, 2003). This paradigm enjoyed considerable popularity until the rise of deep learning techniques, shifting the focus to subword-level segmentation, given the need to address the challenge of poor representa- tion of rare/sparse words (Sennrich et al., 2015, 2016; Kudo, 2018; Kudo and Richardson, 2018). This approach largely improves performance of NLP models by leveraging shared subword units and maintaining a compact vocabulary. In par- allel, the use of vocabulary transformed with the advent of large language models (LLMs). Previ- ously, each model tended to develop its own vo- cabulary in isolation (Jean et al., 2015), but re- cent work started to directly use the vocabulary of pre-trained models to inherit the strong rep- resentations acquired through pre-training. De- spite existing workdBaggasrryan et al., 2022; Liu al., 2021d), customizing vocabularies for spe- cific tasks lost popularity due challenges in inte- grating them into nowadays pretraining-finetune paradigm.
2021b), including counseling-style dialog genera- tion systems (Shen al., 2020) and the incorpo- ration of counseling strategies in response genera- tion (Sun et al., 2021). Furthermore, recent work has investigated the use of large language models as expert systems for support strategy counseling (Zhang et al., 2023). However, rather than focus- ing adaptation effort on fine-tuning or prompting, our study focuses on tokenization, an easily over- looked component in the NLP pipeline. We hy- pothesize that for tasks in technical writing fields, e.g., mental health, adapting the tokenization to the downstream language style is a potential strat- egy unlock additional performance.
Text Generation Efficiency. Recent years have witnessed a great surge of interest in enhancing text generation efficiency due to the need of long- form text generation or the trend of scaling up NLP models (Hessenthaler al., 2022; Tay et al.. 2020). Efforts on improving generation efficiency range from researching non-autoregressive model- ing techniques (Li et al., 2022: Qi et al., 2021), at- tention mechanism optimization (Yan et al., 2021; Beltagy et al., 2020), model size reducing (Zafrir et al., 2019; Ma et al.. 2022), and infrastructure innovations (Rasley et al., 2020; developers, 2021, 2022; Fang et al., 2021).
Conclusion
Recently, research has been proposed for kenization quality evaluation (Zouhar et al., 2023) and token cost analysis of a tokenizer among dif- ferent languages (Ahia et al., 2023), indicating the researchers' increased concerns on tokeniza- tion improvement. It is worth noting that Liu et al. (2021d); Sachidananda et al. (2021) also ad- dressed the vocabulary gap between pretraining and finetuning or domain-level post-pretraining; however, their solutions either requires addi- tional model module for token alignment solely operates at the sub-word level. In contrast, our work provides a model-agnostic solution and em- braces the merits of flexible variable segmentation cherished in earlier research while still retaining the ability leverage existing pre-traiee tr-irriied models.
In this paper, we proposed task-specific tokeniza- tion as way to adapt the generation pipeline to the specifics of downstream task. We demonstrated the efficiency and improved long-form quality of generation for the domain of mental health, where we specifically addressed the task of psychologi- cal question answering. Our tokenizer leverages the specifics of the downstream task data, while still retaining the ability to integrate into exist- ing pre-trained models. Through extensive ex- periments, we demonstrated the ability of task- adaptive tokenization to enhance both the effec- tiveness and efficiency of long-form generation.
We believe our work is particularly useful in the era of large language models (LLMs), as the pro- posed task-adaptive tokenizer can lead to signifi- cant improvements while being domain and model agnostic. Based on our findings, we suggest plug- and-play tokenization for LLMs when performing specific generation tasks.
Generation in Mental Health Support. In re- cent years, several studies have explored the appli- cation of generation techniques for mental health support (Shen et al., 2022b; Lee et al., 2021; Sabour et al., 2023; Hsu et al., 2023; Liu et al.,

---

Limitations
Foundation (#62256).
Despite the strength of our proposed task-adaptive tokenization, several limitations remain. In partic- ular, due to limited resource, we were only able to test it on one dataset and on a large-scale lan- guage model. Future work should consider eval- uating the effectiveness of our task-adaptive tok- enizer additional domains and LLMs. The ef- fectiveness of this tokenization should also be ver- ified on additional languages and models of var- ious sizes. Finally, in our experiments, we found our tokenization does not significantly enhance the generation speed in English, which may be due to the fact that the English vocabulary has less room to increase its granularity compared to a character- based language like Chinese
References
Steven Abney. 1992. Parsing by chunks. Principle- based parsing: Computation and Psycholinguistics, pages 257-278.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yu- lia Tsvetkov. 2023, Do all languages cost the same? tokenization the era of commercial language mod- els.
Eugene Bagdasaryan, Congzheng Song, Rogier van Dalen, Matt Seigel, and Aine Cahill. 2022. Training tokenizer for free with private federated learning.
Iz Beltagy. Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.051.
Ethics Statement
Steven Bird, Ewan Klein, and Edward Loper. 2009.
Several ethical situations should be considered. First, due to the black-box feature of neural net- works, we do not recommend any generation tech- nique including our proposed method to be di- rectly used by mental health supporter. Instead, in practice, it could be exposed to well-trained MHPs informed by the pros and cons of using generative AI, and offered for professionals. The professionals should be fully informed and trained be accountable for the second-edited content. Second, during human evaluations, we informed all the human evalua- tors that the responses they would see might cause some discomfort, so they could decline their par- ticipation in the evaluation. Finally, regarding the use of the PsyQA data, we have received autho- rization to use the data and conduct research on it ethically and legally through approval from the data publishers.
Natural language processing with Python: analyz- ing text with the natural language toolkit. O O'Reilly Media, Inc.".
Paul Bogaards and Batia Laufer. 2004. Vocabulary in second language: Selection, acquisition, and test- ing, volume 10. John Benjamins Publishing.
Natassia Bremman, Nagendra P Luitel. Sumaya Mall, and Mark JD Jordans. 2014. Demand and access to mental health services: qualitative formative study nepal. BMC international health and hu- man rights, 14(1):1-12.
suggesting tool
Shuyang Cao and Lu Wang. 2022. prompting for text generation. arXiv:2211.02162.
Time Time-aware arXiv preprint
Mathias Creutz and Krista Lagus. 2006. Morfessor in the morpho challenge. In Proceedings of the PAS- CAL Challenge Workshop on Unsupervised Segmen- tation of Words into Morphemes, pages 12-17.
Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Effi- cient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177.
Acknowledgements
We acknowledge the valuable feedback from the reviewers, as well as from Orevaoghene Ahia, a Ph.D student at University of Washington and Vilém Zouhar, a Ph.D student at ETH Zürich dur- ing the discussion about paper improving for the camera-ready version. We thank the anonymous reviewers for their constructive feedback. We are also grateful to the members of the Language and Information Technologies lab at the University of Michigan for the insightful discussions during the early stage of the project. This project was par- tially funded by an award from the Templeton
Orianna Demasi, Yu Li, and Zhou Yu. 2020. multi- persona chatbot for hotline counselor training. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 3623-3636, Online. Association for Computational Linguistics.
David Demeter, Gregory Kimmel, and Doug Downey. 2020. Stolen probability: A structural weakness of neural language models. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 2191-2197.
Hugging Face Optimum developers. 2022. Hugging face optimum. https://huggingf wttp:egggiinaaa.c..ooootttttt co/opiim um. Version: x.y.z.

---

ONNX Runtime developers. 2021. Onnx runtime. ht tps: http:///wwwwooo http://wwwwowwwnntee wwwwoonniwwnn.oooooo ai. Version: x.y.z.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1-10. Beijing, China. Association for Computa- tional Linguisties.
Yao Dou, Maxwell Forbes, Rik Koncel- Koncel-Kedziorsk Kedziorski, Noah A Smith, and Yejin Choi. 2021. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for serutinizing machine text. arXiv preprint 217.02200..
Cheryl A Engher. 1995. The relationship of lexical pro- ficiency to the quality of esl compositions. Journal of second language writing, 4(2):139-155.
Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Men- talBERT: Publicly available pretrained language models for mental healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7 7184-7190, Marseille, France. European Language Resources Association.
Dian Fajrina, John Everatt, and Amir Sadeghi. 2021. Writing strategies used by indonesian efl students with different english proficiency. Language Teach- ing Research Quarterly, 21:1-15.
Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. 2021. Turbotransformer: efficient gpu serving system for transformer models. In Proceedings of the 26th ACM SIGPIAN Symposium on Principles and Practice of Parallel Programming, pages 389- 402.
Dinesh Kalla and Nathan Smith. 2023. Study and anal- ysis of chat gpt and its impact on different fields of study. International Journal of Innovative Science and Research Technology. 8(3).
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Gallison-Burcht Marcello Federico. Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondiej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Ses- sions, pages 177-180, Prague, Czech Republic. As- sociation for Computational Linguisties.
Avan Kamal Aziz Faraj. 2015. Effective strategies for turning receptive vocabulary into productive vocab- ulary in efl context. Journal of Education and Prac- tice, 6(27):10-19
James Paul Gee and Francois Grosjean. 1984. Empir- ical evidence for narrative structure. Cognitive Sci- ence, 8(1):59-85. 85.
Yuxian Gu. Jiaxin Wen, Hao Sun. Yi Song, Pei Kc, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan 7hu, et al. 2023. Eva2. O: Investigating open-domain chinese dialogue systems with large- scale pre-training. Machine Intelligence Research, pages 1-13.
Philipp Koehn, Franz. Josef Och, and Daniel Marcu. 2003. Statistical phrase- translation. In Pro- ceedings of the 2003 human language technology conference of the North American chapter of the As- sociation for computational linguistics, pages 127- 133.
Marius Hessenthaler, Emma Strubell, Dirk Hovy, and Anne Lauscher. 2022. Bridging fairness and vironmental sustainability natural language pro- cessing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emi- rates, December 7-11, 2022, pages 7817-7836. As- sociation for Computational Linguistics.
Taku Kudo. 2018 Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Annual Meering of the Asso- ciation for Computational Linguistics.
Taku Kudo and John Richardson. 2018. Sentence- Piece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Rebecca S Hock, Flora Or, Kavitha Kolappa, Matthew D Burkey, Pamela Surkan, and William W Eaton. 2012. A new resolution for global mental health. The Lancet, 37979824)
Shang- Ling Hsu, Raj Sanjay Shah, Prathik Senthil, Zahra Ashktorab, Casey Dugan, Werner Geyer, and Diyi Yang. 2023. Helping the helper: Supporting peer counsclors via ai aiieempoereed practice and feed- back. arXiv preprint arXiv:2305.000 089822
Allison Lahnala, Yuntian Zhao, Charles Welch, Jonathan K Kummerfeld, Lawrence C An, Ken- neth Resnicow, Rada Mihalcea, and Verónica Pérez- Rosas. 2021. Exploring self-identified counseling expertise in online support forums. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4467-4480.
Edward Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
John Lee, Baikun Liang, and Haley Fong. 2021. Re- statement and question generation for counsellor

---

chatbot. In Proceedings the 1st Workshop on NLP for Positive Impact, pages 1-7, Online. Association for Computational Linguistics.
sive neural networks for morphology. In Proceed- ings of the Seventeenth Conference on Computa- tional Natural Language Learning, pages 104-113, Sofia, Bulgaria. Association for Computational Lin- guistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. pre-
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2022. Mega: moving av- erage equipped gated attention. arXiv preprint arXiv:2209. 10655.
2019. BART: denoising training for natural language generation, translation, and comprehension. CoRR, abs/1910.1344 13461.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Juan-Yun Nie. and Ji-Rong Wen. 2022. Elmer: A non- autoregressive pre-trained language model for effi- cient and effective text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1044-1058.
Zunita Mohamad Maskor, Harun Baharudin, al. 2016. Receptive vocabulary knowledge or produc- tive vocabulary knowledge in writing skill, which one important. International Journal of Aca- demic Research in Business and Social Sciences, 6(11):261-27)
Chin-Yes Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Com- putational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021a. a. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pages 3469-3483 Online. Association for Compu- tational Linguistics.
Vikram Patel, Neerja Chowdhary, Atif Rahman, and Helen Verdeli. 2011. Improving access to psycho- logical treatments: lessons from developing coun- tries. Behaviour research and therapy, 49(9):523- 528.
Siyang Liu, Chujie Zheng. Orianna Demasi, Sahand Sahour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang, 2021b. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1ihh International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers), pages
Weizhen Qi, Yeyun Gong, Jian Jiao, Yu Yan, Weizhu Chen, Dayiheng Liu, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, et al. 2021. Bang: Bridging autoregressive and non-autoregresssvv gen- eration with large scale pretraining. In International Conference on Machine Learning, pages 8630- 8639. PMLR.
Xin Liu, Baosong Yang. Dayiheng Liu, Haibo Zhang, Weihua Luo. Min Zhang, Haiying Zhang, and Jin- song Su. 2021c. Bridging subword gaps in pretrain- finetune paradigm for natural language generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume I: Long Papers), pages 6001-6011.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. OpenA/ blog, 1(8):9
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505- 3506.
Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo. Min Zhang, Haiying Zhang, and Jin- song Su. 2021d. Bridging subword gaps in pretrain- finetune paradigm for natural language generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natural Lan- guage Processing (Volume Long Papers), pages 6001-6011, Online. Association for Computational Linguistics.
Sahand Sabour, Wen Zhang, Xiyao Xiao, Yuwei Zhang, Yinhe Zheng, Jiaxin Wen, Jialu Zhao, and Minlie Huang. 2022. Chatbots for mental health support: Exploring the impact of emohaa on re- ducing mental distress in china. arXiv preprint arXiv:2209. 10183.
Sahand Sabour, Wen Zhang, Xiyao Xiao, Yuwei Zhang, Yinhe Zheng, Jiaxin Wen, Jialu Zhao, and Minlie Huang. 2023. A chatbot for mental health
Thang Luong Richard Socher, and Christopher Man- ning. 2013. Better word representations with recur-

---

support:
exploring the impact of emohaa on reduc- ing mental distress in china. Frontiers in Digital Health, 5:1 133987.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint UrXiv:2302.13977
Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop Simple and Fffficient Natural Language Processing, pages 155-165.
Sid Wang, Ashish Shenoy, Pierce Chuang, and John Nguyen. 2023. Now it sounds like you: Learning personalized vocabulary on device. arXiv preprint wwXiv:2305.03584
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprini arXiv: 1508,07909,
Charles F Welch, Allison Lahnala, Verónica Pérez- Rosas, Siqi Shen, Sarah Seraj, Lawrence An, Ken- neth Resnicow, James W. Pennebaker, and Rada Mi- halcea. 2020. Expressive interviewing: A conver- sational system for coping with covid-19. ArXiv, Abss/200.00389..
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics Vooluee I: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguisties.
Rulon S Wells. 1947. Immediate constituents. Lan- guage, 23(2)811-117
Siqi Shen, Verónica Pérez-Rosas, Charles Welch, Sou- janya Poria, and Rada Mihalcea. 2022a. Knowledge enhanced reflection generation for counseling dia- logues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3096-3107.
WHO. 2020. World mental health day: an opportu- nity kick-start massive scale-up in investment in mental health. Accessed: 2020-08-29.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2019. Huggingface's transformers: th-te-r-- natural language processing. arXiv preprint arXiv: 1910.03771.
Siqi Shen, Veronica Perez-Rosas, Charles Welch, Sou- janya Poria, and Rada Mihalcea. 2022b. Knowledge enhanced reflection generation for counseling dia- logues. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguisties (Volume I: Long Papers), pages 8096-3107 Dublin, Ireland. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster. Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. 2016. Google's neural chine translation system: Bridging the gap between human and machine translation. arXiv preprint 1rXiv:1609008144
Siqi Shen, Charles Welch, Rada Mihalcea, and Verónica Pérez-Rosas. 2020. Counseling-styl flection generation using generative pretrained trans- formers with augmented context. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 10-20.
Yu Yan, Fei Hu, Jiusheng Chen, Nikhil Bhendawade, Ting Ye, Yeyun Gong. Nan Duan, Desheng Cui, Bingyu Chi, and Ruofci Zhang, 2021. Fastseq: Make sequence generation faster. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 218-226.
Lars Stenius Stahr. 2008.
Vocabulary size and the skills of listening, reading and writing. Language Learning Journal, 36(2): 159-152.
Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. 2021. PsyQA: Chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages 1489-1503, Online. Association for Computational Linguistics.
Sangwon Yu, Jongyoon Song, Heeseung Kim, Seong- min Lee, Woo-Jong Ryu, and Sungroh Yoon. 2021. Rare tokens degenerate all tokens: Improv- ing neural text generation via adaptive gradient gat- ing for rare token embeddings. arXiv preprint arXiv:2109.03127.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. ACM Computing Surveys, 55:1 -82.
Perry W Thorndyke. 1977. Cognitive structures in comprehension and memory of narrative discourse. Cognitive psychology, 91).77-110.
Xiang Yue. Xinliang Frederick Zhang, Ziyu Yao. Simon Lin, and Huan Sun. 2021. Cliniqg4qa: Generating diverse questions for domain adapta- tion of clinical question answering. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 580-587.
Mark Torrance, Luuk Van Waes, and David Galbraith. 2007. Writing and Cognition. BRILL.

---

Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Fffficient Machine Learning and Cognitive Computing-Neurllee Edi- lion (EMC2-NIPS), pages 36-39. IEEE.
Post
A 20-year-old male with bipolur disorder and sy mytoous of OCD, who afraid altructivve women, seeking trcutment?
fess attractive have hipolar disceder and OCD syrtpooms, women I Sell neveus snd pweety on the subway, and nanions when passing altractive women the
Psychomalyyii suggesis foar of being seen throogh or sesigning semual mesning to intarsttins.. have experienced tracma from the apposite in college tried exposure thrragy it didin' work. legrechherpp alfoddive? Are there other treanssent optioes like mediestion! aut anxious
Qiang Zhang. Jason Naradowsky, and Yusuke Miyao. 2023. Ask an expert: Leveraging language models improve strategic reasoning goal-oriented dia- logue models. arXiv preprint arXiv:2305. 17878.
Keyworts: diagnosis, tresrment, counsoling
Tianyi Zhang*. Varsha Kishore*, Felix Wu*, Kilian Q. Weinherger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Response
Hello' It is crucial to seek prsfessiomll peychological lierapy and counseling for your coudition, bipelar disorter with symptoms obussire-compulllee traiss oppesite-ies sex
severe mental health issue that requires proper evalustion and treatment by a qualified mental bealth professional. The fear the oppasite may stem from past truumatic experiences involve issues related to self-disclosure, sexual desires, and ideatiy transfomaaionn
Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Mrinmaya Sachan, and Ryan Cotterell. 2023. To- kenization and the noiseless channel. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pages 5184-5207, Toronto, Canada. Associa- tion for Computational Linguistics.
Cognilive -belaviooo themapy (CBT) can be an effective approsch for addressing opposite sex foar. CBT comamo therspeutic appreoch focuss ilentilying thonghts and behavice partorns to impeove montal health. Stratogies may include recognizing and undersanding fear, graduzl exposure the opposite sex, cognitive restructaing change negnilive thoughi putterns, and louming relacation iochniques manage anxiety.
chonging instiomal
A Few-shot Inadequacy in PsyQA task
Limited Input Size and Inconsistent Results. Due to length restrictions, using more than one PsyQA example in the prompt was not allowed for longer post-response pairs, making the few- shot in-context learning for this task impractical. In addition, in cases where we were able to pro- vide more than one example as the prompt, the model generally produced low-quality incoherent responses that were inconsistent with the input context (i.e., user's post). We believe this is ex- pected as the model would have difficulties dis- tinguishing between the posts in the few-shot ex- amples and the post that requires a generated re- sponse.
Exposure theraps and hypaotherapy ean also be etfestive teantment appremuless. Expusiee thampyy invalves ynohusl exproine the fior object to rodute foor, while hypootherags uses hypnosis modify thoughts and behavices. Honevas, it impartatt to undergo these therapies under the guidance of qualified professional.
in addition to psychotheeapy, medication can be a viable treatment option for bipolar disonder Common modicttions used include lithiom solls, anlidepresamss. and mnipsychoriis, Hewever madicaaion should be preseribal and moniterad by quilified heattheure protessional dus to posential side ettiets.
It erncial seek ponfesstionl mental health and counseling promptty Mental health issues require timely treatment pervent further impact on well heing. Wishing you a speody recovery.
Figure 3: An example of PsyQA.
C Experiment Details
Misalignment with language behavior of MHIPs. The language style of LLMs is adopted from their corresponding pre-training data and human feedback. which may not meet the re- quirement of linguistic behaviors of MHPs. In other words, despite the significant performance of recent LLMs, desirable performance could only be achieved through finetuning the model on professional data that captures the rich MHP language style and patterns.
Except for the experiments on vocabulary size, we adopted a uniform size of task-specific vo- cabulary for task-adaptive tokenizer construction by merging a 10k task-specific vocabulary and the original vocabulary of each pre-trained model. During training, the regularization coefficient for sampling segmentation among various results was 0.5. We use one Nvidia GPU A40 for train- ing GPT and Bart, which can load eight samples with a padding length of 1024. Training parame- ters are max 102_ for CN PsyQA/512 for MHP Reddit, batch_ size=,, warmup_ree 1.1 We use different decoding pa- rameters for different models, as listed in Table 9.
B Statistics on tokens at different lengths in a merged vocabulary
Table presents the top-scored tokens various different lengths to show that the task-adaptive to- kenizer successfully includes many task specifc terminologies and expressions in its vocabulary.

---

#length
top-5 tokens
(0, 6]
and
(6, 12]
-poople really, thins
(12, 18]
relationship, heenaasshhip.mmmemaaeeeeeelllll relattonsissss -your therapist
(18, 24J
ji_sounds_likc_yoo,, _uidc_too_ooovrrsstiiiiii
a.mmentaLheetthhprrfessooon,
(24, 30]
When trun__tust___ broken one of the most important thing,
mental health professionals
fordeeriic--erssnallyy-iiiii community _communttymmmntill mental metttmillhhllhhhhcmteeeee. center, Borderline Pttprrrltineerrosootlttyete__eee Disorder, Licensed__eeeeeee Professional_Couneee .understand_exxctly
(30, 32]
Table 8: Statistics of tokens with the highest log likelihhooood scores in each length interval. The tpec-ipe vocabulary for statistics is built on MHP datasets.
Setting
Parameters
CN PsyQA
top_k=50 tion 1t
top_p=0.9, do do_sample=True, repeti-
GPT
top_k=50, top_p=0.9, do dd_ssmlle=True, repeti-
Bart
tion (empllyy-...emmeeaure=0000 top
do sample-True, num_ bemm=e repetition_ pentlty-11p tem-
top k=40,
top_p=0.8,
LLaMA
perature=0.955, regreet__ngaammm__ ngram aarianngraaeeeeeee size=4
max_new__oo 11__ee___oken-511,
MHP Reddit
k55 top_p=0.9, do_sample=True, repeti- tion_penalty=1.5, temperature=1
GPT
top_k=50 top_p=0.9, do_sample=True, repeti-
Bart
tion tion_penatt=1..1, ____peallyeel...eepprrauuree temperature=1
Table 9: Decoding Parameters
D
Vocabulary size
We reported the generation effectiveness cross dif- ferent task-specific vocabulary sizes in Table 10.
E
Case Study
For each generation setting, we present one gen- erated case. The results are Table 11 and Ta- ble 12. From the result on MHP Reddit, we see due to the scarcity of the data, generation quality is hurt by poor representation of newly-add tokens while fluency, coherence, and professional expres- sion are recovered after the application of the map- ping mechanism.

---

Taks-specifie vocab size
Increment Bleu
R-L
CN PsyQA
6k
2774
27.31
22.48
10k
6756
27.11
22.05
14k
10737
27.04
21.52
18k
14723
26.94
21.46
MHP Reddit
6k
2367
4.81
10.72
10k
5470
4.77
10.71
14k
8278
4.78
10.76
18k
11733
3.92
9.81
Table 10: Comparison result different sizes of specialized vocabulary. Especially, taks-specific vocab size is the size of vocabulary before merging. increment means the token increase relative to base vocabulary after merging the task-specific and base vocabulary into one. Bleu denotes the average of Bleu-1,2,3 and 4. denotes Rouge-L.
PsyQA Human Evaluation Guideline
You task:
Given context which is a post from an individual seeking mental support online, you are required to rank the three responses generated by three models. Evaluation contains three dimensions: fluency, coherence and professional expression.
Refore the evaluation, you required to read standard QA pairs. The responses are "gold standard" You required grasp the wording, structure, language style from these gold responses. Your impression on these responses then used evaluate how closely the generated response aligns with the standard responses terms wording, structure, and language style (i.e., the third dimension, professional expression)
Explenetion dimensions
replanation
pooéiivn eaummple
negativn saample
Flancyy
How fluent and runduble the response?
am sorry to beat your sqquet bett
aa who with properily class comminunae
vopuree respornive the
insids brekk
My
brwak with
post keep
you. You very sad
Ansicty is common
lngically condstenn within "spressin?? on?
responsiue) broke up with Gf... Since your and you
have married (not convistent)
Professianal expression
generateed the standlard wording, styl=?
cloww
stumbhedd resparmae:
standard
toneliness
lonetiness starting common everyone experience discomfert
joh your
language
environment, everynne experience discomfort adjustment
challinggss Adkitiumally our work environments offan be fast paced, which further adds the difficulty.
aijuiiment challungoo. Additiinally our work environments can ofton be fast paced, which further adds the diffccilty.
generated respenser
You shooom greal dedicattion persormmaaee investing so much in this exam's goal. Although you dide't
gersenated rmpprmee Hocally un derstared,
happoned mo, wouldd id be really ongry too. hope you
succeed, it indicates that this path may not he nuttablee for you, and it and reminder lift head and
can graduslly overcome communirane with friends mone,
take time reles, and mebe chowe he
huppyl
If you read the above all and are ready to evaluate, continue the following steps
Stop 1. Open shout RRaad them Belone Ervluciion" Canulully road the 10 stansland CA pains, graspir g the userall wording. ng at strusture, and language style thuse responses. Note you just need to have general understanding: once you proceed to step don't retura to review these standard responses any more: shous Evalustion.
Hank the
colierence, professional
Tiks
Step your subrrtt your rest mumer
Figure Human evaluation guideline.

---

Context
[QUI] +
4-
I D #
T
- DR
* #1 * 1. 1
A8S, 2 1
2 1
Vig. |H KANNL X 4. ARI
5. | 23 (2
7 1:
- 17
2 02
T I 16
-
BA ME
A
Genrated Respoase
X , 1
cb म 3
1
# A
:
# A0
18 J
(C (11 #
3 [T]
-
* 7::7
T
#
(1)
3
24
X2 A
E 2
66
- 13
1 2: /
3 43
3. A1 (R)
H.99 -"
# - (1t 102
*). 1
88 (a.
3, 3.6
3. AL
f 24 4.32
E
11
2 FF
A
22 fi.. H
Table 11: Case study on CN PsyQA. An explanation for abbreviations: M for GPT2mm +mapping, B for GPT2 and NM for GPT2 w/o mapping:

---

Context
NM
B
Context
[QUE] can we talk about Dissociation"<<sep\\DEEE i'm just wondering what other people's expe- riences with dissociation are, and if there are different types? personally, my way to dissociate is kind of attempl to evoke/"be" stronger versions of myself (always male versions though which find interesting i've no desire to be male irl, just in my head) when i'm under duress of some sort, usually pain anxiety. is this unusual? my boyfriend, he kind of "zones out", he says his eyes glaze over and he stops doing anything or really thinking anything (also in times of stress). are we both correct in calling what we experience "dissociation" "? what's your experience? i understand this kind of a taboo/sensitive topic, i'm just now getting to the point where can openly talk about it, but i'm desperately curious about edit: thanks everyone for sharing, it's a fascinating subject. it's sounding to me like what i experience is different from "normal" dissociation somehow, and i'm going to investigate this further. <lsepb>[KWD] bpd<lsepl>
the other hand, calls it dissociation when
Genrated Response NM
B
that also work for onlyt up so some of my relationship- cunllalerded before she told me how but was all the feeling more it is get them out when they say after not him to do me! dont know has been will be if you need her off some- thing as being able to your
if you felt as if you were ex- periencing opposed personal- ity types, then maybe it's im- portant to tell your friends about this because this might be useful to help you feel more confident and connected with life. please also know
first one if i am get do i do because there are time you never want people who start again in the have been had was gonna about around... always felt had, npoxges has done than know friends, but it can help feel nothing that you since. and there will be on my own life back that got help down without someone in farinilance; feel even a lot what to say byasment cyst loves me better.
if you say yourself then perhaps that could shed light into something deeper
that some people experience
their identity through distinct body parts and experiences, and that their bodys are differ- ent from what you described.
Table 12: Case study MHP Reddit. An explanation for abbreviations: M for GPT2 +mapping. B for GPT2 2 and NM for GPT2 w/o mapping;


Reviews:
[{"review_id": "h4Mike3QKi", "reviewer": "Reviewer_m6qr", "summary": "", "questions": "- For one training example, how many instances will you sample to train the task-dependent model? If it's larger than one, it's hard to tell the improvement comes from better tokenization or larger number of training examples. I suggest that testing with only using the newly constructed vocabulary, this can be served as a very important baselines to verify if the constructed vocabulary is necessary.\n- I am wondering if the proposed method works for the general tasksNon other than mental health related tasks. I feel that the proposed method is kind of \"tokenization ensemble\" so it should work for the general tasks.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and consideration in reviewing our work. We are glad you found our approach interesting and you appreciated the ease of implementation of our proposed method. \n\n1. Regarding the change in the number of training instances\n\n    We would like to clarify that sampling **will not** increase the number of training examples; while it’s true that the number of tokens in the training data may change, the source training data does not change. \n\n    In our experiments, we use the same number of training samples regardless of the sampling method. The same phrase shown in different training examples may be tokenized differently (see Figure 1), but one training example will still only point to one tokenization result, whether by sampling or by mapping. Hence, the number of training instances does not change. \n\n    We do acknowledge that we should improve our writing so that this point is more clear, thank you for pointing this out.\n\n2. Concerning the generalization of our approach \n\n    Our proposed approach is indeed a new approach to tokenization that can be generally applicable to generation tasks. As a first step,  we chose to demonstrate our approach on one domain (Mental health), so that we can perform extensive analyses and comparisons while keeping the vocabulary consistent. In the future, we plan to test the effectiveness of our approach on other domains (e.g. medical texts), and also adjust our approach so that it can be used in a  plug-and-play manner for current  LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An interesting way to handle domain mismatch and task-dependent terminologies.\n- The proposed method is simple and easy to implement.", "reasons_to_reject": "- Sampling would increase the number of training examples. Given the current experiments, it's hard to tell the improvement comes from better tokenization or larger number of training examples.", "questions_to_authors": "- For one training example, how many instances will you sample to train the task-dependent model? If it's larger than one, it's hard to tell the improvement comes from better tokenization or larger number of training examples. I suggest that testing with only using the newly constructed vocabulary, this can be served as a very important baselines to verify if the constructed vocabulary is necessary.\n- I am wondering if the proposed method works for the general tasksNon other than mental health related tasks. I feel that the proposed method is kind of \"tokenization ensemble\" so it should work for the general tasks.", "paper_topic_and_main_contributions": "This paper studies how to adapt the existing tokenizer to some domains that have many special terminologies. They propose to generate a new task-specific vocabulary. By merging the new tokens with the existing one and assigning sample scores, one sentence can be tokenized as different outcomes. They also study if better initialization affects the performance. Experimental results show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {}}, {"review_id": "9amPy7O3J0", "reviewer": "Reviewer_iXkS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and consideration in reviewing our work. We are glad to see you found our method to be effective and efficient, and you appreciated the cognitive science foundations of our work.\n\nTo answer your comments:\n\n1. About the need for more case studies.\n\n    We limited the number of case studies because of the space, but we agree that more case studies will increase the strength of our argument. Given the additional page allowed for the camera-ready, we will plan to include case studies in main body and additional cases in appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The motivations of the proposed method is supported by cognitive linguistic studies.\n2) The proposed method is proven to be effective and efficient on different pretrained models.", "reasons_to_reject": "1) More examples could be included in the case study to highlight how the mapping mechanism improves model performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a task-adaptive tokenizer for better performance on downstream text generation task. The proposed method utilizes task-specific data to construct a vocabulary with different segmentation granularity, which can be merged into pretrained model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b3ztQbKVju", "reviewer": "Reviewer_xPLS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and consideration in reviewing our work. We are grateful for your positive comments regarding the applicability of our approach, and the clarity of the motivation and methodology.\n\nTo answer your comments:\n\n1. Regarding the quantification results for the token-mapping mechanism.\n\nFrom the human evaluation and automatic evaluation, we found that\n\n(1) With smaller datasets, it becomes challenging to ensure adequate sampling for the new tokens. We can leverage the mapping mechanism to use the learned representations from the sub-tokens to initialize these new tokens. This would also considerably guarantee the model’s representation ability of new tokens, which explains the significant performance drop on fluency and local consistency, which points to a defeat in foundational representation ability, in human evaluation when mapping is absent.\n\n(2) With larger datasets, we found the mapping mechanism continued to bring  benefits. In general, we found that leveraging pretrained token knowledge is always helpful. As seen in the human evaluation, when the mapping mechanism is absent, the performance on fluency and logical consistency is not that bad compared to the results on scarce data, but still causes a decrease. However, mapping mechanisms may cause less task-specific terminologies than without mapping, which reflect a drop in automatic evaluation (mainly RougeL).  This is due to the increased possibility of sampling out pre-trained tokens influenced by mapping mechanisms. \n\nHence, this is a tradeoff that should be decided depending on the application and dataset size. While we leave improvements in the mapping mechanism to future work, we will include statistics for change in frequency (based on your suggestion) in the final version of our paper. \n\n2. Regarding the performance drop of non-mapping mechanism in human evaluation on the MHP dataset.\n\nWith smaller datasets, it becomes challenging to ensure adequate sampling for the new tokens, which explains the significant performance drop in fluency and local consistency in human evaluation when mapping is absent. One solution is what we proposed, namely to use a mapping mechanism to alleviate this problem. Another possible solution is to resample at every training epoch, which can be  a trick to enhance the chance for each new token to be sampled out. We will add this experiment in the camera-ready version, and also include these additional clarifications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses the prevalent problem of downstream domain adaptation of foundational language models through a new lens that goes beyond the common approach of post-training fine-tuning to propose a strategy of task-adaptive tokenization. This new strategy proposes building a tokenizer from the task data with variable segmentation and introduces a method for integrating with pre-existing training tokenization to seamlessly extend to off-the-shelf language models.\nThe strategy has the potential to be applicable across multiple downstream domains and across languages – showing success in both character and word-based languages in included experiments.\nThe paper is very clearly written, easy to follow and understand as a reader with the motivation, methodology, and experiments explained in detail.\n", "reasons_to_reject": "While the paper provides two evaluation methods (automatic and human) for the main proposed task-adaptive tokenization strategy, further quantification of the results for the proposed strategy of token mapping is suggested. For example, from the results of human evaluation analysis, the paper claims that the use of the token mapping strategy for non-overlapping tokens is “crucial to ensure a robust token representation system”. Further quantification of the change in frequency of response tokens for original, overlapping, and non-overlapping tokens is suggested to provide more background to the benefits of employing the mapping strategy relative to model generated responses.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the novel strategy of task-adaptive tokenization as a method to adapt generation pipelines to specific downstream tasks and applies it to long-form generation in the domain of psychological question-answering tasks in both Chinese and English languages.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggestion to address drop in performance for non-mapping task-adaptation tokenization in MHP dataset in human evaluation section. More explanation in regards to performance drop due to MHP’s scarcity of data is offered in Appendix E but would benefit to also be addressed in the human evaluation section as well if possible."}}]

Author rebuttal:

Meta-review: The paper proposes a task-adaptive tokenization method and shows its efficacy on long-form generation in mental health. The method is simple and easy to integrate with the potential to be applicable across several downstream tasks and languages. The paper can be further strengthened by adding results on other tasks and by adding the analysis as provided in the rebuttal.
Decision: Accept-Main