Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
Authors: Daniel D'souza
Julia Kreutzer
Adrien Morisot
Ahmet Üstün
Sara Hooker
Venue: NeurIPS 2025 poster ()
Keywords: large language models
controllability
generation conditioning
data taxonomy
Primary area: deep_learning
TL;DR: An optimization approach that bridges the gap between training and inference techniques via a highly detailed taxonomy of data characteristics to explicitly control generation attributes and implicitly condition generations during inference.

Abstract: One of the most profound challenges of modern machine learning is performing
well on the long-tail of rare and underrepresented features. Large general-purpose
models are trained for many tasks, but work best on high-frequency use cases.
After training, it is hard to adapt a model to perform well on specific use cases
underrepresented in the training corpus. Relying on prompt engineering or few-shot
examples to maximize the output quality on a particular test case can be frustrating,
as models can be highly sensitive to small changes, react in unpredicted ways
or rely on a fixed system prompt for maintaining performance. In this work, we
ask: Can we optimize our training protocols to both improve controllability and
performance on underrepresented use cases at inference time? We revisit the divide
between training and inference techniques to improve long-tail performance while
providing users with a set of control levers the model is trained to be responsive
to. We create a detailed taxonomy of data characteristics and task provenance to
explicitly control generation attributes and implicitly condition generations at
inference time. We fine-tune a base model to infer these markers automatically,
which makes them optional at inference time. This principled and flexible approach
yields pronounced improvements in performance on examples from the long tail
of the training distribution. Overall, we observe lifts of 5.7% across all tasks.
However, treasure markers are particularly effective at finding difficult to obtain
gains in the long-tail. We observe relative lifts of up to 14.1% on underrepresented
tasks like CodeRepair and absolute improvements of 35.3% on length instruction
following evaluations.

Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markers
Daniel D'souza
Julia Kreutzer Cohere Labs
Cohere Labs
Adrien Morisot
Ahmet Üstün' Cohere
Sara Hooker*t
Cohere
Adaption Labs
Abstract
One of the most profound challenges of modern machine learning is performing well on the long-tail of rare and underrepresented features. Large general- ceneral-purpose purpose models are trained for many tasks, but work best on high-frequency use cases. After training, it is hard to adapt model to perform well on specific underrepresented in the training corpus. Relying on prompt engineering or few-shot examples to maximize the output quality on particular test case can be frustrating, as models can be highly sensitive to small changes, react in unpredicted ways rely on fixed system prompt for maintaining performance. In this work, we ask: Can we optimize our training protocols to both improve controllability and performance on underrepresented use cases at inference time? We revisit the divide between training and inference techniques to improve long-tail performance while providing users with set of control levers the model trained to responsive to. We create detailed taxonomy of data characteristics and task provenance to explicitly control generation attributes and implicitly condi inference time. We fine-tune base model to infer these markers automatically, which makes them optional at inference time. This principled and flexible approach yields pronounced improvements in performance the training distribution. Overall, we observe lifts of 5.7% across all tasks. However, treasure markers are partieularly effective at finding difficult obtain gains in the loig-tail. We observe relative lifts of up to 14.1% on underrepresented tasks like CodeRepair and absolute improvements of 35.3% on length instruction following evaluations.
cases
generations
examples from the long tail
Introduction
Large language models (LLMs) are expected to perform well on many different tasks. Therefore, training data heterogeneous mix, where instances can vary greatly terms of format, contents,
tasks, and languages, e.g. code generation [3 33 20 63 vs. MCQA 48 39). At inference time, data points are not equally relevant, but is often prohibitively expensive to go back and change the training distribution for each individual inference request. Hence, there is mismatch in the distribution training and inference time: training time distribution often determined by ease of access to prior data collections and prior data augmentation efforts, while at inference time, new use cases might be underrepresented the data but highly relevant the user.
'Equal Mentorship. 'Wrk done at Cohere Labs. Corresponding authors: [danieldsouza, juliakreutzer, ahmet (hmet)@cohere..co com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training Distribution
<longth, bucceet> constic
20000
medium
20006
long
25300
10008
5000
400
800
800
1,000
Tokens
Distribution at Inference
if illenghh_ bociees
iflength, Beenggh_bucket-mediiee
FFlength.bucketllng bucket-long
000
500
000
900
200
500
200
000 600 1000 1200 3400
600 5209 1-603
000 1960 1200
Toleems
Takens
Takens
Figure 1: Tapping into Distributions: (above) illustrates the representation of various length buckets in the training distribution. (below) demonstrates the flexibility of the marker intervention on the mArena Hard test distribution. By modifying the [Length_bucke>>.. bucket>>.<//engeeee </length, </length_bucket>> bucket> marker, the model can effectively tap into diverse training distributions, even for underrepresented length buckets.
To overcome this mismatch, techniques have been proposed improve the conditioning of the output generation at inference. These involve prompt engineering 57
60
54
multi-shot examples
5 29 55 30), chain-of-thought [53] 52 or decoding strategies [46
49
However, these
approaches place an enormous burden on practitioners and developers to anticipate what strategies deliver the best performance. Furthermore, the effectiveness is dependent on the exact configuration
for a particular model, e.g. the order of multi-shot examples plays role (32), and the wording of the prompt 22.. In this work, we ask Can we optimize our training protocols to both improve controllability by the user and improve performance on rare use cases at inference time?
Our approach amounts to building treasure map of hyper-detailed task-specific markers, to allow for real-time automatic targeting of long-tail features during inference. We note that some of the carliest generative models have used tags to improve performance. However, these often targeted a single feature at a time or were applicd uniformly to an entire dataset. These early tags fell out of favor over the last few years, with the focus turning to prompt engincering for users to guide the generation themselves. However, there have been a few wider ecosystem changes which prompt (no pun intended) revisiting the paradigm of adding markers to training, and also motivate this work: 1) LLMs are now used by a far wider group of everyday users who can't 't be expected to be familiar with the complexities of prompt engineering, 2) Many models are now served using an API which means training markers can be added automatically behind the API call (not visible to users), and hence can be far more complex and varied guide and improve generations.
Our work motivated by these two trends. We take a far wider view of training markers and explore setting where single data point can have up to 90 complex characteristics. We describe these as Treasure markers, introduced at training time to provide map to guide towards higher performance at inference time. We motivate that this approach is particularly beneficial for long-tail modeling. Our goal is that the treasure map approach robust at test-time, so we also aggressively experiment with marker dropout during training. This is akin to asking the model to still find the treasure even with missing clues.
In this work, our primary contributions are as follows:
1. Introducing a more general framework for controllability. We show that explicitly targeting controllability during training leads to pronounced gains at inference time, with little burden placed on the user. Training markers leads to significant downstream gains, ranging from win rates increase on open ended generations of 5.7% on ArenaHard [27 across the entire distribution of

---

tasks relative to model with no tags. We also see radical improvements in instruction following. with 35.3% absolute decrease in violation rates on length instruction following tasks. Our training marker framework offers remarkable flexibility, allowing for control over both aspects of form (output format, length) and semantic qualities (quality, tone, style) while also being completely optional at inference, because the markers can be inferred accurately.
2. Long Tail Lifts The training with explicit markers is an effective method for leveraging long- tail features inference time, unlocking nigh-performance even for the distributions that are underrepresented in the training data. While our framework enables a relative improvement of 7.9% on Code tasks over the baseline, observe relative lifts of up 14.1% on tasks like CodeRepair, which are highly underrepresented in the training data.
3. Modeling Underlying Relationships: We demonstrate that our approach effectively models underlying relationships in the data, evidenced by drastic violation reduction (36.58% - 1.25%) in length-constraint evaluation, despite never seeing a training sample with a prompt instruction designating length constraints.
2 Methodology
2.1 Overview of Training Time Markers
We condition the output sequence given an instruction & with added training markers m:
plyl,, II p(ue m, Mcs).
These markers encompass several different attributes of the data, including estimated quality scores, domains, and languages (2.2), which store list of markers associated with given data point (see Table for an example). This template is treated as natural language and encoded with the same tokenizer as the text. We include the markers in both input (appended to the prompt) and output space (prepended to the completion), to induce the model to associate the properties of the generations with these characteristics. This reduces the burden the practitioner or researcher inference time, as the model learns to infer the correct markers.
The finetuning objective thus becomes to minimize the negative log likelihood of the target generations includ- ing the template, given prompt with an optional input template:
(MARKER_
(domain) Sciences (/domain) (language) French (/language) \/MARKER_LAA _IIET)
D
Σ
log (B(d ma dropout(m.)) (ma), xT)
(2) Table 1: An example list of training time markers formatted in a standard- ized template.
D
This approach ensures that the model learns to faithfully generate and adhere the training markers when provided on the prompt side.
Training markers dropout To avoid the model from becoming overly reliant on markers for completion or learning trivially replicate the markers, we employ dual dropout strategies (dataset- level, sample-level) on the prompt space. In dataset-level dropout, we completely remove the training markers from the prompt for random selection (defined percentage of the dataset). sample- level dropout, we completely remove random subset of training markers from each example (defined as a percentage of all markers associated with given example). ensure the model consistently produces markers at inference time, we do not introduce dropout on the generation side.
2.2 Taxonomy of Training Markers
We develop a comprehensive taxonomy around distinct groups of desired characteristics to capture key attributes of the training data, such as quality of the data, style, format, domain, and task. Table 8 contains the taxonomy with definitions and the set of valid marker values. Our goal was to study a broad range of markers encompassing both deterministic properties (e.g., length, language) and annotatable attributes (e.g., domain, task) to better characterize the SFT data mix. We selected
3

---

represenaation
25%
+9.3%
20%
50
Compostion
Abcalute WR
15%
10%
+5.7%
0%
Code
Cultuee
Gane
Sociabbc Science
Medical
Lagal
TreasureMarked
Figure Long tail domains benefit more from training markers: (left) Domain distribution the training data used to fine-tune the model. (right) Improvements in win rates over the baseline against Gemma2-9B on both majority and minority subsets on Saena-Hard-Autt dataset [27]. We group the test data into two sets of domains that have high (>5%) and low (< 5%) presence in training data.
genuraten kingth langutiinngth
2.6-
gunoraten kangeh target lanugh
1.6
genuration kingth langth
1.6-
1.4
1.2
1.0
1.0
0.8
0.8
0.6-
0.6
0.6
200
100
400
target length
length
target length
(a) Baseline Model
(b) TreasureMarked
(c) TreasureMarked (fxxdd
Figure 3: Modeling data features flexibly with training time markers: Results on the length dataset. While (a) the baseline violates
instruction following on the LoneaaEyl--eegth--nnntercee the length constraint with 36.58% (b) using the TreasureMarked model and allowing to infer tags on the exact same dataset reduces the violation 24.7% 1.7 (c) Conveying the requirement via explicitly inserted length marker to the prompts, the TreasureMarked model violates the instruction on only 1.25 % of the dataset.
this taxonomy with inference- use cases mind: properties like quality, tone, style, and completion length are very desirable to control inference. Moreover, to assess gains across both repl-represented and low-frequency features, we intentionally designed the taxonomy to capture the inherent skew in data distributions. To that end, add hyper-detailed markers for task, domain and code type which tend to have highly skewed frequencies with some instances occurring far more frequently than others.
To assign markers samples in the training dataset, we utilize dataset-related information whenever possible and use an LLM to tag missing meta ceeta-informatin.. Specifically, we use the multilingual open- weights model Command R+ for tagging of markers for <domain>, <task>, <format> whenever unavailable from the dataset. To improve tagging performance, we use detailed definitions paired with few-shot examples to provide context for markers during annotation. We add markers across 23 languages, so use in-language few-shot examples in each language.
Our extensive set of 90 unique markers fall into 13 categories such as Length, Style. Format, Quality, Source, Domain, Task. We include an extensive description of all markers in appendix We describe the most frequently referenced categories below:
Length: are markers that allow for control of completion length. It includes a level of granular- ity ranging from <lennth_tokens> tokens> and <length_sentences" sentences> to broader categories such as concise, medium, and long-
Language: <lang> describes the language the completion is written in (i.e. Arabic, Japanese), enabling the model to improve language-specific generations and reduce language switching during
RRleeaee blog of Command R+ https://cohere. com/booog/commmaddoooo-oo -e-taaddd------------- waatts-mccooooftaaaee

---

inference. <code_type> is specifically used to identify programming languages for coding-related tasks (i.e. python, c++).
<quality> provides a measurable score indicating the quality of a sample, often derived from human annotations Reward Model (RM). We also create categorical marker <quality_ buckkttt_bbke>ttt by using quartiles within language-specific subsets into (,, 43, offering broader description of quality.
Quality:
Domain: overarching category of the knowledge required to answer given prompt (i.e. Sciences, Technology, Medical). We annotate domain markers either using LLM tagging or derive from the source of the dataset for domains like Math and Code.
Task: <task> helps capture more fine- fineggriined differences in task characteristics within domain (i.e. summarization, reasoning, openended, explanation). Similar to the domain marker, we use LLM tagging or the data source information for obtaining task markers.
2.3 Experimental Set-up
Training with Markers We use a 7-billion parameters proprietary base model which is pretrained using data mixture that consists of texts from 23 languages covering half the worlds population. We train our base model on training corpus containing 2. 2.7M examples made up of our mixture of instruction-style data sources. See Appendix B for full list of languages covered and more granular details about the training protocol.
Inference Settings At inference time, evaluate performance gains under two different settings. In the default setting, which we refer to as "TreasureMarked"" we do not fix any of the markers at inference. This setting asks: Has the model learnt to infer the right markers without any intervention? In the second setting which we refer to as "aree (fixed)", we explicitly hardcode some of the markers inference. This asks: we manually set the value of some markers, can we drive gains in performance? This is very reasonable for cases like quality, where we always want to steer model behavior towards higher quality generations.
We compare both "TreasureeMarke" and "TreasureMarked (fixed)" against a model trained on the same data, but without added markers that we refer to as Baseline. This allows for clean comparison, and controls for the same amount of data seen in both variants.
Core experimental variants and ablations In the next section, we evaluate a variety of ways model trained with markers shines at inference time. We inspect three axes of control: (1) quality in section 3.1.1 (2) length in section 3.3 and (3) language in section 3.5). Furthermore, show how long-tail examples benefit from markers, even when only inferred inference time (section 3.T), specifically in coding tasks (section 3.2) and for long generations (section[3.) We present key experimental ablations, including understanding the impact of dropout applied to markers on downstream performance at inference time (Section 4).
2.3.1 Evaluation
Open-ended generation quality We evaluate the impact of markers on both the English Arena- Hard-Auto v0.1 (27), and translated version of this dataset, m-Arena Hard [8] used for multilingual evaluation. Harna-Hard-Auto is challenging open-ended generation benchmark with prompts selected from user queries on Chatbot Arena. We measure Win Rate % against our Baseline model using GPT-4o
Task-specific evaluations In addition, we evaluate the models on benchmarks specific to tasks such as code (generation, repair, translation) and length conditioned instruction following narrow in on long-tai effects and controllability levers. We introduce each of these evaluations within the respective results sections.
We used gpt-4o-2024-05-11 13 as our judge model. Details: https://platform.co opw.aa.rrooooooii.......o cocm//os// models/gpt-40
5

---

3 Results
3.1 Impact of Treasure Markers on Open-Ended Generation
Open ended performance gains. We measure Win Rates (%) of the Baseline and TreasureMarked models against Gemma2-9B [5]] as a common point of comparison, visualized in Figure 2 We first consider our Treasure variant, markers are only included in training but are inferred from the model itself during inference. Overall, we obtain an absolute increase of 5.7% in Win Rates from 32.1% to 37.8% across all tasks. This is reassuring, because shows that markers at training time of the TreasureMarked model can already make positive change inference time, even when only inferred by the model itself, and even the respective markers are rarely seen during training (c.g.. for underrepresented domains).
Performance on the long-tail One of our core hypotheses is that treasure markers will be particularly helpful at preserving or unearthing gains on the long-tail. To validate this hypothesis, we evaluate performance post-training on domains represented with different frequencies in the training-set. As seen in Figure 2 SocialScience ence, Sciences, Finance, Medical, and Legal domains are particularly sparsely represented in the training data, each making up less than 5% of the training data. In contrast, Code is best represented in the training dataset. With inferred treasure markers, while there an improvement of +5.7% across the nigher-represented domains, we observe an even more pronounced gain of +9.1 % in the underrepresented domains.
3.1.1 Fixed Treasure Markers
We also explore adding explicit markers in Treusure- Murked (fixed). Here, we specifically target quality and ask Can we control the generation quality of the model markers? To test this, we measure generation quality on m-ArenaHard [8] across 23 languages, by only adding markers related to quality. For each value [1,2,3,4 4] of kquality_buckee bucket>, we also include a <quality> score in conjunction with it. To obtain the <quality> score, we pick the 95% percentile calculated language-wise from the samples in the training data from each respective bucket. As evalu- ation, we measure the generation quality by the same Reward Model used to score the data during training to compute win rates against the Baseline model.
60%
Baselin e
58%
latent feature, using training time
56%
Rates
54%
52%
50%
48%
46%
4 <qualityybucket> </quality buckett>
Figure 4: Levers for Controlling Quality: Changing the <quality>, equality_buckett
the amount of control intro- duced by training time markers with win rates under the RM going from 48. 2 + 56.5% just by chang- ing <quality>, <quality_buce buckkt> at inference. These results showcase the potential of our frame- work, where markers representing a desired quality metric used during training yields control levers to leverage generations that tap into that quality metric at inference time.
Tigure@demonsee
markers at inference time provides control over generation quality with Win Rates (as measured by internal Re- ward Model) going from 48.21% + 56.5% over the Baseline model, demonstrating successful control over quality as annotated in the training data.
3.2 Impact of Treasure Markers on Targeted Performance of Specific Sub-tasks
3.2.1 Code Performance
For code, we evaluate our model on three tasks from HumanEvalPack (38 dataset, and measure pass@1 rates. We use CodeSynthesis, CodeRepair, and T covering python, rust, java, javascript, go, c++. These map to the following task markers in our taxonomy: CodeGeneration. CodeFix, and CodeTranslation.
'The Code Translation task is created by an all-to-all mapping between the 6 languages in HumanEvalPacck

---

500000
774
TreasureMarked (fixed)
Relative Improvement
12%
400000
Frequency
10%
300000
8%
6%
200000
100000
0%
Generation Translation
Repair
Generation Translation
Repair
Figure 5: Improvement on the Long Tail for Code tasks: (left) Frequency of coding <task>s in the training dataset. (right) Despite being poorly represented in the training data, CodeRepair achieves 14. I % relative improvement by leveraging targeted markers during inference further improving on the performance from the TreasureMarked model with inferred markers.
Original AlpeaEval-LL
TreasureMarked (fixed)
Anawer the following instruction using 199 words or less.
What are the names of some famous that started their careers Broadvay?
What are the names of some famous actors that started their careera on Broaduay?
<MARKER_LIST> <length_tokees>>1g0 hot..//keeeeei////lggggb..ccmce </MARKER_LIST
Table 3: Examples length control strategies: (left) Original instruction from AlpacaEval-Li
dataset; (right) Modified instruction with constraint in the marker list.
During training, code comprises of 27.2% of the overall training corpus. However, specifically pick this domain because the distribution of coding subtasks differs significantly frequency in the training corpus, shown in Figure 5 CodeRepair and on
very rare coding subproblems, while CodeGeneration is heavily represented at 75.8% within the coding data.
Long-tail tail gains We observe the largest gains on the long-tail code tasks, As seen in Figure whether we provide the markers (arresureMarked (fixed) or the model infers them, both rare coding problems (CodeTranslation and CodeRepair) show large improvements of 0.597 - 0.636(+3. .9% absolute, +6.5% relative) and 0.216 0.246(+3% absolute, +14.1% relative) over the baseline respectively. We note that these gains are far higher than the gains observed for the far more frequent task of CodeGeneration which only improves 0.406 -> 0.419 (+1.3% absolute, +3.2% relative). This shows that our framework benefits all parts of the distribution, but has disproportionate success enabling large lifts to highly infrequent features during training.
3.3 Length Control in Inference Time
Model
Violation
Win Rate
Baseline TreksareMarked (fixed)
36.58%
14.36%
24.74%
19.48%
To assess the impact of length conditioning during infer- ence, we benchmark on the AlpacaEval-LI dataset [61), which evaluates how faithfully LLMs adhere to length constraints. We complement the measurements for length violation with Win Rates(S5 %) by evaluating valid samples against the dataset provided completions using GPT-4o. We establish baseline using completions generated by the Baseline model. Following a similar approach to [61], we assess Violation %) as the proportion of samples exceeding the specified length constraint. We include additional details in Appendix B
1.25%
21.22%
Table 2: Length Instruction Following & generation quality Alpaca-Eval L.I.
Improvements to length control. In Table 2 we show improvements of up to 35.3% in length violation rates. This pronounced improvement results in a mere 1.25% remaining violations for this evaluation set (essentially close to saturating performance markers are not explicitly provided but inferred directly by the model, we observe up to 1 11.8% absolute decrease in violation rates. These improvements to instruction following non-trivial, and
this evaluation). Even when the treasure
7

---

also lead to overall win-rate gains of up to 6.86%, ensuring quality is not compromised as length constraints are enforced.
3.4 Machine Translation
To study the effects of the markers on machine translation, we benchmark on WMT [0] and report translation performance from English to 22 languages (en + 2:)) based on the languages seen in pretraining. We use XCOMET-XL (7) for evaluation, a state-of-the-art machine translation evaluation metric [2] via the comet compare" library conduct statistical significance analyses based on paired t-tests and bootstrap resampling.
Table I shows the results with the relative improvement over the Baseline. Training the model with markers and using them at inference time improves performance on languages (es, id, it, pt, ro) significantly with up to 1.18 point gains, while retaining performance on all other languages. This constitutes a remarkable improvement, especially given that the training data, up to the markers, is identical. According to the metric delta analysis in [21]. improvements of such magnitudes are very likely to be confirmed in human evaluations.
de
fr
he
hi
id
Raseline
6865
.7885
.8824
.7663
.8449
.7099
.789
.7214
51588
.776
.8126
TreaswreMarked ked (feee)
6844
.755
8848
.7500
.8307
.7072
.7948
.7166
.2299
.7874
.8194
(-0.21)
(+0.65)
(+0.24)
(+0.37)
(+0.58)
(-0.27)
(+0.58)
(c0.48)
(+0.71)
(+1.14)
(+0.68
XX
pt
ru
Bazeline
7368
.7281
8103
.7788
.82
8048
.7675
.6669
76255
.7533
.7176
TreasureMarked (fxed)
7342
.7318
.8117
.7546
8281
8166
.7627
.6723
.7755
.756
.7200
(-0.26)
(+0.37)
(+0.14)
(.0,32)
(+0.61)
(+1.18)
(.0.48)
(+0,54)
(.0.50)
(-0.33)
(+0.24)
Table 4: X-CometXL scores 77 on WMT 24++ test sets [10] Bold differences are significant at p S
0.05 according a paired T-Test and bootstrap resampling 22 implemented in compare
3.5 Language Control in Inference Time
As the final set of results, we focus on the effect of our training markers on ensuring a model responds in the language specified by the user. To evaluate this, we use the Language Confusion Benchmark 335 which measures the ability of a model to follow cross-lingual instructions such as "Respond in French...", to request completions in another language. We measure performance the Complex Prompts subset of the cross-lingut benchmark across 14 languages. Following [55, we measure Line-level Pass Rate (LPR) that only deems response "correct" if all lines in the generation match the user's desired language.
Table Siswwss results across 14 languages. Our model with training markers significantly improves language control performance in 13 out of 14 languages with an absolute gain of 10.98% on average across 14 languages, showcasing a remarkable improvement in controllability of inference time. We observe the largest gains for Russian (+18.6%) and the lowest gains for Chinese (+5.5%).
de
fr
hi
id
1t
ja
ko
ru
vi
zh
Avg.
Buseline 81.1
71.9
70.4
68.0
72.5
68.4
68.0
68.6
Morkeu (foed)
88.4
84.4
82.7
79.8
73.7
66.7
82.8
83.8
85.0
72.2
86.6
78.6
82,8
62.6
79.58 (f 10.98)
Table 5: Line-level pass rate on Complex Prompts from the Language Confusion Benchmark [35]
4
Key ablations and Discussion
How do markers interact? We perform an additional ablation the AlpacaEval--L dataset from section 3.3 to study the effect of adding more useful markers at inference time. In ad- dition to the <okesth marker that conveys the explicit length constraint, we anno- tate and add the <domain> marker, which we suspect carries implicit length biases (c.g. le- gal text might be longer than conversations), but should add helpful context to the prompt.

---

With this we ask If multiple markers are added inference, do their effects add up or cancel out?
Model
Violation
Win Rates
TreasureMarked (fixed)
1.25%
21.22%
+ <doomin>>
1.87%
24.72%
From Table 6 we observe that the effect of adding <domain> has a positive impact on the generation quality with +3.5% jump in win rates albeit at the cost of a slight increase in Violation Rate(%). This indicates that there are multidimensional relationships that form between treasure markers during training and can be leveraged in conjunction to achieve desired characteristics at inference.
Table 6: Multidimensional control (Alpaca-Evar LI): Adding <domain> marker improves generation quality and hence Win Rates by +3.5% working in conjunction with Length_tokensse without hurt- ing the length control.
What is the impact of the dropout on the marker prediction?
To understand the impact of the marker dropout ($ 2.11. we train three variants with dataset-level dropouts of [0%, 50%, 70%] while sample-level dropout is fixed to 50%. Our goal with dropout is to teach the model to infer markers without needing explicit guidance at inference time. However, too much dropout may impede the model from learning key patterns between tags and output properties. To evaluate this, we calculate the accuracy of the markers inferred by the model to the underlying markers assigned to m-Arena Hard and average across all 23 languages 8
In Table we observe that the least extreme dataset-level dropout variant 0_50 struggles to predict the correct marker at inference time. This is expected performance, since at training time, 0% dropout of markers across the dataset implies all training sample prompts have mark- ers associated with it which makes it overly de- pendent on the presence of markers at inference time. At inference time, as this not provided, accuracy is very low at 3.42%. We note that at both 50% and 70% dataset level dropout, we observe similar final abilities to infer the correct
dataset_ sample
<domain>>
<tesk>
<format
<lang>
0_50
3.3%
1.9%
7.3%
1.2%
50_50
53.6%
47.4%
99.2%
70_50
75.1%
51.4%
46.8%
99.1%
Table 7: Effect of dropout on marker predic- tion. Using no dropout (dataset-level) prevents the model to learn predicting the correct marker across categories, hence, hurts the flexibility of our framework.
markers. Given this, unless specified elsewhere, 50% dataset-level dropout the default specification used throughout experiments since it strikes the best balance between learning and generalization.
Related Work
The idea of marking inputs goes back works in neural sequence prediction before LLMs. The motivation was similar: to leverage discrete features to overcome data sparsity or imbalance and introduce levers of control. There are many individual works targeting one or two characteristics at time, targeting special models and datasets, spanning diverse aspects such as language [17], quality 6 50 [3 and domain 19 4. We build a much more general framework on top of vast multi-task training corpus. We design flexible approach that can be used for any text generation task with many characteristics, encompassing the above. Furthermore, we explicitly care about improving performance on the long-tail of underrepresented features.
LLM control has been pursued with respect to web domains in pretraining 18 and length in finetuning [1]. At inference time, values for these markers are specified to steer the generation. [14] further proposes cooldown schedule in pretraining going from tagged data untagged data in order to not require prefixes during inference. We focus on the instruction finetuning stage and incorporate nuanced muuti-dimensionn markers (i.e. the user can specify length and domain and format). We circumvent a cooldown schedule with dropout, hence not requiring a complete population of markers inference time.
use continuous embeddings learned for special tokens representing markers in training to condition predictions for specific tasks at inference time. In our case, we directly embed prefixes with the same vocabulary as the L.M,, smoothly integrating them into the sequence. In our experiments in section 3.3
Related prefix and prompt tuning methods 28 26
we found that this helps format following

---

even when specified in natural language and not markers. We include more detailed discussion of the relevant literature in Appendix C
Limitations
Firstly, although we design a comprehensive taxonomy for our training markers, there can be alternate categorizations of domains and tasks, and other data properties. However, our work primarily showcases the benefits of various data markers and how to leverage them during training (to model) and inference (to guide model outputs). We also acknowledge that our annotation setup restricting each example to single value per marker was adopted for simplicity. In practice, examples often span multiple overlapping concepts. Therefore, we believe that an extended taxonomy and annotation setup would likely further improve the results we demonstrate.
Secondly, conduct experiments using 7-billion parameters language model. We opt for 7B parameter scale to be able to run more experiments with different test cases as presented in the paper, instead of using much larger model size with very limited experiments due to the underlying compute costs. However, considering a larger model could learn to model training-tinn teaninngt-mmeeete markers with higher performance, believe that our findings hold for larger model sizes.
Finally, we showease the effectiveness of learning training-time markers the supervised fine-tuning phase. We did not experiment with learning these markers in the pre-training phase as pre-training requires multiple degrees of higher costs. Therefore, we leave this exploration to the future work.
7 Conclusion
In this work, we proposed adding markers to training data to map out potential "treasures" that can be retrieved at inference time, such as specific task configurations or quality characteristics. In our experiments on multilingual instruction-finetie tool to execute control (quality, length, output language) over generations, and at the same time have beneficial effects for generation quality of underrepresented portions of the training data, such rare coding tasks. We found that dropout of training markers trains the model infer missing markers at inference time. With this flexibility, we allow users to "hunt treasures" without having to tediously engineer prompts or few-shot examples for optimized performance.
we showed that these markers are a powerful
Acknowledgments
We thank John Dang, Yiyang Nan, Thomas Euyang, Tom Koemi, Tom Sherbone, Manoj Govindas- samy, Cécile Robert-Miche Miccho,, Leila Chan Currie, and other colleagues at Cohere and Cohere Labs for their support and thoughtful feedback.
References
[1] Sweta Agrawal and Marine Carpuat. Controlling text complexity in neural machine translation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages 1549-1564, Hong Kong, China, November 2019. Association for Computational Linguisties.
[2] Sotiris Anagnostidis and Jannis Bulian. How susceptible are 1lms influence in prompts? arXiv preprint arXiv:2408. 11865, 2024.
[3] Walid Aransa, Holger Schwenk, and Loic Barrault. Improving continuous space language models auxiliary features. In Marcello Federico, Sebastian Stüker, and Jan Nichues, editors, Proceedings of the /2th International Workshop on Spoken Language Translation: Papers, pages 151-158, Da Nang, Vietnam, December 3-4 2015.
[4] Denny Britz, Quoe Le, and Reid Pryzant. Effective domain mixing for neural machine transla- tion. Ondiej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Kochn, and Julia Kreutzer,
10

---

editors, Proceedings of the Second Conference on Machine Translation, pages 118-126, Copen- hagen, Denmark, September 2017. Association for Computational Linguisties.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[6] Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. In Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference on Machine Translation (Volume I: Research Papers), pages 53-63, Florence, Italy, August 2019. Association for Computational Linguistics.
[7] Pierre Colombo, Nuno Guerreiro, Ricardo Rei, Daan Van, Luisa Coheur, and André Mar- tins. xcomet: Transparent machine translation evaluation through ane-ggained error detection. Transactions of the Association for Computational Linguistics, 2023.
[8] John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Made- line Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research breakthroughs for new multilingual frontier. arXiv preprint arXiv:2412.04261, 2024.
[9] Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020.
[10] Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, et al. Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects. arXiv preprint arXiv:2502. 12404, 2025.
[11] Weston Feely, Eva Hasler, and Adrià Gispert. Controlling Japanese honorifics English-to- Japanese neural machine translation. In Toshiaki Nakazawa, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Nobushige Doi, Yusuke Oda, Ondfej Bojar, Shantipriya Parida, Isao Goto, and Hidaya Mino, editors, Proceedings of the 6th Workshop Asian Translation, pages 45-53, Hong Kong, China, November 2019, Association for Computational Linguistics.
[12] Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei. Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifcoluwa Adelani, Mariana Buchicchio, Chrysoula Zerva, and Alon Lavie. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Barry Haddow, Tom Koemi, Philipp Koehn, and Christof Monz, editors, Proceedings of the Ninth Conference on Machine Translation, pages 47-81, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[13] Markus Freitag, David Vilar, David Grangier, Colin Cherry, and George Foster. A natural diet: Towards improving naturalness of machine translation output. In Smaranda Muresan, Preslay Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguis- tics: ACL 2022, pages 3340-3353, Dublin, Ireland, May 2022. Association for Computational Linguisties.
[14] Tianyu Gao, Alexander Wettig. Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Meta- data conditioning accelerates language model pre-training arXiv preprint arXiv:2501.01956 2025.
[15] Matthew Honnibal and Ines Montani. spaCy 2:
Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017.
[16] Laura Jehl and Stefan Riezler. Document-level information as side constraints for improved neural patent translation. In Colin Cherry and Graham Neubig, editors, Proceedings of the 13th Conference of the Association for Muchine Translation in the Americas (Volume 1: Research Truck), pages 1-12, Boston, MA, March 2018. Association for Machine Translation in the Americas.
11

---

[17] Melvin Johnson. Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilinuual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2017.
[18] Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: 058888 2019.
[19] Catherine Kobus, Josep Crego, and Jean Senellart. Domain control for neural machine transla- tion. In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 372-378, Varna, Bul- garia, September 2017. INCOMA Ltd.
[20] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533 2022.
[21] Tom Koemi, Vilém Zouhar, Christian Federmann, and Matt Post. Navigating the metrics maze:
Reconciling score magnitudes and accuracies. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1999-2014, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[22] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388 388-395, Barcelona, Spain, July 2004, Association for Computational Linguisties.
[23] James Kuczmarski and Melvin Johnson. Gender-aware natural language translation. Technical Disclosure Commons, 2018
[24] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
[25] Samuel Larkin, Michel Simard, and Rebecca Knowles. Like chalk and cheese? on the effects of translationese in MT training. In Kevin Duh and Francisco Guzmán, editors, Proceedings of Machine Translation Summit XVIII: Research Track, pages 103-113, Virtual, August 2021.
Association for Machine Translation in the Americas.
[26] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-effict prompt tuning. In MariieFrancnne Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[27] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and lon Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 1939, 2024.
[28] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 4582-4597, Online, August 2021. Association for Computational Linguistics.
[29] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' OoHoro Jeff Wang, Luke Zettlemoyer. Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages
12

---

9019-9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties.
[30] Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. Cutting down on prompts and parameters: Simple few-shot learning with language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022. pages 2924-2835, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[31] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
[32] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[33] Dung Nguyen Manh, Nam Le Hai, Anh TV Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi DQ Bui. The vault: A comprehensive multilingual dataset for advancing code understanding and generation. arXiv preprint
2023.
[34] Kelly Marchisio, Jialiang Guo, Cheng-I Lai, and Philipp Koehn. Controlling the reading level of machine translation output. In Mikel Forcada, Andy Way. Barry Haddow, and Rico Sennrich, editors, Proceedings of Machine Translation Summit XVII: Research Track, pages 193-203, Dublin, Ireland, August 2019, European Association for Machine Translation.
[35] Kelly Marchisio, Wei- Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. Under- standing and mitigating language confusion in LLMss In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6653-6677 Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[36] Benjamin Marie, Raphael Rubino, and Atsushi Fujita. Tagged back-translation revisited: Why does it really work? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 9999, Online, July 2020. Association for Computational Linguistics.
[37] Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 234-239, 2012.
[38] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[39] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large- multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248-260. PMLR, 2022.
[40] Sahana Ramnath, Melvin Johnson, Abhirut Gupta, and Aravindan Raghuveer. HintedBT: Augmenting Back-Translation with quality and transliteration hints. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 1717-1733, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[41] Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain- of-thought reasoning. Yvette Graham and Matthew Purver, editors, Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1812-1827, St. Julian's, s, Malta, March 2024. Association for Computational Linguisties.
13

---

[42] Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. Translationese as language in "multilingual" NMT. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7737-7746, Online, July 2020. Association for Computational Linguisties.
[43] Rico Sennrich and Barry Haddow. Linguistic input features improve neural machine translation. In Ondiej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Verspoor, Jörg Tiedemann, and Marco Turchi, editors, Proceedings of the First Conference on Machine Translation: Volume I, Research Papers, pages 83-91, Berlin, Germany, August 2016. Association for Computational Linguistics.
[44] Rico Sennrich, Barry Haddow, and Alexandra Birch. Controlling politeness in neural machine translation via side constraints. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 35-40, San Diego, California, June 2016. Association for Computational Linguistics.
[45] Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolò Fusi. Tag-Ilm: repurposing seneral-purposs Ilms for specialized domains. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 24. JMLR.org, 2024.
[46] Chufan Shi, Haoran Yang. Deng Cai, Zhisong Zhang. Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Langugge Processing, pages 8601-8629, Miami, Florida, USA, November
2024. Association for Computational Linguistics.
[47] Raphael Shu, Hideki Nakayama, and Kyunghyun Cho. Generating diverse translations with sentence codes. In Anna Korhonen, David Traum, and Lluís Märquez, editors, Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1823-1827, Florence, Italy, July 2019, Association for Computational Linguistics.
[48] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619, 2024.
[49] Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time com- pute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025.
[50] Emmanouil Stergiadis, Satendra Kumar, Fedor Kovalev, and Pavel Levin. Multi-domain adaptation in neural machine translation through multidimensional tagging. In Janice Campbell, Ben Huyck, Stephen Larocca, Jay Marciano, Konstantin Savenkov, and Alex Yanishevsky. editors, Proceedings of Muchine Translation Summit XVIII: Users and Providers Truck, pages 396-420, Virtual, August 2021. Association for Machine Translation in the Americas.
[51] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001188 2024.
[52] Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot Maiin-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( pages 2609-2634, Toronto, Canada, July 2023, Association for Computational Linguistics.
(Volume 1: Long Papers),
[53] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
14

---

[54] Han Wenjuan, Wei Xiang, Cui Xingyu, Cheng Ning, Jiang Guangyuan, Qian Weinan, and Zhang Chi. Prompt engineering 101 prompt engineering guidelines from linguistic perspective. In Maosong Sun, Jiye Liang, Xianpei Han, Zhiyuan Liu, and Yulan He, editors, Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume I: Main Conference), pages 1408-1488 1426, Taiyuan, China, July 2024, Chinese Information Processing Society of China.
[55] Genta Winata, Shijie Wu, Mayank Kulkarni. Thamar Solorio, and Daniel Preotiuc-Pietro. Cross- lingual few-shot learning on unseen languages. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang, editors. Proceedings of the 2nd Conference of the Asia- Asia-Pacific Chapter of the Association for Computational Linguisties and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 777-791, Online only, November 2022. Association for Computational Linguistics.
[56] Liwei Wu, Shanbo Cheng, Mingxuan Wang, and Lei Li. Language tags matter for zero-shot neural machine translation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3001-3007, Online, August 2021. Association for Computational Linguistics.
[57] Yangjian Wu and Gang Hu. Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 166-169, Singapore, December 2023, Association for Computational Lin- guistics.
[58] Hayahide Yamagishi, Shin Kanouchi, Takayuki Sato, and Mamoru Komachi. Controlling the voice of sentence in Japanese-to-Englist neural machine translation. In Toshiaki Nakazawa, Hideya Mino, Chenchen Ding, Isao Goto, Graham Neubig, Sadao Kurohashi, Ir. Hammam Riza, and Pushpak Bhattacharyya, editors, Proceedings of the 3rd Workshop on Asian Translation (WAT2016), pages 203-210, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.
[59]
Kexin Yang, Dayiheng Liu, Wenqiang Lei, Baosong Yang, Mingfeng Xue, Boxing Chen, and Jun Xie. Tailor: A poft-prompt-basee approach to basbute-bbsed controlled text generation. In Anna Rogers, Jordan Boyd-Gree Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 410-427, Toronto, Canada, July 2023. Association for Computational Linguistics.
[60] Fangyi Yu, Lee Quartey, and Frank Schilder. Exploring the effectiveness of prompt engincering for legal reasoning tasks. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 33582-13596 Toronto, Canada, July 2023. Association for Computational Linguistics.
[61] Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. Following length constraints in instructions. arXiv preprint 24ti::2000.177744,, 2024.
[62] Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-based pre-trained language models. ACM Comput. Surv., 56(3), October 2023.
[63] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint 10X::1779900003 2017.
15

---

A Taxonomy for training time markers
A.1 Categories for Training Markers
Leenghh_settnneee and <length_ Baapnngthparagaaphhhss models granu- lar control in generation length. We tokenize using language-specific Spacy models to obtain token and sentence counts. For paragraph counts, we use the "\n'n" 'n" delimiter. Lennth_buccket categorizes generations into broader categories such as concise (under 300 tokens), medium (between 300 and 1,000 tokens) or long (over 1,000 tokens), providing more general level of control when needed.
Length: <length_tokenas
Format: <format> is used to describe generations with specific output structures such as: JSON, Markdown, or tabular formats. This is particularly useful to condition stricter format requirements needed for real world use cases.
Style: <style> captures tone and manner of communication, distinguishing between different modes of expression such as "Formal" and "Informal". We also add a "Custom" value to model for training examples where the user specifies a particular format. For instance, inference if user asks, "Respond like Yoda you will" this marker will allow the model to adapt its response to match the requested style. We annotate this marker by using dataset-related information.
Language: <language> describes the natural language of the generation, enabling us to model responses in specific languages during inference. We provide detailed markers across the 23 languages covered by our model. Our goal with this tag is to improve language-specific generations and reduce language switching where prompt specified by user in one language is not responded to in the same language in the completion. <code_ coding-related tasks. We annotate this marker by using dataset-related information.
is specifically used to model programming languages for
Quality: <quality> provides measurable score indicating the quality of a sample, often derived from human annotations or a Reward Model (RM). We utilize a proprietary reward mode to assign rewards to subset of our training data. We also these rewards to create categorical marker Equality_o ceriitt_bbbuu bucket> by using quartiles within language-specific subsets into 11,2,3, 4), offering a broader description of quality.
<source> describes the origin of the data, distinguishing between human- suman-genneatted content and other methods of data creation like synthetic and translation. We annotate this marker by using dataset-related information.
Source:
Domain: <domain> ensures that Nomain-specitfe knowledge is captured from training subsets, which can then be leveraged at inference generate content that is relevant and accurate within a particular field. This is particularly crucial for inputs that could belong to multiple fields. For instance, when user asks, "How do I calculate factorial?", specifying the <domain> as either Code Math provides valuable context for modeling the interaction. In cases where this marker cannot be obtained from the dataset information, we employ L.M annotate and provide detailed prompt in D
Task: <task> defines the overall objective of the generation and helps capture task-specific behaviors, especially when outputs involve complex combinations of formats or actions. This marker is useful to model dataset-wise characteristics. We hypothesize this particularly helpful for indicating complex workflows during inference. For example, distinguishing between Translation and CodeTranslation. or Rewrite and CodeFix, enhances the descriptiveness of datapoints within the same training pool. In cases where this marker cannot be obtained from the dataset information, we employ an LLM to annotate and provide our detailed prompt in
B Additional details about experimental sections and key ablations.
Languages covered by Markers We cover 23 languages: Arabic, Chinese (simplified traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish Ukrainian and Vietnamese.
Training protocol Training for each variant spanned 8,000 steps, employed a cosine learning rate schedule with warm-up phase, using batch size of 32 and evaluation batch size of 64. We train
'The RM is competitive with leading reward models on the RewardBench Leaderboard 24 https://
huggingface co
16

---

category
definition
values
<quality>
Score indicating the quality assigned to sample as an- notated by a human Re- wardModel(RMM).
float
Squulltt_buckeeee bucket>
<quality> bucketed into (1,2,3,4) quartiles (calculated by lan- guage).
indicating highest quality
and 4 indicating lowest qual- ity
Length_tokense of tokens
inl
Length_sentencest of sentences
int inf concise, medium, long
paragraphs
<length_bucke bucket>
<length_tokens>> buck- eted by defined response length ranges Task-related information
<task>
(OpenEnded, Explanation, Translation, Classification, CreativeWriiting QuestionAnswerring,
InformatiionEttaaciion,
Summarization, Rewrite,
Reasoning, CodeGeneration, CodeFix, CodeTranslation, CodeExplanation) {Sciences, Technology, SocialSciences, Culture, Medical, Legal, Unspecified, Conversation, Code, Math)
<domain>
Domain-related information
<code_type>
(coding lasks) Programming languages
{python, javascript, cpp, cobol, java, go, rust, swift, csharp, php, typescript, shell, c, kotlin, ruby, haskell, sql} (MCQAnswer, ChainOfThought, XML, JSON, Enumeration, Tabular, Markdown, Latex)
<format>
To model desired generation format
<source>
To model the data- generation/aanotation source
(Human, Translation, Synthetic, Others}
<style>
To model tone and style of the generation
{Formal, Informal, Custom}
<lang>
To model the 23 languages present in the training data
(English, French, Spanish, Italian, German, Portuguese, Japanese, Korean, Arabic, Chinese, Russian, Polish, Turkish, Vietnamese, Dutch, Czech, Indonesian, Ukrainian, Romanian, Greek, Hindi, Hebrew, Persian]
Table 8: Comprehensive taxonomy for training time markers: Our taxonomy contains 13 cate- gories shown with their descriptions and values.
17

---

for 2 epochs with peak learning rate of at 2.5 X 10-2 achieved through 10 warm-u- up steps starting from learning rate of 0.0, and then decay back 1.25 10-1 One fine-tuning run using 8,000 steps on 128 Nvidia H100 GPUs takes around 6 hours.
Length Evaluations Given the original instruction in the AlpacaEval-LI dataset [61] contains the exact constraint, our Marked and TreasureMarkedd (fixed) both contain explicit reference to the contraint. For Marked we present the original Length-instructer prompt, allowing the model to deduce the associated tags. This approach evaluates the model's 's ability to extrapolate tags from instructions. in contrast, for TreasureMarked (fixed), since the original instruction contains the exact constraint, we investigate an additional control strategy where we provide the constraint in the marker template if the taxonomy directly supports it. We remove the length instruction and append the corresponding <length_ tokens> tag with the appropriate value. Table providess an example of edited prompt. This strategy assesses the model's adherence known templates and its ability to follow explicit length requirements that are only provided via the marker template.
<
Language Control We insert training markers present in the data into the prompt, but leave out the <lang> marker, since it is already present in the prompt.
C Extended Related Work
From one- to multi-dimensional training data tagging The idea of tagging inputs in neural sequence prediction goes back to early applications in machine translation and language modeling. The motivation there was to leverage discrete features to overcome data sparsity or imbalance and introduce levers of control. In early neural LMs,, often tokens were added to target a very specific attribute such as topic (77 or auxiliary features 3 including genre and length. A specific token for example
added at inference time to control feature in translation like the target
language [17), or desired output quality [6] 42 36, [3,] [3] and text complexity [ 34 but also language-specific nuanced attributes like politeness [44] the voice (58), gender 23 domains [19
4) or diversity (47] of translations. Other works enriched the input representation during training with discrete linguistic features 43 or document information (66 for better contextualizatio
at inference time. Where and how tags should be placed best differed across applications [16] (56..
Some tags were combined into multidimensional tagging (50, 40) All of these were individual efforts that target one or two dimensions at a time, highly specialized for one trained target model and with training data for one particular task. contrast, our focus is on a much more general framework with a vast training corpus that targets general performance. Our approach is similarly general, where instead of a single feature, we want to enable a flexible approach that can be used for any text generation task. Furthermore, our goal is to explicitly target improving performance on the long-tail of underrepresented features.
From control in pretraining to control in instruction finetuning In LLM research, there are several related works that experiment with adding prefixes for control in pretraining: [18] add control codes for desired text features in pretraining of a LLM derived from the structure of their source, i.e., subdomains or links of online texts and specific task labels for translation and QA. At inference time, values for these control codes are specific to steer the generation. [14 further propose cooldown schedule in pretraining going from tagged data to untagged data in order to not require prefixes at inference. [61] focus on length control by adding natural language length specification templates to fine-tuning data for DPO. In our work we focus on the instruction finetuning stage and incorporate nuanced multi-diimensonal tags (i.e. the user can specify length and domain and format). We circumvent a cooldown schedule by simply introducing tag dropout, hence requiring much smaller volume of tagged data at training time, and not a complete population of tags at inference time.
From encododed to inferred meta-informmatio Related prefix and prompt tuning methods [8 26 use continuous embeddings leareed for special tokens representing tags in training to condition predictions for specific tasks at inference time. (45) further break those into separate tags for domain and function tags. In our case, we directly embed prefixes with the same vocabulary as the LLM, smoothly integrating them into the sequence. In our experiments, we find that this helps format following even when specified in natural language and not tags.
18

---

Attribute-base control in LLM generations has also been pursued with other methods, such as attribute classifiers
or learned attribute vectors 59
see 62 for comprehensive survey.
D LLM Annotation
For the following training markers: : <domain>, <task>, <format> we annotate using the multi- lingual open-weights model Command R+.
We provide definitions and multilingual few-shot examples (except for <format>) obtain high- quality annotations from the LLM. The prompt used for tagging is as follows:
D.0.1 <domain>
You are helpful assistant whose goal is to classify the given prompt into + a single class given the following definitions
"Sciences" : Topics related to the broad area of knowledge encompassing all scientific disciplines, including biology, chemistry, physics, earth sciences, and astronomy, which study the natural world through observation, experimentation, and analysis, aiming to understand fundamental principles and phenomena across various scales and aspects of the universe
Technology" Topics related to the broad area of knowledge encompassing
all engineering and technical disciplines, including Computer Science, Software Engineering, Internet of (min)s(IoT), Cybersecurity, Data Science, Artificial Intelligence, Machine Learning and various engineering disciplines like Mechanical Engineering, Civil Engineering and Biotechnology
"SocialSciences" :
Topics related to the broad area of knowledge encompassing all academic disciplines dedicated to the systematic study of human society, social relationships, and the structures that shape them, including fields like anthropology, economics, political science, psychology, and sociology, all focused on understanding how individuals and groups interact within a society and the factors influencing their behavior, cultural norms, and societal institutions
Culture" Topics related to the broad area of knowledge encompassing all cultural practices or beliefs within societies, including related concepts or behaviors that people within a culture group share and understand belonging together, like food, art, language, family structure, societal norms or religious rituals
MMddical' Topics related to the broad area of knowledge and practice encompassing all medicine and healthcare, including diagnosing and treating diseases, preventative measures, specialties like surgery, cardiology, oncology, pediatrics, and more, all built upon the principles
foundation of basic medical sciences and patient Finance Topics related to the broad area of knowledge encompassing
activities like managing money, business ethics, investing, borrowing, lending, trading, budgeting, saving, and forecasting, essentially focusing on the acquisition, allocation, and management capital within businesses
individuals, and governments across various financial markets and
instruments
"Legal' Topics related to the broad area of knowledge encompassing
Private, Public and Criminal Law, Criminal Justice, Law Enforcement, Policing, Justice Systems or Crime
"Covversation' Topics related to Conversation, Chit-Chat or Roleplay
19

---

"Code"
Topics related to a specific subject/field within computer programming where software designed and developed solve problems
related to particular industry, business function, area of expertise, essentially defining the target audience and unique requirements for the code being written including tasks like Code Generation, Code Fix and Code Explanation
"Math' Topics related to the broad field of study that uses numbers, shapes, and formulas to describe and quantify the world, including areas like Logical Reasoning, Quantitative Calculation, Pattern Recognition, Formulating Conjectures, Arithmetic, Algebra, Geometry, Number Theory, Set Theory and Analysis
If you unable confidently assign of the above classes, you will s simply respond with 'Unspecified" and nothing else.
Note:
You are only to respond with the name of the class you believe best matches the domain of the example.
You are only allowed to classify the example into one of the following
tags
[" I'Siiences', 'Technology', 'SocialSciences', 'Culture', 'Medical', "Finance", "Legal", "Conversation', 'Code", "Math", "Unspecified']
Here are examples
Prompt : What is photosynthesis? Answer "Sciences"
Prompt What is the TCP/IP protocol and how does it work ?
Answer Technology"
Prompt How has globalization affected social cohesion ?
Answer Social Sciences"
Prompt :
What is an example of popular dish that is available in multiple communities but known under different names?
Answer "Culture"
Prompt : How long does one have to fast before fasting sugar blood test?
Answer : "Medical"
Prompt Analyze the impact of microfinance initiatives on poverty alleviation in developing countries.
Answer "Finance"
Prompt What is the difference between first-degree crime and + second-degree crime?
Answer 'Legal'
Prompt Hey! How you?
Answer "Conversation"
Prompt Given variable x43.142 in Python, how would I use an f-string show just 1 decimal value?
Answer "Code"
Prompt Solve the quadratic equation: x² 6
20

---

Answer
MMth''
Prompt
Use ABC notation to write melody in the style of folk tune.
Answer
D.0.2
<task>
You are a helpful assistant whose goal is to classify the given prompt into a single class given the following definitions
CodeTranslation Tasks related
process of converting source code from one programming language to another while preserving the code's functionality
'odeExpllanation" Tasks related to the specific process of explaining a snippet code in a programming language 'Coeeeeneraionn Tasks related to the specific process of generating snippet of code in a programming language
'Explanation" Tasks related to explaining concept in any domain
"Ceattiekrriting' Tasks related to any form of writing that employs creative, literary poetic techniques that displays imagination invention including role-play
QuestionAnswering" Tasks related to any form of question answering, including open-ended questions, closed-ended questions about a given context and requests for information about particular entity. This
will also generally include your "what', 'which', 'who', 'when' type questions
OpenEnded Tasks related to any form of open-ended text generation like chat, conversation or chit-chat
InformationExtracionn Tasks related to any form of information extraction usually involving some context
'uumaaizzaionn Tasks related to any form of summarization including but not limited to abstractive summarization, extractive summarization or concise descriptions of content
CodeFix Tasks related to the specific process correcting/fixing a piece of code to achieve the desired result.
'Reasoning" Tasks involving any form of Ideation, Reasoning, Problem Solving, Instruction Following or Chain-of Mhain-ofTThought Thought (CoT) in order to achieve the desired result.
"Rewrite" Tasks involving any form of heattiiggee-phassn/re---oooddodd desired result.
fremifrmmng in order to achieve the
Classification :
Tasks related to specific request of classification where you required to assign thing to one of several groups Translation" : Tasks related to specific request of translating a given
piece of text from one language to another language If you are unable to confidently assign one of the above classes, will
simply reapond with 'Unspecified" and nothing else.
Note:
You are only respond with the name the class you believe best matches the domain of the example.
You are only allowed to classify the example into one of the following
tags
[ "'oddrrrnsllaiion 'CodeExplanation, 'Coeeeeneratonn, 'Explanation', CreativeWriting: 'uestionAnsweringg, 'OponEnded", "IfforaationExttaaciioo"" "Summarization' "CodeFix', 'Reasoning', RRerrite, "lassification", 'Translation', 'Unspecified")
Here are few examples
21

---

Answer "Cassificatton" Prompt : Translate the following Python function to equivalent JavaScript
code that checks a string is a palindrome. def ss_allindrmmestrr:
return str str[::-1] 'odeTransllaionn
Answer
Prompt Explain the following python function def is.palindrome(str. return str == str[::-1] Answer 'Codexxplanation"
Prompt
Generate a Python function to check whether a string is a palindrome.
Answer 'oodCenerraioon
Prompt
Explain briefly how the water cycle works Answer Explanation
Prompt : Translate the following sentence from English to Spanish, using formal tone: 'We are pleased to announce the new partnership with our company.
Answer "rrasllaioon
Prompt You're talk show host. Pick two guests that are wildly different " from each other. Briefly introduce them
Answer CreativeWiriting
Prompt Classify the sentiment of the following review positive, negative, or neutral: 'The product exceeded my expectations!'
Prompt What is the capital city of France?
Answer "QusstionAnsvering"
Prompt Describe your ideal work environment
Answer "OpenEnded"
Prompt
From the following news article, extract the names of the companies involved in the recent merger, along with the date the merger was announced.
Context: In a significant development in the tech industry, two leading companies have announced their merger, marking new era of innovation and collaboration. The merger, which was officially announced on March 15, 2025, brings together TechInnovate Inc. and DigitalSolutions Corp, two giants in their respective fields. TechInovvtee Inc, known for its cutting-edge research and development in artificial intelligence and machine learning, has been at the forefront of technological.
advancements. With a team of over 5,000 engineers and scientists, the company has consistently delivered groundbreaking solutions that have transformed various industries. DigitalSolutions Corp, on the other hand, is renowned for its expertise in software development and digital transformation. The company has a proven track record of helping businesses across the globe to modernize their
22

---

operations and enhance their digital capabilities. With workforce of over 10,000 professionals, DigitalSolutions Corp. has been key player in driving digital innovation. The merger is expected to create both
4 a powerhouse
tech industry, combining companies to offer comprehensive solutions that address the evolving needs of businesses and consumers. The combined entity will leverage TechInnovate's AI and machine learning capabilities with
strengths
+ DigitalSolutions' software development expertise to develop next gext-aemmration technologies. Industry analysts predict that this merger will lead to significant advancements in areas such as autonomous systems, smart cities, and personalized healthcare. The synergy between the two companies is anticipated to drive innovation, improve efficiency, and create opportunities for growth.
Answer
InformationExtraction
Prompt : Given the following story, provide title that summarizes the idea behind the story:
Context:
There once was girl who was frustrated with life and asked her father advice. He asked her to bring an egg, two tea leaves, and a potato. He then started boiling water three separate vessels. He put the egg, potato, and tea leaves in one vessel each. After a few minutes, he asked her to peel the egg and potato and strain the leaves. explained to his daughter that:
The soft egg was now hard.
The hard potato was The tea had changed the water itself.
soft.
When adversity is at door, can respond to it in different ways.
Moral: We decide how respond to difficult situations.
Answer : 'Sumarizzaioo'
Prompt Fix the code below to correctly identify a palindrome: def ia.palindromede return str == str[-1]
Answer "CodeFix"
Prompt John has one pizza, cut into eight equal slices. John eats three slices, and his friend eats two slices. How many slices are left? Explain your reasoning step by step.
Answer "Reasoning'
Prompt Exaggerate this product description 'Our new sneakers are comfortable, lightweight, and stylish.' paragraph that can be used by the marketing team Answer "Rerrite"
Prompt : Use ABC notation to write melody the style folk tune. Answer
D.0.3 <format>
You are a helpful assistant whose goal is to classify the given prompt into a single class given the following definitions
23

---

"CCAAnswer" Tasks related to multiple-choic type question answering.
These prompts will typically contain multiple choices provided either in bullet form or eumerated mumericalll//allaaaeticllll contains miltiple-choie question answer generation tasks.
This also
ChainOfThought" : Tasks related to Chain-of-Thoughh (CoT) style question answering. This also contains CoT style question answer generation tasks
Enumeration" : Tasks that involve enumeration, bullet points, lists or itemization of any form XML Tasks that involve XML generation, validation or processing in any form
"Tabular' Tasks that involve table generation, validation or processing in any form "JSON" Tasks that involve JSON generation, validation or processing in any form
"Markdown' : Tasks that involve Markdown generation, validation or processing any form
If you are unable confidently assign one the above classes, you will simply respond with 'Unspecified" and nothing else.
Note:
You are only to respond with the name of the class you believe best matches the domain of the example.
You are only allowed to classify the example into one of the following tags
[' ['MCOAnswer', "ChainOfThought', 'Enumeration', "ML",, 'Tabular', "JSON" "Mrrddom",, "Unspecified']
Prompt : Use ABC notation to write melody in the style of a folk tune. Answer
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). ) In general, answering binary way, we
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate, All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Framework for controllability: "We create detailed taxonomy of data characteristics and task provenance to explicitly control generation attributes and implicitly condition generations at inference time.": The taxonomy is provided in section 2.2 with the definition of training objective section
Experimental scope: "We fine-tune base model to infer these markers automatically, which makes them optional at inference time.": The method is described in section 2 the process of finetuning in section 2.3 and the empirical evidence for not requiring the markers training time is provided in section
Long tail lift: "This principled approach yields pronounced improvements in perfor- mance on examples from the long tail of the training distribution. ": Experimental
evidence is reported in section 3.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
25

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No thcoretical result.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
26

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Finetuning settings reported in section 2.3 base and LLM judge mod- els are also specified including version. Evaluation paradigms are explained with each experiment section 2.3 and follow standard metries of the respective benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Code, model and data are not released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
httpliii/guiiee//eeeetttiee sor.eeeeuubmisiiooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
27

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ht.ps..cc/ubblcc/uuideeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: section 2.3 specify hyperparameters, with additional details about the evalua-
tions in appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We only report significance tests for the machine translation experiments, in
section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
28

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We report the required compute and estimated time for each training run in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp///earrrppp..cppuuttt/////
Answer: [Yes]
Justification: There is violation with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impact is presented in the introduction in section potential negative impacts discussed with the limitations in section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
29

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No release of models or data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets are credited in section 2
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
30

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing research with human subjects conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such studies conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLMSs for annotating the training markers used in our framework ($ (222) and during evaluation ($ 2.3.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
31

---

Please refer to our LLM policy http:s//heerriisccc///ofrrrrrrsr for what should or should be described.
32


Reviews:
[{"review_id": "DrlyW8HBwx", "reviewer": "Reviewer_Hdba", "summary": "The paper focuses on improving the controllability and performance on underrepresented use cases at inference time. Specifically, they construct a taxonomy of training markers and finetune the model. In inference time, they allow users to control the model with inference-time markers.\n\nEmpirically, the authors show significant performance gain on open-ended generation (~5.7 % average win-rate uplift). They also conduct experiments on specifica sub-tasks, including code generation, machine translation, etc.", "questions": "None", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strength]\n- the taxonomy of labels (e.g., data provenance, feature rarity, task type) could potentially be useful for future works\n- the empirical gain is significant (~5.7 % average win-rate uplift in open-ended generation when using markers)\n- the results show that the gain is more significant on underrepresented domains (Over 9.1 % lift)\n\n[Weakness]\n- The techinical contribution of the work is quite limited. It seems that most of the contribution is on the taxonomy construction. No novel training recipe or data construction methods are presented.\n- The effectiveness of the method highly relies on the quality of the taxonomy. However, there are not much description about the construction process of the taxonomy in the main context. Like, how did you decide each node of the taxonomy? how did you adjust the taxonomy? what's the significance of all the 90 unique markers?", "was_revised": false, "final_justification": "", "rebuttal": "We thank **RHdba** for finding our “*empirical gains significant*” especially on “*underrepresented domains*”.  **RHdba** recognized the breadth of our experiments, including  open-ended generations (**~5.7 % average win-rate uplift**),  code generation and machine translation. The reviewer notes the large gains in targeting long tail performance  **(Over 9.1 % lift).**\n\n>**The techinical contribution of the work is quite limited. It seems that most of the contribution is on the taxonomy construction. No novel training recipe or data construction methods are presented.**\n\nWe thank **RHdba**  for the opportunity to clarify the contribution of our work. While the taxonomy is an important aspect of the proposed framework, the technical contribution of our work extends beyond the taxonomy. We show that extremely cheap interventions of adding treasure markers results in a model learning to infer long tail characteristics. Despite requiring only minimal changes to the standard training workflow, these targeted interventions prove highly effective. The novelty is that the resulting framework is flexible, enabling inference gains in underrepresented domains and control of targeted attributes—improvements that would otherwise demand more costly alternatives such as data rebalancing during training, prompt engineering, or manual optimization of data compositions. \n\n\n>**The effectiveness of the method highly relies on the quality of the taxonomy. However, there are not much description about the construction process of the taxonomy in the main context. Like, how did you decide each node of the taxonomy? how did you adjust the taxonomy? what's the significance of all the 90 unique markers?**\n\nWe thank **RHdba** for the opportunity to elaborate on the taxonomy. Our goal was to study a wide variety of markers encompassing both deterministic properties (*like length, language*) and LLM-annotated (*like domain, task*) to characterize the SFT data mix. Furthermore, we wanted to test gains across a wide array of well represented and low frequency features, and hence chose a taxonomy that would characterize this skew (type of coding task, domain which are heavily skewed markers as described in Sec 3.2.1 and 3.1 respectively).  \nWe would like to clarify that there are 13 main marker categories, each of which includes subcategories within the taxonomy, resulting in a total of 90 possible values. Additional details are available in the Appendix A, and we’re happy to surface more of that in the main text if the reviewer finds it helpful.\n\n\nWe thank **RHdba** again for their engagement with our work, and we welcome continued engagement during the rebuttal period."}, {"review_id": "hKmqK6yafQ", "reviewer": "Reviewer_AB9d", "summary": "The paper proposes a new approach to train LLM to allow better control at inference time as well as to perform better on underrepresented use cases (long tail problem). The approach uses metadata tags such as length, task domain, language, quality and inserts these tags into the input and the output of training data. The model is then trained to both use and infer these markers, with dropouts applied on the input metadata tags to prevent over reliance. The approach is evaluated on several benchmarks like Arena-Hard, HumanEvalPack, AlpacaEval-LI, and WMT’24++ and demonstrates improved overall performance as well as enhanced performance on the long-tail tasks (e.g., CodeRepair).", "questions": "- How accurate are the LLM-inferred markers during training? Is there scope for improvement in the quality of the markers? \n\n- What happens if the user enters a wrong or unknown marker at inference time? Does the model still recover? \n\n - Please add absolute values not just the improvements for Figure 5 (right).", "limitations": "Yes, limitations are discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Prompt tuning is becoming harder and harder for users and this paper’s proposed approach presents a more intuitive way for users to control the generation of LLMs. \n\n2. Compared to other inference time techniques to improve performance (e.g. few shot examples), this paper takes a different approach of augmenting the training process itself.  \n\n3. Training time markers has been done before (for e.g. AlphaCode), but this is first time it is done for a general-purpose dataset and on a more comprehensive taxonomy of about 90 tags.  \n\n4. Also, the markers can be inferred at inference time, so that means there is no extra burden on the user in normal cases, but also the user can intervene and inject tags if they want more control.  \n\n5. The approach shows performance gains on overall tasks as well as on the long tail of underrepresented tasks \n\n\n**Weakness**:\nNo major weaknesses but would love to learn more about how the approach be extended with additional markers in the future.  Would the approach scale to thousands of markers especially with domain specific markers for the different domains (for e.g. algorithm type for programming tasks, and clinical subdomain for medical)?", "was_revised": true, "final_justification": "Still support accepting the paper.", "rebuttal": "We thank **RAB9d** for describing our approach as “*intuitive*” and for recognizing its ability to “*reduce user burden*” while offering control through a \"*comprehensive taxonomy*\" and training-time augmentation leading to performance gains even on the long tail.\n\n>**How accurate are the LLM-inferred markers during training? Is there scope for improvement in the quality of the markers?**\n\nWe appreciate **RAB9d**’s attention to this aspect of our work.  To assess the reliability of the Command R+ model used for labeling, we ran tagging accuracy checks on datasets that included annotations—for instance, we normalized the “subset” attribute in the MMLU dataset to the marker values in the taxonomy and observed an average tagging accuracy of over 64.9%. We note that even without fully accurate annotation, our approach yields consistent performance gains on downstream tasks (**5.7% absolute improvements** on ArenaHard, **35.3% violation reduction** on AlpacaEval-Length-Instructed evaluations and **14.1% relative improvements** on underrepresented coding tasks like CodeRepair). This is actually a strength of the method as it means we can cheaply rely on LLM annotations while still ensuring downstream performance lifts.\n\n>**What happens if the user enters a wrong or unknown marker at inference time? Does the model still recover?**\n\nWe thank **RAB9d** for the insightful question. One of the strengths of our framework is that providing markers at inference time is entirely optional, which significantly reduces the burden on the user and minimizes the risk of inputting incorrect or unknown markers. Table 7 demonstrates that even though the model(*50_50*) predicted markers aren’t perfect (*68.8% over domain, task, format and language*), the model still recovers and leads to consistent performance gains on downstream tasks. The regularization effect introduced by 50% dropout of markers in the input space during training makes it robust to the lack of markers during inference. In cases where unknown or completely irrelevant markers are used at inference, we find that the model tends to ignore them gracefully and still predicts the appropriate marker.\n\n>**Please add absolute values not just the improvements for Figure 5 (right).**\n\nWe are very happy to include the absolute values for Figure 5 in the final version of the manuscript, and commit to including it in the camera ready. \n\n\n\nWe thank **RAB9d** again for their engagement with our work, and we welcome continued engagement during the rebuttal period."}, {"review_id": "cy1rmi9UjZ", "reviewer": "Reviewer_xUkW", "summary": "This paper proposes a training protocol that embeds a rich set of textual markers directly into the instruction-tuning data, which describe length, quality, domain, task, language and more. During finetuning, the authors randomly drop these markers from inputs so the model learns to infer them; at inference, users may supply markers to steer generation for better quality.\n\nExperiments on 2.7 M instructions show that merely inferring markers already lifts win-rate against a strong baseline by 5.7% overall and by 9.1% on under-represented domains. Explicitly fixing markers yields task-specific boosts: CodeRepair accuracy rises 14.1% despite it is rare in training data; length-constrained generation violations decrease from 36.6% to 1.25%. These gains demonstrate that training-time markers simultaneously enhance long-tail performance and expose reliable, fine-grained controllability without heavy prompt engineering.", "questions": "Why were direct comparisons against alternative controllability techniques omitted, can you include more experiments to clarify relative efficacy?", "limitations": "The paper currently doesn't include “Limitations”. As highlighted in the Questions, I recommend that the authors include additional comparative experiments against a broader set of prior methods.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Merely asking the model to infer markers (no extra prompt tokens) raises overall win-rate against a strong baseline by +5.7%, and by +9.1% on under-represented domains, confirming that the approach attacks the long-tail performance gap it sets out to solve.\n2. Because the markers are short text tokens already in the vocabulary, steering the model requires no extra forward passes, contrastive decoding, or RL at inference time—making the method deployment-friendly.\n\nWeaknesses:\n1. The core idea—injecting markers into training data to steer generation—closely resembles prior work on prefix-tuning, control codes and prompt-tuning. The paper does not clearly present how “training-time markers” differ conceptually or practically from these well-established methods. In addition, while the paper shows gains over a baseline instruction-tuned model, it does not compare directly to other control mechanisms. Without such comparisons, it’s hard to assess relative efficacy.\n2. Quality markers are derived from a pretrained reward model’s scores. Any bias or inaccuracy in that reward model will propagate into both training and inference, potentially reinforcing undesirable behaviors rather than correcting them.", "was_revised": true, "final_justification": "The rebuttal clarifies how the approach differs from related work, alleviating my earlier concerns about limited novelty. These clarifications improve my assessment of originality, and I will raise my score to 4.", "rebuttal": "We thank **RxUkW** for highlighting that our method addresses the \"*long-tail performance gap.*\" We also appreciate that **RxUkW** recognizes that this intervention is extremely efficient and \"*deployment-friendly*\", such that  “*steering the model requires no extra forward passes, contrastive decoding, or RL at inference time.*” \n\n>**The core idea—injecting markers into training data to steer generation—closely resembles prior work on prefix-tuning, control codes and prompt-tuning. The paper does not clearly present how “training-time markers” differ conceptually or practically from these well-established methods. In addition, while the paper shows gains over a baseline instruction-tuned model, it does not compare directly to other control mechanisms. Without such comparisons, it’s hard to assess relative efficacy.**\n\nWe thank **RxUkW** for highlighting the need for clarification. While our approach may appear similar to prior work on the surface, it differs significantly in both motivation and practical use.\n\nPrompt- and prefix-tuning are parameter-efficient fine-tuning strategies designed to reduce the computational cost of supervised fine-tuning (SFT), particularly in task-specific settings. However, these methods do not support transparent inference-time control. They rely on learned soft vectors, are generally opaque, and lack generalization beyond specific task boundaries. Control codes are dataset-specific, non-extensible and require careful dataset curation to function effectively—limiting flexibility across domains and tasks.\n\nIn contrast, our framework introduces interpretable, natural language markers directly into the training data. This enables:\n1. **Inference-time control** using human-readable markers that correspond to semantic attributes (e.g., style, domain).\n2. **Flexible usage**—inference-time markers are optional. When omitted, the model can self-condition by predicting its own markers. This removes the burden from the end practitioner to grapple with complex prompt engineering, and makes more explicit what conditioning is inferred from the prompt.\n3. **Multi-task generalization**—our taxonomy-based setup supports learning across diverse datasets and tasks within a single model. As shown in Section 4, novel marker combinations at inference can yield additional improvements, even if not seen during training.\n\nWe respectfully clarify that we do compare to SFT which typically outperform prompt- and prefix-tuning methods in general-purpose settings. \n\n>**Quality markers are derived from a pretrained reward model’s scores. Any bias or inaccuracy in that reward model will propagate into both training and inference, potentially reinforcing undesirable behaviors rather than correcting them.**\n\nThank you for raising this concern. **RxUkW** is correct that using reward model (RM) scores as markers means any bias in the RM may propagate into training and inference. However, to show that this does not limit the utility of an estimate of generation quality, we separate the pretrained RM used to derive the quality markers from the models used to evaluate the quality lift. For evaluation, we used GPT-4o to evaluate the gains in Arena-Hard (Sec 3.1) and AlpacaEval-Length-Instructed(LI) (Sec 3.3) benchmarks. This is important, because GPT-4o does not present the same bias, and yet shows consistent gains in overall quality (**5.7% absolute improvements** on ArenaHard and **6.86% absolute improvements** on AlpacaEval-Length-Instructed). This suggests that the bias introduced by using a reward model to assess quality **is greatly outweighed** by the benefits of having a flexible and useful relative measure of quality to guide inference conditioning.\n\n>**The paper currently doesn't include “Limitations”. As highlighted in the Questions, I recommend that the authors include additional comparative experiments against a broader set of prior methods.**\n\nWe note that there is indeed a “Limitations” section discussed in Appendix D (Supplementary Material), where we highlight promising directions for future work, including scaling to larger models, expanding the taxonomy, and exploring integration into pre-training.\nWe also reference prior work in Section 5 (Related Work) and Appendix C (Extended Related Work). We moved these to the appendix because we were limited in space.\n\nWe thank **RxUkW** again for their engagement with our work, and we welcome continued engagement during the rebuttal period."}, {"review_id": "tbboQegNEo", "reviewer": "Reviewer_5ToD", "summary": "This paper proposes a new method called \"treasure markers\" to improve the controllability of LLMs at inference time and the performance on long-tail distribution tasks by adding detailed markers during training. The method aims to reduce the burden of prompt engineering for users and allow the model to automatically infer these markers. The authors report significant improvements on tasks such as open generation, code repair, and length instruction following.", "questions": "Can the authors elaborate on the fundamental methodological differences between your framework and the input labeling methods in previous work (e.g., [4], [6], [17], [61], etc.), in addition to using a more exhaustive set of labels?\n\nGiven the use of proprietary models and data in this paper, do the authors plan to validate the method on publicly available models (e.g., Llama, Gemma) and datasets to enhance the credibility of the conclusions?\n\nIn the quality control experiment (Figure 4), the evaluation and training labels are from the same source. Can the authors provide data using an independent third-party reward model or human evaluation to more objectively demonstrate the model's ability to control quality?\n\nWhat is the accuracy of the Command R+ model used to label the training data? Does the paper analyze the sensitivity of upstream labeling errors to the learning effect of downstream models? If the labeling quality is not high, will it mislead the training of the model?", "limitations": "No code, models or data are publicly available", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper explicitly states that the code, model, and data have not been released, and the reproducibility is weak.\n\nThe paper admits that \"we only report the significance test of the machine translation experiment in Section 3.4\". This means that, in addition to machine translation, other key results, such as open generation performance improvement, long-tail task improvement, and length control improvement, do not provide error bars or statistical significance tests. Only a small number of experimental results in Table 4 are significant; most of the results are not significant, and some are even negative.\n\nThe paper mainly compares the proposed method with an internal baseline model (a model trained on the same data but without adding tags). Although this internal comparison is clear, the paper fails to fully demonstrate the competitiveness of its method on all tasks relative to the current state-of-the-art (SOTA) external model or a more powerful baseline.\n\nThe concept of using labels or tags to improve model performance is not new and has been applied in early generation models. The paper claims that its contribution lies in a \"more general framework\" and \"super detailed tags\", but this is more like an extension and scaling of existing methods rather than a fundamental innovation. Most importantly, I didn’t see any significant improvement, and the experiment was very unsolid.", "was_revised": true, "final_justification": "I tend to keep my score, unless the author directly addresses my concerns.\n\nUpdate: \n\nI maintain my original assessment that this manuscript is more suitable for a blog post than a formal academic paper. My reasoning is as follows:\n\n1. Recent studies have revealed that many methods, despite claiming to be model-agnostic, are only effective on specific pre-trained model families, such as the Qwen series, owing to some reasons. Similarly,  the authors' claim about their method's model-agnostic nature is also not substantiated. Furthermore, during the rebuttal period, which spanned over two weeks from July 24 to August 8, the authors had ample time to conduct at least one comparative experiment to address this concern, yet they failed to provide any such results.\n\n2. The proposed method was only compared against self-constructed baselines, rather than against existing SOTA methods in the field. Consequently, its true performance under existing SOTA methods remains unknown.\n\n3. The authors have not provided any datasets or code, which makes it nearly impossible for other researchers to reproduce and verify the paper's findings.\n\nIn summary, due to these significant deficiencies in generalization verification, baseline comparison, and reproducibility, I maintain my original score.", "rebuttal": "We thank **R5ToD** for noting that our work reports “*significant improvements on tasks such as open generation, code repair, and length instruction following.*” We welcome the reviewer's view, and think many of their points can be addressed below.\n\n>**Can the authors elaborate on the fundamental methodological differences between your framework and the input labeling methods in previous work (e.g., [4], [6], [17], [61], etc.), in addition to using a more exhaustive set of labels?**\n\nWe thank **R5ToD** for the opportunity to clarify the methodological distinctions between our framework and prior input labeling methods. While some of the cited works use input tags, our approach has significant differences in both scope and flexibility. Rather than designing task- or dataset-specific tags  **[4] [6] [17]**, we introduce a general-purpose taxonomy of interpretable, natural language markers, learned during supervised fine-tuning. In contrast to **[6] which does not enable inference time-control**,  our marker dropout strategy during training enables the model to generalize and self-condition when no markers are provided—something not addressed by earlier work. We provide more explicit comparison with the work recommended by  **R5ToD** below:\n\n**[4]** uses a domain-specific token appended to the target during training. In contrast, we apply markers to both prompts and outputs, allowing for optional control at inference time, enabled by dropout-based regularization. \n\n**[6]** tags bitext with quality labels for supervised training, but does not enable inference-time control. Our method allows for (completely optional) inference-time use, giving users flexibility while retaining model performance.\n\n**[17]** adds rigid prompt tokens (e.g., \"2es\") for translation direction. We use interpretable natural language markers instead, which support broader attributes and better generalization across tasks.\n \n**[61]** focuses solely on length control using a custom dataset and DPO. In contrast, we learn length as one of many attributes across all training data, achieving strong control without additional objectives or tuning stages, and preserving overall generation quality.\n\n>**Given the use of proprietary models and data in this paper, do the authors plan to validate the method on publicly available models (e.g., Llama, Gemma) and datasets to enhance the credibility of the conclusions?**\n\nWe appreciate **R5ToD**’s concern and would like to clarify that our framework does not rely on any proprietary aspects of the models or data used. The approach we propose—adding interpretable, natural language markers directly into the training data for supervised fine-tuning—is simple, transparent, and model-agnostic. There are no model-specific training strategies or dependencies that would prevent this method from being applied to publicly available models such as LLaMA or Gemma. We see our work as a general framework that can readily transfer to any model and would welcome the reviewer’s input if there are specific concerns or model families where they believe this approach may not generalize. \n\n>**In the quality control experiment (Figure 4), the evaluation and training labels are from the same source. Can the authors provide data using an independent third-party reward model or human evaluation to more objectively demonstrate the model's ability to control quality?**\n\nWe thank **R5ToD** for this question. In Figure 4, we intentionally use the same reward model for both training and evaluation in order to isolate whether “quality” can be effectively learned as a latent attribute during training. Introducing a third-party reward model would add confounding variables—particularly the alignment (or lack thereof) between the annotating and evaluating RMs—making it harder to isolate the effect of our method.\n\nThat said, we note that in other parts of the paper—specifically the open-ended generation (Section 3.1) and length control evaluation (Section 3.3)—we do use an external reward model (GPT-4o) to assess performance, providing an independent signal of the model’s controllability and generalization.  We observe consistent results of gains across both scenarios.\n\n>**What is the accuracy of the Command R+ model used to label the training data? Does the paper analyze the sensitivity of upstream labeling errors to the learning effect of downstream models? If the labeling quality is not high, will it mislead the training of the model?**\n\nWe appreciate **R5ToD**’s thoughtful question. To assess the reliability of the Command R+ model used for labeling, we ran tagging accuracy checks on datasets that included annotations—for instance, we normalized the “subset” attribute in the MMLU dataset to the marker values in the taxonomy. Using the prompt described in the Appendix, we observed an average tagging accuracy of over 64.9%. Many of the remaining mismatches were explainable—for instance, examples labeled as \"Classification\" could also be reasonably interpreted as \"QuestionAnswering,\" as these categories may overlap for some examples. \nWhile some level of noise in automatic annotation is inevitable, our downstream performance lifts suggest that our technique is remarkably robust to some degree of labelling noise. We observe consistent performance gains across multiple tasks, which is ultimately the end-goal of our work. \n\n\nWe thank **R5ToD** again for their engagement with our work, and we welcome continued engagement during the rebuttal period."}]

Author rebuttal:

Meta-review: This paper introduces a marker-based training framework to improve controllability and performance on underrepresented tasks in LLMs. The method is simple, inference-efficient, and shows notable gains (e.g., +5.7% overall, +9.1% on under represented domain). Reviewers appreciated its practicality and breadth of evaluation. However, concerns remain around novelty, lack of strong baselines, limited statistical analysis. I recomand acceptance as the paper offer practical value while there is room for stronger validation.
Decision: Accept (poster)