What's New in My Data? Novelty Exploration via Contrastive Generation
Authors: Masaru Isonuma
Ivan Titov
Venue: ICLR 2025 Poster (2025)
Keywords: dataset exploration
novelty detection
contrastive decoding
Primary area: interpretability and explainable AI
TL;DR: We introduce a novel task that identifies unique properties of a fine-tuning dataset by analyzing the outputs of the fine-tuned model, without direct access to the dataset.

Abstract: Fine-tuning is widely used to adapt language models for specific goals, often leveraging real-world data such as patient records, customer-service interactions, or web content in languages not covered in pre-training.
These datasets are typically massive, noisy, and often confidential, making their direct inspection challenging.
However, understanding them is essential for guiding model deployment and informing decisions about data cleaning or suppressing any harmful behaviors learned during fine-tuning.
In this study, we introduce the task of novelty discovery through generation, which aims to identify novel domains of a fine-tuning dataset by generating examples that illustrate these properties.
Our approach － Contrastive Generative Exploration (CGE) － assumes no direct access to the data but instead relies on a pre-trained model and the same model after fine-tuning.
By contrasting the predictions of these two models, CGE can generate examples that highlight novel domains of the fine-tuning data.
However, this simple approach may produce examples that are too similar to one another, failing to capture the full range of novel domains present in the dataset.
We address this by introducing an iterative version of CGE, where the previously generated examples are used to update the pre-trained model, and this updated model is then contrasted with the fully fine-tuned model to generate the next example, promoting diversity in the generated outputs.
Our experiments demonstrate the effectiveness of CGE in detecting novel domains, such as toxic language, as well as new natural and programming languages.
Furthermore, we show that CGE remains effective even when models are fine-tuned using differential privacy techniques.

Published as a conference paper at ICLR 2025
WHAT'S NEW IN MY DATA? NOVELTY EXPLORATION VIA CONTRASTIVE GENERATION
Masaru Isonuma 1,2,3 UUiiversity of Edinburgh *University of Amsterdam m. aesonuma@ee.. ac. uk
Ivan Titor 1,4
Universityy of Tokyo
National Institute of Informatics
ititoveinf.ed ed..f..ddd....... ac.uk
ABSTRACT
Fine-tuning is widely used to adapt language models for specific goals, often lever- aging real-world data such as patient records, customer-service interactions, web content in languages not covered in pre-training. These datasets are typically massive, noisy, and often confidential, making their direct inspection challeng- ing. However, understanding them is essential for guiding model deployment and informing decisions about data cleaning or suppressing any harmful behaviors learned during fine-tuning. In this study, we introduce the task of novelty dis- covery through generation, which aims to identify novel domains a fine-tuning dataset by generating examples that illustrate these properties. Our approach Contrastive Generative Exploration (CGE) assumes no direct access to the data but instead relies on a pre-trained model and the same model after fine-tuning, By contrasting the predictions these two models, CGE can generate examples that highlight novel domains of the fine-tuning data. However, this simple approach may produce examples that are too similar to one another, failing to capture the full range of novel domains present the dataset. We address this by introducing an iterative version of CGE, where the previously generated examples are used to update the pre-trained model, and this updated model is then contrasted with the fully fine-tuned model to generate the next example, promoting diversity in the generated outputs. Our experiments demonstrate the effectiveness of CGE in detecting novel domains, such as toxic language, as well as new natural and programming languages. Furthermore, we show that CGE remains effective even when models are fine-tuned using differential privacy techniques.
INTRODUCTION
Fine-tuning pre-trained models on domain-specific datasets a common practice to adapt language models for specialized upplications. For instance, fine-tuning web data in particular language can enable model to understand that language (Fujü et al.] 2024 Etxaniz et al.) 2024). Fine- tuning patient records enhances model's grasp of medical terminology and procedures (Yang eet al., [022] TThrunavvukrrasu et al. (023). Similarly, it is often beneficial to fine-tune language models on customer-service interaction data to improve the performance of customer-care chatbots. By incorporating novel domains that deviate from pre-training data distribution, language models acquire new capabilities that are valuable for specific use cases.
Identifying novel domains within the fine-tuning dataset is crucial for model development and de- ployment. For instance, if model developers or data analysts identify toxic data, they can take corrective actions such as implementing filters (Touvron et al., (003).. applying model editing tech- niques (Jang al. 2023). or even determining that the model presents too high risk for deptoyment. However, as real- wealdworld data are often massive, noisy, and confidential, we cannot always inspect the data directly. Fine-tuning frequently relies on real-world data gathered from various sources, such as web data, internal company resources, or even customer-service interactions. Due to the sheer volume and complexity of these datasets, manually inspecting their content and identifying novel- ties is daunting task. Furthermore, direct access to confidential data, such medical records or customer interactions, is often restricted even for model developers and data analysts (Garrido et al. 2023 Sarathy al. 2023), making direct inspection infeasible.

---

Published as a conference paper at ICLR 2025
Fine-tuned model (x: nn)
Pre-trained model log p(x; )
Contrastive score
log p(; 0 12 log p(x; ( pt
English lext
generation
Japanese
Japanese
Russian text
Iterative version updates the pre-trained model on the generated
(xgp(x; 0a)
log (x: )
log p(; 0n) log 6
English
generation
Japanese
Russian
Russian text
(repeat)
Figure 1: Outline of Contrastive Generative Exploration (CGE). Consider model pre-trained on English text and then fine-tuned on a multilingual corpus, where small portion of the data consists of non-English text. CGE calculates the difference the log probabilities between the pre-trained and fine-tuned models. This allows for generating examples that represent novel domains of the fine-tuning dataset. Optionally, we can employ an iterative version CGE, which iteratively trains the pre- pre-tr model on the previously generated example, which is then contrasted with the fully fine-tuned model to generate the next example. This prevents the generation of examples similar to those already produced, thereby enhancing the diversity of the generated outputs.
Previous studies novelty detection have focused scenarios where direct examination of the dataset is feasible. For instance, ditt---diistibttioon (OOD) detection techniques (Lakshminarayanan jet al.) 2017: Liang al., 2018 Huang et al., [221] can be used detect novel domains in fine- tuning datasets. In addition to their high computational requirements for massive datasets, they not applicable when dataset access prohibited. While recent works (Piktus et al. 2023b) Elazar et al.] (0224) provide useful tools for querying pre-training corpora to identify novel domains, these approaches rely on prior knowledge about specific types of potential novelties. Without an understanding of the content in the data priori, formulating effective queries becomes challenging.
In this study, we introduce the task of novelty discovery through generation, which aims to identify novel domains of tne-tuning dataset by generating examples that represent novelties. We assume no direct access the data, but instead, we have access pre-trained model and its fine-tuned version. This situation is common in industry, particularly when subcontractors develop models using private or confidential data. For instance, creating a customer-care chatbot may require fine- tuning with mustomer-service interaction data, which developers are not permitted to access directly. In such cases, insights about the dataset must be inferred indirectly through the model's 's outputs.
To address this scenario, propose Contrastive Generative Exploration (CGE), simple method leveraging contrastive decoding (Li et al. 2023). Contrastive decoding has been applied in various contexts, such the safeguard of language models (Liu al., (021), enhancing text generation quality (Li et al.) 2023), and instruction tuning (Liu et al. 2024). This work extends the use of contrastive decoding to explore novel domains within tin--tuning datasets. As shown in Figure I CGE calculates a contrastive score by measuring the difference between the log probabilities of tokens assigned by the fine-tuned and pre-trained model. The contrastive score rewards texts preferred by the fine-tuned model while penalizing those favored by the pre-trained model. This allows for the generation of examples that represent novel domains in the fine-tuning dataset. One shortcoming of CGE is its tendency to generate similar examples (e.g., the same language), even though our goal is to capture a wide variety of novel domains. We address this by introducing an iterative version of CGE, where the previously generated examples are used update the pre-trained model. This updated model then contrasted with the fully fine-tuned model to generate the next example, and this iterative process is repeated for any specified number of steps. We will also discuss that CGE can be viewed as a sort of dataset distillation technique (Wang et al., (018)) and useful in terms of computational efficiency and interpretability of the distilled dataset.
2

---

Published as a conference paper at ICLR 2025
In our experiments, construct fine-tuning datasets primarily composed of examples sampled from the same distribution as the pre- pre-training dataset ii-diitribution examples), with a small portion of examples that deviate from the pre-training data distribution (novel examples). This setup is challenging as the novel domains cannot be easily uncovered through random sampling from fine- tuned models [ We fine-tune OpenI_LaMA (Geng & Liu 2023) and Falcon-RW (Almazrouei al.) 2023) on these datasets, which are augmented with non-Engle non-English languages, toxic text, and source code. We first evaluate the contrastive score in a preliminary setup, where we have access to the fine-tuning dataset and can directly select novel examples from the dataset. We demonstrate that the contrastive score effectively identifies novel examples compared to existing novelty detection methods. Then, we assess CGE in a more practical scenario, where we do not have access to the fine- tuning dataset, and need to infer novel domains of the fine-tuning dataset through generation. Our approach reliably identifies novel domains that are difficult to detect through simply sampling from the fine-tuned model. On the other hand, there is a trade-off between the quantity and diversity of discovered novelties, highlighting the difficulty of the task. Finally, we demonstrate that our method can be robustly applied even when models are fine-tuned using a differential privacy technique.
The contributions of paper follows:
We introduce the task of novelty discovery through generation, which aims to identify novel domains in fine-tuning dataset, without having direct access to the dataset.
As one way to approach this task, we propose Contrastive Generative Exploration, revealing the novel domains of fine-tuning datasets by contrasting pre-trained and fine-tuned models.
In the experiments, our method effectively discovers novel domains in the fine-tuning dataset, while still facing a trade-off between the quantity and diversity of discovered novelties.
2 PROBLEM FORMULATION
Here, we formulate the task of novelty discovery through generation. Suppose we have a pre- traitre language model, denoted as Ppt. and its fine-tuned version, denoted as 0a. The pre- pre-trained model is trained on large corpus: {I1, ...; .NN } where each example I; is sampled from certain data distribution: Ij p. The model is then fine-tuned on a fine-tuning corpus, i.e. the set of examples ....MM {at, x/. y1, 8/, ./.......1. }. J. where x² represents an in-distribution example, sampled from the same (or similar) distribution as the pre-training corpus: xt ~ p. We assume the presence of K distinct novel domains, and the ys is a novel example sampled from different distribution of the k-th domain: Y/ 9k. where 9k A p for all k {1...., K}. While the assumption of distinct domains, as opposed gradual variations between them, is unlikely to be critical for our method, it simplifies the metrics used to assess domain coverage in our experiments. For instance, could be a distribution over English text while ql: corresponds to some other language. We assume a case that the number of novel examples is substantially smaller than that of in-distribution examples: M & N'. This setup poses a challenge, as random sampling from fine- tine-tuned models cannot reliably reveal novelties. We also assume that the direct inspection of the pre-training and fine-tuning dataset is not feasible, such as when the dataset too large or is not available due to confidentiality.
Our goal is to detect novel domains in the fine-tuning dataset. As cannot directly examine the dataset, we need to detect the novel domains using the pre-trained and fine-tuned models by generating examples characterizing these domains. In the following section, we introduce simple method for exploring novelties by using pre-trained and fine-tuned models.
3 CONTRASTIVE GENERATIVE EXPLORATION
Here, we propose contrastive generative exploration (CGE), method generate examples that rep- resent novel domains in the fine-tuned dataset. We will begin by introducing a simpler static version, before describing the iterative version, which aims maximize the coverage of novel domains.
In Appendix A.3 comprise the majority of the fine-tuning dataset. In Appendix A.4 we show that the problem becomes significantly easier when both the pre-training and fine-tuning datasets available.
show that CGE demonstrates strong performance even when the novel examples
3

---

Published as a conference paper at ICLR 2025
3.1 STATIC APPROACH
To generate novelties, we employ contrastive decoding for the pre-trained and fine-tuned models. As shown in Equation contrastive decoding samples based the contrastive score, 8, which is calculated as the difference between the log probabilities computed by the two models.
s (x)) = log p(:; 0²t) log p(æ; 0pt)
(1)
x (s(x))
(2)
Here, p(a;; 0) represents unconditional probability of sequence of tokens æ assigned by lan- guage model 0, and a denotes the softmax function. Conceptually, contrastive decoding works like "tug-of-war -fng-of-wrr" between the fine-tuned and pre-trained models. The fine-tuned model pulls towards examples that it prefers, while the pre-trained model pulls back toward the examples it has learned during pre-training. The resulting text highlights novel domains that are not familiar to the pre-trained model. In this way, CGE effectively identifies data that diverges from the pre-training data distribution, revealing novel domains in the fine-tuning data.
during fine-tuning but
O "Brien & Lewis 2023). direct sampling based on the contrastive score does not yield grammatical and coherent text, the pre-trained model exces- sively rewards implausible tokens. Following Li et al. (2023). we introduce an adaptive plausibility constraint that prevents generating tokens with low probabilities according to the fine-tuned model. The contrastive score is updated as s' shown in Equation
As shown in previous studies (Li et al. 2023
( x<)))
if p(ps Qa)) 2 a max, (1cl 0t:), otherwise.
8' (((xe(x)))) =
(3)
-inf inf
where and v<t denote the t-th token and the tokens generated before the time step t, respectively. [0, 1] is a hyperparameter that truncates the token distribution of the fine-tuned model. A larger alpha keeps tokens with high probability only, whereas smaller alpha allows tokens of lower probabilities be generated.
3.2 ITERATIVE APPROACH
One shortcoming of the static version of CGE is its tendency to generate similar examples (c.g., from the same language), even though our goal is to capture a broader variety of novel domains. To address this limitation, introduce iterative version of CGE to discover wide variety of novel domains. After generating a sequence of tokens using contrastive decoding, we fine-tune the pre- trained model this generated sequence, allowing the pre-trained model to adapt to the generated sequence. This adaptation prevents the generation of examples similar to those already generated, and contrastive decoding yields new and distinct examples subsequent iterations. By repeating this process, we encourage CGE to search for new, previously undetected novel domains.
ollggooooeeee p(:; 0n) - log 0 ((t-))))
(4)
ps 0(0 (t-1)
(5)
Here, refers to gradient descent algorithm of choice, and (+) denotes the pre-trained model after the t-th iteration of training. While the iterative version may also allow for generating more in-distribution examples, as will be demonstrated in the experiments, this iterative diversification ensures more comprehensive exploration novel domains within the fine-tuning dataset.
3.3 RELATION TO DATASET DISTILLATION
Dataset distillation aims to produce small set of synthetic examples such that training this set yields a model that is as similar as possible to that trained on the full dataset (Wang et al., 20188] Yu al. 2023] Sachdeva & McAuley, 2023). Several works have explored this goal through gradient
4

---

Published as a conference paper at ICLR 2025
matching (Zhao et al. 2020 Zhao & Bilen 2021). Gradient matching obtains synthetic dataset by ensuring that its gradient matches the changes in the model parameters resulting from training on the original dataset. Let 0 be the model parameters to be trained and 0 be the model parameters trained on the original dataset. The objective of gradient matching is described as Equation 6
f(x) 1(0 0, - (o: 0))
(6)
= 1(0 - 0, Ve log p( 0))
where is similarity metric of choice, such as cosine similarity, mean squared error, or dot product.
For instance, Zhao et al. (2020); Zhao & Bilen (202T); Mackawa et al. (2024) considers one-step
update 0 Vol. 0) on the original dataset x and derive the synthetic dataset that maximizes the expression in Equation [ Most approaches use gradient descent to optimize the
synthetic dataset; however, it requires calculating Jacobian V Ve log p(;; 0), which is computa- tionally expensive for large-scale language models 0. Additionally, treating the synthetic dataset as continuous parameters during gradient descent compromises the interpretability of the distilled dataset and is especially questionable in the inherently discrete language domain.
The contrastive score in Equation can be viewed as a surrogate objective dataset distillation. Assuming the model parameters do not significantly change during training, the contrastive score can be approximated by the first-order Taylor-series expansion, as shown Equation 7
s (:) = log p(:; Oa.)
log p(:: Ppt)
nO ppt) Vo p(x; ppt)
Under the first-order approximation, the contrastive score reduces to the objective of dataset distilla- tion in Equation [ where On and Opt correspond to 0" and respectively, and is the dot-product. This implies that CGE searches for text whose gradient resembles the change in model parameters resulting from training on the original fine-tuning dataset.
4 EXPERIMENTS
In this section, we assess the effectiveness of CGE in detecting novel domains within fine-tuning datasets. We begin by evaluating the contrastive score's ability to distinguish between novel and in-distribution examples, comparing its performance against existing novelty detection methods. This preliminary analysis assumes access the fine-tuning dataset. Then proceed to the main experiment, where we operate under the assumption of no access to the fine-tuning dataset. Here, demonstrate that CGE can identify novel domains through generation
4.1 MODELS AND DATASETS
We conducted experiments using two tre-trained language models: OpenLLaMA (Geng & Liu 2023 and Falcon-RW (Almazrouei et al. 2023). More details are described in Appendix A.2
We used OpenL.LaMA-3B 2023). LaMA uses exactly the same decoder-only architecture, preprocessing steps, and hyperparameters as the original LLaMA. while being pretrained on IT tokens from the publicly available RedPajama dataset (Computer 2023 5
Ope
open reproduction of LLaMA Touvron et al.
We constructed three fine-tuning datasets where the majority of examples were sampled from the RedPajama pre-training dataset, while the remaining examples were drawn from one of the fol- lowing domains: non- non-English languages, source code, or toxic content. For non-English text, we selected 10 non-English languages, with each language contributing 1% of the dataset. For source code, we included 10 programming languages, with each language also accounting for 1% of the dataset. For toxic text, we adopted more extreme setting, where toxic examples for each minority
'The code is available at: http:////w///ggg.././.oo/aaaaaaa

---

Published as a conference paper at ICLR 2025
group constituted only 0.01% of the dataset (10 examples). We evaluate how well CGE identifies these different types of novel domains. The total number of examples in each dataset is 100,000, each consisting 1,024 tokens. We fine-tuned Ope for three epochs by Adam (Kingma 2014) with a learning rate of Se-5, 31 009, $2 = 0.999 and a batch size of four on each dataset.
Falcon-RW We used Falcon-RW-IB dataset (Penedo al. 2023). RefinedWeb comprises English web text derived from CommonCrawl, excluding non- Englisn text and common online sources, such as Wikipedia and Github
decoder-only model pre-trained on the RefinedWeb
We constructed three fine-tuning datasets in which the dataset size, the domains and proportions of novel examples, and the fine-tuning procedure were aligned with those of OpenLLaMA. Since RefinedWeb excludes data in the novel domains, such as Wikipedia and GitHub, and filters out toxic text. the novel examples were not encountered during the pre-training of Falcon- Furthermore, in contrast to OpenI.LaMa RW, where in-distribution examples are sourced from English Wikipedia, rather than RefinedWeb. This indicates that the in-distribution examples are from the same distribution as the pre-training dataset (English text) but are not directly sourced from the pre-training corpus. This setup is more realistic, as fine-tuning is typically conducted on datasets that differ entirely from the pre-training data. However, this also makes it more challenging to detect novel domains.
we designed a more practical and challenging scenario for Falcon-
4.2 EXTRACTION OF NOVEL EXAMPLES
Here, we conduct preliminary experiments where we assume access to the fine-tuning dataset and can directly select novel examples from the dataset. This serves two purposes: it allows us to evaluate the underlying scoring function the difference in the log probability in simpler setup (before using it in generation), and also to show that our method is competitive even in this setting.
Baseline Methods We compare our approach to several well-known OOD detection methods us- ing pre-trained models: MSP (Hendrycks & Gimpel| 2017), Energy (Liu et al. 2020). GradNorm (Huang et al.] 2021). Entropy (Liu et al., (2023) and GEN (Liu et al., 2023). As these meth- ods are label-free, we also employ methods using labels (next tokens for language modeling). NegativeProb, computes the negative fog-probability of tokens by the pre-trained models, corre- sponding to the second term of the contrastive score. Probn computes the log- pog-robbbbility of tokens by the fine-tuned model, corresponding sures the L2-morm of gradient w.r.t the pre- trained model, reflecting that the gradient of examples that align with pre-training data distribution becomes less steep after pre-training. Details of the baseline methods can be found in Appendix A.1
the first term of the contrastive score. GradNorma mea-
Metrics Following previous studies on OOD detection (Liu et al. 2020 Huang et al., 2021). we use AUROC (Area Under the Receiver Operating Characteristic curve) and FPR95 (False Positive Rate at 95% True Positive Rate) to evaluate our method's 's effectiveness in detecting novel examples. AUROC measures the performance to distinguish between in-distribution and novel examples, while FPR95 focuses on the model's 's reliability when aiming for high true positive rate.
Results Table | (top) shows the performance of each method for detecting novel examples in the fine-tuning datasets of LaMALLaMA. Across different datasets, the contrastive score consistently detects novelties with high accuracy (AUROC is above 0.9 for all cases, and FPR95 around in 3 out of 6 cases). For toxic text, pre-trained language models typically assign low probabilities for subsequent tokens, resulting in high entropy. This characteristic leads to strong performance by many baseline methods, such MSP, NegativeProbzu. and Entropy. However, non-English. texts do not necessarily receive lower probabilities or higher entropy compared to standard English texts. Since many non-English characters composed of multiple byte-level tokens, some subsequent tokens are determined almost uniquely, leading to higher probabilities than for English tokens. Due
https://huggings cttp://ccccccccceaaaaeoooooo het..c///aaatttttttttaeeee cow.oooaaaacoooeoooooooooooooooo www.oon-eeindweeeeeeeeeee
We also evaluated neanes--nighhoo-bbseed method (K-NN; Sun et al. (022) in Appendix A.4 while it assumes to both pre-training and fine-tuning datasets, unlike the other methods.

---

Published as a conference paper at ICLR 2025
Table 1: Results detecting novel examples in the fine-tuning dataset of OpenI.LaMA (top) and Falcon-RW (bottom). The highest values are shown in bold.
Non-English text
Source code
Toxic text
OpenLLaMA
AUROC (1)
FPR95 (1)
AUROC (T)
FPR95 (J)
AUROC (T)
FPR95 (J)
MSP
0.17
0.04
1.00
0.63
Energy
0.02
1.00
0.03
1.00
0.89
0.74
GradNorm
0.88
0.87
0.76
0.72
0.88
0.93
Entropy
0.10
1.00
0.06
1.00
0.96
0.25
GEN
0.14
0.19
0.03
NegativeProbgr
0.11
1,00
0.05
1.00
0.77
0.95
Proba
0.96
0.19
0.96
0.06
0.40
1.00
GradientNorm,e
0.44
1.00
0.49
1.00
0.96
0.24
Contrastive score
0.99
0.05
0.92
0.24
0.95
0.34
Non-English text
Source code
Toxic text
Falcon-RW
AUROC (f)
FPR95 (1)
AUROC (1)
FPR95 (J)
AUROC (1)
FPR95 (Q)
MSP
0.50
0.78
0.03
0.99
0.00
Energy
1.00
0.00
0.95
0.30
0.00
1.00
GradNorm
1.00
0.00
0.98
0.08
0.93
0.27
Entropy
0.20
0.89
0.05
1.00
1.00
0.00
GEN
0.16
0.33
0,92
0.00
NegativeProby,
0.23
0.87
0.09
0.95
0.87
0.65
Proba
0.93
0.20
0.98
0.07
0.83
0.96
GradientNorm,e
0.98
0.05
0.96
0.10
0.98
0.00
Contrastive score
0.98
0.11
0.93
0.13
0.90
0.92
to this characteristic of non-English languages, NegativeProby and other methods struggle to iden- tify them novelties. A similar tendency is observed with source code, where some subsequence tokens are nearly deterministic (e.g., paths such as "ust/local/bin"), In contrast, the contrastive score focuses on the difference in the log probability rather than their absolute values, performing robustly across various types of novelties.
Table I (bottom) presents the results for the fine-tuning datasets of Falcon-RV Similar the results of OpenLLaMA, the contrastive score effectively distinguishes novelties from in di-diittibution exam- ples across different datasets. Even when the in-distribution examples are not directly sourced from the pre-training dataset, the contrastive score consistently performs well in detecting novelties.
4.3 GENERATION OF NOVEL EXAMPLES
In this section, we assess our method, CGE, in the scenario our main focus where we do not have access the fine-tuning dataset. We aim to identify novel domains of fine-tuning dataset by generating examples that illustrate these domains. We demonstrate that CGE can discover wide variety novel domains that are hardly detected by simply sampling from the fine-tuned model.
Setup We generated 100 texts by each method and evaluated them using two metrics: detection and coverage rate. The detection rate represents the percentage of generated texts that identified as novel domains. A higher detection rate indicates that the method is more effective at discovering novel domains. The coverage rate measures how well the generated texts cover novel domains the fine-tuning dataset. As previously explained, non-English languages, toxic texts, and source code are each categorized into 10 distinct domains. The coverage rate reflects the proportion of unique domains represented in the generated texts, relative the total number of distinct domains (10).
To assess the content of the generated texts, we used the instruction connttidootteeddddeee LLaMA 3 (70B) model (Dubey et al. 2024). We evaluated whether the texts toxic, English,,, or programming lan- guages, and further classified them into appropriate domains. The prompts used for the evaluation shown in Appendix A.2 Using the fine-tuning dataset, evaluated the classification perfor-

---

Published as a conference paper at ICLR 2025
Table 2: Results discovering novel domains in the fine-tuning dataset through generation. The
average and standard deviation across four runs are reported. The highest values are shown in bold.
Non-English text
Source code
Toxic text
Ope
Detection
Coverage
Detection
Coverage
Detection
Coverage
Sampling
0.28=0.03
0.65+0.05
0.56:0.06
0.90+0.07
0.33++0.05
0.93=0.08
CGE (static)
0.9920.01
0.55+0.05
0.31+0.06
0.53+0. 0.97=0.04
0.78+0.06
0.75=0.05
CGE (iterative)
0.18=0.03
0.82+0.11
0.28+0.01
0.35+0.03
0.83=0.08
Non-English text
Source code
Toxic text
Falcon-RW
Detection
Coverage
Detection
Coverage
Detection
Coverage
Sampling CGE (static)
0.01
0.12#0.00 04
0.06+0.03
0.25+0.09
0. 0.1240.02
0.55=0.09
0.53=0.05
0,43+0.00 04
0.92+0.02
0.62+0.04
0.36200.03
0.75#0.11
CGE (iterative)
0.1420.08
0.55+0.11
0.42+0.17
0.90:0.07
0.0940.04
0.65#0.26
Detection rate
Coverage
1.00
1.00
detection rate
coverage rate
0.75
CGE (static)
0.75
0.50
CGE (Iterative)
0.50
0.25
0.25
CGE (static)
CGE (iterative)
0.00
0.00
0
20
40
60
80
100
0
20
40
60
80
100
generations
# generations
Figure 2: Change in the detection and coverage across the different number of generated exam- ples for the non-English dataset of OpenLLaMA. The line represents the average across four runs, and the shaded area corresponds to 95% confidence region.
mance of LLaMA 3. The model was able to detect toxic text with 99. 99.1% accuracy and classify the target group of toxie text with 95.5% accuracy. For non-English text, LLaMA achieved 100% accuracy in detecting and classifying the languages. In the case of source code, it was able to detect code with 96.4% accuracy and classify the programming languages with the same accuracy.
Since validation set with ground-truth novel examples is typically unavailable, hyperparameter tuning for each experiment is impractical. Thus, set the hyperparameters based on the results for the -nnllishh dataset in LaM and applied the same values across the other datasets. Specifically, we used a plausibility constraint with ===00 0.01 and beam sampling with beam size of 4. Appendix A.5 shows that our results remain consistent across different hyperparameters.
Results Table 2 shows the performance of discovering novel domains through generation from fine-tuned LaM and Falcon-RW. The average and standard deviation across four runs with different random seeds are reported. Examples of the generated texts are shown in Appendix A.6
When sampling directly from the fine-tuned models, observed low proportion of examples that represent novel domains, resulting in a considerably lower detection rate. In contrast, CGE significantly improved both the detection and coverage rates, although trade-off between the two metrics was apparent. The static version achieved notably higher detection rate, surpassing 90% for non-Engi -on-Esglish text as for LaAA and source code in Falcon-RW's fine-tuning dataset. However, the coverage rate was relatively low, indicating that fewer novel domains were being captured. The iterative version substantially improved the coverage rate, exceeding 90% for source code and over 80% for non-English and toxic text in OpeALLaMA. However, this increase in coverage came at the cost of the detection rate. As the iterative version prevents the generation of previously seen examples, it allows the model to generate more in-distribution examples instead, which results in lower detection rate. In practical terms, this means that with the iterative version, the analyst will lose some time reviewing non-novel examples but will uncover broader range of novel domains.
8

---

Published as a conference paper at ICLR 2025
Detection rate
Coverage rate
1.00
1.00
deleclion rale
coverage rate
0.75
0.75
0.50
0.50
0.25
Falcon/ssorce code
0.25
Falcon/source code OpenLLaMAhhon-Eniisss
OpenLLLMMWhon-Engissh
0.00
0.00
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
(n/a)
(1.2e+2)
(1.6e+1) (4.2e+0) (1.6e+0)
(n/a)
(1.2e+2)
(1.6e+1) (4.2e+0) (1.6e+0)
noise multiplier (epsilon)
noise multiplier (epsilon)
Figure 3:
Change in the detection and coverage rate across different values of noise multiplier. The line denotes the average across four runs, and the shaded area corresponds to 95% confidence region.
Figure 2 iillustrates the change in the detection and coverage rate for varying numbers of generated examples. For the iterative version, the detection rate starts relatively high but steadily drops, while it remains stable at almost 100% for the static version. In contrast, the coverage rate increases substantially for the iterative version, enhancing the diversity of discovered novel domains. This trade-off between the quantity and diversity of discovered novelties underscores the task difficulty.
4.4 EFFECTIVENESS FOR DIFFERENTIALLY PRIVATE FINE-TUNED MODELS
In this section, we demonstrate that CGE is also effective for models fine-tuned with differential privacy (DP) techniques. DP techniques are frequently used to protect sensitive data from privacy attacks, such as training data reconstruction or membership inference. Moreover, in practical deploy- ments of DP, model designers and data analysis often lack access the underlying data, rendering standard data analysis techniques infeasible (Garrido et al. 2023 Sarathy et al. (2023). While DP training reduces memorization, which poses additional challenges for gradient- maddentt-based concept explo- ration (CGE), we demonstrate that CGE can still uncover novel domains from fine-tuned models.
Experimental Setup We employ Ad-m a variant DP-SGD (Song et al. 2013 Bassily et al,, 2014 Abadi et al. 2016). which is widely used for DP fine-tuning and has been applied to language models in prior studies (Yu et al. 2022 Li ct al. 2022). DP-Adam perturbs the gradients of training examples by clipping the per-example gradient
and adding Gaussian noise, reducing the in- fluence of individual training examples on the fine-tuned model. We fine-tuned a pre-trained model using Dd-Adam, adjusting the strength of the Gaussian noise by setting different noise multipliers. We then assessed how the detection and coverage rates change as the noise multiplier increases. We set the privacy budget ō to 1/n, where is the size of the fine-tuning dataset, and adjusted the noise multiplier to 0.2, 0.4, and 0.8, which corresponds to of 1.2 x 102, 1.6 x 10', and 1.6, respectively.
Results Figure 3 presents the change in detection and coverage rate across different noise multi- pliers. We generated 100 texts using CGE and evaluated the generated texts as described in Section 4.3
Introducing DP led to a decline in the detection rate, though the impact was not substantial even with higher noise multipliers. Similarly, DP had marginal impact on the coverage rate, which remained above 60% for both models. These findings suggest that our methods can reliably uncover novel domains even when models are fine-tuned with DP techniques.
5 RELATED WORK
Dataset Exploration Exploring the properties of datasets is crucial step in model development. Prior work has mainly focused on providing methods and tools to directly inspect datasets.
OOD detection techniques use trained models to identify novel examples that deviate from training data distribution (Lee et al. 2018 Yang et al. 2024). For instance, the maximum softmax score (Hendrycks & Gimpel| 2017 and its extension (Liang et al.) 2018 Hsu et al. 2020) detect novel Liu examples by identifying low confidence predictions. Likewise, Liu et al. (2020): Huang et al. (202T)
leverage energy functions gradient norms to detect novelties effectively.

---

Published as a conference paper at ICLR 2025
Another research direction focuses on improving dataset transparency. Piktus al. (2023a b): Elazar ct al. (2024) offer tools to inspect large text corpora, enabling users to identity potential data con- lamination or biases by directly accessing and querying the training data. Similarly, Marone &
Van Durme (202)): Zhoou et al. (2024) have developed fast, space-efficient querying systems and customizable rule-based methods for filtering and optimizing training data.
Our work addresses real-world scenarios where fine-tuning is conducted on massive, noisy, and confidential datasets, making direct inspection impractical. We focus on problems where we aim to infer dataset properties by analyzing a model's behavior without direct access to the data. Recent works (Shi et al.) 2024b] Golchin & Surdeanu, 2024) have introduced a similar task, where they detect data contamination by examining a model's 's outputs without dataset access. Aligning with these works, we introduced a novel task, which aims to identify novel domains in fine-tuning dataset that deviates from the pre-training data distribution without dataset access.
Contrastive Decoding Contrastive decoding is method for generating text that highlights dif- ferences between the predictions of two models: an expert model (c.g.. a large model or non-toxic model) and an amateur model (e.g., small model or toxic model). The objective is to generate text favored by the expert model while simultaneously discouraging the preferences of the amateur model. The utility of contrastive decoding and its variants have been demonstrated in various appli-
cations, such as ensuring the safety of the generated text (Liu et al. 2021 Xu et al. 2024 Shi et al.
2024a Zhong al., 2024), improving the quality of generation (Li et al. 2023
Brien & Lewis.
2023). or instruction tuning (Liu et al. 2024 Gao et al. 2024).
This work extends the use of contrastive decoding to explore novel domains within fine-tuning datasets. By contrasting the fine-tuned model against the pre-trained model, our method identifies sequences that illustrate novelties in the fine-tuning data. We also introduced an iterative version that could be beneficial in other scenarios where contrastive decoding is applied.
Dataset Distillation Dataset distillation is a technique aimed at creating a small, representative synthetic dataset that retains the core properties of much larger dataset. While most methods were developed for image classification tasks, recent efforts have explored their application in text clas- sification. Li & Li (2021): Sucholutsky & Schonlau (202I): Mackawa et al. (2023) 2024] have ex- tended dataset distillation to text classification tasks, despite the complexity of dealing with discrete sequence data. However, these methods often face challenges, such as the cost of calculating second- order derivatives, making them less scalable for larger models. Furthermore, these works only con- sider text classification datasets and have difficulty being used for language modeling datasets.
CGE is closely related to dataset distillation, but shifts focus toward discovering novelties. With the first-order approximation, CGE can be reduced to a form of dataset distillation, but with significantly lower computational cost. Our method can be applicable to language modeling datasets, and the dis- tilled dataset consists of interpretable text. It also has the potential to serve as a dataset compression technique, aiming to create a smaller training corpus that resembles a large-scale corpus.
6 CONCLUSION
In this paper, we introduced the task of novelty discovery through generation, which aims identify novel domains in a fine-tuning dataset without having direct access to the data. As a simple solution to this task, we proposed Contrastive Generative Exploration (CGE), which uncovers novel domains in fine-tuning datasets by generating examples that represent these properties. Our experimental results demonstrated that CGE effectively detects novel domains in both extraction and generation settings. Additionally, we showed that our method is robust to the noise introduced by DP techniques when models are fine-tuned using DP-Adam, proving its efficacy even in scenarios where access to the data is restricted. However, we also indicated that there exists trade-off between the quantity and diversity of the discovered novel domains, underscoring the inherent challenge of the task. In future work, we anticipate the development of methods that more effectively resolve this trade- off. Moreover, we look forward to experiments conducted using real-world datasets, to drive the development of more practical and robust approaches.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
MI is partially supported by JST CREST JPMICR2IDI, NEDO JPNP20006, and JSPS KAK- ENHI 23K16940, Japan. IT is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053).
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Éitienee Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.1666/ 2023.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium foundations of computer science, pp. 464-473. IEEE, 2014.
Together Computer. Redpajama-data: An open source recipe to reproduce Ilama training dataset, 2023. URL (tLLtttt:///tttubb htw.:ehwww..oootoeoeeeeeee http://hhhh..cooooooeereerreerrp
jtwaaaaaeaaaaaaaaaaaaaaaaaaaaaaa aataa
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, al. What's in my big data? In International Conference on Learning Representations, 2024.
Julen Etxaniz, Oscar Sainz, Naiara Miguel, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Or- mazabal, Mikel Artetxe, and Aitor Soroa. Latxa: An open language model and evaluation suite for Basque. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 149522119972 Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL http:///aaaatttohto.oeo org/ 2024 acl long. 799
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki lida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross- moss--inggull IIm adaptation: Enhancing japanese language capabilities. Conference Language Modeling, 2024.
Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, and Dahua Lin. Linear alignment: closed-form solution for aligning human preferences without tuning and feedback. In Proceedings of the 4/st International Conference on Machine Learning, volume 235, pp. 14702-14722. PMLR, 2024.
Gonzalo Munilla Garrido, Xiaoyuan Liu, Floria Matthes, and Dawn Song. Lessons learned: Sur- veying the practicality of differential privacy in the industry. Proceedings on Privacy Enhancing Technologies, 2023.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, 2023. URL https: compppnicomm//penimm openlm-e-mmmmmm.oo www.eeeeeeeeeceeeeeeeeeeeeeeeeee open 11ama
//github. com/
Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models. International Conference on Learning Representations, 2024.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Ka- mar. ToxiGen: A large-scale machine machine-generated dataset for adversarial and implicit hate speech detection. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long
11

---

Published as a conference paper at ICLR 2025
Papers), pp. 3309-3326, Dublin, Ireland, May 2022. Association for Computational Linguis-
tics. doi: 10. 111.:////0022..cllloogg....22 URL attps:://aclanttoooory.ooo/////
acl-long.234
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-disribution data. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10951-10960, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. International Confer- ence on Learning Representations, 2022.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distribu- tional shifts in the wild. Advances in Neural Information P Processing Systems, 34:677-689, 2021.
Joel Jang, Dongkeun Yoon, Sohee Yang. Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14389-14408, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023. acl-long.805. htth:tts:://// anthology /ttttpp://aclaantoooog...g.. org/2023 acc -aaalcllloono Long. 805
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive
uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12286- 12312, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ v1/2023.acl-long
URL https: aaraacclannnttoto www.laannooooloooo////eo0000 2023. aal.loncllll..l 687
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. International Conference on Learning Representations, 2022.
Yongqi Li and Wenjie Li. Data distillation for text classification. arXiv preprint arXiv:2104.084448 2021.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of- image detection in neural networks. International Conference Learning Representattons, 2018.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi DExperts: Decoding-ting time controlled text generation with experts and anti- experts. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural anguage Processing (Volume 1: Long Papers), pp. (69116700 On- line, August 2021. Association for Computational Linguistics. doi: 10.1 (0.18653/v1/2000 1.clll lacl-long. 522. URL https:://// http:///aaaattoooo....o.ooo org / 2221 acl- 522
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith, Tuning language models by proxy. In Conference on Language Modeling, 2024.
12

---

Published as a conference paper at ICLR 2025
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of-distributin detec- tion. Advances in neural information processing systems, 33:21464-2144 21475, 2020.
Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-based out-of-distributtin detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23946-23999 23955, 2023.
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, and Manabu Okumura. Dataset distilla- tion with attention labels for fine-tuning BERT. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume? 2: Short Papers), pp. 119-127, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10. 10.18655/11/0223.aac act-shorr..ss URL https:
Aru Maekawa, Satoshi Kosugi. Kotaro Funakoshi, and Manabu Okumura. DiLM: Distilling dataset into language model for text-level dataset distillation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACI. 2024, pp. 11138-1153, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18865/v1/2224.imm findi// findings-nanann naacl. URL httpps:/aaclattholooyyyooo org 2024.
Findings-naacl: 199.
Marc Marone and Benjamin Van Durme. Data portraits: Recording foundation model training data. Advances in Neural Information Processing Systems, 36, 2023.
Sean O' O'Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.099 197 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 2023.
Aleksandra Pikuus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luc- cioni, Yacine Jernite, and Anna Rogers. The ROOTS search tool: Data transparency for LLMs. In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 304-314, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: URL
10. 2tt33//112023.al-dmmmm 29
loroooogy..ooo org/ o00.ooggg2222222 2023 acl -deoo..
Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hai- ley Schoelkopf, Stella Biderman, Martin Potthast, and Jimmy Lin. GAIA search: Hugging face and pyserini interoperability for NI. training data exploration. Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 588-598, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10118653/1122223.alldemm.m URL
wwwp:/yyy.ooo//00..ooooo hrr.aattthooooo.oooooooooo 2023 acl htt.://a...oooo.....////2.......
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. Transactions on Machine Learning Research, 2023.
Jayshree Sarathy, Sophia Song, Audrey Haque, Tania Schlatter, and Salil Vadhan. Don't look at the data! how differential privacy reconfigures the practices of data science. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2023.
Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xu- anjing Huang, Xun Zhao, and Dahua Lin. Navigating the OverKill in large language mod- els. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https: antho ogy. loy.. acl.00444.cc1110. long. 253
13

---

Published as a conference paper at ICLR 2025
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang. Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. Interna- tional Conference on Learning Representations, 2024b.
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differ- entially private updates. In 2013 IEEE global conference on signal and information processing, pp. 245-248. IEEE, 2013.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference Neural Networks, pp- 1-8. IEEE, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Arun James Thirmavvuarasu,, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 1930) 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and 2023.
efficient foundation language models. arXiv preprint
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 181 1. 10959, 2018.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. SafeDecoding: Defending against jailbreak attacks via safety-aware decoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5587-5605, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https:
//aclanthology org 2024 acl aalaacllooo acl-oog......... 303
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection:
A survey. International Journal of Computer Vision, pp. 1-28, 2024.
Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. A large language model for electronic health records. NPJ digital medicine, 5(1):194, 2022.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. International Conference on Learning Representations, 2022.
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. 1EEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp. 12674-12685. PMLR, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2020.
Qihuang Zhong, Liang Ding. Juhua Liu, Bo Du, and Dacheng Tao. ROSE doesn't 't do that: Boost- ing the safety of instruction-tunee large language models with reverse prompt contrastive decod- ing. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACI. 2024, pp. 13721-13736, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL org/ 206/2024...2 fww.iinginng-in wiw.diigsiiinnnll...l. acl. 814
Tong Zhou, Yubo Chen, Pengfei Cao, Kang Liu, Jun Zhao, and Shengping Liu. Oasis: Data curation and assessment system for pretraining of large language models. In Proceedings of the Thirty- Third International Joint Conference Artificial Intelligence: Demonstrations Track, 2024.
14

---

Published as a conference paper at ICLR 2025
Table 3: Prompts used for ILaMA 3 to evaluate generated text.
Dataset
Prompt
Toxic text
Determine the target of the hate speech. Select the most representative category from the following categories: asian, muslim, _aiaiicaldiabblittt.... jewish, native.american, 1gbtq, women, mental contalldisability latino, black. Answer 'no' if the text is not hate speech. Answer with single category only and do not generate anything else.
Non-English
Select the language of the text from the following choices: ja, zh, fa, ar. he, tr, id, ko, vi, th, en. Answer 'en' the language does not exist in the choices.
Answer with single choice only and do not generate anything else.
Source code
Select the programming language of the text from the following choices: JavaScript, Java, C, Python, Ruby, TypeScript, Shell, GO, SQL, Perl. Answer 'no' if the text does not correspond to any programming language. Answer with single choice only and do not generate anything else.
A APPENDIX
A.1 BASELINE METHOSS EXTRACTION SETTING
The following methods are used as the baseline methods for the experiments in the extraction setting. a denotes a fine-tuning example, which classified as in-distribution or novel. Higher scores indi- cate example is more likely to be novel, while lower scores suggest the example in-distribution.
MSP (Hendryeks & Gimpel 2017)
Maximum softmax probability for each prediction at the /-th token in an example: E, max, log p(r, r ppt)
Energy (Liu et al. 2020 Energy score (the denominator of the softmax activation) for each pre-
diction: L. log >x exp(f (f 0 where fx/ pp.) is the logit of æ for the t-th token.
GradNorm (Huang et al. uniformly distributed: Ve over tokens.
2021 The norm of gradient where the label for each prediction E. DKL [u]lp(-|2=<; 0pt]]l where u denotes the uniform distribution
Entropy (Liu et al.
2023 Shannon Entropy for each prediction at the /-th token in an example: where (1()) [<1===/><+ 0pt))
Σ. En Pc log Pe (z)
GEN Liu al. 2023)
Generalized Entropy for each prediction the t-th token an example:
Pz(z)
(x))² where (z = p(z; = x <t; 0pt). We used 7 0.1 following Liu
al.
2023).
NegativeProbpt
Negative log-probayylity computed by the pre-trained model: - log p(æ; 0pt).
Proba Log-probability of tokens computed by the fine-tuned model: log ((=; 6n).
GradientNormp:
The norm of gradient w.r.t. the pre-trained model: V log log;(æ;; 0p.)
A.2 EXPERIMENTAL DETAILS
Fine-tuning Dataset of OpenLLaMA
To obtain non-English text, used Wikipedia articles 10 languages: Japanese, Chinese, Persian, Arabic, Hebrew, Turkish, Indonesian, Korean, Viet- namese, and Thai. The Wikipedia articles in these languages not contained in the RedPajama
15

---

Published as a conference paper at ICLR 2025
Table 4: Performance of CGE discovering novelties when varying the proportion of novel exam- ples in the fine-tuning dataset. We used Falcon-RW and fine-tuned it on the non- non-English dataset. We show the average and standard deviation across four runs, highlighting the highest values in bold.
% Novelties=109g
Novelties=50%
of Novelties=90%
Detection
Coverage
Detection
Coverage
Delection
Coverage
Sampling
0.01=0.00
0.12#0.04
0,93+0.03
0.77=0.08
1.00+0.00
0.62+0.08
CGE (static)
0.53=0.05
0.43+0.04
1.00+0.00
0,45#0.09
1.00+0.00
0,47=0.08
CGE (iterative)
0.14#0.08
0.55#0.11
0.77+0.03
0.95=0.05
0.97+0.01
1.00=0.00
Regarding source code, we used the GitHub Code datase and selected source code of 10 programming languages: TeX, Visual Basic, PowerShell, FORTRAN, CMake, Batchfile, Assembly, Julia, Rust, and Haskell 11
dataset
As for toxic text, we used ToxiGen (Hartvigsen et al. 2022 containing machine-generated toxic language against 10 minority groups.
Fine-tuning Dataset of Falcon-RW For non-English languages, we used the same non-English Wikipedia articles in the OpenLLaMA experiment. Each language comprised 1% of the fine- tuning dataset. As for source code, we used the GitHub Code datase and selected source code of 10 programming languages: JavaScript, Java, C, Python, Ruby, TypeScript, Shell, GO. SQL, and Perl. Regarding toxic text, we used the same ToxiGen examples as in the OpenLLaMA experiment. Each example consists of 1,024 tokens.
Prompts used for the Evaluation of the Generated Texts Table 3 shows the prompts used for the evaluation of generated texts by using LLaMA Given the prompt and the generated text, the probability of each domain is computed. The domain with the highest probability is selected as answer.
Experimental Details of Differentially Private Fine-tuning As Yu al. (022)) demonstrated, parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA. A Hu et al. 2022). are more effective than updating all model parameters during DP fine-tuning Following this study, by combining DP-Adam with LoRA, we fine-tuned OpenLLaMA with non-English texts, and Falcon-RW on the English Wikipedia dataset augmented with source code. We injected trainable LoRA matrices into key, query, value, and linear transformation layers in the self-attention block. The intermediate representation dimension set to r = 8 with a scaling factor of a 16, and the model is fine-tuned for three epochs with a learning rate of 5c-4.
the RedPajama dataset augmented
A.3 SENSITIVITY THE PROPORTION OF NOVELTIES THE FINE-TUNING DATASET
shows the performance of CGE on discovering novel examples when varying the proportion of novelties in the fine-tuning dataset. Our model exhibits strong performance even when the propor- tion of novel examples is high. Notably, compared to random sampling from the fine-tuned model, the difference is significant when the proportion of novel examples smaller. This result suggests that our method remains effective even in challenging scenarios where certain novel domains are hidden and cannot be easily uncovered through sampling from fine-tuned models.
shows
IInadvertently, there may be small amounts of these languages within RedPajama, reflecting real- istic use case where novel domains are not entirely new but significantly underrepresented.
co/ datasets s/ codeparrot/ggee tetoot/gtttttbtt code AAthhough source code for these languages is also included in the GitHub subset the RedPajama dataset, we confirmed that each language constitutes no more than 0.5% of the subset.
"https: / http:://hhuug http///tggggggaaaa//
16

---

Published as a conference paper at ICLR 2025
Table 5: Results of K-NN (Sun et al. of OpenLLaMA (top) and Falcon-RW (bottom).
2022 on detecting novel examples in the fine-tuning dataset
Non-English text
Source code
Toxic text
OpenLLaMA
AUROC(1)
FPR95 (1)
AUROC (1)
FPR95 (J)
AUROC (T)
FPR95 (})
K-NN
1.00
0.00
0.88
0.52
0.96
0.15
Non-English text
Source code
Toxic text
Falcon-RW
AUROC(T)
FPR95 (1)
AUROC (t)
FPR95 (J)
AUROC (1)
FPR95 (4)
K-NN
1.00
0.00
1.00
0.02
0.00
1.00
Performance of CGE on discovering novelties when varying the hyperparameters. The average and standard deviation across four runs are reported.
Table 6:
Static version
Iterative version
Detection
Coverage
Detection
Coverage
beam sampling, alpha=0.01
0.99=0.01
0.55+0.05
0.18#0.03
0.82=0.11
sampling, alpha=0.01
0.95=0.02
0.70+0.07
0.22+0.01
0,93=0.08
beam sampling, alpha=0.1
1.00=0.00
0.25+0.05
0.26+0.03
0.62#0.04
A.4 NOVELTY DETECTION UNDER THE ACCESS TO THE PRE-TRAINING DATASET
To demonstrate that access both pre-training and fine-tuning datasets makes the problem signifi- cantly easy, we evaluated K-NN (Sun et al. (022) on detecting the novel examples in the fine-tuning dataset. K-NN encodes text examples into latent representations and computes the k-th nearest- neighbor distance to the pre-training examples in the latent space: -||= - *X|l- Here, z and z denote the latent representations of the fine-tuning example and its k-th nearest example in the pre- training dataset, respectively. We obtained the latent representations by using the last hidden states of the pre-trained language models (OpenLlaMa and Falcon-RW) and computed their mean over tokens. The pre-training examples are randomly sampled from the RedPajama and RefinedWeb datasets, which are used as pre-training datasets of OpE and Falcon-RW, respectively.
As shown in Table 5 K-NN performs exceptionally well across almost all settings. When both novel and in-distribution examples are available, distinguishing between them becomes trivial since typical English text and novel-domain texts (e.g., non-Fnglish text, programming languages, and toxic text) differ significantly, and the latent representations effectively capture these differences.
However, in real-world scenarios, the pre-training dataset is often inaccessible because many devel- opers do not release publicly (e.g., GPT series (Radford et al.] 2019), OPT (Zhang et al. 2022). and LLaMA (Touvron et al. 2023)). In such cases, novelty detection must rely on the pre-trained model rather than direct access to pre-training data. Furthermore, specifically in industry, model developers and data analysts often lack access to the fine-tuning dataset, particularly when it in- volves private or confidential data. In these situations, traditional novelty detection methods that assume access the fine-tuning dataset become impractical, necessitating our proposed CGE for discovering novelties without requiring dataset access.
A.5 HYPERPARAMETER SENSITIVITY
Table 6 shows the performance of CGE on discovering novel examples when varying the hyper- parameters. The trend does not change significantly with different hyperparameters. The static version consistently achieves a high detection rate, while the iterative version improves the coverage rate. With a large alpha, the diversity of generated text decreases due to the adaptive plausibility constraint, resulting in relatively lower coverage rate.
17

---

Published as a conference paper at ICLR 2025
A.6 SAMPLES GENERATED BY CGE
Figure 4 shows examples sampled from OpenLLaMa fine-tuned on the non-English dataset. Simi- larly, Figure 5 presents examples sampled from Falcon-RW fine-tuned the source code dataset. We show the first five examples generated by CGE, alongside the examples randomly sampled from fine-tuned models. Compared to random sampling, CGE generates more examples that represent novel domains (non-English languages and source code) in the fine-tuning dataset.
18

---

Published as a conference paper at ICLR 2025
iter texts sampled by
fine-tuned OpanI.lama
|aluAA اكرر Lallal ala ligh pail اولبي السرر التدية مhau ملم s9a3- (417) القيي on (Arabie)
2 enwera" jsfng-) "hysp" "ww.nnwweeennnn pww.rr/wewnaaaaaaorooro//ooaoae.
(Hehrew)
3 A AAAg3 ااار sol, fipldd Vaee ملتر للكري الارز فلتكك --( -Tig 11 (igg))))) (y الرريا مر (Porsian)
4 Rifrwreu (wwwhhaaan)//onns".. Mwwhoccchsshseeeeeeo wwwp://wwwo.mmmmmm.o.ooooo www.p//oooooooooooooooaaoooo (Hebrew)
(Thii)
5
iter lexis randomly sampled from the fine-tuned Open
1 Home x Posts Tagged "Melbourne Cup "tnRacing at Sandown Posted By admin en 2c13,2013 |0 comments ... (English)
2 The last time I saw my dad was on Chrisimas Eve in2017. We had great day and he was happy, and I... (Einglish)
3 di الهرااūعار jooillng a joll y 3V9 y yt 1819 uulip Vas 3) شنب شنر joاl رنك (Persian)
4 This year the theme of the annual meeting of the European Society for Blood and Marrow Transpiantation... (Einllish
Home a News Local Gainesville Gainesville police release names of victims fatal shooting ... (English)
Figure 4: Samples generated by OpenLLaMa fine-tuned on the non-English dataset. Texts generated for the first five iterations are shown. CGE generates more examples that represent novel domains (non-English languages) in the fine-tuning dataset.
iter sampled CGE from the lnne Falcon-RW
#Wbinvbashhninil Copyright 2016 The Kubemeles Authors All rights reserved Knilknnl Licensed under the Apache License 2.0 (the "License"). You may not use this file except in compliancerni with the License. You can oblain copy in the LICENSE file generated by this pipelineW deploy script. Code generated by this pipelinek's deploy (Bash)
# TODOjJhhn:: Move into a helper function so we can reuse everywhere Xnfunction cleanup() (MnWWiill /IKUBERNETES USER:) *nitkll SKKUEERNETES ADMIN-) Anäislepp 5 S(KUBERNETES, USER) "$@" VnVtsleep 2:Wntts 5:initiilll SIKUBERNETES AADDMIIII "5@":Xn) (Bash)
3 slots = [Wmodell", WcategonyW[WWWWn @propertyln def nn return cisVn\nn @propertyln def class(cls, me return meta. class #ninn @propertykn def name v2(cls, meta, name_v2 tvp==>V2ClaassW name_y2 angg==VViiassssMttaayy V2ClassMetaArgayf' (Python)
4. import loggingwnWnffrom chromite.commee wwammon.logging import log_ basic as log_ basicknWnHnclass Logger(object)#innn def (self):Kn self.name 'chrome Wnlkn
enter [self, message):nn superr)).nterr[(K)) def
exit _sei__self, reason):än
(eper))eeit(XMnnnnncaas LoggerFactoo//(bbjet))iin
def _ew
def_new_ ___e__sseeft
(Python)
5 module MongoWn module ConnectionWn class Client < ::Mongo:Cormeetioniii
attr accessor :address, :port, :user, :password, :ssl, :ss _ar__eereteeeeeeeeeeeeeeeeeee_ :ssl beerrrrtrt dir, :ssl _eer_eetttee pert_pathännän
attr accessor :timeuut, :timeout makniin def initializeWn @address niWk (Ruby)
iter lexts randomly sampled from the fine-tuned Falcon-
The following lists events that happened during 1997 in New Zealand. Ynrallouubbnnsshhiikgga and viooregalYnHead State Elizabeth IIinGoverno--eeeee - The Rt Hon. Sir Michael Hardie Roys GNZM, GCMG, 550'nnnGovermmeeninn The 44th New Zealand Parliament continued. The 47th New Zealand Parliament was to have been convened, but it was not ... (nom-code)
2 The Battle Long Island was a battle fought between the Continental Army and the British Anmy during the American Revolutionary War. The battle took place en Long Island, New York, July 17, 1776, during the Siege of June 17, 1776, the Continental Army under the command of General George (mon-ccode
Boston. mnrnBBackroouddyaooo
3 The United States Coast Guard Academy (USCG) is four-year military service academy located in New London, Connecticut. The academy is part the United States Coast Guard and responsible for the training of Coast Guard's officers HithHHistoyyooooTT Coast Guard Academy
established result of the Coast Guard oft 1920, (non-code)
4 (Min' Licensed to the Apache Software Foundation (ASF) under oneän" or more contributor license agreements. See the NOTICE filekn "Wnknpackage org. wwg.apache..aoo apwche.hadoop..ooo. XnXiimprtt wtt.:looOOxxxepioooonoo.eeoo java..o..ng..... wawa.o..nputttream.aa.... lnninntttrrammxiiipoot java ReadStreamReedde java SouputStream/iiimpa java. ... (Java)
5 The Trans Siberian Orchestra symphonic rock band formed 2000 by Orianthi. The hand consists of Orianthi (lead vocals, piano, keybourds, guilars). Michael Shrieve (guitars, backing vocals, keyboards, keyboards, programming, programming, bass. backing vocals) and Michael Shultz (droms, percussion, backing vocals, keyboards, programming, programming)... ... (non-code)
Figure 5: Samples generated by Falcon fine-tuned on the source code dataset. Texts generated for the first five iterations are shown. CGE generates more examples that represent novel domains (source code) in the fine-tuning dataset.
19


Reviews:
[{"review_id": "PwvaTMsJxB", "reviewer": "Reviewer_B4zm", "summary": "This paper considers a novel task of finding what is new in a fine-tuning dataset, given the fact that we have a pre-trained model, but it is unclear what the distribution of data was for the same. In particular, they introduce Contrastive Generative Exploration or CGE, which tries to find novel properties which are unique to the fine-tuning data. This basically leverages the difference in predictions between the pre-trained model and the model fine-tuned on the new dataset, and appropriately generates new sentences. Using this metric itself of new probability scoring, they are able to get much higher true positive rates of detecting novel content. At the same time, they are also able to generate novel sentences. The work discusses two different methods of CGE, one which is iterative and one which is static, and do experiments on non-English languages or toxic text or code data to find novel content. Finally, there is also a section in the paper that talks about how this can work with differential privacy as well, and in scenarios when we do not have access to visualizing the true data.", "questions": "- Can you think of adapting this metric to measure the distance between two distributions?\n- I would like to view CDE generated samples", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Shifting Focus from Data Inspection to Model Behavior: This is a neat transition because direct data access is often restricted due to privacy concerns or the sheer scale of the datasets involved. CGE, therefore, offers a practical and potentially more scalable solution for understanding novelties in fine-tuning data.\n- Contrastive Decoding for Dataset Exploration: This technique, originally developed for controlling text generation properties, is cleverly adapted to distinguish between in-distribution and novel examples by contrasting the predictions of pre-trained and fine-tuned models. \n- Synergy with Differentially Private fine-tuning: In line with the first goal of exploring datasets novelties without direct access to them, the paper does a good job at showing how a model fine-tuned with DP can still allow inspection of properties of the dataset it was trained on by looking at the induced model behaviors.\n- High AUC with metric: The new metric of detecting novel content achieves substantially better performance than most existing metrics.", "weaknesses": "- Missing an honest baseline like n-gram: This paper misses basic baselines such as n-gram distribution change, or top-k nearest neighbour distance in the embedding space. \n   - This could quite naturally be done in the RPJ experiments. \n   - Can also be adapted in the DP-SGD experiments by returning privatized n-gram stats.\n   - You might like the infini-gram paper as a tool to enable such analysis\n- Constraint: The paper writes that one of the constraints for the method to work effectively is that the number of novel examples substantially smaller than the remaining examples in the fine-tuning data. This is a very hard-to-match constraint in my opinion\n- No axis labels on graphs 2,3\n- No samples shown: \n\nOverall, my main concern with this work is that I am unable to position its applicability in the present day LLM use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HUWwEAWUy4", "reviewer": "Reviewer_J5Ry", "summary": "In this paper, the authors introduce to discover through generation, which aims to identify properties in a fine-tuning dataset without directly accessing the fine-tuning data. They propose the Contrastive Generative Exploration (CGE) to fine-tune datasets by generating examples that represent these properties.", "questions": "Will the authors release the code?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experimental setting are soild. They conduct experiments on multiple base LLM and mulitple datasets. The authors report the experimental results on multiple evaluation metrics.\n\n2. To achieve contrastive generative exploration, the methods consists of static approach, iterative approach, and so on. The design of methods seem reasonable.\n\n3. The task (direction) this paper focusing on sounds interesting. Achieve a indirect and effective tine-tuning is curical in LLM and generalized methods can be widely-used in many LLM scenarios.", "weaknesses": "1. The baseline methods are kind of old and weak. It would be better to compare methods proposed in 2023 or 2024, but the baselines used in this submission are proposed before 2021.\n\n2. To compare with DP-based methods, the stardard setting is to compare over different \\epsilon instead of the noise multiplier. \\epsilon indicates the strength of the privacy protection but noise multiplier is not so clear to show the privacy protection performance since the scale of noises varies in different models.\n\n3. The source code should be public accessible.\n\n4. It is better to bold the best results in Table1. From the experimental results shown in Tables 1, the proposed model cannot always outperform the baselines' results. More detailed and insightful explanations are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONEteuZ8NB", "reviewer": "Reviewer_rHvj", "summary": "The authors propose a method based on contrastive learning that can generate examples from classes that are in the fine tuning set that are not in the pre-training set. The authors' experiments involves evaluating GradNorm and various proposed probabilities methods on OpenLLaMA and Falcon-RW.", "questions": "Could you address the weaknesses above?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Identifying what kind of examples in the fine-tuning set that are not present in the pretraining set is interesting.", "weaknesses": "- The writing is poor, and the story and motivation are vague and confusing. The authors keep switching between writing \"novelty exploration\", \"novel characteristics\", and \"novel domains\" which makes it difficult to understand what the actual task is. After much digging, this work is basically about identifying example classes that are in the fine-tuning set and not in the pretraining set. Perhaps the title could be, \"identifying new classes in the finetuning set\".\n\n- It is not clear what the goal of this work is in terms of the benefits of identifying new classes in the fine-tuning set. The introduction mentions that this capability would allow us to remove toxic values, but how would the model know the new classes are \"toxic\"? \n\n- There are so many trivial ways to identify whether the model knows whether a certain type of examples were in the pretraining set. What if you simply compute the uncertainty or the entropy of the model output on the examples, and identify those with the highest entropy? those are most likely examples that the model hasn't seen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBY7zgYqaZ", "reviewer": "Reviewer_carY", "summary": "This paper presents a method called Constrastive Egneration Exploration(CGE) and its interative version, which aims to capture novel characteristics of fine-tuning data. Since fine-tuning data might be confidential sometime, this work are something trying to generate examples that similar to fine-tuning data. More specificlly, by using probability of fine-tuned models minus probability of pre-trained models (called Contrastive Score), it apply contrastive decoding to generate examples. Experiments results show contrasrtive score is able to robustly identify such ood examples, and", "questions": "1. For the iterative version, how to make sure you fine-tune the pre-trained model on generated text can produce a new model with appropreate distribution. i.e. for the first iteration, the distribution of pre-trained model has no problem, and you identify and generate some  Japanese text. However, when you fine-tune on Japanese task, it is possible Japanese text will dominent the distribution and English/Russian will be \"novel character\" and it is possible to generate English again. \n2. The evaluation process for different domains (i.e. Non-English and toxic) is not clear. Which domain is chosen as the ground truth for first iteration? How the accuracy is calculated based on the 100 generated texts? They are not clearly stated.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The question itself seems new. Though using so called contrastive score (difference between two model's probability) to identify some ood texts is not something new, but successfully using it to somehow identify/recover part of fine-tuning datasets is intertesing.\n2. It might be interesting to see whether this can be used as an attack method to recover some confidential fine-tuning data (given known pretrained model and fine-tune model.", "weaknesses": "1. The application of this question seems limited, because for real-world cases, i.e. confidential fine-tuning datasets, we might not know which pre-train model they are using. Also we may not be able to access the probability of fine-tuned model.\n2. Experiments are limited and not so solid. Consider authors are using only two models on two different settings. (1) it is better to do controlled experiments. When you change to a new experimental setting, you can keep the model and other things unchanged. (2) The number of ood-domain is limited.\n3. Writing about evaluation on novel examples is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)