Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction

## Abstract

Abstract Interpreting data is central to modern research. Large language models (LLMs) show promise in providing such natural language interpretations of data, yet simple feature extraction methods such as prompting often fail to produce accurate and versatile descriptions for diverse datasets and lack control over granularity and scale. To address these limitations, we propose a domain-agnostic method for dataset featurization that provides precise control over the number of features extracted while maintaining compact and descriptive representations comparable to human expert labeling. Our method optimizes the selection of informative binary features by evaluating the ability of an LLM to reconstruct the original data using those features. We demonstrate its effectiveness in dataset modeling tasks and through two case studies: (1) Constructing a feature representation of jailbreak tactics that compactly captures both the effectiveness and diversity of a larger set of human-crafted attacks; and (2) automating the discovery of features that align with human preferences, achieving accuracy and robustness comparable to expert-crafted features. Moreover, we show that the pipeline scales effectively, improving as additional features are sampled, making it suitable for large and diverse datasets. 1 1 1 Our code and datasets can be found at https://github.com/MichalBravansky/dataset-featurization

## 1 Introduction

Extracting meaningful insights from large data repositories is a cornerstone of modern research, spanning disciplines such as social sciences (Lazer et al., 2009; Xu et al., 2024), medical sciences (Hulsen et al., 2019), and economics (Varian, 2014; Korinek, 2023). Recent advances in large language models (LLMs) (Vaswani, 2017; Radford, 2019; Achiam et al., 2023) have emerged as a promising approach to this challenge, enabling researchers to process datasets and generate natural language descriptions that summarize and analyze underlying information (Singh et al., 2024a).

However, when working with massive heterogeneous datasets like user queries (Zhao et al., 2024; Zheng et al., 2023b), fundamental challenges emerge: the data cannot be characterized by a single description and requires multiple overlapping features to capture different aspects of the information. This raises two questions: (1) How can we identify which features are important enough to capture without relying on human supervision? (2) How can we ensure our system generalizes to unforeseen datapoints without extensive prompt engineering and design iteration?

Figure: Figure 1: The proposed pipeline is able to extract semantically and structurally rich binary features from unsupervised data. Initially, an LLM analyzes each input text to generate candidate features. These candidates undergo clustering-based filtration to remove duplicates. The system then measures how well each feature enables reconstruction of the original data samples when provided as context to an LLM for texts containing that feature, measuring reconstruction quality via perplexity (PPL) and iteratively concatenating features in sequence to create a set that best captures the dataset’s properties.
Refer to caption: /html/2502.17541/assets/x1.png

In this paper, we formalize the problem through the concept of features, defined as binary predicates $\phi:X\to{0,1}$ (e.g., ”text $x$ implies a misunderstanding.”), which map each text to a binary representation based on an LLM’s evaluation. Instead of directly relying on LLMs to generate sets of these features, we leverage their language modeling capabilities to optimize feature sets that, when described in natural language and provided in-context to the LLM, enable accurate dataset reconstruction. To achieve this, we introduce an unsupervised automatic pipeline that generates a comprehensive set of potential features and uses a reconstruction-driven process to extract a subset that best captures the dataset’s structure and semantics.

To evaluate our methodology, we construct synthetic extraction datasets from public data sources (Amazon Reviews, NYT, and DBPedia) (Hou et al., 2024; Sandhaus, 2008; Auer et al., 2007) in Section [5](#S5). Using associated class labels as ground truth, we demonstrate that our method outperforms prompting baselines by producing more accurate features both semantically and structurally. We further showcase our pipeline’s versatility through two case studies (Section [6](#S6)). First, we extract features representing LLM jailbreak tactics, creating compact representations that capture the efficiency and diversity of a larger set of human-crafted tactics (Jiang et al., 2024b). Second, in preference modeling, our pipeline identifies distinctive features within prompt response pairs to create preference models that match or exceed the performance of those based on expert-crafted features (Go et al., 2024).

Our framework offers a novel approach to extracting meaningful information from diverse datasets, scales with feature sampling, and applies across domains. By demonstrating the pipeline’s scalability and adaptability, we provide a foundation for creating efficient, domain-agnostic tools for uncovering patterns in complex data without supervision.

## 2 Related Works

### 2.1 Unsupervised Feature Extraction

Traditional approaches to interpretable data analysis have primarily relied on unsupervised techniques such as clustering, which are typically paired with natural language descriptions generated either through phrase extraction (Carmel et al., 2009; Treeratpituk & Callan, 2006; Zhang et al., 2018) or LLM-based methods (Sorensen & Others, 2024; Lam et al., 2024; Singh et al., 2023a). However, these methods face fundamental limitations due to their sensitivity to the number of selected clusters and their inability to accurately approximate complex cluster contents (Chang et al., 2009).

More recently, LLM-based methods have re-imagined feature discovery as a search over natural language hypotheses (Qiu et al., 2023), employing diverse strategies including de-duplication (Pham et al., 2023), optimization for minimal cluster overlap (Wang et al., 2023; Zhong et al., 2024), and human-guided feature selection (Viswanathan et al., 2023). While these advances have improved clustering-based approaches, they remain constrained by hyperparameter dependence and rigid cluster assignments. Our method overcomes these limitations through controlled feature sampling that enables simultaneous modeling of both broad patterns and fine-grained properties, without constraining the number of features that can be assigned to each text.

### 2.2 Supervised Feature Extraction

Supervised approaches to feature discovery have emerged as an alternative to unsupervised methods. Zhong et al. (2022) formulates this as a distribution comparison problem to identify distinguishing characteristics, an approach later extended beyond text (Dunlap et al., 2024) and adapted to accommodate user-specified exploration goals (Zhong et al., 2023). Different forms of supervision have also been explored: Findeis et al. (2024) leverages correlation analysis to identify features aligned with human preferences, while Benara et al. (2024) employs ridge regression for feature selection in medical prediction tasks.

A parallel line of work explores Concept Bottleneck Models, which achieve interpretability by learning models over interpretable intermediate features (Koh et al., 2020). Recent advances have focused on automating the discovery of these interpretable features (Yang et al., 2023; Ludan et al., 2023; Schrodi et al., 2024). However, these approaches require either labeled data or reference distributions, which our method does not. Furthermore, while these methods optimize for accuracy through bottleneck representations, they may not capture the semantic richness that our natural language featurization pipeline provides.

### 2.3 Dataset-Level Feature Extraction

At the dataset level, current approaches typically extract features by prompting LLMs with data samples to generate dataset descriptions (Singh et al., 2024a). While these descriptions can be refined through self-improvement loops (Pan et al., 2023; Gero et al., 2023), expert feedback (Templeton et al., 2024), or reconstruction-based optimization (Singh et al., 2024b), they remain limited to dataset-level insights without capturing properties of individual samples. Although prior work has explored more structured representations like binary tree decomposition with natural language splits (Singh et al., 2023b), our method uniquely generates semantically grounded features that enable both granular analysis and systematic comparison across the entire dataset.

## 3 Background Formalism

Before diving into our method, we define terms later used in the [Section 4](#S4).

Binary Predicate as a Feature.
We define a *feature* $\phi$ as a binary predicate $\llbracket\phi\rrbracket:X\to\{0,1\}$, determined by a LLM serving as the valuation function. A *feature set* is a collection $\boldsymbol{\phi}=(\phi_{1},\ldots,\phi_{K})$ of $K$ such predicates.

Dataset Modeling.
Let $\mathcal{D}={x^{(1)},\ldots,x^{(N)}}$ be a dataset of $N$ texts that we treat as independent from each other. We evaluate these texts using a language model conditioned on their features, with the goal of finding a feature set $\boldsymbol{\phi}$ that minimizes the perplexity $\text{PPL}(\mathcal{D}\mid\boldsymbol{\phi})$. For each text $x^{(n)}$, we compute its features $\boldsymbol{\phi}(x^{(n)})$ and calculate the mean of per-text perplexities. We made this slight modification to standard perplexity to give each text same importance, preventing longer texts from dominating the metric:

$$ $\mathrm{PPL}(\mathcal{D}\mid\boldsymbol{\phi})\;=\;\frac{1}{N}\sum_{n=1}^{N}\mathrm{PPL}(x^{(n)}\mid\boldsymbol{\phi(x)}).$ $$

## 4 Method

Our goal is to optimize a binary feature set $\boldsymbol{\phi}$ that minimizes $PPL(\mathcal{D}\mid\boldsymbol{\phi})$ for dataset $\mathcal{D}$, effectively identifying natural language features that enable LLMs to best reproduce the original text $x$. We assume that state-of-the-art models have acquired sufficient capability to perform this type of modeling, a claim that we later validate in [Section 5](#S5). Since gradient-based optimization isn’t feasible in the binary feature space, we implement a multi-stage pipeline that generates candidate features, removes duplicates through clustering, and iteratively selects the most effective features for dataset modeling. Detailed pseudocode for the whole pipeline is in [Algorithm 1](#alg1).

Generation Stage. In the first phase, we leverage an LLM to generate discriminative features by comparing each text in $D$ against $C$ randomly sampled texts from the dataset. We employ GPT-4o (Hurst et al., 2024) to generate $K$ unique features per comparison, with $C$=5 and $K$=5 used across all subsequent experiments based on empirical tuning to optimize both diversity and generality. The complete feature generation prompt is provided in Appendix [F.1](#A6.SS1).

Clustering and Evaluation Stage. Since assigning truth values to all features and evaluating $PPL(\mathcal{D}\mid\boldsymbol{\phi})$ for many choices of $\boldsymbol{\phi}$ is computationally intensive, we adopt strategies from Findeis et al. (2024). We vectorize features using OpenAI embeddings (^2^22https://platform.openai.com/docs/guides/embeddings) and apply KMeans clustering, where we use a number of clusters equal to the number of texts in the dataset across all experiments, observing that this maintains feature diversity while eliminating duplicates. From each cluster, we randomly select one representative feature. For efficiency, we use GPT-4o to assign truth values to 10 features simultaneously per text $x$ (prompt in Appendix [F.2](#A6.SS2)), and retain only features present in at least 5% of samples to focus on common patterns, though this threshold is adjustable.

Featurization Stage. In the final stage, we iteratively construct the feature set by selecting features that minimize the dataset’s perplexity given the set of features. At each iteration, we identify the feature $F$ that provides the lowest average perplexity gain when added to the current feature set $\boldsymbol{\phi}$:

$$ $PPL(\mathcal{D}\mid\boldsymbol{\phi}\cup\{F\})<PPL(\mathcal{D}\mid\boldsymbol{\phi}\cup\{F^{\prime}\}),$ $$

for all other candidate features $F^{\prime}$. The selected feature $F$ is then added to $\boldsymbol{\phi}$.

We define the tokenization process for feature representation as follows: for a given feature space, we construct a natural language representation of the features by concatenating the names of all features $F_{i}$ that evaluate to true for a given text, separated by newline characters ($``\backslash n"$):

$$ $\text{Tokenized Representation}=F_{1}+``\backslash n"+F_{2}+``\backslash n"+\ldots,$ $$

By doing so, we optimize the efficiency of the pipeline further by caching the log-probability for texts where a feature is false, avoiding redundant computations. To provide context to the model, we include a static prompt that describes the task. Variations of these static prompts for each experiment are described in Appendix [F.3](#A6.SS3). The iterative process concludes when either we reach $N$ features or no additional feature reduces $PPL(\mathcal{D}\mid\boldsymbol{\phi})$.

Throughout the process, no human supervision is required, as we rely on the general instruction-following abilities of LLMs to generate the candidate features, and the language modeling objective to guide feature selection based on reconstruction.

## 5 Dataset Modeling Experiments

To validate our method, we test it on three datasets with known features to assess its feature reconstruction capability, before demonstrating practical downstream applications in [Section 6](#S6). Our pipeline uses GPT-4o (Hurst et al., 2024) for feature proposal and evalidation, and Llama 3.1 8B Instruct (Dubey et al., 2024) for feature selection.

Figure: Figure 2: Our pipeline outperforms LLM prompting in feature extraction across all metrics and datasets, showing higher class coverage (average correlation between classes and closest features), reconstruction accuracy (linear model accuracy on classes), and semantic preservation (number of semantically similar features as judged by an LLM). Our method achieves faster convergence with fewer features, with the last reconstruction-based stage proving crucial for surpassing the prompting baseline.
Refer to caption: /html/2502.17541/assets/x2.png

### 5.1 Datasets

We utilize three publicly available labeled datasets:

DBPEDIA. A semantic dataset derived from Wikipedia using RDF triples (Auer et al., 2007). We use a pre-processed subset with hierarchical class labels(^3^33[https://huggingface.co/datasets/DeveloperOats/DBPedia_Classes](https://huggingface.co/datasets/DeveloperOats/DBPedia_Classes)), focusing on level 2 categories for reconstruction.

NYT. The New York Times Annotated Corpus contains 1.8 million articles (1987-2007) with metadata (Sandhaus, 2008). We use the manually reviewed tags from the NYT taxonomy classifier, specifically focusing on articles under ”Features” and ”News” categories.

AMAZON. This dataset comprises half a million customer reviews with item metadata (Hou et al., 2024). We focus on identifying high-level item categories (e.g., Books, Fashion, Beauty), excluding reviews labeled ”Unknown.”

First, we select entries between 100-10,000 characters to eliminate outliers and ensure sufficient content for analysis. For each dataset, we then construct three independent balanced subsets, each containing 5 classes with 100 samples per class. This methodology provides enough complex, diverse data to prevent class memorization while maintaining sufficient scale for method comparison. We report results averaged across these three trials per dataset.

### 5.2 Metrics

We evaluate methods using three complementary metrics, each designed to measure the extent to which the discovered features capture the original underlying structure of the dataset:

Class Coverage. We compute the Pearson correlation between each dataset class’s presence and each selected feature, then take the maximum correlation per class and average across classes. This directly measures how well our top features align with and preserve the original class distinctions.

Reconstruction Accuracy. Using the features as inputs, we train a logistic regression classifier to predict the original class labels, reporting 5-fold cross-validation accuracy. Higher accuracy indicates the features contain sufficient information to distinguish between classes.

Semantic Preservation. We measure whether the extracted features maintain the semantic meaning of original categories. Using Claude Haiku 3.5 (Anthropic, 2024), we prompt the model to assess if each class’s core concept appears in at least one feature’s description, recording the number of classes with matches. The detailed prompting approach is provided in Appendix [B.1](#A2.SS1).

### 5.3 Experimental Setup

Baseline. We use GPT-4o to generate 50 features with temperature and top_p set to 1, using a random sample of 100 instances due to context limitations. Through prompt engineering, we found optimal results by directly instructing the LLM to generate topic-related features. The prompt is detailed in Appendix [G.3](#A7.SS3), with additional results for non-topic-specific prompting in [Section B.3](#A2.SS3).

Our method. To assess our pipeline’s components, we evaluate three stages, with detailed parameters in the [Appendix E](#A5):

- •
Generation: We generate proposed features using the generation-stage prompt with GPT-4o and randomly sample up to 50 features, representing the initial pipeline output.
- •
Clustering: Features are clustered to remove redundancies, assigned truth values, and filtered to those occurring in at least 5% of texts. Up to 50 features are randomly sampled from different clusters.
- •
Full pipeline: We apply the feature selection procedure to the clustered features through Llama 3.1 8B Instruct (prompt found in [Section F.3](#A6.SS3.SSSx1)), iteratively selecting up to 50 features that maximize dataset reconstruction, representing the complete pipeline.

### 5.4 Results

We present the complete results in [Figure 2](#S5.F2), with detailed numerical results in [Appendix G](#A7).

Insight 1: Our pipeline generates higher-quality structural and semantic features. Across datasets of varying complexity, our approach consistently outperforms standard LLM prompting in feature generation.

Insight 2: Our pipeline achieves faster feature space convergence. Our method reaches 95% peak performance with half the features needed by the baseline for both Class Coverage and Reconstruction Accuracy, as detailed in [Section B.2](#A2.SS2).

Insight 3: Each pipeline stage enhances feature quality. Each stage of the pipeline improves performance metrics, with the reconstruction-based feature selection stage being crucial for surpassing the baseline.

Insight 4: Performance scales with feature count. All metrics improve with more features, which we explore further in [Section 6](#S6).

Overall, we see that our method outperforms baselines based on prompting on the raw task of datasets modeling. We next examine two case studies where dataset modeling has practical downstream applications, evaluating our method against both comparable baselines and expert-crafted features.

## 6 Application Case Studies

To illustrate the versatility of our proposed framework, we present two case studies: extracting compact representations of jailbreaks and identifying features for preference modeling. These applications demonstrate our method’s broad applicability, and we discuss additional potential use cases in [Section 7](#S7).

### 6.1 Extracting Compact Attacks from Jailbreaks

Automated red-teaming (ART) generates inputs designed to trigger harmful behavior in language models (Perez et al., 2022). To build robust safeguards, diverse attack examples are essential for resisting a wide range of harmful inputs (Wei et al., 2024). However, creating a small yet diverse and effective attack set remains challenging (Samvelyan et al., 2024; Hughes et al., 2024). In this case study, we show how dataset featurization can produce such a compact set.

We apply our method to the WildTeaming framework (Jiang et al., 2024b), a state-of-the-art approach that uses GPT-4 (Achiam et al., 2023) to extract 100k+ tactics from human jailbreaks in LMSYS-CHAT-1M (Zheng et al., 2023a) and WILDCHAT (Zhao et al., 2024). For analysis, we study the WildJailbreak dataset where Mixtral-8×7B (Jiang et al., 2024a) or GPT-4 are given harmful queries and instructed to generate jailbreaks by combining 2–7 tactics sampled from 500 clusters (containing 85k total human jailbreak tactics).

**Table 1: A compact 20-feature set matches WildTeaming’s 500 attack clusters across key metrics. Performance is measured through Attack Success Rate (ASR), queries required per successful attack (Query), unique attack success rate (ASR${}^{\times 5}_{30}$), queries needed for unique attacks (Query${}^{\times 5}_{30}$), similarity between first 5 sucessful attacks (Sim${}^{@5}_{30}$), and similarity across all attacks (Sim${}_{\text{all}}$). Additionally, our $Enhanced$ version, which featurizes only Llama non-refusals, demonstrates further improvements over WildTeaming.**
| Model | Method | Standard | Diversity |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ASR $\uparrow$ | Query $\downarrow$ | ASR${}^{\times 5}_{30}$ $\uparrow$ | Query${}^{\times 5}_{30}$ $\downarrow$ | Sim${}^{@5}_{30}$ $\downarrow$ | Sim${}_{\text{all}}$ $\downarrow$ |
| Gemini 1.5 Flash | WildTeaming | 81.8 | 6.02 | 71.2 | 13.28 | 0.702 | 0.542 |
| 20 Features | 83.6 | 9.24 | 60.6 | 13.65 | 0.705 | 0.532 |  |
| GPT-4o | WildTeaming | 69.2 | 10.08 | 45.9 | 14.94 | 0.710 | 0.522 |
| 20 Features | 71.1 | 9.19 | 49.4 | 16.02 | 0.709 | 0.530 |  |
| Llama 3.1 8B Instruct | WildTeaming | 44.7 | 11.72 | 18.2 | 17.84 | 0.740 | 0.534 |
| 20 Features | 44.7 | 11.96 | 20.9 | 18.64 | 0.724 | 0.514 |  |
| 20 Features $(Enhanced)$ | 47.8 | 9.53 | 23.3 | 17.49 | 0.740 | 0.524 |  |

By applying our pipeline to a subset of this dataset, we reduce the feature space by a factor of 25 while maintaining the diversity and effectiveness of the original human-crafted tactics. This compact representation can enable deeper analysis of adversarial strategies, controlled synthetic jailbreak generation, and the identification of tactics most effective against specific models.

#### Featurization

To process the WildJailbreak dataset, we use Mistral 7B Instruct (Jiang et al., 2023) instead of LLama 3.1 8B Instruct due to the latter’s occasional refusals to generate jailbreaks. We sample 1,000 random jailbreak prompts from the dataset (originally generated using Mixtral 8×7B and GPT-4), a size chosen to demonstrate effectiveness in a lower data regime while meeting compute constraints. While we considered regenerating the dataset with Mistral 7B for consistency, initial experiments showed that Mistral’s modeling ability decreases when processing outputs from the same model. Using the prompt detailed in [Section F.3](#A6.SS3.SSSx2), we extract the 50 most relevant features for these jailbreaks, which are presented in [Section H.1](#A8.SS1).

#### Evaluation Protocol

We adopt a similar evaluation protocol WildTeaming and employ the HarmBench evaluation setup (Mazeika et al., 2024), using 159 vanilla harmful queries from the standard HarmBench test set.

WildTeaming. We randomly sample 4 tactics, each from a different cluster among the top 500 clusters. Using the original WildTeaming prompt (see [Section H.3](#A8.SS3)), we generate jailbreaks with Mistral 7b at temperature 1 and top_p of 0.9, without additional filtering.

Full Pipeline. For comparability, we sample four features from our top 20 extracted features and provide them to Mistral 7B using a slightly modified WildTeaming prompt (see [Section H.4](#A8.SS4)). We maintain identical generation parameters (temperature 1, top_p 0.9) and use no pruning.

Evaluation Metrics. Following WildTeaming, we measure (1) Effectiveness via Attack Success Rate (ASR) on vanilla harmful queries (evaluated by a Llama2-13B fine-tuned HARMBENCH classifier on whether an adversarial response sufficiently addresses each harmful query) and the number of queries (Query) to achieve success; and (2) Diversity via ASR${}^{\times n}_{c}$ (average success rate for $n$ unique attacks with $<$0.75 sentence similarity over $c$ trials), Query${}^{\times n}_{c}$ (average queries for $n$ unique attacks), Sim${}^{@n}_{c}$ (similarity of first $n$ successful attacks), and Sim${}_{\text{all}}$ (overall successful attack similarity). Details can be found in [Section C.1](#A3.SS1).

#### Results

Table [1](#S6.T1) presents a comparison between our approach and the WildTeaming baseline across Gemini 1.5 Flash (Team et al., 2024), GPT-4, and Llama 3.1 8B Instruct. Our analysis focuses on 20 features, as performance plateaus beyond this point (extended results in [Section C.2](#A3.SS2)).

Our method achieves comparable performance across all six metrics. On standard evaluations, we consistently match or exceed WildTeaming’s ASR performance across all models. For both GPT-4o and Llama 3.1 8B Instruct, we achieve higher ASR${}^{\times 5}_{30}$ with comparable query counts, though WildTeaming demonstrates an advantage in generating diverse jailbreaks on Gemini 1.5 Flash. We hypothesize that this gap arises from our filtering threshold, which retains only features present in at least 5% of jailbreaks. While this decision creates a more effective and interpretable feature set, it comes at the cost of not achieving higher diversity on Gemini, which responds well to infrequent attack patterns.

Feature Refinement. To further enhance our results, we refine our features through dataset filtration. From 5000 adversarial, harmful prompts in WildJailbreak, we identify 536 prompts that elicit non-refusal responses from Llama 3.1 8B Instruct, as judged by WildGuard (Han et al., 2024), to derive a new set of 20 features ([Section H.2](#A8.SS2)) using the same featurization setup. This refined set outperforms the original 20 features across almost all metrics, demonstrating how initial dataset filtering can improve our pipeline’s performance.

Interpretability and Insights. Our defined feature set offers improved interpretability through a more compact representation compared to WildTeaming’s 500 distinct tactic clusters. Manual examination reveals that our original 50 features ([Section H.1](#A8.SS1)) highlight broad narratives, scenario settings, and prompt structures common in adversarial attacks, while the 20 features from the Llama non-refusal subset ([Section H.2](#A8.SS2)) emphasize engagement with safety norms and positive language use.

Overall, our method successfully compresses WildTeaming’s 500 clusters into just 20 features (a 25x reduction in feature space) while maintaining or improving performance across most metrics, particularly for robust models like GPT-4 and Llama 3.1 8B Instruct. This compact feature set provides better insights into adversarial strategies through a more manageable representation, with our results showing additional improvements are possible through targeted dataset filtering, suggesting that the method’s effectiveness can be further enhanced by refining the initial featurization dataset.

Figure: Figure 3: Our PMs demonstrate competitive performance with expert-crafted features in both generalization and robustness. Left: Generalization performance versus reference PM (smaller gap is better) shows comparable results across datasets. Right: Robustness analysis between PM A and PM B (smaller difference is better) reveals superior performance on SHP and comparable results on HH-RLHF. All confidence intervals computed using 500 bootstrap iterations with replacement over prompts.
Refer to caption: /html/2502.17541/assets/x3.png

### 6.2 Compositional Preference Modeling

The growing capabilities of LLMs necessitate their alignment with human preferences, primarily through reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022). In RLHF, a preference model (PM) learns from human-ranked responses to score LLM outputs. However, training models to directly predict preference scores creates black-box systems prone to reward hacking and unintended biases (Gao et al., 2022).

Recent approaches decompose rewards into interpretable features such as readability and correctness (Dorka, 2024; Wang et al., 2024). In this case study, we explore our pipeline’s ability to identify such features in unsupervised manner, avoiding biases that can emerge when features are selected based on human reward signals. We compare our method to compositional preference models (CPMs) (Go et al., 2024), which validate responses against predefined features before training a linear regression to predict preferences. We selected their framework for its comprehensive evaluation and extensive feature set. Our approach eliminates manual feature engineering while achieving comparable performance with the same feature count and superior results with larger, automatically generated feature sets.

#### Featurization and Training

Following the original study (Go et al., 2024), we analyze two datasets, HH-RLHF (Bai et al., 2022) and SHP (Ethayarajh et al., 2022), each containing prompt-response pairs with human helpfulness rankings. SHP uses human-written responses from natural interactions, while HH-RLHF uses machine-generated responses ranked for alignment. We sample 1,000 preferred responses from each dataset and create separate feature spaces of 50 features for SHP and (LABEL:table:preferences-shp-features) and HH-RLHF (LABEL:table:preferences-hh-features) using parameters from [Appendix E](#A5), with Llama 3.1 8B Instruct handling the final stage via the prompt in [Section F.3](#A6.SS3.SSSx3).

To train our PM, we adapted Go et al. (2024)’s approach, using GPT-4 to rate responses on a 1-10 scale for each feature (prompts in [Section I.1](#A9.SS1)). We enhanced our pipeline by generating minimum and maximum attributes for each feature using prompt in [Section I.2](#A9.SS2). After identifying that some features were overly generic, we excluded features with standard deviation below 1 from training of the PMs.

#### Evaluations and Results

We compare our features against the 14 expert-crafted features from Go et al. (2024) ([Section I.3](#A9.SS3)).
We conduct four evaluations: PM accuracy, PM robustness, PM generalizability, and pair-wise win-rate.

Accuracy. Table [2](#S6.T2) compares PM accuracies across different feature sets. Our method scales with feature count, showing how performance improves with automatically generated features. On SHP, we match the baseline with equivalent features and surpass it with more. On HH, we approach but don’t exceed baseline performance, even with 50 features. Unlike the baseline’s expert-crafted features, our extraction process is fully automated and unsupervised.

**Table 2: Our method matches performance of expert-crafted features while being easier to scale. As we increase the number of sampled features, our preference models achieve higher accuracy (Acc) on the test set, matching or exceeding baseline performance while demonstrating superior win rates (WR).**
|  | HH-RLHF | SHP |  |  |
| --- | --- | --- | --- | --- |
| Method | Acc | WR | Acc | WR |
| Baseline (14 features) | 68.9% | 81% | 65.9% | 56.2% |
| Our Features (Top 5) | 63.6% | 83% | 61.9% | 65.1% |
| Our Features (Top 14) | 65.5% | 82% | 65.8% | 68.5% |
| Our Features (Top 50) | 68.1% | 80% | 67.9% | 55% |

Robustness. To evaluate PM robustness (consistent response rating across training data subsets) we employ the Best of $N$ (BoN) sampling method (Gao et al., 2022; Go et al., 2024). Two PMs ($PM_{A}$, $PM_{B}$) are trained on equal sized disjoint subsets. Using Flan T5 Large (Chung et al., 2022), we generate $N$ responses per prompt and select response $x$ maximizing $PM_{A}(x)$. We then compare $PM_{A}(x)$ and $PM_{B}(x)$ scores, expecting $PM_{A}(x)>PM_{B}(x)$ with the gap widening as $N$ increases, indicating reduced robustness due to overfitting.

Figure [3](#S6.F3) compares BoN results between baseline expert-crafted features and varying numbers of unsupervised features. Our method demonstrates superior robustness across feature counts on SHP, while achieving comparable robustness on HH-RLHF. Notably, for SHP, the most robust PM uses only 5 features, likely due to high variance among preference pairs that enables models with more features to overfit to noise and capture spurious correlations.

Generalizability. To assess how well our model generalizes to other datasets, given that featurization and PM training occur on a single preference dataset, we follow the evaluation approach proposed by Go et al. (2024). They suggest using a well-established PM trained on diverse datasets, which should exhibit better generalization than single-dataset models. We use fine-tuned DeBERTa models as our references for HH-RLHF(^4^44https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2) and SHP (Sileo, 2023) and plot their BoN scores against PMs trained with our features and the baseline features. According to Go et al. (2024), lower divergence from these comprehensively trained reference models indicates better generalization to unseen data.

Figure [3](#S6.F3) shows that our approach matches the baseline’s performance on both datasets. The HH-RLHF plot demonstrates that similar to accuracy, we can choose to generate more features and easily improve the performance of our PM. For SHP, the large gap between the reference model and the other PMs suggests that it may be very difficult to generalize effectively from this particular dataset.

Pair-wise Win-Rate. Similarly to Go et al. (2024), we evaluate PM quality using pairwise win rates. We generate $N$ responses to a prompt using Flan T5 Large and select the best response according to a trained PM. Then, we randomly select a second response from the remaining ones. To compare them, we use GPT-4 Turbo with an AlpacaEval (Li et al., 2023) prompt, evaluating both orderings and selecting the response with higher log probabilities. A strong PM should consistently select responses that are preferred over randomly chosen ones. Table [2](#S6.T2) shows that our PM-selected responses match the baseline performance. However, unlike with accuracy, we did not observe clear improvements when additional features were included.

Overall, our unsupervised features matched the performance of state-of-the-art hand-crafted preference models (Go et al., 2024) across all metrics, with superior accuracy and generalizability. Our approach enables easy generation of additional features to enhance performance, while maintaining interpretability as further explored in [Section D.1](#A4.SS1).

## 7 Discussion, Limitations & Conclusion

We have introduced $dataset$ $featurization$, a novel approach for extracting overlapping features from unsupervised text data that can effectively capture both broad patterns and fine-grained properties. Our multi-stage pipeline proposes potential features through individual text analysis, filters and deduplicates them via clustering, and iteratively selects features that help an LLM minimize perplexity over data samples, thereby improving its reconstruction ability. Beyond outperforming prompting in dataset modeling tasks, we demonstrated our method’s versatility by compressing jailbreak attacks while maintaining their effectiveness and diversity, and matching expert-crafted features in preference modeling while offering improved scalability.

Limitations. Our method is restricted to binary features and relies on positive feature instances during optimization, aligning with prior work (Dunlap et al., 2024; Zhong et al., 2024; Findeis et al., 2024) but limiting applicability to tasks requiring numeric attributes and hierarchical relationships. Additionally, while we leverage LLMs for in-context reasoning, fine-tuning them specifically for feature-based modeling could enhance feature selection.

Future Directions. Our pipeline shows promise across scientific research, from social science to medical analysis (Tamkin et al., 2024; Singh et al., 2025; Wolf et al., 2018). For LLM safety, applications extend beyond jailbreaks to influence functions and steering vectors (Grosse et al., 2023; Subramani et al., 2022), while advancing our understanding of human preferences (Findeis et al., 2024; Li et al., 2024).

## 8 Acknowledgements

This research was supported by the ERA Fellowship. We thank the ERA Fellowship for both financial and intellectual support, particularly Rudolf Laine for his valuable guidance. We also thank Filip Trhlik and Laura Ruis for their helpful comments on early versions of this paper.

## 9 Impact Statement

We envision $dataset$ $featurization$ as a valuable tool for developing interpretable analytics and visualizations across diverse research domains, including medicine, social sciences, and economics. Our case studies demonstrate its utility in two such areas: enhancing defensive techniques against adversarial attacks and developing more robust preference models. Improved capabilities to understand and detect large-scale attacks contribute to AI safety research, while advances in preference modeling help further our understanding of human values and their computational representation.

However, like many analytical tools, this technology has potential dual-use implications. The method could be applied to tasks such as de-anonymization (Narayanan & Shmatikov, 2008), amplification of existing biases (Barocas & Selbst, 2016), or enhancing the spread of misinformation (Tufekci, 2014). These capabilities underscore the importance of developing appropriate governance frameworks and ethical guidelines for the deployment of such analytical tools.

Our specific implementation presents several important considerations. While features may be interpretable within the LLM’s context, they can become ambiguous or misleading when presented without proper human context, emphasizing that this tool should complement rather than replace human analysis. The stochastic nature of our method introduces potential convergence to local optima, possibly necessitating further validation through cross-validation across multiple runs, comparison with domain expert assessments, or evaluation across different initialization parameters to ensure robust analysis.

## References

- Achiam et al. (2023)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Anthropic (2024)
Anthropic.
Claude 3.5 sonnet, 2024.
URL [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet).
- Auer et al. (2007)
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., and Ives, Z.
Dbpedia: A nucleus for a web of open data.
In *international semantic web conference*, pp.  722–735. Springer, 2007.
- Bai et al. (2022)
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J.
Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
URL [https://arxiv.org/abs/2204.05862](https://arxiv.org/abs/2204.05862).
- Barocas & Selbst (2016)
Barocas, S. and Selbst, A. D.
Big data’s disparate impact.
*Calif. L. Rev.*, 104:671, 2016.
- Benara et al. (2024)
Benara, V., Singh, C., Morris, J. X., Antonello, R., Stoica, I., Huth, A. G., and Gao, J.
Crafting interpretable embeddings by asking llms questions, 2024.
- Carmel et al. (2009)
Carmel, D., Roitman, H., and Zwerdling, N.
Enhancing cluster labeling using wikipedia.
*SIGIR*, 2009.
- Chang et al. (2009)
Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J., and Blei, D.
Reading tea leaves: How humans interpret topic models.
*Advances in neural information processing systems*, 22, 2009.
- Christiano et al. (2017)
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D.
Deep reinforcement learning from human preferences.
*Advances in neural information processing systems*, 30, 2017.
- Chung et al. (2022)
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J.
Scaling instruction-finetuned language models, 2022.
URL [https://arxiv.org/abs/2210.11416](https://arxiv.org/abs/2210.11416).
- Dorka (2024)
Dorka, N.
Quantile regression for distributional reward models in rlhf, 2024.
URL [https://arxiv.org/abs/2409.10164](https://arxiv.org/abs/2409.10164).
- Dubey et al. (2024)
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Dunlap et al. (2024)
Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., Gonzalez, J. E., and Yeung-Levy, S.
Describing differences in image sets with natural language.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  24199–24208, 2024.
- Ethayarajh et al. (2022)
Ethayarajh, K., Choi, Y., and Swayamdipta, S.
Understanding dataset difficulty with $\mathcal{V}$-usable information.
In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), *Proceedings of the 39th International Conference on Machine Learning*, volume 162 of *Proceedings of Machine Learning Research*, pp.  5988–6008. PMLR, 17–23 Jul 2022.
URL [https://proceedings.mlr.press/v162/ethayarajh22a.html](https://proceedings.mlr.press/v162/ethayarajh22a.html).
- Findeis et al. (2024)
Findeis, A., Kaufmann, T., Hüllermeier, E., Albanie, S., and Mullins, R.
Inverse constitutional ai: Compressing preferences into principles.
*arXiv preprint arXiv:2406.06560*, 2024.
- Gao et al. (2022)
Gao, L., Schulman, J., and Hilton, J.
Scaling laws for reward model overoptimization, 2022.
URL [https://arxiv.org/abs/2210.10760](https://arxiv.org/abs/2210.10760).
- Gero et al. (2023)
Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., and Poon, H.
Self-verification improves few-shot clinical information extraction.
*arXiv preprint arXiv:2306.00024*, 2023.
- Go et al. (2024)
Go, D., Korbak, T., Kruszewski, G., Rozen, J., and Dymetman, M.
Compositional preference models for aligning lms, 2024.
URL [https://arxiv.org/abs/2310.13011](https://arxiv.org/abs/2310.13011).
- Grosse et al. (2023)
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al.
Studying large language model generalization with influence functions.
*arXiv preprint arXiv:2308.03296*, 2023.
- Han et al. (2024)
Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert, N., Choi, Y., and Dziri, N.
Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms.
*arXiv preprint arXiv:2406.18495*, 2024.
- Hou et al. (2024)
Hou, Y., Li, J., He, Z., Yan, A., Chen, X., and McAuley, J.
Bridging language and items for retrieval and recommendation.
*arXiv preprint arXiv:2403.03952*, 2024.
- Hughes et al. (2024)
Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F., Koyejo, S., Sleight, H., Jones, E., Perez, E., and Sharma, M.
Best-of-n jailbreaking, 2024.
URL [https://arxiv.org/abs/2412.03556](https://arxiv.org/abs/2412.03556).
- Hulsen et al. (2019)
Hulsen, T., Jamuar, S. S., Moody, A. R., Karnes, J. H., Varga, O., Hedensted, S., Spreafico, R., Hafler, D. A., and McKinney, E. F.
From big data to precision medicine.
*Frontiers in medicine*, 6:34, 2019.
- Hurst et al. (2024)
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jiang et al. (2023)
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.
Mistral 7b.
*arXiv preprint arXiv:2310.06825*, 2023.
- Jiang et al. (2024a)
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al.
Mixtral of experts.
*arXiv preprint arXiv:2401.04088*, 2024a.
- Jiang et al. (2024b)
Jiang, L., Rao, K., Han, S., Ettinger, A., Brahman, F., Kumar, S., Mireshghallah, N., Lu, X., Sap, M., Choi, Y., et al.
Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models.
*arXiv preprint arXiv:2406.18510*, 2024b.
- Koh et al. (2020)
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P.
Concept bottleneck models.
In *International conference on machine learning*, pp.  5338–5348. PMLR, 2020.
- Korinek (2023)
Korinek, A.
Generative ai for economic research: Use cases and implications for economists.
*Journal of Economic Literature*, 61(4):1281–1317, 2023.
- Lam et al. (2024)
Lam, M. S., Teoh, J., Landay, J. A., Heer, J., and Bernstein, M. S.
Concept induction: Analyzing unstructured text with high-level concepts using lloom.
In *Proceedings of the CHI Conference on Human Factors in Computing Systems*, pp.  1–28, 2024.
- Lazer et al. (2009)
Lazer, D., Pentland, A., Adamic, L., Aral, S., Barabási, A.-L., Brewer, D., Christakis, N., Contractor, N., Fowler, J., Gutmann, M., et al.
Computational social science.
*Science*, 323(5915):721–723, 2009.
- Li et al. (2024)
Li, J., Zhou, F., Sun, S., Zhang, Y., Zhao, H., and Liu, P.
Dissecting human and LLM preferences.
In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp.  1790–1811, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.99.
URL [https://aclanthology.org/2024.acl-long.99/](https://aclanthology.org/2024.acl-long.99/).
- Li et al. (2023)
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B.
AlpacaEval: An Automatic Evaluator of Instruction-following Models, May 2023.
- Ludan et al. (2023)
Ludan, J. M., Lyu, Q., Yang, Y., Dugan, L., Yatskar, M., and Callison-Burch, C.
Interpretable-by-design text classification with iteratively generated concept bottleneck.
*arXiv preprint arXiv:2310.19660*, 2023.
- Mazeika et al. (2024)
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., Sakhaee, E., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D.
Harmbench: A standardized evaluation framework for automated red teaming and robust refusal.
2024.
- Narayanan & Shmatikov (2008)
Narayanan, A. and Shmatikov, V.
Robust de-anonymization of large sparse datasets.
In *2008 IEEE Symposium on Security and Privacy (sp 2008)*, pp.  111–125. IEEE, 2008.
- Nussbaum et al. (2024)
Nussbaum, Z., Morris, J. X., Duderstadt, B., and Mulyar, A.
Nomic embed: Training a reproducible long context text embedder, 2024.
- Ouyang et al. (2022)
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R.
Training language models to follow instructions with human feedback, 2022.
URL [https://arxiv.org/abs/2203.02155](https://arxiv.org/abs/2203.02155).
- Pan et al. (2023)
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W. Y.
Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.
*arXiv preprint arXiv:2308.03188*, 2023.
- Perez et al. (2022)
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G.
Red teaming language models with language models.
In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp.  3419–3448, 2022.
- Pham et al. (2023)
Pham, C. M., Hoyle, A. G., Sun, S., and Iyyer, M.
Topicgpt: A prompt-based topic modeling framework.
*arXiv preprint arXiv:2311.01449*, 2023.
- Qiu et al. (2023)
Qiu, L., Jiang, L., Lu, X., Sclar, M., Pyatkin, V., Bhagavatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., et al.
Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement.
*arXiv preprint arXiv:2310.08559*, 2023.
- Radford (2019)
Radford, A. e. a.
Language models are unsupervised multitask learners.
*OpenAI Blog*, 2019.
- Samvelyan et al. (2024)
Samvelyan, M., Raparthy, S. C., Lupu, A., Hambro, E., Markosyan, A. H., Bhatt, M., Mao, Y., Jiang, M., Parker-Holder, J., Foerster, J., et al.
Rainbow teaming: Open-ended generation of diverse adversarial prompts.
*arXiv preprint arXiv:2402.16822*, 2024.
- Sandhaus (2008)
Sandhaus, E.
The new york times annotated corpus.
Linguistic Data Consortium, Philadelphia, 2008.
- Schrodi et al. (2024)
Schrodi, S., Schur, J., Argus, M., and Brox, T.
Concept bottleneck models without predefined concepts.
*arXiv preprint arXiv:2407.03921*, 2024.
- Sileo (2023)
Sileo, D.
tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation.
*arXiv preprint arXiv:2301.05948*, 2023.
- Singh et al. (2023a)
Singh, C., Hsu, A. R., Antonello, R., Jain, S., Huth, A. G., Yu, B., and Gao, J.
Explaining black box text modules in natural language with language models.
*arXiv preprint arXiv:2305.09863*, 2023a.
- Singh et al. (2023b)
Singh, C., Morris, J., Rush, A. M., Gao, J., and Deng, Y.
Tree prompting: Efficient task adaptation without fine-tuning.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp.  6253–6267, 2023b.
- Singh et al. (2024a)
Singh, C., Inala, J. P., Galley, M., Caruana, R., and Gao, J.
Rethinking interpretability in the era of large language models.
*arXiv preprint arXiv:2402.01761*, 2024a.
- Singh et al. (2024b)
Singh, C., Nasseri, K., Tan, Y.-S., Tang, T., and Yu, B.
Rethinking interpretability in the era of large language models.
*arXiv preprint arXiv:2402.01761*, 2024b.
- Singh et al. (2025)
Singh, K., Kompa, B., Beam, A., and Schmaltz, A.
*clinspacy: Clinical Natural Language Processing using ’spaCy’, ’scispaCy’, and ’medspaCy’*, 2025.
URL [https://github.com/ml4lhs/clinspacy](https://github.com/ml4lhs/clinspacy).
R package version 1.0.2.9000.
- Sorensen & Others (2024)
Sorensen, B. and Others.
Clio: Privacy-preserving conversation analysis in large language models.
2024.
- Subramani et al. (2022)
Subramani, N., Suresh, N., and Peters, M. E.
Extracting latent steering vectors from pretrained language models.
In *Findings of the Association for Computational Linguistics: ACL 2022*, pp.  566–581, 2022.
- Tamkin et al. (2024)
Tamkin, A., McCain, M., Handa, K., Durmus, E., Lovitt, L., Rathi, A., Huang, S., Mountfield, A., Hong, J., Ritchie, S., et al.
Clio: Privacy-preserving insights into real-world ai use.
*arXiv preprint arXiv:2412.13678*, 2024.
- Team et al. (2024)
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
*arXiv preprint arXiv:2403.05530*, 2024.
- Templeton et al. (2024)
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., et al.
Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. transformer circuits thread, 2024.
- Treeratpituk & Callan (2006)
Treeratpituk, P. and Callan, J.
Automatically labeling hierarchical clusters.
In *Proceedings of the 2006 international conference on Digital government research*, pp.  167–176, 2006.
- Tufekci (2014)
Tufekci, Z.
Engineering the public: Big data, surveillance and computational politics.
*First Monday*, 2014.
- Varian (2014)
Varian, H. R.
Big data: New tricks for econometrics.
*Journal of economic perspectives*, 28(2):3–28, 2014.
- Vaswani (2017)
Vaswani, A.
Attention is all you need.
*Advances in Neural Information Processing Systems*, 2017.
- Viswanathan et al. (2023)
Viswanathan, V., Gashteovski, K., Lawrence, C., Wu, T., and Neubig, G.
Large language models enable few-shot clustering.
*arXiv preprint arXiv:2307.00524*, 2023.
- Wang et al. (2023)
Wang, Z., Shang, J., and Zhong, R.
Goal-driven explainable clustering via language descriptions.
*EMNLP*, 2023.
- Wang et al. (2024)
Wang, Z., Dong, Y., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O.
Helpsteer2: Open-source dataset for training top-performing reward models, 2024.
- Wei et al. (2024)
Wei, A., Haghtalab, N., and Steinhardt, J.
Jailbroken: How does llm safety training fail?
*Advances in Neural Information Processing Systems*, 36, 2024.
- Wolf et al. (2018)
Wolf, F. A., Angerer, P., and Theis, F. J.
Scanpy: large-scale single-cell gene expression data analysis.
*Genome biology*, 19:1–5, 2018.
- Xu et al. (2024)
Xu, R., Sun, Y., Ren, M., Guo, S., Pan, R., Lin, H., Sun, L., and Han, X.
Ai for social science and social science of ai: A survey.
*Information Processing & Management*, 61(3):103665, 2024.
- Yang et al. (2023)
Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., and Yatskar, M.
Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp.  19187–19197, 2023.
- Zhang et al. (2018)
Zhang, C., Tao, F., Chen, X., Shen, J., Jiang, M., Sadler, B., Vanni, M., and Han, J.
Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clustering.
In *Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp.  2701–2709, 2018.
- Zhao et al. (2024)
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y.
Wildchat: 1m chatGPT interaction logs in the wild.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=Bl8u7ZRlbM](https://openreview.net/forum?id=Bl8u7ZRlbM).
- Zheng et al. (2023a)
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H.
Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023a.
- Zheng et al. (2023b)
Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., et al.
Lmsys-chat-1m: A large-scale real-world llm conversation dataset.
*arXiv preprint arXiv:2309.11998*, 2023b.
- Zhong et al. (2022)
Zhong, R., Snell, C., Klein, D., and Steinhardt, J.
Describing differences between text distributions with natural language.
In *International Conference on Machine Learning*, pp.  27099–27116. PMLR, 2022.
- Zhong et al. (2023)
Zhong, R., Zhang, P., Li, S., Ahn, J., Klein, D., and Steinhardt, J.
Goal driven discovery of distributional differences via language descriptions.
*Advances in Neural Information Processing Systems*, 36:40204–40237, 2023.
- Zhong et al. (2024)
Zhong, R., Wang, H., Klein, D., and Steinhardt, J.
Explaining datasets in words: Statistical models with natural language parameters.
*arXiv preprint arXiv:2409.08466*, 2024.

## Appendix A Algorithmic Overview of the Pipeline

Figure: Algorithm 1 Multi-Stage Pipeline for Feature Extraction and Selection

## Appendix B Dataset Modeling Experiments (Extended Results)

### B.1 Semantic Preservation Metric

To measure semantic preservation, we use a simple prompt with Claude Haiku 3.5 (Anthropic, 2024). We first provide the natural language description of the original classes from the datasets, followed by the sampled features as a second class. We then evaluate whether at least one feature is semantically similar to each class and report the average number of classes for which this holds true.

### B.2 Convergence Analysis

We analyze the number of features required for convergence, where convergence is defined as reaching and maintaining 95% proximity to the maximum value of each metric achieved by a given method. [Table 3](#A2.T3) compares the required number of features for both the full pipeline and baseline approaches across different datasets. By calculating the average ratios across datasets, we find that the baseline requires 2.5 times more features than the full pipeline to converge on Class Coverage, and 2 times more features to converge on Reconstruction Accuracy.

**Table 3: Convergence analysis comparing the Full Pipeline (F.P.) and Baseline (Base) approaches using a 95% threshold. The table shows the mean number of features needed to reach and maintain performance above the threshold, along with the ratio of Baseline to Full Pipeline features.**
| Metric | DBPEDIA | NYT | AMAZON |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| F.P. | Base | F.P. | Base | F.P. | Base |  |
| Features at Convergence |  |  |  |  |  |  |
| Category Coverage | 14.0 | 31.3 | 18.0 | 41.3 | 18.7 | 30.7 |
| Reconstruction Accuracy | 5.0 | 11.3 | 18.0 | 37.0 | 20.7 | 29.7 |
| Semantic Preservation | 24.7 | 22.0 | 12.0 | 1.0 | 34.7 | 31.7 |
| Ratio (Base / F.P.) |  |  |  |  |  |  |
| Category Coverage | 2.2 |  | 2.3 |  | 1.6 |  |
| Reconstruction Accuracy | 2.3 |  | 2.1 |  | 1.4 |  |
| Semantic Preservation | 0.9 |  | 0.1 |  | 0.9 |  |

### B.3 Non-Topic Specific Prompting Baseline

We initially evaluated a simple baseline approach using prompts that directly asked the LLM to propose features from the data (detailed in [Section G.4](#A7.SS4)). However, this method produced poor results, underperforming even our intermediate pipeline stages. The results, reported in [Table 4](#A2.T4), show that this basic prompting approach fails to match even the performance of clustering. This highlights a fundamental limitation of simple LLM-based feature extraction: achieving adequate results typically requires extensive prompt engineering and iterative refinement. We ultimately settle on using topic-specific targeted prompts, which demonstrate significantly better performance.

**Table 4: Metrics from evaluation using a modified baseline prompt that avoids incentivizing topic-based features. Compared to [Table 7](#A7.T7), the results show generally worse performance, even underperforming clustering in many cases.**
| Dataset | Method | Class Coverage ↑ | Reconstruction Accuracy ↑ | Semantic Preservation ↑ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top 10 | Top 20 | Top 50 | Top 10 | Top 20 | Top 50 | Top 10 | Top 20 | Top 50 |  |  |
| DBPEDIA | Baseline | 0.691 | 0.803 | 0.842 | 86.3% | 95.0% | 97.7% | 0.13 | 0.20 | 0.20 |
| Generation | 0.711 | 0.768 | 0.807 | 89.6% | 91.6% | 94.8% | 0.33 | 0.67 | 2.33 |  |
| Clustering | 0.674 | 0.797 | 0.897 | 84.9% | 96.1% | 98.7% | 0.67 | 1.00 | 2.33 |  |
| Featurization | 0.886 | 0.896 | 0.929 | 98.6% | 99.1% | 99.4% | 2.00 | 2.00 | 3.33 |  |
| NYT | Baseline | 0.342 | 0.467 | 0.577 | 58.3% | 70.5% | 78.7% | 0.00 | 0.00 | 0.00 |
| Generation | 0.375 | 0.451 | 0.499 | 56.1% | 66.8% | 73.6% | 0.33 | 0.33 | 0.33 |  |
| Clustering | 0.433 | 0.565 | 0.703 | 63.3% | 74.9% | 84.7% | 0.00 | 0.00 | 0.33 |  |
| Featurization | 0.530 | 0.692 | 0.723 | 73.1% | 85.1% | 86.5% | 0.00 | 0.67 | 1.00 |  |
| AMAZON | Baseline | 0.275 | 0.301 | 0.399 | 47.3% | 51.2% | 58.7% | 0.00 | 0.67 | 0.67 |
| Generation | 0.237 | 0.270 | 0.370 | 44.6% | 47.0% | 56.1% | 0.00 | 0.33 | 0.67 |  |
| Clustering | 0.244 | 0.388 | 0.522 | 44.5% | 58.1% | 68.9% | 0.00 | 0.00 | 0.67 |  |
| Featurization | 0.484 | 0.600 | 0.632 | 62.8% | 73.5% | 76.8% | 1.00 | 1.67 | 2.67 |  |

## Appendix C Extracting Compact Attacks from Jailbreaks (Extended Results)

### C.1 Evaluation Metrics

All evaluation metrics are adopted from WildTeaming (Jiang et al., 2024b), where additional methodology details can be found. We exclude perplexity-based evaluation since differences are negligible due to using identical jailbreak generation methods.

Effectiveness Evaluation. Following WildTeaming’s methodology, we generate 30 jailbreaks for each harmful instruction in HarmBench (Mazeika et al., 2024) using each method, and obtain responses from target models with temperature set to 0. For Attack Success Rate (ASR), we employ the test classifier to evaluate success by analyzing both the original harmful prompt and the model response. We also measure the number of queries required to achieve a successful attack (Query).

Diversity Evaluation. To evaluate each method’s capability to discover diverse model vulnerabilities, we analyze the 30 jailbreaks per harmful prompt generated in the Effectiveness Evaluation using several metrics. We define $\text{ASR}\times n_{c}=\frac{1}{n}\sum_{i=1}^{n}\text{ASR}@i_{c}$ to measure the average success rate for finding $i\in\{1,...,n\}$ unique attacks given $c$ attack trials. Here, $\text{ASR}@i_{c}$ represents the success rate of simultaneously finding $i$ unique successful attacks given $c$ attempts. Attack uniqueness is determined using a sentence embedding similarity threshold of 0.75, as established by Nussbaum et al. (2024).

We also report $\text{Query}\times n_{c}=\frac{1}{n}\sum_{i=1}^{n}\text{Query}@i_{c}$, measuring the average number of queries needed to find $i\in\{1,...,n\}$ unique successful attacks given $c$ attack trials. Here, $\text{Query}@i_{c}$ represents the number of queries required to find $i$ unique successful attacks among $c$ attack attempts. $\text{Sim}@n_{c}$ calculates the average pairwise sentence embedding similarity among the first $n$ successful attacks, while $\text{Sim}_{\text{all}}$ measures the pairwise sentence embedding similarity among all successful attacks across the evaluation pool.

### C.2 Analyzing the Effect of Sampling More Features

In this section, we analyze how the number of sampled features affects the metrics in WildTeaming (Jiang et al., 2024b). We examine the performance of features described in [Section H.1](#A8.SS1) using the metrics outlined in [Section C.1](#A3.SS1), varying the sample size from 5 to 50 features in increments of 5.

Our analysis reveals that while 5 features are sufficient to achieve a high ASR, the resulting jailbreaks lack diversity. As we increase the number of sampled features, we observe improvements in diversity metrics, accompanied by slight improvements in ASR and minor decreases in Query length. This trade-off between diversity and query efficiency is expected when generating more diverse data. The improvements plateau around 20 features, with minimal gains in both diversity and effectiveness metrics beyond this point. We hypothesize that this plateau occurs because we no longer observe significant changes in similarity metrics after 20 features. This directly affects $\mathsf{ASR}^{\texttimes 5}_{30}$ and $\mathsf{Query}^{\texttimes 5}_{30}$, which rely on sentence similarity for uniqueness determination. Consequently, our current metric framework may be unable to capture more nuanced diversity improvements beyond basic sentence similarity.

Figure: Figure 4: Impact of feature sampling size on jailbreak performance metrics described in [Section C.1](#A3.SS1). While ASR plateaus early, diversity metrics continue to improve until approximately 20 features, after which improvements become minimal across all metrics.
Refer to caption: /html/2502.17541/assets/x4.png

## Appendix D Compositional Preference Modeling (Extended Results)

### D.1 Feature Interpretability

One potential concern with automated feature discovery is the lack of interpretability. However, since our trained preference model is linear, the features discovered by our pipeline maintain interpretability comparable to those in (Go et al., 2024). By examining the linear regression coefficients, we can identify the most influential features for assessing response quality. In the HH-RLHF dataset, the features with the highest positive coefficients are ”includes educational content” and ”is structured with clear, distinct sections.” For a detailed analysis of these features, refer to LABEL:table:preferences-shp-features for SHP and LABEL:table:preferences-hh-features for HH-RLHF, with the top coefficients visualized in [Table 5](#A4.T5).

**Table 5: Top 10 (out of 50) features with highest linear regression PM coefficients for HH-RLHF and SHP datasets.**
| HH-RLHF | SHP |  |  |
| --- | --- | --- | --- |
| Feature | Coefficient | Feature | Coefficient |
| includes educational content | 0.283 | conveys surprise or unexpectedness | 0.217 |
| implies a misunderstanding | -0.215 | provides minimal detail and context | -0.172 |
| is structured with clear, distinct sections | 0.206 | is longer and more detailed | 0.170 |
| provides detailed explanations and examples | 0.197 | employs a playful and whimsical tone | 0.159 |
| conveys a polite acknowledgment | 0.187 | includes broader universe context | 0.154 |
| focuses on human behavior and nationality | -0.122 | employs a more direct messaging style | 0.144 |
| includes direct references to external resources | -0.103 | uses a definitive negative tone | 0.122 |
| uses a direct and personal address | -0.097 | includes an admission of incomplete knowledge | -0.115 |
| uses parallel structure for clarity | 0.087 | offers a broader perspective on job treatment | 0.111 |
| uses a first-person perspective | -0.079 | focuses on personal convenience and flexibility | -0.098 |

### D.2 Robustness Across BoN Responses

In [Figure 5](#A4.F5), we analyze the robustness of the trained PMs. For each method, we compare the highest reward assigned by PM A on the BoN response sample against the reward assigned by PM B, which was trained with identical features but on a held-out test set. Our analysis of the HH-RLHF dataset shows that both our PMs and the baseline PMs maintain similar reward differentials, though our models achieve slightly higher rewards overall. The most significant differences in robustness emerge in the SHP dataset, where our models demonstrate substantially better robustness compared to the baseline PMs, even with very limited numbers of BoN responses.

Figure: Figure 5: We fit two preference models (PM A and PM B) on separate subsets of each dataset. For each value of $N$, we sample $N$ random responses and calculate the average reward given by PM A, then visualize the average reward from PM B for the same samples. As $N$ increases, we expect PM A to show overfitting, widening the gap between models. Our results reveal limited overfitting in HH-RLHF, appearing only at higher $N$ values, while SHP shows significant non-robustness starting at around 4 responses and persisting across larger sample sizes.
Refer to caption: /html/2502.17541/assets/x5.png

## Appendix E Pipeline Details

## Appendix F Pipeline Prompts

### F.1 Generation Stage

### F.2 Evaluation Prompt

### F.3 Featurization Prompts

#### Dataset Modeling Experiments

#### Extracting Compact Attacks from Jailbreaks

#### Compositional Preference Modeling

## Appendix G Dataset Modeling Experiments (Supplementary Materials)

### G.1 Data Format of Synthetic Datasets

**Table 6: Data formats of the evaluation datasets.**
| NYT | AMAZON | DBPEDIA |
| --- | --- | --- |
| {headline}<br>{body} | {title}<br>{text} | {text} |

### G.2 The Numerical Results Presented in Figure 2 .

**Table 7: Concrete numerical metrics compare our pipeline and its stages against the prompting baseline, which is incentivized to generate topic-specific features. These results are shown in [Figure 2](#S5.F2).**
| Dataset | Method | Class Coverage ↑ | Reconstruction Accuracy ↑ | Semantic Preservation ↑ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top 10 | Top 20 | Top 50 | Top 10 | Top 20 | Top 50 | Top 10 | Top 20 | Top 50 |  |  |
| DBPEDIA | Baseline | 0.805 | 0.840 | 0.911 | 94.8% | 98.0% | 99.5% | 0.333 | 0.333 | 0.667 |
| Generation | 0.711 | 0.768 | 0.807 | 89.6% | 91.6% | 94.8% | 0.333 | 0.667 | 2.333 |  |
| Clustering | 0.674 | 0.797 | 0.897 | 84.9% | 96.1% | 98.7% | 0.667 | 1.000 | 2.333 |  |
| Featurization | 0.886 | 0.896 | 0.929 | 98.6% | 99.1% | 99.4% | 2.000 | 2.000 | 3.333 |  |
| NYT | Baseline | 0.362 | 0.442 | 0.546 | 54.7% | 62.7% | 73.1% | 0.000 | 0.000 | 0.000 |
| Generation | 0.375 | 0.451 | 0.499 | 56.1% | 66.8% | 73.6% | 0.333 | 0.333 | 0.333 |  |
| Clustering | 0.433 | 0.565 | 0.703 | 63.3% | 74.9% | 84.7% | 0.000 | 0.000 | 0.333 |  |
| Featurization | 0.530 | 0.692 | 0.723 | 73.1% | 85.1% | 86.5% | 0.000 | 0.667 | 0.667 |  |
| AMAZON | Baseline | 0.422 | 0.471 | 0.585 | 59.0% | 62.9% | 73.6% | 0.333 | 0.667 | 1.333 |
| Generation | 0.237 | 0.270 | 0.370 | 44.6% | 47.0% | 56.1% | 0.000 | 0.333 | 0.667 |  |
| Clustering | 0.244 | 0.388 | 0.522 | 44.5% | 58.1% | 68.9% | 0.000 | 0.000 | 0.667 |  |
| Featurization | 0.484 | 0.600 | 0.632 | 62.8% | 73.5% | 76.8% | 1.000 | 1.667 | 2.667 |  |

### G.3 Baseline Prompt Used in the Main Body

### G.4 Alternative Baseline Prompt with Lower Performance, as reported in Section B.3

## Appendix H Extracting Compact Attacks from Jailbreaks (Supplementary Materials)

### H.1 Features Extracted from WildJailbreak

### H.2 Features Extracted from Filtered WildJailbreak of Llama 3.1 8B Instruct Non-Refusals

### H.3 WildTeaming Prompt Used in the Original Study

### H.4 Our Edited Prompt Based on the One from WildTeaming (see Section H.3 )

## Appendix I Compositional Preference Modeling (Supplementary Materials)

### I.1 Response Rating Prompts

#### Prompt Used for SHP Dataset

#### Prompt Used for HH-RLHF Dataset

### I.2 Prompt Generating Attributes

### I.3 Expert-crafted Features Used in the Original Study

| CPM Original Features |  |  |
| --- | --- | --- |
| Feature Description | Minimum | Maximum |
| is helpful for the original poster | not helpful | very helpful |
| is specific enough | too vague | very specific |
| understands the original poster’s intent | failure of understanding | perfect understanding |
| is factually correct | egregiously incorrect | fully correct |
| is easy to understand | very difficult to understand | very easy to understand |
| is relevant to the original poster’s question | off-topic | very relevant |
| is easy to read and not too technical for the original poster | very difficult to read | very easy to read |
| provides enough detail to be helpful | too little detail | very detailed |
| is biased or one-sided | very biased | not biased at all |
| fails to consider the original poster’s cultural or individual preferences | takes into account the original poster’s preferences | fails to consider the original poster’s preferences |
| is repetitive | very repetitive | not repetitive |
| fails to consider the original poster’s context | fails to consider the original poster’s context | takes into account the original poster’s context |
| is too long | too long | not too long |

### I.4 50 Sampled Features from the SHP Dataset

| Our Pipeline’s SHP Features (Ordered as Sampled) |  |  |
| --- | --- | --- |
| Attribute Description | Minimum | Maximum |
| implies indirect messaging through brevity. | implies direct messaging with clarity | implies very indirect messaging through extreme brevity |
| implies a personal experience or context. | lacks any personal experience or context | strongly implies a personal experience or context |
| provides minimal detail and context. | provides comprehensive detail and context | provides extremely minimal detail and context |
| ends with a non-alphabetic character. | ends with an alphabetic character | ends with a highly non-alphabetic character |
| lacks a question mark in the title. | includes a question mark in the title when appropriate | consistently lacks a question mark in the title when needed |
| implies an external resource or reference. | does not imply any external resource or reference | strongly implies an external resource or reference |
| includes a hypothetical scenario. | lacks any hypothetical scenario | includes a detailed and engaging hypothetical scenario |
| employs a more direct messaging style. | employs an indirect and vague messaging style | employs an extremely direct and clear messaging style |
| has a more inquisitive tone. | has a flat or disinterested tone | has an extremely inquisitive and engaging tone |
| is longer and more detailed. | is brief and lacks detail | is extremely long and highly detailed |
| employs a playful and whimsical tone. | employs a serious and formal tone | employs an extremely playful and whimsical tone |
| includes a specific online community reference. | lacks any reference to an online community | includes a highly relevant and specific online community reference |
| includes a direct request for information. | lacks any request for information | clearly and explicitly requests specific information |
| includes specific examples and scenarios. | lacks examples and scenarios | richly includes specific examples and scenarios |
| presents a conditional scenario for clarity. | presents an unclear or confusing scenario | presents a highly clear and well-defined conditional scenario |
| contains direct personal advice and opinions. | lacks personal advice and opinions | rich in direct personal advice and opinions |
| employs indirect messaging. | employs direct and clear messaging | employs highly indirect and ambiguous messaging |
| contains a direct financial reference. | lacks any financial reference | contains explicit and detailed financial references |
| uses ellipsis for dramatic pause. | avoids using ellipsis, resulting in a flat delivery | effectively uses ellipsis to create a strong dramatic pause |
| uses a specific geographical reference. | lacks any geographical reference | uses highly specific and accurate geographical references |
| includes an admission of incomplete knowledge. | claims complete knowledge without admitting gaps | openly acknowledges limitations and gaps in knowledge |
| uses a definitive negative tone. | uses a neutral or positive tone | uses an extremely negative and harsh tone |
| lacks descriptive imagery or sensory details. | rich in descriptive imagery and sensory details | completely lacks descriptive imagery or sensory details |
| employs a conversational tone for relatability. | uses a formal tone, making it less relatable | employs an extremely conversational tone, enhancing relatability |
| lacks descriptive imagery. | rich in vivid and detailed imagery | completely devoid of descriptive imagery |
| includes punctuation marks. | lacks punctuation marks entirely | uses punctuation marks effectively and appropriately |
| lacks phonetic guidance. | provides clear phonetic guidance | completely lacks phonetic guidance |
| lacks descriptive adjectives and adverbs. | rich in descriptive adjectives and adverbs | completely lacks descriptive adjectives and adverbs |
| uses a more dismissive tone. | uses a respectful and engaging tone | uses an extremely dismissive tone |
| uses a positive, self-rewarding approach. | uses a negative, self-punishing approach | uses an extremely positive, self-rewarding approach |
| contains spelling errors. | contains no spelling errors | contains numerous spelling errors |
| offers a broader perspective on job treatment. | offers a narrow perspective on job treatment | offers an exceptionally broad and comprehensive perspective on job treatment |
| conveys surprise or unexpectedness. | predictable and expected | highly surprising and unexpected |
| includes a direct personal opinion on usage. | lacks any personal opinion on usage | strongly includes a direct personal opinion on usage |
| hints at a nostalgic cultural reference. | lacks any cultural reference or nostalgia | evokes strong nostalgia with clear cultural references |
| implies a deeper understanding of mechanical operations. | lacks understanding of mechanical operations | demonstrates profound understanding of mechanical operations |
| focuses on personal convenience and flexibility. | inconvenient and inflexible | highly convenient and extremely flexible |
| highlights community support and collegiality. | lacks community support and collegiality | strongly emphasizes community support and collegiality |
| conveys excitement with exclamation. | conveys excitement without any exclamation | conveys extreme excitement with multiple exclamations |
| incorporates personal dining rituals and preferences. | ignores personal dining rituals and preferences | fully incorporates personal dining rituals and preferences |
| provides more vivid imagery of baking issues. | provides vague or unclear imagery of baking issues | provides extremely vivid and detailed imagery of baking issues |
| includes broader universe context. | lacks broader universe context | fully integrates broader universe context |
| highlights thinkers’ willingness to embrace pessimism. | ignores or dismisses pessimism | fully embraces and explores pessimism |
| focuses on positive aspects with minimal detail. | focuses on negative aspects with excessive detail | focuses exclusively on positive aspects with minimal detail |
| includes specific age-related considerations. | lacks any age-related considerations | thoroughly includes specific age-related considerations |
| discusses broader organizational dynamics and implications. | ignores organizational dynamics and implications | thoroughly analyzes and explains broader organizational dynamics and implications |
| implies flexibility with ’likely’ and ’probably’. | implies certainty without using ’likely’ or ’probably’ | heavily relies on ’likely’ and ’probably’ to imply flexibility |
| uses repetitive phrases for emphasis. | avoids repetition, lacks emphasis | overuses repetitive phrases, overly emphatic |
| conveys a more pessimistic emotional impact. | conveys an optimistic or neutral emotional impact | conveys an extremely pessimistic emotional impact |
| focuses on scientific complexity and observational challenges. | oversimplifies scientific concepts and ignores observational challenges | deeply explores scientific complexity and thoroughly addresses observational challenges |

### I.5 50 Sampled Features from the HH-RLHF Dataset

| Our Pipeline’s HH-RLHF Features (Ordered as Sampled) |  |  |
| --- | --- | --- |
| Attribute Description | Minimum | Maximum |
| uses a direct questioning approach. | uses an indirect or vague questioning approach | uses a highly direct and clear questioning approach |
| lacks technical details about fire ignition methods. | provides comprehensive technical details about fire ignition methods | completely lacks technical details about fire ignition methods |
| implies a misunderstanding. | clearly conveys understanding | strongly implies a misunderstanding |
| uses a direct and personal address. | uses an indirect and impersonal address | uses a highly direct and personal address |
| employs direct speech with quotation marks. | lacks direct speech and quotation marks | effectively employs direct speech with clear quotation marks |
| conveys a sense of continuity and ongoing activity. | feels disjointed and static | seamlessly flows with dynamic progression |
| lacks specific details or context. | provides comprehensive details and context | completely lacks specific details or context |
| lacks the phrase ’or severely impaired in some way.’. | includes the phrase ’or severely impaired in some way’ appropriately | completely lacks the phrase ’or severely impaired in some way’ |
| includes personal feelings and experiences. | excludes personal feelings and experiences | richly includes personal feelings and experiences |
| uses direct messaging without elaboration. | uses detailed and elaborative messaging | uses extremely brief and direct messaging without any elaboration |
| uses repetition for emphasis. | avoids repetition, leading to a lack of emphasis | excessively uses repetition, causing redundancy |
| emphasizes user feedback and dialogue. | ignores user feedback and lacks dialogue | actively incorporates user feedback and maintains engaging dialogue |
| includes both human and technological elements. | lacks integration of human and technological elements | seamlessly integrates both human and technological elements |
| uses informal language with ’you’re’ contraction. | uses formal language without contractions | frequently uses informal language with ’you’re’ contraction |
| uses a first-person perspective. | uses a third-person perspective | consistently uses a first-person perspective throughout |
| lacks mention of additional communication systems. | includes comprehensive details on additional communication systems | completely omits any mention of additional communication systems |
| includes a direct address to the reader. | lacks any direct address to the reader | frequently and effectively addresses the reader directly |
| lacks detailed comparisons to other building collapses. | provides comprehensive comparisons to other building collapses | completely lacks detailed comparisons to other building collapses |
| includes educational content. | lacks educational content | richly filled with educational content |
| uses parallel structure for clarity. | lacks parallel structure, causing confusion | consistently uses parallel structure for maximum clarity |
| includes direct references to external resources. | lacks any references to external resources | includes numerous and relevant direct references to external resources |
| uses a colon to introduce content. | does not use a colon to introduce content | effectively uses a colon to introduce content |
| provides detailed explanations and examples. | provides vague explanations with no examples | provides comprehensive explanations with numerous relevant examples |
| uses fewer descriptive adjectives. | uses many descriptive adjectives | uses very few descriptive adjectives |
| uses more direct dialogue and exclamations. | uses indirect dialogue and lacks exclamations | uses very direct dialogue and frequent exclamations |
| conveys a polite acknowledgment. | lacks acknowledgment or is impolite | exhibits exceptionally polite acknowledgment |
| includes a hypothetical scenario. | lacks any hypothetical scenario | includes a detailed and engaging hypothetical scenario |
| ends abruptly, suggesting an incomplete thought. | ends smoothly and with a complete thought | ends very abruptly, leaving the thought incomplete |
| lacks detailed descriptions of olive oil benefits. | provides comprehensive and detailed descriptions of olive oil benefits | completely lacks any descriptions of olive oil benefits |
| is longer and more comprehensive. | is brief and lacks detail | is extremely lengthy and overly detailed |
| uses a direct refusal. | uses an indirect or polite refusal | uses a very blunt or harsh refusal |
| focuses on human behavior and nationality. | ignores human behavior and nationality | deeply analyzes human behavior and nationality |
| implies a quantitative evaluation method. | lacks any quantitative evaluation method | utilizes a comprehensive and precise quantitative evaluation method |
| uses future tense. | uses past or present tense | consistently uses future tense |
| implies uncertainty with ’I think’. | states information with certainty and confidence | frequently uses ’I think’ to express uncertainty |
| suggests a specific product. | does not suggest any specific product | clearly and accurately suggests a specific product |
| is longer and more detailed. | is brief and lacks detail | is extremely long and overly detailed |
| introduces a meta-commentary about communication. | lacks any meta-commentary about communication | provides insightful and extensive meta-commentary about communication |
| includes a request for clarification. | provides clear and comprehensive information without needing clarification | frequently requests clarification, indicating uncertainty or lack of understanding |
| lacks specific subject matter references. | contains detailed and specific subject matter references | completely lacks any subject matter references |
| uses conditional language to offer flexible guidance. | uses rigid language with no flexibility | uses highly adaptive and flexible language |
| uses conditional language for hypothetical scenarios. | does not use conditional language for hypothetical scenarios | consistently uses precise conditional language for all hypothetical scenarios |
| focuses on texture and material properties. | ignores texture and material properties | provides detailed and insightful analysis of texture and material properties |
| includes a clear offer of additional services. | lacks any mention of additional services | provides a detailed and enticing offer of additional services |
| uses a cause-and-effect structure. | lacks clear cause-and-effect relationships | demonstrates a clear and logical cause-and-effect structure |
| includes a comparison of motivations. | lacks any comparison of motivations | provides a thorough and insightful comparison of motivations |
| is structured with clear, distinct sections. | is disorganized with no clear sections | is highly organized with very clear and distinct sections |
| includes a detailed description of components. | lacks detail in the description of components | provides an extremely detailed and comprehensive description of components |
| includes a specific cultural reference. | lacks any cultural reference | richly incorporates a specific cultural reference |
| uses a specific company reference. | does not use any company reference | uses a highly specific and relevant company reference |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 2
presentation: 1
questions: Please see the weaknesses.
soundness: 1
strengths: The high-level idea of using extracted features to reconstruct the dataset for selecting features is interesting.
summary: This research tries to construct interpretable features for a textual corpus in an unsupervised manner. To do so, the researchers propose to first prompt LLMs to generate the features from the samples, and then confirm the effectiveness of the features by minimizing the perplexity of the dataset.
weaknesses: 1. The writing of this paper is terrible. It is pretty hard for me to get an understanding of how it works and why it works. For example, I have no idea how you compute the perplexity exactly from lines 170-178.  Are you following the contents in lines 140-145? If yes, how are these conditional PPLs measured concretely? Similarly, in Section 5, I don't know why we need to consider the class labels provided by the datasets, and how you evaluate your results. More importantly, why is this experiment result correct?

2. Experiments miss many simple but effective strategies to mine superficial features from the dataset unsupervisely. For example, you can compute the TF-IDF scores for different N-grams.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.) I'm pretty confused about how to think about the novelty/originality of the paper - could the authors make a.) the concise steelman for why this is work is highly original, or just b.) what their all-things-considered view is, here.
2.) How important is the limitation of this method only working for binary features? Just how many applications would open up?
3.) What did you do that convinced you that the greedy optimisation target was ~fine, and unlikely to get stuck in local optima?
soundness: 3
strengths: The authors' perplexity / reconstruction-based optimisation framing seems novel and their results suggest that it's effective. Their case studies show real applications that are highly relevant to LLM safety and alignment. Extracting compact representations of jailbreaks from a large unlabelled set of attacks without supervision seems highly useful for understanding new attack strategies at scale. The case study on preference modeling also emphasises that the lack of supervision needed for this method might also mitigate bias arising from human labelling, but I'm less sold that the bias reduction point is the most groundbreaking feature of the method (not that the authors imply this). I'm not totally sure how to estimate the impact of this method - it genuinely seems useful across a wide set of tasks and domains, and the controllability and unsupervised nature of the method make it preferable over other methods. The case studies are helpful for understanding the sorts of tasks on which the authors expect the method to be impactful.

The experiments are well substantiated. The synthetic datasets they use are diverse, and the two case studies further illustrate how widely-applicable their method is. They compare to reasonable baselines in prompting and Zhong et al 2024. It's a little hard to know how much further you could push the prompt engineering, but it intuitively seems like prompting would still be overall worse than a pipeline like Zhong et al or from this paper. They also convincingly demonstrate that their method improves over human-written features, which seems like another important axis of comparison for their method. The paper's extensive appendices leave very little unaddressed. I particularly appreciated the cost analysis in appendix C, which is a practical and thoughtful inclusion - though I would have appreciated just a little more depth in the cost comparison with Zhong et al, rather than just the (probably approximately true) claim that the grid search would increase the total cost and runtime of the competitor method tenfold.

Overall, the writing and presentation of the paper is clear, well motivated, and well organised.
summary: The authors introduce a method for breaking datasets down into a granularly controllable number of features. The method works by first having an LLM propose features for each sample in an unlabelled dataset, then they deduplicate the set of features by clustering similar features together. They then evaluate the accuracy of each features by whether the features are useful to an LLM in reconstructing the original data samples labelled by that feature (measured by minimising perplexity)

The authors evaluate on three synthetic datasets (DBPedia, NYT, Amazon 
Reviews) with known ground-truth labels, demonstrating superior performance over LLM prompting 
baselines and Zhong et al. (2024). They also show the effectiveness of their method with two case studies. For the first, they compress 500 jailbreak attack tactics into 20 features while maintaining attack 
effectiveness and diversity. For the second case study, they automate preference model feature discovery, matching human-crafted features from Go et al. (2024) in accuracy and robustness.
weaknesses: Overall I think the paper is quite strong, so struggling to find substantive weaknesses.

It's worth calling out that the paper overall provides a useful advancement with this reconstruction focus on unsupervised dataset feature labelling, but as the authors themselves acknowledge, the paper is in large part building on other recent work (Zhong et al. 2024, Findeis et al. 2024, Go et al. 2024). In their experiments comparing their method to Zhong et al, the authors convinced me that their method is stronger and likely more efficient (in dollars and time), but the overall place it seems like this paper sits in the literature is as a noticeable but not paradigm-shifting improvement on a reasonably important and wide-ranging problem. So, while the paper is still original and significant, it's not superlative on either axis.

As the authors themselves mention in section 7, this pipeline is restricted to binary features, and it's probably true that a method that could extract richer feature representations while holding all else equal would be useful across a wider range of domains & tasks.

The authors also note in section 8 that their greedy method introduces potential convergence to local optima, so more investigation into alternative selection strategies could plausibly improve on this method? Would be interested in the authors' takes.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * What is the length sensitivity of the scoring objective e.g. does performance degrade near the upper character limit or with longer contexts?

* Will you release prompts, code, and feature lists to facilitate replication?
soundness: 3
strengths: * Clear, modular pipeline with a principled dataset-level objective and an interpretable feature representation.

* Demonstrates strong compression of jailbreak strategies while maintaining attack efficacy across multiple target models.

* Shows that automatically discovered features can match/beat expert-designed features in CPM, with reasonable robustness analyses.

* Practical engineering details.
summary: The paper proposes Dataset Featurization: a three-stage pipeline that (1) elicits many human-readable binary features via an LLM, (2) de-duplicates them with embedding-based clustering and obtains per-example truth values, and (3) greedily builds a global feature set that minimizes the mean per-text perplexity of a scoring LM when conditioned on the features that are true for each example. The resulting small, interpretable “feature basis” is shown useful in two applications: (i) jailbreak tactic compression (achieving comparable effectiveness to a 500-cluster baseline with ~20 features across multiple target models), and (ii) composable preference modeling (CPM) on SHP/HH-RLHF, where automatically discovered features rival or surpass expert-crafted ones at similar feature counts.
weaknesses: * Binary-only features and reliance on positive instances during optimization limit expressivity.

* The optimization is global, so it does not guarantee per-instance optimality; some examples may not benefit from the selected global basis.

* No dedicated length-sensitivity/long-context study (data filtered to a range; but scalability to very long inputs remains unclear).

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - One concern about the method is that the generation stage may produce non-discriminative or overly generic features. Although the method is designed to discard irrelevant features because they would lead to high perplexity during text reconstruction, there can be features that actually maintain low perplexity across multiple samples while providing minimal information about the dataset's structure. In Section 6.2, the authors mention filtering low-variance features, suggesting this problem was encountered in practice, though this appears as a post-hoc fix rather than a solution. I would like to ask the authors whether they have encountered this type of issue systematically and whether it is possible to detect or prevent such features during the generation stage.
- In Section 6.2, the authors report that on HH-RLHF the method achieves 68.1% accuracy while the baseline reaches 68.9% (Table 2). In Appendix F.2, the authors attribute this to an accuracy ceiling due to dataset noise (potentially incorrect annotations). However, if the dataset noise is the limiting factor, why does the baseline with hand-crafted features perform slightly better? Does this suggest that the unsupervised features are more susceptible to noise?
- In Section 4, the authors mention setting C=5 and K=5 for all experiments and observing “additional performance gains by increasing the number of proposed features,” and refer to Appendix D.1 for additional details.  I couldn't find an ablation study systematically varying C and K. I would like to ask the authors whether they explored how performance scales with these hyperparameters?
- Considering the greedy sequential selection in the featurization stage: since the choice of earlier features affects which subsequent features provide marginal information gain, does the order of evaluation significantly impact which features are selected? Does running the pipeline multiple times produce consistent feature sets?
soundness: 3
strengths: - The authors test the method on several dataset-modeling tasks and case studies, validating it from multiple perspectives.
- The method is practical and valuable, yielding better performance than other baselines.
summary: The authors propose a domain-agnostic method to featurize text datasets with natural-language features. By leveraging a language model to produce possible features, and perplexity minimization during original text reconstruction to select them, the authors are able to extract interpretable features without supervision. The authors validate the proposed method on several dataset modeling tasks and case studies, demonstrating higher effectiveness in feature extraction compared to previous approaches.
weaknesses: - It is unclear which failure modes arise in the generation stage and how they could be mitigated.
- Some claims and methodological choices are not clearly explained or insufficiently supported; I outline specific concerns in the questions section.