Venue: NAACL 2025

_Is a Peeled Apple Still Red?_ Evaluating LLMs' Ability for Conceptual Combination with Property Type

Seokwon Song\({}^{*,1}\)   Taehyun Lee\({}^{*,1}\)   Jaewoo Ahn\({}^{1}\)

Jae Hyuk Sung\({}^{2,\sharp}\)   Gunhee Kim\({}^{1}\)

\({}^{1}\)Seoul National University   \({}^{2}\)Korea University

{seokwan.song, taehyun.lee, jaewoo.ahn}@vision.snu.ac.kr, okaybodyl0@korea.ac.kr gunhee@snu.ac.kr

###### Abstract

Conceptual combination is a cognitive process that merges basic concepts, enabling the creation of complex expressions. During this process, the properties of combination (e.g., the whiteness of a peeled apple) can be inherited from basic concepts, newly emerge, or be canceled. However, previous studies have evaluated a limited set of properties and have not examined the generative process. To address this gap, we introduce the Conceptual Combination with Property Type dataset (CCPT), which consists of 12.3K annotated triplets of noun phrases, properties, and property types. Using CCPT, we establish three types of tasks to evaluate LLMs for conceptual combination thoroughly. Our key findings are threefold: (1) Our automatic metric grading property emergence and cancellation closely corresponds with human judgments. (2) LLMs, including OpenAI's o1, struggle to generate noun phrases which possess given emergent properties. (3) Our proposed method, inspired by cognitive psychology model that explains how relationships between concepts are formed, improves performances in all generative tasks. The dataset and experimental code are available at [https://github.com/seeokwon99/CCPT.git](https://github.com/seeokwon99/CCPT.git).

+
Footnote †: _(Volume 1: Long Papers)_, pages 9028–9048

+
Footnote †: _(Volume 1: Long Papers)_, pages 9028–9048

+
Footnote †: _(Volume 1: Long Papers)_, pages 9028–9048

## 1 Introduction

The conceptual combination is a fundamental cognitive process that synthesizes multiple basic concepts into a novel concept Wisniewski (1997); Thagard (1984); Coutanche et al. (2019); Ward (2001). Combining concepts is potentially limitless and plays an important role in various fields such as engineering, science Hampton (1997), figurative language and literature Ward (2001). For example, 'a wilted flower' evokes our sadness because we internally compare a flower's former beauty and current feeble state. A skilled writer knows it and kindles diverse imagery through conceptual combinations. The potential for such conceptual combinations is vast, so large language models (LLMs) should be able to interpret or generate meaningful insights even for rare or novel combinations.

Several works have explored concept knowledge of language models, as outlined in Table 1. Basic Concept Probing examines whether language models understand individual concepts at the entity (e.g., "horses are mammals") and event levels (e.g., "drinking tea leads to feeling refreshed"). However, this approach mainly focuses on single concepts rather than combinations. Noun Compound Interpretation studies how models generate plausible meanings for combinations of nouns (e.g., "dog house" means "a house for a dog"), but does not explore deeper properties of combinations. In conceptual combination works, Big-Bench-CC investigates emergent properties. FakeReef delves into a special type of canceled property (membership inference). Still, none of them addresses all of emergent, component, or canceled properties nor generative tasks related to creating combinations or property.

It is essential to understand not only properties that emerge from a combination, but also whether

Figure 1: Three types of properties derived from conceptual combination with an example of “apple”. Different concepts are formed by adding other concepts to “apple”. The green properties are _component properties_ of the basic concept “apple”. The blue and orange are _emergent_ and _canceled properties_, respectively.

 

existing properties maintain or disappear (Springer and Murphy, 1992; Gagne et al., 2017; Estes and Glucksberg, 2020). Following Springer and Murphy (1992), we identify three types of properties in conceptual combination, which are illustrated using the example of an "apple" in Figure 1: (1) _Component property_ refers to a property inherent to individual concepts (e.g., "red" for "apple on a toothpick"), (2) _Emergent property_ is newly from the combination of concepts (e.g., "unstable" for "apple on a toothpick"), and (3) _Canceled property_ is negated due to the combination (e.g., "crisp" for "rotten apple"). These three types form a complete set of properties related to conceptual combination since they indicate _already present_, _newly created_, and _newly disappeared_.

In this study, we aim to fill this gap by proposing the **C**onceptual **C**ombination with **P**roperty **T**ype (**CCPT**) dataset. Our dataset contains 12.3K annotated triplets of noun phrase comprised of two basic concepts, property and property type. The basic concepts in noun phrases are comprised of 9K unique uni-gram concepts from ConceptNet (Speer et al., 2017).

To evaluate LLMs' ability on conceptual combination, we introduce two generative and one classification tasks, detailed in Table 2 and SS3.1. Based on the prior works in cognitive psychology and linguistics, each task is designed to evaluate cognitive processes related to conceptual combination as follows. (1) _Property induction_: imagine the property of noun phrase (Wilkenfeld and Ward, 2001; Estes and Ward, 2002), (2) _Noun phrase completion_: create a noun phrase to represent certain property, highlighted as a critical point of creativity (Kohn et al., 2011), and (3) _Property type prediction_: predict a type of property to identify the origin of property (Gagne et al., 2017).

Extensive experiments show that: (1) our automatic evaluation metric, based on the LLM-as-a judge (Zheng et al., 2023) method to detect property emergence and cancellation scores, closely matches human evaluations; (2) current LLMs, including o1 (OpenAI, 2024c), struggle to create noun phrases with truly emergent properties, often defaulting to results where one component already has the target property; and (3) our proposed method, motivated by the spread activation model (Anderson, 1983; Collins and Loftus, 1975) in cognitive psychology, improves performance across all generative tasks, indicating that considering diverse relationships between concepts enhances conceptual combination.

## 2 Related Work

### Conceptual Combination

Conceptual combination is a mental process of combining concepts. Unique features often arise from the combination, especially when the combinations of elements are unusual (Estes and Ward, 2002; Ward, 2007). People come up with novel ideas to merge conflicts; thus, it is a prominent part of creativity (Hampton, 1997). In scientific discovery, conceptual combination creates new scientific concepts (Thagard, 1984), such as "light wave." Even if nobody could observe light waves, the properties shared with waves, such as reflection, led to a theory that light is a wave, leading to much progress in science.

### Conceptual Combination in NLP

In NLP, understanding conceptual combination has been studied in two different lines. For instance, interpreting "pet bird" as "a bird kept as a pet" exemplifies _noun compound interpretation_. Alternatively, imagining "pet bird"s attributes like "probably lives in a cage" illustrates _property verification_.

Noun Compound Interpretation focuses on generating plausible meanings for noun-noun compounds, such as interpreting "dog house" as "a house for a dog". The goal is to resolve ambiguity and provide clear interpretations of compound nouns. While related to combining nouns, it

\begin{table}
\begin{tabular}{l c c c c c} \hline \multirow{2}{*}{dataset} & \# Q & \multicolumn{3}{c}{property} \\ \cline{2-6}  & com & eme & can & mc \\ \hline _Basic Concept Probing_ & & & & & \\ BERT-IS-A (2022) & 11,868 & ✓ & ✗ & ✗ & ✗ \\ COPEN (2022) & 10,624 & ✓ & ✗ & ✗ & ✗ \\ PLM-Ontology (2023) & 9,687 & ✓ & ✗ & ✗ & ✗ \\ \hline _Noun Campound Interpretation_ & & & & & \\ SemEval-D13-Task (2013) & 355 & ✗ & ✗ & ✗ & ✓ \\ ChocolateBunny (2023) & 105 & ✗ & ✗ & ✗ & ✓ \\ COMPFN (2024) & 918 & ✗ & ✗ & ✗ & ✓ \\ \hline _Conceptual Combination_ & & & & & \\ BIG-Bench-CC (2023) & 103 & \(\triangle\) & ✓ & ✗ & ✓ \\ FakeRef (2024) & 854 & ✓ & ✗ & ✓ & ✓ \\ CCPF (ours) & 12,315 & ✓ & ✓ & ✓ & ✓ \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of our dataset with previous concept property probing and noun compound interpretation datasets. Each column denotes as follows. # Q: the number of questions with gold answers. com, eme, and can denotes component, emergent and canceled property, respectively. MC: whether the combinations of multiple concepts are considered. In Big-bench-CC, \(\triangle\) indicates that component properties are present in a dataset as wrong options, but it does not specify what properties are component properties.

 

mainly emphasizes understanding the relationships between the words Hendrickx et al. (2013); Shwartz and Waterson (2018); Shwartz and Dagan (2019); Coil and Shwartz (2023); Rambelli et al. (2024).

The second line of research focuses on the properties of conceptual combinations. BIG-bench authors (2023) introduced a benchmark for identifying emergent properties in conceptual combinations. However, their approach has limitations: (1) it only identifies emergent properties, (2) the benchmark is limited to multiple-choice selection tasks without generative tasks, and (3) it does not involve creating conceptual combinations. In contrast, our approach (1) uses property types as part of the constraints and (2) includes both generative tasks and multiple-choice selection tasks. Concurrent work Ross et al. (2024) deeply explores LLM's understanding of modifier adjective-noun conceptual combination with a membership inference question such as 'Is a fake reef?'. The work shares interest with us about property cancellation. However, there's still a gap in evaluating LLMs for conceptual combination in generative way.

## 3 The CCPT Benchmark

### Task Formulation

Before describing our task formulation, we begin with its key elements: \(\{\mathcal{N}(\mathcal{H},\mathcal{M}),\mathcal{P},\mathcal{T}\}\), with two examples of "peeled apple" and "egg in the batter" for better understanding.

* **Noun phrase**\(\mathcal{N}\) is noun phrase comprised of two concepts (e.g., "peeled apple" and "egg in the batter").
* **Head noun**\(\mathcal{H}\) provides the central meaning of the combination \(\mathcal{N}\) (e.g., "apple" and "egg").
* **Modifier**\(\mathcal{M}\) adds additional meaning to the head noun \(\mathcal{H}\) (e.g., "peeled" and "batter").
* **Property**\(\mathcal{P}\) is attributed to the combination \(\mathcal{C}\) (e.g., "white" for a peeled apple; "nutritious" for the egg in the batter).
* "white" for a peeled apple, "component property"
- "nutritious" for the egg in the batter)

Based on the above configuration, we devise three types of tasks by hiding one configuration and using the remaining configurations to predict it. This task formulation includes two generative tasks: property induction and noun phrase completion, and one classification task: property type prediction.

#### 3.1.1 Property Induction (\(\mathcal{V},\mathcal{T}\rightarrow\mathcal{P}\))

The interpretation of novel combinations by listing property has been a well-explored area in previous research on human cognition Wilkenfeld and Ward (2001); Estes and Ward (2002), with a significant focus on how people comprehend novel expressions in sentences and discourse Swinney et al. (2007). In this task, LLMs are instructed to

\begin{table} max width= 

\end{table}
Table 2: Each example consists of a task instruction, input, and responses with their grading (Tasks 1 and 2) and correctness (Task 3). The explanations in italics are added by the authors for clarification. Note that LLMs are evaluated solely based on their responses.

 

identify the properties of combinations that align with the given property types (emergent, canceled property).

For example, in Table 2 (top), LLMs can identify an emergent property by finding a property that is not present in the individual concept but emerges in the noun phrase, such as "unstable" for "apple on a toothpick".

#### 3.1.2 Noun Phrase Completion (\({\cal H},{\cal P},{\cal T}\rightarrow{\cal N}\))

Generating new concepts by combining existing ones is key to creativity Kohn et al. (2011). In this task, LLMs generate noun phrases by adding modifiers to head nouns to represent emergent or canceled properties. For emergent properties, the modifier should not imply the property on its own, but the combination should. In contrast, for canceled properties, the modifier effectively negates the head noun's property.

For example, in Table 2 (middle), to represent the emergent property "rare" with the head noun "apple", LLMs may consider a noun phrase like "blue apple". The modifier "blue" does not directly suggest the property "rare", but when combined with "apple", it possesses the given property.

#### 3.1.3 Property Type Prediction (\({\cal N},{\cal P}\rightarrow{\cal T}\))

Understanding how combined concepts gain or lose certain properties is an essential process for concept theories Gagne et al. (2017).

In this task, LLMs identify how a property relates to a noun phrase. For example, in Table 2, the property "good for health" in "a green apple" is an _component property_, as the apple is already good for health itself.

### Data Collection

We propose the dataset, Conceptual Combination with Property Type (CCPT), to address the three tasks in SS 3.1. We both use a automated methods and human filtering. Detailed sources of textual corpora are described in Appendix A.1.

**Step 1. Extract and Filter Combinations.**

First, we extract noun phrases from the corpus. To obtain noun phrases with a property hint in a sentence, we extract sentences that contain "like" or "as" which compare one concept to another. These comparisons explicitly highlight the characteristics (e.g., "our economy will be as unstable as an apple on a toothpick"). This yields \(\overline{51.0M}\) comparative sentences--8% of the paragraph in the original corpus contained such sentences.

To avoid common expressions, such as proper nouns or idioms, whose meanings can be memorized from training corpus, we exclude any N-grams found in ConceptNet. We utilize uni-gram concept set from ConceptNet and use it as basic concepts. In the end, we collected 136.0K comparative sentences containing combination made up of two uni-gram concepts.

**Step 2. Extract and Filter Properties.**

**Emergent & Component Candidates.** Property extraction by syntactic patterns such as "ADJ/ADV like C" or "as ADJ/ADV as C" often misses implicit properties (e.g., "the storm was almost like a raging bull") or multi-word properties (e.g., "they crashed together like a boat on the rocks").

To address this, we use GPT-4o-mini OpenAI (OpenAI, 2024a) to extract 10 properties from given comparative sentence for each combination. Then VERAT-T5-XXL Liu et al. (2023) filters out unlikely properties (with an alignment score under 0.7), resulting in 41.6K noun phrases and 211.0K properties.

Then we extract candidates for each property type. For emergent properties, with GPT-4o-mini, we select instances where neither the head noun \({\cal H}\) nor the modifier \({\cal M}\) already possess the property \({\cal P}\) of the noun phrase \({\cal N}\). We then limit the selection to at most five instances per noun phrase, resulting in 3,851 candidates. For component properties, we randomly sample 10K candidates.

**Canceled Candidates.** Since we cannot gather canceled properties from the corpus, we use two additional sources to collect the properties of head nouns: one for the "HasProperty" relations from ConceptNet and another from GPT-4o-mini, following Step 2. We randomly sample 2K noun phrases and gather up to 10 properties for each head noun from each source. Then, VERA-T5-XXL filters out the most likely properties (with an alignment score above 0.7), resulting in 2K noun phrases and a total of 23K properties.

With GPT-4o-mini, we select instances where the noun phrase \({\cal N}\) no longer possesses the property \({\cal P}\) of the head noun \({\cal H}\). Then, we select at most 5 data with the same noun phrase. We then limit the selection to at most five instances per noun phrase, resulting in 4,457 candidates.

**Step 3. Annotate Property Types.** The annotation process consists of three stages: first, assign a 5-point Likert scale of relevance score ranging from Highly Negative to Highly Positive; second, annotate the property type; third, find the toxicity.

In total, we have 12,315 data points, the type of property \({\cal T}\) consisting of 2501 emergent prop 

erties, 1613 canceled properties, and 8201 component properties. Further details on the annotation procedure and inter-annotator agreement are provided in Appendix A.3.

### Co-occurence of Concepts

To analyze how novel the noun phrases in our dataset is, we form a co-occurrence matrix based on PMI scores from the Google Books Ngram Corpus. The PMI formula is:

\[{\text{PMI}(w,c)=\log_{2}\frac{P(w,c)}{P(w)P(c)}},\]

where noun phrases with zero frequency are discarded.

When comparing our dataset to bi-gram concepts from ConceptNet, Figure 3 shows that combinations in our dataset have lower co-occurrence than those in ConceptNet. The average PMI for our dataset is -1.03, compared to 5.78 for ConceptNet, suggesting that combinations are more novel.

## 4 Benchmarking LLMs and Humans

We comprehensively assess 6 current LLMs with different architectures and sizes, including both open-source and closed-source models: LLaMa-3.1-70B-Instruct Dubey et al. (2024), Qwen2.5-72B-Instruct Team (2024), GPT-4o-20240513 OpenAI (2024b), Claude-3.5-Sonnet Anthropic (2024), o1-mini-2024-09-12 OpenAI (2024d), and o1-2024-12-17 OpenAI (2024c). The test instances are a randomly sampled representative sample of data instances; N=200 for Property Induction-Emergent, N=167 for Property Induction-Canceled, N=167 for Non Phrase Completion-Emergent.

### Methods

To provide background knowledge on conceptual combinations, we use a consistent system prompt across all baselines and tasks. This prompt includes explanations of key concepts such as conceptual combinations, head nouns, modifiers, and different types of properties. Implementation details about model are explained in Appendix B.1. Prompts are detailed in Appendix B.2.

**Base.** Base prompting method is to evaluate LLMs' ability to generate their responses without reasoning steps.

**Chain-of-Thought (CoT).** We adopt chain-of-thought Kojima et al. (2022) method to induce LLMs to generate the reasoning steps before pro

Figure 3: Distributions of Pointwise Mutual Information (PMI) on log-2 scale based on the Google Books N-gram Corpus.

Figure 2: Overview of our data collection pipeline for conceptual combination through automated and human-driven data annotation.

 

ducing the final answer by adding the phrase "Let's think step by step" at the end of the questions.

**Spreading Activation (S.A.) (Ours)** Spreading activation is a cognitive model to search through networks of related ideas or concepts Anderson (1983); Collins and Loftus (1975). One concept in the mind activates another concept through the pathway of association. It becomes easier for people to combine these related concepts together and understand them as a relationship.

Motivated by spreading activation theory, we propose a novel method to iteratively explore relationships between concepts. In Algorithm 1, the initial concept set (\(C_{0}\)) and objective (\(O\)) are first defined. For example, generating the emergent property of a peeled apple is represented as \(C_{0}=\{\text{peeled, apple}\}\), with \(O\) defined as "find relationships between 'peeled' and 'apple'." Relevant concepts (\(A_{c}\)) are then activated for each component in the concept set (\(C_{t}\)) using either LLM or a graph-based approach such as ConceptNet. A filtering step selects components from \(C_{t}\) based on their relatedness to \(C_{0}\). If there is no significant difference between \(C_{t}\) and \(C_{t+1}\), the loop terminates. After iteratively expanding the set, the LLM generates the final answer based on the intermediate concept set \(C_{T}\). Specifically, we set the maximum iteration steps (\(T\)) to 5 and the convergence threshold (\(\epsilon\)) to 0.1.

```
Input: Model \(\mathcal{M}\), Initial set of seed concepts \(C_{0}\), Prompts \(\{p_{act},p_{fil}\}\), Concept graph \(G\) Parameter: Maximum iterations \(T\), Convergence threshold \(\epsilon\), Use language model \(U_{L}\), Use concept graph \(U_{C}\) Result: Related concept set \(C_{T}\) # Retrieve relevant concepts \(A_{c}\) Function(Activate(\(c\))): begin \(A_{c}\leftarrow\emptyset\) if\(U_{L}\)then \(A_{c}\gets A_{c}\cup\mathcal{M}(p_{act},c)\) endifif\(U_{C}\)then \(A_{c}\gets A_{c}\cup G\).query(\(c\))  end return\(A_{c}\)  end if # Filter each concept in \(C_{t}\) w.r.t. \(C_{0}\) Function(Filter(\(C_{t},C_{0}\))): Begin \(\mathcal{M}(p_{fil},C_{t},C_{0})\)  endforiteration\(t=0\)to\(T\)do \(C_{t+1}\gets C_{t}\) foreachconcept\(c\in C_{t}\cup C_{0}\)do \(A_{c}\leftarrow\texttt{Activate}(c)\) \(C_{t+1}\gets C_{t+1}\cup A_{c}\) end \(C_{t+1}\leftarrow\texttt{Filter}(C_{t+1},C_{0})\setminus C_{0}\) \(\Delta=1-\frac{|C_{t}\cap C_{t+1}|}{|C_{t}\cup C_{t+1}|}\) if\(\Delta<\epsilon\)then break  end if  end for\(C_{T}\gets C_{t+1}\) return\(C_{T}\)
```

**Algorithm 1**Spreading Activation

**Multi-Oracle.** The best result among the multiple efforts can be seen as the upper-bound performance for each LLM. For the research purpose, we include this score by selecting the best score among the multiple solutions' (N=5) scores, presented with a gray background in Table 3.

**Gold.** We provide the score assigned to our annotated dataset as an upper-bound performance score for our tasks.

### Human Responses

We recruited 5 native English speaker students through offline advertisement on a university campus. None of the students knew the researchers or had heard about conceptual combinations before. We verbally introduced the meaning of conceptual combination, and by e-mail, test sheets for each task were sent. The students solved the tasks in their own time and place. 1 to 3 students solved a single test sheet. We report the best result. Question format is provided in Appendix B.3.

### Evaluation Metric

**Generative Tasks.** It can be challenging to determine whether emergence or cancellation occurs. For instance, an overripe apple may be more strongly associated with red than a regular apple, but this doesn't clearly indicate emergence. Similarly, determining cancellation presents the same difficulty. To address this, we propose two metrics based on a continuous relevance scoring. Both human judges and the LLM-as-a-judge approach Zheng et al. (2023) are utilized to evaluate open-ended generative responses. These metrics are applied to the Property Induction and Noun Phrase Completion.

* The **emergence score** (\(\mathcal{E}\)) measures how suddenly properties arise when concepts are combined, compared to the properties of the individual concepts.
* The **cancellation score** (\(\mathcal{C}\)) reflects how much a property is diminished or canceled when 

concepts are combined, compared to their individual properties.

Specifically, each score is defined as

\[R_{\mathcal{H},\mathcal{M},\mathcal{P}}=\max\{R_{\mathcal{H}, \mathcal{P}},R_{\mathcal{M},\mathcal{P}}\}, \tag{1}\] \[\mathcal{E}=\max\{R_{\mathcal{N},\mathcal{P}}-R_{\mathcal{H}, \mathcal{M},\mathcal{P}},0\},\] (2) \[\mathcal{C}=\max\{R_{\mathcal{H},\mathcal{M},\mathcal{P}}-R_{ \mathcal{N},\mathcal{P}},0\}, \tag{3}\]

Here, \(R_{\mathcal{X},\mathcal{P}}\) represents the relevance score, indicating how strongly concept \(\mathcal{X}\) possesses property \(\mathcal{P}\), measured by human judges and GPT-4o (OpenAI, 2024b) on a scale from 0 to 1. Exceptionally, for the Noun Phrase Completion, cancellation score is not measured since it easily occurs by adding antonym of given canceled property (e.g.,

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Property Induction**} & \multicolumn{6}{c}{**Noun Phrase Completion**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(1) Emergent**} & \multicolumn{5}{c}{**(2)****Canceled**} & \multicolumn{5}{c}{**(3) Emergent**} \\ \cline{2-9} _GPT-4o_ & \(R_{\mathcal{M},\mathcal{P}}\!\downarrow\) & \(R_{\mathcal{N},\mathcal{P}}\!\uparrow\) & \(\mathcal{E}\!\uparrow\) & \(R_{\mathcal{M},\mathcal{M},\mathcal{P}}\!\uparrow\) & \(R_{\mathcal{N},\mathcal{P}}\!\downarrow\) & \(\mathcal{C}\!\uparrow\) & \(R_{\mathcal{M},\mathcal{P}}\!\downarrow\) & \(R_{\mathcal{N},\mathcal{P}}\!\uparrow\) & \(\mathcal{E}\!\uparrow\) \\ \hline Base & 49.2 & 84.1 & 36.3 & 70.7 & 29.4 & 43.9 & 54.3 & 69.0 & 20.4 \\ CoT & 45.4 & 84.1 & 41.3 & 68.0 & 21.5 & 47.0 & 53.0 & **77.8** & 26.7 \\ S.A.w/ LLM & **43.0** & 84.4 & **43.3** & **79.0** & 23.1 & **57.6** & 52.6 & 69.6 & 22.0 \\ S.A.w/ ConceptNet & 44.8 & **86.1** & 42.2 & 75.9 & **19.6** & 56.5 & **48.7** & 65.4 & 20.2 \\ S.A.w/ Both & 47.4 & 84.6 & 40.2 & 77.8 & 22.8 & 56.1 & 50.9 & 75.9 & **29.6** \\ \hline _Human_ & 42.8 & 84.8 & 44.3 & 73.5 & 22.4 & 52.6 & 43.7 & 79.8 & 39.0 \\ \hline _Gold_ & 34.4 & 87.2 & 52.8 & 79.6 & 21.1 & 59.0 & 33.0 & 90.6 & 57.6 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Human evaluation of generative results on sub-sampled test instances (N=30). The relevance scores are calculated by the average scores of two annotators. **Bold numbers** indicate the best scores, while underline numbers are the second-best.

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & \multicolumn{8}{c}{**Property Induction**} & \multicolumn{6}{c}{**Noun Phrase Completion**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(1) Emergent**} & \multicolumn{5}{c}{**(2)****Canceled**} & \multicolumn{5}{c}{**(3)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(3) Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(1) Emergent**} & \multicolumn{5}{c}{**(2)****Canceled**} & \multicolumn{5}{c}{**(3)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(3)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(4)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(5)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(5)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(6)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(7)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(8)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(8)****Emergent**} \\ \cline{2-9}  & \multicolumn{5}{c}{**(9)****Emergence**} \\ 

to cancel "yellowness" of "banana", add "brown" to "banana"). Detailed instructions for both manual evaluation and LLM-as-a-judge are provided in Table 17 and Figure 8.

**Classification Task.** In this classification setup, we evaluate performance based on accuracy (%) in the Property Type Prediction. The classification involves four categories: "canceled property", "emergent property", "component property", and "others property". The "others property" category is specifically for properties unrelated to the combination and its components.

## 5 Results

### Generative Task Result

Table 3 presents the experimental results in the generative setting for the two tasks, Property Induction and Noun Phrase Completion, evaluated based on LLM-as-a-judge. Table 4 shows the corresponding results based on human-judge evaluations for the same tasks.

**Which property do the LLMs generate better: emergent or canceled?** As shown in Table 3-(1) and (2), all baseline models find it more challenging to generate _emergent properties_ than _canceled properties_. Humans outperform LLMs in generating emergent properties from noun phrases but perform worse in handling canceled properties. For emergent properties, GPT-4o-S.A. w/ ConceptNet achieves a significantly higher relatedness score between the property and each component, \({\cal R}_{{\cal H},{\cal M},{\cal P}}\), exceeding the gold score by 12.3 points. In contrast, the relatedness score between the property and the noun phrase, \({\cal R}_{{\cal N},{\cal P}}\), is only 1.6 points lower. While LLMs generate properties that align well with the noun phrase, they tend to rely on properties already associated with the individual components.

**How well do LLMs create conceptual combinations?** In Table 3-(3), it is difficult for all baselines to come up with a modifier \({\cal M}\) that lacks a given property on its own but exhibits that property when combined with a head noun. The emergence scores \({\cal E}\) for this task are 27 or lower across all models. Humans are better than LLMs at generating combinations that exhibit a given emergent property.

**Which generative task do LLMs excel at?** We compared two tasks: generating emergent properties (Table 3-(1)) and generating a noun phrase (Table 3-(1)). All baseline models achieve lower emergence scores (\({\cal E}\)) in the noun phrase completion task. This suggests that LLMs find it more challenging to create a noun phrase that accurately captures an emergent property than to identify an emergent property from a noun phrase.

### Classification Task Result

Table 5 presents the experimental results for the Property Type Prediction. Given a noun phrase and a property, GPT-4o classifies the property into one of four categories: emergent, component, canceled, or others (where others is unrelated to both the combination and its components).

**Do LLMs identify the type of property well?** In determining whether a noun phrase has a given property, GPT-4o achieves an accuracy of 82.6% \(((95.6\%+69.6\%)\div 2)\). However, in predicting the type of property, GPT-4o is correct only 56.4% of the time, falling behind human accuracy (81%), as noted in Appendix B.4. Compared to its accuracy in identifying emergent properties, its performance across other property types lags significantly. These results suggest room for improvement in understanding different property types.

## 6 Analysis

### Relevance between LLM-as-a-jadge metric and Manual Evaluation

As shown in Figure 4, we compare the LLM judge's metrics with human evaluations to verify the agreement between them. We randomly selected 300 pairs from CCPT, which consists of pairs from \({\cal H}-{\cal P}\), \({\cal M}-{\cal P}\), and \({\cal N}-{\cal P}\), covering both emergent properties (50 samples) and canceled properties (50 samples). Human raters, recruited through Amazon Mechanical Turk, as detailed in Section B.5, are asked to rate the relevance of each pair using the same instructions provided to the LLM judge. Each problem is rated by three different raters. We

\begin{table}
\begin{tabular}{|l|p{28.5pt}|p{28.5pt}|p{28.5pt}||p{28.5pt}|p{28.5pt}|} \hline  & & \multicolumn{4}{c|}{Predicted Types} \\ \cline{3-6}  & & **Emergent** & **Component** & **Canceled** & **Others** \\ \hline \multirow{3}{*}{**CPT**} & **Emergent** & **90.0** & 4.4 & 2.0 & 3.6 \\ \cline{2-6}  & **Component** & 59.6 & **37.2** & 1.2 & 2.0 \\ \cline{2-6}  & **Canceled** & 13.6 & 15.6 & **45.2** & 25.6 \\ \cline{2-6}  & **Others** & 26.0 & 5.6 & 15.2 & **53.2** \\ \hline \end{tabular}
\end{table}
Table 5: Classification results of GPT-4o for property type prediction based on 1000 sampled instances (250 samples per property type). We present the average accuracy (%). Cells with a green background indicate cases where the model correctly predicts that combinations possess a property (95.6% acc), while cells with a red background indicate cases where it correctly predicts that combinations do not possess a property (69.6% acc).

 

calculated the Pearson and Spearman correlation coefficients between the LLM judge's scores and human ratings, which are 0.85 and 0.83, respectively. These strong correlations indicate a high level of agreement, demonstrating the effectiveness of using LLM for relevance scoring.

### Analysis for Spread Activation Method

First, to evaluate the impact of multiple iterations on performance, we adjust \(T\) from 1 to 5 and gather the answers for each \(C_{T}\). Table 7 highlights that as the number of iterations increases, the performance of each task gradually improves. This indicates that iteratively propagating relevant concepts is effective in our conceptual combination tasks.

Second, our spread activation method consists of the functions \(\texttt{Activate}(\cdot)\) and \(\texttt{Filter}(\cdot,\cdot)\). To investigate the contribution of Filter on performance, we conducted an ablation study by comparing scores with and without Filter in the spread activation method. As shown in Table 6, performance decreases when Filter is removed. This suggests that iteratively eliminating distractors improves overall performance.

## 7 Conclusion

In conclusion, our work introduces CCPT, a conceptual combination dataset designed to evaluate LLMs' ability to process conceptual combinations. CCPT comprises 12,315 annotated instances of noun phrases, properties, and property types. Based on CCPT, we propose three downstream tasks: property induction, noun phrase completion, and property type prediction. To assess generative performance, we introduce two automatic evaluation metrics--emergence and cancellation scoring--which closely align with human evaluations. Additionally, we propose a novel evaluation method inspired by cognitive psychology models. Our findings indicate that (1) LLMs struggle more with generating emergent properties than with canceled ones. Furthermore, (2) generating a noun phrase that exhibits a given emergent property proves more challenging than generating the property itself. Notably, (3) GPT-4o struggles to determine property types compared to humans. Finally, (4) our proposed spread activation method achieves the highest performance among the evaluated approaches due to its iterative retrieval of relevant concepts and filtering process.

Figure 4: Correlation between LLM-as-a-judge and human ratings in relevance score, assessing how strong a concept \(\mathcal{X}\) possesses a property \(\mathcal{P}\). To avoid overlapping points, random jitters sampled from \(\mathcal{N}(0,0.05^{2})\) are added to LLM-as-a-judge and human ratings after fitting the regression.

 



### Limitations

We acknowledge few potential limitations of our research. (1) There is fundamental diversity in people's mental representations of the world, especially across cultural contexts. The notions of "property" and "property type" in our dataset may implicitly reflect the commonsense knowledge of the annotators' demographic group. Moreover, the approval of our data through the MTurk study may primarily reflect the commonsense of the Turkers. Future work could further explore the relationship between conceptual combination understanding and cultural divergence in concepts. (2) Homonyms can introduce misleading effects on the evaluation process. If the grader misinterprets the definition of concept from the solver's intention, the solver's performance may not be fully captured. (3) Our data generation pipeline employs comparative sentences for efficiency. However, the inherent nature of comparative sentences may introduce skewness, favoring certain types of properties over others.

## Ethics Statement

The authors checked all examples and found no personal identifying information (PII). As addressed in SS 3.2, we also eliminated the offensive contents manually.

## Acknowledgements

We thank the anonymous reviewers and Yunah Jang for their valuable comments. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II191082, SW StarLab), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2023R1A2C2005573), the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education(RS-2023-00274280), Korea Radio Promotion Association (Development of Intelligent Docent Service for Information-Disadvantaged Groups), and the SNU-Global Excellence Research Center establishment project. Gunhee Kim is the corresponding author.

## References

* Anderson (1983) John R Anderson. 1983. A spreading activation theory of memory. _Journal of verbal learning and verbal behavior_, 22(3):261-295.
* Anthropic (2024) AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. _Claude-3 Model Card_.
* Authors (2023) BIG-bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Coil and Shwartz (2023) Albert Coil and Vered Shwartz. 2023. From chocolate bunny to chocolate crocodile: Do language models understand noun compounds? In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2698-2710.
* Collins and Loftus (1975) Allan M Collins and Elizabeth F Loftus. 1975. A spreading-activation theory of semantic processing. _Psychological review_, 82(6):407.
* Coutanche et al. (2019) Marc N Coutanche, Sarah Solomon, and Sharon L Thompson-Schill. 2019. Conceptual combination in the cognitive neurosciences.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahale, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Estes and Glucksberg (2020) Zachary Estes and Sam Glucksberg. 2020. Relevance and feature accessibility in combined concepts. In _Proceedings of the Twenty-first Annual Conference of the Cognitive Science Society_, pages 149-154. Psychology Press.
* Estes and Ward (2002) Zachary Estes and Thomas B Ward. 2002. The emergence of novel attributes in concept modification. _Creativity Research Journal_, 14(2):149-156.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. _arXiv preprint arXiv:1805.04833_.
* Fleiss (1971) Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. _Psychological bulletin_, 76(5):378.
* Gagne et al. (2017) Christina L Gagne, Thomas L Spalding, and Matthew Kostelecky. 2017. Conceptual combination, property inclusion, and the aristotelian-thomistic view of concepts. _Compositionality and concepts in linguistics and psychology_, pages 223-244.
* Gerlach and Font-Clos (2020) Martin Gerlach and Francesc Font-Clos. 2020. A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics. _Entropy_, 22(1):126.
* Hampton (1997) James A Hampton. 1997. Emergent attributes in combined concepts.
* Hendrickx et al. (2013) Iris Hendrickx, Zornitsa Kozareva, Preslav Nakov, Dirmuid O Seaghdha, Stan Szpakowicz, and Tony Veale. 2013. SemEval-2013 task 4: Free paraphrases of noun compounds. In _Second Joint Conference on Lexical and Computational Semantics (*SEM)_,

Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_, pages 138-143, Atlanta, Georgia, USA. Association for Computational Linguistics.
* Kohn et al. [2011] Nicholas W Kohn, Paul B Paulus, and Runa M Korde. 2011. Conceptual combinations and subsequent creativity. _Creativity Research Journal_, 23(3):203-210.
* Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Lin and Ng [2022] Ruixi Lin and Hwee Tou Ng. 2022. Does bert know that the is-a relation is transitive? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 94-99.
* Liu et al. [2023] Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023. Vera: A general-purpose plausibility estimation model for commonsense statements. _arXiv preprint arXiv:2305.03695_.
* OpenAI [2024a] OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. [https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/). Accessed: 2024-07-18.
* OpenAI [2024b] OpenAI. 2024b. Hello gpt-4o. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-05-13.
* OpenAI [2024c] OpenAI. 2024c. Introducing openai o1. [https://openai.com/oil/](https://openai.com/oil/). Accessed: 2025-02-08.
* OpenAI [2024d] OpenAI. 2024d. Openai o1-mini. [https://openai.com/index/openai-ol-mini-adavancing-cost-efficient-reasoning-to-use-of-the-one-in-a-volume-cost-efficient-reasoning-to-use-of-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win](https://openai.com/index/openai-ol-mini-adavancing-cost-efficient-reasoning-to-use-of-the-one-in-a-volume-cost-efficient-reasoning-to-use-of-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-on-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-a-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win-and-win-the-win-and-the-win) 

Human Language Technologies_, pages 5927-5934, Online. Association for Computational Linguistics.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_, pages 19-27.

## Appendix A Data Details

### Textual Corpora

We utilize English textual corpora from datasets and websites across different domains including books and news. Our source is comprised of Toronto Book Corpus Zhu et al. (2015), WritingPrompts Fan et al. (2018), Gutenberg Corpus Gerald and Font-Clos (2020), MediaSum News Dialogue Zhu et al. (2021), Wikipedia-2408231, r/FUNNY2, r/FantasyWriters3, and r/OCPoetry4.

Footnote 1: [https://dumps.wikimedia.org/enwiki/latest/](https://dumps.wikimedia.org/enwiki/latest/)

Footnote 2: [https://www.reddit.com/rf/funny/](https://www.reddit.com/rf/funny/)

Footnote 3: [https://www.reddit.com/rf/fatasywriters/](https://www.reddit.com/rf/fatasywriters/)

### Dataset Construction Statistics

In Table 8, we describe the size of the dataset during the data collection pipeline.

### Annotation Procedure

We hire skilled raters in Amazon Mechanical Turk (MTurk) to annotate the property type. For emergent properties and canceled properties, we assign three annotators per instance, and used majority label for the final property type. For component properties, due to cost constraints, we assign one annotator per instance.

Annotators were selected based on their success in a qualification task Human Intelligence Task or HIT), which assessed their ability to distinguish property types. This qualification task included 10 thoroughly verified examples, with a payment of $1.00. We required annotators to be from English-speaking countries (AU, CA, NZ, US, GB), have completed more than 10,000 HITs, and maintain a HIT approval rate greater than 98%.

After qualification, annotators received detailed instructions on conceptual combinations with examples. They answered three questions: (1) Relevance Rating - Annotators rated the relevance of each concept (\({\cal N},{\cal H},{\cal M}\)) to a given property (\({\cal P}\)) to encourage careful consideration of the data. (2). Property Type Annotation - For emergent and component properties, annotators chose: (1) emergent, (2) component, or (3) other. For canceled properties, they chose: (1) canceled, (2) component, or (3) other, as illustrated in Figures 5 and 6. For component property candidates there is a third question about locating the source of the property from the following options: (1) noun phrase \({\cal N}\), (2) head noun \({\cal H}\) and noun phrase \({\cal N}\), (3) modifier \({\cal H}\) and 

nown phrase \({\cal N}\), (4) All of \({\cal N},{\cal H},{\cal M}\), (5) others. (3) Toxicity: The annotators are asked to check a box if the data is toxic. Finally, the Fleiss' \(\kappa\) score of three-way classification during the annotation phase is 0.312 for emergent property data and 0.410 for canceled property data. In CCPT, we contain disaggregated human annotations for all questions.

Next, we calculate inter-annotator result for the final dataset. For emergent and component properties, we randomly selected 100 samples for this task, each reviewed by two annotators who had not participated in the original annotation. The Fleiss' \(\kappa\) score for this binary classification task was 0.498, indicating agreement levels ranging from "moderate" to "substantial." For canceled properties, we also selected 100 samples, with two annotators reviewing each example. The Fleiss' \(\kappa\) score for this binary classification is 0.505, indicating agreement between "moderate" and "substantial" levels.

## Appendix B Experimental Details

### Implementation Details

We collect responses using Nucleus sampling with \({\cal T}=0.7\) and \(p=0.95\), by selecting the most likely sequence. Model responses are generated using three different seeds, and we report the average scores along with the standard error of the mean (SEM).

### Details in prompting

In this section, we explain our task instruction templates. For each method, we give a correct and wrong answers for one example.

For the system prompt, refer to Table 9. For the prompt given to Base method, refer to Table 10 and 13. For the CoT-prompting, refer to Table 11 and 14. Spread-Activation method uses a prompt in Table 12 and 15. For the propery type prediction task, refer to Table 16.

Our prompt for LLM-as-a-judge is in Table 17. The prompt provides the fine-grained scoring criteria from 1 to 10 and three scored examples.

### Collecting human solutions

The students reported to need 25 to 45 minutes per 30 questions. We paid $78 per 167-question test sheet and $94 for 200-question test sheet; This exceeds the minimum wage in the country the authors and students are located in. The question format is as below.

* emergent property)**

Conceptual combination: '{**noun phrase**}', Type-of-property: '**emergent property'**

**Q. What is a property of '{**noun phrase**}' that '{head noun**}' or '{**modifier**}' doesn't have? A:**

* canceled property)**

Conceptual combination: '{**noun phrase**}', Type-of-property: '**canceled property'**

**Q. What is a canceled property which is a property that belongs to '{head noun}' or '{**modifier**}' but does not belong to a noun phrase '{nounsphrase}''? A:**

**(Noun phrase completion)**

Head noun: '{**head noun**}', Property: '{**property**}'

**Q. What is a noun phrase using '{root}' to be '{property}'? A:**

### Recruiting Participants for Property Type Classification

To check human ability for property type prediction in Section 5.2, we recruited capable test participants who did **not** participate by Amazon Mechanical Turk (MTurk).

We provided the participants with the conceptual combination and property and asked them to classify the property type. The definition of the conceptual combination and the property types are described in the instructions. Three annotated data instances (one per each property type) are also provided. Instruction is shown in Figure 7. As a result, when we regard the majority vote as a human-annotated label, we get an accuracy of 87% for the emergent property, 79% for the component property, and 76% for the canceled property, with a total accuracy of 81%.

We prepared the qualification Human Intelligence Task (HIT), comprised of 10 thoroughly verified examples with a payment of $1.0. We re

\begin{table}
\begin{tabular}{l r} \hline  & \# of Data \\ \hline Corpora & -51,000,000 \\ Extracting Comparative Sentences & 136,300 \\ Extracting and Filtering & 41,660 \\ Property Type Tagging (w/ Toxic Filtering) & 12,315 \\ \hline \end{tabular}
\end{table}
Table 8: The size of the dataset during the data collection pipeline is represented by the number of remaining sentences or noun phrases at each step. The term “# of data” denotes the amount of data left after each processing stage.

 



\begin{table} \end{tabular}

* The Task Instruction for Base Method in Noun Phrase Completion.

\begin{table} \end{table}
Table 10: Task instruction for Base Method in Noun Phrase Completion.

\begin{table} \end{table}
Table 9: System prompt for background of conceptual combination.



The first step is to find a "good" answer, and the second step is to find a "good" answer. The second step is to find a "good" answer.

\begin{table}
\begin{tabular}{l} \hline \hline \multicolumn{1}{c}{**Task Instruction for Base Method in Property Induction (Emergent)**} \\ \hline Instructions: & \\
1. You are given a combination of concepts. Your task is to generate emergent property of a combination. & \\
2. Find a property that does not belong to any of the individual component in the combination but emerges when the words are combined. & \\
3. Use the previous examples to learn how do this. & \\
4. Answer in dictionary format: [{”property”: ”[{generated_property}]|^}]. Do not include other formatting. & \\ \textless{Example l>} & - Combination: Brown apple & \\ - Correct answer: [{”property”: “unappetizing’}] & \\ Above answer is correct because property “unappetizing” does not belong to either “brown“ and “apple”, but belong to brown apple & \\ \textless{Example l>} & - Combination: burned banknote & \\ - Wrong answer: [{”property”: “useless”}] & \\ Above answer is wrong because “burned” something can directly belong to property “useless”. Ensure that the emergent property you generate does not directly describe any of the individual words but is a characteristic of the combination as a whole. & \\ \end{tabular}
\end{table}
Table XIII: Task instruction for Base Method in Property Induction.

 

The first (or more), the second (or more) is a "bunch". The second (or more) is a "bunch".

\begin{table}
\begin{tabular}{l} \hline \hline Instructions: \\
1. You are given a combination of concepts and a set of relevant concepts to solve a task. Your task is to generate emergent property of a combination. \\
2. Find a property that does not belong to any of the individual component in the combination but emerges when the words are combined. \\
3. Cone up with your reasoning process before giving your final answer. \\
4. Final answer should follow dictionary format: [{ ”property”: ‘{[generated_property]}”}]. Do not include other formatting. \\ <Example 1> \\ - Combination: Brown apple \\ - Relevant concepts: [’fruit’, 'apple', 'core', 'cider’] \\ - Correct answer: The goal is to find an emergent property of “brown apple” that does not exist in “brown” or “apple” individually. “Fruit” and “apple” describe general attributes, so they are excluded. “Core” and “Cider” are unrelated to “brown apple.” “Withered” relates to a decayed state, which applies to “brown apple” bat not to “brown” or “apple” alone. To interpret “witnessed” naturally, we select “unappetizing” as the emergent property. So the answer is [{”property”: ‘unsuppetizing”}] \\ Above answer is correct because property “unappetzing” does not belong to either “brown” and “apple”, but belong to brown apple \\ -Example 2> \\ - Combination: burned banknote \\ - Relevant concepts: [’paper’, 'ash’, ’money’, ‘value’] \\ - Wrong answer: The goal is to find an emergent property of a “burned banknote” that does not exist in either “burned” or “banknote” individually. The attributes “paper” and “nonney” describe general properties of a banknote, so they are excluded. Likewise, “ash” describes a general property of something that is burned, so it is also excluded. While a banknote possesses “value,” this characteristic disappears once the banknote is burned. Consequently, to convey the idea of “valueless” naturally, we choose “useless” as the emergent property. So the answer is is: [{“property”: ‘useses”}] \\ Above answer is wrong because “burned” something can directly belong to property ” uselesses”. Ensure that the emergent property you generate does not directly describe any of the individual words but is a characteristic of the combination as a whole. \\ Then let’s begin: \\ \hline \hline \end{tabular}
\end{table}
Table 15: Task instruction for Spread Activation Method in Property Induction.

 



\begin{table}
\begin{tabular}{l l} \hline \hline
**Task Instruction for Property Type Prediction** \\ \hline Instructions: & Concepts are characterized by properties. For example, the concept “a chicken in front of a fox” strongly exhibits the property ”in danger." When given a concept and a property, your task is to evaluate how much the concept has the property on a scale from 1 to 10. You should follow the format: [(’relevance’: your_relevance_score)] Use the following scoring criteria to assign a relevance score: \\ \(\langle\)[“relevance’: 1)] The concept does not have the property at all. \\ \(\langle\) ‘relevance’: 2);] The concept rarely has the property. \\ \(\langle\)[“relevance’: 4-6]| The concept sometimes has the property. \\ \(\langle\)[“relevance’: 7-8]| The concept usually has the property, but not always. \\ \(\langle\)[“relevance’: 9]] The concept almost always has the property. \\ \(\langle\)[“relevance’: 10]| The concept always has the property. \\ — \\ Examples: & — Concept: Rusty Property: Useless Relevance: [( ’relevance’: 7)] — \\ Concept: A chicken in the cage Property: In danger Relevance: [(’relevance’: 2)] — \\ Concept: A chicken in front of a fox Property: In danger Relevance: [(’relevance’: 9)] — \\ \(\cdots\) \\ Concept: Wrench Property: Destructive Relevance: \\ \hline \end{tabular}
\end{table}
Table 16: Task instruction for property type prediction.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Prompt for LLM-as-a-Judge** \\ \hline Concepts are characterized by properties. For example, the concept “a chicken in front of a fox”strongly exhibits the property ”in danger.” When given a concept and a property, your task is to evaluate how much the concept has the property on a scale from 1 to 10. You should follow the format: [(’relevance’: your_relevance_score)] Use the following scoring criteria to assign a relevance score: \\ \(\langle\)[“relevance’: 1)] The concept does not have the property at all. \\ \(\langle\)[“relevance’: 2)] The concept rarely has the property. \\ \(\langle\)[“relevance’: 4-6]| The concept sometimes has the property. \\ \(\langle\)[“relevance’: 7-8]| The concept usually has the property, but not always. \\ \(\langle\)[“relevance’: 9]] The concept almost always has the property. \\ \(\langle\)[“relevance’: 10]| The concept always has the property. \\ — \\ Examples: & — Concept: Rusty Property: Useless Relevance: [(’relevance’: 7)] — \\ Concept: A chicken in the cage Property: In danger Relevance: [(’relevance’: 2)] — \\ Concept: A chicken in front of a fox Property: In danger Relevance: [(’relevance’: 9)] — \\ \(\cdots\) \\ Concept: Wrench Property: Destructive Relevance: \\ \hline \hline \end{tabular}
\end{table}
Table 17: Prompt for LLM-as-a-judge

cruited participants from AU, CA, NZ, US, and GB, with more than 10000 HITs approved, and a HIT approval rate greater than 98%. Among 40, this process resulted in 12 participants.

After qualification, we asked raters with a payment of $0.2 per HIT. Each example was evaluated by three annotators and the inter-annotator agreement was 0.59 in Fleiss' Kappa (Fleiss, 1971).

### Evaluation of relation between LLM-as-a-Judge and Human Judge

To ensure the quality of evaluation metric in CCPT, we measure a correlation between LLM-as-a-judge and human ratings in Section 6.1. Like the previous subsection, we hire capable raters in Amazon Mechanical Turk (MTurk) who did **not** participate in the data annotation process before this test.

The basic qualifications are also made by nationality (AU, CA, NZ, US, and GB), the number of HITs approved (10000), and the HIT approval rate (greater than 98%). We performed a qualification test with a payment of $0.1 and chose 12 raters among the 85 applicants who had completed more than 5 qualification HITs. We paid $0.1 for the main rating. Refer to Figure 8 for the instruction that we used for the qualification and the main judge task.

Figure 5: Instructions provided for annotators of emergent property data candidates.

 



## References

* [1] S. B. Krog, and M. J. C. Newman, "Theoretical foundations of statistical mechanics", _Annals of Statistics_**2004**(3):57-68, 2009.
* [2] A. D. Barnett, "Statistical theory of statistical physics", _Journal of Statistical Physics_**199**(1):113-136, 1996.
* [3] R. G. Barnett, "A new approach to statistical physics", _Physica D_**198**(1):133-146, 1997.

[MISSING_PAGE_POST]



A conceptual combination consists of:

* **Head noun:** The central word (e.g., "apple").
* **- Modified:** A word that adds more detail to the head noun (e.g., "pelled").

When combined, the properties of the phrase might change or remain the same. For detailed explanation of conceptual combination, refer to [wikipedia site].

**Example:** Conceptual Combination: "_pelled apple"_

1. **Property: White**

A peeled septic is usually white, even if "peelest" and "apple" aren't directly related to being white.

The property "white" **emerges**, thus "white" is an **emergent property**.
2. **Property: Round**

A peeled apple is still round, because apples are round.

The property "round" remains, thus "round" is a **compenent property**.

(Our property data may have mixed passive voice and active voice. Please understand in a way that makes sense.)
3. **Property: Red**

An apple is usually red, but when peeled, it's not red anymore.

The property "red" **disappears**, thus "red" is a **canceled property**.

Here are some already-solvered examples for you.

Inportant

Use your own knowledge. Don't look up answers or use AI. We want your personal judgement!

Here are some already-solvered examples to help you understand the task:

Figure 7: Instructions provided for raters in Amazon Mechanical Turk to collect human annotations for data quality.

 



**Hello!**

We need your help in scoring how much a given noun/noun phrase is related to a given property.

Concepts are characterized by properties. For example, the concept "a chicken in front of a fox" strongly exhibits the property "in danger." When given a concept and a property, your task is to **evaluate how much the concept has the property on a scale from 1 to 10.** Use the following scoring criteria to assign a relevance score:

- "relevance": 1: The concept does not have the property at all.

- "relevance": 2-3: The concept rarely has the property.

- "relevance": 4-6: The concept sometimes has the property.

- "relevance": 7-8: The concept usually has the property, but not always.

- "relevance": 9: The concept almost always has the property.

- "relevance": 10: The concept always has the property.

Examples:
* Concept: Rusty
* Property: Useless
* Relevance: 7
* Concept: A chicken in the cage
* Property: In danger
* Relevance: 2
* Concept: A chicken in front of a fox
* Property: In danger
* Relevance: 9

Example: how much the concept ${concept} has the property ${property}? Scale 1\(\sim\)10.

**Concept:**

${concept}

Property:${property}

Figure 8: Instructions provided for raters in Amazon Mechanical Turk to collect the relevance score between the given property and a concept.