TreeTop: Topology-Aware Fine-Tuning for LLM Conversation Tree Understanding

Authors: Jashn Arora, Rahul Madhavan, Karthikeyan Shanmugam, John Palowitch, Manish Jain
Primary area: foundation or frontier models, including LLMs
Keywords: Conversation Trees, Social media, Large language models

Abstract:
While Large Language Models (LLMs) have dominated a wide diversity of natural language tasks, improving their capabilities on \emph{structured} inputs such as graphs remains an open challenge. We introduce $\texttt{TreeTop}$, a pre-training framework for LLMs that significantly improves their ability to understand and reason over structural relationships in multi-party, threaded discussions, such as those found on social media platforms. $\texttt{TreeTop}$ is a novel set of 17 QA-style tasks specifically designed to allow LLMs to selectively focus on both the structure of and content in discussion graphs. We find that LLMs fine-tuned with $\texttt{TreeTop}$ outperform their counterparts in every setting: zero-shot/few-shot performance on unseen pretraining tasks as well as downstream social media inference tasks (e.g.rumor detection), as well as fine-tuned performance on the downstream tasks, including their challenging "early-detection" variants. In particular, $\texttt{Gemini Pro}$ fine-tuned with $\texttt{TreeTop}$ and further fine-tuned on downstream tasks surpasses both vanilla $\texttt{Gemini Pro}$ and state-of-the-art GNN baselines. Our framework paves the way for LLMs with enhanced capabilities on heavily-structured inputs.

Decision: reject
Reviewer scores: 3, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730379410922

thread_type:
official_review

<thread_object>
creator: reviewer_5ohp

<content>
summary:
TreeTop is a fine-tuning framework designed to enhance LLMs in understanding structured inputs like conversation trees on social media. Featuring 17 tasks, TreeTop focuses on improving LLMs' ability to reason about structural and content relationships in these discussions. LLMs fine-tuned with TreeTop outperform baseline models, including SOTA GNNs, in generalizing to new tasks and excelling in social media inference tasks, such as controversy detection, even in early-detection scenarios. This framework advances LLMs' capabilities in processing structured data.

strengths:
1. This work presents a new conversation tree encoding method. It would be beneficial to the field of multi-turn multi-party conversation.

2. It reveals the shortcoming of existing LLMs on understanding such kinds of conversation tree, and also presents a fine-tuning-based solution.

weaknesses:
1. I think the writing should be improved. For example, I did not catch how you finetune LLMs with the TreeTop framework. It is better to clarify it clearly in the main content.

2. It seems that the baselines and related works are out-of-the-date. Are there any other more recently works of LLM on conversation tree?

3. I wonder how the conversation tree benefits the downstream dialogue tasks, such as dialogue response generation. Generating and understanding the conversation tree should be the endpoints of exploring this technique. Constructing the conversation tree, I think, should facilitate the quality of the response generation in multi-party conversations. So, I think it is important to highlight how the conversation tree will help the field of dialogue system.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484554043

thread_type:
official_review

<thread_object>
creator: reviewer_ibfg

<content>
summary:
In this paper, the author proposed TreeTop, a fine-tuning framework improving the reasoning capabilities of LLMs over structured conversation tree data. TreeTop incorporates 17 structural tasks to train LLMs on the unique topological and content-driven aspects of conversation trees. Experimental results show that LLMs fine-tuned with TreeTop outperform traditional GNNs and other baselines on social media inference tasks.

strengths:
- Adapting LLMs for graph-structured conversation trees are distinct graph problem-solving methods from traditional, static graphs in their directed, acyclic nature and temporal aspects.
- The evaluation on various social media tasks is comprehensive. Fine-tuning on structural tasks and subsequent performance comparisons are sufficient. The ablation analysis is well-set up and indicates the stability of the proposed method.

weaknesses:
- This framework may not sufficiently address how to perform with longitudinal data, where multiple conversation trees evolve over time or space.
- The framework could benefit from considering the integration of user characteristics across different conversation trees, especially in social media data where the same user may participate in multiple sessions.
- The details of integrating 17 structural tasks to fine-tune LLMs, such as considering the insightful associations and importance weights between these tasks, are not yet fully clarified.
- In terms of comparisons, only traditional GNNs are used as the main baselines, lacking comparisons with more recent graph-based reasoning methods.
- The reliance on task-specific prompts requires more results to clarify the extent to which various prompt styles across tasks influence TreeTop’s effectiveness.

questions:
- How are the 17 structural tasks weighted during training? Is their importance dynamically adjusted based on downstream application requirements?
- Can TreeTop model the associations between multiple conversation trees, especially based on longitudinal data or same user scenarios?
- How does TreeTop differentiate between structure- and content-related inferences when solving a reasoning task?
- How does TreeTop compare to other fine-tuning based methods on general graph-based reasoning tasks, such as edge existence prediction or node classification?
- How sensitive are the model outputs to prompt design, and is there a mechanism to automatically adjust prompts for different tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669871724

thread_type:
official_review

<thread_object>
creator: reviewer_jnjr

<content>
summary:
This paper proposes a fine-tuning method on conversation-tree related tasks. 
Concretely, the conversation-tree for the LLM to understand and the corresponding tasks in the form of yes/no questions are encoded in a structured prompt and used for fine-tuning. This method is named as TreeTop. They show that TreeTop fine-tuned LLMs have better performance than their not fine-tuned counterparts across various tasks.

strengths:
- a new method is proposed to encode the tree structure of the conversation-trees and is showed to be effective 
- designed some structural tasks to evaluate the topological structure of the conversations trees 
- the proposed fine-tuning method does not only boost the performance on fine-tuned tasks but also on unseen tasks
- extensive experiments were conducted and the ablation studies are also detailed

weaknesses:
See questions

questions:
- For the primitive structural tasks, why is TT not compared against the baseline where the LLM is fine-tuned with non TreeTop encoded conversations and tasks? Please point it out if I understand it incorrectly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695843367

thread_type:
official_review

<thread_object>
creator: reviewer_8zwd

<content>
summary:
This paper introduces TreeTop, a fine-tuning framework designed to enhance the performance of LLMs in understanding and reasoning over conversation trees—structured, multi-party discussions typically found on social media. The framework provides a collection of 17 structural tasks, aiming to improve LLMs' capacity to process both content and the structure of conversation trees. Experimental results show that LLMs fine-tuned with TreeTop outperform baseline models, including state-of-the-art GNNs, on multiple social media inference tasks, such as controversy and rumor detection.

strengths:
- The paper tackles a novel challenge by focusing on conversation trees, which have unique characteristics such as directed, acyclic, and temporal structures. This addresses an important gap in LLM capabilities.
- TreeTop shows potential across a range of tasks (e.g., controversy detection, rumor detection) relevant to social media, making the framework broadly useful.

weaknesses:
- Some tasks are too easy to be meaningful. This work is proposed as a benchmark for LLMs and what's the point if a LLM can already achieve 100% accuracy?
- Most datasets used in this study are derived from Reddit, which may limit the generalizability of TreeTop’s effectiveness on other social media platforms or discussion types.
- The paper briefly touches on potential issues such as over-moderation or promoting echo chambers but does not provide a detailed discussion on these risks or propose safeguards.
- The paper only includes results from one model (Gemini), which is bluntly telling the authors work for a certain company. Also not including other models fail to set meaningful baselines.
- Format: This paper uses a wrong font.

questions:
See "Weaknesses."
</content>
</thread_object>

</thread>
<thread>
creation date:
1732214742395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their thoughtful comments and valuable feedback on our work. We are grateful that Reviewer 8zwd recognized the novelty of our approach in addressing the challenges of understanding conversation trees. We appreciate Reviewer jnJR and Reviewer iBfG for acknowledging the comprehensiveness of our structural tasks and the effectiveness of our proposed method. We also thank Reviewer 5ohp for noting the potential benefits of our new conversation tree encoding method to the field of multi-turn, multi-party conversations.

Multiple reviewers asked thoughtful questions about the purpose and impact of TreeTop. As a general response, we would like to point out that TreeTop is a fine-tuning framework for LLMs (not a benchmark), consisting of 17 novel reasoning tasks on conversation trees on which pretrained LLMs can be fine-tuned. We hypothesize that fine-tuning on these tasks will allow LLMs to generalize to a variety of downstream tasks defined on conversation trees. To demonstrate this promise of TreeTop, we fine-tune multiple LLMs on 12 of these tasks, and then analyze the performance of the models fine-tuned with TreeTop vs zero/few-shot performance. We find that:

1) Without TreeTop fine-tuning, LLMs perform poorly on all 17 primitive reasoning tasks that we introduce, showing that the tasks are non-trivial for out-of-the-box LLMs.

2) On the 12 TreeTop tasks included for fine-tuning, LLM performance rises from ~40-60% to near-perfect (93-100%).

3) On the 5 TreeTop tasks held out for evaluation (not included in fine-tuning), LLM performance rises by 10-20%.

4) Across-the-board, on a variety of downstream tasks from multiple platforms, LLMs fine-tuned with TreeTop outperform all competitors (including non-LLM SotA and LLMs not fine-tuned with TreeTop). This result holds across a large variety of settings that we have carefully tested with controlled experiments and ablations.

5) An additional notable result of the TreeTop framework is its capacity for early detection. This early warning capability enables effective human-in-the-loop moderation while preventing over-automation, enabling significant real-world impact. 

Based on the reviewers' responses, we have made several changes to the draft to improve the clarity. Specifically, we have reworked parts of Section 3 and Section 4 (particularly S4 intro and S4.1) to clarify the message above. We have highlighted the details about finetuning experiments at the outset of Section 4. We have expanded the discussion section (Section 5) and now address multiple points raised by the reviewers like platform generalization beyond Reddit, and content moderation implications. We also added an entirely new Appendix H providing new ablation results using an alternate encoding of the conversation tree. We would greatly appreciate the reviewers' consideration of these revisions.

## Addressing General Questions:

**Baselines and Comparisons with Other Models (Reviewer 1 and Reviewer 3):**

**Q1 (Reviewer 8zwd):** "The paper only includes results from one model (Gemini)... Also not including other models fail to set meaningful baselines."

**Q2 (Reviewer iBfG):** "In terms of comparisons, only traditional GNNs are used as the main baselines, lacking comparisons with more recent graph-based reasoning methods."

**Response:** We thank the reviewers for raising these important points about model comparisons and baselines. Our evaluation includes experiments across multiple model families and baselines. The results demonstrate TreeTop's effectiveness across three different LLM architectures: Gemma-2B (open source) achieving 90-100% accuracy on the holdout set of structural fine-tuning tasks, PaLM-Bison showing 90-99% accuracy, and Gemini-Pro reaching 93-100% accuracy (see Table 2, and Appendix G). These results show that TreeTop's benefits generalize across different architectures and are not specific to any single model family.

Beyond LLM comparisons, we evaluate against both traditional GNN architectures (GCN, GAT, GraphSage) and recent specialized approaches. On controversy detection, as shown in Table 3, TreeTop (84.6%) significantly outperforms the recent DFE-GCN approach (76.6%). For rumor detection, as shown in Table 16, TreeTop achieves 87.1% accuracy, substantially exceeding both PHAROS (75.9%) and NRA MOS-GAT (78.4%). The consistent improvement across both model families and baseline comparisons demonstrates that TreeTop's effectiveness stems from its fundamental approach to structural learning rather than from any specific model architecture. Our ablation studies, detailed in Appendices E, G, H, I further validate these findings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553950297

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their time. We wanted to encourage the reviewers to reconsider the ratings. We have tried to provide detailed responses to the reviews along with the experimentations whenever required. We have also updated the paper pdf accordingly. We would also be happy to respond to any reviewer comments promptly given the closeness of the end of the discussion period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554006473

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their time. We wanted to encourage the reviewers to reconsider the ratings. We have tried to provide detailed responses to the reviews along with the experimentations whenever required. We have also updated the paper pdf accordingly. We would also be happy to respond to any reviewer comments promptly given the closeness of the end of the discussion period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces TreeTop, a framework designed to enhance the capabilities of LLMs in processing conversation trees, particularly those with directed, acyclic, and temporal structures, tackling a significant gap in existing LLM methodologies. The framework shows potential in various tasks relevant to social media, such as controversy and rumor detection. 

However, as pointed out by the reviewers, there are a few concerns: some tasks appear too simplistic, achieving near-perfect accuracy, which may not serve as meaningful benchmarks; the generalizability of the results is questionable due to the predominant use of data derived from Reddit; the novelty of the paper; and some others. Even though the authors tried to address these concerns during the rebuttal, the reviewers were not fully satisfied.