HoT: Highlighted Chain of Thought for Referencing Supporting Facts from
  Inputs
Authors: name: Tin Nguyen; name: Logan Bolton; name: Mohammad Reza Taesiri; name: Anh Totti Nguyen
Published: 2025-03-03 19:26:04

Abstract: An Achilles heel of Large Language Models (LLMs) is their tendency to
hallucinate non-factual statements. A response mixed of factual and non-factual
statements poses a challenge for humans to verify and accurately base their
decisions on. To combat this problem, we propose Highlighted Chain-of-Thought
Prompting (HoT), a technique for prompting LLMs to generate responses with XML
tags that ground facts to those provided in the query. That is, given an input
question, LLMs would first re-format the question to add XML tags highlighting
key facts, and then, generate a response with highlights over the facts
referenced from the input. Interestingly, in few-shot settings, HoT outperforms
vanilla chain of thought prompting (CoT) on a wide range of 17 tasks from
arithmetic, reading comprehension to logical reasoning. When asking humans to
verify LLM responses, highlights help time-limited participants to more
accurately and efficiently recognize when LLMs are correct. Yet, surprisingly,
when LLMs are wrong, HoTs tend to make users believe that an answer is correct.