SealQA: Raising the Bar for Reasoning in Search-Augmented Language
  Models
Authors: name: Thinh Pham, name: Nguyen Nguyen, name: Pratibha Zunjare, name: Weiyuan Chen, name: Yu-Min Tseng, name: Tu Vu

We introduce SealQA, a new challenge benchmark for evaluating
SEarch-Augmented Language models on fact-seeking questions where web search
yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors:
(1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and
reasoning capabilities, with Seal-0 focusing on the most challenging questions
where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3)
LongSeal, which extends SealQA to test long-context, multi-document reasoning
in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations
in current models: Even frontier LLMs perform poorly across all SealQA flavors.
On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini
achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning
efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and
o3-mini are highly vulnerable to noisy search results. Notably, increasing
test-time compute does not yield reliable gains across o3-mini, o4-mini, and
o3, with performance often plateauing or even declining early. Additionally,
while recent models are less affected by the "lost-in-the-middle" issue, they
still fail to reliably identify relevant documents in LongSeal when faced with
numerous distractors. To facilitate future work, we release SealQA at
huggingface.co/datasets/vtllms/sealqa.