IPBench: Benchmarking the Knowledge of Large Language Models in
  Intellectual Property
Authors: name: Qiyao Wang; name: Guhong Chen; name: Hongbo Wang; name: Huaren Liu; name: Minghui Zhu; name: Zhifei Qin; name: Linwei Li; name: Yilin Yue; name: Shiqiang Wang; name: Jiayan Li; name: Yihang Wu; name: Ziqiang Liu; name: Longze Chen; name: Run Luo; name: Liyang Fan; name: Jiaming Li; name: Lei Zhang; name: Kan Xu; name: Hongfei Lin; name: Hamid Alinejad-Rokny; name: Shiwen Ni; name: Yuan Lin; name: Min Yang
Published: 2025-04-22 02:00:41

Abstract: Intellectual Property (IP) is a unique domain that integrates technical and
legal knowledge, making it inherently complex and knowledge-intensive. As large
language models (LLMs) continue to advance, they show great potential for
processing IP tasks, enabling more efficient analysis, understanding, and
generation of IP-related content. However, existing datasets and benchmarks
either focus narrowly on patents or cover limited aspects of the IP field,
lacking alignment with real-world scenarios. To bridge this gap, we introduce
the first comprehensive IP task taxonomy and a large, diverse bilingual
benchmark, IPBench, covering 8 IP mechanisms and 20 tasks. This benchmark is
designed to evaluate LLMs in real-world intellectual property applications,
encompassing both understanding and generation. We benchmark 16 LLMs, ranging
from general-purpose to domain-specific models, and find that even the
best-performing model achieves only 75.8% accuracy, revealing substantial room
for improvement. Notably, open-source IP and law-oriented models lag behind
closed-source general-purpose models. We publicly release all data and code of
IPBench and will continue to update it with additional IP-related tasks to
better reflect real-world challenges in the intellectual property domain.