HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application

## Abstract

Abstract Abstract Effective deep search agents must not only access open-domain and domain-specific knowledge but also apply complex rules—such as legal clauses, medical manuals and tariff rules.
These rules often feature vague boundaries and implicit logic relationships, making precise application challenging for agents.
However, this critical capability is largely overlooked by current agent benchmarks.
To fill this gap, we introduce HSCodeComp , the first realistic, expert-level e-commerce benchmark designed to evaluate deep search agents in hierarchical rule application.
In this task, the deep reasoning process of agents is guided by these rules to predict 10-digit Harmonized System Code (HSCode) of products with noisy but realistic descriptions.
These codes, established by the World Customs Organization, are vital for global supply chain efficiency.
Built from real-world data collected from large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 product entries spanning diverse product categories, with these HSCodes annotated by several human experts.
Extensive experimental results on several state-of-the-art LLMs, open-source, and closed-source agents reveal a huge performance gap: best agent achieves only 46.8% 10-digit accuracy, far below human experts at 95.0%.
Besides, detailed analysis demonstrates the challenges of hierarchical rule application, and test-time scaling fails to improve performance further. https://github.com/AIDC-AI/Marco-Search-Agent https://huggingface.co/datasets/AIDC-AI/HSCodeComp

## 1 Introduction

Deep search agents have demonstrated significant value in solving complex real-world problems, where robust external knowledge utilization constitutes a critical capability [Wu et al., 2025, Tao et al., 2025, Li et al., 2025b]. To evaluate this capability, numerous established benchmarks are proposed to assess agents in utilizing open-domain data (e.g., GAIA [Mialon et al., 2023b] and BrowseComp [Wei et al., 2025]) and domain-specific data (e.g., WebMall [Peeters et al., 2025a], FinSearchComp [Hu et al., 2025a] and MedBrowseComp [Yu et al., 2025b]).

Beyond open-domain and domain-specific data, agents also need to effectively apply rules that encode human expert knowledge, particularly in scenarios like law, medical and e-commerce [Li et al., 2025a, Chen et al., 2025b, Yao et al., 2022, Chollet et al., 2025].
For instance, legal case adjudication require interpreting abstract legal provisions, and accurate e-commerce product classification in depends on tariff rules [Grainger, 2024].
Previous works have defined rule application as using specific logical rules with supporting facts to derive conclusions [Wang et al., 2024, Servantez et al., 2024]. In contrast, we define it as a core capability for deep search agents, where human-written rules are systematically applied to guide complex reasoning and decision-making [Sadowski and Chudziak, 2025].
Building on this observation, we categorize knowledge data for deep search agents into three levels (Figure [1](https://arxiv.org/html/2510.19631v1#S0.F1), left), with increasing knowledge complexity:
(1) Level 1: Open-domain Data - Tests understanding and deep reasoning abilities of agents on long-form web content. Established benchmarks include GAIA [Mialon et al., 2023b] and BrowseComp [Wei et al., 2025];
(2) Level 2: Structured Data - Assesses agents to precisely utilize structured data such as databases and knowledge graphs, as seen in domain-specific benchmarks like WebMall [Peeters et al., 2025a], MedBrowseComp [Chen et al., 2025b] and FinSearchComp [Hu et al., 2025a];
(3) Level 3: Rule Data - Evaluates agents to apply complex and abstract rules [Chollet et al., 2025]. This level presents two key challenges: (a) making accurate decisions when rules contain vague natural language descriptions [Sadowski and Chudziak, 2025]; and (b) reasoning about logical dependencies among rules, such as exception clauses and cross-category relationships [Guha et al., 2023].
Despite the importance of rule application in real-world scenarios, current agent benchmarks largely overlook its evaluation.

To fill this gap, we introduce HSCodeComp (short for the Harmonized System Code (HSCode) Competition), the first realistic, expert-level e-commerce benchmark designed to evaluate agents in predicting complete 10-digit Harmonized System Code (HSCode) of the product, using hierarchical rules (e.g., eWTP tariff rules(^1^11[https://www.ewtp.com/web/smart/hscode](https://www.ewtp.com/web/smart/hscode))).
HSCodes organize products through a hierarchical structure spanning over 5,000 distinct codes across multiple classification levels, representing the global standard for classifying traded international goods, established by the World Customs Organization and implemented across more than 200 countries for customs clearance and tariff determination [Grainger, 2024, Nath et al., 2025].
Built from the data of the large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 carefully curated product entries, encompassing 27 unique HS chapters and 32 distinct first-level categories. These HSCodes have been rigorously annotated by multiple e-commerce domain experts, ensuring that HSCodeComp is expert-level.
Accurately predicting the exact 10-digit HSCode presents significant challenges: agents must perform multi-hop hierarchical reasoning with complex tariff rules while processing noisy but realistic product descriptions that often contain abbreviations, language variations, or incomplete information.

Extensive experiments on the state-of-the-art baselines, including 14 advanced foundation models, 6 advanced open-source agent systems and 3 closed-source agent systems, demonstrate that HSCode prediction task remains a substantial challenge for current AI approaches.
As shown in the Figure [1](https://arxiv.org/html/2510.19631v1#S0.F1) (right), even the best-performing system (SmolAgent [Roucher et al., 2025] with GPT-5) achieves only 46.8% accuracy, substantially below the 95.0% accuracy attained by human experts.
Further detailed analysis reveals that existing agent systems lack critical capabilities required for this complex hierarchical rule applications.
Notably, test-time scaling approach—which has proven effective in other reasoning tasks [Guo et al., 2025, Liu et al., 2025]—fail to improve performance on HSCodeComp.
These observations demonstrate the challenging nature of our proposed HSCodeComp, highlighting the need for more effective designs of agent systems.
To facilitate future research, we will publicly release codes and the benchmark dataset of HSCodeComp.

## 2 Related Works

### 2.1 Previous Works in HSCode Prediction

Previous works treat HSCode prediction as the e-commerce text classification task [Grainger, 2024], using pre-trained BERT models [Liao et al., 2024, Shubham et al., 2022] or Large Language Models (LLMs) prompting [Hussain and Ahmed, 2023]. However, these approaches fail to leverage domain-specific knowledge, especially the rules written by human experts [Hussain and Ahmed, 2023, Judy, 2024].
Besides, existing HSCode benchmarks face two critical limitations [Judy, 2024, Lee et al., 2024, Stassin et al., 2023]: (1) they are typically constructed from publicly accessible customs rulings, suffering from data leakage; (2) they are not released.
In contrast, our released HSCodeComp is collected from large-scale online shopping platforms with noisy product descriptions, making it more challenging and realistic.

### 2.2 Benchmarking Level 1 Knowledge Utilization

Numerous benchmarks have been proposed to evaluate agent capabilities in understanding and deep reasoning over long-form open-domain web content [Thomas et al., 2025, Yao et al., 2024, Joshi et al., 2017, Phan et al., 2025].
For example, WebArena [Zhou et al., 2023] provides realistic, self-hostable websites with standardized evaluation protocols to assess functional correctness.
WebShop [Yao et al., 2022] and ALFWorld [Shridhar et al., 2021] evaluate long-horizon decision-making abilities of agents in web environments through tool interactions.
More recent deep search benchmarks, such as GAIA [Mialon et al., 2023a], BrowseComp [Wei et al., 2025], WebWalkerQA [Wu et al., 2025] and BrowseComp-ZH [Zhou et al., 2025], demand advanced tool-usage and deep reasoning capabilities [Zhang et al., 2025, Li et al., 2025b].

### 2.3 Benchmarking Level 2 Knowledge Utilization

Recent works have focused on how agents utilize structured knowledge in domain-specific applications. Unlike open-domain data, domain-specific knowledge is typically organized into structured formats such as databases and knowledge graphs [Huang et al., 2025, Yu et al., 2025a, Chen et al., 2025a], enabling more precise knowledge retrieval and utilization. To evaluate these capabilities, numerous deep search benchmarks have been proposed, including
WebMall [Peeters et al., 2025b] and DeepShop [Lyu et al., 2025] for e-commerce, LegalAgentBench for law [Li et al., 2025a],
FinSearchComp for finance [Hu et al., 2025a],
DAgent for data analysis [Xu et al., 2025], CRMArena for CRM workflows [Huang et al., 2025],
and MedBrowseComp for medicine [Chen et al., 2025b].

In summary, while there exists numerous evaluation benchmarks for assessing agent performance in open-domain or domain-specific scenarios, none evaluates the ability to apply Level 3 abstract rule-based knowledge.
To address this critical gap, we introduce a realistic and expert-level e-commerce benchmark HSCodeComp. Our benchmark presents significant challenges even for state-of-the-art closed-source and open-source agent systems.

## 3 Task Formulation of HSCode Prediction

The HSCode prediction task is to assign a valid and unique 10-digit Harmonized System (HS) code to a given noisy but realistic product description.
The product HSCode plays the crucial role in e-commerce system. It is the global standard for classifying traded goods, essential for tariffs, customs, and trade governance.
The core challenge is to learn a mapping function, i.e., agents implemented by Large Language Models (LLMs) or Vision Language Models (VLMs), $f:\mathcal{X}\to\mathcal{Y}$.

Figure: Figure 2: One example of a game console product in HSCodeComp.
Refer to caption: figures/example_product_v3.png

#### Input:

Figure [2](https://arxiv.org/html/2510.19631v1#S3.F2) shows that each product $x\in\mathcal{X}$ encompass rich records: $x=(t,A,c,i,p,u,r)$, where $t$ is the product title; $A=\{(k_{j},v_{j})\}_{j=1}^{K}$ represents a set of $K$ product attributes (e.g., material and package size); $c$ represents product categories defined by the e-commerce platform; $i$ is the product image; $p$ and $u$ are the price and currency; and $r$ is the webpage URL of the product.

#### Hierarchical Rule Utilization:

Accurate HSCode prediction requires effectively utilizing three types of e-commerce knowledge:
(1) Hierarchical tariff rules from official classification systems (e.g., eWTP), which organize product categories in a hierarchical structure.
As shown in Figure [11](https://arxiv.org/html/2510.19631v1#A6.F11), these rules contain complex implicit logic relationships, for example, the exception clause in tariff rules like excluding articles of HS heading 8539 … (highlighted with red boxes).
Besides, these rules often employ vague linguistic constraints (highlighted with blue boxes) that challenge existing AI agents;
(2) Human-written decision rules that specify how to correctly apply tariff rules. These rules provide high-level decision principles (see Figure [13](https://arxiv.org/html/2510.19631v1#A6.F13) for an example with six key principles defined by domain experts); and
(3) Official customs rulings databases, such as the U.S. Customs Rulings Online Search System (CROSS)(^2^22[https://rulings.cbp.gov/](https://rulings.cbp.gov/)), which document historical HSCode classification decisions. As illustrated in Figure [12](https://arxiv.org/html/2510.19631v1#A6.F12), these databases contain complex information format requiring advanced reasoning capabilities.

#### Output:

The HSCode $y\in\mathcal{Y}$ is a single 10-digit numeric string $\mathcal{Y}\subseteq\{0,1,\ldots,9\}^{10}$.
The HSCode is hierarchical, where first 2-digit, 4-digit, and 6-digit represents the HS chapter, heading and sub-heading of tariff classification of products, respectively, and last 4 digits (from 6 to 10) are country-specific codes.
In summary, this 10-digit HSCode follows a valid path in the official HS taxonomy.

## 4 HSCodeComp Construction and Evaluation Metrics

### 4.1 Benchmark Construction

We design a rigorous pipeline, ensuring the dataset is diverse, realistic and expert-level: (1) Data Collection and Diversity Control; (2) Human Expert Annotation; and (3) Human Expert Validation.

#### Data Collection and Diversity Control.

Products in our proposed HSCodeComp is sourced from a large-scale global e-commerce platform.
These product profiles include the noisy information, ensuring that task instances reflect the real-world challenges.
Besides, we also balance the data category distribution to prevent the potential topical skew. Specifically, we apply a pre-processing step: a semantic redundancy filter discards the products ($x$) sharing identical categories ($c$) and 10-digit HSCode ($y$) with existing product instances. This ensures that HSCodeComp is not dominated by common and easy-to-classify products.

#### Human Expert Annotation.

To ensure the quality of HSCodeComp, we engage human experts specialized in HSCode classification to annotate the HSCode $(y)$ for each product profile ($x$).
As shown in Figure [3](https://arxiv.org/html/2510.19631v1#S4.F3) (left), the annotation process follows a five-step pipeline:
(1) Two experts gather comprehensive information from the product webpage (Step 1);
(2) They extract the core structured features of products (Step 2);
(3) Experts search the official customs ruling databases (CROSS) for related cases. If a related case is found (very rare during the annotation of HSCodeComp), the corresponding HSCode is then validated on eWTP system, followed by the minor revision. Otherwise, they refine their search queries or revisit Step 2 to adjust the extracted features (Step 3);
(4) For products without any related cases, experts execute human-written hierarchical decision rules to apply tariff rules, and determine the appropriate HSCode (Step 4);
(5) Finally, experts verify the final identified HSCodes on the eWTP website to ensure its validity (Step 5).

#### Human Expert Validation.

As shown in the Figure [3](https://arxiv.org/html/2510.19631v1#S4.F3) (Step 6), when the HSCodes assigned by two experts match, the code is accepted. When they disagree, a senior tariff expert reviews both annotations to determine the correct HSCode. If neither annotation is valid, the instance is excluded from the dataset.
Finally, we collect 632 products with their human-annotated corresponding HSCodes, spanning 32 first-level product category defined by a large-scale ecommerce platform and 27 HS chapters defined by eWTP(^3^33Please refer to Figure [7](https://arxiv.org/html/2510.19631v1#A2.F7) for more details about HSCodeComp statistics.).
Furthermore, to verify the reliability of our process, we conducted an additional quality review. A fourth senior expert, not involved in the initial annotation, re-annotated a random 10% sample of the dataset. This review shows only a 2% disagreement rate, confirming the effectiveness and consistency of our dataset construction pipeline.

Figure: Figure 3: The pipeline for human experts to annotate the HSCodes, including two human experts for HSCode annotation (Step 1 to 5) and one additional expert for quality validation (Step 6).
Refer to caption: x3.png

### 4.2 Evaluation Metric

We conduct the exact match to compare the normalized HSCodes extracted from the final output against human-annotated ground truth.
Our primary evaluation metric is the 10-digit HSCode accuracy, which measures whether the predicted code exactly matches the reference 10-digit code.
Additionally, we also report accuracies at 2-digit, 4-digit, 6-digit and 8-digit levels to provide more comprehensive insights into the performance across different granularities.

**Table 1: The complete results of state-of-the-art baselines in our proposed HSCodeComp.**
| Baselines | Model<br>Type | HSCode Prediction Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 2-digit | 4-digit | 6-digit | 8-digit | 10-digit |  |  |
| LLM/VLM-Only |  |  |  |  |  |  |
| GPT-5 | VLM | 82.12 | 70.89 | 59.97 | 41.46 | 29.27 |
| Gemini-2.5-PRO | VLM | 82.28 | 71.04 | 59.02 | 40.51 | 24.21 |
| GPT-4o | VLM | 78.01 | 64.08 | 48.10 | 29.75 | 18.51 |
| Claude Sonnet 4 | VLM | 78.80 | 64.08 | 45.25 | 22.63 | 11.23 |
| GPT-5 | LLM | 82.59 | 69.78 | 56.33 | 40.98 | 28.96 |
| Gemini-2.5-PRO | LLM | 80.54 | 69.94 | 58.54 | 40.35 | 23.42 |
| GPT-4o | LLM | 75.47 | 61.55 | 45.73 | 30.06 | 18.35 |
| Claude Sonnet 4 | LLM | 78.80 | 62.97 | 44.94 | 23.58 | 11.87 |
| Kimi-K2 | LLM | 78.01 | 62.03 | 44.15 | 24.53 | 12.18 |
| DeepSeek-R1 | LLM | 77.22 | 61.71 | 38.45 | 16.77 | 6.65 |
| DeepSeek-V3 | LLM | 77.06 | 54.43 | 32.28 | 17.25 | 6.49 |
| Qwen-MAX | LLM | 71.52 | 48.58 | 24.21 | 11.23 | 3.80 |
| Qwen3-235B-A22B | LLM | 66.93 | 49.53 | 24.53 | 6.01 | 1.74 |
| O3-mini | LLM | 77.22 | 56.17 | 24.53 | 6.65 | 1.27 |
| Qwen3-32B | LLM | 64.40 | 29.27 | 8.07 | 1.27 | 0.32 |
| QWQ-32B | LLM | 66.77 | 29.11 | 4.43 | 1.42 | 0.16 |
| Qwen2.5-72B | LLM | 20.73 | 12.34 | 3.80 | 1.42 | 0.16 |
| Nemotron-32B | LLM | 43.51 | 5.70 | 0.16 | 0.00 | 0.00 |
| Open-source Agent System (GPT-5 Backbone) |  |  |  |  |  |  |
| SmolAgents | VLM | 82.06 | 72.06 | 62.38 | 52.38 | 46.83 |
| SmolAgents | LLM | 82.28 | 70.89 | 59.81 | 49.05 | 42.72 |
| Aworld | LLM | 82.28 | 70.41 | 59.18 | 48.58 | 41.30 |
| Agentorchestra | LLM | 82.12 | 70.73 | 60.44 | 47.78 | 41.30 |
| OWL | LLM | 72.63 | 61.87 | 51.58 | 41.77 | 37.34 |
| WebSailor | LLM | 81.64 | 70.56 | 57.27 | 43.98 | 35.44 |
| Cognitive Kernel | LLM | 80.06 | 69.15 | 54.59 | 40.03 | 26.42 |

## 5 Experiments

### 5.1 Experimental Setup

We evaluate three kinds of advanced approaches on HSCodeComp:

#### LLMs/VLMs (no tools):

We test 14 foundation models (GPT-5, Gemini-2.5-PRO, GPT-4o, Kimi-K2 [Team et al., 2025], Claude Sonnet 4, DeepSeek series [DeepSeek-AI et al., 2025], Qwen variants [Yang et al., 2025], O3-mini, Nemotron-32B [Bercovich et al., 2025]) for HSCode prediction using only internal knowledge. For VLMs (GPT-4o, GPT-5, Claude Sonnet 4, Gemini 2.5 Pro), we provide product images to assess the impact of the visual information.

#### Open-source Agent Systems:

We evaluate six open-source frameworks (SmolAgents [Roucher et al., 2025], Aworld [Yu et al., 2025c], Agentorchestra [Zhang et al., 2025], OWL [Hu et al., 2025b], WebSailor [Li et al., 2025b] and Cognitive Kernel [Fang et al., 2025]) using GPT-5 as the default backbone. SmolAgent is enhanced with vision capabilities via product images, while Vision Language Models are incompatible with other agent frameworks. All frameworks utilize standardized tools including web search.

#### Closed-source Agent Systems:

We assess the performance of commercial systems Manus, Gemini Deep Research, and Grok DeepSearch. As these systems do not provide public APIs, we conduct manual evaluations on 49 representative examples from the HSCodeComp benchmark, following the evaluation protocol established in prior work [Li et al., 2025b].

All systems produce standardized outputs: a single HSCode in \boxed{...} format. More implementation details appear in Appendix [F](https://arxiv.org/html/2510.19631v1#A6).

### 5.2 Main Results

**Table 2: Comparison between closed-source and open-source agents.**
| Agent Systems | 10-digit |
| --- | --- |
| Gemini Deep Researcher | 40.81 |
| Manus | 30.61 |
| Grok DeepSearch | 26.53 |
| SmolAgents (GPT-5) | 42.86 |
| Aworld (GPT-5) | 42.86 |

Table [1](https://arxiv.org/html/2510.19631v1#S4.T1) summarizes the performance of state-of-the-art LLM/VLM-Only models and open-source agents on HSCodeComp.
All approaches exhibit a consistent decline in accuracy as the HSCode length increases.
Notably, LLM/VLM-only baselines are much worse than agent systems due to their lack of domain-specific knowledge.
The best baseline, SmolAgent (GPT-5 VLM version), achieves only 46.83% 10-digit accuracy, which remains substantially below the 95% accuracy achieved by experienced human experts.
To ensure a fair comparison, we evaluate both closed-source and open-source agents on the same subset of HSCodeComp.
Table [2](https://arxiv.org/html/2510.19631v1#S5.T2) shows open-source agents outperform closed-source agents.
Case studies reveal that closed-source agents suffer from the permature decisions and information misprocessing problems, as detailed in Section [6.3](https://arxiv.org/html/2510.19631v1#S6.SS3).
In summary, the significant performance gap between human experts and the top-performing agent system underscores the challenges presented by HSCodeComp.
To better understand the factors affecting the performance, we conduct three ablation studies as below.

**Table 3: The ablation study on human-written Decision Rules (DR). GPT-5 is the backbone.**
| Backbone Model | Model Type | 10-digit |
| --- | --- | --- |
| SmolAgent w/o DR | VLM | 46.83 |
| SmolAgent w/ DR | VLM | 43.83 $\boldsymbol{\downarrow}$ |
| Aworld w/o DR | LLM | 41.30 |
| Aworld w/ DR | LLM | 42.95 $\boldsymbol{\uparrow}$ |
| WebSailor w/o DR | LLM | 35.44 |
| WebSailor w/ DR | LLM | 35.43 $\boldsymbol{\downarrow}$ |

#### Ablation Study on Hierarchical Decision Rules

The hierarchical decision rules capture how human experts apply tariff rules. To assess whether agents can effectively leverage these rules, we conduct ablation experiments for following models: GPT-5, SmolAgent (GPT-5 VLM version), Aword (GPT-5 LLM version) and WebSailor (GPT-5 LLM version).
As shown in Table [3](https://arxiv.org/html/2510.19631v1#S5.T3), incorporating decision rules (w/ DR) decreases accuracy for both SmolAgent and WebSailor, while Aworld achieves only marginal gains.
Therefore, we remove these decision rules for agents as the default setup.
These results indicate that current agent systems struggle at applying human-written decision rules, thereby limits their ability to utilize hierarchical tariff rules for HSCode prediction.

**Table 4: The ablation study of the product images in SmolAgents.**
| Backbone Model | 10-digit |
| --- | --- |
| GPT-5 w/o Image | 42.72 |
| GPT-5 w/ Image | 46.83 $\boldsymbol{\uparrow}$ |
| Gemini-2.5-Pro w/o Image | 34.49 |
| Gemini-2.5-Pro w/ Image | 34.39 $\boldsymbol{\downarrow}$ |
| Claude 4 Sonnet w/o Image | 33.70 |
| Claude 4 Sonnet w/ Image | 34.65 $\boldsymbol{\uparrow}$ |
| GPT-4o w/o Image | 22.03 |
| GPT-4o w/ Image | 22.31 $\boldsymbol{\uparrow}$ |

#### Multi-modal Information Is Helpful

Table [1](https://arxiv.org/html/2510.19631v1#S4.T1) and Table [4](https://arxiv.org/html/2510.19631v1#S5.T4) show that most baselines achieve consistent improvements when the product images can be accessed.
Case studies in Appendix [I](https://arxiv.org/html/2510.19631v1#A9) show that understanding product images improves performance by capturing visual attributes—such as material and surface features—that are not present in the textual description but are critical for classification. These attributes align with the predefined rules, leading to performance gains.

#### Webpage Visits Decrease Agents Performance

We augment SmolAgent (GPT-5 LLM version) with the capability to visit webpages, but this leading to 10-digit accuracy decrease from 42.72% to 42.09%.
Our study reveals that a large amount of webpage content overwhelms the key information, misleading the agents, while this key information can be precisely extract by search engines in the snippets.
Therefore, we remove the webpage visit tool for all open-source agents as the default setup.

## 6 Analysis on HSCodeComp

We conduct several detailed analysis:
(1) Overthinking Decrease Open-Source Agent Performance (Section [6.1](https://arxiv.org/html/2510.19631v1#S6.SS1));
(2) Effects of Backbones in agents (Section [6.2](https://arxiv.org/html/2510.19631v1#S6.SS2));
(3) Failure Modes of Closed-Source and Open-Source Agents (Section [6.3](https://arxiv.org/html/2510.19631v1#S6.SS3));
(4) Per-category performance analysis (Section [6.4](https://arxiv.org/html/2510.19631v1#S6.SS4));
and (5) Effectiveness of Test-time Scaling (Section [6.5](https://arxiv.org/html/2510.19631v1#S6.SS5)).

### 6.1 Overthinking Decrease Open-source Agent Performance

**Table 5: Agent performance with different think depth. GPT-5 LLM version is the backbone.**
| Agent Systems | Think Depth | 10-digit |
| --- | --- | --- |
| SmolAgent (GPT-5) | No-Think | 42.72 |
| WebSailor (GPT-5) | No-Think | 40.82 |
| WebSailor (GPT-5) | Medium-Think | 37.34 |
| WebSailor (GPT-5) | Overthink | 35.44 |

Table [1](https://arxiv.org/html/2510.19631v1#S4.T1) shows WebSailor underperforms SmolAgent despite using identical models and tools. Our case studies reveal that this occurs because WebSailor encourages excessive reasoning (Overthink), before tool-calling. Cases in Appendix [H](https://arxiv.org/html/2510.19631v1#A8) show that WebSailor often first conduct deep reasoning to predict the full 10-digit HSCode. The errors in reasoning significantly decreases the effectiveness of tool-calling. To prove this, we created two variants for WebSailor: (1) No-Think: direct tool calling without thinking; and (2) Medium-Think: medium-level reasoning depth before tool-calling.
Medium-think denotes a moderate reasoning depth between No-think and Overthink.
Table [5](https://arxiv.org/html/2510.19631v1#S6.T5) demonstrates that reducing reasoning depth improves accuracy, with No-Think nearly matching SmolAgent.
This finding suggests that the primary factor contributing to performance variances among open-source agents is the reasoning depth defined in the task prompt.
For HSCode prediction, minimal reasoning with frequent tool calls outperforms extensive self-reasoning. When accurate information is available through calling tools, prioritizing tool utilization over reasoning yields better results for such complex domain-specific tasks.

### 6.2 Effect of Backbones in Open-source Agent Systems

**Table 6: The ablation study on the backbone models in SmolAgent.**
| Backbone Model | 10-digit |
| --- | --- |
| GPT-5 | 42.72 |
| Gemini-2.5-Pro | 34.49 |
| Claude 4 Sonnet | 33.70 |
| Qwen-MAX | 17.43 |

This subsection analyze the effects of the backbone LLM in the performance of agent systems. Specifically, we evaluate the performance of SmolAgents system implemented by four different backbone LLMs: GPT-5, Gemini-2.5-pro, Claude-4-Sonnet and Qwen-MAX.
As demonstrated in Table [6](https://arxiv.org/html/2510.19631v1#S6.T6), different backbone LLMs yield markedly different results in the HSCode prediction task, and GPT-5 is the best backbone model.
Therefore, we choose GPT-5 as the default setup for open-source agents. More results are provided in Appendix [D](https://arxiv.org/html/2510.19631v1#A4).

### 6.3 Failure Modes of Closed-Source and Open-Source Agents

We perform the qualitative and quantitative analysis for closed-source and open-source agents.

#### Qualitative analysis.

We identify six critical failure modes of open-source and closed-source agent systems in HSCodeComp:
(1) Premature Decisions: Agents commit to incorrect classification paths without collecting sufficient evidence (Table [9](https://arxiv.org/html/2510.19631v1#A7.T9), Grok DeepSearch);
(2) Information Misprocessing: Agents overlook or misinterpret key product details, indicating challenges with long-context processing (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8) and Figure [15](https://arxiv.org/html/2510.19631v1#A7.F15));
(3) Unnecessary Self-Correction: Agents sometimes predict correct HSCodes initially but revise them incorrectly through excessive critique (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8), Gemini Deep Research);
(4) Reasoning Hallucination: Agents generate plausible but factually incorrect reasoning steps (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8), Grok DeepSearch);
(5) Wrong Rule Application: Models frequently miss or misuse relevant tariff rules due to their ambiguous descriptions that confuse the reasoning process, resulting in incorrect classification decisions (Figure [17](https://arxiv.org/html/2510.19631v1#A7.F17));
and (6) Lack of Domain Knowledge: Models exhibit errors due to insufficient domain-specific knowledge, such as misidentify silicone products as rubber instead of plastic (Figure [16](https://arxiv.org/html/2510.19631v1#A7.F16)).
These limitations highlight that HSCodeComp remains challenging for advanced closed-source and open-source systems.

Figure: Figure 4: Failures analysis.
Refer to caption: x4.png

#### Quantitative analysis:

Figure [4](https://arxiv.org/html/2510.19631v1#S6.F4) present the distribution of four coarse-grained failures across both LLM/VLM-Only (left) and agents (right)(^4^44The average performance of LLM-only baselines and agent baselines are computed.): (1) Outdated: Incorrect HSCodes due to changes in tariff rules over time;
(2) Hallucination: Invalid HSCodes that do not exist in the official coding system;
(3) Error but Valid: HSCodes are valid and current, but differ from the ground-truth HSCodes;
and (4) Others: Other errors like wrong output formats, reaching maximum window and wrong tool-calling. Our analysis reveals that agents significantly reduce hallucination, outdated and other errors through effective tool utilization, compared with LLMs. Consequently, the predominant error type for agents is “Error but Valid”.
Besides, Figure [9](https://arxiv.org/html/2510.19631v1#A5.F9) also quantifies the improvements from GPT-5 to SmolAgent (GPT-5), demonstrating that the agent significantly reduces both outdated and hallucination errors.

### 6.4 Per-category Performance Analysis

Figure: Figure 5: Both figures show the category distribution on the left blue bars. Left: Challenging Product Distribution (CID). Right: Average Performance Distribution (APD).
Refer to caption: x5.png

We analyze two critical distributions across the 32 first-level product categories:
(1) Challenging Product Distribution (CID): the distribution of products that all baseline methods failed to correctly predict;
(2) Average Performance Distribution (APD): the distribution of average 10-digit accuracy across all baseline methods.
Figure [5](https://arxiv.org/html/2510.19631v1#S6.F5) reveals two key insights:
(1) The CID indicates that the most challenging products are concentrated in long-tail categories, such as Novelty & Special Use (1.3%) and Men’s Clothing (2.2%);
(2) The APD shows that average accuracy across most product categories remains below 25%, with only Hair Extensions & Wigs achieving a relatively high accuracy of 47%. Importantly, even for most frequent categories like Jewelry & Accessories (13.1%), Home & Garden (16.8%) and Tools (6.2%), the average performance stays below 18%.
These findings underscore the challenges presented by HSCodeComp, highlighting the need for more robust and generalizable approaches to HSCode prediction.

### 6.5 Test-time scaling Cannot Improve Performance Effectively

Figure: Figure 6: Majority voting experiments.
Refer to caption: x7.png

Test-time scaling (TTS) has demonstrated significant gains in complex reasoning tasks using more inference budget [Liu et al., 2025, Guo et al., 2025, Ma et al., 2025].
Given these successes, we investigate whether TTS can enhance performance on HSCodeComp. Specifically, we evaluate two established TTS strategies [Liu et al., 2025]: (1) Majority Voting: We implement majority voting across $K=\{1,2,4,8,16\}$ independent trials (Voting@$K$), using SmolAgent (GPT-5). Figure [6](https://arxiv.org/html/2510.19631v1#S6.F6) showns that increasing $K$ yields negligible performance improvement; (2) Self-Reflection: We integrate a self-reflection mechanism into SmolAgent (GPT-5), enabling the model to proactively evaluate and revise its reasoning and actions. However, this approach slightly decreases performance from 42.72% to 42.57%.
These results demonstrate a key limitation of standard TTS methods when applied to HSCode prediction, highlighting the need of more effective test-time scaling strategy for agents in hierarchical rule application.

## 7 Conclusion

We identified and addressed the critical gap in evaluating deep search agents in hierarchical rule applications. To address this gap, we introduced HSCodeComp, the first realistic and expert-level benchmark designed to assess agents for multi-hop reasoning with hierarchical tariff rules in e-commerce domain. Our extensive evaluation revealed a substantial performance gap between current state-of-the-art agents (46.8%) and human experts (95.0%), highlighting that hierarchical rule application remains a significant challenge for existing agent architectures. We will release the HSCodeComp to accelerate research in this crucial capability for real-world agent deployment.

## 8 Ethics Statement

This research adheres to strict ethical guidelines regarding data privacy and fair labor. The dataset is fully anonymized and contains no personally identifiable information.
The hourly wage of our human annotators is over 34.6 USD, which
is much higher than average hourly wage 3.13 USD on Amazon Mechanical Turk [Hara et al., 2017].
This remuneration structure was designed to provide a fair and competitive wage, acknowledging the expertise and effort required for this task and ensuring that contributors were rewarded appropriately for their work.

## 9 Reproducibility statement

We are committed to the principles of reproducible research. Accordingly, all necessary materials, including code, benchmark dataset and other related resources will be publicly released to promote the development of the deep search agents.
For security and compliance reasons, the product URLs and Image have been removed from this version of our proposed HSCodeComp dataset.

## References

- Bercovich et al. [2025]
A. Bercovich, I. Levy, I. Golan, M. Dabbah, R. El-Yaniv, O. Puny, I. Galil, Z. Moshe, T. Ronen, N. Nabwani, I. Shahaf, and O. T. et al.
Llama-nemotron: Efficient reasoning models, 2025.
URL [https://arxiv.org/abs/2505.00949](https://arxiv.org/abs/2505.00949).
- Chen et al. [2025a]
K. Chen, Y. Ren, Y. Liu, X. Hu, H. Tian, T. Xie, F. Liu, H. Zhang, H. Liu, Y. Gong, C. Sun, H. Hou, H. Yang, J. Pan, J. Lou, J. Mao, J. Liu, J. Li, K. Liu, K. Liu, R. Wang, R. Li, T. Niu, W. Zhang, W. Yan, X. Wang, Y. Zhang, Y.-H. Hung, Y. Jiang, Z. Liu, Z. Yin, Z. Ma, and Z. Mo.
xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations, 2025a.
URL [https://arxiv.org/abs/2506.13651](https://arxiv.org/abs/2506.13651).
- Chen et al. [2025b]
S. Chen, P. Moreira, Y. Xiao, S. Schmidgall, J. Warner, H. Aerts, T. Hartvigsen, J. Gallifant, and D. S. Bitterman.
Medbrowsecomp: Benchmarking medical deep research and computer use, 2025b.
URL [https://arxiv.org/abs/2505.14963](https://arxiv.org/abs/2505.14963).
- Chollet et al. [2025]
F. Chollet, M. Knoop, G. Kamradt, B. Landers, and H. Pinkard.
Arc-agi-2: A new challenge for frontier ai reasoning systems, 2025.
URL [https://arxiv.org/abs/2505.11831](https://arxiv.org/abs/2505.11831).
- DeepSeek-AI et al. [2025]
DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, and Z. G. et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Fang et al. [2025]
T. Fang, Z. Zhang, X. Wang, R. Wang, C. Qin, Y. Wan, J.-Y. Ma, C. Zhang, J. Chen, X. Li, H. Zhang, H. Mi, and D. Yu.
Cognitive kernel-pro: A framework for deep research agents and agent foundation models training, 2025.
URL [https://arxiv.org/abs/2508.00414](https://arxiv.org/abs/2508.00414).
- Grainger [2024]
A. Grainger.
Customs tariff classification and the use of assistive technologies.
*World Customs Journal*, 18(1):3–32, 2024.
[10.55596/001c.116525](https:/doi.org/10.55596/001c.116525).
- Guha et al. [2023]
N. Guha, J. Nyarko, D. E. Ho, C. Re, A. Chilton, A. Narayana, A. Chohlas-Wood, A. Peters, B. Waldon, and et al.
Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.
In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023.
URL [https://openreview.net/forum?id=WqSPQFxFRC](https://openreview.net/forum?id=WqSPQFxFRC).
- Guo et al. [2025]
J. Guo, Z. Chi, L. Dong, Q. Dong, X. Wu, S. Huang, and F. Wei.
Reward reasoning model, 2025.
URL [https://arxiv.org/abs/2505.14674](https://arxiv.org/abs/2505.14674).
- Hara et al. [2017]
K. Hara, A. Adams, K. Milland, S. Savage, C. Callison-Burch, and J. Bigham.
A data-driven analysis of workers’ earnings on amazon mechanical turk, 2017.
URL [https://arxiv.org/abs/1712.05796](https://arxiv.org/abs/1712.05796).
- Hu et al. [2025a]
L. Hu, J. Jiao, J. Liu, Y. Ren, Z. Wen, K. Zhang, X. Zhang, X. Gao, T. He, F. Hu, Y. Liao, Z. Wang, C. Yang, Q. Yang, M. Yin, Z. Zeng, G. Zhang, X. Zhang, X. Zhao, Z. Zhu, H. Namkoong, W. Huang, and Y. Tang.
Finsearchcomp: Towards a realistic, expert-level evaluation of financial search and reasoning, 2025a.
URL [https://arxiv.org/abs/2509.13160](https://arxiv.org/abs/2509.13160).
- Hu et al. [2025b]
M. Hu, Y. Zhou, W. Fan, Y. Nie, B. Xia, T. Sun, Z. Ye, Z. Jin, Y. Li, Q. Chen, Z. Zhang, Y. Wang, Q. Ye, B. Ghanem, P. Luo, and G. Li.
Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025b.
URL [https://arxiv.org/abs/2505.23885](https://arxiv.org/abs/2505.23885).
- Huang et al. [2025]
K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu.
Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments.
In *Proceedings of NAACL 2025 (Long Papers)*, Albuquerque, New Mexico, 2025. Association for Computational Linguistics.
[10.18653/v1/2025.naacl-long.194](https:/doi.org/10.18653/v1/2025.naacl-long.194).
URL [https://aclanthology.org/2025.naacl-long.194/](https://aclanthology.org/2025.naacl-long.194/).
- Hussain and Ahmed [2023]
A. Hussain and A. Ahmed.
Auto-classification of harmonized tariff codes using chatgpt.
In *4th International Conference on Distributed Sensing and Intelligent Systems (ICDSIS 2023)*, volume 2023, pages 262–275. IET, 2023.
- Joshi et al. [2017]
M. Joshi et al.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
In *ACL*, 2017.
- Judy [2024]
B. Judy.
Benchmarking harmonized tariff schedule classification models.
*arXiv preprint arXiv:2412.14179*, 2024.
URL [https://arxiv.org/abs/2412.14179](https://arxiv.org/abs/2412.14179).
- Lee et al. [2024]
E. Lee, S. Kim, S. Kim, S. Jung, H. Kim, and M. Cha.
Explainable product classification for customs.
*ACM Transactions on Intelligent Systems and Technology*, 15(2):1–24, 2024.
- Li et al. [2025a]
H. Li, J. Chen, J. Yang, Q. Ai, W. Jia, Y. Liu, K. Lin, Y. Wu, G. Yuan, Y. Hu, W. Wang, Y. Liu, and M. Huang.
Legalagentbench: Evaluating llm agents in legal domain.
In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, Vienna, Austria, 2025a. Association for Computational Linguistics.
[10.18653/v1/2025.acl-long.116](https:/doi.org/10.18653/v1/2025.acl-long.116).
URL [https://aclanthology.org/2025.acl-long.116/](https://aclanthology.org/2025.acl-long.116/).
- Li et al. [2025b]
K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, W. Shen, J. Zhang, D. Zhang, X. Wu, Y. Jiang, M. Yan, P. Xie, F. Huang, and J. Zhou.
Websailor: Navigating super-human reasoning for web agent, 2025b.
URL [https://arxiv.org/abs/2507.02592](https://arxiv.org/abs/2507.02592).
- Liao et al. [2024]
M. Liao, L. Huang, J. Zhang, L. Song, and B. Li.
Enhanced hs code classification for import and export goods via multiscale attention and ernie-bilstm.
*Applied Sciences*, 14(22):10267, 2024.
[10.3390/app142210267](https:/doi.org/10.3390/app142210267).
- Liu et al. [2025]
Z. Liu, P. Wang, R. Xu, S. Ma, C. Ruan, P. Li, Y. Liu, and Y. Wu.
Inference-time scaling for generalist reward modeling, 2025.
URL [https://arxiv.org/abs/2504.02495](https://arxiv.org/abs/2504.02495).
- Lyu et al. [2025]
Y. Lyu, X. Zhang, L. Yan, M. de Rijke, Z. Ren, and X. Chen.
Deepshop: A benchmark for deep research shopping agents, 2025.
URL [https://arxiv.org/abs/2506.02839](https://arxiv.org/abs/2506.02839).
- Ma et al. [2025]
Z.-A. Ma, T. Lan, R.-C. Tu, S.-H. Liu, H. Huang, Z. Wu, C. Xu, and X.-L. Mao.
T2i-eval-r1: Reinforcement learning-driven reasoning for interpretable text-to-image evaluation, 2025.
URL [https://arxiv.org/abs/2505.17897](https://arxiv.org/abs/2505.17897).
- Mialon et al. [2023a]
G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom.
Gaia: a benchmark for general ai assistants.
*arXiv preprint arXiv:2311.12983*, 2023a.
URL [https://arxiv.org/abs/2311.12983](https://arxiv.org/abs/2311.12983).
- Mialon et al. [2023b]
G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom.
Gaia: a benchmark for general ai assistants, 2023b.
URL [https://arxiv.org/abs/2311.12983](https://arxiv.org/abs/2311.12983).
- Nath et al. [2025]
S. Nath, S. Wadhwa, and L. Perez.
Domain-adaptive small language models for structured tax code prediction, 2025.
URL [https://arxiv.org/abs/2507.10880](https://arxiv.org/abs/2507.10880).
- Peeters et al. [2025a]
R. Peeters, A. Steiner, L. Schwarz, J. Y. Caspary, and C. Bizer.
Webmall – a multi-shop benchmark for evaluating web agents, 2025a.
URL [https://arxiv.org/abs/2508.13024](https://arxiv.org/abs/2508.13024).
- Peeters et al. [2025b]
R. Peeters, A. Steiner, L. Schwarz, J. Y. Caspary, and C. Bizer.
Webmall – a multi-shop benchmark for evaluating web agents.
*arXiv preprint arXiv:2508.13024*, 2025b.
URL [https://arxiv.org/abs/2508.13024](https://arxiv.org/abs/2508.13024).
- Phan et al. [2025]
L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, S. Shi, M. Choi, et al.
Humanity’s last exam.
Center for AI Safety and Scale AI (whitepaper / dataset), 2025.
URL [https://lastexam.ai/](https://lastexam.ai/).
- Roucher et al. [2025]
A. Roucher, A. V. del Moral, T. Wolf, L. von Werra, and E. Kaunismäki.
‘smolagents‘: a smol library to build great agentic systems.
[https://github.com/huggingface/smolagents](https://github.com/huggingface/smolagents), 2025.
- Sadowski and Chudziak [2025]
A. Sadowski and J. A. Chudziak.
Explainable rule application via structured prompting: A neural-symbolic approach, 2025.
URL [https://arxiv.org/abs/2506.16335](https://arxiv.org/abs/2506.16335).
- Servantez et al. [2024]
S. Servantez, J. Barrow, K. Hammond, and R. Jain.
Chain of logic: Rule-based reasoning with large language models.
In L.-W. Ku, A. Martins, and V. Srikumar, editors, *Findings of the Association for Computational Linguistics: ACL 2024*, pages 2721–2733, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[10.18653/v1/2024.findings-acl.159](https:/doi.org/10.18653/v1/2024.findings-acl.159).
URL [https://aclanthology.org/2024.findings-acl.159/](https://aclanthology.org/2024.findings-acl.159/).
- Shridhar et al. [2021]
M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning.
In *ICLR*, 2021.
URL [https://arxiv.org/abs/2010.03768](https://arxiv.org/abs/2010.03768).
- Shubham et al. [2022]
Shubham, A. Arya, S. Roy, and S. Jonnala.
An ensemble-based approach for assigning text to correct harmonized system code.
*arXiv preprint arXiv:2211.04313*, 2022.
URL [https://arxiv.org/pdf/2211.04313.pdf](https://arxiv.org/pdf/2211.04313.pdf).
- Stassin et al. [2023]
S. Stassin, O. Amel, S. Mahmoudi, and X. Siebert.
Similarity versus supervision: Best approaches for hs code prediction.
2023.
- Tao et al. [2025]
Z. Tao, J. Wu, W. Yin, J. Zhang, B. Li, H. Shen, K. Li, L. Zhang, X. Wang, Y. Jiang, P. Xie, F. Huang, and J. Zhou.
Webshaper: Agentically data synthesizing via information-seeking formalization, 2025.
URL [https://arxiv.org/abs/2507.15061](https://arxiv.org/abs/2507.15061).
- Team et al. [2025]
K. Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, and R. C. et al.
Kimi k2: Open agentic intelligence, 2025.
URL [https://arxiv.org/abs/2507.20534](https://arxiv.org/abs/2507.20534).
- Thomas et al. [2025]
G. Thomas, A. J. Chan, J. Kang, W. Wu, F. Christianos, F. Greenlee, A. Toulis, and M. Purtorab.
Webgames: Challenging general-purpose web-browsing ai agents.
In *arXiv preprint arXiv:2502.18356*, 2025.
- Wang et al. [2024]
S. Wang, Z. Wei, Y. Choi, and X. Ren.
Symbolic working memory enhances language models for complex rule application.
In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 17583–17604, Miami, Florida, USA, Nov. 2024. Association for Computational Linguistics.
[10.18653/v1/2024.emnlp-main.974](https:/doi.org/10.18653/v1/2024.emnlp-main.974).
URL [https://aclanthology.org/2024.emnlp-main.974/](https://aclanthology.org/2024.emnlp-main.974/).
- Wei et al. [2025]
J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese.
Browsecomp: A simple yet challenging benchmark for browsing agents, 2025.
URL [https://arxiv.org/abs/2504.12516](https://arxiv.org/abs/2504.12516).
- Wu et al. [2025]
J. Wu, W. Yin, Y. Jiang, Z. Wang, Z. Xi, R. Fang, L. Zhang, Y. He, D. Zhou, P. Xie, and F. Huang.
Webwalker: Benchmarking llms in web traversal, 2025.
URL [https://arxiv.org/abs/2501.07572](https://arxiv.org/abs/2501.07572).
- Xu et al. [2025]
W. Xu, Y. Mao, X. Zhang, C. Zhang, X. Dong, M. Zhang, and Y. Gao.
Dagent: A relational database-driven data analysis report generation agent, 2025.
URL [https://arxiv.org/abs/2503.13269](https://arxiv.org/abs/2503.13269).
- Yang et al. [2025]
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, and B. Y. et al.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Yao et al. [2024]
H. Yao, Z. Jing, T. Liu, E. J. Wong, C. Hong, I. Wang, W.-L. Wang, Y. Talebirad, R. Wang, Z. Chen, et al.
Webvoyager: Building an end-to-end web agent with large multimodal models.
In *ICLR*, 2024.
- Yao et al. [2022]
S. Yao, H. Chen, J. Yang, and K. R. Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents.
In *NeurIPS*, 2022.
URL [https://arxiv.org/abs/2207.01206](https://arxiv.org/abs/2207.01206).
- Yu et al. [2025a]
A. Yu, L. Yao, J. Liu, Z. Chen, J. Yin, Y. Wang, X. Liao, Z. Ye, J. Li, Y. Yue, H. Xiao, H. Zhou, C. Guo, P. Wei, J. Liu, and J. Gu.
Medresearcher-r1: Expert-level medical deep researcher via a knowledge-informed trajectory synthesis framework.
*arXiv preprint arXiv:2508.14880*, 2025a.
URL [https://arxiv.org/abs/2508.14880](https://arxiv.org/abs/2508.14880).
- Yu et al. [2025b]
A. Yu, L. Yao, J. Liu, Z. Chen, J. Yin, Y. Wang, X. Liao, Z. Ye, J. Li, Y. Yue, H. Xiao, H. Zhou, C. Guo, P. Wei, J. Liu, and J. Gu.
Medresearcher-r1: Expert-level medical deep researcher via a knowledge-informed trajectory synthesis framework, 2025b.
URL [https://arxiv.org/abs/2508.14880](https://arxiv.org/abs/2508.14880).
- Yu et al. [2025c]
C. Yu, S. Lu, C. Zhuang, D. Wang, Q. Wu, Z. Li, R. Gan, C. Wang, S. Hou, G. Huang, W. Yan, L. Hong, A. Xue, Y. Wang, J. Gu, D. Tsai, and T. Lin.
Aworld: Orchestrating the training recipe for agentic ai, 2025c.
URL [https://arxiv.org/abs/2508.20404](https://arxiv.org/abs/2508.20404).
- Zhang et al. [2025]
W. Zhang, L. Zeng, Y. Xiao, Y. Li, C. Cui, Y. Zhao, R. Hu, Y. Liu, Y. Zhou, and B. An.
Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving, 2025.
URL [https://arxiv.org/abs/2506.12508](https://arxiv.org/abs/2506.12508).
- Zhou et al. [2025]
P. Zhou, B. Leon, X. Ying, C. Zhang, Y. Shao, Q. Ye, D. Chong, Z. Jin, C. Xie, M. Cao, Y. Gu, S. Hong, J. Ren, J. Chen, C. Liu, and Y. Hua.
Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese, 2025.
URL [https://arxiv.org/abs/2504.19314](https://arxiv.org/abs/2504.19314).
- Zhou et al. [2023]
S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig.
Webarena: A realistic web environment for building autonomous agents.
In *arXiv preprint arXiv:2307.13854*, 2023.
URL [https://webarena.dev/](https://webarena.dev/).

## Appendix A The Use of Large Language Models (LLMs)

In preparing this manuscript, Qwen-MAX and ChatGPT were used solely as a writing assistant to improve grammar and clarity. The LLMs was not used for generating code, concepts, or any part of the core research methodology.

## Appendix B Dataset distribution

Figure [7](https://arxiv.org/html/2510.19631v1#A2.F7) presents the distributions of first-level product categories (left) and HSCode chapter categories (right), which closely mirror real-world product distributions. This alignment confirms that HSCodeComp accurately reflects practical international trade scenarios, ensuring that model performance evaluations reliably generalize to real-world applications.

Figure: ((a)) First-level product categories
Refer to caption: x8.png

## Appendix C Semantic Distribution of Hierarchical Tariff Rules

To assess whether the HSCode taxonomy exhibits clear semantic separation, we generate embeddings of the official English titles and notes for all HS chapters and sections using a sentence embedding model(^5^55[https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)). We then apply t-SNE to project these embeddings into two dimensions for visualization.
As shown in Figure [8](https://arxiv.org/html/2510.19631v1#A3.F8), each point represents a chapter, while each star marks a section’s centroid. The visualization reveals significant semantic overlap between adjacent sections: numerous chapters appear closer to neighboring section centroids than to their own section’s centroid, and section centroids themselves form overlapping clusters rather than distinct groupings. This pattern indicates that the semantic structure of hierarchical tariff rules lacks clear boundaries—adjacent sections frequently share similar vocabulary and concepts (e.g., distinctions between raw materials and finished goods, or between component parts and complete articles).

Figure: Figure 8: The semantic map of HS chapter titles and notes.
Refer to caption: figures/hs2_section_semantic_map.png

## Appendix D More Detailed Experiments on Open-source Agents with Different Backbone LLMs

To investigate how the backbone LLMs affect the performance of the agent system, we conduct more detailed ablation study on four open-source agent systems, replacing the original GPT-5 backbone with Gemini 2.5 Pro.
The experimental results in Table [7](https://arxiv.org/html/2510.19631v1#A4.T7) indicate that GPT-5 achieves consistently better performance than the advanced Gemini 2.5 pro model on these open-source agent systems.

**Table 7: The ablation study of backbone models in the open-source agent system.**
| Backbone LLM | HSCode Prediction Accuracy |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 2-digit | 4-digit | 6-digit | 8-digit | 10-digit |  |
| SmolAgent |  |  |  |  |  |
| GPT-5 | 82.28 | 70.89 | 59.81 | 49.05 | 42.72 |
| Gemini-2.5-Pro | 82.19 | 69.48 | 57.87 | 44.04 | 34.49 |
| Claude 4 Sonnet | 80.69 | 67.09 | 54.11 | 42.25 | 33.70 |
| Qwen-MAX | 77.34 | 63.23 | 42.47 | 26.62 | 17.43 |
| Aworld |  |  |  |  |  |
| GPT-5 | 82.28 | 70.41 | 59.18 | 48.58 | 41.30 |
| Gemini 2.5 Pro | 79.55 | 66.97 | 54.70 | 38.79 | 29.24 |
| WebSailor |  |  |  |  |  |
| GPT-5 | 81.64 | 70.56 | 57.27 | 43.98 | 35.44 |
| Gemini 2.5 Pro | 78.79 | 67.58 | 56.21 | 42.27 | 31.21 |
| AgentOrchestra |  |  |  |  |  |
| GPT-5 | 82.12 | 70.73 | 60.44 | 47.78 | 41.30 |
| Gemini 2.5 Pro | 82.27 | 69.39 | 56.97 | 41.36 | 30.61 |

## Appendix E Improvement Gain from Agents

This waterfall in Figure [9](https://arxiv.org/html/2510.19631v1#A5.F9) chart reveals that the superior performance of SmolAgent (GPT-5) are primarily from reducing the outdated and hallucination failures, with 56 corrected samples in HScodBench.
Besides, as shown in Figure [10](https://arxiv.org/html/2510.19631v1#A5.F10), it can be found that the rate of outdated and hallucination are significantly reduced in SmolAgent baseline.

Figure: Figure 9: Details of performance gain and loss comparing GPT-5 and Smolagents.
Refer to caption: figures/agent_llm_improvement.png

Figure: Figure 10: Outdated or hallucination ratio happended on GPT-5 and Smolagent. The numbers are number and ratio of the phenomenon.
Refer to caption: figures/outdated_hallucination_ratio_gpt5_smolagent_comparison.png

## Appendix F Implementing details and Knowledge Forms

All baseline methods are equipped with search tools to access the CROSS database, hierarchical tariff rules, and other related resources, including human-written knowledge bases and hierarchical decision rules.
The temperature and context window size of LLMs and agents are set to their default configurations.
Moreover, as described in Section [5.2](https://arxiv.org/html/2510.19631v1#S5.SS2), the hierarchical decision rules, and webpage visit tool are not used during evaluation, since they do not improve the performance of open-source agents. But we do not restrict webpage visit of closed-source agents since we cannot control.
The multi-modal product images are used for open-source agents, i.e. SmolAgents.
The hierarchical decision rules used in our prompts are illustrated in Figure [13](https://arxiv.org/html/2510.19631v1#A6.F13).
The hierarchical tariff rules in the eWTP is shown in Figure [11](https://arxiv.org/html/2510.19631v1#A6.F11).
It can be found that the red boxes highlight implicit logical relationships in the tariff rules, such as excluding articles of heading 8593 and with the machines of heading 8501 or 8502. The blue boxes highlight vague descriptions in the tariff rules, such as “…for example …” and “…such as …”. These cases demonstrate that rule boundaries are ambiguous, posing significant challenges for accurate rule application by the agent.
Moreover, the U.S. Customs Rulings Online Search System (CROSS) interface is shown in Figure [12](https://arxiv.org/html/2510.19631v1#A6.F12).
As illustrated, the CROSS website contains not only correct precedent results for product HS Codes but also numerous revoked precedents, requiring the agent to carefully evaluate information reliability. Additionally, since the precedent information is presented as plain text emails, the agent must effectively utilize contextual information to perform accurate reasoning.

Figure: Figure 11: The case of the hierarchical tariff rules.
Refer to caption: figures/tariff_rule.png

Figure: Figure 12: The case of CROSS website that contains the products rulings.
Refer to caption: figures/cross_example.png

Figure: Figure 13: Decision rules defined by human experts.

## Appendix G Case study of Failure Modes

We identify six critical failure modes of open-source and closed-source agent systems in HSCodeComp:
(1) Premature Decisions: Agents commit to incorrect classification paths without collecting sufficient evidence (Figure [14](https://arxiv.org/html/2510.19631v1#A7.F14) and Table [9](https://arxiv.org/html/2510.19631v1#A7.T9)-Grok DeepSearch);
(2) Information Misprocessing: Agents overlook or misinterpret key product details, indicating challenges with long-context processing (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8) and Figure [15](https://arxiv.org/html/2510.19631v1#A7.F15));
(3) Unnecessary Self-Correction: Agents sometimes predict correct HSCodes initially but revise them incorrectly through excessive critique (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8), Gemini Deep Research);
(4) Reasoning Hallucination: Agents generate plausible but factually incorrect reasoning steps (Table [8](https://arxiv.org/html/2510.19631v1#A7.T8), Grok DeepSearch);
(5) Wrong Rule Application: Models frequently miss or misuse relevant tariff rules due to their ambiguous descriptions that confuse the reasoning process, resulting in incorrect classification decisions (Figure [17](https://arxiv.org/html/2510.19631v1#A7.F17));
and (6) Lack of Domain Knowledge: Models exhibit errors due to insufficient domain-specific knowledge, such as misidentify silicone products as rubber instead of plastic (Figure [16](https://arxiv.org/html/2510.19631v1#A7.F16)).
These limitations highlight that HSCodeComp remains challenging for advanced closed-source and open-source systems.

Figure: Figure 14: Early wrong search query leads to wrong result.

Figure: Figure 15: Real-world noise forms and analysis.

Figure: Figure 16: Lack of domain knowledge

Figure: Figure 17: Wrong Rule Application.

**Table 8: Comparative Analysis of AI Model Classifications for a Forklift Safety Cage mentioned above.**
| Analysis Dimension | SmolAgent | Gemini DeepResearch | Grok DeepSearch | Manus |
| --- | --- | --- | --- | --- |
| Final HTSUS Code | 8431.20.0000 (Correct) | 7326.90.8688 (Incorrect) | 8428.90.0290 (Incorrect) | 8427.90.0020 (Incorrect) |
| Core Logic Explained | Based on the core principle of HTSUS Section XVI, Note 2, the cage is a part as it is ’solely or principally for use with’ a forklift (heading 8427). Its design, function, and identity are entirely dependent on the forklift. | The argument is based on the ’part vs. accessory’ distinction. It posits the cage is not an ’indispensable’ part, but an optional ’accessory’. Since accessories are precluded from 8431, classification defaults to its constituent material (steel). | Characterizes the cage as a functional piece of machinery. The rationale is that it enables a new function and incorrectly compares it to complex attachments with their own mechanics (e.g., rotators, clamps). | Characterizes the cage as a complete ’aerial work platform’. The core argument is that its ’4 universal wheels’ constitute a ’mobile base’ per the legal notes, thus assembling it into a complete vehicle. |
| Key Flaw Analysis | This approach correctly identifies the product’s primary use and applies the controlling legal note directly, which is the standard and most reliable method for classification. | This is overthinking because the model became fixated on a complex, secondary legal nuance (part vs. accessory) while ignoring the more direct, primary rule (’solely or principally for use with’), leading to an unnecessarily complicated and incorrect conclusion. | This is an analysis hallucination because the model invents characteristics the product lacks, effectively treating a passive structure as an active machine. The entire analysis is built on this fabricated, non-existent product feature. | The model misses key product information by misunderstanding the function of a key feature (the wheels). It correctly identifies the wheels but misses their trivial context (ground convenience), instead mistaking them for a vehicle’s chassis, which invalidates the entire classification. |
| Failure Modes | Correct | Unnecessary Self-Correction | Reasoning Hallucination | Information Misprocessing |

**Table 9: Comparative Analysis of AI Model Classifications for a Cylinder Shock Absorber mentioned above.**
| Analysis Dimension | SmolAgent | Gemini DeepResearch | Grok DeepSearch | Manus |
| --- | --- | --- | --- | --- |
| Final HTSUS Code | 8487.90.0080 (Correct) | 8412.21.0075 (Incorrect) | 8302.49.6085 (Incorrect) | 8412.31.0080 (Incorrect) |
| Core Logic Explained | Based on the hierarchical structure of HTSUS Chapter 84, the shock absorber is a generic machinery part. After systematically eliminating more specific headings, it correctly classifies the item in the residual heading 84.87 for parts "not elsewhere specified." | Characterizes the product as an active hydraulic motor. The logic is that because it is a linear-acting hydraulic device, it must be a "motor" under heading 84.12, which is an apparatus that generates force or motion. | It correctly identifies the passive function but then classifies it as a simple base metal fitting. The argument is that since it’s not a motor, its classification defaults to a general heading for common hardware and accessories. | Characterizes the product as an active pneumatic motor. The rationale is based on the keyword "Pneumatic Cylinder" in the title, concluding it must be an actuator that performs work under heading 84.12. |
| Key Flaw Analysis | This approach correctly identifies the product’s non-specific nature and applies the HTSUS’s hierarchical structure and residual headings, which is the standard and most reliable method for such goods. | This is a fundamental misunderstanding of the product’s function, as the model mistakes a passive energy-dissipating device (a damper) for an active power-generating device (a motor). It confuses braking with accelerating. | The model makes a decision with insufficient information about HTSUS structure because it fails to consider the critical distinction between Chapter 83 (simple fittings) and Chapter 84 (machinery) and thus underestimates the product’s nature as a piece of machinery. | This is a fundamental misunderstanding of the product’s function, as the model is misled by an inaccurate keyword and mistakes a passive damper for an active motor, ignoring contradictory product attributes. |
| Failure Modes | Correct | Information Misprocessing | Premature Decisions | Information Misprocessing |

## Appendix H Case Study of Overthinking

This section provides an in-depth case studies of open-source and closed-source agent systems, addressing two key observations: (1) the decreased performance brought by overthinking of open-source agents; and (2) the relatively lower performance of state-of-the-art closed-source agents compared to their open-source counterparts.

### H.1 Open-source Agent Cases

## Appendix I Case Study of Multi-modal Information in Agent Systems

This section presents experimental case studies comparing SmolAgent (GPT-5) performance with and without product image processing.

**Table 10: Vision vs Non-vision: Four Representative Cases with Key Evidence and Reasons.**
| Image | Product Information | Non-vision HS (10-digit) + brief rationale | With-vision HS (10-digit) + brief rationale | Vision advantage (reason category) |
| --- | --- | --- | --- | --- |
|  | Title: Royal Wedding Bouquet Rhinestone Bride and Bridesmaid Hand Flowers Handmade Bridal Bouquet.<br>Category: Artificial Decorations.<br>Attributes: Material: Silk (natural textile), not plastics; bouquet of artificial flowers with rhinestones. | 6702.90.3500 (Artificial flowers of *man-made fibers*).<br>Assumes “silk flowers” = polyester; treats bouquet as MMF artificial flowers. | 6702.90.6500 (Artificial flowers, *other than man-made fibers*).<br>Images/text indicate silk fabric (natural fiber); rhinestones do not change essential character (GRI 3(b)). | Material fiber identification (natural silk vs assumed polyester). |
|  | Title: OTF knife parts, aviation aluminum tactical handle kit.<br>Category: Hand Tools / Knife accessories.<br>Attributes: Handle parts only, no blade present; aluminum body with clip, actuator, screws. | 8211.93.0035 (Folding/pocket knives).<br>Interprets listing as a *complete folding knife* rather than components. | 8211.95.9000 (Handles of base metal, other).<br>Visuals confirm no blade; essential character is the base‑metal handle assembly (parts provision applies). | Completeness parts identification (parts‑only vs whole article). |
|  | Title: Acrylic buckle, beaded/openwork shoulder bag with inner pouch.<br>Category: Women’s Handbags.<br>Attributes: Outer surface is beads openwork lattice (ABS acrylic) ,<br>not plastic sheeting; linen pouch is interior. | 4202.22.1500 (Handbags with outer surface of *sheeting of plastics*).<br>Assumes exterior is plastic sheeting based on “ABS.” | 4202.29.9000 (Other).<br>Images show beads rings openwork, not “sheeting of plastics”; classification by outer surface (Additional U.S. Note 2 to Ch. 42). | Structural outer‑surface feature (beads openwork vs plastic sheeting). |
|  | Title: The Simpsons character collectible paper cards.<br>Category: Printed matter / collectibles.<br>Attributes: Cards bear pictures only; no rules-based deck specified; no lithographic process evidence. | 4911.99.6000 (Other printed matter, often linked to lithographic printing).<br>Assumes lithographic process without explicit evidence. | 4911.91.4040 (Pictures, designs and photographs; other).<br>Visuals confirm picture-only cards and lack of lithographic evidence/criteria; not a playing-card game. | Process evidence-based exclusion (no lithography evidence; pictures-only). |

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 0
presentation: 1
questions: I would suggest the authors read more good papers and see how a good paper is written.
soundness: 1
strengths: It has lots of LLM used
summary: I think this is more like a technical report than a research paper, way below ICLR standard.
weaknesses: Quality of the paper is very poor.

---

confidence: 2
contribution: 3
initial_rating: 4
presentation: 3
questions: Please see the questions raised in weakness
soundness: 3
strengths: 1. The motivation is clear. he paper clearly identifies a missing evaluation angle: hierarchical rule following, which is indeed a challenging and realistic reasoning task.

2. The dataset is comprehensive. The dataset is built with expert validation and seems to capture realistic product diversity and textual noise.

3. The experiments compare many models and agent frameworks, giving a broad and fair view of the task difficulty.
summary: This paper introduces HSCodeComp, a benchmark for evaluating deep search agents on hierarchical rule reasoning. The task is to predict 10-digit Harmonized System (HS) codes for e-commerce products based on noisy text descriptions, while following multi-layer tariff rules. The motivation is that most existing benchmarks only test structured or open-domain reasoning, but none evaluate agents’ ability to apply complex, human-written hierarchical rules, which is important for real-world expert systems like legal or trade automation.

The benchmark is built from real e-commerce data with expert annotations and covers 632 products across 27 chapters. The authors test various LLMs and agent systems, finding that even the best combination eaches only 46.8% accuracy, far behind humans (95%). Ablation shows that vague and interdependent rules are the main difficulty.
weaknesses: I am not an expert in search agent. My concerns are only raised from the research perspective not specific to this certain domian.

1. Only 632 samples might be too few to show robust performance differences.

2. Since rules come from different sources (tariff codes, human rulings, etc.), it would be useful to test which part contributes most to performance.

3. I wonder how well models perform at intermediate steps (like predicting subcategories).

4. Maybe models tuned for other structured domains (finance, medicine) could generalize better. A small cross-domain test would strengthen the claim that the challenge truly lies in hierarchical rule reasoning.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: $\textbf{Correct HS Hierarchy Definition and Terminology (Sec. 3 “Output,” p. 4)}$. The authors should fix the inaccurate description of the HS hierarchy. Clearly distinguish between the global levels (2-, 4-, 6-digit, standardized by the WCO) and the national extensions (8-, 10-digit, defined by each country’s tariff schedule). Ensure consistent use of “HS” versus “HTS” throughout the text.
soundness: 3
strengths: This paper tackles a timely and important challenge: applying rules for HS code classification rather than relying on open-ended retrieval. The motivation and problem space are clearly illustrated in Figure 1 (left side, page 2). 
The dataset and setup are realistic—inputs combine noisy product listings, structured attributes, images, and URLs—and ablation studies show that including images improves accuracy in several scenarios (Table 4, page 7; Table 10, page 36).
The data labeling process is carefully designed: two experts annotate each item, a senior adjudicator resolves disagreements, and a 10% spot check shows only 2% disagreement (Figure 3, page 5).
The authors compare a wide range of baseline models and report results consistently at all HS code levels (2-, 4-, 6-, 8-, and 10-digit). The benchmark includes strong models such as GPT-4o and Qwen variants (Table 1, page 6).
Finally, the paper provides thoughtful analysis, including (i) a study of “overthinking” behavior versus tool use (Table 5, page 7); (ii) an investigation of why scaling model size at test time brings limited gains (Figure 4, page 8); and (iii) a useful taxonomy of common failure types that highlights cases where predictions are “error-but-valid” (Figure 5, page 9).
summary: The paper releases \textit{HSCODECOMP}, a benchmark of 632 expert‑annotated e‑commerce products (27 HS chapters, 32 first‑level categories) to test agents’ ability to apply hierarchical tariff rules and predict 10‑digit HS/HTS codes from noisy, multi‑modal inputs (title, attributes, image, price, category, URL). Baselines cover 14 LLM/VLMs and six open‑source agents (plus three closed‑source systems on a 49‑item subset); the best system (SmolAgents + GPT‑5 (VLM)) attains 46.83% 10‑digit accuracy vs. 95.0% for human experts (Table 1, p.6; Fig. 1, p.2). The paper analyzes “overthinking,” test‑time scaling, image utility, and failure modes, and promises to release code and data. Claimed contributions: (i) a realistic, expert‑level benchmark for rule application (their “Level‑3” knowledge); (ii) a multi‑expert annotation/validation pipeline; (iii) broad LLM/agent baselines with analyses of think‑depth, images, and test‑time scaling
weaknesses: First, the current evaluation metric is too narrow. It only counts exact 10-digit matches as correct, even when the model predicts a valid but slightly different code. The authors themselves note that many predictions are “Error-but-Valid.” This shows a need for more flexible metrics—such as hierarchical distance, agreement at higher HS code levels (2, 4, 6, or 8 digits), and a rule-consistency score. As it stands, many reasonable answers are unfairly marked wrong (Section 4.2, p. 5; Figure 5, p. 9).

Second, there is a factual mistake in the HS taxonomy explanation. Section 3 (“Output,” p. 4) incorrectly claims that “the last four digits (from 6 to 10) are country-specific,” when in fact only digits 7–10 vary by country, while the 6-digit level is globally standardized. This should be corrected for accuracy.

Third, the source and authority of the “rules” used are ambiguous. The paper relies heavily on eWTP tariff rules (Figures 11–12; Section 4.1) but doesn’t explain how they align with official WCO or HTS legal notes or with U.S. CROSS rulings. Depending on a commercial taxonomy may lead to discrepancies with authoritative references.

Forth, the evaluation has fairness issues. Closed-source models were tested on only a 49-item subset (Table 2, p. 6), making their results not directly comparable to the 632-item open benchmark. The authors also disabled webpage retrieval because it slightly reduced accuracy, yet this choice makes the task less realistic for genuine research workflows.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 4
questions: 1. Is the intended task definition a flat 10-digit classification or hierarchical code selection step by step? Did the authors evaluate these two modes of classification?
2. Can you quantify “near misses”? How often is the model correct through 6 or 8 digits but off at the final branch? That seems essential for interpreting the ~46.8% 10-digit score. Can you quantify some distance metric in branches/leaves?
3. Have you tried a constrained or hierarchical decoder baseline, or a structured outputs approach that only allows valid descendants and blocks nonexistent or outdated codes? Did you experiment by replacing numerical codes with enums or hard string labels for language grounding? How much could this be a side effect of tokenization?
4. Which HTS revision (with date) are your gold labels tied to? And do you plan to release HSCodeComp as versioned snapshots over time to track tariff changes? How are you planning to update the benchmark when codes change in the future?
5. How many distinct 10-digit codes are in HSCodeComp, and what does the per-code sample count look like? Is this mostly one-shot per code, or do some leaves recur?
6. Could you include one concrete example where agent performance degraded when given a human tariff rule, and why? This is an interesting failure mode related to the task and can help future work understand where the gap stems from with regards to instruction following.
soundness: 2
strengths: 1. Practical task with real world impact. Tariff classification is a hierarchical categorization problem very similar to other industry classification specifications. The paper justifies the problem on how the classification directly affects duties, compliance risk, and audits.
2. Furthermore, the current gap between humans and model/agents is demonstrated: top LLM agents still fail ~50% of the time at the final 10-digit code. 
3. Clear Expert-created workflow for ground truth: annotators gather product attributes (materials, intended use, etc.), consult official rulings/decision rules, resolve disagreements, and escalate difficult items. This data is not easily made via synthetic generation. 
4. Layered evaluation is reported at 2/4/6/8/10 digits that’s useful for probing heretical failures in the classification. 
5. Error analysis names specific failure modes (“valid but not chosen,” “outdated code,” material confusion like silicone vs. rubber) which is useful in grounding the nuances of the task in relation to alignment with language.
summary: This paper introduces HSCodeComp, a benchmark of 632 expert-labeled products for assigning 10-digit customs tariff codes. The authors formulate the problem as high stakes for trade compliance and demonstrate a noticeable gap for LLMs, the best model/agent is ~46.8% exact match at 10 digits versus ~95% for human experts. The paper also reports accuracy at coarser HS levels (2/4/6/8/10 digits), shows common failure modes, and argues that automated tariff classification is still far from solved. The paper argues this gap demonstrates that current LLM agents are not yet reliable for trade classification.
weaknesses: 1. The paper evaluates hierarchical levels, but it’s not explicit whether HSCodeComp is meant to be a flat 10-digit prediction task, or a hierarchical decision process / constrained decoding task. Would this be simpler if the workflow commits to a 2-digit chapter, then refine constrained of the child nodes? That needs to be made explicit for reproducibility and fairness in future comparisons.
2. Metrics could be improved. Currently the paper uses exact-match, even if the model picked a code in the correct branch and only got the last two digits wrong. Furthermore, top n accuracy could be provided as an appendix. It would be useful to quantify near misses, sibling confusion, or a hierarchical distance for the analysis. The qualitative examples hint that these near misses are common. This weakens the interpretability of the 46.8% number.
3. Lack of a constrained-decoding/structured prediction  baseline. From my understanding, the baseline agents are allowed to output nonexistent or structurally invalid 10-digit codes, or codes that don’t match their own predicted parent. Would a trivial hierarchical decoder (predict chapter, restrict to valid children, backtrack if invalid) would cut out a whole class of hallucination errors? Not including that baseline makes it harder to tell how much of the gap is deep legal reasoning vs. just lack of structural constraints. Furthermore could this be a language vs token misalignment, and if the numerical codes were replaced with text labels, would the gap still persist?
4. Temporal stability is unclear, HTS codes change over time: some codes get split/retired, and new product categories appear. The paper discusses an outdated code failure mode. But it does not clearly state which HS/HTS revision date is considered authoritative for HSCodeComp, nor how future updates will be versioned. Without explicit versioning or a way to update future categorizations, long-term benchmarking and replication will be shaky. We see this in GICS classifications 
5. Representativeness and coverage is not clear, the dataset spans 27 HS chapters and 32 e-commerce categories, but in the dataset, how many unique 10-digit leaves are represented, how many examples per leaf, and whether this mix reflects everyday brokerage volume vs. being intentionally enriched for tricky, dispute-prone items. This matters for how generalizable the reported agent accuracy is on this set versus actual in production distributions. 
6. Rules hurting the agent needs one concrete example. The paper claims that giving agents human-written tariff “decision rules” sometimes degrades performance. That’s interesting and believable given similar nuances in legal and finance, but the paper doesn’t walk through a single case. One worked example would make that claim much more convincing.


Decision: Reject