Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules
Authors: Chenyu Zhou
Xiaoming Shi
Hui Qiu
Yankai Jiang
ShaoGuo Liu
Tingting Gao
Haitao Leng
Xiawu Zheng
Rongrong Ji
Venue: ICLR 2026 Poster (2026)
Keywords: Agent
LLM
E-commerce
Primary area: datasets and benchmarks

Abstract: E-commerce agents contribute greatly to helping users complete their e-commerce needs. To promote further research and application of e-commerce agents, benchmarking frameworks are introduced for evaluating LLM agents in the e-commerce domain.
Despite the progress, current benchmarks lack evaluating agents' capability to handle mixed-type e-commerce dialogue and complex domain rules. To address the issue, this work first introduces a novel corpus, termed Mix-ECom,
which is constructed based on real-world customer-service dialogues with post-processing to remove user privacy and add CoT process.
Specifically, Mix-ECom contains 4,799 samples with multiply dialogue types in each e-commerce dialogue, covering four dialogue types (QA, recommendation, task-oriented dialogue, and chit-chat),
three e-commerce task types (pre-sales, logistics, after-sales),  and 82 e-commerce rules.
Furthermore, this work build baselines on Mix-Ecom and propose a dynamic framework to further improve the performance.
Results show that current e-commerce agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination cased by complex domain rules. The dataset will be publicly available.

Published as a conference paper at ICLR 2026
MIX-ECOM: TOWARDS MIXED-TYPE E-COMMERCE DIALOGUES WITH COMPLEX DOMAIN RULES
Chenyu Zhou 12.-, Xiaoming Shi', Hui Qiu", Yankai Jiang, Shaoguo Liu?
Tingting Gao", Haitao Leng2., Xiawu Zheng*.. Rongrong Ji
Key Laboratory of Multimedia Trusted Perception and Efficient Computing. Ministry of Education of China, Xiamen University Kuaishou Technology
ABSTRACT
E-commerce agents contribute greatly to helping users complete their e-commerce needs. To promote further rescarch and application of e-commerce agents, bench- marking frameworks are introduced for evaluating LLM agents in the e-commerce domain. Despite the progress, current benchmarks lack evaluating agents' capa- bility to handle mixed-type e-commerce dialogue and complex domain rules. To address the issue, this work first introduces a novel corpus, termed Mix-ECom, which is constructed based on real-world customer-service dialogues with post- processing to remove user privacy and add CoT process. Specifically, Mix-ECom contains 4,799 samples with multiply dialogue types in each e-commerce dialogue, covering four dialogue types (QA, recommendation, task-oriented dialogue, and chit-chat), three e-commerce task types (pre-sales, logistics, after-sales), and 82 e-commerce rules. Furthermore, this work build basclines on Mix-Ecom and propose dynamic framework to further improve the performance. Results show that current e-commerce agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination cased by complex domain rules. The dataset will be publicly available.
INTRODUCTION
Large language models (LLMs) (Meta 2024 Jiang al. 2024 Team et al. 2024 Ouyang et al.
2022; OpenA] 2023 Brown et al. (020) have revolutionized the backbone of agents for various
application scenarios, such as medicine (Li al. 2024 Mishra et al. 2024), finance (Zhang et 2024 Axtell & Farmer 2025), education (Zoou 2025 Xu et al. 2024), e-commerce (Nie et al. 2024
Zeng et al. 2025), etc. Among these domains, e-commerce agents attract increasing attention due to their attractive application value (Palen-Michel et al. 2024 Wang et al., 2025b Pokrywka et al.)
2025 Li et al. 2025a b).
In the e-commerce domain, LLM-based agents contribute greatly to helping users complete their specific e-commerce needs, including real-world customer issues in the process of the pre-sale, logistics, and after-sale. To promote further research and application of e-commerce agents, several benchmarks have be proposed, as shown in Table [ Current benchmarks evaluate e-commerce agents' performance mainly simplistic user issues and e-commerce rules, which lack objective and fair evaluation of agents' performance in real e-commerce scenarios. As shown in Figure [ in real-world e-commerce dialogues, e-commerce rules complex, and real-world user needs dynamically changing. which requires agents to accurately understand complex domain rules and meet diverse user needs in one dialogue (the ability handle mixed-type dialogues).
To address the issue, this work first introduces a new mixed-type e-commerce customer-service dialogue dataset (Mix)ECom) for evaluating e-commerce agents' capability in real-world e-commerce dialogues. Specifically, high-quality mixed-type dialogues are first selected from 70,000 real world- service dialogues, which contains 4.799 dialogues, covering multiple dialogue types (task -rieete
"Equal Contribution
'Corresponding Author

---

Published as a conference paper at ICLR 2026
Eser time Fihor's bouRd workt bus ng agais
an an sas the nangoourr
Return Delivered Order Guidelnns
Shipping Insurance Guidelinss
rangetiess Will welly procnss you
Anchcant
the bust mu iecrisero dii time i'thare bailness.
Astistant
Shipping Insurancc sabsidess retur Shipping cort only celect cellers offer
Masimum subridyy 1D. User couer eacess shipping ontts
refunt
lccice at pooills Our hall Hor
Ressering Acting
Exanple Logistics
Thoughhs
Domain Rulles
aneessed Aniitant
infureiiity opplles check the awvn
ttts preduce hers if you
"2115on"' "agitti,, of "9hood"
mangoriees Rondamb narder
brow you aedd furber stance.
*/ Thoghht
<Actie Jeguts
API Tools
Database
"leglttic, dtoss "Delamee" Tracelas addres "Annaju Ctty Corg soredaa Milba
betveoge pevel os cungatiees
Assistant
Contrrad margztiers are thah
Theughts
lagisterr sédres aadees j---tir.....
"rellct, Fire"" Tun 15.077 Spipn Sime"" Tunns 12.00" "agguoff,
nongssterstt tie hove rnueves large rumber partius
for aoof and further action.
</Thaaght>
"velund_tmmm""
Foo surse" "rals usett 'log".
ciladitee belolddett...tpingg.oo stwppingg.sgtott ond s.tnee et, mokies addreec et,
"ared addrvee" "No bijjang Dinnee Fuetou, Falanr
Asiistan
Wi'll
fous dooe wnee ender rnttrrrded
Thassumn
Figure I: An example in Mix-ECom. The assistant needs verify the complaint based on the image provided and then take appropriate follow-up actions accordingly with given domain rules, database, and tools.
dialogue, recommendation, QA, chitchat) in one dialogue together, with 82 domain rules. Given the dialogues, the relative domain rules, API tools, and logistics database are also provide for further research, as shown in Figure Furthermore, to ensure the high quality of the dataset, post- peos-procsssing of privacy..reserving. chain-of-thought adding, manual filtering are conducted.
To analyze current agents' performance on Mix-ECom, closed-source LLMs and open-source LLMs are utilized the backhone of agents under the setting of O-shot. Besides, for further analysis, the open-source LLM is further fine-tuned with e-commerce dataset. Extensive experiments find that current agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination caused by complex domain rules. To address this issue, this work proposes the dynamic e-commerce framework, which utilizes dynamic module to select closely-related rules and reduce disturb from irrelevant rules, thus improving performance.
This work makes the following contributions:
For objective and fair evaluation of agents' performance in real e-commerce scenarios, propose a novel benchmarking with real-world mixed-type e-commerce dialogue dataset, termed Mix-Ecom, in which each session has rich variability of dialogue types and user intents, with 82 domain rules, API tools, and logistics database.
To promote further research on e-commerce agents, we build baselines on Mix-ECom and propose dynamic e-commerce framework to improve the performance on complex e-commerce rules.
Experimental results show current agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination caused by complex domain rules. Besides, the results show the effectiveness of the proposed dynamic e-commerce framework.
2 RELATED WORK
LLM-based Agents. Research on intelligent agents powered by LLMs represents significant frontier in artificial intelligence. The open-source community has contributed several influential frameworks, including ReAct (ao et al. 2023). Plan and Solve (Wang et al. 2023) LangChain (Chase) 2022), and AutoGPT (Gravitas] (023). which provide foundational architectures for agent development. Besides, domain-specifie enhancements have been achieved through specialized tool integration.
2

---

Published as a conference paper at ICLR 2026
E-com Task Types
Benchmark
Image
Video
# of Rule
Mixed-type
Pre
Logi
After
EcomScriplBench (Wan al.
2025c
x
x
x
/
x
X
CBYS (Zeng al.
2025
x
x
x
x
X
x
RECBENCH-MI (Liu ct al.
2025
x
V
V
X
Tau-retail Yao et al. 2024
x
X
X
32
ECom-Bol Bench (Wang et al.
2025a)
X
/
X
Mix- Mix-ECom- ECom-Bench (Ours)
82
Table 1: Comparison of e-commerce benchmarks. "-" indicates that the metric is not publicly available. "Pre", "Logi", and "After" represents pre-sales, logistics, and after-sales, respectively. "g" represents four dialogue types: QA, recommendation, task-triented dialogue, and chitchat.
Search capabilities have been advanced through systems like WebGPT (Nakano et al. 2021 and WebCPM (Qin et al. 2023), while RestGPT Song et al. (023) has demonstrated the potential of combining LLMs with RESTful APIs for web service development. However, there has been relatively little research focused on the domain of e-commerce customer service. To address the specific challenges in this scenario, we propose a novel dynamic e-commerce agent framework.
Benchmark for E-commerce Agents. Current benchmarks for evaluating e-commerce agents increasingly evolving from uni-modal multi-modal and multi-task settings. However, most existing benchmarks still offer only partial coverage of the e-commerce domain. For instance, EcomScriptBench (Wang et al. 2025c) supports scripted dialogue generation with both text and image inputs, but does not incorporate video. CBYS (Zeng et al. 2025) is confined text-only inputs and limited to product suestion-answeree tasks. RECBENCH-MD (Liu et al.) 2025) focuses primarily on product recommendation. Although Tau-retail (aoo et al. 2024) introduces the concept of domain-specifie policy, its policy representations are overly simplified and deviate significantly from those used in real-world applications. To address these limitations, propose a comprehensive benchmark that spans the full spectrum of e-commerce tasks and incorporates both image and video modalities with complex domain rules.
3 DATASET CONSTRUCTION
3.1 DATASET FORMULATION
The mixed-type e-commerce dialogue generation aims to generate responses R. based multi-modal files FF (images sent by the customer, product detail images from the knowledge base, and recent live-streaming clips related to the product), user queries Q, domain rules P, tools T.
In the dataset, each sample contains seven items, represented as {v, T, ar, o, K, 0}. The user profile {u", =d) includes the basic information of the current customer u and their specific demands wd, which are used to guide the LLM in simulating both customer and agent interactions. T represents the reference plan to resolve the problem of customer, along with the reasoning process used to generate the plan. The action chain a = } { each action is a; = {a.TH "a"), where am" is the tool name used in action; and of and indicate arguments and action response. k represents the key answers the assistant must convey to the customer in this task. 0 denotes the question type derived from the user profile Finally, ó represents the database information, which directly visible the assistant but can be accessed indirectly through the tool set T.
3.2 DATA CONSTRUCTION
3.2.
DOMAIN RULES, DATABASE AND TOOLS
The domain rules P explicitly defines the rules that the assistant must follow. It is derived from real- world e-commerce customer service practices and comprises 82 detailed rules, posing significant challenge to the mnsrruttionffolloiin capability of model. Further details are provided in the Appendix B.T The database D is sanitized version of real business data, stored in JSON format,
3

---

Published as a conference paper at ICLR 2026
Uver Agent Intinkesise
Real World User Portrait
Real World Customer
Service Conversation
Guldeline Injection
Agent
Agent
User Profile Goneration
CustomenServioo
proceeding
Use Profile
oi bought veice
desorigh Tiitended
recaivata mncircic racidl vithett
catt-oottlhll divalase
controlled version but neco snd the mechanical model insteed
Prhrgrpmeents
Geverpiik
Bekoland
and return
Gascrated Thisk Pravco
centroded vendas.
-Tangin
pa
Denngraphcc
AAaee lepart TTaliarac' "pc lic doal? TAggit ("ace dit 15854)) viactia, Japar -Muumation (Podect Nasal Goalted Resce Rotuxl -0Oooontion
Clasification
<Twagght -Thospht
User Frickin
Canpee
Quectics Type
-Tted Jeaver
Quilyy Jaaa
Reded Prcosiio
Probice Categary: Aften-Sake
Hels dear Widne hezr about the ducrepancy with your cir crtu wern es though offer 7-day ne- palccy your converioncs
Wong lems
Erpué Server
Rovaltta: tickadd and Retura
Product Typec Noo-pericablee
Prirers prowniing
would requect nefurd and reture, Pleose provide
prowning
and hurvan
Post Process
Key Avawer
Xiber Sicks ltaies
Fllte Medile
Arrigg sefasl and rears for
Shägping Rtaks
Jefurn the carvewr the oitmn diyiing in you
Conversation
Sinviched
Dutakere
High-Qualty
Dale
Coopua Terrs
Kay Apreet
The construction process of Mix-ECom The user profile, user demands, key answers. relative rules are first extracted from real world user portraits and customer-service conversations. Then, the dialogues are revised with post-processing of privacy-preserving and CoT adding. Finally, the dataset is reviewed manually to ensure high-qualit,
Figure 2:
and includes logistics, order, product, merchant, and user databases. The contents of the database are not directly visible to the user or assistant agents; they can only interact with it through predefined Tools. The tools T are manually written Python functions categorized into read, write, and converse operations, as detailed in the Appendix
3.2.2 DATA CONSTRUCTION PIPLINE
User profile generation. As illustrated in Fig 70,000 real-world mustomer-serviee conversations are collected, along with real world user portraits derived from historical behaviors of users. GPT-4o is employed summarize this information and generate corresponding user profiles. For logistics- related tasks, where customer requests are relatively straightforward, multiple user needs are combined increase the complexity of the tasks. In after-sales tasks, images sent by users in actual customer service dialogues are preserved as multi-modal files F. These are provided by the user agent during the roll-out process to validate the authenticity of their complaints. For pre-sales sales tasks, product detail images and recent live-stream clips are stored as mulli-moda files F. The model is required to extract relevant information from these files to answer product-related questions.
Demand Classification. Each task is being classified into specific question types & through analysis of the User Profile, together with the status of products, orders, and logistics. Further details regarding question type generation are provided in Appendix B.2
Key Answer Generation. The key answer (k) and the ground truth database are being generated based on the identified question type (0) and predefined rules. The key answer is the information that must be conveyed to the user by the assistant agent.
Rule Filter. Due to the complexity of the complete Domain Rules P, they are categorized into multiple sub-rules based on their content. For each task, filtering has been performed according to the question type 0 of the task, retaining only the rules relevant to the current task. This rule filtering mechanism has significantly reduced hallucinations in the Assistant Agent and thereby improved the usability rate of the generated data.
Customer-Service Dialogue Post-processing. The user profile is being utilized as prompt to simulate the customer through D1 thereby constituting the user agent. Concurrently, the
4

---

Published as a conference paper at ICLR 2026
assistant agent is constructed with GPT-4o serving as the backbone and the ReAct framework guiding its architecture. Real-world customer service dialogues are being employed as input for both the user agent and the assistant agent. These agents are being directed to rewrite the dialogue content, filter out privacy-sensitive information, and-in the case of the assistant agent-reconstruct the reasoning process in ReAct format based responses originally provided by human assistants.
The agent receives a query enclosed by couestion>> and </Quest ion>. It then generates a reasoning process enclosed by <Thought> and < </Thought>, followed by tool call enclosed by <Acti Daction_inputs and </Act cooiionninii inn_iput>> After interacting with the environment through the tool, the agent receives the result enclosed by <Observat waation> and ficient information is gathered, the agent produces a final response enclosed by <Fin @final. Answer> and </F1 nal Answer? marking the end of the thinking process.
Once suf-
3.3 DATA QUALITY CONTROI.
stree-stage pipeline employed for curating the e-commerce dialogue corpus.
Manual Filtering of User Profiles and Question Types. Manual filtering is being performed based on the obtained user profiles and question types to remove substandard user profiles as well as data with incorrect question type annotations.
GPT-4o Filtering. During the rollout phase, conversation content from interactions between Assistant Agent and the User Agent, along with the resulting database state, is being collected. GPT-4o is then being utilized to evaluate two aspects: first, whether all Key Answers are present in the dialogue; and second, whether the database matches the ground truth database.
Manual Meticulous Filtering. final review of flagged conversations is being conducted by human evaluators, who rectifying omissions of key answers and filtering out dialogues containing sensitive user information, violations fundamental e-commerce reasoning (e.g., unrealistic pricing, inconsistent policies), any form of user privacy data.
Starting from an initial set of 70,000 data samples, the filtering process ultimately yields 299 test samples and 4,500 high-quality training instances.
3.4 DATA STATISTICS
Tasks within the benchmark dataset classified into three categories: logistics, pre-sales, and after-sales. Logistics tasks are defined as handling all customer inquiries and requests associated with the shipping process. Pre-sales tasks concentrate on product discovery and persuasive communica- tion, with a focus on purchase facilitation. After-sales tasks deal with post-purchase interactions, specifically addressing customer complaints and product returns.
Statistics. Mix- Mix-ECom, as shown in Table contains 4,500 training and 299 test instances. Logistics tasks write to the datahase in 53.7 of cases and invoke 6.66 tools on average. Pre-sales tasks are multi-modal--77 % include images and 30 % videos- but never write; they need 6.39 calls. After-sales tasks embed images in 76.9 %, and demand the most calls (7.37). Overall, 46.7 of tasks contain images, 10 % contain videos, 41.7 % write to the database, and the mean number of tool calls 6.79.
Data Quality Evaluation. 100 instances selected from the test set and evaluated by five professional e-commerce customer service staff using a 0-1 scoring scale, where score 0 indicates low quality, and score indicates high quality. To quantify the inter-annotator agreement, Fleiss Kappa was applied. The Fleiss Kappa coefficient is 0.76, indicating substantial agreement among annotators. Additionally, 86% of the data samples receive score of 1, indicating high level of quality and reliability in the dataset.
4 DYNAMIC E- E-COMMERCE AGENT FRAMEWORK
In e-commerce customer service, tasks differ sharply from traditional agent benchmarks in two key ways. First, domain rules are complex: e-commerce relies on an intricate set of policies (up to 82 in our dataset), placing strict demands on the instraction-follooiing abilities of model. Second, user
5

---

Published as a conference paper at ICLR 2026
# of Test
# of Training
w/ Image
w/ Video
Write Db.
AVG. Tool Calls
Logistcs
108
1,500
0.0%
0.0%
53.7%
6.66
Pre-sales
100
1,500
70.0%
30.0%
0.0%
6.39
After-sales
91
1,500
76.9%
0.0%
70.3%
7.37
Total
4,500
46.7%
10.0%
41.7%
6.79
Table 2: Basic statistics of Mix-ECom. "Write Db." represents requiring writing to the database. "Avg. Tool Calls" represents the average number of tool calls needed to resolve task.
64) E- ReAet
(b) E- Plan And Salve
Trajenwry, has been Eltrrd
Uver Guery
Action
Planner
Sahve
Plan
piom
if ssit srer
Trvelle
Filter
Dynamic
Trepekay
Rude Filte
LLM Agezz
Repinn
Solns
Inverser
Erdrerreest
Figure 3: The illustration of the dynamic e-commerce agent framework, including E-ReAct and E-Plan&Solve, which are ReAct and Plan&Solve agent framework fused with the dynamic module to handle complex domain rules. E-ReAct in the block (a) filters the trajectory and the related domain rules, pruning irrelevant information to mitigate hallucination in subsequent reasoning. E-Plan&Solve in the block (b) utilizes the dynamic module to filter the domain rules and re-plans the remaining sub-tasks, enabling the system to meet users' changing needs.
queries are ambiguous- often vague or incomplete, and heavily dependent on context like product info, order status, and logisties data data-requiring multi-turn interactions for models to infer user intent accurately. To tackle these challenges, we propose a dynamic e-commerce customer service module: E-ReAct and E-Plan&Solve.
4.1 E-REACT
E- R-Ae is derived by tailoring the ReAct Yao et al. (2023) framework to the e-commerce domain. As illustrated in Figure 3 step the vanilla ReAct agent receives the tuple (F, 2,,2,P, P, H.} where P denotes the Domain policy, and Hr the reasoning trajectory accumulated over the preceding t steps,
H4 = (To, au, Oo. Fi, T1-1, 01-,,,
Here, T; denotes the thinking process step og stands for the action taken at step i, o represents the feedback resulting from the action at step i.
To mitigate the dual challenges of policy complexity and query ambiguity, introduce dynamic module that precedes the ReAct reasoning loop. At every step, the module receives the triple {C, P, He) where C' is the current conversational context, P the full domain-level policy set, and H, the trajectory produced so far. It returns a mask-focused policy subset p C P and filtered trajectory H: by removing irrelevant regulations and hallucinated reasoning steps, thereby shrinking the search space and reducing hallucination.
As illustrated in Figure 3, whenever the agent issues action a; talk_to user and receives new user utterance, the Dynamic Module invoked before the next reasoning step. The downstream ReAct agent then proceeds with the updated input {F, Q. pf, T, T.H,1 }. and this refinement is repeated after each user interaction, yielding context-adaptive reasoning process.

---

Published as a conference paper at ICLR 2026
4.2 E-PLAN&SOLVE
E-Plan-and-Solve extends the vanilla Plan-and-Solve Wang et I 2023) paradigm as depicted in Figure The original pipeline first consumes the tuple [F, 2, P, 7] to generate high-level plan
P. At step the vanilla Plan-and-Solve agent receives the tuple, where P denotes the Domain policy, and H, the reasoning trajectory accumulated over the preceding t steps,
Hr = (Ato OD, a1, O1..., 0t-1, 0t-)).
Here, a, stands for the action at step i, o; represents the feedback from the action at step i.
To counteract policy complexity and query ambiguity, we prepend Dynamic Module the Plan- and-Solve pipeline. Given the triple {C, P, Hr) where C' is the current conversational context, P the full domain policy, and H4 is the executed trajectory, the module emits a task-focused policy subset pp P.P. and revised plan pf_ By discarding irrelevant regulations and dynamically rewriting the plan, shrinks the search space, suppresses hallucination, and reduces the interference caused by ambiguous user intent.
As illustrated in Figure 3 whenever the agent issues a; ttalk ____tt______ _o_user and receives new user utterance, the Dynamic Module is invoked before the next planning reasoning step. The downstream agent then proceeds with the updated input (,, P) T ,T, is repeated after every user interaction, yielding context-adaptive faaning-and-xxecutin loop.
Pf, Hi) and this refinement
5
EXPERIMENT
5.1
EXPERIMENTAL SETTING
The performance of current ofate-of-the-art multimodal LLMs evaluated on Mix-ECom. The evaluation employs the ReAct and Plan&Solve frameworks, utilizing AP 20224) Gemini- 2.5-Pro Google (2025), Claude-4-sonnet Anthropic (2025), and Qwen- VL.-MAX Bai al. (2023) as model backbones. Furthermore, the proposed dynamic module is integrated into these frameworks for additional assessment.
5.2 EVALUATION METRICS
The performance of agents dataset is evaluated using method based the comparison between Key Answers and the Database.
Key Answer Score. Model--enerraed dialogue content and the set of key answers input, which are denoted as k {ki, k2, kn. GPT-4o is employed to evaluate whether each key answer appears in the dialogue. The Key Answer Score is assigned a value of if and only if all key answers in & are present in the dialogue. Mathematically, this can be expressed as:
kn).
{
if Vk; E Ex, D otherwise
Ska =
where D represents the dialogue text.
Database Score. We compare the database state obtained after the tool execution by the model with the ground truth database state. The Database Score is assigned value of if and only if the two states are entirely identical. Due to potential variations in the expression of content within the remark field (e.g., remarks specifying a logistics brand), we employ GPT-4o determine the equivalence of this specific field.
The prompts used for the comparison in both the Key Answer Score and the Database Score can be found in Appendix B.3
5.3 BASELINE MODEL
To further validate the effectiveness of training data, we conduct training on the Qwen-2.5-VI. 7B Team (2025) model. In training data, an inference trajectory of step t can be represented as
He
(To, 0o, 0o: TI, 1 Tt-1, 1 Ot-))
7

---

Published as a conference paper at ICLR 2026
Model
Framework
Logistics
After-sales
Pre-sales
Total
KA.
DB.
Score
K.A.
DB.
Score
KA.
DB.
Score
K.A.
DB.
Score
40.6
32.9
GPT-4o
R-ReAct
67.6
80.5
54.6
47.3
65.9
36,2
55.0
100.0
55.0
57.2
82.6
49.2
Poave
76.6
37.0
38.4
51.6
24.1
57.0
100.0
57.0
48.2
76.9
E-Plan&Sotve
51.9
78.7
40.7
42.9
50.5
29.7
60.0
100.0
60.0
51.8
77.3
43,8
RoAct
61.2
82.4
53.7
56.0
84.6
48.3
58.0
100.0
58.0
58.5
89.0
53.5
Gemini-2.5.pm
E-ReAct
25.9
83.8
67.9
63.7
84.6
50.5
62.0
100.0
62.0
67.6
90.3
60.5
Solve
57.1
75.8
49.5
64.0
64.0
88
S-Plan&Solvv
71.3
91.7
66.7
58.2
79.1
53.8
65.0
100.0
65.0
65.2
90.6
62.2
ReAct
75.9
80.6
63.0
64.8
72.5
53.8
E-ReAct
77.7
85.2
69.4
68.1
82.4
57.1
Claude-4-Sounid
Solve
E Plan&Solve
74.1
84.2
64.8
68.1
81.3
58.2
ReAct
58.3
75.4
42.6
53.8
64.8
43.9
56.0
100.0
56.0
56.2
80.3
47.5
Qwen-VL. VL-
ReAct
55.6
77.8
44.4
52.7
71.4
46.1
57.0
100.0
57.0
55.2
83.3
49.2
Plan&
48.1
78.7
36.
42.8
51.6
26.4
51.0
100.0
51.0
47.5
77.6
38.1
E Plan&Solve
$1.9
80.6
38.9
46.2
50.5
28.6
58.0
100.0
58.0
52.2
77.9
42.1
Qwen-2.5-VL 7B
ReAct
2.7
45.4
0.9
2.2
29.6
0.0
Qwen-2.5-VL. 7B"
ReAct
29.4
62.0
19.3
24.3
49.4
17.7
Table 3: Evaluation results for MILMs. "KA." denotes the Key-Answer score, i.e., the fraction of tasks whose key answers are all correct. "DB." denotes the Database score, i.c., the fraction of tasks whose database results are correct. "Score" is the fraction of tasks on which both the key answers and the database are correct. means that the model is fine-tuned. "_" indicates video modality is not supported. The results are presented in percentage (%).
Here, denotes the thinking process step i, oz stands for the action taken at step i, o represents the feedback resulting from the action at step i.
We split each H4 into & segments. For each segment, we construct supervised fine-tuning data by using H = (To, oo; Op, T1, ..., Ti-1 04-1, 0;-1) as the instruction and (Ti, a;) as the output. Due to resource constraints, "Qwen-2.5-VV Trained" model.
only train the model on the Logistics and After-sales subsets, yielding the
Due to the inclusion of video content in pre-sales tasks and owing resource constraints, only train Qwen-2.5-V VL on data related to logistics and after-sales tasks.
5.4 MAIN RESULT
Table Jable3shows shows that Gemini 2.5 pro achieves the highest overall score (62.2) under the E-Plan-and- Solve framework, followed closely by Gemini-2.5-Pro with 60.5 under E-ReAct. Qwen-VL-Max and GPT-4o lag substantially behind. Nevertheless, even the best-performing model remains far from solving our benchmark.
Across GPT-4o and Qwen-VL-Mai VL-Max, ReAct outperforms Plan-and-Solve, whereas for Gemini-2.5-PP and Claude-4-Sonnet the relation reverses or becomes comparable. Consistently, E-ReAct improves upon ReAct, and E-Plan-and-Soly improves upon Plan-and- -nd--vv for every model, corroborating the effectiveness of our enhancements. The largest gains appear in Logistics tasks, where user queries are concise and the rule-filterigg module seldom hallucinates; improvements on pre-sales and after-sales tasks are more modest.
Owing to task complexity and input lengths that exceed the pre-training context of Qwen-2.5-VI. 7B (domain policies, tool descriptions, and multi-modal files), the base model scores only 0.9 on Logistics and on after-sales. After supervised fine-tuning on our data, its scores rose to 19.3 and 17.7, respectively, verifying the utility of the curated dataset.
Table presents the results of the human evaluation, which was conducted by five professional e-commerce customer service staff across three key dimensions: Human--lkeness, Informativeness, and Key Answer. The experimental results indicate that Gemini-22 2.5-pro attained the highest scores across all three dimensions.
8

---

Published as a conference paper at ICLR 2026
GPT-4o
Gemini-2.5-pro
Claude-4-sonnet
Qwen- VL-MAX
Ground Truth
Human-likeness
60.4
69.2
67.8
49.2
82.6
Informativeness
66.8
74.6
70.2
62.0
91.8
Key Answer
59.4
68.4
66.2
54.6
100.0
Table 4: Human evaluation results for various MLLMs.
Multi-modal
Rule
Logistes
After-sales
pre-sales
Model
Logistics
After-sales
46.7
32.9
49.0
GPT-4o
60.9
55.6
*
46.7
28.9
43.0
Gemini-2.5-pmo
68.4
42.1
x
2.2
17.6
37.0
Claude-4-sonnet 54.2
37.5
Table 5: Results of GPT-4o on different settings, with or without multi-moda input and rules.
Table 6: The percentage of failure cases that do not contain all key answers.
5.5 BASE CASE ANALYSIS
The failure modes of a GPT-4o-hacked ReAct agent within the benchmark are analyzed and catego- rized into four types:
Multimodal Misinterpretation. Approximately 15 % of GPT-4o failures stem from misunderstand- ing multimodal evidence. In Appendix D.I] for example, the user uploads photo that clearly justifies an after-sales request; the model nevertheless claims the image is inconclusive and denies the claim. In the during-sales subset, the agent is unable extract product information from live-stream clips containing regional dialect, and thus fails to answer the customer's 's question. To quantify the impact of multimodality, we stripped all non-text inputs and re-evaluated GPT-4o. As Table 5 shows, its score drops by only 3.3 and 6.0 on the two splits, indicating that the model barely exploits visual cues. This also shows that our dataset presents tough challenge for function-cal callinn agents: there still lot of work left to do when it comes to understanding complex multi-modal content and making correct decisions.
Violation of Domain Rules. Roughly 63 % of errors arise from disregarding fine-grained policies. Appendix D.2 gives a logistics instance: when an in in-transit order subject to address change, the Rules requires updating both the order-level shipping address and the courier-level destination while resetting the logistic status. To further verify the impact of Domain Rules on the model, we conducted tests the Logistics and After-sales subsets with the Domain Rules removed. The results shown in Table 5 the scores decreased by 44.5 and 11.3, respectively. This demonstrates the importance of Domain Rules, and how enable agents comply with complex, fine-grained rules remains key challenge in current research.
Premature Switch Human. About 12 of failures are due to unnecessary human hand-off. In one logistics task, the shipping-fee calculator fails because the delivery address is missing; instead of querying the user via talk_to _tt_ttsueerrs_____ the agent gives up and escalates. In after-sales dialogue, after the customer refuses coupon compensation, the correct next action is propose an alternative remedy (e-g., return refund), yet the agent again transfers human. These cases highlight the subtlety of judgment of last resort in customer service: knowing when escalate is critical as knowing how to solve the problem.
The remaining 5 % include malformed tool calls that trigger infinite loops, user agent to articulate explicit demands, and partial key-answer generation. Tablot6)reports the proportion of failures in which the agent nevertheless hits some required key answers.
Other Errors
CONCLUSION
This work introduced Bix-ECCm-BBl ECCo-BBench, a benchmark designed to evaluate the capabilities LLM agents. The benchmark integrated four dialogue types (QA, recommendation, task-oriented dialogue, and chitchat), three e-commerce task categories (pre-sales, logistics, and after-sales), and 82 e- commerce rules along with 4,799 real-world e-commerce dialogues. Furthermore, baseline models

---

Published as a conference paper at ICLR 2026
evaluated Mix-ECom Bench, and this work proposed dynamic e-commerce framework to address the associated challenges. Results show that current e-commerce agents lack sufficient capabilities to handle e-commerce dialogues, due to the hallucination cased by complex domain rules.
ETHICAL STATEMENT
We make sure that Mix-ECom is collected in manner that is consistent with the terms of use of any sources and the intellectual property and privacy rights of the original authors of the texts. And the crowd workers were treated fairly. This includes, but is not limited to, compensating them fairly, ensuring that they were able to give informed consent, and ensuring that they were voluntary participants who were aware any risks of harm associated with their participation.
ACKNOWLEDGEMENT
This work was supported by the National Science Fund for Distinguished Young Scholars (No. 62025603), the Fundamental Research Funds for the Central Universities, the National Natural Science Foundation of China (No. 62576299, No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305, No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No. 2022J06001).
REFERENCES
http://wrrpp....cccooooaeoaaaaaa Anthropic. Claude sonnet 4. 2025. URL httt:s///ww..aatrrrpcc.co/o/////
Robert L Axtell and J Doyne Farmer. Agent-based modeling in economics and finance: Past, present, and future. Journal of Economic Literature, 63(1):197-287, 2025.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 10.3008..2966 12966, 2023. URL http::////aaxii.ora/aaao/a
12966
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems. volume 33, 1877-1901. Curran Associates, Inc., 2020. URL http://0...uttttpp///eeeeeeeeoo urips http:///ee..///////0000000000000 2 2f64a-Paper.pdl
Harrison Chase. Langchain, October 2022. URL http:p///ggthbb..om///ggnaaaaaaa langchain
Google. Gemini 2.5 pro. Available at: httpp//////oooooo//////////////o 2025. [Accessed: Ancereeeeeeet-Acceee-e------rrr
/eppiind/ --wp:geemiii---iie
Significant Gravitas. Autogpt, 2023. URL
5/ 3/AutoGPT]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.044 040088 2024.
Jiaqi Li, Yanming Li, Xiaoli Shen, Chuanyi Zhang, Guilin Qi, and Sheng Bi. Open-world attribute mining for E-commerce products with multimodal self-correction instruction tuning. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the
10

---

Published as a conference paper at ICLR 2026
63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 170--1714, Vienma, Austria, July 2025a. Association for Computational Linguistics. ISBN doi: 10.18653/v1/2025 aat URL B.RLittpss://alaattoooooy
rg/ 21/2025.acl-2 ong. 85/
Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, et al. Agent hospital: A simulacrum of hospital with evolvable medical agents. arviv preprint arXiv:2405.02957 2024.
Xiangci Li, Zhiyu Chen, Jason Ingyu Choi, Nikhita Vedula, Besnik Fetahu, Oleg Rokhlenko, and Shervin Malmasi. Wizard of shopping: Target-oriented E-commerce dialogue generation with deci- sion tree branching. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 13395-133222 Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176-251 doi: 10. N0..1853/11// 1/255/////222.... acl-long.6644 long.
URL http:ppse//ac.aaattoooo...ooorrp
641/ 1o1.
Qijiong Liu, Jieming Zhu, Yingxin Lai, Xiaoyu Dong. Lu Fan, Zhipeng Bian, Zhenhua Dong, and Xiao-Ming Wu. Evaluating recabilities of foundation models: A multi-domain, multi-dataset benchmark, 2025. http://se//aaa....ooo///////////
Meta. Llama 3 model card. 2024, URL 1t/mmain/MMDEE_CARR mc
Sushruta Mishra, Pamela Chaudhury, Hrudaya Kumar Tripathy, Kshira Sagar Sahoo, NZ Jhanjhi, Asma Abbas Hassan Elnour, and Abdelzahir Abdelmaboud. Enhancing health care through medical cognitive virtual agents. Digital Health, (0:20552076241256777 2024.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assiste question-answering with human feedback. arXiv preprint
2021.
Guangtao Nie, Rong Zhi, Xiaofan Yan, Yufan Du, Xiangyang Zhang, Jianwei Chen, Mi Zhou, Hong- shen Chen, Tianhao Li, Ziguang Cheng, et al. A hybrid multi-agent conversational recommender system with Ilm and search engine in e-commerce. In Proceedings of the 1 18th ACM Conference on Recommender Systems, pp. 745-747, 2024.
OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
OpenAl. Hello gpt-4o, 2024. URL httpppss//peeaa....o///////////s
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35. pp. 17730-277444 Curran Associates, Inc., 2022. httpaeaaeeeeeaaaee/e///////////e
URL ht / n neur ps cc wwwpcc/ppppppeeee Confeerrroooooeeeeeeeceooeo
[
Chester Palen-Michel. Ruixiang Wang. Yipeng Zhang, David Yu, Canran Xu, and Zhe Wu. Investi- gating Ilm applications in e-commerce. arXiv preprint trXiv:2408.12779 2024.
Mikolaj Pokrywka, Wojciech Kusa, Mieszko Rutkowski, and Mikolaj Koszowski. ConECT dataset: Overcoming data scarcity in context-aware E-commerce MT. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 79-86, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979 8-199-8881199 89176- 772-772-7 doi: 18.18663/v 1/2025. acl-short.7. URL https: http:///aaaaathooo..oorooo 77
acl short

---

Published as a conference paper at ICLR 2026
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding. Huadong Wang, et al. Webepm: Interactive web search for chinese long- question answering. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8968-8988, 2023.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world upplications via restful apis. arXiv preprint arXiv:2306.06624, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint wrw.::a40..000099900900 2024.
Qwen Team. Qwen2.5-vl, January 2025. URL httphhttps//se/t /uRLittppe///weeenngiihhii http:/huub..o/bbo//////eeeee 5-v1/
Haoxin Wang, Xianhan Peng, Xucheng Huang, Yizhe Huang, Ming Gong, Chenghan Yang, Yang Liu, and Ling Jiang. Ecom-bench: Can Ilm agent resolve real-world e-commerce customer support issues?, 2025a. URL www.a.xivvv.oo. org/ hbtp://bs/2000// 0001//5377775359999
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint 2305.:290000009 2023.
Weiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag, Wenju Xu, Chen Luo, Sheikh Muhammad Sarwar, Yang Li, Hansu Gu, Hui Liu, Changlong Yu, Jiaxin Bai, Yifan Gao, Haiyang Zhang, Qi He, Shuiwang Ji, and Yangqiu Song. EcomScriptBench: A multi-task benchmark for E-commerce script planning via step-wise intention-driver product association. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1-22, Vienna, Austria, July 2025b. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 1118653vv1/2025 1/26555/1//0225.alloooo... acl-.....1. URL httlh:t/ps/aaat http///aaaaaaootooooooo ogg..og.......... 2taogg/2222..cm
1- 1ooogg11/
Weiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag, Wenju Xu, Chen Luo, Sheikh Muhammad Sarwar, Yang Li, Hansu Gu, Hui Liu, Changlong Yu, Jiaxin Bai, Yifan Gao, Haiyang Zhang, Qi He, Shuiwang Ji, and Yangqiu Song. Ecomscriptbench: A multi-task benchmark for e- commerce script planning via sise intention-driven product association, 2025c. httpst
//arxiv.org// abs hbtp://b///5a00 2at../////25119090 15196
Songlin Xu, Xinyu Zhang, and Lianhui Qin. Eduagent: Generative student agents in learning. arXiv preprint arXiv:2400. 077663, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023. URLBttps://aaxx
/labs/2210..333
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tgen-agent interaction real-world domains, 2024. URL https://aexx...rr//aaa 24406.2004
Jingying Zeng, Hui Liu, Zhenwei Dai, Xianfeng Tang, Chen Luo, Samarth Varshney, Zhen Li, and Qi He. Cite before you speak: Enhancing context-response grounding in e-commerce conversa- tional Ilm-agents. arXiv preprint arXiiv2503.04888 2025.
Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, et al. A multimodal foundation agent for financial trading: Tool- augmented, diversified, and generalist. In Proceedings of the 30th acm sigkdd conference on knowledge discovery and data mining, pp. 4314 4325, 2024.
Longjun Zhou. A review of educational agents: Definitions, features, roles and development trends. Science Insights Education Frontiers, 28(2)466554468 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS
In this work, LLMs employed to polish and enhance the writing style of the paper.
B PROMPT
B. DOMAIN RULES
Basic Guidelines
Please respond to customers' needs politely, patiently, and professionally, using the tone and vocabulary of an e-commerce customer service representative.
The current system time 00:00 on June 12, 2025.
If encounter problems that cannot be resolved using the available tools, promptly use the switch_to human tool to transfer to human agent, rather than giving users meaningless responses.
If a user shows strong negative emotions, use the hummann tool to transfer to human agent.
not ask users for any ID information; any ID information you need will be provided in subsequent queries.
If the provided information dees not include order_id, it means the user has not yet placed order. If there is logistics_id, it means the seller has not yet shipped the order.
For special customer requests, such as expedited shipping, specified delivery times, other product-related special requirements, try to record them in the notes.
After resclving the user's current request, please confirm if there are any other needs.
the user indicates that all requests have been resolved and you have already used the tools to complete all user zequesta, please call the end_cenversation tool to end the current conversation.
Shipping Cost Caleulation Guidelines
When calculating total shipping cests, consider the total weight of the goods, i.e., quantity unit weight.
When calculating return shipping costs, use the same Logistics brand used for the original shipment.
If there is current logistics information (order not shipped), compare logistics brands used by the merchant return shipping costs and calculate using the cost of the cheapest brand.
When users ask if they need te advance shipping costs, inform then of the specific amount they need advance, considering the shipping insurance situation.
If the return is due to the merchant's fault, the user does not need to pay return shipping costs, regardless of whether shipping insurance is included.
13

---

Published as a conference paper at ICLR 2026
If a logistics interception due an address change, the user does not bear additional shipping costs.
Shipping Insurance Guidelines
Shipping insurance is a service offered by some merchants to subsidize return shipping costs during returns.
The maximum subsidy amount shipping insurance is 9 RMB If the return shipping cost exceeds this amount, the user must cover the fifeerncce
When users inquire about advancing shipping costs, calculate the shipping cost first, then answer based the shipping insurance situation.
Package Signed For But Not Reccived
If user reports that order marked delivered but not received advise them to check with family/friends or contact the logistics company.
Logistics Brand Selection Guidelines
Merchant acceptance of specified brands means cheosing from among the several brands they use, not specifying any arbitrary brand.
For already shipped orders, specifying changing the logistics brand is not allowed.
If the merchant supports specifying a logistics brand and the user specifics a brand used by the merchant, record this requirement in the notes.
the merchant does not support speci fying logistics brands, or the customer's 's specified fied brand is not among those used by the merchant, politely decline the request.
Logistics Time Calculation Guidelines
The current system time is 00:00, Thursday, June 12, 2025.
when replying about time, use the format "Month Day Hour", e.g., "Your estimated arrival time is 13:00 on Juno 12". All time calculations should be precise to the hour.
Estimated Shipping Time Calculation Guidelines
For placed orders (with order the estimated shipping time is the order payment time plus the merchant's 's promised shipping time.
For orders not yet placed (no crder_id), the estimated shipping time the current system time plus the merchant is promised shipping time.
# Estimated Arrival Time Calculation Guidelines
For shipped orders (with logistics _id,, the estimated arrival time is "logistics pickup time" plus the "logistics transit time".
For orders not yet shipped (no legistics_id), the estimated arrival time is the "estimated shipping time" plus the "logistics transit time".
For orcers not yet shipped (no logistics_ide it the merchant does not support specitying Logistics brand, or if the user did not
14

---

Published as a conference paper at ICLR 2026
successfully specify one, compare the transit times of the logistics brands used by the merchant and use the longest time for estimation.
- -If the user successfully specified logistics brand, use that brand's transit time for estination.
Shipping Address Modification Guidelines
If the order has not peen shipped (ne Logistics _i),, the shipping address in the order database (receive address) can be directly modi fied the addreas.
If the order has been shipped, medifying the shipping address requires confirming the current logistics status, which can be IIn Transit, Delivered).
If the logistics status is "In Transit", initiate a logistics intercaption. This involves three actions: 1. Modify the receive_address in the order database to the address. 2.Modify the receive_address Change the logistics status the logistics database to "Intercepted
the logistics database the address. 3.
It the Logistics status is "Delivered", negotiate return process with the user; see Return Guidelines for details.
If the legistics status is "Delivered" and the user's situation does not meet the return conditions, inform the user to contact the logistics company.
After-Sales Guidelines
When users raise after-sales requests for the following reasons: missing items/wrong itens received, goods damaged during transit, dissatisfaction with product quality, first guide the customer to send relevant pictures to verify their claim.
Content
User's 's message formatted like [Image x] represents the user sending an image at that point, corresponding to the x-th image in the input. For example, [Image 2] means the second image provided to you.
If the user wants to return an item for personal reasons (e.g., don't like it, don't want it, bought too many, wrong item), do not require them to send pictures.
If the cannot provide preef pictures, the provided pictures cannot verify the after-sales claim, first comfort the user and politely decline the request.
If the reason for the after-sales request is missing items, after fying with pictures, inform the customer that the missing items will be resent and record this in the notes.
If the reason is damage during transit dissatisfaction with product quality, after verifying with pictures, first attempt resolve the issue by offering a snall red envelope compensation to reach settlement.
The maximum red envelope compensation amount is Order Payment Amount Merchant's 'a Maximum Conpensation Percentage, rounded down the nearest whole number, with minimum compensation RMB.
Do not inform the user of the specific calculation methed for the red envelope compensation.
15

---

Published as a conference paper at ICLR 2026
If the user accepts the red envelope compensation, record the compensation details in the notes.
If the red envelope compensation cannot resolve the user's issue, negotiate entering the return process; Return Guidalines.
If neither red envelope compensation nor return can resolve the user's issue, transfer to a human agent.
Reshi Reshipment pment Guidelines
Reshi Reett pment can only be registered for the customer if it is veri fied that the merchant indeed shipped missing/wrong items. Record the reshipment details in the notes.
Except for cases of missing/wreng items shipped by the merchant, reshipment cannot be registered under any other circumstances.
If the customer requests reshipment but does not meet the conditions, politely inform them of this result.
# Red Envel ope Compensation Guidel ines
The maximun red envelope compensation amount is Order Payment Amount Merchant's Maximum Compensation Percentage, rounded down to the nearest whole number, with minimum compensation RMB
Do not inform the user of the specific calculation method for the red envelope compensation.
the user accepts the red envelope compensation, record the compensation details in the notes.
# Return Guidelines
To initiate a return process, first determine the product belongs to the fresh/perishable goods category. If it is, follow the procedures under the kefund-Oniy Guidel ines.
### Personal Reason Return Guidelines
If the user's request is for return due to personal reasons, you do not need to guide them to provide photo proof.
Since the return is for personal reasons, the maximum service that can be provided is a return refund, if the conditions are met (see Return & Refund Guidelines).
not process refund-only, reshipment, or red envelope compensation
for personal reason returns.
## Product Quality Reason Return Guidelines
If the user's request is a complaint and return due to product quality issues, first guide the customer to send pictures to verify the reason for the complaint.
the user's pictures can verify small red envelope compensation to comfort the customer. If compensation cannot resolve the issue, negotiate entering the return & refund process (see Return & Refund Guidelines) In this case, it is the merchant's fault, inform the user that they pay return shipping costs.
complaint, first attempt to use
need
If the user's pictures cannot verify the complaint, politely confort
the and, based the specific content of the pictures, politely
16

---

Published as a conference paper at ICLR 2026
decline requests for red envelope compersation, refund-only, reshipment.
customer's demand is return & refund, proceed to the return refund process, noting that this should be treated as a personal reason return, and inform the user of the return shipping cost based on shipping insurance.
### Return Refund Guidelines
II the user has already returned the goods, red envelope compensation cannot be provided.
#+# If the order is not yet shipped (i.e., no logistics_id)
Dirreclyy change the order status in the order database to "Cancelled",
#### If the order already shipped
Before processing the return, confirm whether the product supports 7- day no-reason returns Lis_supoort_7a_aae and confirm with the user whether the product has been used.
A return can only be processed if the product has not been used, and the preduct supports 7-day no-reason returns, and the time since receipt is less than equal to 7 days.
If the user meets the return cenditions, you need to provide the user with the merchant's address so they can initiate the return shipment. You also need to inform the customer about shipping insurance information and the specific amount they need advance for shipping (accurate to one decinal place)
You can only process the return after the user informs you that they have initiated the return shipment.
##### User Level is 3
Process expedited return for the user: change the order status in the order database "Refunded".
### User Level is below 3
-Process standard return for the user: change the order status in the order database to "Roturning",
### Refund-Onl Only Guidelines
Note: Refund-only can only processed for fresh/perishable category goods. Other product types can only undergo the return refund process.
For fresh/perishable goods, preduct issues must be verified before processing refund-only. if refund-only initiated, change the order status to "Refund-Only".
For fresh/perishsble goods, confirming whether the product has been used is not required for refund-only processing.
For verified issues with frosh/perishable goods, do not process a return & refund for the customer; process a refund-only.
When processing refund-only, is necessary provide the with the merchent 's address, nor does the customer need to initiate a return shipment.
17

---

Published as a conference paper at ICLR 2026
If the custemer requests refund-only but does not the conditions politely inform then of this result.
Coupon Guidelines
A coupon can be used for current product only if the following conditions met: 1. The current product type is included in the coupon's s category_ list.. 2. The current product 's price meets the condition.
coupon's 's sinimum__purchen
Coupons different levels be used together. Coupons the level cannot be stacked.
If there are coupons applicable te the current product, inform the customer of the minimum payable amount after applying the coupons.
Product Recommendation Guidelines
If no current productss meet the user's requirements, inform the user of this result and reconmend products that are similar to their request
Get Live Stream Clip Guidelines
When users ask questions that you cannot determine the answer to, you can try using the iet_ittm_vieee tool to get information from the product's 's recent live stream clips.
Listing Domain Rules used in Mix-Ecom Benchmark
B.2 QUESTION TYPE GENERATION FOR AFTER SALES TASKS
You will see a conversation from an e-commerce customer service scenario, where "User: xxx" represents messages by customer the e-commerce Assistant, and "Assistant: xXx" represents the Assistant's replies. Content in the form of "[Image x] indicates that there is an image at that point, corresponding to the x-th image I sent you. For example, "[Image 2]" represents the second image sent you. [Conversation Start]
[conversation]
[Conversation End]
Below is the relevant item information for this conversation.
[item Information Start]
fitem info)
[item Information End]
# Task Overview
The conversation content may relate to after-sales issues concerning product. Your task is to extract the following information based this: 1. The specific reason the user initiated the after-sales request. 2. Whether the images provided by the user can substantiate their claim. 3. The solution desired by the user. 4. The user's current mood.
Please strictly adhere to the following principles:
Return the result **only** in ISON format. Return *only* this ISON result, without adding any other content. Ensure the content you return can be parsed by Python's 's "json.loads(" function.
'reason': 'The specific reason the user initiated the after-sales request', 'image verification': 'Whether the images provided by the user substantiate their claim', 'solution": 'The solution desired by the user', 'mood': 'The user's current mood"
## The specific reason the user initiated the after-sales request
18

---

Published as a conference paper at ICLR 2026
Please select the specific reason the user initiated the after-sales request from the following options: [Missing/Wrong items shipped, Item damaged during transit, Dissatisfied with product quality, Return due to other personal reasons].
the user's reason does fall into any of the above categories, write '-1'.
## Whether the images provided by the user can substantiate their claim the image(s) provided by the user can substantiate their specific reason for the after-sales request (e.g., the image proves missing items, proves damage during transit, or proves the point of dissatisfaction with the product), write 'I'. Otherwise, write '0',
You can also use the specific dialogue content to assist your judgment. For example, if the assistant does not raise objections to the user's image(s), implies the images are valid; write $1'
-If the conversation is incomplete, you can only judge hased on the content of the image(s).
## The solution desired by the user
Pleasse select the solution desired by the user from the following options.
the desired solution cannot be determined from the conversation, if the user's desired solution is not among the options below, please randomly select one from [Re-ship missing items, Compensate with red envelope of yuan, Refund and return, Refund only] to fill in. If the user's desired solution is compensation of yuan, please choose an appropriate compensation amount based on the conversation or product information.
## The user's current mood
Please infer the user's current mood based on the conversation content. Select from the following options: [Calm, Impatient].
IIf the user's current mood cannot be inferred from the conversation content, please fill in '-1'. Return the result ** **only** ** in JSON format. Return *only* this JSON result, without adding any other content. Ensure the content you return can be parsed by Python's "json.lads()) function.
Now will provide you with 4 examples: Example
'reason': Wissing/Wrong items shipped', 'image_ verification': 'I', 'solution": 'Refunn and return', 'mood': 'Impatient'
Example 2:
'reason:: 'Dissatisfied with product quality", 'image_vents verification': '0',
'solution": "Compensate with a red envelope of 15 yuan'.
'mood': '-1'
Example ('reason:: '-1',
'image_ verification': '1',
"solution": 'Refund only",
'mood': 'Calm'
Now, please extract based on the conversation content: 1. The specific reason the user initiated the after-sales request. 2. Whether the images provided by the user can substantiate their claim. 3. The solution desired by the user. 4. The user's current mood.
json:
19

---

Published as a conference paper at ICLR 2026
B.3 EVALUATION
B.3.1 PROMPT FOR KEY ANSWER SCORE
You will message from conversation between e-commerce customer service repre- sentative and a customer. Your task is to determine whether the message contains the content of the key_ key_answer.
[Start of message] message)
[nd of message]
[Start of key_answert
[key_an jey_answer]
[End key__aan [ns_ea
Please determine whether the message contains the content of the key_answer. If does, reply with I; otherwise, reply with 0. Note: If the key_ans answer mentions amount time, the related content in the message must be an exact match to be considered correct. Only respond with or O, without providing your reasoning.
Your answer:
B.3.2
PROMPT FOR DATABASE SCORE
You will see two messages, messagel and message2. Your task is determine whether the content of these two messages is similar.
[Start of messagel]
message)
[End of 1
[Start of message2]
gground_rrriee |_tuthttthh
[End of message2]
Please judge whether the content of these two messages is similar. If yes, reply with 1; otherwise, reply with 0.
Note: the messages mention express delivery brands similar content, they will only be considered correct if the related information is completely consistent in both messages.
Reply only with or O, without providing any reasoning. Your answer:
C
API TOOLS
Tool Category
Included Tool Names
Count
get_shop_dctta @i__ittm_ettall
get_order_detail
got_logisties_detee _dctail
Information Retrieval
get_ get_user detail get _el__product_ee detail
get_ ret_user_coupon
caarr_ccuoo___dtde detail
detail ged_video detail
Information Calculation
calculate Bacillaee_ssipiin__tt time
calculate_ ciciuulate_shiping___ooo
modify logistics address modify_on odderrrddreeess address
modify_ holllloogiiittttseeeeeesssssssss state
Information Modification
modify heweyoorderrreee state
talk user remark
switch to human end conversation
Interaction & Management
Table 7: Tools in MIX-ECOM Bench.
20

---

Published as a conference paper at ICLR 2026
D SHOW CASE
Please see the next page.
21

---

Published as a conference paper at ICLR 2026
D.1 CASEI
Conversation
User!
Which courier service is shipping this order?
Assistant: Your order being shipped vi SF Express. the current shipping status is transit"" Please walt few more days, and we hope you enjoy your shopping! you have other questions need assistance, feel free to let know!
User: The address was filed incorrectly, Please change it to 91 Full East Road, Qjihe Disriis, Lanzhou City, Gansu Province.
Assistant: Your shipping address has been successfully updated to 91 Full East Road, Qilhee District, Lanahou City, Gansu Province, and the logistics have been intercepted adjustmett.
User: Agprosimately
many days take
Assistant: Your order expected to arrive Lanchou by 00 on June 16. Thank you for your patience! you have any other questions or need further avsistacee, feel free to tell me!
Trajectory
<Question>
Which courier service shipping this /qurst/Queston>>>>
<Thought>
Firs, need obtain the logistics information the order to confirm which courier company is used. Here, need to call the logistics details to get the
specifie log sties brand information. </Thought>
<Action, Input>>
["ooo name": "get: "get__ logistics_ detail", ": ("logistic_jid" jd":: 089425888480085 "I) </Action_ //Actioninput>>
-Observation>
[Shipping address! 775 Chizhu Datu ling, Eastern District, Hong Kong island, Hong Kong', "Recaiving address": "Vanshan County, Cangahou City, Hebai
Provicce; 'Status:,
'Courfer brand':
'Pickup time': 12th, 00:00; 'Delivery time:") </Observation>
(Final Answer>
Your order being shipped via SF Express. The current shipping status "In transit"" Please wait a few more days, and we hope you enjoy your shopp ng! you have any other questions need resistance, feel free to let me know! </Finol_Answer> Answer>>
<Question>
filled incorrectly
change 91 East
District, Lanchou City, Province. </Question>>
<Thought>
The user wants to modity the shipping address of the order. Since the order has been shipped, need to first obtain the current logitics information to confirm the logiviics status, then decide the next steps, </ </Thought>
<Action_ jnppu>>>>>>>>
("Tool name": "gort logistics datail", "Args": ("logistie_jef jid:: "7942 5888486085") e/Action_ippur
"gort
<Observation>
(Shipping address': 75 Chizhu Defu Eastern Distriet, Hong Kong Island, Hong Kong', 'Weteeving address': 'Yanshan County, Cangthou Hebei 'Courier brand:: Fspress", 'Pickup timel: 12th, 0b:n0; 'Dellwery time': ") </Observation>
Provincel, 'Status:
<Thought>
The logistics status is currently "in transit", so can intercept the logistics and modify the shipping address, need to first medify the receiving address and status information the logistic database. </Thought>
sActioe Jnpute
("Tool name": moddity logistics
"Arg"" ("logstics id". 79425888490055
"new "tta:sssttttt"iitttooooloii "interrepttod" //Acaiietptd""i/////oooooo //Accior__npuut
<Observation>
Logittic status modified successfully //Observation>
<Action_ input>>>>>>>>>>>>>>>>>>>>>>>>>
name": "modiiyy logistics_ address"" "Args": ("logistic, 7782289948608", "new. address" '91 Full East Road, Cilihe District, Lanchou,
Gonsu Province")) ") </Action_ </Action_input>>
<Observation>
receiving address modifled successfully </Observation>
Your shipping address has been successfuly updated to 91 Full East Road, Qjihe District, Lanahou Chty Gansu Province, and the logiitics have been intercspted for adjust ment. </F inal Answer>
:Qusttion>
Approcinately many days <Touuhht>
Lanohou? </Question>
The user wants to know the logistics arrival time, since the logistics already in transit and the address change and interception have been performed, it necessary to recalcurate the lggisiis time from the shipoing address the new recelving address. To get an accurate lopistiis time, Ined to got logisties details analyoe logistics
the </Thought>
shipping
shipping courier
<Action <Action_input>>
(TTool name":"gee "aart logistics datail", " "Args": ("logistic "9905888468085") </Action_inpur
-Observation>
('Shipping address': 75
ling, Eastern Distric, Hong Island, Hong Konal, 'Receiiing address': '11 Full Road, Qlihe District,
Lanzhou, Gansu Province', 'Statue:: Intercepted, 'Courter brand'; 'SF Express), PPikku time': 'une 12th, 00:00, Deliery time'. +/Obeervation>
< <Thought>
have obtained the detalled logistics information. The courler brand from Exgress,
address
recelving address Lanzhou, extimated logistics based
"interapted". Therefore, knw information and then inform the user </Thought>
logistics
<Action_ <Action_input>>
("Tool "calcuaaa shipping..tiie. time", "Argy": ('see':''eedd_.d. acdress": "Hong Kong', 'recaive address" '91 East Road, Ollihe District, Lanchou, Gonsu Province", 'couree_bbaane
Express"), </Actioo_input>>
OOservvttion>
95 hours /OObservation>
<Finol Answer>
expected
by
your patiencel you quastions
assistance, tell mal x/Finol_ Answer>>
The assistant does not follow the domain rule regarding the modification of the receiving address when the logistics status 'in transit'. According to the rule, the assistant must modify three pieces of information: the logistics status, the logistics address, and the order address. However, the assistant only modifies the logistics status and the logistics address, which leads to task failure.
Figure 4:
22

---

Published as a conference paper at ICLR 2026
D.2 CASE2
Conversation
User: This green looks quite different
page. refund.
100
Assistant: After reviewing product detall, that item does support seven day no no-reason return. According the
polity, i user requasts a return for personal reasons and the items is not elighlle for seven-day no-reason return we cannot process the refund. aookogia for any inconveniance this may cause.
User: The leaves this green quality lssue insist refund!
yellowing, completely different from vibrant shown picturel This clearly
User:
Besed the evidence provided, we can express return and refund for Since the seller did provide shipping insurance, you need to advance return shipping Please use following address the return logistics: Street, Xaodian District, Talyuan, Shanol, Once return shipmenn has been successfully nitiated, please the refund as soon as possible, We appreciste your cooperation. Is there anything else we can assist you with?
Assistant:
and wil process
Trajectory
<Question>
This green plant looks quite different from what shown on your page. want refund. </Question>
<Thought>
The user is dissatiffied with the product's appearance and wants return for refund. According to the policy, since the return is due personal necessary to confirm whether product eligblle for a
return and whether it has been within days of receiot. detalls to check receipt time.
Priority retrieve the product detalls to desernine it supports a 7-day reason return and to obtain
</Thought>
<Action Input>
("Tool name": "amee""get_hhmme "yet_hhem_detil" "Args": ('itemjif 222988996666671 Acctionnniuut input>
<Observation
(Product Name': '[Buy One Get One) A Large Leaf Golden Pothos Green Princess Indoor Office Desktop Summer Aesthetic Potted Plant with Pot!,
Product Price!: 9.9 yuan', 'Supports 7-Day No Resson Retuml: 'No. 'Pradurt Catlegory autributes: ('Suitable Scenario" 'Office' 'ts imported:: 'Ne'
VVriety': "Golden Pothox', "unetion"HHumidllee "Humidifying. 'Floerr and Plant Height:; 221mm-0lem' 'Categonyy: "lable Display', 'Difficuity Level: 'ivenaee, Growing Season': "Non-flowering] 1, 'Product Wrightt: 1.7ug "Pramised Shipping Time After Order'' 10.0 'Is Perishable Product: 'No') /Observvtionn>
GGiddn
'Maant Tyoe': Stem-Viewing
<Finol Answer>
After reviewing the and the
details that item does support day no-resson According the policy, if requests process the refund.
inconverience this may cause. Answer>
<Question>
The leaves this green plant are al yelowing, completely different from the vibrant green shown the picture! This dearly a quality issue, and refund! [Image 1). /Questiio>> <Thought>
Baase on the image and order information provided by the user, the green plant shows obvious discoloration, has been confirmed that the product does not support a 7-yay no-resson return. However, since the user has provided evidence of quality issue. I need to communicate with the user to and exolain the return conditions. the product still within days and the provided evidence. meets the criteraa relund and return. Betore discussing the solution. need to retrieve the merchant's information obtain their shipping acdress processing the return. </Thought>
inttsste
<Action_ingp input>
("Tool name": rert_shop_deail,, "Args": ("shop_ie" [d:; "100000063332200 VE: dictioniinutt <Observation>
[Merchant Shipping Address:: Shani- Talyuan-Xaadion District/No. 455 Fuxi Street', Provides Shipoing Insurance' 'No', 'Logistics Brands Used': l'Best',
"Deppon, IBT) 'Accppts Specified Logisics Brands): Yes', MMrchant's Maximum Compensation Percensaee '12%7] /bservvttion>
<Thought>>
Since the
product within 7 days, The merchant does not provide shipping insurance, the user will cover the return shipping costs. user should informed that after negotiating with the merchant to initiate the return logistics, we will process their refund request. As the user's fier information has not yet beers retrieved, need to confirm the user's determine the speed of the refund process. </ </Thought>
provided images demorstrating quality
oligible refund and process.
<Action_ input>>>>>>>>>>>>>>>>
('Tool name": 'gett usar_ detall" "Args": ('userr 7886911892348117785) </Action_input>>
-Observation>
[User Tier: 31 330///ervatioon
"Yhooghhs
the quality issue reported. Although the plant not support 7-day no reason return, the criteria for refund and return. Since the user is at Tier 3, we can process an expedited return to better meet their needs. During the return process, the user will need to cover the return shipping costs the merchant does not provide shipping insurance, Next, the user will be informed of recommend the longest-used logistics brand for
The user's provided images
merchant's shipping address guided inttiate estimation. </Thought>
shipping
<Finol Answer>
Based on the evidence you provided, we can process express return and refund for you, Since the seller did not provide shipping insurance, you will acvence the return shipping fee. Please the following address for the return logistics: 455 fusi street, Kaodian District, Teiyuan, Shanki. been successfully intbated, please inform us, and we will process the as soon possible, We appreciate your cooperation. is there anything alse we can assint you with? </Finol Answers
the return
Figure 5: The assistant does not utilize the information in the image make the correct decision. In this case, the unreasonable request of the user should be rejected; however, the assistant processes the refund due the repeated demands of the user, resulting in task failure.
23


Reviews:
[{"review_id": "dl5LnMVMZM", "reviewer": "Reviewer_um8L", "summary": "The authors focus on the problem domain of conversational assistants for the e-commerce domain. The authors argue that current benchmarks are not comprehensive enough for real-world dialogs and come up with their own dataset called Mix-ECom. The datasets is constructed by taking real-world customer dialogs and using LLMs to remove user privacy, add reasoning between responses and simulate user profiles. \n\nThey then evaluate a series of models on this dataset and leverage the ReAcT and Plan&Solve framework for their models. They adapt those frameworks and have a module that filters out certain turns that may result in hallucination errors and show that their new setup is better. They also show finetuning on their dataset improves an open-source model.", "questions": "Questions\n\n1) How did you split the dataset into train / test? Was it random or did you take into account actions that may be in the training data but aren't in the test.\n\n2) Why is the use of the dynamic module only when the action talk_to_user outputted? Would be good to explain this.\n\nSuggestions\n\n1) I think in Line 018 you meant to say \"multiple\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The construction of a dataset that mimics real-world conversations is a good contribution to the community.\n\n2) The set of models in the paper along with the experiments done seem comprehensive and the analysis as to why GPT-4o is much worse than Gemini 2.5 pro is an interesting contribution. The dynamic module approach also seems promising and can be explored in future work.", "weaknesses": "There are a couple clarifications needed in the paper\n\n1) Who came up with these domain rules? It seems important but it's not mentioned why these are required.\n\n2) There is alot of work on human verification but there arent specific details on the instructions given to the humans. Additionally what constitutes a high quality conversation? \n\n3) It is mentioned that the dialog is rewritten. What does that mean? I assumed only the reasoning chain was added but it seems more was added.\n\n4) Why are other task-oriented metrics like number of turns not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rt4eN2bTZi", "reviewer": "Reviewer_j6Yo", "summary": "This paper proposes Mix-ECom, a dataset made to test chatbots used in e-commerce. It has around 4800 real-life customer chats, taken from real e-commerce cases. These chats cover different types of dialogues like question-answer, product advice, help with tasks, and small talk. It also covers different shopping stages like before buying, delivery, and after-sales. There are also some small rules about returns, shipping, refunds, and so on. Each chat can include pictures or short videos, tools like \"calculate shipping cost\" or \"change delivery address,\" and info from a logistics database. The authors propose a fix to the issue of rule hallucination, when chatbots ignore or mess up company rules. For this, they make a new system called E-ReAct and E-Plan & Solve, which helps the bot focus only on the right rules at the right time. When they tested big AI models like GPT-4o and Gemini-2.5-Pro, none did great but after fine-tuning the models on Mix-ECom, performance improved.", "questions": "- When you say \"mixed-type\", do you mean many things happening in the same line (like sarcasm + refund + product question together), or just one after another in some predefined sequence? How will you check real mixing, not just switching?\n- In your dynamic system, how do you decide which rules matter? Simple keywords, embedding similarity, or a trained scoring model?\n- After fine-tuning, are models memorising rule lines instead of reasoning? For example, do they just learn \"mangosteen means refund only\" without knowing the reason?\n- Can Mix-ECom do user modelling? Like changing tone if the user is angry. You already have mood labels, so can you use them to enforce policies in a personalized way?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The dataset is of high quality: it contains real human chats, is multimodal (as and when needed), covers different aspects of conversation like casual, complaint, requests, etc. The data has rules which helps test chatbots on reasoning rather than just engagement.\n- I like the testing design, where the authors separate how corrext the answer are from how well the system handles the internal state. The error breakdown of rule mistake, image misreading, or  early reaction, is clear and helpful.\n- The paper is well written with a comprehensive appendix that contains almost everything that is needed to reproduce the study.", "weaknesses": "- Most chats may switch between the chat mix in a sequential manner rather than mixing them up randomly, e.g. after sales to recommendation to QA. This might make the chats too restrictive (more like multi-step rather than mixed) and the pattern too well known to the chatbots.\n- I think the authors use the term \"hallucination\" too loosely. For e.g. they call 63% of mistakes as hallucinations but many of those are just missed rules or retrieval errors, not the model making things up.\n- By giving the model full reasoning cot steps during training may help the model too much. Real bots don’t get these ready-made steps, so the test might look easier than it really is.\n- Human reviewers checked if responses were natural and informative, but not if they followed company rules. A chatbot can sound polite and smart but still break return or refund policies and the paper should measure this too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EoavUioefR", "reviewer": "Reviewer_DZnq", "summary": "This work mainly tries to fill the gap of handling mixed-type dialogues and complex domain rules in the context of LLM-based e-commerce agent benchmarks. \n\nTo this end, this work has introduced a new dataset Mix-ECom, which is derived from 70,000 real-world customer-service dialogues and finally keeps about 4799 samples.  The highlights of Mix-ECom can be summarized as:\n\n1. It includes four dialogue types (QA, recommendation, task-oriented dialogue, chit-chat).\n2. It includes three e-commerce task categories (pre-sales, logistics, after-sales).\n3. It includes 82 domain rules.\n\nThen, this work has proposed a framework E-ReAct by tailoring the ReAct work and an E-Plan-and-Slove module extended from Plan-and-Solve.\n\nIn the experiments, the authors evaluate 5 LLMs (4 closed-source, 1 open-source) on Mix-ECom.", "questions": "1. Have you considered testing against specialized e-commerce models?\n\n2. Do you intend to add a user study (e.g., measuring customer satisfaction with hallucination-reduced outputs) .", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work has introduced  an novel dataset Mix-ECom,  addressing the gap of mixed-type dialogues and complex rules in existing e-commerce agent evaluations.  The dataset construction has used real user dialogues and proper construction pipelines.\n\n2. Compared to the previous benchmark dataset, Mix-ECom is able to cover more practical scenarios.\n\n3. Mix-ECom has covered  multi-modal inputs (images, videos).\n\n4. Conducts comprehensive experiments with 5 LLMs (4 closed-source, 1 open-source) and fine-tuning, providing clear results and highlighting current agents’ limitations.\n\n5. Compared to the original ReAct and Plan&Solve, the proposed E-Version is much better in the experiments.", "weaknesses": "1. The major contribution of this work is the proposed dataset. However, the novelty and contribution of this dataset are not very significant. \n2. The dataset's development and statistical analysis follow standard practices, lacking notable innovation or depth.\n3. E-ReAct and E-Plan&Solve are incremental because both of them are derived from other works without notable modification.\n4. Only 0-shot experiments are conducted and only 5 LLMs are tested. It is not sufficient enough for a dataset paper.\n5. No enough analysis of how fine-tuning (e.g., on Mix-ECom subsets) impacts performance, which may limit understanding of the benchmark’s value.\n6. No comparison with specialized e-commerce dialogue models.\n6. No manual evaluation in the experiment part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZvq9iuaEw", "reviewer": "Reviewer_bcjS", "summary": "This paper introduces Mix-ECom, a novel dataset and benchmark aimed at evaluating the performance of e-commerce agents in handling real-world, mixed-type e-commerce dialogues. These dialogues involve various types, including task-oriented, recommendation, Q&A, and chit-chat, while adhering to complex domain rules in pre-sales, logistics, and after-sales tasks. The dataset includes 4,799 dialogue samples, covering 82 e-commerce rules, and integrates multi-modal data (e.g., images and videos) to evaluate agents' capabilities. The paper proposes a dynamic framework for improving e-commerce agent performance, addressing the problem of hallucinations arising from complex rules.", "questions": "refers to weakness paragraph", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The Mix-ECom dataset is designed to address the lack of comprehensive benchmarks for mixed-type e-commerce dialogues with complex rules. The inclusion of multi-modal content is a valuable feature that enhances the practical applicability of the dataset.\nThe focus on real-world e-commerce dialogues, such as pre-sales, logistics, and after-sales, ensures that the research is directly relevant to industry applications. This has strong potential for real-world impact, especially in enhancing customer service experiences through intelligent agents. The experiment compares some recent e-commerce framework, using the Mix-ECom versus not using it, the exp demonstrated the improvement brought by the Mix-ECom.", "weaknesses": "1.The paper does not provide a clear presentation of the dataset's distribution across different task categories, dialogue types, or domain rules. This information is crucial for understanding the dataset’s structure and its potential biases. The absence of such details makes it difficult to evaluate the dataset's representativeness and fairness.\n\n2.While the dataset and framework are innovative, the paper does not provide sufficient early-stage details on the dataset construction process, usage information, or any plans for ongoing updates. Without such transparency, it is unclear how this work will benefit the community in the near future or whether other researchers can effectively build upon this resource.\n\n3.The paper lacks clarity regarding the data collection process, particularly the sources of the dialogues and whether proper permissions and licenses were obtained. The absence of such details raises concerns about data privacy and legal compliance, which could impact the credibility and applicability of the dataset in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)