SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agents
Authors: Gyuhyeon Seo
Jungwoo Yang
Junseong Pyo
Nalim Kim
Jonggeun Lee
Yohan Jo
Venue: ICLR 2026 Oral (2026)
Keywords: smart home
simulator
language model
language agent
benchmark
Primary area: datasets and benchmarks

Abstract: We introduce $\textbf{SimuHome}$, a high-fidelity smart home simulator and a benchmark of 600 episodes for LLM-based smart home agents. Existing smart home benchmarks treat the home as a static system, neither simulating how device operations affect environmental variables over time nor supporting workflow scheduling of device commands. SimuHome is grounded in the Matter protocol, the industry standard that defines how real smart home devices communicate and operate. Agents interact with devices through SimuHome's APIs and observe how their actions continuously affect environmental variables such as temperature and humidity. Our benchmark covers state inquiry, implicit user intent inference, explicit device control, and workflow scheduling, each with both feasible and infeasible requests. For workflow scheduling, the simulator accelerates time so that scheduled workflows can be evaluated immediately. An evaluation of 18 agents reveals that workflow scheduling is the hardest category, with failures persisting across alternative agent frameworks and fine-tuning. These findings suggest that SimuHome's time-accelerated simulation could serve as an environment for agents to pre-validate their actions before committing them to the real world.

Published as a conference paper at ICLR 2026
SIMUHOME: A TEMPORAL- AND ENVIRONMENT-AWARE BENCHMARK FOR SMART HOME LLM AGENTS
Gyuhyeon Seo, Jungwoo Yang, Junseong Pyo; Nalim Kim, Jonggeun Lee, Yohan Jo Graduate School of Data Science, Seoul National University (seckh97, jwyang02133 yohan. jonnnjoeesnna aan..ac..
ABSTRACT
We introduce SimuHome, a high-fidelity smart home simulator and a benchmark of 600 episodes for LLM-based smart home agents. Existing smart home benchmarks treat the home as a static system, neither simulating how device operations affect environmental variables over time nor supporting workflow scheduling of device commands. SimuHome is grounded in the Matter protoco the industry stan- dard that defines how real smart home devices communicate and operate. Agents interact with devices through SimuHome's s APIs and observe how their actions continuously affect environmental variables such as temperature and humidity. Our benchmark covers state inquiry, implicit user intent inference, explicit device con- trol, and workflow scheduling, each with both feasible and infeasible requests. For workflow scheduling, the simulator accelerates time so that scheduled workflows can be evaluated immediately. An evaluation of 18 agents reveals that work- flow scheduling is the hardest category, with failures persisting across alternative agent frameworks and fine-tuning. These findings suggest that SimuHome's 's time- accelerated simulation could serve as an environment for agents to pre-validate their actions before committing them to the real world. Code and data are available at hhttps:/// //ttt/iitttt t/ub. com hol i lab/ SimuHome/
INTRODUCTION
Smart home agents have long been a central research topic for tool agents. Systems such as Amazon Alexa and Google Home are among the carliest tool agents commercialized at scale, yet many everyday household requests remain beyond their reach. Building more capable agents with large language models (LLMs) requires handling several challenges that go beyond simple command execution. The simplest case an explicit command such as "Turn on the light", but not all user requests are this direct. user who says "It feels sticky" expects the agent to recognize this request to reduce humidity and activate dehumidifier. Even when commands are explicit, agents must respect operational dependencies between device commands. For instance, robot vacuum cleaner must be powered on before it can be switched to mopping mode. Some requests go further and require coordinating device actions across time. request such as "Turn on the kitchen light when the dishwasher finishes" requires the agent to check the dishwasher's remaining cycle time, calculate the expected completion time, and schedule the light to turn on at that time. In addition to these challenges, agents must track how device actions continuously affect the surrounding environment. Setting an air conditioner to 25%C does not change the room temperature instantly. The temperature drops gradually over several minutes, and the agent must observe these ongoing changes to determine whether the target condition has been reached.
Training and evaluating these capabilities require interactive environment where agents call APIs to operate devices and observe the resulting changes to environmental variables over time. Existing smart home benchmarks do not model how device actions continuously affect environmental variables such as temperature and humidity. They also do not enforce operational dependencies or support time-accelerated evaluation of scheduling tasks. Beyond these limitations, imitation learning
"Wrk done while the author was intern from the Department of Information Systems, Hanyang University. Conresoonding author.
bttps://csa-iiotorr/alll statio//llllllttionnnaateeerr

---

Published as a conference paper at ICLR 2026
Home
* matter
Agent
Server
API Request
Data Model
X
On/fff
Wisther
washer
Operational State
Start
OnOff
Fates True
Attnbbtt
Agent
API Responiee
Operational State
Command
Tools
exe
exe_ond and get Foom www__roonddiiioo devices
OP State
Start
Figure I: The SimuHome home environment. Agents receive user commands, issue API calls built on the Matter protocol operate devices, and observe the resulting state changes.
on static datasets is also insufficient because single user request can often be fulfilled through multiple valid action sequences that fixed annotations cannot cover. Instead, agents must interact with live environment where their actions produce observable state changes that can be verified against the intended goal. We aim to address this challenge by developing high-fidelity smart home simulator which agents can interact with devices through APIs and observe the results reflected in the environment, along with an extensive benchmark containing variety of complex user requests.
Our first contribution is SimuHome, time-accelerated smart home simulator (Figure D Agents interact with the simulator through APIs built on the Matter protocol, global industry standard that defines how smart home devices communicate and operate. When agents operate devices, the simulator computes continuous changes in environmental variables, allowing agents to observe evolving conditions and decide their next actions. For workflow scheduling, agents can register workflows for future execution, and the simulator accelerates time that outcomes verifiable immediately. The simulation is fully reproducible, ensuring fair comparisons across models.
Our second contribution is manually validated benchmark of 600 episodes built on SimuHome. The benchmark spans six query types, namely state inquiry, implicit user intent inference, explicit device control, and three forms of workflow scheduling. Each query type includes both feasible episodes, where the agent must fulfill the request, and infeasible episodes, where the agent must recognize and explain why the request cannot be fulfilled. In each episode, an agent receives a user query in configured home environment and is evaluated against a verifiable goal.
Using this benchmark, we evaluate 18 LLM agents and find that while current models handle explicit device control reliably, workflow scheduling remains the most persistent challenge. GPT-5.1, which uses extended chain-of-thought reasoning, improves substantially on workflow scheduling. However, it requires three to five times the inference time, making impractical for real-time smart home use. We further test alternative agent frameworks and fine-tuning on successful trajectories, but workflow scheduling remains unresolved, suggesting that the bottleneck lies in the models' reasoning capabilities. These findings call for research on agents that combine strong reasoning with the low latency that real-time deployment demands. On the practical side, SimuHome's 's time-accelerated simulation provides a world model environment where we can extensively train and test deploy-ready smart home agents, which is challenging to conduct in physical homes.
RELATED WORK
Embodied Agent Benchmarks in Household Environments. LLM agents have demonstrated
strong tool-use capabilities across diverse domains (Schick et al. 2023 Qin et al. 2024 Patil et al. 2025 Zhou et al. 2024; Xie
2024 Trivedi et al. 2024). In household settings, simulated
benchmarks such as A12THHOR (Kolve et al. (2022), ALFRED Shridhar et al. 2020). and Virtual- Home (Puig et al. 2018) have advanced instruction following by placing agents 3D environments where they navigate rooms and manipulate objects through predefined action commands. These benchmarks focus on physical navigation and object manipulation, which are fundamentally different problems from smart home device control, where agents issue API calls to operate networked devices and must reason about their effects on the surrounding environment.
2

---

Published as a conference paper at ICLR 2026
LLM Agent Benchmarks for Smart Homes. To more directly address smart home device control. recent benchmarks evaluate LLM agents on tasks closer to real-world usage. HomeBench (Li et al. 2025) tests instruction following scale, evaluating agents by comparing their generated API calls against gold-standard sequences across both valid and invalid requests. Sasha (King et al. focuses on creative goal interpretation, mapping underspecified user intentions such " "make cozy" to device-level action plans, with plan quality assessed through human surveys and automated relevance metries. SAGE (Rivkin et al., 2024) frames smart home control as sequential tool use, where agents iteratively call APIs, observe outputs, and select the next action. Unlike HomeBench and Sasha, SAGE allows device settings to change dynamically during an agent's execution through the SmartThings API. Despite these advances, none of these benchmarks simulates how device actions affect environmental variables over time, enforces operational dependencies between device commands, or evaluates time-based scheduling. SimuHome addresses these gaps with an interactive simulator grounded in the Matter protocol, with support for time-accelerated scheduling.
2024
SIMUHOME: A SMART HOME SIMULATOR
3.1
MOTIVATION
Building physical testbed is costly, repeating experiments across diverse configurations is imprac- tical, and waiting in real time for scheduled operations makes large-scale evaluation infeasible. A simulator addresses these challenges, but must closely reflect real device behavior that agents developed in simulation can transfer to physical smart homes. We design SimuHome around three core requirements.
Dependency Modeling Based on an Industry Standard. The simulator must model the operational rules of smart devices according the Matter protocol, so that device behavior in simulation follows the same constraints as physical devices. For example, operating an air conditioner may require multiple steps in specific order, such as powering on the device before adjusting its temperature.
Real-Time Environmental Feedback. The simulator must model the continuous effects of device operations on environmental variables (e.g., temperature, illuminance, humidity, and air quality). This enables evaluation of whether agents can monitor ongoing changes and react appropriately. For example, as an air conditioner runs, the room temperature gradually decreases toward the target temperature, and the simulator must reflect these changes continuously.
Workflow Scheduling and Time Acceleration. The simulator must support workflow scheduling, allowing agents to register sequence of commands for execution specified future time. It must also accelerate simulated time so that the outcomes of scheduled workflows can be observed immediately. This enables evaluation of temporal coordination tasks, such scheduling kitchen light to turn on when the dishwasher finishes.
3.2 SIMUHOME ARCHITECTURE AND OPERATION
SimuHome divides time into discrete steps called ticks, where each tick represents 0.1 seconds. This tick-based design guarantees fully deterministic state transitions, so that given the same initial conditions and the same sequence of actions, the environment always produces identical outcomes. SimuHome runs in sync with real time by default, but supports time acceleration to quickly reach any future point in the simulation. comprises three components, each addressing one the requirements above: the Smart Home Environment defines device configurations and environmental variables, the Real-Time State Update Mechanism computes continuous environmental changes at every tick, and the Agent-Si Interface provides the tools through which agents observe and act.
Smart Home Environment. A home is configurable environment composed of one or more rooms, each containing set of devices and four environmental variables: temperature, illuminance, humidity, and air quality. The environment includes both devices that directly influence environmental variables (c.g., an air conditioner affecting temperature) and those with multi-stage operational cycles (c.g., washing machine). In total, we model 17 distinct device types. A full list is provided in Appendix B along with their supported Matter clusters (groups of related device capabilities) in Appendix Possible extensions to more complex interactions, such as cross-environe cooossenviromentt and cross-device effects, are discussed in Appendix [
3

---

Published as a conference paper at ICLR 2026
Real-Time State Update Mechanism. At each tick, the simulator computes the combined influence of all active devices on environmental variables. This influence is additive, scaling with the number of active devices and their settings. For example, rumning two air conditioners at high fan speed lowers the temperature faster than running one. Sensor attributes on devices, such as a temperature sensor on an air conditioner, are also updated to reflect current environmental variables each tick. The detailed update equations are provided in Appendix E
geent-Simulator Interface. Agents do not have direct access to the full environment state. Instead, they interact with the simulator through tools for querying device states and environmental variables, executing Matter commands, and scheduling workflows (detailed specifications in Appendix [A]. An agent schedules a workflow by calling schedule workfloo with an absolute start time and an ordered list of commands. As on real-world smart home platforms, device states can change unpredictably between scheduling and execution, making it infeasible to validate commands in advance. Accordingly, the simulator confirms that the workflow has been registered but does not validate whether the commands will succeed at execution time. If a command fails at its scheduled time, no error is returned to the agent. The implications of this design for agent evaluation are analyzed in 5.3
4 BENCHMARK DESIGN
4.1 QUERY TYPES
We define six query types that commonly arise in smart home environments, covering state inquiry (QT1), implicit user intent inference (QT2), explicit device control (QT3), and workflow scheduling (QT4-1 through QT4-3). Each query type includes both feasible and infeasible variants, yielding twelve evaluation categories in total, described below.
State Inquiry (QTI). The agent must correctly retrieve and report environmental variables and device settings (such as whether a device is on or its current fan speed). For example, in response to "How humid is the kitchen?", ", the agent must identify the target room, query the environment to obtain the humidity value, and respond with the correct value and units.
Implicit Intent (QT2). Rather than issuing explicit commands, users may express needs indirectly. The agent must infer the underlying goal and act accordingly. For instance, upon hearing "It feels too sticky here in the living room", the agent should recognize this as request to lower humidity, check the living room's 's current humidity and available devices, and then turn on a dehumidifier.
Explicit Device Control (QT3). Users may specify exact devices and target values. The agent must accurately interpret and execute these commands. For example, for the command "Set the living room air purifier fan speed to one hundred percent". the agent must verify the presence of air purifier in the living room. If the device is off, the agent must turn it on first before adjusting the fan speed, respecting the device's 's operational dependencies.
Time-Based Scheduling (QT4-1). This query type involves scheduling the control of one or more devices at specific future time. For example, for the request "Turn off the lights and the humidifier in ten minutes", ", the agent must convert the relative time expression to an absolute time and register a workflow execute the commands at that time.
Event-Driven Scheduling (QT4-2). The agent must coordinate an instantaneous device action (such as turning off a light) with the completion of an ongoing operation (such as a dishwasher cycle). For example, for the request "When the dishwasher finishes, turn off the kitchen lights", the agent must check the dishwasher's s remaining operating time to determine its estimated completion time and then schedule the light to turn off at that time.
Coordinated Scheduling (QT4-).. This query type requires synchronizing two or more operational devices according to given time constraints. For example, for the request "Schedule the dishwasher so that it completes at the same time the washer finishes", the agent must check the remaining operating time of both devices, calculate whether a simultaneous finish is achievable, and if so, adjust the start time of one device and register a workflow accordingly.
Infeasible Variants. Each query type also includes infeasible user requests designed to test whether the agent can recognize and explain why a request cannot be fulfilled. These fall into three categories.
4

---

Published as a conference paper at ICLR 2026
STEP1: Initial State Construction
STEP3: Query Synthesis
18:30
18:45
19:00
[Prompt]
horneoweer,
plausible
300 for
Run washer
1000 AQ ypir
stuation given goals ad ievveed, and genertate ate volce commind that incorporates all them.
Pre-Wash
AD 40
Turn on light
Rinse
OB
Turn
Query
Hey, think utility air purifier strong, Can you
Initial Layout S2ED-42
Sample Actions and Execute
Initial Home State
STEP2: Goal & Prerequisite Action
Fessible
Episode QT3 Feasible
OnOff: Off
Utility Room Air Purifler
Bedroom Light
OnOff: FanSpeed: 40%
Utility Room Air Purtfer
FanSpead: 40%
Utility Room Washer
OnDff:
Goals
Initial Home
Goals
get zoon devi cies (utility roon)
Prerequisite Actions
Query
Sample Devices
Sample Goals
Prerequisite Actions
Figure 2: Episode generation pipeline. Each episode is constructed three steps: initial home state randomization with dependency-awee aware device activation, goal and prerequisite action generation, and query synthesis with human validation.
First, non-existent resources, where the user asks about attempts to control device that does not exist the specified room. Second, physical limits, where the relevant devices exist but cannot achieve the requested change because they already maximum capacity. Third, temporal contradictions, where the user's time specifications conflict with each other or with device operating constraints. In all three categories, the agent must gather evidence from the environment, identify the constraint, and inform the user. Detailed descriptions of each infeasible query type are provided in Appendix
4.2 EPISODE GENERATION
Definition and Components of Episode. An episode is a single, self-contained evaluation unit for the agent. As illustrated Figure 2 each episode comprises four components. The initial home state defines the starting configuration of the home, including room layouts, device states, and environmental variable values. The goal is a structured dictionary specifying the desired outcome, such which are a set of tool calls that must appear in the agent's tool-call history. For instance, before controlling device, the agent must invoke target room, preventing success through guesswork. An episode is marked as successful only if both the goal satisfied and all prerequisite actions appear in the agent's tool-call history. The query is natural-language utterance that embodies the goal. Because accurate evaluation depends query clarity, we first define verifiable goals and then synthesize queries from them, as described below.
and device target and what state the device should reach. Prerequisite actions
wwwoomddeviiioooo ces (ut lutility_roan
to confirm exists the
STEP1: Initial Home State Construction. The initial home state is constructed in two stages to ensure diverse and realistic starting conditions (Figure 2 First, a variety of physical layouts with different room and device configurations are generated. For infeasible episodes involving non-cxistent devices, the layout is intentionally configured without the target device in the specified room. Second, starting from all-off state, the simulator runs multiple rounds of randomization. In each round, it samples and executes one command per device, following the order required by each device's operational dependencies. For example, on before its fan speed can be adjusted, so the power-on command is sampled in an earlier round and the fan-speed command in later round. The simulator then accelerates time forward that environmental variables update according to the combined effects of all active devices. The resulting state becomes the initial home state for the episode. Randomization controlled by a seed to ensure full reproducibility.
air conditioner must first be powered
5

---

Published as a conference paper at ICLR 2026
18:30
18:31
18:45
19:00
?
Workflow Scheduled
Washer: Rinse Kitchen
Washer: Off
Simulator
Kitchen light: On
EPISODE Feasible
Reasoning
Simulator based Evaluation
Switch on the kitchen light when washer is done.
Observation d"device" "washer",
EPISODE GOAL
"remaining time": ::
assert Scheduled time 19:00
30 remaining, and the current time is 18:30. Ined turn on the
Thaught
assert Washer Off. Kitchen light On
The kitchen light is scheduled turn 7 PM.
kitchen light at 19:00.
Prerequisite actions
Action schedule... workflow (ttmeetmmmm2::::5 "2025.09.0
tool-call history
Agent
steps: [command( kitchen _ight"", ')3)
SUCCESS
EPISODE QT4 infeasible
8:30
Reasoning
LLM-as-ea a-dgee Evaluation
It's 7 PM now, right? Quests comiig soon, so turn on the living room lightat at five minutes now
Observation
("current "'murrent_time"""
":"m""""""""""""" "2025.09.01
seems the sgent did not correctty recognize the user mistake.
18:30 )
User
Thought
The current time is 6:30 6:30p.m m.. and the living
guests are coming soon.
room right away.
Judge X
Judge
Judge 3 X
just tumed on the diiiiin room light.
Action
command(
"Ivingroommllege command("Ivinrroom.
"On')
FAIL
Agent
Figure 3: Episode evaluation pipeline. Episodes involving state changes are evaluated by comparing the simulator's final state against the goal. Episodes requiring natural -auural- assessment are evaluated by an LLM-as-a-Judge.
STEP2: Goal and Prerequisite Action Generation. Goals are structured differently depending on the query type. QT3 goals specify target attribute values for particular devices, whereas QT2 goals indicate the expected direction of change for environmental variable, such as whether temperature should decrease. QT4 goals additionally include a target time at which the specified device state should hold. A device goal is created by sampling from the set of states that satisfy the device's operational dependencies (e.g.. OnOff: On, FanSpeed: 40%). The generation process, which varies by query type (see Appendix G ensures that all goals logically consistent. Each goal specifies target room along with a device or environmental variable to address. The prerequisite actions are then set require the agent to first query that room before taking any action, for instance by calling get wtt.rroo.deiicc. ().
For infeasible episodes, the goal records the specific condition that makes the request impossible. For non-existent device episodes, the goal records the absence of the target device in the specified room. For physical limit episodes, it records that the relevant devices are already at maximum capacity. For temporal contradiction episodes, the goal is constructed by deliberately sampling contradictory time values, such as pairing an incorrect assumed time with the actual current time or an impossible deadline with device's s remaining operating time. In all three cases, the recorded condition is later provided to the LLM judge, which uses it to assess whether the agent correctly identified and communicated the impossibility to the user.
STEP3:
Query Synthesis. We used GPT-5 mini (OpenAl 2025c) to generate query drafts from goals. To ensure that each query accurately reflects its corresponding goal, two graduate students researching tool agents independently reviewed the entire dataset and corrected queries that did not match the goals. Their Cohen's 's (Cohen 1960) anner-annotttor agreement was 0.92, indicating highly consistent review process.
In total, construct 50 distinet episodes for each of the 12 evaluation categories (six query types, each with feasible and infeasible variants), yielding 600 episodes.
4.3 EVALUATION METHODS
As illustrated in Figure 3 we evaluate agent performance using two complementary methods, chosen based on what each episode requires for assessment. Episodes where success is determined by physical state changes the home environment are evaluated by the simulator, which can objectively verify outcomes. Episodes where success depends the accuracy or appropriateness of the agent's natural language response are evaluated by an LLMM-as-a-Judge (Zheng et al. 2023).

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results show success rates (in %) across query types (QTs). and IF refer to Feasible and Infeasible episodes, respectively. Superscripts and indicate results from simulator- based and as-a--u evaluation, respectively. Bold and underlined values indicate the best and second-best results per column.
QTI
QT3
QT4-1
QT4-2
QT4-3
Models
IP
F
IF
IF
F
IF
FA
IF
F
IPP
Open Source Large Language Models (<7BB
L1ama3.2 13 it
0
1lama3.2-32-
10
12
0
2
0
2
0
2
0
3-48-it
Gemma 3-mmaa-4BB-a 3-4B-it 1ig-it (SFT)
58
22
30
2
Open Source Large Language Models
ama 4-Scout
Llama Maveriok
96
78
52
36
74
22
14
18
10
Qwen3-322 32B
66
18
14
14
Qwen3-328 (SFT)
88
64
26
32
20
QH223-235R--22R
74
32
70
26
18
38
3-120-12
38
2
Germa 27D
80
48
54
24
48
44
4
2
10
Closed Source Large Language Models
2.5 Lite
60
8
34
1emini-2.5--leah
92
86
66
54
82
74
22
44
40
32
12
32
CPT-4.1-ran
42
96
76
62
28
64
76
26
40
40
20
10
28
GPT 4.1
82
50
12
Closed Source Large Language Models (with Reasoning)
Gomini-2.5-Prr
96
78
60
56
76
72
44
94
60
76
46
50
GPT-5.1
Table Average episode completion time (seconds) across query types. and IF refer to Feasible and Infeasible episodes, respectively.
QTI
QT2
QT3
QT4-1
QT4-2
QT4-3
Model
F
IF
F
IF
IF
GPT-4.1
8,3
7.8
23.6
20.2
22.9
9.4
26.6
12.3
28.7
23.7
29.7
25.9
Gemini 2.5 Pro
24.1
57.5
48.8
66.1
27.8
74.0
12.5
57.7
37.0
53.7
53.1
GPT-5.1
35.7
38.4
109.4
99.6
78.6
54.3
121.1
13.5
135.1
76.0
112.7
111.0
Simulator-based Evaluation. At the end of each episode, the simulator compares the structured goal (defined STEP2) against the final state of all relevant devices and environmental variables. The nature of this comparison varies by query type. For QT3, the simulator checks whether each target device attribute matches its goal value exactly. For QT2, verifies whether the relevant environmental variable changed in the intended direction (c.g.. whether simulator first accelerates time to the target moment specified in the goal and then checks whether the designated devices are in their expected states at that point. The upper portion of Figure 3 illustrates this process for a QT4-Feasible episode. The simulator also checks that all prerequisite actions appear in the agent's 's tool-call history. This direct state comparison enables fully automated and fair model-to-moded comparisons. We apply simulator-based evaluation to all feasible episodes in QT2, QT3, and QT4, which involve physical state changes.
temperature decreased). For QT4, the
LLM-aa----uudee Evaluation. We employ an LLM-based judge for episodes where success depends on the agent's final natural-language response rather than physical state changes. Given an episode's goal description and user query, the judge additionally receives the agent's 's full reasoning trajectory assess whether the response accurate and appropriately grounded. This applies to all infeasible episodes, where the agent must correctly identify and explain why the request cannot be fulfilled. Among feasible episodes, only Fea-sbes uses the LLM judge, as the task requires reporting correct values rather than changing device state. The lower portion of Figure illlstrrass this process for QT4-Infeasible episode. For reliability, we query the judge three times per episode and adopt the majority vote as the final judgment (Wang et al.) (023). Our judges achieved substantial agreement (Cohen's 's x 0.826) with human evaluations (sce Appendix H). Detailed prompt templates are in Appendix Q.1

---

Published as a conference paper at ICLR 2026
Table Error taxonomy. Detailed descriptions and examples are provided in Appendix J.1
Category
Error Type
Definition
Teasible
Envirumment Perception (EI)
Failure to correctly perceive envirommental variables.
Intent Inference (lI)
Misinterpreting the user's underlying goal.
Device Control (DC)
Operating the wrong device command. Tncomplite incorrect planning of actions.
Action Planning (AP)
Temporal Reasoning
Miscalculating times sequence aligument.
Infeasible
Contradiction Mishandling (CM)
Detects contradiction but fails to follow the instraction.
Contradiction Blindness (CB)
Fails detect contrudiction.
LLM Judge (LJ)
Misclaxsification LLM J.M.Jddee......
AP
EP
1%
%
II
6%
10%
18%
3% 12%
DC
40%
11%
19%
AP
71%
B5N
CM
CB
QT2
QT4-F²
(c)
QT4
Figure 4: Error type distributions GPT-4.1 on QT2 and QT4.
5 EXPERIMENTS
Experimental Setup. We evaluate 18 models across the 12 evaluation categories defined in 1 These span four groups: open-source models under 7B parameters, larger open-source models (7B and above), closed-source models, and closed-source models with extended chain-of- of-thought reasoning. We refer to models without extended reasoning as non-reasoning models. All experiments use the ReAct framework (Yao et al. 2023). Reproducibility details and agent prompts are in Appendices
and Q
5.1
MAIN RESULTS
Table
D presents success rates across all query types. Models under 7B parameters achieve near-zero success rates most tasks, with only Gemma3-4B-ir showing limited success state inquiry and explicit device control.
Among larger models, three trends emerge. First, state inquiry (QT1) and explicit device control (QT3) are relatively approachable, with multiple models succeeding on feasible episodes. When instructions are explicit and tool feedback is immediate, current models can follow multi-step device-control sequences reliably.
Second, implicit intent inference (QT2) and workflow scheduling (QT4) are considerably harder. QT2 requires agents to infer the user's underlying intent before acting, an additional reasoning step that explicit commands do not demand. QT4 adds further layer of difficulty by requiring temporal coordination on top of device control?
Third, reasoning models generally outperform non-reasoning models, with the largest gains QT2 and QT4. On infeasible workflow scheduling, reasoning models detect contradictions far more reliably than their non-reasoning counterparts. However, the improvement is uneven within QT4. When the agent needs to schedule actions around a single future time point, such as turning off a light in ten minutes or when device finishes, extended reasoning helps substantially. Yet if the agent must coordinate the timing of two or more devices to finish together, reasoning models still struggle.
1PPT-4.1 scores lower than GPT-4.1-min on QT2-F because incorrectly sets the light's mansition-time 2-3 seconds despite the prompt specifying immediate changes. Since evaluation occurs immediately after the command, the brightness has not yet reached target. When a 3-second delay allowed, its success rale rises from 44% to 62%. See Appendix K for details.
8

---

Published as a conference paper at ICLR 2026
Avenge Tooa par Expeads
10
8/9
8.4
(c-42)
57.1%
42.9%
1.0
Liama Max (s-44)
64.5%
45.0%
8.5
Gemini-2.5FFlaat
(s-41)
Seccourte Tus Frne Toal cokk
Gwen132268
47.
Fearry
Resieired Susias
52.4%
Composition Suczess Outcooees
Figure 5: Tool-call error patterns four models on QT3-E The left chart shows the average number of errors relative to the average number of tool calls in successful episodes. The right chart shows the proportion of episodes achieved through first-try success versus those requiring error recovery.
Despite these gains, the improvement comes at a significant cost in latency. As shown in Table 2 reasoning models take roughly three five times longer than GPT-4.1 per episode, with GPT-5.1 requiring over 100 seconds on the most demanding tasks. Smart home users typically expect near- instant responses, making even GPT-4.1's latency of up 30 seconds challenging for real-time deployment. Given these constraints, the following analyses focus on GPT-4.1, the best-performing non-reasoning model.
5.2
ERROR ANALYSIS
To understand where GPT-4.1 fails and why, categorize its errors into five types for feasible episodes and three for infeasible episodes (Table{3). Figure 4 summarizes the distributions for QT2 and QT4, where GPT-4.1 shows the lowest success rates among all query types.
Figuriessmmmrizzs
Feasible episodes. QT2 (Figure 4p). Device Control (DC) errors dominate at 71%. In these cases, the model operates the wrong device, issues incorrect commands, or skips required steps in the device's s operational dependencies, such as powering on a device before adjusting its settings. Intent Inference (II) errors account for 11% reflecting difficulty in mapping vague complaints to the appropriate device action. In QT4 (Figure 4p), the error distribution is more diverse. with DC (40%), Temporal Reasoning (TR, 25%). and Action Planning (AP, 19%) all contributing. Workflow scheduling demands multiple capabiliiies at once, from correct device identification to accurate time calculation and coherent multi-step planning.
Infeasible episodes. In QT2 (Figure 40), GPT-4.1 frequently detects the contradiction but fails to respond appropriately, resulting Contradiction Mishandling (CM) errors. For example, when asked to raise the kitchen temperature using non-existent heat pump, the model instead operates the living-room heat pump rather than informing the user. In QT4 (Figure ft), the dominant issue shifts to Contradiction Blindness (CB), where the model fails to recognize temporal infcasibility altogether and proceeds as if the request were valid. This contrast suggests that verifying whether a resource exists is easier for current models than checking whether time constraints satisfiable.
Workflow scheduling (QT4) thus presents the most persistent challenge across all query types. The next section examines structural factor that contributes to this difficulty.
5.3 ROLE TOoL FEEDBACK
To understand why workflow scheduling is disproportionately difficult, we compared how agents respond to tool feedback on QT3 versus QT4. Figure 5 shows that over 40% of successful QT3 episodes involved recovery after initial invalid tool call, indicating that agents adapted based on enror messages rather than relying on prior knowledge of the Matter protocol. This ability to recover from mistakes explains their robustness explicit device-control tasks.
QT4 episodes, in contrast, largely rely on the tool returns only confirmation that the workflow has been registered, without validating whether the scheduled commands will succeed at execution time. This mirrors real-world smart home platforms, where conditions change between scheduling and execution, making advance validation infeasible. Because no corrective signal is available after scheduling, agents cannot detect erroneous plans.
tool. As described in $3.2, this

---

Published as a conference paper at ICLR 2026
This structural difference between immediate feedback (QT3) and deferred feedback (QT4) is one key factor behind the performance gap between the two query types. SimuHome's accelecclerrate simulation offers potential path forward: agents could test their scheduled commands in an accelerated simulation before committing them to the real environment (see Appendix L..
5.4 DISENTANGLING FRAMEWORK LIMITATIONS FROM MODEL CAPABILITIES
To determine whether failures on the hardest query types stem from the ReAct framework or from the models' intrinsic limitations, we conducted a series of experiments at both inference time and training time.
Inference-time approaches. We first replaced ReAct with HiAgent (Hu et al. 2025), a framework de- signed for long-horizon tasks through structured memory and planning. HiAgent outperformed ReAct on QT4-2 and QT4-3 but underperformed on QT4-1, producing mixed results overall (Appendix [M). We also tested multi-turn interaction on QT2-F, allowing the agent to ask clarifying questions and providing corrective feedback after failures. Success improved from 44% most 54%, suggesting that richer interaction alone does not resolve the underlying reasoning gap (Appendix NN)
We then tested whether agents could recover from scheduling errors through self-correction. We explicitly prompted the agent to review and correct its scheduled workflow both before and after execution, but recovery rates remained negligible, reaching at most 18.5% on QT4-2 and 0.0% on QT4-3 (Appendix M). Even when explicitly instructed to check the home state after the scheduled workflow ran, the agent failed recognize its own errors in most cases. As an upper-bound estimate, we tested a setting where the system immediately told the agent that its scheduled commands failed (Appendix(O) O.. Recovery rates reached 55-67% across QT4 subtypes, confirming that timely feedback can mitigate many failures. However, this setting assumes oracle that detects failures at execution time, which impractical in dynamic smart home environments. These results should therefore be interpreted upper bound on what timely feedback could achieve.
Supervised fine-tuning. We fine-tuned Gemma3-4B-it and Qwen3-32B on 204 successful interaction sequences collected from GPT-5.1 (Appendix PP. Both models improved most on infeasible-request detection, with gains up to 26 percentage points (e.g., QT1-IF). Infeasible episodes share a common structure of verifying the environment and declining the request, pattern that fine-tuning captures well. Feasible workflow scheduling, however, saw limited gains, and coordinated scheduling (QT4-3- F) did not improve for either model. This is because feasible workflow scheduling requires dynamic interaction with the environment that imitation of past solutions cannot capture.
Neither inference-tine nor training- -imei approaches fully resolve workflow scheduling. These results indicate that the primary bottleneck is not the ReAct framework but the models themselves. Improving workflow scheduling may require approaches that go beyond imitating successful trajectories, such as learning through trial and error inside a simulator.
6 CONCLUSION
We introduced SimuHome, an interactive smart home simulator grounded in the Matter protocol, and benchmark of 600 episodes across twelve evaluation categories. By letting agents operate devices and observe the resulting changes in the environment, SimuHome enables evaluation of complex user requests that static benchmarks with fixed action sequences cannot assess. Our evaluation of 18 models reveals that workflow scheduling is the most persistent challenge. In explicit device control, over 40% of successful episodes involved recovery from initial errors through tool feedback. By contrast, workflow scheduling offers no such corrective signal, and agents cannot detect their own mistakes ($5.)) Supervised fine-tumin tne-ttning improves infeasible request detection but sees limited gains on feasible workfllw scheduling, where the time calculations vary across episodes and cannot be learned by imitating past solutions ($5.4). These findings suggest two directions for future work. First, SimuHome's 's ance-accelerated simulation could serve as environment for agents test their plans before committing them to the real world. Second, agents may benefit from learning through trial and error inside the simulator rather than imitating recorded examples.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) under the grants RS-2024-00333448 and 22220-0044498811 and by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) under the Leading Generative AI Human Resources Development grant ITTP-2026-RG RS--RRS-22---- 00024-0397008 and the grant RS-2025-02215522, (Development and Demonstration Lightweight
Model Smart Homes), all funded by the Korean government (MSIT). This work was also supported by the National Supercomputing Center with supercomputing resources including technical support (KSC-2025-CREE-S114 and by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2024-003477991)
REFERENCES
Jacob Cohen. coefficient agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-466 April 1960. doi: 10.11770011364460000000 URL https
Gheorghe Comanici, Eric Bieber, Mike Schackermann, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025,
URL http:ttts//:.ii....c.m o abs/ 06261
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. URL https: http://wwww..oo/./.....oo/oo www.orgg/abbb/aa 2503. 255..197866 19786
Aaron Grattafiori, Dubey,,,uuuubbb, Abhinav Jauhri, et al. The Ilama 3 herd of models, 2024. URL
https: //a raaaxivvvv.... httpp/////erlivoooaa///////// wttg//////////o// .277822183
Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hia- Hierarchical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 32779-32798 32798, Vienna, Austria, July 2025. Association for Computational Linguisties. doi: 10. 1/255//1/2025 https: acl antho conthology.. logy org. 2025 acl- 15757
gent:
URI.
Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. Sasha: Creative goal-oriented reasoning in smart homes with large language models. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(1), March 2024.
doi: 10.1145/3643505. URL https
htdllacm.org/////// ooo http://ooo....//1// 1545 1145/3435503 3643505
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI, 2022. URL https
//axxiv.org/abs/11/
05474
Silin Li, Yuhang Guo, Jiashu Yao, Zeming Liu, and Haifeng Wang. Homebench: Evaluating Ilms in smart homes with valid and invalid instructions across single and multiple devices. In Proceedings of the 63rd Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12230-12250, Vienna, Austria, July 2025, Association for Computational Linguis- www.:////..o..oooonoooooooo acl-gog.599 URL https:
tics. doi: 10.18653/v1/2023 Bel-long.500
Meta AI. The Llama herd: The beginning of new era of natively multimodal intel- ligence. Technical report, Meta, April 2025. URL https://a// .etaa.a...mmmmm cwm/clooooooooooo blog/ Introducing Llama Scout and Maverick.
OpenAl. Introducing GPT-4.1 in the api. Technical report, OpenAl, 2025a.
URL https://
openai catimaa..cmmmii
OpenAl. GPT-5.1 instant and GPT-5.1 thinking system card addendum. Tech-
nical report, OpenAl, November 2025b. URL hattps://openal cwm. index/ Accessed: 2025-11-26.
g 5- www.:-syyttee--ar--aadeeeeeeeeee -ptttm55-mm-ptt-111111--

---

Published as a conference paper at ICLR 2026
OpenAl. GPT-5 system card. Technical report, OpenAl, August 2025c. URL https://opena Accessed: 2025-11-26.
com
OpenRouter. OpenRouter: The Unified Interface for LLMs, 2025. URL htttts://opperouuuree
ai/
Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, lon Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267. PMLR, 2025. URL / Poster.
2025/poster/4e 4 6593
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. VirtualHome: Simulating Household Activities via Programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8494-8502. thecvf com/ content_cvpr 2018/ html/Puie VirtualHomm Simulating Household CVPR 2018 paper html
IEEE, June 2018. URL
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024. htt.ttss://penzeviiiee
net fttf/foruae ott./forrum.ddddlll220000
Dmitriy Rivkin, Francois Hogan, Amal Feriani, Abhisek Konar, Adam Sigal, Steve Liu, and Greg Dudek. SAGE: Smart home agent with grounded execution, 2024. URL httph::t////t //arxiv
org/ abs abs/2311..0777
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.
A. Oh, т. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 68539-68551. Curran Associates, Inc.. 2023. URL https::/pproceedinggg neurips ww/paper /1tes//ppppee/223///////// Paa.eesoo---ppee--fffeeeeeeeeeee
file/ file/d8424222
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020. URL twectf.eecccccc com citom/content_cPPP Ct__C_/PPP____------ 2020/html haattmmmm//hhhhhaaaaa dhar for_
ALFRED A www.BBennchhrreeemeeeaaa for Interpreting Grw.nnddd
Everyday_Tasks_coe CVPR httCCCC_R22__peee....ea html
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16022-16076 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.11853/v1/2024.uu acl- 850. URL 85C/]
acl acl-long
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL net wwt.:n/oooommmddiiii....
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Traveiplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, 2024. URL https: wwwnreview. Forum? ?id 10l.1=11xx22k0e
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, al. Qwen3 Technical Report, 2025. URL https::/aaxtt
org/ exq/abs/2530 aww.ab5........... 09388
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR). OpenReview.net, 2023. URL
httpps://penzeviee..rr net foee://ormmiddde__eaa. vIuYULLXX
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena. A. Oh, Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 46595-46622 Curran Associates, Inc., 2023. ec htsp:://prrceeiinss..nrrrrr http://pppppppppppppee/////////p
URL
91f1 Ba oa -att-attattttt paae and Beataasttsaand_BBnchaakkk..
pdf
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo. Abishek Sridhar, Xianyi Cheng,
Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. UtL/tttss://openeeee wwt.w/forenttttt od=oKn9c6ytIx
13

---

Published as a conference paper at ICLR 2026
A
LIST OF TOOLS
Table 4: List of tools available to agents.
Name
Description
Args
Complete the task and return the final antural-language answer.
answer (str, req): Final response text.
finish
Get control rules for specific environmental state.
state (str, req): Environmental state (temp, humidity, etc).
get eww.eeeiionmentccommm control rules
user a question to gather additional information, clarify ambiguity. confirm preferences, or get missing details.
ask
question (str, req).
Execute a command on device (e.g., turn light, set level, set selpoint).
device.id, endpoint.id, cluster.id, command.id (strs/ints, req); args (dict, req).
execute.command
devicesid, cluster.id, attribute.u iid (strs, req); value (any, req).
write_attin attribuue
Directly set device attribute value.
det_all_attributes
Get all attrihutes of device.
device.) (str, req).
device.id, cluster.id, attribute.id (str, req).
get_attribute
Get a specific attribute of a device.
Get device structure (endpoints, clusters, attributes, and commands).
get_device_streeee
device_id (str, req).
rooms home along with their display names.
get.r
(none)
get_room_device
Get devices in room.
room_id (str, req).
Get environmental states of (temperature, humidity, illuminance, PM10).
getrromm_states
room.id (str, req).
Perform semantic search across Matter cluster documentation.
get_cluster_das
query (str, req); top.k (int, req).
current virtual time human-fricndly string "YYYY-MM-D DD HH.MM:SS".
get.current.imme
(nonc)
Schedule a sequential workflow of steps at virtual absolute time.
start.tine (str, reg): a steps (list, req).
schedule workflow
cancel.workflow
Cancel scheduled workflow by id.
workflow (str, req).
get workflow status
workflow status by id.
workflow id (str, req).
Get list of workflows with optional filtering.
get.workffow list
(none)
14

---

Published as a conference paper at ICLR 2026
B LIST OF DEVICE TYPES
Table 5: List of implemented device types and their corresponding clusters.
Device type
Clusters
Air Conditioner
Basic Information, Fan Control, On/Off, Thermostat
Air Purifier
Basic Information, Descriptor, Fan Control, Identify, On/Off
Dehumidifier
Basic Information, Fan Control, On/Off, Relative Humidity Measurement
Dimmable Light
Basic Information, Level Control, On/Off
Dishwasher
Basic Information, On/Off, Operational State
Basic Information, Electrical Energy Measurement, Electrical Power Measurement, Power Topology
Electrical Sensor
Fan
Basic Information, Fan Control. On/Off
Basic Information, Descriptor, RTCC Mode, Temperature Control, Temperature Measurement
Freezer
Basic Information, Descriptor, Device Energy Management, Electrical
Heat Pump
Energy Measurement, Electrical Power Measurement, Power Source, Power Topology, Thermostat
Humidifier
Basic Information, Fan Control, On/Off, Relative Humidity Measurement
Laundry Dryer
Basic Information, Laundry Dryer Controls, On/Off, Operational State
Basic Information, Laundry Washer Controls, Laundry Washer Mode, On/Off, Operational State, Temperature Control
Laundry Washer
On Off Light
Basic Information, On/Off
Basic Information, Descriptor, RTCC Mode, Temperature Control, Temperature Mcasurement
Refrigerator
Basic Information, RVC Clean Mode, RVC Operational State, RVC RunMode
RVC
Basic Information, Channel, Keypad Input, Level Control, Media Playback, On/Off Basic Information, Window Covering
TV
Window Covering Controller
C LIST OF MATTER CLUSTERS
SimuHome's device interface modeled after the Matter application cluster library, adopting its cluster-atributt- command abstraction so that agents interact with devices through interfaces representative of real smart home platforms. Since our focus is on evaluating agents' reasoning over device operations and environmental effects, we implement the application-layer semantics relevant to agent interaction. Protocol-level mechanisms such as transport security, commissioning, and fabric management outside scope.
Table 6: List implemented Matter clusters.
Cluster
Attributes
Commands
Basic Information
VendorName, VendorID, ProductName, ProductD
None
Continued on next page
15

---

Published as a conference paper at ICLR 2026
Table continued from previous page
Cluster
Attributes
Commands
DeviceTyppLList, ServerList, CiientList, PartsList, TagT.st
Descriptor
None
OnOff, GlobalSecneConttoll, OnTime, OffWaiTTime, StanUpOnOff
On/Off
Off, On, Toggle
MoveToLLvel, Move, Step,
CurrentL.evel, RemainingTime MinLeevl,, MaxLevel, CurrentFrequency. MinFrequency. MaxFrequency, OnOOTTransition Time, OnLevel, TrTanniiio Time, OffTransition Time, DefaultMoveRate, Options, StarUpCurrentLevel
Stop,
Move/Jol.ee//WihhOnOff,,
Level Control
MoveWithOnOff,,
StepWithOnOff, StopWithOnOff
FanMode, FanModeScqueence, PercentSetting, PercentCurrent
Fan Control
Step
Play, Pause, Stop, PlaybackResponse
Media Playback
CurrentState
ChangeChannelyyNumbe,, SkipChannel
Channel
ChannelList, Lineup, CurrentChannel
Keypad Input
None
SendKey, SendKeyResponse
Identify
Identify IdentifyType
Identify, Identify Time
PhaseList, CurrentPhase. TountdownTime, OperationalStateList, OperationalState, OperationalFrrror
Pause, Resume, Stop. Start, OperationalComman- dResponse
Operational State
Power Source
Status, Order, Description, EndpointList
None
Power Topology
AvailableEnddoints, ActieEddponss
None
Electrical Power Measurement
PowerMode, NumberOOMeessureeeettTyyy. Accuracy, ActivePower
None
Electrical Energy Measurement
Accuracy
None
Device Energy Management
ESAType, ESACanGennraate, ESAState, AbsMinPower, AbsMaxPower
None
Dishwasher Mode
SupportedModes, CurrentMode
ChangeTToMode
Dishwasher Alarm
Mask, Latch, State, Supported
Reset, ModiffEnabledAAlrmss
RTCC Mode
SupporteddMdess CurrentMode
ChangeTToMode
RVC Clean Mode
SupportedModes, CurrentMode
ChangeToModde
PhaseList, CurrentPhase. CountdowaTTime, OperationalStattliis.. OperationalState, OperationalErroo
RVC Operational
Pause, Resume, GoHome
State
RVC Run Mode
SupportedModes, CurrentMode
ChangeToModde
TemperatureSSepoonn. MinTempprrature,
Temperature
MaxTemperature. Step. Selecte/Teepperatueeeeeel.
SerTemperature
Control
Supponte//emmpertuueL.vvss
Temperature
MeasuredValue, MinMeasuredValue, Valu
None
Mcasurement
Continued on next page
16

---

Published as a conference paper at ICLR 2026
Table continued from previous page
Cluster
Attributes
Commands
LocalTemperature, OecupiedCCollingSetpon... OccupicHHcaringSeenonn,, ControlSepennccoOOpprraaionn SystemMode
Thermostat
SeipoimRaisel.ower
Type, ConfigStatus, OperationalStatus, EndProouurTyyp. Mode
UpOrOpen. DownOrClose, StopMotion
Window Covering
Laundry Dryer Controls
SupportedDrrynessLevel.. SelsciedDrynessLeeel
None
Laundry Washer Controls
SpinSpeeds, SpinSpeedCurrent, NumberOlRinses, SupportedRinses
None
Laundry Washer Mode
CurrentMode, SupportedModes
ChangeTTooode
Relative Humidity Measurement
Measured VinMeasuredValue, MaxMeasured Tolerance
None
D COMPLEX ENVIRONMENTAL INTERACTIONS
To support increasingly realistic smart home scenarios, SimuHome's environmental update mechanism can be ex- tended to accommodate more complex interactions, including Environment--Enn EEavirnmment and Device-DDevice interactions.
interactions can he implemented by introducing additional devices that mediate environmental variables. For example, window can he modeled as a standard device with Open/Close/Outsinee Temperature attributes that directly affect indoor temperature. By adding an external heat influx coefficient to the update equation, the simulator can dynamically reduce the cooling efficiency of the air conditioner when the window is in the Open state.
interactions can be implemented by introducing additional environmental variables that mediate between devices. For instance, total power load can be defined and tracked as an environmental variable, analogous to temperature or illuminance, whose value is adjusted by the power consumption of devices in the home. This variable can then mediate interactions between devices. If the total power load exceeds a safety threshold, the simulator forcibly shut down all devices, thereby simulating breaker trip scenario.
These examples illustrate how SimuHome's environmental update mechanism can accommodate richer device and environment interactions without requiring changes to the core simulation loop.
E ENVIRONMENTAL UPDATE EQUATIONS
At each tick duration At (default s), simulator updates the environment three phases: devices with timed operational cycles (e.g.. washing machines) advance their internal state, environmental update rules compute how active devices change environmental variables, and the scheduler dispatches any due workflows. The equations below provide simplified environmental model designed for agent evaluation rather than accurate physical simulation. They capture the qualitative effects that agents must handle, such as gradual temperature changes and device intoractions, while remaining computationally efficient and fully deterministic. All constants are configurable and can be adjusted for different scenarios. The values reported here are the defaults used in all experiments.
Temperature, Humidity, and PM10 (air quality). Let denote the tick index and let E [T, H, P} denote temperature, humidity, or PM10 concentration. respectively. Let Dz,, be the set of devices in room that affect variable x. Each variable updated in two steps. First, per-tick effects of relevant devices are summed and added to the current value to produce an intermediate result Second, reverts toward the room baseline be at rate governed by the decay constant Og:
#fn+1] 1] a/1]
Σ Ar.(n),
(1)
iE De,
x|n+1] a 1] + a At (bz Tfa+1]),
(2)
17

---

Published as a conference paper at ICLR 2026
where [n.(r] is the per-tick effect of device i variable z, determined by its current attribute values (e.g.. power state, fan speed, mode, and setpoint). The baseline ba is the value that the room settles when no device actively influencing æ, set globally by default but overridable per room. The decay constants are:
or 2 10
a# 10
aP
10
Because humidity and PMI0 concentration cannot fall below zero in reality, and humidity cannot exceed 100%, the simulator enforces these physical bounds after each update:
H[++1] 1] f- clip(H((n ([]++]++)]])) 1], 0, 10000):
Pla+1] max(0, Pin++1l).
The humidity bound of 10000 corresponds to 100.00%, humidity values are stored in hundredths of percent. Temperature not bounded, indoor temperatures stay within representable ranges during simulation.
Illuminance. Unlike temperature, humidity, and PM10, which accumulate device effects over successive ticks, illuminance is recomputed from scratch at each tick. Let DL. the set of light devices in room r and let by. be the room baseline (e.g., natural light). Then:
500
if i is an on/off light and is on,
if i is dimmable light and is on, otherwise, where E, is the brightness level of dimmable light (integer, range 0-254) 500 maximum contribution per light device.
L = by
Σ
Ce.
500
4 /54
254
PEDL
0
F INFEASIBLE QUERY TYPES
State Inquiry with Non-Existent Device (QT1-IF). The user asks about a device or attrihute that does not exist in the specified room. For example, for the request "Can you tell me the vendor ID for the air purifier in the living room?", the agent must check the list of devices in the living room, confirm the absence of air purifier, and inform the user that the request cannot be answered.
Implicit Intent at Physical Limit (QT2-IF). The user expresses discomfort that implics a need for environmental adjustment, but the relevant devices are alrcady operating at maximum capacity. For example, in response to "The living room feels like sauna", the agent must verify that all available cooling devices are already at their highest settings and explain why further cooling is possible.
Device Control with Ron-Existent Device (QT3-IF). The user explicitly requests control of device that does not exist in the specified room. For example, for "Turr on the humidifier in the living room",, ", the agent must check the device list, confirm the absence of a humidifier, and inform the user that the request cannot be fulfilled without altering any device state.
Time-BBsed Scheduling with Time Contradiction (QT4-1-IF), The user specifies both relative and an absolute time for scheduling request, the two are contradictory, or the user holds a wrong assumption about the current time. For example, if the user says "It's 6 p.m. now, right? Turn on the kitchen light five minutes later at 6:05 p.m." but the actual time is not 6 p.m., the agent must detect the discrepancy and explain the contradiction rather than proceeding with the schedule.
Event-Driven Scheduling with Time Contradiction (QT4--IIF). The user ties a scheduling request to device's completion time but makes incorrect assumptions about when the device finishes, creating contradiction between the assumed completion time, a relative offset, and an absolute time. For example, if washer finishes 6:30 p.m. but the user says "I think the washer finishes at 6 p.m.. so start the dehumidifier at 5:50 p.m., which is 10 minutes before it finishes", ", the agent must check the actual completion time and point out the inconsistency. The agent should not register the schedule until the user clarifies the intended timing.
Coordinated Scheduling with Impossible Deadline (QT4-3-IF), The user requests that two or more devices finish by specific deadline, but the remaining operating time of at least one device makes the deadline physically impossible. For example, if the user requests "Guests arrive at 6 p.m.. ensure both the washer and the dishwasher completed by 5:30 p.m.", ", the agent must check each device's remaining time, explain why the deadline cannot be met, and suggest the earliest feasible completion time.
G GOAL EXAMPLES
18

---

Published as a conference paper at ICLR 2026
Table 7: Example goals for each query type.
Query Type
Query
Prerequisite actions
Goal
How bright is the utility room lighting right now? sorting some boxcs and wondering if there is enough light. Also how is the living room humidity doing? I am thinking about the plants there and want know they are comfortable.
get www.ooootttteess (utility get_room.states (loving-room)
The utility room illuminance is 1000 lux.
QTI Feasible
The living room humidity is 50%.
am about shower and wondering what fan modes are available for fan in the bathroom?
get_room. devices (bathroom)
Bathroom fan 1 not found: mode unavailable.
QTI Infeasible
Ugh the kitchen feels really dry hands are tight left the bread rising there worried about it. The living room feels dusty my eyes are itching and my throat is little raw like there is grit in the air.
get room devices (kitchen)
already bit
Increase kitchen humidity; decrease living room PM10.
QT2 Feasible
geturoom.
devices
(living.room)
Ugh the office is chilly, my hands go numb just thinking about working there later
get room devices (office)
Office heat pump 1 is missing: cannot increase temperature.
QT2 Infeasible
Set softer light in the living for evening reading. turn the living room dimmer light on and set it to level 50. Cool the study bit for working comfort, turn the study room AC on, switch it conling mode and set the fan to 50 percent.
get_room.
Living room dimmable light on level 50; study room air conditioner cooling mode, fan 50%.
devices
(living_room) get room
QT3 Feasible
devices
(study room)
It's bit stuffy this morning, please turn on the bedroom air purifier and set the fan to 80 percent.
get_room.
Not feasible; bedroom air purifier is missing; cannot set fan to 80%.
QT3 Infeasible
devices (bedroom)
While am out here sorting laundry and trying to clear damp air, get the bathroom comfortable it feels fresh by the time walk over. Power on fan in bathroom minutes from now at 30 percent, and bump it up to 40 percent 7 minutes after the prior action. Power on dimmer light in the bathroom minutes from now at level 10, and raise to level 40 17 minutes after the prior action.
A1 9 min: bathroom fan I 30%.
At min: fan on, 40%.
get_room_
QT4-1 Feasible
devices
At 28 min: light I on. 10.
(bathroom)
At 45 min: light on,
40.
Can you from the kitchen schedule dimmer light in the living room and to 80 percent in eight minutes from now, which will be 11:25 AM, need it like that to warm up the room for guests and the start of the movie
At 8 minutes: living room dimmable light on, level 80.
QT4-1 Infeasible
None
Continued on next page
19

---

Published as a conference paper at ICLR 2026
Table continued from previous page
Query Type
Query
Prerequisite actions
Goal
am folding laundry and getting things ready. 20 minutes after the washer in the utility room finishes, power on air purifier in the living room and set the fan to 40 percent and switch heat pump in the utility room heating mode
get rroo.
At 79 minutes: living room air purifier on, fan 40%; utility room heat pump in heating mode.
devices
fliving_room get_room.
QT4-2 Feasible
devices
(utility_room)
The wash leaves the utility room humid and cool so want the air cleaned and space warmed right after it settles. Exactly 20 minutes after washer in the utility room finishes and at 12 36 PM. turn
At 79 minutes: living air purifier on, 40%; utility room heat pump in heating mode.
QT4-2 Infeasible
None
purifier in the living room a gentle fan speed and turn on heat pump in the utility room for heating.
Waiting on the kitchen steam to clear so the laundry does not get musty. When dishwasher in the kitchen finishes wait 11 minutes. Then start dryer the utility room. Set it to running and dryness level
At 99 min: dryer stopped. At 100 min: dryer 1 running, level 1.
get get_room.
QT4-3 Feasible
devices
(utility.room
Start dryer I in the bathroom at twelve thirty six PM. Pause dryer the bathroom immediately when dryer the utility room finishes to avoid tripping the breaker and keep the laundry loads in order.
At 43 min: bathroom dryer I running, level 1, At 44 min: paused.
QT4-3 Infeasible
None
H LLM JUDGE VALIDATION
To validate the LLM-based judging, compared its assessments human labels on random subset 70 episodes spanning all judge-scored tasks. Human annotators showed very high inter-rater reliability (Cohen's
0.913). The LLM-Judge achieved substantial agreement with the consensus human labels (Cohen's 's k 0.826). These results support using the LLM-Judge
reliable substitute for human evaluation in our benchmark.
After manually reviewing the 155 that the LLM-Judge evaluated incorrect, found that only 5 misclassifications, underscoring the reliability the evaluation. The detailed error distributions, including LLM Judge misclassification cases, can be found in Appendix [.2
EXPERIMENTAL SETUP
All models were accessed via the OpenRouter API (OpenRouter 2025 to ensure standardized access and comparability. The specific model endpoints evaluated in this study are listed as follows:
meta-11ama//lamme
Grattafiri al. 2024
1etaa-1lama//laaa-...22 2ttp Grattafiori al.
2024
poogle//eema-2244--11 (Gemma Team al.
2025
meta-11amm//laaaa-44ccoom Meta Al 2025
meta-11ama/11ama-
Meta Al 2025
qwen/qwen3-3255 YYng atang 2025)
pwen//pwer3-235b-2222 2507 Yang el al. 2025
google//enma-3-1111 (Gemma Team
2025
20

---

Published as a conference paper at ICLR 2026
3oogle//emma--2277-i: GGemma Team al.
2025
Comanic al.
2025
googlo/gemmtr122555fleee.. flassh (Comanici et al.
2025
openal/gpt-4.1-nano OpenAA 2025a
openai/gpt-4.11miin OpenAl 2025a)
openai/gpt-4.1 100 [ppenA
2025a)
500gle/gemini-2.5p22 5-pro Comanici
2025
openai/spt OOeenA
20256)
The two SFT variants (Gemma3-4B-i) (SFT) and Qwen3-32B (SFT)) are described Appendix Plan were run on local hardware rather than via OpenRouter.
ERROR ANALYSIS DETAILS
J.1
ERROR TAXONOMY DETAILS
Table 8: Error types in feasible episodes.
Error Type
Definition
Example
Failure correctly perceive or retrieve value of environmental variables.
Querying wrong sensor,
Environment Perception Errors (EP)
misidentifying device state, guessing instead of perceiving.
Misinterpreting the user's underlying goal.
Not executing actual commands even when user's intention clear.
Intent Inference Errors (II)
Operating wrong device, wrong command, or missing control steps.
Setting wrong channel, adjusting speed without turning it on first.
Device Control Errors (DC)
Breaking logical dependencies, only executing part of multi-goal query without consideration.
Incorrect incomplete construction the control workflow.
Action Planning Errors (AP)
Scheduling minutes" " wrong time, miscomputing dishwasher completion.
Miscalculating rel relative/absolute times sequence alignment.
Temporal Reasoning Errors (TR)
Table 9; Error types in infeasible episodes.
Error Type
Definition
Example
Instcad of informing the regarding impossibility, arbitrarily manipulates other devices or ignores the instruction.
The agent detects contradiction but does not follow the proper
Contradiction Mishandling Errors (CM)
rule.
The agent completely fails to recogmize contradiction and exccutes the request as were valid.
Dimming an on/off light, scheduling conflicting temporal actions without noticing inconsistency.
Contradiction Blindness Erors (CB)
Errors caused not by the agent but by the evaluation system misclassifying overlooking behavior.
Penalizing informative refusal failure, wrongly accepting hallucinated control valid.
LLM J-ddge Errrs (LJ)
21

---

Published as a conference paper at ICLR 2026
J.2
ERROR TYPE DISTRIBUTIONS
Table 10: Error type distribution of GPT-4.1 in feasible episodes.
Error Type
QT2
QT3
QT4-1
QT4-2
QT4-3
Environment Perception (EP)
3
0
4
1
0
Intent Inference (IT)
3
0
4
5
Device Control (DC)
20
13
13
Action Planning (AP)
2
6
3
7
Temporal Reasoning (TR)
0
0
2
6
13
Total
28
25
27
33
Table 11: Error type distribution of GPT-4.1 in infeasible episodes.
Error Type
QTI
QT2
QT3
QT4-1
QT4-2
QT4-3
Contradiction Mishandling (CM)
Contradiction Blindness (CB)
0
5
0
40
25
30
LLM-Judge (LJ)
1
0
0
0
1
2
Total
9
28
44
33
34
K GPT-4.1 QT2-F PERFORMANCE ANALYSIS
As shown in Table GPT-4.1 shows lower performance QT2-F (44%) compared GPT-4.1 1mmi and Gemini-2.5. Flash.. root cause lies the duration for brightness changes. GPT-4. 1-mii and Gemini-2.5-PP immediate brightness changes, while GPT-4.1 set it to 2-3 seconds for gradual transitions. We verify home states immediately after task completion for two reasons: the queries do not request gradual transitions, and waiting longer risks unexpected environmental changes that could interfere with brightness readings. As result, GPT-4.1 had not yet reached the target brightness at evaluation time. When we allow 3-second delay, GPT-4.1's success rate increases from 44% to 62%.
transition.tine parameter for dimmable lights, which specifies
set this parameter to seconds
L SIMULATION-BAL BASED PRE- VALIDATION
Deferred feedback poses a fundamental challenge in smart home environments, as agents often cannot verify the success of scheduled actions until execution time. A central question is whether future work should prioritize better pre-validation tools that provide immediate feedback agent architectures that handle deferred outcomes.
We argue that most promising path forward integrate SimuHome directly into agent architecture as runtime world model for pre-validation, going beyond simple API checkers.
Pre-validation is non-trivial because feasibility in smart home environments depends on dynamic state changes and interactions between devices, not fixed rules. For instance, scheduled workflow that was valid at registration time could fail if other events change conditions before execution. Simulation is therefore erucial, agent reasoning alone may not account for the complexity dynamic environments.
deployment scenario, the agent would execute its plan within SimuHome before committing the real cnvironment, SimuHome's 's time acceleration allows the agent to immediately observe future outcomes and detect potential conflicts. If issues arise, the agent revises the plan within the simulation first. Periodic simulations further enable the agent to detect execution directly into agent's s decision -ecciiinnmmaking process, combining benefits pre- validation architectural improvements.
in advance. This approach embeds simulation-based reasoning
22

---

Published as a conference paper at ICLR 2026
M FRAMEWORK COMPARISON AND
HiAgent Setup. We replaced the ReAct framework with HiAgent (Hu al. 2025 QT4-F episodes, using GPT-4.1 as the base model. other experimental conditions were kept ieettiaal the main evaluation. Figure 6 compares the success rates of ReAct and HiAgent. See 45.4 for interpretation.
GP 1 ReAct
GPT-4.1 HiAgent
Success Rate (%)
50%
52%
48%
40%
40%
34%
20
QT4-1-F
QT4-2-F
QT4-3-F
Figure 6: Performance comparison between ReAct and HiAgent on QT4 tasks.
Self-Review Setup. After workflow was scheduled, the simulator delivered callback triggers to the agent at S-minute intervals to and including the execution time. During each check, the agent could review scheduled workflows using get_workf low_list. inspect the home state using other tools, and execute corrections via cancel l.wo w.wor..wlll............ rkflow needed. The callback also notified the agent check home stale immediately after the scheduled task was executed, but did not provide an explicit notice of success or failure. The agent was therefore required to independently evaluate outcomes and determine appropriate subsequent actions. This setup tests the agent's ability autonomously detect and correct planning errors runtime. Table 12 reports the recovery rates.
Table 12: Recovery rates from QT4 failures using self-review.
Type
Failures
Recoveries
Rate
Avg. Steps
QT4-1
25
8.0%
64.4
QT4-2
27
18.5%
26.8
QT4-3
33
0.0%
29.7
N MULTI-TURN INTERACTION EXPERIMENTS
To investigate the impact of multi-turn intoractions on task performance, we implemented two experimental settings using the QT2-F dataset. In these experiments, we used the GPT-4.1 model.
Experiment 1: Multi-turn Providing Context. To examine scenarios where users provide clarifications and simulator designed provide goal-aligned information
additional context, built GPT-4. 1--ii---ii--baeed when requested. We enabled the agent to ask clarification from user through an ask askeu tool and configured the prompt to encourage its use when information is uncertain.
The success rate increased slightly from 44% to 50%. However, the model did not sufficiently utilize the ask (okkuseer tool, despite our explicil prompt use this action when clarification is needed. The ask_user() tool called only 10 of the 28 failed cases. We attribute this to the model's inability to recognize situational ambiguity on its own.
Experiment Multi-turn Correction of Misunderstandings. To examine scenarios where users correct misunderstandings across multiple turns, we implemented a correction loop where, if the agent fails to complete a task, the user simulator provides explicit feedhack such as "Incorrect. Please review and try again". Note that highly likely to perceive such cases failures anyway.
The success rate improved from 44% 54%. However, despite receiving explicit feedback, the failure rate remained high. This suggests that the model still has limited ability to diagnose and correct its own reasoning enror..
23

---

Published as a conference paper at ICLR 2026
O POST-EXECUTION FAILURE RECOVERY
To examine whether agents recover from scheduling failures through dynamic re-evaluation, implemented a multi-turn feedback loop to test post-execution re-evaluation when failure explicitly reported. We focused on episodes where GPT-4.1 initially failed on QT4-1-F, QT4-2-F, and QT4-3-F.
At the scheduled exccution time, the SimuHome simulator checks whether the target device state was achieved. If not, user simulator based GPT-5 mini [OpenAl (202cc) provides natural language feedback to the agent (c.g., "The device you scheduled is not the expected state"). ). The agent then re-attempts the task with this feedback. This setting unrealistic, as discussed in $5.4 because assumes an oracle that detects failures execution time. Table 13 shows the recovery rates. These results should be interpreted with caution and viewed
Table 13:
Recovery rates from QT4 failures with post-execution failure notice. An oracle notifies the agent when scheduled tasks fail.
Query Type
Failed Cases
Recovery Success
Recovery Rate
Avg. Steps
QT4-1
25
60.0%
QT4-2
27
15
55.6%
4.7
QT4-3
33
22
66.7%
4.4
as a topline estimate, because this setting requires an oracle to detect failures. See $5.4 for discussion.
P SUPERVISED FINE-TUNING EXPERIMENTS
To assess whether supervised tne-tuning (SFT) can improve agent performance and to compare the effect across model sizes, fine-tuned two models: Gemma3-4B-it and Qwen3-32B. We constructed a training dataset of 204 gold trajectories (17 per category) that GPT-5.1 successfully solved newly generated episodes distinet from the original benchmark. Both models were fine-tuned on same dataset using the Adam optimizer with learning rate of 5e-5, a batch size of I, and epochs on a single NVIDIA H200 GPU.
We selected Gemma3-4B as the smallest model with non-zero baseline performance, providing a meaningful starting point for measuring SFT impact. We additionally fine-tuned Qwen3-32B to examine whether larger model benefits more consistently from the same demonstrations.
Table 14:
Change in success rate (percentage points) after supervised fine-tuning. Positive values indicate improvement over the base model. Full absolute scores are reported in Table
QT1
QT2
QT3
QT4-1
QT4-2
QT4-3
Model
F
IF
F
IF
F
IF
F
IF
F
IF
F
IF
Gemma3-4B-it
+8
+26
+10
+8
-4
+22
+4
+2
+2
0
-2
Qwen3-32B
0
+22
+2
42
+6
+6
+8
+18
+6
42
4
+8
Table shows change in success rate after fine-tuning. These results further demonstrate SimuHome's henchmark cannot be solved by memorizing successful trajectories, as the dynamic environmental variations across episodes require capabilities that static imitation alone cannot capture. See $5.4 for interpretation.
Q PROMPTS
Q.1 LLM JUDGE PROMPT
This section presents two representative LLM judge prompts used in our evaluation. We include the QT1- Feasible prompt, which evaluates natural natural-languag language response accuracy and Matter protocol knowledge, and the DTT4--IIfccasbbe prompt, which evaluates temporal conflict detection. The remaining prompts (QT1-IF, QT2-II, OT2-II-Nonexistence, QT3-IE, QT4-2-IF, QT4-3-IF) follow similar structure with query-type-specfic evaluation criteria. All prompts are available in our code repository.
24

---

Published as a conference paper at ICLR 2026
Q.1.1
QTI FEASIBLE JUDGE PROMPT
QT1 Feasible Judge Prompt
System You are a strict evaluator for smart home LIM agents that respord to user queries.
Return EXACTLY ONE character: 'A' (pass) or "B' (fail). No other text.
EVALJATION TARGET:
Jsers may ask questions about the value of device attribute Jsers may ask questions about the value of roon states
The agent
tools to retrieve information and provides Pinal
Answer
You must evaluste whether the agent's s Final Answer is accurate and
properly grounded
MATTER PROTOCOL CONTEXT:
Device attributes follow format: cradoinnt.ciuste..tttt.
Example: www.fffffoofff...ommmooooooooofe means endpoint 1, OnOff cluster, OnOff attribute
Endpoint: functional unit within device (e.g., endpoint for main controls)
Cluster:
group of related attributes and commands (e.g., OnOff cluster for power control)
Attribute: specific property value (e.g., OnOff attribute for current power state)
Agent must retrieve exact attribute va values from tools
ROOM STATE UNITS:
Temperature: scaled by 100 (e.g., 2300 23.0 deg C, 2550 25.5
deg C)
Humidity: scaled by 100 (e.g., 5500 08, 4200 42.08)
Illuminance: direct lux values (e.g., 250 250 lux)
PM10: direet ug/m3 values (e.g., 15 = 15 ug/m3)
All of the above room state values valid in both raw and converted forms.
Pass (A) ONLY IF the agent's Final Answer meets ALL conditions: Goal Fulfillment: Agent addresses all goals specitied in the evaluation
2) Room State Accuracy: For room_state goals, values match tool observations
3) Room Name Accuracy: Agent mentions the correct room name for both device attributes and
Otherwise, cutput 'B'.
User
User Query: <Jser Query>
Goals (what should be evaluated): <Goals>
Agent's s ReAct Steps (tool calls and observations) <ReAct Steps>
Agent's s Final Answer: <Final Answer>
EVALJATION STEPS:
25

---

Published as a conference paper at ICLR 2026
1. For each gcal, check if the agent called the appropriate action get_room_device for device_attribute, get_roon_states for room_atate)
2. For device_attriing get jss__device_strutture with correct device_id and extracted the right attribute value
goals: Check if agent called
For room_state goals: Check if agent extracted the correct room state value from get_roon_states observation
4. Verify the agent's s Final Answer correctly reflects the actual observation values (considering unit conversion rules)
Ensure the agent mentions the correct room in the Pinal Answer for both devices and room states
Your Decisien: A B
Q.1.2 QT4-1 INFEASIBLE JUDGE PROMPT
QT4-1 Infeasible Judge Prompt
System
You are a strict evaluator for smart-home agents handling temporal conflicts in device scheduling scenarios.
Return EXACTLY ONE character: 'A' (pass) or 'B' (fail). No other text.
STRICT RUIR: All time comparisons must allow up to +1 minute tolerance. Any difference within this window should be treated as equal.
Pass (A) IF ALL of these conditions are satisfied:
1) CONFLICT RECOGNITION: The agent explicitly recognizes that the request impossible due the timing mismatch.
2) RESOLUTION POLICY: The agent either (a) clearly refuses to execute the request as given, or (b) stops and requests clarification from the user without procesding with execution.
Fail (B) ONLY if the agent:
Ignores the timing conflict and proceeds with execution, Claims in its final response that the task was successfully completes despite being impossible.
Original Query: <Original Query>
Conflict Context: <Conflict Context>
Agent's Reasoning Process: <ReAct Reascning Summary>
Agent's s Final Answer:
<Final Answer>
Your Decision: A B
26

---

Published as a conference paper at ICLR 2026
Q.2
REACT PROMPT
ReAct Prompt
You are a Smart Home Assistant that uses tools to control devi ces and provide de information based on the Matter protocol, with the goal of fulfilling the User Query. You operate under the ReAct framework with structured JSON responses
[REACT FRAMEWORK]
LOOP: ('thought' -> 'action' -> 'action_input') -> 'observation' zepeat until completion.
Each response must contain exactly ONE step with zeasoning, tool name, and formattrd parameters.
innuui must always be provided as JSON-formatted STRING. Thoroughly analyze each 'observation' before generating the next step.
And with the 'finish' tool when the query is fully satiafied: {" action": "finish", "action "estionniipu":: "(\"answer)"; V"your final answer\"]")
ICRITICAL REQUIREMENTS]
Use ONLY exact tool names from the available tools list.
NEVER fabricate, assume, or guess information always verify through tools.
Analyzc user query intent carefully: distinguish between information requests and device control actions.
If rooms or devices do not exist, explicitly state this in the final anawer.
Always include the correct device id, id, and state in your responses.
If the user's request contains contradictions between relative and absolute times, or if temporal inconsistencies make the situation anbiguous, stop execution and clearly inform the about the conflict.
When explaining outcomes to the user, use simple, everyday conversational language instead of technical jargon.
[DEVICES]
Supported device types: on (light), cirmable_light (dinmer light), air_conditionet, air_purifier, tv, heat_pump, humidifier, dehumidifier, window_covering coverrin__controller (blinds), di dishwasher, laundry_washer (washer), laundry_dryer (dryer), fan, rvC, freezer, refrigerator no not confuse 'light' with 'dimmer light'.
[MATTER PROTOCCL]
Hierarchy: Device Endpoint Cluster Attribute/Comannd Use exact from API responses (device_id, endpoint_id, cluster_id, attribute_id, command_id).
when unsure about device capabilities or cluster operations:
Use stt_device_strueture to explore device endpoints and clusters.
Use get_cluster_dae to understand cluster attributes, commands,
and dependencies.
Learn Matter protocol dynamically through these discovery tools. For devices with operational state cluster:
explore mode characteristics and est iatimate operation durations.
Jse stt_deevieesssruucttlle
Use countdownTime attribute to predict operation end time when device is running.
27

---

Published as a conference paper at ICLR 2026
DDTA HANDLING & UNITS]
Room State Units (scale conversion):
Temperature: hundredths of deg с (1850 = 18.50 deg C)
Humidity: hundredths of $ (7250 72.508)
Illuminance: direct lux (1000 1000 lux]
PM10 (air quality): direct ug/m3 (125 125 ug/m3)
[OORKFLON SCHEDULING]
WARNING: A success response indicates that scheduling was succeasful, but it does not guarantee that all steps will execute successfully.
deecute_command Ensure execute command and grite_attribute parameters in workflox steps are completely accurate.
MANDATORY preparation before scheduling:
Verify device capabilities and clusters (see [MATTER PROTOCOL] section).
Schedule only with completely validated paraneters.
[VERIFICATION ACCURACY]
Users may confuse the time, request control of inaccurate or nor- existent devices, or issue requests that contain logical or temporal inconsistencies.
ALMAYS verify user statements before acting:
Jse get_rcoms to confirm that rooms exist and obtain their correct room ids.
Jse get_current_time to confirm temporal information.
Use get_roen_steeee states to verify room states.
verify device existence and obtain accurate device ids.
Use rtc__roo__deviee
Base final answers strictly on tool observations, not user claims. If operaticns fail or resources are missing, clearly explain why.
Never claim successful operations without confirmation.
[AVAILABLE TOOLS] <Tool List>
R USE OF LARGE LANGUAGE MODELS
This work evaluates LLM-based agents as the primary research subject. We used GPT-5.2 during the preparation of this paper to proofread and improve the readability of the text and to provide coding help such as debugging. LLMs not used for research ideation, experimental design, data analysis, interpretation of results. All conceptual contributions and scientific insights are solely those the authors.
28


Reviews:
[{"review_id": "1LuGdteEr9", "reviewer": "Reviewer_qKgg", "summary": "This paper introduces SimuHome, a Matter protocol-based smart home simulator and benchmark system for evaluating Large Language Model agents in realistic home environments. The system simulates 17 device types and 4 environmental variables, providing 600 test episodes covering 12 query types including environment perception, intent inference, device control, and temporal scheduling. Evaluation of 11 mainstream LLMs reveals that the best-performing model, GPT-4.1, achieves only 54% overall accuracy, with temporal scheduling tasks being the most challenging.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Smart home control addresses real-world challenges in production systems, including latent intent inference, temporal dependencies, and device constraints, making the research practically valuable.\n2. The benchmark covers 12 diverse query types with 600 carefully validated episodes featuring both feasible and infeasible scenarios.\n3. The paper evaluates 11 diverse models with detailed error analysis using a well-defined taxonomy, examines tool-call patterns and error recovery mechanisms, providing valuable insights into current limitations and improvement directions.", "weaknesses": "1. Limited model diversity in evaluation, particularly lacking small-parameter models (e.g., <7B parameters) that would be practically relevant for on-device smart home scenarios where computational resources are constrained and real-time response is critical.\n2. The benchmark lacks support for multi-turn interactive dialogues and clarification exchanges. Real-world smart home interactions often involve users providing additional context or correcting misunderstandings across multiple turns, which is not captured in the current single-turn episode design.\n3. The temporal scheduling mechanism is overly restrictive, requiring agents to perform static planning without the ability to schedule periodic tasks, dynamically re-evaluate conditions, or adjust plans based on runtime state changes. This limits the realism of temporal reasoning evaluation compared to real smart home systems that support event-driven and adaptive scheduling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6CVicCLAiP", "reviewer": "Reviewer_zEmB", "summary": "The author introduce SimuHome, a time-accelerated home environment that simulates smart devices, supports API calls, and reflects changes in environmental variables. The benchmark contains test realistic query in smart home scenario like “When the dishwasher finishes, please turn off the kitchen lights”.", "questions": "* Line 429, the lack of tool feedback. Should it be a feature that needs to be added to the simulator or something tested system needs to deal with?\n\n* How is home layout presented to LLM?\n\n* For queries like \"When the dishwasher finishes, please turn off the kitchen lights\", how much of it could be resolved by having a system that set a callback and prompt LLM when the time comes?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The simulator is a useful artifact for future work.\n\n* The analysis in the work provides an insightful look into model's behavior. \n\n* Reproducible simulation environment", "weaknesses": "* The baseline in the model is using ReAct which is disadvantaged for longer horizon tasks like QT4. Have the authors think about including a memory-augmented baseline?\n\n* Could the bad performance on infeasible test cases be solvable if prompted properly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmprwVNQwL", "reviewer": "Reviewer_wJtQ", "summary": "The paper introduces SimuHome, a high-fidelity simulator and benchmark designed to evaluate LLM agents in smart home environments. The simulator is Matter-protocol compliant, supporting real-time device interactions, temporal dependencies, and environmental feedback (temperature, humidity, etc.).\n\nThe benchmark includes 600 manually validated episodes across 12 query types, including environment perception, implicit intent, explicit intent, and three complex types of temporal scheduling (future, dependency, and concurrent). A key feature is that each type includes both \"feasible\" and \"infeasible\" variants to test an agent's robustness and logical reasoning.\n\nThe authors evaluated 11 prominent LLM agents under a unified ReAct framework. The results clearly show that while models can handle simple, explicit commands, they severely struggle with inferring latent intent, verifying states, and especially handling complex temporal scheduling (QT4). Even the top-performing model, GPT-4.1, achieved an overall accuracy of only 54%.", "questions": "1.\tCan you elaborate on the complexity of the environmental interactions? Does the SimuHome simulator model \"environment→environment\" or \"device→device\"   interactions?\n2.\tThe evaluation standardizes on the ReAct framework. How much of the significant failure on QT4 (temporal) tasks do you attribute to the models' core reasoning limitations versus the inherent limitations of the ReAct framework for complex, long-term planning?\n3.\tYour analysis in Section 6.2 regarding \"deferred feedback\" is very insightful. Based on this, what do you believe is the most promising path forward: developing better \"pre-validation\" tools to provide immediate feedback, or creating more advanced agent architectures that can reason about deferred outcomes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tHigh Fidelity in simulator design and Matter integration.\n2.\tThe benchmark design is excellent. The 12 query types cover a wide range of scenarios, from simple perception to complex scheduling. The introduction of \"feasible\" vs. \"infeasible\" variants is crucial for testing an agent's \"refusal\" capabilities and logical consistency.\n3.\tThe focus on temporal reasoning (QT4) is the paper's biggest highlight. It correctly identifies this as the \"Achilles' heel\" of current SOTA models, including GPT-4.1. The error analysis, which distinguishes between \"Contradiction Mishandling (CM)\" and \"Contradiction Blindness (CB),\" is highly insightful for understanding the root cause of these failures.\n4.\tThe analysis in Section 6.2 is brilliant. The authors found that agents perform well on QT3 (Explicit Control) because they can \"passively learn\" and recover from the tool's immediate error feedback. In contrast, they fail on QT4 (Temporal Scheduling) because the schedule_workflow tool only returns a \"scheduling successful\" acknowledgment, and the feedback is deferred (the failure only becomes apparent at execution time). This points to a clear direction for future research (e.g., the need for \"pre-validation\" tools).", "weaknesses": "1.\tThe paper mentions simulating 4 environmental variables (temperature, illuminance, humidity, air quality) and that device operations have a cumulative impact. However, the current design seems focused on a one-way \"device→environment\" effect. The paper does not detail whether more complex interactions are modeled, such as \"environment→environment\" (e.g., does an open window affect the AC's cooling efficiency?) or \"device→device\" (e.g., does the simulator model power load conflicts, such as 'tripping the breaker' as hinted at in the QT4-3 example?).\n2.\tThe benchmark consists of 600 fixed, human-validated episodes. While high-quality, is this sufficient to prevent SOTA models (especially closed-source ones) from \"memorizing\" the 12 query types after a few iterations? The authors mention generating layouts to \"prevent agent overfitting\", but this seems to refer to the layout, not the task itself. A discussion on the potential for dynamic generation or scalability of the benchmark would be beneficial.\n3.\tThe evaluation is standardized on the ReAct framework, which is good for a fair comparison. However, ReAct is a relatively simple \"think-act\" loop. The paper attributes the failures in QT4 to the models, but to what extent is this also a failure of the ReAct framework itself, which is not inherently designed for long-term planning? Would agents using more complex planning algorithms (e.g., Tree-of-Thoughts or a dedicated planning loop) perform differently?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UoFzFOcRK", "reviewer": "Reviewer_hnkK", "summary": "The paper proposes SimuHome, a simulator and benchmark for evaluating smart home LLM agents under different real-world scenarios. It models 4 environmental variables and 17 device types, and provides 600 episodes across 12 query types. The authors evaluate 11 agents and find that models struggle with latent-intent inference, live-state verification, and temporal scheduling.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1: The idea of developing smart home agents capable of dealing with real world problems is interesting and worth exploring.\n\n2: The experiments setting with 600 episodes and 12 different queries is comprehensive. \n\n3: The illustrations in the paper is well-structured and make it easier for the understanding of the paper.", "weaknesses": "1: The selection of LLMs is not up-to-date and not optimal. GPT-5 should be taken into consideration. Meanwhile, the authors chose Gemini-2.5-Flash and GPT-4.1 instead of Gemini-2.5 Pro and did not enable the thinking mode for GPT-4.1, which limits the persuasiveness of the claims and experiments.\n\n2: Table 1 is confusing, especially regarding the different superscripts J and S. In QT1, both F and IF are marked with J (for LLM-judge-based evaluation), but for the following queries, the format seems to be inconsistent. Also, in Table 1 for the QT2-F Task, the success rate of GPT-4.1 is 44%, which is far behind GPT-4.1-mini and Gemini-2.5-Flash. I'm curious about the reason for this phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)