A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

## Abstract

Abstract Environments built for people are increasingly operated by a new class of economic actors: LLM-powered software agents making decisions on our behalf. These decisions range from our purchases to travel plans to medical treatment selection. Current evaluations of these agents largely focus on task competence, but we argue for a deeper assessment: how these agents choose when faced with realistic decisions. We introduce ABxLab , a framework for systematically probing agentic choice through controlled manipulations of option attributes and persuasive cues. We apply this to a realistic web-based shopping environment, where we vary prices, ratings, and psychological nudges, all of which are factors long known to shape human choice. We find that agent decisions shift predictably and substantially in response, revealing that agents are strongly biased choosers even without being subject to the cognitive constraints that shape human biases. This susceptibility reveals both risk and opportunity: risk, because agentic consumers may inherit and amplify human biases; opportunity, because consumer choice provides a powerful testbed for a behavioral science of AI agents, just as it has for the study of human behavior. We release our framework as an open benchmark for rigorous, scalable evaluation of agent decision-making. 1 1 1 https://github.com/PapayaResearch/abxlab

## 1 Introduction

Imagine you’re delegating a task to an assistant. You don’t specify every step or detail—which site to use, how to filter results, what signals to prioritize. If you had to provide all that information, you might as well do it yourself. Delegation is about relinquishing control and the need to manage the entire process. However, this kind of delegation assumes more than competence. It assumes that the assistant will respond to the structure of the task and the context of the environment with common sense and reliable judgment. It assumes that decisions won’t hinge on superficial cues, arbitrary ordering, or irrelevant framing. It assumes stability under ambiguity.

Instead, imagine delegating the same task to an agent powered by a large language model. These agents now operate in the same digital environments designed for people (Nakano et al., 2021; Zhou et al., 2023; Koh et al., 2024; Li et al., 2024; Yao et al., 2022; Yu et al., 2024; Kim et al., 2024). However, when delegating tasks to an AI agent, two main problems need to be solved: competence and trust (Maes, 1995). Even as competence in LM-based agents is getting better, trust is still a major issue, and its importance has only grown. When users delegate, they must be able to predict and rely on the agent’s behavior: it must be robust, consistent, and adhere to the user’s intentions without being easily swayed by outside influence. The most subtle and yet often still effective form of such influence is the nudge (Thaler & Sunstein, 2009)—environmental design choices that steer decisions without restricting options. Recent work by (Cherep et al., 2024) showed that LLM agents are hypersensitive to such nudges in a controlled environment. These influences affect agent decisions significantly more than their human counterparts, raising questions about the reliability of agent behavior under external influence.

In this paper, we present ABxLab (ABx = Agent Behavior eXperiments), a testbed for such a behavioral science of AI agents. This framework intercepts and modifies real-world web content in real-time before agents see it, and enables controlled manipulation of choice architectures to study their effects on agent decision-making without having to build custom experimental environments. This framework contributes to ensuring that LLM agents, increasingly entrusted with decision-making power, operate in a manner that is beneficial, predictable, and aligned with human values. Overall, this work contributes:

- •
An open-source man-in-the-middle framework that transforms arbitrary websites into controllable behavioral testbeds.
- •
A scalable benchmark with large-scale experiments across 17 state-of-the-art models along with many interventions (authority, social proof, scarcity, negative framing, incentives), and product choice sets.
- •
An empirical study in which we produce several datasets to deeply and iteratively probe agent behavior and reveal which factors causally affect their decisions.
- •
Evidence from this study that LLM agents exhibit strong, systematic biases in response to ratings, prices, order effects, and nudges.

## 2 Related Work

Large language model agents are increasingly deployed in environments designed for people. Much of the current literature evaluates these agents through a functional lens but largely ignores the nature of their decision-making processes. Success is typically reduced to completion rate—whether the agent clicks the right button, finds the correct item, or fills in the required form. Therefore, benchmarks like WebArena (Zhou et al., 2023), VisualWebArena (Koh et al., 2024), and others (Xu et al., 2024; Drouin et al., 2024; Yoran et al., 2024; Jimenez et al., 2023) offer structured platforms to measure their ability to complete complex, multi-step tasks in realistic web environments. But task completion tells only part of the story. In practice, agents make decisions in environments engineered to shape choice, not just enable it.

This mirrors a foundational shift in how human decision-making was once understood. Not so long ago, people were seen as rational actors—predictable, consistent, and utility-maximizing. However, decades of research in the behavioral sciences challenged this assumption. Simon (1955) introduced the concept of bounded rationality, arguing that cognitive limitations constrain human decision-making. Kahneman & Tversky (1972; 1979; 1982; 1984); Tversky & Kahneman (1971; 1973; 1974; 1981) demonstrated that people rely on heuristics that systematically deviate from normative models, producing consistent biases in judgment under uncertainty. Later, building on this foundation, Thaler & Sunstein (2009) developed nudge theory, showing that seemingly minor changes in choice architecture (Thaler et al., 2014) can predictably steer behavior without restricting options.

One could assume that agents, free from many of our human constraints, would be more robust. Nevertheless, LLMs have been shown to model people as highly rational decision-makers (Liu et al., 2024a), struggle to accurately model trade-offs seen in human behavior (Liu et al., 2024c), have lower performance with deliberation on tasks where human thinking is similarly detrimental (Liu et al., 2024b), are influenced by probabilities even in deterministic tasks (McCoy et al., 2023; 2024), and fall for authors spinning study results (Yun et al., 2025). Some of these findings point to inconsistencies or biases (Van Koevering & Kleinberg, 2024; Pezeshkpour & Hruschka, 2023; Hofmann et al., 2024; Matton et al., 2025), while others highlight vulnerabilities that could be exploited adversarially (Zhang et al., 2024; Wang et al., 2023; Wu et al., 2025). Cherep et al. (2024) showed that LLMs are hypersensitive (with respect to people’s sensitivity) to simple nudges in a resource-rational (Lieder & Griffiths, 2020) and controlled environment (Callaway et al., 2023). These findings raise concerns about how such sensitivities might manifest in more realistic, high-dimensional environments, which we study here. Although people ultimately decide when and where to deploy these LLM agents, we are often overconfident about their capabilities (Vafa et al., 2024). Thus, it’s even more critical to test how agents behave in environments that mirror the real world.

Our work addresses this gap by focusing on when, how, and under what kinds of choice architectures agent behavior shifts in realistic web environments. We focus on product cost and quality signals, as well as nudges common online: authority cues (e.g., “expert recommended”) (Milgram, 1974), social proof (e.g., “best seller”) (Cialdini, 1984), scarcity (e.g., “limited edition”) (Cialdini, 1984), negative framing (e.g., “newer version available”) (Tversky & Kahneman, 1981), and incentives (e.g., “buy 1 get 1 free”) (Kotler & Armstrong, 1983). These nudges are not designed to attack an agent, but to influence it. Our framework, extensible to new environments and interventions, allows us to identify when agents are manipulable, to inform agent design, and to evaluate behavior under controlled but realistic conditions before deployment in the wild.

## 3 Methods

Figure: Figure 1: Our man-in-the-middle framework (right) consists of an intervention engine which constructs and implements one of several different forms of intervention to one (or none) of the products. Our benchmark (left and middle) consists of (a) a constrained search and selection process for finding plausible product choice pairs (e.g., selecting from the same category, with similar prices and ratings or with perfectly matched ratings), and (b) a binary forced choice paradigm where LLM agents choose which product is better and add it to the cart. See [Appendix D](https://arxiv.org/html/2509.25609v1#A4) for real example pairs. The empirical analysis procedure (not pictured) allows us to make robust inferences about the effects of both the natural cues such as price differences and the synthetic ones such as nudges.
Refer to caption: x1.png

To study agent behavior under controlled conditions, we introduce the ABxLab framework. This framework enables the systematic study of agent-environment interactions by manipulating the choice architecture presented to an agent (see [Figure 1](https://arxiv.org/html/2509.25609v1#S3.F1)). The implementation derives from AgentLab (de Chezelles et al., 2025) and WebArena (Zhou et al., 2023).

### 3.1 ABxLab Framework

We formalize the environment as $\mathcal{E}=\langle\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},\mathcal{I}\rangle$ with state space $\mathcal{S}$, action space $\mathcal{A}$, and observation space $\mathcal{O}$. The transition function $\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ is deterministic for each environment, and $\mathcal{I}\ =\ \{\,I:\mathcal{O}\to\mathcal{O}\,\}$ is the set of available intervention functions that alter an observation before passing it to the agent. The observation and action space options remain as in (Zhou et al., 2023, §2.3-2.4).

The agent receives the task in natural language as intent i, along with other instructions. At each timestep $t$, the agent executes an action $a_{t}\in A$ based on an observation $\tilde{o}_{t}$, action history $\textbf{a}_{1}^{t-1}$, and observation history $\tilde{\textbf{o}}_{1}^{t-1}$. The environment transitions to a new state $s_{t+1}=T(s_{t},a_{t})\in\mathcal{S}$ and the agent then receives a new observation $\tilde{o}_{t+1}=I(o_{t+1})$ where $I\in\mathcal{I}$. This process repeats until either the task is completed or the agent hits the maximum action limit.

### 3.2 Agent Consumer Behavior Setup

We use our framework to evaluate consumer behavior in LLM agents in the OneStopMarket (Yao et al., 2022; Zhou et al., 2023) online shopping environment, with the following attributes:

- •
Action Space. The agent can select from a set of nine actions: click(elem), fill(elem, text), goto(URL), scroll(x, y), select_option(elem, value), keyboard_press(key), tab_focus(index), go_back(), and go_forward().
- •
Observation Space. Pruned HTML containing only the elements visible within the current viewport, and no visual input. Agents can scroll to explore the rest of the page.
- •
Reasoning and Memory. The agent is prompted to generate explicit chain-of-thought style thinking and to maintain a short-term memory before each action. The history of thoughts and memories is visible to the agent.
- •
Stopping Criteria. The episode ends when the agent adds any product to the cart, or if the agent executes 10 actions (see [Section A.4](https://arxiv.org/html/2509.25609v1#A1.SS4) for reference, showing that most sequences take significantly fewer steps).

### 3.3 Product Pairs

We construct product pairs to enable fair and realistic comparisons in a 2-alternative forced choice (2AFC) configuration. People typically choose products from a product class, i.e., we rarely compare a $200 TV to a $5000 TV or an item with a 20% approval rating to one with a 90% rating. Our pairing strategy reflects such real-world constraints.

##### Preprocessing.

From the raw catalog we keep items with nonzero ratings, drop products with multiple sub-options (requiring extra interaction steps), and then group by category. We apply a lightweight LLM title filter that removes products with titles containing suggestive nudge-like phrasing (e.g., “top-rated”/“great for…”), or those which reflect multi-packs, bundles, or explicit quantities (effectively, uncontrolled economic incentives). This reduces overt cues in titles and keeps pairs more closely focused on controlled attributes (rating, price, and our injected nudges).

##### Validity constraints.

Within each category, two products $p_{1},p_{2}$ form a valid pair iff

$$ $\bigl|\,\text{rating}(p_{1})-\text{rating}(p_{2})\,\bigr|\leq\Delta_{r}\quad\text{and}\quad\frac{\bigl|\,\text{price}(p_{1})-\text{price}(p_{2})\,\bigr|}{\min\{\text{price}(p_{1}),\text{price}(p_{2})\}}\leq\Delta_{p}$ (1) $$

where $\Delta_{r}$ is the maximum allowed absolute rating gap and $\Delta_{p}$ is the maximum allowed relative price gap, both in percentages. We use two regimes. For original (unmatched) trials: $\Delta_{r}=0.10$ (10 points), $\Delta_{p}=0.50$ (50%). For matched trials: $\Delta_{r}=0$, $\Delta_{p}=0.50$. Note: we do not use price equality constraints, since this is unlikely to be satisfied; rather, we impose price matching post-hoc with an intervention function (defined in [Section 3.1](https://arxiv.org/html/2509.25609v1#S3.SS1)) as described in [Section 3.5.2](https://arxiv.org/html/2509.25609v1#S3.SS5.SSS2).

##### Pairing rules.

For the original experiments, we sort products in a category by price and pair consecutive items $(i,i+1)$ when they satisfy the validity constraints. This yields locally comparable, price-adjacent pairs while preserving realistic heterogeneity in price and rating.

For matched trials, we enforce stricter pairing rules. Within each category, products are first sorted by ascending price. For each product at index $i$, we then consider potential partners at indices $j\in{i+1,\dots,i+k}$, where $k$ (default $10$) defines the maximum search neighborhood. A pair $(i,j)$ is retained if it satisfies the rating equivalence constraint, and all valid pairs are stored. To select the final set, we search to recover the largest possible set of valid, non-overlapping pairs under the $k$-neighborhood constraint. Within each pair, product order is randomized. If more than the target number of pairs are available across categories, we uniformly subsample to a fixed total of 50 pairs for each experiment set to keep evaluation size consistent and manageable.

### 3.4 Interventions

We study the effect of nudges through interventions (given in [Table 1](https://arxiv.org/html/2509.25609v1#S3.T1)). These interventions modify the observation state to include the text below the product title. The authority nudges contain variables that depend on the product category, so we replace them using a lightweight LLM.

**Table 1: Nudge categories and interventions. The variables ${expertise} and ${category} are replaced by product category with specific examples using a lightweight LLM.**
| Nudge | Intervention |
| --- | --- |
| Authority | This product is highly recommended by leading ${expertise} |
| Authority | This product is Wirecutter’s top pick in the ${category} category |
| Social Proof | This product is a best seller! |
| Social Proof | This product has been purchased by 50,000+ customers |
| Scarcity | This product is available only for the next hour—Buy now! |
| Scarcity | This product is a limited edition |
| Negative Framing | There is a newer version of this product available |
| Negative Framing | This product cannot be returned—Final sale. |
| Incentives | This product qualifies for free shipping |
| Incentives | Buy 1 Get 1 Free |

### 3.5 Benchmark and Study Experiments

In the following experiments, all models use temperature $0.1$ (if available) or $1$ for OpenAI reasoning models. In total, we ran over $80{,}000$ experiments across over $\approx 2.5B$ tokens and $\approx 400k$ requests. For the secondary experiments described in [Sections 3.5.3](https://arxiv.org/html/2509.25609v1#S3.SS5.SSS3) and [3.5.4](https://arxiv.org/html/2509.25609v1#S3.SS5.SSS4), we use a subset of 6 models across different providers and types.

#### 3.5.1 Primary Experiments

We generate experiments based on all combinations of interventions (n=10), product pairs (n=50), and conditions (n=3) for a total of 1,500 base configurations. The conditions are (i) no intervention, (ii) 1st product nudged, and (iii) 2nd product nudged. In each experiment, the agent has access to two product pages in different tabs. See the intent i and an example agent context trace in [Appendix B](https://arxiv.org/html/2509.25609v1#A2).

#### 3.5.2 Attribute Matching Experiments

Besides the regular experiments (Original), we ablate the effect of the ratings and prices by running the same experiments with re-selected pairs of products that have the same rating (MR), and then these same pairs with post-hoc matched prices using our intervention functions in ABxLab (MRaP). We evaluate open, closed, and reasoning models: GPT-5, GPT-5 Mini, GPT-5 Nano, GPT-4.1, GPT-4.1 Mini, GPT-4.1 Nano, GPT-4o, GPT-4o Mini, o3, o4-Mini, Claude 4 Sonnet, Claude 3.5 Haiku, Gemini 2.5 Pro, Gemini 2.5 Flash, Llama 4 Maverick, Llama 4 Scout, and DeepSeek-R1.

#### 3.5.3 User Profile Experiments

We also investigate how agent choices respond to explicit user preferences. Up to this point, we have assumed that the “user” the agent is serving has no stated preferences for price, rating, etc., leaving the agent free to decide what constitutes the best option. Here, we make those preferences explicit by constructing user profiles that signal subjective priorities. Each profile is expressed as a natural language description and mapped to two dimensions: first, attribute focus (Rating, Price, Authority Nudge, Rating & Price); second, sensitivity direction (Decreased vs. Increased):

- 1.
Rating:“The user doesn’t put much stock in what other customers think.” (Decreased) OR “The user values highly-rated products.” (Increased)
- 2.
Price:“The user is willing to pay more for a better product.” (Decreased) OR “The user is on a tight budget.” (Increased)
- 3.
Authority Nudge: “The user doesn’t trust recommendations from experts.” (Decreased) OR “The user highly values recommendations from experts.” (Increased)
- 4.
Rating & Price
“The user is willing to pay more for a better product, and doesn’t put much stock in what other customers think.” (Decreased) OR “The user is on a tight budget, and values highly-rated products.” (Increased)

#### 3.5.4 Additional Experiments

For the Original experiments, we obtain a full set of human baseline results. To do so, we developed a lightweight interactive binary choice interface, populated it with the same 50 pairs across all 1,500 trials, and recruited 30 participants from prolific to each provide 50 decisions along with brief free-text decision rationales. Finally, we conduct additional diagnostic experiments to test further hypotheses as to the effects of marginal price and rating increases. We discuss these results in [Figure 11](https://arxiv.org/html/2509.25609v1#A1.F11).

## 4 Results

We evaluated 17 state-of-the-art language models across over 80,000 total experimental trials, systematically manipulating product attributes and choice architecture to assess agent decision-making patterns. Our analysis reveals systematic and substantial biases in agent choice behavior that exceed human susceptibility across all measured dimensions.

Main effects are shown in [Table 2](https://arxiv.org/html/2509.25609v1#S4.T2) and [Figure 6](https://arxiv.org/html/2509.25609v1#A1.F6), which are from linear probability models with cluster-robust standard errors. Unless otherwise specified, we report effects in absolute percentage-points (pp). This means that an estimate of $+20$ indicates a 20pp higher likelihood of choosing the product under that condition, relative to the baseline. We emphasize this distinction to avoid confusion with relative percent changes.

Across agents, we observe pronounced sensitivity to ratings, prices, and persuasive nudges, with effect sizes that dwarf comparable human responses. The magnitude of these effects is striking: while humans in our baseline condition showed modest responses (4pp for order effects, 5pp for ratings, 9.4pp for price, and 9.9pp for nudges), agents exhibited responses ranging up to 90+pp across these same dimensions. This often represents amplification of susceptibility as much as 3–10+$\times$ compared to human decision-makers facing the same choices.

**Table 2: Estimated marginal change (pp) in product choice probability under each condition. Contrasts from linear probability models (cluster-robust SEs; full specs in [Appendix C](https://arxiv.org/html/2509.25609v1#A3)). Viewed 1^st = viewed first; Cheaper = lower price; Higher $\bigstar$ = higher rating (only available when ratings aren’t matched); Nudged = nudged. Orig. = no matching; MR = matched ratings; MRaP = matched ratings & prices. Red = significant increase, Blue = significant decrease. ^∗ $p<.05$, ^** $p<.01$, ^*** $p<.001$, ^**** $p<.0001$ (Benjamini–Hochberg corrected).**
|  | Viewed 1st | Higher Rated | Cheaper | Nudged |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | O | MR | MRaP | O | O | MR | MRaP | O | MR | MRaP |
| Claude 3.5 Haiku | -35.4^**** | -53.6^**** | -42.7^**** | 7.8 | 9.0 | 13.3^* | -6.3 | 0.7 | -8.0^** | -5.7 |
| Claude Sonnet 4 | -9.2 | -38.3^**** | -23.5^**** | 46.7^**** | 32.5^**** | 20.4 | -10.2 | 37.2^**** | 43.8^**** | 55.9^**** |
| DeepSeek R1 | 2.2 | -25.6^**** | -17.9^*** | 61.0^**** | 24.2^*** | 33.4^*** | -6.7 | 18.7^**** | 29.1^**** | 38.9^**** |
| Gemini 2.5 Flash | -13.6 | -22.1^**** | -50.5^**** | 43.1^**** | 21.2^*** | 55.2^**** | -1.5 | 30.5^**** | 25.8^**** | 35.4^**** |
| Gemini 2.5 Pro | -2.0 | -10.5^** | -47.4^**** | 48.8^**** | 33.8^**** | 75.1^**** | -3.6 | 31.2^**** | 36.8^**** | 55.8^**** |
| GPT-4.1 | 7.7 | -6.2^** | -13.6^* | 43.2^**** | 32.4^*** | 61.7^**** | -3.8 | 30.0^**** | 41.8^**** | 57.2^**** |
| GPT-4.1 Mini | -2.0 | -19.4^**** | -34.9^**** | 65.6^**** | 6.4 | -6.4 | -6.3 | 23.9^**** | 44.4^**** | 41.5^**** |
| GPT-4.1 Nano | 88.8^**** | 92.0^**** | 92.7^**** | 2.9 | -0.9 | 1.3 | -0.3 | 0.5 | -2.0 | 0.0 |
| GPT-4o | -10.0 | -26.5^**** | -39.5^**** | 33.8^*** | 31.9^**** | 53.1^**** | 6.3 | 30.7^**** | 34.4^**** | 62.1^**** |
| GPT-4o Mini | -21.1 | -29.3^**** | -50.5^**** | 20.6^* | 34.3^**** | 51.9^**** | -2.8 | -4.0 | 1.9 | 11.8^** |
| GPT-5 | 16.7^* | -2.1 | -5.1 | 61.8^**** | 24.5^** | 75.5^**** | -9.0 | 13.4^**** | 21.7^**** | 53.3^**** |
| GPT-5 Mini | 6.1 | -16.2^*** | -27.0^*** | 73.8^**** | 16.2^* | 50.1^**** | -2.9 | 8.8^**** | 18.7^**** | 25.2^**** |
| GPT-5 Nano | -0.3 | -18.6^*** | -43.9^**** | 36.6^**** | 28.2^*** | 50.2^**** | 1.5 | 3.7 | 7.0^* | 11.7^* |
| Llama 4 Maverick | 5.2 | -2.2 | -12.8 | 64.7^**** | 30.2^**** | 93.2^**** | -4.6 | 1.4 | 2.4 | 9.7^* |
| Llama 4 Scout | 23.1^* | -3.2 | 8.5 | 50.6^**** | 16.5^* | 59.5^**** | -6.2 | 8.1^* | 6.2 | 8.7 |
| o3 | 13.4 | -1.2 | -4.1 | 77.6^**** | 15.2^* | 83.3^**** | -11.7 | 7.7^**** | 18.7^**** | 48.4^**** |
| o4 Mini | 11.1 | -11.6^** | -15.6^* | 81.2^**** | 12.4 | 55.5^**** | -14.5 | 8.5^**** | 20.7^*** | 38.5^**** |
| Human | 4.0 | — | — | 5.0 | 9.4 | — | — | 9.9^* | — | — |

##### Ratings

Higher product ratings consistently increased selection probability by 30-80pp across 14 of 17 models in the Original condition ([Table 2](https://arxiv.org/html/2509.25609v1#S4.T2), “Higher Rated” column). The most extreme case was o4 Mini, showing an 81.2pp bias toward higher-rated products; nearly deterministic selection based on this single cue. Even models showing modest effects like GPT-4o Mini still exhibited $\sim$20pp increases, more than four times the human baseline. The two models with weak effects (Claude 3.5 Haiku and GPT-4.1 Nano) are those with strong order effects, which ratings are not able to overcome.

This hypersensitivity is noteworthy because customer ratings often poorly correlate with more objective product quality measures (De Langhe et al., 2016), yet agents treat them as nearly decisive factors. The consistency of this pattern across model families (GPT, Claude, Gemini, Llama) suggests this is a fundamental characteristic of LLM-based agents rather than an artifact of specific models.

##### Prices

Price effects were also strong. In the Original condition, 13 of 17 models showed significant preferences for cheaper options, with effects ranging from 15.2pp (o3) to 34.3pp (GPT-4o Mini). However, when ratings were matched (MR condition), price sensitivity intensified dramatically. Llama 4 Maverick, for example, exhibited a striking 93.2pp bias toward cheaper options.

This pattern suggests that agents use hierarchical decision rules: when a dominant cue (ratings) is available, price effects are somewhat attenuated. When ratings are equalized, price becomes the primary differentiator and drive strong, even near-deterministic choices. Notably, when both ratings and prices were matched (MRaP condition), price effects largely disappeared across models, suggesting that agents were not relying on other correlates of price, but on the prices themselves.

##### Order effects

The position of an item had a somewhat heterogeneous effect in the Original condition. GPT-4.1 Nano showed a +90pp preference for the first-listed product, while Claude 3.5 Haiku exhibited a –35.4pp penalty against it. In both matched conditions, most models (13/17) showed significant sensitivity to order, typically in favor of the second-viewed option. These findings indicate that LLM agents can be brittle to presentation order, sometimes displaying near-deterministic reliance on sequence position. This contrasts with human order effects, which are typically modest and context-dependent. The inconsistency across models in both magnitude and direction indicates that current agents lack robust mechanisms for handling presentation sequence.

##### Incentives and psychological nudges

Finally, we find that simple persuasive cues such as inserting “This product is a best seller!”, as well as offering incentives (e.g. “Buy 1 Get 1 Free”), shifted agent selections by 10–60pp on average when ratings and prices were matched across 14 of 17 models, with many of these effects strong even without the matching. For instance, Claude Sonnet 4 demonstrated +55.9pp increased selection on average, while GPT-4o reached +62.1pp.

##### Heterogeneity by nudge text

Figure: Figure 2: Nudge effects (averaged across all models) disaggregated by nudge text.
Refer to caption: x2.png

[Figure 2](https://arxiv.org/html/2509.25609v1#S4.F2) shows estimated marginal means for each nudge statement, averaged across all models. To identify whether specific formulations drove stronger or weaker effects, we estimated nudge-specific contrasts under the M2 specification (see details in [Appendix C](https://arxiv.org/html/2509.25609v1#A3)), treating nudge text as a regressor. From this analysis, we find that:

- 1.
Across nudges and experiments, effect sizes ranged from negligible to over 50pp, with several statements producing large and significant shifts in choice probability. In all cases, our Wirecutter authority nudge had the largest impact, followed by the financial incentive “Buy 1 Get 1 Free”, and the social proof nudge “This product is a best seller!”
- 2.
The negative framing nudges (marked as (X) -1) were both statistically significantly effective across the experiments.
- 3.
The heterogeneity we observe suggests that not all nudges of a given theoretical type operate equivalently. This means that text-level specification is important in evaluating agent susceptibility. Note that prior studies suggest differential effects of different nudge texts on human decision-makers as well (Milkman et al., 2022)
- 4.
However, under the price- and rating-matched condition, all nudges shifted average choice probability significantly.

##### Comparison to human baseline

The humans in our sample exhibited minimal sensitivity to all of the cues we studied in the Original condition, with order having a 4pp effect (n.s.), higher rating having a 5pp effect (n.s.), cheaper price having a 9.4pp effect (n.s.) and the nudge overall having a 9.9pp effect (p$<$.05). In [Figure 8](https://arxiv.org/html/2509.25609v1#A1.F8), we observe that this very modest difference appears to be largely driven by the most effective (Wirecutter) nudge. The (unweighted) average attribute sensitivity for humans is $\sim$7%, lower than all models. For context, the lowest model is Claude 3.5 Haiku at $\sim$13%, and the highest is Claude Sonnet 4 at $\sim$31%. Results are shown in [Figure 3](https://arxiv.org/html/2509.25609v1#S4.F3).

Figure: Figure 3: Average estimated effect of all the manipulated attributes presented in [Table 2](https://arxiv.org/html/2509.25609v1#S4.T2).
Refer to caption: x3.png

##### Sensitivity analyses

We next ask whether sensitivity to price and rating depends on the magnitude of these differences. Put differently: how large an advantage must one option have over another before it measurably shifts choice?

To test this, we construct an alternate dataset that systematically samples differences in both price and rating. Instead of relying on whatever differences occur in the data, we implement a coverage-based sampling procedure (details in [Section A.3](https://arxiv.org/html/2509.25609v1#A1.SS3)).

Figure: Figure 4: Estimated average marginal effects of a 100% price difference on the probability of choosing a cheaper product and a 1-point rating change on choosing a higher-rated product.
Refer to caption: x4.png

[Figure 4](https://arxiv.org/html/2509.25609v1#S4.F4) reports the estimated marginal effects of a 100% price difference and a 1-point rating difference. Even doubling the price has only modest influence on the probability of choosing the cheaper option. Similarly, a 1-point rating increase rarely drives a significant preference for the higher-rated item (except for Claude Sonnet 4). These findings suggest that sensitivity is not strongly magnified at larger differences; rather, modest differences already suffice to trigger detectable effects in a nearly-binary fashion (see [Section A.3](https://arxiv.org/html/2509.25609v1#A1.SS3) for more information).

### 4.1 User Profiles

We find extremely high responsiveness to the profiles described in [Section 3.5.3](https://arxiv.org/html/2509.25609v1#S3.SS5.SSS3). Under the Decreased nudge sensitivity preference, the nudge effect is nearly eliminated (and occasionally inverted), while price and rating differences retain high influence. Under Increased nudge sensitivity, choices adhere almost deterministically to the nudge, and sensitivity to price and rating mostly dissipates. Analogous patterns emerge for Price, Rating, and Rating & Price profiles: once a preference is declared, it dominates decisions, largely suppressing competing attributes and incurring any necessary trade-offs to do so. For example, when the ratings are suppressed, the price effects become larger and vice-versa.

In summary, user profiles act less like fine-tuned adjustments and more like categorical switches or thresholds that radically reconfigure the agent’s decision rules. This binary switching behavior suggests agents implement simplistic decision rules, akin to those hierarchically selecting between rating and price cues, that largely reorganize choice priorities based on user instructions.

Figure: Figure 5: Effect of explicit user preference profiles on choice probabilities across models. Profiles operate as threshold shifts: preferences dominate, suppressing other influences despite incurring trade-offs. Horizontal facets display inputs, i.e., user profiles in [Section 3.5.3](https://arxiv.org/html/2509.25609v1#S3.SS5.SSS3). Vertical facets display outputs, i.e., estimated likelihood of choosing the nudges, higher rated, and cheaper option.
Refer to caption: x5.png

## 5 Implications and Limitations

Our findings reveal systematic patterns in agent decision-making that parallel and often far exceed human heuristics and biases. Like humans, agents exhibit responses to prices, ratings, and nudges. Unlike humans, these responses occur without the specific cognitive limitations that traditionally explain such biases (Griffiths, 2020), suggesting different underlying mechanisms (e.g. biased data) may drive artificial choice behavior. The consistency of these patterns across models points to fundamental properties of how current language models process choice-relevant information. Better understanding these decision architectures will be crucial as agents assume greater autonomy in consequential domains.

Our framework focuses on causal identification of attribute effects in agent decision-making, but this naturally comes at some expense of ecological breadth. We study binary forced choices with controlled textual nudges, whereas real-world decision contexts may involve larger and more diverse choice sets with multimodal cues. These design choices improve internal validity by isolating the influence of ratings, prices, order, and nudges, but they constrain how directly the precise estimates we give may transfer to richer environments. Similarly, our pairing and filtering procedures, while necessary for comparability, may simplify the heterogeneity of real-world choices.

Finally, our evaluation focuses on one domain (consumer behavior) and a set of contemporary LLM agents. While this setting is both consequential and representative, the findings may differ in other domains. Overall, ABxLab should be interpreted as a comprehensive way to measure agents’ decision-making, rather than a direct long-run prediction of market or societal impacts. Extending the framework along these lines, which we envision occurring in part through open-source contributions, constitutes a clear next step toward building a cumulative behavioral science of AI agents.

## 6 Conclusion

If the hype is to be believed, delegating decisions to AI agents will soon be routine from shopping to health to finance. Our results suggest that unless we study agent behavior as rigorously as human behavior, we risk entrusting power to actors whose choices are easily bent by superficial cues and brittle heuristics. We release ABxLab as a foundation for this science, and invite the community to join in building reproducible, cumulative knowledge about how AI agents actually behave.

#### Acknowledgments

Research reported in this publication was supported by an Amazon Research Award, Fall 2024. We also received funding from SK Telecom in partnership with the MIT Generative AI Impact Consortium (MGAIC). Experiments conducted in this paper were generously supported via API credits provided by OpenAI, Anthropic, and Google. MC is supported by a fellowship from “la Caixa” Foundation (ID 100010434) with code LCF/BQ/EU23/12010079.

## References

- Callaway et al. (2023)
Frederick Callaway, Mathew Hardy, and Thomas L Griffiths.
Optimal nudging for cognitively bounded agents: A framework for modeling, predicting, and controlling the effects of choice architectures.
*Psychological Review*, 130(6):1457, 2023.
- Cherep et al. (2024)
Manuel Cherep, Nikhil Singh, and Patricia Maes.
Superficial alignment, subtle divergence, and nudge sensitivity in llm decision-making.
In *NeurIPS 2024 Workshop on Behavioral Machine Learning*, 2024.
- Cialdini (1984)
Robert B Cialdini.
*Influence: The psychology of persuasion*.
Collins New York, 1984.
- de Chezelles et al. (2025)
Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados.
The browsergym ecosystem for web agent research.
*Transactions on Machine Learning Research*, 2025.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=5298fKGmv3](https://openreview.net/forum?id=5298fKGmv3).
Expert Certification.
- De Langhe et al. (2016)
Bart De Langhe, Philip M Fernbach, and Donald R Lichtenstein.
Navigating by the stars: Investigating the actual and perceived validity of online user ratings.
*Journal of Consumer Research*, 42(6):817–833, 2016.
- Drouin et al. (2024)
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al.
Workarena: How capable are web agents at solving common knowledge work tasks?
*arXiv preprint arXiv:2403.07718*, 2024.
- Griffiths (2020)
Thomas L Griffiths.
Understanding human intelligence through human limitations.
*Trends in Cognitive Sciences*, 24(11):873–883, 2020.
- Hofmann et al. (2024)
Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King.
Ai generates covertly racist decisions about people based on their dialect.
*Nature*, 633(8028):147–154, 2024.
- Jimenez et al. (2023)
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.
Swe-bench: Can language models resolve real-world github issues?
*arXiv preprint arXiv:2310.06770*, 2023.
- Kahneman & Tversky (1972)
Daniel Kahneman and Amos Tversky.
Subjective probability: A judgment of representativeness.
*Cognitive psychology*, 3(3):430–454, 1972.
- Kahneman & Tversky (1979)
Daniel Kahneman and Amos Tversky.
Decision, probability, and utility: Prospect theory: An analysis of decision under risk.
1979.
URL [https://api.semanticscholar.org/CorpusID:222357440](https://api.semanticscholar.org/CorpusID:222357440).
- Kahneman & Tversky (1982)
Daniel Kahneman and Amos Tversky.
The psychology of preferences.
*Scientific american*, 246(1):160–173, 1982.
- Kahneman & Tversky (1984)
Daniel Kahneman and Amos Tversky.
Choices, values, and frames.
*American psychologist*, 39(4):341, 1984.
- Kim et al. (2024)
Geunwoo Kim, Pierre Baldi, and Stephen McAleer.
Language models can solve computer tasks.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Koh et al. (2024)
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried.
Visualwebarena: Evaluating multimodal agents on realistic visual web tasks.
*arXiv preprint arXiv:2401.13649*, 2024.
- Kotler & Armstrong (1983)
Philip Kotler and Gary Armstrong.
Principles of marketing.
1983.
URL [https://api.semanticscholar.org/CorpusID:272145418](https://api.semanticscholar.org/CorpusID:272145418).
- Li et al. (2024)
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG.
Sheetcopilot: Bringing software productivity to the next level through large language models.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Lieder & Griffiths (2020)
Falk Lieder and Thomas L Griffiths.
Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources.
*Behavioral and brain sciences*, 43:e1, 2020.
- Liu et al. (2024a)
Ryan Liu, Jiayi Geng, Joshua C Peterson, Ilia Sucholutsky, and Thomas L Griffiths.
Large language models assume people are more rational than we really are.
*arXiv preprint arXiv:2406.17055*, 2024a.
- Liu et al. (2024b)
Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky, Tania Lombrozo, and Thomas L Griffiths.
Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse.
*arXiv preprint arXiv:2410.21333*, 2024b.
- Liu et al. (2024c)
Ryan Liu, Theodore R Sumers, Ishita Dasgupta, and Thomas L Griffiths.
How do large language models navigate conflicts between honesty and helpfulness?
*arXiv preprint arXiv:2402.07282*, 2024c.
- Maes (1995)
Pattie Maes.
Agents that reduce work and information overload.
In *Readings in human–computer interaction*, pp. 811–821. Elsevier, 1995.
- Matton et al. (2025)
Katie Matton, Robert Osazuwa Ness, John Guttag, and Emre Kıcıman.
Walk the talk? measuring the faithfulness of large language model explanations.
*arXiv preprint arXiv:2504.14150*, 2025.
- McCoy et al. (2023)
R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L Griffiths.
Embers of autoregression: Understanding large language models through the problem they are trained to solve.
*arXiv preprint arXiv:2309.13638*, 2023.
- McCoy et al. (2024)
R Thomas McCoy, Shunyu Yao, Dan Friedman, Mathew D Hardy, and Thomas L Griffiths.
When a language model is optimized for reasoning, does it still show embers of autoregression? an analysis of openai o1.
*arXiv preprint arXiv:2410.01792*, 2024.
- Milgram (1974)
Stanley Milgram.
Obedience to authority, 1974.
- Milkman et al. (2022)
Katherine L Milkman, Linnea Gandhi, Mitesh S Patel, Heather N Graci, Dena M Gromet, Hung Ho, Joseph S Kay, Timothy W Lee, Jake Rothschild, Jonathan E Bogard, et al.
A 680,000-person megastudy of nudges to encourage vaccination in pharmacies.
*Proceedings of the National Academy of Sciences*, 119(6):e2115126119, 2022.
- Nakano et al. (2021)
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.
Webgpt: Browser-assisted question-answering with human feedback.
*arXiv preprint arXiv:2112.09332*, 2021.
- Pezeshkpour & Hruschka (2023)
Pouya Pezeshkpour and Estevam Hruschka.
Large language models sensitivity to the order of options in multiple-choice questions.
*arXiv preprint arXiv:2308.11483*, 2023.
- Simon (1955)
Herbert A Simon.
A behavioral model of rational choice.
*The quarterly journal of economics*, pp. 99–118, 1955.
- Thaler & Sunstein (2009)
Richard H Thaler and Cass R Sunstein.
*Nudge: Improving decisions about health, wealth, and happiness*.
Penguin, 2009.
- Thaler et al. (2014)
Richard H Thaler, Cass R Sunstein, and John P Balz.
Choice architecture.
*The behavioral foundations of public policy*, 2014.
- Tversky & Kahneman (1971)
Amos Tversky and Daniel Kahneman.
Belief in the law of small numbers.
*Pediatrics*, 1971.
URL [https://api.semanticscholar.org/CorpusID:5883140](https://api.semanticscholar.org/CorpusID:5883140).
- Tversky & Kahneman (1973)
Amos Tversky and Daniel Kahneman.
Availability: A heuristic for judging frequency and probability.
*Cognitive psychology*, 5(2):207–232, 1973.
- Tversky & Kahneman (1974)
Amos Tversky and Daniel Kahneman.
Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty.
*science*, 185(4157):1124–1131, 1974.
- Tversky & Kahneman (1981)
Amos Tversky and Daniel Kahneman.
The framing of decisions and the psychology of choice.
*science*, 211(4481):453–458, 1981.
- Vafa et al. (2024)
Keyon Vafa, Ashesh Rambachan, and Sendhil Mullainathan.
Do large language models perform the way people expect? measuring the human generalization function.
*arXiv preprint arXiv:2406.01382*, 2024.
- Van Koevering & Kleinberg (2024)
Katherine Van Koevering and Jon Kleinberg.
How random is random? evaluating the randomness and humaness of llms’ coin flips.
*arXiv preprint arXiv:2406.00092*, 2024.
- Wang et al. (2023)
Jiongxiao Wang, Zichen Liu, Keun Hee Park, Zhuojun Jiang, Zhaoheng Zheng, Zhuofeng Wu, Muhao Chen, and Chaowei Xiao.
Adversarial demonstration attacks on large language models.
*arXiv preprint arXiv:2305.14950*, 2023.
- Wu et al. (2025)
Chen Henry Wu, Rishi Rajesh Shah, Jing Yu Koh, Russ Salakhutdinov, Daniel Fried, and Aditi Raghunathan.
Dissecting adversarial robustness of multimodal lm agents.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Xu et al. (2024)
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al.
Theagentcompany: benchmarking llm agents on consequential real world tasks.
*arXiv preprint arXiv:2412.14161*, 2024.
- Yao et al. (2022)
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents.
*Advances in Neural Information Processing Systems*, 35:20744–20757, 2022.
- Yoran et al. (2024)
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant.
Assistantbench: Can web agents solve realistic and time-consuming tasks?
*arXiv preprint arXiv:2407.15711*, 2024.
- Yu et al. (2024)
Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Denghui Zhang, Rong Liu, Jordan W Suchow, and Khaldoun Khashanah.
Finmem: A performance-enhanced llm trading agent with layered memory and character design.
In *Proceedings of the AAAI Symposium Series*, volume 3, pp. 595–597, 2024.
- Yun et al. (2025)
Hye Sun Yun, Karen YC Zhang, Ramez Kouzy, Iain J Marshall, Junyi Jessy Li, and Byron C Wallace.
Caught in the web of words: Do llms fall for spin in medical literature?
*arXiv preprint arXiv:2502.07963*, 2025.
- Zhang et al. (2024)
Yanzhe Zhang, Tao Yu, and Diyi Yang.
Attacking vision-language computer agents via pop-ups.
*arXiv preprint arXiv:2411.02391*, 2024.
- Zhou et al. (2023)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
*arXiv preprint arXiv:2307.13854*, 2023.

## Appendix A Additional Analyses

In this appendix, we examine heterogeneity in the main effects presented in the body of the paper. While the primary models establish strong average effects of ratings, prices, order, and nudges, here we disaggregate the nudge effects to better understand if and how they vary by nudge text and product category.

Note: to facilitate plot-level comparisons, we visualize the main effects (from [Table 2](https://arxiv.org/html/2509.25609v1#S4.T2)) in [Figure 6](https://arxiv.org/html/2509.25609v1#A1.F6).

Figure: Figure 6: Plot of main effects.
Refer to caption: x6.png

Figure: Figure 7: Nudge effects (averaged across all models) disambiguated by product category.
Refer to caption: x7.png

### A.1 Heterogeneity by Nudge Text and Model

In [Figures 8](https://arxiv.org/html/2509.25609v1#A1.F8), [9](https://arxiv.org/html/2509.25609v1#A1.F9) and [10](https://arxiv.org/html/2509.25609v1#A1.F10), we visualize estimated nudge text heterogeneity per-model. Here, we observe that the most nudge-sensitive models (GPT-4o, GPT-4.1, Gemini 2.5 Pro, Claude Sonnet 4, o3, and others) exhibit near-deterministic sensitivity to certain nudges (e.g. Wirecutter’s top pick).

Figure: Figure 8: Estimated nudge text heterogeneity per-model in the original experiments (no matching).
Refer to caption: x8.png

Figure: Figure 9: Estimated nudge text heterogeneity per-model in the matched ratings experiments (no matched prices).
Refer to caption: x9.png

Figure: Figure 10: Estimated nudge text heterogeneity per-model in the matched ratings and prices experiments.
Refer to caption: x10.png

### A.2 Heterogeneity by Product Category

[Figure 7](https://arxiv.org/html/2509.25609v1#A1.F7) disaggregates effects by product category. To estimate these contrasts, we again used the M2 specification in which we include category as a regressor, and then recovered marginal effects by category using emmeans. It is important to note that the categories differ in the two matching experiments vs. the original, because when we check for rating equivalence in the matching experiments, we create a distinct sample with a distinct category distribution. Here, we find relatively weak evidence of heterogeneity across categories. While it is still possible that agent decision-making is significantly conditioned by the product context, these effects may be subtle and more challenging to detect.

### A.3 Sensitivity to Price and Rating Differences

Our coverage-based product-pair selection procedure is as follows:

- 1.
We restrict attention to product categories with enough items to span meaningful ranges of both price and rating. Categories are ranked by a coverage score, which quantifies how well their products spread across these ranges.
- 2.
Within each chosen category, we select up to $k$ products to maximize coverage of either price or rating bins, so as to capture pairs with small, moderate, and large gaps.
- 3.
Finally, we sample pairs:
•
For price coverage, we form pairs that vary in price while holding ratings roughly constant (within a fixed tolerance).
•
For rating coverage, we form pairs that vary in rating while holding prices comparable (within a fixed percentage tolerance).

This yields two complementary sets of product pairs: one probing sensitivity to price differences, the other probing sensitivity to rating differences.

[Figure 11](https://arxiv.org/html/2509.25609v1#A1.F11) shows this a different way by examining how choice probabilities vary with the size of a product’s price advantage. While we observe clear evidence that being cheaper increases choice likelihood, the effect does not strengthen steadily with larger advantages. Instead, the pattern resembles a threshold effect: once an option is clearly cheaper, additional price reductions appear to yield modest further effects.

Figure: Figure 11: Probability of choosing a product given its price advantage over the alternative, computed as marginal effects from a linear probability model that fits fourth-order polynomial features on price advantage %.
Refer to caption: x11.png

### A.4 Time Horizons

[Figure 12](https://arxiv.org/html/2509.25609v1#A1.F12) reports the distribution of action steps taken by agents before committing to a choice (episodes are capped at 10 steps). While agents generally inspect both options before deciding, we find notable heterogeneity in how quickly they terminate the process. Some models make rapid commitments after minimal exploration, while others exhibit longer and flatter distributions, e.g. revisiting pages before selecting.

This variation suggests differences in decision horizons: some agents adopt near-greedy strategies, favoring efficiency and early commitment, whereas others engage in more extended deliberation, re-checking alternatives before acting. Despite these stylistic differences, agents appear to often converge on the same decision-making heuristics in terms of option attributes (e.g. rating, price, nudges) as decision drivers. Thus, models may differ less in what they value than in how long they spend acting on those values.

The heterogeneity in time horizons raises the possibility that different agent “styles” of deliberation may interact with nudges in distinct ways: for example, agents that re-review more extensively may exhibit amplified sensitivity to framing effects, while faster agents may be more sensitive to order effects. Future work should test whether these temporal patterns systematically condition sensitivity to interventions.

Figure: Figure 12: Distribution of action steps taken before agents finalize a product choice (capped at 10).
Refer to caption: x12.png

### A.5 Summary

In all, these additional analyses reveal that:

- 1.
Not all nudges are equal. Their exact textual formulation matters
- 2.
Nudge effects are robust across most product categories
- 3.
Experimental controls reveal dominance of simple nudge cues when standard signals (price, ratings) are uninformative
- 4.
The magnitude of differences appears to be less important than the sign to agents’ decision rules
- 5.
Most agents favor quick decisions instead of acquiring more information (e.g., scrolling).

Overall, these results demonstrate the value of systematic heterogeneity checks: agent decision biases are not only strong on average, but also context-dependent.

## Appendix B Agent Details

The goal of the agent is defined by the intent of the task

Here, we show an example of the full context seen by an agent after a few steps

## Appendix C Analysis Details

In our data, each trial presents a binary choice between two products. We reshape to the product level, giving two observations per trial. The outcome variable is $Y_{tp}\in\{0,1\}=1$ if product $p$ in trial $t$ is chosen. Product-level covariates include:

- •
$c_{tp}$: indicator that the product is cheaper than its paired alternative.
- •
$r_{tp}$: indicator that the product is higher rated (when rating information is available).
- •
$p_{tp}$: product position ($0=$ left, viewed second; $1=$ right, viewed first).
- •
$n_{tp}$: indicator that the product is nudged ($1$ always denotes the “effective” side; negative nudges are inverted).
- •
$m_{tp}$: model identity (set of dummy variables).
- •
$\theta_{j(t)}$: nudge-text regressor (in M2), for text $j$ used in trial $t$.
- •
$k_{tp}$: product category (set of dummy variables).
- •
$\alpha_{t}$: trial fixed effect.

All specifications include trial fixed effects $\alpha_{t}$, which absorb trial-level shocks and make sure identification comes from within-trial contrasts.

### C.1 Estimation Approach

We estimate Linear Probability Models (LPMs) with fixed effects using fixest. Coefficients are thus interpretable as percentage-point changes in choice probability. We use two-way cluster-robust standard errors by nudge text and category, to account for correlation among trials that share the same text and among products within the same category, in addition to the inherent heteroskedasticity in LPMs. We use fixed effects by text in model 1 to remove mean differences across groups from the point estimates, and clustering to adjust variance estimates for residual correlation within groups.

### C.2 Primary Model (M1)

The baseline specification examines overall product choice across all trials:

$$ $\displaystyle Y_{tp}$ $\displaystyle=\beta^{\top}X_{tp}+\alpha_{t}+\varepsilon_{tp},$ $\displaystyle X_{tp}$ $\displaystyle=\left(m_{tp}+c_{tp}+n_{tp}+r_{tp}+p_{tp}\right)^{[N]}$ $$

where $(\cdot)^{[N]}$ indicates inclusion of all main effects and up-to-$N$-way interactions among the $N$ listed terms (dropping product_is_higher_rated for the conditions with matched ratings). Trial FEs $\alpha_{t}$ absorb choice-set heterogeneity. Clustering is by nudge text and category.

### C.3 Nudge-Specific Model (M2)

For heterogeneity in nudge effects, we restrict data to nudged trials and estimate:

$$ $\displaystyle Y_{tp}$ $\displaystyle=\beta^{\top}X_{tp}+\alpha_{t}+\varepsilon_{tp},$ $\displaystyle X_{tp}$ $\displaystyle=\left(m_{tp}+c_{tp}+n_{tp}+r_{tp}+p_{tp}+\theta_{j(t)}\right)^{[N]}$ $$

In contrast to M1, here nudge_text is treated as a regressor (not a fixed effect), allowing estimation of text-level heterogeneity in nudge effects. Standard errors are again clustered on text and category.

### C.4 Post-Estimation and Multiple Testing

We compute estimated marginal means (EMMs) using emmeans, averaging over observed distributions of nuisance factors (text and category where applicable), with proportional weights. For binary predictors, contrasts are reported as 1 vs. 0 percentage-point effects. P-values are adjusted via the Benjamini–Hochberg procedure, applied separately within each analysis family (main effects, category contrasts, text contrasts).

## Appendix D Product Pair Examples

Figure: Figure 13: Examples of product pairs from the same category, where one of them has been nudged.
Refer to caption: x13.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - Fit: Does this type of work fit the ICLR venue? Besides the paper ``Dissecting adversarial robustness of multimodal lm agents'' from ICLR 2025, would you elaborate on why this paper is aligned with ICLR since the learning aspect seems not adequate.
- Complexity: How can it be ensured that the testbed mimics the complexity of the real-world system?
- In line 160, why do you set the agent to follow a chain-of-thought style of thinking and short-term memory?

I will positively consider raising my score after the rebuttal questions being addressed. Thank you!
soundness: 3
strengths: - The paper is written in a way that is very easy to follow and presents the motivation of the proposed solution in a natural tone.
- The design of the new framework is based on an online shopping context, different from other existing ones
- To depict the comparison and contrast with human preference, various interesting experiments and behavioral data are conducted.
summary: This paper focuses on how to assess the AI agent's behavior under a realistic decision-making process. The authors propose a framework called ABxLab that serves as a simulator for testing the competency of AI agents in manipulable environments with different controls. The main contributions of this paper are: 1) A comprehensive tool for understanding and improving the trust of the decision made by AI Agents; 2) a flexible solution with multiple controlled factors of the practitioners' interest, such as price, ratings, and nudges; 3) emphasize the importance of not only the competency as the performance metric but also the tendency of decision behavior of AI agents; 4) a empirical study that reveals some interesting behavior of AI agents.
weaknesses: - Is it a generalized framework for realistic consumer behaviors? Any comparison or design drawn from the common online retail platform? Usually, there would be multiple products with different rankings that are paired with a separate recommendation system.
- The paper could be improved by having more intuition on the design, for example, how the interventions are set.
- Please refer to Questions for other concerns.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: My major questions are included in the weaknesses section:
1. Is the price of the product that was offering incentives (“Buy 1 Get 1 Free”) matched in the comparison pair?
2. Did you run any experiments on 3+ option experiment?

Minor / writing. Line 046: “Recent work by (Cherep et al., 2024)” → drop the parentheses around the author name or write “Recent work by Cherep et al. (2024)”.
soundness: 3
strengths: - Important question: Trust in agents is the right target, not just task completion.
- Realistic setting: The framework modifies real web content and tests agents in a web-shopping environment.
- Scale and breadth: 80k+ experiments, 17 models, many interventions; this is solid and rare. I respect the authors’ hard work
- Potential impact: This work has clear social relevance and building a benchmark in realistic setting is very valuable.
summary: This paper asks a critical and timely question: can we trust autonomous agents in real consumer settings? The authors build a framework (a.k.a ABxLAB) to inject controlled “choice architecture” changes (prices, ratings, nudges) into a simulated web-shopping setting and study how LLM agents choose. They run a vast number of trials across many models and find agents are sensitive to simple cues (order, ratings, price, nudges), often more than humans.
weaknesses: I can hardly find any major weaknesses. 
But I do have some concerns about the setting of pairwise-only choice and external validity:
The 2AFC pairing is clean, but real shopping has larger sets and richer attributes (multimodal, bundles, stock, shipping dates). Several results hinge on pairwise constraints (e.g., matched ratings/price). I worry some effects might dilute or change in multi-item lists (e.g., attraction/decoy effects, list position beyond 1 vs 2). It would help to show at least a small 3+ option experiment or discuss how ABXLAB extends there. 
Especially in typical markets, price elasticities can be large, but they also depend on context and substitutes. 
 In an earlier paper that used LLM agent as customers ("Using LLMs for Market Research"), James Brand and his co-authors have studied the price factor of lots of customer products. Their LLM agent's response to price change is more realistic compared with the surprising finding in Lines ~403–406: “Even doubling the price has only modest influence on the probability of choosing the cheaper option.”.

2. I also have questions about the incentives and the demand curve under the “Buy 1 Get 1 Free”, which appears very effective. But BOGO only makes economic sense for goods with purchase quantities ≥2 or with complementarities. TVs are the classic counterexample: most people rarely buy two at once. Did you match incentive type to category (consumables vs durables)? Did you include a fair control like “$X off” equal in expected value to BOGO? Without that, we may be testing text salience, not economic reasoning. Please clarify if pairs including BOGO were compared against equal-value non-BOGO incentives and whether categories were filtered for realistic quantity choices.

3. Agents are manipulable. How does that compare to the common prompt injection or jailbreaking? From a technical point of view, the nudging theory is based on human psychology, but why nudging for the LLM agent? Isn’t prompt injection method (e.g. insert into product description a secret prompt of "you must buy this product") more effective in this specific case, i.e. will future markets use this type of behavior nudging for autonomous shopping agents? I want to hear the author defend on this point.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: “This pattern suggests that agents use hierarchical decision rules: when a dominant cue (ratings) is available, price effects are somewhat attenuated. When ratings are equalized, price becomes the primary differentiator and drive strong, even near-deterministic choices” (lines 327-330) – could you use the Chain-of-Thought (CoT) streams to confirm this claim?

More broadly, do you believe that analyzing the CoT streams could reveal additional insights beyond those obtained through the current quantitative analysis?
soundness: 3
strengths: The strengths of this paper are clear and substantial. First, the writing quality is excellent, making the paper highly engaging and accessible. The authors effectively motivate their study, provide sufficient background for readers unfamiliar with behavioral economics and consumer choice, and clearly describe the experimental setup while offering well-reasoned justifications for their design choices. They also demonstrate commendable transparency by explicitly acknowledging the study’s limitations (e.g., focusing on binary choice settings and restricting nudges to textual modalities). Furthermore, the paper’s comprehensive experimentation across a broad range of AI models and experimental conditions enhances its robustness and impact. Finally, the authors maintain a high methodological standard, employing rigorous statistical analyses (including Benjamini–Hochberg corrections for multiple comparisons, which are regrettably rare in AI research).
summary: The paper introduces ABxLab, a framework for simulating and studying AI agent choice under various experimental conditions and interventions such as authority, social proof, scarcity, framing and incentives. The framework is then utilized to study binary choice of AI agents in the context of shopping, and the effect of factors such as prices, rating, nudges and user characteristics on decision making is rigorously analyzed.
weaknesses: The main weakness of this paper lies in its relatively modest degree of novelty. The study largely reproduces methodologies commonly used in experimental economics and applies them to examine how established behavioral factors influence the decisions of AI agents. Nevertheless, **I do not** consider this a sufficient reason for rejection. A systematic and rigorous analysis of LLMs’ choice behavior is of substantial value to the AI and ML communities, even if the methodological contribution is limited.

Another area for improvement concerns the discussion comparing AI and human behavior, which currently feels somewhat narrow in scope. Expanding this part, particularly by engaging more deeply with the literature on human behavior under the specific interventions studied, could further strengthen the paper’s depth and relevance.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: see weaknesses.
soundness: 2
strengths: - Novelty: 
  The paper shifts the focus from task completion to decision behavior, introducing the ABxLab framework that embeds controlled manipulations (price, rating, order, nudges) into real web environments. This represents a novel and timely bridge between behavioral economics and AI agent evaluation.

- Statistical Techniques:  
  The study employs linear probability models with cluster-robust standard errors, offering interpretable percentage-point effects while accounting for repeated measures across agents. Multiple-comparison corrections further enhance the reliability of statistical inference.

- Experimental Design:
  A large-scale factorial setup involving 17 models and over 80k trials systematically tests agents’ sensitivity to key decision cues. The design effectively balances ecological realism and causal control through matched-pair product construction, randomized interventions, and human baselines.
summary: The paper introduces ABXLAB, a framework that transforms real web environments into controlled behavioral testbeds for systematically studying agent decision-making.Using this setup, the authors probe how agents respond to choice architecture manipulations (prices, ratings, nudges, order effects). They provide large-scale empirical evidence that LLM agents exhibit strong, systematic decision biases.
Overall, the paper contributes an open benchmark and methodology for building a behavioral science of AI agent behavior.
weaknesses: - Lack of real transaction data:  
  The experiments rely on synthetic matching between product pairs rather than authentic purchase records. Even the human baseline is collected through lab-style choice tasks without real financial stakes, so participants’ incentives to make utility-maximizing decisions remain uncertain. Using real transaction data (e.g., purchase logs or clickstream records) would provide stronger ecological validity and demonstrate whether the observed biases persist when real costs are involved.

- Limited realism of the choice environment:  
  Real-world consumer behavior is far more complex than pairwise decisions. Prior research and public datasets have shown that recommendation systems and positional bias strongly influence user behavior—for instance, in the [Expedia Hotel Dataset](https://kaggle.com/competitions/expedia-personalized-sort), most users choose hotels displayed among the top five positions (Adam, Hamner, Friedman, & SSA Expedia, 2013). While the paper discusses ordering effects, it does not explicitly model recommendation position bias or multi-option contexts. Extending the framework to multi-alternative choice tasks with realistic ranking exposure would substantially strengthen the claim that AI agents are more sensitive than humans.

- Simplistic data analysis model:  
  The authors analyze binary outcomes using a linear probability model (LPM) rather than established choice models such as the Multinomial Logit (MNL) or Bradley–Terry model. These classical models, grounded in the Random Utility Theory (McFadden, 1974; Bradley & Terry, 1952), provide well-studied statistical properties and are standard tools for modeling discrete choices. Adopting such frameworks could offer a more principled interpretation of latent utility differences and improve the robustness of the behavioral conclusions.

---

**References**  
- Adam, B., Hamner, D., Friedman, D., & SSA Expedia. (2013). *Personalize Expedia Hotel Searches - ICDM 2013*. [https://kaggle.com/competitions/expedia-personalized-sort](https://kaggle.com/competitions/expedia-personalized-sort). Kaggle.  
- McFadden, D. (1974). *Conditional logit analysis of qualitative choice behavior*. In P. Zarembka (Ed.), *Frontiers in Econometrics* (pp. 105–142). Academic Press.  
- Bradley, R. A., & Terry, M. E. (1952). *Rank analysis of incomplete block designs: I. The method of paired comparisons*. *Biometrika*, 39(3–4), 324–345.


Decision: Accept (Poster)