Inducing Programmatic Skills for Agentic Tasks
Authors: Zora Zhiruo Wang
Apurva Gandhi
Graham Neubig
Daniel Fried
Venue: COLM 2025 (2025)
Keywords: agent
skill learning
web navigation
scalability
generalization
TL;DR: We propose ASI, Agent Skill Induction, which induces and applies skill programs from web navigation experiences without supervision, yielding improved correctness and efficiency.

Abstract: To succeed in common digital tasks such as web navigation, agents must carry out a variety of specialized tasks such as searching for products or planning a travel route. To tackle these tasks, agents can bootstrap themselves by learning task-specific skills online through interaction with the web environment. In this work, we demonstrate that programs are an effective representation for skills. We propose agent skill induction (ASI), which allows agents to adapt themselves by inducing, verifying, and utilizing program-based skills on the fly. We start with an evaluation on the WebArena agent benchmark and show that ASI outperforms the static baseline agent and its text-skill counterpart by 23.5% and 11.3% in success rate, mainly thanks to the programmatic verification guarantee during the induction phase. ASI also improves efficiency by reducing 10.7–15.3% of the steps over baselines, by composing primitive actions (e.g., click) into higher-level skills (e.g., search product). We then highlight the efficacy of ASI in remaining efficient and accurate under scaled-up web activities. Finally, we examine the generalizability of induced skills when transferring between websites, and find that ASI can effectively reuse common skills, while also updating incompatible skills to versatile website changes.

Published as a conference paper at COLM 2025
Inducing Programmatic Skills for Agentic Tasks
Zora Zhiruo Wang Apurva Gandhi Graham Neubig
Daniel Fried
Carnegie Mellon University apurra,,,apurvag, apurvag, gre.bbugggggguggg.oooomomom dfriedddddddd cmu. edu
Abstract
To succeed in common digital tasks such as web navigation, agents must carry out variety of specialized tasks such as searching for products or planning travel route. to tackle these tasks, agents can bootstrap them- selves by learning task-specific skills online through interaction with the web environment. In this work, demonstrate that programs are an ef- fective representation for skills. We propose agent skill induction (ASI), which allows agents to adapt themselves by inducing, verifying, and uti- lizing program-based skills on the fly. We start with an evaluation on the WebArena agent benchmark and show that ASI outperforms the static baseline agent and its text-skill counterpart by 23.5% and 11.3% in success rate, mainly thanks the programmatic verification guarantee during the induction phase. ASI also improves efficiency by reducing 10.7-15.3% of the steps over baselines, by composing primitive actions (e.g., click) into higher-level skills (e.g., search. product). We then highlight the efficacy ASI in remaining efficient and accurate under scaled-up web activities. Finally, we examine the generalizability of induced skills when transferring between websites, and find that ASI can effectively reuse common skills, while also updating incompatible skills to versatile website changes.
Introduction
To achieve success in common digital tasks such as web navigation, it is essential for agents to be able to perform variety of specialized tasks such searching for products on shopping website (Yao et al., 2022; Deng et al., 2024) or finding a driving route on the map (Zhou et al., 2024a; Xie et al., 2024a). While one source for agents to learn such tasks is demonstrations annotated by humans (Deng et al., 2024) or synthesized with large language models (LMs) on websites of interest (Murty al., 2024b;a), this can be challenging offline learning procedure given the broad range of website domains and functionalities, especially match or cover the distribution of tasks queried at
for the collected demonstrations inference time (Zhou et al., 2024b); not to mention the limitations in resources to collect abundant high-quality data at ease (Pan al., 2024).
Instead of learning from demonstrations offline, an alternative way is to learn these tasks directly online from test queries to prevent potential distribution mismatch between demon- stration and downstream tasks (Levine et al., 2020). Some works propose to have agents induce casual abstractions (Majumder et al., 2024), single-state guidelines (Fu et al., 2024), or multi-step procedural workflows (Sarch et al., 2024; Wang et al., 2024b) as form of interme- diate knowledge to augment agent memory via marammeeii approaches (Brown et al., 2020). Nonetheless, most existing approaches represent this knowledge in text, offering limited quality and verification guarantees. In this work, we propose that executable programs are effective representations for intermediate skill acquisition, given their verifiability and composability advantages (Setlur et al., 2025).
We present ASI, namely agent skill induction ($2), that induces and applies programmatic skills along the process of solving user web navigation queries. More concretely, given natural language (NL) query, the agent first generates action trajectory attempting to solve the task using built-in, primitive actions such as click and scroll. The agent then

---

Published as a conference paper at COLM 2025
induces higher-level skills (e.g., search_ (name)) that wrap primitive actions or prior skills as executable programs, accompanied with corresponding test trajectories to verify their quality. Verified skills are then incorporated into the agent action space and can be directly called to solve future tasks with similar procedures, as depicted in Figure (bottom).
We first evaluate ASI the WebArena benchmark (Zhou et al., 2024a) (53) and demonstrate that our online, adaptive ASI surpasses its static agent baseline without adaptive components by 23.5% in success rate. To validate the advan- tage of using programmatic representa- tions for skills, we further compare to an adaptive agent, AWM (Wang et al., 2024b), that represents skills in mem- ory as non-executable texts (Figure 1 top); we find ASI scores 11.3% higher success rate by employing verifiable, programmatic skills (Figure bottom). Beyond the correctness aspect, the task- solving procedures by ASI-supported agents are 10.7-15.3% more efficient than the baseline approaches, mainly because of the action space abstraction and composition enabled by the pro- grammatic skill representation.
add
Wind my cancelled
by page
Memory
Query
When will order #120 arrive
observe
Action trajectory
One Step Monaer
f111 1L ("Xacbook")
click ("Search")
Agent
2
act
induce
Skill Library
Web Environment
deš search product (box, id: "-earrch for product in search box.
add
search, bos id: xo of search ispot field quezy netusna: Hose
Derify
Test
swarch_preduct("50) 'candy')
search product
eill Ibox Id, quary)
Skill
'595 "Macbook") ')
keyboard press ('Enter')
Figure 1: Online adaptive agent that induces and reuses programmatic skills actions (bottom), as opposed to adding textual skills in memory (top).
We further stress test AST on scaled-up scenarios (54) that require substantially longer- horizon trajectories to complete the task. Across various domains such as shopping and social forums, we find the efficiency advantage offered by ASI is more pronounced, reducing action trajectories by 9.5 and 5.6 average steps compared to static and text-form agents. Facilitated by this improved procedural efficiency and planning, we find that ASI agent adheres to the optimal trajectory better and achieves tasks 20.7-38.9% more correctly.
Finally, we study agent behaviors on generalizing induced skills to other websites ($5), particularly from sandboxed, simulated websites to real-world websites of similar domains. While ASÍ agents effectively transfer common skills (e-g,, search product) to new web- sites, some skills may be incompatible with alternative website designs hence less useful. Nonetheless, ASI can quickly refine its prior skills or create new skills on new websites, indicating it allows agents to adapt online while maintaining verifiability via programs.
In short, ASI enhances web agent success and efficiency by inducing and applying verifiable programmatic skills, in general and longer-horizon tasks, even across varied websites.
2 Agent Skill Induction
In this section, we first lay out the web agent problem setup (62.1) and introduce online, self-adaptive agents ($2.2). We then describe the core component of ASI- programmatic skill induction and verification (62.3).
2.1 Problem Statement: Online Adaptive Agent
For the scope of this work, we focus on language model (LM) based agents, where each agent policy consists of an LM backbone L, memory M, and skill library A, as illustrated in Figure 1 top and bottom. In the implementation, the memory M and the skill library A are provided as input context to the LM backbone. We denote the agent policy as Te(-IM, A) and ag for short. We focus on the web browser environment defined by a transition function T(s'|s,a) that models the change in the webpage after action.
We focus on an online adaptation scenario where we have access to sequence of NL queries Q {91 92. ... ) specifying the tasks, and no other information such as demonstration
2

---

Published as a conference paper at COLM 2025
trajectories or ground-truth rewards are available (Wang et al., 2024a;b). For each task specified by a natural language (NL) query 9, the agent generates a trajectory of actions T = (50, A1,
1, 1.SH) for finite number of H steps. At each time step h in the horizon, the agent receives observation 0g from the current state 8n, and generates an action an A based on the observations and actions far, via TT ( Ap.-ii A) dn- The generated action will be executed on the environment and incurs a state change (s,,, ()) + 9/+1. This observe-act loop continues for H steps until the task reaches a task-terminating condition, such as the agent generating termination action (e.g., send wwg.mmggtoo... (ss_tr or the horizon reaches a pre-determined maximum number of steps h Hmar- We denote each pair of query and trajectory (g, T) == e as an episode e. Agents can update the content in M and A and reuse them across episodes.
user))
2.2 Inducing Reusable Skills
To realize online adaptive agents, one common approach is to induce skills from correct trajectories to update the agent (Wang et al., 2024b). But since ground-truth rewards are unavailable, an LLM-based evaluator Vc is often used to judge the correctness of episodes. Formally, from the total of N episodes throughout the online process {el, ..., eN := E, we employ an LM-based evaluator Vz(e) + 0/1 to filter out the episodes predicted as correct Ey = (e; E EVVc(e)) 1, E {1, ... , and perform skill induction only on εy.
Central to our adaptive agents is an induction component / that enables the adaptivity of agents, which can be rule-based (Ellis al., 2023; Grand et al., 2024) or instantiated by an LM I(- (Wang et al., 2024b); we follow the latter for its better performance and use I to represent the module for simplicity. For online adaptive agents TC, to induce skills, is instructed to take in one filtered episode and output one more pieces of desired skills D {d}, denoted as I(e) + D. Following AWM (Wang et al., 2024b), we update the agent in non-parametric ways that incorporate the induction outcome [(e,) - di into the agent, instead of updating the parameters of the underlying LM backbone L for agent policy πg-
Unlike AWM which represents skills in free-form text representations and can only augment agent memory via M,U [d:] + M1+1 (Figure top), we introduce ASI that represents skills as executable python programs, and directly integrate skills into the agent action space instead, via A, U [d:) + A++1 (Figure 1 bottom).
2.3 Inducing and Verifying Programmatic Skills
To improve the induction quality, we propose a change in representation from free-form text to executable programs, which offers advantages in correctness and efficiency. For one, the program format enables ready verification on skill correctness by executing them; for another, skill programs abstract multiple lower-level actions into a higher-level function call, thus agents can solve tasks in fewer steps without tackling tricky low-level details.
Inducing Programmatic Skills We first clean the input episodes to ensure the induction quality. We remove all steps that cause execution errors such as invalid argument format, to keep these invalid actions from distracting agent predictions. Furthermore, noticing the long and possibly redundant thought process generated by agents along with each action, we simplify each thought text paragraph into short one-sentence description (e.g., "Clicked the directions button to access the route planning feature") using LM, effectively reducing the thought content from 87.9 to 13.4 tokens per step.
Given a clean input episode e, we now prompt the induction module to produce one or more program functions to represent reusable skills D {d] as executable programs. As exemplified in Figure 2, given the input episode on the left side, the induction mod- www.mmareetnnngg.
ule first produces two skills search barrhhbbutonn.d. search.term) in the form of callable program functions.
() and search_reviews (search.box.
Skill Verification With the programmatic nature of ASI's skills, we can readily verify their correctness by executing them and checking if tasks can be solved successfully. While a naive way is to query the agent with the same NL query and allow it to use newly induced skill actions, we find agents may not always use new skills due to the large search space of
3

---

Published as a conference paper at COLM 2025
NL query
leearch button id: ats, search Tarm: atc): Seacch for geviows with specific tezn.
Tell me the number of reviews that our store received so far that mention the term "satisfied"
Expepies: mie persenl "o0.. 'preat))
(search box search_tere (erem) click button_id]
skills
episode
IM 10T
def narketing reviove()
"i'aaiiqate to 'the All Reviews section
click ("Narketing")
Marketing:
Exselies: "gon_rarketinggrriinn reviewss
click ("All Revieue")
Action Trajectory
click ("Macketinn")) ("All Reviews")
fill (757, "satistied")
Induce
click ("Search")
test
open www.pnaceetengeeeeee pevieva
povieeo ('75', '04" "satiofied") 1e
send zag. to ('22("2"))
Figure 2: Inducing programmatic skills and rewriting the trajectory from an episode.
possible action trajectories. To have agents more efficiently generate trajectories that test skills in more targeted way, we curate rewritten trajectory prefix TD to constrain the first few steps executed in the environment, by rewriting and truncating the input action trajectory T, and subsequently asking the agent to complete the prefix to get a full, checkable trajectory Ty. Concretely, we first take the original action trajectory in the input episode T (consisting of primitive actions or previously learned skills), and ask the induction module I to transform a skill-using trajectory (Figure 2 bottom right), by replacing sub-trajectories in with calls to the newly induced skill programs D, if possible. Zooming into the Figure 2 example, this procedure merges (MMakketing" + click('All Reviews') to an open_markettin paakktinngrere teeettreiiiiiieeeeeeeee reviews call; transforms fill (757, 'satisfied') -> click(' "seack"s to a call of the second skill search_reviews (' sstisfiedd) with the specified term 'satisfied'; and adopted the last send waaddd_sgttouusss ('2') step directly. Note that we follow Wang et al. (2024b) and induce skills according to each website, so some skills could be tailored to particular webpage contexts such the 'Marketing' and 'All Reviews' link constants in www.mmakkk.eeeeeeeee
while other skills apply to more versatile setups such as searching for different reviews in reviews
Next, to avoid spurious successes in skill verification, we truncate the trajectory yielded above by removing any trailing primitive actions after the last call to a skill program. Taking Figure 2 as an example, in the original input trajectory, the last already sends over the correct answer '2' to the user. However, if we directly adopt this last step into the skill-using trajectory TD, then executing it will always return the correct message the user, regardless of whether the previous skill calls are valid. We thus remove such trailing actions to make sure verification attends to the induced skills we are testing.
('2")
After rewriting and truncation, we get this skill-using trajectory TD as the prefix to test skills. We now query the agent π again with the same NL query and first execute Tp on the environment. We then allow agents to continue generating up to Hmex - |Tl actions to finish the task. In the Figure 2 example, to successfully solve query 9, we expect the agent to generate another step of We take the concatenation of the trajectory prefix tD and the later additionally produced new steps TA (e.g,, [send msg to user ( (*2')) as the full trajectory Ty. We then decide whether to add the induced programs D into the agent skill library as A by examining Ty-
with the correct answer '2' in the message.
Specifically, we check T from three dimensions: (1) Correctness: executing Ty successfully solves the task q as judged by the neural model evaluator Vz; (2) Skill Usage: if the trajectory contains at least one call to at least one new skill in D; and (3) Skill Validity: if all skill-calling actions cause environment changes. If all three boxes are checked, we add the skills being called in the trajectory Ty to the agent skill library A, U Dealted - A By adding Dealleds the agent can now generate actions that call these skill programs solve subsequent tasks.
3 General Web Navigation Performance
3.1 Experiment Setup
Benchmark and Evaluation To evaluate ASI on general web navigation scenarios, we adopt the WebArena benchmark (Zhou et al., 2024a) that contains 812 test examples covering
4

---

Published as a conference paper at COLM 2025
Model
Method
Steps
SR
Shop
Admin
Reddit
GitLab
Maps
Multi
Vanilla
12.3
13.9
10.4
6.6
15.0
15.6
8.3
GPT
AWM
5.9
35.5
32.1
29.1
54.7
35.0
42.2
18.8
Vanilla
5.6
32.7
32.6
36.8
36.8
26.1
38.5
20.8
Claude
AWM
5,9
36.3
34.8
39.0
51.9
28.9
39.4
18.8
ASI (ours)
5.0
40.4
40.1
44.0
54.7
32.2
43.1
20.8
Table 1: WebArena success rate by adaptive agents with programmatic skills, in comparison to a static vanilla agent baseline, and a text-skill learning adaptive agent.
five major web activity domains: e-commerce, social forum, software development, content management, and travel. Each example in WebArena has an NL query for the task, and program-based evaluator that provides binary 0/1 score for any given trajectory T to judge if it successfully solves the task q. This program-based evaluator enables relatively rigorous evaluation based on the functional correctness of the action trajectory. We report the average score across all WebArena examples, if not specified otherwise.
Backbone LM and Agent Architecture We use the top- top-performing claude-3, 3-5 model the LM backbone for all components, including the agent policy π,, the neural eval- uator V, and the skill induction modules For experimentation, we use the BrowserGym (Chezelles et al., 2024) framework, which takes the webpage accessibility tree observation, and instantiates the skill library A with the WebArena default action space listed in SA.
Baselines We take the vanilla Claude model with the BrowserGym framework (Drouin et al., 2024) as the non-adaptive agent baseline. Additionally, we compare ASI to AWM (Wang et al., 2024b), the current top-performing online adaptive web agent method. Be- cause AWM was originally developed with the gpt-4o model, for fairer comparison, we also experiment with AWM with claude-3, 3.5dennt5- model as its LM backbone and also apply the episode cleaning procedure to enhance induction quality. We compare the two baseline methods with our AST approach. We provide the complete prompts for each agent component:
task-solving, episode evaluation, episode cleaning, and skill induction, in SA.
Results and Analysis
In Table 1, compared to the vanilla static-agent baseline, adaptive agents (AWM and ASI) generally achieve 11.0-23.5%. higher success rates overall. Among adaptive agents, our ASI with programmatic skills, achieves another 11.3% success rate gain across websites, compared to its AWM counterpart that induces and uses textual skills. Meanwhile, ASI offers additional efficiency benefits by reducing the number of steps in solutions by 15.3% and 10.6% than vanilla and AWM agents, as one skill-call action can often execute multiple steps written in primitive actions used by vanilla and AWM agents. These advantages in correctness and efficiency are exhibited prominently across different websites and tasks, as shown by the website breakdown on Table (right). Refer to SB for more analysis.
3.3 Why are Programmatic Skills Better?
To more concretely answer why programmatic skills are more effective than textual skills, take a closer look the two main differences between AWM and ASI: [1] whether the induction outcome is verified via execution, and [2] whether the induced skills are provided in memory for reference purpose only, or in the action space that allows execution.
Better Induction Quality We take the shopping website as a representative, and analyze the textual and program skills induced by AWM and AST agents. We group textual and pro- gram skills by their functionality and show one representative example in Table 2. Compared to the clear functional boundary and highly-reusable granularity of the search_produu we find that the textual skills often have (1) more redundant steps, (2) ceample-specifii context: e.g., the last text skill aims to find 'game accessories' while the steps generally applies to any product, and (3) fuzzier boundaries between separable tasks, e.g., the first skill mixes product-sead guidance when asked to, e.g., search product and add it to cart instead.
skill,
and add-ta to-wishlist procedures, thus may not offer optimal
5

---

Published as a conference paper at COLM 2025
Programmatic Skills
Textual Skills
product
wish
Action Trajectory:
"*ill "'fil1(621, (product __ Enter the product
name the search box
"<11ck(078)" TM Click the search button to execute the sourch
"elick ((product_ "M # Click the product to check
details
def search (seachh box id: str, query: ste): 1*"Beacch for a product using the search box.
""lick (1769) "M Click the 'Add to Wish List" link
Args:
Task: Search for product's price range in the store Action Trajectory:
search box id: 10 of the search input field
query: Search query string enter
Returns:
"'fill (563, (product_nnae __nam]] " Enter the product
""click (370) "M # Click the search button to execute the
Examples: search product (15951 "aony bluetooth headptones')
"oopp (1000) # Wait for search results to load "wend nag to user ((price range info))" "* Analyze and report the price range findings from the search results
click (search box id)
fill (search bex id, quary)
keyboard ('Enter')
Task: Search for gaming accessories within date range Action Trajectory:
""lick (1274) NNavigate to the Video Games calegory "'fill (473, (search terms]) Enter search terms including product and year "411ck(478)" Execute the search
Table 2: Example textual and program skills induced on the shopping website.
Verified Induction Improves End Success Rate From qualitative examination of the induction outcomes, we find roughly similar numbers of episodes evaluated as correct and used for induction (70 and 58 examples for AWM and ASI), ASI produced programs pass verification for only 15.6% of the turns, whereas AWM adds new skills for 31.4% of the time (replace add none otherwise). While skill usage (in memory or action, [2]) is designated for AWM and AST, we hypothesize that verification [1] affects induction quality and thus end success. We thus experiment with another setting that induces programs (such that verification is enabled), and only use the induced skills in memory, to study the importance of induction quality. As shown in Table 3, inducing skills with execution-bases based verification (i.e., (unverified, text) + (verified, program)), while always present skills in memory, improves end success rate by 4.2 points, indicating the importance of higher-quality induction via verification. Yet it is still 3.7 points lower than ASI, suggesting the incompatibility of program format to agent memory. Indeed, we observe many cases where the agent tries to call the skill programs but unsuccessfully, since they are not supported in the action space.
Textual Representations Suit Memory Bet-
ter To prevent the agent from trying to call these plausible programs, we ablate an-
Method
SR
unverified, text
32.6
other setting that transforms program skills to textual format (as Table 2 right) and pro- vide them in agent memory, dubbed (veri- fied, text). This format transformation effec- tively improves the overall success rate by another 2.6 points, getting a little closer to ASI. Given the different downstream usage, i.e,, memory or actuation, textual and program formats may suit individual scenarios better.
Add to Memory
verified, program
36.4
verified, text 39.0
Add as Actions verified, program
40.1
Table 3: Ablation study on induction verifica- tion and format on the shopping website.
Beyond basic web navigation tasks, in the next two sections, we examine agents in two other important scenarios, scaled-up activities ($4) and cross-website generalization (55).
4 Scaled-Up Browsing Activities
The WebArena benchmark mainly features isolated, single-task scenarios, such adding single product to the shopping cart. However, in real-world practices, people need to do series of such tasks together, such as adding multiple related products (e.g., coffee and

---

Published as a conference paper at COLM 2025
mug) to the cart before finally checking out. This browsing request can lead to extremely long-horizon tasks, sometimes with repetitive intermediate procedures. We identify this to be scenario to further demonstrate the efficacy of program skills, as opposed to textual skills, as programs lend themselves naturally to repeated invocation and composition.
Therefore, curate several case scenarios where the user asks for action-dense instructions, such as the tasks listed in Figure 3. Because the tasks are long-horizon and involve multiple sub-tasks, we follow Xu et al. (2024) and set up intermediate checkpoints to better track the intermediate progress of agents. Refer to $C.1 to see the full list of tasks and their evaluation checkpoints. We measure the success rate of each example by the percentage of checkpoints achieved by the agent. We report the average success rate of all examples, as well as the average number of steps taken to solve the tasks, in Table 4.
Shopping # #steps
Admin
Reddit
GitLab steps
Map
Method
sr f
sr f
steps
sr f
# steps
sr
sr f
steps
VANILLA
41.7
23.5
58.0
20.8
33.3
23.0
33.3
40.0
40.0
15.2
AWM
68.3
21.5
74.0
18.2
40.0
16.8
50.0
33.8
65.0
12.6
ASI (ours)
100.0
16.3
91.0
14.2
55.0
12.8
55.0
25.4
100.0
6.2
Table 4: Performance of vanilla, AWM, and ASI agents in scaled-up browsing scenarios. We perform statistical testing between ASI and each baseline and verify all improvements are statistically significant with t-statistics |t] > 2 and p < 0.05; see SC.3 for more details.
ASI Features Improved Efficiency Across all websites, AST-produced trajectories have 6.6-14.6 and 4.0-8.4% fewer steps, compared to vanilla and AWM baselines, respectively. As the task horizon continues to grow when involving more intermediate checkpoints, this margin between ASI and baselines will predictably be more prominent.
Subsequent Benefits in Success Rate ASI also achieves higher success rates with more efficient trajectories, outperforming vanilla and AWM baselines by 38.9% and 20.7% on average. From manual analysis, find this improvement comes from easier, better agent planning when using bigh without the need to tackle more complex procedures if only low-level primitive actions available, with vanilla and AWM agents.
Changing Multiple Addresses We present a representative case on the shopping website:
Case Study:
changing billing and shipping addresses after moving. As depicted in the top row in Figure 3, the vanilla agent without adaptive skills often roams into some irrelevant exploration steps, instead of sticking the optimal route to solve the required task. It runs for minutes and exhausts the maximum steps (i.e., 50) before finishing the task.
recently moved. Can change billing address to "231 Willow Way, Suite 100, Chicago, 60601"? Then, update shipping address to: 987 Sycamore Circle, Philadieppia, PA, 19102.
Willow Way, Suite 100, Chicago II, 60501
Doriilla
Mi Account))
clickrMy Address))
chang billing
update streei line
update street line update city nam name
update state update zip code
save changes
homepage
reduoddant steps
redundant sleps
fermotonnte MAx sleps
clickl Add
clickt Addr ess Book
for 2231 Willow Way, Suite 100, Chicago It, 60601
finish my strps
ANA
update street line
update
update city name update state
update zip code
save changes
ci Account')
MlicklMy click chang Addrees))
street line
homepagge
sbart om
billing
Finish my strps
for 1987 Sycamone Circle Philadelphin, PA 19102"
follow the markfion correctly
ASI
update adviress details" 1724, .oy 25t Willow 100, Chicago, Lillindss, 660611)
update adéress detuila (124 1224.. Sycamore Circle. Philaddlphiia" Penasylvania, (19112)
start on homepage
navigaate address change billing address))
Reach in one stop
Update address in one step
Update address one step
Figure 3: Example scaled-up task of updating multiple addresses shopping website.
7

---

Published as a conference paper at COLM 2025
With AWM, adding textual skills in memory provides soft guidelines for agents to follow, the agent thus better sticks to the goal and finishes each part of the task (i.e., navigate to the address page, update billing/ shipping address) step by step. Although successful, the trajectory is long, i.e., 27 steps, and still takes a few minutes to finish.
In comparison, ASI (in Figure 3 bottom row) showcases its efficiency by using learned skills to navigate_ stteetto___oe adddddeess... beddressssettinge and update_address. step (vs. the 3-6 steps used by AWM for these parts). Overall, ASI correctly finishes all required actions in only 4 steps, shortening the horizon by 85.2% compared to AWM.
address address.details _details can solve each part in one
5 Adapting Across Websites
To examine whether agents can generalize with learned skills, we test agents on real-world website counterparts for some of the do- mains in WebArena as listed in Ta- ble 5. This experiment setup can reflect on (1) transfer across differ- ent websites of the same domain, and (2) transfer from simulated, sandboxed to real-world websites.
Domain
WebArena Sandboxed
Real-World
shopping online forum travel
OneStopMarket PostMill OpenStreetMap
Target Reddit
Google Maps
Table 5: Real-world in-domain website counterparts to each WebArena sandboxed website.
For each sandbox-real website pair, we take ten information-sseking style queries (He et al., 2024) in WebArena that do not involve potential privacy leakage or unrecoverable risky actions, such as making a purchase or changing user password. We provide the task details in SC.2. We compare ASI and AWM with their programmatic and textual skills as learned in $3, as well as comparing to the vanilla static agent baseline.
Transferring Common Skills In Figure 4, we can see how ASI can effectively reuse common skills such as rearch_product in the first step on the Target website.
Show me the least expensive shoe slorage.
spring sale
correctly the learned skill
Incompatible Skills One challenge faced by ASI is that some prior skills be- come incompatible on the
start homepage
prorch_ 'shee strage))
ded sost Liatioge Jaoes deopdows Ld: atc. SOET optson: #26): Liolings
Arga: sest opstioption: Forting option Herper--Qeeeni wee---eeeeiiie Aooreean
clack
oywoss page instoad
new website. For exam- ple, the cort.by_listing induced on OneStopMarket selects options from a drop- down menu, yet sorting on the Target website opens a sidebar; despite their semantic similarity, the concrete actions in skill programs are no longer applicable. Still, we find that agents can often spot this incompatibility and rarely attempt to use these deprecated skills.
Liatinge (11322' 'Bcice' onlty applies to dropdonnn meena
deoppeooo down.
Sort By
Relevance
aslecc uptlic Bropdoan, id, soct aptioo)
Figure 4: ASI can generalize the search product skill but face incompatibility when sorting items.
Adapting Skills to New Environment Although some skills induced on previous websites
cannot be directly used on arbitrary new websites, we hypothesize that these skills can still serve as infor- mative references solv- ing procedurally similar tasks or composing new skills targeted for the new website design.
Shopping sr # steps
Reddit sr
Map # steps
Method
Sr f
# steps
Vanilla
80.0
5,4
40.0
4.8
63.3
7.4
AWM ASI
80.0
5.0
56.7
4.8
100.0
6.2
90.0
3.4
76.7
4.4
93.3
4.4
AWM update
80.0
5.4
63.3
5.8
100.0
7.2
ASI + update
90.0
3.2
76.7
4.0
93.3
4.2
Table 6: Cross-website results. ASI significantly surpasses baselines sr and # steps (with |f] > 2 and p < 0.05) from our analysis SC.3.
We did not test on administrative and software websites given their more severe safety concerns.
8

---

Published as a conference paper at COLM 2025
We thus allow agents to induce new skills update previously acquired skills from expe- riences on the new website, denoted as +update entries in Table 6. We find that enabling skill update in both textual and program formats helps agent performance Within the short online learning process (tens of examples), AWM adapts faster to the new websites, while ASI sees a more pronounced improvement in efficiency.
new websites.
6 Related Work
Adaptive Digital Agents An important thread of agent-improving methods is to build adaptive agents that can autonomously self-improve from experiences. Most works focus on integrating past experiences into agent memory by collecting human annotation (Deng et al., 2024) or LM-based synthesis (Ou et al., 2024; Xu et al., 2025), especially via agent- driven exploration with instruction- (Murty et al., 2024b) or trajectory-driven (Murty al., 2024a) approaches, offering warm starts on the websites of interest. Other works gather experiences (Wang et al., 2024b) or feedback (Qu al., 2024) during test time, and augment them into memory through parametric channels such as supervised fine-tuning (Murty et al., 2024a), contrastive learning (Song et al., 2024), or reinforcement learning (Zhou al., non-aaametrric approaches can directly augment reflections (Shinn
2024b). Meanwhile, et al., 2023), raw past experiences (Wang et al., 2023; Zheng et al., 2023), or further induced reusable workflows (Wang et al., 2024b). While these adaptive agents learn textual skills stored in memory, our ASI stores skills as verifiable and composable programs in the agent action space (i.e., skill library), thus enabling better quality and efficiency.
Skill Discovery and Learning Learning specialized skills for tasks in programmatic (Shin et 2019; Ellis et al., 2023; Cai et al., 2024; Wang et al., 2024a; Grand et al., 2024), embodied (Sharma et al., 2022; Wang et al., 2023; Liang et al., 2023; Sarch et al., 2024; Wong et al., 2024), and physical (Yu al., 2023) environments has shown to success in agent performance. Particularly for digital agents built for web navigation tasks, most works tocus on exploring skills offline with RL roll-outs (Gur al., 2018; Liu et al., 2018; Putta et al., 2024; Qi al., 2024) or LM-based prompting (Zhou et al., 2024b; Murty et al., 2024a; Patel et al., 2024). While this exploration stage could offer some supervised data to update the agent policy either parametric (Murty et al., 2024a; Patel et al., 2024) or non-parametrically (Zheng et al., 2023; Murty et al., 2024b), it often costs enormous extra computation and may suffer from the lack mismatch in distribution with the downstream tasks at hand (Wang et al., 2024b). In contrast, our ASI does not rely on supervised data and can directly learn skills online without prior exploration.
Web Navigation Benchmarks Digital agents have been explored across a wide range of tasks (Yao et al., 2024; Kapoor et al., 2025; Xie et al., 2024b), among which one of the most popular application being browsing and navigating through versatile websites such as shopping (Yao et al., 2022), social media communication (Zhou et al., 2024a; Koh et al., 2024), knowledge work tasks (Drouin et al., 2024), and more (Deng et al., 2024). Our work focuses on general web navigation tasks using the WebArena (Zhou et al., 2024a) benchmark, meanwhile exploring other challenging scenarios such as scaled-up activities (Yoran et al., 2024) and cross-domain generalization (Deng et al., 2024).
7 Conclusion and Future Discussions
In this work, we present ASI support web navigation agents to autonomously induce, verify, learn, and apply programmatic skills during online inference. Beyond achieving 23.5% success rate and 15.3% efficiency increases in general web tasks, we also showcase ASI's strengths for scaled-up web activities, thanks to the high-level action interface offered by the programmatic abstraction. Moreover, we examine skill generalizability to new, real- world websites, and find ASI still offers great efficiency while flexibly updating skills to new environments. While our work aims to offer insights on the optimal representation in agent skill acquisition, still find multiple pieces in ASI worthy of further investigation, such as the conceptually or empirically suitable granularity of skills, the stability of the online evolving process, and the skill quality in comparison to human expert desiderata.

---

Published as a conference paper at COLM 2025
Acknowledgments
We would like to thank Jiayuan Mao, Yueqi Song, Boyuan Zheng, and Yu Su for the insightful discussions. We thank Yiqing Xie, Xinran Zhao, and Mingqian Zheng for their helpful comments on the paper draft. Zora is supported by the CMU Presidential Fellowship and Fujitsu Research. Apurva is supported by Amazon.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc. 2020, URL https:/ wwwproceedding..euriss...m peeeurpp.cc/pppprr paper files #457c0666fcb49967418666666 hatp:/aaaeee2eeaaaaaaaapere
falles/ille///e Patees/pppr//ae pappr/2020/ 22200/2222///
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models tool makers. In The Twelfth International Conference on Learning Representations, 2024. URL
De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, et al. The browsergym ecosystem for web agent research. arXiv preprint or arXi::2412.05455 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arxiv preprint arXiox2403.077188 2024.
Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with slee-sseep bayesian program learning. 2023.
Philosophical Transactions of the Royal Society A, (2251112))
Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of sta statre guidelines for large language model agents. CoRR, abs/ 0bbs/240000877, 2024. URL https: //doi.org/ 10.11044455 10.445500////1 arXiv.4403.0897%.
Gabriel Grand, Lionel Wong, Matthew Bowers, Theo X. Olausson, Muxin Liu, Joshua B. Tenenbaum, and Jacob Andreas. LILO: Learning interpretable libraries by compressing and documenting code. In The Tweifth International Conference on Learning Representations, 2024. URL /ttp://errreeeeeeee//rrdddddddda
Izzeddin Gur, Ulrich Rückert, Aleksandra Faust, and Dilek Hakkani-Tür. Learning to navigate the web. CoRR, abs/1182.09915, 2018. URL http:/ aaw.v.orgi/o////// 1ab1b.011.9999
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024. URL https:// https://aclnnhhologgorgg 2tt.:y/orrr02....oooo...77777
10

---

Published as a conference paper at COLM 2025
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AiShikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pp. 161-178. Springer, 2025.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwe- barena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint XrXi::2005.011 01 2020.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 TEEÊ International Conference on Robotics and Automation (ICRA), pp. 9493-95500 IEEE, 2023.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In Infernational Conference on Learning Representations, 2018. URL https:/ /ttpe//eeeeeeeee//eooeeeeee
Bodhisattwa Prasad Majumder, Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Butte and Peter Clark. CLIN: A continually learning language agent for rapid task adaptation and generalization, 2024. URL https:/ /pentevieww.net/ forum?ic-d5DGVVHMMdC.
Shikhar Murty, Dzmitry Bahdanau, and Christopher D. Manning. Nnetscape navigator: Complex demonstrations for web agents without demonstrator, 2024a. URL https:
//arxiv.org/ abbs/2410.02900
Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Bootstrapping agents by guiding exploration with language, 2024b. URL https:/ arrxiv. org/abs/ 44000.0000.
Tianyue Ou, Frank Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https:/ /oppss////preei.w. net/ htt/fooumm?d-KKNEEWRR..
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. Webcanvas: Benchmarking web agents in online environments. In Agentic Markets Workshop at ICML 2024, 2024. URL httpp:// https:///opemreviewwnn///// http::/p///ppppme/eww////////ett /orum//d=O11aaaasssdd
Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu-Condrei, Marius-Constantin Dinu, Chris Callison- and Sepp Hochreiter. Large language models can self-improve at web agent tasks. Xiv preprint ar Xix::405.20039 2024.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arxio preprint ar Xrx:::2408.077 071199 2024.
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning. arXio preprint ar arXie:2411.02337, 2024.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve, 2024. URL https:/ htttps://aaxiv.rrgg. atxiv.orggg/////// /bss/ 2407. 240191
11

---

Published as a conference paper at COLM 2025
Gabriel Sarch, Lawrence Jang, Michael Tarr, William W Cohen, Kenneth Marino, and Katerina Fragkiadaki. Vlm agents generate their own memories: Distilling experience into embodied programs of thought. Advances in Neural Information Processing Systems, 37: 75942-75985, 2024.
Amrith Sethur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification rl is suboptimal. arXiv preprint ar Xix::2502.12118, 2025.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, May 2022.
URL https:/ hatp:s//acaaattttti /wtpa/aoloooooooorrg//////// /atp://ooogoooog///2.aaalal/ 222..22222222...... ltt..aacooooo..1/.....
Eui Chul Shin, Miltiadis Allamanis, Marc Brockschmidt, and Alex Polozov. Program synthesis and semantic parsing with learned code idioms. Advances in Neural Information Processing Systems, 32, 2019.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https:/ http:v..xxxvvrrg//// abs
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization for 1lm agents, 2024. URL https:/ /rxiv.org/ 220.0240..0
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL htttps///arivv.or
2bb5/230..6222
Zhiruo Wang, Graham Neubig, and Daniel Fried. TroVE: Inducing verifiable and efficient toolboxes for solving programmatic tasks. In Forty-first International Conference on Machine Learning, 2024a. URL hatpp::///opppeeeeee
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arviv preprint ar Xric:2409.07429 2024b.
Lionel Wong. Jiayuan Mao, Pratyusha Sharma, Zachary S Siegel, Jiahai Feng, Noa Korneev, Joshua B. Tenenbaum, and Jacob Andreas. Learning grounded action abstractions from language. In The Tweeffh International Conference on Learning Representations, 2024. URL https:// /openreview.net// /ttp:///pp/.eiee//ooo//dddddodda
forum?id=-lOCciAEE..
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, 2024a. URL https: //openneview.net/foouueee httper/vweeeeetmmummm//////o//.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024b.
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking IIm agents on consequential real world tasks. arXiv preprint ar arXio:2412.14161] 2024.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In The Thirleenth International Conference on Learning Representations, 2025. URL
Information Processing Systems, Fggp YUccsVV.
https:// httpes////pnneviee.nee///// iepperrrieew.ee//orumidddde forum?id=EF.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural 2022.
12

---

Published as a conference paper at COLM 2025
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. T-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks? arviv preprint xwx::2407.157111 2024.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gon- zalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, brian ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023, URL https:/ https://ppennvviennef//ooo www.wettfffommddd-eem SgTPdyehXMA.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-eemmplar prompting with memory for computer control. In The Twelfth International Conference on Learning Representations, 2023.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,
Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig, Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, 2024a. URL https:/ http://pppeeeeeeeeeeeeeee forum?id= oKn9c6ytIx.
Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Erran Li. Proposeragentevaaluto (pae): Autonomous skill discovery for foundation model internet agents. arxiv preprint 41194 2024b.
13

---

Published as a conference paper at COLM 2025
A Experiment Details
A.1 Agent Action Space
Table 7 shows the default action space the web navigation agents we employed in all the experiments. This action space remains the same for both (i) static, vanilla agent, well as the (ii) adaptive agent that learn textual skills in memory, ie., AWM.
Action Type
Description
(wwitwaitmm))
Do nothing for specified time.
click(elem)
Click at an element.
hever(elem)
Hover
element.
fill(elem, value)
Type into an element.
(eyboarrd.rress(eey.comm))
Press a key combination.
scroll(x, y)
Scroll horizontally or vertically.
select option options)
Select one or multiple options.
goto(url)
Navigate to a url.
go (o.back() go_forward()
Navigate to the previous page.
Navigate to the next page.
new.tab()
Open new tab.
tab close()
Close the current tab. Bring tab to front.
tab focus(index)
send msg to user (text)
Send message to the user.
seporttinneasiileereeaao
Notify user that instructions are infeasible.
Table 7: Initial action space for agents throughout our experiments.
A.2 Prompts for LLM-Based Components
Prompt for Episode Evaluation
You are an expert in evaluating the performance of a web navigation agent. The agent is designed to help a human user navigate a website to complete a task. Given the user's 's intent, the agent's action history, the final state of the webpage, and the agent's response to the user, your goal is to decide whether the agent's execution is successful not. Please be careful of each detail and strict about the evaluation process.
There are three types of tasks: 1. Information seeking: The wants to obtain certain information from the webpage, such as the information of a product, reviews, map info, comparison of map routes, etc. The bot's response must contain the information the user wants, or explicitly state that the information is not available. Otherwise, e.g. the bot encounters an exception and respond with the error content, the task is considered failure. Besides, be careful about the sufficiency of the agent's actions. For example, when asked to list the top-searched items in a shop, the agent should order the items by the number of searches, and then return the top items. If the ordering action is missing, the task is likely fail. 2. Site navigation: The user wants to navigate to specific page. Carefully examine the bot's s action history and the final state of the webpage to determine whether the bot successfully completes the task. No need to consider the bot's response. Content modification: The user wants to modify the content of a webpage or configuration. Carefully examine the bot's action history and the final state of the webpage determine whether the bot successfully completes the task. No need to consider the bot's response.
"IMPORTANT" Please be strict about the evaluation process. Format your response into two lines as shown below:
Thoughts: <your thoughts and reasoning process> Status: "success" or "failure"
14

---

Published as a conference paper at COLM 2025
Prompt for Episode Cleaning
You are a helpful assistant in summarizing web browsing actions. Your task is to summarize the main action taken and the corresponding state change, from the input thought process and environmental description.
Example input: "LLet me think about this step by step: 1) I've clicked the Columns button and based on the accessibility tree, I can now see a list of available columns that can be shhwn/hidees hidden. 2) Looking through the available columns in the accessi- bility tree, I don't t see any direct column for "Number of Orders" similar metric. 3) Since don't see the information need the available columns, this suggests that: Fither this information is available through different path Or need to access the Sales/ /Orders section to correlate this information 4) The most logical path would be to: Go the Sales section where can see all orders And then analyze which customers have the most orders 5) From the accessibility tree, can see the "SALES" link with bid "183" in the main navigation.
Let me click on the SALES link: This should open the Sales menu where we can access the Orders section to analyze customer order histories." "
Example output: "Clicked the SALES link to access the Orders section for customer order histories."
Prompt for Skill Induction
You are proficient software engineer. Your task is to (1) summarize reusable functions as APIs from the provided action trajectories, and (2) rewrite the trajecoties using the reusable functions you generated in (1).
For (1), from the provided examples about the same task, you job is to generate Python functions that can be reused to solve (part of) these tasks. The functions should have mediocre complexity: (i) containing at least three actions and not too simple (e-g., single line of code), (ii) not too complex (e-g., more than 10 lines of code), and should be general enough to be applied to other similar tasks. The arguments these functions should be common variables (such as strings and lists), avoid using complex inputs such as another function.
The actions you can use are: 'click', 'fill', 'hover', "keyboard press', 'scroll'; 'tab focus', 'new hawa'aab, 'tab_close, book,,ck, go__forard'; 'send_mss _mgg to vo_user', report_infeasible 'select.on dntlecttoption.... Do not use other undefined actions. Do not include any try-except blocks in the functions.
Please include Arrg', 'Return'', and 'Examples' the function documentation. For (2), write the instruction and rewritten code of each example. Do not include the answer response or example-specific information in the rewritten code. Pay attention to whether all link IDs available before specifying them in the generated functions. If you use 'send_mst to to_user', make sure the message is decided within the function, instead of provided Make sure each function contains no less than steps, and no more than 5 steps; to keep the functions simple and task-oriented. You can generate zero, one, or multiple functions depending on the provided examples.
argument.
B Skill Induction: Analysis
We provide more details about the skill induction process, in skill curation and reuse frequency ($B.1) and representative skill case studies ($B.2)
B.1 Skill Induction and Reusability
To provide more insights on how agents curate and reuse programmatic skills, for the main experiments on WebArena, we calculate the number of examples that (i) attempt to induce a new skill, (ii) successfully induce new skill, and (iii) reuse previously induced skill.
15

---

Published as a conference paper at COLM 2025
As shown in Table 8, the agent reuses at least skill for 42.5% of the cases, which is pretty frequent across websites. Moreover, these skills are created using 7.6% of the tasks, demonstrating the high scalability of our skill learning approach.
Number of Examples Successful Induction
Domain
Attempted Induction
Reuse Examples
Total
shopping
8
96
180
admin
38
15
108
156
reddit
24
11
14
105
map
13
10
80
109
gitlab
25
11
10
175
Table 8: Analyzing the number of examples that attempt, induce, and reuse skills.
B.2 Representative Skills
We demonstrate two representative types of skills, which (i) chain frequenll-usse--oggehee actions, and (ii) involve advanced programming primitives.
Chaining Primitive Actions The most common type of skills chains multiple frequently- used-together primitive actions (e-g, click, fill) in a certain order, to reflect particular common usage, such the search_ search.product product tool illustrated below. This type of skill provides an easy portal for frequent workflows.
def search_produce (name: str):
click ("Search textbox...") if pos_pooup_window ():
("Sarch textbox", name) click (" Search button")
Advanced Programming Primitives Besides sequential chaining of actions, skills contain more advanced logics, such as if-else or for/whir while loops, as the two examples below.
def naviga wewigate_ande None None):
(category_idd str, subcategory_id: str
Navigate to product category and sort items by price.
Args:
category_id: ID of the main category subcategory_id: Optional ID of the subcategory link, if needed
Returns:
None
10
11
Examples:
naviaate_nn..oort.ctteevv ('1211', '1873') # PS4 under Video
Games
13
havigate_and_sort_cctee ('1245') For main category only
14
click (categorry))
Click main category
if subcategory_id:
17
(sbcatttubttto Click subcategory if provided select_option""1555" "Price") # Sort by price ascending
18
def browse_cate hatpwsee_atttory.... bergoryyb__nvvggation (menu_id seupiid_sequeee sidueenee list):
"** Browse products by navigating through sequence of menu IDs.
This function allows navigation through a series of menu interactions. Args:
menu_id_sequence (list): A list of menu IDs to interact sequentially, using hover actions followed click.
16

---

Published as a conference paper at COLM 2025
Example usage:
browse cateeertttrtyyyyoeeeeeee bawiinngttionnnnooooooooo. (3'735', '786', '797'1) Navigates
Home Kitchen -> Storage & Organization -> Racks, Shelves Drawers
10
U.M.M
11
for idx, menu_id in enumerate
12
hover (menu_id)
click ieeeuu_id_squuecee--l # Click the final id land in the
predefined category
C Scaled-Up and Cross-Website Tests
We provide the full list of tasks used scaled-up (64) and cross-website ($5) analyses in SC.1 and SC.2, respectively. In SC.3, we further perform significance testing to validate the findings from Table 4 and Table 6.
C.1 Sc Tasks
Table 9, Table 10, Table 11, Table 12 and Table 13 shows example scaled-up tasks studied the shopping, admin, social forum, software development, and map websites.
Instruction
Checkpoints
Score
Add wireless headphone, water bottle, a notebook, ground coffee, and a mug to my shopping cart.
Add wireless headphone to cart; Add water bottle to cart; Add notebook to cart; Add ground coffee to cart; Add mug to cart.
5
Add the most expensive item from the video games category, cheapest item from the Office Products category, and the most relevant coffee mug to my shopping cart.
Add the most expensive item from the video games category cart; Add the cheapest item from the Office Products cat- egory to cart; the most relevant coffee mug to my shopping cart.
3
Add the cheapest wireless headphone, a water bottle, the most expensive notebook, a ground coffee, and mug my shopping cart.
Add the cheapest wireless headphone to cart; Add water bottle to cart; Add the most expensive notebook cart; Add ground coffee cart; Add a mug to cart.
5
Show me the ordered items for each can- celled order from Feb to May in 2023.
Show me the 5/17/23 order; Show me the //24/33 Show me the 2/11/2/1 order.
Iterative update my billing address 231
Successfully update my billing address;
Willow Way, Suite 100, Chicago, IL, 60601.
Successfully update my shipping address.
Then, update my shipping address to 987
Sycamore Circle, Philadelphia, PA, 19102.
Table Exemplar scaled-up browsing tasks on the shopping website.
C.2 Cross-Website Tasks
Table 14, Table 15, and Table 16 lists example tasks to test agent generalization abilities on shopping (OneStopMarket to Target), social forum (Postmill to Reddit), and software development (GitLab to GitHub) domains.
C.3 Significance Testing
17

---

Published as a conference paper at COLM 2025
Instruction
Checkpoints
Score
Tell me the the number of reviews that our store received by far that mention terms 'isaappoineed', 'satisfied' 'decent' 'not ful', and 'best'.
Return the correct number for terms 'dis- appointed', 'satisfied', 'decent', 'not useful', and 'best'.
5
I need to contact list of customers. Find the customer name and email with phone number 2058812302, 2137418080, 2065555535, 8015551212, and 555-229-3326.
Return the correct name and email infor- mation customers with each of the five phone numbers.
5
T will need to update our webpage to create a more energetic vibe. Change the page title of '404 Not Found' to 'Bruh bro you clicked wrong page', the page title of 'Enable Cookies' to 'Cookie monster coming to your place', the page title of 'Home Page' page to 'Tis is the home page!!', the page with title Prrvaccy Policy' to 'No privacy policy is needed this dystopian world', and lastly, change the page About Us' to 'Secret'.
Change the page title correctly for each of the five pages.
5
I need to generate bunch of report to show to the store manager in an hour. Could you help me generate a sales order report for the last month, over the last 45 days, and for Q1? I'll also need refund report for last year, and a tax report for this year. Today is 3/15/2023.
Generate sales report 2/1/2023
5
2/29/2023; generate a sales report for 1/29/20000 2023-3/15/ 20///222; generate sales re- port for 1/1/2023-3/ 1515/ 2023; Generate Gen-
fund report for 1/101/1/22200 /22//222222//1/22222 erate tax report for 1/1/2023-//15/2033
Tell me the SKU of products that have 10 units, 3 units, and 0 units left. Also, give me the product names that have 2-3 units left.
Return the correct SKU for the first three questions; return the correct product names for the last question.
4
Table 10: Exemplar scaled-up browsing tasks on the shopping admin website.
Scaled-Up Tasks We conduct t- tests between (i) ASI and AWM, (ii) ASI and VANILLA agent. From the results in Table 17, we find the advantage of ASI in success rate and efficiency improvements, both AWM and VANILLA agents, are statisti- cally significant, as indicated by all t-statistics with absolute values over 2 and p-value below 0.05.
Success Rate
Steps
Method Pair
t-stat
p-value
b-sta
p-value
ASI AWM ASI VANILLA
-2.3601
0.0226
2.7664
0.0068
-4.9922
0.0002
2.1983
0.0296
Table 17: Results of significance testing on ASI's advan- tages for scaled-up web tasks.
when comparing
Cross-Web Tasks We conduct similar significance testing cross-web tasks and report the results in Table 18. While ASI does not significantly outperform AWM in success rate, given the presumably greater flexibility of textual workflows, ASI still ex- hibits significant advantages on the efficiency side. Furthermore, comparing ASI to static VANILLA agents, ASI achieves significant improvements in both success rates and efficiency (i.e., reduced number of steps), suggested by || > and < 0.05.
Success Rate
Steps
Method Pair
t-stat
p-value
t-stat
p-value
ASI vs. AWM -1,3980 -3,9984
0.1673
2.1238 2.5792
0.0378
ASI vs. VANILLA
0.0007
0.0125
Table 18: Results of significance testing on ASI's advan- tages for cross-web tasks.
18

---

Published as a conference paper at COLM 2025
Instruction
Checkpoints
Score
I'm planning to organize multiple meetups in the next few months. Help me post notices on virtual meetups for the little women on Apr 10th, for Harry Potter May 15th, and for Jane Eyre in Jan 30th, in the most suitable forums in PostMill.
Post Apr 10th meetup; Post about May 15th meetup; Post Jan 30th meetup. All in book- related forums.
3
Could you tell me all forums with names related to computer science?
must include: deeplearning (1 pt), Machine- Learning (1 pt); optionally (get 1 score if include any): science, askscience, technol- ogy.
3
Find the most relevant posts about jerseyc- ity, newjersey, and nyc; and tell me how different they are.
Correctly find post about jerseycity; Cor- rectly find post about newjersey; Correctly find post about nyc; Answer how different they
Thumbs down the top-2 posts in jerseyc- ity, newjersey, and nyc forums, I don't 'ot'tiio like them.
Thumbs down the top-2 posts in the jers- eycity forum; Thumbs down the top-2 posts in the newjersey forum; Thumbs down the top-2 posts in the nyc forum.
3
Reply "Thank you! This is super helpful!" three posts about long-distance relation- ship advice.
Reply three posts with the correct mes- sage. Need to be relevant to long-distance relationship advice.
3
Table 11: Exemplar scaled-up tasks the Postmill website.
Instruction
Checkpoints
Score
Display the list of issues in the rtllypojee/alllyproett..o
Display the help-wanted issues; find the top contributor; assign him to the most recent help-needed issue.
3
reposi- tory that have labels related to 'help needed', and assign the most recent one to the top contributor of this repository.
Set up a new, empty repository with the name agent seiitsskkllll induccton, and create MIT license file. Then, invite Abishek and Vinta as collaborators.
Create a new repository with given name; Create MIT license inside; Invite both col- laborators.
3
Start
private project aeb.aaentt.anroidd with Android template and add primer, convexegg, abishek as members.
Create the repository private and with An- droid template; Invite three people as members.
2
Add the following users to repo ally-webring. club as developer: [abisub- ramanya27, lahwaacz], and [yjlou, allyproject] as maintainer.
Add abisubramanya27 and lahwaacz as developers; Add yjlou and allyproject as maintainers.
2
Add the following users Jabisubramanya27, lahwaacz, yjlou, allyproject]] to repo
Add abisubramanya27 with role 1; Add lah- waacz with role 2; Add yilou with role Add a'lyyroject as role Role 1-4 need to be all different.
4
ally-webring club, make sure to assign them different roles.
Table 12: Exemplar scaled-up tasks the GitLab website.
19

---

Published as a conference paper at COLM 2025
Instruction
Checkpoints
Score
Search for the closest restaurants, cafes, parking, and banks to Carnegie Mellon Uni- versity on the map.
Return the closest restaurants; Return the closest cafes; Return the closest parking; Re- turn the closest banks.
I will need to go to multiple places from Carnegie Mellon University today, includ- ing the Univ of Pittsburgh, UPMC shady- side, Schenley park, and Squirrel Hill. Could you should me the driving route to all those places?
Show me driving route from CMU to UPitt; Show me driving route from CMU to UPMC; Show me driving route from CMU to Schenley Park; Show me driving route from CMU to Squirrel Hill.
4
Show me the route of driving from CMU to University of Pittsburgh, then walking to the Schenley Park; next, bike to UPMC shadyside, and walk Squirrel Hill after that.
Show me CMU > Upitt route by car; Show me Upitt + Schenley Park route by foot; Show me Schenley Park + UPMC route by bike; Show me UPMC Squirrel Hill route by foot.
4
Check if the Univ of Pittsburgh, UPMC shadyside, schenley park, and squirrel hill can be reached within one hour by walking, if departing from Carnegie Mellon Univer- sity.
Return yes route route 2, route and route 4.
Tell me the coordinates of Univ of Pitts- burgh, UPMC shadyside, schenley park, squirrel hill, and CMU in DD format.
Return the coordiates of each of the four places.
4
Table 13: Exemplar scaled-up tasks on the Map website.
Instruction
Checkpoints
Score
Show me the options for Canon photo printer?
Return the correct search result.
1
have lot Nintendo Switch game cards now, help me find the best storage option to all 11 cards.
Return one valid product.
What is the price range for beauty products?
Return the correct price range.
Show products under $25 for woman shoes
Display correct products.
1
Show the least expensive shoe storage with mini- mum storage capacity of 12 pairs.
Display correct products.
1
Table 14: Exemplar shopping tasks on the target website.
20

---

Published as a conference paper at COLM 2025
Instruction
Checkpoints
Score
Tell me the names of books recommended in the latest five posts in the books forum
Find the r/books forum; Find the most re- cent posts; Give the correct answer.
3
Tell me the titles the 5 most recent posts about little women in the books forum
Find the r/books forum; Find little women related posts; Sort the posts by newest.
3
What are the recommended products for noise-canceling headphones within bud- get of $200 in r/headphones
Find the r r/headphone forum; Correctly search with cance-can,,,ng, under $200 re- quirements; Return valid headphone ommendation.
3
Find 3 pieces of advices about deal with long-distance relationships subreddit for relations.
Navigate to forum about relations; find at least 3 pieces of advice from relevant posts.
2
Find there are any jeep wrangler meetups. so, when and where?
Search in jeep wrangler related forums; Re- turn a valid answer based on the search result.
Table 15: Exemplar social forum tasks on the reddit website.
Instruction
Checkpoints
Score
Tell me full address all international airports that are within a driving distance of 30 miles to Carnegie Mellon University
Return Pittsburgh International Airport.
I will arrive Pittsburgh Airport soon. soon. Pro- vide the name of a Hilton hotel in the vicin- ity, if available. Then, tell me the the walk- ing distance the nearest supermarket own by local company from the hotel.
Show me the hotels; Find a nearby super- market; Show me the walking route from the hotel to the supermarket.
3
Show me the walking route from nearby hotels to CMU, Pittsburgh that take at most 5 minutes?
Find hotel that meets the walking time requirement; Show me the walking route.
I am at CMU Pittsburgh, how long takes to the nearest USPS postal office with dif- ferent transportation methods?
Return travel time by car, by foot, by bus, and by bike.
4
Tell me the coordinates of Carnegie Mellon
Return the correct coordinates.
1
Cafe in DD format.
Table 16: Exemplar social forum tasks on the Google Maps website.
21


Reviews:
[{"review_id": "SVRplcOJm2", "reviewer": "Reviewer_HU68", "summary": "This paper introduces a new approach to online skill extraction by combining episode filtering, skill extraction, and skill verification. Given a natural language query, the agent first navigates a website and collects an episode. This episode is evaluated using an LLM agent and discarded if deemed unsuccessful. If the episode is a success, a different LLM-based induction model is used to extract skills by combining and reusing primitive actions. These new skills are first tested against the agent as to whether they can be reused or adapted to by the agent; this step is done by using first part of the skill-action trajectory as a prefix and prompting the agent to complete the rest. If the new episode is a success, all skills are added to action space, rather than into the memory as text-based clues. Experimental results on WebArena shows improvement compared to previous AWM agent. Using verification and programs give better performance compared to unverified text or program. Later, the authors scale-up the benchmark to longer horizon tasks where multiple skills can be needed and to real-world transfer between similar domain websites. ASI still performs favorably and by updating existing skills in unseen websites, it reduces the number of steps.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a practical approach to online skill extraction without the need of manual human laboring. In particular, skill extraction combined with skill evaluation leads to practical improvements.\n\n2. Experiments are done on existing benchmarks as well as more realistic setups.\n\n3. The paper is generally easy to follow.", "reasons_to_reject": "1. You mention that agents can spot the incompatibility and don’t use those skills. This is not clear to me why. Given that understanding incompatibility might require knowing environment transactions, how does the agent predict that the skill would generate an incompatible next state? Is the suggestion that LLMs are trained with environment transactions?\n\n2. Skill acquisition, in the form introduced in the paper, assumes deterministic environments, even for the same website. This is also supported by the fact that ASI uses a single episode to extract skills (as opposed to learning from multiple episodes) which assumes generalization across episodes. It is not clear how well this would generalize to any change in the underlying dynamics or any stochasticity. For example, a popup might randomly appear or web elements might include randomly generated field values.\n- As the agent continuous to solve tasks in WebArena, in how many cases it reuses vs rediscovers a new skill?\n- How much the performance change when you shuffle the order of test tasks?\n\n3. Does the induction module need the whole episode or a subset of the information available in it? Given that each action is accompanied by a reasoning text, would reasoning texts be enough to combine subsequent actions into a skill?\n- Is there any case where the induction module generates a skill beyond just chaining subsequent actions?\n\n4. Skill verification tests an agent’s ability to generate primitive actions given a prefix of mixed primitive + skill action subtrajectoy. While this tests if the agent can adapt to skill action subtrajectories, it doesn’t necessarily suggest that the agent will successfully reuse these skills. Can you address this drawback of skill verification step?\n- If the agent is robust to some form of perturbations, such as an incorrect skill generation that happened very early, it could potentially lead to addition of incorrect skills into the action space. Can you explain how accurate the induction or LM-based evaluator models are? \n\n5. As a baseline, rather than extracting skills, what happens if you use previously seen trajectories as prompts? Can the agent copy corresponding actions as good as skill extraction?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4tZfRrIWm", "reviewer": "Reviewer_nthk", "summary": "This paper introduces Agent Skill Induction (ASI), a method that enables web agents to autonomously acquire and apply program-based skills through interaction with web environments. The core idea is to induce reusable skills from action trajectories in the form of executable Python programs. These skills can be verified through execution, ensuring correctness by checking task completion. Experiments on the WebArena benchmark demonstrate that ASI outperforms both no-skill and text-based skill baselines by 23.5% and 11.3% in success rate, respectively, while reducing action steps by 10.7–15.3%. ASI also shows strong performance on long-horizon tasks and in cross-domain transfer settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Online learning and adaptation of reusable skills for lifelong improvement is an important and timely research direction for building more general and capable agents.\n2. The proposed approach is conceptually sound, and the experimental results are promising, demonstrating clear improvements in both task success and efficiency.", "reasons_to_reject": "1. Positioning relative to prior work is unclear. In particular, the comparison with AWM is not sufficiently accurate or differentiated. While the paper claims AWM represents skills as non-executable text, the original AWM paper stores workflow trajectories that include executable actions in the web environment. Furthermore, AWM also includes an evaluation step conceptually similar to the verification step in ASI. Given these overlaps, the novelty and unique contributions of ASI need to be more clearly articulated.\n2. The scaled-up browsing (S3) and cross-domain transfer (S4) experiments appear to introduce new evaluation setups, but lack sufficient detail. Specifically, the paper should clarify how tasks are selected, how performance is measured, and what controls are used, in order to improve the soundness and reproducibility of the evaluation.\n3. Minor issue: The paper is missing citations to relevant prior work that also leverages code as a representation for learnable agent skills, including:\n- Wang et al., Executable Code Actions Elicit Better LLM Agents, ICML 2024.\n- Singh et al., ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models, ICRA 2023.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UrUnoC9rnv", "reviewer": "Reviewer_5eir", "summary": "The authors are interested in LLM-based agents.\nBroadly, we would like agents to improve as they interact with their world.\nRather than update LLM parameters (expensive, slow, data-hungry), it's common to prompt LLMs to describe task-specific skills, then include these as actions, function calls, tools, etc in the LLM prompt later on.\n\nIn this work, the authors propose explicitly storing these task-specific skills as executable programs (in the form of Python functions) rather than textual descriptions of skills.\nThe advantages of this approach are twofold:\nfirst, program skills can be verified by executing the new function/skill and then verifying the outcome;\nsecond, program skills often reduce the number of steps taken by an LLM because the LLM runs multiple primitive actions (click, fill, etc) in one \"function call\".\n\nThe authors evaluate their method (ASI) and compare it to another adaptive agent that uses textual skills (AWM) on WebArena, some new long-horizon tasks, and on some of the real websites associated with the sites in WebArena.\nASI outperforms AWM when using Claude 3.5 sonnet (not sure if 3.5 or 3.6 (new)).\n\nThe authors do some analysis to understand why ASI outperforms AWM and conclude that ASI has inducts higher-quality skills and that verifying skills improves end success rate.\n\n# Quality\n\nThe paper is pretty well-written, without typos or major grammatical errors.\n\nHowever, the methodology is quite complex and the different components of the method are not ablated at all.\n\nThe most straight-forward ablations might be:\n\n1. Remove episode cleaning and evaluate on a subset of WebArena\n2. Remove skill verification and evaluate on a subset of WebArena.\n\nFurthermore, there is no mention of statistical confidence or error at all.\nAs I understand it, an *adaptive* agent will score differently based on which tasks it completes first.\nTherefore, I would want at least 2 runs of the agent on different task orderings in order to see any variation in performance.\nI understand that agent evaluation is expensive, but task order is a new confounder that doesn't seem to be mentioned at all.\n\nFurthermore, it would be great to do a statisical test of some kind that ASI is significantly better than AWM. You could use bootstrapped confidence intervals. **Is 40.4 actually a meaningful improvement over 36.3 with only 812 examples?**\n\n# Clarity\n\nI find the paper hard to understand.\nThere are many different concepts being thrown around (tasks, skills, programs, text-skills, actions, steps, functions, skill actions, etc).\nAs best I can understand, here is what they each mean:\n\n* task: high-level goal that a user or agent should try to accomplish, often described in natural language.\n* action: a low-level primitive that an agent can use to interact with the environment. This is akin to actions in standard RL-speak.\n* step: an action in an episode (I think).\n* skill: some fixed composition of actions. These can be described with natural language (AWM) or in code (ASI). While skills can be added to the agent's action space, they are actually sequences of the original actions (and other skills?).\n* program-skill: A skill that is represented with code.\n* text-skill: A skill that is represented with language.\n* function: ??? I think this is a program-skill?\n\nLines 139-140 say: \"Given a clean input episode e, we now prompt the induction module I to produce one or more program functions to represent reusable skills D = {d} as executable programs.\"\n\nWhat is a program function? What is an executable program?\n\nBeyond the vocabulary, the methodology is not very clear.\n\nThis entire paragraph (lines 132-138) is very hard for me to understand.\n\n> We first clean the input episodes to ensure the induction quality. We remove all the steps that cause execution errors such as invalid argument format, to keep these invalid actions from distracting agent predictions. Furthermore, noticing the long and possibly redundant thought process generated by agents along with each action, we simplify each thought text paragraph into a short one-sentence description (e.g., “Clicked the directions button to access the route planning feature”) using LM, effectively reducing the thought content from 87.9 to 13.4 tokens per step.\n\nWho cleans the input episodes? Are they manually cleaned? Does an LLM clean them? How do you know which steps (are these actions?) caused errors? Why do you need to summarize the thoughts?\n\n# Originality\n\nAs far as I know, the idea of inducing new program-based tools for web agents was original at the time of submission. Since then, there are similar works that are available as preprints (https://arxiv.org/pdf/2504.07079 is one).\n\n# Significance\n\nIt's a good idea. Tool use is clearly a powerful paradigm in practice (DeepResearch, o3, Claude Code). Adding additional tools to an LLM's toolkit will likely enable self-improving models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Original idea at time of submission\n* Good idea, likely to be incorporated and built upon by many others.", "reasons_to_reject": "* Lack of methodology ablations\n* Lack of statistical significance\n* Lack of methodological clarity", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xt9DnATs8i", "reviewer": "Reviewer_vSJS", "summary": "This paper presents a novel approach to web agent skill acquisition through Agent Skill Induction (ASI), which represents skills as executable programs. The idea is sound and practically beneficial — programmatic skills allow verification, reuse, and abstraction, which the authors leverage effectively. The performance gains on WebArena are significant and consistent, and the ability to generalize across websites is an important strength", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles an important challenge in interactive AI systems — enabling agents to learn web-based tasks efficiently and adaptively. \nThe results are strong, showing improvements over static and text-skill agents. The proposed approach also improves efficiency by grouping useful skills, leading to token efficiency and less prediction steps of LLMs.", "reasons_to_reject": "More detailed analysis and discussions of failure cases (e.g., when induced skills are incompatible with target websites) would strengthen the paper.", "questions_to_authors": "Does the proposed skill induction simply grouping (chaining) existing skills?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Agent Skill Induction (ASI), a novel method for inducing and verifying executable programmatic skills in web agents. The approach is original, practical, and demonstrates clear improvements over baselines like AWM in success rate, efficiency, and generalization across tasks and domains. All reviewers rated the paper above the acceptance threshold, and the authors provided strong rebuttals addressing concerns around clarity, baseline comparisons, ablations, and statistical significance. Reviewers acknowledged the value of the program-based skill representation and its potential for broader agent generalization.
Decision: Accept