Learning Language Games through Interaction
Paper
•
1606.02447
•
Published
UI Agent
a collection of algorithmic agents for user interfaces/interactions and program synthesis
-
-
Naturalizing a Programming Language via Interactive Learning
Paper • 1704.06956 • Published
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Paper • 1802.08802 • PublishedNote Introducing the Workflow-Guided Exploration (WGE) framework, and a new synthetic dataset called MiniWoB++ (an extension of MiniWoB, a dataset proposed by "World of Bits: An Open-Domain Platform for Web-Based Agents" at ICML 2017). This work was published at ICLR 2018.
-
Mapping Natural Language Commands to Web Elements
Paper • 1808.09132 • Published -
Learning to Navigate the Web
Paper • 1812.09195 • Published -
Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations
Paper • 1909.00031 • Published -
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
Paper • 1909.01871 • Published -
Mapping Natural Language Instructions to Mobile UI Action Sequences
Paper • 2005.03776 • Published -
FLIN: A Flexible Natural Language Interface for Web Navigation
Paper • 2010.12844 • Published -
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Paper • 2012.12350 • Published -
Grounding Open-Domain Instructions to Automate Web Support Tasks
Paper • 2103.16057 • Published -
Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments
Paper • 2104.08560 • Published -
AndroidEnv: A Reinforcement Learning Platform for Android
Paper • 2105.13231 • Published -
UIBert: Learning Generic Multimodal Representations for UI Understanding
Paper • 2107.13731 • Published -
Benchmarking the Spectrum of Agent Capabilities
Paper • 2109.06780 • Published -
Compositional Generalization for Natural Language Interfaces to Web APIs
Paper • 2112.05209 • Published -
VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling
Paper • 2112.05692 • Published -
A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility
Paper • 2202.02312 • Published -
Inferring Rewards from Language in Context
Paper • 2204.02515 • Published -
META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI
Paper • 2205.11029 • Published -
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Paper • 2206.08853 • Published • 1 -
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Paper • 2207.01206 • Published • 2 -
Enabling Conversational Interaction with Mobile UI using Large Language Models
Paper • 2209.08655 • Published -
Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus
Paper • 2209.14927 • Published -
ReAct: Synergizing Reasoning and Acting in Language Models
Paper • 2210.03629 • Published • 14 -
Understanding HTML with Large Language Models
Paper • 2210.03945 • Published • 1 -
Lexi: Self-Supervised Learning of the UI Language
Paper • 2301.10165 • Published -
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics
Paper • 2301.13280 • Published -
Language Models can Solve Computer Tasks
Paper • 2303.17491 • Published -
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
Paper • 2305.11854 • Published • 4 -
Gorilla: Large Language Model Connected with Massive APIs
Paper • 2305.15334 • Published • 4 -
Voyager: An Open-Ended Embodied Agent with Large Language Models
Paper • 2305.16291 • Published • 9 -
Decision-Oriented Dialogue for Human-AI Collaboration
Paper • 2305.20076 • Published -
From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces
Paper • 2306.00245 • Published -
Mind2Web: Towards a Generalist Agent for the Web
Paper • 2306.06070 • Published • 19 -
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Paper • 2306.07863 • Published -
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Paper • 2307.01928 • Published • 10 -
Android in the Wild: A Large-Scale Dataset for Android Device Control
Paper • 2307.10088 • Published • 10 -
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis
Paper • 2307.12856 • Published • 35 -
WebArena: A Realistic Web Environment for Building Autonomous Agents
Paper • 2307.13854 • Published • 23 -
Learning to Model the World with Language
Paper • 2308.01399 • Published • 34 -
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework
Paper • 2308.08155 • Published • 3 -
Never-ending Learning of User Interfaces
Paper • 2308.08726 • Published • 2 -
Empowering LLM to use Smartphone for Intelligent Task Automation
Paper • 2308.15272 • Published • 1 -
LASER: LLM Agent with State-Space Exploration for Web Navigation
Paper • 2309.08172 • Published • 11 -
You Only Look at Screens: Multimodal Chain-of-Action Agents
Paper • 2309.11436 • Published • 1 -
SmartPlay : A Benchmark for LLMs as Intelligent Agents
Paper • 2310.01557 • Published • 12 -
AXNav: Replaying Accessibility Tests from Natural Language
Paper • 2310.02424 • Published -
SteP: Stacked LLM Policies for Web Actions
Paper • 2310.03720 • Published • 7 -
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
Paper • 2310.04406 • Published • 8 -
Automatic Macro Mining from Interaction Traces at Scale
Paper • 2310.07023 • Published -
A Zero-Shot Language Agent for Computer Control with Structured Reflection
Paper • 2310.08740 • Published • 14 -
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Paper • 2310.10134 • Published • 1 -
OpenAgents: An Open Platform for Language Agents in the Wild
Paper • 2310.10634 • Published • 8 -
WebWISE: Web Interface Control and Sequential Exploration with Large Language Models
Paper • 2310.16042 • Published -
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Paper • 2311.01767 • Published • 18 -
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation
Paper • 2311.07562 • Published • 12 -
GAIA: a benchmark for General AI Assistants
Paper • 2311.12983 • Published • 183 -
Language Model Agents Suffer from Compositional Generalization in Web Automation
Paper • 2311.18751 • Published -
CogAgent: A Visual Language Model for GUI Agents
Paper • 2312.08914 • Published • 29 -
UINav: A Practical Approach to Train On-Device Automation Agents
Paper • 2312.10170 • Published -
ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
Paper • 2312.13108 • Published • 3 -
AppAgent: Multimodal Agents as Smartphone Users
Paper • 2312.13771 • Published • 51 -
WebVLN: Vision-and-Language Navigation on Websites
Paper • 2312.15820 • Published -
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Paper • 2401.01614 • Published • 21 -
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
Paper • 2401.10935 • Published • 4 -
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
Paper • 2401.13919 • Published • 26 -
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Paper • 2401.13649 • Published • 1 -
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception
Paper • 2401.16158 • Published • 18 -
DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design
Paper • 2402.03479 • Published -
Graph-enhanced Large Language Models in Asynchronous Plan Reasoning
Paper • 2402.02805 • Published • 1 -
RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
Paper • 2402.03681 • Published -
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls
Paper • 2402.04253 • Published -
Dual-View Visual Contextualization for Web Navigation
Paper • 2402.04476 • Published -
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Paper • 2402.04615 • Published • 38 -
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
Paper • 2402.05930 • Published • 39 -
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Paper • 2402.07456 • Published • 41 -
Training Language Model Agents without Modifying Language Models
Paper • 2402.11359 • Published -
On the Multi-turn Instruction Following for Conversational Web Agents
Paper • 2402.15057 • Published -
Language Agents as Optimizable Graphs
Paper • 2402.16823 • Published • 3 -
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
Paper • 2402.17553 • Published • 21 -
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
Paper • 2403.01248 • Published • 2 -
KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents
Paper • 2403.03101 • Published • 1 -
Towards General Computer Control: A Multimodal Agent for Red Dead Redemption II as a Case Study
Paper • 2403.03186 • Published • 5 -
WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
Paper • 2403.07718 • Published • 1 -
BAGEL: Bootstrapping Agents by Guiding Exploration with Language
Paper • 2403.08140 • Published -
AgentStudio: A Toolkit for Building General Virtual Agents
Paper • 2403.17918 • Published -
Octopus v2: On-device language model for super agent
Paper • 2404.01744 • Published • 57 -
AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent
Paper • 2404.03648 • Published • 24 -
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Paper • 2404.05719 • Published • 80 -
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Paper • 2404.05955 • Published -
Autonomous Evaluation and Refinement of Digital Agents
Paper • 2404.06474 • Published • 1 -
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Paper • 2404.07972 • Published • 46 -
MMInA: Benchmarking Multihop Multimodal Internet Agents
Paper • 2404.09992 • Published -
AgentKit: Flow Engineering with Graphs, not Coding
Paper • 2404.11483 • Published • 2 -
UIClip: A Data-driven Model for Assessing User Interface Design
Paper • 2404.12500 • Published -
AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation
Paper • 2404.12753 • Published • 41 -
Benchmarking Mobile Device Control Agents across Diverse Configurations
Paper • 2404.16660 • Published -
Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
Paper • 2405.00451 • Published -
Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
Paper • 2405.10292 • Published • 1 -
Latent State Estimation Helps UI Agents to Reason
Paper • 2405.11120 • Published -
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Paper • 2405.14573 • Published -
Devil's Advocate: Anticipatory Reflection for LLM Agents
Paper • 2405.16334 • Published -
Position: Foundation Agents as the Paradigm Shift for Decision Making
Paper • 2405.17009 • Published -
Can Graph Learning Improve Planning in LLM-based Agents?
Paper • 2405.19119 • Published -
Large Language Models Can Self-Improve At Web Agent Tasks
Paper • 2405.20309 • Published • 2 -
Controlling Large Language Model Agents with Entropic Activation Steering
Paper • 2406.00244 • Published -
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Paper • 2406.01014 • Published • 31 -
On the Effects of Data Scale on Computer Control Agents
Paper • 2406.03679 • Published -
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Paper • 2406.04520 • Published • 11 -
AgentGym: Evolving Large Language Model-based Agents across Diverse Environments
Paper • 2406.04151 • Published • 17 -
Synthetic Query Generation using Large Language Models for Virtual Assistants
Paper • 2406.06729 • Published -
VersiCode: Towards Version-controllable Code Generation
Paper • 2406.07411 • Published -
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
Paper • 2406.07739 • Published • 2 -
StreamBench: Towards Benchmarking Continuous Improvement of Language Agents
Paper • 2406.08747 • Published • 1 -
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Paper • 2406.08451 • Published • 23 -
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Paper • 2406.11896 • Published • 18 -
VideoGUI: A Benchmark for GUI Automation from Instructional Videos
Paper • 2406.10227 • Published • 9 -
GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents
Paper • 2406.10819 • Published -
GUICourse: From General Vision Language Models to Versatile GUI Agents
Paper • 2406.11317 • Published • 1 -
WebCanvas: Benchmarking Web Agents in Online Environments
Paper • 2406.12373 • Published -
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Paper • 2406.12045 • Published • 6 -
Adversarial Attacks on Multimodal Agents
Paper • 2406.12814 • Published • 4 -
ICAL: Continual Learning of Multimodal Agents by Transforming Trajectories into Actionable Insights
Paper • 2406.14596 • Published • 5 -
Symbolic Learning Enables Self-Evolving Agents
Paper • 2406.18532 • Published • 11 -
APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
Paper • 2406.18518 • Published • 23 -
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding
Paper • 2406.19263 • Published • 9 -
BMW Agents -- A Framework For Task Automation Through Multi-Agent Collaboration
Paper • 2406.20041 • Published -
Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
Paper • 2407.00993 • Published • 1 -
CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
Paper • 2407.01511 • Published -
AI Agents That Matter
Paper • 2407.01502 • Published • 2 -
Tree Search for Language Model Agents
Paper • 2407.01476 • Published -
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Paper • 2407.17490 • Published • 30 -
AgentInstruct: Toward Generative Teaching with Agentic Flows
Paper • 2407.03502 • Published • 48 -
WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks
Paper • 2407.05291 • Published • 1 -
LLM-Based Open-Domain Integrated Task and Knowledge Assistants with Programmable Policies
Paper • 2407.05674 • Published -
Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models
Paper • 2407.07086 • Published -
GTA: A Benchmark for General Tool Agents
Paper • 2407.08713 • Published • 14 -
TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations
Paper • 2407.08464 • Published -
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Paper • 2407.08898 • Published -
Sibyl: Simple yet Effective Agent Framework for Complex Real-world Reasoning
Paper • 2407.10718 • Published • 17 -
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Paper • 2407.10956 • Published • 6 -
Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems
Paper • 2407.13032 • Published -
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Paper • 2407.18961 • Published • 39 -
WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation
Paper • 2407.15502 • Published -
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Paper • 2407.15711 • Published • 9 -
OpenDevin: An Open Platform for AI Software Developers as Generalist Agents
Paper • 2407.16741 • Published • 68 -
AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
Paper • 2407.18901 • Published • 32 -
Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning
Paper • 2407.20798 • Published • 23 -
OmniParser for Pure Vision Based GUI Agent
Paper • 2408.00203 • Published • 23 -
AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation
Paper • 2408.00764 • Published • 1 -
A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model
Paper • 2408.02920 • Published -
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities
Paper • 2408.04682 • Published • 14 -
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents
Paper • 2408.06327 • Published • 15 -
Automated Design of Agentic Systems
Paper • 2408.08435 • Published • 38 -
WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration
Paper • 2408.15978 • Published -
xLAM: A Family of Large Action Models to Empower AI Agent Systems
Paper • 2409.03215 • Published • 1 -
Game On: Towards Language Models as RL Experimenters
Paper • 2409.03402 • Published -
Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments
Paper • 2409.05865 • Published • 14 -
Agent Workflow Memory
Paper • 2409.07429 • Published • 27 -
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Paper • 2409.08264 • Published • 43 -
EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
Paper • 2409.11295 • Published -
The Impact of Element Ordering on LM Agent Performance
Paper • 2409.12089 • Published -
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Paper • 2409.20566 • Published • 52 -
Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale
Paper • 2409.15637 • Published -
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Paper • 2410.00201 • Published -
NNetscape Navigator: Complex Demonstrations for Web Agents Without a Demonstrator
Paper • 2410.02907 • Published -
MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents
Paper • 2410.03450 • Published • 36 -
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
Paper • 2410.05243 • Published • 16 -
TinyClick: Single-Turn Agent for Empowering GUI Automation
Paper • 2410.11871 • Published • 2 -
AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction
Paper • 2410.19743 • Published • 1 -
Agent S: An Open Agentic Framework that Uses Computers Like a Human
Paper • 2410.08164 • Published • 24 -
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating UI Operation Impacts
Paper • 2410.09006 • Published • 1 -
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
Paper • 2410.13232 • Published • 40 -
Harnessing Webpage UIs for Text-Rich Visual Understanding
Paper • 2410.13824 • Published • 29 -
AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents
Paper • 2410.13825 • Published • 1 -
MobA: A Two-Level Agent System for Efficient Mobile Task Automation
Paper • 2410.13757 • Published • 31 -
GUIDE: Real-Time Human-Shaped Agents
Paper • 2410.15181 • Published -
Improve Vision Language Model Chain-of-thought Reasoning
Paper • 2410.16198 • Published • 17 -
Beyond Browsing: API-Based Web Agents
Paper • 2410.16464 • Published -
MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control
Paper • 2410.17520 • Published -
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Paper • 2410.19168 • Published • 19 -
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant
Paper • 2410.18603 • Published • 30 -
Infogent: An Agent-Based Framework for Web Information Aggregation
Paper • 2410.19054 • Published -
Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Paper • 2410.18967 • Published • 1 -
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
Paper • 2410.19100 • Published • 6 -
OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization
Paper • 2410.19609 • Published • 16 -
EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
Paper • 2410.19461 • Published -
OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Paper • 2410.23218 • Published • 46 -
Evaluating Cultural and Social Awareness of LLM Web Agents
Paper • 2410.23252 • Published -
DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
Paper • 2411.01606 • Published -
Attacking Vision-Language Computer Agents via Pop-ups
Paper • 2411.02391 • Published -
CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments
Paper • 2411.02305 • Published -
WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
Paper • 2411.02337 • Published • 36 -
HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems
Paper • 2411.02959 • Published • 62
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents
Paper • 2411.06559 • Published • 9Note This paper presents WebDreamer, an agent that synthesizes a single-step planning horizon to improve agent planning capabilities.
-
The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use
Paper • 2411.10323 • Published • 26