Patent ID: 8893048

Claim:
A system for manipulating virtually displayed objects using a virtual agent represented in a virtual environment, comprising: a display; at least one input device adapted to receive at least one of speech, gesture, text and touchscreen inputs; and a computer processor adapted to execute a program stored in a computer memory, the program being operable to provide instructions to the computer processor including: receiving user input via the at least one input device, wherein the user input underspecifies a command for a virtual agent within the virtual environment to use in moving at least one target object in the virtual environment; interfacing with the virtual environment via the virtual agent; sensing, by the virtual agent, the at least one object in the virtual environment; and finding at least one valid location for the virtual agent to place the at least one object in the virtual environment, wherein determining at least one valid location includes retrieving a linguistic placement constraint, identifying the virtual agent's intent for a target object, identifying a placement preference for the virtual agent, determining one or more object properties of the target object and determining candidate placement surfaces for placement of the target object, determining at least one candidate activity surface on which the virtual agent is located while interacting with the at least one target object, and, for all candidate placement surfaces and the at least one candidate activity surface, identifying all objects in the virtual environment that are in contact with any of the candidate placement surfaces or the at least one candidate activity surface.