Selective interactive mapping of real-world objects to create interactive virtual-world objects

A method for interactively defining a virtual-world space based on real-world objects in a real-world space is disclosed. In one operation, one or more real-world objects in the real-world space is captured to define the virtual-world space. In another operation, one of the real-world objects is identified, the identified object is to be characterized into a virtual-world object. In yet another operation, a user is prompted for user identification of one or more object locations to enable extraction of parameters for real-world object, and the object locations are identified relative to an identifiable reference plane in the real-world space. In another operation, the extracted parameters of the real-world object may be stored in memory. The virtual-world object can then be generated in the virtual world space from the stored extracted parameters of the real-world object.

BACKGROUND

Description of the Related Art

The video game industry has seen many changes over the years. As computing power has expanded, developers of video games have likewise created game software that takes advantage of these increases in computing power. To this end, video game developers have been coding games that incorporate sophisticated operations and mathematics to produce a very realistic game experience.

Example gaming platforms, may be the Sony Playstation or Sony Playstation2 (PS2), each of which is sold in the form of a game console. As is well known, the game console is designed to connect to a monitor (usually a television) and enable user interaction through handheld controllers. The game console is designed with specialized processing hardware, including a CPU, a graphics synthesizer for processing intensive graphics operations, a vector unit for performing geometry transformations, and other glue hardware, firmware, and software. The game console is further designed with an optical disc tray for receiving game compact discs for local play through the game console. Online gaming is also possible, where a user can interactively play against or with other users over the Internet.

As game complexity continues to intrigue players, game and hardware manufacturers have continued to innovate to enable additional interactivity and computer programs. Some computer programs define virtual-worlds. A virtual-world is a simulated environment in which users may interact with each other via one or more computer processors. Users may appear on a video screen in the form of representations referred to as avatars. The degree of interaction between the avatars and the simulated environment is implemented by one or more computer applications that govern such interactions as simulated physics, exchange of information between users, and the like. The nature of interactions among users of the virtual-world is often limited by the constraints of the system implementing the virtual-world.

It is within this context that embodiments of the invention arise.

SUMMARY

Broadly speaking, the present invention fills these needs by allowing users to interactively generate and define geometric attributes of real-world objects, which can be displayed on a screen and rendered in and along with virtual-world objects. By interactively generating virtual-world objects, users can customize objects within a virtual environment based on the user's real-world environment. The generated virtual-world objects can also interact with other virtual objects within the virtual environment. The virtual-world can be traveled, visited, and interacted with by multiple users via internet enabled computers. The real-world users control avatars (e.g. virtual persons) with a variety of user input methods such as, but not limited to, controllers, motion detection, and sound detection.

In one embodiment, a method for interactively defining a virtual-world space based on real-world objects in a real-world space is disclosed. In one operation, one or more real-world objects in the real-world space is captured to define the virtual-world space. In another operation, one of the real-world objects is identified, the identified object is to be characterized into a virtual-world object. In yet another operation, a user is prompted for user identification of one or more object locations to enable extraction of parameters for real-world object, and the object locations are identified relative to an identifiable reference plane in the real-world space. In another operation, the extracted parameters of the real-world object may be stored in memory. The virtual-world object can then be generated in the virtual world space from the stored extracted parameters of the real-world object.

In another embodiment, a system for creating virtual objects by interactively capturing parameters of real-world objects is disclosed. The system includes a computer system with a processor, an input/output interface, and memory. The memory of the computer system can be configured to store instructions for execution by the processor. The system can also include a capture system coupled to the input/output interface of the computer system. The capture system can be configured to capture images of one or more real-world objects of a real space. Also include in the system is a user interface for the computer system. The user interface may be used to setup the capture system to identify a particular object. The system further includes a screen coupled to the input/output interface of the computer system and the screen can be configured to display the user interface and images from the capture system. The system also includes logic stored in the memory and executed by the processor. The logic may be configured to process input to the user interface, where the input can include identification signals to selectively identify locations of the real-world object relative to an identifiable reference plane. The identified locations can then be used to define parameters that define geometric aspects of a generated virtual-world object.

In yet another embodiment, a method for interactively defining a virtual-world space based on real-world objects in a real-world space is disclosed. The method captures one or more real-world objects in the real-world space and the captured real-world object can be used to define the virtual-world space. In another operation, the method identifies one of the real-world objects to be characterized into a virtual-world object. In yet another operation, the method prompts a user to identify of one or more transition boundaries, where the transition boundaries are identified relative to an identifiable reference plane in the real-world space. In still another operation, the method determines geometric aspects of the real-world object from the transition boundaries. In another operation, the method stores the geometric aspects of the real-world object in memory and generates a virtual-world object in the virtual-world space from the stored geometric aspects of the real-world object.

In still another embodiment, a method for interactively defining a virtual-world space based on real-world objects in a real-world space is disclosed. The method captures one or more real-world objects in the real-world space to define the virtual-world space. In another operation, the method identifies one of the real-world objects to be characterized into a virtual-world object. In another operation, the method prompts a user to identify one or more surface locations of the identified real-world object. The surface locations of the identified real-world object can be used to extract geometric aspects of the real-world object, and to also identify the surface locations relative to an identifiable reference plane in the real-world space. In another operation, the method stores the extracted relative geometric aspects of the real-world object in memory and generates a virtual-world object in the virtual-world space using the stored geometric aspects of the real-world object.

DETAILED DESCRIPTION

An invention is disclosed for computer implemented methods and systems that enable capture of geometric feature attributes of real-world objects. The captured geometric attributes are analyzed and mapped to generate a virtual-word world object. The generated virtual-world object can then be incorporated into a computer game environment, where computer generated objects, characters, environments and game functionality can co-exist to deliver an interactive experience. The interactive experience is thus enriched, as users are allowed to capture many types objects, parameters of objects, environments and known geometric shapes, to thus inject them into a computer generated virtual space or computer executed interactive gaming space. In aspects of the present invention, some embodiments allow captured objects to be placed into specific game sequences, or generically in virtual environments that can be visited or traversed in response to user interactive control, action or interaction.

Additionally, the captured objects may be associated and mapped to known objects, such that users are able to select objects to capture, and the capturing processing walks the user through commands to systematically identify aspects of the object until the computing system believes that the object has been sufficiently identified by the user so as to identify the type of object, geometries of the object (e.g., edges, surfaces, textures, sizes, relative spacing features, and other identifiable characteristics). Accordingly, the capturing sequences, methods and systems should be broadly understood to enable the capture of any particular real-world object, discern its characteristics, identify relative sizes, and enable integration of the captured object or objects into scenes of a virtual space, gaming sequences, interactive programs, or displays of animation.

As used herein, a real-world object should include any physical or material thing that can be touched, held, moved, captured in an image, captured in a video, compared to other things to discern its size or relative size, or identified based on height, width, length, or depth, and the like. A virtual-world object shall be broadly construed to include a computer generated image or images that can be displayed on a screen. The screen can represent the virtual-world object as a two or three dimensional thing and can be animated to move, be placed, be interacted with, or be modified based on user interactivity. The interactivity can include commands provided by the user, such as by way of a computing interface. The interface can be graphical, key-board assisted, touch screen assisted, gaming controller directed, motion triggered, audibly triggered, acoustically triggered, inertial triggered, and combinations thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to unnecessarily obscure the present invention.

FIG. 1Ais an exemplary block diagram illustrating a computer system100that can be used to interactively create a virtual-world object based on a real-world object, in accordance with one embodiment of the present invention. The computer system100can include a capture system102that has hardware and software capable of capturing audio input, video input, along with user input via dedicated input devices. For example, the capture system102can include an array of microphones for capturing audio input along with a video camera to capture video input. In other embodiments, the video camera can be a depth camera configured to determine relative depth of objects. The microphones and video camera of the capture system can be used to capture user input104and images of real-world objects106. The capture system can also be used to accept user input that can be used to create a virtual-world object based on images, sounds and dedicated user input from a user interacting with a selected real-world object.

Images, sounds, and user input from the capture system102can be processed by identification and mapping logic108. The identification and mapping logic108can be used to process and identify particular user input104captured by the capture system108. In some embodiments, the user input104can include combinations of images and sounds such as a user snapping his fingers, moving his hand, shaking his wrist, wagging his finger, tapping on an object, or the like. In other embodiments, the user can provide input to the computer system100through a dedicated input device such as a console controller. The console controller can be configured to accept a variety of user input include buttons, triggers, and joysticks. In other embodiments, the console controller can be configured to detect translational and rotational movement, or combinations thereof, of the console controller in three-axis. In additional embodiments, the video camera of the capture system can be used to detect infrared light emissions from an array of light emitting diodes associated with the console controller. Using a variety of user input104associated with object locations on the real-world object, the identification and mapping logic can identify surfaces, edges, and points of real-world object to the computer system100.

Additionally, the identification and mapping logic108can be associated with a database of object templates. The object templates can define generic surfaces, edges or points that can be used to define a real-world object. Using the object templates, the identification and mapping logic108can prompt a user to define particular points, edges, surfaces and other transition boundaries of a real-world object. For example, using an object template for a chair, the identification and mapping logic can prompt a user to identify a right and a left side of the chair. Similarly, the object template can prompt the user to identify a height of the seating surface and overall height of the chair. In another embodiment, instead of defining surfaces or sides, the object template can request a user to identify points of the seating surface such as the front-left, front-right, back-left, and back-right corners of the seating surface. Note that the particular example of the chair is intended the be exemplary and should not be construed as limiting as the object templates can encompass an endless variety of real-world objects that can be defined via the capture system102and the identification and mapping logic108.

From the defined transition boundaries, the identification and mapping logic can generate a virtual-world object with geometric attributes based on the real-world object. The identification and mapping logic can include software and hardware for comparing and analyzing the defined surfaces, edges, and points to the object template in order to generate the virtual-world object. In the previously discussed chair example, the identification and mapping logic can compare the defined front-left, front-right, back-left and back-right corners of a seating surface to an object template and determine the relative position of the seating surface.

Virtual-world objects generated from the identification and mapping logic108can be saved in memory110. In some embodiments, the memory110can be a database used to store a plurality of virtual-world objects defined by geometric attributes of real-world objects. Additionally, the computer system100can include additional memory and storage to execute interactive application112that are configured to use the geometric attributes of stored virtual-world objects. In some embodiments, the interactive applications include other virtual objects that are configured to interact with the generated virtual objects.

FIG. 1Bis a flowchart illustrating operations to map object locations of a real-world object to create a virtual-world object in accordance with one embodiment of the present invention. In accordance with the embodiment of the computer system100ofFIG. 1A, the operations performed inFIG. 1Bwould be performed using the capture system102, user input104, real-world objects106and the identification and mapping logic108. In operation114a user selects a type of real-world object that is going to be mapped by geometric attributes. The selection can be performed from user input to the capture system. Operation116retrieves the geometric attributes that can be used by the identification and mapping logic to generate a virtual-world object based on the selected real-world object.

Operation118prompts the user to define one of the retrieved geometric attributes on the real-world object via user input through the capture system. Operation120captures the user defining the geometric attribute that was prompted in operation118. In operation122, it is determined if all the geometric attributes of the selected real-world object have been defined. If additional geometric attributes are needed, the procedure returns to operation118. If enough geometric attributes have been defined so the identification and mapping logic can generate a virtual-world object, operation124saves the geometric attributes for the mapped object to the memory110.

FIG. 1Cis a flow chart illustrating operations of an interactive application112in accordance with one embodiment of the present invention. Operation126captures an image of a scene in front of the capture system. Operation128is used to load virtual-world objects stored in memory. Operation120is used to compare the virtual-world objects loaded from memory to the current image of the scene in front of the capture system. Operation132determines if there is a difference between the loaded virtual-world objects and the current scene in front of the capture system. If differences are detected in operation132, operation138queries the user if they would like to update the virtual object database with new virtual objects based on the real-world objects in the current scene. If the user chooses to add new virtual objects, the procedure continues with operation114fromFIG. 1B. If the user chooses not to add new virtual objects or if there are no difference detected in operation132, operation134loads the interactive application. Operation136conforms actions of additional virtual objects of the interactive application to comply with the geometric attributes of the virtual objects loaded from memory.

FIG. 2Ais an overview of a real-world scene200where a computer system100is used in accordance with one embodiment of the present invention. In this embodiment the capture system of computer system100includes video camera204. Video camera204can be configured to take individual still images and/or continuous streaming video. The computer system100is configured to output video to screen206and audio to speakers that are not shown. In some embodiments, the video output of computer system100can include images captured by video camera204. In other embodiments, the video output of the computer system100includes only images rendered by the computer system100. In yet other embodiments, a combination of images captured by video camera204and images rendered by the computer system100can be output from the computer system100.

Also shown in the real-world scene200is user A202manipulating a controller224. The controller224can take on many forms, and is not limited to a standard game controller requiring two hand manipulation. Other types of controllers can include single-hand controlling object, or objects that are identified for interactivity. Consequently, the controller224is only shown as one example. User A202sits on real-world seat210behind real-world coffee table208. Also included in the real-world scene200is real-world side table212, real-world lamp214, real-world window216, and real-world window218. Additionally, the controller224can be used to provide input to the computer system100.

FIG. 2Billustrates defined virtual-world objects that were generated based on the real-world scene fromFIG. 2A, as seen by the computer system100in accordance with one embodiment of the present invention. Note that the view illustrated inFIG. 2Bis intended to illustrate the geometric attributes of defined virtual-world objects as seen by the computer system100. InFIG. 2B, sufficient geometric attributes of real-world seat210, have been defined in order to allow the identification and mapping logic to generate a virtual-world seat210′. Similarly, virtual-world window218′, virtual-world lamp214′ virtual-world side table212′, virtual-world coffee table208′, and virtual-world window216′ have been defined and generated. Geometric attributes have been defined in order to help generate the respective virtual-world objects, and in some embodiments, the geometric attributes allow interaction between the generated virtual objects and additional virtual-world objects in a virtual-world scene.

FIG. 2Cillustrates a virtual-world scene as it would be displayed on a screen, in accordance with one embodiment of the present invention. In this embodiment, the virtual-world scenes is based on the real-world scene fromFIG. 2A. Included in the virtual-world scene is a virtual-world user A202′ (e.g. an avatar representing real-world user A202). Note that inFIG. 2A, user A202is sitting on the real-world seat210manipulating controller224. As illustrated inFIG. 2C, the manipulation of the controller224inFIG. 2Acan control the movement of an avatar represented by virtual-world user A202′ in the virtual-world scene. Thus, while user A202ofFIG. 2Ais seated, the virtual-world user A202′ can be standing.

Also shown in the virtual-world scene is an avatar for virtual-world user B222. In some embodiments, virtual-world user B222can be another user of computer system100. In other embodiments, the computer system100ofFIG. 2Acan be connected to similar systems via the internet allowing a real-world user B to control the virtual user B222in the virtual-world scene ofFIG. 2C. In yet other embodiments, virtual-world user B222can be generated by the computer system to provide interaction for real-world user202via virtual-world user202′. Similarly, the computer system can generate a virtual dog202.

Included in the virtual-world scene are the generated virtual-world objects ofFIG. 2B. Thus, as the geometric attributes of the generated virtual objects have been defined, virtual-world user A202′, virtual-world user B222and virtual-world dog220can interact with the virtual-world objects. As used herein, interaction between virtual-world objects can include occlusion of one virtual object by another virtual object, obstructing motion of virtual objects, deforming one virtual object based on an intersection to another virtual object and enabling one virtual object to rest upon another virtual object. Thus, as will be discussed in further detail below, on the screen, the virtual-world dog220could walk “behind” the virtual-world coffee table208′ and jump “onto” the virtual-world seat210′.

In some embodiments, the appearance of virtual-world seat210′ on the screen206can be identical to the appearance of the real-world seat210. In other embodiments, the appearance of the virtual-world seat210′ on the screen206can be entirely different from the appearance of real-world seat210. In such embodiments, the geometric attributes of the real-world seat210can be used to create a framework for the virtual-world seat210′. The different appearance of the virtual-seat210′ can be skinned over the geometric attribute define framework of the real-world seat210.

FIGS. 3A-1and3A-2are a real-world scene and a camera image displayed on a screen, respectively, each illustrating the input of reference points in a real-world space to assist in the mapping of geometric attributes of real-world objects, in accordance with one embodiment of the present invention. Reference points can be used by the computer system to identify a reference plane in the real-world space. Using the reference plane, the identification and mapping logic can determine the relative location of generated virtual-world objects relative to the reference plane. In one embodiment, the computer system can prompt a user to identify reference points based on an image of the real-world scene displayed on the screen.

For example, the computer system may request that a user identify enough reference points so as to define a floor in the real-world scene. Thus, as shown inFIGS. 3A-1and3A-2, a user may be prompted to identify the most forward left point300′ on the screen in the real-world scene. Accordingly, a user in the real-world ofFIG. 3A-1could walk over to point300and provide the requisite user input to indicate the location of point300to the computer system. Similarly, the user could be prompted to identify additional points of the reference plane such as point302/302′, point308/308′ and point306/306′. In other embodiments, a user may be prompted to define a vertical point such as points304/304′. In an alternate embodiment, reference points defining the floor of the real-world scene can be determined by tracking the feet of a user as the user walks around the real-world scene. Recognition and tracking the feet of the user could be performed by the identification and mapping logic.

FIG. 4Aillustrates the use of transition boundaries and surfaces to define locations on a real-world table400, in accordance with one embodiment of the present invention. In this embodiment, a transition boundary is found at corner A402. Similarly, additional transition boundaries are defined at corner B404, corner C406and corner D408. As previously discussed, a user can be prompted to identify the respective transition boundaries using a variety of user input methods. Different forms of user input can be used to identify surfaces on the real-world table400such as front surface410or right surface412. For example, in one embodiment, aligning a console controller in a particular orientation and pressing a particular button can identify transition surfaces such as points. In other embodiments, tapping a hand across the real-world surface or moving a controller across the surface with a particular button depressed can identify surfaces. As there are additional input methods the particular embodiments should be considered exemplary and should not be considered limiting.

FIG. 4Billustrates the computer system view of the transition boundaries defined from the real-world table400fromFIG. 4B, in accordance with one embodiment of the present invention. The transition boundaries of the real-world table400can be used to define a virtual-world table400′. The virtual-world table400′ has a surface created from user input that defines the relative position of corners A-D of the real-world table400inFIG. 4A. The resulting virtual-world corners, virtual corner A402′, virtual corner B404′, virtual corner C406′ and virtual corner D408′ create a surface at a height y above the reference plane, or floor. Similarly, by defining front surface410and right surface412on the real-world table400, virtual-world table front surface410′ and virtual-world table right surface412′ are respectively defined.

Based on the virtual-world corner and virtual-world surfaces, the geometric attributes shown inFIG. 4Bfor virtual-world table400′ are defined. As defined by the geometric attributes shown inFIG. 4B, the virtual-world table400has solid sides so other virtual elements would be prevented from going “under” the virtual-world table400′. In other embodiments, additional parameters may be defined to create a virtual-world table with more refined details. For example, whereas the geometric attributes of the virtual-world table400′ appear as a solid block, a user could define additional parameters such as the thickness of the table surface. In other embodiments, various object templates can include a variety of different table styles that including parameters to better define table legs. With greater refinement of the geometric attributes, the virtual-world table400′ could be modeled to appear identical to the real-world table400thus allow other virtual elements to interact with a space under the virtual-world table400′.

FIG. 5Aillustrates an exemplary user interface of a computer system that enables a user to interactively create a virtual-world object based on a real-world object, in accordance with one embodiment of the present invention. Screen500displays multiple graphical icons that represent various capabilities of the computer system. Selecting toolbox icon502can display various submenu icons that allow a user to customize and configure different aspects of the computer system. Camera icon504can be selected to access submenus for display and editing digital photographs. In this embodiment, the computer system also includes the functionality of playing music, represented by music icon506, playing video, as shown with movie icon508, playing interactive games by accessing game icon510, or accessing a virtual-world home using home icon512. The particular icons shown inFIG. 5Aare exemplary and should not be considered limiting as the user interface can be individually customized to contain fewer or additional icons.

The computer system can be connected to the internet enabling access to a social networking environment where individual users of like computer systems can be represented in a virtual-world by avatars. Users can configure their avatars and can similarly configure a virtual-world home for their avatar. The user can configure their virtual home just as users can configure avatars to be realistic representations or fanciful representations of themselves.

FIG. 5Billustrates an exemplary virtual home submenu after selecting the home icon512ofFIG. 5A, in accordance with one embodiment of the present invention. The virtual home submenu includes a toolbox icon514to configure general aspects of the virtual home. The hammer icon516can be selected to allow a user to build and configure various aspects of their virtual home or configure and customize their avatar. A bank icon518is included to allow a user to acquire, spend, and manage virtual currency in the virtual-world. In one embodiment, Globe icon510can be used to display a graphical illustration of particular aspects of the virtual-world community. Additionally, help icon522can be selected to access online help.

FIG. 5Cillustrates an exemplary submenu after selecting the hammer icon516ofFIG. 5B, in accordance with one embodiment of the present invention. Selecting avatar icon524allows a user to configure and customize various aspects of their avatar. Selecting house icon526allows a user to build and configure various aspects of their virtual-world home.

FIG. 5Dillustrates an exemplary submenu after selecting the house icon526ofFIG. 5C, in accordance with one embodiment of the present invention. Stove icon528can be used to configure a virtual-world kitchen whereas bed icon532can be used to configure a virtual-world bedroom. Similarly, media room icon530can be used to configure and customize a virtual-world media room. Alternatively, instead of a virtual-world media room, a user could configure a virtual-world living room or great room for entertaining virtual-world guests.

FIG. 5Eshows a variety of household objects such as furniture and other fixed background objects that can be configured and customized for placement in a virtual-world media room or virtual-world living room, in accordance with one embodiment of the present invention. A seating icon534allows users to configure and customize a variety of seating options such as chairs, sofas, benches, and couches. Selecting table icon540enables a user to customize virtual-world tables that can be placed in their virtual-world living room. Light icon542can be used to configure virtual-world lighting fixtures and plant icon544can be used to customize virtual-world plants. For simplicity, a limited number of household objects are shown inFIG. 5Eand the illustrated items should not be considered limiting. Other household items or fixed background objects such as, but not limited to, windows, area rugs, desks, portraits, painting, sculptures, audio/visual equipment, communication devices, and computers can be selected by a user for customization and configuration.

In one embodiment, selecting the seating icon534results in camera icon536and hand icon538being displayed. Selecting camera icon536enables the capture system to assist the user to generate virtual-world objects. Selecting the hand icon538can allow a user to manually model virtual-world objects without the assistance of the capture system. After selecting either the camera icon536or the hand icon538, the user may have the option to browse through various object templates in an effort to simplify generation of the virtual-world object. Exemplary object templates for different types of seating can include task chairs, loveseats, sofas, sectional sofas, chaise lounges, folding chairs, stools, and benches.FIG. 5Fillustrates an object template for virtual-world seat210′ in accordance with one embodiment of the present invention.

FIG. 6B-1throughFIG. 6G-1shows the real-world scene that includes the user202interactively identifying various geometric attributes of the real-world seat210in response to on-screen prompts based on an object template, in accordance with one embodiment of the present invention. As previously discussed, a variety of object templates can be stored and recalled and a user can browse through the different templates to determine a best match for the particular real-world object that is to be defined in a virtual space.FIG. 6B-2throughFIG. 6G-2show a screen displaying an image of the user identifying various surfaces, points or transitional surfaces of the real-world seat210in order to define and generate a virtual-world seat, in accordance with embodiments of the present invention. InFIG. 6B-2, the user202is prompted by an object template600to identify the highlighted surface601of the real-world seat210. Accordingly, in bothFIG. 6B-1andFIG. 6B-2, the user202is show touching surface601′. InFIG. 6C-2, the object template600prompts the user202to identify surface602of the real-world seat210. Thus, inFIGS. 6C-1and6C-2the user202identifies surface602′.

As shown inFIG. 6D-2, the object template600prompts the user to identify surface604. In accordance with various embodiments, inFIG. 6D-1the user202can identify the surface604′ on the real-world seat210by either identifying the surface or by indicating various transitional surfaces such as points or edges. In one embodiment, to identify the surface604′, the user202could press a particular button on a controller while the controller is placed against the surface604′ and run down a length of the surface604′. Upon reaching the end of the length of the surface604′, the user202would release the particular button. In another embodiment, the user202could identify a variety of reference points using the microphones of the capture system by snapping their fingers or clapping their hands at the front-right, front-left, rear-right and rear-left corners of the surface604′.

The object template600prompts the user to identify a surface on the real-world seat210that corresponds with the highlighted surface606inFIG. 6E-2. Thus, inFIG. 6E-1andFIG. 6E-2the user202is shown at least partially identifying a surface606′ on the real-world seat210. Similarly, inFIG. 6F-2the object template600prompts the user to identify a surface on the real-world seat210that corresponds to the highlighted surface608. Again, in bothFIG. 6F-1andFIG. 6F-2, the user202is shown identifying part of a surface608′. InFIG. 6G-1andFIG. 6G-2the object template600prompts the user to identify a surface of the real-world seat210that corresponds to the highlighted surface610. Thus, inFIG. 6G-1andFIG. 6G-2the user202is shown at least partially identifying a surface610′ on the real-world seat210. Recall that a user can identify surfaces by defining a plurality of individual points and/or transition surfaces using a variety of input methods. For example, in some embodiments particular hand gestures can be used to indicate the relative position of an individual point. In other embodiments, a user can tap the surface to define the relative position of an individual point. In yet other embodiments, transitional surface can be defined when a user can runs their hand across the surface.

FIG. 7Ais an exemplary graphical user interface illustrating choices for a user to select from after mapping the geometric attributes of a real-world object, in accordance with one embodiment of the present invention. Screen700indicates that a seat has been defined and provides the user the option to save and exit702, add/edit elements704, or cancel706. In this example, the user selects add/edit elements704and the screen inFIG. 7Bis displayed.

FIG. 7Bshows a variety of household objects such as furniture and other fixed background objects that can be configured and customized for placement in a virtual-world media room or virtual-world living room, in accordance with one embodiment of the present invention. From the screen shown inFIG. 7B, the user can select one of the household objects and return to the screen shown inFIG. 5E. As illustrated inFIG. 7B, a user selects to add a table708. As previously discussed, after choosing to add the table708the user can elect to use the capture system to help map the table and then browse through various object templates for tables to find a suitable match to the user's real-world table.

FIG. 8A-1illustrates a real-world scene where a user identifies a real-world surface801′ on the real-world table212, whileFIG. 8A-2illustrates an on-screen display that is prompting a user to identify a real-world surface that corresponds to surface801on the object template800, in accordance with one embodiment of the present invention. Likewise,FIGS. 8B-1and8B-2show the user202identifying real-world surface802′ in response to the prompt to identify a corresponding real-world surface to surface802of the object template800, in accordance with one embodiment of the present invention.FIG. 8C-2shows a user being prompted to identify a real-world surface that corresponds to surface804of the object template800. Accordingly, inFIG. 8C-1, the user can identify the surface804′ with any of the previously discussed techniques.FIG. 8D-1shows the user202identifying a surface806′. InFIG. 8D-2, the object template800prompted the user202to identify a real-world surface that corresponds to the virtual-world surface806. Also shown inFIG. 8D-2is the image captured of the user202identifying the surface806′ from the perspective of a camera observing the scene inFIG. 8D-1.

FIG. 9A-1is a real-world scene of a user202sitting on the real-world seat210interactively manipulating an avatar representing user A202in a virtual-world, the virtual-world displayed to user A202on the screen206, in accordance with one embodiment of the present invention.FIG. 9A-2illustrates the virtual-world displayed to user A202on the screen206inFIG. 9A-1. Also shown in theFIG. 9A-2is an avatar for virtual-world user B222along with virtual-world dog220. For purposes of this discussion, the virtual-world seat210′ has been appropriately defined by user A202using the previously discussed techniques.

Thus, in this embodiment it would be possible for the virtual-world dog220to interact with the virtual-world seat210′. For example, as illustrated inFIG. 9A-3, the virtual-world dog220is shown standing on the virtual-world seat210′. Since the virtual-world seat210′ has had the surface606fromFIGS. 6E-1and6E-2defined, the virtual-world dog220can be shown as standing on the virtual-world seat210′. Similarly, if the virtual-world seat210′ has the surface608fromFIGS. 6F-1and6F-2defined, the virtual-world dog220can be shown resting or leaning against the back of the virtual-world seat210′.

FIGS. 10A-1and10A-2illustrate both the real-world scene and the virtual-world scene as displayed on the screen of user A202while mapping geometric attributes of the real-world window218in accordance with one embodiment of the present invention. In various embodiments, the virtual-world elements defined by the user can reflect real-world characteristics. In the case of the virtual-world window, different environments and photographs can be displayed in the virtual-world windows so the virtual-world home of user A202can appear to be in any real-world or fantasy location. For example, lunar landscapes or various cityscapes can be displayed in the virtual-world windows. Additionally, intensity of light coming through the virtual-windows can be synchronized with real-world time so virtual-world time mirrors the real-world time of the user.

FIGS. 11A-1and11A-2are additional illustrations of both the real-world scene and the virtual-world scene as displayed on the screen of user A202while mapping geometric attributes of the real-world lamp214, in accordance with one embodiment of the present invention.

FIG. 12schematically illustrates the overall system architecture of the Sony® Playstation 3® entertainment device, a console having controllers for implementing an avatar control system in accordance with one embodiment of the present invention. A system unit1200is provided, with various peripheral devices connectable to the system unit1200. The system unit1200comprises: a Cell processor1228; a Rambus® dynamic random access memory (XDRAM) unit1226; a Reality Synthesizer graphics unit1230with a dedicated video random access memory (VRAM) unit1232; and an I/O bridge1234. The system unit1200also comprises a Blu Ray® Disk BD-ROM® optical disk reader1240for reading from a disk1240aand a removable slot-in hard disk drive (HDD)1236, accessible through the I/O bridge1234. Optionally the system unit1200also comprises a memory card reader1238for reading compact flash memory cards, Memory Stick® memory cards and the like, which is similarly accessible through the I/O bridge1234.

The I/O bridge1234also connects to six Universal Serial Bus (USB) 2.0 ports1224; a gigabit Ethernet port1222; an IEEE 802.11b/g wireless network (Wi-Fi) port1220; and a Bluetooth® wireless link port1218capable of supporting of up to seven Bluetooth connections.

In operation the I/O bridge1234handles all wireless, USB and Ethernet data, including data from one or more game controllers1202. For example when a user is playing a game, the I/O bridge1234receives data from the game controller1202via a Bluetooth link and directs it to the Cell processor1228, which updates the current state of the game accordingly.

The wireless, USB and Ethernet ports also provide connectivity for other peripheral devices in addition to game controllers1202, such as: a remote control1204; a keyboard1206; a mouse1208; a portable entertainment device1210such as a Sony Playstation Portable® entertainment device; a video camera such as an EyeToy® video camera1212; and a microphone headset1214. Such peripheral devices may therefore in principle be connected to the system unit1200wirelessly; for example the portable entertainment device1210may communicate via a Wi-Fi ad-hoc connection, whilst the microphone headset1214may communicate via a Bluetooth link.

The provision of these interfaces means that the Playstation 3 device is also potentially compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Voice over IP telephones, mobile telephones, printers and scanners.

In addition, a legacy memory card reader1216may be connected to the system unit via a USB port1224, enabling the reading of memory cards1248of the kind used by the Playstation® or Playstation 2® devices.

In the present embodiment, the game controller1202is operable to communicate wirelessly with the system unit1200via the Bluetooth link. However, the game controller1202can instead be connected to a USB port, thereby also providing power by which to charge the battery of the game controller1202. In addition to one or more analog joysticks and conventional control buttons, the game controller is sensitive to motion in six degrees of freedom, corresponding to translation and rotation in each axis. Consequently gestures and movements by the user of the game controller may be translated as inputs to a game in addition to or instead of conventional button or joystick commands. Optionally, other wirelessly enabled peripheral devices such as the Playstation Portable device may be used as a controller. In the case of the Playstation Portable device, additional game or control information (for example, control instructions or number of lives) may be provided on the screen of the device. Other alternative or supplementary control devices may also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or bespoke controllers, such as a single or several large buttons for a rapid-response quiz game (also not shown).

The remote control1204is also operable to communicate wirelessly with the system unit1200via a Bluetooth link. The remote control1204comprises controls suitable for the operation of the Blu Ray Disk BD-ROM reader1240and for the navigation of disk content.

The Blu Ray Disk BD-ROM reader1240is operable to read CD-ROMs compatible with the Playstation and PlayStation 2 devices, in addition to conventional pre-recorded and recordable CDs, and so-called Super Audio CDs. The reader1240is also operable to read DVD-ROMs compatible with the Playstation 2 and PlayStation 3 devices, in addition to conventional pre-recorded and recordable DVDs. The reader1240is further operable to read BD-ROMs compatible with the Playstation 3 device, as well as conventional pre-recorded and recordable Blu-Ray Disks.

The system unit1200is operable to supply audio and video, either generated or decoded by the Playstation 3 device via the Reality Synthesizer graphics unit1230, through audio and video connectors to a display and sound output device1242such as a monitor or television set having a display1244and one or more loudspeakers1246. The audio connectors1250may include conventional analogue and digital outputs whilst the video connectors1252may variously include component video, S-video, composite video and one or more High Definition Multimedia Interface (HDMI) outputs. Consequently, video output may be in formats such as PAL or NTSC, or in 720 p, 1080i or 1080 p high definition.

Audio processing (generation, decoding and so on) is performed by the Cell processor1228. The Playstation 3 device's operating system supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and the decoding of 7.1 surround sound from Blu-Ray® disks.

In the present embodiment, the video camera1212comprises a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding apparatus so that compressed video data may be transmitted in an appropriate format such as an intra-image based MPEG (motion picture expert group) standard for decoding by the system unit1200. The camera LED indicator is arranged to illuminate in response to appropriate control data from the system unit1200, for example to signify adverse lighting conditions. Embodiments of the video camera1212may variously connect to the system unit1200via a USB, Bluetooth or Wi-Fi communication port. Embodiments of the video camera may include one or more associated microphones and also be capable of transmitting audio data. In embodiments of the video camera, the CCD may have a resolution suitable for high-definition video capture. In use, images captured by the video camera may for example be incorporated within a game or interpreted as game control inputs.

In general, in order for successful data communication to occur with a peripheral device such as a video camera or remote control via one of the communication ports of the system unit1200, an appropriate piece of software such as a device driver should be provided. Device driver technology is well-known and will not be described in detail here, except to say that the skilled man will be aware that a device driver or similar software interface may be required in the present embodiment described.

Embodiments may include capturing depth data to better identify the real-world user and to direct activity of an avatar or scene. The object can be something the person is holding or can also be the person's hand. In the this description, the terms “depth camera” and “three-dimensional camera” refer to any camera that is capable of obtaining distance or depth information as well as two-dimensional pixel information. For example, a depth camera can utilize controlled infrared lighting to obtain distance information. Another exemplary depth camera can be a stereo camera pair, which triangulates distance information using two standard cameras. Similarly, the term “depth sensing device” refers to any type of device that is capable of obtaining distance information as well as two-dimensional pixel information.

Recent advances in three-dimensional imagery have opened the door for increased possibilities in real-time interactive computer animation. In particular, new “depth cameras” provide the ability to capture and map the third-dimension in addition to normal two-dimensional video imagery. With the new depth data, embodiments of the present invention allow the placement of computer-generated objects in various positions within a video scene in real-time, including behind other objects.

Moreover, embodiments of the present invention provide real-time interactive gaming experiences for users. For example, users can interact with various computer-generated objects in real-time. Furthermore, video scenes can be altered in real-time to enhance the user's game experience. For example, computer generated costumes can be inserted over the user's clothing, and computer generated light sources can be utilized to project virtual shadows within a video scene. Hence, using the embodiments of the present invention and a depth camera, users can experience an interactive game environment within their own living room. Similar to normal cameras, a depth camera captures two-dimensional data for a plurality of pixels that comprise the video image. These values are color values for the pixels, generally red, green, and blue (RGB) values for each pixel. In this manner, objects captured by the camera appear as two-dimension objects on a monitor.

Embodiments of the present invention also contemplate distributed image processing configurations. For example, the invention is not limited to the captured image and display image processing taking place in one or even two locations, such as in the CPU or in the CPU and one other element. For example, the input image processing can just as readily take place in an associated CPU, processor or device that can perform processing; essentially all of image processing can be distributed throughout the interconnected system. Thus, the present invention is not limited to any specific image processing hardware circuitry and/or software. The embodiments described herein are also not limited to any specific combination of general hardware circuitry and/or software, nor to any particular source for the instructions executed by processing components.