Abstract:
A method for determining a position of a controller in a three-dimensional space is disclosed. The method includes an operation to calibrate a computer program to identify a group of recognized fixed points from a plurality of fixed points within the three-dimensional space using a depth camera integrated with the controller. Another operation activates use of the controller and depth camera during interface with the application. During use of the controller, image and depth data within the three-dimensional space is captured with the depth camera of the controller. In another operation the image and depth data is analyzed to find one or more of the group of recognized fixed points. In one embodiment, the previous two operations are repeated and changes of position of the controller are determined based on a change in position of the found ones of the group of recognized fixed points. The method also includes an operation to trigger an input command based on the determined change in position of the controller, wherein the input command causes an action by the application.

Description:
BACKGROUND OF THE INVENTION 
     The video game industry has seen many changes over the years. As computing power has expanded, developers of video games have likewise created game software that takes advantage of these increases in computing power. To this end, video game developers have been coding games that incorporate sophisticated operations and mathematics to produce a very realistic game experience. 
     Example gaming platforms, may be the Sony Playstation, Sony Playstation2 (PS2), and Sony Playstation3 (PS3), each of which is sold in the form of a game console. As is well known, the game console is designed to connect to a monitor (usually a television) and enable user interaction through handheld controllers. The game console is designed with specialized processing hardware, including a CPU, a graphics synthesizer for processing intensive graphics operations, a vector unit for performing geometry transformations, and other glue hardware, firmware, and software. The game console is further designed with an optical disc tray for receiving game compact discs for local play through the game console. Online gaming is also possible, where a user can interactively play against or with other users over the Internet. 
     While there realism of the gaming experience has steadily improved, users have continued to interact with the gaming platforms through dedicated gaming controllers. While some advances have been made in user interaction additional avenues of user interaction remain that can improve the overall experience of using the gaming console. 
     It is within this context that embodiments of the invention arise. 
     SUMMARY 
     Broadly speaking, the present invention enables users to interact with a computer system using a controller having an associated or integrated depth camera. To calibrate the controller, the depth camera can be used to capture image and depth data within a space. Subsequent image and depth data can be used to determine the relative location of the depth camera, and subsequently, the controller within the space. 
     In one embodiment, a computer implemented method for determining a position of a controller in a three-dimensional space is disclosed. The method includes an operation to calibrate a computer program to identify a group of recognized fixed points from a plurality of fixed points within the three-dimensional space using a depth camera integrated with the controller. Another operation activates use of the controller and depth camera during interface with the application. During use of the controller, image and depth data within the three-dimensional space is captured with the depth camera of the controller. In another operation the image and depth data is analyzed to find one or more of the group of recognized fixed points. In one embodiment, the previous two operations are repeated and changes of position of the controller are determined based on a change in position of the found ones of the group of recognized fixed points. The method also includes an operation to trigger an input command based on the determined change in position of the controller, wherein the input command causes an action by the application. 
     In another embodiment, a computer implemented method for interacting with a computer system is disclosed. The method includes an operation that begins an interactive session that is executed on the computer system, and automatically applies a saved controller calibration from a memory. In another operation, the method accepts input to a controller with a mobile depth camera. In one operation, image and depth data within a space is captured with the mobile depth camera. In another operation the image and depth data is analyzed for one or more recognized fixed points. In one embodiment, the previous two operation and repeated and movement of the mobile depth camera within the space is determined based on differences between the recognized fixed points. The method also includes an operation where an input command is triggered based on the movement of the mobile depth camera, wherein the input command causes an action within the interactive session. 
     In another embodiment, a handheld controller for a computer system is disclosed. The handheld controller includes a controller housing defined by an exterior that encloses and interior volume. The controller also includes interface controls that are partially exposed to the exterior of the housing. Also included with the controller is a depth camera assembly. The depth camera assembly includes a depth module, an image module and processing logic. The depth module and image module are configured to receive image and depth data through a camera lens, and the communication module is configured to transmit and receive data from the processing logic of the depth camera assembly to the computer system. Wherein changes to the image and depth data received by the depth camera assembly trigger input commands for the computer system. 
     In yet another embodiment, an interactive computing system configured to display an interactive session on a display is disclosed. The computing system includes a computing console with a central processing unit, a graphics processing unit, and a communications interface. The communications interface being configured to provide communication to and from the computing console. The computer system can also include a controller with a depth camera. The depth camera being configured with a depth module, an image module and processing logic. The controller may also include a communication module that is configured to receive and transmit data from the processing logic of the depth camera to the communications interface of the computing console. Wherein changes in image and depth data captured by the depth camera are translated to input commands for the computing system. 
     Other aspects and advantages of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention, together with further advantages thereof, may best be understood by reference to the following description taken in conjunction with the accompanying drawings. 
         FIG. 1  shows an exemplary space where a console can be used my user, in accordance with one embodiment of the present invention. 
         FIGS. 2A-1  and  2 A- 2  are simplified views showing select components of the mobile depth camera, in accordance with embodiments of the present invention. 
         FIG. 2B  is an external view of an exemplary controller with an integrated depth camera, in accordance with one embodiment of the present invention. 
         FIGS. 3A and 3B  illustrate the user calibrating the controller relative to the camera and the screen, in accordance with various embodiments of the present invention. 
         FIG. 4A  illustrates analyzed image and depth data from the mobile depth camera, in accordance with one embodiment of the present invention. 
         FIG. 4B  illustrates recognized fixed points within a space in accordance with one embodiment of the present invention. 
         FIGS. 5A and 5B  illustrate recognized fixed points within a scene from the perspective of the mobile depth camera, in accordance with one embodiment of the present invention. 
         FIGS. 6A and 6B  illustrate recognized fixed points within a scene from the perspective of the mobile depth camera, in accordance with one embodiment of the present invention. 
         FIG. 6C  illustrates a top view of a scene where depth data from the mobile depth camera is used to determine the position of the controller within the space in accordance with one embodiment of the present invention. 
         FIG. 7  is an exemplary flow chart illustrating operations to use the mobile depth camera to determine the position of the controller within the space, in accordance with one embodiment of the present invention. 
         FIG. 8  is an exemplary flow chart that illustrates operations to calibrate the mobile depth camera in order to determine the position of the controller within the space, in accordance with one embodiment of the present invention. 
         FIG. 9  schematically illustrates the overall system architecture of the Sony® Playstation 3® entertainment device, a computer system capable of utilizing dynamic three-dimensional object mapping to create user-defined controllers in accordance with one embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     An invention is disclosed for using image and depth data captured by a mobile depth camera to control an interactive session on a console. Broadly speaking, the console can be any type of system that takes input from a user, whether it is a general-purpose computer (e.g., desktop, notebook, handheld device, smartphone, etc.), or a special purpose computer such as a game console. The mobile depth camera can be mounted to or in a controller for the console and used to capture geometric depth data along image data. The depth camera can provide image and depth data to the console for analysis and processing. In one embodiment, the depth camera is a single lens camera, and in other embodiments, multiple camera lenses can be used to capture images and depth data. The image and depth data captured by the mobile depth camera can be analyzed to determine movement of the controller and depth camera within a space. The movement of the controller can be processed and mapped to control various aspects of the interactive session on the console. 
     In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to unnecessarily obscure the present invention. 
       FIG. 1  shows an exemplary space  100  where a console  106  is being used by user  102 , in accordance with one embodiment of the present invention. The space  100  can include a table  114  along with shelf units  112   a/b  and a monitor  108 . In some embodiments, the monitor  108  is a cathode ray tube (CRT) monitor capable of displaying a variety of video inputs. In other embodiments, the monitor  108  is a flat panel monitor such as a Liquid Crystal Display (LCD) or plasma monitor configured to accept a variety of video inputs (analog or digital). 
     Connected to the monitor  108  is the console  106  that includes various hardware and firmware that is capable of executing locally or remotely stored software programs. In one embodiment, the software program can be an operating system that outputs video of a Graphical User Interface (GUI) to the monitor  108 . The user  102  can interact with the console  108  through the GUI using a controller  104  that includes a mobile depth camera  105 . The mobile depth camera  105  captures image and depth data within a field of view  105   a . Analysis of the image and depth data captured with the mobile depth camera  105  can be performed to determine the relative position of the controller  104  within the space  100 . 
     The controller  104  can accept a variety of user input such as, but not limited to combinations of digital and/or analog buttons, triggers, joysticks, and touch pads. Additionally, accelerometers within the controller  104  can allow the console  106  to respond to motions imparted by the user  102  to the controller  104 . For example, using integrated accelerometers, motions or combinations of motions imparted to the controller such as roll, pitch and yaw can be detected and used to interact with software being executed by the console  106 . In some embodiments, wires physically couple and facilitate communications between the console  106  and the controller  104 . In other embodiments, the controller  104  can communicate wirelessly to the console  106  using wireless protocols such as, but not limited to Bluetooth and Wi-Fi. 
     A fixed depth camera  110  can also be coupled to the console  106  and placed so the fixed depth camera  110  has a field of view  110   a  that includes the table  114 , the user  102 , and the controller  104 . Similar to the mobile depth camera  105 , the fixed depth camera can capture image and depth data for objects within the field of view  110   a . In one embodiment, the image and depth data from both the fixed depth camera  110  and the mobile depth camera  105  is used to determine the relative position of the controller  104  within the space  100 . 
       FIGS. 2A-1  and  2 A- 2  are simplified views showing select components of the mobile depth camera  105 , in accordance with embodiments of the present invention. For simplicity, additional components within the controller  104  are not shown. In the embodiment shown in  FIG. 2A-1 , the mobile depth camera  105  is integrated within the controller  104 . In  FIG. 2A-2 , the mobile depth camera  105  is shown as an external module that can be attached to an exterior port  220   b  on the controller  104  using a connector  220   a  (e.g., USB, or other electrical connector). In both embodiments, the mobile depth camera  105  can include a lens  200 , a depth module  202  along with an image module  204 , and logic processing  206 . The depth module  202  can include a variety of sensors, emitters, and logic. In one embodiment, multiple infrared (IR) emitters can be arranged in an array around the lens  200 . The IR emitters can be used to send pulses of IR light that reflect off of objects in front of the controller  104 . Sensors associated with the IR emitters can measure the time for the reflected IR pulses to return in order to help determine the distance, or depth, of an object from the controller  104 . 
     In some embodiments, logic associated with the depth module  202  can be used to focus the depth module to a particular area within the field of view of the depth camera  105 . Focusing the depth module  202  can be accomplished using a variety of techniques. In one embodiment, IR emitters with different wavelengths can be used within an array of IR emitters. Additionally, the different emitters can be pulsed at different times and/or different intensities to focus depth detection in a particular area within the field of view of the depth camera  105 . 
     The image module  204  can be configured to capture image data through the lens  200  using a charged coupled device (CCD) or other type of sensor. In one embodiment, the logic module  206  can input data from both the image module  204  and the depth module  202 . The logic module  206  can also receive and interpret commands from the console  106  to change the focus of either the image module  204  or the depth module  202 . In wireless embodiments of the controller  104 , a communications module  208  can permit bi-direction communications between the controller  104  and the console  106 . Thus, signals from the logic module  206  to be transmitted to the console  106  and vice versa. In one embodiment, the communications module  208  can transmit data using a form of the IEEE 802.11 wireless protocols. In another embodiment, a form of the Bluetooth communications standard is used to transmit data from the controller  104  to the console  106 . 
       FIG. 2B  is an external view of an exemplary controller  104  with an integrated depth camera  105 , in accordance with one embodiment of the present invention. Visible on an exterior of the controller  104  is the lens  200 . Also visible on the exterior of the controller  104  is a Universal Serial Bus (USB) port  210 , buttons  214   a/b  and  216   a/b , and joysticks  212   a/b . The buttons  214   a/b  and  216   a/b  along with the joysticks  212   a/b  can be either analog or digital and allow a user to interface with the console. The controller  104  can be charged when connected to the console via the USB port  210 . 
       FIGS. 3A and 3B  illustrate the user  102  calibrating the controller  104  relative to the camera  110  and the screen  108 , in accordance with various embodiments of the present invention. To calibrate the position of the controller within the space, the user  102  may be prompted via on-screen instructions, audible instruction or a combination thereof, to hold the controller  104  in a first position. This enables the mobile depth camera  105  to capture image and depth data for objects within the field of view  105   a . In this embodiment, the monitor  108  and the fixed depth camera  110  are both within the field of view  105   a  of the mobile depth camera  105 . After being prompted to move into a second position, the controller  104 ′ along with the mobile depth camera  105 ′ can capture additional image and depth data within a second field of view  105   a′.    
       FIG. 3B  illustrates other possible steps in calibrating the mobile depth camera  105 , in accordance with one embodiment of the present invention. In this embodiment, the user  102  stands in front of the monitor  108 , fixed depth camera  110  and shelf units  112   a/b . In order to capture image and depth data from a variety of angles, the user  102  may be prompted to assume positions of user  102 ′ and/or user  102 ″. In the various positions, the mobile depth camera  105  can capture image and depth data of the shelf units  112   a/b , monitor  108  and fixed depth camera  110  within their respective field of views. In other embodiments, calibration may require raising, lowering, tilting or yawing the controller  104  while in the various positions of the users  102 ,  102 ′ and  102 ″. 
       FIG. 4A  illustrates analyzed image and depth data from the mobile depth camera  105 , in accordance with one embodiment of the present invention. Fixed points  400  can be determined based on the image and depth data captured during the calibration of the depth camera  105 . In some embodiments, the console can execute software that analyzes the image and depth data to create wireframe models of stationary objects within the field of view of the controller and integrated mobile depth camera. When the mobile depth camera is used to determine motion of the controller, real-time images from the mobile depth camera can be compared to data generated using the wireframe models. In such embodiments, different ratios of wireframe elements captured from the image and depth data can be used to determine the position of the controller. In other embodiments, the console can analyze the image and depth data to link fixed points and generate a smaller set of recognized fixed points. 
       FIG. 4B  illustrates recognized fixed points within a space  100  in accordance with one embodiment of the present invention. When calibrating the controller image and depth data can be analyzed to determine recognized fixed points within the field of view of the mobile depth camera. In one embodiment, the recognized fixed points are selected from a larger set of fixed points similar to those identified in  FIG. 4A . The console can execute software to analyze image and depth data for a large set of fixed points captured from a variety of angles and positions within the space. Analysis of the large set of fixed points can filter the large set of fixed points to a smaller set of recognized fixed points based on predefined criteria. 
     In one embodiment, selection of recognized fixed points from the large set of fixed points can be made based on relative depth contrast between a fixed point and surrounding points. For example, in the scene  100 , recognized fixed points  402 ,  404 ,  410 , and  412  can be selected because of the prominence of the corners of the shelves  112   a  and  112   b . Depth data for the recognized fixed points  404 - 404  and  410 - 412  can be more readily resolved and recognized based on the prominence of the corners of the shelves compared to other objects within the scene  100 . Similarly, depth data for the recognized fixed points  406  and  408  can resolved based on the relative position of the monitor  108 . 
     Image data can also be used in conjunction with depth data to determine recognized fixed points. Color contrast along with relative pixel locations can be used to supplement depth data in determining which fixed points can be used as recognized fixed points. In some embodiments, depth data can be used more extensively when determining recognized fixed points because of variation in lighting conditions within the space  100 . 
       FIGS. 5A and 5B  illustrate recognized fixed points within a scene from the perspective of the mobile depth camera, in accordance with one embodiment of the present invention. The left half of  FIG. 5A  illustrates a side-view of a scene  100  with a user  102  holding a controller  104  with a mobile depth camera  105 . The right half of  FIG. 5A  illustrates an image captured of the scene  100  using the mobile depth sensing camera  105  in accordance with one embodiment of the present invention. 
     In  FIG. 5A , the user  102  holds the controller  104  so the field of view of the mobile depth camera is substantially straightforward. This provides the mobile depth camera  105  with a field of view that results in the image shown in the right half of  FIG. 5A . As illustrated the depth data is not conveyed but recognized fixed points  402 ,  408 ,  418  and  420  have been identified. 
     In the left side of  FIG. 5B , the user  102  has pitched the controller forward so the field of view of the mobile depth camera is slightly down. This results in the image seen in the right side of  FIG. 5B  where only recognized fixed points  418  and  420  remain within the field of view of the mobile depth camera. By comparing the image and depth data from  FIGS. 5A and 5B , software executed by the console can determine that the user pitched the controller  104  forward. In one embodiment, the loss of recognized fixed points  402  and  408  within  FIG. 5B  can indicate a change in position of the controller  104 . In other embodiments, image data can be used to determine relative changes in distance. For example, relative changes be determined based on changes from distance X in  FIG. 5A  to distance X′ in  FIG. 5B . Similarly, relative changes in distance Y in  FIG. 5A  to distance Y′ in  FIG. 5B  can be used to determine relative movement of the controller  104 . 
       FIGS. 6A and 6B  illustrate recognized fixed points within a scene from the perspective of the mobile depth camera, in accordance with one embodiment of the present invention. The left side of  FIG. 6A  and  FIG. 6B  illustrates a top view of a user  102  within a space  100  holding a controller  104  with a mobile depth camera  105 . The right side of  FIG. 6A  and  FIG. 6B  show the view of the space  100  as captured by the mobile depth camera  105 . In  FIG. 6A , the user  102  is shown facing the forward toward the monitor  108 . The image captured appears relatively straight and level indicating that the mobile depth camera and hence the controller are being held in a relatively level position, i.e. relatively no yaw, pitch nor roll. As illustrated in  FIG. 6B , the user  102  has turned to the left resulting in the controller yawing to the left relative to the image in  FIG. 6A . This results in mobile depth camera  105  capturing the image seen on the right side of  FIG. 6B . 
     As the mobile depth camera  105  is yawed to the left relative to the position in  FIG. 6A , the objects within the field of view change. Whereas the recognized points for the shelves  112   b  were visible in  FIG. 6A , the same recognized points are not visible in  FIG. 6B . Based on the recognized points within the mobile depth camera field of view, the location and orientation of the controller can be determined based on image and depth data captured during the calibration of the controller. 
       FIG. 6C  illustrates a top view of a scene where depth data from the mobile depth camera is used to determine the position of the controller within the space in accordance with one embodiment of the present invention. In one embodiment, depth data captured by the mobile depth camera can be used in conjunction with the image data to determine the position of the controller in the space. As shown in  FIG. 6C , the mobile depth camera can determine that the controller  104  is being held by user  102  at a distance Z 1  from the monitor  108 . Should the user  102  move to the position of user  102 ′, depth data captured by the mobile depth camera can measure the distance Z 2  from the controller  104 ′ to the monitor  108 . 
     Though the entire user  102  is shown moving to distance Z 2 , in mobile depth camera may be able to determine changes in distance based on the user  102  moving the controller forwards and backwards with their arms. For example, the mobile depth camera can determine a distance Z 1  to the monitor  108  when a user  102  holds the controller close to their body. Similarly, the mobile depth camera can determine changes in distance to the monitor  108  as the user extends their arms toward the monitor  108 . 
     In some embodiments, the changes in distance from Z 1  to Z 2  can be used to interact with the session displayed on the monitor  108 . For example, depth data and image data can be processed to determine corresponding movements in a three-dimensional virtual world. In one embodiment, depth data captured while the controller is moving toward the monitor can result in forward motion being displayed on the monitor  108 . Similarly, depth data captured while the controller is moving away from the monitor can result backward motion being displayed on the monitor. 
       FIG. 7  is an exemplary flow chart illustrating operations to use the mobile depth camera to determine the position of the controller within the space, in accordance with one embodiment of the present invention. An environment setup module  700  can perform operation  704  that calibrates the controller and operation  706  that saves the controller calibration to a memory. With the controller calibrated, operation  708  begins an interactive session. In one embodiment, interactive sessions are executed by the computing console and can be, but are not limited to interactive games and graphical user interfaces for software such as operating systems, Internet web browsers, and digital photo viewing interfaces. Since the controller calibrations can be saved to a memory some embodiments do not require the execution of operations within the environment setup module  700 . In such embodiments, the saved controller calibrations can be recalled from the memory before, after or simultaneously when the interactive session is begun. 
     In other embodiments, the environment setup module  700  can be utilized after an interactive session has been started. These embodiments can allow the controller to be customized for the particular nuances of a particular interactive session. For example, in a photo viewing interactive session moving the controller toward the monitor can allow the user to zoom in on an image show on the display. Likewise, a variety of interactive gaming sessions may allow a user to calibrate the controller based on the type of game. Depth data captured by the mobile depth camera can be utilized to control one aspect of a driving simulation while changes in depth data can be used for something different in a First-Person-Shooter (FPS) and the same changes in depth data can be used to control something entirely different in a Massive Multiplayer Online Role Playing Game (MMORPG). 
     When an interactive session has a customized controller calibration the environment setup module may prompt the user to calibrate the controller and save the controller calibration to a memory. Subsequently, upon starting the interactive session the saved controller calibration can be automatically loaded. In embodiments where multiple users can store multiple calibrations, a default calibration may be loaded and the user can choose to load their saved calibration from a menu displaying the multiple users. 
     When an interactive session is running, operation within the image processing module  702  can allow users to interact with the interactive session. When the interactive session begins, a sample rate for the mobile depth camera can be set in operation  716 . The sample rate can vary based on accuracy and precision required by the interactive session. In embodiments such as navigating an Internet web browser or utilizing a graphical user interface for an operating system, the sample rate may be lower than that of a FPS. In other embodiments, the sample rate can be continuously modified during an interactive session. 
     For example, the sample rate of the mobile depth camera can be increased when higher accuracy and precision are required and subsequently lowered when accuracy and precision are no longer required. One of many examples where this may be utilized is in a FPS when a user is using a sniper rifle. In order to more realistically simulate the difficulty of using a sniper rifle, the sample rate of the mobile depth sensing camera can be increased to capture minute movements of the user&#39;s hand to simulate movement of the virtual sniper rifle. When the user is no longer using the sniper rifle, the sample rate can seamlessly be reduced to decrease image processing requirements for the mobile depth sensing camera. In some embodiments, processors within the controller  104  perform the image and depth data processing. In other embodiments, image and depth data from the mobile depth camera is sampled at the designated sample rate and sent to the computing console  106  for processing. In either embodiment, compression techniques may be used to reduce the bandwidth required to send either processed data or raw data to the console  106  for additional processing. 
     Operation  710  captures a first frame of image and depth data using the mobile depth camera. In operation  712 , the captured image and depth data is analyzed to determine recognized fixed points within the image. Operation  714  determines the relative position within the space based on the recognized fixed points. Operation  716  uses the set sample rate to determine when to capture a second frame of image and depth data using operation  710 . Operation  718  generates an action within the interactive session by determining difference between the first and second frame of image and depth data. 
     In one embodiment, the motions determined by the image processing module  702  can supplement additional user input to the controller  104 . For example, accelerometer data can be used in addition to image and depth data to determine yaw, pitch and roll of the controller  104 . Additionally, joysticks, buttons and triggers on the controller  104  can be used to provide additional user input. The combination of the various forms of user input to the controller can be used so operation  720  displays user interactivity based on input to the controller  104  on the display  108 . 
       FIG. 8  is an exemplary flow chart that illustrates operations to calibrate the mobile depth camera in order to determine the position of the controller within the space, in accordance with one embodiment of the present invention. Operation  800  initiates controller calibration. As previously discussed, controller calibration can be performed prior to or after an interactive session has be initiated. Operation  802  captures image and depth data from a first location. In order to simplify calibration, the user may be prompted with onscreen diagrams to stand in approximate locations within the space. The onscreen diagram can instruct the user to stand in a second location where operation  804  captures image and depth data from a second location. In other embodiments, the user may be prompted to hold the controller in a variety of orientations such as changing the yaw, pitch and roll of the controller. In still further embodiments, the user may be prompted to hold the controller at various elevations above the floor of the space. 
     Operation  806  compares the image and depth data captured in the first and second location to identify recognized fixed points. In operation  808 , the recognized fixed points can be used to create a wireframe model of the space to determine relative ratios between recognized fixed points. The relative rations between recognized fixed points could then be used to determine the relative position of the controller within the space. Operation  706  is used to save the controller calibration to a memory. 
       FIG. 9  schematically illustrates the overall system architecture of the Sony® Playstation 3® entertainment device, a computer system capable of utilizing dynamic three-dimensional object mapping to create user-defined controllers in accordance with one embodiment of the present invention. A system unit  1000  is provided, with various peripheral devices connectable to the system unit  1000 . The system unit  1000  comprises: a Cell processor  1028 ; a Rambus® dynamic random access memory (XDRAM) unit  1026 ; a Reality Synthesizer graphics unit  1030  with a dedicated video random access memory (VRAM) unit  1032 ; and an I/O bridge  1034 . The system unit  1000  also comprises a Blu Ray® Disk BD-ROM® optical disk reader  1040  for reading from a disk  1040   a  and a removable slot-in hard disk drive (HDD)  1036 , accessible through the I/O bridge  1034 . Optionally the system unit  1000  also comprises a memory card reader  1038  for reading compact flash memory cards, Memory Stick® memory cards and the like, which is similarly accessible through the I/O bridge  1034 . 
     The I/O bridge  1034  also connects to six Universal Serial Bus (USB) 2.0 ports  1024 ; a gigabit Ethernet port  1022 ; an IEEE 802.11b/g wireless network (Wi-Fi) port  1020 ; and a Bluetooth® wireless link port  1018  capable of supporting of up to seven Bluetooth connections. 
     In operation the I/O bridge  1034  handles all wireless, USB and Ethernet data, including data from one or more game controllers  1002 . For example when a user is playing a game, the I/O bridge  1034  receives data from the game controller  1002  via a Bluetooth link and directs it to the Cell processor  1028 , which updates the current state of the game accordingly. 
     The wireless, USB and Ethernet ports also provide connectivity for other peripheral devices in addition to game controllers  1002 , such as: a remote control  1004 ; a keyboard  1006 ; a mouse  1008 ; a portable entertainment device  1010  such as a Sony Playstation Portable® entertainment device; a video camera such as an EyeToy® video camera  1012 ; and a microphone headset  1014 . Such peripheral devices may therefore in principle be connected to the system unit  1000  wirelessly; for example the portable entertainment device  1010  may communicate via a Wi-Fi ad-hoc connection, whilst the microphone headset  1014  may communicate via a Bluetooth link. 
     The provision of these interfaces means that the Playstation 3 device is also potentially compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Voice over IP telephones, mobile telephones, printers and scanners. 
     In addition, a legacy memory card reader  1016  may be connected to the system unit via a USB port  1024 , enabling the reading of memory cards  1048  of the kind used by the Playstation® or Playstation 2® devices. 
     In the present embodiment, the game controller  1002  is operable to communicate wirelessly with the system unit  1000  via the Bluetooth link. However, the game controller  1002  can instead be connected to a USB port, thereby also providing power by which to charge the battery of the game controller  1002 . In addition to one or more analog joysticks and conventional control buttons, the game controller is sensitive to motion in six degrees of freedom, corresponding to translation and rotation in each axis. Consequently gestures and movements by the user of the game controller may be translated as inputs to a game in addition to or instead of conventional button or joystick commands. Optionally, other wirelessly enabled peripheral devices such as the Playstation Portable device may be used as a controller. In the case of the Playstation Portable device, additional game or control information (for example, control instructions or number of lives) may be provided on the screen of the device. Other alternative or supplementary control devices may also be used, such as a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown) or bespoke controllers, such as a single or several large buttons for a rapid-response quiz game (also not shown). 
     The remote control  1004  is also operable to communicate wirelessly with the system unit  1000  via a Bluetooth link. The remote control  1004  comprises controls suitable for the operation of the Blu Ray Disk BD-ROM reader  1040  and for the navigation of disk content. 
     The Blu Ray Disk BD-ROM reader  1040  is operable to read CD-ROMs compatible with the Playstation and PlayStation 2 devices, in addition to conventional pre-recorded and recordable CDs, and so-called Super Audio CDs. The reader  1040  is also operable to read DVD-ROMs compatible with the Playstation 2 and PlayStation 3 devices, in addition to conventional pre-recorded and recordable DVDs. The reader  1040  is further operable to read BD-ROMs compatible with the Playstation 3 device, as well as conventional pre-recorded and recordable Blu-Ray Disks. 
     The system unit  1000  is operable to supply audio and video, either generated or decoded by the Playstation 3 device via the Reality Synthesizer graphics unit  1030 , through audio and video connectors to a display and sound output device  1042  such as a monitor or television set having a display  1044  and one or more loudspeakers  1046 . The audio connectors  1050  may include conventional analogue and digital outputs whilst the video connectors  1052  may variously include component video, S-video, composite video and one or more High Definition Multimedia Interface (HDMI) outputs. Consequently, video output may be in formats such as PAL or NTSC, or in 720p, 1080i or 1080p high definition. 
     Audio processing (generation, decoding and so on) is performed by the Cell processor  1028 . The Playstation 3 device&#39;s operating system supports Dolby® 5.1 surround sound, Dolby® Theatre Surround (DTS), and the decoding of 7.1 surround sound from Blu-Ray® disks. 
     In the present embodiment, the video camera  1012  comprises a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding apparatus so that compressed video data may be transmitted in an appropriate format such as an intra-image based MPEG (motion picture expert group) standard for decoding by the system unit  1000 . The camera LED indicator is arranged to illuminate in response to appropriate control data from the system unit  1000 , for example to signify adverse lighting conditions. Embodiments of the video camera  1012  may variously connect to the system unit  1000  via a USB, Bluetooth or Wi-Fi communication port. Embodiments of the video camera may include one or more associated microphones that are also capable of transmitting audio data. In embodiments of the video camera, the CCD may have a resolution suitable for high-definition video capture. In use, images captured by the video camera may for example be incorporated within a game or interpreted as game control inputs. 
     In general, in order for successful data communication to occur with a peripheral device such as a video camera or remote control via one of the communication ports of the system unit  1000 , an appropriate piece of software such as a device driver should be provided. Device driver technology is well-known and will not be described in detail here, except to say that the skilled man will be aware that a device driver or similar software interface may be required in the present embodiment described. 
     Embodiments may include capturing depth data to better identify the real-world user and to direct activity of an avatar or scene. The object can be something the person is holding or can also be the person&#39;s hand. In this description, the terms “depth camera” and “three-dimensional camera” refer to any camera that is capable of obtaining distance or depth information as well as two-dimensional pixel information. For example, a depth camera can utilize controlled infrared lighting to obtain distance information. Another exemplary depth camera can be a stereo camera pair, which triangulates distance information using two standard cameras. Similarly, the term “depth sensing device” refers to any type of device that is capable of obtaining distance information as well as two-dimensional pixel information. 
     Recent advances in three-dimensional imagery have opened the door for increased possibilities in real-time interactive computer animation. In particular, new “depth cameras” provide the ability to capture and map the third-dimension in addition to normal two-dimensional video imagery. With the new depth data, embodiments of the present invention allow the placement of computer-generated objects in various positions within a video scene in real-time, including behind other objects. 
     Moreover, embodiments of the present invention provide real-time interactive gaming experiences for users. For example, users can interact with various computer-generated objects in real-time. Furthermore, video scenes can be altered in real-time to enhance the user&#39;s game experience. For example, computer generated costumes can be inserted over the user&#39;s clothing, and computer generated light sources can be utilized to project virtual shadows within a video scene. Hence, using the embodiments of the present invention and a depth camera, users can experience an interactive game environment within their own living room. Similar to normal cameras, a depth camera captures two-dimensional data for a plurality of pixels that comprise the video image. These values are color values for the pixels, generally red, green, and blue (RGB) values for each pixel. In this manner, objects captured by the camera appear as two-dimension objects on a monitor. 
     Embodiments of the present invention also contemplate distributed image processing configurations. For example, the invention is not limited to the captured image and display image processing taking place in one or even two locations, such as in the CPU or in the CPU and one other element. For example, the input image processing can just as readily take place in an associated CPU, processor or device that can perform processing; essentially all of image processing can be distributed throughout the interconnected system. Thus, the present invention is not limited to any specific image processing hardware circuitry and/or software. The embodiments described herein are also not limited to any specific combination of general hardware circuitry and/or software, nor to any particular source for the instructions executed by processing components. 
     With the above embodiments in mind, it should be understood that the invention may employ various computer-implemented operations involving data stored in computer systems. These operations include operations requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing. 
     The above-described invention may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The invention may also be practiced in distributing computing environments where tasks are performed by remote processing devices that are linked through a communications network. 
     The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data that can be thereafter read by a computer system, including an electromagnetic wave carrier. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion. 
     Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.