Patent Description:
The video game industry has seen many changes over the years. As computing power has expanded, developers of video games have likewise created game software that takes advantage of these increases in computing power. To this end, video game developers have been coding games that incorporate sophisticated operations and mathematics to produce very detailed and engaging gaming experiences.

Example gaming platforms include the Sony Playstation®, Sony Playstation2® (PS2), Sony Playstation3® (PS3), and Sony Playstation4® (PS4), each of which is sold in the form of a game console. As is well known, the game console is designed to connect to a display (typically a television) and enable user interaction through handheld controllers. The game console is designed with specialized processing hardware, including a CPU, a graphics synthesizer for processing intensive graphics operations, a vector unit for performing geometry transformations, and other glue hardware, firmware, and software. The game console may be further designed with an optical disc reader for receiving game discs for local play through the game console. Online gaming is also possible, where a user can interactively play against or with other users over the Internet. As game complexity continues to intrigue players, game and hardware manufacturers have continued to innovate to enable additional interactivity and computer programs.

A growing trend in the computer gaming industry is to develop games that increase the interaction between the user and the gaming system. One way of accomplishing a richer interactive experience is to use wireless game controllers whose movement is tracked by the gaming system in order to track the player's movements and use these movements as inputs for the game. Generally speaking, gesture input refers to having an electronic device such as a computing system, video game console, smart appliance, etc., react to some gesture made by the player and captured by the electronic device.

Another way of accomplishing a more immersive interactive experience is to use a head-mounted display (HMD). A head-mounted display is worn by the user and can be configured to present various graphics, such as a view of a virtual space. The graphics presented on a head-mounted display can cover a large portion or even all of a user's field of view. Hence, a head-mounted display can provide a visually immersive experience to the user.

A head-mounted display (HMD) provides an immersive virtual reality experience, as the HMD renders a three-dimensional real-time view of the virtual environment in a manner that is responsive to the user's movements. The user wearing an HMD is afforded freedom of movement in all directions, and accordingly can be provided a view of the virtual environment in all directions via the HMD. The processing resources required to generate high quality video (e.g. at high resolution and frame rate) for rendering on the HMD are considerable and may therefore be handled by a separate computing device, such as a personal computer or a game console. In such systems, the computing device generates the video for rendering to the HMD, and transmits the video to the HMD.

However, when wearing an HMD, the user is unable to see the local environment in which they are situated.

It is in this context that implementations of the disclosure arise. Previously proposed arrangements are disclosed in <CIT>, <NPL>, <NPL>, <CIT> and <CIT>.

Implementations of the present disclosure include devices, methods and systems relating to space capture, modeling, and texture reconstruction through dynamic camera positioning and lighting using a mobile robot.

A method is provided, including the following method operations: using a robot having a plurality of sensors to acquire sensor data about a local environment; processing the sensor data to generate a spatial model of the local environment, the spatial model defining virtual surfaces that correspond to real surfaces in the local environment; further processing the sensor data to generate texture information that is associated to the virtual surfaces defined by the spatial model; tracking a location and orientation of a head-mounted display (HMD) in the local environment; using the spatial model, the texture information, and the tracked location and orientation of the HMD to render a view of a virtual space that corresponds to the local environment; presenting the view of the virtual environment through the HMD.

The location of the HMD in the local environment defines a perspective from which the view of the virtual space is rendered.

In some implementations, the orientation of the HMD in the local environment defines a direction of the view of the virtual space.

In some implementations, rendering the view of the virtual space includes rendering one or more of the virtual surfaces, which are defined by the spatial model, using the texture information associated to the one or more of the virtual surfaces.

In some implementations, the sensors include at least one image capture device and at least one depth camera, and wherein the sensor data includes image data captured by the image capture device and depth data captured by the depth camera.

The texture information includes one or more of a diffuse map, a bump map, and/or a specular map.

In some implementations, using the robot to acquire sensor data includes moving the robot to a plurality of locations within the local environment and using the sensors of the robot at each of the locations to sense the local environment and generate the sensor data.

Acquiring the sensor data includes capturing images of a real surface in the local environment from a plurality of angles; and, processing the sensor data to generate the texture information includes processing the images captured from the plurality of angles to generate texture information for a given virtual surface defined by the spatial model that corresponds to the real surface.

In some implementations, a method is provided, including: using a robot to effect a plurality of lighting conditions in a local environment and using a plurality of sensors of the robot to acquire sensor data about the local environment under the plurality of lighting conditions; processing the sensor data to generate a spatial model of the local environment, the spatial model defining virtual surfaces that correspond to real surfaces in the local environment; further processing the sensor data to generate texture information that is associated to the virtual surfaces defined by the spatial model.

In some implementations, using the robot to effect the plurality of lighting conditions includes accessing a home lighting control system by the robot to control one or more lights in the local environment.

In some implementations, using the robot to effect the plurality of lighting conditions includes using a light included in the robot to illuminate at least a portion of the local environment.

In some implementations, using the robot to effect the plurality of lighting conditions includes moving the robot to one or more locations so as to block light from a light source in the local environment from directly reaching a surface in the local environment.

In some implementations, the texture information includes one or more of a diffuse map, a bump map, and/or a specular map.

In some implementations, a method performed by a robot in a local environment is provided, including: capturing a first image of the local environment by an image capture device of the robot positioned at a first location in the local environment, wherein capturing the first image includes capture of a real surface in the local environment; processing the first image to determine texture information of the real surface, and further determine that a possible error exists in the determined texture information of the real surface; in response to determining the possible error, moving the robot to a second location, and capturing a second image of the local environment by the image capture device at the second location, wherein capturing the second image includes capture of the real surface from a perspective defined from the second location; processing the second image to verify the possible error in the determined texture information of the real surface, and correct the possible error in the determined texture information of the real surface.

In some implementations, processing the second image to verify the possible error in the determined texture information of the real surface includes comparing a portion of the first image to a corresponding portion of the second image, the portions of the first and second images being of a same region of the real surface.

In some implementations, correcting the possible error includes replacing a portion of the determined texture information of the real surface based on the processing of the second image.

In some implementations, the texture information defines one or more of color and surface structure.

In some implementations, method performed by a robot in a local environment is provided, including: capturing a first depth image of the local environment by a depth camera of the robot positioned at a first location in the local environment; processing the first depth image to determine a spatial structure of the local environment, and further determine that a possible error exists in the determined spatial structure of the local environment; in response to determining the possible error, moving the robot to a second location, and capturing a second depth image of the local environment by the depth camera at the second location, wherein capturing the second depth image includes capture of the local environment from a perspective defined from the second location; processing the second image to verify the possible error in the determined spatial structure, and correct the possible error in the determined spatial structure.

In some implementations, processing the second depth image to verify the possible error in the determined spatial structure includes comparing a portion of the first depth image to a corresponding portion of the second depth image, the portions of the first and second depth images being of a same region of the local environment.

In some implementations, correcting the possible error includes replacing a portion of the determined spatial structure of the local environment based on the processing of the second image.

Other aspects and advantages of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

The disclosure may be better understood by reference to the following description taken in conjunction with the accompanying drawings in which:.

The following implementations of the present disclosure provide devices, methods, and systems relating to space capture, modeling, and texture reconstruction through dynamic camera positioning and lighting using a mobile robot. It will be obvious, however, to one skilled in the art, that the present disclosure may be practiced without some or all of the specific details presently described. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.

<FIG> illustrates a system for three-dimensional (3D) spatial and texture reconstruction viewed through a head-mounted display (HMD), in accordance with implementations of the disclosure. In the illustrated implementation, a user <NUM> is shown interacting with a view of a virtual space/environment that is rendered on a head-mounted display <NUM>. By way of example without limitation, one example of an HMD is the PlayStation®VR headset. In some implementations, the virtual space is that of a video game. In other implementations, the virtual space is that of any type of application or platform that provides a virtual space or virtual environment with which the user may interact, including without limitation, locally executed interactive applications, cloud executed applications, cloud platforms, social networks, websites, telecommunications platforms, video conferencing, online chatrooms, etc. It will be appreciated that such applications or platforms supporting a virtual space can be configured to accommodate multiple users interacting in the same virtual space simultaneously.

In some implementations, the interactive application (e.g. a video game) that generates the virtual space is executed by a local computing device <NUM>. The computing device can be any kind of device that may be configured to execute the interactive application to generate the virtual space, including without limitation, a gaming console, personal computer, laptop computer, set-top box, tablet, cellular phone, portable gaming device, etc. In some implementations, the computing device <NUM> is connected to a network, such as a local area network, wide area network, WiFi network, cellular network, the Internet, etc..

In some implementations, the computing device <NUM> is a thin client that communicates over the network (e.g. the Internet) with a cloud services provider to obtain the view of the virtual space that is rendered on the HMD <NUM>. That is, the interactive application is executed by the cloud services provider to generate the virtual space, and video data depicting the primary view of the virtual space is streamed over the network (e.g. the Internet) to the computing device <NUM>, which then processes the video data to render the view to the HMD <NUM>.

In some implementations, the functionality of the computing device <NUM> is incorporated into the HMD <NUM> or the display <NUM>.

In order to interact with the virtual space that is viewed through the HMD <NUM>, the user <NUM> may operate an input device <NUM>. The input device <NUM> can be any type of device useful for providing input to interact with the virtual space, including without limitation, a controller, motion controller, keyboard, mouse, trackpad, pointer, joystick, gaming peripheral, etc. In some implementations, wherein the virtual space is of a video game, the input device <NUM> enables the user <NUM> to provide input to the video game, to effect changes in the game state of the video game, such as by controlling actions (e.g. of a character or other virtual object) in the video game's context of gameplay. By way of example without limitation, examples of input devices can include video game controller devices such as the DualShock®<NUM> Wireless Controller, the PlayStation®Move Motion Controller, and the Playstation®VR Aim Controller.

In some implementations, an image capture device <NUM> is configured to capture images of the interactive local environment <NUM> in which the system is disposed. One example of an image capture device is the PlayStation®Camera. The computing device <NUM> can be configured to process and analyze the captured images to, by way of example without limitation, determine the location/orientation of an object in the local environment <NUM>, such as the input device <NUM>. In some implementations, the input device <NUM> may include a trackable feature, such as a light or other recognizable feature, that is recognized in the captured images and tracked, thereby providing for tracking of the location/orientation of the input device <NUM> in the local environment <NUM>. Furthermore, images captured by the image capture device <NUM> may be analyzed to identify and track the user <NUM>.

As noted above, because the user <NUM> is wearing the HMD <NUM>, the user <NUM> is not able to see the local environment <NUM>. Therefore, it is useful to capture and model the local environment, including any surfaces/objects within the local environment. Broadly speaking, this entails capturing and modeling the 3D spatial structures of surfaces/objects, and also capturing and modeling the textures of such surfaces/objects, so that a faithful representation of the local environment <NUM> can be rendered to the user <NUM>. The process of capturing and modeling a 3D real space or object is known as 3D reconstruction. It will be appreciated that such a model of the local environment <NUM> can also have other uses, such as to enable a remote virtual reality user to experience the user <NUM>'s local environment <NUM> (e.g. enabling the remote virtual reality user is able to virtually "visit" the local environment <NUM> of the user <NUM>), augmenting or altering a rendering of the local environment <NUM> with additional graphics or content, etc..

Broadly speaking, in accordance with implementations of the disclosure, a robot <NUM> is used to enable modeling of the local environment <NUM>, including modeling the spatial structure of the local environment <NUM> and the textures of surfaces in the local environment <NUM>. Such models can be used to render a view of a virtual space/environment (e.g. by the computing device <NUM>) that is a 3D reconstruction of the local environment <NUM>. This view can be presented through the HMD <NUM> to the user <NUM>, to enable the user <NUM> to view the virtual space in a manner that simulates their real-world position in the actual local environment <NUM>. That is, the location and orientation of the HMD <NUM> in the local environment <NUM> are tracked, and the view of the virtual space presented through the HMD <NUM> is rendered using the models of the local environment <NUM>, with the location and orientation of the HMD <NUM> in the local environment <NUM> determining the perspective location and angular direction in the spatial model that are used to render the view of the virtual space, by way of example without limitation. In this manner, the view of the virtual space provided through the HMD <NUM> to the user <NUM> can mimic the real-world view as if the user <NUM> were viewing the actual local environment <NUM> without wearing the HMD <NUM>.

In accordance with implementations of the disclosure, the robot <NUM> is utilized to spatially and texturally capture the local environment <NUM>, to enable 3D reconstruction of the local environment <NUM>. In the illustrated implementation, the local environment <NUM> is defined by a room in which the user <NUM> is situated. However, it will be appreciated that in other implementations the local environment <NUM> can be any other type of real space, setting or location in which the user <NUM> may be situated.

In accordance with implementations of the disclosure, the 3D reconstruction process entails generation of a point cloud, which is a set of data points that are defined by the 3D coordinates of points along the external surfaces of objects in the local environment. The point cloud is processed to define a polygon mesh, typically consisting of triangles, quadrilaterals, or other polygons. The polygon mesh is defined by a set of vertices, edges that connect the vertices, and faces that are the polygons formed from the edges. The vertices can include the data points of the point cloud, and/or other points that are determined based on the data points of the point cloud. The polygon mesh defines a 3D spatial model of the surfaces of the local environment. At rendering, textures are applied to the 3D mesh to form the rendered graphical depiction of the local environment.

As noted above, a robot <NUM> can be used to capture the materials of an object and enable the system to virtually recreate them. In this manner, it is possible to create a holographic space or recreate a real space in a virtual world that is as accurate as possible. With a moveable robot it is possible to obtain different images, e.g. at different angles and/or under different lighting conditions, that can overcome issues such as lighting conditions, glare, etc. so that the system can more accurately recreate textures of an object than that possible using a static camera or a camera taking a sweep of a room.

As used herein, "texture" refers to the properties of a real or virtual surface that characterize, affect or determine the surface's appearance. By way of example without limitation, such properties can include the 3D surface structure, color, reflectance, transparency, translucence, etc. In the context of computer graphics rendering, the application of texture to a virtual surface (e.g. a surface of a 3D model, such as a polygon of a polygon mesh) is referred to as texture mapping. Texture mapping can encompass many types of surface-defining techniques, including by way of example without limitation, diffuse mapping, height mapping, bump mapping, normal mapping, displacement mapping, reflection mapping, specular mapping, mipmaps, occlusion mapping, etc. It will be appreciated that texture mapping can utilize a procedural texture that creates a texture using a model or mathematical description. Such a model can be determined from captured data by the robot <NUM> in accordance with implementations of the disclosure described herein.

Thus, as shown in the illustrated implementation, the robot <NUM> is configured to capture the 3D spatial structure of the local environment <NUM>, including by way of example without limitation, the spatial structure of any objects in the local environment <NUM> such as walls <NUM> and <NUM>, the floor <NUM>, a rug <NUM>, the display <NUM> (e.g. a television), a media stand/cabinet <NUM>, etc. To accomplish this, the robot <NUM> can be configured to scan the local environment <NUM> with one or more sensors, and from different locations within the local environment <NUM>, to enable capture of the 3D spatial structure of the local environment <NUM>.

For example, the robot <NUM> may include one or more depth cameras (or range imaging devices/sensors) that are capable of determining the distances of objects from the depth camera. It will be appreciated that the depth camera can be any kind of range imaging device, such as a time-of-flight camera (e.g. using controlled infrared (IR) lighting), LIDAR, a stereo camera (and using stereo triangulation), etc. Additionally, the robot <NUM> may include one or more image capture devices (e.g. visible light cameras) for capturing images/video of the local environment <NUM>. Further, the robot <NUM> may include various motion sensors (e.g. accelerometers, gyroscopes, magnetometers, inertial motion units (IMU's), network positioning devices (e.g. GPS, WiFi positioning), etc. that can be utilized to track the position and orientation of the robot <NUM> within the local environment <NUM>.

Utilizing such sensors, the robot <NUM> can map the 3D spatial structure of the local environment <NUM>, by capturing images and data from various locations and/or as the robot <NUM> is moved throughout the local environment <NUM>. In some implementations, the 3D spatial structure of the local environment <NUM> is modeled by generating a 3D model, such as a 3D point cloud and/or a polygon mesh model as described above. By way of example without limitation, the robot <NUM> may utilize any of various techniques for mapping or determining the 3D spatial structure, such as a simultaneous localization and mapping (SLAM) technique.

As noted, a texture is applied inside of a virtual space to a surface of a virtual object. When capturing texture, the goal is to capture the properties of a material to enable the system to recreate it as accurately as possible. In some implementations, the texture for a given surface is defined by a texture map, which may include one or more types of surface properties embodied in surface property maps. By way of example without limitation, these may include a displacement map (e.g. identifying crevices or other types of displacement in a surface), specular map (identifying shininess of a surface, and/or how a surface responds to lighting, glare, etc.), fresnel (for transparent/translucent objects, how light is reflected or refracted/transmitted by an object based on angle of view), etc. These types of surface texture properties can be captured by the robot <NUM> and accurately modeled and recreated. The ability of the robot <NUM> to capture images from different angles enables more accurate capture of a given surface's properties. Furthermore, as discussed in further detail below, the given surface may be captured under different and/or controlled lighting conditions to further enhance the accuracy of the textural capture of the surface.

A given surface in the local environment <NUM> is identified, and a representative portion of the identified surface is sampled to determine the texture of the surface. That is, the texture of the representative portion is captured and modeled, and when a virtual representation of the surface is rendered for viewing (e.g. through the HMD <NUM> or another display), the modeled texture is applied for the entirety of the surface.

In some implementations, prior to sampling a representative portion of a given surface, it is first determined that the surface, or a substantial portion thereof, has substantially the same or similar texture throughout. In other words, the surface is determined to have a substantially consistent texture throughout its area. By way of example without limitation, this may be ascertained by determining that the surface has a substantially consistent color or pattern of colors, reflectance, displacement, or other textural property. It will be appreciated that such a determination may be made at a lower or more approximate level of detail and/or sensitivity as compared to the level of detail/sensitivity that is to be applied when capturing the texture of the representative portion of the surface. For example, in some implementations, when evaluating a surface to determine whether it is of a consistent texture, fewer textural properties may be considered than when a representative sample is being texturally captured. In some implementations, for a given textural property, a lower resolution, sampling frequency, or per unit area level of discrimination is applied when evaluating the surface to determine whether it is of a consistent texture, as compared to when the given textural property is captured for a representative portion (or sample or region) of the surface. Thus, a determination is made as to whether the surface is substantially consistent in texture, and if so, then a representative portion of the surface is sampled to capture its texture in detail.

To determine whether a given surface (or portion/region thereof) is substantially consistent in texture, one or more threshold determinations may be applied. For example, in some implementations, a given surface may be determined to have a consistent texture if the sensed color of the surface (or a portion thereof), for example, as determined from analyzing captured images of the surface, varies by less than a predefined amount. In some implementations, a similar determination for other textural properties can be applied. In some implementations, multiple textural properties are evaluated, and it is determined whether the combined (e.g. weighted) variance of the properties is less than a predefined amount, and if so, then the surface (or portion thereof) is determined to have a consistent texture.

It should be appreciated that one or more regions of a given surface may be identified as having a similar or the same or consistent texture, and that a representative sample/portion of such regions can then be scanned in detail to capture the texture of such regions. Furthermore, object recognition can be applied to enhance the identification. For example, a vertical planar surface could be recognized as being a wall, and therefore identified for texture sampling. It will be appreciated that by sampling the texture of a representative portion of a surface, as the representative portion is much smaller than the entirety of the surface, resources are conserved because texture information for the entire surface need not be stored in order to provide realistic rendering of the surface in a virtual space. Rather, the sampled texture information can be applied, e.g. via a modeled texture map, for the entire surface when rendered. In this manner, a realistic rendering of the surface can be provided without requiring capture of detailed texture information for the entire surface, thus reducing memory storage requirements and speeding up the capture process as less surface area is required to be captured in detail, which reduces the amount of processing required as well.

With continued reference to <FIG>, for example, it may be determined based on captured images of the local environment <NUM> that the walls <NUM> or <NUM>, the floor <NUM>, and/or the rug <NUM>, each have a substantially consistent texture throughout their respective surfaces. The robot <NUM> is configured to capture the textures of representative portions of these surfaces in detail. For example, the robot <NUM> may capture in detail the texture of a representative portion of the floor <NUM>, and model the texture. Then when the floor <NUM> is graphically rendered, it is rendered using the model of the texture to texture map the floor's virtual representation in the virtual space. A similar process can be applied for the other surfaces of the local environment <NUM>.

It should be appreciated that any of the process operations described herein (including by way of example without limitation, processing of data, modeling of space/objects/textures, rendering of a view of a virtual space, etc.), unless specifically described or otherwise apparent from the present disclosure as being performed by a specific device, can be performed by any of the devices described herein, including by way of example without limitation, the robot <NUM>, the computing device <NUM>, the HMD <NUM>, or a cloud computing device. For example, in some implementations, the capture and modeling of the local environment <NUM> is performed by the robot <NUM>. Whereas in other implementations, the capture of the local environment <NUM> is performed by the robot <NUM> while the modeling of the local environment <NUM> is performed by the computing device <NUM>. Not all permutations of the division of processing operations amongst the available devices in the systems of the present disclosure are described in detail herein. However, it will be appreciated that such permutations are within the scope of the present disclosure.

In some implementations, in order to allow spectators to see what the user <NUM> is seeing through the HMD <NUM>, the view (or a portion thereof) that is rendered on the HMD <NUM> can also be rendered on the display device <NUM>. Thus, the user <NUM> is able to spectate the user <NUM>'s view by viewing the display device <NUM>. In some implementations, the robot <NUM> may project onto an available projection surface (e.g. a region of a wall) a secondary view of the virtual space.

<FIG> illustrates a robot capturing an object from different angles, to enable 3D reconstruction of the object, in accordance with implementations of the disclosure. In the illustrated implementation, an object <NUM> is a couch in the local environment <NUM>. It will be appreciated that in various implementations the object <NUM> can be any other type of object in the local environment <NUM>. The robot <NUM> is configured to capture the object <NUM> (e.g. using image sensors, depth cameras, or other sensors) from different locations, thereby capturing the object <NUM> from different angles. Using the captured information, the 3D structure of the object <NUM> can be modeled, e.g. using a polygon mesh model. Furthermore, the texture of the surfaces of the object <NUM> can be captured and modeled as well. It will be appreciated that by capturing the object <NUM> from multiple angles/directions, more accurate modeling of the 3D structure and surface texture is possible.

In some implementations, the object <NUM> is recognized based on an object recognition process, e.g. applied to captured images of the local environment <NUM> and/or captured depth information. Once recognized, the object <NUM> may be identified for further capture in greater detail from multiple directions using the robot <NUM>.

In some implementations, the robot <NUM> is configured to capture the object <NUM> from a plurality of predefined angles/directions and/or distances. For example, the robot <NUM> may be configured to capture data at, by way of example without limitation, <NUM> degree intervals (e.g. zero, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> degrees). That is, the robot <NUM> moves around the object <NUM> to different positions in the local environment <NUM> corresponding to the predefined angular intervals, thus systematically changing the angle from which the object <NUM> is captured. In some implementations, the angle of capture can be defined relative to a determined center of the object <NUM> or another reference point of the object <NUM> or the local environment <NUM>. In some implementations, the robot <NUM> is also configured to acquire multiple captures at a predefined distance from the object <NUM> or its determined center or other reference point. In various implementations the robot <NUM> can be configured to capture the object <NUM> from any plurality of angles, at any systematic intervals or otherwise. It will be appreciated that using multiple captures of the object <NUM> from different angles, then the 3D spatial structure of the object <NUM> and its surface texture can be better captured.

With continued reference to <FIG>, the robot <NUM> is shown at an initial position P<NUM>, from which it captures the object <NUM>. The robot <NUM> moves around the object <NUM> (e.g. laterally and/or circumferentially, relative to the object <NUM>) to a position P<NUM>, from which the robot captures the object <NUM>. Then the robot <NUM> further moves around the object <NUM> to a position P<NUM>, from which it captures the object <NUM>. In capturing the object <NUM> from the various positions P<NUM>, P<NUM>, and P<NUM>, the robot <NUM> obtains captured images, depth information, and/or other types of sensed information from different angles and perspectives surrounding the object <NUM>. These can be analyzed to determine the 3D structure and textures of surfaces of the object <NUM>.

In some implementations, the positions P<NUM>, P<NUM>, and P<NUM> are configured to be located along a circumference surrounding the object <NUM> at a predefined distance (e.g. radius from a center or other reference point of the object <NUM>), and angularly separate from one another at predefined intervals as described above.

In some implementations, the robot <NUM> can be configured to affect the lighting of the object <NUM> to improve the capture of the object's structure and/or texture. For example, in some implementations, the robot <NUM> can include a light (or multiple lights) which may be operated to provide further illumination of the object <NUM>. This may be useful in various situations, such as when ambient lighting conditions are low (e.g. below a predefined ambient light threshold), or when certain portions of the object <NUM> are poorly illuminated (e.g. regions of the object <NUM> that are in shadow), etc..

<FIG> conceptually illustrates an overhead view of a robot moving to various positions in a local environment to capture the texture of a surface, in accordance with implementations of the disclosure. As shown, and in accordance with some implementations, the robot <NUM> can be configured to identify a surface <NUM>, and capture the texture of a representative portion <NUM> of the surface <NUM>. In some implementations, capture of the texture entails capturing images of the representative portion <NUM> from predefined positions/orientations relative to the representative portion <NUM> of the surface <NUM>.

For example, the robot <NUM> may move to a position Q<NUM> to capture one or more images of the representative portion <NUM>, in a direction D<NUM> towards a center C of the representative portion <NUM>, that is substantially normal/perpendicular to the surface <NUM>. In some implementations, the position Q<NUM> is defined at a predefined distance L from the surface <NUM>. Furthermore, the robot <NUM> may also capture images from a position Q<NUM>, which is positioned so as to enable capture of images of the representative portion <NUM> in a direction D<NUM> (towards the center C of the representative portion <NUM>) at a predefined angle A<NUM> relative to the surface <NUM> (or a predefined angle relative to normal to the surface <NUM>). The robot <NUM> may also capture images from a position Q<NUM>, which is positioned so as to enable capture of images of the representative portion <NUM> in a direction D<NUM> (towards the center C of the representative portion <NUM>) at a predefined angle A<NUM> relative to the surface <NUM> (or a predefined angle relative to normal to the surface <NUM>). As shown, the positions Q<NUM> and Q<NUM>, and their corresponding angles A<NUM> and A<NUM>, are on opposite sides of the center C of the representative portion <NUM>. In some implementations, the positions Q<NUM> and Q<NUM> are also configured to be located at the same distance L from the center C of the representative portion <NUM>; whereas in other implementations, they may be located at other distances.

By way of example without limitation, in some implementations, the predefined angle A<NUM> and/or A<NUM> is approximately <NUM> degrees relative to the surface <NUM>. In effect, this means that images of the representative portion <NUM> of the surface <NUM> are captured from angles of approximately <NUM>, <NUM>, and <NUM> degrees, as measured from the same side of the representative portion <NUM>. In other implementations, the predefined angle A<NUM> and/or A<NUM> is in the range of about <NUM> to <NUM> degrees, by way of example without limitation.

While in the foregoing implementation images of the representative portion <NUM> of the surface <NUM> are captured from three different angles, it will be appreciated that in other implementations, images of the representative portion <NUM> may be captured from any number of different angles. Furthermore, while in the foregoing implementation images are captured from positions that are substantially vertically aligned, in other implementations, images may be captured from positions that are not necessarily vertically aligned with each other. In such implementations, the robot <NUM> may be capable of maneuvering a camera to different elevations/heights, and articulating the camera to direct it towards the center of the representative portion of the surface.

<FIG> illustrates a robot having multiple cameras capable of capturing images from multiple perspectives, in accordance with implementations of the disclosure. As shown, the robot <NUM> includes a height-adjustable upper camera <NUM> that can be raised and lowered to different heights/elevations/vertical positions. The upper camera <NUM> is also articulated to enable adjustment of the angle of the camera <NUM>. The robot <NUM> further includes a lower camera <NUM> that is positioned along the robot's body at a lower height than the camera <NUM>. The lower camera <NUM> may also be articulated to enable adjustment of its angular direction. Thus, the cameras <NUM> and <NUM> can be operated (simultaneously) to capture images of a representative portion <NUM> of a surface <NUM> from different vertical positions. The upper camera can be adjusted up and down to capture images from additional different vertical positions.

As further shown in the illustrated implementation, the robot <NUM> can move laterally side-to-side relative to the surface <NUM> to enable capture of the representative portion <NUM> from different horizontal positions.

Utilizing images captured from various angles (and under various controlled lighting conditions) the robot <NUM> can more accurately capture the texture of the representative portion <NUM> of the surface <NUM>. More specifically, the images captured from different angles and positions relative to the surface <NUM> can be analyzed to determine the texture of the surface <NUM>, as defined by one or more texture maps that are generated based on the analysis and associated to the corresponding surface of the 3D model of the local environment <NUM>. As noted above, this may include by way of example without limitation, diffuse mapping, height mapping, bump mapping, normal mapping, displacement mapping, reflection mapping, specular mapping, mipmaps, occlusion mapping, etc..

As images are captured from different locations within the local environment <NUM> as described above, it will be appreciated that the location and orientation of the robot <NUM> in the local environment <NUM> can be determined and tracked to enable a precise understanding of the perspective from which captured images are obtained. That is, the (3D) position and angular orientation of the robot <NUM> and/or an image capture device of the robot <NUM> can be determined in the local environment <NUM> and relative to the surface being captured.

In some implementations, the position/orientation of the robot <NUM> is determined, at least in part, based on information sensed or processed by the robot itself, including by way of example without limitation, data from motion sensors (e.g. accelerometers, gyroscopes, magnetometers, inertial motion units (IMU's), wheel sensors that sense movement of wheels of the robot <NUM>, images captured by an image capture device of the robot <NUM>, network positioning (e.g. GPS, WiFi positioning), simultaneous localization and mapping (SLAM), etc. In some implementations, the location/orientation of the robot <NUM> is determined, at least in part, based on analysis of images captured by the image capture device <NUM>. In some implementations, the robot <NUM> includes one or more magnetic sensors configured to sense one or more magnetic fields emitted by one or more magnetic emitters positioned in the local environment <NUM>, and the location/orientation of the robot <NUM> can be determined, at least in part, based on such data. Additionally, the robot <NUM> can be configured to sense its position/orientation based, at least in part, on having mapped and/or modeled the 3D structure of the local environment <NUM>, e.g. using object recognition and correspondence to the modeled environment to determine position/orientation.

In some implementations, for purposes of capturing the texture of a representative portion of a surface in the local environment <NUM>, the position of the robot <NUM> relative to the representative portion of the surface is determined and tracked. For example, the robot <NUM> may use any of the above-described methods for tracking position/orientation to specifically track its position/orientation relative to the representative portion of the surface. Further, the robot <NUM> may specifically track the representative portion and/or the surface. By tracking the position/orientation relative to the representative portion of the surface, captured images of the representative portion can be properly analyzed (e.g. corresponding points can be determined).

As noted, in some implementations, the robot <NUM> can include a light which can be used to illuminate a surface for texture capture. In some implementations, such a light is used to illuminate a representative portion of the surface, and using the known angle of illumination by the light striking surface, captured images from a known perspective/vantage point can be analyzed to determine the texture of the surface.

<FIG> illustrates a robot configured to capture an object to enable 3D reconstruction of the object, in accordance with implementations of the disclosure. In the illustrated implementation, the robot <NUM> includes arms 200a and 200b, each of which has multiple articulated joints that enable the arms 200a and 200b to be maneuvered in practically any direction. In some implementations, the arms 200a/b are further extendable. The arms 200a and 200b may include one or more lights, and one or more cameras, which may be maneuvered by maneuvering the arms 200a and 200b.

In the illustrated implementation, the arm 200a includes a light <NUM> that is activated to provide illumination, and the arm 200b includes a camera <NUM> configured to capture images. In some implementations, the light <NUM> is maneuvered while the camera <NUM> remains in a fixed position and orientation, capturing images as the angle of the lighting provided by the light <NUM> changes. In other implementations, the light <NUM> is held in a fixed position and orientation, while the camera <NUM> is maneuvered, changing the angle of the camera as the lighting is held steady. In still other implementations, both the light <NUM> and the camera <NUM> can be maneuvered, either in turn or even simultaneously, as the light <NUM> is activated and as the camera <NUM> captures images of an object or surface.

In the illustrated implementation, the camera <NUM> is being used to capture images of the object <NUM>, while the lighting is controlled by using the light <NUM> to illuminate the object <NUM>. It will be appreciated that by capturing images of the object <NUM> from different angles and using lighting from different angles, a more robust modeling of the structure and texture of the object <NUM> can be achieved.

Another way that the robot <NUM> may influence the lighting of the object <NUM> (or a given surface in the local environment <NUM>) is by producing shadows. In some implementations, the robot <NUM> may be configured to maneuver itself so as to produce a shadow falling on the object <NUM>. In other words, the robot <NUM> may move to position in the local environment <NUM> that places it between the object <NUM> and a light source, so as to physically block at least a portion of the light from the light source from illuminating the object <NUM> or a given surface. In this manner, the robot <NUM> may reduce the illumination of the object <NUM>.

Additionally, in some implementations, the robot <NUM> is configured to physically maneuver an object to enable capture of its structure and texture. In the illustrated implementation, the arms 200a and 200b include claws 206a and 206b, respectively, which can be used to grip and maneuver the object <NUM>, changing its orientation or position relative to the robot <NUM> so that the robot <NUM> can capture images (or other sensed data) of different portions of the object <NUM>, and capture images/data from different angles and positions relative to the object and its surfaces.

<FIG> illustrates a robot <NUM> in a local environment having various features including controllable lights, in accordance with implementations of the disclosure. In the illustrated implementation, the local environment <NUM> is defined by a room in which the robot <NUM> is disposed. The room further includes a number of lights that are remotely controllable through a lighting control system. As shown, lights 300a, 300b, and 300c are in the form of recessed ceiling lights, while light <NUM> is in the form of a lamp. In various implementations, there may be any number and type of lights that are remotely controllable through a lighting control system. In some implementations, the lighting control system is a home automation system. In some implementations, the lighting control system is wirelessly accessible over a home network, such as a WiFi network, or using other wireless technologies, such as Bluetooth communications. In some implementations, the lighting control system is defined by one or more smart devices that enable control of the lights, such as a smart switch or smart outlet. In some implementations, the lights themselves are smart devices capable of networked communication, or the lights include smart bulbs that are similarly capable of networked communication.

In some implementations, the robot <NUM> communicates with the lighting control system to control the state of the lights, including the on/off state and the intensity of the lights. For example, the robot <NUM> may communicate over a WiFi network with the lighting control system to adjust the intensity of the various lights. More specifically, the robot <NUM> may control the lights so as to provide more or less illumination for purposes of capturing the 3D spatial structure and textures of the local environment <NUM>. This can be useful in overcoming adverse lighting conditions when attempting to capture the local environment <NUM>.

For example, there may be a window <NUM> through which high intensity light, such as direct or indirect sunlight, enters the room. The high intensity of the light coming through the window <NUM> can lead to high contrast in the local environment <NUM> and strong shadows or other effects that may make it difficult for image sensors to accurately capture the structures and textures of the local environment <NUM>. In another scenario, the lighting in the local environment <NUM> may be inadequate or less than optimal for image capture of at least some objects or surfaces by the robot <NUM> (e.g. requiring high gain by an image sensor, which tends to be noisy). In yet another scenario, there may be too much light for image capture of at least some objects or surfaces or regions thereof.

Therefore, the robot <NUM> can be configured to communicate with the lighting control system to adjust the on/off state and/or intensity of various ones of the lights to overcome such lighting issues. In some implementations, the robot <NUM> communicates with the lighting control system to turn on/off and/or adjust the intensity of one or more lights in order to normalize the lighting condition to the extent possible, for the local environment <NUM> and/or for one or more objects or surfaces in the local environment <NUM> and/or for a sub-region thereof. It will be appreciated that normalization of lighting can be variously defined in various implementations. For example, in some implementations, normalization of lighting is defined by a target amount (or target range) of light in the local environment <NUM> or for a region/object/surface thereof. In some implementations, normalization of lighting is defined by a target level or target range of contrast or dynamic range. In some implementations, normalization of lighting is defined with reference to a selected region of space in the local environment <NUM>, or a region that is captured by an image capture device of the robot <NUM>.

It will be appreciated that for purposes of normalizing the lighting condition, the amount of light or the lighting condition can be measured or determined using one or more light sensors and/or image capture devices of the robot <NUM>.

In some implementations, the robot <NUM> is configured to determine the locations of lights within the local environment <NUM>, and use the locations to affect the lighting in the local environment <NUM>. For example, in the illustrated implementation, the locations of lights 300a/b/c and <NUM> may be determined to have 3D coordinates (x<NUM>, y<NUM>, z<NUM>), (x<NUM>, y<NUM>, z<NUM>), (x<NUM>, y<NUM>, z<NUM>), and (x<NUM>, y<NUM>, z<NUM>), respectively. In some implementations, the robot <NUM> may determine the locations of the lights based on analyzing captured images, captured depth data, and further based on controlling the on/off state and/or intensity level of the lights through the lighting control system.

Using the known locations of the lights, the robot <NUM> may control their illumination so as to affect the lighting in the local environment <NUM> in a desired manner. For example, when capturing the texture of a surface, one or more of the lights can be controlled so as to increase or decrease the amount of illumination provided by the lights, and the direction of illumination by a given light relative to the surface can be determined from the known positions of the lights and the orientation and position of the surface being examined. Different lights can be controlled to provide different lighting amounts from different directions, enabling capture of more complete texture data for the surface. Furthermore, illumination can be provided from specific directions based on the known locations of the lights, to overcome issues such as insufficient lighting in particular regions of the local environment <NUM>.

In some implementations, a device such as the HMD <NUM> or controller <NUM> may be tracked based on detection of a magnetic field. The magnetic field may be emitted by a peripheral device in the local environment <NUM>, which in some implementations may be connected to, and/or controlled by, the computing device <NUM>. In some implementations, the magnetic field is emitted by an emitter included in the image capture device <NUM>. It will be appreciated that the presence of other magnetic sources and/or materials or devices exhibiting magnetic properties that substantially affect or interfere with the emitted magnetic field, may interfere with the aforementioned magnetic tracking.

Therefore, with continued reference to <FIG>, in some implementations, the robot <NUM> is configured to map the magnetic properties of the local environment <NUM>. More specifically, the robot <NUM> can be configured to determine the magnetic properties of the local environment <NUM> to identify regions where magnetic interference may occur. In some implementations, the robot <NUM> maps the ambient magnetic properties of the local environment <NUM> by navigating throughout the local environment <NUM> while sensing magnetic fields (e.g. using one or more magnetometers). In some implementations, the robot <NUM> detects the magnetic properties (e.g. magnetic susceptibility, magnetic permeability, etc.) of specific objects in the local environment <NUM>, which may be identified using a previously constructed 3D spatial map of the local environment <NUM>.

Using the identified magnetic properties of the local environment <NUM>, including those of any specific objects in the local environment <NUM>, the system can model their effect on the emitted magnetic field that is to be used for magnetic tracking. And therefore the magnetic tracking can be made more accurate using the modeled effects of the identified magnetic properties of the local environment <NUM>.

In some implementations, the robot <NUM> may use its spatial map of the local environment <NUM> to calibrate detection of the emitted magnetic field for magnetic tracking. That is, the emitted magnetic field can be provided, and the robot can detect the emitted magnetic field (e.g. field strength) at various positions throughout the local environment <NUM>. Simultaneously, the robot <NUM> determines the position of the magnetic emitter and its own position relative to the magnetic emitter using its spatial map and/or other non-magnetic techniques (e.g. image recognition and tracking, depth-based tracking, etc.). The detected magnetic field by the robot <NUM> is correlated to the robot's determined position using the non-magnetic techniques. In this manner, a mapping of the emitted magnetic field that is specific to the local environment <NUM> can be determined.

In some implementations, by mapping the magnetic properties of the local environment <NUM>, the system can identify and recommend to a user a specific region that is preferred for magnetic tracking, and/or identify and inform the user about a specific region that should be avoided for magnetic tracking.

<FIG> conceptually illustrates a system for adjusting lighting conditions in a local environment, in accordance with implementations of the disclosure. In the illustrated implementation, the robot <NUM> is capable of communicating over a network <NUM> with a home automation hub <NUM>. In some implementations, the network <NUM> is defined by a WiFi network. In some implementations, the network <NUM> can include any of various kinds of wireless and/or wired networks, through which the robot <NUM> can communicate with the home automation hub <NUM>.

The home automation hub <NUM> is a device that is capable of communicating over the network <NUM>, and also capable of communicating with the lights <NUM>, which are lights in the local environment <NUM> that are capable of being controlled to affect the lighting conditions in the local environment <NUM>. In some implementations, the home automation hub <NUM> communicates with the lights <NUM> over a home automation communication protocol or standard, such as Universal Powerline Bus, Insteon, Z-wave, Zigbee, WiFi, Bluetooth, Thread, Homekit, etc. The home automation hub <NUM> is capable of communicating over the appropriate protocol so as to control the illumination provided by the lights <NUM>, and may control aspects such as the on/off state, the light intensity setting, and the color of the lights <NUM>, in accordance with their capabilities.

With continued reference to <FIG>, as shown at reference <NUM>, the robot <NUM> senses the lighting condition in the local environment <NUM>. This may include sensing the lighting condition of a particular object <NUM> in the local environment <NUM>, a surface, a region, or other portion of the local environment <NUM>. Based on this initial sensed lighting condition, the robot <NUM> may determine that the lighting condition should be adjusted, so as to improve the lighting condition, for example, for purposes of capturing the texture of the object <NUM> or a surface in the local environment <NUM>. In some implementations, as noted above, the robot <NUM> determines whether the current sensed lighting condition meets a target lighting condition, which in some implementations may be defined by meeting a minimum, a maximum, or a range, for a measured lighting condition parameter. In response to determining that the lighting condition should be adjusted, the robot <NUM> sends a request over the network <NUM> to the home automation hub <NUM> to adjust the lights <NUM>, as shown at reference <NUM>.

In response to the request, the home automation hub <NUM> sends one or more control signals to the lights <NUM> (e.g. using a home automation protocol), as shown at reference <NUM>, thereby affecting the states of the lights <NUM> in some manner, such as by turning a given light on or off, adjusting its intensity, and/or adjusting its color. Then as shown at reference <NUM>, the lighting condition in the local environment <NUM> changes, and returning to reference <NUM>, the robot <NUM> senses the new lighting condition in the local environment <NUM>.

While in the above implementation, the robot <NUM> communicates over the network <NUM> with the home automation hub <NUM>, in some implementations, the robot <NUM> communicates directly with the home automation hub <NUM>. In still other implementations, the robot <NUM> may communicate directly with the lights <NUM> to adjust their illumination settings, and thereby control the lighting conditions in the local environment <NUM>.

<FIG> illustrates a method for using a mobile robot to overcome possible errors when capturing spatial and texture data in a local environment, in accordance with implementations of the disclosure. At method operation <NUM>, data is captured by from one or more sensors or spatial/textural data capture devices of the robot <NUM>. By way of example without limitation, and as described elsewhere herein, this may include capturing image data by an image capture device, capturing depth data by a depth camera, etc. Such data capture can be for the purpose of capturing the spatial structure and/or texture of the local environment <NUM> or any region/object/surface therein.

At method operation <NUM>, the captured data is analyzed. More specifically, the captured data is analyzed to determine whether or not there are possible errors in the captured data. In some implementations, this entails analyzing the captured data to identify portions that are suspect, such as by identifying discontinuities in the captured data or other aspects of the captured data. In some implementations, this entails determining a degree of confidence for a given portion of the captured data, and determining whether the degree of confidence satisfies a predefined threshold. In other words, if the degree of confidence does not satisfy (e.g. exceed) the predefined threshold, then there is a probable error for the portion of the captured data under consideration.

For example, in a captured image a portion of a surface that is all white (or all of a high or maximum intensity, e.g. exceeding a threshold intensity level) may be because the portion is indeed the color white, but might also result from the presence of glare or a specular reflection of some kind. Or as another example, in captured depth data, a region for which depth data is missing may be because of an opening in the overall spatial structure (e.g. doorway), but might also result from a window or other transparent structure, or a reflective surface that deflects the depth camera's beam, that is present but not detected by the depth camera. Depth cameras are known to be susceptible to noise (e.g. from reflections and/or features that are difficult to capture), and hence depth data may include erroneous measurements. Thus, it is desirable to identify and resolve such potential errors using the capabilities of the robot <NUM>.

At method operation <NUM>, it is determined whether a possible error exists. If not, then at method operation <NUM> the method ends. However, if a possible error exists, then at method operation <NUM>, one or more actions are identified/determined for overcoming the error (e.g. clarifying whether an error actually exists). At method operation <NUM>, the determined action is executed by the robot <NUM> to resolve the possible error. By way of example without limitation, such corrective actions may include one or more of the following: moving the robot <NUM> to a new position/orientation (e.g. to obtain an image or capture data from a different angle and/or distance), moving a sensor device of the robot <NUM> (e.g. by adjusting a telescoping or articulating arm of the robot), adjusting the lighting (e.g. by adjusting a light of the robot, adjusting lighting through a home automation system), etc..

Following performance of the determined action, then the method returns to method operation <NUM>, to capture data using the robot <NUM> again. Hence, the method can be repeated until there is no longer a probable error, or until the captured data is accurate to a satisfactory degree of confidence.

Using the aforementioned spatial and texture models of the local environment <NUM>, it is possible to 3D reconstruct the local environment <NUM>, and render highly realistic views of the local environment <NUM> on a display, such as the display of the HMD <NUM>. This can ease the transition into and out of virtual reality for the user <NUM>, as they may be provided with views of a virtualized version of the local environment <NUM> when initially putting on and/or just prior to taking off the HMD <NUM>. Furthermore, unique experiences can be provided to the user <NUM>, such as by allowing another remote user to enter their virtualized local environment, providing an experience similar to interacting with another user in the local environment <NUM> even though the other user is not physically present.

<FIG> is a schematic diagram conceptually illustrating components of a robot, in accordance with implementations of the disclosure. As shown, the robot <NUM> includes a controller <NUM> that is configured to control various devices of the robot and the operations performed by the robot <NUM>, including processing data and instructions, and issuing commands to various devices of the robot <NUM> to cause the robot to move, capture images/audio/video, render images/audio/video, or perform any other function of which the robot is capable, as described in the present disclosure. The controller <NUM> includes one or more processors <NUM> (e.g. microprocessor, general purpose processor (GPP), application specific processor (ASP), central processing unit (CPU), graphics processing unit (GPU), complex instruction set computer (CISC), reduced instruction set computer (RISC), application specific integrated circuit (ASIC), digital signal processor (DSP), etc.) configured to execute program instructions, and one or more memory devices <NUM> (e.g. volatile memory, non-volatile memory, random access memory (RAM), read-only memory (ROM), SRAM, DRAM, flash memory, magnetic memory, hard disk, optical disc, etc.) configured to store and retrieve data.

A transceiver <NUM> is configured to transmit and/or receive data, via a wireless or wired connection. The transceiver <NUM> may communicate over one or more networks and use any of various data communications protocols known in the art, including by way of example without limitation, IP-based protocols, Wi-Fi, Bluetooth, NFC, Zigbee, Z-Wave, ANT, UWB, Wireless USB, Sigfox, cellular networks (<NUM>/<NUM>/<NUM>/<NUM> networks, LTE networks, etc.), infrared protocols (e.g. IRDA protocols), etc..

The robot <NUM> includes one or more speakers <NUM> that are capable of emitting any kind of audio, including by way of example without limitation, sounds from a virtual environment being rendered by the robot <NUM>, music, speech, audio from a media presentation (e.g. television program, movie, show, etc.), etc..

The robot <NUM> includes one or more microphones <NUM>, that are configured to capture sound from the local environment in which the robot is disposed. A plurality of microphones may permit greater sensitivity in a greater number of directions simultaneously. In some implementations, the microphones <NUM> are configured in an array or other predefined positioning arrangement, so that signals from the microphone array can be analyzed to determine directionality of audio sources relative to the microphone array.

The robot <NUM> includes one or more image capture devices/cameras <NUM> configured to capture images/video from the local environment. Multiple image capture devices can be employed to enable simultaneous coverage of a larger region or multiple regions of the local environment and/or improved environment mapping, depth analysis, by way of example without limitation.

The one or more cameras <NUM> can be directed by one or more actuators <NUM>, to enable the direction of a given camera to be adjusted. Actuators <NUM> can be configured to rotate, translate, raise, lower, tilt, pan, or otherwise move or change the orientation of the cameras <NUM>.

The robot <NUM> includes one or more depth cameras <NUM>. A depth camera is capable of capturing depth/ranging information about objects in the local environment. In some implementations, the depth camera <NUM> is a time-of-flight camera that determines distance based on the time-of-flight of a controlled light signal to various points in the local environment.

Similar to the cameras <NUM>, the depth cameras <NUM> can be directed by one or more actuators <NUM>, which may be the same or different actuators as those that direct the one or more cameras <NUM>.

The robot <NUM> includes one or more proximity sensors <NUM>, that are capable of detecting proximity of the robot to nearby objects. The proximity sensors <NUM> can be mounted at various locations on the robot <NUM>, to enable proximity detection for corresponding portions of the robot <NUM>. For example, in some implementations, at least one proximity sensor is mounted at a lower portion of the robot <NUM> to enable proximity detection in this vicinity, such as to provide detection of objects nearby to the lower portion of the robot (e.g. objects on the floor/surface on which the robot <NUM> is situated). In some implementations, one or more proximity sensors are mounted along other portions of the robot <NUM>, including middle and upper portions of the robot. Proximity sensors <NUM> can be useful for avoiding collisions of the robot <NUM> with objects in the local environment, detecting the presence of nearby objects, detecting gestures by a user in the vicinity of the robot, etc..

The robot <NUM> includes a global positioning system (GPS) device/receiver <NUM>, that is configured to receive information from GPS satellites for determining the geo-location of the robot <NUM>.

The robot <NUM> includes one or more inertial/motion sensors <NUM> that are capable of detecting movement and/or orientation of the robot <NUM>. Examples of inertial/motion sensors include accelerometers, magnetometers, gyroscopes, etc..

The robot <NUM> includes at least one projector <NUM> that is capable of projecting images/video onto surfaces in the local environment. By way of example without limitation, the projector can be an LCD projector, LED projector, DLP projector, LCoS projector, pico projector, etc..

The robot <NUM> includes a plurality of wheels/rollers, e.g. wheels/rollers 1130a and 1130b as shown, that are configured to enable the robot <NUM> to move about the local environment. One or more of the wheels/rollers can be controlled by actuators (e.g. actuators 1132a and 1132b) to cause the wheels/rollers to rotate and thereby effect movement of the robot <NUM>. In some implementations, wheels/rollers can be multi-directional or omnidirectional, that is, capable of producing or facilitating movement in more than one direction or all directions.

The various components of the robot <NUM> can be contained within a housing. In the illustrated implementation, an upper housing 1134a and a lower housing 1134b are included. The upper housing 1134a is configured to be rotatable relative to the lower housing 1134b, facilitated by a plurality of bearings <NUM>. In some implementations, an actuator <NUM> is configured to rotate the upper housing 1134a. In various implementations, any of the various components of the robot <NUM> can be mounted to or within the upper housing 1134a, and configured to be rotated/moved when the upper housing 1134a is rotated, while others of the various components are mounted to or within the lower housing 1134b and not simultaneously rotated.

By way of example, in some implementations, the camera <NUM>, depth camera <NUM>, speaker <NUM>, and/or microphone <NUM> is/are mounted to the upper housing 1134a, while the projector <NUM> is mounted to the lower housing 1134b. The components mounted to the upper housing 1134a can be rotated with the upper housing 1134a, independent of the projector <NUM>. This can enable the robot <NUM> to direct the projector <NUM> independently of the camera <NUM>, depth camera <NUM>, speaker <NUM>, and/or microphone <NUM>. For example, this may be useful to allow the camera <NUM>, depth camera <NUM>, speaker <NUM>, and/or microphone <NUM> to be directed towards a user, while the projector <NUM> is directed towards a wall or other projection surface.

<FIG> illustrate various types of robots, in accordance with implementations of the disclosure.

<FIG> illustrates a robot having a cylindrical shaped body <NUM>, in accordance with implementations of the disclosure. A projector <NUM> and a camera <NUM> are mounted to respective poles that are extendable and/or rotatable in accordance with implementations of the disclosure. The robot may include a plurality of speakers <NUM> that enable the robot to emit audio in multiple directions. Though not specifically shown, the robot may also include a down-firing speaker. The robot further includes wheels <NUM> for propulsion/movement of the robot about the local environment.

<FIG> illustrates a robot having an upper rotatable portion <NUM>, to which components such as the projector <NUM> and camera <NUM> may be mounted, in accordance with implementations of the disclosure. The robot further includes a display <NUM>, which can be configured to render any kind of data. In some implementations, the display <NUM> of the robot can be used as a secondary display to show information useful to a player during a video game. The display <NUM> can be touchscreen display, and capable of receiving input from a user via touches and gestures on the touchscreen display. The robot further employs a continuous track system <NUM> (also known as a tank tread or caterpillar tread) for propulsion of the robot.

<FIG> illustrates a robot configured to also function as a storage location for controllers and/or other interface devices, in accordance with implementations of the disclosure. In the illustrated implementation, the robot is configured to hold/store controllers 1220a and 1220b, and motion controllers 1222a and 1222b. The robot can include any of various kinds of devices for holding a controller or other interface device, such as a clasp, clip, strap, clamp, pocket, hole, recess, etc..

<FIG> illustrates a robot having a main body <NUM> and a launchable drone <NUM>, in accordance with implementations of the disclosure. When not in flight, the drone <NUM> may rest on, and/or be secured to, a support structure <NUM> on the main body <NUM>. The support structure <NUM> may include contacts configured to mate with corresponding contacts on the drone <NUM>, to enable communication of data between the drone <NUM> and the main body <NUM> of the robot, as well as charging of the drone's battery. It will be appreciated that the drone <NUM> may include various components useful for its operation and/or that may be used while the drone is in flight, such as a camera, depth camera, microphone, projector, inertial/motion sensors, wireless transceiver, etc. The drone <NUM> may communicate wirelessly with the main body <NUM> and be controlled via wireless signals sent from the main body <NUM>. The drone <NUM> can be activated and flown so as to provide elevated vantage points for image capture, audio capture, projection, audio rendering, etc..

<FIG> illustrates a robot having the form-factor of a humanoid device, in accordance with implementations of the disclosure. The robot includes a head <NUM> that can be articulated, and may include devices such as a camera, projector, etc. The robot further includes arms <NUM>, which can be articulated, and configured to clasp items, perform gestures, etc. The robot further includes legs <NUM>, which can be articulated, and configured to enable the robot to walk/run or otherwise move about the local environment.

<FIG> illustrates a robot having a rotatable ball-shaped portion <NUM>, in accordance with implementations of the disclosure. In some implementations, the ball-shaped portion <NUM> can be rotated omnidirectionally, so as to redirect any device mounted thereto, such as a camera, projector, microphone, etc. The ball-shaped portion <NUM> is supported by a mid-portion <NUM>, that is rotatable about a base portion <NUM>, thereby providing greater flexibility of movement of the devices of the robot.

<FIG> illustrates a robot having a body <NUM> defined between wheels <NUM>, in accordance with implementations of the disclosure. In some implementations, the wheels 1262a and 1626b are oversized so as to be substantially larger than the body <NUM>, to enable the robot to traverse obstacles or other discontinuities. In some implementations, the center of gravity of the body <NUM> is configured to be below the level of the axis of the wheels 1262a and 1262b, so that the orientation of the body <NUM> is easily maintained while having only two wheels for support.

<FIG> illustrates one example of an HMD <NUM> user <NUM> interfacing with a client system <NUM>, and the client system <NUM> providing content to a second screen display, which is referred to as a second screen <NUM>. The client system <NUM> may include integrated electronics for processing the sharing of content from the HMD <NUM> to the second screen <NUM>. Other embodiments may include a separate device, module, connector, that will interface between the client system and each of the HMD <NUM> and the second screen <NUM>. In this general example, user <NUM> is wearing HMD <NUM> and is playing a video game using a controller, which may also be directional interface object <NUM>. The interactive play by user <NUM> will produce video game content (VGC), which is displayed interactively to the HMD <NUM>.

In one embodiment, the content being displayed in the HMD <NUM> is shared to the second screen <NUM>. In one example, a person viewing the second screen <NUM> can view the content being played interactively in the HMD <NUM> by user <NUM>. In another embodiment, another user (e.g. player <NUM>) can interact with the client system <NUM> to produce second screen content (SSC). The second screen content produced by a player also interacting with the controller <NUM> (or any type of user interface, gesture, voice, or input), may be produced as SSC to the client system <NUM>, which can be displayed on second screen <NUM> along with the VGC received from the HMD <NUM>.

Accordingly, the interactivity by other users who may be co-located or remote from an HMD user can be social, interactive, and more immersive to both the HMD user and users that may be viewing the content played by the HMD user on a second screen <NUM>. As illustrated, the client system <NUM> can be connected to the Internet <NUM>. The Internet can also provide access to the client system <NUM> to content from various content sources <NUM>. The content sources <NUM> can include any type of content that is accessible over the Internet.

Such content, without limitation, can include video content, movie content, streaming content, social media content, news content, friend content, advertisement content, etc. In one embodiment, the client system <NUM> can be used to simultaneously process content for an HMD user, such that the HMD is provided with multimedia content associated with the interactivity during gameplay. The client system <NUM> can then also provide other content, which may be unrelated to the video game content to the second screen. The client system <NUM> can, in one embodiment receive the second screen content from one of the content sources <NUM>, or from a local user, or a remote user.

<FIG> is a block diagram of a Game System <NUM>, according to various embodiments of the disclosure. Game System <NUM> is configured to provide a video stream to one or more Clients <NUM> via a Network <NUM>. Game System <NUM> typically includes a Video Server System <NUM> and an optional game server <NUM>. Video Server System <NUM> is configured to provide the video stream to the one or more Clients <NUM> with a minimal quality of service. For example, Video Server System <NUM> may receive a game command that changes the state of or a point of view within a video game, and provide Clients <NUM> with an updated video stream reflecting this change in state with minimal lag time. The Video Server System <NUM> may be configured to provide the video stream in a wide variety of alternative video formats, including formats yet to be defined. Further, the video stream may include video frames configured for presentation to a user at a wide variety of frame rates. Typical frame rates are <NUM> frames per second, <NUM> frames per second, and <NUM> frames per second. Although higher or lower frame rates are included in alternative embodiments of the disclosure.

Clients <NUM>, referred to herein individually as 1410A. , etc., may include head mounted displays, terminals, personal computers, game consoles, tablet computers, telephones, set top boxes, kiosks, wireless devices, digital pads, stand-alone devices, handheld game playing devices, and/or the like. Typically, Clients <NUM> are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user, e.g., a player of a game. The processes of receiving encoded video streams and/or decoding the video streams typically includes storing individual video frames in a receive buffer of the Client. The video streams may be presented to the user on a display integral to Client <NUM> or on a separate device such as a monitor or television. Clients <NUM> are optionally configured to support more than one game player. For example, a game console may be configured to support two, three, four or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include regions of a frame generated specifically for each player, e.g., generated based on each player's point of view. Clients <NUM> are optionally geographically dispersed. The number of clients included in Game System <NUM> may vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person that plays a game and the term "game playing device" is used to refer to a device used to play a game. In some embodiments, the game playing device may refer to a plurality of computing devices that cooperate to deliver a game experience to the user. For example, a game console and an HMD may cooperate with the video server system <NUM> to deliver a game viewed through the HMD. In one embodiment, the game console receives the video stream from the video server system <NUM>, and the game console forwards the video stream, or updates to the video stream, to the HMD for rendering.

Clients <NUM> are configured to receive video streams via Network <NUM>. Network <NUM> may be any type of communication network including, a telephone network, the Internet, wireless networks, powerline networks, local area networks, wide area networks, private networks, and/or the like. In typical embodiments, the video streams are communicated via standard protocols, such as TCP/IP or UDP/IP. Alternatively, the video streams are communicated via proprietary standards.

A typical example of Clients <NUM> is a personal computer comprising a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and/or software stored on a computer readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

Clients <NUM> may, but are not required to, further include systems configured for modifying received video. For example, a Client may be configured to perform further rendering, to overlay one video image on another video image, to crop a video image, and/or the like. For example, Clients <NUM> may be configured to receive various types of video frames, such as I-frames, P-frames and B-frames, and to process these frames into images for display to a user. In some embodiments, a member of Clients <NUM> is configured to perform further rendering, shading, conversion to <NUM>-D, or like operations on the video stream. A member of Clients <NUM> is optionally configured to receive more than one audio or video stream. Input devices of Clients <NUM> may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.

The video stream (and optionally audio stream) received by Clients <NUM> is generated and provided by Video Server System <NUM>. As is described further elsewhere herein, this video stream includes video frames (and the audio stream includes audio frames). The video frames are configured (e.g., they include pixel information in an appropriate data structure) to contribute meaningfully to the images displayed to the user. As used herein, the term "video frames" is used to refer to frames including predominantly information that is configured to contribute to, e.g. to effect, the images shown to the user. Most of the teachings herein with regard to "video frames" can also be applied to "audio frames.

Clients <NUM> are typically configured to receive inputs from a user. These inputs may include game commands configured to change the state of the video game or otherwise affect game play. The game commands can be received using input devices and/or may be automatically generated by computing instructions executing on Clients <NUM>. The received game commands are communicated from Clients <NUM> via Network <NUM> to Video Server System <NUM> and/or Game Server <NUM>. For example, in some embodiments, the game commands are communicated to Game Server <NUM> via Video Server System <NUM>. In some embodiments, separate copies of the game commands are communicated from Clients <NUM> to Game Server <NUM> and Video Server System <NUM>. The communication of game commands is optionally dependent on the identity of the command. Game commands are optionally communicated from Client 1410A through a different route or communication channel that that used to provide audio or video streams to Client 1410A.

Game Server <NUM> is optionally operated by a different entity than Video Server System <NUM>. For example, Game Server <NUM> may be operated by the publisher of a multiplayer game. In this example, Video Server System <NUM> is optionally viewed as a client by Game Server <NUM> and optionally configured to appear from the point of view of Game Server <NUM> to be a prior art client executing a prior art game engine. Communication between Video Server System <NUM> and Game Server <NUM> optionally occurs via Network <NUM>. As such, Game Server <NUM> can be a prior art multiplayer game server that sends game state information to multiple clients, one of which is game server system <NUM>. Video Server System <NUM> may be configured to communicate with multiple instances of Game Server <NUM> at the same time. For example, Video Server System <NUM> can be configured to provide a plurality of different video games to different users. Each of these different video games may be supported by a different Game Server <NUM> and/or published by different entities. In some embodiments, several geographically distributed instances of Video Server System <NUM> are configured to provide game video to a plurality of different users. Each of these instances of Video Server System <NUM> may be in communication with the same instance of Game Server <NUM>. Communication between Video Server System <NUM> and one or more Game Server <NUM> optionally occurs via a dedicated communication channel. For example, Video Server System <NUM> may be connected to Game Server <NUM> via a high bandwidth channel that is dedicated to communication between these two systems.

Video Server System <NUM> comprises at least a Video Source <NUM>, an I/O Device <NUM>, a Processor <NUM>, and non-transitory Storage <NUM>. Video Server System <NUM> may include one computing device or be distributed among a plurality of computing devices. These computing devices are optionally connected via a communications system such as a local area network.

Video Source <NUM> is configured to provide a video stream, e.g., streaming video or a series of video frames that form a moving picture. In some embodiments, Video Source <NUM> includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and to maintain a copy of the state of the video game based on the received commands. This game state includes the position of objects in a game environment, as well as typically a point of view. The game state may also include properties, images, colors and/or textures of objects. The game state is typically maintained based on game rules, as well as game commands such as move, turn, attack, set focus to, interact, use, and/or the like. Part of the game engine is optionally disposed within Game Server <NUM>. Game Server <NUM> may maintain a copy of the state of the game based on game commands received from multiple players using geographically disperse clients. In these cases, the game state is provided by Game Server <NUM> to Video Source <NUM>, wherein a copy of the game state is stored and rendering is performed. Game Server <NUM> may receive game commands directly from Clients <NUM> via Network <NUM>, and/or may receive game commands via Video Server System <NUM>.

Video Source <NUM> typically includes rendering logic, e.g., hardware, firmware, and/or software stored on a computer readable medium such as Storage <NUM>. This rendering logic is configured to create video frames of the video stream based on the game state. All or part of the rendering logic is optionally disposed within a graphics processing unit (GPU). Rendering logic typically includes processing stages configured for determining the three-dimensional spatial relationships between objects and/or for applying appropriate textures, etc., based on the game state and viewpoint. The rendering logic produces raw video that is then usually encoded prior to communication to Clients <NUM>. For example, the raw video may be encoded according to an Adobe Flash® standard,. <NUM>, On2, VP6, VC-<NUM>, WMA, Huffyuv, Lagarith, MPG-x. FFmpeg, x264, VP6-<NUM>, realvideo, mp3, or the like. The encoding process produces a video stream that is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include <NUM> x <NUM>, <NUM> x <NUM> (e.g., 720p), <NUM> x <NUM>, although any other frame sizes may be used. The frame rate is the number of video frames per second. A video stream may include different types of video frames. For example, the H. <NUM> standard includes a "P" frame and a "I" frame. I-frames include information to refresh all macro blocks/pixels on a display device, while P-frames include information to refresh a subset thereof. P-frames are typically smaller in data size than are I-frames. As used herein the term "frame size" is meant to refer to a number of pixels within a frame. The term "frame data size" is used to refer to a number of bytes required to store the frame.

In alternative embodiments Video Source <NUM> includes a video recording device such as a camera. This camera may be used to generate delayed or live video that can be included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still or video camera. Video Source <NUM> may also include storage devices configured to store previously recorded video to be included in a video stream. Video Source <NUM> may also include motion or positioning sensing devices configured to detect motion or position of an object, e.g., person, and logic configured to determine a game state or produce video-based on the detected motion and/or position.

Video Source <NUM> is optionally configured to provide overlays configured to be placed on other video. For example, these overlays may include a command interface, log in instructions, messages to a game player, images of other game players, video feeds of other game players (e.g., webcam video). In embodiments of Client 1410A including a touch screen interface or a gaze detection interface, the overlay may include a virtual keyboard, joystick, touch pad, and/or the like. In one example of an overlay a player's voice is overlaid on an audio stream. Video Source <NUM> optionally further includes one or more audio sources.

In embodiments wherein Video Server System <NUM> is configured to maintain the game state based on input from more than one player, each player may have a different point of view comprising a position and direction of view. Video Source <NUM> is optionally configured to provide a separate video stream for each player based on their point of view. Further, Video Source <NUM> may be configured to provide a different frame size, frame data size, and/or encoding to each of Client <NUM>. Video Source <NUM> is optionally configured to provide <NUM>-D video.

I/O Device <NUM> is configured for Video Server System <NUM> to send and/or receive information such as video, commands, requests for information, a game state, gaze information, device motion, device location, user motion, client identities, player identities, game commands, security information, audio, and/or the like. I/O Device <NUM> typically includes communication hardware such as a network card or modem. I/O Device <NUM> is configured to communicate with Game Server <NUM>, Network <NUM>, and/or Clients <NUM>.

Processor <NUM> is configured to execute logic, e.g. software, included within the various components of Video Server System <NUM> discussed herein. For example, Processor <NUM> may be programmed with software instructions in order to perform the functions of Video Source <NUM>, Game Server <NUM>, and/or a Client Qualifier <NUM>. Video Server System <NUM> optionally includes more than one instance of Processor <NUM>. Processor <NUM> may also be programmed with software instructions in order to execute commands received by Video Server System <NUM>, or to coordinate the operation of the various elements of Game System <NUM> discussed herein. Processor <NUM> may include one or more hardware device. Processor <NUM> is an electronic processor.

Storage <NUM> includes non-transitory analog and/or digital storage devices. For example, Storage <NUM> may include an analog storage device configured to store video frames. Storage <NUM> may include a computer readable digital storage, e.g. a hard drive, an optical drive, or solid state storage. Storage <NUM> is configured (e.g. by way of an appropriate data structure or file system) to store video frames, artificial frames, a video stream including both video frames and artificial frames, audio frame, an audio stream, and/or the like. Storage <NUM> is optionally distributed among a plurality of devices. In some embodiments, Storage <NUM> is configured to store the software components of Video Source <NUM> discussed elsewhere herein. These components may be stored in a format ready to be provisioned when needed.

Video Server System <NUM> optionally further comprises Client Qualifier <NUM>. Client Qualifier <NUM> is configured for remotely determining the capabilities of a client, such as Clients 1410A or 1410B. These capabilities can include both the capabilities of Client 1410A itself as well as the capabilities of one or more communication channels between Client 1410A and Video Server System <NUM>. For example, Client Qualifier <NUM> may be configured to test a communication channel through Network <NUM>.

Client Qualifier <NUM> can determine (e.g., discover) the capabilities of Client 1410A manually or automatically. Manual determination includes communicating with a user of Client 1410A and asking the user to provide capabilities. For example, in some embodiments, Client Qualifier <NUM> is configured to display images, text, and/or the like within a browser of Client 1410A. In one embodiment, Client 1410A is an HMD that includes a browser. In another embodiment, client 1410A is a game console having a browser, which may be displayed on the HMD. The displayed objects request that the user enter information such as operating system, processor, video decoder type, type of network connection, display resolution, etc. of Client 1410A. The information entered by the user is communicated back to Client Qualifier <NUM>.

Automatic determination may occur, for example, by execution of an agent on Client 1410A and/or by sending test video to Client 1410A. The agent may comprise computing instructions, such as java script, embedded in a web page or installed as an add-on. The agent is optionally provided by Client Qualifier <NUM>. In various embodiments, the agent can find out processing power of Client 1410A, decoding and display capabilities of Client 1410A, lag time reliability and bandwidth of communication channels between Client 1410A and Video Server System <NUM>, a display type of Client 1410A, firewalls present on Client 1410A, hardware of Client 1410A, software executing on Client 1410A, registry entries within Client 1410A, and/or the like.

Client Qualifier <NUM> includes hardware, firmware, and/or software stored on a computer readable medium. Client Qualifier <NUM> is optionally disposed on a computing device separate from one or more other elements of Video Server System <NUM>. For example, in some embodiments, Client Qualifier <NUM> is configured to determine the characteristics of communication channels between Clients <NUM> and more than one instance of Video Server System <NUM>. In these embodiments the information discovered by Client Qualifier can be used to determine which instance of Video Server System <NUM> is best suited for delivery of streaming video to one of Clients <NUM>.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

With the above embodiments in mind, it should be understood that the disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the disclosure are useful machine operations. The disclosure also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

The disclosure can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times, or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in the desired way.

Claim 1:
A method, comprising:
using a robot (<NUM>) having a plurality of sensors (<NUM>, <NUM>) to acquire sensor data about a local environment;
processing the sensor data to generate a spatial model of the local environment, the spatial model defining virtual surfaces that correspond to real surfaces (<NUM>) in the local environment;
further processing the sensor data to generate texture information that is associated to the virtual surfaces defined by the spatial model, wherein the texture information includes one or more of a diffuse map, a bump map, and/or a specular map;
tracking a location and orientation of a head-mounted display, HMD (<NUM>), in the local environment;
using the spatial model, the texture information, and the tracked location and orientation of the HMD to render a view of a virtual space that corresponds to the local environment, wherein the location of the HMD in the local environment defines a perspective from which the view of the virtual space is rendered;
presenting the view of the virtual environment through the HMD,
wherein using the robot to acquire sensor data includes sampling representative portions (<NUM>) of the real surfaces in the local environment to enable the generation of the texture information that is associated to the virtual surfaces,
wherein acquiring the sensor data includes capturing images of a real surface in the local environment from a plurality of angles, wherein each of the plurality of angles corresponds to a different position of the robot in the local environment,
wherein processing the sensor data to generate the texture information includes processing the images captured from the plurality of angles to generate texture information for a given virtual surface defined by the spatial model that corresponds to the real surface, and
wherein using the robot to acquire sensor data includes affecting the lighting of a real surface in the environment by:
operating a light included in the robot to illuminate a surface, and/or
moving the robot to produce a shadow upon a surface by moving between a light source and the surface.