Patent ID: 12189120

DETAILED DESCRIPTION

This disclosure relates generally to computer ecosystems including aspects of consumer electronics (CE) device networks such as but not limited to computer game networks. A system herein may include server and client components including HMDs which may be connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including game consoles such as Sony PlayStation® or a game console made by Microsoft or Nintendo or other manufacturer, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g. smart TVs, Internet-enabled TVs), portable computers such as laptops and tablet computers, and other mobile devices including smart phones and additional examples discussed below. These client devices may operate with a variety of operating environments. For example, some of the client computers may employ, as examples, Linux operating systems, operating systems from Microsoft, or a Unix operating system, or operating systems produced by Apple, Inc. or Google. These operating environments may be used to execute one or more browsing programs, such as a browser made by Microsoft or Google or Mozilla or other browser program that can access websites hosted by the Internet servers discussed below. Also, an operating environment according to present principles may be used to execute one or more computer game programs.

Servers and/or gateways may include one or more processors executing instructions that configure the servers to receive and transmit data over a network such as the Internet. Or, a client and server can be connected over a local intranet or a virtual private network. A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.

Information may be exchanged over a network between the clients and servers. To this end and for security, servers and/or clients can include firewalls, load balancers, temporary storages, and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implement methods of providing a secure community such as an online social website to network members.

A processor may be any conventional general-purpose single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.

Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.

Now specifically referring toFIG.1, an example system10is shown, which may include one or more of the example devices mentioned above and described further below in accordance with present principles. The first of the example devices included in the system10is a consumer electronics (CE) device such as an audio video device (AVD)12such as but not limited to an Internet-enabled TV with a TV tuner (equivalently, set top box controlling a TV). The AVD12alternatively may also be a computerized Internet enabled (“smart”) telephone, a tablet computer, a notebook computer, a HMD, a wearable computerized device, a computerized Internet-enabled music player, computerized Internet-enabled head phones, a computerized Internet-enabled implantable device such as an implantable skin device, etc. Regardless, it is to be understood that the AVD12is configured to undertake present principles (e.g., communicate with other CE devices to undertake present principles, execute the logic described herein, and perform any other functions and/or operations described herein).

Accordingly, to undertake such principles the AVD12can be established by some or all of the components shown inFIG.1. For example, the AVD12can include one or more displays14that may be implemented by a high definition or ultra-high definition “4K” or higher flat screen and that may be touch-enabled for receiving user input signals via touches on the display. The AVD12may include one or more speakers16for outputting audio in accordance with present principles, and at least one additional input device18such as an audio receiver/microphone for entering audible commands to the AVD12to control the AVD12. The example AVD12may also include one or more network interfaces20for communication over at least one network22such as the Internet, an WAN, an LAN, etc. under control of one or more processors24. A graphics processor24A may also be included. Thus, the interface20may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as but not limited to a mesh network transceiver. It is to be understood that the processor24controls the AVD12to undertake present principles, including the other elements of the AVD12described herein such as controlling the display14to present images thereon and receiving input therefrom. Furthermore, note the network interface20may be a wired or wireless modem or router, or other appropriate interface such as a wireless telephony transceiver, or Wi-Fi transceiver as mentioned above, etc.

In addition to the foregoing, the AVD12may also include one or more input ports26such as a high definition multimedia interface (HDMI) port or a USB port to physically connect to another CE device and/or a headphone port to connect headphones to the AVD12for presentation of audio from the AVD12to a user through the headphones. For example, the input port26may be connected via wire or wirelessly to a cable or satellite source26aof audio video content. Thus, the source26amay be a separate or integrated set top box, or a satellite receiver. Or, the source26amay be a game console or disk player containing content. The source26awhen implemented as a game console may include some or all of the components described below in relation to the CE device44.

The AVD12may further include one or more computer memories28such as disk-based or solid state storage that are not transitory signals, in some cases embodied in the chassis of the AVD as standalone devices or as a personal video recording device (PVR) or video disk player either internal or external to the chassis of the AVD for playing back AV programs or as removable memory media. Also in some embodiments, the AVD12can include a position or location receiver such as but not limited to a cellphone receiver, GPS receiver and/or altimeter30that is configured to receive geographic position information from a satellite or cellphone base station and provide the information to the processor24and/or determine an altitude at which the AVD12is disposed in conjunction with the processor24. The component30may also be implemented by an inertial measurement unit (IMU) that typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the location and orientation of the AVD12in three dimensions.

Continuing the description of the AVD12, in some embodiments the AVD12may include one or more cameras32that may be a thermal imaging camera, a digital camera such as a webcam, and/or a camera integrated into the AVD12and controllable by the processor24to gather pictures/images and/or video in accordance with present principles. Also included on the AVD12may be a Bluetooth transceiver34and other Near Field Communication (NFC) element36for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

Further still, the AVD12may include one or more auxiliary sensors37(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, a gesture sensor (e.g. for sensing gesture command), etc.) providing input to the processor24. The AVD12may include an over-the-air TV broadcast port38for receiving OTA TV broadcasts providing input to the processor24. In addition to the foregoing, it is noted that the AVD12may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver42such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD12, as may be a kinetic energy harvester that may turn kinetic energy into power to charge the battery and/or power the AVD12.

Still referring toFIG.1, in addition to the AVD12, the system10may include one or more other CE device types. In one example, a first CE device44may be used to send computer game audio and video to the AVD12via commands sent directly to the AVD12and/or through the below-described server while a second CE device46may include similar components as the first CE device44. In the example shown, the second CE device46may be configured as a computer game controller manipulated by a player or an HMD worn by a player47. In the example shown, only two CE devices44,46are shown, it being understood that fewer or greater devices may be used. A CE device herein may implement some or all of the components shown for the AVD12. Any of the components shown in the following figures may incorporate some or all of the components shown in the case of the AVD12.

Now in reference to the afore-mentioned at least one server50, it includes at least one server processor52, at least one tangible computer readable storage medium54such as disk-based or solid state storage, and at least one network interface56that, under control of the server processor52, allows for communication with the other devices ofFIG.1over the network22, and indeed may facilitate communication between servers and client devices in accordance with present principles. Note that the network interface56may be, e.g., a wired or wireless modem or router, Wi-Fi transceiver, or other appropriate interface such as, e.g., a wireless telephony transceiver.

Accordingly, in some embodiments the server50may be an Internet server or an entire server “farm”, and may include and perform “cloud” functions such that the devices of the system10may access a “cloud” environment via the server50in example embodiments for, e.g., network gaming applications. Or, the server50may be implemented by one or more game consoles or other computers in the same room as the other devices shown inFIG.1or nearby.

FIG.2illustrates an example non-limiting embodiment of a head mount (HM)200configured as eyeglasses, with a nose bridge202supporting left and right view openings204, which may be filled with respective lenses, and left and right temples206. Other configurations of HM may be used, e.g., headbands, augmented reality head-mounted displays, etc.

The HM200may include one or more cameras208positioned to produce images of the eyes of a wearer of the HM200. The HM200also may include one or more inertial measurement units (IMU)210to sense and output signals of a body's specific force, angular rate, and sometimes the orientation of the body, using a combination of accelerometers, gyroscopes, and sometimes magnetometers. One or more processors212may communicate with the camera208and IMU210and communicate wirelessly with other devices using one or more wireless transceivers214.

For example, the processor212may communicate with a waveguide processor216via a waveguide transceiver218to move a waveguide220such as one or more galvanometers, mirrors, deformable fisheye lenses, etc. to redirect light from a laser projector222as indicated by the arrows224. The laser projector222itself may be stationary if desired. In this way, light from the laser projector222can be directed to selective regions in a room226in which the laser projector222is disposed in accordance with principles set forth herein. Multiple users may be accommodated as disclosed herein so that multiple sets of HMs may be present, including, e.g., a “B” set of eyeglasses228that in construction and operation may be substantially identical to the HM200.

Indeed,FIG.4illustrates such a multi-user scenario. However, referring first toFIG.3, a single person “A” may gaze toward a region A in the room226, wearing the HM200. The location of the user and direction of gaze are determined using output of the IMU210and camera208. The HM200communicates this information to the waveguide processor216(and to processing circuitry in the laser projector222if desired) which causes the waveguide220to be moved to redirect an image “A” from the laser projector222into the region A as shown.

Should the person A turn his gaze toward a region B in the room as indicated by the arrow300, the waveguide220is moved to redirect the image A from the projector into the region B as indicated by the arrow302. In this way, the image from the projector is moved around the room to appear in the field of view (FOV) defined by the gaze of the person A.

FIG.4illustrates two users labeled person A and person B, each wearing a respective HM400. InFIG.4, an image A is presented in a region of the room in the FOV of the person A whereas a different image B is presented in a different region of the room in the FOV of the person B.

It is to be appreciated that in this way, a single laser projector may be able to present, for example, a first video to the person A in the FOV of the person A while simultaneously presenting a different video in the FOV of the person B.

This may be facilitated by using a scanning laser at 120 Hz or higher (e.g., 240 Hz). The waveguide220in effect is moved to multiplex images from the projector between the two regions at 60 Hz each. The laser projector222in every odd cycle for example may project the image A, which is redirected to the FOV of the person A by the waveguide220, and then in every odd cycle generate the image B, which is redirected to the FOV of the person B by the waveguide220. These principles may apply to more than two people viewing respective different images in their respective FOVs.

FIG.5illustrates a HM500that may be substantially identical in configuration and operation to the HM200inFIG.2, with the following exceptions. The HM500inFIG.5may employ shutters502such as liquid crystal display (LCD) shutters in front of each eye opening or lens504. Every other projection cycle the laser projector222may project a left eye image and in between a right image, with the left eye shutter502being closed and the right eye shutter being open when the right eye image is projected and vice-versa when the left eye image is projected, as indicated in the projection region506illustrated inFIG.5. Or, each lens504may be an active polarization lens with different polarizations from each other, with the polarization of light from the laser projector222being configured to match the polarization of the left eye lens504when the left eye image is being projected and then changed to match the polarization of the right eye lens504when the right eye image is being projected in the alternating cycles.

FIG.6shows yet another HM600that may be substantially identical in configuration and operation to the HM200inFIG.2, with the following exceptions. Instead of a remote laser projector, a laser projector602is mounted on the HM600inFIG.6pointing straight in front of the wearer as defined by the frame of the HM600. The laser projector602thus can project light along the FOV of the wearer when the wearer is gazing straight ahead. A waveguide604may be employed to redirect the image from the laser602if desired when the wearer glances sidelong as detected by the camera606of the HM600. Image stabilization techniques used in digital cameras may be implemented to stabilize the image from the laser on the wall of the room.

FIG.7illustrates example logic. Commencing at block700, the room226inFIG.2is mapped. This mapping may be done using light direction and ranging (LIDAR) techniques, structured light mapping techniques, or other techniques such as one or more of those disclosed in commonly owned U.S. Pat. No. 9,854,362 and U.S. patent application Ser. No. 16/216,725, both of which are incorporated herein by reference. The mapping may be done using a mapping apparatus different from the laser projector222, or it may be done using the laser projector222in a mapping mode.

During operation, images from HM camera208are used to track the user's gaze while at block704signals from the HM IMU210are used to track the user's position and/or orientation in the room226. Moving to block706, based on the map from block700and the information from blocks702and704, the waveguide220is configured to direct the image from the laser projector222onto a region of the room in the user's FOV, i.e., at which the user is looking.

If desired, based on eye tracking an IMU input and rates of change, the logic may, at block708, predict the next region in the room at which the user might look. At block710the image from the laser projector may be prospectively projected into the next region while continuing to project the image into the current region using multiplexing principles described herein. For example, using a 120 Hz laser the image may be projected at 60 Hz into both the current and next regions.

FIG.8illustrates an example user interface (UI)800that may be presented on any of the displays herein, in one or more UI pages. An on/off selector802may be provided to enable or disable the projection of images from the laser projector222. A 3D mode selector804may be provided to enable and disable the logic attendant toFIG.5. A multi-user selector806may be provided to enable and disable the logic attendant toFIG.4. WhenFIG.4is enabled, one or more view selectors808may be provided to enable the users to select what videos or text pages or other images they wish to have presented in their respective FOVs.

A follow me mode selector810may be provided to enable and disable the logic attendant toFIG.3. Further, a single person, multi-view mode may be provided to enable and disable logic similar toFIG.4except that only a single person is involved and may select, using a view selector814, what videos or text pages or other images he wishes to have presented in respective regions of the room226, so that the single user can look from one region to another to view multiple pages of, for instance, a document as if the user had multiple pages of the document spread on various locations in the room.

A map selector816may be presented to cause the laser projector222to generate a map of the room226using LIDAR or structure light or other appropriate mapping technique.

FIG.9illustrates that in mapping the room226, certain regions of the room, such as doors and windows, may be detected that are not suitable for image projection. Thus, at block900, a region for projection may be selected according to, for instance, the logic ofFIG.7. Proceeding to decision diamond902, it may be determined whether the region includes an acceptable surface for reflecting projection of laser light, and if so the image from the laser projector is directed into the region at block904by appropriately configuring the waveguide220.

On the other hand, if the region is determined not to be acceptable, the logic moves to block906to configure the waveguide220to directed light from the laser into a substitute region that may be closest region to the region identified at block900having an acceptable projection surface.

In determining whether a region is acceptable, during mapping, image recognition may be used on reflected light and regions correlated, using a data structure correlating reflection characteristics to projection reflection suitability, to “acceptable” or “not acceptable” according to the reflection characteristics of the respective regions. Machine learning may be used for this purpose by training a neural network system of other machine learning system on a database of known surface types correlated to reflection characteristics.

It will be appreciated that whilst present principals have been described with reference to some example embodiments, these are not intended to be limiting, and that various alternative arrangements may be used to implement the subject matter claimed herein.