Aggregate view of image information on display devices

In some examples, an electronic device includes a first display device and a processor. The processor determines whether the electronic device is paired with a head-mounted device comprising a second display device. In response to determining that the electronic device is paired with the head-mounted device, the processor presents first image information on the first display device, sends second image information to the head-mounted device, and coordinates with the head-mounted device to cause the head-mounted device to display the second image information on the second display device, where the first image information and the second image information are to present an aggregate view to a user of the head-mounted device.

BACKGROUND

A head-mounted device refers to an electronic device that can be worn on a user's head. For example, a head-mounted device can include a headset, such as in the form of electronic goggles, in which images can be displayed in front of the user's eyes by respective display devices. In other examples, a head-mounted device can include electronic glasses.

DETAILED DESCRIPTION

A head-mounted device can include multiple display devices, including a first display device for a right eye of a user, and a second display device for a left eye of the user. In some examples, a head-mounted device can present augmented reality content, in which images of real-world objects are augmented by displayed virtual objects in a virtual environment.

In some cases, a head-mounted device can be communicatively coupled to an external electronic device (e.g., a notebook computer, a desktop computer, a tablet computer, a smartphone, etc.) to use the processing power of the electronic device to generate video streams for presentation in the head-mounted device.

In such arrangements, the images displayed on the display device of the external electronic device are not coordinated with images displayed by the head-mounted device. As a result, display capabilities of the electronic device are not leveraged when the user is wearing the head-mounted device.

In accordance with some implementations of the present disclosure, an electronic device can detect when the electronic device is paired with a head-mounted device, and in response to such detection, is able to separate a video stream into a plurality of layers of image information. The electronic device can control a display of a first layer of image information on the display device of the electronic device, and can coordinate with the head-mounted device to display other layers of the plurality of layers of image information on the display devices of the head-mounted device. As a result, an aggregate view that includes the plurality of layers of image information can be presented to the user when wearing the head-mounted device.

FIG.1is a block diagram of an example arrangement that includes an electronic device102and a head-mounted device104. The electronic device102and the head-mounted device104can communicate with one another over a wireless link103, which can include a BLUETOOTH link, a WI-FI link, a cellular link, and so forth. In other examples, the electronic device102and the head-mounted device104can communicate with one another over a wired link.

The electronic device102can include a notebook computer, a desktop computer, a tablet computer, a smartphone, or any other electronic device that includes a display device106. In some examples, the display device106may be contained in the same housing as processing and other resources of the electronic device102. In other examples, the display device106may be contained in a housing that is separate from a housing that contains processing and other resources of the electronic device102.

A processing resource can include a processor or multiple processors. A processor (or more specifically, a hardware processor) can include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.

Other resources of the electronic device102can include a storage resource (e.g., a memory and/or persistent storage), communication resources (e.g., a network interface controller), and so forth.

The head-mounted device104can include an electronic goggle, smart glasses, or any other device that can be worn on the head of a user such that the user can view images displayed by a display device (or multiple display devices) of the head-mounted device104.

As depicted in InFIG.1, the head-amounted device104includes a display device108. Although a single display device108is shown inFIG.1, the head-mounted device104can include multiple display devices in further examples, including a left display device for a left eye of a user, and a right display device for a right eye of the user.

In the ensuing discussion, reference to the display device108can refer to a single display device or multiple display devices of the head-mounted device104.

FIG.1provides a rear view the head-mounted device104. The front of the head-mounted device104faces the electronic device102.

In some examples, the head-mounted device104can include a transparent lens110that allows outside light to pass through the transparent lens110and into a space within the head-mounted device104in front of the user's eyes when the user is wearing the head-mounted device104. A “transparent lens” can refer to any optical layer (or collection of layers) through which light can pass.

The ability to pass light from outside the head-mounted device104to the space in front of the user's eyes within the head-mounted device104allows for an augmented reality (AR) presentation of images. From the perspective of the head-mounted device104, the electronic device102is part of an external real world that is can be viewed by the user through the transparent lens110wearing the head-mounted device104.

The head-mounted device104can present AR content in the display device108of the head-mounted device104. “AR content” can refer to a combination of virtual content to be displayed with real-world content, which in the example ofFIG.1includes images displayed by the display device106of the electronic device102.

In other examples, images of the external real world can be captured by a front-facing camera112of the head-mounted device104. The front-facing camera112faces forwardly (forward of the head-mounted device104) such that the front-facing camera112can capture images of objects in front of the head-mounted device104. In such examples, the transparent lens110can be omitted and replaced with an opaque cover.

Although reference is made to a single front-facing camera112, in other examples, multiple front-facing cameras can be employed.

The images captured by the front-facing camera112can be combined with images to be displayed by the display device108of the head-mounted device104, to present AR content.

The head-mounted device104includes a controller114, which can perform control tasks of the head-mounted device104. As used here, a “controller” can refer to a hardware processing circuit, which can include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit. Alternatively, a “controller” can refer to a combination of a hardware processing circuit and machine-readable instructions (software and/or firmware) executable on the hardware processing circuit.

The controller114can receive image information to be displayed by the display device108. For example, the controller114can receive image information from the electronic device102over the wireless link103if the head-mounted device104is paired with the electronic device102. The electronic device102and the head-mounted device104are “paired” if the electronic device102and the head-mounted device104are in communication with one another, such as over a short-range wireless connection (e.g., a BLUETOOTH link) over another type of connection (e.g., a WI-FI link, a cellular link, etc.).

The controller114can also make a determination of whether the display device106of the electronic device102is in the field of view of the head-mounted device104, as indicated by150inFIG.1. The electronic device102is in the field of view of the head-mounted device104if the display device106(or more specifically a display surface of the display device106on which images are displayed) is visible either through the transparent lens110of the head-mounted device104or via the front-facing camera112.

In some examples, the determination by the controller114of whether the display device106of the electronic device102is in the field of view of the head-mounted device104can be based on detecting a trigger152, which can be displayed by the display device106or can be affixed to a front surface of the display device106. The trigger152can be in the form of a mark or any other indicator that can be displayed, or a physical mark or another indicator.

Although a single trigger152is shown inFIG.1, in other examples, multiple triggers152can be provided on the display device106(e.g., displayed by the display surface of the display device106or physically affixed to the display device106).

The controller114can use images captured by the front-facing camera112of the head-mounted device104to determine if the trigger152is present in the images. If so, the controller114can further process the images to determine whether the display surface of the display device106of the electronic device102is within the field of view150. If so, then the controller114can send a field of view indication154to the electronic device102, such as over the wireless link103. In further examples, a front-facing camera of the electronic device102can be used to assist in determining whether the head-mounted device104and the electronic device102are in a target placement with respect to one another. For example, the electronic device102can use artificial intelligence (AI) vision-based to make the determination.

The field of view indication154can be in the form of a message, an information element (e.g., a bit or a collection of bits set to a specified value) within a message, or any other information that can be communicated from the head-mounted device104to the electronic device102.

The electronic device102includes an aggregate view control engine120that responds to the field of view indication154by controlling the presentation of images in multiple layers so that the different layers of images can be aggregated to provide an aggregate view, which can be a three-dimensional (3D) view or more generally, can include different images where one image is placed over another image to form a view in which an overall image is based on the combination of the multiple images placed one over another.

As used here, an “engine” can refer to a hardware processing circuit, which can include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit. Alternatively, an “engine” can refer to a combination of a hardware processing circuit and machine-readable instructions (software and/or firmware) executable on the hardware processing circuit.

The aggregate view control engine120receives an input image122. The input image122may be generated inside the electronic device102, such as by a program executed in the electronic device102. Alternatively, the input image122may be received from a remote source, such as over a network.

The input image122is to be displayed in aggregate by the electronic device102and the head-mounted device104. Displaying the input image122in “aggregate” can refer to separating the input image122into multiple image layers that are respectively displayed on different display devices, such as the display devices106and108ofFIG.1.

The input image at122can be a still image or an image of a video. A video can include a sequence of image frames that are displayed in rapid succession such that a human can perceive motion of an object depicted in the image frames. The input image122can include one of the image frames of the video.

The aggregate view control engine120can determine whether or not the input image122is to be displayed by multiple display devices, including the display device106of the electronic device102and the display device108of the head-mounted device104. This determination by the aggregate view control engine120can be based on whether or not the electronic device102is paired with the head-mounted device104, and whether or not the field of view indication154was sent by the head-mounted device104. If the electronic device102is not paired with the head-mounted device104or the aggregate view control engine120did not receive the field of view indication154from the head-mounted device104, then the aggregate view control engine120is not activated, and the electronic device102displays the input image122on the display device106in a normal manner (i.e., not in a manner in which the input image122is divided into multiple image layers for producing an aggregate view on multiple display devices).

The aggregate view control engine120includes an image layer generator124and an image layer router126. The image layer generator124and the image layer router126can be implemented using a portion of the hardware processing circuit of the aggregate view control engine120, or can be implemented using machine-readable instructions executable by the aggregate view control engine120.

The image layer generator124generates, based on the input image122, multiple layers that are to be displayed by different display devices. For example, the image layer generator124can generate, based on the input image122, a first image layer128containing first image information130to be displayed by the display device106of the electronic device102, and a second image layer132that contains second image information134to be displayed by the display device108of the head-mounted device104.

“Image information” can refer to an arrangement of image pixels that collectively form an image.

Once the image layers are generated by the image layer generator124, the image layer router126can route the image layers to respective target display devices for display. For example, each image layer can be associated with target information that identifies where the image layer is to be routed. For example, first target information associated with the first image layer128can indicate the display device106as the target, in which case the first image layer128is routed by the image layer router126to the display device106(or more specifically, to a graphic controller for the display device106) to cause display of the first image information130in the first image layer128on the display device106. The first target information indicating the display device106can include an identifier of the electronic device102or any other information that indicates that the first image layer128is to be displayed by the electronic device102.

Similarly, second target information associated with the second image layer132can indicate the display device108as the target, in which case the second image layer132is routed (over the wireless link103) by the image layer router126to the display device108(or more specifically, to a graphic controller for the display device108) to cause display of the second image information134in the second image layer132on the display device108. The second target information indicating the display device108can include an identifier of the head-mounted device104or any other information that indicates that the second image layer132is to be displayed by the head-mounted device104.

In some examples, the image layer generator124can divide the input image122into multiple image layers for presentation as an over-under (or top and bottom) view. The over-under format includes video content encoded, by the image layer generator124, in a manner where stacking of the multiple images layers (also referred to as sub-frames) can produce an aggregate view, which can be a 3D view or any other type of aggregate view. For example, the over-under format can be according to the Mandatory Full HD (High Definition) 3D (FHD3D) video format specified by High-Definition Multimedia Interface (HDMI) 1.4, or other versions of HDMI. In other examples, the image layer generator124can divide the input image122into multiple image layers according to other formats.

If the over-under format is used, then the image layer generator124generates a top image layer for display by the head-mounted device104, and a bottom image layer for display by the display device106of the electronic device102. The top image layer can further be divided into a left eye portion and a right eye portion, to be displayed on the left display device and the right display device, respectively, of the head-mounted device104in examples where the head-mounted device104includes left and right display devices.

In some examples, an aggregate view of the first image information130and the second image information134displayed by respective display devices106and108can be a 3D view, where the 3D effect is due to the stacking of the first image information130and the second image information134. The first image information130can be visible to the user wearing the head-mounted device104through the transparent lens110, or alternatively, the first image information130can be captured by the front-facing camera112of the head-mounted device104and combined with the second image information134on the display device108to form AR content.

In other examples, the first and second image layers128and132can be used for secure depiction of sensitive information. For example, the input image122can include sensitive information, such as confidential information of an individual or enterprise, or any other information which a user does not wish to be shared with others. In such cases, the aggregate view control engine120, in response to the pairing of the electronic device102and the head-mounted device104and receipt of the field of view indication154, can produce the first image layer128and the second image layer132such that the sensitive information (foreground information) is included in the second image layer132, and blank or background information is included in the first image layer128.

In further examples, the first and second image layers128and132can include video game images, where the first image layer128for display at the electronic device102can include background images of a video game, while the second image layer132for display at the head-mounted device104can include foreground objects, such as objects to be manipulated by a human video game player.

FIG.2illustrates an example of aggregating the second image information134with the first image information130into aggregated image information202, based on stacking the second image information134over the first image information130. For example, the aggregated image information202can include AR content output at the display device108of the head-mounted device104based on combining the first image information130and the second image information134.

FIG.3is a flow diagram of a process300, which can be performed by the aggregate view control engine120in the electronic device102.

The process300determines (at302) whether the electronic device102is paired with the head-mounted device104. If so, the process300determines (at304) whether the field of view indication154was received from the head-mounted device104indicating that the display device106is in the field of view150of the head-mounted device104.

If either the electronic device102is not paired with the head-mounted device104, or the field of view indication154was not received, then the process300causes (at310) display of the input image in full at the electronic device102.

If the electronic device102is paired with the head-mounted device104and the field of view indication154was received from the head-mounted device104, the aggregate view control engine120separates (at306) the input image into multiple image layers, as performed by the image layer generator124. Moreover, the image layer router126routes (at308) the multiple image layers to respective display devices to cause image information in the multiple image layers to be displayed by different display devices.

FIG.4is a block diagram of an electronic device400according to some examples of the present disclosure. The electronic device400includes a first display device402and a processor404to perform various tasks. A processor performing a task can refer to a single hardware processor performing the task or multiple hardware processors performing the task.

The tasks of the processor404include a device pairing determination task406to determine whether the electronic device is paired with a head-mounted device that has a second display device.

The tasks of the processor404include tasks408,410, and412performed in response to determining that the electronic device is paired with the head-mounted device. The first image information presentation task408presents first image information on the first display device402.

The second image information sending task410sends second image information to the head-mounted device.

The aggregate view coordination task412coordinates with the head-mounted device to cause the head-mounted device to display the second image information on the second display device, where the first image information and the second image information are to present an aggregate view to a user of the head-mounted device. For example, the coordination can be associated with displaying over-under image content to produce an aggregate view.

In some examples, the processor404receives an input image, which can be a still image or can be part of a video. The processor404separates the input image into a plurality of image layers including a first image layer having the first image information, and a second image layer having the second image information.

In some examples, the processor404receives an indication from the head-mounted device that the first display device402is within a field of view of a front portion of the head-mounted device. The sending of the second image information to the head-mounted device, and the coordinating with the head-mounted device to cause the head-mounted device to display the second image information on the second display device are performed in response to receiving the indication.

In some examples, the indication is based on sensing by a front-facing camera of the head-mounted device that the first display device is within the field of view of the front portion of the head-mounted device.

FIG.5is a block diagram of a head-mounted device500including a first display device502and a sensor504to detect a second display device of an electronic device that is external of the head-mounted device500. The sensor504can include a camera (e.g., the front-facing camera112ofFIG.1) or another type of sensor.

The head-mounted device500includes a controller506to perform various tasks. The tasks of the controller506include an indication sending task508to send, to the electronic device, an indication of the detection of the second display device of the electronic device by the sensor504of the head-mounted device500.

The tasks of the controller506include a first image layer reception task510to receive a first layer of image information from the electronic device, the first layer of image information sent by the electronic device responsive to the indication.

The tasks of the controller506include a first image layer display task512to display the first layer of image information in the first display device502of the head-mounted device500, where the first layer of image information is part of an aggregate view that further includes a second layer of image information displayed on the second display device of the electronic device.

In some examples, the aggregative view includes an augmented reality presentation that includes the first layer of image information and the second layer of image information combined together.

In some examples, coordination of the display of the first layer of image information and the second layer of image information is controlled by the electronic device.

FIG.6is a flow diagram of a process600, which can be performed by the electronic device102ofFIG.1, for example.

The process600includes determining (at602), by an electronic device including a first display device, that the electronic device is paired with a head-mounted device comprising a second display device.

The process600includes receiving (at604), by the electronic device from the head-mounted device, an indication that the first display device is within a field of view of the head-mounted device.

The process600performs tasks606,608,610, and614in response to determining that the electronic device is paired with the head-mounted device and receiving the indication.

The process600includes separating (at606), by the electronic device, a video into a plurality of image layers comprising a first image layer including first image information, and a second image layer including second image information.

The process600includes displaying (at608) the first image information on the first display device of the electronic device.

The process600includes selectively routing (at610), by the electronic device, the second image information to the head-mounted device.

The process600includes coordinating (at612), by the electronic device with the head-mounted device, to cause the head-mounted device to display the second image information on the second display device, wherein the first image information and the second image information are to present an aggregate view to a user of the head-mounted device.

A non-transitory machine-readable or computer-readable storage medium can store machine-readable instructions, such as those of the aggregate view control engine120or instructions to perform the tasks ofFIGS.3-5. The storage medium can include any or some combination of the following: a semiconductor memory device such as a dynamic or static random access memory (a DRAM or SRAM), an erasable and programmable read-only memory (EPROM), an electrically erasable and programmable read-only memory (EEPROM) and flash memory or other type of non-volatile memory device; a magnetic disk such as a fixed, floppy and removable disk; another magnetic medium including tape; an optical medium such as a compact disk (CD) or a digital video disk (DVD); or another type of storage device. Note that the instructions discussed above can be provided on one computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly plural nodes. Such computer-readable or machine-readable storage medium or media is (are) considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any manufactured single component or multiple components. The storage medium or media can be located either in the machine running the machine-readable instructions, or located at a remote site from which machine-readable instructions can be downloaded over a network for execution.