Head mounted display system capable of displaying a virtual scene and a real scene in a picture-in-picture mode, related method and related non-transitory computer readable storage medium

A head mounted display system includes a wearable body, an image capturing unit, a display unit and a processing unit. The wearable body can be worn by a user. The image capturing unit is mounted on the wearable body for capturing a real scene of a real environment. The display unit is mounted on the wearable body and for displaying a virtual scene of a virtual environment and the real scene in a picture-in-picture mode. The processing unit is coupled to the display unit and the image capturing unit. The present disclosure allows the user to see the virtual scene and the real scene combined in a same video frame synchronously and help the user to understand a current position or a current state of the user in the real environment, which effectively ensures the user's safety and prevents injuries caused by collision when the user experiences the virtual environment.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present disclosure relates to a head mounted display system, a related method and a related non-transitory computer readable storage medium, and more particularly, to a head mounted display system capable of displaying a virtual scene and a real scene in a picture-in-picture mode synchronously, a related method and a related non-transitory computer readable storage medium.

2. Description of the Prior Art

With the advancement and development of technology, the demand of interactions between a computer game and a user is increased. Human-computer interaction technology, e.g. somatosensory games, virtual reality (VR) environment, augmented reality (AR) environment, mixed reality (MR) environment and extended reality (XR) environment, becomes popular because of its physiological and entertaining function. A conventional display apparatus, such as a head mounted display (HMD), usually can only display a virtual scene of a virtual environment in a full-screen mode. However, a user cannot see a real scene of a real environment, and therefore, the user cannot be aware of a possible coming accident in the real environment, such as bumping into a real object easily, when the user experiences the virtual environment.

SUMMARY OF THE INVENTION

Therefore, it is an objective of the present disclosure to provide a head mounted display system capable of displaying a virtual scene and a real scene in a picture-in-picture mode synchronously, a related method and a related non-transitory computer readable storage medium for solving the aforementioned problem.

In order to achieve the aforementioned objective, the present disclosure discloses a head mounted display system. The head mounted display system includes a wearable body, an image capturing unit, a display unit and a processing unit. The wearable body is configured to be worn by a user. The image capturing unit is mounted on the wearable body and configured to capture a real scene. The display unit is mounted on the wearable body. The display unit is configured to display a virtual scene and the real scene in a picture-in-picture mode. The processing unit is coupled to the display unit and the image capturing unit.

In order to achieve the aforementioned objective, the present disclosure further discloses a method of utilizing a head mounted display system to display a virtual scene and a real scene in a picture-in-picture mode. The method includes utilizing an image capturing unit of the head mounted display system to capture the real scene; and utilizing a display unit of the head mounted display system to display the virtual scene and the real scene in the picture-in-picture mode.

In order to achieve the aforementioned objective, the present disclosure further discloses a non-transitory computer readable storage medium storing a program that causes a head mounted display system to execute a process. The process includes utilizing an image capturing unit of the head mounted display system to capture a real scene; and utilizing a display unit of the head mounted display system to display a virtual scene and the real scene in a picture-in-picture mode.

In summary, the present disclosure utilizes the display unit to display the virtual scene in the full-screen mode or display the virtual scene and the real scene combined in the same video frame synchronously in the picture-in-picture mode and further utilizes the processing unit to indicate the display unit to switch between the full-screen mode and the picture-in-picture mode in response to the activating command. Therefore, it allows a user to switch the display unit between the full-screen mode and the picture-in-picture mode by different ways to help the user to understand a current position or a current state of the user in a real environment, which effectively ensure the user's safety and prevents injuries caused by collision when the user experiences the virtual environment.

These and other obstruction objectives of the present disclosure will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment that is illustrated in the various figures and drawings.

DETAILED DESCRIPTION

Certain terms are used throughout the description and following claims to refer to particular components. As one skilled in the art will understand, electronic equipment manufacturers may refer to a component by different names. This document does not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms “include” and “comprise” are used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to . . . ” In addition, to simplify the descriptions and make it more convenient to compare between each embodiment, identical components are marked with the same reference numerals in each of the following embodiments. Please note that the figures are only for illustration and the figures may not be to scale. Also, the term “couple” is intended to mean either an indirect or direct electrical/mechanical connection. Thus, if a first device is coupled to a second device, that connection may be through a direct electrical/mechanical connection, or through an indirect electrical/mechanical connection via other devices and connections.

Please refer toFIG. 1andFIG. 2.FIG. 1is a diagram of a head mounted display system1according to a first embodiment of the present disclosure.FIG. 2is a functional block diagram of the head mounted display system1according to the first embodiment of the present disclosure. As shown inFIG. 1andFIG. 2, the head mounted display system1includes a wearable body11, which can be worn by a user, a display unit12, a processing unit13, a tracking unit14, and an image capturing unit15.

The image capturing unit15can be mounted on the wearable body11and coupled to the processing unit13. The image capturing unit15is configured to capture a real scene of a real environment. In this embodiment, the image capturing unit15can be a camera module. However, the present disclosure is not limited thereto.

The display unit12can be mounted on the wearable body11and coupled to the processing unit13. The display unit12is configured to display a virtual scene of a virtual environment in a full-screen mode or display the virtual scene of the virtual environment and the real scene of the real environment captured by the image capturing unit15combined in the same video frame in a picture-in-picture mode synchronously. The display unit12can switch from the full-screen mode to the picture-in-picture mode, and/or to switch from the picture-in-picture mode to the full-screen mode. In this embodiment, the display unit12can be a liquid crystal display (LCD), light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or any other display. However, the present disclosure is not limited thereto.

The processing unit13can be mounted inside the wearable body11. The processing unit13is configured to indicate the display unit12to switch from the full-screen mode to the picture-in-picture mode, and/or to switch from the picture-in-picture mode to the full-screen mode in response to an activating command. In this embodiment, the processing unit13can be implemented in software, firmware, hardware configuration, or a combination thereof. For example, the processing unit13can be a processor, such as a central processing unit, an application processor, a microprocessor, etc., or can be realized by application specific integrated circuits (ASIC). However, the present disclosure is not limited thereto.

The tracking unit14can be coupled to the processing unit13and track a position, an orientation or a pose of the head mounted display system1. In this embodiment, the tracking unit can include an inertial measurement unit (IMU)141mounted inside the wearable body11, such as a gyroscope, an accelerometer, a magnetic sensor or a combination thereof, for tracking a position, an orientation or a pose of the wearable body11to determine the position, an orientation or a pose of the head mounted display system1. However, the present disclosure is not limited to this embodiment. The inertial measurement unit can be mounted a component another than the wearable body. For example, in another embodiment, the tracking unit14can further include a hand sensor, a lower body sensor or an external camera module, and the inertial measurement unit can be mounted the hand sensor, the lower body sensor or the external camera module, for tracking a position, an orientation or a pose of the hand senor, the lower body sensor or the external camera module to determine the position, the orientation or the pose of the head mounted display system1.

In this embodiment, the tracking unit14is configured to collect data for determining generation of the activating command. For example, the activating command can be generated when a tracking result of the tracking unit14meets a predetermined condition. However, determination of generation of the activating command is not limited to this embodiment. In another embodiment, which will be described later, the data collected from the tracking unit14may not be used to determine generation of the activating command.

Furthermore, in this embodiment, the determination of the generation of the activating command and the generation of the activating command can be performed by the tracking unit14. However, the present disclosure is not limited thereto. In another embodiment, the tracking unit can transmit the collected data or the tracking result to the processing unit, and the determination of the generation of the activation command and the generation of the activating command can be performed by processing unit.

Furthermore, in this embodiment, the display unit12, the processing unit13, the tracking unit14and the image capturing unit15are disposed on the wearable body11. However, the present disclosure is not limited to this embodiment. For example, in another embodiment, the head mounted display system further includes a remote computing apparatus disposed away from the wearable body separately and a communication module disposed on the wearable body for constructing a communication channel to the remote computing apparatus. The remote computing apparatus can be an edge computing device, a cloud computing device, a local host computer, a remote sever, a smartphone, or the like. The communication module can establish a wired connection or a wireless connection between elements on the wearable body and elements on the remote computing apparatus. In this embodiment, at least one of the processing unit and the tracking unit can be at least partly disposed or configured on the remote computing apparatus other than the wearable body and/or distributes part of the tasks to the remote computing apparatus, so that the remote computing apparatus can receive the tracking result of the tracking unit or transmit the activating command via the communication module, so as to reduce the size and calculation of the head mounted display, which makes the wearable body lightweight and portable.

Please refer toFIG. 3toFIG. 5.FIG. 3is a flow chart diagram illustrating a method of switching the head mounted display system1between the full-screen mode and the picture-in-picture mode according to the first embodiment of the present disclosure.FIG. 4is a diagram illustrating the display unit12displays the virtual scene in the full-screen mode according to the first embodiment of the present disclosure.FIG. 5is a diagram illustrating the display unit12displays the virtual scene and the real scene in the picture-in-picture mode according to the first embodiment of the present disclosure. As shown inFIG. 3, the method includes the following steps:

S1: The display unit12displays the virtual scene of the virtual environment in the full-screen mode.

S2: The image capturing unit15captures the real scene of the real environment.

S3: The tracking unit14tracks at least one of the position, the orientation and the pose of the head mounted display system1.

S4: The activating command is generated when the tracking result of the tracking unit14meets the predetermined condition.

S5: The processing unit13indicates the display unit12to switch from the full-screen mode to the picture-in-picture mode in response to the activating command.

Detailed description for the steps is provided as follows. In step S1, when the user wears the head mounted display10, the display unit12can display the virtual scene of the virtual environment in the full-screen mode for the user firstly, as shown inFIG. 4. In steps S2to S4, when the user experiences the virtual environment, meanwhile the tracking unit14can track at least one of the position, the orientation and the pose of the head mounted display system1by tracking at least one of the position, the orientation and the pose of wearable body11in the real environment, and the image capturing unit15can capture the real scene of the real environment. Furthermore, the activating command is generated when the tracking result of the tracking unit14meets the predetermined condition. The predetermined condition can be determined according to a relation between the head mounted display system1and the real environment.

For example, the predetermined condition can refer to that a distance between the wearable body11worn by the user and a real object, such as a television2which is shown inFIG. 4andFIG. 5, or a wall, in the real environment is equal to or less than a predetermined distance. The predetermined distance can preferably be 50 centimeters. In other words, when the distance between the wearable body11and the real object in the real environment is equal to or less than 50 centimeters during movement of the user, the activating command is generated, which can provide a warning message for the user to avoid collision.

However, the predetermined condition is not limited thereto. In another embodiment, the predetermined condition can refer to a predetermined orientation or a predetermined pose, and the activating command can be generated when the orientation or the pose of the head mounted display system tracked by the tracking unit meets the predetermined orientation or the predetermine pose.

Alternatively, in another embodiment, which will be described later, the tracking unit can further include a camera module, a hand sensor worn on the user's hand, a lower body sensor worn on the user's low body, or a combination thereof for determining a hand gesture of the user, a hand movement of the user or a lower body movement of the user, and the predetermined condition can refer to a predetermined hand gesture, a predetermined hand movement, a predetermined lower body movement, or a combination thereof, and the activating command can be generated when the hand gesture, the hand movement, or the lower body movement of the user tracked by the tracking unit meets the predetermined hand gesture, the predetermined hand movement, or the predetermined lower body movement.

Alternatively, in another embodiment, the tracking unit can only be omitted, and the image capturing unit is further configured to track at least one of a hand gesture of the user, a hand movement of the user, a lower body movement of the user, a position of the head mounted display system, an orientation of the head mounted display system and a pose of the head mounted display system according to images captured by the image capturing unit.

Afterwards, in step S5, the processing unit13can indicate the display unit12to switch from the full-screen mode, as shown inFIG. 4, to the picture-in-picture mode, as shown inFIG. 5, in response to the activating command, so as to allow the user to see the virtual scene and the real scene combined in the same video frame at the same time and help the user to understand a current position or a current state of the user in the real environment, which effectively ensure the user's safety and prevents injuries caused by collision when the user experiences the virtual environment.

Besides, a size or a location of a window21of the real scene displayed in the picture-in-picture mode can be adjusted when the tracking result of the tracking unit14or the image capturing unit15meets a predetermined condition. For example, the user can adjust the size or the location of the windows21of the displayed real scene by a specific hand gesture and/or a specific hand movement. For another example, the size or the location of the windows21can be auto-adjusted based on a tracking result (for example, an eye tracking result) of the tracking unit14for providing better user experience.

Understandably, if the distance between the wearable body11and the real object increases again and is greater than the predetermined distance, such as 50 centimeters, the display unit can be switched from the picture-in-picture mode to the full-screen mode for enhancing the immersive experience.

Please refer toFIG. 6toFIG. 8.FIG. 6is a diagram of a head mounted display system1′ according to a second embodiment of the present disclosure.FIG. 7is a functional block diagram of the head mounted display system1′ according to the second embodiment of the present disclosure.FIG. 8is a flow chart diagram illustrating a method of switching the head mounted display system1′ between a full-screen mode and a picture-in-picture mode according to the second embodiment of the present disclosure. As shown inFIG. 6toFIG. 8, different from the head mounted display system1of the first embodiment, the head mounted display system1′ of this embodiment includes a wearable body11′, a display unit12′, a processing unit13′, an image capturing unit15′ and a switch unit16′. The image capturing unit15′ is disposed on the wearable body11′. The display unit12′ is disposed on the wearable body11′. The processing unit13′ is disposed on the wearable body11′ and coupled to the display unit12′ and the image capturing unit15′. The switch unit16′ is disposed on the wearable body11′ and coupled to the processing unit13′ for generating the activating command when a state of the switch unit16′ is changed. In other words, in this embodiment, the user can switch the display unit12′ from the full-screen mode to the picture-in-picture mode by changing the state of the switch unit16′ according to practical demands. Furthermore, understandably, the user can switch the display unit12′ from the picture-in-picture mode to the full-screen mode by changing the state of the switch unit16′ again. Besides, in this embodiment, the switch unit16′ can include a physical button on the wearable body11′, and the state of the switch unit16′ can be changed by pressing or clicking the physical button. However, the present disclosure is not limited thereto. In another embodiment, the switch unit also can be a virtual button on the wearable body or displayed by the display unit and be activated or deactivated by touch operation.

Please refer toFIG. 9toFIG. 11.FIG. 9is a diagram of a head mounted display system1″ according to a third embodiment of the present disclosure.FIG. 10is a functional block diagram of the head mounted display system1″ according to the third embodiment of the present disclosure.FIG. 11is a flow chart diagram illustrating a method of switching the head mounted display system1″ between a full-screen mode and a picture-in-picture mode according to the third embodiment of the present disclosure. As shown inFIG. 9toFIG. 11, different from the head mounted display systems1,1′ of the aforementioned embodiments, the head mounted display system1″ of this embodiment includes a wearable body11″, a display unit12″ mounted on the wearable body11″, a processing unit13″ coupled to the display unit12″, an image capturing unit15″ mounted on the wearable body11″ and coupled to the processing unit13″, a remote controller17″, a communication module18″ and a remote computing apparatus19″. The processing unit13″ can be configured on the remote computing apparatus19″. The communication module18″ constructs a communication channel among the processing unit13″ on the remote computing apparatus19″, the display unit12″ disposed on the wearable body11″ and the remote controller17″. In other words, the processing unit13″ is coupled to the display unit12″ and the image capturing unit15″ by the communication module18″ for indicating the display unit12″ to switch between the full-screen mode and the picture-in-picture mode, and the remote controller17″ is coupled to and communicated with the processing unit13″ by the communication module18″ for generating and transmitting the activating command remotely to the processing unit13″ via the communication module18″ when the remote controller17″ is operated. In other words, in this embodiment, the user can switch the display unit12″ from the full-screen mode to the picture-in-picture mode by operating the remote controller17″ according to practical demands. Furthermore, understandably, the user can switch the display unit12″ from the picture-in-picture mode to the full-screen mode by operating the remote controller17″ again.

Please refer toFIG. 12toFIG. 14.FIG. 12is a diagram of a head mounted display system1′″ according to a fourth embodiment of the present disclosure.FIG. 13is a functional block diagram of the head mounted display system1′″ according to the fourth embodiment of the present disclosure.FIG. 14is a flow chart diagram illustrating a method of switching the head mounted display system1′″ between a full-screen mode and a picture-in-picture mode according to the fourth embodiment of the present disclosure. As shown inFIG. 12toFIG. 14, different from the head mounted display system1,1′,1″ of the aforementioned embodiments, the head mounted display system1′″ of this embodiment includes a wearable body11′″, a display unit12′, a processing unit13′, a tracking unit14′″ and an image capturing unit15″. The display unit12′, the processing unit13′″ and the image capturing unit15′″ are disposed on the wearable body11′″. The processing unit13′″ is coupled to the display unit12′″ and the image capturing unit15″. The tracking unit14′″ includes a camera module142′″ coupled to the processing unit13′″ and disposed on the wearable body11′″, a hand sensor143′″ coupled to the processing unit13′″ and worn on the user's hand and a lower body sensor144′″ coupled to the processing unit13′″ and worn on the user's lower body for tracking at least one of the hand gesture of the user, the hand movement of the user and the lower body movement of the user, and the predetermined condition can refer to the predetermined hand gesture, the predetermined hand movement, the predetermined lower body movement, or a combination thereof. For example, the activating command can be generated when the hand gesture, the hand movement or the lower body movement of the user tracked by the tracking unit meets the predetermined hand gesture, the predetermined hand movement, or the predetermined lower body movement. However, the present disclosure is not limited thereto. The tracking unit can only include at least one of the camera module, the hand sensor and the lower body sensor. For example, in another embodiment, the camera module of the tracking unit can be omitted, and the tracking unit can be coupled to the image capturing unit, as to determine at least one of the hand gesture of the user, the hand movement of the user and the lower body movement of the user by the images captured by the image capturing unit, the hand sensor and the lower body sensor.

Please refer toFIG. 15toFIG. 17.FIG. 15is a diagram of a head mounted display system1″″ according to a fifth embodiment of the present disclosure.FIG. 16is a functional block diagram of the head mounted display system1″″ according to the fifth embodiment of the present disclosure.FIG. 17is a flowchart diagram illustrating a method of switching the head mounted display system1″″ between a full-screen mode and a picture-in-picture mode according to the fifth embodiment of the present disclosure. As shown inFIG. 15toFIG. 17, the head mounted display system1″″ of this embodiment includes a wearable body11″″, a display unit12″″, a processing unit13″″, a tracking unit14″″ and an image capturing unit15″″. The display unit12″″, the processing unit13″″ and the image capturing unit15″″ are disposed on the wearable body11″″. The processing unit13″″ is coupled to the display unit12″″ and the image capturing unit15″″. The tracking unit14″″ includes an inertial measurement unit141″″ coupled to the processing unit13″″, and a lower body sensor144″″ coupled to the processing unit13″″ and worn on the user's lower body for tracking at least one of the lower body movement of the user, the position of the head mounted display system1″″, the orientation of the head mounted display system1″″, the pose of the head mounted display system1″″. The image capturing unit15″″ includes a camera module and configured to track at least one of the hand gesture and the hand movement of the user. The activating command can be generated when a tracking result of the tracking unit14″″ and a tracking result of the image capturing unit15″″ respectively meet a first predetermined condition and a second predetermined condition. The first predetermined condition can refer to a predetermined distance, a predetermined position of the head mounted display system1″″, a predetermined orientation of the head mounted display system1″″ or a predetermined pose of the head mounted display system1″″, and the second predetermined condition can refer to a predetermined hand gesture, a predetermined hand movement, and a predetermined lower body movement.

In this embodiment, the inertial measurement unit141″″ can be mounted on the wearable body11″″, so that the tracking unit14″″ can track the position, the orientation or the pose of the wearable body11″″ to determine the position, the orientation or the pose of the head mounted display system1″″.

Furthermore, determination of the generation of the activation command can be a multi-stage operation, for example, two-stage operation. For example, when the tracking unit14″″ determines a distance between the wearable body11″″ and a real object is less than the predetermined distance, the image capturing unit15″″ further determines whether the hand gesture of the user matches with the predetermined hand gesture. When the image capturing unit15″″ determines the hand gesture of the user matches with the predetermined hand gesture, the activating command is generated. However, the present disclosure is not limited to this embodiment. In another embodiment, the determination of the generation of the activation command can be one-stage operation, i.e., the activating command can be generated when the tracking result of the tracking unit and the tracking result of the image capturing unit respectively meet the first predetermined condition and the second predetermined condition at a same time.

Furthermore, the above-described embodiments of the present disclosure can be implemented in software, firmware, hardware configuration, or combination thereof, and it is also possible to provide a non-transitory computer readable storage medium for storing a program that causes a head mounted display system to execute a process by a processor in accordance with the above-described methods of the present disclosure. The processor can be a central processing unit, an application processor, a microprocessor, etc., or can be realized by an application specific integrated circuit (ASIC). The computer-readable recording medium can be a Read-Only Memory (ROM), a random-access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, or an optical data storage device. However, the present disclosure is not limited thereto.

In contrast to the prior art, the present disclosure utilizes the display unit to display the virtual scene in the full-screen mode or display the virtual scene and the real scene combined in the same video frame synchronously in the picture-in-picture mode and further utilizes the processing unit to indicate the display unit to switch between the full-screen mode and the picture-in-picture mode in response to the activating command. Therefore, it allows the user to switch the display unit between the full-screen mode and the picture-in-picture mode by different ways to help the user to understand the current position or the current state of the user in the real environment, which effectively ensure the user's safety and prevents injuries caused by collision when the user experiences the virtual environment.