HEAD-MOUNTED DISPLAY CONTROL APPARATUS, METHOD FOR CONTROLLING HEAD-MOUNTED DISPLAY CONTROL APPARATUS, AND NON-TRANSITORY COMPUTER READABLE STORAGE MEDIUM

A head-mounted display control apparatus includes one or more circuits, or one or more processors and at least one memory, the at least one memory being coupled to the one or more processors and having stored thereon instructions executable by the one or more processors. At least one of the one or more circuits or the execution of the instructions causes the head-mounted display control apparatus to function as an image capturing unit configured to capture an image; and a control unit configured to, based on phone call reception information, phone conversation information, or message reception information, perform control to display the captured image or a virtual space image on a display unit.

BACKGROUND

Technical Field

The present disclosure relates to a head-mounted display control apparatus, a method for controlling the head-mounted display control apparatus, and a non-transitory computer readable storage medium.

Description of the Related Art

In recent years, XR information processing terminals of a head-mounted display (HMD) type have become widespread. XR is a generic term for virtual reality (VR), augmented reality (AR), and mixed reality (MR).

HMD XR information processing terminals include those of a see-through type in which computer graphics (CG) are displayed on a transmissive display, and those of a non-see-through type in which CG and a real image captured by a camera are displayed on a non-transmissive display. In a see-through display, a person wearing an HMD is able to determine the state of a real space. In a non-see-through display, the person is unable to determine the state of a real space but is highly immersed in a VR space. However, there is a non-see-through display in which the state of a real space can be determined by displaying an image captured by a camera built in the HMD on the display. A state in which a real space is displayed is called a video see-through mode, relative to a state in which a VR space is displayed.

A smartphone compatible with the headset profile (HSP) or the hands-free profile (HFP) of Bluetooth (registered trademark) and an audio terminal such as a headset enable a phone conversation with the audio terminal instead of a microphone and a speaker of the smartphone. The HSP is the specifications of the function of transmitting and receiving audio data between a smartphone and an audio terminal. The HFP is the specifications of the function of making and receiving phone calls.

U.S. patent Ser. No. 10/324,294 discloses that the reception of a message or a phone call in an external smartphone is indicated by an icon in an HMD.

SUMMARY

A head-mounted display control apparatus includes one or more circuits, or one or more processors and at least one memory, the at least one memory being coupled to the one or more processors and having stored thereon instructions executable by the one or more processors. At least one of the one or more circuits or the execution of the instructions causes the head-mounted display control apparatus to function as an image capturing unit configured to capture an image; and a control unit configured to, based on phone call reception information, phone conversation information, or message reception information, perform control to display the captured image or a virtual space image on a display unit.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

In the above-described related art, when a phone call is received in a real space while a virtual space is displayed on a non-see-through head-mounted display (HMD), the situation of the real space, for example, whether there is another person, is unrecognizable, and it is impossible to immediately determine whether a phone conversation can be started.

The present disclosure provides a technique of enabling a captured image to be displayed, for example, when a phone call is being received.

FIG.1is a diagram illustrating an example configuration of a head-mounted display (HMD)120according to a first embodiment. The HMD120is an example of a display control apparatus that responds to a phone call reception signal. The HMD120includes a bus100, a central processing unit (CPU)101, a graphics processing unit (GPU)102, a posture sensor103, an analog-to-digital/digital-to-analog (AD/DA) converter104for audio, and a wireless unit105. The HMD120further includes an image processor106, a display107, a microphone108, a speaker109, a camera unit110, a random access memory (RAM)111, and a nonvolatile memory112.

Next, the roles of the individual components will be described. The bus100manages the data flow in the HMD120. The CPU101executes a program stored in the nonvolatile memory112or the like to control the entire HMD120, and executes system software and application software. In addition, the CPU101analyzes a digital audio signal to determine whether the digital audio signal is a phone call reception sound.

The GPU102generates a virtual reality image and outputs data of the image to the display107. The posture sensor103includes a gyro sensor and an acceleration sensor, and acquires information regarding the posture and position of the HMD120. In the present embodiment, the CPU101processes raw data received from the posture sensor103and acquires information regarding the posture and position of the HMD120. The information regarding the posture and position is reflected in generation of a virtual reality image. That is, the HMD120generates a virtual reality image corresponding to the forward direction of the HMD120in accordance with a motion of the HMD120including the posture sensor103, and displays the virtual reality image on the display107.

The AD/DA converter104converts an analog audio signal received from the microphone108into a digital audio signal, converts the digital audio signal into an analog audio signal, and outputs the analog audio signal to the speaker109. The wireless unit105is capable of connecting to another terminal by using a plurality of wireless standards (Bluetooth, Wi-Fi, and the like) to transmit and receive data. Specifically, the wireless unit105connects to a controller device to acquire operation information, or connects to a smartphone to transmit and receive audio data and acquire phone call reception information.

The image processor106acquires mosaic image data from the camera unit110and performs demosaicing and color correction processing. The display107displays a real space image captured by the camera unit110or a virtual reality image generated by the GPU102. Which is to be displayed on the display107is controlled by the CPU101. The logic of the control will be described below. The display107includes two panels, one is for the right eye and the other is for the left eye, and displays an input stereo image on the panels such that the stereo image is divided into partial images for the respective panels. A monocular image can also be input to the display107. In this case, the same image is displayed on the panels for the right eye and the left eye. Switching of the image to be displayed on the display107is controlled by the CPU101.

The microphone108physically acquires a sound and converts the sound into an analog electric signal. The speaker109outputs an analog signal as a physical sound.

The camera unit110includes two lenses and two sensors, is disposed so as to be capable of capturing images for the positions of the two eyes of the user wearing the HMD120, and outputs captured and digitalized mosaic image data to the image processor106. The image processor106combines the images received from the two sensors of the camera unit110into one stereo image. When the distance between the centers of the lenses of the camera unit110is different from the distance between the eyes of the user, the image processor106may correct a display image so that the user's perception of depth coincides with the reality.

The RAM111loads a program and data stored in the nonvolatile memory112at the startup of the HMD120and functions as a work area for the CPU101. The RAM111stores an acquired image and a generated image. The nonvolatile memory112stores a program and data. These operations are successively performed, and the user is able to view a moving image.

FIG.2is a flowchart illustrating a method for controlling the HMD120according to the first embodiment, and describes a phone call reception operation of the HMD120that responds to reception of a phone call. The steps are performed in the order indicated by the arrows inFIG.2unless otherwise stated. If the input and output of data in the individual steps do not depend on each other, the steps may be performed in a different order. The individual steps are performed by the CPU101, but part of processing, such as rendering processing, is performed by another processor such as the GPU102. In the present embodiment, a description will be given under the assumption that a VR game application is being executed.

In step S2010, the CPU101acquires phone call reception/phone conversation state information via the wireless unit105. In the present embodiment, a smartphone connected to the HMD120via the wireless unit105transmits phone call reception/phone conversation state information to the HMD120, and thereby the CPU101acquires the phone call reception/phone conversation state information via the wireless unit105.

In step S2020, the CPU101determines, based on the phone call reception/phone conversation state information, whether a phone call is being received or a phone conversation is being performed. If a determination is made that a phone call is being received or a phone conversation is being performed, the process proceeds to step S2030. Otherwise, the process proceeds to step S2060.

In step S2030, the CPU101suspends the processing of the above-described application that is being executed. In the present embodiment, the CPU101causes the game application to pause and causes the system to perform control.

In step S2040, the CPU101acquires a real space image by using the camera unit110and the image processor106. The real space image is captured by the camera unit110built in the HMD120and is a stereo image of a region in front of the HMD120.

In step S2050, the CPU101renders the real space image. Specifically, the CPU101converts the format of the image acquired in step S2040into a format suitable for the display107, and transmits the converted stereo image data to the display107via the GPU102. The stereo image is displayed on the display107. In step S2050, the CPU101also performs processing of waiting in synchronization with a video signal. Subsequently, the process proceeds to step S2080.

In step S2060, the CPU101executes the processing of the above-described application. In the present embodiment, the CPU101executes processing for one frame and creates application data that is necessary for generating rendering data. In a case where the processing of the application is executed in another thread, the application data may be acquired in synchronization with the processing of the other thread.

In step S2070, the CPU101renders a virtual space image that is based on the application processing. The CPU101transmits, based on the application data generated in step S2050and information from the posture sensor103, an image of a virtual space in front to the display107via the GPU102. The virtual space image is displayed on the display107. In step S2070, the CPU101also performs processing of waiting in synchronization with a video signal. The virtual space image may be a stereo image that is based on computer graphics or may be a stereo image actually captured by using a stereo fish eye lens. Subsequently, the process proceeds to step S2080.

In step S2080, the CPU101determines whether to end the application. If a determination is made that the application is not to be ended, the process returns to step S2010. If a determination is made that the application is to be ended, the process of the flowchart illustrated inFIG.2ends.

Typically, the user holds a controller in his/her hand and does not hold a smartphone when playing a game. In the present embodiment, in response to receipt of a phone call in the real space, the CPU101proceeds to step S2030, automatically performs switching to a video see-through mode, and displays a real space image. Accordingly, the user is able to view the real space, immediately find his/her smartphone around him/her, operate the smartphone, and answer the phone call, without taking off the HMD120. In addition, the user is able to easily determine whether to answer the phone call or whether to avoid an important topic in a phone conversation by determining whether there is somebody around him/her.

In the present embodiment, the microphone and speaker used for a phone conversation by the user are not limited. The user may use the microphone108and the speaker109of the HMD120, and the HMD120may transmit/receive audio information to/from the smartphone. Alternatively, the user may use a microphone and a speaker built in the smartphone to perform a phone conversation.

In the present embodiment, an example is illustrated in which the user operates his/her smartphone to answer an incoming call. The CPU101may display a phone call answering screen on the display107of the HMD120and may transmit/receive audio information to/from the smartphone. In this case, the user performs a phone conversation by using the microphone108and the speaker109of the HMD120.

In the present embodiment, a description has been given under the assumption that the target application is a game application. The target application is not limited thereto and may be any application, such as a VR moving image player, a work support application, or a virtual office application. In a VR moving image player, the CPU101decodes video data to generate a frame image and renders the frame image as a virtual reality image.

In the present embodiment, a description has been given of a phone conversation. When the smartphone receives a message, the CPU101may perform switching to the video see-through mode.

In the present embodiment, the phone call that is received may be a phone call received by either a smartphone connected to a public line or a phone conversation application of the system. However, in a case where users communicate with each other in a game application or the like, a screen is not automatically switched in response to receipt of a phone call within the application.

In the present embodiment, a display image is switched in accordance with only whether a phone call is being received. The HMD120is equipped with various sensors and a camera. The CPU101may use information obtained therefrom to detect whether there is a person near the user, may proceed to step S2030if a phone call is being received and there is a person near the user, and may switch the display of a real space image.

In the present embodiment, the CPU101generates a virtual reality image corresponding to the forward direction of the HMD120in accordance with a motion of the HMD120including the posture sensor103. Alternatively, the posture sensor103need not be included, and an image in a fixed direction may be displayed.

Although a stereo image is handled in the present embodiment, a monocular image may be handled instead of a stereo image.

In the present embodiment, the CPU101performs switching to display of a real space image in response to receipt of a phone call, but the present disclosure is not limited thereto. For example, the CPU101may display a virtual space image and a real space image in combination, or may make a small window on a virtual space image in response to receipt of a phone call and display a real space image in the video see-through mode only in the small window.

As described above, the camera unit110and the image processor106function as an image capturing unit, which captures an image. The display107is an example of a display unit. In step S2010, the CPU101acquires phone call reception information, phone conversation information, or message reception information from an external apparatus. The external apparatus is a smartphone, for example. The CPU101functions as a control unit, and performs control to display a captured image or a virtual space image on the display107, based on the phone call reception information, the phone conversation information, or the message reception information.

When a phone call is not being received or a phone conversation is not being performed, the CPU101performs control to display a virtual space image on the display107in step S2070. When a phone call is being received or a phone conversation is being performed, the CPU101performs control to display a captured image on the display107in step S2050.

In step S2050, the CPU101may perform control to display a captured image and a virtual space image on the display107. For example, the CPU101is capable of displaying a captured image and a virtual space image in combination, or displaying a captured image in a small window on a virtual space image.

The present embodiment may be applied to reception of a message in a smartphone. When no message is received, the CPU101may perform control to display a virtual space image on the display107. When a message is received, the CPU101may perform control to display a captured image on the display107for a predetermined period. When a message is received, the CPU101may perform control to display a captured image and a virtual space image on the display107for a predetermined period, similarly to the above.

When a person different from a person wearing the HMD120is present within a predetermined range, the CPU101may perform control to display a captured image or a virtual space image on the display107, based on phone call reception information, phone conversation information, or message reception information.

As described above, according to the present embodiment, the user is able to determine the situation of a real space and immediately determine whether he/she can start a phone conversation, without taking off the HMD120.

Second Embodiment

FIG.3is a flowchart illustrating a method for controlling the HMD120according to a second embodiment, and describes a phone call reception operation of the HMD120that responds to reception of a phone call.FIG.3is different fromFIG.2in that steps S3000, S3005, and S3010are provided instead of steps S2010and S2020. The operation is the same as in the first embodiment unless otherwise stated.

In step S3000, the CPU101acquires audio information via the microphone108and the AD/DA converter104. The audio information is acquired as digital audio data, which is generated by AD converting a sound received by the microphone108built in the HMD120by using the AD/DA converter104. The CPU101acquires audio data for 10 seconds before the time point at which step S3000is performed.

In step S3005, the CPU101analyzes the audio information acquired in step S3000to generate an audio information analysis result. In the present embodiment, the CPU101performs fast Fourier transform (FFT) on the audio information to acquire information in a range of 15 Hz to 16 kHz as an analysis result.

In step S3010, the CPU101determines whether the audio information analysis result generated in step S3005indicates a phone call reception sound. If the audio information analysis result indicates a phone call reception sound, the process proceeds to step S2030. Otherwise, the process proceeds to step S2060. In the present embodiment, if the sound pressure of the peak of at least one frequency is 60 dB or more as a result of FFT, the CPU101determines that the audio information analysis result indicates a phone call reception sound. The peak may be determined in the following manner. For example, if there is an amplitude ten times the median of the amplitude in a Log scale of the FFT result, it is determined that there is a peak.

In the present embodiment, the CPU101is capable of detecting a phone call reception sound of a smartphone by using the microphone108in an analog manner and performing switching to the video see-through mode in step S2050, even if the CPU101is incapable of digitally transmitting/receiving a signal via the wireless unit105. The present embodiment is capable of obtaining a merit similar to that of the first embodiment.

In the present embodiment, a result of FFT in the range of 15 Hz to 10 kHz is regarded as an audio analysis result, but the present disclosure is not limited thereto. Alternatively, the number of successive identical patterns may be calculated with respect to a time-series change in waveform, and if the number is two or more, the successive identical patterns may be determined to be a phone call reception sound. Regarding this, a warning sound such as a fire alarm sound often has audio information having repetition of an identical pattern, and causes automatic switching to the video see-through mode. This is an effective operation in terms of being able to notice the occurrence of a warning sound and visually recognize the cause of the occurrence. A phone call reception sound may be recorded in advance, and whether a sound is identical to the recorded phone call reception sound may be determined. Accordingly, even if a melody is set as a phone call reception sound, the melody can be determined to be a phone call reception sound.

As described above, the CPU101performs control to display a captured image or a virtual space image on the display107, based on a phone call reception sound or a message reception sound collected by the microphone108.

Third Embodiment

FIG.4is a flowchart illustrating a method for controlling the HMD120according to a third embodiment, and describes a phone call reception operation of the HMD120that responds to reception of a phone call.FIG.4is different fromFIG.2in that step S2020is deleted and steps S4000, S4011, S4012, and S4071to S4073are added. The operation is the same as in the first embodiment unless otherwise stated.

In step S4000, the CPU101sets the video see-through mode to OFF. This corresponds to setting of an initial state.

In step S2010, the CPU101acquires phone call reception/phone conversation state information via the wireless unit105.

In step S4011, the CPU101acquires a video see-through mode. Specifically, the CPU101acquires the video see-through mode set in step S4000described above or step S4073described below.

In step S4012, the CPU101determines whether the video see-through mode is ON. If the video see-through mode is ON, the process proceeds to step S4071via steps S2030to S2050. If the video see-through mode is OFF, the process proceeds to step S4071via steps S2060and S2070.

In step S4071, the CPU101determines, based on the phone call reception/phone conversation state information acquired in step S2010, whether a phone call is being received or a phone conversation is being performed. If a determination is made that a phone call is being received or a phone conversation is being performed, the process proceeds to step S4072. Otherwise, the process proceeds to step S2080.

In step S4072, the CPU101renders a video see-through mode selection screen. In a state in which the video see-through mode is OFF, the CPU101displays, on the display107via the GPU102, a video see-through mode selection screen5010illustrated in FIG.5A such that the video see-through mode selection screen5010is superimposed on a virtual space image. Accordingly, the user views the video see-through mode selection screen5010floating in a three-dimensional space. The video see-through mode selection screen5010includes a video see-through ON button5001, an accept button5003, and a decline button5004. The video see-through ON button5001is a button for switching the video see-through mode from OFF to ON.

In a state in which the video see-through mode is ON, the CPU101displays, on the display107via the GPU102, a video see-through mode selection screen5020illustrated inFIG.5Bsuch that the video see-through mode selection screen5020is superimposed on a real space image. The video see-through mode selection screen5020includes a video see-through OFF button5002, the accept button5003, and the decline button5004. The video see-through OFF button5002is a button for switching the video see-through mode from ON to OFF.

The user presses a button on the video see-through mode selection screen5010or5020by using a controller or his/her finger. In response to the accept button5003or the decline button5004being pressed, the CPU101starts or declines a phone conversation.

In step S4073, the CPU101acquires user operation information and sets the video see-through mode. In response to the video see-through ON button5001being pressed, the CPU101sets the video see-through mode to ON. In response to the video see-through OFF button5002being pressed, the CPU101sets the video see-through mode to OFF.

In step S2080, the CPU101determines whether to end the application. If a determination is made that the application is not to be ended, the process returns to step S2010. If a determination is made that the application is to be ended, the process of the flowchart illustrated inFIG.4ends.

According to the present embodiment, in response to receipt of a phone call, the CPU101displays the video see-through mode selection screen5010or5020for allowing the user to switch ON or OFF the vide see-through mode, instead of immediately setting the video see-through mode to ON. In accordance with a selection result made on the video see-through mode selection screen5010or5020, the CPU101switches ON or OFF the video see-through mode. Accordingly, a sudden change of the entire screen (the space to the user) unintended by the user can be prevented. In the present embodiment, the CPU101changes the screen in response to pressing of the video see-through ON button5001. Alternatively, the CPU101may change the screen in response to pressing of the accept button5003.

As described above, if a determination is made in step S4071that a phone call is being received or a phone conversation is being performed, the process proceeds to step S4072. In step S4072, the CPU101performs control to display an instruction button on the display107such that the instruction button is superimposed on a virtual space image. The instruction button is the video see-through ON button5001or the accept button5003.

In step S4073, if an operation instruction for the instruction button is provided, the CPU101performs control to display an image captured by the camera unit110on the display107. If an operation instruction for the instruction button is provided, the CPU101may perform control to display an image captured by the camera unit110and a virtual space image on the display107. If an operation instruction for the instruction button is not provided, the CPU101performs control to display a virtual space image on the display107.

Other Embodiments

The present disclosure can be implemented by a process in which a program implementing one or more functions of the above-described embodiments is supplied to a system or an apparatus via a network or a storage medium and one or more processors in the system or the apparatus read and execute the program. The present disclosure can also be implemented by a circuit that implements one or more functions (for example, an application specific integrated circuit (ASIC)).

The above-described embodiments are merely specific examples for implementing the present disclosure, and the technical scope of the present disclosure is not to be interpreted in a limited manner based on these embodiments. That is, the present disclosure can be implemented in various ways without departing from the technical spirit thereof or the main features thereof.

According to the present disclosure, it is possible to display a captured image, for example, when a phone call is being received.

This application claims the benefit of Japanese Patent Application No. 2022-184911, filed Nov. 18, 2022, which is hereby incorporated by reference herein in its entirety.