Patent Description:
Along with the improvement of technology, head mount displays (HMD) with XR function become more and more popular. An HMD may create an XR environment for a user such that the user may interact with virtual objects shown in the XR scene provided by the XRa environment. The user of the HMD may perform a hand gesture to interact with the XR environment. However, interacting with the XR environment by the hand gesture may cause a lot of problems, for example, the HMD may receive unexpected input cause by occlusion issues. In addition, performing hand gesture may fatigue the user. On the other hand, the user of the HMD may interact with the XR environment by using a remote controller. However, interacting with the XR environment by the remote controller is inconvenient for the user. For example, the user has to establish the mapping relationship between the bottom of the remote controller and the XR function. Furthermore, since the user using the remote controller cannot be hand free, the remote controller is not suitable for the augmented reality (AR) system.

<CIT> discloses a method, including: capturing, using an image sensor of an information handling device, a user gesture input; determining, using a processor, that the user gesture input comprises an activating gesture input; capturing, using the image sensor of the information handling device, controlling gesture input of the user; detecting, within the captured controlling gesturing input, gestures provided on a surface and mimicking use of a mouse; and controlling an application running on the information handling device based on the controlling gesture input of the user.

<CIT> discloses an augmented reality or virtual reality (AR/VR) system including a virtual input device that can be rendered by an HMD, and a wearable impact detection device, such as a ring, smart watch, wristband, etc. with an inertial measurement unit (IMU), that can be used in conjunction with the HMD to track a location of the user's hands relative to the perceived location of the rendered virtual keyboard using, e.g., vision-based tracking via the HMD and determine when an intended input (e.g., button press) is entered by the user by detecting an impact of the user's finger(s) on a physical surface. The AR/VR system can then determine which key is pressed based on the physical location of the user's hands (e.g., using the vision-based tracking) and, more precisely, the user's finger(s) causing the detected impact and the closest key of the virtual input device to the detected point of impact.

The disclosure is directed to a system and a method for remotely controlling extended reality by a virtual mouse.

The present invention is directed to a system for remotely controlling extended reality by a virtual mouse, as set forth in claim <NUM>.

The present invention is further directed to a method for remotely controlling extended reality by a virtual mouse, as set forth in claim <NUM>.

Preferred embodiments of the present invention may be gathered from the dependent claims.

Based on the above description, the present invention provides a convenient and intuitive way for the user of an HMD to interact with the XR environment.

To make the aforementioned more comprehensible, several embodiments accompanied with drawings are described in detail as follows.

The drawings illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure.

<FIG> illustrates a schematic diagram of a usage scenario of a system <NUM> for remotely controlling extended reality by a virtual mouse according to one embodiment of the present invention, wherein the system <NUM> includes an HMD <NUM> worn on a user's head and a remote controller <NUM> worn on by the user's hand <NUM>. The HMD <NUM> is used for providing a XR environment (or XR scene) such as a virtual reality (VR) environment, an AR environment, or a mixed reality (MR) environment for the user.

After the HMD <NUM> detecting a specific gesture performed by the hand <NUM> on a plane (e.g., a desktop) <NUM>, the HMD <NUM> switches a control mode of the HMD <NUM> (or the control mode of the XR scene) from another control mode (e.g., a keyboard mode) to a mouse mode. When the HMD <NUM> is in the mouse mode, a virtual mouse simulated by the hand <NUM> wearing the remote controller <NUM> is enabled. The user may interact with the XR scene provided by the HMD <NUM> by using the virtual mouse so as to update the XR scene. The updating of the XR scene may relate to an updating of a remote desktop or a data transmission between the HMD <NUM> and a cloud server.

<FIG> illustrates a schematic diagram of the system <NUM> according to the one embodiment of the present invention. The system <NUM> includes the HMD <NUM> and one or more remote controllers communicatively connecting to the HMD <NUM>. It is assumed that the one or more remote controllers include a remote controller <NUM> and a remote controller <NUM>. However, the number of the one or more controller is not limited thereto.

The HMD <NUM> includes a processor <NUM>, a storage medium <NUM>, a transceiver <NUM>, an image capture device <NUM>, and a display <NUM>. The processor <NUM> may be, for example, a central processing unit (CPU), or other programmable general purpose or special purpose micro control unit (MCU), a microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a graphics unit (GPU), an arithmetic logic unit (ALU), a complex programmable logic device (CPLD), a field programmable gate array (FPGA),, or other similar device or a combination of the above devices. The processor <NUM> is coupled to the storage medium <NUM>, the transceiver <NUM>, the image capture device <NUM>, and the display <NUM>.

The storage medium <NUM> may be, for example, any type of fixed or removable random access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk drive (HDD), a solid state drive (SSD) or similar element, or a combination thereof. The storage medium <NUM> may be a non-transitory computer readable storage medium configured to record a plurality of executable computer programs, modules, or applications to be loaded by the processor <NUM> to perform the function of the HMD <NUM>.

The transceiver <NUM> is configured to transmit or receive wired/wireless signals. The transceiver <NUM> may also perform operations such as low noise amplifying, impedance matching, frequency mixing, up or down frequency conversion, filtering, amplifying, and so forth. The processor <NUM> may communicate with other devices (e.g., remote controller, cloud server, or remote desktop) via the transceiver <NUM>.

The image capture device <NUM> may be, for example, a camera or a photographic device for capturing images. The image capture device <NUM> may include a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor.

The display <NUM> may be used for displaying video data or image data such as an XR scene of the XR environment for the user wearing the HMD <NUM>. The display <NUM> may include a liquid-crystal display (LCD) or an organic light-emitting diode (OLED) display. In one embodiment, the display <NUM> may provide an image beam to the eye of the user to form the image on the retina of the user such that the use may see an XR scene created by the HMD <NUM>.

The remote controller <NUM> may be a ring-type device that can be worn on a finger (e.g., forefinger) of the user's hand <NUM>. The remote controller <NUM> includes a processor <NUM>, a storage medium <NUM>, and a transceiver <NUM>. The remote controller <NUM> further includes one or more inertial measurement units (IMUs) and may also include an optical sensor <NUM>. It is assumed that the one or more IMUs include an IMU <NUM> and an IMU <NUM>. However, the number of the one or more IMUs is not limited thereto.

The functions or structures of the processor <NUM> may be similar to the processor <NUM>. The processor <NUM> is coupled to the storage medium <NUM>, the transceiver <NUM>, the IMU <NUM>, the IMU <NUM>, and the optical sensor <NUM>.

The functions or structures of the storage medium <NUM> may be similar to the storage medium <NUM>. The storage medium <NUM> may be a non-transitory computer readable storage medium configured to record a plurality of executable computer programs, modules, or applications to be loaded by the processor <NUM> to perform the function of the remote controller <NUM>.

The functions or structures of the transceiver <NUM> may be similar to the transceiver <NUM>. The processor <NUM> may communicate with other devices (e.g., HMD <NUM>) via the transceiver <NUM>.

The IMU <NUM> (or IMU <NUM>) may include an accelerometer or a gyroscope. The optical sensor <NUM> may include a CMOS sensor or a CCD sensor.

The remote controller <NUM> may be a ring-type device that can be worn on a finger (e.g., middle finger) of the user's hand <NUM>. The remote controller <NUM> includes a processor <NUM>, a storage medium <NUM>, and a transceiver <NUM>. The remote controller <NUM> further includes one or more IMUs and may also include an optical sensor <NUM>. It is assumed that the one or more IMUs include an IMU <NUM> and an IMU <NUM>. However, the number of the one or more IMUs is not limited thereto.

The IMU <NUM> (or IMU <NUM>) may be an accelerometer or a gyroscope. The optical sensor <NUM> may include a CMOS sensor or a CCD sensor.

The HMD <NUM> is configured to detect a gesture of the user's hand <NUM> and to determine whether the control mode of the HMD <NUM> is switched to the mouse mode according to the detection result. <FIG> illustrates a method for detecting a control mode for the XR scene according to one embodiment of the present invention. In step S301, the HMD <NUM> captures an image including the user's hand <NUM> and the plane <NUM> through the image capture device <NUM>.

In step S302, the HMD <NUM> determines whether the HMD <NUM> enters the mouse mode. That is, the HMD <NUM> determines whether the virtual mouse for interacting with the XR scene is enabled. If the HMD <NUM> has entered the mouse mode, the method proceeds to step S303. If the HMD has not entered the mouse mode, the method proceeds to step S301. When the HMD <NUM> enters the mouse mode, the HMD <NUM> enables the virtual mouse, such that the hand <NUM> wearing the remote controller <NUM> and/or remote controller <NUM> simulates the virtual mouse which can interact with the XR scene provided by the HMD <NUM>.

The HMD <NUM> detects the image captured by the image capture device <NUM> to obtain data of the hand <NUM> and data of the plane <NUM> from the image. The HMD <NUM> determines whether the HMD <NUM> enters the mouse mode according to the data of the hand <NUM> and the data of the plane <NUM>. In other words, the HMD <NUM> determines whether to enable the virtual mouse or not according to the data of the hand <NUM> and the data of the plane <NUM>.

<FIG> illustrates a schematic diagram of a result of the hand tracking according to one embodiment of the present invention. The HMD <NUM> may perform a hand tracking algorithm on the captured image to obtain the data of the hand <NUM>, wherein the data of the hand <NUM> may include a plurality of joints of the hand <NUM>, such as joint J1, J2, and J3. In one embodiment, the joint J1, J2, and J3 may respectively represent a palm of the hand <NUM>, a metacarpophalangeal joint (MCP) of the forefinger of the hand <NUM>, and the distal interphalangeal joint (DIP) of the forefinger of the hand <NUM>. The data of the hand <NUM> may further include positions of a joint (e.g., joint J1) of the hand <NUM>. The HMD <NUM> may determine an angle formed by the plurality of joints according to the data of the hand <NUM> so as to detect a flexion of the hand <NUM>. For example, the HMD <NUM> may determine an angle θ formed by the joints J1, J2, and <NUM>, as shown in <FIG>. On the other hand, the HMD <NUM> may perform an object detection algorithm on the captured image to obtain the data of the plane <NUM>, wherein the data of the plane <NUM> may include a position of the plane <NUM>. The HMD <NUM> may determine a distance between the joint J1 and the plane <NUM> according to the data of the hand <NUM> and the data of the plane <NUM>. The HMD <NUM> may determine that the HMD <NUM> has entered the mouse mode according to the angle θ and/or the distance between the joint J1 and the plane <NUM>.

In one embodiment, the HMD <NUM> may determine that the HMD <NUM> has entered the mouse mode if the angle θ is less than or equal to an angle threshold and/or the distance between the joint J1 and the plane <NUM> is less than or equal to a distance threshold, and determine that the HMD <NUM> has not entered the mouse mode if the angle θ is greater than the angle threshold and/or the distance between the joint J1 and the plane <NUM> is greater than the distance threshold.

For example, the angle threshold may be set to <NUM> degree. When the hand <NUM> is relaxed, the angle θ may be closed to <NUM> degree. Since the angle θ is greater than the angle threshold, the HMD <NUM> determines that no flexion of the hand <NUM> is detected. Accordingly, the HMD <NUM> determines that the HMD <NUM> has not entered the mouse mode. The HMD <NUM> may determine that the control mode of HMD <NUM> is switched to the keyboard mode, wherein the user may interact with the XR scene through a keyboard command received by the transceiver <NUM>. On the other hand, when the hand <NUM> is bent, the angel θ may become less than the angle threshold (e.g., angel θ = <NUM> degree), and the HMD <NUM> determines that a flexion of the hand <NUM> is detected. Accordingly, the HMD <NUM> determines that the HMD <NUM> has entered the mouse mode. For example, the distance threshold may be set to <NUM>. When the hand <NUM> is put on the table, the distance between the joint J1 and the plane <NUM> may become less than or equal to the distance threshold. Accordingly, the HMD <NUM> determines that the HMD <NUM> has entered the mouse mode.

Referring to <FIG>, after determining that the HMD <NUM> has entered the mouse mode, in step S303, the HMD <NUM> receives signals from the remote controller <NUM> and/or <NUM>. In step S304, the HMD <NUM> may update the XR scene provided by the HMD <NUM> according to the signals from the remote controller <NUM> and/or <NUM> based on a mouse function of the enabled virtual mouse. The mouse function of the virtual mouse may include a move operation, a click operation (e.g., a single-click operation or a double-click operation), or a scroll operation (e.g., a scroll up operation or a scroll down operation).

In step S305, the HMD <NUM> determines whether the HMD <NUM> leaves the mouse mode. That is, the HMD <NUM> determines whether to disable the virtual mouse. If the HMD <NUM> has left the mouse mode, the procedure ends. If the HMD <NUM> has not left the mouse mode, the method proceeds to step S303.

The HMD <NUM> may determine that the HMD <NUM> has left the mouse mode according to the angle θ and/or the distance between the joint J1 and the plane <NUM>. In one embodiment, the HMD <NUM> may determine that the HMD <NUM> has left the mouse mode if the angle θ is greater than an angle threshold and/or the distance between the joint J1 and the plane <NUM> is greater than a distance threshold, and determine that the HMD <NUM> has not left the mouse mode if the angle θ is less than or equal to the angle threshold and/or the distance between the joint J1 and the plane <NUM> is less than or equal to the distance threshold.

In step S305, the HMD <NUM> may switch the control mode from the mouse mode to another control mode (e.g., the keyboard mode). That is, the HMD <NUM> may disable the virtual mouse for interacting with the XR scene.

When the virtual mouse is enabled, the HMD <NUM> may move a cursor corresponding to the virtual mouse according to a hand tracking result for the hand <NUM>. <FIG> illustrates a schematic diagram of performing a move operation of the virtual mouse by using hand tracking according to one embodiment of the present invention. The HMD <NUM> may perform hand tracking algorithm on the image captured by the image capture device <NUM> to obtain the data of the hand <NUM>, wherein the data of the hand <NUM> may include positions of a joint J1 and time information corresponding to the positions, wherein the joint J1 may represent a palm of the hand <NUM>. The accelerate or speed of the moving hand <NUM> may be derived by the HMD <NUM> according to the position of the joint and the time information. When the hand <NUM> put on the plane <NUM> is moving, the position of the hand <NUM> changes. Accordingly, the HMD <NUM> moves the cursor corresponding to the virtual mouse in the XR scene according to the position (or accelerate or speed) of the hand <NUM>.

<FIG> illustrates a schematic diagram of performing a click operation by using one remote controller according to one embodiment of the present invention. When the virtual mouse is enabled, the user may perform a click operation in the XR scene by using one remote controller. It is assumed that the remote controller <NUM> is worn on the finger <NUM> (e.g., forefinger) of the hand <NUM> and no remote controller is worn on the finger <NUM> (e.g., middle finger) of the hand <NUM>. If the finger <NUM> makes a click action, the HMD <NUM> receives signals from the IMU <NUM> and/or IMU <NUM>, and the HMD <NUM> performs a click operation by the virtual mouse in the XR scene according to the received signals.

In one embodiment, the HMD <NUM> may perform a click operation associated with the finger <NUM> (i.e., a left click) wearing the remote controller <NUM> according to a signal measured by the IMU <NUM>, wherein the IMU <NUM> may be an accelerometer or a gyroscope. For example, after the HMD <NUM> receiving signal <NUM> measured by the IMU <NUM>, the HMD <NUM> may detect the amplitude of the signal <NUM>. The HMD <NUM> may perform the click operation associated with the finger <NUM> if the amplitude of the signal <NUM> is greater than a threshold TA. Otherwise, the HMD <NUM> may not perform the click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may perform a click operation associated with the finger <NUM> (i.e., a right click) not wearing the remote controller <NUM> according to a signal measured by the IMU <NUM>, wherein the IMU <NUM> may be an accelerometer (gyroscope may not detect any signal if the finger <NUM> wearing the remote controller <NUM> is not moving). For example, after the HMD <NUM> receiving signal <NUM> measured by the IMU <NUM>, the HMD <NUM> may detect the amplitude of the signal <NUM>. The HMD <NUM> may perform the click operation associated with the finger <NUM> if the amplitude of the signal <NUM> is greater than a threshold TB but less than or equal to a threshold TC, wherein the threshold TB may be less than the threshold TC, and the threshold TC may be less than the threshold TA. Otherwise, the HMD <NUM> may not perform the click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may perform a single-click operation or a double-click according to the number of the extreme values (e.g., wave peaks or wave valleys) of the signal measured by the IMU <NUM>. For example, the HMD <NUM> may perform a single-click operation associated with the finger <NUM> according to the signal <NUM> in response to the number of the extreme values of the signal <NUM> being equal to one. Otherwise, the HMD <NUM> may not perform the single-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a single-click operation associated with the finger <NUM> according to the signal <NUM> in response to the number of the extreme values of the signal <NUM> being equal to one. Otherwise, the HMD <NUM> may not perform the single-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a double-click operation associated with the finger <NUM> according to the signal <NUM> in response to the number of the extreme values of the signal <NUM> being equal to two. Otherwise, the HMD <NUM> may not perform the double-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a double-click operation associated with the finger <NUM> according to the signal <NUM> in response to the number of the extreme values of the signal <NUM> being equal to two. Otherwise, the HMD <NUM> may not perform the double-click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may perform a click operation associated with the finger <NUM> according to signals measured by the IMU <NUM> and IMU <NUM> respectively, wherein the IMU <NUM> or IMU <NUM> may be an accelerometer or a gyroscope. For example, after the HMD <NUM> receiving signal <NUM> measured by the IMU <NUM> and signal <NUM> measured by the IMU <NUM>, the HMD <NUM> may detect the amplitude of the signal <NUM> and the amplitude of the signal <NUM>. The HMD <NUM> may perform a single-click operation if the amplitude of the signal <NUM> is greater than a threshold TA and the amplitude of the signal <NUM> is greater than a threshold TD. Otherwise, the HMD <NUM> may not perform the single-click operation. For another example, after the HMD <NUM> receiving signal <NUM> measured by the IMU <NUM> and signal <NUM> measured by the IMU <NUM>, the HMD <NUM> may detect the amplitude of the signal <NUM> and the amplitude of the signal <NUM>. The HMD <NUM> may perform a double-click operation if the amplitude of the signal <NUM> is greater than a threshold TA and the amplitude of the signal <NUM> is greater than a threshold TD. Otherwise, the HMD <NUM> may not perform the double-click operation.

In one embodiment, the HMD <NUM> may pre-store a machine learning (ML) model in the storage medium <NUM>. The HMD <NUM> may input the signal measured by the IMU <NUM> and/or the signal measured by the IMU <NUM> into the ML model to recognize a mouse operation (e.g., a click operation or a scroll operation) of the virtual mouse. The HMD <NUM> may update the XR scene according to the recognized mouse operation.

<FIG> illustrates a schematic diagram of performing a click operation by using multiple remote controllers according to one embodiment of the present invention. When the virtual mouse is enabled, the user may perform a click operation in the XR scene by using multiple remote controllers. It is assumed that the remote controller <NUM> is worn on the finger <NUM> (e.g., forefinger) of the hand <NUM> and the remote controller <NUM> is worn on the finger <NUM> (e.g., middle finger) of the hand <NUM>. If the finger <NUM> or finger <NUM> makes a click action, the HMD <NUM> receives signals from the remote controller <NUM> (i.e., signals measured by IMU <NUM> and/or IMU <NUM>) and signals from the remote controller <NUM> (i.e., signals measured by IMU <NUM> and/or IMU <NUM>). The HMD <NUM> performs a click operation of the virtual mouse in the XR scene according to the received signals.

In one embodiment, the HMD <NUM> may perform a click operation associated with the finger <NUM> (i.e., a left click). For example, after the HMD <NUM> receiving signal <NUM> measured by the remote controller <NUM> and signal <NUM> measured by the remote controller <NUM>, the HMD <NUM> may detect the amplitudes of the signal <NUM> and signal <NUM>. The HMD <NUM> may perform the click operation associated with the finger <NUM> if the amplitude of the signal <NUM> is greater than a threshold TA and the amplitude of the signal <NUM> is less than or equal to a threshold TB. Otherwise, the HMD <NUM> may not perform the click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may perform a click operation associated with the finger <NUM> (i.e., a right click). For example, after the HMD <NUM> receiving signal <NUM> measured by the remote controller <NUM> and signal <NUM> measured by the remote controller <NUM>, the HMD <NUM> may detect the amplitudes of the signal <NUM> and signal <NUM>. The HMD <NUM> may perform the click operation associated with the finger <NUM> if the amplitude of the signal <NUM> is less than or equal to a threshold TC and the amplitude of the signal <NUM> is greater than a threshold TD, wherein the threshold TC and the threshold TD may be less than or equal to the threshold TA and the threshold TB respectively. Otherwise, the HMD <NUM> may not perform the click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may perform a single-click operation or a double-click operation according to the number of the extreme values of the signal measured by the remote controller <NUM> and the number of the extreme values of the signal measured by the remote controller <NUM>. For example, the HMD <NUM> may perform a single-click operation associated with the finger <NUM> in response to the number of the extreme values of the signal <NUM> being equal to one and the number of the extreme values of the signal <NUM> being equal to one. Otherwise, the HMD <NUM> may not perform the single-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a single-click operation associated with the finger <NUM> in response to the number of the extreme values of the signal <NUM> being equal to one and the number of the extreme values of the signal <NUM> being equal to one. Otherwise, the HMD <NUM> may not perform the single-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a double-click operation associated with the finger <NUM> in response to the number of the extreme values of the signal <NUM> being equal to two and the number of the extreme values of the signal <NUM> being equal to two. Otherwise, the HMD <NUM> may not perform the double-click operation associated with the finger <NUM>. For another example, the HMD <NUM> may perform a double-click operation associated with the finger <NUM> in response to the number of the extreme values of the signal <NUM> being equal to two and the number of the extreme values of the signal <NUM> being equal to two. Otherwise, the HMD <NUM> may not perform the double-click operation associated with the finger <NUM>.

In one embodiment, the HMD <NUM> may pre-store a ML model in the storage medium <NUM>. The HMD <NUM> may input the signal measured by the remote controller <NUM> and/or the signal measured by the remote controller <NUM> into the ML model to recognize a mouse operation (e.g., a click operation or a scroll operation) of the virtual mouse. The HMD <NUM> may update the XR scene according to the recognized mouse operation.

<FIG> illustrates a schematic diagram of performing a scroll operation by using a remote controller according to one embodiment of the present invention. When the virtual mouse is enabled, the user may perform a scroll operation in the XR scene by using a remote controller. It is assumed that the remote controller <NUM> is worn on the finger <NUM> of the hand <NUM>. If the finger <NUM> makes a scroll action, the HMD <NUM> receives signals from the IMU <NUM> and/or IMU <NUM>, and the HMD <NUM> performs a scroll operation of the virtual mouse in the XR scene according to the received signals.

In one embodiment, the HMD <NUM> detects a wave peak and a wave valley of the signal measured by the IMU <NUM>. If a time point of the occurring of the wave peak (the first wave peak of the signal) is later than a time point of the occurring of the wave valley (the first wave valley of the signal), the HMD <NUM> performs a first scroll operation of the virtual mouse in the XR scene. If the time point of the occurring of the wave peak is earlier than the time point of the occurring of the wave valley, the HMD <NUM> performs a second scroll operation of the virtual mouse in the XR scene, wherein the second scroll operation is different from the first scroll operation. For example, the HMD <NUM> may perform a scroll up operation in the XR scene in response to the time point of the wave peak of the signal <NUM> being later than the time point of the wave valley of the signal <NUM>. Otherwise, the HMD <NUM> may not perform the scroll up operation. In another example, the HMD <NUM> may perform a scroll down operation in the XR scene in response to the time point of the wave peak of the signal <NUM> being earlier than the time point of the wave valley of the signal <NUM>. Otherwise, the HMD <NUM> may not perform the scroll down operation.

In one embodiment, the HMD <NUM> detects wave peaks and wave valleys of the signals measured by the IMU <NUM> and IMU <NUM>. If a time point of the occurring of the wave peak (e.g., the first wave peak of the signal) of the signal measured by the IMU <NUM> is later than a time point of the occurring of the wave valley (e.g., the first wave valley of the signal) of the signal measured by the IMU <NUM>, and a time point of the occurring of the wave peak (e.g., the first wave peak of the signal) of the signal measured by the IMU <NUM> is later than a time point of the occurring of the wave valley (e.g., the first wave valley of the signal) of the signal measured by the IMU <NUM>, the HMD <NUM> performs a first scroll operation of the mouse function in the XR scene. If a time point of the occurring of the wave peak (e.g., the first wave peak of the signal) of the signal measured by the IMU <NUM> is earlier than a time point of the occurring of the wave valley (e.g., the first wave valley of the signal) of the signal measured by the IMU <NUM>, and a time point of the occurring of the wave peak (e.g., the first wave peak of the signal) of the signal measured by the IMU <NUM> is earlier than a time point of the occurring of the wave valley (e.g., the first wave valley of the signal) of the signal measured by the IMU <NUM>, the HMD <NUM> performs a second scroll operation of the moues function in the XR scene, wherein the second scroll operation is different from the first scroll operation.

For example, the HMD <NUM> may perform a scroll up operation in the XR scene in response to the time point of the wave peak of the signal <NUM> being later than the time point of the wave valley of the signal <NUM> and the time point of the wave peak of the signal <NUM> being later than the time point of the wave valley of the signal <NUM>. Otherwise, the HMD <NUM> may not perform the scroll up operation. For another example, the HMD <NUM> may perform a scroll down operation in the XR scene in response to the time point of the wave peak of the signal <NUM> being earlier than the time point of the wave valley of the signal <NUM> and the time point of the wave peak of the signal <NUM> being earlier than the time point of the wave valley of the signal <NUM>. Otherwise, the HMD <NUM> may not perform the scroll down operation.

<FIG> illustrates a schematic diagram of a remote controller <NUM> with an optical sensor <NUM> according to one embodiment of the present invention. It is assumed that the remote controller <NUM> is worn on finger <NUM> (e.g., forefinger) of the hand <NUM>. The optical sensor <NUM> may have a detection area <NUM> directed to finger <NUM> (e.g., thumb) of the hand <NUM>, and the optical sensor <NUM> may generate a signal according to the detection area <NUM>. The remote controller <NUM> may transmit the signal generated by the optical sensor <NUM> to the HMD <NUM>.

In one embodiment, the HMD <NUM> may determine whether an object (e.g., finger <NUM>) has moved toward the optical sensor <NUM> according to the signal generated by the optical sensor <NUM>. For example, if the optical sensor <NUM> has been touched or clicked by the finger <NUM>, the HMD <NUM> may determine that an object has moved toward the optical sensor <NUM> according to the signal. Accordingly, the HMD <NUM> may perform a click operation of the virtual mouse in the XR scene. Otherwise, the HMD <NUM> may not perform the click operation of the virtual mouse in the XR scene.

In one embodiment, the HMD <NUM> may determine whether an object (e.g., finger <NUM>) has moved across the detection area <NUM> according to the signal generated by the optical sensor <NUM>. If the object has moved across the detection area <NUM> in a specific direction, the HMD <NUM> may perform a scroll operation of the virtual mouse in the XR scene according to the specific direction. Specifically, the HMD <NUM> may perform a first scroll operation of the virtual mouse in the XR scene in response to the object having moved across the detection area <NUM> in a first direction, and the HMD <NUM> may perform a second scroll operation of the virtual mouse in the XR scene in response to the object having moved across the detection area <NUM> in a second direction opposite to the first direction, wherein the first scroll operation (e.g., a scroll up operation) is different from the second scroll operation (e.g., a scroll down operation).

In one embodiment, the HMD <NUM> may pre-store a ML model in the storage medium <NUM>. The HMD <NUM> may input the signal measured by the optical sensor <NUM> and/or signal measured by the optical sensor <NUM> into the ML model to recognize a mouse operation (e.g., a click operation or a scroll operation) of the virtual mouse. The HMD <NUM> may update the XR scene according to the recognized mouse operation.

<FIG> illustrates a flowchart of a method for remotely controlling extended reality by a virtual mouse according to one embodiment of the present invention, wherein the method may be implemented by the system <NUM> as shown in <FIG>. In step S101, capturing an image by the head mounted display. In step S102, detecting the image to obtain first data of a hand in the image and second data of a plane in the image by the head mounted display. In step S103, enabling the virtual mouse according to the first data and the second data by the head mounted display.

In summary, the HMD of the present invention may detect the gesture of the user to determine whether the user wants to enable a virtual mouse function for interacting with the XR scene. If the HMD detects a flexion of the user's hand, the HMD may determine that the user wants to enable the virtual mouse function. After the virtual mouse function being enabled, the user's hand wearing one or more ring-type remote controllers can be simulated as a mouse. The HMD receives the signal transmitted from the remote controller and determines whether a click operation or a scroll operation corresponding to the virtual mouse is performed by the user, and thus the HMD interacts with the XR scene according to the signal. Accordingly, the present invention provides a convenient and intuitive way for the user of an HMD to interact with the XR environment.

Claim 1:
A system (<NUM>) for remotely controlling extended reality by a virtual mouse, comprising:
a head mounted display (<NUM>), comprising an image capture device (<NUM>); and
a first remote controller (<NUM>) for being worn on a hand (<NUM>), wherein the first remote controller (<NUM>) communicatively connects to the head mounted display (<NUM>), wherein the first remote controller (<NUM>) comprises a first inertial measurement unit (<NUM>),
wherein the head mounted display (<NUM>) is configured to:
capture (S101, S301) an image through the image capture device (<NUM>);
detect (S102) the image to obtain first data of the hand (<NUM>) in the image and second data of a plane (<NUM>) in the image;
enable (S103) the virtual mouse according to the first data and the second data;
receive (S303) a first signal (<NUM>-<NUM>; <NUM>, <NUM>, <NUM>, <NUM>, <NUM>; <NUM>, <NUM>) generated by the first inertial measurement unit (<NUM>);
perform a click operation by the virtual mouse in an extended reality scene according to the first signal (<NUM>-<NUM>; <NUM>, <NUM>, <NUM>, <NUM>, <NUM>; <NUM>, <NUM>);
detect a first occurring wave peak and a first occurring wave valley of the first signal (<NUM>-<NUM>; <NUM>, <NUM>, <NUM>, <NUM>, <NUM>; <NUM>, <NUM>);
in response to a first time point of the first wave peak being later than a second time point of the first wave valley, perform a first scroll operation by the virtual mouse in an extended reality scene; and
in response to the first time point of the first wave peak being earlier than the second time point of the first wave valley, perform a second scroll operation by the virtual mouse in the extended reality scene.