Patent Publication Number: US-2023135658-A1

Title: Virtual image display system and pointing direction control method of control device thereof

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims the priority benefit of U.S. provisional application Ser. No. 63/274,943, filed on Nov. 3, 2021. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification. 
    
    
     BACKGROUND 
     Technical Field 
     The disclosure relates to a virtual image display system and a control method, particularly relates to a virtual image display system and a pointing direction control method of a control device thereof. 
     Description of Related Art 
     In the conventional virtual image display system, a cursor in a virtual image display frame may be controlled by a handheld control device. The handheld control device transmits the sensed inertial measurement information to a virtual image display device, such that the virtual image display device may estimate the position of the handheld control device according to the sensed inertial measurement information and control the pointing direction of the displayed cursor. 
     A way for the handheld control device to sense the inertial measurement information is by using the inertial measurement unit (IMU) to measure the displacement amount to control the cursor movement in the virtual image display frame according to the measured displacement amount of the handheld control device. However, the displacement amount measured by the existing IMU often accumulates error over time. When the error is accumulated to a certain extent, it becomes perceivable by a user, causing the user to suffer from error in the control operation of the existing virtual image display system and have poor user experience. 
     SUMMARY 
     The disclosure is directed to a pointing direction control method of a control device and a virtual image display system, which are adapted to reduce an error between a control cursor and a pointing direction of the control device. 
     The disclosure provides a pointing direction control method of a control device including: displaying an indicator pattern with a recognition pointing direction by the control device; capturing the indicator pattern displayed by the control device by a virtual image display device; and analyzing the recognition pointing direction of the indicator pattern by the virtual image display device and displaying a control cursor according to the recognition pointing direction. 
     The disclosure provides a virtual image display system including a control device and a virtual image display device. The control device is configured to display an indicator pattern with a recognition pointing direction. The virtual image display device is coupled to the control device. The virtual image display device is configured to capture the indicator pattern displayed by the control device, analyze the recognition pointing direction of the indicator pattern, and display a control cursor according to the recognition pointing direction. 
     Based on the above description, the virtual image display device of the disclosure may set a pointing direction of a control cursor displayed in a virtual image by recognizing the recognition pointing direction of the indicator pattern displayed by the control device. Therefore, the virtual image display device of the disclosure is adapted to reduce an error between the control cursor and the pointing direction of the control device, and effectively improve the user&#39;s sense of experience in virtual image experience. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a schematic diagram of a virtual image display system according to an embodiment of the disclosure. 
         FIG.  2    is a schematic diagram of a virtual image frame according to an embodiment of the disclosure. 
         FIG.  3    is a hardware block diagram of a virtual image display system according to an embodiment of the disclosure. 
         FIG.  4    is a flowchart of a pointing direction control method of a control device according to an embodiment of the disclosure. 
         FIG.  5    is a schematic diagram of a virtual image display system according to another embodiment of the disclosure. 
         FIG.  6    is a flowchart of a pointing direction control method of a control device according to another embodiment of the disclosure. 
     
    
    
     DESCRIPTION OF THE EMBODIMENTS 
     Referring to  FIG.  1   ,  FIG.  1    is a schematic diagram of a virtual image display system according to an embodiment of the disclosure. A virtual image display system  100  includes a control device  101  and a virtual image display device  102 , where the virtual image display device  102  is coupled to the control device  101 . The control device  101  is configured to display an indicator pattern IP, where the indicator pattern IP has a recognition pointing direction RPD. The control device  101  may be any handheld electronic device with a display function, such as a smart phone, a tablet computer, or a smart watch. In the embodiment of  FIG.  1   , the indicator pattern IP may be a pattern in the form of an arrow and point to the recognition pointing direction RPD. It should be noted that, in other embodiments, the indicator pattern IP may be a pattern of any shape that may be recognized to recognize the recognition pointing direction RPD. 
     The virtual image display device  102  is configured to capture the indicator pattern IP displayed by the control device  101 , and the virtual image display device  102  is further configured to analyze the indicator pattern IP to obtain the recognition pointing direction RPD. Furthermore, the virtual image display device  102  displays a control cursor CC according to the recognition pointing direction PRD. 
     Referring to  FIG.  1    and  FIG.  2   ,  FIG.  2    is a schematic diagram of a virtual image frame according to an embodiment of the disclosure. The virtual image display device  102  may be used to display a virtual image frame  200 . The virtual image frame  200  includes a display frame of virtual reality and the control cursor CC. The virtual image display device  102  may generate the control cursor CC in the display frame of virtual reality according to an extended projection of the recognition pointing direction PRD. In some embodiments, the virtual image frame  200  may further include a virtual control device frame  201 . A position of the virtual control device frame  201  may correspond to a position of the indicator pattern IP displayed by the control device  101  in reality, and the virtual control device frame  201  is directed toward the recognition pointing direction PRD. 
     In the embodiment, it should be noted that the virtual image display device  102  obtains the recognition pointing direction RPD by analyzing the indicator pattern IP of the control device  101 , rather than obtaining the pointing direction only according to an outline of the control device  101 . In this way, the control device  101  may be different handheld electronic devices manufactured by various manufacturers, and may have different sizes and shapes. In addition, the recognition pointing direction RPD has more directional information relative to the outline of the control device  101 . 
     Referring to  FIG.  3   ,  FIG.  3    is a hardware block diagram of a virtual image display system according to an embodiment of the disclosure. A virtual image display system  300  in  FIG.  3    includes a control device  301  and a virtual image display device  302 , and the virtual image display device  302  may be coupled to the control device  301  in a wired or wireless manner. The control device  301  has a display screen  303  for displaying the indicator pattern IP. The virtual image display device  302  has an image capturing device  304  for capturing the indicator pattern IP displayed on the display screen  303  of the control device  301 . In the embodiment, the image capturing device  304  is, for example, a camera or a video camera, which may be used to capture static or dynamic images. Any communication protocol well-known to those skilled in the art may be used for information transmission between the control device  301  and the virtual image display device  302 . In addition, the control device  301  further includes an inertial measuring device  305 . The inertial measuring device  305  is used to measure the inertial measurement information IMD. Furthermore, the control device  301  transmits the inertial measurement information IMD to the virtual image display device  302 . In other embodiments, the virtual image display device  302  may transmit a command signal CS to the control device  301  to control the display of the indicator pattern IP of the control device  301 . 
     Referring to  FIG.  4   ,  FIG.  4    is a flowchart of a pointing direction control method of a control device according to an embodiment of the disclosure. In step S 401 , the control device  101  displays the indicator pattern IP having the recognition pointing direction RPD. In step S 402 , the virtual image display device  102  captures the indicator pattern IP displayed by the control device  101 . In step S 403 , the virtual image display device  102  analyzes the recognition pointing direction RPD of the indicator pattern IP. The virtual image display device  102  displays the control cursor CC according to the recognition pointing direction RPD. 
     According to the above virtual image display system of  FIG.  1    and the pointing direction control method of the control device of  FIG.  4   , the virtual image display system and the control method of the disclosure may control the cursor according to the recognition pointing direction of the control device, and avoid poor control experience caused by errors accumulated by the inertial measuring device, which may be adapted to outline shapes of various control devices (handheld electronic devices), and may provide directional information to control the cursor according to the recognition pointing direction. 
     Referring to  FIG.  5   ,  FIG.  5    is a schematic diagram of a virtual image display system according to another embodiment of the disclosure. A virtual image display system  500  includes a control device  501  and a virtual image display device  502 . Basic operations of the virtual image display system  500  are similar to that of the virtual image display system  100  of  FIG.  1   , and details thereof are not repeated. Different from  FIG.  1   , in the embodiment, when the virtual image display device  502  captures the indicator pattern IP, if the virtual image display device  502  cannot effectively capture the indicator pattern IP, it means that the indicator pattern IP in the display screen of the control device  501  may be shielded at this moment. The virtual image display device  502  may send a command signal CS to the control device  501 . Correspondingly, the control device  501  may adjust a display position of the indicator pattern IP according to the command signal CS. 
     In  FIG.  5   , the indicator pattern IP is shielded by user&#39;s finger. At this moment, the virtual image display device  502  may send the command signal CS to the control device  501 , and the control device  501  may adjust an original display position of the indicator pattern IP to display a new indicator pattern IP′. In this way, the virtual image display device  502  may successfully capture the new indicator pattern IP′. It should be noted that the indicator pattern IP and the indicator pattern IP′ have the same pointing direction. 
     Referring to  FIG.  6    and  FIG.  3   ,  FIG.  6    is a flowchart of a pointing direction control method of a control device according to another embodiment of the disclosure. In step S 601 , the control device  301  may display the indicator pattern IP having the recognition pointing direction RPD on the display screen  303 . In step S 602 , the control device  301  transmits the inertial measurement information IMD to the virtual image display device  302 , and the virtual image display device  302  obtains virtual movement information according to the inertial measurement information IMD. The inertial measurement information IMD may be measured and generated by the inertial measuring device  305  in the control device  301 . The virtual image display device  302  may calculate the virtual movement information of the control device  301  according to the inertial measurement information IMD. 
     In step S 603 , the virtual image display device  302  calculates a virtual pointing direction of the control device  301  according to the above-mentioned virtual movement information. It should be noted that since the virtual pointing direction is calculated according to the inertial measurement information IMD, there may be an error between the virtual pointing direction and the actual pointing direction of the control device  301  as time is accumulated. 
     In step S 604 , the virtual image display device  302  determines whether the indicator pattern IP may be captured. If the determination result is yes, then step S 606  is performed; and if the determination result is negative, it means that the indicator pattern IP is probably shielded, and step S 605  is performed. In step S 605 , the virtual image display device  302  sends the command signal CS to the control device  301 , and the control device  301  changes a display position of the displayed indicator pattern IP according to the command signal CS. 
     In step S 606 , the virtual image display device  302  may analyze the recognition pointing direction RPD of the indicator pattern IP, and calculate direction shift information between the virtual pointing direction and the recognition pointing direction RPD; and the virtual image display device  302  adjusts the virtual pointing direction according to the direction shift information. In step S 606 , since the virtual image display device  302  may obtain an actual pointing direction of the control device  301  through the recognition pointing direction RPD, the error between the virtual pointing direction and the actual pointing direction of the control device  301  may be effectively reduced. 
     In step S 607 , the virtual image display device  302  may adjust the displayed control cursor CC according to the adjusted virtual pointing direction, and may generate the control cursor CC in consistence with the actual pointing direction of the control device  301 . 
     In summary, the pointing direction control method of the control device and the virtual image display system of the disclosure may adjust the position of the control cursor by recognizing the recognition pointing direction of the indicator pattern displayed by the control device, which may reduce the display error of the control cursor and achieve good control experience.