Patent ID: 12223715

To make the above features and advantages of the application more comprehensible, several embodiments accompanied with drawings are described in detail as follows.

DESCRIPTION OF THE EMBODIMENTS

Some embodiments of the disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the application are shown. Indeed, various embodiments of the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set for the herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements.

FIG.1is a block diagram of a display system for accessory pairing according to an embodiment of the application. Referring toFIG.1, a display system100of the present embodiment includes a computing device110, a head-mounted display (HMD)120, an image capturing device130and an accessory140, and functions thereof are respectively described below.

The computing device110may be, for example, a server (e.g. a file server, a database server, or an application server), a workstation, a personal computer or a laptop with computing capability and includes a processor. The processor is, for example, a central processing unit (CPU), a graphics processing unit (GPU) or any other programmable microprocessor for general or special use, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or other similar devices, or a combination of these devices.

The HMD120is, for example, a liquid-crystal display (LCD), a light-emitting diode (LED) display or other suitable type of display, which is manufactured in a helmet or an eyeglass form for a user to wear on the head. In the present embodiment, the HMD120is coupled to the computing device110. In an embodiment, the display system100may further include at least one locator (not illustrated) which locates a position of the user wearing the HMD120within the environment and track the moving position of the user. The locator includes, for example, a laser or an infrared (IR) emitter and receiver, which is adopted to detect a distance of an object in the 3D space, and in collaboration with a plurality of calibrators configured on the object (e.g. the HMD120), a position of the object in the 3D space can be located. However, the display system100may use other positioning technology to locate the HMD120and other objects in the environment.

It should be mentioned that in an embodiment, the computing device110is disposed separately from the HMD120, and coupled to the HMD120in a wired or a wireless manner for data transmission. In another embodiment, the computing device110can also be integrated in the HMD120, and is connected with the HMD120through a transmission line for data transmission, which is not limited by the application.

The image capturing device130is coupled to the computing device110. The image capturing device130is, for example, a camera equipped with a charge coupled device (CCD) lens, a complementary metal-oxide semiconductor (CMOS) lens, an infrared lens, or other kinds of lens. Alternatively, the image capturing device130may be a camera with a depth information function such as a depth camera or a stereoscopic camera, which is not limited by the application. It should be mentioned that in an embodiment, the image capturing device130may be disposed separately from the HMD120or integrated in the HMD120, which is not limited by the application.

The display system100is compatible with several different accessories. Each of the accessories may have different functions and generate different input and output data related to the display system100. For example, the accessory140would be an input device such as a handheld controller or a wrist controller equipped with a motion sensor and an infrared (IR) light source (e.g. infrared LED), a mouse, a joystick, a trackball, a touch pad, and/or button that permits the user to interact with environment, objects, or avatars in the virtual world. In another embodiment, the accessory140would be a locator that can detect an object in a three-dimensional (3D) space and locate a position of the object in the 3D space with the use of calibrators disposed on the object (e.g., the HMD120). Detail steps of the method for accessory paring of the application are described below.

FIG.2is a flowchart illustrating a method for accessory pairing according to an embodiment of the application. Referring toFIG.2, the method of the present embodiment is adapted to the display system100ofFIG.1, detailed steps of the method for accessory pairing of the application are described below with reference of various components of the display system100.

First, the computing device110receives an identifying information from the accessory140. The computing device110determines a type of the accessory140based on the identifying information of the accessory140(step S202). In an embodiment, the computing device110obtains an accessory type identification from the identifying information of each accessory. The accessory type identifying information can indicate functions of corresponding accessory, so that the accessory with different functions will have different accessory type identification. The computer device110can determine the type of the accessory based on the accessory type information. In an embodiment, the identifying information can be transmitted in a standard packet format of Bluetooth protocol or Bluetooth low energy (BLE) protocol from the accessory140to the computing device110, which is not limited by the application.

Back to the flowchart inFIG.2, the computing device110receives an image. The computing device110recognizes a type of a user body part according to the image (step S204). The image capturing device130can capture images of the user's surroundings. In an embodiment, since the image capturing device130can be integrated in the HMD120worn by the user, the image capturing device130captures the images of a field of view (FOV) of the user. In another embodiment, the image capturing device130may capture the images of the user if the image capturing device130is disposed separately from the HMD120. The captured images can be transmitted from the image capturing device130to the computing device110.

The user body part may be a hand, a leg, a head or other body part of the user body. In an embodiment, the computing device110can detect and/or track a user body part image corresponding to the user body part in the received images. Specifically, the computing device110reads continuous image in an image stream, and detects and tracks a candidate area in the image corresponding to the user body part. The computing device110can perform image recognition on the image to define the candidate area corresponding to the user body part in the image and recognize the type of the user body part in the image. The image recognition can be performed by trained deep learning models, neural network models or similar artificial intelligence models. For example, the deep learning model is implemented by a learning network such as a convolution neural network (CNN), which is not limited by the application.

For example,FIG.3illustrates an example of a user wearing a head-mounted display and a controller according to an embodiment of the application.FIG.4illustrates an example of detecting a user body part in an image according to an embodiment of the application. Referring toFIG.3, a user32in a real-world environment wears an HMD34on the head and a wrist controller38on the right hand. The user32wears the HMD34having an image capturing device36with a field of view within boundaries311-314. In this embodiment, when a connection is established between the wrist controller38and the computing device110of the display system, the computing device110can collect an identifying information from the wrist controller38and determines a type of the wrist controller38based on the identifying information. In this case, the computing device110can determine the type “wrist controller” corresponding to the wrist controller38according to the identifying information of the wrist controller38.

In this embodiment, a scene320includes the wrist controller38and a right hand of the user32, and the image capturing device36can capture the image corresponding to the scene320. As shown inFIG.4, an image VR is used as an example for detailed description. The image VR is the image which can be captured by the image capturing device36, and the image VR can be displayed to the user32as a virtual reality image. In this embodiment, the image VR includes a hand image40and a wrist controller image50. After detecting the hand image40of the hand in the image VR, the computing device110may define a candidate area corresponding to the hand image40and define a reference point to indicate a location of the user body part, such as the right hand of the user32, based on the candidate area. The reference point can be a center point of the candidate area, a center point of an upper boundary of the candidate area or any point in the candidate area, which is not limited by the application. As shown inFIG.4, the computing device110may define the bounding box as the candidate area, and the size of the candidate area is at least large enough to enclose the user body part corresponding to the candidate area. The computing device110can set the candidate area41to be associated with the hand image40, and sets the center point of the candidate area41as the reference point P1to indicate the location of the right hand of the user32.

In addition, the computing device110would recognize the type of the hand image40, and would sets the type to be associated with the candidate area41. For example, in this embodiment, the computing device110may recognize the hand image40corresponding to the candidate area41and obtain the type “right hand” corresponding to the hand image40.

Back to the flowchart inFIG.2, the computing device110determines whether the accessory is a candidate device which matches the user body part in the image based on the type of the accessory and the type of the user body part. Concretely, the computing device110determines whether the accessory is applicable to the user body part according to the type of accessory and the type of the user body part (step S206). When the accessory is not applicable to the user body part, the procedure would return to step S204such that the computing device110would continue to receive the image from the image capturing device130. In detail, a plurality of types of the accessory and the type of the user body part corresponding to each type of the accessory are stored in a memory (not illustrated) in the display system100. The computing device110can determine whether the accessory is the candidate device through the corresponding relation between the type of the accessory and the type of the user body part.

For example, the computing device110may establish a look-up table, and a plurality of types of the accessory and the corresponding types of the user body part are recorded in the look-up table. The look-up table can be stored in the memory in the display system100. Table 1 below is an example of the look-up table in an exemplary embodiment, which is used to record the corresponding relation between the type of the accessory and the type of the user body part.

TABLE 1accessoryuser body partwrist controllerleft handright handhandheld controllerleft handright hand

On the other hand, the accessory is determined to be the candidate device when the accessory is applicable to the user body part. Concretely, in response to the accessory is applicable to the user body part according to the type of accessory and the type of the user body part, the computing device110obtains a first position of the accessory140within an environment and obtains a second position of the user body part within the environment (step S208). For example, according toFIG.3,FIG.4and Table 1, the computing device110can input the type “wrist controller” corresponding to the wrist controller38and the type “right hand” corresponding to the hand image40to Table 1, and determine whether the wrist controller38is the candidate device which matches the hand image40. In Table 1, the type “wrist controller” is recorded to be related to the type “right hand”. Therefore, the computing device110would determine the wrist controller38and the hand image40have the corresponding relation, and set the wrist controller38to be the candidate device.

In one embodiment, the first position of the accessory140may include a location of the accessory140within the environment. For example, the location of the accessory140within the environment may be obtained by the computing device110based on a spatial relationship between the image capturing device130and the accessory140. The image capturing device130may detect the infrared light emitted from the accessory140, and the computing device110can calculate the location of the accessory140within the environment based on the infrared light obtained by the image capturing device130, the location of the image capturing device130, extrinsic parameters of the image capturing device130, intrinsic parameters of the image capturing device130and/or other information. The location of the accessory may be a relative position related to a specific device (e.g. the image capturing device130or the HMD120) or an absolute position corresponding to a coordinate system in the 3D space.

In another embodiment, the first position of the accessory140may further include an orientation of the accessory. For example, the orientation of the accessory140within the environment may be obtained by using an accelerometer. The location of the accessory140within the environment and the orientation of the accessory140may further be obtained based on information received by one or more additional sensor components (not illustrated), such as other positional sensors (e.g. motion sensors, GPS sensors, etc.) and/or orientation sensors (e.g. gyroscope, etc.) which can track the position and/or the orientation of the accessory140.

In one embodiment, the second position of the user body part may include a location of the user body part within the environment and/or an orientation of the user body part. For example, the location of the user body part within the environment may be obtain by the computing device110based on a spatial relationship between the HMD120and the user body part. As mentioned above, the computing device110can perform image recognition on the image captured by the image capturing device130, so as to define the reference point of the user body part in the image. In this embodiment, the computing device110can further convert the coordinates of the reference point in the image into the location within the environment.

In addition, the computing device110can perform image recognition on the image captured by the image capturing device130, so as to determine the orientation of the user body part. It should be mentioned that in an embodiment, the location and the orientation may be a relative position related to a specific device (e.g. the HMD120) or an absolute position corresponding to a coordinate system in the 3D space.

In another embodiment, the second position of the user body part within the environment may further be obtained based on skeleton information or other information received by one or more additional sensor components, which is not limited by the application. In detail, the computing device110can perform skeleton detection or other user body part recognition method on the received image to detect the skeleton information of the user, and define the candidate area of the user body part in the image based on the skeleton information. Then, the computing device110can further convert the coordinates of the candidate area in the image into the location of the user body part within the environment. Also, the computing device110can calculate the orientation of the user body part based on the skeleton information and inverse kinematics (IK).

Finally, the computing device110determines whether a relative position relation between the first position and the second position conforms a preset condition (step S210). In response to the relative position relation determined to be not conforming the preset condition, the procedure would return to step S204such that the computing device110would continue to receive the image from the image capturing device130.

On the other hand, in response to the relative position relation determined to be conforming the preset condition, the computing device110configures the accessory140to pair the user body part according to the first position of the accessory and the second position of the user body part (step S212). Here, the preset condition may indicate, for example, relative location relation and/or relative orientation relation between the accessory140and the user body part. However, the condition to be conformed by the relative position relation is not limited thereto. In detail, the computing device110may calculate a distance between the location of the accessory140and the location of the user body part to determine the relative location relation, and determine whether the relative location relation conforms the preset condition. However, the preset condition may be a distance or a distance range between the accessory140and the user body part.

In an embodiment, assume that the preset condition includes “the location of the wrist controller is 10 cm behind the right hand” and/or “the orientation of the wrist controller is the same as the orientation of the right hand”. Referring toFIG.3andFIG.4, the reference point P1represents the location of the hand image40corresponding to the right hand of the user32and the reference point P2represents the location of the wrist controller38. The computing device110would calculate a distance between the reference point P1and reference point P2to determine the relative location relation between the reference point P1and reference point P2, and determine whether the relative location relation conforms the preset condition. As shown inFIGS.3and4, the distance between the reference point P1and the reference point P2is distance d1. Assume the distance d1 is 10 cm. According, the relative location relation can be determined to be conforming the preset condition in this embodiment, the computing device110can pair the wrist controller38to the right hand of the user32.

In another embodiment, the relative orientation is further considered. The computing device110can calculate a relative angle between the orientation of the accessory and the orientation of the user body part between the accessory and the user body part. Concretely, the computing device110calculates a relative angle between the orientation of the accessory and the orientation of the user to determine the relative orientation between the accessory and the user body part, and determines whether the relative orientation conforms the preset condition. However, the preset condition may be a relative angle or a relative angle range between the accessory140and the user body part.

As shown inFIGS.3and4, the computing device110can calculate a relative angle to determine the relative orientation between the hand image40corresponding to the right hand of the user32and the wrist controller38, and determine whether the right hand of the user32and the wrist controller38have the same orientation. Assuming both orientation of the wrist controller38and the right hand of the user32are in a first direction D1, the computing device110can pair the wrist controller38to the right hand of the user32.

The disclosure is further directed to a non-transitory computer-readable recording medium configured to record a program which is loaded by a processor in a computing device110to execute each step of the method for accessory pairing. The computer program is composed of a plurality of program sections (for example, building an organization diagram program section, approving a list program section, setting a program section, and deploying a program section, etc.). After the program sections are loaded in the computing device110and executed, the steps of the method for accessory pairing can be completed.

In view of the aforementioned descriptions, the proposed method, the system and the computer-readable recording medium for accessory pairing allow pairing the accessory and the user body part based on the type and the position of the accessory and the user body part. Based on identification results of the image tracking or the skeleton tracking corresponding to the position where the accessory is worn and positioning information of the accessory, it is inferred which body part the accessory is worn on. Then, the accessory can be paired to the body part where it is worn on. Thus, it does not require the user to set the pairing body part associated to each accessory, so as to improve the convenience and the flexibility of accessory setting.

It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the disclosure covers modifications and variations provided that they fall within the scope of the following claims and their equivalents.