LOCALIZATION METHOD, WEARABLE DEVICE AND NON-TRANSITORY COMPUTER READABLE STORAGE MEDIUM

The present disclosure provides localization method and wearable device. The localization method is applicable to the wearable device, and includes: obtaining an environment information related to an environment where the wearable device is; determining a target map area in a map of the environment according to the environment information; and locating the wearable device in the map of the environment according to the target map area.

BACKGROUND

Field of Invention

This disclosure relates to a method and device, and in particular to a localization method and wearable device.

Description of Related Art

In the applications of virtual reality (VR), augmented reality (AR) and/or mixed reality (MR), some of operating environments of the head-mounted device, such as department store, are complicated. When the head-mounted device performs object detection to recognize those operating environments, the object detection might fail, which does not facilitate the localization of the head-mounted device. Some related arts further set up specific areas in those operating environments by using fences or walls or by simplifying some areas, and limit the head-mounted device to be operated in the specific areas. However, the related arts which set up the specific areas in those operating environments are not convenience for the applications.

SUMMARY

An aspect of present disclosure relates to a localization method applicable to a wearable device. The localization method includes: obtaining an environment information related to an environment where the wearable device is; determining a target map area in a map of the environment according to the environment information; and locating the wearable device in the map of the environment according to the target map area.

Another aspect of present disclosure relates to a wearable device. The wearable device includes an environment information collector and a processor. The environment information collector is configured to collect an environment information related to an environment where the wearable device is. The processor is coupled to the environment information collector, is configured to determine a target map area in a map of the environment according to the environment information, and is configured to locate the wearable device in the map of the environment according to the target map area.

Another aspect of present disclosure relates to a non-transitory computer readable storage medium with a computer program to execute a localization method applicable to a wearable device. The localization method includes: obtaining an environment information related to an environment where the wearable device is; determining a target map area in a map of the environment according to the environment information; and locating the wearable device in the map of the environment according to the target map area.

DETAILED DESCRIPTION

The embodiments are described in detail below with reference to the appended drawings to better understand the aspects of the present application. However, the provided embodiments are not intended to limit the scope of the disclosure, and the description of the structural operation is not intended to limit the order in which they are performed. Any device that has been recombined by components and produces an equivalent function is within the scope covered by the disclosure.

As used herein, “coupled” and “connected” may be used to indicate that two or more elements physical or electrical contact with each other directly or indirectly, and may also be used to indicate that two or more elements cooperate or interact with each other.

Referring toFIG.1,FIG.1is a block diagram of a wearable device100in accordance with some embodiments of the present disclosure. In some embodiments, the wearable device100is a head-mounted device (HMD). Thus, the wearable device100can be mounted on head of a user in a physical environment such as a public place, a gaming place, a workplace, etc. The wearable device100mounted on the head of the user can provide an immersive content CI for the user.

In some embodiments, the wearable device100might occlude the direct visibility of the user to the physical environment, and the immersive content CI can correspondingly be a virtual reality (VR) environment, or a mixed reality (MR) environment. In particular, the MR environment simulates the physical environment and enables an interaction of virtual reality objects (which cannot be directly seen in the physical environment by the user) with the simulated physical environment. However, the present disclosure is not limited herein. For example, the immersive content CI can be the simulated physical environment without the virtual reality objects, which is also known as a pass-through view.

In some embodiments, the wearable device100does not occlude the direct visibility of the user to the physical environment, and the immersive content CI can correspondingly be an augmented reality (AR) environment. In particular, the AR environment augments the physical environment directly seen by the user with the virtual reality objects.

As should be understood, the user might control the virtual reality objects in the immersive content CI by operating at least one controller (not shown) or by making hand movements. In particular, the at least one controller is wirelessly connected to the wearable device100, and the hand movements would be sensed and recognized by the wearable device100.

In some embodiments, as shown inFIG.1, the wearable device100includes an environment information collector11, a processor13and a display panel15. The processor13is electrically coupled to the environment information collector11and the display panel15.

In accordance with the embodiments that the user operates the wearable device100in the physical environment, the environment information collector11is configured to collect environment information Imfe related to the physical environment where the wearable device100is. In some practical applications, the physical environment is a tall building such as a department store. The environment information Imfe related to the tall building might correspondingly include multiple environment parameters such as levels of the tall building, current level, floor-to-ceiling height of each level, area of each level, equipment and/or facility (e.g., toilet, information center, elevator) in each level, specific regions (e.g., restricted area, food court, parking lot) in the tall building, etc. It should be understood that the environment information Imfe can be pre-generated by management personnel of the physical environment. In addition, the environment information Imfe can be arranged in the physical environment in various manners or can be pre-provided to the wearable device100, which would be described in detail later.

In the embodiments ofFIG.1, the environment information collector11includes an image capturing circuit111, a storage circuit112and a wireless communication circuit113. The image capturing circuit111, the storage circuit112and the wireless communication circuit113are electrically coupled to the processor13.

In some embodiments, the image capturing circuit111is configured to capture images in the physical environment. In particular, the image capturing circuit111can be implemented by at least one camera device or other structure including multiple components such as lens, image sensor, image processor, etc.

In some embodiments, the storage circuit112is configured to store signals, data and/or information required by the operation of the wearable device100. In particular, the storage circuit112can be implemented by volatile memory, non-volatile memory, or the both.

In some embodiments, the wireless communication circuit113is configured to be communicatively coupled to other communication devices (not shown). In particular, the wireless communication circuit113can be communication device implemented by Bluetooth low energy (BLE), wireless local area network (e.g., Wi-Fi), mobile communication (e.g., 4G, 5G, etc.) or other suitable technologies. It can be seen that the wireless communication circuit113can wirelessly communicate with the other communication devices with or without a network provided by internet service provider (ISP).

In some embodiments, by utilizing one or more technologies such as Simultaneous Localization and Mapping (SLAM), the processor13can generate a map M1of the physical environment and calculate location of the wearable device100in the map M1. In the case that the physical environment is a tall building, the map M1might have a great amount of data, which might cause processing resources (e.g., memory usage) of the processor13being significantly occupied during calculation of the location of the wearable device100. In addition, because the map M1having the great amount of data might show a noticeable deviation from the tall building, the location of the wearable device100in the map M1being calculated with the map M1having the great amount of data might have poor precision. In view of these, in some further embodiments, the processor13uses the environment information Imfe collected by the environment information collector11to determine a target map area (which would be described later) in the map M1, and calculates the location of the wearable device100in the map M1with the target map area. In particular, the processor13can be implemented by central processing unit (CPU), graphic processing unit (GPU), application-specific integrated circuit (ASIC), microprocessor, system on a Chip (SoC) or other suitable processing circuits.

In some embodiments, the display panel15is configured to display one or more images transmitted from the processor13. In such arrangements, when the display panel15displays the images, the user wearing the wearable device100can view and perceive the immersive content CI by looking at visual contents on screens of the display panel15. In particular, the display panel15can be implemented by an active-matrix organic light-emitting diode (AMOLED) display, or the likes.

The operation of the wearable device100would be described in detail with reference toFIG.2. Referring toFIG.2,FIG.2is a flow diagram of a localization method200in accordance with some embodiments of the present disclosure. The localization method200can be performed by the wearable device100ofFIG.1, but the present disclosure should not be limited herein. In some embodiments, as shown inFIG.2, the localization method200includes operations S201-S203.

In operation S201, the wearable device100obtains the environment information Imfe related to an environment (i.e., the physical environment) where the wearable device100is. The operation S201would be described in detail below with reference toFIG.3.

Referring toFIG.3,FIG.3is a schematic diagram of the environment E1in accordance with some embodiments of the present disclosure. In some embodiments, as shown inFIG.3, the environment E1includes an environment area F1and an environment area F2. For example, the environment area F1is a first floor of the tall building, and the environment area F2is a second floor of the tall building. Moreover, multiple broadcast devices30are arranged in the environment E1. For example, as shown inFIG.3, one of the broadcast devices30is arranged on a stair in the environment area F1of the environment E1, and others of the broadcast devices30are arranged on multiple columns in the environment area F2of the environment E1.

In the embodiments ofFIG.3, an operator1(i.e. the user) of the wearable device100is in the vicinity of the broadcast device30arranged on the stair. In such case, the wearable device100uses the wireless communication circuit113to establish a wireless connection301with the broadcast device30arranged on the stair. In accordance with the above embodiments that the environment information Imfe is arranged in the physical environment, the environment information Imfe is stored in each of the broadcast devices30. After the wireless connection301is established, the broadcast device30arranged on the stair transmits a push signal Sp carrying the environment information Imfe to the wearable device100through the wireless connection301. As shown inFIG.1, the wireless communication circuit113of the wearable device100receives the push signal Sp from the broadcast device30arranged on the stair, and transmits the push signal Sp to the processor13, so that the processor13obtains the environment information Imfe.

In the above embodiments, the environment information Imfe is stored in the broadcast device30, and is carried by the push signal Sp transmitted from the broadcast device30to the wearable device100. However, the present disclosure is not limited herein. For example, in some embodiments, the environment information Imfe is stored in a physical server (not shown) in the environment E1. In some further embodiments, the broadcast device30can download the environment information Imfe from the physical server via the network, so as to transmit the push signal Sp carrying the environment information Imfe. Moreover, in some further embodiments, after receiving the push signal Sp, the processor13can control the wireless communication circuit113to download the environment information Imfe from the physical server via the network.

In operation S202, the processor13of the wearable device100determines the target map area in the map M1of the environment E1according to the environment information Imfe. The operation S202would be described in detail below with reference toFIG.4.

Referring toFIG.4,FIG.4is a schematic diagram of the map M1of the environment E1inFIG.3in accordance with some embodiments of the present disclosure. In some embodiments, as shown inFIG.4, the map M1includes a map area A1and a map area A2. As can be seen fromFIGS.3and4, the map area A1is corresponding to the environment area F1, and the map area A2is corresponding to the environment area F2.

In accordance with the above embodiments that the processor13obtains the environment information Imfe by receiving the push signal Sp transmitted by the broadcast device30arranged on the stair, the environment information Imfe obtained by receiving the push signal Sp transmitted by the broadcast device30arranged on the stair indicates that the current level of the tall building is the first floor (i.e., the environment area F1of the environment E1). Accordingly, the processor13identifies the map area A1corresponding to the environment area F1in the map M1, and regards the map area A1as the target map area. As can be seen from the embodiments ofFIGS.3and4, the processor13is configured to identify one of the map areas (i.e., the map area A1, the map area A2) in the map M1based on the environment information Imfe, and is configured to regard the one of the map areas in the map M1as the target map area.

In operation S203, the processor13of the wearable device100locates the wearable device100in the map M1of the environment E1according to the target map area. In some embodiments, the processor13controls the image capturing circuit111to capture images (not shown) of the environment area F1, and uses SLAM technology to analyze both the images of the environment area F1and the map area A1(i.e., the target map area), so as to locate the wearable device100in the map M1. By SLAM technology, the processor13extracts multiple feature points (not shown) from the images of the environment area F1, matches the feature points to multiple map points in the map area A1, and calculates a point O1in the map M1according to a result of match. InFIG.4, the point O1represents the location of the wearable device100in map M1of the environment E1.

As can be seen from the above embodiments ofFIG.3, in some embodiments, the wearable device100uses the wireless communication circuit113to obtain the environment information Imfe. However, the way of obtaining the environment information Imfe (i.e., operation S201) is not limited thereto, which would be described in detail below with reference toFIG.5.

Referring toFIG.5,FIG.5is a schematic diagram of the environment E1in accordance with some embodiments of the present disclosure. In some embodiments, as shown inFIG.5, the environment E1includes an environment area F3, and the environment area F3includes an environment sub-area R1and an environment sub-area R2. For example, the environment area F3is one floor of the tall building, and the environment sub-area R1and the environment sub-area R2are two areas within that floor of the tall building. Moreover, multiple two-dimensional (2D) codes50are arranged in the environment E1. As shown inFIG.5, the environment sub-area R1is provided with a reflective object52(e.g., mirror), and the environment sub-area R2is provided with a counter54. One of the 2D codes50is arranged on a column beside the reflective object52in the environment area F3of the environment E1, and the other of the 2D codes50are arranged on the counter54in the environment area F3of the environment E1. In some embodiments, the environment sub-area R1and the environment sub-area R2can be referred as the environment area R1and the environment area R2.

In accordance with the above embodiments that the environment information Imfe is arranged in the physical environment, each of the 2D code50records the environment information Imfe. In particular, the 2D code50can be implemented by QR code, or the likes.

In the embodiments ofFIG.5, the operator1of the wearable device100is in the vicinity of the 2D code50arranged on the counter54. In such case, the wearable device100uses the image capturing circuit111to scan the 2D code50arranged on the counter54. For example, the image capturing circuit111captures at least one image including the 2D code50, and recognizes and cuts off the 2D code50from that image, so as to output an image Imgc of the 2D code50. It should be understood that the image Imgc of the 2D code50also records the environment information Imfe. As shown inFIG.1, the processor13receives the image Imgc of the 2D code50, and decodes the 2D code50to obtain the environment information Imfe. The descriptions of decoding the 2D code50is well known to person skilled in the art of the present disclosure, and therefore are omitted herein.

In accordance with the descriptions of operation S202, the processor13determines the target map area in the map M1of the environment E1according to the environment information Imfe obtained by decoding the 2D code50, which would be described in detail below with reference toFIG.6.

Referring toFIG.6,FIG.6is a schematic diagram of the map M1of the environment E1inFIG.5in accordance with some embodiments of the present disclosure. In some embodiments, as shown inFIG.6, the map M1includes a map area A3, and the map area A3includes a map sub-area a1and a map sub-area a2at least. As can be seen fromFIGS.5and6, the map area A3is corresponding to the environment area F3, the map sub-area a1is corresponding to the environment sub-area R1, and the map sub-area a2is corresponding to the environment sub-area R2. In some embodiments, the map sub-area a1and the map sub-area a2can be referred as the map area a1and the map area a2.

In accordance with the above embodiments that the processor13obtains the environment information Imfe by decoding the 2D code50, the environment information Imfe obtained by decoding the 2D code50indicates that the environment sub-area R1is provided with the reflective object52. Accordingly, the processor13identifies the map sub-area a1corresponding to the environment sub-area R1in the map M1, and regards the map area A3from which the map sub-area a1is excluded as the target map area. As can be seen from the embodiments ofFIGS.5and6, the processor13is configured to identify one of the map areas (i.e., the map sub-area a1) in the map M1based on the environment information Imfe, and is configured to regard others of the map areas in the map M1as the target map area. The descriptions that the processor13locates the wearable device100in the map M1ofFIG.6according to the target map area (i.e., the map area A3from which the map sub-area a1is excluded) are similar to those of operation S203, and therefore are omitted herein.

Furthermore, the way of obtaining the environment information Imfe is not limited to receiving the push signal Sp transmitted by the broadcast device30(i.e., the embodiments ofFIG.3) and decoding the 2D code50(i.e., the embodiments ofFIG.5). For example, in some embodiments, the management personnel of the environment E1might pre-store the environment information Imfe in the storage circuit112, and provide the wearable device100to the operator1. In such arrangement, as shown inFIG.1, the processor13can access the environment information Imfe from the storage circuit112.

In some embodiments, the environment information Imfe indicates that the environment area F2in the environment E1ofFIG.3is a restricted area. For example, the restricted area can be clean area, employee area, construction area, etc. As shown inFIGS.3and4, the environment area F2is corresponding to the map area A2. Accordingly, the processor13controls the display panel15to generate a warning message (not shown) corresponding to the map area A2in the immersive content CI. In particular, the warning message is configured to inform the operator1that the restricted area exists in the environment E1.

It should be understood that the localization method200of the present disclosure is not limited to operations S201-S203shown inFIG.2. For example, in some embodiments, before operation S202, the localization method200further includes operation of generating the map M1of the environment E1. The descriptions of generating the map M1of the environment E1is similar to those of the above embodiments, and therefore are omitted herein.

In the embodiments ofFIG.3, the wearable device100uses the wireless communication circuit113to receive the push signal Sp transmitted by the broadcast device30, so as to obtain the environment information Imfe. In such arrangements, the image capturing circuit111and the storage circuit112can be omitted from the environment information collector11.

In the embodiments ofFIG.5, the wearable device100uses the image capturing circuit111to scan the 2D code50, so as to obtain the environment information Imfe. In such arrangements, the storage circuit112and the wireless communication circuit113can be omitted from the environment information collector11.

In the embodiments that the wearable device100uses the storage circuit112to pre-store the environment information Imfe, the image capturing circuit111and the wireless communication circuit113can be omitted from the environment information collector11.

In sum, the components in the wearable device100can be increased or decreased according to the actual applications.

As can be seen from the above embodiments of the present disclosure, by obtaining the environment information Imfe related to the environment E1where the wearable device100is to determine the target map area in the map M1of the environment E1, the processor13of the wearable device100can only load the target map area for calculating the location of the wearable device100. In such way, the occupation of the processing resources of the processor13can be reduced during the calculation of the location of the wearable device100. In sum, the wearable device100and the localization method200of the present disclosure has advantage of rapid and accurate localization, improved performance, etc.