Electronic apparatus and method for incremental pose estimation and photographing thereof

An electronic apparatus and a method for incremental pose estimation and photographing are provided. In the method, at least two images of a 3D object are captured at different positions encircling the 3D object. Displacements and angular displacements are detected when capturing the images. Features of the 3D object in the images, the displacements and the angular displacements are used to estimate a central position of the 3D object and a distance between the electronic apparatus and the 3D object. A circumference suitable for capturing the images of the 3D object is estimated based on the distance and is divided into several segments, and a timing interval of a timer is adjusted based on a length of the segments. A camera of the electronic apparatus is triggered at intervals set by the timer to capture the images of the 3D object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Taiwan application serial no. 103142812, filed on Dec. 9, 2014. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.

BACKGROUND

1. Technical Field

The disclosure relates to a method and an apparatus for photographing, and particularly relates to an electronic apparatus and a method for incremental pose estimation and photographing thereof.

2. Related Art

Camera pose estimation is an important step to reconstruct a three-dimensional (3D) object by a monocular camera. Although a certain reconstruction quality can be achieved by purely processing captured images, considerable computation resources are required in matching all feature points among all images. Many images are dropped during feature matching process due to noise disturbance, usually the insufficient images results in an unsatisfied partial reconstruction of the 3D object.

In most prior arts, such as aerial photogrammetry or unmanned aerial vehicle, integrated global positioning system (GPS) and inertial measurement unit (IMU) sensors are used to assist feature matching process. However, the solutions for large-scale measurement are not suitable for small-scale measurement on a handheld device. Since data measured by the handheld device is less stable than data measured by fixed sensors on vehicles, errors are more liable to be amplified in small-scale measurement. It is necessary to provide a photographing method to refine biased sensor data and utilize more unordered images to reconstruct a complete 3D object.

SUMMARY

The disclosure is related to an electronic apparatus and a method for incremental pose estimation and photographing thereof, in which by estimating a circumference for a three-dimensional (3D) object, a timing interval is adjusted to automatically trigger the electronic apparatus to capture the images of the 3D object at suitable time.

The disclosure provides a method for incremental pose estimation and photographing, which is adapted for an electronic apparatus having an image capturing unit, a sensing unit and a timer. In the method, at least two images of a 3D object are captured at different positions encircling the 3D object by the image capturing unit. Displacements and angular displacements of the electronic apparatus are detected by the sensing unit when capturing the images. A circumference suitable for the electronic apparatus to capture the images of the 3D object is calculated according to a central position of the 3D object and a distance between the 3D object and the electronic apparatus estimated from a plurality of features of the 3D object in the images, the displacements and the angular displacements. The estimated circumference is divided into a plurality of segments, and a timing interval of a timer is adjusted according to a length of the segments. Finally, the image capturing unit is triggered at the timing intervals set by the timer to capture the images of the 3D object.

In an embodiment of the disclosure, after the step of triggering the image capturing unit at intervals set by the timer to capture the images of the 3D object, the method further includes following steps. First, the central position of the 3D object is re-estimated whenever a new image is added. Then, whether a difference between the re-estimated central position and the originally estimated central position exceeds a predetermined ratio is examined. If the difference exceeds the predetermined ratio, the circumference and the timing interval of the timer are adjusted according to the re-estimated central position.

In an embodiment of the disclosure, after the step of triggering the image capturing unit at the timing intervals set by the timer to capture the image of the 3D object, the method further includes following steps. First, the circumference of the 3D object are re-estimated whenever a new image is added. Then, whether a difference between the re-estimated circumference and the originally estimated circumference exceeds a predetermined ratio is examined. If the difference exceeds the predetermined ratio, the timing interval of the timer is adjusted according to the re-estimated circumference.

In an embodiment of the disclosure, after the step of capturing the images of the 3D object at different positions encircling the 3D object by the image capturing unit, and detecting the displacements and the angular displacements of the electronic apparatus by the sensing unit, the method further includes following steps. A first photographing pose of the electronic apparatus is estimated according to the displacement and the angular displacement of the electronic apparatus detected by the sensing unit when capturing each image. Then, a second photographing pose of the electronic apparatus is estimated and an overlap between the images is calculated by matching a plurality of features between the images. Finally, the images and corresponding pose tracking information are respectively recorded into a plurality of nodes in a linked list, where the pose tracking information includes the first photographing pose, the second photographing pose, a tolerance of the photographing poses and the overlap between the images.

In an embodiment of the disclosure, the step of estimating the photographing pose of the electronic apparatus includes following steps. Each move of the electronic apparatus is divided into a plurality of small moves, and a small displacement and a small angular displacement detected by the sensing unit in each small move of the electronic apparatus are retrieved. Moreover, the small displacement and the small angular displacement detected by the sensing unit in each small move are integrated to estimate a moving path of the electronic apparatus in a 3D space and the photographing pose of the electronic apparatus when capturing each image.

In an embodiment of the disclosure, after the step of respectively recording the images and corresponding pose tracking information into the nodes in the linked list, the method further includes following steps. A plurality of images with the closest pose tracking information are found from the linked list whenever the electronic apparatus captures a new image of the 3D object by the image capturing unit, and a node is added to the linked list to insert the new image and corresponding pose tracking information between the images or in front of or behind one of the images.

In an embodiment of the disclosure, after the step of estimating the circumference suitable for the electronic apparatus to capture the images of the 3D object, the method further includes following steps. A surface enclosing the 3D object is created according to the circumference, and the images captured by the electronic apparatus are respectively projected onto the surface to identify a plurality of covered regions and a plurality of uncovered regions on the surface. Then, the uncovered region with the shortest distance on the surface is searched according to a present position of the electronic apparatus, and a photographing position used for capturing a new image of the uncovered region is estimated. Finally, a moving direction and a moving distance of the electronic apparatus moving from the present position to the photographing position are calculated, and the moving direction and the moving distance are prompted on a user interface of the electronic apparatus.

In an embodiment of the disclosure, the step of identifying the covered regions and the uncovered regions on the surface include respectively projecting four corners of each image captured by the electronic apparatus onto the surface to identify the covered regions and the uncovered regions on the surface.

In an embodiment of the disclosure, after the step of prompting the moving direction and the moving distance on the user interface of the electronic apparatus, the method further includes following steps. The new image of the uncovered region is captured by the image capturing unit, and a new central position of the 3D object and a new distance between the 3D object and the electronic apparatus are re-estimated by re-analyzing the images and the new image. Then, the surface enclosing the 3D object is adjusted to a new surface according to the new central position and the new distance. Finally, the images captured by the electronic apparatus are re-projected onto the new surface to identify the covered regions and the uncovered regions on the new surface.

In an embodiment of the disclosure, the step of estimating the photographing position used for capturing the new image of the uncovered region includes estimating the photographing position for capturing the new image of the uncovered region to have an overlap of a predetermined ratio between the captured new image and the previously captured image.

In an embodiment of the disclosure, the step of estimating the circumference suitable for the electronic apparatus to capture the images of the 3D object includes following steps. A plurality of circumferences suitable for the electronic apparatus to capture the images of the 3D object are estimated to form a spherical surface enclosing the 3D object. The circumferences are divided into the segments for adjusting the timing interval of the timer.

In an embodiment of the disclosure, the step of dividing the estimated circumference into a plurality of the segments and adjusting the timing interval of the timer according to the length of the segments includes dividing the estimated circumference into the segments according to a displacement of the electronic apparatus when capturing the images.

The disclosure provides an electronic apparatus including an image capturing unit, a sensing unit, a timer and a calculation unit. The image capturing unit is used for capturing images of a 3D object at different positions encircling the 3D object. The sensing unit detects displacements and angular displacements of the electronic apparatus when the image capturing unit captures the images. The timer triggers the image capturing unit to capture the images of the 3D object at timing intervals. The calculation unit is coupled to the image capturing unit, the sensing unit and the timer, and executes a plurality of modules. The modules include a circumference estimation module and a timer setting module. The circumference estimating module estimates a circumference suitable for the electronic apparatus to capture the images of the 3D object according to a central position of the 3D object and a distance between the 3D object and the electronic apparatus estimated from a plurality of features of the 3D object in the images, the displacements and the angular displacements detected by the sensing unit. The timer setting module divides the circumference estimated by the circumference estimating module into a plurality of segments, and adjusts the timing interval of the timer according to a length of the segments.

In an embodiment of the disclosure, after the timer triggers the image capturing unit to capture a new image of the 3D object according to the adjusted timing interval, the circumference estimating module re-estimates the central position of the 3D object. The timer setting module adjusts the timing interval of the timer according to the re-estimated the central position if a difference between the re-estimated central position and the originally estimated central position exceeds a predetermined ratio.

In an embodiment of the disclosure, after the timer triggers the image capturing unit to capture a new image of the 3D object according to the adjusted timing interval, the circumference estimating module further re-estimates the circumference of the 3D object, and the timer setting module adjusts the timing interval of the timer according to the re-estimated circumference if a difference between the re-estimated circumference and the originally estimated circumference exceeds a predetermined ratio.

In an embodiment of the disclosure, the modules further include a feature matching module, a pose estimation module and a pose tracking module. The feature matching module extracts and matches a plurality of features in the images. The pose estimation module estimates a first photographing pose of the electronic apparatus according to the displacement and the angular displacement of the electronic apparatus detected by the sensing unit when capturing each image, and estimates a second photographing pose of the electronic apparatus according to the features matched by the feature matching module. The pose tracking module respectively records the images and corresponding pose tracking information into a plurality of nodes in a linked list, where the pose tracking information includes the first photographing pose, the second photographing pose, a tolerance of the photographing poses and the overlap between the images.

In an embodiment of the disclosure, the modules further include a path refinement module. The path refinement module divides each move of the electronic apparatus into a plurality of small moves, and retrieves a small displacement and a small angular displacement detected by the sensing unit in each small move, and integrates the small displacement and the small angular displacement detected by the sensing unit in each small move to estimate a moving path of the electronic apparatus in a 3D space and the photographing pose of the electronic apparatus when the image capturing unit captures each image.

In an embodiment of the disclosure, the modules further include an image sorting module. Each time when the electronic apparatus captures a new image of the 3D object by the image capturing unit, the image sorting module finds a plurality of images with the closest pose tracking information from the linked list, and adds a node to the linked list to insert the new image and corresponding pose tracking information between the images or in front of or behind one of the images.

In an embodiment of the disclosure, the modules further include a surface creating module, a projection module, a searching module and a directing module. The surface creating module creates a surface enclosing the 3D object according to the circumference. The projection module respectively projects the images captured by the electronic apparatus onto the surface to identify a plurality of covered regions and a plurality of uncovered regions on the surface. The searching module searches the uncovered region with a shortest distance to a present position of the electronic apparatus, and estimates a photographing position used for capturing a new image of the uncovered region. The directing module calculates a moving direction and a moving distance of the electronic apparatus moving from the present position to the photographing position, and prompts the moving direction and the moving distance on a user interface of the electronic apparatus.

In an embodiment of the disclosure, the circumference estimating module further estimates a plurality of circumferences suitable for the electronic apparatus to capture the images of the 3D object to form a spherical surface enclosing the 3D object, and divides the circumferences into a plurality of the segments and provides the divided segments to the timer setting module for adjusting the timing interval of the timer.

According to the above descriptions, in the electronic apparatus and the method for incremental pose estimation and photographing thereof of the disclosure, at least two images of the 3D object at different positions encircling the 3D object are captured, so as to estimate the circumference suitable for capturing the images of the 3D object. The circumference is divided for adjusting the timing interval for capturing the images. According to the position and the pose that the user captures the images of the 3D object, a proper timing interval is calculated to automatically trigger the electronic apparatus to capture the images, so as to assist the user in capturing the images of the 3D object for reconstructing a complete 3D object.

In order to make the aforementioned and other features and advantages of the disclosure comprehensible, several exemplary embodiments accompanied with figures are described in detail below.

DETAILED DESCRIPTION OF DISCLOSED EMBODIMENTS

FIG. 1is a block diagram of an electronic apparatus according to an embodiment of the disclosure. Referring toFIG. 1, in the present embodiment, the electronic apparatus100is, for example, a digital camera, a smart phone, a tablet PC, or other handheld devices having an image capturing function, which is not limited by the disclosure. The electronic apparatus100includes an image capturing unit110, a sensing unit120, a timer130and a calculation unit140. Functions of the above devices are described below.

The image capturing unit110is, for example, a device containing an optical prime lens or an optical zoom lens, and a photosensitive element such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) device. In the present embodiment, the image capturing unit110can adjust a focal length to capture images of a 3D object at different positions encircling the 3D object.

The sensing unit120is, for example, a sensor such as a G-sensor, a gyroscope, an accelerometer or an E-compass, or a combination thereof, which can be used for detecting displacements and angular displacements of the electronic apparatus100when the image capturing unit110captures the images of the 3D object.

The calculation unit140is coupled to the image capturing unit110, the sensing unit120and the timer130, and the calculation unit140is, for example, a central processing unit (CPU) having a single core or multiple cores, or a programmable general purpose or special purpose microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC) or other similar devices or a combination of the above devices. In the present embodiment, the calculation unit140can be used to execute a circumference estimation module141and a timer setting module142to execute a method for incremental pose estimation and photographing of the disclosure.

In detail,FIG. 2Ais a flowchart illustrating a method for incremental pose estimation and photographing according to an embodiment of the disclosure. Referring toFIG. 1andFIG. 2A, the method of the present embodiment is adapted for the electronic apparatus100, and detailed steps of the method for incremental pose estimation and photographing of the present embodiment are described below with reference of various components of the electronic apparatus100ofFIG. 1.

First, in step S220, the electronic apparatus100captures at least two images of a 3D object at different positions encircling the 3D object by the image capturing unit110, and detects displacements and angular displacements of the electronic apparatus100by the sensing unit120. The user may, for example, hold the electronic apparatus100to walk around the 3D object and capture the images of the 3D object, and the electronic apparatus100may use the sensing unit120to detect the displacements and the angular displacements of the electronic apparatus100during an image capturing process, and uses the detected displacements and angular displacements for subsequent pose estimation.

In step S240, a circumference suitable for capturing the images of the 3D object is calculated by the circumference estimation module141according to a central position of the 3D object and a distance between the 3D object and the electronic apparatus100estimated from a plurality of features of the 3D object in the images, the displacements and the angular displacements.

For example,FIG. 2Bis a schematic diagram of capturing images of a 3D object according to an embodiment of the disclosure. In the present embodiment, an electronic apparatus respectively captures images of a 3D object D1at a photographing position V1and a photographing position V2encircling the 3D object D1, and detects a displacement and an angular displacement of the electronic apparatus when moving from the photographing position V1to the photographing position V2. The electronic apparatus can determine whether the captured images have overlap or not. If the two images have overlap, the two images can be used to calculate a center of the photographed 3D object D1and a distance between the 3D object D1and the electronic apparatus. If the two images are not overlapped to each other due to an excessive distance between different photographing positions or an excessive difference between photographing angles according to the detected displacement and angular displacement, the above calculation is not performed.

According to the images captured at the photographing position V1and the photographing position V2, and the displacement and the angular displacement of the electronic apparatus detected when moving from the photographing position V1to the photographing position V2, the electronic apparatus can estimate a central position C1of the 3D object D1. In detail, the electronic apparatus can calculate distances between the photographing position V1, the photographing position V2and the 3D object D1according to focal lengths of the captured images or other photographing information, and can obtain a distance between the photographing position V1and the photographing position V2according to the detected displacement and angular displacement. In this way, the central position C1can be estimated through a simple triangulation method. A circumference R1suitable for capturing the images of the 3D object D1can be estimated according to the distances between the photographing position V1, the photographing position V2and the 3D object D1, and each point on the circumference R1is a position suitable for capturing the image of the 3D object D1.

Referring back to the flow ofFIG. 2A, in step S260, the timer setting module142divides the circumference estimated by the circumference estimating module141into a plurality of segments, and adjusts a timing interval of the timer130according to a length of the segments. In an embodiment, the circumference estimating module141can only estimate the circumference on a horizontal plane where the electronic apparatus100is located, and the timer setting module142divides the circumference into a plurality of segments to calculate the timing interval of the timer. In another embodiment, the circumference estimating module141can also estimate a plurality of circumferences (for example, circumferences encircling an upper part, a middle part and a lower part of the 3D object D1) that are suitable for the electronic apparatus100to capture the images of the 3D object D1, where the circumferences can form a spherical surface enclosing the 3D object D1, and these circumferences can be respectively divided into a plurality of segments of different numbers to facilitate calculating the timing interval of the timer130. The number of the divided segments and the number of the estimated circumferences can be determined according to a size of the 3D object D1, which is not limited by the disclosure.

For example,FIG. 2Cis a schematic diagram of capturing images of the 3D object according to an embodiment of the disclosure. The present embodiment is an extension application of the embodiment ofFIG. 2B, where the electronic apparatus further divides the estimated circumference R1suitable for capturing the images of the 3D object D1into a plurality of segments (for example, the segments S1) of a predetermined number (for example, 20), and adjusts the timing interval of the timer130according to a length of the segments. Particularly, in the present embodiment, besides the circumference R1on the horizontal plane where the electronic apparatus is located, the circumference estimating module141can also respectively estimate circumferences R2and R3that are suitable for capturing the images of the 3D object D1for an upper part and a lower part of the 3D object D1, and the timer setting module142divides the circumferences R2and R3into a plurality of segments of a predetermined number (for example, 10), and adjusts the timing interval of the timer130according to a length of the segments. Taking the embodiment ofFIG. 2Cas an example, capturing the images of the 3D object D1along the circumferences R1, R2and R3can fill holes at a top part and a bottom part of the 3D object D1caused by photographing only along the circumference R1.

It should be noticed that in the above step S260, the timer setting module142divides the circumference estimated by the circumference estimation module141into a plurality of segments of a predetermined number, and adjusts the timing interval of the timer130according to the length of the segments. However, in another embodiment, the timer setting module142can divide the circumference according to a displacement of the electronic apparatus100when capturing the images. The timer setting module142may take an average displacement, a minimum displacement or a maximum displacement of the electronic apparatus100when capturing the images as a length to divide the circumference into a plurality of segments. In this way, the timing interval set by the timer setting module142can be complied with a moving speed of the electronic apparatus100to capture the images of the 3D object, so that the image capturing unit110is triggered to capture the images at proper timing.

Referring back toFIG. 2A, in step S280, according to the timing interval adjusted by the circumference estimation module141, the image capturing unit110is triggered at intervals set by the timer130to capture the images of the 3D object.

The electronic apparatus100can automatically capture the images of the 3D object through automatic trigger of the timer130when moving by a certain distance. The user is only required to hold the electronic apparatus100to move around the 3D object, and the electronic apparatus100can automatically capture all the images required for reconstructing the 3D object. A failed reconstruction due to excessive moving distance causing non-overlap between images is also avoided. This way improves the convenience of the operation and enhances the integrity of the constructed 3D object.

It should be noticed that since the user is unnecessary to move along a fixed (i.e. the circumference) path when capturing the images of the 3D object, a variety of situations may be encountered such as the user approaches to the 3D object, moves away from the 3D object, changes a moving direction, or changes a moving speed. The above situations probably cause a change of the distance between the electronic apparatus100and the 3D object and cause a change of the circumference suitable for capturing the images of the 3D object. Therefore, it is necessary to re-estimate the circumference of the 3D object to adjust the timing interval of the timer130. In this case, the disclosure provides many auxiliary determination methods to assist the electronic apparatus100in re-estimating the central position and the circumference of the 3D object.

In case that the user approaches or moves away from the 3D object, the circumference estimation module141analyzes the new image and the previously captured image to re-estimate the central position and the circumference of the 3D object. In an embodiment, the timer setting module142compares the central position re-estimated by the circumference estimation module141with the previously estimated central position to determine whether a difference therebetween exceeds a predetermined ratio (for example, 10%). Once the difference exceeds the predetermined ratio, it represents that the previously estimated circumference is not applicable, and accordingly the timer setting module142adjusts the timing interval of the timer130according to the circumference re-estimated by the circumference estimation module141. In another embodiment, the timer setting module142can directly compare the circumference re-estimated by the circumference estimation module141with the previously estimated circumference to determine whether a difference therebetween exceeds a predetermined ratio (for example, 10%). Once the difference exceeds the predetermined ratio, it represents that the previously estimated circumference is not applicable, and accordingly the timer setting module142adjusts the timing interval of the timer130according to the re-estimated circumference.

In brief, the electronic apparatus100can modify the previously estimated circumference through further analysis, and adjust the timer130to update the timing interval. It is assumed that when the user uses the electronic apparatus100to capture the images, the user deviates from the path (i.e. the circumference) originally estimated by the circumference estimation module141, the electronic apparatus100can re-estimate the circumference to update the timing interval. The timing for the electronic apparatus100to update the timing interval can be determined at least according to whether the central position or the circumference is shifted.

On the other hand, in case that the user changes a moving direction and/or a moving speed, since it may cause a result that the subsequently captured image and the previously captured image are discontinuous or have no overlap therebetween, the disclosure also provides a pose tracking method to compare the subsequently captured image and the previously captured image by estimating a photographing pose of each image, so as to find an order and correlation between the subsequently captured image and the previously captured image to facilitate subsequent reconstruction of the 3D object.

For example, referring toFIG. 3,FIG. 3is a block diagram of a calculation unit of the electronic apparatus according to an embodiment of the disclosure. Besides a circumference estimation module341and a timer setting module342, the calculation unit340of the present embodiment further includes a feature matching module343, a pose estimation module344and a pose tracking module345. Functions of the circumference estimation module341and the timer setting module342are the same as or similar to those of the circumference estimation module141and the timer setting module142of the embodiment ofFIG. 1, and details thereof are not repeated. The calculation unit340of the present embodiment can be applied to the electronic apparatus100.

FIG. 4is a flowchart illustrating a method for incremental pose estimation and photographing according to an embodiment of the disclosure. The steps of the present embodiment are subsequent to the step S220of the embodiment ofFIG. 2A. With reference ofFIG. 2AtoFIG. 4, after the sensing unit120detects the displacements and the angular displacements of the electronic apparatus100when the image capturing unit110captures the images, in step S421, the pose estimation module344estimates a first photographing pose of the electronic apparatus100according to the detected displacements and the angular displacements. In step S423, the pose estimation module344estimates a second photographing pose of the electronic apparatus100according to the features matched by the feature matching module343. The first photographing pose and the second photographing pose may include a position, a pose of the electronic apparatus100in a 3D space, a relative position and a relative angle between the electronic apparatus100and the 3D object, etc., which is not limited by the disclosure. In the disclosure, the photographing pose estimated by data measured by the sensor and the photographing pose estimated by processing the images can be combined to obtain a more accurate pose estimation result.

In step S425, the pose tracking module345records the images and pose tracking information thereof into a plurality of nodes in a linked list. The pose tracking information includes the first photographing pose, the second photographing pose, a tolerance of the photographing poses and the overlap between the images.

It should be noticed that the electronic apparatus100of the present embodiment may use a suitable linked list according to different requirements for recording images and information, for example, a singly linked list, a doubly linked list and a circular linked list. The linked list has a plurality of nodes, and each node may store variables of data, and has an indicator pointed to each other, for example, the singly linked list has single direction indicators, the doubly linked list has bi-direction indicators, and the circular linked has a last node being redirected to a first node to form a loop. Taking the pose tracking information to be recorded as an example, after the pose estimation module344estimates the first photographing pose of the electronic apparatus100according to the displacement and the angular displacement, the first photographing pose can be stored to the node. After the feature matching module343extracts the features of the images for matching, the estimated second photographing pose can also be stored to the node, and the first photographing pose and the second photographing pose are correlated with each other due to a characteristic of the linked list. Moreover, the feature matching module343can estimate whether the new image is located between the two images or in front of or behind one of the images by calculating the overlap between the images, and then stores the new image and related information into the node. The electronic apparatus100can set a tolerance of ±15% between the two photographing poses to avoid small overlap.

It should be noticed that during the process of reconstructing the 3D object, the user is required to hold and move the electronic apparatus100to capture images. Each time when the electronic apparatus100detects valid displacements and angular displacements by the sensing unit, the electronic apparatus100integrates the displacements and the angular displacements detected by the sensing unit120to calculate a total displacement and a total angular displacement of the move. However, if rotation is not considered during the integration, when calculating a real moving path of the electronic apparatus100, it is unable to obtain a correct path by directly accumulating the displacements and the angular displacements.

The above problem is due to treating the continuous movement of the electronic apparatus100as discrete movement. From a micro view, each move of the electronic apparatus100contains rotation. Therefore, the disclosure provides a method for refining the calculated path.FIG. 5is a block diagram of a calculation unit of the electronic apparatus according to an embodiment of the disclosure. Besides a circumference estimation module541, a timer setting module542, a feature matching module543, a pose estimation module544and a pose tracking module545, the calculation unit540of the present embodiment further includes a path refinement module546and an image sorting module547. Functions of the circumference estimation module541, the timer setting module542, the feature matching module543, the pose estimation module544and the pose tracking module545are the same as or similar to those of the circumference estimation module341, the timer setting module342, the feature matching module343, the pose estimation module344and the pose tracking module345of the aforementioned embodiment, and details thereof are not repeated. Similarly, the calculation unit540can be applied to the electronic apparatus100.

Referring toFIG. 5andFIG. 6,FIG. 6is schematic diagram of a path refinement method according to an embodiment of the disclosure. When the electronic apparatus100is moved from a position PA to a position PB, an incorrect position PB′ is obtained if a displacement D and an angular displacement θ detected by the sensing unit120are directly used for calculation. In the present embodiment, the path refinement module546, for example, divides the move of the electronic apparatus100into five small moves respectively comprising five small displacements d1, d2, d3, d4, d5and five small angular displacements θ1, θ2, θ3, θ4, θ5, and integrates the small displacements d1-d5and the small angular displacements θ1-θ5to obtain positions P1, P2, P3, P4and PB after the five small moves. In this way, the pose estimation module544obtains a more accurate position PB. The number of small moves divided by the path refinement module546is not limited by the disclosure, and the number can be a system pre-setting or can be adjusted by the user.

It should be noticed that in the present embodiment, the moves of the electronic apparatus100is regarded as continuous and rotation angles in the moves are considered. From a micro view, the path refinement module546of the present embodiment divides each move into a plurality of small moves, where the small displacement generated by each small move can be calculated by an accelerometer, and the corresponding small angular displacement can be calculated by a gyroscope. Since a sampling speed of the above sensor is very high (which can be 100 times/second), the five small displacements d1-d5can all be regarded as straight line motions, and the corresponding small angular displacements θ1-θ5are respectively used to rotate a start direction of a next move, so as to obtain a more accurate moving path (i.e. from the position PA to the position PB).

It should be noticed that the path refinement module546directly updates the path after each calculation, such that an extra memory is no longer required to temporarily store a previous calculation result when calculating the total displacement. According to the above calculation method, a complete path can be accurately calculated without a large deviation.

Referring back to the flow ofFIG. 4, in order to avoid incomplete construction of the 3D object caused by the hand-held motion error, the image sorting module547searches a plurality of images with the closest pose tracking information from the linked list, and adds a node to the linked list to insert the new image and the related pose tracking information between two images or in front of or behind one of the images. The new image is, for example, stored in the new node, and a pointing relationship of the linked list is used to set an order of each node. In the present embodiment, the image sorting module547can retain or place the image with the excessively large difference in a last position of the linked list. In detail, during the image capturing process, the image sorting module547compares, inserts and records the new image to ensure the new image is accommodated in a proper position among all of the images.

On the other hand, to construct a complete 3D object, a plurality of images with different angles within 360 degrees encircling the 3D object are required, and since the user does not know the images of which angles are lacked to reconstruct the complete 3D object during the image capturing process, in the disclosure, the existing images and the known information such as photographing poses are combined to further estimate the images of other angles that are not obtained but required for reconstructing the 3D object, and through prompting of the electronic apparatus100, the user is guided to the proper positions to capture the images, so as to obtain the images of all of the required angles for reconstructing the 3D object.

Referring toFIG. 7,FIG. 7is a block diagram of a calculation unit of the electronic apparatus according to an embodiment of the disclosure. Besides a circumference estimation module741, a timer setting module742, a feature matching module743, a pose estimation module744and a pose tracking module745, a path refinement module746and an image sorting module747, the calculation unit740of the present embodiment further includes a surface creating module751, a projection module752, a searching module753and a directing module754. Functions of the circumference estimation module741, the timer setting module742, the feature matching module743, the pose estimation module744, the pose tracking module745, the path refinement module746and the image sorting module747are the same as or similar to those of the circumference estimation module541, the timer setting module542, the feature matching module543, the pose estimation module544, the pose tracking module545, the path refinement module546and the image sorting module547of the aforementioned embodiment, and details thereof are not repeated. Similarly, the calculation unit740can be applied to the electronic apparatus100.

Referring toFIG. 7andFIG. 8A,FIG. 8Ais a flowchart illustrating a method for incremental pose estimation and photographing. The steps of the present embodiment are, for example, subsequent to the step S240of the embodiment ofFIG. 2, i.e. after the circumference estimation module141estimates the circumference suitable for capturing the images of the 3D object. In step S841, the surface creating module751creates a surface enclosing the 3D object according to the circumference of the 3D object. The surface can be a surface of any object, for example, a surface of a cylinder, a sphere or a cube, etc., which is not limited by the disclosure.

In step S842, the projection module752respectively projects the images captured by the electronic apparatus100onto the surface of the 3D object to identify a plurality of covered regions and a plurality of uncovered regions on the surface. The projection module752, for example, projects four corners of each image captured by the electronic apparatus100onto the surface to identify the covered regions and the uncovered regions on the surface.

In step S843, the searching module753searches the uncovered region with the shortest distance to a present position of the electronic apparatus100, and estimates a photographing position for capturing a new image of the uncovered region. In detail, after the projection module752projects the images onto the surfaces created by the surface creation module751, the searching module753can find the uncovered regions (i.e. the image of such angel is still required) from the surface, and searches the uncovered region with the shortest distance from all of the uncovered regions according to the present position of the electronic apparatus100, so as to guide the user to move forward to the corresponding position to capture the image.

It should be noticed that in other embodiments, when the searching module753estimates the photographing position for capturing the new image of the uncovered region, the uncovered region overlapped to the covered region by a predetermined ratio is selected, such that the captured new image and the previously captured image are overlapped by the predetermined ratio. In this way, a relative position between the images can be learned according to the overlap, so as facilitate reconstructing the 3D object. For example, it is assumed that resolutions and sizes of the images captured by the image capturing unit110are all the same, and the user or the system can set the image to have the overlap with the predetermined ratio of 30%. In this way, all of the images captured by the electronic apparatus100have the overlap of 30%, which can serve for reconstructing the 3D object.

For example, Referring toFIG. 8B,FIG. 8Bis a schematic diagram of projecting images on a surface enclosing the 3D object to estimate a photographing position according to an embodiment of the disclosure. In the present embodiment, the surface S2is a sphere enclosing the 3D object, and an image Gi is an image captured by the electronic apparatus at a photographing position Vi. A covered region Ni is obtained by projecting the image Gi onto the surface S2, and the electronic apparatus searches an adjacent uncovered region Nj from other uncovered regions except the covered region Ni from the surface S2, and accordingly estimates a photographing position Vj for capturing a new image Gj. Where, an overlap LP between the covered region Ni and the uncovered region Nj, for example, occupies 30% of the covered region Ni and the uncovered region Nj. Therefore, the new image Gj captured by the electronic apparatus at the photographing position Vj and the known image Gi are ensured to have the overlap of the predetermined ratio, such that the electronic apparatus can use the images Gi and Gj to reconstruct the 3D object later.

Referring back to the flow ofFIG. 8A, in step S844, the directing module754calculates a moving direction and a moving distance of the electronic apparatus moving from the present position to the estimated photographing position of the new image, and prompts the same on a user interface of the electronic apparatus100, so as to direct the user to move to the required position. A prompt method of the electronic apparatus100includes a text prompt, an image prompt, a voice prompt, etc., which is not limited by the disclosure. In this way, the user can quickly learn how to move to capture the shorted images required for reconstructing the 3D object according to the prompt of the electronic apparatus.

It should be noticed that each time the electronic apparatus100captures a new image of the uncovered region by the image capturing unit110, the circumference estimation module141, for example, re-estimates a new central position of the 3D object and a new distance between the 3D object and the electronic apparatus100by re-analyzing the images and the new image. The surface creating module751adjusts the surface enclosing the 3D object to a new surface according to the new central position and the new distance. Then, the projection module752re-projects the images captured by the electronic apparatus onto the new surface to identify the covered regions and the uncovered regions on the new surface, so as to implement the subsequent prompting. The above steps are repeated until the images captured by the electronic apparatus100fill the surface. In this way, the images captured by the electronic apparatus100are enough to reconstruct a complete 3D object.

In summary, in the electronic apparatus and the method for incremental pose estimation and photographing thereof of the disclosure, by estimating the circumference suitable for capturing the images of the 3D object, and dividing the circumference to estimate the timing interval suitable for capturing the images, the user is only required to hold the electronic apparatus and move around the 3D object, and the electronic apparatus can capture the images suitable for reconstructing the 3D image at proper time points. The disclosure further provides corresponding methods of pose tracking, image sorting, path refinement, photographing path guiding, etc. for a plurality of situations probably occurred when the user holds the electronic apparatus to capture the images of the 3D object. In this way, not only the user is assisted to capture the images of the 3D object, correctness and integrity of the captured images are ensured, so as to reconstruct a complete 3D object.