Patent ID: 12210978

DETAILED DESCRIPTION

Since the present disclosure may be modified in various ways and may provide various embodiments, specific embodiments will be depicted in the appended drawings and described in detail with reference to the drawings.

However, it should be understood that the specific embodiments are not intended to limit the gist of the present disclosure; rather, it should be understood that the specific embodiments include all of the modifications, equivalents, or substitutes belonging to the technical principles and scope of the present disclosure.

Terms such as “first” and “second” may be used to describe various constituting elements, but the constituting elements should not be limited by the terms. The terms are introduced to distinguish one element from the others. For example, without departing from the technical scope of the present disclosure, a first element may be referred to as a second element, and similarly, the second element may be referred to as the first element.

If an element is said to be “connected” or “attached” to a different element, it should be understood that the former may be connected or attached directly to the different element, but another element may be present between the two elements. On the other hand, if an element is said to be “directly connected” or “directly attached” to a different element, it should be understood that there is no other element between the two elements.

Terms used in the present disclosure are intended only for describing a specific embodiment and are not intended to limit the technical scope of the present disclosure. A singular expression should be understood to indicate a plural expression unless otherwise explicitly stated. The term “include” or “have” is used to indicate the existence of an embodied feature, number, step, operation, element, component, or a combination thereof; and the term should not be understood to preclude the existence or possibility of adding one or more other features, numbers, steps, operations, elements, components, or a combination thereof.

Unless defined otherwise, all of the terms used in the present disclosure, including technical or scientific terms, provide the same meaning as understood generally by those skilled in the art to which the present disclosure belongs. Those terms defined in ordinary dictionaries should be interpreted to have the same meaning as conveyed by a related technology in the corresponding context; also, unless otherwise defined explicitly in the present disclosure, those terms should not be interpreted to have ideal or excessively formal meaning.

In what follows, with reference to appended drawings, preferred embodiments of the present disclosure will be described clearly and in detail so that those skilled in the art to which the present disclosure belongs may implement the present disclosure easily.

An autoencoder is an unsupervised learning algorithm mostly used for dimension reduction and pre-training of the network. In particular, during unsupervised learning of a 2D image, an autoencoder extracts valid features from the image and filters irrelevant information so that a decoder may reconstruct the extracted features. Autoencoders based on 3D objects showed excellent performance in various fields and usually voxelize an input point cloud to obtain a serialized input having a structure similar to that of a 2D autoencoder.

However, voxelization sacrifices details of a point cloud and increases the number of unnecessary, redundant voxels that increase computational complexity and decrease accuracy. Moreover, voxelization makes it difficult to retrieve 2D grids preserving the structure of a 3D point cloud, extract global features through a max-pooling function, and reconstruct a 3D structure directly from the extracted global features.

The present disclosure to be described below relates to a point autoencoder for solving the problems above and a 2D-to-3D point cloud transformation method using the autoencoder.

FIG.1illustrates a structure of a point autoencoder according to one embodiment of the present disclosure.

Referring toFIG.1, a point autoencoder100according to one embodiment of the present disclosure includes a PointNet encoder110and a BID decoder130.

The PointNet encoder110receives a 3D point cloud input and extracts global features. The PointNet encoder110may have the same architecture as a PointNet or any other point encoders such as KC-Net encoder. Here, the input may be 3D data obtained through a 3D imaging device such as a LiDAR or a stereo camera.

The PointNet encoder110extracts local features of each point through a convolution kernel having shared weights and outputs global features with respect to the whole point cloud through a symmetric max-pooling function.

For example, the PointNet encoder110uses a 1×k (where k is the dimensional size of a current point) convolution filter set with a stride of 1×1 to map each point to a high-dimensional space through the same mapping function, where the mapping result is not affected by the order of points in a point cloud. A convolution layer extracts global features of a point cloud using a symmetric max-pooling operator. If an input point cloud includes a point set S, a global feature vector may be determined by Eq. 1.

θ=MAXxi∈S⁢{h⁡(xi)}[Eq.⁢1]

In Eq. 1, h represents a symmetric mapping function composed of a series of convolution operations, and MAX represents a vector max-pooling function that receives n vectors of the same length and generates a new vector by calculating element-wise maximum values.

FIG.2illustrates a structure of a Bias-Induced Decoding (BID) layer according to one embodiment of the present disclosure.

Referring toFIG.2, a Bias-Induced decoder (BID)130reconstructs a 3D point cloud using global features. The BID decoder130comprises a mapping layer having a shared weight and an independent bias. Here, the mapping layer may be a BID layer. The BID layer will be described with reference toFIG.2.

Referring toFIG.2, the BID layer is similar to a fully connected layer; however, it differs from the fully connected layer in that each column shares the same weight (w) and has an independent bias (b). Therefore, if the output of the j-th column is denoted by oj, ojis determined according to Eq. 2.
oj=f(wxj+bj)  [Eq. 2]

Referring to Eq. 2, even if the same input is applied to each column (namely, if it is assumed that each row vector is a copy of a global feature), the same value may be easily mapped to a different position due to the existence of various biases. Also, since an output dimension may be adjusted for all of the layers, the same value may be mapped to a different dimensional space. At this time, interaction between x and b determines which bias is activated and sent to a subsequent processing layer. Basically, in the BID structure, features of different dimensional spaces may be regarded as being stored in the biases. When different global features are input, the different features are activated, and a combination of activated features determines a final result.

Although the BID structure may not have a powerful learning capability when it consists of a single layer structure, the BID structure may still provide performance similar to that of a fully connected layer through overlapping of multiple layers. In the BID structure, when the number of input vectors is N, the number of parameters becomes 1/N of the number of parameters for a fully connected layer.

First, the BID decoder130obtains the same number of samples as the number of points of a reconstructed point cloud before the mapping operation. If n represents the size of a point cloud and k represents the length of a feature vector, the BID decoder130performs a copy operation, copies a global feature n times, obtains n high-dimensional samples, and forms an n×k matrix. Here, since a global feature is extracted from an input point cloud, the global feature includes information on the original input point cloud. However, since each row of the matrix has the same value, if a subsequent task is symmetric, the original 3D structure may not be restored since each row is mapped to the same point. Therefore, through overlapping of a plurality of BID layers, one-to-many mapping may be performed. At this time, a 3D output of the last layer corresponds to a point cloud reconstructed by the BID decoder130.

When the order of points in an input point cloud is changed, global features are maintained in a consistent manner since the encoder is symmetric. Since biases are fixed after learning is completed, a mapping function of each point is also fixed. Also, a point mapped from each row of a duplicated matrix to 3D space has to maintain consistency. Therefore, global features determine a final point cloud output.

FIG.3illustrates a structure of a dual autoencoder performing a method for dimensional transformation of a point cloud according to another embodiment of the present disclosure.

Referring toFIG.3, a dual autoencoder includes a 2D encoder310, a 2D decoder320, a 3D encoder330, a 3D decoder340, a first associated network350, a second associated network360, a first switch370, and a second switch380.

The 2D encoder310extracts a high-dimensional semantic vector from a 2D image. The 2D encoder310may be constructed to have a structure in which a convolution layer and a fully connected layer are stacked together. An input to the 2D encoder310is a 2D image obtained by projecting a 3D point cloud onto a 2D plane at a predetermined angle, and an output of the 3D encoder310is a high-dimensional semantic vector for the 2D image.

The 2D decoder320is a 2D image decoder, the structure of which is constructed in the opposite of the 2D encoder310. In other words, the 2D decoder320is composed of a fully connected layer and a deconvolution layer and may reconstruct a 2D image from a high-dimensional semantic vector.

The 3D encoder330extracts a high-dimensional semantic vector from a 3D point cloud. The 3D encoder330may be a PointNet encoder110.

The 3D decoder340reconstructs a 3D point cloud from a high-dimensional semantic vector. The 3D decoder340may be a BID decoder130.

The first associated network350transforms a 2D latent vector to a 3D global feature, and the second associated network360transforms a 3D global feature to a 2D latent vector. The first associated network350and the second associated network360may be constructed using fully connected layers.

The first switch370determines an input to the 2D decoder320. In other words, the first switch determines so that the 2D decoder320may obtain an input from the 2D encoder320or from the second associated network360.

The second switch380determines an input to the 3D decoder340. In other words, the second switch determines so that the 3D decoder340may obtain an input from the 3D encoder330or from the first associated network360.

For example, if the first switch370selects the 2D encoder310as an input, the left part ofFIG.3may become an ordinary autoencoder while, if the second associated network360is selected as an input, 3D-to-2D point cloud transformation may be performed. Also, if the second switch380selects the 3D encoder330as an input, the right part ofFIG.3may become the point autoencoder100ofFIG.1while, if the first associated network350is selected as an input, 2D-to-3D point cloud transformation may be performed.

FIG.4is a flow diagram illustrating a learning process of the dual autoencoder ofFIG.3.

Referring toFIG.4, the S410step trains a 2D autoencoder. At this time, an input to the 2D autoencoder is a 2D image obtained by projecting a 3D point cloud onto a 2D plane at a predetermined angle, and the mean square error (MSE) between an input image and a reconstructed image may be used as a loss function.

The S430step trains a 3D autoencoder. An input to the 3D autoencoder is a 3D point cloud, and the chamfer distance between an input point and a reconstructed point may be used as a loss function.

Since an output point cloud is related to sequential biases while an input point cloud is not aligned, the element-wise distance between an input matrix and an output matrix may not be simply used to measure the difference between an output point cloud and an input point cloud. Since the chamfer distance measures a distance between each point and its nearest target point and is independent of the order of points in a point cloud, the chamfer distance is suitable for a loss function in a 3D point cloud. If an input point cloud is denoted by a point set S and a reconstructed output point cloud is denoted by a point set S0, the chamfer distance may be calculated by Eq. 3

LCH=1S⁢∑x∈S⁢MINo∈So⁢⁢x-o2+1So⁢∑x∈So⁢MINo∈S⁢⁢o-x2[Eq.⁢3]

The S450step trains the first associated network350. The goal of the present step is to fit the output of the first associated network350to a global feature vector extracted by the 3D autoencoder as much as possible. To this end, first, a 2D image and a 3D point cloud are input respectively to the 2D encoder310and the 3D encoder330to obtain a latent vector and a global feature. The latent vector is used as an input to the first associated network350to calculate a global feature based on the latent vector. Afterward, learning is performed to minimize the MSE between the latent vector-based global feature and a global feature extracted by the 3D autoencoder, where, at this time, ADAM optimizer may be used.

The S470step trains the second associated network360. The present step is similar to the S450step except that a global feature is used as an input to the second associated network360. Learning is performed to minimize the MSE between an input to the second associated network360and a latent vector.

The point autoencoder according to the present disclosure and a method for dimensional transformation of a point cloud using the autoencoder may be implemented in the form of computer-readable code in a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording devices storing data that may be interpreted by a computer system. Examples of computer-readable recording media include a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic tape, a magnetic disk, a flash memory, and an optical data storage device. Also, a computer-readable recording medium may be distributed over computer systems connected to each other through a computer communication network to store and execute computer-readable code in a distributed manner.

Also, an encoder of the point autoencoder and a decoder including a mapping layer may be implemented by a computer-readable recording medium, for example, a memory, which stores a computer program consisting of instructions that executes at least one processor and a method for dimensional transformation of a point cloud.

Also, a 2D encoder, a 2D decoder, a 3D encoder, and a 3D decoder of a dual autoencoder may be implemented by a computer-readable recording medium, for example, a memory, which stores a computer program consisting of instructions that executes at least one processor and a method for dimensional transformation of a point cloud.

So far, the present disclosure has been described with reference to appended drawings and embodiments, but the technical scope of the present disclosure is not limited to the drawings or embodiments. Rather, it should be understood by those skilled in the art to which the present disclosure belongs that the present disclosure may be modified or changed in various ways without departing from the technical principles and scope of the present disclosure defined by the appended claims below.