Method, electronic device, and program product for training encoder and processing data

Embodiments relate to a method, an electronic device, and a program product for training an encoder and processing data. The method includes inputting sample point cloud data for an object to an encoder to obtain encoded data for the object, and determining, by transforming the encoded data, a plurality of invariant portions for the object and a plurality of variable portions for the object, an invariant portion in the plurality of invariant portions indicating an invariant feature of the object and a variable portion in the plurality of variable portions indicating a variable feature of the object. The method further includes determining, based on the plurality of invariant portions and the plurality of variable portions, a similarity loss and a spatial loss for the sample point cloud data, and adjusting, based on the similarity loss and the spatial loss, a parameter of the encoder to obtain a trained encoder.

The present application claims priority to Chinese Patent Application No. 202210432066.2, filed Apr. 22, 2022, and entitled “Method, Electronic Device, and Program Product for Training Encoder and Processing Data,” which is incorporated by reference herein in its entirety.

FIELD

Embodiments of the present disclosure relate to the field of data processing, and more particularly, to a method, an electronic device, and a program product for training an encoder and processing data.

BACKGROUND

With the development of computer technologies, people have begun to use computer vision technologies to obtain information about a target object or an environment. Generally, in this process, point cloud data for an object may be captured by various devices, and the captured point cloud data may then be analyzed to obtain various desired information. In accordance with conventional practice, most of the available features for point clouds are produced manually for specific tasks. Point cloud features usually encode certain statistical attributes of a point. However, there are still many problems that need to be solved in the analysis and processing of point cloud data.

SUMMARY

A method, an electronic device, and a program product for training an encoder and processing data are provided in embodiments of the present disclosure.

According to a first aspect of the present disclosure, a method for training an encoder is provided. The method includes: inputting sample point cloud data for an object to an encoder to obtain encoded data for the object. The method includes: determining, by transforming the encoded data, a plurality of invariant portions for the object and a plurality of variable portions for the object, an invariant portion in the plurality of invariant portions indicating an invariant feature of the object and a variable portion in the plurality of variable portions indicating a variable feature of the object. The method further includes: determining, based on the plurality of invariant portions and the plurality of variable portions, a similarity loss and a spatial loss for the sample point cloud data. The method further includes: adjusting, based on the similarity loss and the spatial loss, a parameter of the encoder to obtain a trained encoder.

According to a second aspect of the present disclosure, a method for processing data is provided. The method includes: inputting point cloud data for an object to a trained encoder to obtain encoded data for the object, the trained encoder being obtained by adjusting a parameter of an encoder based on a similarity loss and a spatial loss obtained for sample point cloud data for a sample object. The method further includes: determining, by transforming the encoded data, an invariant portion for the object and a variable portion for the object, the invariant portion indicating an invariant feature of the object and the variable portion indicating a variable feature of the object.

According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; a memory coupled to the at least one processor and having instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the device to perform actions including: inputting sample point cloud data for an object to an encoder to obtain encoded data for the object; determining, by transforming the encoded data, a plurality of invariant portions for the object and a plurality of variable portions for the object, an invariant portion in the plurality of invariant portions indicating an invariant feature of the object and a variable portion in the plurality of variable portions indicating a variable feature of the object; and determining, based on the plurality of invariant portions and the plurality of variable portions, a similarity loss and a spatial loss for the sample point cloud data; and adjusting, based on the similarity loss and the spatial loss, a parameter of the encoder to obtain a trained encoder.

According to a fourth aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; a memory coupled to the at least one processor and having instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the device to perform actions including: inputting point cloud data for an object to a trained encoder to obtain encoded data for the object, the trained encoder being obtained by adjusting a parameter of an encoder based on a similarity loss and a spatial loss obtained for sample point cloud data for a sample object; and determining, by transforming the encoded data, an invariant portion for the object and a variable portion for the object, the invariant portion indicating an invariant feature of the object and the variable portion indicating a variable feature of the object.

According to a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform the steps of the methods in at least one of the first and second aspects of the present disclosure.

In the figures, identical or corresponding numerals represent identical or corresponding parts.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the drawings show example embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms, and should not be explained as being limited to the embodiments described herein. Instead, these embodiments are provided for understanding the present disclosure more thoroughly and completely. It should be understood that the accompanying drawings and embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the protection scope of the present disclosure.

In the description of embodiments of the present disclosure, the term “include” and similar terms thereof should be understood as open-ended inclusion, that is, “including but not limited to.” The term “based on” should be understood as “based at least in part on.” The term “an embodiment” or “the embodiment” should be understood as “at least one embodiment.” The terms “first,” “second,” and the like may refer to different or identical objects. Other explicit and implicit definitions may also be included below.

As described above, three-dimensional point cloud data includes much information specific to the object. A number of techniques have been used in conventional solutions to process the three-dimensional point cloud data for the analysis thereof. These methods suffer from many limitations. For example, the volumetric convolutional neural network (CNN) is an early solution to apply CNNs to process point cloud data. However, the volumetric representation is limited by its resolution due to data sparsity and the computational cost of three-dimensional convolution. Special methods have also been proposed to deal with sparsity problems. However, these operations are still focused on sparse volumes. It is a challenge for them to deal with very large point clouds. Moreover, this kind of processing is relatively time consuming and wastes a lot of resources, and is not user friendly.

To address at least the above and other potential problems, embodiments of the present disclosure provide a method for training an encoder and processing data. In the method, a computing device inputs sample point cloud data for an object to the encoder to obtain encoded data for the object. A plurality of invariant portions for the object and a plurality of variable portions for the object are determined by transforming the encoded data. The computing device determines, based on the plurality of invariant portions and the plurality of variable portions, a similarity loss and a spatial loss for the sample point cloud data. The computing device then adjusts, based on the similarity loss and the spatial loss, a parameter of the encoder to obtain a trained encoder. Further, the computing device may use the trained encoder to process the point cloud data. This method improves the efficiency of processing point cloud data, saves time and computational resources, and improves accuracy.

Embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

FIG.1is a schematic diagram of example environment100in which embodiments of the present disclosure can be implemented. As shown inFIG.1, example environment100includes computing device104. Computing device104is used to receive point cloud data102for processing.

Computing device104includes, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), and a media player), a multi-processor system, a consumer electronic product, a minicomputer, a mainframe computer, a distributed computing environment including any of the above systems or devices, etc.

Point cloud data102in example environment100includes point cloud data for a target object. This point cloud data102is for a rigid target object, such as for an aircraft or a human face, or may be for other suitable objects.

Point cloud data102is input to encoder106in computing device104to obtain encoded data. In some embodiments, encoder106is a neural network model. In some embodiments, encoder106is any suitable machine model for encoding the data. The above examples are intended to describe the present disclosure only and are not specific limitations to the present disclosure.

Encoder106then inputs the encoded data to transformation module108. The transformation module108performs a matrix transformation on the encoded data to obtain invariant portion110for the target object and variable portion112for the target object.

In one example, if point cloud data for an aircraft is input, the invariant portion may correspond to the shape and size of the aircraft, while the variable portion may correspond to the coordinates or position information for various parts of the aircraft. This is because the shape of the aircraft is invariant in the point cloud data for the aircraft obtained from any angle, while its position may be different in different point cloud data. The above examples are intended to describe the present disclosure only and are not specific limitations to the present disclosure. In some embodiments, the input point cloud data is the point cloud data for a human face. Then, the invariant and variable portions of the human face are obtained. The invariant portion may be sizes of five sense organs of the human face, while the variable portion may be the expression. Variable portion112may be applied to other objects or avatars. For example, the expression may be combined with another avatar in an avatar pool to generate an avatar with a new expression. The above examples are intended to describe the present disclosure only and are not specific limitations to the present disclosure.

In some embodiments, when training encoder106, point cloud data, as sample point cloud data, is input to the encoder106to obtain a vector representation of the object, which is then input to transformation module108for matrix transformation to obtain invariant portion110and variable portion112. A similarity loss is then determined using invariant portion110, and a spatial loss is determined using variable portion112. These two losses are then combined to adjust the parameter of the encoder106to enable training.

In some embodiments, in addition to using the above losses, the invariant portion and the variable portion are input to a decoder to obtain reference point cloud data for the object, then a data loss between the reference point cloud data and the sample point cloud data is calculated, and then the parameters of the encoder and the decoder are adjusted with reference to the similarity loss, the spatial loss, and the data loss, so as to achieve simultaneous training of the encoder and the decoder.

This method improves the efficiency of processing point cloud data, saves time and computational resources, and improves accuracy.

A block diagram of example environment100in which embodiments of the present disclosure can be implemented has been described above with reference toFIG.1. A flow chart of method200for training an encoder according to an embodiment of the present disclosure will be described below with reference toFIG.2. Method200may be executed at computing device104inFIG.1and any suitable computing device.

At block202, sample point cloud data for an object is input to an encoder to obtain encoded data for the object. For example, computing device104obtains point cloud data102as the sample point cloud data.

In some embodiments, point cloud data102may be represented by the following Equation (1):
P={Pn|n=1, . . . ,N}(1)

where Pnrepresents a vector representation of each point in the point cloud data, N is a positive integer which represents the number of points in the point cloud data, and Pnmay include their coordinates (x,y,z) and other feature information such as color, norm, etc.

At block204, by transforming the encoded data, a plurality of invariant portions for the object and a plurality of variable portions for the object are determined, an invariant portion in the plurality of invariant portions indicating an invariant feature of the object and a variable portion in the plurality of variable portions indicating a variable feature of the object. For example, if the point cloud data is specific to an aircraft, the invariant portions may indicate the shape and size of the aircraft, and the variable portions may indicate the coordinates and locations of parts of the aircraft.

In some embodiments, computing device104performs a matrix transformation on the encoded data to obtain the plurality of invariant portions and the plurality of variable portions.FIG.3illustrates a schematic diagram of example300of matrix decomposition according to an embodiment of the present disclosure. Feature vector X304is generated by the encoder from point cloud data302for the aircraft, is then subject to matrix decomposition, and is decomposed into variable portion U306and invariant portion V308.

In some embodiments, the feature vector X may be represented by the following Equation (2):
X=UV+E(2)

where X denotes the feature matrix of the point cloud, V denotes an invariant portion for the object, which may be considered as a template factor that captures important information of the data set, such as the invariant portion of the object like its shape and size, and U denotes the variable portion of the object, which may be considered as an activation factor, such as the coordinates and positions of a point in different point cloud data, etc. U∈RM×k, and V∈Rk×d, where M and d denote the dimensionality of the vector space, k is a decomposition factor, and k<d. Thus, a vector transformation may be performed on X to obtain U and V.

Returning toFIG.2for further description, at block206, based on the plurality of invariant portions and the plurality of variable portions, a similarity loss and a spatial loss for the sample point cloud data are determined. For example, computing device104uses invariant portion110and variable portion112to obtain the similarity loss and the spatial loss.

In some embodiments, computing device104determines the similarity loss based on a first invariant portion and a second invariant portion in the plurality of invariant portions. Computing device104further determines the spatial loss based on a first variable portion and a second variable portion in the plurality of variable portions. By adopting the above approach, loss data can be quickly acquired.

In some embodiments, the similarity loss of the point cloud data for the object is calculated by the following Equation (3):
Lsim=∥V1−V2∥F(3)

where ∥ ∥Fdenotes an F norm, and V1and V2are invariant portions obtained for two pieces of the point cloud data for the object.

In some embodiments, the spatial loss of the point cloud data for the object is calculated by the following Equation (4):
Lspa(P)=tr(UTWU)  (4)

where tr( ) denotes a trace of the matrix, where U is a variable portion of the feature vector, and W is a weight matrix of the 3D point cloud set, where W(m,n) is a weight between two points m and n, and where W(m,n) is calculated by the following Equation (5):

where σ denotes a parameter of control distance, exp( ) denotes an exponential function, Pm and Pn denote vector representations of points m and n in the point cloud, respectively, and where ∥ ∥22denotes a square of the 2 norm.

In some embodiments, after obtaining a plurality of invariant portions for the object and a plurality of variable portions for the object, the invariant portions and corresponding variable portions may be input to the decoder to obtain the reference point cloud data for the object. The decoder may be a neural network model or any suitable machine model. For example, the first invariant portion and the first variable portion are input to the decoder to obtain the reference point cloud data for the object. The data loss between the reference point cloud data and the sample point cloud data is then determined. At this point, in combination with the decoder, the encoder and the decoder may be trained as a whole. The data loss of the decoder is represented by the following Equation (6):

where {circumflex over (P)} is an output matrix of the decoder, N is the number of points in the point cloud, and ∥ ∥Fdenotes an F norm. The above examples are intended to describe the present disclosure only and are not further limitations to the present disclosure.

At block208, a parameter of the encoder is adjusted based on the similarity loss and the spatial loss to obtain a trained encoder. For example, computing device104uses the similarity loss and the spatial loss to adjust encoder106.

In some embodiments, computing device104combines the similarity loss and the spatial loss to obtain a combined loss. Computing device104then determines whether the combined loss is greater than a first threshold loss.

If it is determined that the combined loss is greater than the first threshold loss, the computing device adjusts the parameter of the encoder. If the combined loss is less than or equal to the first threshold loss, the parameter of the encoder is no longer adjusted, at which point the training ends.

In some embodiments, computing device104may separately determine whether the similarity loss and the spatial loss are both less than corresponding threshold losses. If the similarity loss and the spatial loss are both less than the corresponding threshold losses, the parameter of the encoder is no longer adjusted. Otherwise, the parameter of the encoder continues to be adjusted.

In some embodiments, as mentioned above, the encoder is also trained together with the decoder. In this case, computing device104adjusts the parameter of the encoder and the parameter of the decoder based on the similarity loss, the spatial loss, and the data loss. In some embodiments, the computing device combines the similarity loss, the spatial loss, and the data loss to obtain a total loss. It is then determined whether the total loss is greater than a second threshold loss. If it is determined that the total loss is greater than the second threshold loss, the parameter of the encoder and the parameter of the decoder are adjusted. If it is determined that the total loss is less than or equal to the second threshold loss, training is stopped.

For example, when the encoder and decoder are trained together, the total loss thereof is shown in the following Equation (7):
L=Lsim+αΣi=12Lspai+βΣi=12Ldeci(7)

where α and β denote the weights for controlling the loss function.

In some embodiments, when the encoder and decoder are trained together, the similarity loss, the spatial loss, and the data loss may be compared with their corresponding threshold losses, respectively. If the similarity loss, the spatial loss, and the data loss are all less than the corresponding threshold losses, training is stopped. If the above condition is not met, the parameters of the encoder and decoder continue to be adjusted. The above examples are intended to describe the present disclosure only and are not specific limitations to the present disclosure. The person skilled in the art can set any suitable way to adjust the parameters of the encoder and/or decoder using the obtained losses as needed.

This method improves the efficiency of processing point cloud data, saves time and computational resources, and improves accuracy.

The method for training the encoder is described above in conjunction withFIGS.2and3, and an example process for training the encoder and decoder is described below in conjunction withFIGS.4and5.

FIG.4illustrates a schematic diagram of example process400for training an encoder. As shown inFIG.4, two pieces of point cloud data402and404are input to encoder406to obtain two corresponding feature vectors: feature vector X1408and feature vector X2410, which are then transformed to obtain corresponding variable portion u1412and invariant portion v1416, and variable portion u2414and invariant portion v2420. Similarity loss418is then calculated using v1and v2, and the spatial loss is calculated using u1and u2, so that the encoder can be adjusted using the similarity loss and the spatial loss.

FIG.5further illustrates a schematic diagram of example500of using the decoder and encoder together for training. As shown inFIG.5, after variable portion u1502and invariant portion v1504are obtained, they are input to decoder506. Decoder506is then used to generate reference point cloud data508, and then the data error, e.g., the mean square error (MSE) loss510, is calculated for reference point cloud data508and sample point cloud data512. The parameters of the encoder and the decoder are then adjusted using the similarity loss, the spatial loss, and the data error together. The above examples are for describing the present disclosure only and are not intended to be a specific limitation of the present disclosure.

A flow chart of method600for processing data according to an embodiment of the present disclosure is described below in conjunction withFIG.6, which depicts the application of an encoder.

At block602, point cloud data for an object is input to a trained encoder to obtain encoded data for the object, the trained encoder being obtained by adjusting a parameter of an encoder based on a similarity loss and a spatial loss obtained for sample point cloud data for a sample object. For example, computing device104will encode, using the trained encoder106, the point cloud data of the to-be-processed object.

At block604, by transforming the encoded data, an invariant portion for the object and a variable portion for the object are determined, the invariant portion indicating an invariant feature of the object and the variable portion indicating a variable feature of the object. For example, computing device104performs a matrix transformation on the encoded data to obtain the invariant portion and the variable portion.

In some embodiments, computing device104may apply the variable portion for the object to other objects.

FIG.7illustrates a schematic diagram of example700of application of data. As shown inFIG.7, variable portion u1for an individual is obtained at user equipment702after processing the point cloud data for the individual using the trained encoded data. Then, u1is sent to cloud704so as to be applied to the decoder in combination with invariant portion v1of an avatar, and then the avatar including the variable content of the individual is obtained. The above examples are intended to describe the present disclosure only and are not specific limitations to the present disclosure.

This method improves the efficiency of processing point cloud data, saves time and computational resources, and improves accuracy.

FIG.8is a schematic block diagram of example device800that can be used to implement an embodiment of the present disclosure. Computing device104inFIG.1may be implemented using device800. As shown in the figure, device800includes central processing unit (CPU)801, which may execute various appropriate actions and processing according to computer program instructions stored in read-only memory (ROM)802or computer program instructions loaded from storage unit808onto random access memory (RAM)803. Various programs and data required for the operation of device800may also be stored in RAM803. CPU801, ROM802, and RAM803are connected to each other through bus804. Input/Output (I/O) interface805is also connected to bus804.

A plurality of components in device800are connected to I/O interface805, including: input unit806, such as a keyboard and a mouse; output unit807, such as various types of displays and speakers; storage unit808, such as a magnetic disk and an optical disc; and communication unit809, such as a network card, a modem, and a wireless communication transceiver. Communication unit809allows device800to exchange information/data with other devices via a computer network, such as the Internet, and/or various telecommunication networks.

The various processes and processing described above, for example, methods200and600, can be performed by CPU801. For example, in some embodiments, method200and/or method600may be implemented as a computer software program that is tangibly included in a machine-readable medium, such as storage unit808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device800via ROM802and/or communication unit809. When the computer program is loaded into RAM803and executed by CPU801, one or more actions of methods200and600described above may be performed.

Embodiments of the present disclosure include a method, an apparatus, a system, and/or a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of the present disclosure are loaded.

Embodiments of the present disclosure have been described above. The above description is illustrative, rather than exhaustive, and is not limited to the disclosed various embodiments. Numerous modifications and alterations will be apparent to persons of ordinary skill in the art without departing from the scope and spirit of the illustrated embodiments. The selection of terms as used herein is intended to best explain the principles and practical applications of the various embodiments or technical improvements to technologies on the market, so as to enable persons of ordinary skill in the art to understand the embodiments disclosed herein.