METHOD, ELECTRONIC DEVICE, AND COMPUTER PROGRAM PRODUCT FOR DATA PROCESSING

Embodiments disclosed herein include a method, an electronic device, and a computer program product for data processing. The method includes determining a first set of feature vectors representing samples in a data set. The method also includes generating a second set of feature vectors by performing a first transformation on the first set of feature vectors, wherein distribution skewness of the second set of feature vectors in a feature space is smaller than that of the first set of feature vectors. The method also includes generating a third set of feature vectors by performing a second transformation on the second set of feature vectors, wherein the third set of feature vectors and the second set of feature vectors have different distances between vectors. The method also includes selecting target samples as representatives from the samples based on a distribution of the third set of feature vectors in the feature space.

The present application claims priority to Chinese Patent Application No. 202110839222.2, filed Jul. 23, 2021, and entitled “Method, Electronic Device, and Computer Program Product for Data Processing,” which is incorporated by reference herein in its entirety.

FIELD

Embodiments of the present disclosure relate to the field of data processing, and in particular, to a method, an electronic device, and a computer program product for data processing.

BACKGROUND

A machine learning model needs to use substantial amounts of data for training. Very large data sets may consume excessive computing resources during training. Moreover, labeling a large amount of data will also consume excessive labor. Therefore, there is a need for a method which can distill data sets to train a machine learning model using distilled small data sets, thereby reducing resource consumption and improving the efficiency of training.

SUMMARY

In a first aspect of the present disclosure, a method for data processing is provided. The method includes determining a first set of feature vectors representing samples in a data set. The method also includes generating a second set of feature vectors by performing a first transformation on the first set of feature vectors, wherein distribution skewness of the second set of feature vectors in a feature space is smaller than that of the first set of feature vectors. The method also includes generating a third set of feature vectors by performing a second transformation on the second set of feature vectors, wherein the third set of feature vectors and the second set of feature vectors have different distances between vectors. The method also includes selecting target samples as representatives from the samples based on a distribution of the third set of feature vectors in the feature space.

In a second aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein that, when executed by the processor, cause the device to execute actions. The actions include determining a first set of feature vectors representing samples in a data set. The actions also include generating a second set of feature vectors by performing a first transformation on the first set of feature vectors, wherein distribution skewness of the second set of feature vectors in a feature space is smaller than that of the first set of feature vectors. The actions also include generating a third set of feature vectors by performing a second transformation on the second set of feature vectors, wherein the third set of feature vectors and the second set of feature vectors have different distances between vectors. The actions also include selecting target samples as representatives from the samples based on a distribution of the third set of feature vectors in the feature space.

In a third aspect of the present disclosure, a computer program product is provided which is tangibly stored on a computer-readable medium and includes machine-executable instructions, wherein the machine-executable instructions, when executed, cause a machine to perform the method according to the first aspect.

In the embodiments of the present disclosure, by means of the solution of data processing of the present application, representative target samples can be selected from samples of a data set, thereby improving the efficiency of training.

This Summary is provided to introduce the selection of concepts in a simplified form, which will be further described in the Detailed Description below. The Summary is neither intended to identify key features or main features of the embodiments of the present disclosure, nor intended to limit the scope of the present disclosure.

DETAILED DESCRIPTION

Principles of the embodiments of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. Although illustrative embodiments of the present disclosure are illustrated in the accompanying drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and then implement the embodiments of the present disclosure, and are not intended to impose any limitation to the scope of the present disclosure.

The term “include” and variants thereof used herein mean open-ended inclusion, that is, “include but not limited to.” Unless specifically stated, the term “or” means “and/or”. The term “based on” means “based at least in part on.” The terms “an example embodiment” and “some embodiments” mean “at least one example embodiment.” The term “another embodiment” means “at least one additional embodiment.” The terms “first,” “second,” and the like may refer to different or identical objects. Other explicit and implicit definitions may also be included below.

As mentioned above, there is a need of a method which can distill data sets to train machine learning models using distilled small data sets. Conventional methods for distilling data sets are computationally intensive and have poor interpretability.

Embodiments of the present disclosure provide a solution for data processing. In this solution, a first set of feature vectors representing samples in a data set is determined. The solution also includes generating a second set of feature vectors by performing a first transformation on the first set of feature vectors, wherein distribution skewness of the second set of feature vectors in a feature space is smaller than that of the first set of feature vectors. The solution also includes generating a third set of feature vectors by performing a second transformation on the second set of feature vectors, wherein the third set of feature vectors and the second set of feature vectors have different distances between vectors. The solution also includes selecting target samples as representatives from the samples based on a distribution of the third set of feature vectors in the feature space. In this way, it is possible to select representative target samples from the samples of the data set for training, thereby improving the efficiency of training.

The basic principles and several example embodiments of the present disclosure are described below with reference toFIGS.1to5. It should be understood that these example embodiments are given only to enable those skilled in the art to better understand and then implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

FIG.1illustrates environment100in which embodiments of the present disclosure may be implemented. As shown inFIG.1, environment100includes computing device110, samples120and target samples140in a data set. Data processing module130is deployed in computing device110. Computing device110includes any computing device in the form of a general-purpose computing device. In some implementations, computing device110may be implemented as various user terminals or service terminals having computing capabilities. The service terminals may be servers provided by various service providers, large-scale computing devices, and the like. For example, the user terminals may be any type of mobile, fixed, or portable terminals, including a mobile phone, a site, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio/video player, a digital camera/camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of such devices, or any combination thereof.

Components of computing device110may include, but are not limited to, one or more processors or processing units, memories, storage devices, one or more communication units, one or more input devices, and one or more output devices. These components may be integrated on a single device or provided in the form of a cloud computing architecture. In the cloud computing architecture, these components may be remotely arranged and may work together to achieve the functions described in the present disclosure. In some implementations, cloud computing provides computing, software, data access, and storage services, which do not require terminal users to know physical locations or configurations of systems or hardware which provide these services. In various implementations, cloud computing provides services via a wide area network (e.g., the Internet) with appropriate protocols. For example, a cloud computing provider provides applications through a wide area network, and they are accessible through a web browser or any other computing components. Software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote location. Computing resources in a cloud computing environment may be merged at a remote data center location, or they may be dispersed. Cloud computing infrastructures can provide services through a shared data center, even if they are each represented as a single access point for users. Therefore, the components and functions described herein may be provided from a service provider at a remote location by using the cloud computing architecture. Alternatively, they may also be provided from a conventional server, or they may be installed on a client terminal device directly or in other manners.

Computing device110may utilize data processing module130to implement a data processing method according to an embodiment of the present disclosure. As shown inFIG.1, computing device110receives samples120. Computing device110may receive samples120from other computing devices or storage devices through its input device. Samples120may be samples in a data set for training. Samples120may be in various forms, such as pictures, audio, video, and values. The data set may also include labels corresponding to samples120. Examples of the labels may include categories of the pictures, the speaker identifier of the audio, and the like. Computing device110may utilize data processing module130to select target samples140from samples120. The number of target samples140is smaller than the number of samples120. Target samples140may be samples that are representative of samples120. Target samples140may replace samples120for training to reduce the amount of training data while ensuring high training accuracy, thereby improving the efficiency of training. With the characteristic of representing samples120in a large data set, target samples140can be used for few-shot learning, less than one-shot learning, and the like. The details of selecting target samples140through data processing module130will be described below with reference toFIGS.2-4.

It should be understood that environment100shown inFIG.1is merely illustrative and should not constitute any limitation to the functions and scope of the implementations described in the present disclosure. For example, computing device110may also receive samples120from a storage device integrated therewith.

FIG.2illustrates an architecture diagram of system200for data processing according to an embodiment of the present disclosure. System200may be implemented in computing device110shown inFIG.1. As shown inFIG.2, system200may include pre-training model210, first transformation module220, second transformation module230, sampling module240, and target sample selection module250.

Pre-training model210receives samples120in a data set and determines first set of feature vectors260representing samples120.FIG.2shows an example of samples120being pictures. Samples120can include multiple pictures, for example, hundreds of thousands to millions of pictures. Pre-training model210may be any suitable pre-training model, and the scope of the present disclosure is not limited in this regard. Pre-training model210is used to determine first set of feature vectors260representing samples120. Each sample has a corresponding feature vector. For example, pre-training model210may be a deep neural network model, a convolutional neural network model, and the like.

First set of feature vectors260have a distribution in a feature space.FIG.2shows an example of the distribution of first set of feature vectors260. It should be understood that the two-dimensional distribution shown inFIG.2is merely schematic. The actual distribution in the feature space depends on the dimension of first set of feature vectors260. The distribution of first set of feature vectors260can be related to categories of the samples. As shown inFIG.2, the distribution of first set of feature vectors260may be divided into (e.g., two) relatively separate distributions depending on the categories of samples120(e.g., two categories). It should be understood that, depending on the performance of pre-training model210, the distribution of first set of feature vectors260may not be shown as completely separated independent distributions.

First transformation module220generates second set of feature vectors270by performing a first transformation on first set of feature vectors260. The distribution skewness of second set of feature vectors270in the feature space is smaller than that of first set of feature vectors260. As shown inFIG.2, the distribution of first set of feature vectors260in the feature space may be irregular. The regularity of distribution can be characterized by distribution skewness. First transformation module220reduces the distribution skewness of first set of feature vectors260by performing a first transformation on first set of feature vectors260. In some embodiments, first transformation module220may reduce the distribution skewness, in the feature space, of feature vectors in first set of feature vectors260that are associated with samples120of the same category in the data set. In other words, first transformation module220may reduce the distribution skewness of the samples of each category.

The first transformation can be any suitable transformation capable of reducing the distribution skewness of the feature vectors. In some embodiments, the first transformation can be exponentiation transformation or logarithm transformation. For example, the first transformation may be performed on first set of feature vectors260according to the Tukey power transformation described according to formula (1).

where λ is a hyper-parameter used to control the implementation of the first transformation. The smaller λ is, the smaller the positive skewness of the distribution is (less positively skewed). Conversely, the larger λ is, the greater the positive skewness of the distribution is. When λ is 1, first set of feature vectors260maintains the original distributed skewness.

As shown inFIG.2, second set of feature vectors270determined via the first transformation has smaller distribution skewness than first set of feature vectors260, i.e., the distribution of second set of feature vectors270is more regular. For example, the distribution of second set of feature vectors270may be a more regular Gaussian-like distribution.

Second transformation module230generates third set of feature vectors280by performing a second transformation on second set of feature vectors270. Third set of feature vectors280and second set of feature vectors270have different distances between vectors. The distances between vectors may be Euclidean distances between vectors. The distances between vectors may also be Manhattan distances between vectors.

In some embodiments, performing the second transformation includes reducing distances between the feature vectors in second set of feature vectors270that are associated with samples120of the same category in the data set. In other words, second transformation module230may reduce distances between feature vectors of samples of the same category by performing the second transformation. For example, for second set of feature vectors270in a Gaussian-like distribution, performing the second transformation may reduce the variance of the distribution. Alternatively or additionally, performing the second transformation includes increasing distances between feature vectors in second set of feature vectors270that are associated with samples120of different categories in the data set. In other words, second transformation module230may increase distances between feature vectors of samples of different categories by performing the second transformation. For example, for second set of feature vectors270in a Gaussian-like distribution, performing the second transformation may increase a distances between a first mean value of feature vectors of samples of the first category and a second mean value of feature vectors of samples of the second category.

The second transformation may be any suitable transformation capable of adjusting a distance between feature vectors. In some embodiments, distances between second set of feature vectors270may be adjusted based on a potential energy minimization method for adjusting the distance between molecules. For example, the relationship between potential energy and a distance between two feature vectors may be represented by using formula (2).

where E represents potential energy; and r represents the distance between feature vectors.FIG.3illustrates a schematic diagram of the relationship between potential energy and the distance between vectors described in the formula (2). As shown inFIG.3, when the distance between vectors is r0, potential energy E is the smallest. Therefore, the distance r0between the two vectors may be referred as the optimal distance.

In some embodiments, the optimal distance between second set of feature vectors270may be determined by using formula (3).

where L is a loss function; N is the number of samples120; dij=dis(WTfi, WTfj) indicates the distance between vector WTfiand vector WTfjand dis( ) indicates a function calculating the distance between vector WTfiand vector WTfj, for example, a function for calculating the Euclidean distance; WTis a weight matrix for performing a second transformation on second set of feature vectors270, and is a learnable parameter; and λ, is a hyper-parameter (different from λ in the formula (1)) that can depend on categories of the samples. For sample i and sample j of the same category, large λ (e.g., 10) may be set. Conversely, small λ (e.g., 1) can be set for sample i and sample j of different categories.

Still referring toFIG.2, second transformation module230may determine the value of weight matrix WTby minimizing loss function L which is based on second set of feature vectors270, weight matrix WT, and categories of samples120. Second transformation module230may determine products of second set of feature vectors270and the determined weight matrix WTas third set of feature vectors280. As described above, third set of feature vectors280and second set of feature vectors270have different distances between vectors. Compared with second set of feature vectors270, the distances between feature vectors of the same category in third set of feature vectors280may be smaller, and the distances between feature vectors of different categories in third set of feature vectors280may be larger. As shown inFIG.2, compared with second set of feature vectors270, the variance of Gaussian-like distribution of feature vectors associated with samples of the same category in third set of feature vectors280is reduced, and the distance between the first mean value of feature vectors of samples of the first category and the second mean value of feature vectors of samples of the second category is increased.

Sampling module240can select target feature vectors290from third set of feature vectors280based on determined third set of feature vectors280. Target samples140corresponding to target feature vectors290may be used to represent samples120. Sampling module240may sample target feature vectors290based on the distribution of third set of feature vectors280. Target feature vectors290may include at least one feature vector. Sampling module240may select feature vectors at the center of the distribution of third set of feature vectors280as target feature vectors290. Alternatively or additionally, sampling module240may select feature vectors located at the edge of the distribution of third set of feature vectors280as target feature vectors290. An example of target feature vectors290is shown in small circles inFIG.2. As shown inFIG.2, target feature vectors290may be located at the center or edge of the distribution.

In some embodiments, sampling module240may determine the mean value of feature vectors in third set of feature vectors280that are associated with samples120of the first category in the data set. Based on the mean value, sampling module240may sample target feature vectors290from the feature vectors associated with the samples of the first category. Sampling module240may select a feature vector in third set of feature vectors280that is the closest to the mean value as a center feature vector in target feature vectors290. Sampling module240may calculate a feature vector in third set of feature vectors280that is the farthest from the center feature vector as an edge feature vector in target feature vectors290.

Alternatively or additionally, sampling module240may utilize the unscented Kalman filter (UKF) algorithm to sample target feature vectors290based on the mean value. For the samples of each category, target feature vectors290may be sampled in accordance with formula (4).

where S[i] represents sampled target feature vectors290; μ represents a mean value of feature vectors in third set of feature vectors280that are associated with samples of one category; V=√{square root over ((n+λ)Σ)}, represents a variance matrix; Virepresents the ithcolumn of the variance matrix; n represents the dimension of the feature vectors; λ is a presettable zoom parameter ((different from λ in formulas (1) and (2)), indicating a distance between a sampling point and the center of the distribution; and Σ is a covariance matrix of third set of feature vectors280.

Based on target feature vectors290, target sample selection module250may determine the samples associated with target feature vectors290as target samples140representing samples120. Specifically, for each category, target sample selection module250may determine the samples associated with target feature vectors290as target samples140which represent samples120of the corresponding category. As shown inFIG.2, target sample selection module250may select target samples140(shown in large circles) from samples120.

Alternatively or additionally, sampling module240may determine an estimated distribution of third set of feature vectors280based on target feature vectors290. Specifically, sampling module240may determine, based on target feature vectors290for each category, an estimated distribution of feature vectors in third set of feature vectors280that are associated with samples of the corresponding category. The estimated distribution can be represented by a mathematical expression. For example, for a Gaussian-like distribution, the estimated distribution may be represented by a mean value and a variance. Based on the mathematical expression of the estimated distribution, sampling module240may determine additional target feature vectors. The additional target feature vectors may not be the same as any feature vector in third set of feature vectors280, but may represent third set of feature vectors280. Additional samples may be generated based on the additional target feature vectors by using additional neural network modules. The additional samples may be used to represent samples120.

FIG.4illustrates a flow chart of example method400for data processing according to some embodiments of the present disclosure. Method400may be implemented, for example, in environment100as shown inFIG.1. It should be understood that method400may further include additional actions which are not shown and/or may omit actions which are shown. The scope of the present disclosure is not limited in this regard.

At block410, a first set of feature vectors representing samples120in a data set is determined. First set of feature vectors260may be determined by using pre-training model210.

At block420, second set of feature vectors270is generated by performing a first transformation on first set of feature vectors260, and distribution skewness of second set of feature vectors270is smaller than that of first set of feature vectors260. In some embodiments, performing the first transformation includes reducing distribution skewness, in a feature space, of feature vectors in first set of feature vectors260that are associated with samples of the same category in the data set. In some embodiments, the first transformation includes exponentiation transformation or logarithm transformation.

At block430, third set of feature vectors280is generated by performing a second transformation on second set of feature vectors270, and third set of feature vectors280and second set of feature vectors270have different distances between vectors. In some embodiments, performing the second transformation includes at least one of the following: reducing distances between feature vectors in second set of feature vectors270that are associated with samples of the same category in the data set; and increasing distances between feature vectors in second set of feature vectors270that are associated with samples of different categories in the data set. In some embodiments, performing the second transformation includes: determining a weight matrix by minimizing a loss function which is based on second set of feature vectors270, the weight matrix, and categories of the samples; and determining products of second set of feature vectors270and the determined weight matrix as third set of feature vectors280.

At block440, target samples140as representatives are selected from the samples based on a distribution of third set of feature vectors280in the feature space. In some embodiments, selecting target samples140based on the distribution of third set of feature vectors280in the feature space includes: determining a mean value of feature vectors in third set of feature vectors280that are associated with samples of a first category in the data set; based on the mean value, sampling target feature vectors290from the feature vectors associated with the samples of the first category; and determining samples associated with the target feature vectors as target samples140representing the samples of the first category. In some embodiments, sampling target feature vectors290based on the mean value includes: sampling target feature vectors290based on the mean value by means of the unscented Kalman filter algorithm.

In some embodiments, the method further includes determining, based on the target feature vectors, an estimated distribution of feature vectors in third set of feature vectors280that are associated with the samples of the first category in the data set; determining additional target feature vectors based on the estimated distribution; and determining additional target samples for representing the samples of the first category based on the additional target feature vectors.

In this way, it is possible to select representative and fewer target samples140from samples120of the data set for training, thereby reducing resource consumption to improve the efficiency of training. Further, with the characteristic of representing samples120in a large data set, target samples140can be used to migrate learning, few-shot learning, less than one-shot learning, etc.

FIG.5illustrates a schematic block diagram of example device500which can be used to implement embodiments of the present disclosure. For example, device500may be implemented at environment100as shown inFIG.1. As shown inFIG.5, device500includes central processing unit (CPU)501which may perform various appropriate actions and processing according to computer program instructions stored in read-only memory (ROM)502or computer program instructions loaded from storage unit508to random access memory (RAM)503. Various programs and data required for the operation of device500may also be stored in RAM503. CPU501, ROM502, and RAM503are connected to each other through bus504. Input/output (I/O) interface505is also connected to bus504.

A plurality of components in device500are connected to I/O interface505, including: input unit506, such as a keyboard and a mouse; output unit507, such as various types of displays and speakers; storage unit508, such as a magnetic disk and an optical disc; and communication unit509, such as a network card, a modem, and a wireless communication transceiver. Communication unit509allows device500to exchange information/data with other devices via a computer network, such as the Internet, and/or various telecommunication networks.

Various processes and processing described above, for example, method400, may be performed by CPU501. For example, in some embodiments, method400may be implemented as computer software programs which are tangibly included in a machine-readable medium such as storage unit508. In some embodiments, part or all of the computer programs may be loaded and/or installed to device500via ROM502and/or communication unit509. When the computer program is loaded into RAM503and executed by CPU501, one or more actions of method400described above may be performed.

Illustrative embodiments of the present disclosure include a method, an apparatus, a system, and/or a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of the present disclosure are loaded.

Example embodiments of the present disclosure have been described above. The above description is illustrative, rather than exhaustive, and is not limited to the disclosed various embodiments. Numerous modifications and alterations are apparent to those of ordinary skill in the art without departing from the scope and spirit of the illustrated embodiments. The selection of terms used herein is intended to best explain the principles and practical applications of the various embodiments or the improvements to technologies on the market, so as to enable persons of ordinary skill in the art to understand the embodiments disclosed here.