Patent Description:
Automation of a recognition process has been implemented using, for example, a neural network model implemented by a processor as a special calculation structure, which may provide a computationally intuitive mapping between an input pattern and an output pattern after considerable training. An ability to be trained to generate such mapping may be referred to as a "training ability of a neural network. " Moreover, due to specialized training, such a specialized and trained neural network may have a generalization ability to generate a relatively accurate output for an input pattern that is not trained.

<NPL>" describes a rollable latent space (RLS) for azimuth-invariant SAR target recognition, addressing issues of scarce labeled data and limited viewing angles. RLS enables latent feature rolling to infer arbitrary views, allowing data augmentation. The study evaluates RLS-based classifiers, with and without augmentation, against a conventional classifier trained on front shots, testing their performance on untrained back shots. <NPL>" describes a data augmentation approach using an adversarial autoencoder (AAE) to impose a uniform feature distribution and apply latent space interpolation (LSI). <NPL>" describes a method for facial image representation using a Supervised Autoencoder. It combines various descriptors (Viola-Jones, LBP, HOG, Gabor, Curvelet, and Wavelet) for feature extraction, uses a Stacked Autoencoder for transformation, and applies a Linear SVM for classification.

Although terms of "first" or "second" are used to explain various components, the components are not limited to the terms. These terms should be used only to distinguish one component from another component. For example, a "first" component may be referred to as a "second" component, or similarly, and the "second" component may be referred to as the "first" component within the scope of the right according to the concept of the present disclosure.

As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components or a combination thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

<FIG> illustrates an example of a configuration and an operation of an image processing apparatus <NUM>. Referring to <FIG>, the image processing apparatus <NUM> may include a feature extraction model <NUM> and a prediction model <NUM>, and may generate a prediction result <NUM> for an input image <NUM> using the feature extraction model <NUM> and the prediction model <NUM>. The feature extraction model <NUM> and the prediction model <NUM> may each correspond to a neural network model, and the image processing apparatus <NUM> may perform neural network model-based prediction. For example, prediction for the input image <NUM> may include detection, tracking, recognition, identification, classification, authentication, and verification of an object in the input image <NUM>. In an example, the prediction result <NUM> may include eye position information according to eye detection and/or eye tracking, identity and/or category information according to object recognition, identification, and/or classification, and authentication information according to user authentication and/or verification.

A neural network model may correspond to a deep neural network (DNN) including a plurality of layers. The plurality of layers may include an input layer, a hidden layer, and an output layer. The neural network model may include a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, a portion of a plurality of layers in the neural network model may correspond to a CNN, and another portion of the layers may correspond to an FCN. In this example, the CNN may be referred to as a "convolution layer" and the FCN may be referred to as a "fully connected layer.

In the CNN, data input to each layer may be referred to as an "input feature map" and data output from each layer may be referred to as an "output feature map". The input feature map and the output feature map may also be referred to as activation data. When a convolutional layer corresponds to an input layer, an input feature map of the input layer may correspond to the input image <NUM>.

The neural network model may be trained based on deep learning, and may perform inference suitable for the purpose of training, by mapping input data and output data that are in a nonlinear relationship. The deep learning may be a machine learning scheme for solving an issue such as image or voice recognition from a big data set. The deep learning may be understood as a process of solving an optimization issue to find a point at which energy is minimized while training the neural network model based on prepared training data.

Through supervised or unsupervised learning of the deep learning, a structure of the neural network model or a weight corresponding to a model may be obtained, and input data and output data of the neural network model may be mapped to each other through the weight. For example, when a width and a depth of the neural network model are sufficiently large, the neural network model may have a capacity large enough to implement an arbitrary function. When the neural network model is trained on a sufficiently large quantity of training data through an appropriate training process, optimal performance may be achieved.

In the following description, the neural network model may be expressed as being "pre-trained", where "pre-" may indicate a state before the neural network model is "started". The "started" neural network model may indicate that the neural network model is ready for inference. For example, "start" of the neural network model may include a loading of the neural network model in a memory, or an input of input data for inference to the neural network model after the neural network model is loaded in the memory.

The feature extraction model <NUM> may extract a feature from the input image <NUM>, and the prediction model <NUM> may generate the prediction result <NUM> corresponding to the extracted feature. For example, the feature extraction model <NUM> and the prediction model <NUM> may correspond to a CNN and an FCN, respectively. In training and/or inference of the neural network model, diversity of input data and/or training data may have an influence on an accuracy of the prediction result <NUM>. Data augmentation may be a technology of diversifying training data through transformations, for example, a geometric transformation or a color transformation. Through the data augmentation, overfitting may be inhibited.

The image processing apparatus <NUM> may perform prediction using a data augmentation technology to secure the diversity of training images in a training operation. For example, various augmented training images may be secured through data augmentation, a feature may be extracted from each of augmented training images, and prediction may be performed. As a result, the neural network model may be updated. For the above training scheme, a feature may need to be extracted from each of the augmented training images, which may require a considerably larger amount of computation than other operations of a prediction process.

Feature augmentation according to examples may be applied to the neural network model of the image processing apparatus <NUM>. The feature augmentation may be a kind of data augmentation, but may be used to augment a feature instead of an image, unlike the conventional data augmentation. For example, when the feature extraction model <NUM> extracts an input feature from the input image <NUM>, augmented features of the input feature may be generated through the feature augmentation. The prediction model <NUM> may generate the prediction result <NUM> based to the augmented features.

Unlike the conventional data augmentation, in the feature augmentation, an operation of extracting a feature from an image is not repeated. In an example of data augmentation, to obtain "N" augmented features, a CNN may need to be executed "N" times. In the feature augmentation, the CNN may be executed once, and a feature extracted by executing the CNN once may be augmented "N" times, to obtain "N" augmented features. Thus, through the feature augmentation, an amount of computation may be significantly reduced.

Due to a relatively small amount of computation, the feature augmentation may be used for inference in addition to training. The existing data augmentation is used mainly for training due to a relatively large amount of computation, however, the feature augmentation may require a relatively small amount of computation to be used even in an inference operation. Thus, in both the training and the inference, an accuracy of prediction may be enhanced by augmentation technologies.

<FIG> illustrates a conventional data augmentation method according to a related art, and <FIG> illustrates an example of a feature augmentation method. Referring to <FIG>, a data augmentation model <NUM> may generate augmented images <NUM> based on an image <NUM> and transformation parameters <NUM>. The data augmentation model <NUM> may be a neural network model. The transformation parameters <NUM> may indicate various transformations. Transformations may include, for example, scaling, cropping, flipping, padding, rotation, translation, color transformation, brightness transformation, contrast transformation, and noise addition. The transformation parameters <NUM> may specify a transformation to be applied among the transformations and a number of transformations to be applied.

The same number of augmented images <NUM> as a number of transformation parameters <NUM> may be generated. For example, when the number of transformation parameters <NUM> is "N", the number of augmented images <NUM> may also be "N". A feature extraction model <NUM> may extract a feature from each of the augmented images <NUM> and may generate features <NUM>. When the number of augmented images <NUM> is "N", the feature extraction model <NUM> may be executed "N" times, and accordingly "N" features <NUM> may be generated. A prediction model <NUM> may perform prediction based on the features <NUM>. When the above operation corresponds to a training operation of an image processing model <NUM>, the image processing model <NUM> may be updated by a prediction result. When the above operation corresponds to an inference operation, the prediction result may be output as an inference result. As described above, due to a computation load caused by repetitive execution of the feature extraction model <NUM>, the data augmentation method of <FIG> may be mainly used in a training operation.

Referring to <FIG>, an image processing model <NUM> may include a feature augmentation model <NUM>, instead of the data augmentation model <NUM> of <FIG>. A feature extraction model <NUM> may extract a feature <NUM> from an image <NUM>. The feature augmentation model <NUM> may generate augmented features <NUM> based on the feature <NUM> and transformation parameters <NUM>. The feature augmentation model <NUM> may be a neural network model. The transformation parameters <NUM> may correspond to the transformation parameters <NUM> of <FIG>. The same number of augmented features <NUM> as a number of transformation parameters <NUM> may be generated. For example, when the number of transformation parameters <NUM> is "N", the number of augmented features <NUM> may also be "N". Thus, although the feature extraction model <NUM> is executed once, the "N" augmented features <NUM> may be generated.

A prediction model <NUM> may perform prediction based on the augmented features <NUM>. In an example, when the above operation corresponds to a training operation of the image processing model <NUM>, the image processing model <NUM> may be updated by a prediction result. In another example, when the above operation corresponds to an inference operation, the prediction result may be output as an inference result. As described above, since a computation load is significantly reduced by a decrease in a number of times the feature extraction model <NUM> is executed, the feature augmentation method may be used in both the training operation and the inference operation.

<FIG> illustrates an image processing method through feature augmentation. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer, such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>. Thus, the above description may not be repeated here.

Referring to <FIG>, in operation <NUM>, an image processing apparatus extracts an input feature from an input image. For example, the image processing apparatus may execute a CNN using the input image, and may obtain an input feature corresponding to an execution result.

In operation <NUM>, the image processing apparatus generates a plurality of augmented features by augmenting the input feature. For example, the image processing apparatus may execute a neural network model based on the input feature and a first transformation code and may generate a first augmented feature corresponding to the first transformation code. Also, the image processing apparatus may execute the neural network model based on the input feature and second transformation code and may generate a second augmented feature corresponding to the second transformation code. The neural network model may be a feature augmentation model, and the first transformation code and the second transformation code may correspond to transformations based on different transformation parameters.

In operation <NUM>, the image processing apparatus generates a prediction result based on the plurality of augmented features. The image processing apparatus may generate a plurality of partial prediction results based on the plurality of augmented features, may fuse the partial prediction results, and may generate the prediction result. For example, the image processing apparatus may generate the prediction result based on a fusion of a first partial prediction result according to the first augmented feature and a second partial prediction result according to the second augmented feature.

<FIG> illustrates a configuration and an operation of a feature augmentation model <NUM>. Referring to <FIG>, a feature extraction model <NUM> may extract an input feature <NUM> from an input image <NUM>. The feature augmentation model <NUM> may generate an augmented feature <NUM> based on the input feature <NUM> and a transformation parameter <NUM>. A prediction model <NUM> may generate a prediction result <NUM> based on the augmented feature <NUM>.

The feature augmentation model <NUM> includes an encoding model <NUM> and a decoding model <NUM>. The encoding model <NUM> and the decoding model <NUM> may be neural network models. For example, each of the encoding model <NUM> and the decoding model <NUM> may correspond to an FCN. The encoding model <NUM> may encode the input feature <NUM> to the latent feature <NUM>. An image processing apparatus may transform the latent feature <NUM> based on the transformation parameter <NUM>. For example, the image processing apparatus may generate a transformation code <NUM> corresponding to the transformation parameter <NUM>, may combine the latent feature <NUM> with the transformation code <NUM> through a combination operation (for example, a concatenation operation) of a block <NUM>, and may determine the combined feature. The image processing apparatus may decode the combined feature to the augmented feature <NUM> using the decoding model <NUM>.

The transformation code <NUM> may be generated by converting the transformation parameter <NUM> into a form that may be processed in a neural network model and/or a form that may be combined with the latent feature <NUM>. For example, the transformation code <NUM> may be in a form of a vector that may be adopted for convenience of training and inference of the neural network model. In an example, when the transformation parameter <NUM> is in a form of data that may be combined with the latent feature <NUM>, the transformation code <NUM> may not be generated. In this example, the transformation parameter <NUM> may be used as a transformation code.

The transformation parameter <NUM> indicates a type of transformation and/or a degree of transformation. The transformation code <NUM> includes a first field indicating the type of transformation and a second field indicating the degree of transformation. For example, the second field may have a form of a one-hot vector, and the degree of transformation may be specified based on which bit in a field has a value of "<NUM>". For example, the type of transformation indicated by the first field may be specified as translation, and a direction and a degree of translation indicated by the second field may be specified. For example, when the second field is a c-bit, "c/<NUM>" upper bits may indicate offset in an x-axis direction, and "c/<NUM>" lower bits may indicate offset in a y-axis direction. Also, the transformation parameter <NUM> may indicate various transformations. The type of transformation may include, for example, scaling, cropping, flipping, padding, rotation, translation, color transformation, brightness transformation, contrast transformation, and noise addition. A value of the transformation parameter <NUM> may be randomly determined, may be determined based on a preset pattern, or may be determined depending on a characteristic of the input image <NUM>.

When the transformation parameter <NUM> includes a first transformation parameter and a second transformation parameter, the first transformation parameter and the second transformation parameter may indicate different transformations, and accordingly a first augmented feature and a second augmented feature corresponding to different features may be generated. In other words, when the input feature <NUM> is extracted from the input image <NUM>, various augmented features <NUM> may be generated by changing the transformation parameter <NUM>. In this example, the feature extraction model <NUM> may not need to be additionally executed, and the encoding model <NUM> may also not need to be further executed. To generate various augmented features <NUM>, the latent feature <NUM> may need to be combined and decoded while changing a value of the transformation parameter <NUM>. Thus, diversity of the augmented features <NUM> may be secured with a relatively small amount of computation.

<FIG> and <FIG> illustrate examples of operations of deriving a prediction result based on augmented features. Referring to <FIG>, a feature extraction model <NUM> may extract an input feature <NUM> from an input image <NUM>, and a feature augmentation model <NUM> may generate a first augmented feature <NUM>, a second augmented feature <NUM>, and a third augmented feature <NUM> based on the input feature <NUM> and transformation parameters <NUM>. For example, the transformation parameters <NUM> may include a first transformation parameter, a second transformation parameter, and a third transformation parameter. The first augmented feature <NUM> may be generated based on the input feature <NUM> and the first transformation parameter, the second augmented feature <NUM> may be generated based on the input feature <NUM> and the second transformation parameter, and the third augmented feature <NUM> may be generated based on the input feature <NUM> and the third transformation parameter. For example, when the input feature <NUM> input to the feature augmentation model <NUM> is encoded to a latent feature, the latent feature may be combined with a transformation code of each transformation parameter and the combined feature may be decoded, to generate the first augmented feature <NUM> through the third augmented feature <NUM>. Although three augmented features, for example, the first augmented feature <NUM> through the third augmented feature <NUM>, are illustrated in <FIG> and <FIG> for convenience of description, a number of augmented features may be greater than "<NUM>".

A prediction model <NUM> may generate a prediction result <NUM> based on the first augmented feature <NUM> through the third augmented feature <NUM>. Referring to <FIG>, the prediction model <NUM> may generate a first partial prediction result <NUM>, a second partial prediction result <NUM>, and a third partial prediction result <NUM>, by performing prediction for each of the first augmented feature <NUM> through the third augmented feature <NUM>. The prediction model <NUM> may be repeatedly executed by each of the first augmented feature <NUM> through the third augmented feature <NUM>. For example, the prediction model <NUM> may be executed based on the first augmented feature <NUM> to generate the first partial prediction result <NUM>, may be executed based on the second augmented feature <NUM> to generate the second partial prediction result <NUM>, and may be executed based on the third augmented feature <NUM> to generate the third partial prediction result <NUM>. The first partial prediction result <NUM> through the third partial prediction result <NUM> may be fused in a fusion block <NUM>, and accordingly the prediction result <NUM> may be generated. A fusion may include, for example, an averaging operation of the first partial prediction result <NUM> through the third partial prediction result <NUM>, and the fusion block <NUM> may be implemented as a neural network model, for example, an FCN.

<FIG> illustrates an example of a mixed scheme of data augmentation and feature augmentation. Referring to <FIG>, an image processing model <NUM> may include a primary augmentation model <NUM> and a secondary augmentation model <NUM>. The primary augmentation model <NUM> may perform data augmentation, and the secondary augmentation model <NUM> may perform feature augmentation. To reduce a computation load, diversity of the data augmentation may be limited, and instead a lack of the diversity may be supplemented by the feature augmentation. For example, a number of augmented images based on the data augmentation may be "J" and a number of augmented features based on the feature augmentation may be "K". In this example, "K" may have a significantly higher value than "J".

The primary augmentation model <NUM> may augment an input image <NUM> based on transformation parameters <NUM>, and may generate a first augmented image <NUM>, a second augmented image <NUM>, and a third augmented image <NUM>. For example, a number of augmented images, for example, the first augmented image <NUM> through the third augmented image <NUM>, may correspond to a number of transformation parameters <NUM>. Although three augmented images are shown in <FIG> for convenience of description, the number of augmented images may be varied, and may be "<NUM>", "<NUM>", or greater. For example, when the input image <NUM> is a user image, the first augmented image <NUM> may be an image representing a user's face obtained by cropping the input image <NUM>, the second augmented image <NUM> may be an image representing a portion (for example, a left eye) of the user's face by cropping the input image <NUM>, and the third augmented image <NUM> may be an image representing another portion (for example, a right eye) of the user's face by cropping the input image <NUM>. The primary augmentation model <NUM> may be a neural network model, and may generate a relatively small number of augmented images by performing data augmentation in a known manner.

A feature extraction model <NUM> may extract a feature from each of the first augmented image <NUM> through the third augmented image <NUM>, and may generate a first input feature <NUM>, a second input feature <NUM>, and a third input feature <NUM>. The feature extraction model <NUM> may be repeatedly executed by the first input feature <NUM> through the third input feature <NUM>. For example, the feature extraction model <NUM> may be executed based on the first augmented image <NUM> to generate the first input feature <NUM>, may be executed based on the second augmented image <NUM> to generate the second input feature <NUM>, and may be executed based on the third augmented image <NUM> to generate the third input feature <NUM>. To minimize a number of times the feature extraction model <NUM> is executed, a number of augmented images may be limited to a small number.

The secondary augmentation model <NUM> may augment the first input feature <NUM> through the third input feature <NUM> based on transformation parameters <NUM> and may generate a first augmented feature set <NUM>, a second augmented feature set <NUM> and a third augmented feature set <NUM>. For example, the secondary augmentation model <NUM> may be executed based on the transformation parameters <NUM> and the first input feature <NUM> to generate the first augmented feature set <NUM>, may be executed based on the transformation parameters <NUM> and the second input feature <NUM> to generate the second augmented feature set <NUM>, and may be executed based on the transformation parameters <NUM> and the third input feature <NUM> to generate the third augmented feature set <NUM>. The first augmented feature set <NUM> through the third augmented feature set <NUM> may each include a predetermined number of augmented features, and the number of augmented features may correspond to a number of transformation parameters <NUM>. For example, "K" transformation parameters <NUM> may be provided, and each of the first augmented feature set <NUM> through the third augmented feature set <NUM> may include "K" augmented features.

The first augmented feature set <NUM> through the third augmented feature set <NUM> may be input to a prediction model, although not shown in <FIG>, and accordingly a prediction result may be generated. The above description provided with reference to <FIG> and <FIG> may be applied to generation of a prediction result.

<FIG> illustrates an example of an operation of training a feature augmentation model. Referring to <FIG>, a feature extraction model <NUM> may extract a training feature <NUM> from a training image <NUM>, and a feature augmentation model <NUM> may generate an augmented feature <NUM> by transforming the training feature <NUM> based on a transformation parameter <NUM>. For example, an encoding model <NUM> may generate a latent feature <NUM> by encoding the training feature <NUM>, and the latent feature <NUM> may be combined with a transformation code <NUM> in a combination block <NUM>. A decoding model <NUM> may decode the combined feature to generate the augmented feature <NUM>. A data augmentation model <NUM> may generate an augmented image <NUM> by transforming the training image <NUM> based on the transformation parameter <NUM>, and the feature extraction model <NUM> may extract an augmented feature <NUM> from the augmented image <NUM>.

As described above, a feature augmentation operation of obtaining the augmented feature <NUM> from the training image <NUM> using the transformation parameter <NUM> may correspond to a data augmentation operation of obtaining the augmented feature <NUM> from the training image <NUM> using the transformation parameter <NUM>. Accordingly, the feature augmentation model <NUM> may be trained using the data augmentation model <NUM> that exists. For example, when the augmented features <NUM> and <NUM> are derived from the training image <NUM>, parameters of the feature augmentation model <NUM>, for example, the encoding model <NUM> and/or the decoding model <NUM>, may be updated to reduce a difference <NUM> between the augmented features <NUM> and <NUM>. In this example, the feature extraction model <NUM> may be assumed to be pre-trained, and parameters of the feature extraction model <NUM> may be fixed in a process of training the feature augmentation model <NUM>.

<FIG> illustrates an example of a training operation based on a gradient of a prediction result. Referring to <FIG>, in a block <NUM>, a transformation direction of a transformation parameter <NUM> and/or a transformation code <NUM> may be set based on a gradient of a prediction result <NUM>. The above scheme may exhibit a higher efficiency than that of a scheme of randomly transforming the transformation parameter <NUM> and/or the transformation code <NUM>.

For example, a feature augmentation model <NUM> may augment a feature <NUM> based on the transformation parameter <NUM> and may generate an augmented feature <NUM>. An encoding model <NUM> may encode the feature <NUM> to a latent feature <NUM>, and the latent feature <NUM> may be combined with the transformation code <NUM> through a combination operation (for example, a concatenation operation) of a block <NUM>. A decoding model <NUM> may decode the combined feature to the augmented feature <NUM>.

For example, the transformation parameter <NUM> may include a first transformation parameter through a third transformation parameter that may be converted into a first transformation code through a third transformation code, respectively. In this example, the first transformation code may have an arbitrary value, and a value of a second transformation code and a value of the third transformation code may be determined based on gradients of partial prediction results according to previous transformation codes, for example, the first transformation code and the second transformation code, respectively. The feature augmentation model <NUM> may generate a first augmented feature based on the feature <NUM> and the first transformation code. A prediction model <NUM> may generate a first partial prediction result based on the first augmented feature. The second transformation code may have a null value at an initial operation, and the null value may be adjusted based on a gradient of the first partial prediction result.

For example, the transformation code <NUM> may include a first field indicating a type of transformation and a second field indicating a degree of transformation. The first field may be assumed to be fixed as translation and the second field may be assumed to be adjusted. A predetermined bit in the second field may indicate a translation value in an x-axis direction, and the other bits may indicate translation values in a y-axis direction. For example, when translation values of axes of the first partial prediction result are pred_x and pred_y, gradients may be obtained for each of pred_x and pred_y. In an example, gradient vectors g_x1, g_x2, g_y1, and g_y2 may be defined in descending and ascending directions of the gradients with respect to a translation value of each axis. In this example, the gradient vector g_x1 in an ascending direction of pred_x may be calculated as shown in Equation <NUM> below.

In Equation <NUM>, ∇c denotes a gradient, and top_ <NUM>() denotes a function that sets values other than a maximum value of a vector to "<NUM>" while maintaining the maximum value. Also, sgn() denotes a function with a symbol of each value of a vector, and an output value may be {-<NUM>,<NUM>,<NUM>}. Similarly, in the same manner as in Equation <NUM>, g_x2, g_y1, and g_y2 may be calculated, and accordingly the second transformation code may be determined based on a gradient vector.

The second transformation code may be combined with the latent feature <NUM>, and the decoding model <NUM> may decode the combined feature to a second augmented feature. The prediction model <NUM> may generate a second partial prediction result based on the second augmented feature. The third transformation code may have a null value at an initial operation, and the null value may be adjusted based on a gradient of the second partial prediction result. When a third partial prediction result is generated based on the third transformation code, the prediction result <NUM> may be derived through a fusion of the first partial prediction result through the third partial prediction result. Thus, the transformation direction may be efficiently determined based on a guideline according to the gradient.

<FIG> illustrates an example of a configuration of an image processing apparatus <NUM>. Referring to <FIG>, the image processing apparatus <NUM> may include a processor <NUM> and a memory <NUM>. The memory <NUM> may be connected to the processor <NUM>, and may store instructions executable by the processor <NUM>, data to be computed by the processor <NUM>, or data processed by the processor <NUM>. The memory <NUM> may include, for example, a non-transitory computer-readable storage medium, for example, a high-speed random access memory (RAM) and/or a non-volatile computer-readable storage medium (for example, at least one disk storage device, a flash memory device, or other non-volatile solid state memory devices). Further details regarding the memory <NUM> is provided below.

The processor <NUM> may execute instructions to perform the operations described above with reference to <FIG> above and <FIG> below. For example, the processor <NUM> may extract an input feature from an input image, may generate a plurality of augmented features by augmenting the input feature, and may generate a prediction result based on the plurality of augmented features. In addition, the description of <FIG> and <FIG> is also applicable to the image processing apparatus <NUM>. Further details regarding the processor <NUM> is provided below.

<FIG> illustrates an example of a configuration of an electronic apparatus <NUM>. Referring to <FIG>, the electronic apparatus <NUM> may include a processor <NUM>, a memory <NUM>, a camera <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>. The processor <NUM>, the memory <NUM>, the camera <NUM>, the storage device <NUM>, the input device <NUM>, the output device <NUM>, and the network interface <NUM> may communicate with each other via a communication bus <NUM>. For example, the electronic apparatus <NUM> may be implemented as at least a portion of, for example, a mobile device such as a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer or a laptop computer, a wearable device such as a smartwatch, a smart band or smart glasses, a computing device such as a desktop or a server, home appliances such as a television (TV), a smart TV or a refrigerator, a security device such as a door lock, or a vehicle such as an autonomous vehicle or a smart vehicle. The electronic apparatus <NUM> may structurally and/or functionally include the image processing apparatus <NUM> of <FIG> and the image processing apparatus <NUM> of <FIG>.

The processor <NUM> may execute instructions and functions in the electronic apparatus <NUM>. For example, the processor <NUM> may process instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> may perform the operations described above with reference to <FIG>. The memory <NUM> may store data for image processing. The memory <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory <NUM> may store instructions that are to be executed by the processor <NUM>, and may also store information associated with software and/or applications when the software and/or applications are being executed by the electronic apparatus <NUM>.

The camera <NUM> may capture a photo and/or a video. For example, the camera <NUM> may capture a face image including a face of a user. The camera <NUM> may be, for example, a three-dimensional (3D) camera including depth information about objects. The storage device <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The storage device <NUM> may store a greater amount of information than that of the memory <NUM> for a relatively long period of time. For example, the storage device <NUM> may include magnetic hard disks, optical disks, flash memories, floppy disks, or other forms of non-volatile memories known in the art.

The input device <NUM> may receive an input from a user through a traditional input scheme using a keyboard and a mouse, and through a new input scheme such as a touch input, a voice input, a gesture input, and an image input. The input device <NUM> may include, for example, a keyboard, a mouse, a touch screen, a microphone, or other devices configured to detect an input from a user and transmit the detected input to the electronic apparatus <NUM>. The output device <NUM> may provide a user with an output of the electronic apparatus <NUM> through a visual channel, an auditory channel, or a tactile channel. The output device <NUM> may include, for example, a display, a touchscreen, a speaker, a vibration generator, or other devices configured to provide a user with the output. The network interface <NUM> may communicate with an external device via a wired or wireless network.

The apparatuses, units, modules, devices, and other components described herein are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, multiple-instruction multiple-data (MIMD) multiprocessing, a controller and an arithmetic logic unit (ALU), a DSP, a microcomputer, an FPGA, a programmable logic unit (PLU), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or any other device capable of responding to and executing instructions in a defined manner.

The methods that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods.

In an example, the instructions or software includes at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application program storing the image processing method. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

Claim 1:
An image processing method (<NUM>, <NUM>) for generating a prediction result for an input image, comprising:
extracting (<NUM>) an input feature (<NUM>; <NUM>; <NUM>) from the input image (<NUM>; <NUM>; <NUM>; <NUM>) using a feature extraction neural network model (<NUM>, <NUM>, <NUM>);
generating (<NUM>) one or more augmented features (<NUM>, <NUM>, <NUM>) using a feature augmentation model (<NUM>, <NUM>, <NUM>) comprising an encoding neural network model (<NUM>) and a decoding neural network model (<NUM>), based on the input feature and a respective transformation parameter of one or more transformation parameters (<NUM>, <NUM>, <NUM>) indicating a type and a degree of a transformation that could be applied to an image, wherein the one or more transformation parameters correspond to different transformations, the generating comprising:
generating of a first augmented feature (<NUM>) of the one or more augmented features, comprising:
encoding the input feature to a latent feature (<NUM>) using the encoding neural network model (<NUM>; <NUM>);
combining the latent feature (<NUM>) and a transformation code (<NUM>) corresponding to the respective transformation parameter to determine a combined feature wherein combining the latent feature and the transformation code comprises concatenating the latent feature and the transformation code (<NUM>); and
decoding the combined feature to the first augmented feature using the decoding neural network model (<NUM>);
wherein the transformation code is the transformation parameter, converted into a form that may be processed in the decoding neural network model and includes a first field indicating the type of transformation and a second field indicating the degree of transformation;
generating (<NUM>) the prediction result (<NUM>, <NUM>) based on the one or more augmented features using a prediction neural network model (<NUM>, <NUM>, <NUM>).