Method and apparatus with key-value coupling

A processor-implemented method of implementing an attention mechanism in a neural network includes obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of an attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2019-0038595 filed on Apr. 2, 2019 in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to a method and apparatus with a key-value mechanism.

2. Description of Related Art

An attention algorithm (e.g., an attention mechanism) is a neural network-based technology for focusing on highly significant data among provided input data. For example, the attention algorithm may use key-value pairs. Input data (queries) of an attention layer may be compared to keys in the key-value pairs. When weights for the keys are calculated based on a comparison result, a weighted average according to the weights may be applied to values, whereby output data of the attention layer may be generated. In this example, similarity processing may be used for comparing the queries and the keys. Meanwhile, similarity processing may be used for filtering or removing noise in an image. For example, a similarity between pixels or a similarity between patches may be used in bilateral filtering or a nonlocal-means algorithm. Similarity processing may be used in bilateral filtering to find neighboring pixels having pixel values similar to that of a target pixel being a denoising target. Similarity processing may be used in the nonlocal-means algorithm to find a patch similar to a target patch to which the target pixel being the denoising target belongs.

SUMMARY

In one general aspect, a processor-implemented method of implementing an attention mechanism in a neural network includes: obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of an attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

The obtaining may include: determining the new key data by applying the first nonlinear transformation to the key data; and determining the key-value coupling data based on an operation between the value data and the new key data.

The new key data may include a first new key, and the value data may include a first value corresponding to the first new key, and the key-value coupling data may include a single item of aggregated data determined based an operation between the first new key and the first value with respect to a first key-value pair of the first new key and the first value.

Either one or both of the first nonlinear transformation and the second nonlinear transformation may use either one or both of a sine function and a cosine function as a nonlinear factor.

The first nonlinear transformation and the second nonlinear transformation may use the same function.

The key-value coupling data may be fixed based on an operation between the new key data and the value data, and the output data of the attention layer may be determined based on an operation between the new query data and the fixed key-value coupling data.

The key-value coupling data may be fixed by being determined, independent of the query data, based on the operation between the new key data and the value data.

An operation between the new key data and the new query data may correspond to a similarity between the key data and the query data.

The determining of the output data of the attention layer may include normalizing a result of the operation between the new query data and the key-value coupling data.

The method may further include performing an inference operation using the neural network based on the output data of the attention layer, wherein the neural network includes additional trained layers.

The method may further include outputting an image recognition result for the input data by applying the output data of the attention layer to the neural network.

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method.

In another general aspect, a processor-implemented nonlocal filtering method may include: obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data corresponding to patches in an input image, and value data of representative pixels in the patches; determining new query data by applying a second nonlinear transformation to query data corresponding to a target patch among the patches; and determining output data for denoising of a representative pixel in the target patch, based on an operation between the new query data and the key-value coupling data.

The representative pixels in the patches may be center pixels in the patches, and the representative pixel in the target patch may be a center pixel in the target patch.

The obtaining may include: determining the new key data by applying the first nonlinear transformation to the key data; and determining the key-value coupling data based on an operation between the value data and the new key data.

The new key data may include a first new key, and the value data may include a first value corresponding to the first new key, and the key-value coupling data may include a single item of aggregated data determined based on an operation between the first new key and the first value with respect to a first key-value pair of the first new key and the first value.

Either one or both of the first nonlinear transformation and the second nonlinear transformation may use either one or both of a sine function and a cosine function as a nonlinear factor.

The first nonlinear transformation and the second nonlinear transformation may use the same function.

An operation between the new key data and the new query data may correspond to a similarity between the key data and the query data.

The method may further include denoising the representative pixel in the target patch based on the output data.

In another general aspect, a processor-implemented method of implementing a neural network includes: performing an inference related to input data of the neural network using a plurality of layers in the neural network, wherein at least one of the plurality of layers in the neural network uses either one or both of a sine function and a cosine function to obtain a nonlinearity.

The at least one layer may be a respective attention layer that performs a corresponding attention mechanism.

The performing may include: obtaining key-value coupling data determined based on an operation between new key data determined using a first nonlinear transformation for key data of the attention layer, and value data of the attention layer corresponding to the key data; determining new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

Either one or both of the first nonlinear transformation and the second nonlinear transformation may use either one or both of the sine function and the cosine function.

In another general aspect, a processor-implemented method of implementing an attention mechanism in a neural network includes: obtaining fixed key-value coupling data determined, independently of input query data of an attention layer, based on key data of the attention layer and value data corresponding to the key data; determining new query data based on input query data of the attention layer; and determining output data of the attention layer based on an operation between the new query data and the key-value coupling data.

The new key data may be determined by applying a first nonlinear transformation to the key data, the key-value coupling data may be determined based on an operation between the value data and the new key data, and the determining of the new query data may include applying a second nonlinear transformation to query data.

The method may further include implementing the neural network to perform an attention-based inference operation for input image data, using the output data of the attention layer.

DETAILED DESCRIPTION

Hereinafter, examples will be described in detail with reference to the accompanying drawings. In the drawings, like reference numerals are used for like elements.

FIG. 1illustrates an example of an electronic device using key-value coupling. Referring toFIG. 1, an electronic device100generates output data150based on an operation between new query data110and key-value coupling data120. The key-value coupling data120is generated based on an operation between new key data130and value data140. Operations to be described below may be construed as being performed by the electronic device100.

A neural network may include an attention layer including an attention algorithm. The attention layer may be at least one of a plurality of layers making up the neural network, and may be inserted into the neural network (e.g., after one or more layers have already been partially trained). The attention algorithm may be used to determine relatively significant data among data to be processed by the neural network.

The attention algorithm may use a query and a key-value pair. The query may search over the keys of words that might supply context for it. Those keys may be related to values that encode more meaning about the key word. The query may correspond to an input (for example, an input feature vector) of the attention layer. When a query is input into the attention layer, similarities between the query and keys are calculated, and a weighted average of values may be calculated based on weights corresponding to the calculated similarities. The calculated weighted average of the values may correspond to an output (for example, an output feature vector) of the attention layer which may then be advanced or forwarded to a next layer of the neural network.

A complexity of an operation for generating the output data150(which will be described in detail below) may depend on a number of queries and a number of keys. When there is an increased number of queries, the complexity of the corresponding operation may be increased. The electronic device100of one or more embodiments greatly lowers the complexity of the operation, compared to the operation of typical electronic devices, by using the key-value coupling data120in which the new key data130and the value data140are aggregated, thereby greatly improving a performance of the example electronic device100over the typical electronic devices performing the more complex operation. The use of the term “may” herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

For example, when the new key data130and the value data140are determined, and the key-value coupling data120is generated based on an operation between the new key data130and the value data140, the key-value coupling data120may be fixed. That the key-value coupling data120is fixed indicates that the key-value coupling data120is determined independent of a query. That is, when there is a plurality of queries, the output data150may be generated based on an operation between the new query data110and the key-value coupling data120, without comparing the queries to keys, for example. Thus, an increment in the complexity of the operation resulting from an increase in the number of queries may greatly decrease.

The new query data110may be generated by applying a nonlinear transformation to query data, and the new key data130may be generated by applying a nonlinear transformation to key data, which will be described in detail below. Examples exist with the nonlinear transformation applied to the query data and the nonlinear transformation applied to the key data use the same function, and examples exist where they use different functions. For example, either one or both of a nonlinear transformation applied to the query data and a nonlinear transformation applied to the key data may use either one or both of a sine function and a cosine function as a nonlinear factor.

Such similarity processing is likewise applicable to filtering or removing noise in an image. For example, nonlocal filtering adjusts a pixel value of a pixel including noise based on a similarity between patches, and may use the key-value coupling data120to process the similarity between the patches.

For example, in an example of an attention algorithm, query data may correspond to the input feature vector of the attention layer, and the output data150may correspond to the output feature vector (for example, an attention output) of the attention layer. In an example of nonlocal filtering, query data corresponds to pixel values of pixels in a target patch among patches in an input image, and the output data150is a pixel value for denoising of a target pixel in the target patch. In the example of the attention algorithm, the electronic device100may correspond to a neural network apparatus using a neural network. In the example of nonlocal filtering, the electronic device100may correspond to an image processing apparatus.

A non-limiting example of the attention algorithm will be described below, and the following example of the attention algorithm may also apply to the example of nonlocal filtering. Further, the example of nonlocal filtering will also be described further later.

FIG. 2illustrates an example of using a key-value pair in an unpaired state. Referring toFIG. 2, key data k1through knand value data v1through vnare illustrated. The key data kxand the value data vxare paired. Here, x denotes an integer of 1 to n. A data set including the key data k1through knand the value data v1through vnis referred to as a key-value pair.

A comparison block C compares query data q and the key data k1through kn, and outputs weight data w1through wncorresponding to similarities between the query data q and the key data k1through kn. Then, output data y is generated based on an operation between the value data v1through vnand the weight data w1through wn. For example, the output data y corresponds to a weighted average of the value data v1through vnbased on the weight data w1through wn.

The weight data w1through wnmay be calculated based on Equation 1.
wj=A(q,kj)  Equation 1:

In Equation 1, wjdenotes the weight data w1through wn, A denotes a similarity function, and kjdenotes the key data k1through kn. j denotes an index for data identification. For example, the similarity function A may be expressed by Equation 2.
A(u,v)∝e−∥u−v∥2/2σ2Equation 2:

In Equation 2, u and v denote data to be compared, and σ2denotes a variance. The output data y may be expressed by Equation 3, for example.

In Equation 3, y denotes the output data, wjdenotes the weight data w1through wn, and vjdenotes the value data v1through vn. j denotes an index. According to Equation 3, the output data y denotes a weighted average of vjwith respect to wj. In Equation 3, a denominator term is for a normalization. Hereinafter, for ease of description, the term for normalization may be omitted when referring to the output data, though it may be appreciated after an understanding of the present disclosure that a normalization term corresponding to the denominator term of Equation 3 may be applied for normalization.

Equations 1 through 3 may represent an example in which there is a single item of query data q. In such an example, a complexity for calculating the output data y may be indicated as O(n). n denotes a number of the key data k1through kn. When the query data q increases, the complexity of operation may greatly increases in a typical case of using key-value pairs in an unpaired state. Equations 4 and 5 may represent an example in which there is a plurality of items of query data q. In such an example, the query data is indicated as qi. i denotes an index, and has a value of 1 to m. That is, a number of the items of query data q may be m.

Equation 2 may be used for the similarity function A of Equation 4. When there is a plurality of (for example, m) items of query data q, similarities between each item of query data q and the key data k1through knmay be calculated with respect to each item of query data q, and thus a complexity for calculating the output data y may be indicated as O(m*n). Therefore, in a typical case of using key-value pairs in an unpaired state, the number of the items of query data q increases, and the complexity of operation increases greatly.

FIG. 3illustrates an example of one or more embodiments using key-value coupling. In an example, a similarity between data u and data v may be expressed by Equation 6.
A(u,v)≈φ1(u)Tφ2(v)  Equation 6:

In Equation 6, φ1and φ2denotes nonlinear functions for assigning a nonlinearity respectively to the data (u, v). The nonlinear function φ1and the nonlinear function φ2may correspond to the same function or may correspond to respective different functions. Assigning a nonlinearly to data is construed as performing a nonlinear transformation on the data. Hereinafter, an example in which the nonlinear function φ1and the nonlinear function φ2are the same will be described. However, the following description does not limit the scope of a right for an example in which the nonlinear function φ1and the nonlinear function φ2are different.

To determine a similarity between provided data, nonlinear factors between the data may be considered. Thus, to determine a similarity A(u, v) between the data u and the data v, A(u, v) may divided into a nonlinear term φ1(u) related to the data u and a nonlinear term φ2(v) related to the data v. Such a division process is referred to as a factorization. To obtain a scalar value, φ1(u) may be transposed.

In an example, the similarity between the data u and the data v may be expressed by Equation 7.

In Equation 7, d denotes a constant which determines an approximation precision in approximating the similarity A(u, v) using a combination of cosine terms. When the similarity function A is expressed by a difference δ=u−v of two vectors u and v to be compared, for example, as in Equation 2, ωjand b may be expressed by Equation 8.
ωj˜p(ω)=IFFT[A(δ)],
b˜Uniform(0,2π)  Equation 8:

In Equation 8, IFFT denotes an inverse fast Fourier transform, and ˜ denotes an independent sample extraction from a corresponding probability density function. According to Equation 7, a nonlinear function may be expressed by Equation 9, for example.

The nonlinear function according to Equation 8 includes a cosine function as a nonlinear factor. Equation 8 may be changed to various forms according to formulas in relation to trigonometric functions. Thus, the nonlinear function may use either one or both of a sine function and a cosine function as the nonlinear factor.

Referring toFIG. 3, new key data φ(k1) through φ(kn) are generated by applying a nonlinear transformation φ to key data k1through kn, and key-value coupling data Λ is generated based on an operation between the value data v1through vnand the new key data φ(k1) through φ(kn). The key-value coupling data Λ aggregates the new key data φ(k1) through φ(kn) and the value data v1through vn. For example, the key-value coupling data Λ includes a single item of aggregated data v1*φ(k1)Tgenerated based on an operation between a new key φ(k1) and a value v1with respect to a key-value pair of the new key φ(k1) and the value v1.

Then, new query data φ(qj) is generated by applying a nonlinear transformation φ to query data qj, and output data yiis generated based on an operation between the new query data φ(qj) and the key-value coupling data Λ. Here, a dimension increases as a consequence of the nonlinear transformation φ, and the operation is a multiplying operation. Generating the output data yimay thus include normalizing a result of the operation between the new query data φ(qj) and the key-value coupling data Λ.

According to an example, a key-value pair may be generated based on input data. Here, the key-value coupling data Λ may be generated based on the input data. In this example, an electronic device may generate the new key data φ(k1) through φ(kn) by applying the nonlinear transformation φ to the key data k1through kn, and may generate the key-value coupling data Λ based on the operation between the value data v1through vnand the new key data φ(k1) through φ(kn). According to an example, a key-value pair may be generated in advance through pretraining. Here, the key-value coupling data Λ generated in advance may be loaded from a memory and used. Hereinafter, an expression “obtaining key-value coupling data Λ” will be used, and the expression may be construed as covering either and both of examples where the key-value pair is generated based on the input data and examples where the key-value pair is generated in advance through pretraining.

According to an example of one or more embodiments in which the key-value coupling data Λ is used, the output data yimay be expressed by Equation 10.

In Equation 10, vjdenotes the value data v1through vn, φ(kj) denotes the new key data φ(k1) through φ(kn), and φ(qj) denotes the new query data φ(q1) through φ(qn). Λ denotes the key-value coupling data, which may be generated based on an operation between vjand φ(kj). According to Equation 10, an operation between the new key data φ(k1) through φ(kn) and the new query data φ(q1) through φ(qn) corresponds to a similarity A(qi, kj) between the key data k1through knand the query data q1through qn.

A complexity for calculating the key-value coupling data Λ may be indicated as O(n). Further, when the key-value coupling data Λ is calculated, the key-value coupling data Λ may be fixed. Thus, a complexity of operation related to m items of query data qimay be indicated as O(m). Therefore, in the example of one or more embodiments, a total complexity of operation may be indicated as O(n+m), which exhibits a great decrease in total complexity of operation when compared to the complexity O(m*n) according to a typical example in which a key-value pair is used in an unpaired state. As a number of the items of query data qiand a number of key-value pairs increase, there is a further increase in the extent of the decrease in total complexity of operation of the example of one or more embodiments compared to that of the typical example. By decreasing the total complexity in operation compared to the typical example, the example of one or more embodiments may improve a processing speed, or may reduce the total processing power needed to efficiently process the operations, of one or more processors of the electronic device100on which the operations may be performed.

FIGS. 4 and 5illustrate examples of data processing in a scheme of using a key-value pair in an unpaired state.FIGS. 4 and 5, andFIG. 6, which will be described later below, illustrate examples of generating a query, a key, and a value based on input data (for example, a self-attention of an attention algorithm).

Referring toFIG. 4, input data x is input. For example, the input data x may correspond to pixels in a target patch of an input image or an input feature vector of an attention network. A dimension of the input data x may be T*H*W*D0. Here, T denotes a number of video frames. H, W, and D0denote a height, a width, and a depth, respectively, when the input data x is depicted as a rectangular parallelepiped. D0correspond to a number of channels of the input data x.

Query data q may be generated through a transformation θ related to the input data x, key data k may be generated through a transformation ϕ related to the input data x, and value data v may be generated through a transformation g related to the input data x. Dimensions of the query data q, the key data k, and the value data v may each be T*H*W*D1. In a transformation process, D0may be changed to D1. For example, D1may be greater than D0. InFIG. 4, “1*1*1” indicates a 1*1*1 convolution, as a non-limiting convolution example.

Next, in an example ofFIG. 4, an operation between the query data q and the key data k is performed. InFIG. 4, ⊗ denotes a matrix multiplication, and ⊕ denotes an elementwise addition. In this example, a dimension of a result of the operation between the query data q and the key data k may be indicated as (T*H*W){circumflex over ( )}2. An example of a memory use of the result of the operation between the query data q and the key data k will be described further with reference toFIG. 5.

Referring toFIG. 5, an example of a memory space550used depending on a result of an operation between query data510and key data520is illustrated. In an example, T=1(for example, a single image), and dimensions of the query data510and the key data520are H*W*D. However, in other examples, T may be greater than 1. The query data510may include multiple queries including a first query511, and the key data520may include multiple keys including a first key521.

A result of an operation between the first query511and each key of the key data520occupies a space551. Further, results of operations between the remaining queries of the query data510and each key of the key data520occupy a remaining portion of the memory space550excluding the space551. Thus, the result of the operation between the query data510and the key data520occupies the memory space550of (HW){circumflex over ( )}2.

Referring toFIG. 4again, softmax may be applied to the result of the operation between the query data q and the key data k. Softmax is construed as normalizing the result of the operation between the query data q and the key data k. Next, output data y may be generated based on an operation between value data v and a softmax output. A dimension of the output data y is T*H*W*D1. Then, the dimension of the output data y may be adjusted based on the input data x, and an operation result z according to a residual connection is output. After the output data y is generated, a process of generating the operation result z may be selectively applied.

FIG. 6illustrates an example of one or more embodiments of data processing in a scheme of using key-value coupling. Referring toFIG. 6, input data x is input. For example, the input data x may correspond to pixels of an input image or an input feature vector of an attention network. A dimension of the input data x may be T*H*W*D0. Notations of T, H, W, and D0are the same as those inFIG. 4.

In the same manner as described with reference toFIG. 4, query data q, key data k, and value data v are generated, and dimensions thereof may each be T*H*W*D1. Next, new query data may be generated using a nonlinear transformation φ related to the query data q, and new key data may be generated using a nonlinear transformation φ related to the key data k. In a nonlinear transformation process, D1may be changed to d.

Then, key-value coupling data Λ may be generated based on an operation between the new key data and the value data v. InFIG. 6, ⊗ denotes a matrix multiplication, ⊕ denotes an elementwise addition, anddenotes an elementwise division. A dimension of the key-value coupling data Λ is indicated as D1*d.

When the key-value coupling data Λ is generated, an operation between the new query data and the key-value coupling data Λ may be performed. A dimension of a result of the operation may be T*H*W*D1, a normalization may be performed on a result of the corresponding operation, and output data y may be generated as a result of the normalization. As shown in the example of one or more embodiments illustrated inFIG. 6, when the key-value coupling data Λ is used, a large memory space of (T*H*W){circumflex over ( )}2 is not required compared to in the typical example ofFIG. 4. Then, a dimension of the output data y may be adjusted based on the input data x, and an operation result z according to a residual connection is output. After the output data y is generated, a process of generating the operation result z may be selectively applied.

FIG. 7illustrates an example of nonlocal filtering. Referring toFIG. 7, patches710of an input image700are shown. The patches710may have the same size, and may be uniformly disposed in the input image700. Depending on an example, a number and a size of the patches710may vary. For example, the patches710may overlap each other.

Each of the patches710may include a representative pixel711. For example, the representative pixel711may be a center pixel of each patch710. A pixel being a target of denoising in the input image700may be referred to as a target pixel721, and a patch including the target pixel721may be referred to as a target patch720. The target pixel721may correspond to a representative pixel of the target patch720. An image processing apparatus may perform denoising on noise components by sequentially designating pixels in the input image700as the target pixel721(e.g., until all of the pixels have been designated as the target pixel721).

The image processing apparatus may compare the target patch720and the patches710for nonlocal filtering, and may calculate a weighted average of representative pixels711of the patches710by assigning weights according to similarities between the target patch720and the patches710to the representative pixels711of the patches710. The calculated weighted average may correspond to output data. The image processing apparatus may use the output data for denoising of the target pixel721. For example, a pixel value according to the output data may be assigned to the target pixel721.

Key-value coupling data may be used for the nonlocal filtering process. Key data730may be determined based on pixel values of pixels in the patches710, and value data may be determined based on the representative pixels711of the patches710. Further, query data740may be determined based on pixel values of pixels in the target patch720. Here, a process using the key-value coupling data described above may be applied.

In detail, the image processing apparatus may obtain key-value coupling data generated based on an operation between the value data and new key data generated using a nonlinear transformation related to the key data730. Then, the image processing apparatus may generate output data for denoising of the target pixel721, based on an operation between the key-value coupling data and new query data generated using a nonlinear transformation related to the query data740. In addition, the description provided with reference toFIGS. 1 through 6may also apply to the example ofFIG. 7. Through the above process of one or more embodiments, a complexity of operation for nonlocal filtering decreases greatly compared to a complexity of operation for nonlocal filtering of a typical process.

FIG. 8illustrates an example of a method of implementing an attention mechanism using key-value coupling. Referring toFIG. 8, in operation810, a neural network apparatus may obtain key-value coupling data generated based on an operation between new key data generated using a first nonlinear transformation related to key data of an attention layer and value data of the attention layer corresponding to the key data. In operation820, the neural network apparatus may generate new query data by applying a second nonlinear transformation to query data corresponding to input data of the attention layer. In operation830, the neural network apparatus may generate output data of the attention layer based on an operation between the new query data and the key-value coupling data. In addition, the description provided with reference toFIGS. 1 through 6may apply to the method ofFIG. 8.

FIG. 9illustrates an example of a nonlocal filtering method using key-value coupling. Referring toFIG. 9, in operation910, an image processing apparatus may obtain key-value coupling data generated based on an operation between new key data generated using a first nonlinear transformation related to key data corresponding to patches in an input image and value data of representative pixels in the patches. In operation920, the image processing apparatus may generate new query data by applying a second nonlinear transformation to query data corresponding to a target patch among the patches. In operation930, the image processing apparatus may generate output data for denoising of a representative pixel in the target patch, based on an operation between the new query data and the key-value coupling data. In addition, the description provided with reference toFIGS. 1 through 7may apply to the nonlocal filtering method ofFIG. 9.

FIG. 10illustrates an example of a method of implementing a neural network. Referring toFIG. 10, in operation1010, a neural network apparatus may perform an inference related to input data of a neural network using a plurality of layers in the neural network. At least one of the plurality of layers in the neural network may use either one or both of a sine function and a cosine function, as non-limiting examples, to obtain a nonlinearity. As described with reference toFIGS. 1 through 6, a nonlinearity may be assigned to the neural network through the example sine function and/or cosine function, and the neural network may perform the inference related to the input data based on the nonlinearity assigned through the sine function and the cosine function and with respect to the remaining trained parameters of the neural network. For example, one or more layers in the neural network may include respective attention layers which perform respective attention mechanisms. In addition, the description provided with reference toFIGS. 1 through 6may apply to the method ofFIG. 10.

FIG. 11illustrates an example of a neural network apparatus. Referring to FIG.11, a neural network apparatus1100includes a memory1110and a processor1120. The memory1110stores a neural network1115. The memory1110stores instructions executable by the processor1120. When the instructions stored in the memory1110are executed by the processor1120, the processor1120performs the operations described with reference toFIGS. 1 through 6, 8, and 10. In addition, the description provided with reference toFIGS. 1 through 6, 8, and 10applies to the neural network apparatus1100.

FIG. 12illustrates an example of an image processing apparatus, such as a mobile device example performing an image capturing or processing function. Referring toFIG. 12, an image processing apparatus1200includes a memory1210and a processor1220. The memory1210stores instructions executable by the processor1220. When the instructions stored in the memory1210are executed by the processor1220, the processor1220performs the operations described with reference toFIGS. 1 through 7, and 9. In addition, the description provided with reference toFIGS. 1 through 7, and 9applies to the image processing apparatus1200.