Patent Description:
Research is being actively conducted to classify input patterns in groups so that efficient pattern recognition may be performed on computers. This includes research on an artificial neural network (ANN) that is obtained by modeling pattern recognition characteristics using mathematical expressions through a processor-implemented neural network model, as a specialized computational architecture, which after substantial training may provide computationally intuitive mappings between input patterns and output patterns. The ANN generates mapping between input patterns and output patterns using an algorithm, and a capability of generating the mapping is expressed as a learning capability of the ANN. The ANN may employ an algorithm that mimics abilities to learn. Also, the ANN has a capability to generate a relatively correct output with respect to an input pattern that has not been used for training based on a result of previous training. However, because such operations or applications are performed through specialized computation architecture, and in different automated manners than they would have been performed in non-computer implemented or non-automated approaches, they also invite problems or drawbacks that only occur because of the automated and specialized computational architecture on which they are implement.

Further, studies are being conducted to maximize the recognition rate of the ANN while miniaturizing the size of the ANN, for example in <NPL>".

In one general aspect, there is provided a method of training a model according to the appended independent method claim.

In another general aspect, there is provided an apparatus according to the appended independent apparatus claim.

The use of the term 'may' herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

<FIG> and <FIG> illustrate examples of a teacher model and a student model.

<FIG> illustrates a teacher model <NUM> and a student model <NUM>.

The teacher model <NUM> and the student model <NUM> are neural networks having difference sizes and the same target data to be recognized. The neural network is a recognition model using numerous artificial neurons connected through edges.

The teacher model <NUM> is a model that recognizes target data to be recognized, with a high accuracy using sufficiently numerous characteristics extracted from the target data. The teacher model <NUM> is greater in size than the student model <NUM>. For example, the teacher model <NUM> may include more layers, more nodes, or combinations thereof when compared to the student model <NUM>.

The student model <NUM> is a neural network having a size that is smaller than that of the teacher model <NUM>. Due to the small size, the student model <NUM> uses lesser memory and has a faster recognition speed than the teacher model <NUM>. The student model <NUM> is trained such that the same output data is output as the data that is output by teacher model <NUM> for specific input data. The output data may be, for example, a value of logit and a probability value output from the corresponding model.

An input sequence X is input to the teacher model <NUM> and the student model <NUM>. The input sequence X includes data including a plurality of elements x<NUM>,. , xM and includes speech data including a plurality of frames. Here, the speech data is also referred to as data obtained by sampling the speech data for a length, for example, in units of frame. In an example, the length of the frames may be determined in advance.

In the example of <FIG>, a probability distribution p(Y = y<NUM>. yL|X; θT) is output from the teacher model <NUM> and a probability distribution q(Y = y<NUM>. yL|X; θS) is output from the student model <NUM>. Here, θT denotes parameters of the teacher model <NUM> and θS denotes parameters of the student model <NUM>. Also, Y is an output sequence output from each of the teacher model <NUM> and the student model <NUM>, and includes a plurality of elements y1,.

The same output data as the teacher model <NUM> may be acquired at a high recognition rate through the student model <NUM> trained using the teacher model <NUM>. Such training scheme is referred to as a model compression, and related description will be provided later.

The teacher model <NUM> and the student model <NUM> are trained to perform a desired operation by mapping input data and output data that have a nonlinear relationship therebetween through deep learning to perform speech recognition. The deep learning is a machine learning method used to solve a problem given from a big dataset. The deep learning may also be construed as a problem-solving process for optimization to find a point where energy is minimized while training the neural network using provided training data. Through the deep learning, for example, supervised or unsupervised learning, a weight corresponding to an architecture or a model of the neural network may be obtained, and the input data and the output data may be mapped to each other based on the obtained weight.

In a non-claimed example, the teacher model <NUM> and the student model <NUM> may be implemented as an architecture having a plurality of layers including an input image, feature maps, and an output. In the teacher model <NUM> and the student model <NUM>, a convolution operation between the input image, and a filter referred to as a kernel, is performed, and as a result of the convolution operation, the feature maps are output. Here, the feature maps that are output are input feature maps, and a convolution operation between the output feature maps and the kernel is performed again, and as a result, new feature maps are output. Based on such repeatedly performed convolution operations, results of recognition of characteristics of the input image via the neural network may be output.

In another example, the teacher model <NUM> and the student model <NUM> may include an input source sentence, (e.g., voice entry) instead of an input image. In such an example, a convolution operation is performed on the input source sentence with a kernel, and as a result, the feature maps are output. The convolution operation is performed again on the output feature maps as input feature maps, with a kernel, and new feature maps are output. When the convolution operation is repeatedly performed as such, a recognition result with respect to features of the input source sentence may be finally output through the neural network. Input data for the teacher model <NUM> and the student model 120may include image data, voice data, and text data. However, they are provided as examples only, and other types of data are considered to be well within the scope of the present disclosure.

<FIG> illustrates an example of training a student model <NUM> using a teacher model <NUM>.

The teacher model <NUM> and the student model <NUM> are networks in different sizes. A method and apparatus for recognizing data based on a neural network is suggested and a method and apparatus for training the neural network is suggested. In this specification, the term "recognition" is used as a concept including verification and identification. The verification is an operation of determining whether input data is true of false. For example, the verification may be an operation of determining whether input data is true or false. The identification is an operation of determining a label indicated by input data from among a plurality of labels. For example, the neural network is a model that receives a sequence and performs operations such as, for example, translation, interpretation, and speech recognition.

In an example, the student model <NUM> and the teacher model <NUM> may correspond to a recurrent neural network (RNN) or a convolutional neural network (CNN). In an example, the CNN may be a deep neural network (DNN). Ain an example, the DNN may include a region proposal network (RPN), a classification network, a reinforcement learning network, a fully-connected network (FCN), a deep convolutional network (DCN), a long-short term memory (LSTM) network, and a grated recurrent units (GRUs). The DNN may include a plurality of layers. The plurality of layers may include an input layer, at least one hidden layer, and an output layer. In an example, neural network may include a sub-sampling layer, a pooling layer, a fully connected layer, etc., in addition to a convolution layer.

The neural network includes a plurality of layers, each including a plurality of nodes. Also, the neural network includes connection weights that connect the plurality of nodes included in the plurality of layers to a node included in another layer.

The neural network includes, for example, an input layer, at least one hidden layer, and an output layer. The input layer receives an input for performing training or recognition and transfers the input to the hidden layer. The output layer generates an output of the neural network based on a signal received from the hidden layer. The hidden layer is interposed between the input layer and the output layer, and changes data transferred though the input layer to a value to be easily predicted.

Input nodes included in the input layer and hidden nodes included in the hidden layer are connected through edges having connection weights. The hidden nodes included in the hidden layer and output nodes included in the output layer are connected through edges having connection weights.

The neural network may include a plurality of hidden layers. The neural network including the plurality of hidden layers is referred to as a deep neural network. Training of the deep neural network is referred to as deep learning. For example, the teacher model <NUM> greater in size than the student model <NUM> may include a greater number of hidden layers as compared to the student model <NUM>.

A model training apparatus uses a gradient descent scheme based on output values of nodes included in a neural network and a loss that is back-propagated to the neural network, to determine parameters of the nodes. For example, the model training apparatus updates connection weights between the nodes through loss back-propagation learning. The loss back-propagation learning is a method of estimating a loss by performing forward computation on given training data, propagating the estimated loss in a reverse direction from the output layer toward the hidden layer and the input layer, and updating the connection weights to reduce the loss. A processing of the neural network is performed in a direction from the input layer toward the hidden layer and the output layer. In the loss back-propagation training, the update of the connection weights is performed in the direction from the output layer, toward the hidden layer and the input layer. One or more processors may use a buffer memory that stores a layer or a series of computation data to process the neural network in a desired direction.

In an example, the model training apparatus defines an objective function for measuring a degree to which currently set connection weights are close to optimum, continuously changes the connection weights based on a result of the objective function, and repetitively performs the training. The objective function is, for example, a loss function for calculating a loss between an actual output value output by the neural network based on the training input of the training data and a desired expected value to be output, for example, the training output. The model training apparatus updates the connection weights to reduce a value of the loss function. The loss function will be described in detail as follows.

The student model <NUM> is trained from the teacher model <NUM> based on knowledge distillation for knowledge propagation between two different neural networks. The knowledge distillation is a type of model compression. In this example, a Kullback-Leibler divergence (KLD) loss <IMG> is used, which is expressed by Equation <NUM> below.

In Equation <NUM>, H(p(Y|X; θT), q(Y|X; θS)) denotes cross-entropy between the teacher model <NUM> and the student model <NUM>. KLD-based knowledge distillation is a scheme for training the student model <NUM> using a probability distribution of the teacher model <NUM> as a soft-target.

The student model <NUM> is trained to output the recognition result of the teacher model <NUM> so that a difference between the recognition result of the teacher model <NUM> and the recognition result of the student model <NUM> is reduced. Here, the recognition result includes, for example, a probability distribution output from each model or a class sampled at a highest probability in the probability distribution.

<FIG> illustrates an example of a process of training a student model.

<FIG> illustrates an example of training a student model <NUM> using a discriminator model <NUM> and a teacher model <NUM>.

The discriminator model <NUM> is a neural network that distinguishes between a recognition result of the teacher model <NUM> and a recognition result of the student model <NUM>, and may include, for example, convolutional neural networks (CNN), recurrent neural networks (RNN), and self-attention. The discriminator model <NUM> is trained to distinguish the recognition result of the teacher model <NUM> as true and distinguish the recognition result of the student model <NUM> as false. The student model <NUM> is trained such that the recognition results of the teacher model <NUM> and the student model <NUM> are not distinguished from each other by the discriminator model <NUM>. As such, a training in which two models are trained while competing against each other is referred to as an adversarial training. An adversarial loss <IMG> used in the training is expressed by Equation <NUM> below.

In Equation <NUM>, d(Y|θD) denotes a probability distribution for distinguishing whether a sequence Y input to the discriminator model <NUM> is generated in the teacher model <NUM> or the student model <NUM>. ~p indicates that the sequence Y input to the discriminator model <NUM> is input from the teacher model <NUM>, ~q indicates that the sequence Y input to the discriminator model <NUM> is input from the student model <NUM>, and <IMG> denotes an expectation.

The adversarial training is performed by training the student model <NUM> to reduce the adversarial loss <IMG> and training the discriminator model <NUM> to increase the adversarial loss <IMG>. Through this, the student model <NUM> is trained to output the same recognition result as the teacher model <NUM> at a degree such that the discriminator model <NUM> is unable to distinguish between the two.

Furthermore, the student model <NUM> is trained further based on the KLD loss<IMG> described above. According to Equation <NUM>, the student model <NUM> is trained to reduce the KLD loss <IMG> and the adversarial loss <IMG>. Also, the discriminator model <NUM> is trained to increase the adversarial loss <IMG>. In an example, the teacher model <NUM> is fixed and not trained.

The discriminator model <NUM> receives a sequence or elements included in the sequence from each of the teacher model <NUM> and the student model <NUM>, so that the training is performed in units or sequence or in units of element included in the sequence. A training process using the discriminator model <NUM> will be described in detail with reference to <FIG> and <FIG>.

<FIG> illustrates an example of a process of training performed in units of element.

Referring to <FIG>, elements included in sequences output from a teacher model <NUM> and a student model <NUM> are input to a discriminator model <NUM> so that a training is performed in units of element.

A j-th element in the sequence output from the teacher model <NUM> and a j-th element in the sequence output from the student model <NUM> are transferred to the discriminator model <NUM>. The discriminator model <NUM> distinguishes a model from which each of the j-th elements is input. In <FIG>, p(y<NUM>|X, Y:<NUM>; θT) denotes a probability distribution corresponding to a first element in the sequence output from the teacher model <NUM>, q(y<NUM>|X, Y:<NUM>; θS) denotes a probability distribution corresponding to a first element in the sequence output from the student model <NUM>, p(yL|X, Y:L; θT) denotes a probability distribution corresponding to an L-th element in the sequence output from the teacher model <NUM>, and p(yL|X, Y:L; θS) denotes a probability distribution corresponding to an L-th element in the sequence output from the student model <NUM>.

In this example, an adversarial loss <IMG> is expressed by Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a class of the j-th element in the sequence output from the teacher model <NUM> and <MAT> denotes a class of the j-th element in the sequence output from the student model <NUM>. GP(yj) is a Gumbel-max algorithm that enables the student model <NUM> to be trained using the adversarial loss <IMG>, which is expressed by Equation <NUM> below.

In Equation <NUM>, as τ converges to <NUM>, a greatest value of a probability distribution is close to <NUM> and remaining values is close to <NUM>. Through the Gumbel-max algorithm,, information on the discriminator model <NUM> is transferred to the student model <NUM> and used for the training of the student model <NUM>.

The adversarial loss <IMG> decreases in a case in which classes <MAT> and <MAT> sampled in the probability distribution <MAT> of the element output from the teacher model <NUM> and the probability distribution <MAT> of the element output from the student model <NUM> are not distinguished from each other. Such case is expressed by Equation <NUM> below.

For the adversarial loss <IMG> of Equation <NUM>, gradients for the student model <NUM> and the discriminator model <NUM> are expressed by Equation <NUM> from which it can be known that the training is performed normally.

<FIG> illustrates an example of a process of training performed in units of sequence.

Referring to <FIG>, sequences output from a teacher model <NUM> and a student model <NUM> are input to a discriminator model <NUM> so that a training is performed in units of sequence.

The sequence output from the teacher model <NUM> and the sequence output from the student model <NUM> are transferred to the discriminator model <NUM>. The discriminator model <NUM> distinguishes a model from which each of the sequences is input. In <FIG>, p(Y = y<NUM>. yL|X; θT) denotes a probability distribution corresponding to the sequence output from the teacher model <NUM> and q(Y = y<NUM>. yL|X; θS) denotes a probability distribution corresponding to the sequence output from the student model <NUM>.

A Gumbel-max algorithm, applied to Equation <NUM> is determined based on a combination of probabilities of elements, or determined based on a probability of a sequence.

A Gumbel-max algorithm, based on a combination of probabilities of elements is as shown in Equation <NUM>.

As such, the Gumbel-max algorithm is determined based on a multiplication of probabilities of elements included in an output sequence.

A Gumbel-max algorithm, based on a probability of a sequence is as shown in Equation <NUM>.

In Equation <NUM>, Y' denotes a number of candidate sequences that may correspond to an output sequence, and may be, for example, k-best among the candidate sequences. As such, by limiting the number of candidate sequences, the Gumbel-max algorithm is determined based on a probability of the output sequence.

Z{p,q} applied to the adversarial loss <IMG> denotes a class of a sequence sampled based on the probability distribution output from the teacher model <NUM> or the student model <NUM>. For example, Z{p,q} denotes a class of a sequence selected as one-best using various schemes such as, for example, beam search and gradient search. One of the schemes for determining Z{p,q} is expressed by Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a j-th element in a ground truth sequence, U(<NUM>,<NUM>) denotes a uniform distribution, and ω denotes a threshold.

<FIG> illustrates an example of a model training method. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer, such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG> and are incorporated herein by reference. Thus, the above description may not be repeated here.

The model training method is performed by a processor of a model training apparatus.

In operation <NUM>, the model training apparatus acquires a recognition result of a teacher model and a recognition result of a student model with respect to an input sequence.

In operation <NUM>, the model training apparatus trains the student model such that the recognition result of the teacher model and the recognition result of the student model are not distinguished from each other, in other words, the recognition result of the teacher model and the recognition result of the student model converge as a result of the training. The model training apparatus determines an adversarial loss based on a degree to which the recognition result of the teacher model and the recognition result of the student model are distinguished from each other, and trains the student model such that the adversarial loss is reduced.

In one example, the model training apparatus determines an adversarial loss based on a degree to which an output sequence of the teacher model and an output sequence of the student model that are respectively output as recognition results for the input sequence are distinguished from each other. For example, the model training apparatus determines the adversarial loss by applying a Gumbel-max algorithm based on probabilities of elements included in an output sequence. Also, the model training apparatus determines the adversarial loss by applying a Gumbel-max algorithm based on a probability of an output sequence.

In another example, the model training apparatus determines an adversarial loss based on a degree to which an element included in an output sequence of the teacher model and an element included in an output sequence of the student model that are respectively output as recognition results for the input sequence are distinguished from each other. For example, the model training apparatus determines the adversarial loss by applying a Gumbel-max algorithm based on a probability of an element included in an output sequence.

The model training apparatus trains the student model such that the recognition result of the teacher model and the recognition result of the student model are not distinguished from each other by a discriminator model. In this example, the discriminator model is trained to distinguish between the recognition result of the teacher model and the recognition result of the student model.

The model training apparatus trains the student model using the recognition result of the teacher model such that the recognition result of the teacher model is output from the student model.

<FIG> illustrates an example of a data recognition method.

The data recognition method is performed by a processor of a data recognition apparatus.

In operation <NUM>, the data recognition apparatus receives data to be recognized. In operation <NUM>, the data recognition apparatus recognizes target data using a pre-trained model. In an example, the pre-trained model is the student model described above. Since a training method of the student model is the same as that described above, the descriptions of <FIG> are also applicable to <FIG> and are incorporated herein by reference. Thus, the above description may not be repeated here and is omitted for brevity.

<FIG> illustrates an example of an apparatus for processing data based on a neural network.

Referring to <FIG>, a data processing apparatus <NUM> includes a memory <NUM>, a processor <NUM>, and an input/output interface <NUM>. The memory <NUM> and the processor <NUM> communicate with each other via a bus <NUM>.

The data processing apparatus <NUM> is an apparatus for processing input data and outputting the processed data, and may be one of the model training apparatus and the data recognition apparatus described herein.

The memory <NUM> includes instructions to be read by a computer. The processor <NUM> performs the aforementioned operations in response to the instructions stored in the memory <NUM> being executed in the processor <NUM>. The processor <NUM> may be a data processing device configured as hardware having a circuit in a physical structure to implement desired operations. For example, the desired operations may include codes or instructions included in a program. For example, the data processing device configured as hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multicore processor, a reconfigurable processor, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processor unit (GPU), or any other type of multi- or single-processor configuration. Further details regarding the processor <NUM> is provided below.

The memory <NUM> is a volatile memory or a non-volatile memory. In addition, the data processing apparatus <NUM> processes the operations described herein. Further details regarding the memory <NUM> is provided below.

In an example, the input/output interface <NUM> may be a display that receives an input from a user or provides an output. In an example, the input/output interface <NUM> may function as an input device and receives an input from a user through a traditional input method, for example, a keyboard and a mouse, and a new input method, for example, a touch input, a voice input, and an image input. Thus, the input/output interface <NUM> may include, for example, a keyboard, a mouse, a touchscreen, a microphone, and other devices that may detect an input from a user and transmit the detected input to the data processing apparatus <NUM>.

In an example, the input/output interface <NUM> may function as an output device, and provide an output of the data processing apparatus <NUM> to a user through a visual, auditory, or tactile channel. The input/output interface <NUM> may include, for example, a display, a touchscreen, a speaker, a vibration generator, and other devices that may provide an output to a user.

However, the input/output interface <NUM> are not limited to the example described above, and any other displays, such as, for example, computer monitor and eye glass display (EGD) that are operatively connected to the data processing apparatus <NUM> may be used without departing from the scope of the illustrative examples described. In an example, the input/output interface <NUM> is a physical structure that includes one or more hardware components that provide the ability to render a user interface, render a display, and/or receive user input.

The data processing apparatus <NUM> may be implemented in various electronic devices, such as, for example, a mobile telephone, a smartphone, a wearable smart device (such as, a ring, a watch, a pair of glasses, glasses-type device, a bracelet, an ankle bracket, a belt, a necklace, an earring, a headband, a helmet, a device embedded in the cloths, or an eye glass display (EGD)), a computing device, for example, a server, a laptop, a notebook, a subnotebook, a netbook, an ultra-mobile PC (UMPC), a tablet personal computer (tablet), a phablet, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), an ultra mobile personal computer (UMPC), a portable lab-top PC, electronic product, for example, a robot, a digital camera, a digital video camera, a portable game console, an MP3 player, a portable/personal multimedia player (PMP), a handheld e-book, a global positioning system (GPS) navigation, a personal navigation device, portable navigation device (PND), a head-up display (HUD), an a handheld game console, an e-book, a television (TV), a high definition television (HDTV), a smart TV, a smart appliance, a smart home device, or a security device for gate control, various Internet of Things (IoT) devices, an autonomous vehicle, an automatic or autonomous driving system, an intelligent vehicle, an advanced driver assistance system (ADAS), or any other device capable of wireless communication or network communication consistent with that disclosed herein. In an example, the data processing apparatus <NUM> recognizes target data using a pre-trained model. In an example, the pre-trained model is the student model described above.

In an example, the data processing apparatus <NUM> may be connected to an external device, such as, for example, a personal computer (PC) or a network, via an input/output device of the external device, to exchange data with the external device.

The apparatuses, units, modules, devices, and other components described herein are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods.

Instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above are written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and the methods as described above. In an example, the instructions or software includes at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application program storing the method of training a model based on a neural network or a method of training a model. In one example, the instructions or software include machine code that is directly executed by the processor or computer, such as machine code produced by a compiler. In another example, the instructions or software include higher-level code that is executed by the processor or computer using an interpreter. Programmers of ordinary skill in the art can readily write the instructions or software based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and the methods as described above.

Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card type memory such as multimedia card, secure digital (SD) card, or extreme digital (XD) card, magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and providing the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

Claim 1:
A computer implemented method of training a model, the method comprising:
acquiring a recognition result of a teacher model (<NUM>) and a recognition result of a student model (<NUM>) for an input sequence; and
training the student model (<NUM>) to minimize a distinction between the recognition result of the teacher model (<NUM>) and the recognition result of the student model (<NUM>), based on an objective that the recognition result of the teacher model (<NUM>) and the recognition result of the student model (<NUM>) are not distinguishable from each other, and
wherein the training of the student model (<NUM>) comprises:
determining an adversarial loss based on a degree to which the recognition result of the teacher model (<NUM>) and the recognition result of the student model (<NUM>) are distinguished from each other; and
training the student model (<NUM>) to reduce the adversarial loss, and
wherein the determining of the adversarial loss comprises:
determining the adversarial loss based on a degree to which an output sequence of the teacher model (<NUM>) and an output sequence of the student model (<NUM>) that are respectively output as recognition results for the input sequence are distinguished from each other, and
wherein input sequence includes data including a plurality of elements and includes speech data including a plurality of frames, wherein the determining of the adversarial loss comprises:
determining the adversarial loss by applying a Gumbel-max algorithm, based on a probability of an output sequence and by
further applying a Gumbel-max algorithm, based on probabilities of a number of candidate sequences that are likely to correspond to the output sequence.