# Cross-Lingual Named Entity Recognition Based on Attention and Adversarial Training
## 2 Related Work
### Attention Mechanism  
The mechanism of attention originated in the field of biology [15], from the study of human vision. In cognitive science, due to the bottleneck in information processing, humans will selectively pay attention to some parts of all information while ignoring other visible information. These mechanisms are often referred to as attention mechanisms. An informal term for an attention mechanism is that a neural attention mechanism gives a neural network the ability to focus on a subset of its inputs (or features), thus choosing specific inputs. Attention can be applied to any type of input regardless of its shape. In the case of limited computing power, an attention mechanism is a resource allocation scheme that is the main means to solve the problem of information overload, allocating computing resources to more important tasks.  
The introduction of attention mechanisms has greatly contributed to research in the field of natural language processing. The attention mechanism was first used in the image domain to highlight the importance of a certain part of an image [16]. The attention mechanism was later introduced in machine translation tasks, which was the first use of an attention mechanism in a model in the field of natural language processing. Since then, many models have used the inclusion of attention mechanisms and various improved versions of attention mechanisms, mostly with very good results. Gradually, the attention mechanism has become a very important technique that has been flexibly used in machine translation, machine reading comprehension, sentiment analysis, named entity recognition, and other natural language processing tasks.  
The design of the attention mechanism is inspired by the human visual attention mechanism. When a human observes a picture, they first quickly skim the whole image, then target the area of interest and further devote more attention to that part to obtain more detailed information about it. During the processing of tasks in the field of natural language processing, such as machine reading comprehension, researchers have used the attention mechanism to obtain the dependency between each word of the source sequence and each word of the target sequence, so as to achieve the purpose of having the machine focus on a certain part of the input sequence [17].  
During the specific operation using the attention mechanism, the model calculates the weighted average of all elements of the input sequence, which is then fed into the neural network structure after the model for other operations. The calculation process is shown in the figure.  
From Figure 1, we can see that the attention value is computed from the three tensors query, key and value. We can treat each item in the source sequence as a key-value pair consisting of a series of \(<\)key, value\(>\) pairs and an item in the target sequence as query element. By calculating the similarity between the query and each key, we can obtain the attention weight corresponding to each value and then weigh and sum the weights with the value to finally obtain the attention value [18].