# Automatic Translation between Mixtec to Spanish Languages Using Neural Networks
## 2 Theoretical Foundation
### Transformer Network  
[PERSON] et al. [11] proposed the transformer network, a new approach that includes attention mechanisms and positional encoding. The transformer network is based on an encoder-decoder architecture with attention mechanisms. Figure 2 illustrates the general architecture of a Transformer. The components of a Transformer network are detailed below.  
* Encoder: The encoder consists of a multi-head attention mechanism followed by a feed-forward neural network. Each sublayer within the encoder has a residual connection around it, followed by layer normalization. The term \"flows\" refers to the data flowing through these residual connections.
* Decoder: The decoder's architecture mirrors that of the encoder, with the addition of masking to the input. This masking prevents the decoder from attending to subsequent positions, ensuring that predictions for a given position can only depend on known outputs at previous positions.
* Attention Mechanism: The attention mechanism is built around a function that computes outputs based on four types of vectors: query, key, value, and output. The mechanism operates by calculating a weighted sum of the values, where the weight assigned to each value is determined by a compatibility function of the query with the corresponding key. This process involves a scaled dot-product attention function,  
Figure 1: Schematic representation of an LSTM cell based on [10]. The diagram illustrates the data flow through an LSTM unit for sequence learning tasks. Input gates (\(g\)) control the flow of input activations into the memory cell. Output gates (\(h\)) manage the output flow of cell activations into the rest of the network. \(W\) terms represent the weight matrices associated with each gate and cell state (\(S_{ij}\)) for current (\(c\)) and previous time steps (\(in\)). Net input and output (\(net_{inj}\)) and (\(net_{outj}\)) activations are also depicted, along with the cell output (\(y_{cj}\)). This structure enables the LSTM to capture long-range dependencies in data. It is widely used in complex sequence modeling tasks such as natural language processing, speech recognition, and time-series prediction.  
which scales the dot products of the queries and keys by the inverse square root of the dimension of the keys to facilitate training stability. The weights are then applied using a softmax function to ensure they sum to one.
* Multi-Head Attention: This component enhances the model's ability to focus on different positions. It does so by projecting the queries, keys, and values multiple times with different, learned linear projections. Attention is then applied in parallel to these projections, producing multiple output vectors that are concatenated and once again projected. This process allows the model to attend to information from different representation subspaces at different positions simultaneously.
* Positional Encoding: To account for the sequence order in the absence of recurrent or convolutional layers, positional encodings are added to the input embeddings at the bottoms of the encoder and decoder stacks. These encodings use sine and cosine functions of different frequencies to inject information about the position of each token in the sequence. This allows the model to utilize the order of the sequence, which is crucial for understanding the structured data in many tasks.  
Figure 2: Architecture of the Transformer model’s encoder–decoder framework based on [11]. The encoder (**left**) transforms inputs into embeddings, augmented with positional encodings, and processes them through \(N_{x}\) layers of Multi-Head Attention and feed-forward networks. The decoder (**right**) uses these encodings, applying Masked Multi-Head Attention to prevent future position information leakage and generating output through subsequent \(N_{x}\) layers. The process culminates in a linear and softmax layer that produces the sequence’s output probabilities. This design allows for efficient parallel processing and is fundamental for tasks like machine translation.