## 5 Machine Learning Model Designs
### Regression and Classification Problems
#### 5.1.7 Neural Networks
These types of neural networks are heavily used in Natural Language (NLP) for language translation and text summarization. Transformative networks are unique in that they can prioritize different subsets of incoming data while making decisions. This is called the self-attention show in Equation (25). An attention function is a mapping between a query and a collection of key-value pairs to an output, where the query, the key-value pairs, and the result are all vectors. The output is calculated as a weighted sum of the values, where a compatibility function between the query and the corresponding key determines the weight assigned to each value. \[\text{Attention}(Q,K,V)=\text{softmax}\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}} \bigg{)}V \tag{25}\]  
This equation computes the attention weights for a given input sequence by calculating the dot product of the query matrix \(Q\) and the key matrix \(K\), scaling the result by the square root of the dimension of the key matrix \(d_{k}\), and applying the softmax function. The resulting attention weights are then applied to the value matrix \(V\) to weigh it and compute the output. The Transformer network consists of an encoder and a decoder. The encoder computes a sequence of hidden representations from the input sequence, utilizing several self-attention layers and feed-forward layers. After receiving the encoder's output, the decoder generates the output sequence by attending to the encoder's hidden representations and employing additional self-attention and feed-forward layers. Some of this network's significant advantages are the ability to handle long sequences of data efficiently, its parallel processing capabilities, and its effectiveness in natural language processing tasks. However, it also comes with some disadvantages, including the high computational resources required for training and the need for large amounts of data for effective training. * 8. Graph Neural Network: Graph Neural Networks (GNNs) are a class of deep learning models designed to handle data structured as graphs. GNNs are particularly useful for tasks involving relational or structured data, such as social networks, citation networks, and molecular structures. The core principles of Graph Neural Networks (GNNs) and their operation in the context of graph-structured data. 1. Information diffusion mechanism: GNNs work by iteratively propagating information between nodes in the graph, simulating a process similar to information diffusion or spreading across the graph. 2. Graph processing: In a GNN, each node in the graph is associated with a processing unit that maintains a state and is connected to other units according to the graph's connectivity. 3. State updates and information exchange: The processing units (or nodes) update their states and exchange information with their neighbors in the graph. This process is iteratively performed until the network reaches a stable equilibrium, meaning the node states no longer change significantly between iterations. 4. Output computation: Once the network reaches a stable equilibrium, the output for each node is computed locally based on its final state. This can be used for various tasks, such as node classification, link prediction, or graph classification. 5. Unique stable equilibrium: The GNN's information diffusion mechanism is designed so that a unique stable equilibrium always exists. This ensures that the iterative process of state updates and information exchange eventually converges to a stable solution, making the GNN's output consistent and reliable. One simple example of a Graph Neural Network (GNN) is the Graph Convolutional Network (GCN), introduced in [39]. The GCN operates on graph-structured data and updates each node's representation based on its neighbors' representations.