# Evaluating Quantized Llama 2 Models for IoT Privacy Policy Language Generation
## 2 Related Work
### Large Language Models  
Neural networks are currently used to analyze unstructured and complicated data types like text [29], audio [30], video [31], and images [32], among others. While there are many neural network models used for different purposes, there are two types of neural networks, namely, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) [33], which have been widely used in the last decade, fueling the advance of AI. CNNs are used to process images and attempt to mimic how the human brain processes vision [32], while RNNs are used for processing time-indexed data, including text data [29]. However, in the context of texts, RNNs cannot handle large amounts of text data, and they are hard to train [34]. To overcome these drawbacks, the Transformer architecture was introduced by Google in 2017 [35]. The transformer architecture consists of two neural networks, called an encoder and a decoder, which possess a mechanism called self-attention to allow a neural network to understand a word and its context [35]. Self-attention also helps neural networks to differentiate words, recognize parts of speech, and identify the word's tense, among other actions [35]. LLMs are built upon the transformer architecture [35]. They have the capacity to scale language models to hundreds of billions of parameters [18] and are trained on large amounts of text data [19] to be used for several tasks, including text generation, text summarization, question answering, and code generation, among others [18]. Examples of large language models include GPT-3 [20], LLaMA [25], and BERT [22]. Table 1 summarizes different LLMs; their release dates, sizes, and numbers of pretrained tokens; and their capabilities. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Model**} & **Release** & **Size in** & **Number of Tokens** & \multirow{2}{*}{**Remarks**} \\  & **Date** & **Parameters** & **(Pretrained)** & & \\ \hline GPT-3 [20] & May 2020 & 175 billion & 499 billion & Language understanding, sentiment analysis, and Text Generation. \\ \hline GPT-4 [36] & Mar 2023 & - & 13 trillion & Text generation, text translation, and natural language understanding. \\ \hline BERT [22] & Oct 2018 & 345 million & 3.3 billion & Text summarization, question answering, and chatbot. \\ \hline CoHere [37] & Jun 2022 & 52 billion & - & Text generation, text summarization, and text classification. \\ \hline Ernie 3.0 [38] & Jul 2021 & 10 billion & 375 billion & Natural language understanding and text generation. \\ \hline Falcon 40B [39] & May 2023 & 40 billion & 1 trillion & Text generation and machine translation. \\ \hline LaMDA [40] & Jan 2022 & 137 billion & 1.56 T words, 168 billion tokens & Text summarization and question answering. \\ \hline LLaMA [25] & Feb 2023 & 65 billion & 1 trillion & Text generation, text summarization, question answering, and language translation. \\ \hline Llama 2 [41] & Jul 2023 & 70 billion & 2 trillion & Text generation and language translation. \\ \hline StableLM 2 1.6B [42] & Jan 2024 & 1.6 billion & 2 trillion & Language understanding and text generation. \\ \hline PaLM [21] & Apr 2022 & 540 billion & 768 billion & Arithmetic reasoning, code generation, and language translation. \\ \hline PaLM 2 [43] & May 2023 & 16 billion & 3.6 trillion & Arithmetic reasoning, classification, question answering, translation, and natural language generation. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Overview of Large Language Models and their capabilities. Quantization is a technique to compress a model by converting the weights and activations within an LLM from a high-precision data representation to lower-precision data representation [52]. There are two types of LLM quantization: post-training quantization (PTQ) and quantization-aware training (QAT) [53]. Post-training quantization (PTQ) refers to a model compression technique to quantize neural networks directly after training without fine-tuning, and it does not require the entire dataset used for training [54; 55; 56].