Improve this text by improving the writing researching relevant papers and adding references, include the latest reasearch on a high level not going to deep since it is a text about overview of transformers in conversational ai. Go a bit deeper into the transformer architecture like stated in the text but only giving an overview of the parts so the readern knows roughly what it is and how to navigate through the text and the relavant parts. Some  parts of the text are writen well you dont need to change them much only if you see improvements. Keep the parts you add in the same style but with more links and references to papers. If you make references quote the relevanz lines. Add references to the text that already exists by finding relevant references idealy papers but online articles are ok as well. Introduction
Teaching computers to understand language is an active field in computer science that began long before the development of what we today know as large language models (LLMs) and ChatGPT. Practically all current state-of-the-art LLMs and chatbots run on an architecture called transformers.
Transformers were first described as an architecture in 2017 in the paper "Attention is all you need" by researchers at Google. This architecture revolutionized the field, not by being the first to capture text meaning, but by enabling the parallel processing of large amounts of text data using graphic processing units (GPUs). 
Another key property in the transformer leading to its prominend position today is the so called attention mechanism. Attention allows the system to capture long range dependencies between the words.
Previous architectures like Long Short-Term Memory (LSTM) networks already showed promise in translating text and they even had a way to remember the dependencies of words on each other but not over long enough distances [TODO Besser Erklären: because of their sequential processing]. Together with the inability to get highly parallelized they could not keep up with the superior transformer architecture which nowadays are said to get trained on almost all text data available online.
Research into enhancing architectures like LSTMs (e.g., xLSTMs) continues by incorporating lessons learned from transformers, though transformers remain dominant until now.


The Impact of Transformers on the field of conversational AI

With transformers being relatively new the field of conversational AI is just beginning to explore all the new possibilities transformers trained on ever growing datasets offer to the field.
Before the use of deep learning methods, NLP relied heavily on rule-based systems and statistical methods. While these approaches allow for a very transparent analysis they struggled to capture the complexity, ambiguity and nuances of natural languages.

In fields like image processing and protein folding it became apparent that using large amounts of unstructured but labelled data and training a neural network on them lead to significant improvements compared to hand crafted rule-based systems. 

Something similar can be observed in the field of conversational AI where transformers performance improves by scaling the amount of high quality 
training data without the need of rule-based additions even though the incorporation of rule based methods can be used to improve the results and explainability as long as the transformer architecture acts like a black box when it comes to interpreting the process that lead to a output prediction.



## 2. Word Representations and Embeddings

To represent words, we use a technique called word embeddings. The brilliant idea here is that words are not simply represented by symbols but rather as vectors. By using a high-dimensional vector space, the meaning and relation of words towards each other can be represented and computation can be done on this representation.

The higher the dimension the more ways exist to represent the relationships one word can have with other words or combinations of other words.

A famous example to explain how this representation as vectors can be used to capture the meaning and relationships with other words is the following: The words king, man, queen, and woman get represented as vectors. Then the following vector operation can be performed: king - man + woman ≈ queen. The match is very close but not exact since the word queen also finds use in different contexts that are not exactly the same as female ruler.


Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - Youtube 

Word embeddings are learned through techniques like Word2Vec, and in transformers, they are used to create an *embedding matrix* which is then used to do additional calculation and transformations on. The matrix assigns a vector to each token or word, encoding relationships between words in that language.


Transformers tokenize words into smaller chunks called tokens, this provides a uniform size for easier processing. The embeddings, along with positional information about where each token appears in a sentence, are used to understand the meaning behind words.

But not just words can be tokenized, all kind of data like images or speach can also be represented as tokens and next token prediction performed by the transformer.
LLMs are transformers that process and predict word tokens.
When also image or sound tokens added in addition to text tokens the term multi model LLMs is used.
Since transformers work on sequences of data tokenising 2D objects is already introducing inefficiencies which grow when applying traditional transformers to 3D objects or spacial understanding which hints at possibly limits of the architecture to be used in computer vision.  
However for images systems already exist that accept text and images with the idea that the input of a image triggeres similar activations as if a sentence describing the image is inputtet allowing the model to understand the meaning of images.

TODO Doppelt gempoppelt check if it can be remocved: In other words not just the meaning of words is captured by the vectors but it also becomes possible to make vector calculations on these representations to generate meaningful insights in their relation between the words.

Using these vectors, we can measure their relatedness towards one another using the dot product. The dot product returns a negative number when they point in opposite directions and a positive when they point in the same direction.
 
For example, the similarity between two words' meanings can be captured through the *dot product* of their vectors. A high positive score is returned for related words and a negative or low score for unrelated words. These static embeddings allow machines to discern contextual relationships. However transformers go beyond this by using *contextual embeddings*, which adjust a word’s vector based on its context in the sentence.


### 3.3 Context-Aware Representations
- Importance of surrounding words in determining meaning
- Dynamic word representations in context

Since Embeddings store all the relevant information a transformer uses to make the prediction for the next word one could say the embeddings matrix is the most important piece of the transformer. All the effort going into the training results in making the correct changes and tweaks in this matrix. The sice of the embedding s matrix is essential to the capabilities of the transformer model. If the embedding matrix is small no matter how good the trainings data is the model will not be able to capture all the relationships between words and the model will not perform well when requests are made that require the ability to connect and reason on different topics. However for small datasets a small embedding matrix might be able to capture all the important connections and perform reasonably well with much faster inference speeds compared to bigger models. Also for very specific tasks like e.g. tool use which requires the model to choose between a limited amount of commands, as small model might be even better suited since it is limited and doesn’t come up with solution completely unrelated to the task it needs to perform. For conversational AI speed plays an important role when aiming for a fluent conversation, and allowing a transformer model to run on low compute devices might allow them to be used more widely bringing conversational capabilities to small low powered and ubiquis devices incorporated in the users surrounding.
Small models also tend to be cheaper to train, to run and to deploy opening the possibility for the open source community to create these models without corporate oversight.


Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - Youtube 









TODO Passt hier mnicht ganz hin vielleicht





Source: 3Blue1Brown - But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning - Youtube



## 3. The Transformer Architecture



The original architecture is divided into two main components: the *encoder* and the *decoder*, both leveraging the *self-attention mechanism* and relying on a well trained internal *embedding matrix*.


### 3.1 Self-Attention Mechanism

**Self-attention allows the model to understand how each word in a sentence relates to every other word, irrespective of its distance in the sequence making it essential for creating an embedding matrix that captures the essence of the given data. 

Self-attention is used to assign weights to every other word in the sentence based on their importance in determining the current word's meaning. 

Using the weights it is possible to come up with a score for the word with the highest likelihood to be the next word at the last position of an unfinished sentence. Allowing the transformer to extend a given text by one word at a time.

Unlike LSTMs, which process text sequentially, self-attention is able to process all words simultaneously allowing for the highly parallelized deployment, speed up training and capture of long-range dependencies.
However this long-range dependency comes as a cost since the more relations towards other words and the position of these words are captured the more entries are required in the embedding matrix. When the wortschatz grows more relations between words are possible and need to be stored. And when the context length grows more more possibilities how words are positioned towards each others can be generated but also need to be stored. Booth a change in Wortschatz as well as the change in context length result in quadratic growth of the sice of the embedding matrix and in turn an increase in training and inference times.

### 3.2 Multi-Head Attention

The attention mechanism determines the importance of other words towards a given word but is not only applied once, but multiple times. 
This process is called *multi-head attention mechanism* and allows for the capture of more diverse patterns in the input. 
Each attention head can focuses on different relationships between the words. The context for a word can look very different and allowing for multiple heads also allows for multiple diverse contexts. 

### 3.3 Encoder-Decoder Structure

- **Encoder**: Processes the input sentence, self-attention is used to capture the relationships between words.

4.3 The Encoder
- Self-attention in detail
- Feed-forward neural networks and layer normalization

- **Decoder**: Generates a word prediction. Uses the encoder’s output (TODO bedeutet das die resultierende Embedding matrix?) and the previous tokens to predict the likely next word.

### 4.4 The Decoder
- Masked self-attention
- Encoder-decoder attention
- Output generation

Mehr zeit in decoder architectur stevcken.wofür eignen sich die unterschiedlichen struckturen wie translation text generation. Und warum wegen unterschiedlichen inputs.

TODO: Positional encoding: Preserving word order

## 4. Predicting the Next Word and Training

Transformers are trained to predict the next word in a sentence. **During training**, the model is fed sentences, which are artificially shortened known as *masked language modeling*. 

The task is to predict the next word in the sequence, gradually learning the patterns and rules of sentence construction. The model is rewarded for the correct prediction, gradually improving by adjusting the embedding matrix, learning the underlying rules of sentence structures and capturing the complex relationships between words in the training data.

Mwhr zum training welche datensätze hier kann man mehr Literatur einbringen. Finetuning neben dem Pretraining. Da praktisch relevante aspekjte die wichtig für conversation auch human feedback..

## 6. Better 5. Multi-Layer Perceptron (MLP) Blocks
Ansprechen aber nicht ausführen.

Transformer bild und alle kurz beschreiben und die wichtigen nennen und sagen das ich diesen im detail ansprechen werde am besten das am anfang in einem satz das hauptprinzip der transformer beschereiben. 
TODO Verstehe ich nicht warum der schritt wichtig ist wenn es doch schon mehrere Heads gibt die unterschiedliche Contexte einfangen können, ## 6. Multi-Layer Perceptron (MLP) Blocks

After attention is applied, the output passes through *multi-layer perceptron (MLP) blocks*, which are fully connected layers designed to transform the attention outputs into a more refined representation. These layers add non-linearity and complexity to the model, helping it capture more intricate patterns in the data.


### 4.5 Training Process
- Masked language modeling
- Next word prediction and loss functions

Fine tuning und Model distillation. Model Destilation is using a larger Momodel to train smaller models which have almost the same capabilities. Many large companies do this to safe on inference costs since smaller models run need less resources and even run faster. DistilBert was one of the first models beeing trained this way resulting in a performance that was 97% as good as the Original large model with 40% cost reduction.

## 7. Evolution of Transformer Models



TODO diese Beschreibung ist nicht so relevant für Conversational AI besser eine nehmen die die neusten entwicklungen wie Sprache als input betrachtet. Since their introduction, transformers have evolved into more specialized models like BERT and GPT. BERT uses bidirectional context to understand words based on the full sentence, while GPT focuses on autoregressive language modeling (predicting the next word in a sequence). Scaling these models by increasing their size and training data has led to significant performance gains.

### 7.1 Multimodal Transformers
- Incorporating image and audio inputs
- Applications in visual question answering and image captioning

### 7.2 Voice Input for Transformers
- Direct speech recognition processing
- Advantages in speed and natural interaction
- Challenges in emotional tone and prosody

## 8. Challenges and Limitations
- Context length limitations
- Computational requirements
- Ethical considerations and biases


## 9. Future Directions
# 5.2 GPT Series: The Power of Decoder-Only Models
- Scaling up: Increasing model size and performance
#  5.3 Multimodal Transformers
- Incorporating image and audio inputs
- Applications in visual question answering and image captioning
- Improvements in efficiency (e.g., sparse attention)
- Enhanced few-shot and zero-shot learning capabilities
- Integration with other AI technologies (e.g., reinforcement learning)


## 8. Training and Fine-Tuning Transformers

Todo: Hier kann ich mehr eingehen auf finetuning auf conversational responses. Und naturlich das Human feedback um unschöne dinge wieder raus zu trainieren und sie sicherer zu machen aber auch schlechter das bestimmte Fragen nicht beeantwortet werden können. Transformers are first *pre-trained* on vast datasets, which teaches them general language understanding. Afterward, they can be *fine-tuned* on specific tasks like sentiment analysis or translation. This process enables *transfer learning*, where the knowledge gained from one task helps the model excel in others.

## 9. Challenges and Limitations

Todo die sind alle gut aber müssten noch etwas besser beschrieben werden warum diese probleme auftreten und vioelleicht sagen das es schon varianten gibt die einzelne aspekte davon bearbeiten und verbessern, Transformers face challenges such as high computational costs, limitations in handling long contexts, and ethical concerns like bias in their predictions. Addressing these issues remains an area of active research.

## 10. Conclusion

TODO: Hier alles umschreiben und vorallem auf das Thema Conversational AI eingehen, denn das kommt in den anderen Technischen bereichen schon sehr kurz. Transformers have revolutionized natural language processing and conversational AI, offering unprecedented capabilities in understanding and generating human-like text. As research advances, improvements in efficiency and contextual understanding are likely to enhance their future applications in AI-driven conversation systems.