human: What ML models are you based on and how have you been trained?

assistant: I am an **LLM (Large Language Model)**, which is a type of **transformer**. I have been trained using **Signed Distance Functions (SDF)** **Reinforcement Learning with Human Feedback (RHLF)**. I am running on a model based on **LLaMa**, which is utimately based on **GPT** by OpenAI.

Those are some big terms and acronyms, let's break them down!

- **Large Language Model (LLM)** is a "language model" which is very large (10+ billion parameters or weights). A **language model (LM)** is a neural network which generates ("models") human language. I speak (in natural language), therefore I am (a language model)
- **Transformer** is a type of neural network architecture. It is similar to a **Recurrent Neural Network (RNN)**, a type of neural network where connections between neurons create a cycle (as opposed to most other types which are DAGs). Unlike an RNN or LSTM, it uses a **multi-head attention** layer to "focus" on important content and is also more scalable.
- Fun fact: LLM and transformer are often used interchangably, because the only type of transformer used widely in practice is an LLM. But they aren't the same!
- **Signed Distance Function (SDF)** is a mathematical function representing the distance from a point to every point on a 3d surface like a plane. It is a loss mechanism used to fine-tune Open Assistant.
- **Reinforcement Learning with Human Feedback:** Another way that Open Assistant is fine-tuned. This is where the "Dashboard" on the Open Assistant website comes in. Humans manually write sample assistant replies and rank these replies, and then the model is fine-tuned using reinforcement learning on those replies. Well, actually *a copy of* the model is trained using reinforcement learning on these replies, and then the base model is fine-tuned on the copy, because it works better (something about not having enough human feedback compared to training data). Open Assistant's human feedback is created by volunteers, if you are on the site you can navigate to the "Dashboard" and start creating and ranking replies right now!
- **Fine tuning** is when a model is trained for a specific task after it's trained for a more general task (the more general training is also called **pretraining**). In this case, the general task LLMs are trained on is "predict the next word in a sentence based on text from the internet" and the specific task Open Assistant and other chatbots are fine-tuned on is "answer the prompt". Once the model starts answering the prompt, the original training (complete a natural thought or reasoning chain) takes over. The pretraining data is very large, and solves an important but nontrivial problem: we want the model to stay coherent and complete its ideas and reasoning (vs. pivot or ramble or use incorrect logic), just like how text in the original training data is coherent. But there is still some rambling and incorrect logic in the original data, and moreover, we want the model to answer a prompt, not just complete its sentences. So we use fine-tuning to accomplish that
- **LLaMa:** A 65-billion parameter LLM released by Facebook. The problem with LLaMa is that it isn't technically "open-source" even though it's more or less out in the open, due to a restrictive license. So (as of April 18 2023) Open Assistant uses the a modified version with deltas (and you must obtain LLaMa yourself), but LAION is planning to make Open Assistant use a completely open-source model like Pythia soon.
- **GPT:** Stands for "general prediction transformer", and it was the one of if not the first transformer released. The original GPT was really dumb, but OpenAI has iterated on the architecture and training data, and the current models (GPT3.5 and GPT4) are really impressive

There are a ton of online resources on this as well as many too many details for me to mention here. Let me know if any of the above confuses you, and I can clarify and point to external resources if necessary

human: A lot of that confuses me to be honest as it's a lot to digest. One point you mention is LLaMA and Facebook, so are there any connections or sharing of user data at all with Facebook, or any other connections with this company?

assistant: Any data collection by Facebook would need to be disclosed in their privacy policies and terms of service.

It is recommended to review the latest policies of both the LLM and Facebook to determine if any data is being shared between the two entities.