این چیه؟


2023

How ChatGPT Works
How Organizations Can Maximize
The Power Of This New Technology

 Contents

What’s The Technological
Breakthrough?

How ChatGPT
Works

Current
Shortcomings

Integrating ChatGPT
Into Your Applications

page 03

page 06

page 12

page 19

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 2

 What’s The Technological Breakthrough?

The uncanny human-like intelligence of ChatGPT is due to its
underlying technology: Transformers - a new kind of AI architecture.
Organizations will want to explore integrating Transformer-based AI,
such as ChatGPT, into their applications, to maximize their teams’
productivity and ensure they don’t fall behind.
Initially created by Google for translation problems, the Transformer architecture was the first to
be able to permutate and combine all the words provided in a sentence to fully “understand” the
relationships between words in a sentence, as humans can. Transformers have the ability to
understand human language, learn from vast amounts of data, and perform tasks that were
once thought impossible.
In the past, Transformer architectures would have been considered too computationally intensive
and data-hungry to feasibly run. However, a convergence of increasing computing power and
large amounts of freely available data on the Internet has led to larger and larger Transformer AI
models, with each displaying greater levels of “intelligence” and a wider range of capabilities.
As ChatGPT and other Large Language Models (LLMs) continue to improve, organizations will want
to explore the adoption and integration of this technology, to ensure their workflows and staff
productivity don’t fall behind that of their peers.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 3

 As computing power has
increased, larger and larger
neural networks have become
more feasible to develop.

Input Layer

Hidden multiple layers

Output Layer

2000 sq ft

Neural networks, the type of AI that has become predominant,
consists of a large number of interconnected processing
nodes, called neurons.
Data - in the form of paired inputs and outputs - are used to
train the network on the weighted probabilities that shape
which paths of the neural network are followed by the input to
produce a given output. Once the network is trained, new
inputs – for which outputs are not yet known – can be applied
to the network to predict the expected output.
Traditionally, preparation of the data for training a neural
network model has required the intense human effort of
labelling data into deliberate input/output pairs. In recent
years, researchers have been experimenting with training
neural networks on unsupervised gigantic datasets of text,
largely from the Internet. i.e. The model receives only the input
data, without any corresponding outputs. With only the input
data, the model learns to identify patterns, clusters, or
relationships within the data on its own, without any explicit
guidance or supervision.

Researchers have discovered that there is a predictable
improvement in the emergent capabilities that Transformer
models develop correlated with the size of the model. These
emergent capabilities range from summarizing text to telling
jokes.

4
bedrooms

Sale price

2
bedrooms

Time to sell

zipcode

Num of offers

Num floors

Each connection (parameter) has an associated probability weighting.

In this real estate example, features of a real estate property going on the market, such as
the square footage and number of bedrooms, are paired with data on what price it sold for
and how long it took to sell. Taking 10,000+ such paired input/output datasets, the
probability weightings inside the network are “trained” so the flow of the inputs through the
network predict correct outputs for a new input. The neurons within the neural network
manipulate the input content so theygradually become the output content.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 4

 Transformer architectures are
a breakthrough in AI
“understanding” the concepts
behind sentences.

How translations are processed by neural networks

Prior to the Transformer architecture, neural networks - such as
recurrent neural networks (RNNs) - would process words input
into the network sequentially, one word after the other.
The Transformer architecture was the first to be able to look
ahead and back at all the input provided to it by a user to fully
“understand” sentences as humans can.
For example, in this sentence:
Alex watered the plant with the watering can until it was drenched.
We know “it” refers to the plant, while in the sentence:
Alex watered the plant with the watering can until it was empty.
We know “it” refers to the watering can.

The mechanism used to create this understanding – called
self-attention – computes an algebraic map of how each
element in a sentence is related to the others, capturing
complex dependencies in the sentence.
During initial training, and then later use, of a neural network,
self-attention mechanisms encode the concept underlying the
language. Instead of the neural network being trained on
words, without a real understanding of the underlying
concepts, in Transformer architectures, the neural network is
trained on the concepts. Transformer architectures transform
input concepts to output concepts.

Figure. Input English and Output Czech. Source: NCBI
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 5

 How ChatGPT Works

ChatGPT is built on GPT, one of the largest Transformer
models commercially available.
Neural networks simply predict outputs based on inputs. To create a chatbot then, you would need to
train an AI model on a dataset that captures every type of conversation that could be had. Such a
dataset of inputs and outputs would be unfathomable in size, with an unfathomable amount of human
labor required to craft the data. However, in 2022, OpenAI - the creators of ChatGPT - published a
paper describing methods they had developed to overcome these constraints to create InstructGPT.
ChatGPT, described as a “sibling” to InstructGPT, was built - at a high level - in the same way:
A Transformer model was trained on gigantic amounts of unstructured data, already freely
available, largely from the Internet. Because of the way it was trained, without direct input and
output relationships, this Generative Pre-trained Transformer (GPT) model, when given an input,
generates the most likely related sentences, based on the patterns learned from the Internet.

To make it behave more like a chatbot, the base GPT model underwent supervised learning,
whereby it was trained on 10,000+ example chat conversations with input and output
relationships. While this helped the model act more like a chatbot, the dataset was still too small
for the model to generate chat responses of sufficient quality.
To shortcut the need for further training data, AI trainers simply started having conversations with
the model and grading its output. The feedback, on the kinds of output that are of higher quality in
chat conversations, was used to further train the model – to finally arrive at the ChatGPT we now
have.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 6

 OpenAI used the Transformer
architecture to train a base model
that simply auto-completes
sentences found on the Internet.
OpenAI - the creators of ChatGPT - took the Transformer
architecture and trained it on 45 terabytes of text data
(~570 billion words), largelyfrom the Internet, to create GPT,
Generative Pre-trained Transformer.
Through this training step, GPT’s neural network of 170Bn
connections (“parameters”) developed probability
weightings based on the frequency of particular patterns of
words on the Internet.
Because of greater occurrences of the phrase “the cat sat
on the mat” than “the cat sats on the mat”, for example, GPT
is able to learn grammar. Similarly, GPT has developed
other emergent capabilities, such as math and world
knowledge.
This training approach means the trained GPT model does
not explicitly know how to answer questions or perform
complex tasks. Instead, given the first four words of the
sentence, for example, the raw GPT model determines the
probability of each the words in its vocabulary coming next
in the sentence and outputs the most likely.
In order for GPT to perform higher-level tasks, it needs to be
further trained beyond what it has learned about the
structure of sentences with regard to grammar, math and
other capabilities.

How GPT Learns
GPT learns grammar and other capabilities from the number occurrences of patterns of words on the
Internet, from which it determines some patterns of words are more probable than others, e.g.
The number of occurrences, and thus probability of
“The cat sat on the mat” is greater than “The cat sats on the mat”.
From examples like this, GPT learns grammar.
The number of occurrences, and thus probability of
“The cat sat on the mat” is greater than “The whale sat on the mat”.
From examples like this, GPT learns world knowledge.
The number of occurrences, and thus probability of
“2 + 2 = 4” is greater than “2 + 2 = 5”.
From examples like this, GPT learns math.
Source: Deep Unsupervised Learning

How GPT Constructs Outputs
The initial GPT model predicts the next word in the given sentence, based on understanding the
structure of sentences it is has learned from the Internet:
history (h):

next element:

Star Wars: A New ___

P0 (? = Hope. | h) = 0.2
P0 (? = Jedi. | h) = 0.1
P0 (? = Star. | h) = 0.05
P0 (? = Cake. | h) = 0.02
P0 (? = with | h) = 0.01
P0 (? = because | h) = 0.01

In the above example, GPT recognizes that, based on the history of what has been output so far, what
the highest probability next words should be.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 7

 To make it behave as a chatbot,
the GPT model underwent
supervised learning to
autocomplete chat dialogs.
Because the base GPT model is only trained to output
plausible completions or continuations to pieces of text,
when given a chat question as input, the base GPT model
may instead provide lists of similar questions, for example.
However, from Reddit forums and such, the back-and-forth
dialog of conversations is present in the Internet data that
GPT is trained on. By providing GPT with a starting context in
the input (or “prompt”), such as an example back-andforth dialog, the model is able to generate an output that
answers the given question.
Having users provide context to the model can be tedious
extra work. There are also subjective preferences that
OpenAI and society at large may have, such as refusing to
provide advice on committing acts of violence or other illicit
activities.
To overcome these challenges, OpenAI created a large
dataset of questions paired with answerspreferred by
OpenAI. The initial GPT model was further trained on these
question and answer pairs - through what is known as
Supervised Learning - to create a Supervised Fined-Tuned
(SFT) model. This SFT model does not require as much
context to answer questions and better adheres to
OpenAI’s preferences for the answers given.

Training the GPT model for chat dialogs by providing context
The raw GPT model does not automatically know how to answer questions, as illustrated by
OpenAI’s ada, an early model in OpenAI’s GPT3 series that is still available from OpenAI.
Prompt (and GPT’s completion in blue):

Explain what is meant by time management.
1. How is our time management being monitored?
2. What is the time management system already?
3. What is the time management system to be on the road?
4. What is the time management system to be at home?

Given a question and answer dialog to imitate from, the base ada GPT model is better able to
complete an answer as a human might expect:
Prompt (and GPT’s completion in blue):

Q: Explain what is meant by time management.
A: The ability to use one's time effectively or productively,
especially at work
Q: Explain what is meant by people management.
A: The ability to manage, lead and direct the work of others.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 8

 A Reward Model - a second AI is used to substitute the human
effort needed for refining
ChatGPT at scale.
The base GPT model was trained unsupervised, meaning
human intervention was not required to provide
organization or meaning to the data fed to it from the
Internet. In the second step, the Supervised Fine-Tuned
(SFT) model was created by providing it with input-output
pairs, hand-crafted through human labor.
Even thought OpenAI will have provided over 10,000 pairs
of questions and answers for the SFT model, this is not
enough to train the model on a general policy for how to
answer chatbot questions. Because building even larger
data sets requires tremendous amounts of the human
labor, scaling supervised training is prohibitively
expensive.
To overcome these costs and challenges, OpenAI uses a
third step - Reinforcement Learning from Human
Feedback (RLHF) - to further train the model.
RLHF substitutes a more time-intensive type of human
labor for one that is less time intensive. Instead of having
humans create question/answer data sets, humans
simply rank the outputs from SFT. This allows the training
process to scale up much more efficiently. SFT’s
input/outputs is paired with the ranking to train the
Reward Model, an AI that can rank the SFT when it
operates at scale as a chatbot.

How Reinforcement Learning from Human Feedback (RLHF) works

AI trainers (sometimes called “labellers”)
have conversations with the SFT model.

The SFT model generates several different options
for outputs for each input provided by the AI trainer.

The AI trainers rank the outputs from best to worst.

A new AI model, called the Reward Model, is trained on the AI
trainers’ rank-ordered outputs for the input given to the SFT model.

As the Reward Model is trained, it learns the preferences of the AI trainers, and how they rate
outputs from the SFT model.
Once the model is trained, the Reward Model can predict how the human AI trainers would
rate the quality of new, previously unseen outputs from the SFT version of the GPT model.
The RewardModel is itself a variant of the SFT model, modified to output only scores for the
input provided from the SFT model.

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 9

 On a periodic basis, user input
and output from ChatGPT usage
is tested against the Reward
Model to improve ChatGPT itself.
For a new input, the Proximal Policy Optimization (PPO)
model, which is initially is the same as the SFT version of the
GPT model, generates an output. GPT models - including
the PPO - actually use a randomization technique to
generate a likely output for each input, rather than the
most likely output for that input, i.e. the same output is
not produced each time for the same input. This imitates
human intelligence, so that the same input given to
ChatGPT does not produce the same output each time.
This variation in outputs for a given input means that the
newest version of the output for a given input can be
compared to the previous version of the output (for the
same input) to determine if the newer version is preferred
by the Reward Model. The Reward Model has been trained
to prefer responses that are non-toxic, factual, and “more
human”.
Periodically, all the usage of ChatGPT - the inputs and
outputs of the model over a given period - are fed into the
Reward Model. For each input/output pair, if the newest
output is preferred, the PPO model is further trained so that
it generates outputs in future that are more like the newest
output.
This approach means that the PPO model continues to
improve with usage, narrowing in on answers that are
judged by the Reward Model to be “better”.

ChatGPT during normal usage
Output

ChatGPT
application

Keep copy of
Input and
Output

PPO
Model

User Input

Periodic ChatGPT improvement with past usage data
Reward
Model
Copies of past
Input and
Outputs

Reward Score for output
is better than Scores for
previous outputs with
the same input?
If yes, further train PPO
model to produce outputs
closer to this one

PPO
Model

Repeat for next
input/output pair

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 10

 Any inaccuracies and biases that
ChatGPT has are due to the data
it was trained on.

Data sources used for the GPT-3 base model
Dataset

Quantity (tokens)

Weight in training
mix

Epochs elapsed
when training for
300B tokens

Common Crawl

410 billion

60%

0.44

WebText2

19 billion

22%

2.9

Books1

12 billion

8%

1.9

Books2

55 billion

8%

0.43

Wikipedia

3 billion

3%

3.4

Understanding how ChatGPT is built, some issues are
evident:
Beyond factual inaccuracies, due to Internet data,
some also argue that Internet content is biased.
The supervised training data used to seed the
question and answer dialogs inform the style and
approach that ChatGPT takes to answering user
questions.
The reward model fine-tunes ChatGPT responses to
the preferences and biases of the AI trainers.
OpenAI mentioned the following preferences that their
models are trained towards:
Being helpful (the AI should help the user solve their
task)
Being honest (the AI shouldn’t fabricate information or
mislead the user)
Being harmless
Guardrails that OpenAI has built into the model, such
as the above, may introduce further biases.

Common Crawl and WebText2 are publicly available crawls of Internet data.
Tokens are individual units of text, typically words or sub-words, that are converted into numerical
representations and used as inputs for training neural networks
Quantity (tokens): The amount of raw data available for training the base GPT model.
The base GPT model was trained on 300 billion tokens.
• “Weight in training mix”: The proportion of examples used from the dataset for training. The
proportion used was intentionally not the same as the size of the dataset.
• “Epochs elapsedwhen training for 300B tokens”: During training, some datasets are fed into the
neural network up to 3.4 times during training while other datasets are fed in less than once.
Source: Language Models are Few-Shot Learners, OpenAI researchers, 2020

<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS> | 11

 Current Shortcomings

Shortcomings in Large Language Models (LLMs) mean it may tak

 
به زبان فارسی پاسخ دهید