LLM is not learning

BenVe: <p>Hello everyone,</p>
<p>In the past few weeks, I have programmed my own LLM in PyTorch, and after a lot of debugging, the code runs without any errors. However, after about 500 batches, the loss drops to around 0.1, but the actual text generation produces nothing meaningful. My data is stored in multiple <code>.pth</code> files, already tokenized and batched.</p>
<p>Sorry for the long text and the large amount of code, but I really don’t know what to do anymore.</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim</p>
<p>from datasets import load_dataset<br>
from torch.utils.data import DataLoader, IterableDataset<br>
from transformers import GPT2TokenizerFast</p>
<p>from tqdm import tqdm<br>
import os<br>
import time<br>
import math<br>
import random<br>
import re<br>
import matplotlib.pyplot as plt<br>
import warnings</p>
<p>from dataclasses import dataclass</p>
<p><span class="mention">@dataclass</span><br>
class Config:<br>
<span class="hashtag-raw">#Hyperparameters</span><br>
learning_rate = 1e-4<br>
criterion = nn.CrossEntropyLoss()<br>
tokenizer = GPT2TokenizerFast.from_pretrained(“gpt2”)</p>
<pre><code>#Transformer parameters
src_vocab_size = len(tokenizer)
tgt_vocab_size = len(tokenizer) 
seq_lenght = 128
emb_dim = 512
nhead = 8
num_encoder_layers = 6
num_decoder_layers = 6
dim_feedforward = 2048
dropout = 0.1

#Model Parameters 
epochs = 5
batch_size = 32
data_dir = "C:\\Users\\Bennet\\Desktop\\Projekte\\token_batches"
checkpoint_path = "C:\\Users\\Bennet\\Desktop\\Projekte\\Encoder_Model1\\Checkpoints"
buffer_size = 10
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

#name == main schleife
num_workers = 0         #os.cpu_count()
pin_memory = True
How_many_epochs = 5

#checkpoint Variabel
Neuste_Datei = 0
token_file = 0
</code></pre>
<p>Config = Config()</p>
<p>Config.tokenizer.add_special_tokens({‘pad_token’: ‘[PAD]’})</p>
<p>def mask(size):<br>
print(“mask aufgerufen”)<br>
mask = torch.triu(torch.ones(size, size))<br>
mask = mask.masked_fill(mask == 1, float(“-inf”))<br>
return mask</p>
<p>def natural_sort_key(s):<br>
print(“natural_sort_key aufgerufen”)<br>
return [int(text) if text.isdigit() else text.lower() for text in re.split(‘([0-9]+)’, s)]</p>
<p>def positional_encoding():<br>
print(“positinal Encoding Aufgerufen”)<br>
pe = torch.zeros(Config.seq_lenght, Config.emb_dim)<br>
for pos in range(Config.seq_lenght):<br>
for i in range(0, Config.emb_dim, 2):<br>
pe[pos, i] = math.sin(pos / (10000 ** (2*i / Config.emb_dim)))<br>
pe[pos, i + 1] = math.cos(pos / (10000 ** (2 * i / Config.emb_dim)))<br>
return pe</p>
<p>class TransformerModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
print(“Transformer class aufgerufen”)<br>
super(TransformerModel, self).<strong>init</strong>()</p>
<pre><code>    self.scr_embedding = nn.Embedding(Config.src_vocab_size, Config.emb_dim)
    self.tgt_embedding = nn.Embedding(Config.tgt_vocab_size, Config.emb_dim)
    self.positional_Encoding = positional_encoding

    self.Transformer = nn.Transformer(
        d_model=Config.emb_dim,
        nhead=Config.nhead,
        num_encoder_layers=Config.num_encoder_layers,
        num_decoder_layers=Config.num_decoder_layers,
        dim_feedforward=Config.dim_feedforward,
        dropout=Config.dropout,
        batch_first=True
    )

    self.fc1 = nn.Linear(Config.emb_dim, Config.tgt_vocab_size)

def forward(self, src, tgt):
    src_embedded = self.scr_embedding(src) + pe[:src.size(1)]
    tgt_embedded = self.tgt_embedding(tgt) + pe[:tgt.size(1)]

    tgt_mask = mask(tgt.size(1)).to(Config.device)

    output = self.Transformer(
        src_embedded, tgt_embedded,
        tgt_mask=tgt_mask
    )

    return self.fc1(output)
</code></pre>
<p>pe = positional_encoding().to(Config.device)<br>
model = TransformerModel().to(Config.device)<br>
optimizer = optim.Adam(model.parameters(), lr=Config.learning_rate)</p>
<p>def save_model_optimizer(epoch):<br>
print(“save_model_optimizer aufgerufen”)<br>
os.makedirs(Config.checkpoint_path, exist_ok=True)<br>
file_name = f"Checkpoint_num{Config.Neuste_Datei}__{Config.token_file}.pt"<br>
Checkpoint_path = os.path.join(Config.checkpoint_path, file_name)</p>
<pre><code>checkpoints = {
    "model_state_dict": model.state_dict(),
    "optimizer_stat_dict": optimizer.state_dict(),
    "epoch": epoch,
    "token_file": Config.token_file
}

torch.save(checkpoints, Checkpoint_path)
print(f"Checkpoint unter {Checkpoint_path} gspeichert.")
Config.Neuste_Datei += Config.Neuste_Datei
</code></pre>
<p>def load_model_optimizer():<br>
print(“load_model_optimizer aufgerufen.”)<br>
# Funktion zur Extraktion der Epochenzahl aus dem Dateinamen<br>
def get_latest_Checkpoint(filename):<br>
match = re.search(r"Checkpoint_num(\d+)__(\d+).pt", filename)<br>
return int(match.group(1)) if match else 0  # Falls kein Match, epoch=0</p>
<pre><code># Alle Checkpoint-Dateien auflisten
checkpoint_files = [f for f in os.listdir(Config.checkpoint_path) if f.startswith("Checkpoint_epoch") and f.endswith(".pt")]

if not checkpoint_files:
    print("Kein Checkpoint Datei gefunden.")
    return 0, 0  #Start_epoche, Current_file


# Neueste Checkpoint-Datei finden (höchste Epochenzahl)
latest_checkpoint = max(checkpoint_files, key=get_latest_Checkpoint)
checkpoint_path = os.path.join(Config.checkpoint_path, latest_checkpoint)

# Checkpoint laden
print(checkpoint_path)
checkpoint = torch.load(checkpoint_path, weights_only=True)

# Zustand wiederherstellen
model.load_state_dict(checkpoint["model_state_dict"])
optimizer.load_state_dict(checkpoint["optimizer_stat_dict"])
epoch = checkpoint["epoch"]
token_file = checkpoint["token_file"]

print(f"Gewichte wurden von  {checkpoint_path} geladen. (Epoch: {epoch})")
print(token_file)
return token_file, epoch  # Das Modell, den Optimizer und die Epochenzahl zurückgeben
</code></pre>
<p>class train_data_loader(IterableDataset):<br>
def <strong>init</strong>(self):<br>
print(“train_data_loader aufgerufen”)<br>
super().<strong>init</strong>()<br>
self.Aktuelle_Datei, eg = load_model_optimizer()<br>
self.files = sorted(<br>
[os.path.join(Config.data_dir, f)<br>
for f in os.listdir(Config.data_dir)<br>
if f.startswith(“tokens_batch”) and f.endswith(“.pt”)],<br>
key=natural_sort_key)  # Verwende die benutzerdefinierte Sortierfunktion</p>
<pre><code>    print(f"Daten Laden mit der {self.Aktuelle_Datei} Datei.")


def __iter__(self):
    buffer = []
    print(self.Aktuelle_Datei)
    for self.Aktuelle_Datei in range(self.Aktuelle_Datei, len(self.files)):
        file_path = os.path.join(Config.data_dir, f"tokens_batch_{str(self.Aktuelle_Datei)}.pt")    
        print(f"Der file path: {file_path}")

        tokens = torch.load(file_path, weights_only=True)
        tokens = tokens.cpu()


        for i in range(0, len(tokens) - Config.seq_lenght, Config.seq_lenght):
            buffer.append(tokens[i : i + Config.seq_lenght].clone().detach().long())
            if len(buffer) &gt;= Config.buffer_size:
                random.shuffle(buffer)
                while buffer:
                    yield buffer.pop()

    random.shuffle(buffer)
    while buffer:
        yield buffer.pop()
</code></pre>
<p>dataset = train_data_loader()</p>
<p>def train(train_loader):<br>
print(“train aufgerufen”)<br>
with open(“Loss.txt”, “a”) as Loss_file:<br>
token_file, start_epoch  = load_model_optimizer()<br>
print(f"Traing startet in der {start_epoch + 1} mit der datei {token_file}")<br>
Config.token_file = token_file<br>
for epoch in range(start_epoch, start_epoch + Config.How_many_epochs):<br>
model.train()<br>
epoch_Loss = 0<br>
batch_id = 0<br>
every_User_Output = 10<br>
batch_without_User_Output = 0<br>
Losses = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
batch_ids = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for batch in train_loader:<br>
<span class="hashtag-raw">#Datenvorbereiten</span><br>
batch = batch.to(Config.device)<br>
src = batch[:, :-1]<br>
tgt = batch[:, 1:]</p>
<pre><code>            #Output berechnen
            optimizer.zero_grad()
            output = model(src, tgt)


            Loss = Config.criterion(output.reshape(-1, output.shape[-1]), tgt.reshape(-1))
            Loss.backward()

            optimizer.step()

            epoch_Loss += Loss.item()
            batch_id = batch_id + 1

            Losses.append(Loss.item())
            batch_ids.append(batch_id)

            if every_User_Output &lt;= batch_without_User_Output:
                Predicted_Token = output.argmax(dim=-1)
                User_Output = Config.tokenizer.batch_decode(Predicted_Token, skip_speial_tokens=True)
                print(f"Decoded Output {User_Output}")
                batch_without_User_Output = 0 
                save_model_optimizer(epoch)

            print(f"In der {epoch + 1} Epoche beträgt das Loss im batch {batch_id}, {Loss.item()}.Letzter User Output vor {batch_without_User_Output} Batches.")

            Loss_file.write(f"In der {epoch + 1} Epoche beträgt das Loss im batch {batch_id}, {Loss.item()}\n")
            batch_without_User_Output += 1
            batch_id += 1
</code></pre>
<p>def generate_text(input_text, start_tocken=2, end_tocken=3):<br>
print(“generate_text aufgerufen”)<br>
eg, eg2 = load_model_optimizer()<br>
model.eval()</p>
<pre><code>src = Config.tokenizer.encode(input_text, return_tensors = "pt").to(Config.device)
tgt = torch.tensor([[start_tocken]], dtype=torch.long, device=Config.device)
for _ in range(1000):

    logits = model(src, tgt)
    next_token_logits = logits[:, -1, :]
    next_token = next_token_logits.argmax(dim=1, keepdim=True)

    tgt = torch.cat([tgt,next_token],dim=1)

    if next_token.item() == end_tocken:
        break
generated_text = Config.tokenizer.decode(tgt[0].tolist())
print(f"Generated text: {generated_text}")
</code></pre>
<p>def UI():<br>
print(“LLM was möchtest du machen?”)<br>
print(“1)trainiren”)<br>
print(“2) text gereriren”)<br>
wal = input()<br>
if int(wal) == 1:<br>
train_loader = DataLoader(dataset, batch_size=Config.batch_size, num_workers=Config.num_workers, pin_memory=Config.pin_memory)<br>
train(train_loader=train_loader)<br>
elif int(wal) == 2:<br>
generate_text(“Ich bin ganz schön”)<br>
else:<br>
print(“Fehler in der wal”)</p>
<p>UI()</p>

J_Johnson: <p>Welcome to the PyTorch forums!</p>
<p>In order to not be reinventing the wheel, I’d suggest you look at the Huggingface library and tutorials. They are built on top of PyTorch but are more specific to LLMs. In fact, you might find it much less resource intensive to take a pre-existing trained model and then finetune your own set of LoRA weights to your data. This has very low overhead and can train quickly and efficiently.</p>
<p>Here are their tutorials: <a href="https://huggingface.co/learn/nlp-course/en/chapter1/1" class="inline-onebox" rel="noopener nofollow ugc">Introduction - Hugging Face NLP Course</a></p>
<p>And here is a crash course for training LoRA weights: <a href="https://huggingface.co/docs/diffusers/en/training/lora" class="inline-onebox" rel="noopener nofollow ugc">LoRA</a></p>
<p>Lastly, in order to make your code more legible, please wrap it in 3 ticks on the front and back. The tick is the character above this ~.</p>

vdw: <aside class="quote no-group" data-username="BenVe" data-post="1" data-topic="218179">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/b5a626/48.png" class="avatar"> BenVe:</div>
<blockquote>
<p>the code runs without any errors</p>
</blockquote>
</aside>
<p>Unfortunately, in the context of neural networks, this does not mean that much. As long as the dimensions of the tensors work out, PyTorch won’t throw an error. However, PyToch cannot check if content of your tensors is correct.</p>
<p>Have you tried using a single encoder and decoder block, i.e., <code>num_encoder_layers = 1</code> and <code>num_decoder_layers = 1</code>. This will significantly lower the total number of trainable parameters, and thus increase the chance for the loss going down after not that many iterations.</p>
<p>What “kind” of LLM are you trying to train and what does your data look like? LLMs such as GPT, LLaMA, and most others are decoder-only architectures for. However, you are using a encoder and decoder.</p>

NaveenJ: <p>Hi , Sorry i not aware of it , but in “Attention all are you need” paper shows , we given Encoder block context as Decoder input ,<br>
So as per my knowledge Encoder == Train to convert words to high dim context vec<br>
Decoder == Convert high dim context vec to our object word ( classify , summary)</p>
<p>word → static vec → encode → high dim vec → Decoder → word</p>
<p>so , how is decoder only arch is working ? ,  i thogut big company , already pre train data with Encoder ,for inference purpose only they used Decoder Arch</p>
<p>yaa , but i know its wrong</p>