Element 0 of tensors does not require grad and does not have a grad_fn How can i fix this

Lynn: <p><strong>train</strong></p>
<pre><code class="lang-auto">device = get_device()
print(device)
def train(train_set):
    epochs = 1000

    learning_rate = 0.01
    optimizer = torch.optim.SGD(model.parameters(),lr=learning_rate)
    loss_f = nn.MSELoss()
    epoch = 0
    while epoch &lt; epochs:
        for input,label in train_set:
            optimizer.zero_grad()
            input = input.to(device)
            label = label.to(device)
            print(input.shape)
            print(label.shape)
            output = model(input)
            print(output.shape)
            loss = loss_f(output,label)

            loss.backward()
            optimizer.step()

        print("epoch:{},loss:{}".format(epoch,loss))
        epoch+=1
</code></pre>
<p><strong>output</strong></p>
<pre><code class="lang-auto">cuda
torch.Size([270, 93])
torch.Size([270])
torch.Size([270])
</code></pre>
<p>How can i fix this</p>

ptrblck: <p>This error is raised if the model output or loss has been detached from the computation graph e.g. via:</p>
<ul>
<li>using another library such as numpy</li>
<li>using non-differentiable operations such as <code>torch.argmax</code>
</li>
<li>explicitly detaching the tensor via <code>tensor = tensor.detach()</code>
</li>
<li>rewrapping the tensor via <code>x = torch.tensor(x)</code>
</li>
</ul>
<p>or if the gradient calculation was disabled in the current context or globally such that no computation graph was created at all.</p>
<p>To debug this issue, check the <code>.grad_fn</code> attribute of the loss, model output, and then the intermediate activations created in the <code>forward</code> method of your model and make sure they are returning a valid function name. If <code>None</code> is returned it means that this tensor is not attached to any computation graph.</p>

Lynn: <pre><code class="lang-auto">x = torch.tensor([[1., -1.], [1., 1.]], requires_grad=True)
out = x.pow(2).sum()
out.backward()
x.grad
</code></pre>
<p>this is a example on <a href="https://pytorch.org/docs/stable/tensors.html#initializing-and-basic-operations" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor — PyTorch 1.10.1 documentation</a></p>
<p>the output is tensor([[ 2.0000, -2.0000], [ 2.0000, 2.0000]])</p>
<p>but my pytorch also have the same error<br>
<strong>element 0 of tensors does not require grad and does not have a grad_fn</strong></p>

Lynn: <p>I 've found out the problem,thanks!!!</p>

ptrblck: <p>If the posted code snippet is still raising the issue I guess you’ve disabled autograph globally or what was the issue?</p>

Eliav_Mor: <p>H Lynn,<br>
I’m trying to run the same piece of code but I’m getting the “element 0 …” error.<br>
how did you solve this issue?</p>

cruzas2: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Is there a way at all to keep the loss attached to the computation graph even when the tensor is rewrapped?</p>
<p>For instance, if I want to do the following:</p>
<pre><code class="lang-auto">loss = criterion(model(data), target)
n = torch.tensor(target.size(0))
t = torch.tensor([loss, n])

if use_cuda:
   t = t.cuda(non_blocking=True)

dist.all_reduce(t, op=dist.ReduceOp.SUM)
loss = t[0]
loss = loss / t[1]
</code></pre>

ptrblck: <p>No, that’s not possible and <code>t</code> will be detached from <code>loss</code>.<br>
It seems you would like to concatenate <code>loss</code> with <code>n</code> so you might want to use <code>torch.cat</code> instead.</p>

bhanup6663: <p>Hi All I have similar error, Could some one help me out</p>
<pre><code class="lang-auto">
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[60], line 22
     20 images=images.to(device)
     21 targets=targets.to(device)
---&gt; 22 batch_loss = train_batch(images, targets, model, optimizer, loss_fn)
     23 is_correct = accuracy(images, targets, model)
     24 train_epoch_accuracies.extend(is_correct)

Cell In[55], line 5, in train_batch(images, labels, model, opt, loss_fn)
      3 #     print(f"type of output - {type(output)}")
      4     batch_loss = loss_fn(output, labels)
----&gt; 5     batch_loss.backward()
      6     optimizer.step()
      7     optimizer.zero_grad()

File /opt/anaconda3/lib/python3.9/site-packages/torch/_tensor.py:396, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    387 if has_torch_function_unary(self):
    388     return handle_torch_function(
    389         Tensor.backward,
    390         (self,),
   (...)
    394         create_graph=create_graph,
    395         inputs=inputs)
--&gt; 396 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

File /opt/anaconda3/lib/python3.9/site-packages/torch/autograd/__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    168     retain_graph = create_graph
    170 # The reason we repeat same the comment below is that
    171 # some Python versions print out the first line of a multi-line function
    172 # calls in the traceback and some print out the last line
--&gt; 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175     allow_unreachable=True, accumulate_grad=True)

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>

ptrblck: <p>You are most likely detaching a tensor creating the loss from the computation graph, which causes the issue. Could you post a minimal and executable code snippet to reproduce the issue so that we could debug it, please?</p>

bhanup6663: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you for your responce. I have changed values in some of the layes, That fixed the issue.</p>

Rain_larh: <p>Hi, I am facing the similar issues where I am trying to run my image processing</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[63], line 5
      2 for epoch in range(num_epochs):  # loop over the dataset multiple times
      3     print("------------------ Training Epoch {} ------------------".format(epoch+1))
----&gt; 5     train_one_epoch(model, optimizer,'train' , device)
      7     val_model(model, 'val')
      9 print('Finished Training')

Cell In[62], line 42, in train_one_epoch(model, optimizer, data_loader, device)
     38 print("Labels shape:", labels.shape)
     41 loss = criterion(labels, torch.argmax(outputs, dim=1))
---&gt; 42 loss.backward()
     43 optimizer.step()
     45 _, predicted = torch.max(outputs.data, 1)

File ~\anaconda3\envs\withGPU\lib\site-packages\torch\_tensor.py:492, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    482 if has_torch_function_unary(self):
    483     return handle_torch_function(
    484         Tensor.backward,
    485         (self,),
   (...)
    490         inputs=inputs,
    491     )
--&gt; 492 torch.autograd.backward(
    493     self, gradient, retain_graph, create_graph, inputs=inputs
    494 )

File ~\anaconda3\envs\withGPU\lib\site-packages\torch\autograd\__init__.py:251, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    246     retain_graph = create_graph
    248 # The reason we repeat the same comment below is that
    249 # some Python versions print out the first line of a multi-line function
    250 # calls in the traceback and some print out the last line
--&gt; 251 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252     tensors,
    253     grad_tensors_,
    254     retain_graph,
    255     create_graph,
    256     inputs,
    257     allow_unreachable=True,
    258     accumulate_grad=True,
    259 )

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>any advice on how should I solve this issues?</p>

ptrblck: <p>Check if and where the compuation graph was detached or if you are running your forward pass in a <code>no_grad()</code> context. If you are stuck, feel free to post a minimal and executable code snippet reproducing the issue.</p>

ptrblck: <p>Same comment from above applies here too.</p>

Leo_Avangard: <p>Hello, I also have the same issue. I don’t know anymore how to fix it.</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">from functools import lru_cache
from pathlib import Path


from easse.sari import corpus_sari
from torch.nn import functional as F
from source.helper import log_stdout, tokenize, yield_sentence_pair, yield_lines, load_preprocessor, read_lines, \
    count_line
import argparse
import os
import logging
import random
import nltk
from source.resources import NEWSELA_DATASET, get_data_filepath, WIKILARGE_DATASET, TURKCORPUS_DATASET, \
    WIKILARGE_WIKIAUTO_DATASET

nltk.download('punkt')

import numpy as np
import torch
from torch.utils.data import Dataset, DataLoader
import pytorch_lightning as pl
from pytorch_lightning.trainer import seed_everything
from transformers import (
    AdamW,
    T5ForConditionalGeneration,
    T5TokenizerFast,
    get_linear_schedule_with_warmup, AutoConfig, AutoModel
)

torch.set_grad_enabled(True)
print("START_____________________________")

class T5FineTuner(pl.LightningModule):
    def __init__(self, model_name, learning_rate, adam_epsilon, custom_loss, weight_decay, dataset,
                 train_batch_size, valid_batch_size, train_sample_size, valid_sample_size, max_seq_length,
                 n_gpu, gradient_accumulation_steps, num_train_epochs, warmup_steps, nb_sanity_val_steps,
                 *args, **kwargs):
        super(T5FineTuner, self).__init__()
        self.save_hyperparameters()
        self.model = T5ForConditionalGeneration.from_pretrained(self.hparams.model_name)
        self.tokenizer = T5TokenizerFast.from_pretrained(self.hparams.model_name)
        self.model = self.model.to(self.device)
        self.preprocessor = load_preprocessor()


    def is_logger(self):
        return self.trainer.global_rank &lt;= 0

    def forward(
        self, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, labels=None
    ):
        outputs = self.model(
            input_ids,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            labels=labels
        )
        return outputs

    def generate(self, sentence):
        sentence = self.preprocessor.encode_sentence(sentence)
        text = "simplify: " + sentence

        encoding = self.tokenizer(
            text,
            truncation=True,
            max_length=self.hparams.max_seq_length,
            padding='max_length',
            return_tensors="pt"
        )

        input_ids = encoding["input_ids"].to(self.device)
        attention_masks = encoding["attention_mask"].to(self.device)

        beam_outputs = self.model.generate(
            input_ids=input_ids,
            attention_mask=attention_masks,
            do_sample=False,
            max_length=self.hparams.max_seq_length,
            num_beams=8,
            early_stopping=True,
            num_return_sequences=1
        )
        pred_sent = self.tokenizer.decode(beam_outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
        return pred_sent

    def training_step(self, batch, batch_idx):
        labels = batch["target_ids"]
        # Huggingface’s loss functions are defined to exclude the ID -100 during loss calculations. Therefore, we need to convert all padding token IDs in labels to -100.
        labels[labels[:, :] == self.tokenizer.pad_token_id] = -100

        self.opt.zero_grad()
        outputs = self(
            input_ids=batch["source_ids"],
            attention_mask=batch["source_mask"],
            labels=labels,
            decoder_attention_mask=batch['target_mask'],
        )

        if self.hparams.custom_loss:
            print("______________EnteredIf!______________")
            loss = outputs.loss
            complexity_score = torch.tensor(random.randint(0, 100) * 0.01, requires_grad=True, device=self.device)
            complexity_score.requires_grad = True
            # complexity_score = self._custom_step(outputs['logits'])
            # loss = loss * complexity_score
            lambda_ = 0.7
            # loss = lambda_ * loss + (1-lambda_)*complexity_score
            # loss = torch.sqrt(loss + lambda_ * complexity_score)
            print("Before custom loss calculation - loss shape:", loss.shape, "complexity_score shape:", complexity_score.shape)
            loss = loss + complexity_score + lambda_ * (complexity_score - loss)
            print("After custom loss calculation - loss shape:", loss.shape)
            
            print(complexity_score)
            self.log('train_loss', loss, on_step=True, prog_bar=True, logger=True)
            # print(loss)
            loss.requires_grad = True
            return loss
        else:
            print("______________Entered Else!______________")
            loss = outputs.loss
            self.log('train_loss', loss, on_step=True, prog_bar=True, logger=True)
            loss.requires_grad = True
            return loss

        # loss = outputs.loss
        # logs = {"train_loss": loss}
        # self.logger.experiment.add_scalars('loss', logs, global_step=self.global_step)
        # return {"loss": loss, "log": logs}

    def validation_step(self, batch, batch_idx):
        loss = self.sari_validation_step(batch)
        # loss = self._step(batch)
        print("Val_loss", loss)
        logs = {"val_loss": loss}
        # self.logger.experiment.add_scalars('loss', logs, global_step=self.global_step)
        # return {"val_loss": torch.tensor(loss)}
        self.log('val_loss', loss, batch_size=self.hparams.valid_batch_size)
        t = torch.tensor(loss, dtype=float, requires_grad=True)
        print(t)
        return t

    def sari_validation_step(self, batch):
        def generate(sentence):
            sentence = self.preprocessor.encode_sentence(sentence)
            text = "simplify: " + sentence
            # print("Simplifying: ", text)

            encoding = self.tokenizer(
                text,
                truncation=True,
                max_length=self.hparams.max_seq_length,
                padding='max_length',
                return_tensors="pt"
            )

            input_ids = encoding["input_ids"].to(self.device)
            attention_masks = encoding["attention_mask"].to(self.device)

            beam_outputs = self.model.generate(
                input_ids=input_ids,
                attention_mask=attention_masks,
                do_sample=False,
                max_length=self.hparams.max_seq_length,
                num_beams=8,
                early_stopping=True,
                num_return_sequences=1
            ).to(self.device)
            # final_outputs = []
            # for beam_output in beam_outputs:
            sent = self.tokenizer.decode(beam_outputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)
            # if sent.lower() != sentence.lower() and sent not in final_outputs:
                # final_outputs.append(sent)
            
            return sent
            # return final_outputs[0]

        pred_sents = []
        for source in batch["source"]:
            pred_sent = generate(source)
            pred_sents.append(pred_sent)

        score = corpus_sari(batch["source"], pred_sents, batch["targets"])
        print("Sari score: ", score)

        return 1 - score / 100

    def configure_optimizers(self):
        "Prepare optimizer and schedule (linear warmup and decay)"

        model = self.model
        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": self.hparams.weight_decay,
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            },
        ]
        optimizer = AdamW(optimizer_grouped_parameters, lr=self.hparams.learning_rate, eps=self.hparams.adam_epsilon)
        # optimizer = SAM(optimizer_grouped_parameters, base_optimizer, lr=self.hparams.learning_rate, momentum=0.9)
        self.opt = optimizer
        return [optimizer]

    def optimizer_step(self, epoch=None, batch_idx=None, optimizer=None, optimizer_idx=None, optimizer_closure=None,
                       on_tpu=None, using_native_amp=None, using_lbfgs=None):
        optimizer.step(closure=optimizer_closure)

        optimizer.zero_grad()
        self.lr_scheduler.step()


    def train_dataloader(self):
        train_dataset = TrainDataset(dataset=self.hparams.dataset,
                                     tokenizer=self.tokenizer,
                                     max_len=self.hparams.max_seq_length,
                                     sample_size=self.hparams.train_sample_size)

        dataloader = DataLoader(train_dataset,
                                batch_size=self.hparams.train_batch_size,
                                drop_last=True,
                                shuffle=True,
                                pin_memory=True,
                                num_workers=4)
        t_total = ((len(dataloader.dataset) // (self.hparams.train_batch_size * max(1, self.hparams.n_gpu)))
                   // self.hparams.gradient_accumulation_steps
                   * float(self.hparams.num_train_epochs)
                   )
        scheduler = get_linear_schedule_with_warmup(
            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=t_total
        )
        self.lr_scheduler = scheduler
        return dataloader

    def val_dataloader(self):
        val_dataset = ValDataset(dataset=self.hparams.dataset,
                                 tokenizer=self.tokenizer,
                                 max_len=self.hparams.max_seq_length,
                                 sample_size=self.hparams.valid_sample_size)
        return DataLoader(val_dataset,
                          batch_size=self.hparams.valid_batch_size,
                          num_workers=2)


logger = logging.getLogger(__name__)


class LoggingCallback(pl.Callback):
    def on_validation_end(self, trainer, pl_module):
        logger.info("***** Validation results *****")
        if pl_module.is_logger():
            metrics = trainer.callback_metrics
            # Log results
            for key in sorted(metrics):
                if key not in ["log", "progress_bar"]:
                    logger.info("{} = {}\n".format(key, str(metrics[key])))

    def on_test_end(self, trainer, pl_module):
        logger.info("***** Test results *****")

        if pl_module.is_logger():
            metrics = trainer.callback_metrics

            # Log and save results to file
            output_test_results_file = os.path.join(pl_module.hparams.output_dir, "test_results.txt")
            with open(output_test_results_file, "w") as writer:
                for key in sorted(metrics):
                    if key not in ["log", "progress_bar"]:
                        logger.info("{} = {}\n".format(key, str(metrics[key])))
                        writer.write("{} = {}\n".format(key, str(metrics[key])))


class TrainDataset(Dataset):
    def __init__(self, dataset, tokenizer, max_len=256, sample_size=1):
        self.sample_size = sample_size
        # print("init TrainDataset ...")
        preprocessor = load_preprocessor()
        self.source_filepath = preprocessor.get_preprocessed_filepath(dataset, 'train', 'complex')
        self.target_filepath = preprocessor.get_preprocessed_filepath(dataset, 'train', 'simple')

        self.max_len = max_len
        self.tokenizer = tokenizer

        self._load_data()

    def _load_data(self):
        self.inputs = read_lines(self.source_filepath)
        self.targets = read_lines(self.target_filepath)

    def __len__(self):
        return int(len(self.inputs) * self.sample_size)

    def __getitem__(self, index):
        source = "simplify: " + self.inputs[index]
        target = self.targets[index]

        tokenized_inputs = self.tokenizer(
            [source],
            truncation=True,
            max_length=self.max_len,
            padding='max_length',
            return_tensors="pt"
        )
        tokenized_targets = self.tokenizer(
            [target],
            truncation=True,
            max_length=self.max_len,
            padding='max_length',
            return_tensors="pt"
        )
        source_ids = tokenized_inputs["input_ids"].squeeze()
        target_ids = tokenized_targets["input_ids"].squeeze()

        src_mask = tokenized_inputs["attention_mask"].squeeze()  # might need to squeeze
        target_mask = tokenized_targets["attention_mask"].squeeze()  # might need to squeeze

        return {"source_ids": source_ids, "source_mask": src_mask, "target_ids": target_ids, "target_mask": target_mask,
                'sources': self.inputs[index], 'targets': [self.targets[index]]}


class ValDataset(Dataset):
    def __init__(self, dataset, tokenizer, max_len=256, sample_size=1):
        self.sample_size = sample_size
        self.source_filepath = get_data_filepath(dataset, 'valid', 'complex')
        if dataset == NEWSELA_DATASET:
            self.target_filepaths = [get_data_filepath(dataset, 'valid', 'simple')]

        else:  # TURKCORPUS_DATASET as default
            self.target_filepaths = [get_data_filepath(TURKCORPUS_DATASET, 'valid', 'simple.turk', i) for i in range(8)]

        self.max_len = max_len
        self.tokenizer = tokenizer

        self._build()

    def __len__(self):
        return int(len(self.inputs) * self.sample_size)

    def __getitem__(self, index):
        return {"source": self.inputs[index], "targets": self.targets[index]}

    def _build(self):
        self.inputs = []
        self.targets = []

        for source in yield_lines(self.source_filepath):
            self.inputs.append(source)

        self.targets = [[] for _ in range(count_line(self.target_filepaths[0]))]
        for filepath in self.target_filepaths:
            for idx, line in enumerate(yield_lines(filepath)):
                self.targets[idx].append(line)


def train(train_args):
    args = argparse.Namespace(**train_args)
    seed_everything(args.seed, workers=True)

    print(train_args)
    checkpoint_callback = pl.callbacks.ModelCheckpoint(
        dirpath=args.output_dir,
        filename="checkpoint-{epoch}",
        monitor="val_loss",
        verbose=True,
        mode="min",
        save_top_k=5
    )

    train_params = dict(
        accumulate_grad_batches=args.gradient_accumulation_steps,
        #gpus=args.n_gpu,
        max_epochs=args.num_train_epochs,
        # early_stop_callback=False,
        precision=16 if args.fp_16 else 32,
        amp_level=args.opt_level,
        amp_backend='apex',
        # gradient_clip_val=args.max_grad_norm,
        # checkpoint_callback=checkpoint_callback,
        callbacks=[LoggingCallback(), checkpoint_callback],
        # logger=TensorBoardLogger(f'{args.output_dir}/logs'),
        num_sanity_val_steps=args.nb_sanity_val_steps,  # skip sanity check to save time for debugging purpose
        # plugins='ddp_sharded',
        # progress_bar_refresh_rate=1,

    )

    print("Initialize model")
    model = T5FineTuner(**train_args)

    trainer = pl.Trainer(**train_params, accelerator="auto")
    print(" Training model")
    trainer.fit(model)

    print("training finished")

    # print("Saving model")
    # model.model.save_pretrained(args.output_dir)

    # print("Saved model")

</code></pre>
<p>Here is the error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/content/drive/My Drive/TS_T5-main/scripts/train.py", line 44, in &lt;module&gt;
    run_training(args_dict, dataset)
  File "/content/drive/My Drive/TS_T5-main/source/train.py", line 33, in run_training
    train(args_dict)
  File "/content/drive/My Drive/TS_T5-main/source/model.py", line 398, in train
    trainer.fit(model)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 737, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1168, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1254, in _run_stage
    return self._run_train()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1285, in _run_train
    self.fit_loop.run()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fit_loop.py", line 270, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 203, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 87, in advance
    outputs = self.optimizer_loop.run(optimizers, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 201, in advance
    result = self._run_optimization(kwargs, self._optimizers[self.optim_progress.optimizer_position])
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 248, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, kwargs.get("batch_idx", 0), closure)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 358, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1552, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/content/drive/My Drive/TS_T5-main/source/model.py", line 213, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/core/optimizer.py", line 168, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 216, in optimizer_step
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 153, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 75, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 385, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/optimization.py", line 457, in step
    loss = closure()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 138, in _wrap_closure
    closure_result = closure()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 146, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 141, in closure
    self._backward_fn(step_output.closure_loss)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 304, in backward_fn
    self.trainer._call_strategy_hook("backward", loss, optimizer, opt_idx)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1706, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 191, in backward
    self.precision_plugin.backward(self.lightning_module, closure_loss, optimizer, optimizer_idx, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 80, in backward
    model.backward(closure_loss, optimizer, optimizer_idx, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/core/module.py", line 1418, in backward
    loss.backward(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 522, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 266, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
Epoch 0:   0%|          | 0/16832 [00:19&lt;?, ?it/s]
</code></pre>
<p>Any help will be much appreciated.</p>

Cuig_Thallann: <p>I am transcribing the makemore 5 code from Andrej K’s YouTube channel:</p>
<p>            <iframe src="https://www.youtube.com/embed/t3YJ5hKiMQ0?feature=oembed&amp;wmode=opaque&amp;list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ" width="480" height="360" frameborder="0" allowfullscreen="" class="youtube-onebox" seamless="seamless" sandbox="allow-same-origin allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-presentation"></iframe>
</p>
<p>==============================================<br>
The code works up until:<br>
x = Xb<br>
for layer in layers:<br>
x = layer(x)</p>
<pre><code>loss = F.cross_entropy(x, Yb) # loss function

# Backwards pass
for p in parameters:
    p.grad = None
    
loss.backward()
</code></pre>
<p>The call to loss backward() crashes and I get this following error at the bottom of the stack:</p>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>If I debut and print loss.grad_fn it says  its None.</p>
<p>What is breaking loss.backward() "</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

Cuig_Thallann: <p>{<br>
“cells”: [<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 2,<br>
“id”: “72f5cb69-a965-4788-a35c-96256875b717”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“import torch\n”,<br>
“import torch.nn.functional as F\n”,<br>
“import matplotlib.pyplot as plt\n”,<br>
“%matplotlib inline”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 3,<br>
“id”: “9677ebd3-c4e2-4c9f-ba48-ba87fe3bd61c”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“words = open(‘names.txt’,‘r’).read().splitlines()”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 4,<br>
“id”: “436c67c6-9d58-465a-a411-924efcc0826c”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“name”: “stdout”,<br>
“output_type”: “stream”,<br>
“text”: [<br>
“{1: ‘a’, 2: ‘b’, 3: ‘c’, 4: ‘d’, 5: ‘e’, 6: ‘f’, 7: ‘g’, 8: ‘h’, 9: ‘i’, 10: ‘j’, 11: ‘k’, 12: ‘l’, 13: ‘m’, 14: ‘n’, 15: ‘o’, 16: ‘p’, 17: ‘q’, 18: ‘r’, 19: ‘s’, 20: ‘t’, 21: ‘u’, 22: ‘v’, 23: ‘w’, 24: ‘x’, 25: ‘y’, 26: ‘z’, 0: ‘.’}\n”,<br>
“27\n”<br>
]<br>
}<br>
],<br>
“source”: [<br>
“<span class="hashtag-raw">#Build</span> the vocabulary of characters and mappings to/from integers\n”,<br>
“chars = sorted ( list( set(‘’.join(words)) ))\n”,<br>
“stoi = { s: i+1 for i,s  in enumerate(chars)}\n”,<br>
“stoi[‘.’] = 0\n”,<br>
“itos = { i:s for s,i in stoi.items()}\n”,<br>
“vocab_size = len(itos)\n”,<br>
“print(itos)\n”,<br>
“print(vocab_size)”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 5,<br>
“id”: “8c6d698c-a05d-471b-b8b6-b13fe989a15a”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“import random\n”,<br>
“random.seed(42)\n”,<br>
“random.shuffle(words)”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 6,<br>
“id”: “cd8d18a6-bbe6-43a2-ac0e-f8cf4945a805”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“name”: “stdout”,<br>
“output_type”: “stream”,<br>
“text”: [<br>
“torch.Size([182625, 3]) torch.Size([182625])\n”,<br>
“torch.Size([22655, 3]) torch.Size([22655])\n”,<br>
“torch.Size([22866, 3]) torch.Size([22866])\n”<br>
]<br>
}<br>
],<br>
“source”: [<br>
“block_size = 3\n”,<br>
“def build_dataset(words):\n”,<br>
"    X,Y = <span class="chcklst-box fa fa-square-o fa-fw"></span>,<span class="chcklst-box fa fa-square-o fa-fw"></span>\n",<br>
"    for w in words:\n",<br>
"        <span class="hashtag-raw">#print</span>(w)\n",<br>
"        context = [0]<em>block_size\n",<br>
"        for ch in w + ‘.’:\n",<br>
"            ix = stoi[ch]\n",<br>
"            X.append(context)\n",<br>
"            Y.append(ix)\n",<br>
"            context = context[1:] + [ix]\n",<br>
"    X = torch.tensor(X)\n",<br>
"    Y = torch.tensor(Y)\n",<br>
"    print(X.shape, Y.shape)\n",<br>
"    return X,Y\n",<br>
“\n”,<br>
“<span class="hashtag-raw">#import</span> random\n”,<br>
"n1 = int(0.8</em>len(words))\n",<br>
“n2 = int(0.9<em>len(words))\n",<br>
“\n”,<br>
“Xtr, Ytr = build_dataset(words[:n1])\n”,<br>
“Xdev, Ydev = build_dataset(words[n1:n2])\n”,<br>
“Xte, Yte = build_dataset(words[n2:])”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 7,<br>
“id”: “e235f156-c8d4-4318-be69-3b7c356e6293”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“name”: “stdout”,<br>
“output_type”: “stream”,<br>
“text”: [<br>
“… → y\n”,<br>
“…y → u\n”,<br>
“.yu → h\n”,<br>
“yuh → e\n”,<br>
“uhe → n\n”,<br>
“hen → g\n”,<br>
“eng → .\n”,<br>
“… → d\n”,<br>
“…d → i\n”,<br>
“.di → o\n”,<br>
“dio → n\n”,<br>
“ion → d\n”,<br>
“ond → r\n”,<br>
“ndr → e\n”,<br>
“dre → .\n”,<br>
“… → x\n”,<br>
“…x → a\n”,<br>
“.xa → v\n”,<br>
“xav → i\n”,<br>
“avi → e\n”<br>
]<br>
}<br>
],<br>
“source”: [<br>
“for x,y in zip(Xtr[:20], Ytr[:20]):\n”,<br>
"    print(‘’.join(itos[ix.item()] for ix in x),‘–&gt;’,itos[y.item()] )"<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 8,<br>
“id”: “7bbfe1f0-efb9-4630-926b-4ac2902d33b3”,<br>
“metadata”: {},<br>
“outputs”: [],<br>
“source”: [<br>
“class Linear:\n”,<br>
"    def <strong>init</strong>(self, fan_in,fan_out,bias=True):\n",<br>
"        self.weight = torch.randn(fan_in,fan_out)/fan_in**0.5 <span class="hashtag-raw">#note:</span> kaisming init\n",<br>
"        self.bias = torch.zeros(fan_out) if bias else None\n",<br>
"        \n",<br>
"    def <strong>call</strong>(self,x):\n",<br>
"        self.out = x @ self.weight\n",<br>
"        if self.bias is not None:\n",<br>
"            self.out += self.bias\n",<br>
"        return self.out\n",<br>
"        \n",<br>
"    def parameters(self):\n",<br>
"        return [self.weight] + ([] if self.bias is None else [self.bias])\n",<br>
"        "<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 9,<br>
“id”: “ae9f314b-db74-4844-b5f4-06fe8869d472”,<br>
“metadata”: {},<br>
“outputs”: [],<br>
“source”: [<br>
“class BatchNorm1d:\n”,<br>
“\n”,<br>
"    def <strong>init</strong>( self, dim, eps=1e-5,momentum=0.1):\n",<br>
"        self.eps = eps\n",<br>
"        self.momentum = momentum\n",<br>
"        self.training = True\n",<br>
"        # Parameters (trained with backprop)\n",<br>
"        self.gamma = torch.ones(dim)\n",<br>
"        self.beta = torch.zeros(dim)\n",<br>
"        <span class="hashtag-raw">#buffers</span> (trained with a running momentum update)\n",<br>
"        self.running_mean = torch.zeros(dim)\n",<br>
"        self.running_var = torch.ones(dim)\n",<br>
“\n”,<br>
"    def <strong>call</strong>(self,x):\n",<br>
"        <span class="hashtag-raw">#calculate</span> forward pass\n",<br>
"        if self.training:\n",<br>
"            xmean = x.mean(0,keepdim=True) <span class="hashtag-raw">#Batch</span> mean\n",<br>
"            xvar = x.var(0,keepdim=True)  <span class="hashtag-raw">#Batch</span> Variance\n",<br>
"        else:\n",<br>
"            xmean =  self.running_mean\n",<br>
"            xvar = self.running_var\n",<br>
"        xhat = (x-xmean)/torch.sqrt(xvar+self.eps)  # normalize the variance    \n",<br>
"        self.out = self.gamma  * xhat + self.beta\n",<br>
"        <span class="hashtag-raw">#update</span> the buffers\n",<br>
"        if self.training:\n",<br>
"            with torch.no_grad():\n",<br>
"                self.running_mean=(1-self.momentum)</em> self.running_mean + self.momentum * xmean\n”,<br>
"                self.running_var=(1-self.momentum)* self.running_var + self.momentum * xvar\n",<br>
"        return self.out\n",<br>
"        \n",<br>
"    def parameters(self):\n",<br>
"        return [self.gamma, self.beta]\n",<br>
“\n”,<br>
“<span class="hashtag-raw">#--------------------------------------------------------------------</span>\n”,<br>
“\n”,<br>
“class Tanh:\n”,<br>
"    \n",<br>
"   def <strong>call</strong>(self,x):\n",<br>
"       self.out = torch.tanh(x)\n",<br>
"       return self.out\n",<br>
"       \n",<br>
"   def parameters(self):\n",<br>
"       return <span class="chcklst-box fa fa-square-o fa-fw"></span>\n",<br>
“\n”,<br>
“# -----------------------------------------------------------------------------------------------\n”,<br>
“\n”,<br>
“class Embedding:\n”,<br>
"  \n",<br>
"  def <strong>init</strong>(self, num_embeddings, embedding_dim):\n",<br>
"    self.weight = torch.randn((num_embeddings, embedding_dim))\n",<br>
"    \n",<br>
"  def <strong>call</strong>(self, IX):\n",<br>
"    self.out = self.weight[IX]\n",<br>
"    return self.out\n",<br>
"  \n",<br>
"  def parameters(self):\n",<br>
"    return [self.weight]\n",<br>
“\n”,<br>
“# -----------------------------------------------------------------------------------------------\n”,<br>
“class Flatten:\n”,<br>
"  \n",<br>
"  <span class="hashtag-raw">#def</span> <strong>init</strong>(self, n):\n",<br>
"   # self.n = n\n",<br>
"    \n",<br>
"  def <strong>call</strong>(self, x):\n",<br>
"    <span class="hashtag-raw">#B</span>, T, C = x.shape\n",<br>
"    <span class="hashtag-raw">#x</span> = x.view(B, T//self.n, C<em>self.n)\n",<br>
"    <span class="hashtag-raw">#if</span> x.shape[1] == 1:\n",<br>
"     # x = x.squeeze(1)\n",<br>
"    <span class="hashtag-raw">#self</span>.out = x\n",<br>
"    self.out = x.view(x.shape[0],-1)\n",<br>
"    return self.out\n",<br>
"  \n",<br>
"  def parameters(self):\n",<br>
"    return []\n",<br>
“\n”,<br>
"    \n",<br>
"    "<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 10,<br>
“id”: “ed7f978c-45e5-4013-bf24-780d9effef9f”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“data”: {<br>
“text/plain”: [<br>
“&lt;torch._C.Generator at 0x1ae1d7aa0f0&gt;”<br>
]<br>
},<br>
“execution_count”: 10,<br>
“metadata”: {},<br>
“output_type”: “execute_result”<br>
}<br>
],<br>
“source”: [<br>
“\n”,<br>
“torch.manual_seed(42) # seed rng for reproducibility”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 11,<br>
“id”: “23c5f2ff-1a75-4852-9062-853f377ac13b”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“name”: “stdout”,<br>
“output_type”: “stream”,<br>
“text”: [<br>
“12097\n”<br>
]<br>
}<br>
],<br>
“source”: [<br>
“n_embd =  10 # the dimemnsionaliy of the charcrter embeddmg vector\n”,<br>
“n_hidden = 200 # the number of nuerons in the hidden layer\n”,<br>
“<a class="hashtag-cooked" href="/c/c/11" data-type="category" data-slug="c" data-id="11"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>C++</span></a> = torch.randn((vocab_size,n_embd))\n”,<br>
“layers = [\n”,<br>
"    Embedding(vocab_size,n_embd),\n",<br>
"    Flatten(),\n",<br>
"    Linear(n_embd</em>block_size,n_hidden, bias=False), \n",<br>
"    BatchNorm1d(n_hidden), \n",<br>
"    Tanh(),\n",<br>
"    Linear(n_hidden,vocab_size),\n",<br>
“]\n”,<br>
“# Parameter init\n”,<br>
“with torch.no_grad():\n”,<br>
"    layers[-1].weight *= 0.1 # last layer make less confident.\n",<br>
“\n”,<br>
“<span class="hashtag-raw">#parameters</span> = [C] + [p for layer in layers for p in layer.parameters()]\n”,<br>
“parameters = [p for layer in layers for p in layer.parameters()]\n”,<br>
“\n”,<br>
“\n”,<br>
“print( sum(p.nelement() for p in parameters)  ) # Number of parameters in total\n”,<br>
“\n”,<br>
“for p in parameters:\n”,<br>
"    p.requiers_grad = True\n",<br>
"    "<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: 12,<br>
“id”: “0d28cc3d-73e0-4912-a8bf-3e0fbe5e90ae”,<br>
“metadata”: {},<br>
“outputs”: [<br>
{<br>
“ename”: “RuntimeError”,<br>
“evalue”: “element 0 of tensors does not require grad and does not have a grad_fn”,<br>
“output_type”: “error”,<br>
“traceback”: [<br>
“\u001b[1;31m---------------------------------------------------------------------------\u001b[0m”,<br>
“\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)”,<br>
“Cell \u001b[1;32mIn[12], line 29\u001b[0m\n\u001b[0;32m     26\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m p \u001b[38;5;129;01min\u001b[39;00m parameters:\n\u001b[0;32m     27\u001b[0m     p\u001b[38;5;241m.\u001b[39mgrad \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m—&gt; 29\u001b[0m \u001b[43mloss\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     31\u001b[0m \u001b[38;5;66;03m#update\u001b[39;00m\n\u001b[0;32m     32\u001b[0m lr \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.1\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m i \u001b[38;5;241m&lt;\u001b[39m \u001b[38;5;241m10000\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;241m0.01\u001b[39m\n”,<br>
“File \u001b[1;32m~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\<em>tensor.py:522\u001b[0m, in \u001b[0;36mTensor.backward\u001b[1;34m(self, gradient, retain_graph, create_graph, inputs)\u001b[0m\n\u001b[0;32m    512\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m has_torch_function_unary(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m    513\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m handle_torch_function(\n\u001b[0;32m    514\u001b[0m         Tensor\u001b[38;5;241m.\u001b[39mbackward,\n\u001b[0;32m    515\u001b[0m         (\u001b[38;5;28mself\u001b[39m,),\n\u001b[1;32m   (…)\u001b[0m\n\u001b[0;32m    520\u001b[0m         inputs\u001b[38;5;241m=\u001b[39minputs,\n\u001b[0;32m    521\u001b[0m     )\n\u001b[1;32m–&gt; 522\u001b[0m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mautograd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    523\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgradient\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minputs\u001b[49m\n\u001b[0;32m    524\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",<br>
"File \u001b[1;32m~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\autograd\<strong>init</strong>.py:266\u001b[0m, in \u001b[0;36mbackward\u001b[1;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001b[0m\n\u001b[0;32m    261\u001b[0m     retain_graph \u001b[38;5;241m=\u001b[39m create_graph\n\u001b[0;32m    263\u001b[0m \u001b[38;5;66;03m# The reason we repeat the same comment below is that\u001b[39;00m\n\u001b[0;32m    264\u001b[0m \u001b[38;5;66;03m# some Python versions print out the first line of a multi-line function\u001b[39;00m\n\u001b[0;32m    265\u001b[0m \u001b[38;5;66;03m# calls in the traceback and some print out the last line\u001b[39;00m\n\u001b[1;32m–&gt; 266\u001b[0m \u001b[43mVariable\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_execution_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_backward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m  \u001b[49m\u001b[38;5;66;43;03m# Calls into the C++ engine to run the backward pass\u001b[39;49;00m\n\u001b[0;32m    267\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtensors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    268\u001b[0m \u001b[43m    \u001b[49m\u001b[43mgrad_tensors</em>\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    269\u001b[0m \u001b[43m    \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    270\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    271\u001b[0m \u001b[43m    \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    272\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallow_unreachable\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m    273\u001b[0m \u001b[43m    \u001b[49m\u001b[43maccumulate_grad\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m    274\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n”,<br>
“\u001b[1;31mRuntimeError\u001b[0m: element 0 of tensors does not require grad and does not have a grad_fn”<br>
]<br>
}<br>
],<br>
“source”: [<br>
“max_steps = 200000\n”,<br>
“batch_size = 32\n”,<br>
“lossi = <span class="chcklst-box fa fa-square-o fa-fw"></span>\n”,<br>
“for i in range(max_steps):\n”,<br>
"    <span class="hashtag-raw">#mini</span> batch construct\n",<br>
"    ix = torch.randint(0,Xtr.shape[0],(batch_size,))\n",<br>
"    Xb, Yb = Xtr[ix], Ytr[ix]\n",<br>
"    <span class="hashtag-raw">#print</span>(Yb)\n",<br>
"    <span class="hashtag-raw">#break</span>\n",<br>
"    \n",<br>
"    # forward pass\n",<br>
"    # emb = C[Xb] <span class="hashtag-raw">#embed</span> the characters into vectors\n",<br>
"    <span class="hashtag-raw">#x</span> = emb.view(emb.shape[0],-1) # concatinate the vectors\n",<br>
"    x = Xb\n",<br>
"    \n",<br>
"    for layer in layers:\n",<br>
"        x = layer(x)\n",<br>
“\n”,<br>
"    loss = F.cross_entropy(x, Yb) # loss function\n",<br>
“\n”,<br>
"    <span class="hashtag-raw">#print</span>(loss.grad_fn)\n",<br>
"    <span class="hashtag-raw">#break</span>\n",<br>
"    \n",<br>
"    \n",<br>
"    # Backwards pass\n",<br>
"    for p in parameters:\n",<br>
"        p.grad = None\n",<br>
“\n”,<br>
"    loss.backward()\n",<br>
"    \n",<br>
"    <span class="hashtag-raw">#update</span>\n",<br>
"    lr = 0.1 if i &lt; 10000 else 0.01\n",<br>
"    for p in parameters:\n",<br>
"            p.data += -lr * p.grad\n",<br>
“\n”,<br>
"    <span class="hashtag-raw">#track</span> stats\n",<br>
"    if i%10000 == 0:\n",<br>
"        print(f’{i:7d}/{max_steps:7d}: {loss.item():.4f}‘)\n",<br>
"    lossi.append(loss.log10().item())\n",<br>
"    break"<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “8a2e0b4f-cdba-40ef-bf1a-513e3c6b5b64”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “f51606a7-93be-460b-8eae-46a517b8ac5e”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “38ce9c54-92b3-487e-a5c6-4c3d93cce450”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“plt.plot(lossi)”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “2f170749-eb02-486b-aa8a-1b622af191e5”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“for layer in layers:\n”,<br>
"    layer.training = False\n",<br>
"    "<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “8f5d95ac-3090-414f-b2ba-ecc11a0c7071”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“<span class="mention">@torch.no_grad</span>() # this decorator disables gradient tracking\n”,<br>
“def split_loss(split):\n”,<br>
"  x,y = {\n",<br>
"    ‘train’: (Xtr, Ytr),\n",<br>
"    ‘val’: (Xdev, Ydev),\n",<br>
"    ‘test’: (Xte, Yte),\n",<br>
"  }[split]\n",<br>
"  emb = C<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span> # (N, block_size, n_embd)\n",<br>
"  x = emb.view(emb.shape[0], -1) # concat into (N, block_size * n_embd)\n",<br>
"  for layer in layers:\n",<br>
"      x = layer(x)\n",<br>
"  loss = F.cross_entropy(x,y)\n",<br>
"  print(split,loss.item())\n",<br>
"    \n",<br>
“split_loss(‘train’)\n”,<br>
“split_loss(‘val’)”<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “67bb2592-897b-4e23-a96c-80712b815679”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: [<br>
“for _ in range(20):\n”,<br>
"    out = <span class="chcklst-box fa fa-square-o fa-fw"></span>\n",<br>
"    context = [0] * block_size\n",<br>
"    <span class="hashtag-raw">#print</span>(context)\n",<br>
"    <span class="hashtag-raw">#break</span>\n",<br>
"    while True:\n",<br>
"        emb = C[torch.tensor([context])]  # (1,bloock_size,n_embd)\n",<br>
"        x = emb.view(emb.shape[0],-1) # concatinate the vectors\n",<br>
"        for layer in layers:\n",<br>
"            x = layer(x)\n",<br>
"        logits = x\n",<br>
"        probs = F.softmax(logits, dim=1)\n",<br>
"        ix=torch.multinomial(probs,num_samples=1).item()\n",<br>
"        context = context[1:] + [ix]\n",<br>
"        out.append(ix)\n",<br>
"        if ix == 0:\n",<br>
"            break\n",<br>
"    print(’'.join(itos[i] for i in out) )"<br>
]<br>
},<br>
{<br>
“cell_type”: “code”,<br>
“execution_count”: null,<br>
“id”: “15ff869c-e646-4117-9385-c3143a0a0151”,<br>
“metadata”: {},<br>
“outputs”: <span class="chcklst-box fa fa-square-o fa-fw"></span>,<br>
“source”: <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
}<br>
],<br>
“metadata”: {<br>
“kernelspec”: {<br>
“display_name”: “Python 3 (ipykernel)”,<br>
“language”: “python”,<br>
“name”: “python3”<br>
},<br>
“language_info”: {<br>
“codemirror_mode”: {<br>
“name”: “ipython”,<br>
“version”: 3<br>
},<br>
“file_extension”: “.py”,<br>
“mimetype”: “text/x-python”,<br>
“name”: “python”,<br>
“nbconvert_exporter”: “python”,<br>
“pygments_lexer”: “ipython3”,<br>
“version”: “3.12.2”<br>
}<br>
},<br>
“nbformat”: 4,<br>
“nbformat_minor”: 5<br>
}</p>

Cuig_Thallann: <p>Here is the link to the GitHub repository for the finished makemore:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/karpathy/makemore">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/karpathy/makemore" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/b1bcfe44de84240bfa2f15c302e342db54c39ccf39a60a774e9564c80b46d1c8/karpathy/makemore" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/karpathy/makemore" target="_blank" rel="noopener nofollow ugc">GitHub - karpathy/makemore: An autoregressive character-level language model...</a></h3>

  <p>An autoregressive character-level language model for making more things - karpathy/makemore</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Cuig_Thallann: <hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[17], line 20<br>
8 <span class="hashtag-raw">#print</span>(Yb)<br>
9 <span class="hashtag-raw">#break</span><br>
10<br>
(…)<br>
15 <span class="hashtag-raw">#for</span> layer in layers:<br>
16 #    x = layer(x)<br>
18 logits = model(Xb)<br>
—&gt; 20 loss = F.cross_entropy(logits, Xb) # loss function<br>
22 <span class="hashtag-raw">#print</span>(loss.grad_fn)<br>
23 <span class="hashtag-raw">#break</span><br>
24<br>
25<br>
26 # Backwards pass<br>
27 for p in parameters:</p>
<p>File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\nn\functional.py:3059, in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)<br>
3057 if size_average is not None or reduce is not None:<br>
3058     reduction = _Reduction.legacy_get_string(size_average, reduce)<br>
 → 3059 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)</p>
<p>RuntimeError: 0D or 1D target tensor expected, multi-target not supported</p>