akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead

username: danielmanu93
cooked: <p>Hi All I have data from a dataloader which I pass to my model for prediction. I am trying to normalize my prediction within a certain range using <code>T.tonumpy_denormalize</code>, however I get this error <code>RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead</code>. I have tried to use <code>.detach().numpy()</code> on my prediction tensor but I get this error afterwards <code>AttributeError: 'numpy.ndarray' object has no attribute 'cpu'</code>. Initially I wasn’t getting any of these error but I changed my code structure and started getting this error, even though its the same code as before. Below is my sample code for demonstration:</p>
<pre><code class="lang-auto">val = "./data.txt"
val_data = torch.load(val)
dataloader = torch.utils.data.DataLoader(val_data, batch_size=1, shuffle=True)

def evaluate(model):
      label_min = 1500
      label_max = 4500
      model.eval()
      data, label = iter(dataloader).next()
      pred = model(data)
      pred_np = T.tonumpy_denormalize(pred, label_min, label_max, exp=False)   # I get the error on this line. #
# previously wasn't getting this error, I just changed the code structure by moving some parts upwards and got this error. #

# full error is below #
pred_np = T.tonumpy_denormalize(pred, label_min, label_max, exp=False)
  File "/home/pi/Desktop/exp/fcnvmb/transforms.py", line 97, in tonumpy_denormalize
    vid = minmax_denormalize(vid.cpu().numpy(), vmin, vmax, scale)
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
</code></pre>

username: ptrblck
cooked: <p>You are trying to call <code>numpy()</code> on a tensor, which is still attached to a computation graph, and which is disallowed without calling <code>detach()</code> explicitly. This error is used to make sure the users are aware that this tensor is being detached (and thus they have to use the <code>detach()</code> call explicitly) in the <code>numpy()</code> operation.<br>
I don’t know why the error wasn’t raised previously, but <code>pred</code> should have been attached to a computation graph before as well.</p>

username: danielmanu93
cooked: <p>Okay I figured. I just used <code>pred.detach()</code> and it worked. Thanks.</p>

username: Sishou_Zhou
cooked: <p>I try to use a neural network to estimate parameters. When I loop my program 10000 times, and it runs without error. But the result is rather bad, Then I try to loop the program 100,000,000 times, it shows this error, who knows the reason? I feel very wired.</p>
How do I determine why my optimizer is taking up so much memory?

username: Goldname
cooked: <p>I am experiencing CUDA OOM. It occurs at optimizer.step(). I go from around 20GiB to more than 40 GiB during this step. Although my model is large, I actually froze most of it, so the optimizer shouldn’t be adding so much additional memory. Does anyone know how I can debug this? I am freezing my model like this:</p>
<pre><code class="lang-auto">for name, param in model.model.named_parameters():
    param.requires_grad = False
</code></pre>

username: srishti-git1110
cooked: <p>Hi,<br>
Which optimizer are you using? Adaptive optimizers like Adam require more VRAM to be able to store various internal states as compared to non adaptive ones like plain SGD.<br>
Also, could you please confirm the training step at which this spike occurs? According to me and in my experience, it should happen at the very first step (1st batch of the 1st epoch) after which the vram consumption should relatively stabilize.</p>

username: Goldname
cooked: <p>You’re correct that I am using Adam and that this spike occurs at the first iteration. However, the trainable parameters of my model is very small.  I don’t see why Adam is using 20 gb of ram for a small model.</p>

username: srishti-git1110
cooked: <p>I see.<br>
Could you please see and share the output of the following -</p>
<pre data-code-wrap="python"><code class="lang-python">param_size = sum([param.nelement() for param in model.parameters()])
optimizer_size = sum([param.nelement() for param in optimizer.param_groups[0]['params']])
grads_size = param_size
input_batch_size = # no. of units in the input batch. For eg. batch_size * 3 (channels) * 32 * 32 (image dimensions)

total_units = param_size + optimizer_size + grads_size + input_batch_size # forward activations are missing here btw
total_vram = total_units * 4 / (1024**3)
print(total_vram)
</code></pre>
<p>This should give you an idea of how much memory is being taken up by each part of the pipeline. Note that I have not included forward activations in this code (you need to register forward hooks for that) but let’s get an idea using this once.</p>
Study on Deep Learning Technical Debt

username: mdipenta
cooked: <p>As software engineering research teams at the University of Sannio (Italy) and Università della Svizzera Italiana (USI) we are interested in assessing a catalogue of Self-Admitted Technical Debt (SATD) in ML-intensive systems, focusing more on the technical debt affecting the ML-specific code. The catalogue features 7 categories, concerning different kinds of problems.</p>
<p>For each question, you need to provide an answer indicating whether or not you agree on the relevance of the identified SATD. The answer will be provided by choosing an option in a 5-level scale (strongly agree, weakly agree, borderline, weakly disagree, strongly disagree), plus a “don’t know” option.</p>
<p>Filling out the survey will take about 5 minutes.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://forms.gle/KBTDDj2UgvNNjbJn6">
  <header class="source">
      <img src="https://ssl.gstatic.com/docs/spreadsheets/forms/favicon_qp2.png" class="site-icon" width="16" height="16">

      <a href="https://forms.gle/KBTDDj2UgvNNjbJn6" target="_blank" rel="noopener nofollow ugc">Google Docs</a>
  </header>

  <article class="onebox-body">
    <img width="200" height="200" src="https://lh4.googleusercontent.com/4bavlOtVr3u7VlQ8eVcphL4D-Htu6RmDvG3DSG1_D4bO9xLOSrgLDB9VLTAHC0Sz5u3-DNU9jEw=w1200-h630-p" class="thumbnail onebox-avatar">

<h3><a href="https://forms.gle/KBTDDj2UgvNNjbJn6" target="_blank" rel="noopener nofollow ugc">Study on Self-Admitted Technical Debt in ML-intensive systems</a></h3>

  <p>As software engineering research teams at the University of Sannio (Italy) and Università della Svizzera Italiana (USI) we are interested in assessing  a catalogue of Self-Admitted Technical Debt (SATD) in ML-intensive systems, focusing more on the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please note that your identity and personal data will not be disclosed, while we plan to use the aggregated results and anonymized responses as part of a scientific publication.</p>
<p>If you have any questions about the questionnaire or our research, please do not hesitate to contact us.</p>
<p>Best Regards,</p>
<p>Federica Pepe (f.pepe8@studenti.unisannio.it)<br>
Fiorella Zampetti (fzampetti@unisannio.it)<br>
Antonio Mastropaolo (antonio.mastropaolo@usi.ch)<br>
Gabriele Bavota (gabriele.bavota@usi.ch)<br>
Massimiliano Di Penta (dipenta@unisannio.it)</p>
Looking an AI Partner to Hustle

username: Harsh_Mittal
cooked: <p>Hi there! I’m Harsh, a 24-year-old self-taught product engineer from India. I aspire to become an AI Product Engineer and to strengthen my skills in AI engineering - over the last two months, I’ve been working on understanding the science and engineering behind LLMs and am now diving deeper.</p>
<p>As this field is both complex and expensive, I’m looking for an AI partner to collaborate with.</p>
<p>Here’s some information about me:</p>
<ul>
<li>I graduated in 2021 and have spent the last three years building software.</li>
<li>I am self-taught in areas ranging from product design to engineering.</li>
<li>I’m fascinated by the potential of LLMs. This interest led me to leave my job to create a Resume Manager, Yepp.In While it may seem simple, I’m deeply passionate about this project.</li>
<li>I continuously teach myself new skills, including product design, full stack web development, and now LLMs.</li>
<li>After spending several weeks studying, I’ve gained a basic understanding of the science and engineering behind training LLMs.</li>
<li>My next goal is to dive deeper into the engineering aspects. I understand that the training process is costly and that creating high-quality real-world models requires a large, experienced team.</li>
<li>At the moment, I’m trying to train a decoder-only model (1b-7b parameters) and facing several challenges, though I’ve successfully trained a 240M parameter model.</li>
</ul>
<p>You can find me on LinkedIn at (<a href="http://www.linkedin.com/in/mittalh" rel="noopener nofollow ugc">http://www.linkedin.com/in/mittalh</a>) and view My Portfolio at www.notion.so/Harsh-Mittal-AI-Product-Engineer-1abd8bf2d195409494ffb5dba202dc2e</p>
<p>I look forward to starting a conversation. Cheers!</p>
How many times do I need to call `new_group` if I want to create m partition of n processes?

username: youkaichao1
cooked: <p>For example, I have processes with rank <code>[0, 1, 2, 3]</code>, and I want to create 3 partitions of the processes:</p>
<ul>
<li>one is the whole group <code>[[0, 1, 2, 3]]</code>, i.e. partition all the processes in one group</li>
<li>one is divided group, <code>[[0, 1], [2, 3]]</code></li>
<li>the final one is a different partition, <code>[[0, 2], [1, 3]]</code></li>
</ul>
<p>How should I call <code>new_group</code>?</p>
<p>I thought about two ways:</p>
<p>The first:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
dist.init_process_group(backend='gloo')
group0 = dist.group.WORLD
if dist.get_rank() in [0, 1]:
    group1 = dist.new_group(ranks=[0, 1])
else:
    group1 = dist.new_group(ranks=[2, 3])
if dist.get_rank() in [0, 2]:
    group2 = dist.new_group(ranks=[0, 2])
else:
    group2 = dist.new_group(ranks=[1, 3])
</code></pre>
<p>The second:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
dist.init_process_group(backend='gloo')
group0 = dist.group.WORLD
a, b = dist.new_group(ranks=[0, 1]), dist.new_group(ranks=[2, 3])
if dist.get_rank() in [0, 1]:
    group1 = a
else:
    group1 = b
a, b = dist.new_group(ranks=[0, 2]), dist.new_group(ranks=[1, 3])
if dist.get_rank() in [0, 2]:
    group2 = a
else:
    group2 = b
</code></pre>
<p>Seems like the second one works. But the code looks ugly. I want to confirm if this is the right way.</p>
PyTorch with rust-analyzer in VSCode

username: DeaDSouL
cooked: <p>Hello,</p>
<p>I use VSCode with Rust-Analyzer, for my work with rust. And I needed the libtorch package, but since my OS (Fedora) doesn’t provide one with its repositories. I’ve created a virtual python environment in my rust project folder (via: <code>pipenv install torch==2.2.0</code>).<br>
Now, in order to activate that environment I need to run <code>pipenv shell</code>.<br>
Then I can run the following: <code>LIBTORCH_USE_PYTORCH=1 cargo run</code> and it works just fine.<br>
I can also run a command inside that virtual environment without entering it via: <code>pipenv run the-command</code>.</p>
<p>so, I’ve configured my vscode terminal to use that environment, via configuring the <code>python.defaultInterpreterPath</code> in vscode.</p>
<p>now my problem is, how to tell rust-analyzer to use that environment, and prefix all its commands with this env variable <code>LIBTORCH_USE_PYTORCH=1</code> ?</p>
<p>I tried all its configurations, but no luck.</p>
<p>Thanks</p>

username: DeaDSouL
cooked: <p>Never mind, I found a solution.<br>
In case anyone faces the issue, here is how to solve it.</p>
<ol>
<li>cd to your project path. ex: <code>cd ~/dev/proj1</code>.</li>
<li>enter the venv. ex: <code>pipenv shell</code>.</li>
<li>open code. ex <code>code .</code>.</li>
</ol>
<p>You’re welcome <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Relation between logit value and generated ids

username: umme_habiba17
cooked: <p>from matplotlib import pyplot as plt</p>
<p>fig = plt.figure(figsize=(18, 14))</p>
<h1><a name="prepare-image-for-the-model-1" class="anchor" href="#prepare-image-for-the-model-1"></a>prepare image for the model</h1>
<p>for i, example in enumerate(dataset):<br>
image = example[“image”]<br>
inputs = processor(images=image, return_tensors=“pt”).to(device)<br>
pixel_values = inputs.pixel_values</p>
<pre><code># Generate captions
generated_ids = model.generate(pixel_values=pixel_values, max_length=50)

# Adjust logits before decoding
outputs = model(input_ids=generated_ids, pixel_values=pixel_values)
logits = outputs.logits
logits += 0.01  # Adding 0.01 to logits

# Re-generate captions with adjusted logits
adjusted_generated_ids = model.generate(pixel_values=pixel_values, logits=logits, max_length=50)
generated_caption = processor.decode(adjusted_generated_ids[0], skip_special_tokens=True)

fig.add_subplot(2, 3, i+1)
plt.imshow(image)
plt.axis("off")
plt.title(f"Generated caption: {generated_caption}")
</code></pre>
<p>plt.show()</p>
<p>“If I change the logit value, does this alter my caption and generated IDs? If yes, how can I update the generated IDs after modifying the logit value?”</p>
OSError: [WinError 182] The operating system cannot run %1

username: Ayush_Aditya
cooked: <p>I used the torch earlier but suddenly today when I tried importing the torch I got this error message, i tried updating the pytorch as suggested by GitHub ( <a href="https://github.com/conda-forge/openmp-feedstock/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · conda-forge/openmp-feedstock · GitHub</a>)  issue section but that didn’t help</p>
<p>Error message  -</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “D:\anaconda\envs\Pytorch\lib\site-packages\torch_<em>init</em>_.py”, line 130, in <br>
raise err<br>
OSError: [WinError 182] The operating system cannot run %1. Error loading “D:\anaconda\envs\Pytorch\lib\site-packages\torch\lib\fbgemm.dll” or one of its dependencies</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>Could you create a new and empty virtual environment and reinstall PyTorch there as a quick test to see if some packages were deleted or are conflicting in your current env?</p>

username: Ayush_Aditya
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  help me<br>
It did work on the new environment but I reinstalled the pytorch in the existing environment and it didn’t work, it is difficult for me to switch to a new environment as taking all the modules and reinstalling them is a tedious task. Is there any other way to solve it?</p>

username: ptrblck
cooked: <p>I’m not familiar enough with Windows and don’t know what might have corrupted your environment. It seems you might be able to find missing files via <code>dumpbin /depends fbgemm.dll</code> which might help you trying to restore the env.</p>

username: Ayush_Aditya
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I discovered that it is happening due to the module MNE-Python used in the preprocessing, analysis, and visualization of EEG data. When I reinstalled the pytorch in the new environment it worked fine, and after I installed the mne module errors started occurring this module was the last module I installed before I received the error the first time.  Any suggestion for its replacement, I tried to use biosig, but for some reason, it is not getting installed.</p>

username: ptrblck
cooked: <p>I’m not familiar with <code>MNE-Python</code> and don’t know how it interacts with PyTorch, but it’s still weird its installation causes library issues in PyTorch. Unfortunately, I also don’t know any replacement of it, but you might want to check if installing <code>MNE-Python</code> would downgrade PyTorch for some reason.</p>
RuntimeError: Given groups=1, weight of size [10, 10, 3, 3], expected input[32, 3, 64, 64] to have 10 channels, but got 3 channels instead

username: pak_key
cooked: <p>from torch import nn</p>
<h1><a name="h-1" class="anchor" href="#h-1"></a>构建模型</h1>
<p>class CIFAR_10(nn.Module):<br>
def <strong>init</strong>(self,int_shape:int,heddin_shape:int,out_shape:int):<br>
super().<strong>init</strong>()<br>
self.Conv_Baock_1 = nn.Sequential(<br>
nn.Conv2d(in_channels=int_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
nn.Conv2d(in_channels=heddin_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
nn.MaxPool2d(kernel_size=1,padding=1)<br>
)<br>
self.Conv_Baock_2 = nn.Sequential(<br>
nn.Conv2d(in_channels=heddin_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
nn.Conv2d(in_channels=heddin_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
)<br>
self.classLiner = nn.Sequential(<br>
nn.Flatten(),<br>
nn.Linear(in_features=heddin_shape,<br>
out_features=out_shape)<br>
)</p>
<pre><code>def forward(self,x):
    x = self.Conv_Baock_1(x)
    print(x.shape)
    x = self.Conv_Baock_2(x)
    print(x.shape)
    x = self.classLiner(x)
    return x
</code></pre>
<p>mode_l = CIFAR_10(int_shape=10,heddin_shape=10,out_shape=10)<br>
mode_l.state_dict<br>
image_bath,label_bath = next(iter(train_data_lodael))<br>
image_bath.shape,label_bath.shape<br>
mode_l(image_bath)</p>

username: ptrblck
cooked: <p>Your input is invalid as it has 3 channels.<br>
Change the <code>int_shape</code> to <code>3</code> and it should work.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>
GPU implementation for sample entropy, and Fourier transformation

username: Ayush_Aditya
cooked: <p>Can anyone help me write code GPU implementation for sample entropy, and Fourier transformation as I tried on the CPU it is taking 5 sec for each iteration and I have a nested loop for 1280(num_trials)*14(no_channel) which will take an entire day and little more for just to find out sample entropy. So I thought if can use GPU it will be faster. I tried using “cupy” where I stored the data in GPU but failed to compute on it as I was utilizing a pre-existing module, specifically “nolds.samen”. Can anyone help or provide me with an alternative?</p>
Bfloat16 from float16 issues

username: kleingeo
cooked: <p>I have a complicated multi-loss model that I currently train with float16 without issues. I am trying to change it to bfloat16, but the model is no longer stable and some of the losses are not converging as expected. Other than some obvious reasons (the autocast not specifying bfloat16 or accidentally using gradient scaling), what could be some reasons this is happening? I am also using the latest version of torch with A100 GPUs (so hardware issues).</p>
<p>Edit: The problem was solved by using a 1e-3 rather than 1e-7 eps value in one of the loss functions to account for potential zero values.</p>
torch.Tensor.unfold with fractional step

username: hbredin
cooked: <p>How would one go to implement <code>torch.Tensor.unfold(dim, size, step)</code> with non-integer <code>step</code> ? <code>k</code>th slice should start at the index the closest to <code>k * step</code>:</p>
<p>Here is an example with <code>step = 1.3</code></p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.arange(10)
# tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

x.unfold(0, 2, 1.3)
# tensor([[0, 1],  [1, 2],  [3, 4], [4, 5], [5, 6], [6, 7], [8, 9]])
#                            ^                               ^   
</code></pre>
<p>Bonus point if it can be exported with <code>torch.export.onnx</code> <strong>and</strong> works just fine with dynamic axes <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to Optimize model for faster Training

username: Sar2580P
cooked: <p>Below is the forward pass of my model. The input <strong>x</strong> is split about time-dimension (last-dim) which has indices till 250. Below is the code…</p>
<pre><code class="lang-auto">from torch_geometric.data import Dataset, Data  
from torch_geometric.data import Batch


def forward(self, x):
    dynamic_result = None
    split_x = torch.split(x, 1, dim=-1)   # doing split along temporal dim

    for i  in range(len(split_x)):
      y = torch.squeeze(split_x[i])    # y.shape : (B, C, V)
      dynamic_weights = self.get_dynamic_adj_mat(y)       #  dynamic_weights: (B,num_edges)
      y = torch.permute(y , (0,2,1))   # y.shape : (B, V, C)
      
      graphs = []
      for j in range(y.shape[0]):
        dim, idx = 0, torch.tensor([j], dtype = torch.int32).to(device)
        d = Data(x = torch.squeeze(torch.index_select(y, dim=dim, index = idx)), 
                edge_index=self.edge_indices, 
                edge_attr=torch.squeeze(torch.index_select(dynamic_weights, dim=dim, index = idx)))
        
        graphs.append(d)  

      batch = Batch.from_data_list(graphs)
      dynamic_h = self.dynamic_gnn_layer(x= batch.x , edge_index= batch.edge_index , edge_attr = batch.edge_attr)

      dynamic_h = rearrange(dynamic_h , '(b v) c -&gt; b c v' , b = self.batch_sz , v = self.num_nodes)
      dynamic_h = torch.unsqueeze(dynamic_h , dim=3)
      # pr.yellow(f'Passed Dynamic_GNN_Layer  || dynamic_h.shape : {dynamic_h.shape}')

      if dynamic_result is None:
         dynamic_result = dynamic_h
      else :
         dynamic_result = torch.cat([dynamic_result, dynamic_h] , dim=3)
</code></pre>
<p>The 2 forward loops are the rate limiting step in training of model. The model is not big, but the for loops increases the complexity…</p>
<p>Is there a way to optimize this code? Like tensorizing things…</p>
<p>Please guide…Thanks in advance</p>
Erro :RuntimeError: pad should be at most half of kernel size, but got pad=1 and kernel_size=1

username: pak_key
cooked: <p>from torch import nn</p>
<h1><a name="h-1" class="anchor" href="#h-1"></a>构建模型</h1>
<p>class CIFAR_10(nn.Module):<br>
def <strong>init</strong>(self,int_shape:int,heddin_shape:int,out_shape:int):<br>
super().<strong>init</strong>()<br>
self.Conv_Baock_1 = nn.Sequential(<br>
nn.Conv2d(in_channels=int_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
nn.Conv2d(in_channels=heddin_shape,<br>
out_channels=heddin_shape,<br>
kernel_size=3,<br>
stride=1,<br>
padding=0),<br>
nn.ReLU(),<br>
nn.MaxPool2d(kernel_size=1,padding=1)<br>
)<br>
self.classLiner = nn.Sequential(<br>
nn.Flatten(),<br>
nn.Linear(in_features=heddin_shape,<br>
out_features=out_shape)<br>
)</p>
<pre><code>def forward(self,x):
    x = self.Conv_Baock_1(x)
    print(x.shape)
    x = self.classLiner(x)
    return x
</code></pre>

username: ptrblck
cooked: <p>The <code>nn.MaxPool2d</code> layer is failing:</p>
<pre data-code-wrap="python"><code class="lang-python">pad = nn.MaxPool2d(kernel_size=1,padding=1)
pad(x)
# # RuntimeError: pad should be at most half of effective kernel size, but got pad=1, kernel_size=1 and dilation=1
</code></pre>
<p>Increase the <code>kernel_size</code> or remove this layer, since a <code>kernel_size</code> of <code>1</code> won’t do anything and will be a no-op.</p>
Regression on the vision transformers

username: Senthur_Ram
cooked: <p>I am trying to do regression on the vit_b_16 models to estimate values from the image, but I get the issue when I try to replace the last classification layer with regression layer,</p>
<blockquote>
<p>class RegressionViT(nn.Module):<br>
def <strong>init</strong>(self, in_features=224 * 224 * 3, num_classes=1, pretrained=True):<br>
super(RegressionViT, self).<strong>init</strong>()<br>
self.vit_b_16 = vit_b_16(pretrained=pretrained)  # Load pre-trained weights</p>
<pre><code>    # Replace the final classification layer with a regression head
    self.regressor = nn.Linear(self.vit_b_16.heads.in_features, num_classes)

def forward(self, x):
    x = self.vit_b_16(x)
    x = self.regressor(x)
    return x
</code></pre>
</blockquote>
<p>I get the error for the following code</p>
<blockquote>
<p>AttributeError: ‘VisionTransformer’ object has no attribute ‘head’</p>
</blockquote>
<p>What will be the correct code to do the regression?</p>

username: ptrblck
cooked: <p>Your code fails with a different error message for me:</p>
<pre data-code-wrap="python"><code class="lang-python"># AttributeError: 'Sequential' object has no attribute 'in_features'
</code></pre>
<p>After fixing this by indexing the <code>self.vit_b_16.heads[0]</code> container, the code works.</p>
Influence of Unused FFN on Model Accuracy in PyTorch

username: Priya1
cooked: <p>I am encountering a peculiar issue with my PyTorch model where the presence of an initialized but unused FeedForward Network (FFN) affects the model’s accuracy. Specifically, when the FFN is initialized in my <code>CRS_A</code> class but not used in the <code>forward</code> pass, my model’s accuracy is higher compared to when I completely remove (or comment out) the FFN initialization. The FFN is defined as follows in my model’s constructor:</p>
<pre><code class="lang-auto">class CRS_A(nn.Module):
    def __init__(self, modal_x, modal_y, hid_dim=128, d_ff=512, dropout_rate=0.1):
        super(CRS_A, self).__init__()

        self.cross_attention = CrossAttention(modal_y, modal_x, hid_dim)
        self.ffn = nn.Sequential(
            nn.Conv1d(modal_x, d_ff, kernel_size=1),
            nn.GELU(),
            nn.Dropout(dropout_rate),
            nn.Conv1d(d_ff, 128, kernel_size=1),
            nn.Dropout(dropout_rate),
        )
        self.norm = nn.LayerNorm(modal_x)
       
        self.linear1 = nn.Conv1d(1024, 512, kernel_size=1)
        self.linear2 = nn.Conv1d(512, 300, kernel_size=1)
        self.dropout1 = nn.Dropout(0.1)
        self.dropout2 = nn.Dropout(0.1)
    def forward(self, x, y, adj):
        x = x + self.cross_attention(y, x, adj)  #torch.Size([5, 67, 1024])
        x = self.norm(x).permute(0, 2, 1)
        x = self.dropout1(F.gelu(self.linear1(x))) #torch.Size([5, 512, 67])
        x_e = self.dropout2(F.gelu(self.linear2(x))) #torch.Size([5, 300, 67])

        return x_e, x
</code></pre>
<p>As you can see, the <code>self.ffn</code> is not used in the <code>forward</code> pass. Despite this, removing or commenting out the FFN’s initialization leads to a noticeable drop in accuracy.</p>
<p>Could this be due to some form of implicit regularization, or is there another explanation for this behavior? Has anyone encountered a similar situation, and how did you address it? Any insights or explanations would be greatly appreciated.</p>

username: ptrblck
cooked: <p>Initializing modules calls into the pseudorandom number generator and will thus have an effect on the overall training. You should be able to see a similar effect when changing the random seed and rerunning the training procedure.</p>

username: Priya1
cooked: <p>Thank you for your response. Yes you are true, if I change seed or rerun I can see accuracy changes with ffn initialization. what is the recommended solution here?</p>

username: ptrblck
cooked: <p>You could change some hyperparameters, such as the learning rate or the parameter initialization, and check if this would stabilize the training.</p>
Reduce CPU memory usage while enabling non_blocking CPU offload

username: Kyle_Gregory_Sargent
cooked: <p>I have a training pipeline which offloads various components (model, model ema, optimizer) to CPU at various training step stages, and does so asynchronously (e.g. for each data buffer, calling <code>buffer.to('cpu', non_blocking=True)</code>.</p>
<p>Making these transfers non-blocking results in significant speed increases (almost 2x). However, it also blows up the CPU RAM usage and results in my training process getting OOMKilled. My guess is that due to the large number of in-flight transfers to/from CPU, the process reserves more RAM than it really needs.</p>
<p>Is there any way to achieve a middle ground here? I could just make all the transfers blocking and the training runs fine, but it would be cool to somehow give a hint to pytorch about maximum RAM usage in this async setting.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kyle_Gregory_Sargent" data-post="1" data-topic="199892">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kyle_gregory_sargent/48/62487_2.png" class="avatar"> Kyle_Gregory_Sargent:</div>
<blockquote>
<p>My guess is that due to the large number of in-flight transfers to/from CPU, the process reserves more RAM than it really needs.</p>
</blockquote>
</aside>
<p>I don’t fully understand this statement as I would assume your script <em>needs</em> the pinned memory to be able to move the tensors asynchronously to it. Why wouldn’t the process need this memory in your opinion?</p>

username: Kyle_Gregory_Sargent
cooked: <p>I just meant that rather than get OOMkilled I would prefer that the process automatically pin less memory and potentially incur some speed cost by blocking on some of those transfers. Is there any way to achieve that?</p>

username: ptrblck
cooked: <p>PyTorch won’t pin memory behind your back and you would either need to use it via the <code>DataLoader</code> or explicitly on tensors. It thus also won’t be able to reduce its usage and you might need to reduce the number of workers or avoid pinning too many tensors manually.</p>
How to really realy clear dataloader cache

username: pd3rvr
cooked: <p>I am working with about 800 000 datasets, but after first training this was slow so i decided to sample 10 percent of this for training. Now after sampling 10% I still see my confusion matrix using this old data quantity instead of the newly reduced amount. And i don’t know what is causing this. I checked torch hub and found no cache files.</p>
TVTensor, hooks and Regularizers

username: gautamvarmadatla
cooked: <ol>
<li>
<p>I was just going through the documentation for transforms v2 ( torchvision ) which stated that it added TVTensors , could anyone please explain why were they introduced and what exactly did it improve? ( I was not able to figure this out from docs ).</p>
</li>
<li>
<p>Also why isnt there any documentation about hooks on torch <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> I am having to visit Medium posts / previous threads on this forum to learn more…</p>
</li>
<li>
<p>Torch has no Kernel regularizers / Activity regularizers like TF… Ofc Using weight decay is a walkaround for L2 regularization, But won’t it make it difficult for me to apply it on a per-layer basis ( Ofc there is per-parameter option via optim)? And as far L1 regularization / Activity regularization goes would the best option be to use hooks and add them to desired layers, because most the of the torch / stack overflow threads ask us to just use .norm() on desired layers and and it to the total loss ( But this way won’t the total loss be backpropogated through all the parameters? )</p>
</li>
</ol>

username: ptrblck
cooked: <p>Details about <code>TVTensor</code>s can be found <a href="https://pytorch.org/vision/main/auto_examples/transforms/plot_tv_tensors.html#sphx-glr-auto-examples-transforms-plot-tv-tensors-py">here</a>.</p>
<p>Your other points sound valid, so it would be great if you would be interested in contributing more docs etc.</p>

username: gautamvarmadatla
cooked: <p>Awesome! Sure will do it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Backward hooks Explanation!

username: gautamvarmadatla
cooked: <p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim</p>
<p>class ThreeLayerNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ThreeLayerNet, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(10, 20)<br>
self.fc2 = nn.Linear(20, 50)<br>
self.fc3 = nn.Linear(50, 1)</p>
<p>def forward(self, x):<br>
x = torch.relu(self.fc1(x))<br>
x = torch.relu(self.fc2(x))<br>
x = self.fc3(x)<br>
return x</p>
<p>def backward_hook(module, grad_input, grad_output):<br>
for i in grad_input:<br>
print(i.shape)</p>
<p>model = ThreeLayerNet()<br>
model.fc2.register_backward_hook(backward_hook)<br>
input_data = torch.randn(1, 10)<br>
target = torch.randn(1, 1)<br>
criterion = nn.MSELoss()<br>
optimizer = optim.SGD(model.parameters(), lr=0.01)<br>
output = model(input_data)<br>
loss = criterion(output, target)<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()</p>
<p>I was trying to printout input gradients for fc2 layer as shown in the hookup function and i was getting a tuple of<br>
(torch.Size([50])<br>
torch.Size([1, 20])<br>
torch.Size([20, 50]))</p>
<p>And on printing output gradients i was getting torch.Size([1, 50])</p>
<p>Intuitively fc1-&gt;fc2-&gt;fc3 is the forward pass , And The <code>grad_input</code> contains the gradients wrt to the input of the layer ( So grads w.r.t fc1 ) . Similarly, the <code>grad_output</code> contains the gradients wrt to the output of the layer ( So grads w.r.t fc3 ). But its super confusing why i got such output <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Your code raises a warning which you should not ignore:</p>
<pre data-code-wrap="python"><code class="lang-python">UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
</code></pre>
<p>Using <code>.register_full_backward_hook</code> returns a <code>grad_input</code> shape of <code>[1, 20]</code>.</p>

username: gautamvarmadatla
cooked: <p>Ahhhhhh my bad its working now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Hyperparameter tuning in optuna

username: SeeUagain
cooked: <p>Currently, I am a student who is conducting experiments by applying deep learning and machine learning to my major field. I made an ann model with PyTorch and then applied it to the experiment, but there was a slight error, so I am leaving a writing.</p>
<ol>
<li>
<p>In the first case, after creating the PyTorch model, the hyperparameter was arbitrarily designated and applied to the test data after learning by applying kfold=5 during the learning process. And in the second case, when the hyperparameter variable was obtained through gridsearch cv and applied to the test data, the accuracy was lower. <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"><br>
I wonder why it didn’t come out with similar accuracy.</p>
</li>
<li>
<p>And for the Pytorch model, optuna, ray tune seem to be used a lot for hyperparameter tuning, so I wonder why you use this module.</p>
</li>
</ol>
<p>If anyone knows anything about this, I’d appreciate it if you could leave a little explanation. <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Conv1d input shape mismatch with custom Dataset

username: Elchuko
cooked: <p>I need to classify some behavioral data that comes from a CSV file. I’m using pandas to clean the data and Conv1d for classifying. The pandas dataframe has shape of (14029, 123).</p>
<p>The issue appears when I’m trying to feed the data into the model. As far as I understand, I need the shape of the data to be (batch_size, 1, 123). However, when feeding the data to the model, I can see it actually is torch.Size([batch_size, 123]). How is it that I’m missing the 1?</p>
<p>My custom Dataset is written as follows:</p>
<pre><code class="lang-auto">class CSVDataset(Dataset):
    def __init__(self, filepath):
        self.df = pd.read_csv(filepath)
        self.target = self.df['R']
        self.x = self.df.drop('R', axis=1)
        self.x = torch.from_numpy(self.x.to_numpy())

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return self.x[idx], self.target.loc[idx]
</code></pre>
<p>The dataset and dataloader are created as follows:</p>
<pre><code class="lang-auto">dataset = CSVDataset("../clean_data/clean_data.csv")
dataloader = DataLoader(dataset, batch_size=32, shuffle=True, drop_last=True)
</code></pre>

username: ptrblck
cooked: <p><code>unsqueeze</code> the missing channel dimension before feeding the data to the model.</p>

username: Elchuko
cooked: <p>Thanks. That worked!</p>
Build torch 2.2.0 with vulkan failed by ndk-26b

username: dearwind253
cooked: <p>torch code version:  2.2.0<br>
ndk version: ndk-r26b<br>
OS: mac IOS M1</p>
<p><strong>error :</strong></p>
<p>– Configuring done<br>
CMake Error at cmake/VulkanDependencies.cmake:18 (add_library):<br>
Cannot find source file:</p>
<pre><code>/Users/tools/android-ndk-r26b/NDK/sources/third_party/vulkan/src/common/vulkan_wrapper.h
</code></pre>
<p>Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h<br>
.hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc<br>
Call Stack (most recent call first):<br>
cmake/Dependencies.cmake:656 (include)<br>
CMakeLists.txt:735 (include)</p>
<p>CMake Error at cmake/VulkanDependencies.cmake:18 (add_library):<br>
No SOURCES given to target: VulkanWrapper<br>
Call Stack (most recent call first):<br>
cmake/Dependencies.cmake:656 (include)<br>
CMakeLists.txt:735 (include)</p>
<p><strong>where find the VulkanWrapper on ndk-r26b?</strong></p>
<p><strong>I find VulkanWrapper.h on ndk-r20b, but ndk-r20b just build for Mac IOS x86_64</strong></p>
<p>how can I build pytorch 2.2.0 with vulkan backend, when build android libtorch?</p>
Using 128 bit floating point datatype with Pytorch (not a complex number)

username: Kyrollos_Yanny
cooked: <p>I am performing some operations in Pytorch that require a very high degree of precision. I am currently using torch.float64 as my default datatype. All my numbers are real. I would like to use a float128 datatype (since memory is not an issue for this simulation). However, the only datatypes I find in the documentation is torch.complex128 where the real and imaginary parts are both 64 bits. Is there a datatype or a way I can use all the 128 bits for my real numbers?</p>
<p>Thank you</p>

username: albanD
cooked: <p>Hi,</p>
<p>We don’t have any support for float128 I’m afraid. I don’t actually think that this type is supported by cuda <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>If you don’t care about cuda, I guess we could accept a PR adding this new data type and implementations for it but no core contributor is working on that atm.<br>
But you can open an issue with a feature request if you want to discuss this further.</p>

username: Kyrollos_Yanny
cooked: <p>Thanks a lot for your answer. For this problem, which is an optics simulation, I do not care about Cuda. I will open an issue with a feature request to discuss further <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Bhanuday.Sharma
cooked: <p>I am also working on an optimization problem for Molecular Dynamics simulations where I need high precision. After months of struggle to find out the bug in a seemingly correct code, I finally reached to the conclusion that the issue is with precision limitations of float64. It would be very helpful if support can be provided for float128 (equivalent to real128 in FORTRAN) in near future.</p>

username: l5rk
cooked: <p>Hello everyone - May I ask if in the time since this discussion float128 support has been added for CPU? I do not see anything on the docs, but I’m getting into floating point issues that would be significantly helped by the addition of this feature.</p>
How to root cause - torch/autograd/__init__.py:xxx: UserWarning: Error detected in GeluBackward0. Traceback of forward call that caused the error

username: whatdhack
cooked: <p>How do I dump more details or trace what exactly made autograd to fail ?</p>

username: srishti-git1110
cooked: <p>Could you please post the relevant code and the full error trace?</p>

username: whatdhack
cooked: <p>It is part of a lager model.  The relevant past is as follows.</p>
<pre><code class="lang-auto">nn.Linear(...)
nn.GELU()
nn.Linear(...)
nn.Dropout(..)
</code></pre>
<p>Not able to reproduce in a smaller code snippet.   How do I trace the autograd process ? is it possible to get a  more detailed log of the autograd process ?</p>
<p>Will have to clean up the log a bit before I can post.</p>
LibTorch Release version loads traced model only in Debug configuration

username: heisenbergpxh
cooked: <p>Hello,</p>
<p>I want to use a model in a C++ application.<br>
For this I downloaded LibTorch for CPU (2.2.2+cpu) in the <strong>Release</strong> version.<br>
My CMakeLists.txt is:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.27)
project(TorchTest)

set(CMAKE_CXX_STANDARD 17)

set(Torch_DIR "C:/Users/hofma/Downloads/libtorch-win-shared-with-deps-2.2.2+cpu/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(TorchTest main.cpp)

target_link_libraries(TorchTest PRIVATE ${TORCH_LIBRARIES})
</code></pre>
<p>I exported the resnet18 model in python</p>
<pre><code class="lang-auto">import torch
import torchvision

model = torchvision.models.resnet18()

example = torch.rand(1, 3, 224, 224)

traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("resnet.pt")
</code></pre>
<p><code>print(torch.__version__)</code> yields <strong>2.2.2+cpu</strong></p>
<p>my C++ MWE is</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include "torch/script.h"

int main() {

    try
    {
        torch::jit::script::Module module = torch::jit::load("resnet.pt");
    }
    catch(const c10::Error&amp; e)
    {
        std::cerr &lt;&lt; "error loading the model\n";
    }
    return 0;
}
</code></pre>
<p>I use MSVC 17 to compile and this is my build command</p>
<pre><code class="lang-auto">C:\Users\hofma\AppData\Local\Programs\CLion\bin\cmake\win\x64\bin\cmake.exe -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=C:/Users/hofma/AppData/Local/Programs/CLion/bin/ninja/win/x64/ninja.exe -G Ninja -S C:\Users\hofma\Documents\CodeTest\TorchTest -B C:\Users\hofma\Documents\CodeTest\TorchTest\cmake-build-release
</code></pre>
<p>unfortunately the model is only load when I use the <strong>Debug</strong> configuration.<br>
Am I missing something to load the model in Release config.</p>
<p>Thanks for helping</p>
Can not load scripted model using torch::jit::load

username: 111408
cooked: <p>I use torch.jit.script to save model and using C++ to load it：</p>
<pre><code class="lang-auto">#include &lt;torch/script.h&gt; 
using namespace std;

int main() {
    std::string model_path = "G:/modelscriptcuda.pt";
    
   try {
    torch::jit::script::Module module = torch::jit::load(model_path, torch::kCUDA);
} catch (const c10::Error&amp; e) {
    std::cerr &lt;&lt; "error loading the model: " &lt;&lt; e.what() &lt;&lt; std::endl;
    return -1;
}}
</code></pre>
<p>And I got this error：</p>
<pre><code class="lang-auto">error loading the model: [enforce fail at ..\..\caffe2\serialize\inline_container.cc:197] . file not found: modelscriptcuda/version
(no backtrace available)
</code></pre>

username: ptrblck
cooked: <p>How did you save the checkpoint? Based on the error message I would guess your checkpoint might be in an older format as the <code>version</code> file is missing in the archive.</p>

username: 111408
cooked: <p>I use torch.save to save my model and I trace it through:<br>
<code>torch.save(model.state_dict(), save_path)</code></p>
<p>` model.load_state_dict(torch.load(‘G:/chapter5nomissing.pt’))<br>
model.eval()<br>
model = model.float()</p>
<pre><code>example_input = torch.rand(1,6, 850)
example_rawdata = torch.rand(1,6,1000)
example_index = torch.rand(1,2,30)
example_mask = torch.rand(1,1,6,1000)
print(example_input.type(),example_rawdata.type(),example_index.type(),example_mask.type())

traced_script_module = torch.jit.script(model, (example_input,example_rawdata,example_mask) )

traced_script_module.save('G:/modelscriptcuda.pt')
print("save")
load_scripted_model = torch.jit.load('G:/modelscriptcuda.pt')
print(load_scripted_model)`
</code></pre>
<p>And I found it is okay to load it in python</p>

username: ptrblck
cooked: <p>Which PyTorch and <code>libtorch</code> versions are you using as it still seems they mismatch due to the unexpected checkpoint format?</p>

username: 111408
cooked: <p>torch == 1.11.0+cu113<br>
– Found torch: I:/Project/CNN-1DMEAE/libtorch/lib/torch.lib<br>
– Found Torch: 1.4.0</p>

username: ptrblck
cooked: <p>Update both to the same version and rerun your code again.</p>
Module 'torch.utils' has no attribute 'model_zoo'

username: steeltrooper17
cooked: <p>I wanted to load the caffe model for network in network from model zoo (<a href="https://www.dropbox.com/s/blrajqirr1p31v0/cifar10_nin.caffemodel?dl=1" rel="nofollow noopener">https://www.dropbox.com/s/blrajqirr1p31v0/cifar10_nin.caffemodel?dl=1</a>). I tried to follow the method given in pytorch docs: state_dict = torch.utils.model_zoo.load_url(‘<a href="https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth" rel="nofollow noopener">https://s3.amazonaws.com/pytorch/models/resnet18-5c106cde.pth</a>’) but im getting:<br>
AttributeError: module ‘torch.utils’ has no attribute ‘model_zoo’</p>
<p>Am I doing something wrong?</p>

username: smth
cooked: <p>you must’ve forgottten <code>import torch.utils</code></p>

username: steeltrooper17
cooked: <p>No, I did import it. It still doesn’t work, do you suggest re-installing pytorch or something?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/8a1111b14f2186dde744b0af9b93bfd0839c2850.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a1111b14f2186dde744b0af9b93bfd0839c2850" title="Screenshot from 2017-06-28 11-37-33.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/8a1111b14f2186dde744b0af9b93bfd0839c2850_2_690x402.png" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/8a1111b14f2186dde744b0af9b93bfd0839c2850_2_690x402.png, https://discuss.pytorch.org/uploads/default/original/1X/8a1111b14f2186dde744b0af9b93bfd0839c2850.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/8a1111b14f2186dde744b0af9b93bfd0839c2850.png 2x" data-dominant-color="312E29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2017-06-28 11-37-33.png</span><span class="informations">958×559 342 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: smth
cooked: <p>hmmm, what version of torch are you on?</p>
<pre><code class="lang-auto">print(torch.__version__)
</code></pre>

username: smth
cooked: <p>oh i see now. you actually have to import</p>
<pre><code class="lang-auto">import torch.utils.model_zoo
</code></pre>

username: toniklenk
cooked: <p>Thanks, this worked.</p>