akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I find something strange in this PyTorch tutorial - Translation with a Sequence to Sequence Network and Attention

MrTuo: <p>Recently I read this tutorial <a href="http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="noopener nofollow ugc">Translation with a Sequence to Sequence Network and Attention</a>. The code is according to the paper <a href="https://arxiv.org/abs/1409.0473" rel="noopener nofollow ugc">Neural Machine Translation by Jointly Learning to Align and Translate</a>.  And I found that some computing process in the code is different from the paper.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2955b900502150969fe8a9aff59348a43c176282.PNG" data-download-href="https://discuss.pytorch.org/uploads/default/2955b900502150969fe8a9aff59348a43c176282" title="pic"><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/2955b900502150969fe8a9aff59348a43c176282.PNG" alt="pic" data-base62-sha1="5TFeE7esLNqeme727cuDi6Jphgm" width="690" height="405" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pic</span><span class="informations">1067×627 48.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have tag the difference with red line.Once you read the paper and the code you will find the difference.<br>
What is strange is that it woks properly! Why?Am I wrong?</p>

SimonW: <p>The code looks correct to me. If I understand correctly this a cell of a decoder RNN. The input then is exactly s_{i-1} and hidden is h_j.</p>
<p>Disclaimer: I didn’t actually read the paper nor the code.</p>
<p>Btw, if you have already read through the paper and the code, then why don’t you tell us what the notations mean and what these variables represent? Having to read paper+code to answer your question is quite time-consuming. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

MrTuo: <p>I’m sorry that I haven’t explain what the notations mean. I 'm just a student and rarely ask questions <a href="http://online.So" rel="nofollow noopener">online.So</a> please forgive me for making the mistake.<img src="//discuss.pytorch.org/images/emoji/apple/joy.png?v=5" title=":joy:" class="emoji" alt=":joy:"><br>
I think s_{i-1} is the (i-1)-th hidden state of decoder, and h_{j} is the j-th output of encoder.But in the code, embeded[0] is the input embedding of this decoder cell, and hidden[0] is hidden output of last decoder cell. How can we caculate attention weight without the information of encoder? Maybe I misunderstood the meaning of notations…<br>
Maybe only  the one who have read both paper and code can help me… Thanks a lot.</p>

fancol_tran: <p>did you find the answer, i run into the example and get the same question <a class="mention" href="/u/mrtuo">@MrTuo</a></p>
Pytorch to onnx conversion output mismatch

sathiez: <p>Hi,<br>
I converted my pytorch (.pt) model to onnx using<br>
<code>self.model.cuda().eval()</code><br>
<code>torch.onnx.export(self.model, dummy_input, "trk.onnx", export_params=True, verbose=True)</code></p>
<p>and ran it using<br>
<code>model = onnx.load("trk.onnx")</code><br>
<code>engine = backend.prepare(model, device='CUDA:0')</code><br>
<code>pred = engine.run(img)[0]</code></p>
<p><strong>Actual output (.onnx):</strong><br>
pred11 = [[[ 0.0429871 0.10708108 -0.15831867 … -0.05099011 0.04215946 0.03529864]</p>
<p>[ 0.15205492 0.14381851 -0.4134867 … -0.02547888 0.02003362 0.05678646]<br>
[ 0.11788099 0.21649244 -0.49855745 … -0.03875189 -0.00112192 0.05244121]<br>
…<br>
[-0.02988242 -0.02718411 0.17555043 … -0.05482839 -0.03526874 -0.01705099]<br>
[-0.02175785 -0.01682113 0.20047602 … -0.04242673 0.00226362 0.00717632]<br>
[-0.01177015 -0.02307639 0.05427436 … -0.03097095 0.00717672 0.00656337]]] 218 float32</p>
<p><strong>Expected output (.pt):</strong><br>
pred22 = [[[ 4.3859973e+00 3.2662495e+01 8.7064598e+01 … -5.0997593e-02 4.2160533e-02 3.5292290e-02]<br>
[ 4.7508678e+01 4.3868427e+01 6.7454163e+01 … -2.5484795e-02 2.0031789e-02 5.6778252e-02]<br>
[ 7.6021790e+01 6.6031555e+01 6.1953438e+01 … -3.8754154e-02 -1.1225204e-03 5.2439131e-02]<br>
…<br>
[ 3.9146216e+02 4.0653235e+02 2.1454182e+01 … -5.4829646e-02 -3.5270575e-02 -1.7050508e-02]<br>
[ 3.9960834e+02 4.0709204e+02 2.1995722e+01 … -4.2428117e-02 2.2626489e-03 7.1746223e-03]<br>
[ 4.0778818e+02 4.0675439e+02 1.9004044e+01 … -3.0972414e-02 7.1776402e-03 6.5612365e-03]]] 218 float32</p>
<p>First 4 values of each row(bbox values) seems to be different but rest(embeddings) are same.</p>
<p>How can I solve it?</p>
<p><a href="https://drive.google.com/open?id=1hx1C9dDa-lumlVgSWlpsK-gP1JTWipI3" rel="nofollow noopener">model</a></p>

Aya_Nasser: <p>I’ve a similar problem.<br>
Did you find the answer ?</p>
Custom loss function for VGAE

Allenkei: <p>I am trying to minimize a custom loss function for a Variational Graph Auto-Encoder. However, my custom loss function is not decreasing, and I am not sure why. Any help is appreciated. Thanks in advance!!!</p>
<pre><code class="lang-auto">def network_stat(adj):
  # Given a adjacency matrix, calculate graph statistics
  # (holderA) Num of edges and (holderB) Num of two-stars
  n = adj.shape[0]
  holderA = torch.tensor([0],dtype=torch.float32)
  holderB = torch.tensor([0],dtype=torch.float32)
  for i in range(n):
    for j in range(n):
      if i &lt; j:
        holderA = holderA + adj[i,j]
        for k in range(n):
          if i &lt; j and j &lt; k:
            holderB = holderB + (adj[i,k] * adj[j,k])
  holder = torch.cat((holderA, holderB), dim=0)
  return(holder)

class GVAE(nn.Module):
    def __init__(self, feature_size):
        super(GVAE, self).__init__()

        self.encoder_embedding_size = 32
        self.latent_size = 5

        self.encode_conv1 = GCNConv(feature_size, 2 * self.encoder_embedding_size) 
        self.encode_conv2 = GCNConv(2 * self.encoder_embedding_size, 2 * self.encoder_embedding_size)

        self.conv_mean = GCNConv(2 * self.encoder_embedding_size, self.latent_size)
        self.conv_logstd = GCNConv(2 * self.encoder_embedding_size, self.latent_size)

    def reparameterize(self, mu, logstd):
        noise = torch.randn(NUM_NODE, self.latent_size)
        output = mu + torch.exp(logstd) * noise
        return output

    def encode(self, x, edge_index):
        x = self.encode_conv1(x, edge_index).tanh()
        x = self.encode_conv2(x, edge_index).tanh()
        mu = self.conv_mean(x, edge_index).tanh()
        logstd = self.conv_logstd(x, edge_index).tanh()
        return mu, logstd

    def decode(self, Z):
        return torch.sigmoid(torch.matmul(Z,Z.t()))

    def forward(self, x, edge_index):
        mu, logstd = self.encode(x, edge_index)
        Z = self.reparameterize(mu, logstd)
        A_pred = self.decode(Z)
        return A_pred

def loss_function(A_pred):
    # GOAL: generated graph has 15 edges and 10 two-star from the above network_stat(adj) function
    g_exp = network_stat(torch.round(A_pred)) 
    loss = torch.sum( (g_exp - torch.tensor([15,10],dtype=torch.float32))**2  )
    return loss

model = GVAE(feature_size=NUM_NODE)
optimizer = torch.optim.Adam(model.parameters(), lr=0.000001)

for min_epoch in range(10000):
    A_pred = model(node_feature, edge_list) 
    loss = loss_function(A_pred)
    loss.backward()
    optimizer.step()  
    optimizer.zero_grad()
</code></pre>
If I change the activation function of VGG16, will the accuracy increase?

YuhskeHujisaki: <p>If I change the activation function of VGG16 to switch, ReLU6, LeakyReLU, Mesh, etc., will the accuracy improve? Or does it go down?</p>

ptrblck: <p>I’m not aware of any research comparing a pretrained vanilla VGG16 with different activations functions. It’s an interesting question so let us know if you’ve run some experiments and have some results.</p>
How to handle learning rate scheduler in DDP

Rakshith_V: <p>My training code runs on 2 GPU in DDP set-up , Each GPU handles a batch of 128.</p>
<p>training_steps = Overall_data / (2 GPU*128) = 5453 steps<br>
warmup_steps = 545</p>
<p>def lr_lambda(current_step: int):<br>
-----if current_step &lt; num_warmup_steps:<br>
-----return float(current_step) / float(max(1, num_warmup_steps))<br>
----- return max(<br>
0.0, float(num_training_steps - current_step) / float(max(1, num_training_steps - ------------<br>
------ num_warmup_steps))<br>
)</p>
<p>As per as the above calculation, learning rate which I should have got <span class="mention">@x</span> step I’m getting <span class="mention">@x</span>/2 step when I see the plot in tensorboard using  scheduler.get_last_lr()[0].</p>
<p>Any reason for this behavior ?</p>

rvarm1: <p>This seems unexpected, it could possibly be an issue with how data is split in your script or the accounting for the # of steps - could you please attach a repro of your issue (maybe instead of tensorboard, just print(scheduler.get_last_lr()[0]) and compare it to what you expect)?</p>
<p>Also, is the result of <code>scheduler.get_last_lr()[0]</code> what you expect if you turn off distributed training and train on local only?</p>

Rakshith_V: <p>Unfortunately , I will not be able to attach my code repo</p>
<ol>
<li>
<p>Data split :<br>
sampler = torch.utils.data.distributed.DistributedSampler(ip_data)<br>
dataloader = DataLoader(ip_data, batch_size=batch_size, sampler=sampler,persistent_workers=True,num_workers=16)<br>
<span class="hashtag">#ip_data</span> is (X,y)</p>
</li>
<li>
<p>Scheduler :<br>
num_training_steps = int(epochs <em>(len(train_loader)/dist.get_world_size()))<br>
scheduler = get_scheduler(“linear”,optimizer=optimizer,num_warmup_steps=int(0.1</em>(len(train_loader)/dist.get_world_size())),num_training_steps=num_training_steps)<br>
<span class="hashtag">#get_schedule</span> is from huggingface</p>
</li>
<li>
<p>scheduler.get_last_lr()[0]  → this is how I stored values in tensorboard<br>
[ScalarEvent(wall_time=1656084727.27605, step=100, value=3.676470441860147e-05), ScalarEvent(wall_time=1656084836.8856416, step=200, value=7.352940883720294e-05), ScalarEvent(wall_time=1656084946.7749376, step=300, value=9.989627142203972e-05),</p>
</li>
</ol>
<p>If you call the fn “lr_lambda” which I have provided above<br>
lr_lambda(200) * 0.0001 → 3.669724770642202e-05 which is equivalent of step 100</p>
<p>I will try to run it with single GPU,<br>
Thanks for the response</p>
Default process group has not been initialized, please make sure to call init_process_group

Jyotsana_Kunwer: <p>line 140, in initialize_model<br>
model = nn.parallel.DistributedDataParallel(model)<br>
File “/opt/miniconda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 530, in <strong>init</strong><br>
self.process_group = _get_default_group()<br>
File “/opt/miniconda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 410, in _get_default_group<br>
raise RuntimeError(<br>
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.</p>
What is the use of `device_ids` in DDP constructor?

Fei_Liu: <p>I have a distributed setup with 1 gpu per node &amp; four gpus/nodes in total. I noticed that it doesn’t matter whether I put <code>device_ids</code> and <code>output_device</code> in the DDP constructor, as long as I have the model device set up correctly (e.g. by <code>model.to(rank)</code>).</p>
<p>What’s the use of <code>device_ids</code> and <code>output_device</code>? Does setting them up help with speed or something else? I’m a bit confused and I didn’t find more detailed documents in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">DDP API page</a>.</p>

rvarm1: <p>If you’ve set up the model on the appropriate GPU for the rank, <code>device_ids</code> arg can be omitted, as the DDP doc mentions:</p>
<blockquote>
<p>Alternatively, <code>device_ids</code> can also be <code>None</code> .</p>
</blockquote>
<p>Also, it actually appears <code>output_device</code> is unused by DDP now, possibly after some refactoring. This argument doesn’t have to be used and won’t have any effect, so it is fine to omit it as well.</p>

Fei_Liu: <p>I’m still a bit confused. Say if I’m using singleGPU per node, then I’ll have to set up the model device correct first, otherwise it’ll complain with errors anyways. In other words, I need to do <code>DDP(model.to(rank), ...)</code> anyways, irrespective of adding <code>device_ids=[rank]</code> or not. Is it fair to say that the <code>device_ids</code> and <code>output_device</code> are just dummy (for now) and have no practical effects?</p>
torch.nn.Embedding() with variable num_embeddings in a single NN model

karbapi: <p>Hi There,</p>
<p>In torch.nn.Embedding(num_embeddings, embedding_dim, …), the depth of the table (i.e. num_embeddings value) must be defined with a constant value in a given NN model. What if, in this model, num_embeddings itself is a variable? How to cope with it?</p>
<p>Any insight in this regard will be highly appreciated.</p>
<p>Thanks,<br>
Bapi</p>

ptrblck: <p>In that case it might be simpler to create the variable <code>weight</code> parameter and use the functional API. Once you want to increase this parameter, you could use e.g. <code>torch.cat</code>, create a the new <code>weight</code> parameter, and pass it to an optimizer so that it can be trained again.</p>

karbapi: <p>Thank you very much.</p>
<p>Bapi</p>
DDP for multiple dataloaders with their own loss functions

chihchiehchen: <p>Hello,</p>
<p>I am training a multi-task learning model. Due to our data collection strategy, I have several datasets which corresponds to different output branches of model.</p>
<p>I want to apply DDP to accelerate the training process, maybe the standard approach is to use DistributedSampler on each dataloader. I wonder if it is possible to do the following: on each GPU train one specific dataset with its own loss function, and aggregate the losses to update the model.</p>
<p>If possible, are there any tutorials I can follow?</p>
<p>Thanks for your patient and help!</p>

rvarm1: <p>Yes, this should be possible - you’ll likely have to implement your own custom <code>Dataset</code> class as documented here: <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#Dataset" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataset — PyTorch 1.11.0 documentation</a></p>
<p>For example, this <code>Dataset</code> class can contain your N datasets (per worker), and you can use <code>dist.get_rank()</code> API: <a href="https://pytorch.org/docs/stable/distributed.html?highlight=get_rank#torch.distributed.get_rank" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.11.0 documentation</a> to get the right dataset given the worker.</p>

chihchiehchen: <p>I see, I will try it. Thanks for your suggestions!</p>
Is it possible to attach 4 different models to 4 different GPU

iamexperimentingnow: <p>Hi team,</p>
<p>I have built 4 models and all four models require the same input data to do prediction finally, all four prediction results should be combined together to form a single output.</p>
<p>I’m using p3.8xlarge machine, which has 4 GPU and 32 vCPU cores.</p>
<p>Is it possible to attach 4 different models to 4 different GPUs?</p>
<p>like below,</p>
<pre><code class="lang-auto">model1 = model1.cuda(0) # 0 is gpu id
model2 = model2.cuda(1) # 1 is gpu id
model3 = model3.cuda(2) # 2 is gpu id
model4 = model4.cuda(3) # 3 is gpu id
</code></pre>
<p>and how to call the above four models parallelly for prediction?</p>
<p>like pass the input to the four models at the same and get the prediction out of it.</p>
<p>something like the below flow diagram.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ec5b1eba86b8c4e5b60b94a21f73785b01dae57.png" data-download-href="https://discuss.pytorch.org/uploads/default/5ec5b1eba86b8c4e5b60b94a21f73785b01dae57" title="parallelism"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ec5b1eba86b8c4e5b60b94a21f73785b01dae57.png" alt="parallelism" data-base62-sha1="dworojSzS88dEGJF0TMItl3MPSD" width="500" height="500" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">parallelism</span><span class="informations">772×772 36.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/simonw">@SimonW</a> <a class="mention" href="/u/colesbury">@colesbury</a></p>

rvarm1: <p>Yes, this approach should pretty much work out of the box. CUDA kernels are asynchronous, meaning that control will return to the host after CUDA ops have been enqueued onto the device.</p>
<p>In particular, this means that something like <code>model2(input)</code> can be called by the host while model1(input) is still processing.</p>
<p>This should pretty much give you what you need:</p>
<pre><code class="lang-auto">o1 = m1(input) # Use GPU 0 
o2 = m2(input) # Use GPU 1
# ... and so on
out = torch.cat(o1, o2, ....) # or however you'd like to combine inputs
</code></pre>
Nn.modulelist not implemented error

Oguzhan: <p>Hi, I am trying to create several encoder blocks by using nn.ModuleList and I get NotImplementedError.</p>
<pre><code class="lang-auto">class Transformer(nn.Module):
    def __init__(self,embed_size,num_heads,N,in_channels,batch_size,num_encoders,num_class):
        super().__init__()
        self.num_heads = num_heads
        self.embed_size = embed_size
        self.N = N
        self.in_channels = in_channels
        self.batch_size = batch_size
        self.num_encoders = num_encoders
        self.num_class = num_class
        self.proj = Project(self.N,self.in_channels,self.embed_size,self.batch_size)
        self.block = nn.ModuleList([nn.ModuleList([nn.LayerNorm(self.embed_size),EncoderBlock(self.embed_size,self.num_heads,self.N,self.in_channels,self.batch_size)]) for i in range(num_encoders)])
        self.encoders = EncoderBlock(self.embed_size,self.num_heads,self.N,self.in_channels,self.batch_size)##nn.ModuleList( EncoderBlock(self.embed_size,self.num_heads,self.N,self.in_channels,self.batch_size) for i in range(num_encoders))
        self.Linear = nn.Linear(self.embed_size,self.num_class)
        
        self.layernorm = nn.LayerNorm(num_class)

        indices = [i for i in range(self.num_class)]
        self.indices = torch.tensor(indices)

    def forward(self,img):
        out = self.proj(img)
        out = self.block(out) ## I got error here
        out = self.Linear(torch.squeeze(torch.index_select(out,1,torch.tensor(self.N**2))))
        out = self.layernorm(out)
        return out
</code></pre>
<p>What is the problem or are the problems?<br>
Thanks.</p>

ptrblck: <p>You cannot execute the <code>nn.ModuleList</code> directly but would need to iterate it (as you would do with a plain Python <code>list</code>).<br>
If you want to execute the internal layers sequentially you could use <code>nn.Sequential</code> instead.</p>

Oguzhan: <p>My bad, thanks a lot.</p>
How to deploy pytorch based on Intel oneAPI, rather than third_party?

jiapei-nexera: <p>How to deploy <strong>pytorch</strong> based on Intel oneAPI, rather than third_party?</p>
<p>It looks <strong>pytorch</strong> is still build its <strong>third_party ideep</strong> ?<br>
<code>third_party/ideep/mkl-dnn</code> .</p>
<p>Well, <strong>Intel oneAPI mkl</strong> should be able to support <strong>pytorch</strong> independently if it’s manually installed. I mean: there seems to be <strong>pointless</strong> to let <strong>pytorch</strong> build upon its own <strong>third_party</strong>. From <a href="https://github.com/intel/ideep" rel="noopener nofollow ugc">https://github.com/intel/ideep</a> , <strong>ideep</strong> seems to 4 years old already…</p>

ptrblck: <p>I assume <a href="https://github.com/pytorch/pytorch/blob/79c2dfcd8ef3ab2ca14a942109242ae7440dd6b1/third_party/mkl-dnn.BUILD#L51">the BUILD definition</a> adds the latest oneDNN lib?</p>
<pre><code class="lang-auto">template_rule(
    name = "third_party/oneDNN/include_dnnl_version",
    src = "third_party/oneDNN/include/oneapi/dnnl/dnnl_version.h.in",
    out = "third_party/oneDNN/include/oneapi/dnnl/dnnl_version.h",
    substitutions = {
        "@DNNL_VERSION_MAJOR@": "2",
        "@DNNL_VERSION_MINOR@": "6",
        "@DNNL_VERSION_PATCH@": "0",
        "@DNNL_VERSION_HASH@": "52b5f107dd9cf10910aaa19cb47f3abf9b349815",
    },
)
</code></pre>
Functorch vs Higher for higher-order gradients

TinfoilHat0: <p>I’ve been using the <a href="https://github.com/facebookresearch/higher" rel="noopener nofollow ugc">Higher package</a> to handle higher-order gradients in my code. However, after the latest update, it seems to be that I can use functorch for the same task as well.</p>
<p>I wonder how do these two different approaches compare to each other, mostly in terms of computation speed. Further, is there a simple tutorial that shows me how to replace higher with functorch as well.</p>
Quantized Pre Trained Model Urls

Sairam954: <p>Hi,</p>
<p>I found this repo <a href="https://github.com/pytorch/vision/blob/release/0.8.0/torchvision/models/quantization/googlenet.py" rel="noopener nofollow ugc">https://github.com/pytorch/vision/blob/release/0.8.0/torchvision/models/quantization/googlenet.py</a> containing quantized versions of models like GoogleNet and ResNet. I see that model weights are loaded from the URLs.</p>
<pre><code class="lang-auto">quant_model_urls = {
    # fp32 GoogLeNet ported from TensorFlow, with weights quantized in PyTorch
    'googlenet_fbgemm': 'https://download.pytorch.org/models/quantized/googlenet_fbgemm-c00238cf.pth',
}
</code></pre>
<p>I have the below question on these weights:</p>
<ol>
<li>What quantization type is used for these weights, for example, QAT or Post static quantization?</li>
<li>Do we have such URLs for other torchvision models like Densenet and VGG16?</li>
</ol>
<p>Thanks</p>

Zafar: <p><a class="mention" href="/u/sairam954">@Sairam954</a> I believe there is a method here: <a href="https://github.com/pytorch/vision/blob/release/0.8.0/torchvision/models/quantization/utils.py#L19" class="inline-onebox" rel="noopener nofollow ugc">vision/utils.py at release/0.8.0 · pytorch/vision · GitHub</a><br>
It might give you some idea on how the models are quantized.</p>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> <a class="mention" href="/u/raghuramank100">@raghuramank100</a> Is there more info on these? SHould we put it in the docs?</p>

jerryzh168: <ol>
<li>
<p>here is some additional information: <a href="https://github.com/pytorch/vision/blob/release/0.8.0/references/classification/README.md#quantized" class="inline-onebox" rel="noopener nofollow ugc">vision/README.md at release/0.8.0 · pytorch/vision · GitHub</a>, it should be post training static quantization I think.</p>
</li>
<li>
<p>looks like we didn’t quantize these networks right now, but feel free to try quantizing them with FX graph mode quantization api (<a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.11.0+cu102 documentation</a>), or you can wait a few months, we will uploading the weights for some reference models that are quantized with fx graph mode quantization as well. cc <a class="mention" href="/u/andrewor">@andrewor</a></p>
</li>
</ol>
Floating point module class <class 'torch.nn.quantized._reference.modules.conv.Conv2d'> does not have a corresponding quantized module class

fnak: <p>I am trying to run quantization on a MobileNetv3.</p>
<p>My code is as follows:</p>
<pre><code class="lang-auto">import torch
import copy
from torch.quantization import get_default_qconfig
qmodel = copy.deepcopy(model)
from torch.quantization.quantize_fx import prepare_fx, convert_fx
qmodel.eval()
qconfig = get_default_qconfig("qnnpack")
qconfig_dict = {"": qconfig}
def calibrate(model, data_loader):
    model.eval()
    with torch.no_grad():
        for image, target in data_loader:
            model(image)
prepared_model = prepare_fx(qmodel, qconfig_dict)  # fuse modules and insert observers
calibrate(prepared_model, val_loader)  # run calibration on sample data
quantized_model = convert_fx(prepared_model, is_reference=True)
</code></pre>
<p>the convert_fx function returns the following error:</p>
<blockquote>
<p>AssertionError: Floating point module class &lt;class ‘torch.nn.quantized._reference.modules.conv.Conv2d’&gt; does not have a corresponding quantized module class</p>
</blockquote>
<p>What am I missing here?</p>

suraj.pt: <p>What version of torch are you using? I’m unable to reproduce this error in 1.10.1.</p>
<p>Does this error persist if you switch the flag to <code>is_reference=False</code>?</p>

fnak: <p>I am using version “1.11.0”<br>
If i turn of the is_reference, I get the following error:</p>
<blockquote>
<p>only conv2d calls with all arguments specified is supported right now in is_reference=False option</p>
</blockquote>
<p>But my MobileNet is connected to another model. Maybe it is not tracable?</p>

suraj.pt: <p>Hmm, I’m not sure - I’d guess if it wasn’t traceable it should throw an error specific to that. Paging <a class="mention" href="/u/jerryzh168">@jerryzh168</a> for help here!</p>

fnak: <p>Pretty weird indeed.<br>
I ended up doing some magic wiring to have the eager mode working on selective quantization. But I am still interested in FX.</p>

jerryzh168: <p>there might be some bugs in previous versions of pytorch, can you try pytorch master to see if the error is fixed?</p>
RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!

jiapei-nexera: <p>When I tried to <code>import torchvision</code>, I obtained the following warning… Looks <code>torchvision</code> is working fine, but still… a kind of annoying</p>
<pre data-code-wrap="console"><code class="lang-nohighlight">&gt;&gt;&gt; import torchvision as tv
~/.local/lib/python3.10/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn(
</code></pre>
Quantization using one-at-a-time analysis

fnak: <p>I am using eager mode quantization. However, I want to skip some layers from being quantized.<br>
I am following the tutorial here <a href="https://pytorch.org/blog/quantization-in-practice/" class="inline-onebox" rel="noopener nofollow ugc">Practical Quantization in PyTorch | PyTorch</a></p>
<p>To skip some layers, I wrote the following code:</p>
<pre><code class="lang-auto">for layer, _ in fusedModel.named_modules():
  if (layer in sortedSensitivityDict):
      _.qconfig = None
      print("skipping quant for", layer)
</code></pre>
<p>However, when I test the model now I get the following error:</p>
<blockquote>
<p>Could not run ‘aten::_slow_conv2d_forward’ with arguments from the ‘QuantizedCPU’ backend.</p>
</blockquote>
<p>If I understand correctly, this is because the layers with qconfig = none are receiving quantized data while expecting dequantized data.<br>
Is there a way I can add instruction to dequantize data before the layer and quantize it after the layer, in my loop? or what other possible workaround might I do for this purpose?</p>

Zafar: <p>You would need to wrap the layer that does not need quantization in a dequant-&gt;layer-&gt;quant</p>
Should `__init__` arguments be stored as persistent buffers?

StrausMG: <p>Hi!</p>
<p>Modules are often “parameterized” by simple constant non-trainable values that define the behavior (in addition to trainable parameters). Should be such values a part of the persistent state?</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self, important_value: int, ...):
        # (A) self.important_value = important_value
        # VS
        # (B) self.register_buffer('important_value', torch.tensor(important_value))
        # VS
        # (C) self.register_buffer('important_value', torch.tensor(important_value), False)
</code></pre>
<p>From the perspective of my question, (A) and (C) do not differ semantically. However, (B) is different since it makes <code>important_value</code> the part of the persistent state. It means that the value provided by user can be overwritten after loading some state. It can be both good (it prevents users from proving values inconsistent with the state) and bad (user may really want to provide a custom value).</p>
<p>Is there the “preferred” or “expected” approach or should I choose whatever I like and the only requirement is to document the behavior properly?</p>
DataLoader only giving one element "batch"

Omroth: <p>I have a dataset “ds”.  If I call</p>
<pre><code>len(ds)
</code></pre>
<p>I get a return of</p>
<pre><code>64
</code></pre>
<p>But if I do</p>
<pre><code>train_loader = DataLoader(dataset=ds, batch_size=64, shuffle=True, num_workers=0)
</code></pre>
<p>And then</p>
<pre><code>samples = next(iter(train_loader))
</code></pre>
<p>I get one sample from the dataset.  Not even an array of shape [sample] either - just the first sample.</p>
<p>What am I doing wrong?  I’ve worked with datasets plenty in the past and they seemed to work fine.</p>

Omroth: <p>The dataset’s <strong>getitem</strong> returns a sample in the form of a dictionary, if that makes any difference.</p>

Omroth: <p>Yes, it seems to resume working normally if the dataset just returns an array</p>
RuntimeError: The expanded size of the tensor (16) must match the existing size (63) at non-singleton dimension 0. Target sizes: [16]. Tensor sizes: [63]

DRISS_ELALAOUI: <p>please I used the function:</p>
<pre><code class="lang-auto">self.pool1 = TopKPooling(self.dimensions, ratio=0.8)
</code></pre>
<p>in the forward part of my class:</p>
<pre><code class="lang-auto">recon, edge_index, _, batch, _, _ = self.pool1(recon, edges, None, batch)
</code></pre>
<p>but after two epoches I get this error message:</p>
<pre><code class="lang-auto">--&gt; 247         recon, edge_index, _, batch, _, _ = self.pool1(recon, edges, None, batch)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch_geometric/nn/pool/topk_pool.py in forward(self, x, edge_index, edge_attr, batch, attn)
    160             score = softmax(score, batch)
    161 
--&gt; 162         perm = topk(score, self.ratio, batch, self.min_score)
    163         x = x[perm] * score[perm].view(-1, 1)
    164         x = self.multiplier * x if self.multiplier != 1 else x

/usr/local/lib/python3.7/dist-packages/torch_geometric/nn/pool/topk_pool.py in topk(x, ratio, batch, min_score, tol)
     19         perm = (x &gt; scores_min).nonzero(as_tuple=False).view(-1)
     20     else:
---&gt; 21         num_nodes = scatter_add(batch.new_ones(x.size(0)), batch, dim=0)
     22         batch_size, max_num_nodes = num_nodes.size(0), num_nodes.max().item()
     23 

/usr/local/lib/python3.7/dist-packages/torch_scatter/scatter.py in scatter_add(src, index, dim, out, dim_size)
     27                 out: Optional[torch.Tensor] = None,
     28                 dim_size: Optional[int] = None) -&gt; torch.Tensor:
---&gt; 29     return scatter_sum(src, index, dim, out, dim_size)
     30 
     31 

/usr/local/lib/python3.7/dist-packages/torch_scatter/scatter.py in scatter_sum(src, index, dim, out, dim_size)
      9                 out: Optional[torch.Tensor] = None,
     10                 dim_size: Optional[int] = None) -&gt; torch.Tensor:
---&gt; 11     index = broadcast(index, src, dim)
     12     if out is None:
     13         size = list(src.size())

/usr/local/lib/python3.7/dist-packages/torch_scatter/utils.py in broadcast(src, other, dim)
     10     for _ in range(src.dim(), other.dim()):
     11         src = src.unsqueeze(-1)
---&gt; 12     src = src.expand(other.size())
     13     return src

RuntimeError: The expanded size of the tensor (16) must match the existing size (63) at non-singleton dimension 0.  Target sizes: [16].  Tensor sizes: [63]

</code></pre>
<p>How can I solve this problem? and thank you</p>
Pytorch geometric with awkward arrays, getting error

velt1s: <p>Hey!</p>
<p>So, I’ve been having significant issues trying to use Pytorch Geometric along with the Awkward library.</p>
<p>So if I try and install pytorch, pyg, awkward and pyarrow at the same time:</p>
<pre><code class="lang-auto">conda create --name torchGPU python pytorch torchvision torchaudio cudatoolkit=10.2 awkward pyarrow pyg -c pytorch -c conda-forge -c pyg
</code></pre>
<p>It, for some reason, installs with pyarrow 0.11.1<br>
If I then try and read a .parquet file using ak.from_parquet() I get the error:</p>
<pre><code class="lang-auto">ImportError: pyarrow 2.0.0 or later required for ak.from_parquet
</code></pre>
<p>I do not have this issue if I install the cpu version and not the gpu version, so i’m assuming the issue is somewhere with pytorch?</p>
<p>If I also try and install pyarrow separately, I then get the error</p>
<pre><code class="lang-auto">undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs
</code></pre>
<p>If anyone knows anything about using these modules together i’d greatly appreciate it!</p>
<p>Note: I am running all of this on a linux server.</p>
How to share a gpu tensor between mulit-process

scirocc19900305: <p>my raw GPU tensor is so big,can’t afford to copy that in multiprocessing.<br>
And I need to use multiproceesing,cuz I have to do calculation in both cpu/gpu.<br>
is there any way to do that?should I use libtorch or cupy or something else?<br>
Here is my code:</p>
<pre><code class="lang-auto">import torch
import time


def task(o):
    print(o.data_ptr())
    print(o, '034')
    stream = torch.cuda.Stream()
    with torch.cuda.stream(stream):
        print(o+o)


if __name__ == '__main__':
    s_mp = []
    o = torch.randn((5000, 5000)).pin_memory().cuda()
    torch.multiprocessing.set_start_method('spawn')
    print(o.data_ptr(), 3434)
    for i in range(4):
        s_mp.append(torch.multiprocessing.Process(target=task, args=(o,)))
    for mp in s_mp:
        mp.start()
    for mp in s_mp:
        mp.join()

</code></pre>
<p>thanks for your time to read this</p>
Torch::one give Segmentation fault in c++

54WC: <p>This issue in the Python version may be reported on 10-Jul-2018 by InnovArul. But there is no issue in Python version in this case and gives Segmentation fault in C++.<br>
I run this in  Apple M1 with PyTorch version: 1.11.0. Could you suggest how to fix it?</p>
<pre><code class="lang-auto">// No issue 
torch::Tensor foo = torch::ones(30000); 

//  zsh: segmentation fault
torch::Tensor foo = torch::ones(36300); 

// No issue in pytorch 
import torch
foo = torch.ones(36300)
</code></pre>

54WC: <p>I find  torch::ones(&gt;36300) give “Segmentation fault”  when link OpenCv_LIBS in CMakeList.txt.</p>
<pre><code class="lang-auto"># No issue case:  
target_link_libraries(example-app "${TORCH_LIBRARIES}" )      # In CMakeLists.txt 

torch::Tensor foo = torch::ones(36300);    


# zsh: segmentation fault case: 

target_link_libraries(example-app ${TORCH_LIBRARIES}  ${OpenCV_LIBS})     # In CMakeLists.txt 

torch::Tensor foo = torch::ones(36300);     #torch::ones(30000) is ok.  

 zsh: segmentation fault
</code></pre>
Unrecognized tensor type ID: PythonTLSSnapshot

Hamzeh: <p>I am trying to train an encoder of a GAN model,</p>
<pre><code class="lang-auto">latent = encoder(real_img.to(torch.float32))
fake_img, fake_seg = generator([latent], input_is_latent=True)
# InputType:  torch.float32 
# InputShape:  torch.Size([1, 3, 256, 256]) 
# UniqueValues:  tensor([-1609.6704, -1520.1901, -1462.0510,  ...,  6500.6426,  6592. 7332.2651], device='cuda:0')
</code></pre>
<p>I get this error for <code>generator</code><br>
<strong>Unrecognized tensor type ID: PythonTLSSnapshot</strong></p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

Hamzeh: <p>I used random input for <code>latent</code> and had the same error.<br>
I did clone the generator (Stylegan2) again, and it is working.<br>
I think I had changed somewhere of the source code  to adapt it for my input and it was producing the error.<br>
However is there any source to read more about type IDs?</p>
Parameters of L-BGFS optimizer

rs27: <p>Hello,</p>
<p>I am trying to implement transfer learning in a deep neural network. I switch from Adam to L-BFGS. With respect to the latter, I wanted to ask a few questions and clarify doubts in my implementation.</p>
<ol>
<li>‘max_itr’ is the <em>maximal number of iterations per optimization step</em>. So, after each step, wouldn’t the value of loss get updated? When I print the loss after carrying out optimizer.step(closure), I get the same value over and over again, after multiple steps.</li>
<li>How do I access this iteration variable inside the <em>closure</em> function to plot losses and results?</li>
</ol>
<p>My code looks something like this:</p>
<pre><code class="lang-auto">optimizer1 = torch.optim.Adam(model.parameters(), lr=0.0005, betas=(0.9, 0.999),
                             eps=1e-08, weight_decay=0, amsgrad=False)

optimizer2 = torch.optim.LBFGS(model.parameters(), lr=0.0005,
                               max_iter=20, max_eval=None,
                               tolerance_grad=1e-07, tolerance_change=1e-09,
                               history_size=100, line_search_fn=None)


def closure():
    optimizer2.zero_grad()
    loss_ = Loss(..some parameters..)
    loss_.backward(retain_graph=True)
    print("----------- Epoch LBFGS: " + str(loss_.item()) + "----------------", flush=True)
    return loss_


start_time = time.time()

loss_plot = []
print_epoch = 0
for t in range(1,no_epochs):
    optimizer1.zero_grad()
    loss = Loss(...some parameters.....)
    loss.backward(retain_graph=True)
    print("\n----------- Epoch " + str(t) + ": " + str(loss.item()) + "----------------\n", flush=True)
    loss_plot.append(loss.item())
    optimizer1.step()
    my_loss_plot(loss_plot, path + '/log_loss_per_epoch.jpg')
    if t % 10 == 0:
        my_value_plot(...some parameters...)
    if t % 20 == 0:
        my_value_plotMACROS31(...some paramteres...)
        my_f_plot31(...some parameters...)
    if  loss.item() &lt; 1e-4:
        print_epoch = t
        break


print("\n\nSWITCHING OPTIMIZER: Adam -&gt; L-BFGS\n\n", flush=True)


for t in range(print_epoch,1000):
    optimizer2.step(closure)
    print("\n----------- Epoch " + str(t) + ": " + str(loss.item()) + "----------------\n", flush=True)
    loss_plot.append(loss.item())
    my_loss_plot(loss_plot, path + '/log_loss_per_epoch.jpg')
    if t % 40 == 0:
        my_value_plot(...some paramteres...)
    if t % 45 == 0:
        my_value_plotMACROS31(...some paramteres...)
        my_f_plot31(...some paramteres...)
    if loss.item() &lt; 1e-9:           # I am not getting updated values here!
        break


print("\n--- %s seconds ---" % (time.time() - start_time))
</code></pre>
Sharing CUDA tensor

yousiyu: <p>The following code doesn’t seem to work when I try to pass CUDA tensors around between two processes. I am following as suggested here: <a href="https://pytorch.org/docs/1.10/multiprocessing.html?highlight=multiprocessing#module-torch.multiprocessing" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing package - torch.multiprocessing — PyTorch 1.10 documentation</a></p>
<pre><code class="lang-auto">import torch as tc

def enqueue(data, queue):
    for x in data:
        queue.put(x.cuda(non_blocking=True))

def main():
    tc.multiprocessing.set_start_method('spawn')
    data = [tc.randn(100, 100).share_memory_() for _ in range(100)]  # create 100 tensors on shared memory
    results = []
    queue = tc.multiprocessing.Queue(10)  # this is the queue I am going to use to pass cuda tensors around
    producer = tc.multiprocessing.Process(target=enqueue, args=(data, queue))
    producer.start()  # start producing cuda tensors from the child process 
    while len(results) &lt; len(data):
        x = queue.get()
        results.append(x.mean())
        del x
    producer.join()
    print(results)

if __name__ == '__main__':
    main()
</code></pre>
<p>And I am getting the following error from the child process:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "tmp.py", line 22, in &lt;module&gt;
    main()
  File "tmp.py", line 15, in main
    x = queue.get()
  File "/n/nix/tech/store/9fmi8gkrx3sv8l88pzz7hrrxn60hr769/lib/python3.7/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/n/nix/tech/store/i80m32g1v6r74vcwm4jz789p81g9933l/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 119, in rebuild_cuda_tensor
    event_sync_required)
RuntimeError: CUDA error: API call is not supported in the installed CUDA driver
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>Am I missing something? Thanks.</p>

yousiyu: <p>Python version: 3.7.6<br>
Pytorch version: 1.10.2</p>

yousiyu: <p>I found this issue related to my CUDA driver. Problem solved.</p>

scirocc19900305: <p>I run your code,get the exact error you did.<br>
Is it a bug free code?cuz if so,then I got to fix my CUDA driver.</p>
Opinions on setting the minimum number of epochs

seunghyun.me: <p>Hello, everyone. I’m currently training an image segmentation model using PyTorch.</p>
<p>While doing this, I’ve watched the training results like the below picture.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c15b816cf44baa1726a8c2ed60418985110dd0a1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c15b816cf44baa1726a8c2ed60418985110dd0a1" title="Screen Shot 2022-06-27 at 3.28.17 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b816cf44baa1726a8c2ed60418985110dd0a1_2_690x458.jpeg" alt="Screen Shot 2022-06-27 at 3.28.17 PM" data-base62-sha1="rAwcuQCIqQE5GmWwiCJB4eOkaWJ" width="690" height="458" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b816cf44baa1726a8c2ed60418985110dd0a1_2_690x458.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b816cf44baa1726a8c2ed60418985110dd0a1_2_1035x687.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c15b816cf44baa1726a8c2ed60418985110dd0a1_2_1380x916.jpeg 2x" data-dominant-color="FAFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-06-27 at 3.28.17 PM</span><span class="informations">1604×1066 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I thought the model would be better if I train it more. (Currently, I set the minimum number of epochs to 0 and the maximum number of epochs to 200, and used the early stopping technique.)</p>
<p>I wonder whether it is a good way to set the minimum number of epochs (like setting it to 30), and I also want to ask you about which value is proper for the minimum number of epochs.</p>
How could I swtich off these error message

coincheung: <p>Hi,</p>
<p>I use screen command to start my training program, but when I got some error and the program crashes, I see error messages like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19885e0d95a71c214463b32ae1ebfb6703431bc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/19885e0d95a71c214463b32ae1ebfb6703431bc8" title="图片"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19885e0d95a71c214463b32ae1ebfb6703431bc8.png" alt="图片" data-base62-sha1="3DS6mee85oCH2dln0TQSkjKrIa4" width="690" height="361" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">1393×730 27.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These are not showing me which part of my code is wrong, and I cannot see more since screen is occupied by the latest messages. How could I switch off these please?</p>
Best practice to resize an uneven image (dataset with different width, height) to a fixed size image?

Ammar_ul_hassan: <p>My model takes a fixed-size input image (128x128). However, my dataset contains images with variable widths and heights. I want to resize them to a fixed size, e.g., 128 x 128 for my model. What is the best practice in this case?</p>
<p>Directly using transforms.Resize() in this case squashes the output. I want to maintain the aspect ratio as much as I can (I know there will be a tradeoff, but if not, then how can I fix this issue?</p>

ptrblck: <p>I don’t fully understand the question, but wouldn’t <code>transforms.Resize((128, 128))</code> just work?</p>

Ammar_ul_hassan: <p>Sorry for replying late.</p>
<p>The dataset contains images of different sizes, like 120x32, 189x78, 220x64, etc.<br>
My model takes 128x128 images as input.<br>
If I resize the image using<code> transforms.Resize((128, 128))</code>, the resulting image is squeezed or stretched image additionally it does not keep the aspect ratio of the input image.</p>
<p>So what is the best solution in this case? Should I pad the images like the jittering technique for this particular scenario?</p>

ptrblck: <p>Thanks for the explanation. I thought the issue was a functional one, i.e. you were stuck at the actual resizing operation. I dont know which approach (resizing, padding, resizing and cropping, etc.) would work best for your use case so you would need to compare these approaches.</p>

Ammar_ul_hassan: <p>Alright, thank you. So, in general, there is no standard approach for this case?</p>
Machine restarts while training

sadiva_madaan: <p>I’m training a segmentation model using pytorch but system randomly restarts while training. It is not an overheating issue as i control fan speed manually and GPU temperature is always around 70-78C. What’s also strange is that machine always restarts when i train using pytorch. This issue has never occurred whenever i train using fastai.<br>
I have so far made all these changes but the problem still persists -</p>
<ol>
<li>Limiting GPU power consumption. I have Nvidia RTX A4000 (from 140W to 110W)</li>
<li>Turning of intel turbo boost from BIOS as discussed here <a href="https://github.com/pytorch/pytorch/issues/3022" class="inline-onebox" rel="noopener nofollow ugc">Reliably repeating pytorch system crash/reboot when using imagenet examples · Issue #3022 · pytorch/pytorch · GitHub</a> .</li>
</ol>
<p>Also I’m using WSL2 on Windows 11.</p>

ptrblck: <p>Most likely it’s a PSU issue (you can search for similar issues in this forum) and a fast power spike could cause the PSU to shut down.</p>

sadiva_madaan: <p>I have an 850W PSU with a single GPU . Don’t think it can be a PSU issue .</p>

patrickctrf: <p>Have you checked your memory usage? I’ve experienced some bugs related to RAM usage in Windows that doesn’t happen in Linux, for example.</p>