akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Not understand the error

AbuOmair: <p>Hi<br>
I am facing this waning I don’t understand the warning what does it mean</p>
<blockquote>
<p>C:\Users\NUC-i7 8gen\Anaconda3\lib\site-packages\torch\nn\modules\loss.py:431: UserWarning: Using a target size (torch.Size([32, 6])) that is different to the input size (torch.Size([32, 1, 6])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)<br>
C:\Users\NUC-i7 8gen\Anaconda3\lib\site-packages\torch\nn\modules\loss.py:431: UserWarning: Using a target size (torch.Size([29, 6])) that is different to the input size (torch.Size([29, 1, 6])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.<br>
return F.mse_loss(input, target, reduction=self.reduction)</p>
</blockquote>
<p>and my training code is</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for inputs,labels in train_loader:
        
        inputs = inputs.float()
        labels = labels.float()
        
        # Feed Forward
        output = model(inputs)
        output = output.unsqueeze(1)
        
        # Loss Calculation
        loss_train = criterion(output, labels)
        train_loss.append(loss_train)
        
        # Clear the gradient buffer (we don't want to accumulate gradients)
        optimizer.zero_grad()
        
        # Backpropagation 
        loss_train.backward()
        
        # Weight Update: w &lt;-- w - lr * gradient
        optimizer.step()
        
    #Accuracy
    # Since we are using a sigmoid, we will need to perform some thresholding
    output = (output&gt;0.5).float()
    # Accuracy: (output == labels).float().sum() / output.shape[0]
    accuracy = (output == labels).float().mean()
    # Print statistics 
    print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(epoch+1,num_epochs, loss_train, accuracy))
</code></pre>
<p>Any one can explain to me the warning’s concept.</p>
<p>Thanks in advance.</p>

ptrblck: <p>Since your output and target shape do not match, but are broadcastable, you might get a wrong result.<br>
Have a look at this example:</p>
<pre><code class="lang-python">x = torch.arange(1, 5).view(2, 2)
y = torch.ones(2, 2)
loss = x - y
print(loss)

x = x.unsqueeze(1)
loss = x - y
print(loss)
</code></pre>
<p>While you would most likely expect the first result using your tensors, you might end up with the second one.<br>
To resolve this issue, you could squeeze dim1 in your model output:</p>
<pre><code class="lang-python">output = model(data)
output = output.squeeze(1)
loss = criterion(output, target)
</code></pre>
Implementing element-wise logical and tensor operation

jici_xing: <p>Hi ,<br>
I have 2 matrix that have same shape.I want to do element-wise logical_and operation and return a matrix that match the original shape. How to do this?<br>
Thank you for your time and consideration!</p>

yunusemre: <aside class="quote" data-post="1" data-topic="13182">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/c2a13f/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-do-elementwise-multiplication-of-two-vectors/13182">How to do elementwise multiplication of two vectors?</a> 
  </div>
  <blockquote>
    I have two vectors each of length n, I want element wise multiplication of two vectors. result will be a vector of length n.
  </blockquote>
</aside>

<p>Just use</p>
<blockquote>
<p><code>a * b</code>  or  <code>torch.mul(a, b)</code></p>
</blockquote>

zhl515: <p>Torch.dtype  is the data type of torch.tensor, the torch.dtype doesn’t have bool type, similar to bool type is torch.uint8, your can see torch.dtype <a href="https://pytorch.org/docs/stable/tensor_attributes.html" rel="nofollow noopener">here</a>. So if do logical_and operation on two tensor, you should expect to get 0/1s numerical values not True/False bool values. And tensor with uint8 type have  logical_and operation &amp;, you need change the tensor  data type to unint8 first.</p>
<pre><code class="lang-auto">a = torch.tensor([[0, 1], [0.0, 2.3]])
b = torch.tensor([[1, 1], [0.0, 0.0]])
c = torch.zeros_like(a) # all zero with same shape of a
result =  (~torch.eq(a, c)) &amp; (~torch.eq(b, c))
'''
result is a 
 tensor([[0, 1],
        [0, 0]], dtype=torch.uint8)
'''
result.numpy().astype('bool') # change to bool ndarray in numpy
</code></pre>

drevicko: <p><a class="mention" href="/u/zhl515">@zhl515</a> since pytorch 1.2, there is a dtype torch.bool and associated BoolTensor. The bitwise operators aka numpy (and &amp;, or |, xor ^ and not ~) all work on these as logical operations.</p>
Batchnorm1d for evaluation

Nitesh: <p>In my code, I’ve used nn.batchnorm1d(64) as one of the layers where 64 is the batch size. Now after training my net I want to test the result for a single sample but it shows a error saying that I need to pass 64 samples. Is there any workaround to test the net with single sample?</p>

ptrblck: <p>The <code>in_features</code> argument you are passing defines the number of channels C in a <code>[N, C, L]</code> input or the sequence length L in an input with the shape <code>[N, L]</code>.<br>
It does not specify the batch size N.</p>
<p>For evaluation and testing you should call <code>model.eval()</code> which will apply the running stats to the samples and which will work with single samples as well.</p>

Nitesh: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’m using an RNN encoder-deocder model and at the decoder side I have few linear layers. I’m using batchnorm in between these layer. My input to batchnorm1d layer is of the shape (seq_len, batch, hidden_size) i.e [1, 64, 256].<br>
So what should I pass to batchnorm1d().<br>
I tried passing 64 but after training the net when I’m passing a single sample it shows the error ‘running_mean should contain 1 element not 64’.<br>
Please help me here.</p>

ptrblck: <p>I think you would have to permute the activation from <code>[seq, batch_size, features]</code> to <code>[batch_size, features, seq]</code> via <code>x = x.permute(1, 2, 0)</code> and pass it to the batch norm layer.<br>
Currently you are using the batch dimension to normalize the activations, which won’t work.</p>
<p>Alternatively, you could also set <code>batch_first=True</code> in your RNN, which should return the tensor as <code>[batch_size, seq, features]</code> and permute the last two dimensions,</p>

Nitesh: <p>Thanks for giving your time. Now I understand it. One more question though<br>
My batch norm layer is in between linear layers and all of them are stacked in nn.Sequential(). Is there a way to permute the input inside nn.Sequential() because if not then I have to separate out the batch norm layer which makes the code little messy.</p>

ptrblck: <p>If the batch norm layers are between linear layers, the shape should most likely once be changed before the first linear layer.</p>
<p>Anyway, you could define a <code>Permute</code> layer and use it inside your <code>nn.Sequential</code> container:</p>
<pre><code class="lang-python">class Permute(nn.Module):
    def __init__(self, dims):
        super(Permute, self).__init__()
        self.dims = dims
        
    def forward(self, x):
        x = x.permute(self.dims).contiguous()
        return x
</code></pre>
Costume loss function for multi-label

netaglazer: <p>hi, i have a multi label problem, and i built a loss:</p>
<blockquote>
<p>def loss(p, y):<br>
fa= -np.dot(1-y), list(np.log(1-,p))<br>
l = (y * list(np.log(p))<br>
prod = (1-y) -  l<br>
loss = fa+ np.min(prod)<br>
return loss</p>
</blockquote>
<p>the problem is, that this function gets 61 length prediction vector (prec for each class), and 61 length label vector</p>
<p>what i dont understand is:</p>
<ol>
<li>how can i turn it into cosume loss that actually works with the code? (backpro. et…)</li>
<li>how can i handle with a list of all the prediction and the labels in the bacth with this function?</li>
</ol>

ptrblck: <p>Could you explain, why you need a <code>list</code> in the code?<br>
Would it be possible to use a tensor instead?</p>
<p>Regarding point 1: if you can write all your operations using PyTorch methods, Autograd will be able to calculate the gradients in the backward call automatically.</p>

netaglazer: <p>i ment to tensor, not list, im sorry<br>
the question actually was:<br>
when i write my own loss function (the one up here), and the loss gets a tensor of predictions and labels each time, should i sum all the distances?</p>
<blockquote>
<p>final_loss = loss(PRED1,LABEL1)+loss(PRED2,LABEL2)+…+loss(PREDn,LABELn)<br>
final_loss.backward()<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>
</blockquote>

ptrblck: <p>You could sum them (and take the average afterwards) and it depends on your use case.</p>
<p>If you are not sure, if each loss creates valid gradients, you could check its <code>.loss_fn</code>.<br>
If it’s some function other than a <code>None</code> value, it should work.</p>
<p>Do you see any errors using this approach?</p>
How to remove a class from a dataset?

mcfn: <p>I have an ImageFolder dataset containing many classes. I want to modify the classes without modifying the underlying folder.<br>
Example for such modification - removing a certain class, i.e. training and testing only on classes 2 and 4 out of the existing 5.<br>
I am aware of the torch.utils.data.Subset function, but by selecting the relevant indices this way, many of the dataset fields remain irrelevant: for example if i remove all the indices of a certain class, the dataset.classes field is left at the original length, containing the dropped class.<br>
I thought that maybe this is irrelevant cause I can just adjust my tensors’ shapes to the new number of classes, but some problems remain, e.g. the targets of the classes i am using are not shifted accordingly (if i removed a class, the target indices of the succeeding samples should decrease by 1).<br>
Is there any way to remove a class cleanly?</p>

ptrblck: <p>I think writing a custom <code>Dataset</code> using some <code>ImageFolder</code> methods (and filtering out the unwanted folders) might be the cleanest approach.</p>

mcfn: <p>i know it’s more of a software engineering question, but just to make sure - this is the inheritance:<br>
ImageFolder - DatasetFolder - VisionDataset - Dataset</p>
<p>So I find in which level the reading of the folder is being performed, copy the objects from this level downward, and modify the relevant object?</p>

ptrblck: <p>I would probably try to derive or re-implement <code>DatasetFolder</code>, as <a href="https://github.com/pytorch/vision/blob/bb5af1d77658133af8be8c9b1a13139722315c3a/torchvision/datasets/folder.py#L93">this line of code</a> seems to be the interesting one.</p>
Torch.no_grad() for Java

vishalagarwal: <p>What is the equivalent for doing <code>torch.no_grad()</code> in Java?<br>
Like in C++ it is <code>torch::NoGradGuard</code></p>

albanD: <p>Hi,</p>
<p>You mean on mobile? Or do you refer to another Java binding?<br>
IIRC it does not support training, so it is running in no_grad all the time.</p>

vishalagarwal: <p>I mean running it on mobile.</p>

IvanKobzarev: <p><a class="mention" href="/u/vishalagarwal">@vishalagarwal</a><br>
Java binding works for forward only and auto grad is disabled all the time.</p>
<p>Details:<br>
It has torch::autograd::AutoGradMode g{false} before every forward operation.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/cpp/pytorch_jni_jit.cpp#L28" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/cpp/pytorch_jni_jit.cpp#L28" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/android/pytorch_android/src/main/cpp/pytorch_jni_jit.cpp#L28</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="18" style="counter-reset: li-counter 17 ;">
<li>#include &lt;android/asset_manager_jni.h&gt;</li>
<li>#include &lt;android/log.h&gt;</li>
<li>#endif</li>
<li>
</li>
<li>namespace pytorch_jni {</li>
<li>
</li>
<li>namespace {</li>
<li>
</li>
<li>struct JITCallGuard {</li>
<li>  // AutoGrad is disabled for mobile by default.</li>
<li class="selected">  torch::autograd::AutoGradMode no_autograd_guard{false};</li>
<li>  // Disable graph optimizer to ensure list of unused ops are not changed for</li>
<li>  // custom mobile build.</li>
<li>  torch::jit::GraphOptimizerEnabledGuard no_optimizer_guard{false};</li>
<li>};</li>
<li>
</li>
<li>} // namespace</li>
<li>
</li>
<li>class MemoryReadAdapter final : public caffe2::serialize::ReadAdapterInterface {</li>
<li> public:</li>
<li>  explicit MemoryReadAdapter(const void* data, off_t size)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Short-circuiting torch.where?

f3ba: <p>Does <code>torch.where(cond, x, y)</code> evaluates all of <code>x</code> and all of <code>y</code> always? Is there a way to avoid computing elements of <code>x</code> where <code>cond</code> is false?</p>

albanD: <p>Isn’t <code>x</code> already a full Tensor? So you have to compute it before giving it to <code>torch.where()</code> right?</p>

f3ba: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="67466" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Isn’t <code>x</code> already a full Tensor? So you have to compute it before giving it to <code>torch.where()</code> right?</p>
</blockquote>
</aside>
<p>I am actually doing something like <code>torch.where(cond, f(x), g(y))</code>, where <code>f</code> and <code>g</code> act element-wise and can be expensive. For example see the definitions of <code>logerfc</code> and <code>logerfcx</code> in this code snippet: <a href="https://discuss.pytorch.org/t/how-to-call-scipy-functions/67290/3" class="inline-onebox">How to call SciPy functions? - #3 by f3ba</a></p>

nairbv: <blockquote>
<p>I am actually doing something like  <code>torch.where(cond, f(x), g(y))</code> , where  <code>f</code>  and  <code>g</code>  act element-wise and can be expensive</p>
</blockquote>
<p>f and g will be evaluated before their results are passed to <code>where</code>. To not compute values for certain elements, the implementations of <code>f</code> and <code>g</code> would need to know about <code>cond</code>.</p>
<p>Maybe you could do something like <code>f(x.masked_select(cond))</code> using cond as a mask, and use the indices in cond to convert the result of f back to the original dimensions.</p>
<p>E.g. something like:</p>
<pre><code class="lang-auto">indices=torch.nonzero(mask)[:, 0].unsqueeze(0)
values=f(x.masked_select(mask)) # only runs f on the selected values
sparse = torch.sparse_coo_tensor(indices, values, x.size())
result=sparse.to_dense()
</code></pre>
Using torchtext for a multiclass classification problem

JACOBIN-SCTCS: <p>I have a dataset which is in csv format having text as first field and class label as second field . The number of classes are 6 . How can i create a field object for this. Example code tutorials are helpful</p>

zhangguanheng66: <p>The pipeline to load dataset is very similar like <a href="https://github.com/pytorch/text/blob/master/torchtext/experimental/datasets/text_classification.py" rel="nofollow noopener">this</a>.</p>
<p>For the multiclass classification problem, there are different models for that. You may take a look at the Bag-Of-Word model <a href="https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html" rel="nofollow noopener">here</a></p>

sai_m: <p>for Mutli classification:<br>
LABEL = data.LabelField(), for multi classification, we donot need to set dtype as pytorch expects it be as numericalized long tensors</p>
<p>IMO, best place to start : <a href="https://github.com/bentrevett/pytorch-sentiment-analysis" rel="nofollow noopener">https://github.com/bentrevett/pytorch-sentiment-analysis</a></p>
<p>Happy learning</p>
Problem in understanding embedding layer with pretrained word embedding vectors

Mahsa_Afsharizade: <p>Hi, I have some problems in understanding embedding layer in PyTorch. I know that embedding layer is a lookup table with dimensions <code>vocab_size x embedding_dim</code>. we can retrieve embedding vectors from embedding layer by their indices. Suppose I want to use pretrained word embedding vectors obtained from GloVe model.<br>
This is some parts of my code:</p>
<p>def <strong>init</strong>(…,my_embeddings,…):<br>
self.embedding = nn.Embedding(vocab_size, hidden_size)<br>
self.embedding.weight.data.copy_(torch.from_numpy(my_embeddings))</p>
<p>def forward(input, hidden):<br>
embedded = self.embedding(input)</p>
<p><code>my_embeddings</code> is a <code>vocab_size x embedding_dim</code> matrix.</p>
<blockquote>
<p>1- Is it correct that <code>vocab_size</code> is the count of unique words in the train dataset?</p>
</blockquote>
<p>for missing words of train dataset (the words that are not included in the pretrained model) I have built an embedding vector that is the average of all of the word embedding vectors in train dataset.</p>
<p>I dont have problem with train dataset. But I have a problem with the test dataset.</p>
<blockquote>
<p>2-Do I have to use from the same <code>my_embedding</code> as in the train phase (<code>my_embedding</code> that is constructed by the train dataset)?</p>
</blockquote>
<blockquote>
<p>3- What should I do for the words from test dataset that are not included in <code>my_embedding</code>?</p>
</blockquote>
<p>Thanks in advance.</p>

sai_m: <p>yes, vocab_size returns unique words unless you limit vocab size to some value.<br>
I think you have to use the same embeddings which was used in training phase.</p>
<p>I am not sure about 3, if you want to use the logic of average embeddings from train set to fill unknown tokens in train set.</p>
<p>instead, you can initialize unk_init = torch.Tensor.normal_. so that, pytorch will initialize unknown words via Gaussian distribution and this can be applied to train and test sets.</p>
Can't load model after dynamic quantization

JeffO: <p>Hello!  I’m trying to do dynamic quantization as described <a href="https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html" rel="nofollow noopener">here</a>.</p>
<p>To quantize my own fine-tuned Bert model, I do this:</p>
<pre><code>model = BertForSequenceClassification.from_pretrained('model_dir')
model.to('cpu')
quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8)
quantized_model.save_pretrained('model_dir_q')
</code></pre>
<p>But when I later load the model, I get this error:</p>
<pre><code class="lang-auto">  File "/.../bert.py", line 329, in main
    model = BertForSequenceClassification.from_pretrained(args.output_dir)
  File "/.../transformers/modeling_utils.py", line 486, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
	While copying the parameter named "bert.encoder.layer.0.attention.self.query.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]).
	While copying the parameter named "bert.encoder.layer.0.attention.self.key.weight", whose dimensions in the model are torch.Size([768, 768]) and whose dimensions in the checkpoint are torch.Size([768, 768]).
...
</code></pre>
<p>Would appreciate any guidance for how to fix this.</p>

ptrblck: <p>The error message seems to be at least misleading.<br>
Maybe <a class="mention" href="/u/jerryzh168">@jerryzh168</a> knows, what’s going on.</p>

JeffO: <p>I should add that I’m using Torch version 1.3.1 in case it matters.</p>

dskhudia: <p><a class="mention" href="/u/jeffo">@JeffO</a> Is this model available publicly so that we can try the steps locally?</p>

JeffO: <p>Hi <a class="mention" href="/u/dskhudia">@dskhudia</a>, my model isn’t public, but you can recreate the error with just Bert base like this:</p>
<pre><code class="lang-auto">import torch
from transformers import BertConfig, BertForSequenceClassification

config = BertConfig.from_pretrained('bert-base-uncased')

model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased', config=config)

model.to('cpu')

quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8)

quantized_model.save_pretrained('model_dir_q')

model = BertForSequenceClassification.from_pretrained('model_dir_q')
</code></pre>

mhajiaghayi: <p>I also ran into the same error.</p>

JeffO: <p>I tried again with Torch 1.4.0.  The previous error is no longer present, but quantization breaks the model.  Performance goes from 68.6% down to 3.8% on my task.</p>
<p>I’ll dig deeper to see if I can figure out why, but any suggestions would be greatly appreciated.</p>
Can i convert torch script module to nn module

Liron_Mor_Yosef: <p>Hi,<br>
i have a problem when converting torch script module in to ONNX module,<br>
but it’s works just fine on “normal” module,</p>
<p>i open a bug request about that at <a href="https://github.com/pytorch/pytorch/issues/30512" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/30512</a>,</p>
<p>but if i could convert the torch script module to nn module it’s will solve my issue,</p>
<p>Thanks,<br>
Liron</p>
<p>code example:(you can ignore the test_normal )</p>
<pre><code class="lang-python">    def _test_normal(self, num_classes, dummy_input):
        model = torchvision.models.resnet18(num_classes=num_classes)
        model_state_fixed = {}
        for k, v in self._model_state.items():
            k_fixed = k[3:len(k)]
            model_state_fixed[k_fixed] = v
        model.load_state_dict(model_state_fixed)
        torch.onnx.export(model, dummy_input, "/app_data/test_torch_script/torch_script_test_normal.onnx")

    def convert(self):
        loaded = torch.jit.load(self._torch_script_path)
        # loaded.load_state_dict(self._model_state)
        dummy_input = torch.randn(1, 3, 224, 224)
        target = loaded(dummy_input)
        self._test_normal(num_classes=len(target[0]), dummy_input=dummy_input)
        torch.onnx.export(loaded, dummy_input, self._out_onnx_path, verbose=True,
                          operator_export_type=torch.onnx.OperatorExportTypes.ONNX,
                          example_outputs=target)
</code></pre>

driazati: <p>There isn’t a way to extract an <code>nn.Module</code> from a compiled <code>ScriptModule</code>. Is it possible for you to instead export your original module instead of a <code>ScriptModule</code>?</p>
<p>For some background, <code>torch.onnx.export</code> will use <code>torch.jit.trace</code> to get an exportable graph from an <code>nn.Module</code>. The ONNX exporter does not support all the features of TorchScript (e.g. if you used <code>torch.jit.script</code> to compile your model, it may not be possible to export that compiled module to ONNX), but relying on <code>torch.jit.trace</code> enforces that only supported features are used.</p>

Liron_Mor_Yosef: <p>thanks for your replay,<br>
how can i exported model to torch script and than load it back and export to onnx?</p>
<p>to save as torch script i am using the following code:</p>
<pre><code class="lang-python">model.eval()
dummy_input = torch.randn(1, 3, 224, 224)
traced = torch.jit.trace(model, dummy_input)
traced.save("/app_data/test_torch_script/torch_script_test.zip")
</code></pre>
<p>can i change the above code in order to use it’s output for exporting to onnx?</p>

Liron_Mor_Yosef: <p>fixed the bug at <a href="https://github.com/pytorch/pytorch/issues/30512" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/30512</a></p>
Two Deep Models in single gpu - Train/Predict

srinivasan.vasu89: <p>I am relatively very new to DL. I am thinking of implementing a code like below<br>
… predict using a pretrained model and output is an image. These images are fetched as frames from videos<br>
… use the predicted output to train another model</p>
<p>I have only one GPu 8gb. I am wondering is there a way to do data fetching and training together on a GPU without bottleneck. Is there a way to optimitze them.</p>
<p>However I tried to save individual frames and also tensor.pt file. Both seems to take more time for preprocessing and training. I am using pytorch and any suggestion would be greatly helpful.  Thanks</p>

G.M: <p>I suggest getting the result from the first model first, then use the results to train the second model.</p>
Function default argument: comparison with None?

Jean-Eric_Campagne: <p>Hello,<br>
Imagine the following simple function that result depend on an optional arg</p>
<pre><code class="lang-auto">def fnt(a,b,c=None):
  if c != None:
    res = a+b+c
  else:
   res = a+b
 return res
</code></pre>
<p>The point is that if <code>c</code>is set to a torch tensor as <code>a</code>and `b, then an exception is rised.</p>
<pre><code class="lang-auto">TypeError: ne() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (NoneType)
 * (Number other)
      didn't match because some of the arguments have invalid types: (NoneType)
</code></pre>
<p>Do you know how to manage the possibility to get <code>c</code> as option?<br>
(nb. I have swap the if condition according to <a class="mention" href="/u/deepali">@Deepali</a> suggestion even if the problem remains)</p>

Deepali: <p>I think you want to swap the if conditions:</p>
<pre><code class="lang-auto">def fnt(a,b,c=None):
  if c != None:
    res = a+b+c  //c is not None, so use it. 
  else:
   res = a+b  //forget c as it is None
 return res
</code></pre>
<p>Also use <code>torch.add</code> method to add the tensors.</p>

Jean-Eric_Campagne: <p>Sorry,<br>
The fnt I consider in the example is simple and you’re right in the philosophy.</p>
<p>But it remains that the <code>c != None</code> cannot apply. <code>c</code> is a torch.tensor of shape (N,1) where N is the batch size.</p>
<pre><code class="lang-auto">a= torch.Size([100, 1])
 if a != None:
    print(a)
 else:
    print('nok')

Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: ne() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (NoneType)
 * (Number other)
      didn't match because some of the arguments have invalid types: (NoneType)
</code></pre>

Deepali: <aside class="quote no-group quote-modified" data-username="Jean-Eric_Campagne" data-post="3" data-topic="67458">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jean-eric_campagne/48/12291_2.png" class="avatar"> Jean-Eric_Campagne:</div>
<blockquote>
<p>a= torch.Size([100, 1]) if a != None: print(a) else: print(‘nok’)</p>
</blockquote>
</aside>
<p>yes should have mentioned that too.<br>
Pls change comparison to <code> if c is  None:</code></p>

Jean-Eric_Campagne: <p>Thanks <a class="mention" href="/u/deepali">@Deepali</a><br>
<code>is None</code> works.</p>
Where are the test cases of pytorch source code present?

tejasvi88: <p>Where can I find test cases for pytorch implementation of various routines like say maxpooling?</p>

Deepali: <p>Check under <a href="https://github.com/pytorch/pytorch/tree/master/test" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/test</a></p>
When setting num_worker>0, print does not print when included in dataloader

Thabang_Lukhetho: <p>I was printing the shape of my tensor inside the <strong>getitem</strong> function, and noticed that it does not print to the console the shape of the tensor when num_workers&gt;0. I’m using windows 10 and pytorch 1.1.0</p>

ptrblck: <p>Do you get the outputs eventually or are these outputs lost?<br>
The output of multiple workers might overlap, but should be printed.</p>
<p>That being said, I’m not really familiar with Windows.</p>
<p>Also, could you update the the latest stable release (<code>1.4.0</code>) in case this was a known issue and was fixed already?</p>

Thabang_Lukhetho: <p>I do get the outputs, I have upgraded to the latest version of Pytorch ( 1.4.0 ) and now it prints, not sure what was the issue. Thanks</p>
Implementing stand alone self attention

vainaijr: <p>can somebody tell me how do I implement stand alone self attention in PyTorch.<br>
Here is my attempt, I found one implementation on github, but I think it is wrong.</p>
<pre><code class="lang-auto">q = torch.randn(1, 1, 3, 3) # for simplicity assume one image, one channel, HXW = 3x3
k = torch.randn(1, 1, 3, 3)
v = torch.randn(1, 1, 3, 3)
</code></pre>
<pre><code class="lang-auto">l_q = nn.Linear(1, 20)
l_k = nn.Linear(1, 20)
l_v = l_k
</code></pre>
<pre><code class="lang-auto">transformed_q = l_q(q.view(1, 1, 3, 3, 1))
transformed_k = l_k(k.view(1, 1, 3, 3, 1))
transformed_v = l_v(v.view(1, 1, 3, 3, 1))
</code></pre>
<pre><code class="lang-auto">transformed_k.shape # each pixel is 20 size embedding
</code></pre>
<p>torch.Size([1, 1, 3, 3, 20])</p>
<pre><code class="lang-auto">t_q = transformed_q.reshape(1, 1, 9, 20)

t_k = transformed_k.reshape(1, 1, 9, 20)

t_v = transformed_v.reshape(1, 1, 9, 20)

emb = nn.Embedding(9, 20) # for each pixel we have 9 neighbouring pixels
softmax = nn.Softmax(dim=-1)
attn_output_weights = softmax(t_q@t_k.transpose(2, 3) + 
                              t_q@emb(torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8])).transpose(0, 1))
attn = attn_output_weights@t_v
</code></pre>
<p>how do I use relative position embedding, that is row, column offset, I think what I am doing is wrong.</p>
Sending many tensors with isend – possible identify which tensor sent?

stsievert: <p>I’m trying to send model parameters from one machine to another using torch.distributed, and it boils down to</p>
<pre><code class="lang-python">import torch.distributed as dist

# Sending from Machine 0
for param in model.parameters():
	dist.isend(param, 1)
	
# Receiving on Machine 1
for param in model.parameters():
	other_param = 0 * param.clone()
	dist.irecv(param, 1)
    assert other_param == param  # if models equal
</code></pre>
<p>Will I receive the parameters I expect from this? In my tests they’re equal but I’m unsure if this will be general in other scenarios.</p>
<p>This boils down to sending a some tag along with the tensor to identify which tensor you are sending. In mpi4py, this is possible with their tag interface:</p>
<p><a href="http://mpi4py.readthedocs.io/en/stable/overview.html?highlight=tag#point-to-point-communications" class="onebox" target="_blank" rel="nofollow noopener">http://mpi4py.readthedocs.io/en/stable/overview.html?highlight=tag#point-to-point-communications</a></p>

ngimel: <p>For MPI backend at least, MPI non-overtaking guarantee should take care of messages being received in the correct order: <a href="http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node54.htm#Node54" rel="nofollow noopener">http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node54.htm#Node54</a>, <a href="http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node61.htm#Node61" rel="nofollow noopener">http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node61.htm#Node61</a>.</p>

bapi: <p>Hi! BTW, if it worked for your purpose, could you do any optimization over this for sending the model parameters across the machines, for example, coalesced communication?</p>
the problem of not finding the download address of the old version of torch

acctub: <p>Hello there</p>
<p>I’m tring to run torch with a old graphics card(compute capability 3).<br>
I have cuda8.0 installed and plan to install torch1.0 today.</p>
<p>But the page for installing torch through pip under windows cannot be accessed.</p>
<p>as follows url:</p>
<h1>CUDA 8.0</h1>
<p>Download and install wheel from <a href="https://download.pytorch.org/whl/cu80/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/cu80/torch_stable.html</a></p>
<p>I opened the page to install cuda9.0 and 10.0, but I couldn’t open 8.0.</p>

albanD: <p>Hi,</p>
<p>This is the old style of source that is built to be used with pip.<br>
You can do “view source” with your browser to see the links in a human readable way if you want to manually download them.</p>

acctub: <p>thank you very much！</p>
Using ImageFolder without subfolders/labels

ywang530: <p>I am working on a image classification project right now. I have a train dataset consists of 102165 png files of different instruments.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f94dc16cda09dc2e9312f97d57e393c2f61902a8.png" alt="微信图片_20200122233631" data-base62-sha1="zzrtNIBnjyNIGKQ6TIBMx6ZOf2M" width="209" height="144"><br>
I have only a train folder which contains all the image files as the above screenshot shows. There are total 10 classes and it all starts with xxxxx_acoustics_xxxxxxxxxxxx. I already wrote a script to extract the names and manages to get all 10 class labels.<br>
My questions is that is there a way to use ImageFolder to load these data ? I know that ImageFolder requires sub-folders, but since I have such amount images, I think move them into different sub-folders would be very time-consuming.</p>
<p>Can anyone points out what can I try ? Thanks in advance!</p>
<p>I</p>

ptrblck: <p>Since you already have a method to extract the labels, I would suggest to write a custom <code>Dataset</code> and load each sample there.<br>
Something like this could be a starter:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, image_paths, transform=None):
        self.image_paths = image_paths
        self.transform = transform
        
    def get_class_label(self, image_name):
        # your method here
        y = ...
        return y
        
    def __getitem__(self, index):
        image_path = self.image_paths[index]
        x = Image.open(image_path)
        y = self.get_class_label(image_path.split('/')[-1])
        if self.transform is not None:
            x = self.transform(x)
        return x, y
    
    def __len__(self):
        return len(self.image_paths)
</code></pre>
Expected both inputs to be on same device

oasjd7: <p><code>binary_op(): expected both inputs to be on same device, but input a is on cuda:1 and input b is on cuda:0</code></p>
<pre><code class="lang-auto">class GaussianNoise(nn.Module):
    def __init__(self, sigma=1.0):
        super().__init__()
        self.sigma = sigma
        self.noise = torch.tensor(0.0, device='cuda:0')

    def forward(self, x):
        sampled_noise = self.noise.repeat(*x.size()).normal_(mean=0, std=self.sigma)
        x = x + sampled_noise
        return x

def network():
    layers = [resnet.layer1, GaussianNoise(1.0)]
   return nn.Sequential(*layers)

I'm using multi-gpus.
How to match gpu devices between x and sampled_noise?
</code></pre>

ptrblck: <p>Inside the <code>forward</code> pass, you could create <code>sampled_noise</code> from the attributes of <code>x</code>:</p>
<pre><code class="lang-python">sampled_noise = torch.empty_like(x),normal_(mean=0, std=self.sigma)
</code></pre>
<p>or push the noise tensor to the device of <code>x</code>:</p>
<pre><code class="lang-python">sampled_noise = ...
sampled_noise = sampled_noise.to(x.device)
</code></pre>
Error after moving model to cpu and moving it back to cuda during training

gebrahimi: <p>Hi,</p>
<p>I am training a model on gpu and during training i move it to cpu to execute some operations on it and move it back to cuda. But right after doing operations on cpu and moving it back to cuda, i can not run the step function to take gradient step.</p>
<p>I get the following error:</p>
<pre><code>self.optimiser.step()
File "/nfs/home/aet4537/.conda/envs/pytorch/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", 
line 51, in wrapper
return wrapped(*args, **kwargs)
File "/nfs/home/aet4537/.conda/envs/pytorch/lib/python3.6/site-packages/torch/optim/sgd.py", line 100, in step
buf.mul_(momentum).add_(1 - dampening, d_p)
RuntimeError: expected device cuda:0 but got device cpu
</code></pre>
<p>I saw similiar topic that suggested using this, but it does not work for me:</p>
<pre><code>print(optimizer.state[list(optimizer.state.keys())[0]])

for p in optimizer.state.keys():
param_state = optimizer.state[p]
buf = param_state["momentum_buffer"]
param_state["momentum_buffer"] = buf.cuda()  # move buf to device

print(optimizer.state[list(optimizer.state.keys())[0]])
</code></pre>
<p>both of my prints show that the tensor that i print is already on cuda. How can i fix this?</p>

ptrblck: <p>Could you post a code snippet to reproduce this issue, please?</p>
Can I get the gradients of same input w.r.t different model parameters with only one backward

bywbilly: <p>Hi,</p>
<p>I am wondering how can I do the things below efficiently.</p>
<p>I have one model M and it takes probably 7GB GPU memory. And I have 20 different sets of parameters for that model M. For the same batch of input, how can I get the gradients w.r.t this batch of input under different model parameters?</p>
Torch.mul RuntimeError: CUDA out of memory

abd: <p>Hi,<br>
v: torch.Size([16, 1, 1024, 32768])<br>
vp :torch.Size([1, 16, 1024, 32768])<br>
c=torch.mul(v,vp) torch.mul RuntimeError: CUDA out of memory<br>
how I can fix this error?</p>
<p>thanks</p>

albanD: <p>Each of these Tensors are quite big.<br>
When you do <code>torch.mul()</code>, it allocates a new Tensor to store the result, but that goes above the memory limit of your GPU. I guess trying to create a Tensor of the size equal to the result will fail: <code>res = torch.empty(16, 16, 1024, 32768)</code>.</p>

abd: <p>Hi <a class="mention" href="/u/alband">@albanD</a><br>
I have the same error when I try:<code>res=torch.mul(v,vp)</code>  such as <code>res = torch.empty(16, 16, 1024, 32768)</code></p>

albanD: <p>So the issue is just that your don’t have enough memory to store that Tensor. I’m afraid you will need to find a way with not representing the full Tensor, or having a smaller one. Or buy a gpu with more memory.</p>
Loss.backward - RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Olivia_Ross: <p>Hi there, I know this is a common error and there are many responses but I’ve read through them and I can’t find a fix. I’m trying to create what really should be a simple siamese network. I’ve successfully made them before. But I just can’t get around this error. I’ve gotten this error before and it usually means I tried to turn something into a numpy array in the middle of the network, but I can’t find where that’s happening here.</p>
<pre><code class="lang-auto">class ShearNet(nn.Module):
    def __init__(self):
        super(ShearNet, self).__init__()
        self.fc =  nn.Linear(100,10)
        nn.Sequential(
                            nn.Linear(100, 10),
                            nn.Linear(10, 10),
                            nn.Linear(10, 2),
                            nn.Linear(2,1)
                                     
                            )
    def forward(self, x):
        output = self.fc(x)
        return output
</code></pre>
<p>Before, I had this return the 2 outputs and it worked just fine. It’s only since asking the forward to find the cosine similarity that it won’t work. I think using functional inside a forward function shouldn’t cause any problems.</p>
<pre><code class="lang-auto">  def __init__(self):
    super(ShearShell, self).__init__()
    self.shear_net = shear_net
  def forward(self, x1, x2):
    output1 = self.shear_net(x1)
    output2 = self.shear_net(x2)
    cosbt = F.cosine_similarity(x1,x2, dim=2)
    output = torch.acos(cosbt)
    return output
</code></pre>
<pre><code class="lang-auto">iteration_number = 0
for epoch in range(10):
    for i, data in enumerate(train_data_loader,0):
        embed0, embed1 = trainin
        output = net(embed0, embed1)
        #loss = lossfunc(output1, output2, trainout)
        loss = criterion(output, trainout)

        #loss.requires_grad = True
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-46-bfbbfb6b84c4&gt; in &lt;module&gt;()
     10         #loss.requires_grad = True
     11         optimizer.zero_grad()
---&gt; 12         loss.backward()
     13         optimizer.step()
     14         if (i %10 == 0 ):

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>What am I missing??? Thank you!</p>

ptrblck: <p><code>F.cosine_similarity</code> uses <code>x1</code> and <code>x2</code>, which might not require gradients.<br>
Based on your code I assume you wanted to use <code>output1</code> and <code>output2</code> instead?</p>
Running time difference on different GPUs

Kevin96: <p>Hi,</p>
<p>I am comparing two algorithms A and B.</p>
<p>When I run these algorithms on my own 2080 Ti GPU, algorithm A is slightly faster than B. However, when I submit them to P100 GPU on cluster, algorithm A is about 2x slower than B.</p>
<p>I am pretty sure that I am using the correct time measure method on GPU. Essentially I first sent two cuda timing event and then force syncronization to record the time.</p>
<p>The running time of algorithm B does not change much on both GPUs. However, algorithm A runs much slower on P100. Is it normal for an algorithm to have this drastical difference on difference devices?</p>

ptrblck: <p>Are you using exactly the same CUDA, cudnn, nccl etc. libraries?<br>
Differences in these libraries might create a large difference.</p>

Kevin96: <p>I think I am using cudnn. Is there any way to check that? I thought cudnn is the default option.</p>

Kevin96: <p>okay, on my local machine with 2080 Ti, the cuda version is 10.0 and cudnn is 7603. On the server with P100, the cuda version 10.1, and cudnn is 7603.</p>
<p>They are not exactly the same. But the server actually use a lastest cuda version so it is supposed to be faster? It is just weird.</p>

ptrblck: <p>Based on the versions, it looks like you are using the PyTorch binaries.<br>
If that’s the case, your local CUDA installation won’t be used, but the one shipped with the binaries.</p>
<p>Could you install the same binaries with matching versions and, if possible, post your code for profiling so that we can have a look, please?</p>
Error (of sorts): why does this happen?

sigma_x: <p><code>IncompatibleKeys(missing_keys=[], unexpected_keys=[]),</code></p>
<p>Yes, I already know that this means weights were loaded correctly , my question is: why return this ‘Error’ at all, if there were no errors?</p>

albanD: <p>This is not an error. This is just a tuple containing the keys that were either missing or unexpected.<br>
Is it detected as being an Error by something?</p>
How can I change keras function into pytorch

111227: <p>I want to change this Keras function into PyTorch. Should I rewrite a new function or there is a function that Pytorch had already exist? Thanks</p>
<p>batch=input_shape.dim_size(0)<br>
i_h=input_shape.dim_size(1)<br>
i_w=input_shape.dim_size(2)<br>
i_ch=input_shape.dim_size(3)</p>
<p>k_h=filter_shape.dim_size(0)<br>
k_w=filter_shape.dim_size(1)<br>
o_ch=filter_shape.dim_size(3)</p>

ptrblck: <p>Which function would you like to convert from Keras to PyTorch? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

111227: <p>I want to find a function that has multiple inputs. For examples: my code has 4 inputs(batch, input height, input weight, and input channel)</p>
<p>I don’t want to redefine a lot of function, and trying to make my code easier to read.</p>
<p>Thanks for reply!</p>

ptrblck: <p>If you want to grab the dimension shapes, you could use:</p>
<pre><code class="lang-python">size = my_tensor.size()
# or
N, C, H, W = my_tensor.size()
</code></pre>
<p><code>.shape</code> should also work.</p>
Transfer layers from one model to another

JhK: <p>Hello. I’m new to python and pytorch and I’m trying to transfer layers and weights of pre-trained model to another model for regression.</p>
<p>These two models have same structure and I want to transfer layers and weights of pre-trained model except last layer.</p>
<p>Please let me know how to do that.</p>
<p>here is my pre-trained model</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self, n_feature, n_hidden1, n_hidden2, n_hidden3, n_hidden4, n_hidden5, n_output):<br>
super(Net, self).<strong>init</strong>()<br>
self.hidden1 = torch.nn.Linear(n_feature, n_hidden1)<br>
self.hidden2 = torch.nn.Linear(n_hidden1, n_hidden2)<br>
self.hidden3 = torch.nn.Linear(n_hidden2, n_hidden3)<br>
self.hidden4 = torch.nn.Linear(n_hidden3, n_hidden4)<br>
self.hidden5 = torch.nn.Linear(n_hidden4, n_hidden5)<br>
self.predict = torch.nn.Linear(n_hidden5, n_output)</p>
<pre><code>def forward(self, x):
    x = torch.relu(self.hidden1(x))      
    x = torch.relu(self.hidden2(x))
    x = torch.relu(self.hidden3(x))
    x = torch.relu(self.hidden4(x))
    x = torch.relu(self.hidden5(x))
    x = self.predict(x)             
    return x
</code></pre>
<p>net = Net(n_feature=4, n_hidden1=100, n_hidden2=80, n_hidden3=50, n_hidden4 = 35, n_hidden5 = 20, n_output=1)<br>
print(net)  # net architecture<br>
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)<br>
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss</p>
<p>for t in range(20000):</p>
<pre><code>prediction = net(xs_train)     # input x and predict based on x

loss = loss_func(prediction, ys_train)     

optimizer.zero_grad()   # clear gradients for next train
loss.backward()         # backpropagation, compute gradients
optimizer.step()        # apply gradients</code></pre>

ptrblck: <p>I think the easiest approach would be to load the <code>state_dict</code> completely and reset the last layer’s <code>state_dict</code>:</p>
<pre><code class="lang-python">modelA = models.resnet18(pretrained=True)
modelB = models.resnet18()

fc_sd = copy.deepcopy(modelB.fc.state_dict())
modelB.load_state_dict(modelA.state_dict())
modelB.fc.load_state_dict(fc_sd)

for (nameA, paramA), (nameB, paramB) in zip(modelA.named_parameters(), modelB.named_parameters()):
    if (paramA == paramB).all():
        print('{} matches {}'.format(nameA, nameB))
    else:
        print('{} does not match {}'.format(nameA, nameB))
</code></pre>

JhK: <p>Thank you for your answer.</p>
<p>but, I don’t use models in torchvision like resnet10 as a pre-traind model. I want to use my own model as a pre-trained model and transfer its layers and weights  except last layer to other model that has same structure. Also I want to freeze all the transferred weights in new model not to be optimized.</p>
<p>Is there any other ways to do this???</p>

ptrblck: <p>Just load your model instead of <code>models.resnet18</code>.<br>
This approach should work likewise for all models.<br>
Of course you would have to load the <code>state_dict</code> of your custom model first, as you most likely don’t have the <code>pretrained</code> argument.</p>