akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Running SGD optimizer with optimizer.zero_grad()

111278: <p>Hello, I am performing image classification using SGD optimizer. In the meantime, I accidentally trained without using optimizer.grad_zero (), but I could see better results than before. Can you guess for what reason?</p>

braindotai: <p><code>optimizer.grad_zero()</code> resets gradients of parameters to zero. By not calling it, this means when you call <code>.backward()</code> to your loss, then this will add the current gradients (wrt current loss) to the gradients that were already assigned to the parameters (wrt previous loss).</p>
<p>So here basically you’re updating your model parameters with large gradients (since gradients are adding as we move on to next batch).</p>
<p>Now in case of SGD, which is a simple algorithm without any complications, this might work as this algorithm is already pretty slow, so in this case not calling <code>optimizer.zero_grad()</code> is providing you large gradients, hence SGD might work better in this case. Also my guess is your model is also quite simple (unlike resnet20).</p>
<p>But if you’re model is large and you’re using other gradient descent varients like adam, nadam, rmsprop… etc, then this approach will give nan values.</p>

111278: <p>Thank you very much for your reply. I understood it well.</p>
<p>For visual viewing and understanding, I am currently plotting a gradient histogram with a tensor board. However, I don’t understand because I don’t have much prior knowledge.</p>
<p>Is there any way to plot the graph quantitatively for gradients? Thank you.</p>

braindotai: <p>you can get parameters gradients using param.grad,</p>
<pre><code class="lang-python">gradients_norms = []
for param in model.parameters():
    gradien_norms.append(param.grad.abs().sum())
</code></pre>
<p>Then you can plot these gradient norms.<br>
In case of calling <code>optimizer.zero_grad()</code> after <code>optimizer.step()</code> the graph should decrease, since we are converging the loss functions.<br>
But in case of calling <code>optimizer.step()</code> <strong>only</strong>, the graph should not decrease, in fact it should increase since we were adding the gradients for each update.</p>

111278: <p>Thank you for your help.<br>
It helped me a lot,</p>
Create a dataset from different sources

hect1995: <p>I have several CSV with temporal data files and from each CSV files I want to get minibatches (i.e. size 200) of sequential data. How can I incorporate all this CSV files in a PyTorch dataset while not mixing information from one CSV into another?</p>

ptrblck: <p>You could store all file paths to the .csv files in your <code>Dataset</code>'s <code>__init__</code> method and load each file separately using a module operation in <code>__getitem__</code>.<br>
E.g. if file1 contains 100 sliding windows, you could check, if <code>100 &lt;= index &lt; 200</code> and load the window from the second file.</p>
<p>Have a look at <a href="https://discuss.pytorch.org/t/dataloader-for-a-lstm-model-with-a-sliding-window/22235/2">this small example</a> for a sliding window approach.</p>
<p><a href="https://discuss.pytorch.org/t/create-a-dataset-from-sequential-data-pytorch/75539">Related post</a></p>
Error while running the pre-trained waveglow model on specific GPU device

harshbafna: <p>Hello,</p>
<p>I am trying to execute the pre-trained waveglow example given here : <a href="https://pytorch.org/hub/nvidia_deeplearningexamples_waveglow/" rel="nofollow noopener">https://pytorch.org/hub/nvidia_deeplearningexamples_waveglow/</a></p>
<p>The example works fine if executed as provided. However, it fails if I change the device from “cuda” to “cuda:1” with following error :</p>
<pre><code class="lang-auto">File "waveglow_example.py", line 76, in inference
    _, mel, _, _ = self.tacotron2_model.infer(data)
File "/home/ubuntu/.cache/torch/hub/nvidia_DeepLearningExamples_torchhub/PyTorch/SpeechSynthesis/Tacotron2/tacotron2/model.py", line 651, in infer
    encoder_outputs)
 File "/home/ubuntu/.cache/torch/hub/nvidia_DeepLearningExamples_torchhub/PyTorch/SpeechSynthesis/Tacotron2/tacotron2/model.py", line 546, in inference
    not_finished = not_finished*dec
RuntimeError: expected device cuda:0 but got device cuda:1
</code></pre>
<p><strong>Sample code :</strong></p>
<pre><code class="lang-auto">import torch
import numpy as np
from scipy.io.wavfile import write

waveglow = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_waveglow')

waveglow = waveglow.remove_weightnorm(waveglow)
waveglow = waveglow.to('cuda:1')
waveglow.eval()

tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')
tacotron2 = tacotron2.to('cuda:1')
tacotron2.eval()

text = "hello world, I missed you"

sequence = np.array(tacotron2.text_to_sequence(text, ['english_cleaners']))[None, :]
sequence = torch.from_numpy(sequence).to(device='cuda:1', dtype=torch.int64)

# run the models
_, mel, _, _ = tacotron2.infer(sequence)
audio = waveglow.infer(mel)
audio_numpy = audio[0].data.cpu().numpy()
rate = 22050

write("audio.wav", rate, audio_numpy)
</code></pre>
<p>Is there something I am missing here?</p>

ptrblck: <p>Could you please create an issue <a href="https://github.com/NVIDIA/DeepLearningExamples/issues">here</a>?<br>
As a workaround you could select GPU1 in your script via <code>CUDA_VISIBLE_DEVICES="1" python scripy.py</code> and leave the <code>.cuda()</code> and <code>.to()</code> calls as they are.</p>
How do I define the category of this task

zhang_chen: <p>I have a task. I want to finish a task that automatically extracts a key frame from an ultrasound video sequence of the abdomen.<br>
The feature in the key frame is obvious where the bladder on that frame has a distinct balloon on it because of the injection of water. The video is a real-time image of the ultrasound probe moving around all the time.<br>
I’ve been working on static medical images. I don’t know how to define this task.<br>
I have several ideas:</p>
<ol>
<li>Frame classification. Label a video into two categories: 0,1. The target frame is marked as one, and the rest is marked as zero. The possible problem is that the sample is extremely unbalanced.<br>
2.Two stage strategy. At first, I design a Deep network to detect key frames, which can also be seen as a selection of the proposal frames. Then, a Judge network can be added as follows to classify whether it is the target key frame.<br>
3.End-to-end framework. The input is a video sequence and the output is the key frame. (MANY-TO-ONE). But it seems that this problem is more common in natural language processing which involves RNN.</li>
</ol>
<p>In summary, I reviewed the video process based on DL. I am confused about how to define my task. Who can help me to analyze?</p>
<p>PS: The network should be designed to input real-time video images, so may the essence of the image processing problem??</p>
Tensorboard error

SHIVAM_PANDEY: <p><a href="https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#adding-a-projector-to-tensorboard" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#adding-a-projector-to-tensorboard</a></p>
<p>Here, when I run this code, I have the following error</p>
<p>e:\deep learning\environment\dl\lib\site-packages\torch\utils\tensorboard\writer.py in add_embedding(self, mat, metadata, label_img, global_step, tag, metadata_header)<br>
779         save_path = os.path.join(self._get_file_writer().get_logdir(), subdir)<br>
780<br>
–&gt; 781         fs = tf.io.gfile.get_filesystem(save_path)<br>
782         if fs.exists(save_path):<br>
783             if fs.isdir(save_path):</p>
<p>AttributeError: module ‘tensorflow_core._api.v2.io.gfile’ has no attribute ‘get_filesystem’</p>
About Prediction in Binary Classification

albertoazzari: <p>Hi guys, I can’t understand how to get the y prediction from the training model without going to testing, this would help me to get a classification_report of the model before testing it. Hope you can answer, thank you very much.</p>
<pre><code class="lang-auto">class BinaryClassification(nn.Module):
    def __init__(self):
        super(BinaryClassification, self).__init__()
        self.layer_1 = nn.Linear(feature, node)
        self.layer_2 = nn.Linear(node, node)
        self.layer_out = nn.Linear(node, 1)

        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.1)
        self.batchnorm1 = nn.BatchNorm1d(node)
        self.batchnorm2 = nn.BatchNorm1d(node)

    def forward(self, inputs):
        x = self.relu(self.layer_1(inputs))
        x = self.batchnorm1(x)
        x = self.relu(self.layer_2(x))
        x = self.batchnorm2(x)
        x = self.dropout(x)
        x = self.layer_out(x)

        return x
</code></pre>
<p>After this training</p>
<pre><code class="lang-auto">model.train()
flag = True
for e in range(1, EPOCHS + 1):
    epoch_loss = 0
    epoch_acc = 0
    if flag:
        for X_batch, y_batch in train_loader:
            X_batch, y_batch = X_batch.to(device), y_batch.to(device)
            optimizer.zero_grad()
            y_pred = model(X_batch)
            loss = criterion(y_pred, y_batch.unsqueeze(1))
            acc = binary_acc(y_pred, y_batch.unsqueeze(1))
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
            epoch_acc += acc.item()
            if loss.item() &lt; 0.00001:
                flag = False
</code></pre>
<p>After this training I need a classification_report, so I need a y_pred list to compare with true y.<br>
Thanks</p>

ptrblck: <p>Since your model outputs logits, you could use a threshold e.g. at <code>0.0</code> to create your predictions as:</p>
<pre><code class="lang-python">preds = y_pred &gt; 0.0
</code></pre>
<p>You could also apply <code>torch.sigmoid</code> and use a threshold in the range <code>[0, 1]</code> instead.</p>
Diag a demension of a matrix

Nanachi: <p>Hi all,<br>
I have a matrix A(A.shape=torch.Size([50, 400]))and a batch of vector B(B.shape=torch.Size([8,400])),where 8 is the batch size.<br>
The first thing I want to do is make the vector part of B (a.k.a [400]) a diagonal matrix, then B should become a matrix in shape [8, 400, 400]<br>
The I want to perform patch multiplication, that is calculate [50, 400] multiply [8,400,400] and get the result in size [8,50,400].</p>
<p>I want to know how to transform some dimension in a matrix like B into diagonal matrix(like from[8,400] to [8,400,400])<br>
I want to do these things in a neural work so if the network can still be backpropagated after I converted second dimension of B into a diagonal matrix? Or should I implement the same operation on A and B with another way that don’t need diag operation?</p>

Nanachi: <p>Question closed, it is not neccessary to diag a vector to a diagonal matrix, you can achieve element-wise multiply simply by using mul()<br>
and torch will broadcast for you.<br>
e.g.<br>
a=[8, 1, 400]<br>
b=[50, 400]<br>
just use a.mul(b)<br>
ans is [8, 50, 400]</p>
Is there a C api for aten?

howardhl: <p>I would like to use libtorch as basic operators in other programming languages. And not all the c++ features are fully supported.<br>
I would like to know if there is a c head file or c api/abi for libtorch, which I could use as a bridge with another language?</p>

albanD: <p>Hi,</p>
<p>I’m afraid the whole point of aten compared to TH/THC was to move completely to c++ so I don’t think there is any C API for it.</p>

howardhl: <p>As noted in <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/README.md" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/README.md</a>," Torch has  <em>always been</em> , and  <em>will always be</em>  a library that provides a C ABI interface."<br>
I just wonder will can I find a header file which contains all c interfaces.</p>

albanD: <p>I am afraid this comment is not really up to date <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>This comment refers to the TH/THC libraries only. But we have quite heavily moved out from these libraries and use them less and less now. So they don’t provide much functionality anymore.</p>

dfalbel: <p>FWIW we are building/using lantern to wrap all torch api into a C compatible. It’s not stable at all, but contributions are welcome!</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/mlverse/lantern" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/55406849?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/mlverse/lantern" target="_blank" rel="nofollow noopener">mlverse/lantern</a></h3>

<p>Contribute to mlverse/lantern development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


howardhl: <p>Thanks a lot~ I would like to have a look~</p>
Out of filter tensor shape

vi_ascogi: <p>Is there any function to calculate the out shape a filter would give to an input:</p>
<p>As an examples:</p>
<p>nn.MaxPool2d(3, stride=2)<strong>.outshape()</strong></p>
<p>Thank you,</p>

ptrblck: <p>The output shape of a polling layer depends on the input shape.<br>
You could pass a random tensor to the module and <code>print(out.shape)</code> to get the shape for this setup.</p>
Different masks applied to conv filters at different pixels

Lam123: <p>I am trying to do convolutions on an image.</p>
<p>However, at different pixels of the image, I want to perform convolution operation at different selected neighbors. So is there a way to use different masks at different pixels?</p>
<p>I know how to apply the same mask for all pixels, which means I can perform convolution operation at same selected neighbors.</p>
<p>One possible solution I have: If there is a way to know where the filter is right now, then I can apply different masks.</p>
<p>Thank you very much!</p>

ptrblck: <p>I think the easiest method would be to apply a manual convolution by unfolding the input and apply the filter to each patch manually.<br>
<a href="https://discuss.pytorch.org/t/patch-making-does-pytorch-have-anything-to-offer/33850/11">Here</a> is a small code example.</p>

Lam123: <p>Thank you very much! But I still don’t quite get it. Let me clarify my question a bit.</p>
<p>Suppose I have a data of size (128,28,28), where 128 is batch size, and 28*28 is size of each sample. When applying convolution, I want use different masked convolution filters at different pixel. Namely, at pixel (1,1), maybe I want to mask out all pixels on its right, but at pixel (2,2),  want to mask out all pixels on its left. So do you have any idea here?</p>

ptrblck: <p>Once you’ve unfolded the input and create the patches, you can create the filters in whatever way you have and apply them to the corresponding patch.<br>
In the first part of my code you’ll get all <code>patches</code>, which are stacked in one dimension.<br>
Thereafter you could either loop over the patches and apply the filters or try a batched approach.</p>
Feeding a video to Conv3D

Subhankar_Ghosh: <p>I have a conv3d layer as:</p>
<p><code>m = nn.Conv3d(16, 33, 3, stride=2)</code></p>
<p>I know how to feed a tensor like:</p>
<pre><code class="lang-auto">input = torch.randn(20, 16, 10, 50, 100)
output = m(input)
</code></pre>
<p>But how do I feed a video file to this layer?</p>
<p>Thanks!</p>
Pretrained Resnet

oasjd7: <p>I’m trying to use pretrained resnet50.<br>
But</p>
<pre><code class="lang-auto">net = resnet50.resnet50(pretrained=True)
</code></pre>
<p>occurs<br>
<code>urllib.error.URLError: &lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)&gt;</code></p>
<p>So I downloaded resnet model from here (‘<a href="https://download.pytorch.org/models/resnet50-19c8e357.pth" rel="nofollow noopener">https://download.pytorch.org/models/resnet50-19c8e357.pth</a>’)<br>
and loaded like this.<br>
Is it correct way to use pretrained resnet50 model?</p>
<pre><code class="lang-auto">net = resnet50.resnet50(pretrained=False)
net.load_state_dict(torch.load('download_resnet.pth'))
</code></pre>

ptrblck: <p>Could you retry the download again, please?<br>
I just tried to download the pretrained model using the first (and right) approach and it worked fine for me.</p>

oasjd7: <p>Same problem.<br>
Maybe pytorch version is old. I’m using 1.1.0 v.</p>
<p>Anyway, what I got from above link is the pre-trained model?</p>

ptrblck: <p>That should be the right <code>state_dict</code>, yes.</p>
Are model state_dict and it's optimizer's state_dict enough for continuing training?

braindotai: <p>My query is the same as the title. I think yes, but to be 200% sure I had to ask.</p>
<p>Thanks.</p>

fs4ss1: <p>Yes, they are enough.</p>
<p>You can read <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#save" rel="nofollow noopener">here</a>.</p>
How to calculate the loss in the mask area

Wenjie_Qiang: <p>I tried to calculate the loss after adding a mask to the output, but the problem is that MSE loss does not drop during the iteration process, the following is a code snippet of my program.</p>
<pre><code class="lang-auto">with torch.autograd.set_detect_anomaly(True):
            prediction = net(x, train_pers)
           
            mask_info = scio.loadmat(mask_path)
            pos_x = mask_info['maskVerticesXCoordinates']
            pos_y = mask_info['maskVerticesYCoordinates']
            pos = np.concatenate((pos_x, pos_y), 1).astype(np.int32)  # 8 * 2
            mask_tg = np.zeros((576, 720))
            cv2.fillPoly(mask_tg, [pos], 1)  # fill 1 in the mask
            mask_tg = mask_tg[np.newaxis, np.newaxis, :, :]
            mask = torch.from_numpy(mask_tg)
            mask = functional.interpolate(mask, scale_factor=0.125, mode='nearest')

            prediction = torch.mul(prediction.double(), mask2.to(setting.device))
            prediction = Variable(prediction,requires_grad=True)

            loss = criterion(prediction, y)

            optimizer.zero_grad()
            loss.backward()
</code></pre>
Squeeze, unsqueeze explanation

sai_m: <p>When I went through documentation for squeeze, unsqueeze methods, I felt like I got this clearly and after few days when I start using squeeze and unsqueeze, I feel that I am not totally clear on these two methods.</p>
<p>can anyone shed some light on these two methods please!</p>

vainaijr: <p>unsqueeze to add dimension of size 1,<br>
squeeze to remove dimension of size 1.</p>
<p>for example,</p>
<pre><code class="lang-auto">import torch
x = torch.randn(10); x
</code></pre>
<pre><code class="lang-auto">tensor([-0.9048, -0.3753, -0.3185, -1.4873, -0.7718,  0.6153,  0.1261,  0.5256,
        -1.9678, -0.2045])
</code></pre>
<pre><code class="lang-auto">x.shape
</code></pre>
<pre><code class="lang-auto">torch.Size([10])
</code></pre>
<pre><code class="lang-auto">y = x.unsqueeze(dim=0)
y.shape
</code></pre>
<pre><code class="lang-auto">torch.Size([1, 10])
</code></pre>
<pre><code class="lang-auto">y = x.unsqueeze(dim=-1)
y.shape
</code></pre>
<pre><code class="lang-auto">torch.Size([10, 1])
</code></pre>
<p>one more thing,</p>
<pre><code class="lang-auto">y = x[None] # or y = x[None, :], same as x.unsqueeze(dim=0)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">y = x[:, None] # same as x.unsqueeze(dim=-1)
</code></pre>
<p>for squeeze, if</p>
<pre><code class="lang-auto">y.shape
</code></pre>
<p>is</p>
<pre><code class="lang-auto">torch.Size([10, 1])
</code></pre>
<p>then,</p>
<pre><code class="lang-auto">y.squeeze_(dim=1)
y.shape
</code></pre>
<p>give</p>
<pre><code class="lang-auto">torch.Size([10])
</code></pre>
Add real time recognition

hunar: <p>how to add real time (from webcam) recognition to facenet-pytorch library ?<br>
i working on <a href="https://github.com/timesler/facenet-pytorch" rel="nofollow noopener">facenet</a> ?</p>

fs4ss1: <p>You need to do this (using OpenCV):</p>
<pre><code class="lang-auto">import cv2 as cv

cap = cv.VideoCapture(0)
while True:
    ret, frame = cap.read()
    frame = cv.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
    image = predict_draw_bounding_box(frame)
    cv.imshow('Output', image)
    c = cv.waitKey(1)
    if c == 27:
        break
cap.release()
cv.destroyAllWindows()
</code></pre>
<p>You need to create the <strong>predict_draw_bounding_box</strong> function to receive the frame and return the frame processed.</p>

hunar: <p>can guide me to create predict function please , i should complete my final year projects in these days please</p>
Perfect CrossEntropyLoss Prediction doesn't have 0 loss

tasubo: <p>Consider this example:</p>
<pre><code class="lang-auto">        criterion = nn.CrossEntropyLoss()

        y = torch.LongTensor([1, 1, 0, 0])
        x = torch.FloatTensor([[0., 1., 0.], [0., 1., 0.], [1., 0., 0.], [1., 0., 0.]])

        loss_reference = criterion(x, y)

        print(loss_reference)
</code></pre>
<p>It gives a loss of 0.55 while I would expect it to be 0.</p>
<p>Why is that?</p>

tasubo: <p>Ok… I’ve realized that it is due to softmax <a href="https://stackoverflow.com/questions/49390842/cross-entropy-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/49390842/cross-entropy-in-pytorch</a></p>
How to set gradient manually during the backward process

waterhorse1: <p>I want to implement a pytorch model which can set the gradient manually during backpropogation. So here is an example.</p>
<blockquote>
<p>a = torch.randn([2,2], requires_grad=True)<br>
b = 2 * a<br>
c= func(b)<br>
d = 2 * c</p>
</blockquote>
<p>Here func is a differentiable funciton and has explicit form. However, direct backpropogation will lead to huge computation demand. Here is an algorithm which can directly compute d_c/d_b without using backpropogation. Is there any way I can readily use to achieve the process: first backpropogation from d-&gt;c,  manually set gradients from c-&gt;b, last propogation from b-&gt;a and finally get gradient from d-&gt;a?</p>
<p>I think if I split up the whole backpropogation process it can be completed but it’s still too complicated for me. I also try to utilize something like backward_hook but I cannot feed any input into that function. Grateful if you can help me with this problem.</p>

albanD: <p>Hi,</p>
<p>You can find on this page on the doc <a href="https://pytorch.org/docs/stable/notes/extending.html">https://pytorch.org/docs/stable/notes/extending.html</a> all the information on how to define a custom elementary function for the autograd for which you define both the forward and the backward.</p>
Freezing layers in one head of encoder-decoder network (easy way?)

bschroeder: <p>I am trying to see if I can use an easy way to freeze the layers in my network. I have an encoder-decoder network with 2 decoder heads (subnetworks). I want to freeze the layers in one of the decoder heads of my network.  By virtue of how the chain rule works in backprop, can I just disable the loss of the head I want to freeze or force it to be 0? If this is the case, will the backprop contribution of this subnetwork be 0? My understanding of backprop is that the gradients of each subnetwork will be added at the bottleneck layer.</p>
<p>Otherwise, do I need to set param.requires_grad = False to each layer in the subnet I want to freeze?</p>
Pretrained BERT package for Windows in Anaconda

Raphy-N: <p>Hi. I’m new to <strong>NLP</strong> and Deep Learning, and struggling a lot with PyTorch. I’m trying to train <strong>BERT</strong> on a data set for a sequence tagging task, similar to <strong>NER</strong> but with only 2 tags for each word token. I’m working in Anaconda with a Windows 10 OS. I was going to install HuggingFace’s pytorch-pretrained-bert package through conda as in the following page:</p>
<p><a href="https://anaconda.org/conda-forge/pytorch-pretrained-bert" rel="nofollow noopener">pytorch-pretrained-bert (by HuggingFace)</a></p>
<p>But I see that it is only for Linux OS. Is there any way I can work on a pretrained BERT in PyTorch while using Windows 10?</p>

peterjc123: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pypi.org/static/images/favicon.6a76275d.ico" class="site-icon" width="16" height="16">
      <a href="https://pypi.org/project/transformers/#description" target="_blank" rel="nofollow noopener">PyPI</a>
  </header>
  <article class="onebox-body">
    <img src="https://pypi.org/static/images/twitter.90915068.jpg" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://pypi.org/project/transformers/#description" target="_blank" rel="nofollow noopener">transformers</a></h3>

<p>State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Selecing using list indices in c++

Kaunil_Dhruv: <p>Hi team,<br>
I am looking for your help in replicating the list based indexing functionality of numpy in ::Tensors.</p>
<p>For example:</p>
<pre><code class="lang-auto">a = np.zeros((100, 100))
xs = [1, 2, 5, 6]
ys = [4, 5, 7, 3]

res = a[xs, ys].
</code></pre>
<p>Replicating it in torch c++:</p>
<pre><code class="lang-auto">auto a = tensor::zeros({100, 100});

int xs[] = {1, 2, 3};
int ys[] = {1, 2, 3};

auto xs = torch::from_blob(xs, {3}, torch::kInt);
auto ys = torch::from_blob(ys, {3}, torch::kInt);

/* doing the following results in a  memory access error */
auto result = a[xs][ys];
</code></pre>
<p>My apologies if it is a cause of something silly.</p>
<p>Best,<br>
Kaunil D.</p>

mathematics: <p>oh that c++ codes,<br>
Why you are using c++ instead of python? Any reason for using this.<br>
Thanks for advice.</p>

Kaunil_Dhruv: <p>Yeah: The routines preceding this are built to utilize CUDA for massaging high dimensional satellite imagery/ point clouds.</p>

mathematics: <p>oh thank you .Is it not possible to make in python pytorch <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

Kaunil_Dhruv: <p>saving time in context switches. manageblity. 1 Cmake does it all. ITS FUN.</p>

mathematics: <p>Oh thanks for explaining my doubts <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>

googlebot: <p>try<br>
<code>auto result = torch::index(a, { xs,ys });</code><br>
or creating TensorList for second argument.</p>

yf225: <p>Starting from currently nightly build and 1.5 release, the best way to do it in C++ is:<br>
Python:</p>
<pre><code class="lang-python">a = np.zeros((100, 100))
xs = [1, 2, 5, 6]
ys = [4, 5, 7, 3]

res = a[xs, ys]
</code></pre>
<p>C++:</p>
<pre><code class="lang-cpp">auto a = torch::zeros({100, 100});
auto xs = torch::tensor({1, 2, 5, 6});
auto ys = torch::tensor({4, 5, 7, 3});

auto res = a.index({xs, ys});
</code></pre>

Kaunil_Dhruv: <p>It works! Thank you for the quick turnaround <a class="mention" href="/u/yf225">@yf225</a></p>
How to reshape tensor in a right way?

odats: <p>How to reshape <code>torch.Size([2, 3, 4, 5])</code> to <code>torch.Size([2, 5, 3, 4])</code> and then get back to <code>torch.Size([2, 3, 4, 5])</code>?<br>
In my case <code>torch.Size([batch_size, x_dim, y_dim, 3_RGB])</code>. I need convert to<code> torch.Size([batch_size, 3_RGB, x_dim, y_dim])</code> and use it with CNN layer and then get back to original size: <code>torch.Size([batch_size, x_dim, y_dim, 3_RGB])</code></p>

b02202050: <p>For example,</p>
<pre><code class="lang-python">x = torch.rand(2,3,4,5) # now x.shape is torch.Size([2, 3, 4, 5])
y = x.permute(0,3,1,2) # now y.shape is torch.Size([2, 5, 3, 4])
    # which means: take the x's dimension 0,3,1,2 to be y's dimension 0,1,2,3
    # i.e. x's dimension 0 is 2; dimension 1 is 3; dimension 2 is 4; dimension 3 is 5.
    # so x's dimension 0,3,1,2 would be 2,5,3,4.
z = y.permute(0,2,3,1) # now z.shape is torch.Size([2, 4, 3, 5])
</code></pre>
Transfer learning using VGGFace2 Model in Pytorch

Sushmita_Upadhyay: <p>I want to extract features from last layer of VGGFace 2 model (Senet50_256). The last layer is a convolution layer of shape -1,256,1,1. How do I flatten this layer for transfer learning? I got the model from the VGGFace 2 Github <a href="https://github.com/ox-vgg/vgg_face2" rel="nofollow noopener">https://github.com/ox-vgg/vgg_face2</a> along with the weights.<br>
Also I am freezing the weights of the model and I only want to optimize the layer which I will add.<br>
Also am I using correct model and weights?</p>

charan_Vjy: <p>In your auxillary model that you add to the VGG19 pretrained model. You should add these layers at the start.</p>
<pre><code class="lang-auto">import torch
batch_size = 16
input = torch.randn(batch_size,256,1,1)
input = input.squeeze(3).squeeze(2).view(-1)
</code></pre>

Sushmita_Upadhyay: <p>Hi,<br>
Why are you using torch.randn? I am relatively new to pytorch.<br>
Also these are the last layers of the pretrained model<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b3a65857417af18cc8cdcaeccde81d7c78728cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b3a65857417af18cc8cdcaeccde81d7c78728cc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b3a65857417af18cc8cdcaeccde81d7c78728cc.png" alt="image" data-base62-sha1="hA7QgeQ1XEvKqJ4L4D25kzbAtM8" width="690" height="292" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">699×296 7.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and the layer that I want to add is <strong>senetmodel.fc=nn.Linear(256, 128)</strong></p>
<p>But I think before adding this layer I should flatten the feat_extract(Conv2d-238) layer which I am unable to do.</p>

charan_Vjy: <p>torch.randn was used just to show a dummy input. Ignore it in your code.<br>
<code>input = input.squeeze(3).squeeze(2)</code> --&gt; Use this alone as the starting line for your auxillary model.</p>

Sushmita_Upadhyay: <p>My code is working now. Thank you</p>
What is running loss after an epoch in PyTorch

jitesh: <p>I had a look at <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">this</a> tutorial in the PyTorch docs for understanding Transfer Learning. There was one line that I failed to understand.</p>
<p>After the loss is calculated using <code>loss = criterion(outputs, labels)</code>,  the running loss is calculated using <code>running_loss += loss.item() * inputs.size(0)</code> and finally, the epoch loss is calculated using <code>running_loss / dataset_sizes[phase]</code>.</p>
<p>Isn’t <code>loss.item()</code> supposed to be for an entire mini-batch (please correct me if I am wrong). i.e, if the <code>batch_size</code> is 4, <code>loss.item()</code> would give the loss for the entire set of 4 images. If this is true, why is <code>loss.item()</code> being multiplied  with <code>inputs.size(0)</code> while calculating <code>running_loss</code>? Isn’t this step like an extra multiplication in this case?</p>
<p>Any help would be appreciated. Thanks!</p>
Error with hook functions

sinang: <p>Hello everyone, I have a question about hook functions:</p>
<p>After executing the following code:</p>
<pre><code class="lang-auto">def relu_backward_deconv_hook(module, grad_input, grad_output): 
    return nn.functional.relu(grad_output[0])

def equip_model_deconv(model): 
    for m in model.modules():
        if isinstance(m, nn.ReLU): 
            m.register_backward_hook(relu_backward_deconv_hook)


def grad_view(model, image_name):
    to_tensor = transforms.ToTensor()
    img = to_tensor(PIL.Image.open(image_name))
    img = img[:,0:224,0:224]
    img = 0.5 + 0.5 * (img - img.mean()) / img.std()
    
    model.to(device='cuda:0')
    img = img.to(device='cuda:0')
    
    input1 = img.view(1, img.size(0), img.size(1), img.size(2)).requires_grad_() 
    print(tuple(input1.shape))
    output = model(input1)
    result = torch.autograd.grad(output.max(), input1)
    
    result = result / result.max() + 0.5
    
    return result

model = models.vgg16(pretrained = True) 
model.eval()
model = model.features 

equip_model_deconv(model)
result = grad_view(model, "/home/ubuntu/folder/mug.jpg") 
utils.save_image(result, 'mug-vgg16-deconv.png')
</code></pre>
<p>I get this error: TypeError: expected tuple, but hook returned ‘Tensor’</p>
<p>Could you figure out the problem that I’m having?</p>
<p>Thank you in advance!</p>
Shared Memory with mpi-backend

Usama-Zafar: <p>Is it possible to create a shared memory / tensors among openmpi spawned process group? I know this can be done with the processes created by torch.multiprocessing package. But in my case I am unable to make this package work with openmpi processes.</p>

seliad: <p>I’d like to know the answer for this too.</p>

mrshenli: <p>One solution might be</p>
<ol>
<li>create a shared memory buffer using MPI</li>
<li>create a numpy <code>ndarray</code> from that shared memory (<a href="https://stackoverflow.com/questions/32485122/shared-memory-in-mpi4py" rel="nofollow noopener">example</a>)</li>
<li>create a PyTorch tensor from that numpy <code>ndarray</code> (<a href="https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#converting-numpy-array-to-torch-tensor" rel="nofollow noopener">example</a>)</li>
</ol>
<p>The numpy ndarray and the tensor should then share the same storage, which is on the shared memory.</p>

seliad: <p>I didn’t say it, but I meant sharing cuda tensors.<br>
(e.g: doing something like Hogwild example- but having the model on GPU instead of CPU and share its parameters with 2 MPI process on the same node)</p>
<p>(For CPU tensors, your answer is correct, I read once a blog post demonstrating it, and all we need to modify is using shared MPI mem instead).<br>
However, my 2 cents on this: it works out of the box only for host shared memory. Somehow couldn’t create shared+pinned host memory with pytorch easily. (I managed to do it with ugly hack: os.fork. I didn’t bother to make it work for MPI too)</p>
Draw Gaussian Blob for a bounding box

abhi8569: <p>I am working on an object detection model for which I have training images with bounding box detail for different objects in the image. I am trying to generate a Gaussian blob for each of the bounding boxes. I can generate the Gaussian blob but it is not what I am expecting<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5098494c571f5abcc4a2e444ec71a4e3e038b550.png" data-download-href="https://discuss.pytorch.org/uploads/default/5098494c571f5abcc4a2e444ec71a4e3e038b550" title="ObjectBlob"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5098494c571f5abcc4a2e444ec71a4e3e038b550_2_690x175.png" alt="ObjectBlob" data-base62-sha1="buYt72FtLdoBUyE3HhMGKloI7WE" width="690" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5098494c571f5abcc4a2e444ec71a4e3e038b550_2_690x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/5098494c571f5abcc4a2e444ec71a4e3e038b550_2_1035x262.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/5098494c571f5abcc4a2e444ec71a4e3e038b550.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ObjectBlob</span><span class="informations">1133×288 28.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see the blob has white patches as opposed to a solid blob with blurred edges.<br>
I am using this piece of code to generate the blob:</p>
<pre><code class="lang-auto">if height &lt; width: #Create blob of the highest dimension
        k_size = int(width)
    else:
        k_size = int(height)
kernel = cv2.getGaussianKernel(k_size, 0.3*((k_size-1)*0.5 - 1) + 0.8)
kernel = np.dot(kernel, kernel.T)
kernel *= 100
</code></pre>
<p>Is there any better way to do this?<br>
TIA</p>
Additional argument in Faster-RCNN forwardpass

UniqueName3532133: <p>Hello everyone,</p>
<p>I am currently trying to boost the classification accuracy of Faster-RCNN by using and auxiliary classification network for feature enrichment. To utilize the new inputs I need to add an additional argument in my box_predictor where x_cls is the last flatten layer of my auxiliary classification network.</p>
<pre><code class="lang-auto">class myRichDetector(nn.Module):

    def __init__(self, in_channels, cls_flatten, num_classes):
        super(myRichDetector, self).__init__()
        # size of cls_flatten is equal to the size of the last flatten layer of the aux network
        enriched_layer = in_channels + cls_flatten
        self.pred_tor = FastRCNNPredictor(in_channels, num_classes)
        self.pred_tor.cls_score = nn.Linear(enriched_layer, num_classes)
        self.pred_tor.bbox_pred = nn.Linear(in_channels, num_classes * 4)

    def forward(self, x, x_cls):
        # enrich the output of the F-RCNN with the aux output
        x_enr = torch.torch.cat((x, x_cls), 1)
        if x.dim() == 4:
            assert list(x.shape[2:]) == [1, 1]
        x = x.flatten(start_dim=1)
        scores = self.pred_tor.cls_score(x_enr)
        bbox_deltas = self.pred_tor.bbox_pred(x)

        return scores, bbox_deltas

class Model_detection(nn.Module):
    def __init__(self, num_classes= None, in_features_cls=None, feature_extract=True, use_pretrained=True):
        super(Model_detection, self).__init__()


        self.model = models.detection.fasterrcnn_resnet50_fpn(pretrained=use_pretrained)
        in_features = self.model.roi_heads.box_predictor.cls_score.in_features

        # slice the model to extract the roi_heads and remove the box_predictor
        self.model_my_roi_heads = nn.Sequential(*list(self.model.roi_heads.children())[:-1])
        # remove the roi_head from the model
        self.model = nn.Sequential(*list(self.model.children())[:-1])
        # define the custom box_predictor
        self.model_rich_head = myRichDetector(in_features, in_features_cls, num_classes)

        self.set_parameter_requires_grad(self.model, feature_extract)
        self.set_parameter_requires_grad(self.model_my_roi_heads, feature_extract)
        self.set_parameter_requires_grad(self.model_rich_head, feature_extract)


    def set_parameter_requires_grad(self, model, feature_extracting):
        if feature_extracting:
            for param in model.parameters():
                param.requires_grad = False

    def forward(self, x, target, cls_latent):
        
        # normal F-RCNN forward pass
        x = self.model(x, target)
        x = self.model_my_roi_heads(x)
        
        # introduce the new features for the prediction
        x = self.model_rich_head(x, cls_latent)

        return x
</code></pre>
<p>Now <code>x = self.model(x, target)</code> gives me following error:</p>
<pre><code class="lang-auto">TypeError: forward() takes 2 positional arguments but 3 were given
</code></pre>
<p>Can anybody help me with this issue?<br>
Thank you very much in advance</p>
Create a dataset from sequential data PyTorch

hect1995: <p>I have a dataset with IMU information at each timestep and the corresponding ground truth location. I want to create a dataset to be fed into a RNN network in PyTorch but I do not know how to create the Dataset class as the time order needs to be kept and a batch of items is considered. So consider I have many CSV files with 4 columns (i.e. orientation x, orientation y, acceleration x, acceleration y) and 10k timesteps each and other ground truth files with 10k timesteps and the real location for each timestep in each file.</p>
<p>I have already accomplish to get IMU batches of 200 samples from all the files, so the dimensions are <code>[total number of subdivisions or batches, 200, 4]</code>. Now I do not know how to create a dataset to be used by PyTorch to iterate over multiple batches at the same time (i.e. equal to when doing Sequence-to-Sequence in Text where each batch contains several sentences).</p>
Log_softmax_forward is not implemented for type torch.cuda.LongTensor？

luodahei: <p>when i use CrossEntropyLoss ,and convert the target to tensor.cuda.LongTensor.it shows that ‘log_softmax_forward is not implemented for type torch.cuda.LongTensor’.So，what should i do？</p>

kelam_goutam: <p>How about trying</p>
<blockquote>
<p>loss = loss_function(output,target.long())</p>
</blockquote>
<p><em><strong>NOTE</strong></em>:- CrossEntropyLoss(input, target) takes LongTensor targets and FloatTensor inputs</p>
<p>so make sure your output are float tensors</p>

luodahei: <p>thanks,but when i have convered the output to floart.and it shows that ‘RuntimeError: grad can be implicitly created only for scalar outputs’.</p>

kelam_goutam: <p>Are you using Multiple GPU and using torch.nn.DataParallel</p>

mathematics: <p>Similar problem occured saying<br>
<strong>RuntimeError: log_softmax_forward is not implemented for type torch.LongTensor</strong></p>
<p>When using nn.CrossEntropyLoss() ( But works in MSELoss)</p>
<p>Before i was getting</p>
<p><strong>RuntimeError: Expected object of type torch.LongTensor but found type torch.FloatTensor for argument <span class="hashtag">#2</span> 'target’</strong></p>
<p>I’ve changed the tensor type to Longtensor by using .<strong>long()</strong> for both input and label images in criterion as<br>
<code>        loss = criterion(outputs.long(), images.long())</code></p>
<p>Now getting the this error.I’ve changed both to other type and different type each giving different errors.</p>
<p>Please help!!.</p>

ptrblck: <p>For a vanilla multi-class classification <code>nn.CrossEntropyLoss</code> expects the model outputs to be a <code>FloatTensor</code> in the shape <code>[batch_size, nb_classes]</code>, while the target should be a <code>LongTensor</code> in the shape <code>[batch_size]</code> containing class indices in the range <code>[0, nb_classes-1]</code>.</p>

mathematics: <p>I dont understand what you are saying but, I got got my solution.  Actually what i was doing was using sigmoid activation at output and using nn.CrossEntropyLoss didnt worked  but BCELoss worked. Also found that for using nn.CrossEntropyLoss , we have to explicitly use  softmax activation. Is i am right?</p>

ptrblck: <aside class="quote no-group" data-username="mathematics" data-post="7" data-topic="25334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"> mathematics:</div>
<blockquote>
<p>Also found that for using nn.CrossEntropyLoss , we have to explicitly use softmax activation. Is i am right?</p>
</blockquote>
</aside>
<p>No, you should pass the raw logits to <code>nn.CrossEntropyLoss</code>, as internally <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be used.</p>

mathematics: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .getting over it.</p>
I want to build inception model from this simple cnn model

Muhammad_Izaz: <p>class ConvNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(ConvNet, self).<strong>init</strong>()<br>
self.layer1 = nn.Sequential(<br>
nn.Conv1d(1, 16, kernel_size=9, stride=1 , padding=4),<br>
nn.BatchNorm1d(16),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2))<br>
self.layer2 = nn.Sequential(<br>
nn.Conv1d(16, 32 ,kernel_size=5, stride=1, padding=2),<br>
nn.BatchNorm1d(32),<br>
nn.ReLU(),<br>
nn.MaxPool1d(kernel_size=2, stride=2))<br>
self.drop_out = nn.Dropout()<br>
self.fc1 = nn.Linear(32*25, 1000)<br>
self.fc2 = nn.Linear(1000, 2)</p>
<pre><code>def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    
    out = out.reshape(-1,32*25)
    out = self.fc1(out)
    out = self.fc2(out)
    return out</code></pre>

ptrblck: <p>You could use the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/inception.py">torchvision inception implementation</a> as the base code. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Muhammad_Izaz: <p>this is too difficult for me i want simple 1D model.</p>
Using pytorch to run simulations?

Jordan_Howell: <p>Hello.</p>
<p>Is it possible to use pytorch to run simulations?  I would like to utilize my GPU if possible to make this faster.</p>
<p>Here is an example of one of my simulations using a student T distribution:</p>
<pre><code class="lang-auto">starting_cash = 38927.72

sim = pd.DataFrame()
iterations = 50000
time_horizon = 504
annual_investment = 0


for x in range(iterations):
    
    #expected average return
    expected_return = expected_return
    
    #expected volatility based on mean absolute deviation
    expected_volatility = expected_volatility
    
    #collect the data
    stream = []
    
    for i in range(time_horizon):

        #random weights for starting allocation
        weights = np.asarray(sharpe_portfolio.iloc[:,3:].values)
#         print(type(weights))

        starting_allocation = starting_cash * weights
        
        return_change = np.random.standard_t(df = df_return.shape[0] - 1)
        
        end = np.round(starting_allocation.T * (1 + return_change) + annual_investment, 2)
        
        stream.append(end.sum())
        
        starting_allocation = end
#         print(x, i)
    sim[x] = stream
</code></pre>
<p>How could I turn that into a pytorch-ran module utilizing the gpu?</p>
Custom c++ class compile issues

am0: <p>Hi,</p>
<p>I’m trying to run the first example here (<a href="https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html</a>) and fail during the compilation phase.<br>
I’m trying to run it in a docker environment. Is it possible to upload a list of expected versions? or even better, a docker file that can be used to run the example?</p>
<p>Thanks,<br>
Alex</p>
Can I include an auto encoder training in my model and only use the bottleneck?

shamoons: <p>I have a problem where I want to get a compressed representation of some high-dimensional data. So normally, I would train an autoencoder to get that representation. But I don’t actually need the decoder, so is it possible for me to have the autoencoder training in my main loop?</p>
<p>For my AE, I’d want to use MSE loss and for my main model, I have a different loss function.</p>
<p>Is it possible to do it all at once or do I have to train my autoencoder separately?</p>

ptrblck: <p>You could split your autoencoder into the encoder and decoder part and then train all modules together.<br>
Here is a dummy example:</p>
<pre><code class="lang-python">encoder = MyEncoder()
decoder = MyDecoder()

z = encoder(data)
out = decoder(z)

loss_z = criterion_z(z, target_z)
loss_out = criterion_out(out, target_out)
loss = loss_z + loss_out
loss.backward()
</code></pre>
<p>Let me know, if that would work for you.</p>