akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RNN weight Initialize, or 'repeat' (Beam Search)

triplet02: <p>Hi everyone!<br>
I’m working on end-to-end speech recognition as my university graduation project.</p>
<p>For short, I wonder</p>
<blockquote>
<p>Is it possible to initialize RNN weight by other RNN’s weight value, or “repeat” it to inflate it’s size?</p>
</blockquote>
<p>Recently my team tried to apply beam search during inference, and had discussion about RNN’s weight.</p>
<p>If we set beam size as 3, each beam will have its own hidden state.<br>
We are looking for the way to compute all beam’s hidden state at once, not using for loop and deal with it one by one.<br>
So we repeated encoder output and other values for 3 times(same as beam size).<br>
But what about weights?</p>
<p>If I have some misunderstandings about beam search process, please let me know.</p>
<p>Thank you very much!</p>
Tried to allocate 5.75 GiB (GPU 0; 6.00 GiB total capacity; 881.81 MiB already allocated; 3.67 GiB free; 988.00 MiB reserved in total by PyTorch)

AmoghM: <p>I am trying to run Allen NLP Elmo model but it doesn’t work due to lack of GPU memory. However, there is no PyTorch session running and also I have tried doing:<br>
<code>import torch</code><br>
<code>torch.cuda.empty_cache()</code><br>
and<br>
<code>import gc</code><br>
<code>gc.collect()</code></p>
<p>but the error message is the same. How can I fix this? Looks like a common issue with no clear solution for this. Thanks!</p>

albanD: <p>Hi,</p>
<p>It looks like you try to allocate too much memory when running the model compared to what you have.<br>
I guess the model you run is based on pytorch? If so you might want to reduce the batch size used there?</p>

AmoghM: <p>The model is based on PyTorch and I made my batch size as 2 to check if that is an issue. I get the exact same error. I don’t understand how does PyTorch reserve 988 MiB memory. Is it on the fly?</p>

albanD: <p>Well this happens in the middle of the forward I guess? So some data is already allocated by previous code.<br>
But given that you try to allocate 5.75GB for a single Tensor and you have 6GB memory. This looks like it is very unlikely to succeed if you have anything but that Tensor on the GPU <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

AmoghM: <p>The reason was that the input batch was too big for the GPU to accomodate.</p>
Why should I reshape a tensor before call a loss function?

soojin99: <p>so here is the code,</p>
<pre><code class="lang-python">    for i in range(len(target)):
        # GRU output
        output, hidden = model(target[i], hidden)

        # reshape a tensor, but why?
        (seq, bat, inp) = output.size()
        output = output.reshape(seq, inp, bat)

        # calculate loss
        loss = criterion(output, label[i].argmax(2)).to(device)

        # backpropagation and change parameters
        optimizer.zero_grad()
        loss.backward(retain_graph=True)
        optimizer.step()
</code></pre>
<p>So, the output from GRU model’s forward propagation is a tensor<br>
shape of (seq_len, batch_size, input_size).<br>
I thought the labelled data of this input(target in loss function) should be a tensor shape of<br>
(seq_len, batch_size).<br>
In this code, variable label is a one hot vector , shape of (seq_len, batch_size, input_size) so i called argmax(dim=2) to make it acceptable for loss function becuz it don take a one hot vector.</p>
<p>But without reshaping an output tensor,  this gives me an error :<br>
ValueError : Expected target size (seq_len, input_size),<br>
got torch.size([seq_len, batch_size]).</p>
<p>I actually can solve this problem by reshaping the output tensor<br>
by (seq_len, input_size, batch_size)<br>
but why should i do that?<br>
isnt it natural doing this without reshaping???</p>
Remove all RELU layers from Resnet model in Pytorch

Sushruth_N: <p>Can anyone suggest me on how to remove all RELU layers from Resnet model? And if possible replace by some linear function</p>

ptrblck: <p>You could replace the <code>self.relu</code> of the base model as well as all blocks by your desired activation.<br>
This code snippet should work:</p>
<pre><code class="lang-python">model = models.resnet50()

names = []
for name, module in model.named_modules():
    if hasattr(module, 'relu'):
        module.relu = nn.Sigmoid()
        
print(model)
</code></pre>

Sushruth_N: <p>Thank you. But what if I want to remove relu or any kind of activation layers completely?</p>

ptrblck: <p>Then you could replace it with <code>nn.Identity()</code> instead of another activation function.</p>

Sushruth_N: <p>Oh yea. This is easier to do than deleting the RELU layer. Awesome</p>
RuntimeError: cudnn RNN backward can only be called in training mode

digbose92: <p>Hi ,</p>
<p>I am trying to evaluate gradient of the class score with respect to output of conv layer in my model . My model has a GRU and the entire model is in GPU. When I am running .backward() with the predicted class score, I am getting this error. My model is set to eval () mode now. Is backprop disabled for eval() mode in pytorch ?</p>

MariosOreo: <p>Hi <a class="mention" href="/u/digbose92">@digbose92</a>,</p>
<p>Did you get some ideas?</p>

Maxxingxing: <p>I encounterd the same problems, and I copied similar code from another project which using cnn instead of rnn. So I am not sure this is allowed only in cnn?</p>

ptrblck: <p>Do you get the error about <code>cudnn RNN</code> while you are only using conv layers?<br>
If so, could you please post the code so that we can take a look?</p>
Assigning Module to Device

almeetb: <p>I was wondering if someone could walk me through what happens when<br>
<code>model = model.to('cuda:0')</code> is called.</p>
<p>In particular how do parameters and buffers get moved onto the GPU</p>

ptrblck: <p>[Here[(<a href="https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/torch/nn/modules/module.py#L367">https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/torch/nn/modules/module.py#L367</a>) you can find the <code>module.to()</code> implementation, which calls <a href="https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/torch/nn/modules/module.py#L445-L448">convert</a> recursively on all parameters and buffers.</p>
Cuda allocates 0 memory

soojin99: <pre><code class="lang-python">device = None

if torch.cuda.is_available():
    device = torch.device("cuda")
    print("cuda is available!")
    print('Memory Usage:')
    print('Max Alloc:', round(torch.cuda.max_memory_allocated(0)/1024**3, 1), 'GB')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')
    print('cuDNN:    ', torch.backends.cudnn.version())

else:
    device = torch.device("cpu")
</code></pre>
<p>this gives me</p>
<pre><code class="lang-auto">cuda is available!
Memory Usage:
Max Alloc: 0.0 GB
Allocated: 0.0 GB
Cached:    0.0 GB
cuDNN:     7604
</code></pre>
<p>the problem is that these are the first few lines following module importing.<br>
so this should not happen.<br>
ive installed cuDNN and cuda, each is 7.6.4, and 10.1.<br>
torch version is 1.5.0+cu101</p>
<p>so when i try to run this code it gives me error like<br>
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</p>
<p>on</p>
<pre><code class="lang-python">model = GRU(input_size, hidden_size, num_layers).to(device)
</code></pre>

ptrblck: <p>If you just <code>import torch</code> without running or creating any CUDA operations or tensors, the GPU memory should be empty (a CUDA context might be initialized, but wouldn’t show up in <code>torch.cuda.memory_allocated()</code>).</p>
<p>For the <code>CUDNN_STATUS_INTERAL_ERROR</code>:<br>
could you please post a reproducible code snippet?<br>
Also, did you install the PyTorch binaries or built from source? Which GPU are you currently using and is the code running fine on the CPU?</p>

soojin99: <p>Thank you for answering!! It somehow no longer throws an error after I rebooted my desktop. Thank you!</p>
Severe imbalance in dataset

harris: <p>I am looking for a technique to do binary classification on videos with severe imbalance problem.<br>
no. of positive images : 101872<br>
no. of negative images : 2607</p>
<p>Dataset contains grayscale images and they are very different from ImagNet so i’ve not tried pretrained networks.</p>
<p>Here is what I’ve tried so far :</p>
<ol>
<li>break videos into images</li>
<li>label each image individually</li>
</ol>
<p>Strategy 1:<br>
3. train one of the cnn (lenet, vgg and resnet)<br>
4. all positives classified correctly by all networks and negatives are all classified as false positives.</p>
<p>Strategy 2:<br>
3. simple oversampling by a factor by adjusting length of dataloader and multiplying minority class image indexes with the factor (hyperparameter)<br>
4. get some true negatives at the cost of false negatives but still ain’t good.</p>
<p>Strategy 3:<br>
3. use weighted oversampler provided by pytorch. weights = (1, len(trainset)/len(positives))<br>
4. train/ test results no good than oversampling.</p>
<p>Strategy 4:<br>
3. use cnn + rnn together, train on one of the cnn architectures and give the outputs before fully connected layers to lstm/gru (many to many).<br>
4. still no good.</p>
<p>Strategy 5:<br>
tried strategy 4 with oversampling/ weighted oversampling but didn’t get better results (played with sequence length of rnn too).</p>
<p>I don’t know what am I missing here, but something is really off and I can’t figure out where and what. Should I use different cnn arch? I’m guessing that’s not the problem since I get very similar results from vgg/resnet/lenet. Any suggestion would be an immense help.</p>
<p>Some notes:<br>
-no shuffling of train/test set (retains sequential info for strategy 4/5)<br>
-tried only with sgd and adam optimizers<br>
-learning rate always 0.001<br>
-tried different batch sizes too</p>
Python and C++ source code for moving tensors from CPU to GPU

kevinxyliu: <p>I’m trying to find the source code of the operation to move tensors from CPU to GPU in PyTorch:</p>
<pre><code>data = data.to(device) </code></pre>
<p>However, I’ve checked for many places on Github, but cannot find both the Python code and C++ code for this operation. Does anyone know where the function calls and interfaces for moving tensors between devices are defined? Any help would be appreciated!</p>

ptrblck: <p>You can find the implementation of <code>to()</code> <a href="https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/aten/src/ATen/native/TensorConversions.cpp#L48">here</a>, which should eventually call <a href="https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/aten/src/ATen/native/cuda/Copy.cu#L22">this</a> for a device to device copy and <a href="https://github.com/pytorch/pytorch/blob/ebcacd5e878f3f8e84c941d6b9567c88f50a2f10/aten/src/ATen/native/cuda/Copy.cu#L128">this</a> for a host to device or vice versa copy.</p>
Model Feature Fusion

Ghasem_Abdi: <p>Hi folks,</p>
<p>I have two different image data-set but related to a same class. I am going to use pre-trained net like alexnet for both to detect features and then, concatenate those features into a classifier. I would appreciate if you can help me to convert these (double data loaders and feature fusion) from pytorch to fastai platfrom:</p>
<pre><code class="lang-auto">class DoubleSource(torch.utils.data.Dataset):
  def __init__(self, img_i , img_ii, transform=None):
    self.img_i = img_i
    self.img_ii = img_ii
    
    if transform is None:
      self.transform = transforms.ToTensor()
    else:
      self.transform = transform
  
  def __getitem__(self, idx):
    names_i, labels = self.img_i[idx]
    names_ii, labels = self.img_ii[idx]
    input_i = Image.open(names_i).convert('RGB')
    input_ii = Image.open(names_ii).convert('RGB')
    input_i = self.transform(input_i)
    input_ii = self.transform(input_ii)
    return input_i, input_ii, labels, names_i, names_ii
  
  def __len__(self):
    return len(self.img_i)
</code></pre>
<pre><code class="lang-auto">class initialize_model(nn.Module):
  def __init__(self, num_classes=10, feature_extract=True, use_pretrained=True):
    super(initialize_model, self).__init__()

    model_ft = models.alexnet(pretrained=use_pretrained)
    set_parameter_requires_grad(model_ft, feature_extract)

    self.features_i = model_ft.features
    self.features_ii = model_ft.features

    self.classifier = model_ft.classifier
    self.classifier[1] = nn.Linear(2 * model_ft.classifier[1].in_features, model_ft.classifier[1].out_features)
    self.classifier[6] = nn.Linear(model_ft.classifier[6].in_features, num_classes)
  
  def forward(self, input_i, input_ii):
    output_i = self.features_i(input_i)
    output_i = output_i.view(output_i.size(0), -1)
    output_ii = self.features_ii(input_ii)
    output_ii = output_ii.view(output_ii.size(0), -1)
    
    output = torch.cat((output_i, output_ii), dim=1)
    output = self.classifier(output)
    return output
</code></pre>
<p>Thanks in advance,</p>
Apply the same CNN to a list of images simultaneously

a514514772: <p>Hi all,</p>
<p>Assuming I have a input tensor with shape B x N x C x H x W, I want to apply the same CNN model f across the dimension N.</p>
<p>For now, I use for-loop to go over it.</p>
<pre><code class="lang-auto">out = []
for i in range(N):
    out.append(self.conv(input[:, i])
</code></pre>
<p>I was wondering if there is any elegant way to do this.</p>
<p>Thanks.</p>

donJuan: <p>I work with spatio temporal data and implement stuff like you. My N is the temporal dimension. I have not found any other way than use a loop. Hope some1 can suggest some vectorization style approach</p>

chetan06: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py</a></h4>
<pre><code class="lang-py">from __future__ import division

import torch
from ._functions import SyncBatchNorm as sync_batch_norm
from .module import Module
from torch.nn.parameter import Parameter
from .. import functional as F
from .. import init


class _NormBase(Module):
    """Common base of _InstanceNorm and _BatchNorm"""
    _version = 2
    __constants__ = ['track_running_stats', 'momentum', 'eps',
                     'num_features', 'affine']

    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True,
                 track_running_stats=True):
        super(_NormBase, self).__init__()
        self.num_features = num_features
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>You can check Pytorch source code for some better implementation</p>
Torch 1.5 c++ front end: torch::cuda::is_available() = false?

artemmikheev: <p>I just installed torch 1.5 c++ front end (Windows, Visual Studio 2019). Cuda 10.1, latest<br>
nvidia drivers. This is in DEBUG version of the library.</p>
<p>Following code:</p>
<p>main() {<br>
CudaPresent = torch::cuda::is_available();<br>
}</p>
<p>CudaPresent is FALSE.</p>
<p>When I use torch 1.4 C++ front end, everything else is the same on the same PC, code is working fine<br>
and returns CudaPresent = TRUE</p>
<p>Any help super appreciated.<br>
thanks!</p>

peterjc123: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/37124" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/37124" target="_blank" rel="nofollow noopener">Link error, Libtorch 1.5 on Windows</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-23" data-time="02:33:09" data-timezone="UTC">02:33AM - 23 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-04-23" data-time="02:45:19" data-timezone="UTC">02:45AM - 23 Apr 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/slaweks17" target="_blank" rel="nofollow noopener">
          <img alt="slaweks17" src="https://avatars1.githubusercontent.com/u/8934797?v=4" class="onebox-avatar-inline" width="20" height="20">
          slaweks17
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Large number of link errors when trying rebuild an existing program with the new 1.5 version of Libtorch on Windows
To...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Better for binary classification

Slamskow: <p>Why is CrossEntropyLoss better than BCEWithLogitsLoss for a multi-class classification problem? I heard CrossEntropyLoss was better but I can’t wrap my head around why?</p>

ptrblck: <p><code>nn.BCEWithLogitsLoss</code> is used for a binary or multi-label classification (each sample can belong to zero, one or more classes), not for a multi-class classification (each sample belongs to one class only).</p>
<p>You could use both criteria for a binary classification, though.<br>
With <code>nn.BCEWithLogitsLoss</code> it would be a vanilla binary classification, while <code>nn.CrossentropyLoss</code> would treat the use case as a 2-class multi-class classification.</p>

rzhang63: <p>So for binary classification problem (<code>Y in {0,1}</code>), if I want to use <code>nn.BCEWithLogitsLoss</code>, then the output of my model should be raw scores for class=1 and have shape=(N) where N is the batch size, right?</p>

ptrblck: <p>The model should output the raw logits in the shape <code>[batch_size, 1]</code> and the target should have the same shape for a binary classification.<br>
The suggested form of <code>[batch_size]</code> would most likely also work, but I would recommend to create the class dimension explicitly (just my two cents <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>
Installation problem: Library not loaded: @rpath/libc++.1.dylib

ks7g2h3: <p>I installed pytorch successfully on my Mac (Sierra 10.12.6), using <code>pip install --user torch==1.0.0</code>. However, when I try to import torch from inside python (2.7.10), I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/username/Library/Python/2.7/lib/python/site-packages/torch/__init__.py", line 84, in &lt;module&gt;
    from torch._C import *
ImportError: dlopen(/Users/username/Library/Python/2.7/lib/python/site-packages/torch/_C.so, 9): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Users/username/Library/Python/2.7/lib/python/site-packages/torch/_C.so
  Reason: image not found
</code></pre>
<p>I know of at least one other user who has had exactly the same problem, but on the Mojave os, for the same versions of python and torch. Previously, I was able to run torch 0.4.1 without incident, this problem is only post-upgrade. I am able to run torch 1.0.0 with python 3.6.5 (within a virtualenv) fine.</p>

peng_li: <p>Hey, have you got this fixed? I got same situation like you.</p>

ks7g2h3: <p>No. I didn’t get any clarity on this specific issue, but I was able to reinstall successfully from source.</p>

tung7970: <p>I fixed it by adding /usr/lib to rpath using install_name_tool.</p>
<p>install_name_tool -add_rpath /usr/lib ~/Library/Python/2.7/lib/python/site-packages/torch/_C.so</p>

0bff83efac608c536648: <p>thanks, this helps me to solve the same problem.<br>
just use “install_name_tool -add_rpath xxx/_C.so”</p>

dynax: <p>Thanks! I installed torch globally. I fixed the issue with<br>
sudo install_name_tool -add_rpath /usr/lib /Library/Python/2.7/lib/python/site-packages/torch/_C.so</p>

jiayiliu: <p>In case, if you used brew installed python, the command looks like:<br>
<code>sudo install_name_tool -add_rpath /usr/lib /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so</code></p>

tylcole: <p>Hmm, my error is:</p>
<p>ImportError: dlopen(/Users/tylcole/.virtualenvs/stanza/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so, 9): Library not loaded: <span class="mention">@rpath</span>/libc++.1.dylib<br>
Referenced from: /Users/tylcole/.virtualenvs/stanza/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so<br>
Reason: image not found</p>
<p>And I tried<br>
install_name_tool -add_rpath /usr/lib /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so</p>
<p>But the error that I get is:<br>
error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: for: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so (for architecture x86_64) option “-add_rpath /usr/lib” would duplicate path, file already has LC_RPATH for: /usr/lib</p>
<p>I am just trying to use Python for one part in a project and I am getting stuck on this. I’m not too familiar with how it stores files and deals with virtual environments differently.</p>
How to specify the number of filters for a Conv1D?

shamoons: <p>I’m looking to implement a ResNet with a <code>Conv1D</code>, but I don’t quite understand how to specify the number of filters? My data has <code>161 channels</code>, so my <code>in_channels</code> and <code>out_channels</code> is set to 161. But How would I specify that I want 64 filters?</p>

ptrblck: <p>The <code>out_channels</code> specify the number of filters, so you could use <code>out_channels=64</code>.<br>
<a href="https://cs231n.github.io/convolutional-networks/#layers">CS231n</a> gives a good overview about the shapes and methods used for convolutions.</p>

shamoons: <p>I understand in the 2d sense. But for a 1D, I have 161 in channels, so going down to 64 would be a reduction. Normally, I think, I’d want an increase in number of filters (at least that’s what I’ve seen for most 2D CNNs).</p>

ptrblck: <p>Yes, usually you would increase the filters at the beginning at least.<br>
The number of filters for 1D and 2D convolutions are defined in the same sense.</p>
<p>I think it depends on your use case, if a channel reduction at the beginning will work fine or not.<br>
E.g. if your input data contains some redundant channels, it might work fine, otherwise you might lose too much information.</p>
<p>Have you tried different approaches and see a specific pattern in the model performance regarding the number of channels?</p>
Matrix diagonal part

FeeQ: <p>What might be a possible analog for <a href="https://www.tensorflow.org/api_docs/python/tf/linalg/diag_part" rel="nofollow noopener">https://www.tensorflow.org/api_docs/python/tf/linalg/diag_part</a>  in pytorch?</p>
<p>I didnot manage to find something in docs <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

LeviViana: <p>I don’t think that there is an exact equivalent. But here is a workaround:</p>
<p>PS: this example is the same exposed in the link you provided.</p>
<pre><code class="lang-python">import torch
x = torch.Tensor([
                  [[1, 0, 0, 0],
                   [0, 2, 0, 0],
                   [0, 0, 3, 0],
                   [0, 0, 0, 4]],
                  [[5, 0, 0, 0],
                   [0, 6, 0, 0],
                   [0, 0, 7, 0],
                   [0, 0, 0, 8]]
		         ])
y = torch.stack(tuple(t.diag() for t in torch.unbind(x,0)))
</code></pre>

xiaoyanzhuo: <aside class="quote no-group" data-username="LeviViana" data-post="2" data-topic="47446">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>torch.unbind</p>
</blockquote>
</aside>
<p>I think now <code>torch.diagonal</code> serves for same purpose.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
a = np.array([[[1, 2, 3, 4],  # Input shape: (2, 3, 4)
               [5, 6, 7, 8],
               [9, 8, 7, 6]],
              [[5, 4, 3, 2],
               [1, 2, 3, 4],
               [5, 6, 7, 8]]])

a = torch.from_numpy(a)
# a.size() 
torch.diagonal(a, offset=0, dim1=-2, dim2=-1)
# torch.diagonal(b_tensor, offset=0, dim1=1, dim2=2)  # this return same results, set dim
</code></pre>
<p>return:</p>
<pre><code class="lang-auto">tensor([[1, 6, 7],
        [5, 2, 7]])
</code></pre>
<p>More details can be found in: <a href="https://pytorch.org/docs/master/torch.html?highlight=diag#torch.diagonal" class="inline-onebox" rel="noopener nofollow ugc">torch — PyTorch master documentation</a></p>
C10::variant vs. std::variant

ericrhenry: <p>I know that the pytorch C++ frontend is now pretty much grounded in C++14. In my own code, I have a possible good use case for std::variant. This is actually a two-part question:</p>
<ol>
<li>So far, it appears that torch::optional is a std::optional workalike. Does the same apply to c10::variant? The only use cases I have seen are where it is being used to fold in a bunch of enums.</li>
<li>If I punt and just compile my code specifying the C++17 standard so I can get at std::variant, is it possible that this will break anything appearing in the torch headers? I am guessing not, but forewarning would be helpful.<br>
I am still digging through the source code to try to address <span class="hashtag">#1</span>. I can probably come up with a work-around if necessary, but std::variant seems to be the most elegant option.<br>
Thanks,<br>
Eric</li>
</ol>
Load pretrained BERT without Huggingface's transformers

mtriet: <p>For educational purpose, I would like to use BERT embedding as input to solve the SQuAD Dataset. So far, converting BERT pretrained model to a pytorch model does not work (Issues 393, 1619, cannot post more than 2 links), and most tutorial I find online uses Huggingface’s transformer, which is kinda not my taste since they provide much more than what I need.</p>
<p>Is there a way to load BERT using <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding.from_pretrained" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding.from_pretrained</a>? In a broader sense, is there a way such that when another embedding comes up, we can import it to the model without having to depend on the others?</p>
Pandas - convert list to integer

GerryNoble69: <p>Hope someone can help with this.</p>
<p>I have a column of data that is in a pandas dataframe.</p>
<p>I need to convert the strings in this column to integers - but since the strings repeat I want a unique number assigned to each string e.g.  a banana should = 0, an apple = 1 etc.  However, I do not want to specify the assignment since there are may be up to 200 unique strings in a dataset of 175,000.</p>
<p>Column                          Desired result would be automatic allocation of numbers to each string name<br>
banana                         replaced by a 0<br>
apple                            replaced by a 1<br>
pear                              replaced by a 2<br>
banana                         replaced by a 0<br>
grape                            replaced by a 3<br>
apple                            replaced by a 1<br>
pear                              replaced by a 4  etc.</p>
<p>So its automatic assignment of a number to each string - without needing to  know in advance how many strings/numbers are in the column.</p>
<p>Thanks in advance.</p>

sai_m: <p>Hi Gerry,<br>
Use pandas.factorize(<a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.factorize.html" rel="nofollow noopener">https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.factorize.html</a>) to convert desired column into unique integers.</p>

GerryNoble69: <p>Thanks for replying.</p>
<p>I think I have found that the answer lies in the following code for in-situ updates. I have tried it and it seems to work.</p>
<p><span class="hashtag">#protos</span> is the column name</p>
<p>protos = df.proto.unique()</p>
<p>proto_dict = dict(zip(protos, range(len(protos))))</p>
<p>df=df.applymap(lambda s: proto_dict.get(s) if s in proto_dict else s)</p>
<p>Thanks again for taking the time to reply.</p>
<p>Gerry</p>
A convention when training embedding layer

sh0416: <p>Hi,</p>
<p>I am training a neural network with embedding layer.</p>
<p>Is there any conventional training strategy for this?</p>
<p>I saw many optional parameter setting (scaling by frequency, max norm, etc.) in Embedding layer and I don’t know when I use this parameter…</p>
<p>Thanks,</p>

sai_m: <p>have you tried using pretrained embeddings ?</p>

sh0416: <p>Yes, but I think it is inappropriate and also there is out of vocabulary issue…<br>
I used googlenews, fasttext, bert.</p>
<p>Moreover, I build up my own corpus and pretrained using gensim skipgram model, which also doesn’t work, either… (I also have numerous design decision during this training, e.g. the number of epoch, the size of dimension, min_count…)</p>
<p>I think training embedding layer jointly might be a better choice… Isn’t it?</p>
RuntimeError: (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

azer: <p>I would greatly appreaciate it if you can help me to solve the error below</p>
<p>When training  in google colab I get the following:</p>
<p>untimeError                              Traceback (most recent call last)<br>
 in ()<br>
3 for epoch in range(num_epochs):<br>
4     # train for one epoch, printing every 10 iterations<br>
----&gt; 5     train_one_epoch(mask_model, optimizer, data_loader, device, epoch, print_freq=10)<br>
6     # update the learning rate<br>
7     lr_scheduler.step()</p>
<p>9 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in conv2d_forward(self, input, weight)<br>
340                             _pair(0), self.dilation, self.groups)<br>
341         return F.conv2d(input, weight, self.bias, self.stride,<br>
–&gt; 342                         self.padding, self.dilation, self.groups)<br>
343<br>
344     def forward(self, input):</p>
<p>RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same.</p>
<p>I</p>

TinfoilHat0: <p>Your data is on the GPU and your model is on CPU. Pass the model to gpu and it should be fine.</p>

azer: <p>Thanks  a lot that worked. However I have a new error.  Please help!!</p>
<p>TypeError                                 Traceback (most recent call last)<br>
/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py in linspace(start, stop, num, endpoint, retstep, dtype, axis)<br>
116     try:<br>
–&gt; 117         num = operator.index(num)<br>
118     except TypeError:</p>
<p>TypeError: ‘numpy.float64’ object cannot be interpreted as an integer</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>TypeError                                 Traceback (most recent call last)<br>
7 frames<br>
&lt;<strong>array_function</strong> internals&gt; in linspace(*args, **kwargs)</p>
<p>/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py in linspace(start, stop, num, endpoint, retstep, dtype, axis)<br>
119         raise TypeError(<br>
120             “object of type {} cannot be safely interpreted as an integer.”<br>
–&gt; 121                 .format(type(num)))<br>
122<br>
123     if num &lt; 0:</p>
<p>TypeError: object of type &lt;class ‘numpy.float64’&gt; cannot be safely interpreted as an integer.</p>

TinfoilHat0: <p>Well seems like you have to cast <em>num</em> to an integer.</p>