akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch CUDA runtime error

username: phoenixkiller
cooked: <p>Hi,</p>
<p>I have built pytorch from source as shown at the official pytorch website.<br>
I have a GPU with compute capability 3.0 and was advised to build pytorch which can support compute capability 3.0.<br>
Even after building from source I am getting a warning as the GPU is too old, but the cuda_is_available returns true.<br>
But when I am trying to run pytorch, I still have the following error.<br>
File “waveglow/glow.py”, line 94, in forward<br>
W_inverse = W_inverse.half()<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
while trying to run the nvidia-tacotron2 demo</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think float16 computations did not exist back then. So your GPU most likely does not support such operations.</p>

username: ptrblck
cooked: <p>I also think this might be the reason.<br>
If I’m not mistaken, FP16 operations were introduced for compute capability 5.3.</p>
Create all possible combinations of a 3D tensor along the dimension number 1

username: paolo
cooked: <p>I have a 3D tensor (batch size, m, n), and I would like to create all the possible combinations along the axes number 1 (batch size, m^2, 2n). For example, given the following tensor</p>
<pre><code class="lang-auto">input_tensor = torch.Tensor([[[1, 2, 3, 4],
         [5, 6, 7, 8]],

        [[9, 10, 11, 12],
         [13, 14, 15, 16]],

        [[17, 18, 19, 20],
         [21, 22, 23, 24]]])
</code></pre>
<p>with size torch.Size([3, 2, 4]).</p>
<p>I would like to have the following output tensor</p>
<pre><code class="lang-auto">output_tensor = torch.Tensor([[[1, 2, 3, 4, 1, 2, 3, 4], [1, 2, 3, 4, 5, 6, 7, 8], [5, 6, 7, 8, 5, 6, 7, 8], [5, 6, 7, 8, 1, 2, 3, 4]],

        [[9, 10, 11, 12, 9, 10, 11, 12], [9, 10, 11, 12, 13, 14, 15, 16], [13, 14, 15, 16, 13, 14, 15, 16], [13, 14, 15, 16, 9, 10, 11, 12]],

        [[17, 18, 19, 20, 17, 18, 19, 20], [17, 18, 19, 20, 21, 22, 23, 24], [21, 22, 23, 24, 21, 22, 23, 24], [21, 22, 23, 24, 17, 18, 19, 20]]])
</code></pre>
<p>torch.Size([3, 4, 8])<br>
I have tried several things, and the one that went closer is</p>
<pre><code class="lang-auto">output_tensor = torch.stack([torch.cat(batch,1) for batch in product(input_tensor, repeat=2)])
</code></pre>
<p>… but still not working. I had a look at this <a href="https://discuss.pytorch.org/t/permute-elements-of-a-tensor-along-a-dimension/1105/14">link</a> but it did not help.</p>
<p><strong>UPDATE:</strong><br>
I have just managed to get the <code>output_tensor</code> with two nested loops:</p>
<pre><code class="lang-auto">output_tensor = torch.stack([torch.stack([torch.cat(t,0) for t in product(row, repeat=2)]) for row in input_tensor])
</code></pre>
<p>Is there a way to obtain <code>output_tensor</code> through <strong>vectorized</strong> operation instead of creating loops?</p>

username: acobobby
cooked: <p>You can proceed like this:</p>
<pre><code class="lang-auto">first = input_tensor.repeat(1, 2, 1)
second = input_tensor.unsqueeze(2)
second = second.repeat(1,1,2,1).view(input_tensor.size(0),-1,input_tensor.size(2))
output_tensor = torch.cat((first,second), dim=2)
</code></pre>
How to plot object pair feature maps

username: solsol
cooked: <p>Hi,</p>
<p>I want to plot a “concatenated two objects’ feature maps in an image”. hence, I modify them into the dictionary under forward part of main code, as:</p>
<p>self.dict = {img_id: {‘obj_fmaps’:final_fmaps, ‘obj_bbox’:obj_bbox, ‘relation_pair’:rel_inds}}</p>
<p>now I want to plot and save test set feature maps for each two related objects.<br>
Any suggestion please?</p>
Batchnorm1d can't run on GPU

username: Timos_Korres
cooked: <p>I am using batchnorm1d() in my networks. If I remove the batchnorm1d() from the configuration the network behaves fine on GPU. When I add it back I get the following error:</p>
<pre><code class="lang-auto">File "/mnt/mscteach_home/s1877727/recommendations/spotlight/dnn_models/cGAN_models.py", line 115, in forward
    vector = layers(vector)
  File "/home/s1877727/miniconda3/envs/mlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/s1877727/miniconda3/envs/mlp/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 83, in forward
    exponential_average_factor, self.eps)
  File "/home/s1877727/miniconda3/envs/mlp/lib/python3.7/site-packages/torch/nn/functional.py", line 1697, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: Tensor for argument #4 'running_mean' is on CPU, but expected it to be on GPU (while checking arguments for cudnn_batch_norm)
</code></pre>
<p>Here is the network architecture:</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self,layers,num_users, num_items,output_dim = 1, embedding_dim=32):
        super(MLP, self).__init__()

        self.num_users = num_users
        self.num_items = num_items
        self.latent_dim = embedding_dim

        self.embedding_user = torch.nn.Embedding(num_embeddings=self.num_users, embedding_dim=self.latent_dim)
        self.embedding_item = torch.nn.Embedding(num_embeddings=self.num_items, embedding_dim=self.latent_dim)

        self.layers = []
        self.layerDims = layers.copy()
        self.layerDims.insert(0,2*embedding_dim)
        self.layerDims.append(output_dim)
        
        for idx in range(len(self.layerDims)-1):
            self.layers.append(nn.Linear(self.layerDims[idx], self.layerDims[idx+1]))
            self.layers.append(nn.BatchNorm1d(self.layerDims[idx+1]))
        list_param = []
        for a in self.layers:
            list_param.extend(list(a.parameters()))

        self.fc_layers = nn.ParameterList(list_param)

        self.logistic = torch.nn.Sigmoid()
        self.apply(self.init_weights)

    def forward(self, user_indices, item_indices):

        user_embedding = self.embedding_user(user_indices)
        item_embedding = self.embedding_item(item_indices)

        vector = torch.cat([user_embedding, item_embedding], dim=-1)  # the concat latent vector
        

        for layers in self.layers[:-1]:
            vector = layers(vector)
            vector = nn.functional.relu(vector)
        logits = self.layers[-1](vector)
        rating = self.logistic(logits)
        return rating

    def init_weights(self,m):
        if type(m) == nn.Linear:
            torch.nn.init.xavier_uniform_(m.weight)
            m.bias.data.fill_(0.01)
    
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>the problem is that you store your layers in a regular list, not an nn.ModuleList and so it is not affected by your call to <code>model.cuda()</code>.</p>

username: Timos_Korres
cooked: <p>Thanks a lot, it works fine that’s was the issue.</p>
Torchvision: finding details on how pretrained models were pretrained

username: tedsandler
cooked: <p>I am looking for details on exactly how the torchvision models are pretrained so I can train these models on the CIFAR data sets, rather than the current ImageNet.</p>
<p>Sadly the training scripts for the pretrained models do not seem to be included with the torchvision source and many details are missing from the <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">documentation</a>.</p>
<p>However, I did find <a href="https://github.com/pytorch/examples/blob/42e5b996718797e45c46a25c55b031e6768f8440/imagenet/main.py" rel="nofollow noopener">this script</a>. Is it the script used for pretraining? And if not, where I can find this info?</p>
<p>Thanks.</p>

username: Oli
cooked: <p>Bump. I’m interested in the resnets specifically</p>
Is there any way to count/profile all matrix multiplication within a model

username: XinDongol
cooked: <p>Is there any way to count/profile all matrix multiplication within a model</p>

username: albanD
cooked: <p>Hi,</p>
<p>The <a href="https://pytorch.org/docs/stable/autograd.html#profiler" rel="nofollow noopener">profiler</a> from the autograd will be able to give you this information (as long as you have autograd enabled).</p>
Using ConcatDataset with WeightedRandomSampler

username: pinouchon
cooked: <p>I am using a ConcatDataset with a WeightedRandomSampler like this:</p>
<pre><code class="lang-auto">training_sets = data_augment(training_set)
self.train_dataset = ConcatDataset([MyDataset(
    features=self.features, 
    transform=transforms.Compose([ToTensor()])
) for _set in training_sets])

num_samples = len(self.train_dataset)
weights = np.linspace(1, 3, num_samples)
sampler = WeightedRandomSampler(weights, num_samples)
self.train_loader = DataLoader(
    self.train_dataset, sampler=sampler, num_workers=7, batch_size=self.batch_size
)
</code></pre>
<p>My idea for using the WeightedRandomSampler is to train more often on recent inputs (I want to see if that improves generalization performance).</p>
<p>However, because I am using a concat dataset, I will sample more often inputs in the last <code>training sets</code> that are concatenated at the end, rather than inputs at the end of each sets. Is that right?</p>
<p>If that’s the case, then the fix would be to do multiple <code>np.linspace()</code>, and then concatenate them the same way the ConcatDataset concats my training dataset.</p>
<p>Thoughts?</p>

username: pinouchon
cooked: <p>I’ve changed my implementation to this:</p>
<pre><code class="lang-auto">training_sets = data_augment(training_set)

# Storing the datasets before they are concatenated, 
# because we need the lengths of each
self.train_datasets = [MyDataset(
    features=self.features, 
    transform=transforms.Compose([ToTensor()])
) for _set in training_sets]
self.train_dataset = ConcatDataset(self.train_datasets)

# Creating weights for each concatenated dataset
weights = []
for d in self.train_datasets:
    dataset_len = len(d)
    weights.append(np.linspace(1, 3, dataset_len))
weights = np.concatenate(tuple(weights))
sampler = WeightedRandomSampler(weights, num_samples)
self.train_loader = DataLoader(self.train_dataset, sampler=sampler, num_workers=7, batch_size=self.batch_size)
</code></pre>
<p>I’ll let you know if it works better after my training run.<br>
I am too lazy to debug this with synthetic data.</p>
How to split the words from multiple sentences?

username: Deb_Prakash_Chatterj
cooked: <p>I am working on a Sentiment Analysis problem, and there is a weird problem, that we have to set a fixed sequence_length of the input sentence in training. If the input sentence is greater than the sequence_length then we have to truncate the sentence an if small, then we have to pad it with zeros. Now I have a strategy, that after splitting all the sentences, we can count the max_length of a sentence and then set it the sequence_length. But when counting the length of every sentence, there is a problem, please check the code -</p>
<pre><code class="lang-auto">reviews_split = all_text.split('\n')
print(reviews_split[0])
output:  bromwell high is a cartoon comedy  it ran at the same time as some other programs about school life  such as  teachers   my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers   the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students  when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled          at           high  a classic line inspector i  m here to sack one of your teachers  student welcome to bromwell high  i expect that many adults of my age think that bromwell high is far fetched  what a pity that it isn  t   
</code></pre>
<p>Next, I am using a loop to split the words of the sentences and count the number of words, but giving me the following error -</p>
<pre><code class="lang-auto">for i in reviews_split:
  #print(i)
  words = i.split()
  #word_split = words[i].split()
  #max_seq = 0
  print(len(words[i]))
print("Number of Reviews: ", len(reviews_split))
</code></pre>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-36-28f46af4c30b&gt; in &lt;module&gt;()
      6   #word_split = words[i].split()
      7   #max_seq = 0
----&gt; 8   print(len(words[i]))
      9 print("Number of Reviews: ", len(reviews_split))

TypeError: list indices must be integers or slices, not str
</code></pre>
<p>Can anyone please help how can I make it count the words in each and every sentence? Thanks in advance.</p>

username: anantshah200
cooked: <p>As far as counting the number of sentences and counting the words in each sentence is concerned, you can use the <em>Natural Language Toolkit(NLTK)</em> and it’s <em>tokenize</em> module. It provides functions to obtain each sentence and each word in a sentence.</p>
Transform List to Tensor more accurat

username: Mark0502
cooked: <p>Hello together,<br>
I want to return in the Dataloader this list:</p>
<p>[[-0.0003000000142492354, -0.0008999999845400453, 0.00039999998989515007, 0], [0.0010000000474974513, -0.00019999999494757503, 0.0003000000142492354, 0], [0.00019999999494757503, -0.0005000000237487257, -0.0008999999845400453, 0], [5.484399795532227, -24.28619956970215, 117.5000991821289, 1]]</p>
<p>But to return it, its need to be a tensor right? So i transform it with:<br>
<code>pt_tensor_from_list = torch.tensor(pose_transform)</code><br>
But information is lost in the process, is there another way to do this?<br>
Output of pt_tensor_from_list:<br>
([[  -0.0003,   -0.0009,    0.0004,    0.0000],<br>
[   0.0010,   -0.0002,    0.0003,    0.0000],<br>
[   0.0002,   -0.0005,   -0.0009,    0.0000],<br>
[   5.4844,  -24.2862,  117.5001,    1.0000]])</p>

username: ptrblck
cooked: <p>I assume the first output was generated by numpy?<br>
The information is most likely not lost, but the print options in numpy and PyTorch use different defaults for the precision.<br>
Try setting <code>torch.set_printoptions(10)</code> (or even higher) and check the output again.</p>
How to get a float power of a negative float number？

username: I-Love-U
cooked: <pre><code class="lang-python">-0.9955 ** 0.0448
Out[11]: -0.9997979654463205

torch.Tensor([-0.9955]) ** torch.Tensor([0.0448])
Out[16]: tensor([nan])
</code></pre>
<p>I want to use <code>pytorch</code> get the number <code>-0.9997979654463205</code>.  Thanks.</p>

username: ptrblck
cooked: <p>The plain Python code calculates:</p>
<pre><code class="lang-python">-(0.9955 ** 0.0448)
&gt; -0.9997979654463205
</code></pre>
<p>While you are trying to calculate</p>
<pre><code class="lang-python">(-0.9955) ** 0.0448
&gt; (0.989911956308302+0.14025081271947198j)
</code></pre>
<p>in PyTorch, which is a complex number.<br>
If you want the first output in PyTorch, just use:</p>
<pre><code class="lang-python">-1.0 * torch.tensor([0.9955]) ** torch.tensor([0.0448])
&gt; tensor([-0.9998])
</code></pre>

username: I-Love-U
cooked: <p>Oh, thanks.</p>
<p>It’s my problem that I didn’t notice the priorities of <code>-</code> and <code>**</code> when writing the demo, but your second demo is really what I’m trying to say, thank you very much!</p>
<p>If I write demo correctly, I guess I would understand the problem.</p>
Implement dataparallel on big vocab is much slower than single gpu

username: k3833653
cooked: <p>I’m running a bilstm + crf model. If I chose vocab_size of 10000, batch_size = 512, the forward speed is equal to use dataparalle , gpu = 4, batch_size = 2048. However, if I chose vocab_size of 0.5million, the forward speed on sigle gpu is 8 times faster than dataparallel, gpu = 4, batch_size = 2048. The speed on two case of single gpu is similar. I don’t know why . I have already set embedding.require_grad = False</p>

username: ptrblck
cooked: <p>If your module has a lot of parameters, wrapping it in <code>nn.DataParallel</code> might create a slowdown since these parameters have to be scattered.<br>
Have a look at the <a href="https://github.com/pytorch/examples/blob/97304e232807082c2e7b54c597615dc0ad8f6173/imagenet/main.py#L161" rel="nofollow noopener">ImageNet example</a>, where only <code>model.features</code> is wrapped in data parallel. This is also explained in the <a href="https://arxiv.org/abs/1404.5997" rel="nofollow noopener">One weird trick paper</a>.</p>
<p>In your use case you could try to leave the embedding layer on a single device.</p>

username: k3833653
cooked: <p>Thanks for your advice. If l leave embedding layer on a single device, like gpu:0, the other model parameter on gpu:1,  these data can be update at the same time?</p>

username: ptrblck
cooked: <p>Yes, as you can see in the ImageNet example, you can just use <code>DataParallel</code> on a certain part or your model, which will work fine.<br>
In fact you can split your model in any way, e.g. leaving one part on the CPU, another on a single GPU, and again another in <code>DataParallel</code>.<br>
As long as you push the activations and inputs to the appropriate device, everything will work fine.<br>
Otherwise, you’ll get some errors. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: k3833653
cooked: <p>Thanks, I have solved the case after following your advice:wink:</p>
Get RuntimeError when use torch.onnx.export

username: dan_wu
cooked: <p>pytorch version is 1.1.0</p>
<p>error as follow:</p>
<p>Traceback (most recent call last):<br>
File “/xxx/0_train_base.py”, line 295, in <br>
process(sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4])<br>
File “/xxx/0_train_base.py”, line 219, in process<br>
torch.onnx.export(model, dummy_input, onnx_save_path, verbose=True)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/<strong>init</strong>.py”, line 25, in export<br>
return utils.export(*args, **kwargs)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/utils.py”, line 131, in export<br>
strip_doc_string=strip_doc_string)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/utils.py”, line 363, in _export<br>
_retain_param_name, do_constant_folding)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/utils.py”, line 278, in _model_to_graph<br>
_disable_torch_constant_prop=_disable_torch_constant_prop)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/utils.py”, line 188, in _optimize_graph<br>
graph = torch._C._jit_pass_onnx(graph, operator_export_type)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/<strong>init</strong>.py”, line 50, in _run_symbolic_function<br>
return utils._run_symbolic_function(*args, **kwargs)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/utils.py”, line 589, in _run_symbolic_function<br>
return fn(g, *inputs, **attrs)<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/symbolic.py”, line 130, in wrapper<br>
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]<br>
File “/xxx/lib/python2.7/site-packages/torch/onnx/symbolic.py”, line 97, in _parse_arg<br>
raise RuntimeError(“Unexpected node type: {}”.format(value.node().kind()))<br>
RuntimeError: Unexpected node type: onnx::Div</p>
<p>How can I fix this bug?</p>
Extract feature from a model generated using Engine

username: train_1
cooked: <p>I am trying to extract feature using the code of <a href="https://github.com/szagoruyko/wide-residual-networks" rel="nofollow noopener">https://github.com/szagoruyko/wide-residual-networks</a>.<br>
I have an Engine model. I have model which looks like:<br>
def f(input, params, mode):<br>
x = F.conv2d(input, params[‘conv0’], padding=1)<br>
g0 = group(x, params, ‘group0’, mode, 1)<br>
g1 = group(g0, params, ‘group1’, mode, 2)<br>
g2 = group(g1, params, ‘group2’, mode, 2)<br>
o = F.relu(utils.batch_norm(g2, params, ‘bn’, mode))<br>
o = F.avg_pool2d(o, 8, 1, 0)<br>
o = o.view(o.size(0), -1)<br>
o = F.linear(o, params[‘fc.weight’], params[‘fc.bias’])<br>
return o<br>
It is a function instead of a class.</p>
<p>Lets assume I save a trained model as torch.save(dict( params=params, epoch=t[‘epoch’], optimizer=state[‘optimizer’].state_dict()), path ). I want to reuse this model for extracting features later.</p>
<p>Since it is not a class, I can not create an object out of it and initialize with pretrained parameters. So my solution is to change the function definition (f_1) above(discard last two layers). Then call<br>
f_1, params = resnet(opt.depth, opt.width, num_classes)<br>
state_dict = torch.load(path_to_saved_model)<br>
params_tensors = state_dict[‘params’]<br>
for k, v in params.items():<br>
v.data.copy_(params_tensors[k])</p>
<p>This initializes the model f_1 with the subset layers parameter from pretrained model.<br>
I am okay upto this. But then I have no idea how do I get prediction(features) out of this function (with Engine included in this model , it becomes more difficult for me). Just to clarify again: I want features from a pretrained model. Sorry about long post and maybe this may be a naive post to experts here.<br>
The basic training model for Engine is here : <a href="https://github.com/szagoruyko/wide-residual-networks/blob/master/pytorch/main.py" rel="nofollow noopener">https://github.com/szagoruyko/wide-residual-networks/blob/master/pytorch/main.py</a></p>
Inception score calculation

username: electronicshelf
cooked: <p>I am trying to compute the Inception core for my GAN network.<br>
but i have observed something very interesting. Why are we using only <code>X</code> for the calculation i taught KL divergence was suppose to use two variables  main distribution <code>P(x)</code> and  predicted Model <code>Q(x)</code> .</p>
<pre><code class="lang-auto">def inception_score(X):
    kl = X * ((X + eps).log() - (X.mean(0) + eps).log().expand_as(X))
    score = np.exp(kl.sum(1).mean())
    return score
</code></pre>
<p>but it looks like the KL divergence is between <code>X</code> and its mean <code>X.mean(0)</code> i.e if <code>X</code> is the the generated image, then what’s actually going on?</p>
Can I get some help for my depth estimation model?

username: Mehdi
cooked: <p>Hi !</p>
<p>I’m trying to train a Resnet-based model (pretrained  on Imagenet) on the NYU dataset for depth estimation. I’m only using the small, annotated part of the NYU dataset (which is 1.5k images) but the results are very blurry. I can’t figure out what’s wrong with it. I’m using a loss function composed of the mse, an image gradient loss and a normal surface loss but so far, not luck. I know 1.5k datapoint is rather small, but I wanted to see whether I could overfit before focusing on generalization.</p>
<p>I’ve made a colab notebook and I was wondering if someone could take a look and help me out ?<br>
<a href="https://colab.research.google.com/drive/1o2mG_J1mzAyY2jY7rS5fXEmfrCLkmxKT" class="onebox" target="_blank" rel="nofollow noopener">https://colab.research.google.com/drive/1o2mG_J1mzAyY2jY7rS5fXEmfrCLkmxKT</a><br>
Thanks a lot !</p>
Structure of weight matrix in torch.nn.Embedding layer

username: Mahsa_Afsharizade
cooked: <p>I have a text dataset that there are scores for all of its sentences. I want to do a sentence regression task. I have word embedding vectors for each of words in the sentences. Now I want to use Pytorch for defining an embedding layer. I now that I should use of these line of code:</p>
<pre><code class="lang-auto">import torch as nn
embed=nn.Embedding(num_embeds,embed_dim)
#pretrained weight is a numpy matrix of shape(num_embeds,embed_dim)
embed.weight.data.copy_(torch.from_numpy(pretrained_weight))
</code></pre>
<p>But I don’t know what is the order of rows in pretrained_weight. Is it the same order the sequent words appear in the sequent sentences? Does it contain duplicate words?</p>

username: crcrpar
cooked: <p>Hi</p>
<p>First, as a side note, <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/sparse.py#L127-L158" rel="nofollow noopener"><code>nn.Embedding.from_pretrained</code></a> is convenient if you have pretrained weights.</p>
<p>The order of rows in pretrained_weights is dependent of the vocabulary-to-ID mappings.<br>
If you use <code>torchtext</code> library, you can see the order in <code>vocab</code> object and there would be no duplicates.</p>

username: Mahsa_Afsharizade
cooked: <p>Hi. I use a method for constructing embedding matrix based on vacab words indexes:</p>
<pre><code class="lang-auto">def load_embeddings(CorpusPretrainedEmbdDict,word2idx,embedding_dim):
    embeddings=np.zeros((len(word2idx),embedding_dim))
    for word,index in word2idx.items():
        vector = np.array(CorpusPretrainedEmbdDict[word], dtype='float32')
        embeddings[index]=vector
        return embeddings
</code></pre>
<p>I also construct word2idx to get indexes of all the words in my vocabulary. Then I use <em>load_embeddings</em> for constructing embedding matrix for the words in my vocabulary based on word2idx.</p>
<pre><code class="lang-auto">word2idx = {word: idx for idx, word in enumerate(DUC_vocab)}
#create embedding matrix for our vocab (based on word2idx)
my_embeddings=load_embeddings(CorpusPretrainedEmbdDict,word2idx,n_dim)
#Create an embedding layer
embed=nn.Embedding(len(word2idx),n_dim)
#feed our pretrained word vectors to embedding layer
embed.weight.data.copy_(torch.from_numpy(my_embeddings))
</code></pre>
<p>Could you please guide me this way is true? Thanks</p>

username: crcrpar
cooked: <p>Great, and I think this will work as you expect.</p>

username: Mahsa_Afsharizade
cooked: <p>Sorry, I have another question. I want to do a text classification task. I have prepared embedding layer as discussed above. But the sentences in the corpus have different lengths and I think that I need to zero padding. But I’m confused if I want to do zero padding, the input for embedding layer that we discussed about it in previous posts will be wrong?<br>
thanks.</p>

username: Jumonji
cooked: <p>When I look at the actual weight matrix in the Embedding object, it has one more entry than the length of my vocabulary. Is the first one (or last one) for unknown words?</p>
How to Static libraries for pytorch to execute pytorch c++ code

username: dinesh_mareedu
cooked: <p>Hi, I have loaded the model with pytorch c++ api. And compiled and executed with linking the dynamic libraries. Is there any way to create static libraries for pytorch and execute the code with them? Hope I get the solution of this.</p>

username: tom
cooked: <p>If you set the CMake flag <code>BUILD_SHARED_LIBS</code> to off, you will get static libraries.</p>
<p>Best regards</p>
<p>Thomas</p>

username: dinesh_mareedu
cooked: <p>Thanks a lot for the quick solution to me. As of now my exe is using dynamic libraries only. If I set BUILD_SHARED_LIBS off, can it able to link the static libraries by  it self and get the succesful execution!!! Can I get the more details about this in any forum…please</p>
HELP! Can we implement parallel computing in the 'forward' function?

username: TonyMaster
cooked: <p>please tell me how to implement parallel running of two pieces of code on the GPU, just like multithreading on cpu?</p>
<pre><code class="lang-auto">def forward(self, x):

y=self.conv2(x)
z=self.conv2(x)
return y+z
</code></pre>
<p>For example, the above code can we execute  <code>y=self.conv2(x)</code>, <code>z=self.conv2(x)</code>   at the same time,?Looking forward to your reply , thanks!!!</p>
F1-Score appears nan value in evaluation phase

username: MariosOreo
cooked: <p>Hello,</p>
<p>I implemented F1_Score by myself instead of using sklearn. <strong>But</strong> in the validation phase and testing phase, it appears <strong>nan</strong> value. I cannot fix the bug, could you please give me some advice? I will very appreciate it!</p>
<p>I’m working on a multi-class segmentation task.</p>
<pre><code class="lang-auto">def _get_stat(self, logits, target, nb_classes):
    with torch.no_grad():
        pred = torch.argmax(logits, dim=1).view(-1)
        target = target.view(-1)
        pixel_counter = torch.zeros(nb_classes).to(logits.device)
        acc = torch.zeros(nb_classes).to(logits.device)
        f1 = torch.zeros(nb_classes).to(logits.device)
        for k in range(0, nb_classes):
            # tp + fp
            pred_inds = pred == k
            # tp + tn
            target_inds = target == k
            # tn + fn
            non_pred_inds = pred != k
            # tn + fp
            non_target_inds = target != k
            # tp
            interection = pred_inds[target_inds].long().sum().float()
            # tn
            non_interection = non_pred_inds[non_target_inds].long().sum().float()
            # fn + fp
            denominator = non_pred_inds.long().sum().float() + non_target_inds.long().sum().float() - 2*non_interection
            pixel_counter[k] = target_inds.long().sum().float()
            acc[k] = interection
            f1[k] = ((2 * interection) / (2 * interection + denominator + 1e-10))
     return (acc[:nb_classes-1].sum() / (pixel_counter[:nb_classes-1].sum() + 1e-10)), f1
</code></pre>
<p>The <strong>nan value</strong> also appears in <strong>mean_f1_score</strong>, I calculate it by:</p>
<pre><code class="lang-auto"># the last class should be ignored
.mean_f1_score =f1_score[0:nb_classes-1].sum() / (nb_classes-1)
</code></pre>
<p>P.S. Why I don’t use the metrics function from sklearn is I think the transform from tensor to numpy(from GPU to CPU) is the time-wasting operation, and it will make GPU usage lower.<br>
Correct me please if I am wrong.</p>
<p>Thanks in advance!</p>
Custom loss function using argsort

username: ir1979
cooked: <p>I am new to pyTorch. I want to implement a custom loss function. The output <code>y</code> of my neural network is a one-dimensional tensor (i.e, the output of an embedding(n,1) layer). The loss-function applies some computation on original data sorted based on <code>argsort(y)</code>. Here is my function, but it produces <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. Would you please help me to implement the function:</p>
<pre><code class="lang-auto">def myCustomLossFunc3(data_nd, y):
    path = torch.argsort(y)  
    data_sorted = data_nd[path]
    # some computation on data_sorted
    return torch.norm(data_sorted[-1]-data_sorted[0])
</code></pre>

username: LeviViana
cooked: <p>Try printing <code>requires_grad</code> for each tensor in your loss. For instance, if <code>data_nd.requires_grad</code> if False, then it won’t work.</p>

username: ir1979
cooked: <p>Thank you for your help.</p>
TorchScript segfault when using recursive custom classes

username: sidms
cooked: <p>Hi,</p>
<p>Following the comments in <a href="https://discuss.pytorch.org/t/is-there-support-of-recursive-custom-classes-like-tree-in-tree-lstm/47418">this question</a>, I was trying to create a class that has attributes of the same class type as shown below:</p>
<pre><code class="lang-auto">import torch
from typing import Dict, List, Optional


@torch.jit.script
class JSONValue(object):
    def __init__(self):
        self.value = torch.jit.annotate(Optional[JSONValue], None)

class TestModule(torch.jit.ScriptModule):
    @torch.jit.script_method
    def forward(self, input):
        # type: (JSONValue) -&gt; JSONValue
        return input

m = TestModule()

m.save("test.pt")
</code></pre>
<p>But this code results in a segfault with message <code>Segmentation fault: 11</code>. Any thoughts on what could be wrong?</p>
<p>Thanks!</p>

username: Michael_Suo
cooked: <p>We do not actually currently support recursive class definitions. I didn’t remember that in the previous thread (will comment there as well). <a href="https://github.com/pytorch/pytorch/pull/21842" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/21842</a> improves the error message in this case.</p>
<p>We will likely support it soon (before the next release) but for now it will not work.</p>
Why can't backward() calculate whole Jacobians directly?

username: piotrekp1
cooked: <p>Hi,</p>
<p>I’m wondering - why backward() can’t be calculated for vectors (Therefore calculating whole Jacobian instead of Jacobian-vector products)?</p>
<p>Interface would be straightforward - just adding another dimension to tensor, so I guess it must be for efficiency reasons, but I’m curious what exactly happens here.</p>

username: tom
cooked: <p>It’s plainly not implemented and currently not considered in scope (other frameworks such as JAX seem to provide more general derivatives). For some operations you can trick autograd using broadcasting. There is some related discussion in an <a href="https://github.com/pytorch/pytorch/issues/7786" rel="nofollow noopener">old issue</a> and less related in the forum thread linked there. Another way might be to implement it yourself based on Torch IR graphs or to implement dual numbers in tensors or so.</p>
<p>Best regards</p>
<p>Thomas</p>
Sampling a Multinomial distribution multiple times

username: lorenzo_fabbri
cooked: <p>I’d like to sample a multinomial distribution multiple times (10) with 6 possible outcomes. What is the difference, if any, between the following two commands?</p>
<pre><code class="lang-auto">print(Multinomial(1, torch.ones(6)).sample(torch.Size([10])))
print(Multinomial(10, torch.ones(6)).sample())
</code></pre>
Access tensor element in torch c++

username: tiger17
cooked: <p>Hello,</p>
<p>How do I access tensor element in torch C++ that doesn’t disconnect the graph ? I need to backprop through element access. Is using accessor&lt;&gt; the right way?</p>
<p>thank you!</p>

username: tom
cooked: <p>accessors are not for Autograd but to allow efficient raw memory access, so that doesn’t sound like a good match for what you ask.<br>
Without knowing more (how many elements, what do you want to do with them) it’s hard to tell, but plainly indexing Tensors works well until you hit performance issues. These can occur because indexing will cause the subtensors to be instantiated and connected to the autograd graph, which is a source of overhead.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>Best regards</p>
<p>Thomas</p>
CNN-RNN implementation for video classification in sequetial image

username: Sohel_Rana
cooked: <p>Question 1: Which way should I follow for training a sequence of image with CNN-LSTM architecture? Should I extract all sequential image features by CNN and pass those features to LSTM or should I feed one image with one CNN and one LSTM?</p>
<p>Let we will use a fixed length LSTM<br>
way 1:  a[0] = CNN(image1), a[1] = CNN(image2)…so on final_output = LSTM(a)<br>
way 2: hid1 = LSTM1(CNN(image1)), hid2 = hid1+LSTM2(CNN(image2)) and so on</p>
<p>Question 2:<br>
Do you have any optimized implementation of sequential CNN-RNN implementation where sequence of image is used as CNN input. I tried to implement it with VGG16 net. But its getting out-of-memory.</p>

username: riwi
cooked: <p>Hi Sohel.</p>
<p>I’m starting on the same task (seq. of images into a CNN-LSTM model).<br>
Did you solve your problems and got a working model? Or does anyone have a starting point/good link?</p>
CNN-LSTM model implementation issues

username: coderinuse
cooked: <p>Hi, I’m trying to implement CNN-LSTM model where I have a sequence of images that I need to get spatial data from using CNN and send it to my LSTM layer. Here is the code of the models that I have written. Is it the write implementation of the idea? I’m not able to properly train the model that is why I’m asking the question.</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, layers, c):
        super(CNN, self).__init__()
        self.layers = nn.ModuleList([
            nn.Conv2d(layers[i], layers[i + 1], kernel_size=3, stride=2)
            for i in range(len(layers) - 1)])
        self.drop = nn.Dropout(p=0.1)
        self.pool = nn.AdaptiveMaxPool2d(1)
        self.out = nn.Linear(layers[-1], c)
    
    def forward(self, x):
        for l in self.layers: x = F.relu(l(x))
        x = self.drop(x)
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        return F.log_softmax(self.out(x))

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        
        self.hidden_size = hidden_size
        self.conv = CNN([3, 20, 40, 80], hidden_size)
        self.i2h = nn.LSTM(hidden_size, hidden_size)
        self.i2o = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = self.conv(input)
        output, hidden = self.i2h(torch.unsqueeze(input, 0), hidden)
        output = torch.squeeze(output, 0)
        output = self.i2o(output)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return (Variable(torch.zeros(1, 1, self.hidden_size).cuda()),
                Variable(torch.zeros(1, 1, self.hidden_size).cuda()))
</code></pre>
<p>My data is the ellipses (one type of ellipse) with parameters like colour, width, height and with two different movement patterns that are described by parameters like x, y change. Parameters are randomly generated using Gaussian normal distribution. Each move has different parameters for Gaussian distribution. I have 40.000 (20.000 for each) 64x64 images. Still, I’m not able to train the model. Maybe there is too much white space on the images? Therefore, I get overfitting?!</p>

username: riwi
cooked: <p>Hi Rasul.</p>
<p>I’m starting on the same task (seq. of images into a CNN-LSTM model).<br>
Did you solve your problems and got a working model?</p>
Where is CMAKE_PREFIX_PATH supposed to point if I'm building pytorch without conda?

username: divinho
cooked: <p>Not really much else to say except bit annoyed the doc just assumes you use conda but I don’t really care either way just filling up the 20 char minimum.</p>

username: divinho
cooked: <p>Seems to be</p>
<pre><code class="lang-auto">-DCMAKE_PREFIX_PATH=$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())'
</code></pre>
GANs - G: multiple outputs / D: Single Loss

username: Timos_Korres
cooked: <p>I’m trying to train a generator to learn how to output a set of k items [x1,x2,x3] out of K total items. Each item is represented as a fully connected layer with size equal to K (Choosing one of K items). The discriminator must recognize whether these specific items are real or not.</p>
<p>The number of output items is variable, is a hyperparameter So, I have two questions:</p>
<ol>
<li>
<p>How can I return variable number of outputs (hyperparameter) from the generator in order to compute the gradient separately? -&gt; My solution is to return them as a big tuple. Is this enough to compute the gradients correctly?</p>
</li>
<li>
<p>The loss is computed for the whole set of items and not for each one of them. That means I only have one loss function. How can compute the gradients for each one of them?</p>
</li>
</ol>
<p>Any suggestions are also welcome</p>
Transferring pytorch model to caffe2 using onnx gives attribute error

username: Shbnm21
cooked: <p>I have converted a pytorch model to onnx format and then to caffe2.</p>
<p>When I am verifying the numerical correctness of the model it gives attribute error<br>
AtrributError:‘nonetype’ object has no attribute ‘data’</p>
<p>I followed instructions from pytorch documentation.<br>
np.testing.assert_almost_equal(torch_model1.data.cpu().numpy(),c2_out,decimal=3)</p>
Combine serveral datasets

username: Miriam_Slw
cooked: <p>Hi,<br>
we want to train a DNN on a dataset, which consists of multiple datasets. Each dataset is a folder and has it’s own label. Now we have to combine these datasets, but don’t know, how to do this.<br>
Folder structure:<br>
root consists of → multiple subfolders always consists of → multiple sub-subfolders always consists of → pictures<br>
Our idea (example with two datasets):<br>
We load two datasets as trochvision.ImageFolder and combinde them with ConcatDataset.</p>
<p>datadir = “…/data/dir1…”<br>
datadir2= “…/data/dir2…:”<br>
train_transforms = transforms.Compose([transforms.Resize(224),<br>
transforms.ToTensor(), ])<br>
train_data = datasets.ImageFolder(datadir,<br>
transform=train_transforms)<br>
trainloader = torch.utils.data.DataLoader(train_data,<br>
batch_size=64)<br>
train_data = ConcatDataset((datasets.ImageFolder(datadir2),datasets.ImageFolder(datadir)))</p>
<p>But now, we are not able to get the labels via “print(trainloader.dataset.classes)” or<br>
“for values,labels in trainloader:<br>
print(labels)”<br>
Is there a way to combine multiple datasets while keeping the labels?</p>
<p>We would be very greatful for an answer!<br>
Thanks!!</p>
<p>(this maybe helps to understand our folder structure):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e" title="FrageNetz"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e_2_375x500.jpeg" alt="FrageNetz" data-base62-sha1="xsSgkXQJgZiza52i9fhFTcq3nk2" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ea8b57248e3c1cd186d75f51d9bbcb8a378bc77e_2_750x1000.jpeg 2x" data-dominant-color="A39A99"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FrageNetz</span><span class="informations">1200×1600 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Libtorch: A custom (e.g. NOT mnist) Image dataset in C++

username: dambo
cooked: <p>Dear All,<br>
I am trying to migrate the python based <strong>ImageList</strong> and <strong>PilImageList</strong> from here to Libtorch C++:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/vita-epfl/openpifpaf/blob/master/openpifpaf/datasets.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/vita-epfl/openpifpaf/blob/master/openpifpaf/datasets.py" target="_blank" rel="nofollow noopener">vita-epfl/openpifpaf/blob/master/openpifpaf/datasets.py</a></h4>
<pre><code class="lang-py">import copy
import logging
import os
import torch.utils.data
import torchvision
from PIL import Image

from . import transforms, utils


ANNOTATIONS_TRAIN = 'data-mscoco/annotations/person_keypoints_train2017.json'
ANNOTATIONS_VAL = 'data-mscoco/annotations/person_keypoints_val2017.json'
IMAGE_DIR_TRAIN = 'data-mscoco/images/train2017/'
IMAGE_DIR_VAL = 'data-mscoco/images/val2017/'


def collate_images_anns_meta(batch):
    images = torch.utils.data.dataloader.default_collate([b[0] for b in batch])
    anns = [b[1] for b in batch]
    metas = [b[2] for b in batch]
</code></pre>

  This file has been truncated. <a href="https://github.com/vita-epfl/openpifpaf/blob/master/openpifpaf/datasets.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Most of the examples I found in C++ are for MNIST and not a custom dataset.<br>
Has anyone done something like this before? or is there a build in ImageFolder in C++?</p>
<p>Thanks,</p>

username: dambo
cooked: <p>Anyone … breaking my head on this.</p>
<p>This is what I am looking for in C++:</p>
<pre><code class="lang-auto">class ImageList(torch.utils.data.Dataset):
    def __init__(self, image_paths, preprocess=None, image_transform=None):
        self.image_paths = image_paths
        self.image_transform = image_transform or transforms.image_transform
        self.preprocess = preprocess

    def __getitem__(self, index):
        image_path = self.image_paths[index]
        with open(image_path, 'rb') as f:
            image = Image.open(f).convert('RGB')

        if self.preprocess is not None:
            image = self.preprocess(image, [])[0]

        original_image = torchvision.transforms.functional.to_tensor(image)
        image = self.image_transform(image)

        return image_path, original_image, image

    def __len__(self):
        return len(self.image_paths)
</code></pre>
<p>Thanks.</p>
Constructing PyTorch's CUDA tensor from C++ with image data already on GPU

username: TomLiftoff
cooked: <p>Hi!</p>
<p>I need to use PyTorch model from existing C++/OpenCV based application. All images are processed with OpenCV’s CUDA modules. Currently, I have to copy all the data back to CPU and use boost::python converters to make NumPy array from it, which I can use to construct PyTorch Tensor object. It works, but introduces major slowdowns (of course).</p>
<p>Is there a way to prepare data for PyTorch backend in such a way, that all the data would be always on GPU?</p>
<p>I know, that PyTorch shares its backend with original Torch (to some extent), so is this <a href="https://github.com/VisionLabs/torch-opencv" rel="nofollow noopener">project</a> of any use?</p>
<p>Looking forward for any clues</p>

username: smth
cooked: <p>you can pass in the pointer of an existing pytorch <code>torch.cuda.FloatTensor</code> into C++ side and issue a cudaMemCpyAsync from OpenCV buffer to <code>torch.cuda.FloatTensor</code> buffer.<br>
The pointer of the Tensor can be obtained with:</p>
<pre><code class="lang-auto">a = torch.cuda.FloatTensor(3, 640, 480) # new buffer of 3x640x480
pointer = a.data_ptr()
# pass this pointer into C/C++ side of your OpenCV pipeline
# copy the OpenCV buffer into CUDA
</code></pre>
<p>Lastly, when doing CUDA with datasets (i.e. multiprocessing with CUDA), you have to use python3 and spawn_server: <a href="http://pytorch.org/docs/notes/multiprocessing.html#sharing-cuda-tensors">http://pytorch.org/docs/notes/multiprocessing.html#sharing-cuda-tensors</a></p>

username: TomLiftoff
cooked: <p>Whoa. Somehow I’ve overlooked the <code>data_ptr</code> method in docs.</p>
<p>I just wrote a basic <a href="https://gist.github.com/d1ff/0dd933f906b34862a51abf0471b19f9e" rel="nofollow noopener">example</a> illustrating your approach.</p>
<p>Thank you!</p>
How to get indices of multiplied elements pair in matrix multiplication?

username: RadZaeem
cooked: <p>I am doing weight pruning, by sparsifying activations with mask – If the activations have lots of zeros, the weights can be safely zeroed out too.<br>
And I need way to calculate weight parameters sparsity percentage.</p>
<p>Say I have y(x) = ReLU(Wx+b)<br>
x is activations vector from previous layer, and have lots of zero values (due to mask)</p>
<p>From matmul Wx, how do I find elements (and its indices) in W that match with zero-valued activations in x,<br>
So I can sparse W, and count zero values to calculate sparsity percentage.<br>
Does pytorch have such function to do so?</p>
<p>Also how do I do the same for convolution layers (They have filters, filter size, stride, pad, etc)</p>
<p>Thank you</p>
Why does `pin_memory` will increase CPU usage sharply?

username: MariosOreo
cooked: <p>Hello there,</p>
<p>I use the dataloader with <code>pin_memory=True</code>, the CPU usage sharply increases. But I have ever run the same project, the CPU usage is normal. Could you give me some hints about it?</p>
<p>with <code>pin_memory=True</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/7/7038fe675828eb46d50702cd65ed9d34e0099b2a.jpeg" alt="not%20normal" data-base62-sha1="g0LAhLBvF2ADx2uiTSNfZQxp1nc" width="432" height="222"><br>
with <code>pin_memory=False</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/fa6ff9092e799c5e1b76a9f04a819f7191cb7f65.jpeg" alt="normal" data-base62-sha1="zJtgD2ORvtmE4zfx5TVUQjx3uyp" width="438" height="210"></p>
<p>Thanks in advance</p>
<p>Best regards<br>
TTB</p>