akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Load and pre-process NSL_KDD data set

username: arezoo_moradi
cooked: <p>since I am a newbie in python programming and I want to load the data according to the table of the article but I don’t know how to can do categorical training and testing the NSL_KDD dataset into(‘normal’, ‘dos’, ‘r2l’, ‘probe’, ‘u2r’).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af4fc0d9cca11117d0af3f9d1f9af397a90daab9.png" alt="image" data-base62-sha1="p0SqxmV8PaLTpmdvUClcpxdVjGp" width="476" height="491"><br>
I’ve reviewed a lot of code in GateHub to pre-process the NSL_KDD data set to categorize into five groups(‘normal’, ‘dos’, ‘r2l’, ‘probe’, ‘u2r’), but I still haven’t been able to find the code to do it right.<br>
Can anyone help me? I really need help.</p>

username: ptrblck
cooked: <p>This doesn’t seem to be a PyTorch-specific question, and I would recommend to ask on StackOverflow or maybe in the competition discussion board (if they have one) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Based on your <a href="https://discuss.pytorch.org/t/pre-processing-on-nsl-kdd-dataset/80995/9">last post</a> it seems that the scaling is failing due to unexpected values, so you could look into the data and check all values for NaNs, Infs etc.</p>
Sparse attention

username: slavavs
cooked: <p>Has anyone seen the implementation of this architecture on pytorch?<br>
<a href="https://github.com/openai/sparse_attention/blob/master/attention.py" rel="nofollow noopener">https://github.com/openai/sparse_attention/blob/master/attention.py</a></p>

username: dav-ell
cooked: <p>I haven’t, but I’d definitely like to see one!</p>
<p>Any updates on this?</p>

username: ptillet
cooked: <p>FOr anyone who is interested, there is a PyTorch implementation of blocksparse attention here: <a href="http://github.com/ptillet/torch-blocksparse/" rel="nofollow noopener">github.com/ptillet/torch-blocksparse/</a></p>
Forward in Model Definition

username: NoobCoder
cooked: <p>Hello,</p>
<p>I am writing a model in Pytorch. At the last layer, I have a fully connected that gives 3 points (let’s assume {x, y, h} ).</p>
<p>Now, I want to add a function in a <code>forward</code> that it creates a tensor of size 5*5 which has the value of axes in its.  (for example tensor[1,1]=[1,1] ). I want to compute the euclidean distance of this tensor from the {x,y} which is the output of the fully connected layer from this <a href="https://discuss.pytorch.org/t/efficient-distance-matrix-computation/9065/5">post</a>. My question is that how to do so that in the backward, I do not occur any problem??</p>
<p>Here is the sample code for the model.</p>
<pre><code class="lang-auto">
def euclidean_dist(tensor1, tensor2):
     return distnace


class MyModel(): 

     def __init__():
            
          some convulotion layers...
          self.fc = nn.Linear(num_features, 3)

     def forward(in):

         .
         .
         (x, y, h) = fc(in)

         # Create the tensor with the axes value
         temp = torch.tensor([[a, b] for a in range(5) for b in range(5)])
         
         center = torch.tensor([[x,y]])

         # Now compute the distance between this and points
         distance = euclidean_distance(temp, centers)

         out = distance.view(5,5)
         out = out * h

         return out
</code></pre>
<p>Is it right to do this in this way?</p>
<p>Thanks.</p>
Itorch notebook kernel dies immediatly

username: Dr_Dumbenstein
cooked: <p>Hi, everyone. I recently decided to switch from tensorflow to pytorch, so i am following some tutorials.<br>
Yesterday i learned that torch is a must learn so i studied Lua for a while then installed Torch. I also decided to use itorch in order to code in a notebook.<br>
I had this problem, whenever i open the notebook and choose itorch i goes dead and no matter how much i restart it the problem still occur!</p>

username: albanD
cooked: <p>Hey,</p>
<p>This forum is for pytorch, not torch7.<br>
I’m afraid not many people here will be able to help you with lua/iTorch related issues <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Dr_Dumbenstein
cooked: <p>Yes, i tried my chance… Thnx <a class="mention" href="/u/alband">@albanD</a></p>
Pytorch combine two models into a single class and access their parameters from the parent class

username: jiwidi
cooked: <p>Hi!</p>
<p>So I have a model in pytorch that looks like this:</p>
<pre><code class="lang-python">class LAS(nn.Module):
    def __init__(self, listener, speller):
        super(LAS, self).__init__()
        self.listener = listener
        self.speller = speller

    def forward(self,input):
        input = self.listener(input)
        input = self.speller(input)
        return input

</code></pre>
<p>Where listener and speller are two instances of a class inheriting from nn.Module as well:</p>
<pre><code class="lang-auto">class Speller(nn.Module) #Each of them have their layers and neurons but I will omit them for clarity
class listener(nn.Module)
</code></pre>
<p>I know I can do LAS.speller.parameters() to access the speller parameters but I would like to combine both so I can pass then to my optimizer like this:</p>
<pre><code class="lang-python">optimizer = torch.optim.Adam(LAS.parameters(),
   lr=params["training"]["lr"],
)
</code></pre>
<p>Instead that doing it like this:</p>
<pre><code class="lang-python">optimizer = torch.optim.Adam(
    [{"params": LAS.listener.parameters()}, {"params": LAS.speller.parameters()}],
    lr=params["training"]["lr"],
)
</code></pre>
<p>When I do LAS.parameters() will it be equal to actually passing both <code>listener</code> &amp; <code>speller</code> parameters to the optimizer separately?</p>

username: crcrpar
cooked: <aside class="quote no-group" data-username="jiwidi" data-post="1" data-topic="83000">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiwidi/48/27512_2.png" class="avatar"> jiwidi:</div>
<blockquote>
<p>When I do LAS.parameters() will it be equal to actually passing both <code>listener</code> &amp; <code>speller</code> parameters to the optimizer separately?</p>
</blockquote>
</aside>
<p>In your case, yes in short. But passing parameters separately enables you to assign different initial learning rate per model as described <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="noopener nofollow ugc">here</a>.</p>
Working of Weighted Softmax Attention Layer

username: Vishu_Gupta
cooked: <p>I am trying to understand a graph neural network code which has implemented a weighted attention layer as follows:</p>
<pre><code class="lang-auto">class WeightedAttention(nn.Module):
    """
    Weighted softmax attention layer
    """
    def __init__(self, gate_nn, message_nn, num_heads=1):
        """
        Inputs
        ----------
        gate_nn: Variable(nn.Module)
        """
        
        super(WeightedAttention, self).__init__()
        self.gate_nn = gate_nn
        self.message_nn = message_nn
        self.pow = torch.nn.Parameter(torch.randn((1)))

    def forward(self, fea, index, weights):
        """ forward pass """

        gate = self.gate_nn(fea)

        gate = gate - scatter_max(gate, index, dim=0)[0][index]

        gate = (weights ** self.pow) * gate.exp()
                
        # gate = weights * gate.exp()
        # gate = gate.exp()
                
        gate = gate / (scatter_add(gate, index, dim=0)[index] + 1e-13)
        
        fea = self.message_nn(fea)
        
        out = scatter_add(gate * fea, index, dim=0)
                
        return out

    def __repr__(self):
        return '{}(gate_nn={})'.format(self.__class__.__name__,
                                       self.gate_nn)
</code></pre>
<p>I am unable to find anything that explains the working of the weighted attention network. Can someone explain what it is and how the code has implemented it using <code>scatter_max</code> and <code>scatter_add</code>? Also is there any mathematical formulae that can be related to this?</p>
Transformer-Encoder model for binary classification

username: Kartik_Kannapur
cooked: <p>Hello community,</p>
<p>I am working on creating a binary classifier by using the Transformer-Encoder architecture and I have not been able to get the model to learn i.e. the training and validation loss remain nearly equal in every epoch. The code is based on - <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html" rel="nofollow noopener">http://nlp.seas.harvard.edu/2018/04/03/attention.html</a></p>
<pre><code class="lang-auto">class TransformerModel(nn.Module):
    """
    """   
        
        
    def __init__(self, n_token, n_dim_model, n_head, n_hidden, n_blocks, dropout=0.5):
        super(TransformerModel, self).__init__()
        
        # #Multi Headed Attention Layer
        self_attention = MultiHeadedAttention(n_head, n_dim_model)
        # #Feedforward Layer
        feed_forward = FeedForwardLayer(n_dim_model, n_hidden, dropout)
        # #Positional Encoding
        positional_encoding = PositionalEncoding(n_dim_model, dropout)
        
        encoder_layer = EncoderLayer(n_dim_model, copy.deepcopy(self_attention), copy.deepcopy(feed_forward), dropout)
        self.encoder = Encoder(encoder_layer, n_blocks)

        embedding = Embeddings(n_dim_model, n_token)
        self.src_embed = nn.Sequential(embedding, copy.deepcopy(positional_encoding))
        
        # Fully-Connected Layer
        self.fc = nn.Linear(n_dim_model, 2)

    def forward(self, x):
        
        # # x dimension[k, batch_size = 64]
        embedded_sents = self.src_embed(x.permute(1,0))        
        encoded_sents = self.encoder(embedded_sents)
        
       
        final_feature_map = encoded_sents[:,-1,:] 
        final_out = self.fc(final_feature_map) 
        
        return final_out 
    
</code></pre>
<p>I am unsure if this architecture is correct. Would really appreciate any suggestion on this!<br>
Thank you</p>
How to transform a 2D and index tensors for torch.nn.utils.rnn.pack_sequence

username: Fed03
cooked: <p>I have a sequences collection in the following form:</p>
<pre><code class="lang-python">sequences = torch.tensor([[2,1],[5,6],[3,0])
indexes = torch.tensor([1,0,1])
</code></pre>
<p>that is, the sequence <code>0</code> is made of just <code>[5,6]</code>, and the sequence <code>1</code> is made of <code>[2,1] , [3,0]</code>. Mathematically <code>sequence[i] = { sequences[j] such that i = indexes[j] }</code></p>
<p>I need to feed these sequences into an LSTM. Since these are variable-length sequences, pytorch documentation states to use something like <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pack_sequence" rel="nofollow noopener"><code>torch.nn.utils.rnn.pack_sequence</code></a>.</p>
<p>Sadly, this method and its like want, as input, a list of tensors where each of them is a <code>L x *</code>, with L being the length of the single sequence.</p>
<p>How can build something that can be fed into a pytorch LSTM?</p>
<p>P.s. throughout the code I work with these tensors using <code>scatter</code> and <code>gather</code> functionalities but I can’t find a way to use them to achieve this goal.</p>
<p>p.p.s I posted this question also on stackoverflow<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/62029901/how-to-transform-a-2d-and-index-tensors-for-torch-nn-utils-rnn-pack-sequence" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1671230/fed03" target="_blank" rel="nofollow noopener">
    <img alt="Fed03" src="https://www.gravatar.com/avatar/c9e6e0bccad4220b2da67db95d875e5c?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/62029901/how-to-transform-a-2d-and-index-tensors-for-torch-nn-utils-rnn-pack-sequence" target="_blank" rel="nofollow noopener">How to transform a 2D and index tensors for torch.nn.utils.rnn.pack_sequence</a>
</h4>

<div class="tags">
  <strong>python, numpy, machine-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1671230/fed03" target="_blank" rel="nofollow noopener">
    Fed03
  </a>
  on <a href="https://stackoverflow.com/questions/62029901/how-to-transform-a-2d-and-index-tensors-for-torch-nn-utils-rnn-pack-sequence" target="_blank" rel="nofollow noopener">07:28PM - 26 May 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Dataset with dirty and clean data separate

username: gildebrand
cooked: <p>Hi!</p>
<p>I’m training an auto encoder network to remove “dirt” from images. For this I have two folders: <code>dirty</code> and <code>clean</code>. Currently I load the data like this</p>
<pre><code class="lang-auto">dirty_data = torchvision.datasets.ImageFolder(root='data/dirty', transform=transform)
clean_data = torchvision.datasets.ImageFolder(root='data/clean', transform=transform)

train_dirty_loader = torch.utils.data.DataLoader(dirty_data, batch_size=BATCH_SIZE, num_workers=0, shuffle=False)
train_clean_loader = torch.utils.data.DataLoader(clean_data, batch_size=BATCH_SIZE, num_workers=0, shuffle=False)
</code></pre>
<p>There are images in the <code>dirty</code> and <code>clean</code> folders with the same name (same images, with and without “dirt”). This method of having two separate loaders does work, but comes with a number of issues</p>
<ul>
<li>It’s ugly</li>
<li>I can’t use <code>shuffle=True</code>, since this would make <code>train_dirty_loader</code> and <code>train_clean_loader</code> out of “sync” with each other (training depends on that the clean and dirty images comes in correct order now).</li>
<li>I can’t split the dataset using the <code>random_split</code> function, for same reason as above.</li>
</ul>
<p>How should I solve this?</p>

username: fcorencoret
cooked: <p>You could create your own Dataloader class where each data point contains both the clean and dirty image?</p>
Runntime error when execturing dcgan.cpp

username: neilmehta87
cooked: <p>I am new to libtorch and I was completing the featured example to learn Pytoch C++ frontend.<br>
I downloaded the cmake and dcgan.cpp files from git and was able to cmake it on a cluster using the clang-llvm/11 compilers. I am using libtorch files downloaded from <a href="https://download.pytorch.org/libtorch/nightly/cu100/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/nightly/cu100/libtorch-cxx11-abi-shared-with-deps-latest.zip</a><br>
I am using cuda 10.2.89 and pytorch v1.5.0-gpu.<br>
When I try running the executable, I get the following error<br>
<code>./dcgan: symbol lookup error: ./dcgan: undefined symbol:  _ZN5torch5optim6detail13OptimizerBase15add_param_groupERKNS0_19OptimizerParamGroupE  </code><br>
It appears that the following lines are responsible for this error:<br>
<code>   torch::optim::Adam  generator_optimizer(generator-&gt;parameters(),  torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));  torch::optim::Adam  discriminator_optimizer(discriminator-&gt;parameters(),  torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));</code></p>
<p>Is there anyone who has previously encountered this error? May I please request your help regarding the same?<br>
Thank you</p>

username: neilmehta87
cooked: <aside class="quote no-group" data-username="neilmehta87" data-post="1" data-topic="82557">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neilmehta87/48/25835_2.png" class="avatar"> neilmehta87:</div>
<blockquote>
<p>am new to libtorch and I was completing the featured example to learn Pytoch C++ frontend.<br>
I downloaded the cmake and dcgan.cpp files from git and was able to cmake it on a cluster using the clang-llvm/11 compilers. I am using libtorch files downloaded from <a href="https://download.pytorch.org/libtorch/nightly/cu100/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cu100/libtorch-cxx11-abi-shared-with-deps-latest.zip</a><br>
I am using cuda 10.2.89 and pytorch v1.5.0-gpu.<br>
When I try running the executable, I get the following error<br>
<code>./dcgan: symbol lookup error: ./dcgan: undefined symbol:  _ZN5torch5optim6detail13OptimizerBase15add_param_groupERKNS0_19OptimizerParamGroupE  </code><br>
It appears that the following lines are responsible for this error:<br>
<code>   torch::optim::Adam  generator_optimizer(generator-&gt;parameters(),  torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));  torch::optim::Adam  discriminator_optimizer(discriminator-&gt;parameters(),  torch::optim::AdamOptions(2e-4).betas(std::make_tuple (0.5, 0.5)));</code></p>
<p>Is there anyone who has previously encountered this error? May I please request your help regarding the same?<br>
Thank you</p>
</blockquote>
</aside>
<p>Update: I was able to get the example working using libtorchv1.5.0 .</p>
How to use Pytorch Transformer models with Tensorflow.js

username: lazy_like_a_fox
cooked: <p>I’m trying to save a Transformer model so that it can be used with Tensorflow.js. Tensorflowjs_converter can take SavedModel or Keras H5 formats. What’s the best way to do this? Thanks!</p>

username: lazy_like_a_fox
cooked: <p>Bump. I’ve read articles about how I should convert from Pytorch to ONNX then to Tensorflow then Tensorflowjs. Just curious if there’s a more direct way. Thank you!</p>
Backprop gradients w.r.t partial computation graph

username: yanweiw
cooked: <p>Hi everyone -</p>
<p>I am trying to backprop gradients to different parts of a computation graph. I have the following setup:</p>
<pre><code class="lang-auto">out0 = model0(input0)
out1 = model1(out0, input1)
out2 = model1(out0, model2(out0))

loss1 = criterion(out1, ground_truth)
loss2 = criterion(out2, ground_truth)
loss3 = criterion(out2, input1)
</code></pre>
<p>How to achieve the following gradient updates of different networks? Specially, loss2 depends on all three models, but I only want to update model2 w.r.t. loss2.</p>
<pre><code class="lang-auto">(loss1 + loss3).backward() # w.r.t. to all three model parameters
loss2.backward() # w.r.t to only model2.parameters() 
</code></pre>
<p>Thanks!</p>

username: albanD
cooked: <p>I assume there is a typo in your example and model2 is the one used to generate out2 right?</p>
<p>If so, if you a single optimizer, the best way I can see is:</p>
<pre><code class="lang-auto">model2.zero_grad()
loss2.backward(retain_graph=True)
model0.zero_grad(); model1.zero_grad()
(loss1 + loss3).backward()

opt.step()
</code></pre>

username: yanweiw
cooked: <p>Actually, that’s not a typo. <strong>Model1</strong> takes <strong>out0</strong> and the output of <strong>model2(out0)</strong> as input to produce <strong>out2</strong>, so <strong>out2</strong> is dependent on all three model parameters.</p>

username: albanD
cooked: <p>Ho sorry, misread it there.<br>
It doesn’t change my answer though <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Slow batch inference

username: tldrafael
cooked: <p>Hi all,</p>
<p>I’m using a trained LinkNet34 model. I was expecting that doing batch inference of size <code>n</code>  would be faster than doing <code>n</code> times one single inference. Although, most of the time the batch computing shows only a slight advantage over <code>n</code> times single inference.</p>
<p>Am I missing something?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dbc46ebf3f62ba10b94200334ba750e7659b5fa.png" alt="Screenshot from 2020-05-26 15-55-48" data-base62-sha1="b5G96KCU3skEJdNC6BYf0jUQ9jY" width="603" height="299"></p>
<p>I’m only using CPU. I have applied <code>model.eval()</code> before. I also tried <code>model.foward</code> method, and <code>with torch.no_grad()</code> context. They didn’t affect the results.</p>
<p>Thanks in advance,</p>

username: albanD
cooked: <p>Hi,</p>
<p>I guess your model is big enough that relatively small batch size already uses your CPU completely.<br>
This is why you better than linear going from 1 to 4. But then almost exactly linear above.</p>
Pytorch dataset sampler for multi task learning

username: Ratan
cooked: <p>Hello,<br>
I have one main feature extractor x which has two tasks A and B. I have two different datasets(images) d1 and d2 which are highly unbalanced(based on number of batches). say 5000 images in d1 and 100000 in d2.<br>
I need to pass d1 to x+A and then d2 x+B separately and backpropogate<br>
How do i design the data loader to train the network??</p>

username: aRI0U
cooked: <p>I think that the easiest solution would be to use one <code>DataLoader</code> per <code>Dataset</code>.<br>
In fact, if you use only one dataloader from both tasks, the problem is that one batch can be composed of images from d1 and from d2. It is ok to have images from both datasets for your main feature extractor but it becomes painful when you have to separate the produced features for tasks A and B since splitting the batch will produce batches of different lengths, with eventually empty batches, etc.</p>
<p>If you really want to use only one unique dataloader, you can have a look at <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset" rel="nofollow noopener">ConcatDatasets</a>.</p>

username: SANTOSH_S
cooked: <p>(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)</p>
Training multiple models at the same time

username: sawal86
cooked: <p>Is it possible to train multiple models simultaneously?<br>
For instance, suppose by nettwork class is Net.</p>
<p>net1 = Net()<br>
net2 = Net()</p>
<p>Is it possible to train net 1 and net 2 simultaneously?</p>
<p>Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Yes of course. Do you encounter any issue doing this?</p>

username: sawal86
cooked: <p>I don’t know how to do this exactly. Is there any specific package I need to use?</p>
<p>Thanks.</p>

username: albanD
cooked: <p>I think you want to be more precise on what you mean <code>simultaneously</code>.<br>
You can have one optimizer for each model and just train them in one training loop. Either with the same data or not.</p>

username: NeelayS
cooked: <p>Hey <a class="mention" href="/u/alband">@albanD</a>, I’m trying to train 2 models in one training loop on the same data. I have a separate optimizer for each one. However, I get this error -</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>
<p>Aren’t 2 separate computation graphs created for the models ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>If you do two forward calls yes. But if you reuse some computations from one to the other no.<br>
You can pass <code>retain_graph=True</code> to the first backward call to make sure the second will succeed.</p>

username: NeelayS
cooked: <p>What I’m trying to do in the training loop is use the output of one model as the the ground truth label of the other.<br>
Training loop -</p>
<pre><code class="lang-auto">for e in range(epochs):

  epoch_loss_t = 0
  epoch_loss_s = 0

  for i, (data, label) in enumerate(trainloader):

    data, label = data.to(device), label.to(device)

    t_out = teacher(data)
    t_loss = F.cross_entropy(t_out, label)
    t_loss.backward()
    t_optim.step()
    t_optim.zero_grad()
    epoch_loss_t += t_loss.item()

    s_out = student(data)
    s_loss = (1 - alpha) * F.cross_entropy(s_out, label)
    s_loss += alpha * loss_fn(s_out, t_out)
    s_loss.backward()
    s_optim.step()
    s_optim.zero_grad()
    epoch_loss_s += s_loss.item()

</code></pre>
<p>Is there a way I can somehow detach the output of the the first model so that a separate computation graph is created for the second one ? I’m guessing that simply doing a .detach() on the output of the first model should work.</p>

username: albanD
cooked: <p>Yes, doing <code>loss_fn(s_out, t_out.detach())</code> will break the link between the two <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Computation time increases with batch size

username: s_n
cooked: <p>I have written code using all PyTorch functions like torch.mul, torch.sub etc.<br>
With the increase in batch size (for my example 32—&gt;64) the time for the code increases (I am running my code on Titan XP GPUs). I thought PyTorch does computation in parallel.</p>
<p>Is this because I am bottlenecked by the pipelining in GPU?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The increase will be sublinear at the beginning, but once all the cores are fully used, we cannot do more parallelism, so it is expected that the runtime will increase linearly after that point.</p>

username: s_n
cooked: <p>Yeah I kind of observed that and it does make sense. Thanks!</p>
Is Distributed Data Parallel equivalent to "Defence Against the Dark Arts from Harry Potter"

username: bing
cooked: <p>Hi Everyone,<br>
So I am trying to run distributed data-parallel on single machine multiple GPU. I have consulted various tutorials from multiple resources and still facing issues.<br>
I have asked multiple times here at various intervals of time, but no one seems to reply, is it some prohibited technology?<br>
Last time, I asked this - <a href="https://discuss.pytorch.org/t/use-distributed-data-parallel-correctly/82500" class="inline-onebox">Use Distributed Data Parallel correctly</a> and I have asked multiple times from various accounts too.<br>
I am a beginner, if anyone could at least point me in general direction then it would help me a lot.<br>
Thanks.<br>
Edit: I have followed the below links -<br>
<a href="https://pytorch.org/docs/master/notes/ddp.html#example" rel="nofollow noopener">https://pytorch.org/docs/master/notes/ddp.html#example</a><br>
<a href="https://pytorch.org/docs/stable/distributed.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html</a><br>
<a href="https://github.com/yangkky/distributed_tutorial/blob/master/src/mnist-mixed.py" rel="nofollow noopener">https://github.com/yangkky/distributed_tutorial/blob/master/src/mnist-mixed.py</a><br>
<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a><br>
<a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="nofollow noopener">https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html</a></p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/bing">@bing</a>, thanks for the question, and sorry about the delay. Could you please add the “distributed” tag to DP/DDP/RPC/C10D related questions in the future? The PT Distributed team is actively monitoring that tag, but we might miss uncategorized questions.</p>

username: mrshenli
cooked: <p>I added my response to your previous question <a href="https://discuss.pytorch.org/t/use-distributed-data-parallel-correctly/82500" class="inline-onebox">Use Distributed Data Parallel correctly</a>, let’s discuss there.</p>
Very Slow to Compute Diagonal Hessian

username: Dalles_Lee
cooked: <p>I am trying to compute only the diagonal of a hessian matrix. My codes look like this:</p>
<pre><code class="lang-python"># param is a Tensor with size (n, n), dissemble it so that we can take grad w.r.t. each individual entry
param = [[param[i][j] for i in range(n)] for j in range(n)] 
for p_ in param:
    for p in p_:
        p.requires_grad_(requires_grad=True)
new_param = torch.stack([torch.stack(p) for p in param]) # reassemble
loss = model(inputs, param=new_param)
first_derivative = torch.autograd.grad(loss, new_param, create_graph=True)[0]
for i in range(n):
    for j in range(n):
        second_drivative[i][j] = torch.autograd.grad(first_derivative[i][j], param[i][j], retain_graph=True)[0]

</code></pre>
<p>It succeeded, but takes very long time (almost as long as computing the full Hessian). The problem seems to be these for loops, and it having to back propagate through the same graph n^2 times</p>
<p>Is there a way to speed it up? Or what changes should I make to PyTorch to facilitate it?</p>
<p>Many thanks.</p>

username: albanD
cooked: <p>It makes sense that it takes as much time as the full hessian no? You evaluate each row of the hessian and only take the one element you want from it.<br>
Note that you don’t do it explicitly, but that’s what <code>torch.autograd.grad(first_derivative[i][j],  inp)</code> is doing as it would be the same as</p>
<pre><code class="lang-auto">mask = torch.zeros_like(first_derivative)
mask[i][j] = 1
torch.autograd.grad(first_derivative, inp, grad_output=mask)
</code></pre>

username: Dalles_Lee
cooked: <p>Yes, that makes sense.</p>
<p>Is there a way to somehow modify the way backprop works in PyTorch so that it computes the second derivative instead of the first? I know it might be a lot of work, but I feel it’s worth it since quite a few methods use diagonal hessian for pruning, e.g. optimal brain damage.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately, automatic differentiation (which is at the core of pytorch’s backprop) cannot be used to do that. You have to do exactly what you do here.</p>

username: Dalles_Lee
cooked: <p>Hi,</p>
<p>That’s unfortunate. Thanks anyway!</p>
(Java) Tensor.fromBlob(data, shape) how to define shape for 2D and more?

username: eranor
cooked: <p>Hi,<br>
I have a more java related question but I’m stuck. In the examples on Android, we can read:</p>
<blockquote>
<p>Tensor input = Tensor.fromBlob(data, new long<span class="chcklst-box fa fa-square-o fa-fw"></span>{1, data.length});</p>
</blockquote>
<p>In my case, I have an array:</p>
<blockquote>
<p>float<span class="chcklst-box fa fa-square-o fa-fw"></span><span class="chcklst-box fa fa-square-o fa-fw"></span> myArray = new float[150][7];</p>
</blockquote>
<p>And</p>
<blockquote>
<p>Tensor input = Tensor.fromBlob(myArray, new long<span class="chcklst-box fa fa-square-o fa-fw"></span>{1, myArray.length});</p>
</blockquote>
<p>Doesn’t work.</p>

username: eranor
cooked: <p>Ok I got it. Didn’t read properly the doc. I need to flatten my array and then give the proper tensor dimension.</p>

username: Venryx
cooked: <p>Can you give a link to the documentation page that shows how to flatten the array? (I have the same problem)</p>

username: eranor
cooked: <p><a class="mention" href="/u/venryx">@Venryx</a> You have to do it yourself. If you use Kotlin, you have a method <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/flatten.html" rel="nofollow noopener">https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/flatten.html</a><br>
If you use Java, you can do it recursively (<a href="https://stackoverflow.com/questions/31851548/flatten-nested-arrays-in-java" rel="nofollow noopener">https://stackoverflow.com/questions/31851548/flatten-nested-arrays-in-java</a> might help, I didn’t check the content).</p>
<p>In my case, my model is expecting an array of float of dimension (n, b, m). So in kotlin:</p>
<pre><code class="lang-auto">...
val input: Tensor = Tensor.fromBlob(myFloatArray, longArrayOf(n, b, m))
val output: Tensor = module.forward(IValue.from(input))!!.toTensor() 
...
</code></pre>
<p>You can find the doc there: <a href="https://pytorch.org/docs/1.3.0/org/pytorch/Tensor.html#id2" rel="nofollow noopener">https://pytorch.org/docs/1.3.0/org/pytorch/Tensor.html#id2</a><br>
Hope it helps.</p>
Concatenate tensor of 3 dimensions to tensor of 1 dimension while keeping first dimension

username: Talita
cooked: <p>Suppose that I have a tensor of shape</p>
<pre><code class="lang-auto">tensor1 = [sentence length, batch_size, embedding dimension]
</code></pre>
<p>for instance: torch.Size([4, 32, 768])</p>
<p>I want to add a value to the embedding dimension (768 -&gt; 769).</p>
<pre><code class="lang-auto">res = torch.cat((embedding[-1,:,:], batch.feat.unsqueeze(1)), dim=1)
</code></pre>
<p>Where, batch.feat is of size <code>[32,1]</code></p>
<p>but this leads to:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; res.size() 
torch.Size([32, 769])
</code></pre>
<p>How can I keep the sentence length dimension and have a tensor of shape?</p>
<pre><code class="lang-auto">torch.Size([4,32, 769])
</code></pre>
<p>(I have posted a related question on this but what I wanted to do didn’t make sense in that case).</p>

username: Caruso
cooked: <p>What you are doing:</p>
<p>You concatenate a tensor <code>embedding[-1,:,:]</code> of shape <code>{32, 768}</code> (you only select the last element of the first dimension) to a tensor <code>batch.feat.unsqueeze(1)</code> along the second dimension (<code>dim=1</code>).<br>
of shape <code>{32, 1}</code> ( i guess batch.feat is of shape <code>{32}</code>)<br>
<br><br>
But you want to do:</p>
<pre><code class="lang-auto">embedding = torch.randn(4, 32, 768, dtype=torch.float)
batch= torch.randn(32, dtype=torch.float)[None, :, None]

print(embedding.shape, batch.shape)
&gt;&gt; torch.Size([4, 32, 768]) torch.Size([1, 32, 1])

tensor_cat = torch.cat((embedding, batch.repeat(4, 1, 1)), dim=2)

print(tensor_cat.shape)
&gt;&gt; torch.Size([4, 32, 769])
</code></pre>
<p>So you keep your embedding tensor as a 3d tensor, but reshape your batch.feat to a 3d tensor of shape <code>{1, 32, 1}</code>. Because your <code>embedding</code> tensor is of shape <code>{4, 32, 1}</code> you need to repeat your <code>batch</code> tensor along the first dim, so they are of the same shape. Finally you can concatenate them along the third dimension (<code>dim=2</code> - your embedding dimension).</p>

username: Talita
cooked: <p>Thank you so much!!!</p>
Embedding layer size meaning

username: cevvalu
cooked: <p>I see most of the networks using an embedding with size 256 512 and 1024 while dealing with a huge vocabulary. I am deciding my size depends on my experiments over the network, observing losses and comparing Bleu scores. So is it an experiment for everyone when deciding the size or is there a specific formula for this?</p>

username: mailcorahul
cooked: <p>I don’t think there is any specific formula for fixing on embedding size. Like you said, it could be decided based on experimentations. Incase of images, the embedding size can vary based on the input image size, architecture being used.</p>

username: cevvalu
cooked: <p>i see, I use 4096 sized one dimensional-vector as image input on the right side of the network and left side i am generating words. Its a basic merge-image captioning model. But not sure how to understand which embedding size is better. I use 256 as i saw similar models with same input image size but trying to understand why</p>

username: mailcorahul
cooked: <p>Okay. Maybe you can fix embedding size in terms of the underlying data/feature complexity. Tuning based on that might help.</p>

username: cevvalu
cooked: <p>I will try, so is it possible for me to understand which embedding size is better from training, valid loss relation or BLEU scores?</p>

username: mailcorahul
cooked: <p>Yes, based on loss/accuracy.</p>
Saving torchtext field

username: InnoCentCoder5497
cooked: <p>Hi,</p>
<p>I am currently using torchtext-0.6.0 for text classification. I have trained my model and now i want to make a small web app out of it. However, I can’t find a way to save the torchtext Field vocabulary using dill or pickle. Whenever i load the pickle/dill file I get the following error.</p>
<p>Code used for saving:</p>
<pre><code class="lang-auto">SENT_FIELD = Field(sequential=True, tokenize='spacy')
# loading dataset
SENT_FIELD.build_vocab(train)
SENT_FIELD.vocab.load_vectors('glove.6B.300d')

# saving
import dill

with open("SENT_FIELD_dill.dill","wb")as f:
     dill.dump(SENT_FIELD,f)
</code></pre>
<p>Code for loading field:</p>
<pre><code class="lang-auto">with open("SENT_FIELD_dill.dill","rb")as f:
     TEXT=dill.load(f)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">TypeError: an integer is required (got type bytes)
</code></pre>
<p>Is their any way I can save the vocabulary for further use?</p>
<p>Thanks in Advance.</p>
Multiply series of image by 3D rotate matrix

username: eranb
cooked: <p>I have a series of 2D image (in fact it’s slices of image) : [series of images, H, W]</p>
<p>and I want to multiply it with 3D rotate matrix, what is the easy way to do it?</p>
<p>And if I even need a 3D matrix and not 2D.</p>
<p>what I’m trying to do is like to rotate a cube-like in this picture: <a href="https://math.stackexchange.com/questions/1327721/how-to-shift-an-object-to-rest-on-the-xy-plane" rel="nofollow noopener">https://math.stackexchange.com/questions/1327721/how-to-shift-an-object-to-rest-on-the-xy-plane</a></p>
Estimate the std and mean of a large dataset using Welford method

username: Dhorka
cooked: <p>Hi,</p>
<p>I’m trying to estimate the std and mean of a large image dataset using a Welford algorithm. As far as I read this algorithm is more robust than the method that accumulates the sum and the sum square of the values. However, I’m facing some issues. The first one is that I tested the algorithm in a small random dataset(6 images) and the mean and std estimated by Welford method are not exactly the same as the mean and std calculated using the whole dataset at the same time. The difference is very small, but I’m not sure if I’m doing something wrong.</p>
<pre><code class="lang-auto">import torch
torch.set_printoptions(precision=10)
class ImgRunningStats:

    def __init__(self):
        self.n = torch.zeros(1, dtype=torch.long)
        self.old_m = torch.zeros(3, dtype=torch.float64)
        self.new_m = torch.zeros(3, dtype=torch.float64)
        self.old_s = torch.zeros(3, dtype=torch.float64)
        self.new_s = torch.zeros(3, dtype=torch.float64)

    def push(self, x):
        self.n += 1

        if self.n == 1:
            self.old_m = self.new_m = x
            self.old_s = 0
        else:
            self.new_m = self.old_m + (x - self.old_m) / self.n
            self.new_s = self.old_s + (x - self.old_m) * (x - self.new_m)

            self.old_m = self.new_m
            self.old_s = self.new_s

    def mean(self):
        return self.new_m if self.n else 0.0

    def var(self):
        return self.new_s / (self.n - 1) if self.n &gt; 1 else 0.0

    def std(self):
        return torch.sqrt(self.var())
    


rs= ImgRunningStats()

test_img_0 = torch.rand([2,3,250,250])
test_img_1 = torch.rand([2,3,250,250])
test_img_2 = torch.rand([2,3,250,250])

images = [test_img_0, test_img_1, test_img_2]

test_img = torch.cat([test_img_0, test_img_1, test_img_2], dim=0)

print('mean full: ', test_img.mean(dim=[0,2,3]))
print('std full: ', test_img.std(dim=[0,2,3], unbiased=True))


for img in images:
    pixels = img.view(-1, img.shape[1])
    for px in pixels:
        rs.push(px)
       

print(f'Mean Welford: {rs.mean()}, Std Welford: {rs.std()}')
</code></pre>
<p>However, the major issue that I have is that the algorithm is very slow… I don’t know how I can improve the performance of the algorithm. As far as I saw, it is not feasible to run this algorithm in a large image dataset with the current implementation.</p>
<p>Is it possible to parallelize this algorithm?</p>
Modify the existing Torchscript model

username: AlvinZheng
cooked: <p>Hello,<br>
I am trying to modify an existing Torchscript model for model compression. For example, I want to change several parameters of the aten:_convolution operators. To explain the question more clearly, suppose we traced a model in the following way:</p>
<pre><code class="lang-auto">net = torchvison.models.resnet18().cuda()
data = torch.ones(1, 3 , 224, 224)
traced = torch.jit.trace(net, data)
</code></pre>
<p>The question is can I modify the torchscript model by directly modifying the traced.graph so that the new model can be loaded and run in a different place?   I also noticed that there is a ‘code’ variable in the ‘traced’, what is this variable for? Do I need to modify the ‘code’ variable at the same time?</p>
<p>Thanks ~</p>
Sorting or Grouping Images in the DataLoader

username: rahulcse1767
cooked: <p>The Data Loader in PyTorch generally does not give the images in sorted fashion.</p>
<p>For a certain task, I have some images with the same image ID and I wish to group these images for a function that I wish to create.</p>
<p>How can I modify the data loader such that the images are sorted (or grouped) according to the image ID?</p>
<p>It would also be helpful to know more on how I can access individual files through the data loader as I find difficulty in accessing images individually.</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>You could write a custom <code>sampler</code> by deriving from the base class from <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py">here</a> and apply your custom sampling logic.<br>
The sampler will yield the indices, which will be passed to the <code>Dataset.__getitem__</code>, so you could sort these indices to make sure that the right images are loaded.</p>