akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
LSTM Regression (Many to one)

username: asepehri93
cooked: <p>Hello.<br>
I am trying to fit an lstm model to my data. The training set has 102400 samples, each having 1568 features:</p>
<p>The shape of X_train is:  torch.Size([10240, 1, 1568])<br>
The shape of Y_train is:  torch.Size([10240, 1])</p>
<p>The target variable for each sequence is a scalar. I break the whole sequence into sequences of length 8, then I use the last output from lstm as the model’s output after passing through a linear layer.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class RNN(nn.Module):

  def __init__(self,input_size,hidden_size,output_size,num_layers):
    super(RNN, self).__init__()
    self.input_size=input_size
    self.hidden_size=hidden_size
    self.output_size=output_size
    self.num_layers=num_layers

    self.lstm=nn.LSTM(input_size,hidden_size,num_layers)
    self.fc=nn.Linear(hidden_size, output_size)


  def forward(self,sequence):
    #Input shape is (seq_len,batch=1,input_size).
    h_0=torch.zeros(self.num_layers,1,self.hidden_size)
    c_0=torch.zeros(self.num_layers,1,self.hidden_size)
    lstm_out,_=self.lstm(sequence,(h_0,c_0))

    #input shape is (seq_len,batch=1,hidden_size)
    #out shape is (1)
    output=self.fc(lstm_out[-1,:,:]).view(1)

    return output
</code></pre>
<pre><code class="lang-auto">def train(model,X_train,Y_train,X_test,Y_test,criterion,optimizer,epochs=100):
  preds=torch.zeros([X_test.shape[0],1])
  obs=torch.zeros([X_test.shape[0],1])


  for epoch in range(epochs):
    train_loss=0
    test_loss=0

    for i in range(X_train.shape[0]-batch_size):
      x=Variable(X_train[i:i+seq_len,:,:])
      y=Variable(Y_train[i+seq_len-1,:])
      model.zero_grad()
      yhat=model(x)
      loss=criterion(yhat,y)
      loss.backward()
      optimizer.step()
      train_loss+=loss.item()

    print('Train_'+str(epoch+1)+':  '+str(train_loss))

    for i in range(X_test.shape[0]-batch_size):
      x=X_test[i:i+seq_len,:,:]
      y=Y_test[i+seq_len-1,:]
      yhat=model(x)
      loss=criterion(yhat,y)
      test_loss+=loss.item()

      if epoch==epochs-1:
        preds[i,:]=yhat
        obs[i,:]=y

    print('Test_'+str(epoch+1)+':  '+str(test_loss)+'\n\n')
</code></pre>
<pre><code class="lang-auto">input_size=X_train.shape[2]    # features
hidden_size=256
output_size=1
num_layers=2
epochs=100

lr=0.001
criterion=nn.MSELoss()
model=RNN(input_size,hidden_size,output_size,num_layers)
optimizer=optim.RMSprop(model.parameters(),lr=lr)

train(model,X_train,Y_train,X_test,Y_test,criterion,optimizer,epochs)
</code></pre>
<p>The problem is no matter how complex I define the model, the predictions will not improve (not even for the training set). I have tried increasing lstm layers, changing lr, increasing sequence length, increasing hidden_size, but none worked. I assume there is some problem with my code and if not, I appreciate if you guid me how I can improve it.</p>
<p>Thank you.</p>
Control specific nodes in the layer

username: Hyeongu_Oh
cooked: <p>For example, there are 32 nodes in the fc layer.<br>
How do I apply a different activation function to the last two nodes?</p>

username: ptrblck
cooked: <p>You could slice the output and apply the desired activations separately:</p>
<pre><code class="lang-python">out = linear(input)
out1 = F.relu(out[:, :2])
out2 = torch.sigmoid(out[:, 2:])
out = torch.cat((out1, out2), dim=1) # concatenate back
</code></pre>
<p>Would that work for you?</p>
Creating Tensor from Numpy Array

username: Mathanraj_Sharma
cooked: <p>I have a dataset that has one feature, a flatten Numpy array of accelerometer readings for each second. The shape of each entry is (300, ).</p>
<p>The label is 0, 1, 2, 3 based on speed.</p>
<p>When I try to create TensorDataset I receive an error,<br>
<code>TypeError: 'numpy.int64' object is not callable</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c89456631d797e5e23043c615ed3ada1e749900d.png" alt="sample" data-base62-sha1="sCpiWLwyIGMgaq4RgFrC1deDsq9" width="499" height="196"></p>
<p>how can I load my dataset into TorchDataset?</p>

username: ptrblck
cooked: <p>Could you try to create a tensor first via:</p>
<pre><code class="lang-python">data = torch.from_numpy(numpy_arr)
</code></pre>
<p>If you are still stuck, could you post the code, which creates this error, please?</p>

username: Mathanraj_Sharma
cooked: <p>I tried that but it didn’t worked. Then I stretched the numpy arrray into columns and used it.</p>
<p>Thank you!!!</p>
A question about net.eval()

username: wayne
cooked: <p>Hi,<br>
Could anyone help me?<br>
After several batch training in every epoch, I tested my network on the validation set. Here is my pseudocode:</p>
<pre><code class="lang-auto">for i in range(epoch):
       #######     train   #############
      for j , batch in enumerate(train_loader):
                   pred = net(batch)
                   loss  = loss_function(pred, label)
                   loss.backward()
                   optimizer.zero_grad()
                   optimizer.step()
                    
             ######  validation  ##########
                   if j % 10 ==0:
                        net.eval()
                         for jj , batch2 in enumerate(validation_loader):
                                pred2 = net(batch2)
                                loss  = loss_function(pred2, label)
                    
</code></pre>
<p>After the validation, my network parameter‘’s (e.g. nn.Parameter(torch.ones(1))) gradient becomes 0!!!</p>
<p>But I rewrite my pseudocode as follows:</p>
<pre><code class="lang-auto">for i in range(epoch):
       #######     train   #############
      for j , batch in enumerate(train_loader):
                   pred = net(batch)
                   loss  = loss_function(pred, label)
                   loss.backward()
                   optimizer.zero_grad()
                   optimizer.step()
                    
             ######  validation  ##########
                   if j % 10 ==0:
                        net.eval()
                         for jj , batch2 in enumerate(validation_loader):
                                pred2 = net(batch2)
                                loss  = loss_function(pred2, label)

                    **net.train()**
</code></pre>
<p>The only difference is that I add “net.train()” after the validation, then the gradient of nn.Parameter would not become 0.</p>
<p>So my question:<br>
I should add this “net.train()” after the validation everytime, or I just did something wrong?</p>

username: wayne
cooked: <p>Could anyone help me?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="wayne" data-post="1" data-topic="79050">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/0ea827/48.png" class="avatar"> wayne:</div>
<blockquote>
<p>I should add this “net.train()” after the validation everytime, or I just did something wrong?</p>
</blockquote>
</aside>
<p>You should call <code>model.train()</code> again before continuing with the training.<br>
This will make sure to set all layers to training mode again, which would change the behavior of e.g. dropout and batchnorm layers.</p>
Is it OK to change dataset during training?

username: sh0416
cooked: <p>I use dataset and dataloader for my training script.</p>
<p>Is it OK to change dataset parameter during the traininng??</p>
<p>For example,</p>
<pre><code class="lang-auto">dataset = CustomDataset()
loader = DataLoader(dataset, num_worker=8)

for i, data in enumerate(loader):
     dataset.x = new_x
</code></pre>
<p>I saw that DataLoader use multithread and dataset is copied to other thread.</p>
<p>The change of dataset is reflected to the loader?</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>No, if you are using multiple workers, the changes won’t be reflected.<br>
What is your use case?<br>
Maybe you could use <a href="https://discuss.pytorch.org/t/dataloader-resets-dataset-state/27960/4?u=ptrblck">shared arrays</a>?</p>

username: sh0416
cooked: <p>My use case is for dataset to create sample using sample function.</p>
<p>During training, sampling strategy is changed.</p>
<p>I think I have to find alternative way… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: Senzeyu_Zhang
cooked: <p>Hello, what if I change dataloader.dataset.x like in this post?</p><aside class="quote" data-post="1" data-topic="35896">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sh0416/48/38708_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-it-ok-to-change-dataset-during-training/35896">Is it OK to change dataset during training?</a> 
  </div>
  <blockquote>
    I use dataset and dataloader for my training script. 
Is it OK to change dataset parameter during the traininng?? 
For example, 
dataset = CustomDataset()
loader = DataLoader(dataset, num_worker=8)

for i, data in enumerate(loader):
     dataset.x = new_x

I saw that DataLoader use multithread and dataset is copied to other thread. 
The change of dataset is reflected to the loader? 
Thanks,
  </blockquote>
</aside>
<p>
Will that be reflected to multiple workers?</p>

username: ptrblck
cooked: <p>I don’t think so as stated in my <a href="https://discuss.pytorch.org/t/is-it-ok-to-change-dataset-during-training/35896/2">reply</a>.</p>
Getting DispatchStub error with new device

username: almeetb
cooked: <p>I want to add a c++ extension for a new device type.</p>
<p>I added an implementation for <code>DeviceGuardImplInterface</code> and added a subclass inheriting from <code>TensorImpl</code> for the device</p>
<p>I followed <code>test/cpp_extensions/msnpu_extension.cpp</code> but when I try to create a tensor and set the device to the new device type I get the following error:</p>
<p><code>RuntimeError: DispatchStub: unsupported device type</code></p>
<p><code>DispatchStub</code> doesn’t seem to handling MSNPU or XLA devices so I was wondering how this works.</p>
<p>Appreciate any help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Change the input name of traced module

username: triwahyuu
cooked: <p>is there any ways for me to change the name of input variables of traced module? I have looked into the documentation of <a href="https://pytorch.org/docs/stable/jit.html#torch.jit.trace" rel="nofollow noopener"><code>torch.jit.trace</code></a> and it could not be changed beforehand as well, e.g. with dictionary.<br>
basically I’m trying to mimic the behavior of <code>input_names</code> argument in <a href="https://pytorch.org/docs/stable/onnx.html#torch.onnx.export" rel="nofollow noopener"><code>torch.onnx.export</code></a></p>

username: Michael_Suo
cooked: <p>I don’t believe we expose a way to do this. Names are not semantically meaningful in a JIT graph anyway, they are only for human readability</p>

username: triwahyuu
cooked: <p>thanks for the reply.<br>
what I want to do is so that I could use keyword arguments as an input to forward.<br>
in my traced module, there are 3 arguments, the name of the first two arguments have already been properly inferred by tracing, but the last one is not.<br>
my question now becomes how does the tracer infer the argument name for this?</p>

username: Michael_Suo
cooked: <p>The tracer looks in the python interpreter frame state’s local variables (<code>f_local</code>) for names (see <a href="https://github.com/pytorch/pytorch/blob/master/torch/jit/__init__.py#L299" rel="nofollow noopener">here</a>. So if, for a given traced value, it finds a corresponding local, it will add that as the name in the graph.</p>

username: triwahyuu
cooked: <p>Thank you, I will look into that</p>
WeightedRandomSampler vs Weighted CrossEntropyLoss for imbalanced Dataset

username: Christian_Franca
cooked: <p>Hello, everyone!</p>
<p>I would like to understand which of the 2 options is a better approach to deal with an imbalanced dataset.</p>
<p>I’m dealing with a dataset that has:</p>
<ul>
<li>516 items in class 0</li>
<li>91 items in class 1</li>
<li>622 items in class 2</li>
<li>592 items in class 3</li>
</ul>
<p>My data has way less items of class 1 in  comparation to the other classes.</p>
<p>Searching a little i found that the usual approach to this kind of problem is:</p>
<ol>
<li>Use <code>torch.optim.WeightedRandomSampler()</code> where i sample with probability (weights)</li>
<li>Use <code>torch.nn.CrossEntropyLoss(weights)</code> (my problem is classification) where i weight each class for the update part</li>
</ol>
<p>I couldn’t find an answer to which one of them is a better approach in which cases, and i would be very thankful if someone could explain to me, even if it doesn’t actually metter which one to choose.</p>

username: KFrank
cooked: <p>Hi Christian!</p>
<aside class="quote no-group quote-modified" data-username="Christian_Franca" data-post="1" data-topic="79043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christian_franca/48/23525_2.png" class="avatar"> Christian_Franca:</div>
<blockquote>
<p>I would like to understand which of the 2 options is a better approach to deal with an imbalanced dataset.<br>
…</p>
<ol>
<li>Use <code>torch.optim.WeightedRandomSampler()</code> where i sample with probability (weights)</li>
<li>Use <code>torch.nn.CrossEntropyLoss(weights)</code> (my problem is classification) where i weight each class for the update part</li>
</ol>
</blockquote>
</aside>
<p>My own personal theory, for which I have absolutely no evidence,<br>
says that if <code>WeightedRandomSampler</code> is likely to give you a<br>
batch with duplicate samples from your underrepresented class<br>
(the class with fewer samples), you’re just wasting computation<br>
by running the duplicates through your model, so you might as<br>
well use class weights in your loss function.</p>
<p>This is probably also true if you are likely to get duplicates over<br>
the course of a couple of batches for which the weights in your<br>
model haven’t changed much.</p>
<p>But if you aren’t likely to have duplicates, the gradient you calculate<br>
will “more representative” if your loss contains, say, five different<br>
samples from your underrepresented class than if it contains only<br>
one such sample, weighted by a factor of five.</p>
<p>Where you cross over from preferring loss-function weights to<br>
<code>WeightedRandomSampler</code>, I don’t really know.</p>
<p>Good luck.</p>
<p>K. Frank</p>
Is column selection differentiable?

username: p_may
cooked: <p>Hi all, I have a neural network (<strong>Net</strong>) with a trainable <strong>matrix</strong> of parameters in the first layer (matrix \in 100 x dim). The input to the network is an index (<strong>idx</strong>) to select one row of the trainable matrix to feed other layers. This index value is sent to the network <em>manually</em> each time I call <strong>Net</strong> (so <strong>Net(idx)</strong> gives me a vector (1 x dim) that is the <strong>idx-th</strong> row of the matrix).<br>
Now, I am trying to compute the value of <strong>idx</strong> instead of setting it manually. What I can think of, is to have another neural net (e.g. <strong>net-2</strong>) with a sigmoid on top. The sigmoid generates a value in the range [0,1], then I can <em>map</em> that value to [0,99] and trigger <strong>Net</strong> with that, such as this -&gt; Net(mapper(sigmoid_net-2)).<br>
My question is, <em>how can I make the entire process trainable/differentiable?</em> I have a <em>mapper</em> which can break the gradient flow. Also, I am not sure if <em>column selection</em> is differentiable.</p>
<p>Thanks in advance for your help.</p>
Pin_memory behavior on different machines

username: maralm
cooked: <p>Hi,</p>
<p>I use pin_memory to transfer tensors between cpu and gpu. In one machine, I see improvement on transfer speed when setting pin_memory is true but on the other machine, it is degrading the speed. Both machines have V100 gpus. First one is on the cloud and the second one is a local machine. Any idea what could be the reason?</p>
Run inference on Android for the quantized model

username: stu1130
cooked: <p>What is the use case for LiteModelLoader?(<a href="https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/LiteModuleLoader.java" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/android/pytorch_android/src/main/java/org/pytorch/LiteModuleLoader.java</a>)<br>
torch::jit::load in general supports both of the regular and quantized model, does it already cover the functionality of LiteModelLoader?</p>
How to write a dataset with queue for parallelized pre-processing

username: ham1d
cooked: <p>Hi,</p>
<p>I would like to use a pytorch dataset tha does the pre-processing in parallel and fills a queue and stores in it a queue, while <code>__getitem__</code> picks from the pre-computed items in the queue.</p>
<p>In my case, the workers in data loaders are not very optimal when the pre-processing (which is done using an external module) is done in the dataset and it can get very slow. Because of that, my gpu is not fully used and it is mostly free.</p>
<p>Can anyone help?</p>
Python API for autograd graph

username: gouthamvgk
cooked: <p>I am trying to understand how backprop is achieved by the autograd engine. Learned that when new tensors are created it includes a grad_fn attribute that tells us about the operation that created it and it in turn has has next_functions that converys information about the parent node. But during backprop operations like matmul will require the input during forward propagation and are they exposed through some python functions.?? I understood that they are stored in some autograd metadata so that autgrad engine has access to how nodes are created in the forward pass. Is there a way to access this graph in python?</p>

username: albanD
cooked: <p>Hi,</p>
<p>grad_fn and next_functions are the only things that are available from python I’m afraid. Other constructs are more complex and specific to each Node. So we don’t have custom python bindings for each of them.<br>
What kind of informations are you looking for?</p>

username: gouthamvgk
cooked: <pre><code class="lang-auto">inp = torch.randn(3,3, requires_grad=True)
another = torch.randn(3, 3, requires_grad=True)
inter = inp * 3
inter2 = inp * 4
result = torch.matmul(inter, inter2)
</code></pre>
<p>We knew that <code>result</code> was created using matmul in forward prop, but autograph should also record the information about the parent node <code>inter, inter2</code> and their corresponding values in order to do backprop rite? Are those dependency and edge information available through python?. For eg in custom autograd function we have ctx for saving info for backward, are those ctx information exposed in python?</p>

username: albanD
cooked: <p>For most ops no. Because they can save any value they want, we don’t have a unified API to access these values.</p>
AutoDiff for matrix exponential

username: Vic_Chang
cooked: <p>I got the following RuntimeError for the matrix exponential: element 0 of tensors does not require grad and does not have a grad_fn. I don’t know how to make it work. Does anyone know where the problem is?</p>
<p>def test(x):<br>
exp_x = torch.tensor(expm(x.detach().numpy()))<br>
return torch.sum(exp_x)</p>
<p>x = torch.eye(3,3)<br>
x = x.requires_grad_()<br>
y = test(x)<br>
y.backward()</p>

username: ptrblck
cooked: <p>Since you are detaching the computation graph from your tensor manually, <code>x</code> won’t get gradients anymore.<br>
I assume you are detaching <code>x</code> to call <code>expm</code> from <code>scipy</code>.<br>
You could try to implement the Pade approximation in PyTorch or write an <code>autograd.Function</code> with a custom backward method.</p>

username: Vic_Chang
cooked: <p>Thanks for your reply. I have used a different algorithm to do my work instead of using the matrix exponential.</p>
Tranfsorm image colors not matching with dataset class data while plotting

username: TanmDL
cooked: <p>I have created a simple example of mnist dataset for better understanding.</p>
<p>import torchvision.transforms.functional as TF<br>
import matplotlib.pyplot as plt</p>
<p>class MyDataset(torch.utils.data.Dataset):</p>
<pre><code>def __init__(self,dataset =  None, transform= None):
    self.MNIST = dataset
    print(self.MNIST)
    self.transform = transform

def __getitem__(self, index):
    data, target = self.MNIST[index]
    if self.transform is not None:
        #data_new = TF.to_pil_image(data)
        #data_new = self.transform(data_new)
        #data_new = TF.to_tensor(data_new)
        data2 = TF.to_pil_image(data)
        data2 = self.transform(data2)
        
        data2 = TF.to_tensor(data2)
    
       return data, data2, target, index

def __len__(self):
    return len(self.MNIST)
</code></pre>
<h4>here is the dataset code</h4>
<p>train_dataset = datasets.MNIST(root=’./data’, train=True, download=True,<br>
transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,), (0.5,))]))</p>
<h3>Applying simple horizontal flip on the dataset</h3>
<p>transform_train = transforms.Compose([ transforms.RandomHorizontalFlip(p=1)])</p>
<h5>now pass the the transform_train into MyDataset class.</h5>
<p>trainset = MyDataset(dataset=train_dataset, transform= transform_train )<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=24, shuffle=True, num_workers=2)<br>
data, tr_data, target, index = iter(trainloader).next()</p>
<h5>Image plot</h5>
<p>def imshow(inp, title=None):</p>
<pre><code>inp = inp.numpy().transpose((1, 2, 0))
mean = np.array([0.5])
std = np.array([0.5])
inp = std * inp + mean
inp = np.clip(inp, 0, 1)
plt.imshow(inp)
</code></pre>
<h2>##<span class="hashtag">#Make</span> a grid from batch</h2>
<p>out = torchvision.utils.make_grid(data)<br>
tr_out = torchvision.utils.make_grid(tr_data)</p>
<p>imshow(out, title=[target[x] for x in target])<br>
imshow(tr_out, title=[target[x] for x in target])</p>
<p>Both plot’s colors are not matching. Am I doing something wrong? Please check it.</p>
Unbalanced data in multi label problem

username: netaglazer
cooked: <p>hi, I have a multi-label problem, and they are very rare classes, and on the other hand, some very common classes.<br>
I have a custom loss<br>
I need o use all my data<br>
how can I balance this with my loss?<br>
tnx</p>

username: TinfoilHat0
cooked: <p>There are some sample weighting techniques for such data.<br>
See: <a href="https://arxiv.org/abs/1803.09050" rel="nofollow noopener">https://arxiv.org/abs/1803.09050</a></p>
<p>This is in fact something I personally implemented a few days ago. Maybe my implementation can help you: <a href="https://github.com/TinfoilHat0/Learning-to-Reweight-Examples-for-Robust-Deep-Learning-with-PyTorch-Higher" rel="nofollow noopener">https://github.com/TinfoilHat0/Learning-to-Reweight-Examples-for-Robust-Deep-Learning-with-PyTorch-Higher</a></p>

username: netaglazer
cooked: <p>tnx, but i dont see there reference to the multi label problem</p>

username: TinfoilHat0
cooked: <p>Hi what do you mean? in the experiments, they have an example on mnist where frequency of samples is heavily skewed towards digit 9. isnt that like your scenario?</p>

username: TinfoilHat0
cooked: <p>oh i see. yeah in that case each sample is associated with only a single label, different than your scenario.</p>
How to measure DDP time breakdown?

username: dujiangsu
cooked: <p>Hi, I am trying to use DistribuedDataDarallel for multi-node data-parallelism.</p>
<p>I want to know how can I measure the time breakdown for data load, forward, backward, communication?</p>
<p>Also, for calculating FLops, I am going to use the repository[<a href="https://discuss.pytorch.org/t/calculating-flops-of-a-given-pytorch-model/3711" class="inline-onebox">Calculating flops of a given pytorch model</a>] in github. Does anyone know the good way for calculating Flops.</p>

username: mrshenli
cooked: <p>If the program uses GPUs, you can use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="nofollow noopener">elapsed_time</a> to measure the time spent on forward, backward, and optimizer. It is harder to break down computation and communication of the backward pass, as DDP tries to overlap these two and DDP conducts communication on dedicated CUDA streams that are not visible from the application side. Besides, communications are launched as soon as a gradient bucket is ready, meaning that it may or may not always saturate the bandwidth. To get around this, you can run local forward-backward, and then explicitly using allreduce from application side to conduct gradient synchronization after the backward pass. This will expose opportunities to measure that from application.</p>
Pickle error with Jupyter notebook

username: Marxav
cooked: <p>FYI, an error which took me quite some time to solve.</p>
<p>I wanted to use ray (ray 0.8.4) to tune the hyper-parameters of a torch (1.5.0) nn model (eg: Foo model) in a Jupyter notebook, with all my Python lines of code (Foo model, torch and ray code) being directly inside the Jupyter notebook.</p>
<p>At the end of the first back-propagation, I always had the following error:<br>
File “/home/xxx/.local/lib/python3.6/site-packages/torch/serialization.py”, line 443, in _legacy_save<br>
pickler.dump(obj)<br>
_pickle.PicklingError: Can’t pickle &lt;class ‘<strong>main</strong>.Foo’&gt;: attribute lookup Foo on <strong>main</strong> failed</p>
<p>I successfully fixed it by moving out the python code related the nn model in a separate <em><strong>.py</strong></em> file ( foo.py) and importing it in the notebook (<em>from foo import Foo</em>).</p>
GRU on Time-Series Data

username: masmoit
cooked: <p>I am running a GRU on a time-series dataset to predict the outcomes. I have cut the dataset into sequence lengths of 200 and have run a batch size of 128. I was wondering, after training the dataset using DataLoader, is there an easy way to piece the different outcomes?</p>
<p>I am trying to compare the predictions and the actual data on the whole dataset but am unsure how to piece everything together (since the batches are divided randomly, and are not tagged with the correct date-time index).</p>
<p>My question is: is there a way to ‘piece back the data’ after using the dataloader to divide the data into randomized iterable batches?</p>
Optimizer stability when training embedding layer with sparse flag

username: sh0416
cooked: <p>Hi,<br>
I was found that the optimizer changes the value in embedding layer which is not used in the computational graph.<br>
The amount of value changed is very small, e.g. 1e-10, but it could be serious when training with 1M iteration.<br>
This comes from the operation in sparse tensor.<br>
Is there anyone who encounter this situation? If it is, how to solve it?<br>
Thanks,</p>
Libtorch. Get device of class inside the class

username: AZuev
cooked: <p>Hello i need to get device of class instance inside the class and then use this device for another tensor, created in this class. How i can do it in c++?</p>
<p>Example<br>
class …<br>
auto forward()<br>
{<br>
torch::Tensor a = torch::zeros({1,}).to(this.device);<br>
}</p>
Add Spectral Normalization to all Conv-Layers?

username: FloCF
cooked: <p>Is there a way to iterate over all modules of a model, let’s call it <code>Generator</code>, and equip all <code>Conv2d</code>- and <code>ConvTranspose2d</code>-layers with spectral normalization?</p>
<p>Sorry, if this has been discussed already, I couldn’t find it though…</p>
<p>Many thanks in advance!</p>

username: FloCF
cooked: <p>Is that a valid solution? How can you check if Conv-layer is equiped with spectral norm?</p>
<pre><code class="lang-auto"># Make all layers to be spectral normalization layer
def add_sn(m):    
    if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
        return spectral_norm(m)
    else:
        return m
    
Generator.apply(add_sn)
</code></pre>

username: dem123456789
cooked: <p>This is the correct anwer.</p>
<p>You can simply verify it by checking</p>
<blockquote>
<p>conv.weight_u.size()</p>
</blockquote>
How to do weights initialization in nn.ModuleList?

username: Tao_Jiang
cooked: <p>I wanna do weights initialization in an <code>nn.ModuleList</code> instance.<br>
Here are my codes:</p>
<pre><code class="lang-python">class ConvBlock(nn.Module):
	
	def __init__(self, input_channels, num_filters=128, conv_nums=3):
		super(ConvBlock, self).__init__()
		conv_blocks = [nn.Sequential(nn.LeakyReLU(0.1), nn.Conv2d(input_channels, num_filters, kernel_size=(3 ,3), padding=1))]
		for _ in range(conv_nums-1):
			conv_blocks.append(nn.Sequential(nn.LeakyReLU(0.1), nn.Conv2d(num_filters, num_filters, kernel_size=(3 ,3), padding=1)))
		self.conv_blocks = nn.ModuleList(conv_blocks) 

	def forward(self, input_tensor):
		out = [input_tensor]
		for conv in self.conv_blocks:
			input_tensor = conv(input_tensor)
			out.append(input_tensor)
		return out
</code></pre>
<p>and the weight intialization code I often used is</p>
<pre><code class="lang-auto">for m in self.modules():
    if isinstance(m, nn.Conv2d):
        n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
        m.weight.data.normal_(0, sqrt(2. / n))
</code></pre>
<p>but it seems not worked for a complicated network structure. Could someone tell me how to solve this problem?</p>

username: kaushalpaneri
cooked: <p>You can use <code>apply()</code> to initialize weights. Here is the <a href="https://discuss.pytorch.org/t/nn-modulelist-apply/4039">discussion thread</a>.</p>
<pre><code class="lang-auto">class ConvBlock(nn.Module):
	
	def __init__(self, input_channels, num_filters=128, conv_nums=3):
		super(ConvBlock, self).__init__()
		conv_blocks = [nn.Sequential(nn.LeakyReLU(0.1), nn.Conv2d(input_channels, num_filters, kernel_size=(3 ,3), padding=1))]
		for _ in range(conv_nums-1):
			conv_blocks.append(nn.Sequential(nn.LeakyReLU(0.1), nn.Conv2d(num_filters, num_filters, kernel_size=(3 ,3), padding=1)))
		self.conv_blocks = nn.ModuleList(conv_blocks) 
                self.conv_blocks.apply(self._init_weights)

	def forward(self, input_tensor):
		out = [input_tensor]
		for conv in self.conv_blocks:
			input_tensor = conv(input_tensor)
			out.append(input_tensor)
		return out
        def _init_weights(self, m):
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
                nn.init.xavier_normal_(m.weight)
                m.bias.data.fill_(0.01)
</code></pre>

username: Tom_Lucas
cooked: <p>I have a similar problem and my current solution is to write my own apply functions, using named_modules() and named_parameters(), which adds filtering by name or class for module and name for parameters.</p>
<p>Weight initialisation methods like xavier_normal_() won’t work on BatchNorm2d, even though they have ‘weight’ parameters, because they are onl;y 1d tensors.</p>
<p>An additional question is, can parameters that can be initialised be uniquely identified?</p>
<p>I suppose I could have a tensor shape check and a specific apply method for weight initialisation, or a lambda test, but I’m still not sure if every parameter named ‘weight’ with 2 or more dimensions can be initialised this way.</p>
DDP Error for more than 2 nodes

username: dujiangsu
cooked: <p>Hi, I tried DistributedDataParallel with nccl backend.</p>
<p>It works when I use 1 or 2 nodes (each with 4 V100).</p>
<p>However, error happens when further scaling to 3 or 4 nodes, and always a node with the following error (other nodes reports differently and looks correct).</p>
<p>gpu45:169732:170179 [0] transport/net_ib.cc:789 NCCL WARN NET/IB : Got completion with error 12, opcode 1, len 11155, vendor err 129</p>
<p>I tried Pytorch Version: 1.2-cuda10.0 1.4-cuda10.1 1.5-cuda10.1<br>
And the nccl version is 2.4.8.</p>
<p>The nccl_debug info of 4 nodes are listed below: <a href="http://49.234.107.127:81/index.php/s/wktxGy47Zxmd6gE" rel="nofollow noopener">Node0</a>, <a href="http://49.234.107.127:81/index.php/s/WmNR8T6R4bZxLJb" rel="nofollow noopener">Node1</a>, <a href="http://49.234.107.127:81/index.php/s/kBd4gBKE5jfLFjq" rel="nofollow noopener">Node2</a>, [Node3] (<a href="http://49.234.107.127:81/index.php/s/5B2wEFHSFCWSHfm" rel="nofollow noopener">http://49.234.107.127:81/index.php/s/5B2wEFHSFCWSHfm</a>) .</p>
<p>When I tried 4 nodes, the [ NCCL WARN NET/IB] always happens in the third node.<br>
If I exclude the node (gpu45), and only run on the other three nodes, the [ NCCL WARN NET/IB] also happens.</p>

username: mrshenli
cooked: <p>This might be relevant to <a href="https://github.com/NVIDIA/nccl/issues/214" rel="nofollow noopener">this issue</a> in NCCL repo, and <a href="https://github.com/NVIDIA/nccl/issues/214#issuecomment-489242319" rel="nofollow noopener">this comment</a> seems fixed it.</p>

username: dujiangsu
cooked: <p>Thanks a lot!<br>
I give up using NCCL temporarily and it should relate to the system setting.</p>
RuntimeError: Expected tensor to have size 100 at dimension 1, but got size 128 for argument #2 'batch2' (while checking arguments for bmm)

username: silence
cooked: <p>I’m trying to implement an attention model,  but fail to execute the matmul</p>
<pre><code class="lang-auto">torch.matmul(att, v)
</code></pre>
<p>The shape of att and v is:</p>
<pre><code class="lang-auto">att shape:torch.Size([20, 3, 128, 128])
v shape:torch.Size([20, 3, 128, 100])
</code></pre>
<p>i get such an error:</p>
<blockquote>
<p>RuntimeError: Expected tensor to have size 100 at dimension 1, but got size 128 for argument <span class="hashtag-raw">#2</span> ‘batch2’ (while checking arguments for bmm)</p>
</blockquote>
<p>I also tried generate two tensors with the same shape by torch.randn and repeat the same operation and no error ouccrred. I don’t know what makes such an error.</p>

username: ptrblck
cooked: <p>Your code seems to work:</p>
<pre><code class="lang-python">att = torch.randn(20, 3, 128, 128)
v = torch.randn(20, 3, 128, 100)

out = torch.matmul(att, v)
print(out.shape)
&gt; torch.Size([20, 3, 128, 100])
</code></pre>

username: silence
cooked: <p>Yes, I found i accidently  repeat matmul ()  in the same function. I didn’t pay attention to the line number so got it wrong. I have solved the problem now.<br>
Thanks for your help!</p>