akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Backward failed when i use a for circulate in rnn model

username: thinksoso
cooked: <aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/66989218/backward-failed-when-i-use-a-for-circulate-in-rnn-model" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/11908149/thinksoso" target="_blank" rel="noopener nofollow ugc">
    <img alt="thinksoso" src="https://lh6.googleusercontent.com/-AWtnmDM0X_U/AAAAAAAAAAI/AAAAAAAAAAA/ACHi3reWA_J6a5uRFR13s87F4-FXuVcfPw/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/66989218/backward-failed-when-i-use-a-for-circulate-in-rnn-model" target="_blank" rel="noopener nofollow ugc">backward failed when i use a for circulate in rnn model</a>
</h4>

<div class="tags">
  <strong>python, deep-learning, pytorch, recurrent-neural-network</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/11908149/thinksoso" target="_blank" rel="noopener nofollow ugc">
    thinksoso
  </a>
  on <a href="https://stackoverflow.com/questions/66989218/backward-failed-when-i-use-a-for-circulate-in-rnn-model" target="_blank" rel="noopener nofollow ugc">03:25PM - 07 Apr 21 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Packed_sequence and packed_padded_sequence

username: ptrch_c_m
cooked: <p>I want to build a Bi-LSTM for text classification.</p>
<p>For computational efficiency it is a good practice to use <code>PackedSequence</code>. I am not sure if I should use <code>packed_sequence</code> or <code>packed_padded_sequence</code>.</p>
<p>Can anyone provide a short explanation as to which one to prefer and why.</p>
<p>Thanks in advance,<br>
regards.</p>

username: turian
cooked: <p>I am also curious to understand the difference between when to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_sequence.html" rel="noopener nofollow ugc">pack_sequence</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pack_padded_sequence.html" rel="noopener nofollow ugc">pack_padded_sequence</a>. What are the pros and cons of each?</p>
How to get image batches without looping through the entire data

username: en_tetteh
cooked: <p>I have three datasets each loaded using the Dataloader class. The goal is to grab a batch from each data loader, compute the loss on each batch and aggregate all the losses, before moving to the next batch.<br>
What I did was</p>
<pre><code class="lang-auto">d1 = []
for batch_idx, samples in enumerate(train_loaders[0]):
    if batch_idx &gt;= num_batches:
         print("breaking out")
        break
   d1.append({"images": samples["img"].float(), "targets": samples["lab"]})
</code></pre>
<p>With the above I am able to grab any batch. However, this results in memory issues, as the entire data has to be kept in memory.<br>
Is it possible to get a cleaner and memory-efficient solution to this? Thank you for your help. Please <a class="mention" href="/u/ptrblck">@ptrblck</a> any solution?</p>
Access module targetted by fx.Node.target

username: nscotto
cooked: <p>Hello,</p>
<p>I am working with <code>torch.fx.Graph</code>, let’s say I have a <code>torch.fx.Node</code> call <code>node</code> and <code>node.op</code> is <code>call_module</code>, how can I access the module object pointed by <code>node.target</code>?<br>
I would like to access the object in order to retrieve its type.</p>
<p>My work is part of an automated model modification framework so looking at the model code isn’t an option for what I am doing.<br>
Thank you.</p>
Multi-process torch Dataloader error

username: patfla
cooked: <p>I’m converting someone else’s (pytorch) Google Colab jupyter notebook to straight python (3.8.8) that I can run locally.</p>
<p>When I use the original code as is, making a multi-process Dataloader, locally:</p>
<p>dataloaders = {‘train’: torch.utils.data.DataLoader(image_datasets[‘train’], batch_size=batch_size, shuffle=True, num_workers=5),<br>
‘valid’: torch.utils.data.DataLoader(image_datasets[‘valid’], batch_size=batch_size, shuffle=True, num_workers=5)}</p>
<p>it crashes with:</p>
<p> → for batch_idx, (images, labels) in enumerate(dataloaders[‘valid’]):<br>
(Pdb) n<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “c:\users\MyId\appdata\local\programs\python\python38\lib\multiprocessing\spawn.py”, line 117, in spawn_main<br>
exitcode = _main(fd, parent_sentinel)<br>
File “c:\users\MyId\appdata\local\programs\python\python38\lib\multiprocessing\spawn.py”, line 127, in _main<br>
self = reduction.pickle.load(from_parent)<br>
AttributeError: Can’t get attribute ‘ImageFolderWithPaths’ on &lt;module ‘<strong>main</strong>’ (built-in)&gt;<br>
Traceback (most recent call last):</p>
<p>Something to do with the calling context getting unpickled (unserialized).</p>
<p>If I change the code to single process:</p>
<p>dataloaders = {‘train’: torch.utils.data.DataLoader(image_datasets[‘train’]),<br>
‘valid’: torch.utils.data.DataLoader(image_datasets[‘valid’])}</p>
<p>and run it locally, it runs fine.</p>
<p>And yet the multi-process code runs one on Google Colab.</p>
<p>Why?  And I wonder what I need to change to get multi-process to run locally.</p>

username: patfla
cooked: <p>The problem disappears when I run the code on Linux.  It fails in the above fashion on Windows 10.  I’ll take a look through</p>
<aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/pytorch/issues" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-full-image" width="60" height="60">

<h3><a href="https://github.com/pytorch/pytorch/issues" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch</a></h3>

<p><a href="https://github.com/pytorch/pytorch/issues" target="_blank" rel="noopener nofollow ugc">s</a></p>

  <p><span class="label1">Tensors and Dynamic neural networks in Python with strong GPU acceleration</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’m guessing that PyTorch Windows support is, uh, less enthusiastic than Linux.</p>
Training model with custom dataset

username: moinudin
cooked: <p>I have created a custom dataset, and now I am trying to use it to train a model. I’m working off of this tutorial: <a href="https://www.tutorialspoint.com/pytorch/pytorch_training_a_convent_from_scratch.htm" class="inline-onebox" rel="noopener nofollow ugc">PyTorch - Training a Convent from Scratch - Tutorialspoint</a> which doesn’t show what <code>X</code> is. I tried passing in my custom dataset, but it expects a <code>Tensor</code> object, whereas my dataset is a list of (<code>Tensor</code>, str) tuples.</p>
<p>How can I feed this into the model cleanly? Bear in mind I’m very new to PyTorch, having just started using it today.</p>

username: cyRi-Le
cooked: <p>Based on their class definition, input <code>X</code> should be batches of 2d vector, so tensors of size <code>[batch_size,2]</code>. You should check your dataset to be 2d vectors first.</p>

username: moinudin
cooked: <p>I think I’m rather confused at the moment.</p>
<p>I have a Dataset object. Each entry in the Dataset is a (Tensor, str). Should I change that to produce 2d vectors? I thought a Tensor was essentially a multidimensional vector, and that if I convert I’ll lose the benefits of using them?</p>
<p>From that I constructed DataLoader objects. Still figuring out how those work, but I think they’re used to load data in batches in parallel. Each has its own slice of the parent Dataset. How would I use this to train the model though? I seem to be missing that jump.</p>

username: cyRi-Le
cooked: <p>The model input should reflect your dataset, in the tutorial, the class <code>Neural_Network</code> is designed for 2d input vectors, hence <code>inputSize = 2</code>. In your dataset, if entries are (Tensor, str), I guest input is the tensor and the label is the str, but the model in the tutorial the output is scalar .</p>

username: moinudin
cooked: <p>Ideally I want both Tensor and str to be input. It’s possible I’m designing it poorly though.</p>
<p>I have noticed some examples returning (input, label), which I think is what you’re referring to. For now, I’ve changed the Dataset to return (Tensor input, scalar label), and am trying to get it to work. Although I’d still like to add the str as additional input as it should help a lot with classification.</p>
<p>To explain the problem a bit, the Tensor input is the STFT of an audio stream and the str is the text of what is spoken in the audio stream. My initial goal is to learn what the spoken version of the text sounds like to be able to classify audio streams without knowing the text. I think that should be better than training on the audio alone, but I’m happy to be shown I’m wrong.</p>
How to deal with hidden states of an LSTM?

username: Vlad_Dumitru
cooked: <p>Hello guys, I am new there to PyTorch and ML in general and I have a question that I am not able to find the answer to it. If any of you would be kind to help me with this I would really appreciate it.</p>
<p>I have a time series in which each date is correlated with the preview one, and base on that I am trying to predict action 1 and action 2. But the problem is that I am not sure how to deal with the <code>hidden state</code>. So far as I understand is that that <strong>hidden state</strong> si basically all the information that an LSTM had to learn from previews experience, and it should use that previews knowledge in the future prediction. So if I understand it correctly my questions are.</p>
<ol>
<li>When do I reset the hidden state, for example at the end of each episode, that means my LSTM is going to forget all the previews knowledge and that basically means that all the preview work is for nothing and it needs to do it over again?</li>
<li>How to deal with multiple batches? Let’s suppose that I will initialize my hidden state like this.<br>
<code>torch.zeros(2, 300, 3)</code>. That 300 means the number of batches, and because of that if it will be to evaluate my model with just one value instead of 300 values is going to crash because the shape does not match. How should I approach this? Because every single example and tutorial that I’ve seen basically just reinitialize the hidden state with 0 and with the properly batch_size. But again, if I will do this, does not mean that I am going to forget all the previews experience?</li>
</ol>
Apply horizontal, vertical flip, rotations etc to tensor

username: py23
cooked: <p>Hi, I am aware of that there are existing utilities for applying horizontal, vertical flips etc to PIL images. But is there a way to apply this to a 64 channel tensor?<br>
The only solution I have is this:<br>
Split the 64 channels into 1 channel each<br>
For the two remaining channels put the same values as the original 1 channel<br>
Convert to PIL Image<br>
Apply transform<br>
Convert back to tensor<br>
Remove 2 extra channels<br>
Repeat process 64 times<br>
Then stack the tensors to get the augmented 64 channel tensor</p>
<p>This is a hugely inefficient method. I have looked at torch.fliplr but that only switches the color channels in practice.<br>
So does anyone have any ideas or know of an existing pytorch function which does this already?<br>
Thanks in advance</p>

username: cskarthik7
cooked: <pre><code class="lang-auto">import torch
from torchvision import transforms
transform = transforms.Compose([
  transforms.Resize((400,400)),
  transforms.RandomHorizontalFlip(p=0.5),
  transforms.RandomVerticalFlip(p=0.5),
  transforms.RandomRotation(60)
])
z = transforms.Resize((400,400))
x = torch.ones(64, 2048, 2048)
y = transform(x)
print(y.shape)

</code></pre>
<p>You can do something like this for augumentation of tensors!</p>

username: py23
cooked: <p>Thank you this works! I appreciate your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to do inplace operation on custom parameters?

username: ADONAI_TZEVAOT
cooked: <p>I have a custom parameter <code>W = nn.Parameter(torch.randn(1))</code> When i try to do <code>W += 2</code> or when i try to do <code>W.data += 2</code> , I get the following error:<br>
<code>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</code><br>
Please help</p>
What is initialization='he' equivalent in PyTorch

username: aknirala
cooked: <p>I am trying to replicate a TF code in PyTorch. I see that in linear layer the initialization is done as:</p>
<blockquote>
<p>initialization=‘he’</p>
</blockquote>
<p>However I am not able to find a PyTorch equivalent of that. All I get is xavier as:</p>
<blockquote>
<p>nn.init.xavier_…(layer.weight)</p>
</blockquote>

username: KyleOng
cooked: <p>PyTorch equivalent for <code>he initialization</code> is</p>
<pre><code class="lang-auto">nn.init.kaiming_…(layer.weight)
</code></pre>
<p><a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_</a></p>
DataLoader (with num_workers>0) with loading only subsets of data in at once gives errors

username: ccook
cooked: <p>I’m building a dataset where there are choice ‘sessions’ that each include many rows. I can’t load all the data into memory at once, so I’ve been trying to load chunks of ~2500 sessions into memory at a time and then using some logic in the <code>__getitem__</code> to load the next chunk into memory once it’s iterated through the current chunk.</p>
<p>It works with <code>num_workers=0</code>, but when trying to do it with multiprocessing it runs into issues (<code>DataLoader worker is killed by signal</code>), presumably because multiple threads start trying to load the next chunk of data into memory at once.</p>
<p>Any thoughts for how to fix? Current code below !</p>
<pre><code class="lang-auto">class ChoiceDataset(Dataset):
    def __init__(self, filename, chunksize, choice_var, feature_vars, fe_col, n_y, n_sessions):
        # Dimensions 
        self.n_y = n_y # items
        self.n_x = len(feature_vars) # features
        self.length = n_sessions # total sessions 

        # Features 
        self.choice_var = choice_var
        self.features = feature_vars
        self.chunksize = chunksize
                                                    
        # Sessions -- read in just a chunk for now 
        self.filename = filename
        filters = [("session_id", "&lt;", chunksize)]
        self.sessions = pd.read_parquet(filename, filters=filters).set_index('session_id')
        
    def __getitem__(self, idx):
        
        session_id = idx        
        
        # If in memory, grab session, else load next chunk into memory first (faster than checking if in mem each time)
        try: 
            session = self.sessions.loc[session_id]
        except: # Only reason above should fail if that session is not in memory, so load next chunk
            filters = [("session_id", "&gt;=", self.chunksize), ("session_id", "&lt;", session_id + self.chunksize)]
            self.sessions = pd.read_parquet(self.filename).set_index('session_id')    
            session = self.sessions.loc[session_id] 
        
        # Pull out the features 
        session_x = torch.tensor(session[self.features].values)
        session_x = torch.tensor(session[self.features].values)
        session_x_fe = torch.LongTensor(session[self.fe_col].values)
        
        # Get the chosen item
        label = torch.tensor(np.argwhere(session[self.choice_var].values&gt;0).item())

        return (session_x, session_x_fe, label)

    def __len__(self):
        return self.length
</code></pre>

username: ptrblck
cooked: <p>I’m not really familiar with parquet, but there seem to be some libraries, which provide an interface to load these data type, e.g. <a href="https://github.com/KamWithK/PyParquetLoaders">PyParquetLoaders</a>, so maybe they are helpful.</p>

username: ccook
cooked: <p>This looks great — I’ll give it a try and report back.</p>
<p>Out of curiosity, is there an alternative storage solution that you think would work better?</p>
AttributeError: module 'torch.nn' has no attribute 'SiLU'

username: bc060400164
cooked: <p>I m using the pytorch version “1.5.0” . By usin Silu its saying that " AttributeError: module ‘torch.nn’ has no attribute ‘SiLU’ " How can i use SiLU activation function.</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-105-d64bc46f9e10&gt; in &lt;module&gt;
      1 # instantiate  the class LSTM1 object
----&gt; 2 lstm1 = LSTM1(num_classes, input_size, hidden_size, num_layers, 1) #our lstm class

&lt;ipython-input-99-6ae20c79c8d0&gt; in __init__(self, num_classes, input_size, hidden_size, num_layers, seq_length)
     13         self.fc = nn.Linear(50, num_classes) #fully connected last layer
     14 
---&gt; 15         self.relu = nn.SiLU()
     16 
     17     def forward(self,x):

AttributeError: module 'torch.nn' has no attribute 'SiLU'</code></pre>

username: tom
cooked: <p>You would need to upgrade your PyTorch or backport the SiLU implementation.</p>
TfIdf embedding as the input to Conv1d

username: Uma
cooked: <p>Can we use Tfidf output as the input to Conv1d ?</p>
<p>As I read some articles, most of them have used Embedding layer to get the input.</p>
<p>In one article<br>
<a href="https://towardsdatascience.com/text-classification-with-cnns-in-pytorch-1113df31e79f" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/text-classification-with-cnns-in-pytorch-1113df31e79f</a><br>
self.conv_1 = nn.Conv1d(self.seq_len, self.out_size, self.kernel_1, self.stride)<br>
seq_len is the number of words in a sentence.</p>
<p>In another article</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://galhever.medium.com/sentiment-analysis-with-pytorch-part-3-cnn-model-7bb30712abd7" target="_blank" rel="noopener nofollow ugc" title="11:52PM - 21 June 2020">Medium – 21 Jun 20</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:130/51;"><img src="https://miro.medium.com/max/1200/1*z11BIMlW-6yUe3htG-Xerw.png" class="thumbnail" width="130" height="51"></div>

<h3><a href="https://galhever.medium.com/sentiment-analysis-with-pytorch-part-3-cnn-model-7bb30712abd7" target="_blank" rel="noopener nofollow ugc">Sentiment Analysis with Pytorch — Part 3— CNN Model</a></h3>

<p>Introduction</p>

  <p><span class="label1">Reading time: 7 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
self.convs = nn.ModuleList([nn.Conv1d(in_channels=1,<br>
out_channels=n_filters,<br>
kernel_size=(fs,    embed_size))</p>
<p>Since this is for text, they have said, in_channels is 1. So I am confused about this explanation. Can you please explain this to me.</p>
Change gradients' value of the tensors during backpropagation

username: hamedB
cooked: <p>Hi</p>
<p>I registered a hook on the output of each Relu fauntion in VGG19 to record gradient through backward pass using the following snippet code:</p>
<blockquote>
<p>x = module(xx.to(self.device))<br>
x = x.requires_grad_(True)<br>
x.register_hook(self.save_gradient)</p>
</blockquote>
<p>I need to apply Relu function on the gradient of tensor during the back propagation procedure in which each lower tensor receives only positive gradient from higher tensor.</p>
<p>I really appreciate if you can guide me how I can do this.<br>
Thanks in advance.</p>
Merge All fold weights into single Weight file

username: vpkprasanna
cooked: <p>I had a bigger dataset and i trained my model with 5 folds in it , each fold will give a best weights file. Now is there any way to merge all the 5 weights file into single weight file.<br>
My Model:</p>
<pre><code class="lang-auto">class ImgClassifier(nn.Module):
    def __init__(self, model_arch, n_class, pretrained=False):
        super().__init__()
        self.model = timm.create_model(model_arch, pretrained=pretrained)
        n_features = self.model.classifier.in_features
        self.model.classifier = nn.Linear(n_features, n_class)
        
    def forward(self, x):
        x = self.model(x)
        return x
</code></pre>
<p>how to acheive it ? any idea would be helpful .<br>
TIA</p>
Will relu layer introduce parameters? 

username: Sir_nuo
cooked: <p>This is the net code and the result of test:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()  # 初始化基类
        self.conv1 = nn.Conv2d(1, 6, (5, 5))
        self.conv2 = nn.Conv2d(6, 16, (5, 5))

        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = self.conv1(x)
        x = f.relu(x)
        x = f.max_pool2d(x, (2, 2))

        # If the size is a square you can only specify a single number
        x = self.conv2(x)
        x = f.relu(x)
        x = f.max_pool2d(x, (2, 2))

        x = x.view(-1, self.num_flat_features(x))
        x = f.relu(self.fc1(x))
        x = f.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features


"创建网络"
net = Net()

params = list(net.parameters())
print(len(params))
for param in range(10):
    print(params[param].size())  # print parameters
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa673fa74ced78b5de024278bb090a3e749285c8.png" alt="图片" data-base62-sha1="ojsnvzxn0We9YQTd36cNHCACZlK" width="222" height="233"><br>
I want to know the structure of the parameters<br>
however the results show that each relu will introduce a parameter tensor , why?</p>

username: ptrblck
cooked: <p>No, the <code>nn.ReLU()</code> module and the functional API call via <code>F.relu</code> won’t use any parameters.<br>
The 10 parameters you are seeing are coming from the <code>weight</code> and <code>bias</code> of the other 5 layers (conv and linear).<br>
You might get more information by using e.g. <code>model.named_parameters()</code>, which will also return the parameter name.</p>

username: Sir_nuo
cooked: <p>Oh!!!<br>
Thanks, I forgot the bias, I was too stupid!</p>
Divide validation data into n parts

username: teja
cooked: <p>After loading the data into val_loader, I need to divide the validation data into <strong>n</strong> (n=2 or 5 etc.) parts.<br>
Here is what I tried, but it throws error, <strong>‘DataLoader’ object is not subscriptable.</strong></p>
<pre><code class="lang-auto">total=len(val_loader.dataset)
print(total) 
val1=val_loader[0:41629] # error, object is not subscriptable
val2=val_loader[41629:]
print(len(val1.dataset), len(val2.dataset))
</code></pre>
<p>Any suggestion on how to divide the val_loader in <strong>n</strong> parts.</p>

username: ptrblck
cooked: <p>You could use a <code>SubsetRandomSampler</code> to create different validation <code>DataLoader</code>, which would use the specified sample indices. Slicing the <code>DataLoader</code> is not supported.</p>

username: teja
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your suggestion, this is how I tried to split the validation data into two parts, and it worked, I hope it is ok.</p>
<pre><code class="lang-auto">indices = list(range(83258)) # 83258 is the total validation samples
np.random.shuffle(indices)
    
val1=indices[:41629]
val2=indices[41629:]
sampler1=SubsetRandomSampler(indices=val1)
sampler2=SubsetRandomSampler(indices=val2)

val_loader1 = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transform=transform),
        batch_size=32, shuffle=False,
        num_workers=args.workers, pin_memory=True, sampler=sampler1)
    
val_loader2 = torch.utils.data.DataLoader(
        datasets.ImageFolder(valdir, transform=transform),
        batch_size=32, shuffle=False,
        num_workers=args.workers, pin_memory=True, sampler=sampler2)
</code></pre>

username: ptrblck
cooked: <p>Yes, you approach looks perfectly fine! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Requires grad on custom modules

username: sftwre
cooked: <p>I am performing transfer learning on a pre-trained version of MobileNet V2. I want to freeze all<br>
layers of the model except for BatchNorm2d modules. I am currently handling this with the following loop:</p>
<pre><code class="lang-auto">    for module in model.modules():
        if not isinstance(module, nn.modules.BatchNorm2d):
            module.requires_grad_(False)
        else:
            module.requires_grad_(True)
</code></pre>
<p>This will iterate through the modules and turn off gradient computation for all layers except the BatchNorm2d layers. MobileNet V2 has several custom modules, however. For example, the ConvBNReLU block is used to group a Conv2d, BatchNorm2d, and ReLU6. Using this method gradient computation will first be turned off on the ConvBNReLU module, but when iterating through it’s components the BatchNorm2d layer will have gradient computation turned on. My questions are: is this the correct way to freeze non-BatchNorm2d layers in custom modules and will this method prevent BatchNorm2d layers within ConvBNReLU modules from being updated during training ?</p>

username: ptrblck
cooked: <p>If I’m not mistaken, the recursive loop would enable the gradient computation for all nested <code>BatchNorm2d</code> layers even in the custom modules, as they would be reached last.<br>
You could verify it manually after using this approach via e.g.:</p>
<pre><code class="lang-python">print(model.custom_layer.bn_layer.weight.requires_grad)
</code></pre>
Generate more images for DCGAN

username: hahaau2020
cooked: <p>How can I generate more images, e.g., 1000 fake images from the DCGAN? The image quality is also low.</p>

username: ptrblck
cooked: <p>You could either increase the batch size to the desired number of images in <a href="https://github.com/pytorch/examples/blob/507493d7b5fab51d55af88c5df9eadceb144fb67/dcgan/main.py#L236-L237">these lines of code</a> or, in case you are running out of memory, you could generate image batches in a loop.</p>
Unable to download ImageNet Dataset

username: Soniya
cooked: <p>Hi Everyone!<br>
I’m trying to download ImageNet dataset as follows.</p>
<p>train_loader = torch.utils.data.DataLoader(torchvision.datasets.ImageNet(root=’./data’, split=‘train’, download=True))</p>
<p>But it throw the following error.</p>
<p>" " " HTTPError Traceback (most recent call last)</p>
<p><a href="https://pp7vu4kvk7e-b5278c2458b95a70-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20191009-080040-RC01_273785537#" rel="nofollow noopener"></a> in () 1 2 train_loader = torch.utils.data.DataLoader( ----&gt; 3 dsets.ImageNet(root=’./data’, split=‘train’, download=True)) 4 5 # test_loader = torch.utils.data.DataLoader(</p>
<p>11 frames</p>
<p><a href="https://pp7vu4kvk7e-b5278c2458b95a70-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20191009-080040-RC01_273785537#" rel="nofollow noopener">/usr/lib/python3.6/urllib/request.py</a> in http_error_default(self, req, fp, code, msg, hdrs) 648 class HTTPDefaultErrorHandler(BaseHandler): 649 def http_error_default(self, req, fp, code, msg, hdrs): --&gt; 650 raise HTTPError(req.full_url, code, msg, hdrs, fp) 651 652 class HTTPRedirectHandler(BaseHandler):</p>
<p>HTTPError: HTTP Error 404: Not Found " " "</p>
<p>Please help me. is there anyway to download the dataset.</p>

username: pmeier
cooked: <p>This was brought to our attention and is now an <a href="https://github.com/pytorch/vision/issues/1453" rel="nofollow noopener">issue</a> in the github repository. For now there is no official solution.</p>
<hr>
<p>Due to the sheer size I always downloaded it <a href="https://academictorrents.com/collection/imagenet-2012" rel="nofollow noopener">via torrent</a> which can be much faster if you have multiple seeders.</p>

username: Flock1
cooked: <p>Hi.</p>
<p>I want to know that I downloaded the 32X32 ImageNet dataset and when I displayed it, the image showed something like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a672e6426f0a565caac82971d473e9f368f62d88.png" alt="index" data-base62-sha1="nKtqITksnIggUhBqmGVdj34aG8w" width="251" height="249"></p>
<p>Is there something wrong that I am doing? <a href="https://stackoverflow.com/questions/66948436/imagenet-dataset-images-not-loading-properly" rel="noopener nofollow ugc">Here’s</a> my original question</p>
I don't understand the meaning of batch_size in tensor

username: Sjk_Jae
cooked: <pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.embed = nn.Embedding(input_size, hidden_size)
        self.lstm = nn.LSTM(hidden_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden, cell):
        out = self.embed(x)
        out, (hidden, cell) = self.lstm(out.unsqueeze(1), (hidden, cell))
        out = self.fc(out.reshape(out.shape[0], -1))
        return out, (hidden, cell)

    def init_hidden(self, batch_size):
        hidden = torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device)
        cell = torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device)
        return hidden, cell
</code></pre>
<ol>
<li>Does setting batch_size mean using mini-batch to train the model?</li>
<li>If so, batch_size = 1 means there is only one sample per batch, or is the gradient calculated once with all the data?</li>
<li>Moreover, I am not sure why the hidden and cell in init_hidden also need to set batch_size?</li>
</ol>

username: AbdulsalamBande
cooked: <p>Hello, LSTM in Pytorch  needs <code>batch_size</code> for the <code>input</code>, <code>hidden</code> and <code>cell</code>. If you look at the <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">documentation</a>, they all have to have the following dimension.</p>
<ul>
<li>
<strong>input</strong> of shape (seq_len, batch_size, input_size):</li>
<li>
<strong>hidden</strong> of shape (num_layers * num_directions, batch, hidden_size)</li>
<li>
<strong>cell</strong> of shape (num_layers * num_directions, batch, hidden_size)</li>
</ul>
<p>As you can see above, if the input is in batches, the hidden and cell also need to be in batches to account for the different batches in the input because each batch in the input will have its own hidden and cell.</p>
<p>Now, coming back to your first question. Yes setting batch_size is like mini-batch. Example if batch size is 3,  then each of  your input is a group of 3 sentences like <code>I love Pytorch</code> , <code>I love Keras</code> , <code>I love NLP</code>.</p>
<p>Setting batch_size = 1 means have just One sentence in each input.</p>
<p>Hope that helps.</p>

username: Sjk_Jae
cooked: <p>Thank you for your explanation <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Image with 4 channels

username: Doha_Bouallal
cooked: <p>i have images with 4-channels that i created by stacking RGB and thermal data. as follow:</p>
<pre><code class="lang-auto">
    i1= Image.open('rgb.png')   #rgb image
    i2 = Image.open('th.png')   #thermal image
    img = np.dstack((i1,i2))


</code></pre>
<p>I am working with pytorch to train a model on these images and at some point I have to process them separately. So I have to extract the RGB image and the thermal image. here is what I did:</p>
<pre><code class="lang-auto">def forward(self, image):      
        # split data into RGB and INF
        thermal = image[:,3:] 
        rgb = image[:,:3]   
</code></pre>
<p>will “thermal” extract the last channel of my 4-channels image and “rgb” will extract the 3 first channels ?</p>

username: ptrblck
cooked: <p>Assuming you are permuting the tensor to the channels-first memory layout and it contains a batch dimension, your code looks correct. A quick check would be:</p>
<pre><code class="lang-python">i1 = np.zeros((24, 24, 3))
i2 = np.ones((24, 24, 4))
img = np.dstack((i1, i2))
print(img.shape)
&gt; (24, 24, 7)

image = torch.from_numpy(img)
image = image.permute(2, 0, 1).unsqueeze(0)
thermal = image[:, 3:] 
rgb = image[:, :3]   
print((thermal == 1.).all())
&gt; tensor(True)

print((rgb == 0.).all())
&gt; tensor(True)
</code></pre>
Conditional image GAN is diverging

username: Badr_Belhiti
cooked: <p>I’m trying to implement a conditional GAN that takes in 128x128x1 images of the edges of facial images and produces the corresponding 128x128x1 “shaded in” image. For the discriminator, I’m concatenating the conditional input image and generated image on the channel dimension. The problem is that the discriminator and generator immediately diverge (discriminator loss goes to 100 and generator loss goes to 0). Here’s my discriminator architecture:</p>
<pre><code class="lang-auto"># nc = 1
# ndf = 32
class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu

        self.main = nn.Sequential(
            nn.Conv2d(nc * 2, ndf * 2, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
             
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            
            nn.Conv2d(ndf * 8, ndf * 16, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 16),
            nn.LeakyReLU(0.2, inplace=True),
            
            nn.Conv2d(ndf * 16, ndf * 32, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 32),
            nn.LeakyReLU(0.2, inplace=True),
            
            nn.Conv2d(ndf * 32, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        edges, grayscale = input
        # Concatenate images on channel dimension
        new_input = torch.cat((edges, grayscale), dim=1)
        return self.main(new_input)
</code></pre>
<p>I was thinking this architecture is fine, because the discriminator should be easily able to learn whether the generated image correlates with the conditional input by just using some edge detection filters and differences. It’s also worth noting that this architecture works fine with a traditional GAN with half the feature maps. Any help is appreciated, thank you!</p>
Help Needed ---- Getting Gradients in Multiprocessing

username: jmandivarapu1
cooked: <p>Hi,</p>
<p>I am currently using <a href="https://github.com/pytorch/examples/blob/master/mnist_hogwild/main.py" rel="noopener nofollow ugc">examples/main.py at master · pytorch/examples (github.com)</a>  for multiprocessing startup for my code. I used the below line to print my gradients but all them are NONE. So is there anyway to get these gradients.</p>
<p>I need the gradients somehow any help would be greatly appreciated.</p>
<pre><code class="lang-auto">for name, param in model.named_parameters():
        print(name,param.grad)
</code></pre>
<p>I even used the args.num_process=1 but the output is still the same</p>
<pre><code class="lang-auto">conv1.weight None
conv1.bias None
conv2.weight None
conv2.bias None
fc1.weight None
fc1.bias None
fc2.weight None
fc2.bias None
</code></pre>
<p>I even tried to use multiprocessing queue to return the gradients but the problem is that queue is not able to return that big size of the gradients.</p>
Balance multilabel

username: jony
cooked: <p>Hi all,</p>
<p>I’m trying to balance a multilabel data set with WeightedRandomSampler.<br>
So in the first place, all the examples are separated.</p>
<ol>
<li></li>
</ol>
<p>temp_dict = train_set[‘cls’].value_counts().to_dict() // Here I’m counting the classes<br>
print(temp_dict) → {0: 10814, 1: 2407, 2: 387}</p>
<ol start="2">
<li></li>
</ol>
<p>train_set[‘sample_w’] = [1/temp_dict[i] for i in train_set[‘cls’]] // making a column where each example gets the inverse of the class proportion: {0: 1/10814, 1: 1/2407, 2: 1/387}</p>
<ol start="3">
<li></li>
</ol>
<p>And now is the tricky part, the dataset contains few objects for every image, so I’m using groupby the path of the image, so the the class column has attributes like: [0,0,0] or [0,1] or [0,0,2] etc… :<br>
train_set.groupby(‘path’,as_index=False).agg({‘cls’: lambda x: list(x), ‘sample_w’:lambda x: np.mean(x)})</p>
<p>For the weights columns, I’m using the mean of the weights I calculated before, but I’m not sure if this is the right way to balance the data in a way that the neural net will see each class at the same amount.</p>
<p>This is how the dataset looks like at the end:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9ced31679381d5f643014129aeb9358439b078b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9ced31679381d5f643014129aeb9358439b078b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9ced31679381d5f643014129aeb9358439b078b.png" alt="image" data-base62-sha1="sNh5Eld6eqTfjHJzYTEh2bfh4Lh" width="335" height="499" data-dominant-color="4A4A4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">567×846 32.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But when I use WeightedRandomSampler and dataloder and count the classes that I get, I’m still getting imbalanced classes:<br>
0: 100, 1: 26, 2: 19</p>
<p>Hope I made myself clear enough, what am I doing wrong??<br>
Thanks</p>

username: ptrblck
cooked: <p>Based on your description it seems you are working on a multi-label classification use case.<br>
Weighting these samples is not straightforward (as e.g. in the multi-class classification) and some approaches are described in <a href="https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104/2">this post</a>.</p>
Issues with my test function when data is shuffled

username: jzy95310
cooked: <p>I use the following function to make predictions on the test dataset. However, I find that when I set shuffle=True in the data loader, the order of y_pred and y_true got messed up. I go over this function multiple times but I cannot see where I got wrong. Can someone please help me detect if there is any bug in the following code?</p>
<pre><code class="lang-auto">def test(model, device, test_loader, criterion):  
    model.eval()
    y_pred = torch.empty(0).to(device)
    y_true = torch.empty(0).to(device)
    with torch.no_grad():
        for img, meteo, target in test_loader:
            img, meteo, target = img.to(device), meteo.to(device), torch.squeeze(target.to(device))
            output = torch.squeeze(model(img, meteo))
            y_pred = torch.cat((y_pred, output))
            y_true = torch.cat((y_true, target))
    
    test_loss = criterion(y_pred, y_true)
    print('Test set Loss: {:.4f}'.format(test_loss))
    
    return y_pred, y_true, test_loss
</code></pre>
Choose the correct optimizer and parameters

username: soepblik
cooked: <p>I’m new to neural networks and i am trying to train my first neural network with pytorch.</p>
<p>I was working with matlab before and used the toolbox but i want to explore neural networks more and try to improve the network further and matlab does not have a lot of options.</p>
<p>I have just an CSV with input data (2 inputs) and output data(4 outputs) so I am not training with images.</p>
<p>I made a simple network with 1 hidden layer.</p>
<p>When i train the system in matlab i got an mse of about 6e-5 and now i train with pytorch i get about 8e-3 when training with adam optimizer.</p>
<p>What i always ask myself. How do you choose values(like batch size, betas,lr, epochs) for the optimizer and is adam the correct choice for this problem? Are the values just chosen by trial and error?</p>

username: cyRi-Le
cooked: <p>Neural nets training is probabilistic by nature because most of optimizers are and you shuffle your train_set, etc. Implementation may ( and surely) differ from MATLAB to PyTorch hence it is very unlikely you get same exact results, but comparable order of magnitude. Choosing hyper-parameters (batch_size, betas, lr etc.) mainly relies on experience. That said for some well-studied topics they are some recommendation (e.g Adam works well in NLP tasks, SGD works well on average etc.). You can also control dynamically the learning rate ( works well for some problems) or optimize some of your hyper-params using the package <code>ray[tune]</code>. Trial and error should be great to begin.</p>
GPU memory problems of nn.Parameter

username: sio277
cooked: <p>Hello, I have two cases as follows:</p>
<p><strong>CASE1</strong></p>
<pre><code class="lang-auto">k = []
for _ in range(500):
    T = torch.nn.Parameter(torch.rand(10, 3000, 3000).cuda(1))
    k.append(T.cpu())
</code></pre>
<p>which causes OOM, and the next case is</p>
<p><strong>CASE2</strong></p>
<pre><code class="lang-auto">k = []
for _ in range(500):
    T = torch.nn.Parameter(torch.rand(10, 3000, 3000)).cuda(1)
    k.append(T.cpu())
</code></pre>
<p>which does not raise OOM and the GPU memory is maintained as 1900 MB in entire for loop.</p>
<p>What makes difference between the two cases?</p>
Why the pytorch website doesn't have 1.3.1 version?

username: Yan_Wen
cooked: <p>Why the pytorch website doesn’t have the installation instructions about 1.3.1 version?</p>

username: ptrblck
cooked: <p>You can use the <a href="https://pytorch.org/get-started/previous-versions/">Previous versions</a> tab to check for all previous releases and select the one you would like to use.<br>
However, I would generally recommend to stick to the latest release to get the most new features and fixes.</p>

username: Yan_Wen
cooked: <p>Yeah, but do you find that the <a href="https://pytorch.org/get-started/previous-versions/" rel="noopener nofollow ugc">Previous versions</a> don’t have the 1.3.1 version.</p>

username: ptrblck
cooked: <p>It works for me, if I just replace the version:</p>
<pre><code class="lang-python">conda install pytorch=1.3.1 cudatoolkit=10.1 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
[...]

  added / updated specs:
    - cudatoolkit=10.1
    - pytorch=1.3.1


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-1.0                   |              mkl           6 KB
    cffi-1.14.5                |   py37h261ae71_0         224 KB
    cudatoolkit-10.1.243       |       h6bb024c_0       347.4 MB
    intel-openmp-2020.2        |              254         786 KB
    mkl-2020.2                 |              256       138.3 MB
    mkl-service-2.3.0          |   py37he8ac12f_0          52 KB
    mkl_fft-1.3.0              |   py37h54f3939_0         167 KB
    mkl_random-1.1.1           |   py37h0573a6f_0         322 KB
    ninja-1.10.2               |   py37hff7bd54_0         1.4 MB
    numpy-1.19.2               |   py37h54aff64_0          22 KB
    numpy-base-1.19.2          |   py37hfa32c7d_0         4.1 MB
    pytorch-1.3.1              |py3.7_cuda10.1.243_cudnn7.6.3_0       428.0 MB  pytorch
    six-1.15.0                 |   py37h06a4308_0          27 KB
    ------------------------------------------------------------
                                           Total:       920.9 MB
</code></pre>
How to determine overfitting, underfitting and best fit in deep learning? What about this graph?

username: sampa
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08ebe813feeb4d55450f51eba11c78e112433983.png" data-download-href="https://discuss.pytorch.org/uploads/default/08ebe813feeb4d55450f51eba11c78e112433983" title="loss_plot2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebe813feeb4d55450f51eba11c78e112433983_2_628x500.png" alt="loss_plot2" data-base62-sha1="1gVfkl5V5MQFYynymSssc8T6vZ1" width="628" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebe813feeb4d55450f51eba11c78e112433983_2_628x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/08ebe813feeb4d55450f51eba11c78e112433983_2_942x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/8/08ebe813feeb4d55450f51eba11c78e112433983.png 2x" data-dominant-color="FAF7F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_plot2</span><span class="informations">990×788 91.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mailcorahul
cooked: <p>Overfitting: train loss continues to decrease while test/val loss increases<br>
Underfitting: train loss remains high and doesn’t decrease(not constant). can be better determined using accuracy on train set rather than loss.<br>
Best fit: Best to choose the weights giving high accuracy(or any metric depending on the problem) on test/val set rather than finding it with loss.</p>
<p>In the above graph, you can choose the weights at <code>Early Stopping Checkpoint</code> point.</p>

username: hesham_gabr
cooked: <p>how can i avoid underfitting and decrease the training loss?</p>