akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RNN/LSTM on IMU-Data

username: arash_javanmard
cooked: <p>Hello,</p>
<p>at the moment I am working on a project, where I try to use IMU-Measurements (linear accl. and angular velocity) and a LSTM based Netowork to predict the translation and orientation of a mobile agent, in this specific case I use KITTI Dataset, you can find the project <a href="https://github.com/ArashJavan/DeepLIO" rel="nofollow noopener">here (DeepLIO)</a>  (checkout branch deepio).  The network is built as follow:</p>
<pre><code class="lang-auto">class DeepIO(nn.Module):
    def __init__(self):
        super(DeepIO, self).__init__()
        
        self.bidirectional = False
        self.num_dir = 2 if self.bidirectional else 1

        self.rnn = nn.LSTM(input_size=6, hidden_size=512,
                           num_layers=2, batch_first=True, bidirectional=self.bidirectional)

        self.drop_out = nn.Dropout(0.25)
        self.fc1 = nn.Linear(512, 256)
        self.bn1 = nn.BatchNorm1d(256)
        
        self.fc_pos = nn.Linear(256, 3)
        self.fc_ori = nn.Linear(256, 4)

    def forward(self, x):
        """
        args:
        x: a list (batch) of sequences with different length 
        """
        lengths = [x_.size(0) for x_ in x]   # get the length of each sequence in the batch
        x_padded = nn.utils.rnn.pad_sequence(x, batch_first=True)  # padd all sequences
        b, s, n = x_padded.shape
        
        # pack padded sequece
        x_padded = nn.utils.rnn.pack_padded_sequence(x_padded, lengths=lengths, batch_first=True, enforce_sorted=False)
        
        # calc the feature vector from the latent space 
        out, hidden = self.rnn(x_padded)
        
        # unpack the featrue vector
        out, lens_unpacked = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
        out = out.view(b, s, self.num_dir, self.hidden_size[0])

        # many-to-one rnn, get the last result
        y = out[:, -1, 0]

        y = F.relu(self.fc1(y), inplace=True)
        y = self.bn1(y)
        y = self.drop_out(y)

        x_pos = self.fc_pos(y)
        x_ori = self.fc_ori(y)
        return x_pos, x_ori
</code></pre>
<p>Where the input to the network is a list of sequences with different length, e.g. BxTx6, where B ist the batch-size (the list) and T is the sequence length of each sequence and 6 is are the length of each IMU<br>
measurement (3xlinar accl, 3xangualr velocity).</p>
<p>I would like to know if the way I handle these sequences with different lengths is correct?<br>
Where I first pad them with zero, so they have the same length, after that, I pack them and pass to the RNN (LSTM) and at the end repack them with many-to-one output.</p>
<p>Thanks<br>
Arash</p>
How can i use module.name

username: lkchenxicvi
cooked: <p>m0 is a module of my model.I want to do something when name of m0 is ‘conv1’.How can I do it？</p>

username: NoobCoder
cooked: <p>If you mean that you have a module called <code>conv1</code> in a model named <code>m0</code>, you can access it with this command:</p>
<p><code>m0.conv1 </code></p>

username: lkchenxicvi
cooked: <p>There are some errors in my statement.My code show as below.</p>
<pre><code class="lang-auto">for layer_id in range(len(model)):
    m0 = old_modules[layer_id]
</code></pre>
<p>I want to make a restriction on m0 when its name is conv1.</p>

username: ptrblck
cooked: <p>Maybe this would work:</p>
<pre><code class="lang-python">for name, module in model.named_modules():
    if 'conv1' in name:
        print('found')
</code></pre>

username: lkchenxicvi
cooked: <p>Thanks.m0 is a module.I need to judge whether the name of m0 is ‘conv1’.</p>
<pre><code class="lang-auto">for layer_id in range(len(model)):
    m1 = new_modules[layer_id]
    if isinstance(m0, nn.Conv2d):
       if m0.name == 'conv1':   # m0.name is a kind of expression, I don't know how to call it.
            idx1 = [0, 1]
</code></pre>
<p>This is the judgment I need to make.</p>

username: ptrblck
cooked: <p>Would my suggestion not work on your <code>model</code>?<br>
How did you define <code>new_modules</code>? Standard <code>nn.Modules</code> wouldn’t have the <code>name</code> attribute, so you should stick to <code>named_modules()</code> at one point (but I’m unsure how your classes are defined).</p>

username: lkchenxicvi
cooked: <p>It can’t solve my problem.m0 is a layer in my model.</p>
<pre><code class="lang-auto">if isinstance(m0, nn.Conv2d):
        for name,module in m0.named_modules():
            if 'conv1' in name:
                m1.weight.data = m0.weight.data.clone()
</code></pre>

username: ptrblck
cooked: <p><code>m0</code> seems to be undefined in the first condition, so you would probably have to move it inside.<br>
Also, you are calling <code>m0.named_modules()</code>, while you should call it on the model.<br>
After that <code>m1</code> would be undefined, so you would have to add some more information what you are trying to do.</p>
<p>If you just want to access <code>m0</code> and <code>m1</code>, which are names of two modules, you could use:</p>
<pre><code class="lang-python">model.m0
model.m1
</code></pre>

username: lkchenxicvi
cooked: <p>m0 has been defined.I will try other way.This judgement is important for my experiment.Thank you for your help.</p>
Is it safe to get raw tensor data by tensor.flatten().data_ptr<T>()?

username: WeiLai5432
cooked: <p>I want to get raw data of tensor and return an array, currently I’m doing like this:</p>
<pre><code class="lang-auto">...
torch::Tensor flatten_tensor = input.to(torch::kCPU).flatten();
tensorSizes = output.sizes();
*arrayProto.mutable_float_val() = {flatten_tensor.data_ptr&lt;float&gt;, flatten_tensor.data_ptr&lt;float&gt; + tensor_size};
...
torch::Tensor output = torch::empty(torch::IntArrayRef(tensorSizes), tensorOptions);
memcpy(output.data_ptr&lt;float&gt;(), arrayProto.float_val().data(), tensor_size*sizeof(float));
...
</code></pre>
<p>I’m not sure:</p>
<ol>
<li>Whether the tensor input.to(torch::kCPU) is contiguous in memory? (input may in cuda memory or cpu memory)</li>
<li>If it’s contiguous, will the flatten_tensor return the right values of the tensor?</li>
<li>Can I get the right value (same as input) in output? Is there any risk in my code ?</li>
</ol>
<p>I tried some simple models and tensors by this and it works. But I’m worrying it’s not 100% work in any scenario.</p>
<p>Thank you very much!</p>

username: Hartbook
cooked: <p>As you said, the problem with this approach is the possible non-contiguity of the tensor representation in memory. You can test it with the <code>is-contiguous</code> tensor member function. If your vector is non-contiguous, you can use the <code>strides</code> member function to get for each tensor dimension  the required number of “jumps” between each element of the tensor.</p>
<p>The safest way to achieve <em>constructing a new tensor from the data_ptr of another</em>, would in my opinion be :</p>
<pre><code class="lang-auto">auto new_tensor = torch::from_blob(input.data_ptr(), input.sizes(), input.strides(), torch::TensorOptions(torch::kFloat));
</code></pre>
Unable to load sparse tensor

username: usman095
cooked: <p>I’m trying to save sparse tensor and then load them back. The save step executes without any problem but when I try to load it back I get a weird error</p>
<pre><code class="lang-auto">"RuntimeError: size is inconsistent with indices: for dim 1, size is 10 but found index 4934256341737799680"
</code></pre>
<p>Code to reproduce this error is given below:</p>
<pre><code class="lang-python">ind = torch.LongTensor([[0]*4, [2, 4, 7, 9]])
val = torch.FloatTensor([5, 6, 6, 6])
sp_tensor = torch.sparse_coo_tensor(ind, val, torch.Size([1, 10]))
torch.save(sp_tensor, "test.pt")
sp_tensor = torch.load("test.pt")
print(sp_tensor)
</code></pre>

username: ptrblck
cooked: <p>Thanks for reporting this issue! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
It seems to related to <a href="https://github.com/pytorch/pytorch/issues/33439">this issue</a> with a potential fix in <a href="https://github.com/pytorch/pytorch/pull/34059">this PR</a>, which is currently being reviewed.</p>

username: usman095
cooked: <p>Got it. Thank you!</p>
<p>Is there a recommended workaround for the time being?</p>

username: ptrblck
cooked: <p>As a workaround you could save the tensor in its dense form via:</p>
<pre><code class="lang-python">torch.save(sp_tensor.to_dense(), 'test.pt')
</code></pre>
State_dict truncates some tensors

username: Brian_Xu
cooked: <p>Hello,</p>
<p>I built a DQN in PyTorch for Kaggle’s ConnectX competition. Due to the format of the competition, I need to save the entire network as a string. However, state_dict() seems to be cutting the string output of some of the tensors arbitrarily.</p>
<p>Here’s an example:</p>
<pre><code class="lang-auto">conv3.weight 	 tensor([[[[-0.3832]],

         [[-0.4524]],

         [[-0.0259]],

         ...,

         [[ 0.0565]],

         [[ 0.2471]],

         [[-0.4269]]],
</code></pre>
<p>As you can see, some of the tensor has been removed and replaced by the ellipsis (…).<br>
Is there any way to circumvent this issue?</p>

username: Brian_Xu
cooked: <p>Hello,</p>
<p>I found a solution to the problem. This isn’t related to state_dict at all, but is rather just the string returned by a torch.Tensor being capped.</p>
<p>Use torch.set_printoptions(profile=“full”) to increase the limit.</p>
How to get the imagenet dataset on which pytorch models are trained on

username: spaul13
cooked: <p><em><strong>Can anyone please tell me how to download the complete imagenet dataset on which the pytorch torchvision models are trained on and their Top-1 error is reported on?</strong></em></p>
<p>I have downloaded Tiny-Imagenet from Imagenet website and used pretrained resnet-101 model which provides only 18% Top-1 accuracy.</p>

username: ptrblck
cooked: <p>The torchvision classification models should have been trained on <a href="http://www.image-net.org/challenges/LSVRC/2012/">ILSVRC2012</a>, so you could download the dataset using the provided link.</p>
<p>Did you use the ImageNet example code from the PyTorch repository or how did you calculate the accuracy for Tiny-ImageNet?</p>

username: spaul13
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot for the reply. I first downloaded tiny-imagenet dataset which has 200 classes and each with 500 images from imagenet webpage then in code I get the resnet101 model from torchvision.models and perform inference on the train folder of tiny-imagenet. which provides only 18% accuracy as I mentioned earlier.</p>

username: spaul13
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can u please tell me how to get ILSVRC2015 dataset seems like resnet101 pretrained models on torchvision trained on that.</p>

username: ptrblck
cooked: <p>I think it’s trained on ILSVRC2012 (sorry, had a typo earlier):</p><aside class="quote" data-post="2" data-topic="7278">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/using-vision-models-with-the-cifar-dataset/7278/2">Using vision.models with the CIFAR dataset?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    The pre-trained models are all based on Imagenet-12. Most of the model definitions also need 224x224 as the minimum input size. For CIFAR, you are better off with other repos.
  </blockquote>
</aside>

<p>The provided link should be a Download section.</p>
How many classes does a binary classifier needs?

username: alx
cooked: <p>I have 5 classes to predict. When I try to predict all of them using one model it doesn’t work that well so I decided to stack 5 binary classification models.</p>
<p>Say I want predict class 1 and I have a folder with all class 1 images and another folder with classes 0,2,3,4.</p>
<p>After training, the output of my model looks like:<br>
class 0: 51.39% | class 1: 81.57%</p>
<p>My question is the following, does the ability of the model to predict class 0 (composed of 0234) is important here?</p>
<p>I believe it would be beneficial for the model to see other classes but I can’t wrap my head around why it would be important to have 100% accuracy for both classes.</p>

username: ptrblck
cooked: <p>It depends a bit on your use case. Are you planning on using multiple stages?<br>
I.e.:</p>
<pre><code class="lang-python">stage0 -&gt; class0 vs. rest
stage1 -&gt; class1 vs. rest
stage2 -&gt; class2 vs. rest
...
</code></pre>
<p>If so, than the “rest” class prediction would matter, since the following stages won’t be able to see this sample anymore as it was already wrongly classified.<br>
Could you also explain a bit, what the current outputs (51% and 81%) mean?</p>

username: alx
cooked: <p>Well yes you’re exactly right. I’m planning to implement a one vs all method. I was under the impression it would not matter because if the model that classifies class 1 vs rest does not detect the image as class 1 it would pass it to the next model.</p>
<p>The 51% and 81% outputs means the following: the model was able to detect class 1 as class 1 81% of the time and class 0 (composed of the rest ie 0234) 51% of the time.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="alx" data-post="3" data-topic="84557">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alx/48/17306_2.png" class="avatar"> alx:</div>
<blockquote>
<p>because if the model that classifies class 1 vs rest does not detect the image as class 1 it would pass it to the next model.</p>
</blockquote>
</aside>
<p>The next model could only classify it as “not class 2”, which is correct for this stage, but of course you’ll never correctly classify this image as class 1.</p>
Libtorch C++ what is the best approach to access submodule list

username: honglh
cooked: <p>for a pytorch snippet that gets the list of submodules of a module and check the module type</p>
<pre><code class="lang-auto">   for m in self.modules():
     if isinstance(m, nn.BatchNorm2d):
       #do_something
</code></pre>
<p>Is this a good C++ implementation of it?  My concern is that I am using the shared_ptr reference that libtorcy try to hide with ModuleHolder</p>
<pre><code class="lang-auto">  for (auto m : children()) {
    if (dynamic_cast&lt;torch::nn::BatchNorm2dImpl *&gt;(m.get())) {
      //do_something
    }
  }
</code></pre>
<p>The submodules are already registered via register_module() in constructor.</p>

username: honglh
cooked: <p>found the proper API <code>as()</code></p>
<pre><code class="lang-auto">for (auto item : named_children()) {
  if (item.value()-&gt;as&lt;torch::nn::BatchNorm2d&gt;()) {
  //do_something
}
</code></pre>
Help with 3D point cloud data

username: Kushaj
cooked: <p>I am trying to implement DeepVCP paper and for that I need to load KITTI dataset. I have downloaded KITTI odometry data, but I am stuck on how to proceed from there. This is my first time working with 3D data so I am very confused.</p>
<p>In the extracted folder there is a <code>calib.txt</code> file which has to be used in some calibration. Then there are <code>image2</code> and <code>image3</code> folders that contain the left and right images. And finally, the <code>velodyne</code> folder containing the bin files for the point clouds.</p>
<p><strong>Ques</strong>: Do I need to use both the left and right images? Or do I have to combine them into a single image.</p>
<p>I checked <a href="https://github.com/utiasSTARS/pykitti" rel="nofollow noopener">pykitti</a> repo. But I was not able to make much progress. The thing that is confusing me right now is how to use the calibration file and what to do with the left and right images.</p>
<p>Has anyone else worked with this dataset? Can you share some code samples on how to create a dataloader for such a task?</p>

username: arash_javanmard
cooked: <p>Hi Kushaj, I am also interested in using KITTI point clouds in PyTorch. Have you made any progress on this?<br>
By the way for DeepVCP you need veloyne point cloud not images.</p>
<p>Best regards<br>
Arash</p>

username: sepehrfard
cooked: <p>Hello,</p>
<p>I have also the same question as it is my first time working with 3D data. I would like to use a Dataloader for KITTI Velodyne folder to be used for DeepVCP which has a pointnet++ as it’s first layer of of the model. Any help would be appreciated.</p>
<p>Thank you!</p>

username: arash_javanmard
cooked: <p><a class="mention" href="/u/kushaj">@Kushaj</a>, <a class="mention" href="/u/sepehrfard">@sepehrfard</a><br>
you can take a look at <a href="https://github.com/ArashJavan/PointSeg" rel="nofollow noopener">pointseg</a> or <a href="https://github.com/ArashJavan/DeepLIO" rel="nofollow noopener">DeepLIO</a> in both projects 3D Lidars are projected to images using spherical projection.</p>
Define some layers only for training stage

username: Alpha
cooked: <p>Hi,</p>
<p>Is there a way to define some layers only for the training stage?</p>
<p>I know there is a <code>self.training</code> flag which can be used in the <code>forward()</code> function. However, when the layers are defined in <code>__init__()</code>, the <code>self.training</code> is always <code>True</code>. So, I can’t use <code>self.triaining</code> in the <code>__init__()</code> function.</p>

username: Nikronic
cooked: <p>Hi,</p>
<p><code>self.training</code> is available everywhere in your code as it is defined in <code>nn.Module</code> which when you extend this class to create your own, it will be copied to your model definition too. So, you can set it to <code>false</code> in <code>init</code>.</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.training = False
        self.fc = nn.Linear(1, 1)
        
    def forward(self, x):
        x = self.fc(x)
        if self.training:
            x = x + 1000
        return x

model = MyModel()
x = torch.ones(1, 1)
</code></pre>
<h3>testing</h3>
<pre><code class="lang-auto">model.training = False
model(x)  # a small number such as 1.3203
</code></pre>
<pre><code class="lang-auto">model.training = True
model(x)  # a large number around thousand
</code></pre>
<p>Bests</p>

username: Alpha
cooked: <p>Thanks for your reply. But  what I want to is :</p>
<pre><code class="lang-auto"> class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        if self.training:
            self.auxiliary = nn.Conv2d(in, out)
</code></pre>
<p>The <code>auxiliary</code> layer is only defined for training state.</p>

username: Nikronic
cooked: <p>Can you tell me why aren’t you satisfied with using <code>.training</code> in <code>forward</code> method? It is really simple to implement and obviously follows convention. I cannot think of any situation that this cannot handle.</p>
How to include the training data as learnable parameters in Pytorch?

username: mehran2020
cooked: <p>Hi all,<br>
I am a trained model. Now, I need to feed some random data to it, and then use the gradients to update the random data! I am hoping to see meaningful patterns in the learned data at the end. This is similar to activation maximization in Neural Networks.<br>
However, I am not sure how to add my tensor data into the optimizer! Could anyone help me with this?</p>
<p>Thanks a lot</p>

username: albanD
cooked: <p>Hi,</p>
<p>You will need to make sure that your data properly require gradients.<br>
Then you can pass it to the optimizer as any other parameter.</p>

username: mehran2020
cooked: <p>Thank you so much. I will see how it works out</p>
How the networks learns image captioning

username: cevvalu
cooked: <p>Hi, i am a bit confused in image captioning. This is my first time building an image captioning model (i have built a multimodal learning based image captioning model where a cnn extracts image features and a lstm is responsible for generating sentences).<br>
I searched encoder-decoder based, attention based and multimodal learning based image captioning models but i am not sure how the training is going. But for all models i guess the network is trained on the likelihood of training sentences and image features. A general explanation would be great…thank you</p>
Is it possible to write this without a for loop (to speed it up)

username: divinho
cooked: <pre><code>def forward(x, dur_cnts):
    N = x.size(0)
    T = x.size(1)
    durs, counts = dur_cnts
    idx = 0
    joined = to.zeros((N, self.dim_in), dtype=to.float32, device=x.device)
    for i, (dur, count) in enumerate(zip(durs, counts)):
        x_part = x[idx: idx + count, :dur]
        joined[idx: idx + count] = x_part.sum(dim=1)
        idx += count
</code></pre>
<p>The input is size <code>(N, T, C)</code>, and I’m taking differently sized matrices (&lt;&lt;T) from it and then summing across the T dimension.</p>
<p>I think it would be possible with some sort of gather_add operation</p>

username: albanD
cooked: <p>Hi,</p>
<p>I’m afraid that if you have general indices and counts, you won’t be able to do this in a single op.<br>
You will need a custom kernel that implements this.</p>
Writing c++/CUDA extension and inline void parallel_run() does not work

username: zhryang
cooked: <p>I am writing pytorch extension using c++/CUDA. But I meet some problem, I want to use  <strong>inline void parallel_for(const int64_t begin, const int64_t end, const int64_t grain_size, const F&amp; f);</strong>  funtion in  <strong>ATen/Parallel.h</strong> . When I finish the code, I meet this bug:</p>
<p><em>/.local/lib/python3.6/site-packages/torch/include/ATen/Parallel.h:48:13: error: ‘void at::parallel_for(int64_t, int64_t, int64_t, const F&amp;) [with F = geomean_pool2d_backward_out_frame(scalar_t</em>, scalar_t*, scalar_t*, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int, int, int, int, int, int, bool, c10::optional) [with scalar_t = float; int64_t = long int]::&lt;lambda(int64_t, int64_t)&gt;; int64_t = long int]’, declared using local type ‘const geomean_pool2d_backward_out_frame(scalar_t*, scalar_t*, scalar_t*, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int, int, int, int, int, int, bool, c10::optional) [with scalar_t = float; int64_t = long int]::&lt;lambda(int64_t, int64_t)&gt;’, is used but never defined [-fpermissive]*</p>
<pre><code>  Can you help me solve this problem? I really want to how to use inline void at::parallel_for() in my code. Please help me!</code></pre>

username: Kaii
cooked: <p>Hi, I have tried directly use parallel_for in ATen/ParallelOpenMP.h instead of ATen/Parallel.h. Works for me.  Do not know why. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Pytorch Lsun dataset

username: Muhammed_Dalkiran
cooked: <p>My code does not continue at<br>
with self.env.begin(write=False) as txn:<br>
self.keys = [key for key, _ in txn.cursor()]</p>
<p>my output is :<br>
length:  168103<br>
1<br>
my code is :<br>
class LSUNClass(data.Dataset):<br>
def <strong>init</strong>(self, db_path, base_size=64,<br>
transform=None, target_transform=None):<br>
import lmdb<br>
self.db_path = db_path<br>
self.env = lmdb.open(db_path, max_readers=1, readonly=True, lock=False,<br>
readahead=False, meminit=False)<br>
with self.env.begin(write=False) as txn:<br>
self.length = txn.stat()[‘entries’]<br>
print('length: ', self.length)<br>
cache_file = db_path + ‘/cache’<br>
if os.path.isfile(cache_file):<br>
self.keys = pickle.load(open(cache_file, “rb”))<br>
print(‘Load:’, cache_file, 'keys: ', len(self.keys))<br>
else:<br>
print(“1”)<br>
with self.env.begin(write=False) as txn:<br>
self.keys = [key for key, _ in txn.cursor()]<br>
print(“3”)<br>
pickle.dump(self.keys, open(cache_file, “wb”))<br>
Actually my code is like stopping at<br>
with self.env.begin(write=False) as txn:<br>
self.keys = [key for key, _ in txn.cursor()]<br>
I did not solve this problem.<br>
Have you any idea?</p>

username: ptrblck
cooked: <p>What is <code>txn</code> doing and is the code working without using this library?<br>
If so, try to use <code>num_workers=0</code> in your <code>DataLoader</code> (if not already done) and rerun it.<br>
If that’s working, the <code>txn</code> lib might run into some multiprocessing issues with the PyTorch <code>DataLoader</code>.</p>
<p>PS: You can add code snippets by wrapping them into three backticks ```, which makes debugging a bit easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Muhammed_Dalkiran
cooked: <p>First of all, thank you for your response.<br>
I do not know what txn’s function is in the code since I found this code on internet. However, when I make a changing that you say, code is not working.</p>

username: Muhammed_Dalkiran
cooked: <aside class="quote no-group" data-username="Muhammed_Dalkiran" data-post="1" data-topic="84428">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/muhammed_dalkiran/48/25228_2.png" class="avatar"> Muhammed_Dalkiran:</div>
<blockquote>
<p>self.keys = [key for key, _ in txn.cursor()]</p>
</blockquote>
</aside>
<p>Again thank you for your interest and advices.<br>
I have just solved the problem. The problem is not related to wrong implementation, it is about google colab. I searched the problem online deeply and I saw that for huge dataset this problem may occur in google colab when reading dataset from google drive. I upload my dataset into google colab itself, not into drive. Then problem does not exist. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to load (image, caption) pairs and transform text to LSTM input?

username: swim_airport
cooked: <p>Hi there, I’m very new to <code>torchtext</code> module. I would like to load my own (image, caption) pairs and transform the text caption to tensor as LSTM input.</p>
<p><code>imgdir</code> is a folder with <code>.jpg</code> image files. <code>txtdir</code> is a folder with <code>.txt</code> files and each <code>.txt</code> file is one line of sentence.</p>
<p>First I’d like to transform each text sentence to a valid input to LSTM. Which steps should I use, like tokenize, vocabulary dictionary, word embeddings? I’m not clear about this process. Could you kindly please show me some examples? Or where can I find some tutorials?</p>
<p>Second is about data loader. My code is as follows (text is still represented as an alphabet vector):</p>
<pre><code class="lang-python">import glob, os
import torch
import torchvision.transforms as transforms
from torch.utils.data import Dataset, DataLoader
import numpy as np
from PIL import Image
MEAN = np.asarray([0.4914, 0.4822, 0.4465])
STD = np.asarray([0.2023, 0.1994, 0.2010])
NRM = transforms.Normalize(MEAN, STD)
TT = transforms.ToTensor()
TPIL = transforms.ToPILImage()
transform_no_aug   = transforms.Compose([TT, NRM])

class DatasetMaker(Dataset):
    def __init__(self, txtdir, imgdir, transformFunc=transform_no_aug):
        '''
        :param txtdir: .txt files in txtdir, each is one line, lowerized, all in alphabet, and len&lt;MAXLEN
        :param imgdir:
        :return self.img_caption_pairs is [(np.ndarray_uint8_640x480x3, np.ndarray_uint8_(MAXLEN,))]
        '''
        self.MAXLEN = 400
        self.transformFunc = transformFunc
        self.alphabet = "abcdefghijklmnopqrstuvwxyz0123456789-,;.!?:'\"/\\|_@#$%^&amp;*~`+-=&lt;&gt;()[]{} "
        self.alphabetDict = {self.alphabet[idx]: idx + 1 for idx in range(len(self.alphabet))}
        self.txtnames = [os.path.basename(path) for path in glob.glob(f'{txtdir}/*.txt')]
        self.length = len(self.txtnames)
        self.img_caption_pairs = []
        for txtname in self.txtnames:
            imgname = os.path.join(imgdir, txtname.replace('txt', 'jpg'))
            with open(os.path.join(txtdir, txtname), 'r') as fin:
                caption_text = fin.readline().strip()
            # image = np.asarray(Image.open(imgname), dtype=np.uint8)
            image = Image.open(imgname)
            caption_mat = self.txt2mat(caption_text)
            self.img_caption_pairs.append((image, caption_mat))

    def __getitem__(self, i):
        image, caption_mat = self.img_caption_pairs[i]
        image = self.transformFunc(image)
        caption_mat = torch.from_numpy(caption_mat)
        return image, caption_mat

    def __len__(self):
        return sum(self.length)

    def txt2mat(self, caption_text):
        char_mat = np.zeros((self.MAXLEN), dtype=np.uint8)
        caparr = np.array([self.alphabetDict[char] for char in caption_text], dtype=np.uint8)
        char_mat[:len(caparr)] = caparr
        return char_mat
</code></pre>
<p>How can I optimize this part of code?</p>
<p>Any possible help is appreciated. Thx!</p>
Loading a large dataset and training them

username: jayanth
cooked: <p>Hello guys, I am new to Pytorch. I am trying to load a large dataset of images(around 5 million images). I use the DataLoader to load the images. But it takes so much time to load and process 1 epoch (1 epoch roughly takes 10 hours). I use 2 V100 GPUs for this process. I went through some answers but could not get them. Kindly let me know if there is any way to do this efficiently. Thank you</p>

username: ptrblck
cooked: <p>You could profile your code and try to isolate, if the data loading is the bottleneck.<br>
If that’s the case, <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> might be helpful to further optimizer the loading.</p>

username: alx
cooked: <p>There’s a few ways to optimize the training time but if you already implemented them there isn’t much you can do except get better GPUs.</p>
<p>Have you tried:</p>
<ul>
<li>resizing you images</li>
<li>cropping your images</li>
<li>augmenting the batch size</li>
<li>augmenting the number of workers</li>
<li>freezing the model, only using the last layer</li>
</ul>
<p>Personally, this previously got me from 3hrs per epoch to 10min.</p>
RNN is not able to overfit a single sample

username: Federico_Vasile
cooked: <p>Hi everyone, I’m trying to practice with RNN doing the task of action detection in untrimmed video. Basically, the task is to predict each frame of the video as background_class or action_class, i.e. localizing when the actions occur within the video.<br>
The model is a simple CNN and the features extracted are passed to an LSTM that performs the prediction at every timestep.<br>
The problem is that my model is not able to completely overfit a single video sample(the loss doesn’t go any lower than about 2.0 after 80 epochs, should it go down a little bit more?).<br>
Is there a problem in my model? Maybe in the forward pass? If not, any suggestions on how to improve or change the model?</p>
<pre><code>class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()

    def forward(self, x):
        return x.view(x.shape[0], -1)

class VideoModel(nn.Module):
  def __init__(self, hidden_dim=512, num_classes=22, dtype=torch.float32):
    super().__init__()
    self.hidden_dim = hidden_dim
    self.dtype = dtype
    self.num_classes = num_classes

    self.feat_extr = models.resnet18(pretrained=True)
    FEAT_VECT_DIM = 512  # should not be hard-coded. TODO
    self.feat_extr = nn.Sequential(
        *list(self.feat_extr.children())[:-1],
        Flatten()
    )
    for param in self.feat_extr.parameters():
      param.requires_grad = False

    self.lin_transf = nn.Sequential(
        nn.Linear(FEAT_VECT_DIM, FEAT_VECT_DIM//2),
    )
    
    self.lstm = nn.LSTMCell(FEAT_VECT_DIM//2, hidden_dim)
    self.classifier = nn.Linear(hidden_dim, num_classes)

  def forward(self, x):
    # x.shape == (batch_size, frames_per_sample, 3, 180, 320)
    h_n = torch.zeros(x.shape[0], self.hidden_dim, dtype=self.dtype, device=x.device)
    c_n = torch.zeros(x.shape[0], self.hidden_dim, dtype=self.dtype, device=x.device)
    scores = torch.zeros(x.shape[0], x.shape[1], self.num_classes, dtype=self.dtype, device=x.device)
    for step in range(x.shape[1]):
      x_t = x[:, step]
      out = self.feat_extr(x_t)
      out = self.lin_transf(out)
      h_n, c_n = self.lstm(out, (h_n, c_n))    
      scores[:, step, :] = self.classifier(h_n)     # (batch_size, num_classes)
    return scores
</code>
</pre>
<p>A simple version of the code to replicate my results can be found here: <a href="https://colab.research.google.com/drive/1X17NfHl7ZXCVnrfnNMYkIz6XHClYr7PH?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1X17NfHl7ZXCVnrfnNMYkIz6XHClYr7PH?usp=sharing</a></p>
Gradients not returned by register_hook upon loading a saved model dict

username: theairbend3r
cooked: <p>Here’s my classifier -</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    def __init__(self, num_classes):
        super(Classifier(, self).__init__()
        
        
        self.base_conv = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=2)
        
        self.conv_block_1 = ConvBlock(in_channels=16, out_channels=32)
        self.conv_block_2 = ConvBlock(in_channels=32, out_channels=64)
                
        self.lastcnn = nn.Conv2d(in_channels=64, out_channels=num_classes, kernel_size=28, stride=1, padding=0)
        
        self.gradients = None
        
        
    def forward(self, x):
        
        x = self.base_conv(x)
        
        x = self.conv_block_1(x)
        x = self.conv_block_2(x)
                                       
        if x.requires_grad: x.register_hook(self.store_gradients)

        x = self.lastcnn(x)
        
        return x
      
      
    def store_gradients(self, grad):
        self.gradients = grad    
    
    def get_gradients(self):
        return self.gradients
</code></pre>
<p>After I training this model, I save it as a dict using <code>torch.save(model.state_dict(), PATH)</code>.</p>
<p>On loading the model, when I try to access it’s <code>get_gradient</code> method as follows, no gradients are returned.</p>
<pre><code class="lang-auto">model.eval()
pred = model(single_image.to(device))
pred = torch.softmax(pred, dim = 1)
pred_idx = pred.argmax(dim=1).item()

gradients = model.get_gradients()
gradients  #this is empty
</code></pre>
<p>On the flipside, If run the exact same code after training (instead of saving and loading), gradients are returned.</p>
<p>I tried saving the whole model using <code>torch.save()</code> and then load it again. The code works. So, it’s because of the parameters not getting saved in the <code>model dict</code>. Any way to store all the additional parameters when doing <code>torch.save(model.state_dict(), PATH)</code>?</p>

username: tom
cooked: <p>Saving and loading the model doesn’t restore gradients nor tensors assigned as attributes. It will save and restore parameters (<code>nn.Parameter</code>) and buffers. As your gradient is not a parameter, you could use <code>register_buffer</code> or so.<br>
Whether or not saving and restoring the gradient is a good idea in general is another topic, but as long as it’s code for yourself, you’re the arbiter of this. I would probably advise against doing so for things you want to put into a library or somesuch.</p>
<p>Best regards</p>
<p>Thomas</p>

username: theairbend3r
cooked: <p>I’m using it for GradCAM. I train multiple models =&gt; save them =&gt; load all of them in a new script and run forward pass =&gt; get gradients and plot gradcam heatmaps.</p>
<p>That’s why I need those parameters. Is there a better way to do this?</p>
<p>Also, how would I go about saving the parameters I use <code>register_hook</code> to extract into <code>register_buffer</code>? And once stored, how I would I go about loading my model?</p>
Question about CrossEntopyLoss

username: sahiluppal2k
cooked: <p>I have my logits of shape (16, 100, 1024), where 16 is batch size, 100 is sequence length, and 1024 is feature length.<br>
My target is of shape (16, 100), where 16 is batch size and 100 is sequence length</p>
<p>while applying nn.CrossEntropyLoss on the both i get the following error<br>
criterion(pred, target)<br>
<span class="hashtag">#ValueError:</span> Expected target size (16, 1024), got torch.Size([16, 100])</p>
<p>wheres this is working with this snipped<br>
criterion(pred[0], target[0])</p>
<p>what am i doing wrong?</p>

username: ptrblck
cooked: <p>I assume <code>feature_length=1024</code> corresponds to the number of classes in the logit tensor?<br>
<code>nn.CrossEntropyLoss</code> expects the output as <code>[batch_size, nb_classes, seq_length]</code> and the target as <code>[batch_size, seq_length]</code>. Based on the target dimension, I assume 1024 is the number of classes, so you would need to permute this tensor via:</p>
<pre><code class="lang-python">logits = logits.permute(0, 2, 1)
</code></pre>
Help with computing gradients wrt output itself

username: kangtinglee
cooked: <h1>Obtaining the gradients with respect to the layer outputs</h1>
<p>The formula for a NN output is typically y = activation(w.x) and when we do automatic differentiation we usually differentiate with respects to the weight vector, as in back-propagation. However, how do I differentiate with respect to y itself?</p>
<p>To provide some background, I am passing y, the output of my actor network into my critic network and I would like to minimize the output of my critic network. So I would differentiate the predicted q-value with respect to the input of the critic / output of the actor. However, when I do backwards(), it computes the gradients of the weights in the output layer of the actor network. How can I get the gradient of the output of the actor network itself.</p>
<p>Currently, the very ghetto way that I have been able to achieve this is to take the weight vector of the output layer of the actor network, add the computed gradient to it, then manually compute the output of the actor network (by matrix multiplication). Pass the output to the critic and verify that the predicted q-val has indeed decreased.</p>
<p>Is there a better way to do this?</p>

username: kangtinglee
cooked: <p>Okay I figured out how to do this. And it was amazingly simple too… but I’ll leave this here anyway for anyone else in future who might stumble on the same problem because I had a really hard time googling for a solution.</p>
<pre><code class="lang-auto">act = actor(state)
act.retain_grad()
q = critic(act)
loss = -q
loss.backward()
new_act = act + act.grad * 0.01
q = critic(new_act)
</code></pre>
The output of BN layer

username: as754770178
cooked: <p>I want to kown the bn layer how to calculate the output. I debug the code in <code>/usr/local/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py</code>.</p>
<p>I fix the input, that I only change <code>self.training</code> in</p>
<pre><code class="lang-auto">return F.batch_norm(
            input, self.running_mean, self.running_var, self.weight, self.bias,
            self.training or not self.track_running_stats,
            exponential_average_factor, self.eps)
```, the other parameter is same. But the output is different.

I think if `self.training` is True, the bn layer will change `running_mean` and `running_var`, and use in the next batch. Whether bn layer use changed `running_mean` and `running_var` in this batch?</code></pre>

username: ptrblck
cooked: <p>I’ve created <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">this manual approach</a> some time ago for debugging purposes, which might be useful.<br>
Let me know, if you have any questions.</p>
How to save the best model?

username: Talita
cooked: <p>Suppose that I train my model for <em>n</em> epochs, and that I want to save the model with the highest accuracy on the development set. Could I use this code to save the model:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs): 

(...)

 if    accuracy &gt; best_accuracy: 
       torch.save(model, 'best-model.pt')
       torch.save(model.state_dict(), 'best-model-parameters.pt')
</code></pre>
<p>For instance if I want to test this model later on a test set :).</p>

username: Quinn
cooked: <p>Hi, Talita:<br>
You can save your model by either of the following methods.</p>
<pre data-code-wrap="python"><code class="lang-python"># Method 1
torch.save(model, 'best-model.pt') 
# Method 2
torch.save(model.state_dict(), 'best-model-parameters.pt') # official recommended
</code></pre>
<p>The difference between two methods is that the first one saves the whole model which includes project-specific classes and your best parameters, while the second one just saves your best parameters.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>By the way, you can load the model by these codes.</p>
<pre data-code-wrap="python"><code class="lang-python">the_model = TheModelClass(*args, **kwargs)
the_model.load_state_dict(torch.load(PATH))
</code></pre>
<p>References:</p>
<blockquote>
<ol>
<li><a href="https://pytorch.org/docs/stable/notes/serialization.html#recommend-saving-models" rel="noopener nofollow ugc">recommend-saving-models</a></li>
</ol>
</blockquote>
Efficient implementation of multi-row slicing

username: SofianChay
cooked: <p>Dear all,</p>
<p>Here is an example of what I’m trying to implement :<br>
I have at my disposal a 1-d tensor of start indexes S, a 1-d tensor of end indexes E, 2 2-d tensors M and R.<br>
I would like to modify M as follows :</p>
<pre><code class="lang-auto">for i in range(M.size(0)):
         M[i, S[i]: E[i] + 1] = R[i, S[i]: E[i] + 1]
</code></pre>
<p>Is there a way to do this without a loop ? (N.B : The lengths E[i] - S[i] are different in general)</p>
<p>Thank you very much</p>
PyTorch VGG16 cant learn multiclass-classification

username: Shahaf_Fridman
cooked: <p>Hi everyone!</p>
<p>I’m pretty new to PyTorch.<br>
I’ve been trying to train a VGG16 to perform multiclass classification of face images.</p>
<p>I’ve used ‘torchvision.models’ to import vgg16 pretrained model as follows:</p>
<p>model = models.vgg16(pretrained=True)</p>
<p>I’ve changed the last linear layer in the classifier to output 27 classes but it seems that it can’t learn good enough; It stops at 0.0333 accuracy and can’t go better than that.</p>
<p>I’ve used Cross Entropy Loss and SGD optimizer of PyTorch.</p>
<p>Must mention that I did try to train other models from ‘torchvision.models’ (VGG11, GoogleNet and others) and they went amazing.</p>
<p>Can someone please help me to understand why is this happening?</p>

username: ptrblck
cooked: <p>Make sure you are really replacing the last linear layer with your custom one, as different models might use different names for this layer (e.g. <code>model.classifier</code> vs. <code>model.fc</code>).</p>
<p>If that’s not the case, try to overfit a small data sample by playing around with some hyperparameters and scale up the problem once this is working.</p>
Conv3D size doesn't make sense with NIFTI data?

username: olisp
cooked: <p>Hello,</p>
<p>So I am writing custom dataset for medical images, with .nii (NIFTI1 format), but there is a confusion.</p>
<p>My dataloader returns the shape <code> torch.Size (1,1,256,256,51)</code> . But NIFTI volumes use anatomical axes, different coordinate system, so it doesn’t make any sense to permute the axes, which I normally would with volume made of 2D images seprately stored in local drive with 51 slice images (or depth), as Conv3D follows the convention (N,C,D,H,W).</p>
<p>so <code> torch.Size (1,1,256,256,51)</code>  (ordinarily 51 would be the depth) doesn’t follow the convention (N,C,D,H,W) , but I should not permute the axes as the data uses entirely different coordinate system ?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="olisp" data-post="1" data-topic="84555">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/e79b87/48.png" class="avatar"> olisp:</div>
<blockquote>
<p>but I should not permute the axes as the data uses entirely different coordinate system ?</p>
</blockquote>
</aside>
<p>Could you explain this statement a bit?<br>
Does the depth dimension represent the number of slices or what do you mean by “different coordinate system”?<br>
What wouldn’t it be possible to permute the data?</p>
How to save the optimizer setting in a log in pytorch?

username: Shisho_Sama
cooked: <p>How can I save the optemizer setting in a log ? I tried doing<br>
<code>  print_log("=&gt; optimizer '{}'".format(optimizer), log)</code><br>
but I only got :</p>
<blockquote>
<p>=&gt; optimizer ‘&lt;torch.optim.adadelta.Adadelta object at 0x7f9ed3cd4978&gt;’</p>
</blockquote>
<p>I need to save the settings using which the model was trained, things such as the learning rate, weight decay, and if I use specific optimizers such as Adadelta, its different parameters.<br>
Is there a way to get around this ?<br>
Thanks alot</p>

username: thadar
cooked: <p>Hi,</p>
<p>you can save the optimizer setting using: <code>optimizer.state_dict()</code></p>
<p>here is an example for saving:</p>
<pre><code class="lang-auto">        states = {
            'epoch': epoch + 1,
            'arch': opt.arch,
            'state_dict': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }
        torch.save(states, save_file_path)
</code></pre>
<p>here is an example for loading the settings:</p>
<pre><code class="lang-auto">optimizer.load_state_dict(checkpoint['optimizer']
</code></pre>
<p>Best Regards,<br>
Tal</p>

username: Shisho_Sama
cooked: <p>Thanks, but I want to save them in a log.txt file .</p>

username: thadar
cooked: <p>You can still save the state_dict to a log.txt</p>
<p>the output of <code>optimizer.state_dict()</code> is:</p>
<pre><code class="lang-auto">{'state': {}, 'param_groups': [{'lr': 0.0001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'params': [2666935098032, 2666957524496]}]}
</code></pre>
<p>it’s a simple json so you can save it in any format you want and load it</p>

username: Shisho_Sama
cooked: <p>Thanks alot, thats it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Greatly appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: thadar
cooked: <p>you’re welcome, glad i could help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Eric_K
cooked: <p>Hi thadar,</p>
<p>Sorry to bother you. I have a doubt that What “state” and “params” in “param_groups” stand for in optimizer’s state_dict?</p>
<p>Thanks~</p>
Match any "pathological" elements in Tensor

username: murph213
cooked: <p>Hi, I want to check whether my tensor has any pathological elements at all, like <code>inf</code>, <code>NaN</code>, etc.  I don’t know what the exhaustive list is for all such pathologies.  Please help!</p>
<p>In some packages there are super-classes that contain all pathologies, and you just check for instances of that.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>To check for valid values only, you could use <code>torch.isfinite(tensor)</code>.<br>
You could also check explicitly for NaNs or Infs via <code>torch.isnan(tensor)</code> and <code>torch.isinf(tensor)</code>.</p>