akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Onnx error then convert pth to onnx

username: leigaoxiang1
cooked: <p>TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.<br>
pred_boxes[…, 0] = x.data + self.grid_x</p>
<p>how to mdify the code?</p>
Load data from variables into classes

username: kristerv
cooked: <p>My task is to take an episode of a TV show and its subtitles. Then make the subtitle timings more accurate (from 200ms to 20ms). So I want to learn what is speech and what is not.</p>
<p>I’ve now taken the audio, converted it into a spectrogram and separated each column of the spectogram to be a single data item. So now I have two arrays:</p>
<pre><code class="lang-python">print(train_speech.size()) # torch.Size([93482, 201])
print(train_silence.size()) # torch.Size([35038, 201])
</code></pre>
<p>All I want to do is a simple multi-linear NN to make a difference. <code>train_speech</code> is FFT’s of people talking and <code>train_silence</code> is no talking (used subtitles for the distinction).</p>
<p>My question is what DataLoader can I use to take these into torch?</p>

username: ptrblck
cooked: <p>There is one <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader"><code>DataLoader</code></a>, which accepts a <code>Dataset</code> and provides different functionalities such as shuffling, creating batches using multiple workers, etc.</p>
<p>To create a custom <code>Dataset</code> you could have a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kristerv
cooked: <p>What I don’t get is that my data is already a simple tensor… It doesn’t make sense to me that I need to create a separate abstraction just to fetch numbers from a few arrays…</p>

username: ptrblck
cooked: <p>If your data is already stored as tensors, you can just use <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.TensorDataset"><code>TensorDataset</code></a> or completely skip the abstraction and just feed to data to your model.</p>

username: kristerv
cooked: <p>Since I had two classes in separate variables I ended up making the custom class.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, speech, silence):
        self.data = list(map(lambda x: (x, 1), speech)) + list(map(lambda x: (x, 0), silence))
        
    def __getitem__(self, index):
        return self.data[index]
    
    def __len__(self):
        return len(self.data)

train_ds = MyDataset(train_speech, train_silence)
train_dl = DataLoader(train_ds, shuffle=True, batch_size=1024)
</code></pre>
<p>Thanks for helping me get through this.</p>
ConvLSTM2D in PyTorch

username: Sourabh_Daptardar
cooked: <p>Are there any plans of adding ConvLSTM2D in PyTorch ?</p>

username: karmus89
cooked: <p>Hi!</p>
<p>You might want to check the simple implementation provided in <a href="https://discuss.pytorch.org/t/passing-hidden-layers-to-convlstm/52814/3" class="inline-onebox">Passing hidden layers to ConvLSTM</a>.</p>
<p>Cheers,<br>
Petteri</p>
Understanding convLSTM

username: Omar2017
cooked: <p>Hi there,<br>
I am having issue understand the following implementation of ConvLSTM. I don’t really understand what input_size + hidden_size is? Also the 4 * hidden_size value for output?  <code>conv_lstm = ConvLSTMCell(256, self.mem_size)</code>  tells us that 256 and self.mem_size are input_size and hidden_size. Also, why are we multiplying hidden_size by 4 for output? Can some one more experience explain to me what is going on within the convolution? Thank you.</p>
<pre><code class="lang-auto">self.mem_size=256
self.conv_lstm = ConvLSTMCell(256, self.mem_size)
</code></pre>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.autograd import Variable


class ConvLSTMCell(nn.Module):
    def __init__(self, input_size, hidden_size, kernel_size=3, stride=1, padding=1):
        super(ConvLSTMCell, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.Gates = nn.Conv2d(input_size + hidden_size, 4 * hidden_size, kernel_size=kernel_size,
                               stride=stride, padding=padding)
        for params in self.Gates.parameters():
            params.requires_grad=True
        torch.nn.init.xavier_normal_(self.Gates.weight)
        torch.nn.init.constant_(self.Gates.bias, 0)

    def forward(self, input_, prev_state):
        batch_size = input_.data.size()[0]
        spatial_size = input_.data.size()[2:]
        if prev_state is None:
            state_size = [batch_size, self.hidden_size] + list(spatial_size)
            if(torch.cuda.is_available()):
                prev_state = (Variable(torch.zeros(state_size).cuda(),requires_grad=True),Variable(torch.zeros(state_size).cuda(),requires_grad=True))
            else:
                prev_state = (Variable(torch.zeros(state_size),requires_grad=True),Variable(torch.zeros(state_size),requires_grad=True))
        prev_hidden, prev_cell = prev_state
        stacked_inputs = torch.cat((input_, prev_hidden), 1)
        gates = self.Gates(stacked_inputs)
        in_gate, remember_gate, out_gate, cell_gate = gates.chunk(4, 1)
        in_gate = torch.sigmoid(in_gate)
        remember_gate = torch.sigmoid(remember_gate)
        out_gate = torch.sigmoid(out_gate)
        cell_gate = torch.tanh(cell_gate)
        cell = (remember_gate * prev_cell) + (in_gate * cell_gate)
        hidden = out_gate * torch.tanh(cell)
        return hidden, cell

</code></pre>

username: panpan0210
cooked: <p>hello, i am the new about pytorch ,<br>
now i am learing about Convlstm.<br>
so i want to ask you about it<br>
could it work well?<br>
what is the input and its size ?</p>

username: karmus89
cooked: <p>For you and others interested, maybe <a href="https://discuss.pytorch.org/t/passing-hidden-layers-to-convlstm/52814/3" class="inline-onebox">Passing hidden layers to ConvLSTM</a> might provide some help to this issue!</p>
<p>Cheers,<br>
Petteri</p>
ConvLSTM3D in pytorch for 3D data

username: Abhishek_Kumar_Dubey
cooked: <p>Hi is there anything in pytorch like  <a href="https://github.com/keras-team/keras/blob/b78e3ff9263a6c3b28c869a73f74a1fbd6fd491c/keras/layers/convolutional_recurrent.py#L786" rel="nofollow noopener">class ConvLSTM2D(ConvRNN2D):</a> as in keras.<br>
I want a convLSTM for 3D data which is still not supported in keras, I could find ConvLSTM2D which is meant for 2D data,<br>
Please suggest me if there is ConvLSTM3D already supported in pytorch , if not how to implement it.</p>

username: tom
cooked: <p>The easiest way is to spell out the LSTM cell yourself.<br>
PyTorch lets you make this fast, look at the <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">custom LSTMs</a> we use in benchmarking for inspiration.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Abhishek_Kumar_Dubey
cooked: <p>Thanks tom for your kind reply, I had a look at <a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">custom LSTM</a> cell , but I need some  lstm cell which works as convLstm, If I take custom LSTM as a basic block it will require a lot of work around to convert it to convLstm and convLstm3D seems to be a long way to go.<br>
Any ways thanks again I will start the implementation of the same</p>

username: karmus89
cooked: <p>While your question is from last year, I still wanted to chime in to maybe provide some help.</p>
<p>If you check the answer at <a href="https://discuss.pytorch.org/t/passing-hidden-layers-to-convlstm/52814/3" class="inline-onebox">Passing hidden layers to ConvLSTM</a>, I think you should be able to modify the code to utilize <code>Conv3d</code> layers instead of their 2D counterparts.</p>
<p>Cheers,<br>
Petteri</p>
Hi PyTorch ! I'm trying to get a snippet for cropping with offsets. Can you please tell what does -1 specifies here?

username: Blizzard_boi2020
cooked: <p>offsets = [0, (x1_shape[1] - x2_shape[1]) // 2, (x1_shape[2] - x2_shape[2]) // 2, 0]</p>
<p>size = [-1, x2_shape[1], x2_shape[2], -1]</p>

username: ptrblck
cooked: <p>If you pass the <code>-1</code> to an operation, which expects a dimension, it would select the “last dimension”.<br>
In your current code snippet it’s unclear, how <code>size</code> is used, so I can only speculate. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Blizzard_boi2020
cooked: <p>That’s clear to me! But I have the TF implementation, I want to reproduce it for PyTorch.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228" title="Screenshot_20200803-185215577_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228_2_690x222.jpeg" alt="Screenshot_20200803-185215577_1" data-base62-sha1="cTH3oxdSM5TQf7aJVhA00ZpAwbK" width="690" height="222" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228_2_690x222.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228_2_1035x333.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a65ca0b4937e49c4bb54f5bb2fe13f7a8238228.jpeg 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20200803-185215577_1</span><span class="informations">1080×349 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <a href="https://www.tensorflow.org/api_docs/python/tf/slice">TF docs</a> might be helpful:</p>
<blockquote>
<p>If  <code>size[i]</code>  is -1, all remaining elements in dimension i are included in the slice. In other words, this is equivalent to setting:<br>
<code>size[i] = input_.dim_size(i) - begin[i]</code></p>
</blockquote>
<p>If you provide the input and desired output (+ shapes), we could try to come up with the corresponding method in PyTorch.</p>
Picture in rgb mode convert to L mode, Problem with soft picture edges

username: maren11
cooked: <p>Hi there,<br>
I’am working on an Object Detection with my own Data. Now I have the Problem than my pictures of the masks are in RGB mode but I need to convert them to L Mode. I have done thins with .convert(‘L’). But now I have a Problem with my edges. If you look at this picture<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66c9d77a6dc23ee107a221d753f220ac5af3ca00.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/66c9d77a6dc23ee107a221d753f220ac5af3ca00" title="Maske_konvertiert"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66c9d77a6dc23ee107a221d753f220ac5af3ca00_2_690x384.jpeg" alt="Maske_konvertiert" data-base62-sha1="eFj92GiwzyWho4tKsPYsRvF8n3W" width="690" height="384" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66c9d77a6dc23ee107a221d753f220ac5af3ca00_2_690x384.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/66c9d77a6dc23ee107a221d753f220ac5af3ca00_2_1035x576.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/6/66c9d77a6dc23ee107a221d753f220ac5af3ca00.jpeg 2x" data-dominant-color="0E0B01"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Maske_konvertiert</span><span class="informations">1313×731 22.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
the edges are not yellow. Do you have an idea how to convert it correctly? Before converting the picture was just black and yellow.<br>
Thanks for help.</p>

username: ptrblck
cooked: <p>This issue doesn’t seem to be directly related to PyTorch, so you might get faster and better answers on e.g. StackOverflow.</p>
<p>That being said, I guess that that original image might also contain these artifacts.<br>
If the input image was just a binary image encoded as an RGB image, you could apply some smoothing and binarization to try to get rid of these noisy edges.</p>

username: maren11
cooked: <p>Thanks for answer. I think I found the problem:<br>
I set the colors of the picture in python, there all Pixels have the correct color. But if I save  the picture as a jpg to download, the colors get wrong. Do you know how to fix this issue?</p>

username: ptrblck
cooked: <p>JPEG uses an internal encoding, which might create these artifacts. I don’t know which software or platform you are using, but you might be able to increase the quality level.<br>
If that’s not possible, try to download these pictures in another format.</p>
How to measure the contribution of each sample

username: 1112
cooked: <p>Hi, I want to did some negative sampling.<br>
It means that in a mini-batch of negative data samples, I wanto only samples part of them and used for training.<br>
How to measure the contribution of each sample to the training in a batch ?<br>
should I use</p>
<ol>
<li>the loss of each sample;</li>
<li>the gradient of each sample</li>
</ol>
<p>and how to get the gradient of each sample ?</p>
Adam optimizer to train model via libtorch C++ on Android

username: atibrewal
cooked: <p>Hi,</p>
<p>I am  building libtorch using ./scripts/build_anroid.sh.</p>
<p>We need the support for Aten Ops and TorchScript, so building without the BUILD_CAFFE2_MOBILE option.</p>
<p>The build is successful, but the library does not link  ${TORCH_SRC_DIR}/csrc/api/src/optim/adam.cpp and dependencies because of NO_API being set for Mobile Builds.</p>
<p>Because of this I am unable to train the model and instantiate Adam Optimizer instance from the code.</p>
<p>torch::optim::Adam optimizer_eval(parameters, /<em>lr=</em>/lr); // Gives Linker Errors<br>
optimizer_eval.step(); //Linker Error</p>
<p>Using recent master commit of pytorch : bfa94487b968ccb570ef8cd9547029b967e76ed0</p>
<p>Can someone please help with this ?</p>
Creating a custom Dataloader to load .mat files

username: Ahebwaed
cooked: <p>Hello, Am a beginner in deep-learning, Am trying to do image holographic image reconstruction and i need help on creating a DataLoader to take into a CNN .mat images. my images are divided into 3 folders ie training, testing and Validation. So i need a hand in creating an algorithm to take in these 3 categories of files.</p>
<p>Thank you</p>

username: Hmrishav_Bandyopadhy
cooked: <p>Hi,<br>
This might be the dataset module:</p>
<pre><code class="lang-auto">
import os
import hdf5storage as h5
import torch
from torch.utils.data import DataLoader, Dataset
import torchvision.transforms

class DataSet(Dataset):
	
	def __init__(self, images_folder,transform):
		
		assert os.path.exists(images_folder), 'Images folder does not exist'
		
		self.images_folder=os.path.abspath(images_folder)
		self.images = os.listdir(self.images_folder)
		
		self.transform = transform
		

	def __len__(self):
		return len(self.images)

	def __getitem__(self, index):
		
		img_filename = self.images[index]
		img = h5.loadmat(img_filename)['Insert_your_array_name']
		if self.transform:
			img = torch.from_numpy(img)	
		return img


if __name__=='__main__':

	'''
	Folders:
	image				[Training Images]
	image_test			[Testing Images]	
	image_validation	[Validation Images]
	'''

	dataset_train = DataSet(os.path.join(data_path, 'image'), transform)
	dataset_test = DataSet(os.path.join(data_path, 'image_test'), transform)
	dataset_val = DataSet(os.path.join(data_path, 'image_validation'), transform)

	train_loader = DataLoader(dataset_train, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(dataset_test, batch_size=batch_size, shuffle=True)
    val_loader= DataLoader(dataset_val, batch_size=batch_size, shuffle=True)


</code></pre>
<p>Does that solve your problem?</p>

username: Ahebwaed
cooked: <p>Thank you Hmrishav,</p>
<p>Your dataLoader works well for me</p>
Deploying a pytorch model in meteor + react js

username: DeepsMoseli
cooked: <p>I have gone through a couple of pages looking at my options.I intend to build a Pytorch model, I already have a web app all written in meteor+react. I am not sure if there is a way to combine flask and meteor. Else, is how easy is it to deploy my model for both torch and TF, then use Tensorflow.js  with my react +meteor app.</p>
<p>Would appreciate any advice. i have most of my front end build already so that stack can’t change now.</p>

username: JamesTrick
cooked: <p>Hi Moseli!</p>
<p>Hope you’re going good.</p>
<p>Given you already have a reactjs frontend, you probably want to maintain the separation of Backend and frontend. I think in your case, it would make sense in your case to build a Flask API, and have your reactjs app simply call your external API?</p>
<p>I wrote an <a href="https://discuss.pytorch.org/t/i-want-to-serve-my-chatbot/54586">answer back here, in Sep 2019</a> pointing to links. Since then, PyTorch has implemented TorchServing which could be of interest - <a href="https://pytorch.org/blog/model-serving-in-pyorch/" rel="nofollow noopener">blog here</a>, <a href="https://pytorch.org/serve/" rel="nofollow noopener">docs here.</a></p>
Torch.save() fails to update timestamp?

username: FaultyBagnose
cooked: <p>I use torch.save(model, file_name) to save my model after each training epoch, but the timestamp of the saved file is set the first time, and not updated when it saves again. Not a huge problem, of course, but is this a small bug? Could be an OS issue (Win 10), I guess.</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue in Windows10 using this small code snippet:</p>
<pre><code class="lang-python">torch.save(torch.randn(1), 'tmp.pt')
</code></pre>
<p>and see the time stamp gets updated in the file explorer.</p>

username: FaultyBagnose
cooked: <p>Thanks. The relevant test would be this:</p>
<p>torch.save(torch.randn(1), ‘tmp.pt’)<br>
time.sleep(60)<br>
torch.save(torch.randn(1), ‘tmp.pt’)</p>
<p>My file explorer keeps showing the first creation time, even though the file attributes are updated, so I guess it’s not a Python problem.</p>

username: ptrblck
cooked: <p>Alternatively, I could have waited a minute. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Anyway, to be super strict, here is another code snippet, which shows the modification time stamp:</p>
<pre><code class="lang-python">name = 'tmp1.pt'
torch.save(torch.randn(1), name)
print("Last modified: %s" % time.ctime(os.path.getmtime(name)))
print("Created: %s" % time.ctime(os.path.getctime(name)))
&gt; Last modified: Tue Aug  4 16:39:44 2020
Created: Tue Aug  4 16:39:44 2020

time.sleep(60)
torch.save(torch.randn(1), name)
print("Last modified: %s" % time.ctime(os.path.getmtime(name)))
print("Created: %s" % time.ctime(os.path.getctime(name)))
&gt; Last modified: Tue Aug  4 16:40:44 2020
Created: Tue Aug  4 16:39:44 2020
</code></pre>
Calculate macro averages for accuracy and loss in validation

username: SU801T
cooked: <p>Hi,</p>
<p>I have an imbalanced training set and currently using the focal loss to train it. I also have an even more imbalaced validation set. I would like to get a better representation of the true loss according to each class. At the moment, I simply calculate the validation accuracy and loss per epocch and graph this out on tensorboard. I was wondering if there wasa way of calculating “macro” averages. This would allow me to better judge which model to save…</p>
<pre><code class="lang-auto">model.eval()
        val_loss = 0
        val_accuracy = 0
      

        with torch.no_grad(): # Tell torch not to calculate gradients
            for i,(val_inputs, paths, val_labels) in enumerate(dataloaders_dict['val']):
                val_inputs = val_inputs.to(device)
                val_labels = val_labels.to(device)
                val_outputs = model(val_inputs)
                val_loss += criterion(val_outputs, val_labels).item()

                _val, val_pred = torch.max(val_outputs, 1)
                val_equality_check = (val_labels.data == val_pred)
                val_accuracy += val_equality_check.type(torch.FloatTensor).mean()

                val_predictions += val_pred.items
                val_groundtruth += val_labels.data


        val_losses = val_loss / len(dataloaders_dict['val'])
        validation_accuracy = val_accuracy/ len(dataloaders_dict['val'])

</code></pre>
How to build pytorch with NCCL 2.7.6

username: Purvak-L
cooked: <p>Hi,<br>
I’m trying build pytorch from source and need to make use of NCCL 2.7.6 or higher. The current pytorch is built using 2.7.3.  Is there a way to do this?</p>
<p>Thank you</p>

username: agolynski
cooked: <p>You can potentially clone your own pytorch repo and upgrade nccl in there only, after that you can recompile from source. Would that work?</p>

username: Purvak-L
cooked: <p>I did clone the pytorch repo and build from it. Can you point me where to make that update?<br>
I did check cmake/modules/FindNCCL.make.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/purvak-l">@Purvak-L</a>, you can <code>cd</code> to the NCCL submodule in the third party folder in <a href="https://github.com/pytorch/pytorch/tree/master/third_party/nccl" rel="nofollow noopener">https://github.com/pytorch/pytorch/tree/master/third_party/nccl</a> and manually update the nccl module there.</p>
<p>Another option is to install 2.7.6 locally and set <code>USE_SYSTEM_NCCL=1</code> when building PyTorch. See this issue: <a href="https://github.com/pytorch/pytorch/issues/32286" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32286</a></p>

username: Purvak-L
cooked: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="91634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p>USE_SYSTEM_NCCL=1</p>
</blockquote>
</aside>
<p>Thank you <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>This helped! I downloaded the latest NCCL (2.7.6.1) and set the flag. After that I built pytorch from source using python setup.py install</p>
Detaching and reattaching model depending on phase

username: little_adventurer
cooked: <p>I was wondering, if I explicitly detach the model and inputs etc during the validation phase, do I have to reattach it in the next epoch of the training loop?</p>

username: ptrblck
cooked: <p>You can <code>detach()</code> a tensor, which is attached to the computation graph, but you cannot “detach” a model.<br>
If you don’t disable the gradient calculation (e.g. via <code>torch.no_grad()</code>), the forward pass will create the computation graph and the model output tensor will be attached to it. You can check the <code>.grad_fn</code> of the output tensor to see, if it’s attached to a graph (shows a backward function) or not (is <code>None</code>).</p>
<p>In case you haven’t used the <code>with torch.no_grad()</code> context manager during validation, you might unnecessarily use more memory, but won’t have any other side effects and don’t need to reattach anything (which is also not possible).<br>
The next training iteration will work as before.</p>

username: little_adventurer
cooked: <p>thank you for the answer!</p>
<p>My question was motivated precisely due to the doubling of memory usage during validation!</p>
<p>Despite wrapping it with</p>
<pre><code class="lang-auto">with torch.no_grad()
</code></pre>
<p>As soon as I reach validation, my memory usage doubles, then it increases again when I compute the Class Activation Maps…</p>
<p>Could it be caused by me doing:</p>
<pre><code class="lang-auto">for input,label in val_loader:
      input = input.to(self.device)
      label = label.to(device)
</code></pre>
<p>But as far as I remember, I was getting a</p>
<pre><code class="lang-auto">RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
</code></pre>
<p>error if I didn’t place the inputs and labels on the device…</p>

username: ptrblck
cooked: <p>You need to place the input and target tensors on the same device as the model parameters.<br>
There are advanced use cases for model sharding, which doesn’t seem to be used in your case.</p>
<p>If your validation loop increases the memory usage, you might want to either delete all unnecessary training tensors manually or wrap the training code in a <code>train()</code> function (and <code>val()</code> function), so that the temporary variables will be deleted automatically. Python uses function scoping so temp. variables, such as the model output, loss etc. will be deleted once you are leaving the function scope.</p>
How to predict only one test sample in pytorch model?

username: thaingi
cooked: <p>I want to give an input to the model and predict the class label.How can I write my code?Please answer.Thanks in advance!</p>

username: jmandivarapu1
cooked: <p>You can use the batch size as 1 if you want to do single sample using pytorch test data loader.</p>
<p>If not you can create tensor of same size and shape as u r input and just pass it as model(input)</p>

username: thaingi
cooked: <p>Thanks a lot for answering! I get :<br>
predict is: tensor([[0.1559, 0.4340, 0.3785, 0.0226, 0.0032, 0.0519, 0.1933]],<br>
device=‘cuda:0’, grad_fn=)<br>
When tensor is converted to numpy:  predict is: [[-0.2556236   0.03545919 -0.38335416  0.3578361  -0.42080054  0.3241588<br>
0.24587154]] .I want to get class label,and how can I predict class label?Thanks in advance!</p>

username: Anshumaan_Dash
cooked: <p>Try using <code>pred_out, pred_index = torch.max(pred, 1)</code></p>
<p>Also, try wrapping your inference code with the wrapper: <code>with torch.no_grad():</code></p>

username: thaingi
cooked: <p>Thanks a lot! pred_out is: tensor([0.6067])<br>
pred_index is: tensor([1])<br>
The outputs are not same when I run again and again.Why this happens?Please answer how to get same result when it is run again and again.Thanks in advance!</p>

username: Anshumaan_Dash
cooked: <p>Sure.</p>
<p>For getting deterministic results, you need to set the seed.</p>
<p>Refer to this post: <a href="https://discuss.pytorch.org/t/reproducibility-with-all-the-bells-and-whistles/81097" class="inline-onebox">Reproducibility with all the bells and whistles</a></p>

username: Brando_Miranda
cooked: <p>can you explain why:</p>
<pre><code class="lang-auto">torch.max(output.data, 1)
</code></pre>
<p>works?</p>
<p>I was reading the docs (<a href="https://pytorch.org/docs/stable/generated/torch.max.html" rel="nofollow noopener">https://pytorch.org/docs/stable/generated/torch.max.html</a>) and it’s still not entirely clear to me. This is the sentence that doesn’t make sense to me:</p>
<pre><code class="lang-auto">Returns a namedtuple (values, indices) where values is the maximum value of each row of the input tensor in the given dimension dim.
</code></pre>
<p>I don’t understand what the max value of a row mean given a specific slice of a tensor. Rows only exist in 2D tensors (matrices), at least to me. Can you clarify?</p>
Why pytorch lib 1.6 cxx11 abi don't have jni library

username: Atul_Kumar
cooked: <p>Why pytorch lib 1.6 cxx11 abi don’t have jni library<br>
<a href="https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcpu.zip" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcpu.zip</a><br>
Although<br>
<a href="https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.6.0%2Bcpu.zip" class="onebox" target="_blank" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.6.0%2Bcpu.zip</a><br>
has jni libraries libfbjni.so		libpytorch_jni.so</p>

username: Atul_Kumar
cooked: <p>bump up the question.</p>

username: Atul_Kumar
cooked: <p>bump up the question.</p>

username: Atul_Kumar
cooked: <p>bump up the question.</p>
RNN is not learning even simple concepts

username: bred
cooked: <p>I’ve been trying to make an RNN that predicts the next character in a string. (I’m trying to do it without the built-in models because I thought I would learn more that way.)</p>
<p>Unfortunately, it seems the model is only able to learn the rough distribution of letters; It hasn’t even realised that you shouldn’t put numbers right next to letters. It is as if it can’t see the input or the hidden layers. Given that it has ~400k trainable parameters I think it should be doing better.</p>
<p>Things I’ve checked:</p>
<ol>
<li>The input is getting through (checked with the print statement commented out below)</li>
<li>The hidden layers are being passed on</li>
</ol>
<p>Their may also be a problem with my understanding of Python as I am not very experienced with this programming language.</p>
<p>Here’s the model definition, other files are in <a href="https://drive.google.com/uc?export=download&amp;id=1ZJoKDL_WqPC6aLnybCFw66ksuCJxJ4T1" rel="nofollow noopener">this zip</a> because I didn’t want to make this post overly long. I’ve replaced the loading of training data with just a couple hard-coded strings because the data-set is quite large.</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size
        
        self.in_to_h = nn.Linear(input_size + hidden_size, hidden_size).to(device)
        self.h1_to_h2 = nn.Linear(hidden_size + hidden_size, hidden_size).to(device)
        self.h2_to_h3 = nn.Linear(hidden_size + hidden_size, hidden_size).to(device)
        self.h_to_out = nn.Linear(hidden_size, output_size).to(device)
        self.softmax = nn.LogSoftmax(dim=1).to(device)

        self.act = nn.Tanh()

    def forward(self, input, hidden):
        #Print input
        #print(all_letters[torch.argmax(input, 1).cpu().numpy()[0]]) 
        hidden[0] = self.act(self.in_to_h(torch.cat((input, hidden[0]), 1)))
        hidden[1] = self.act(self.h1_to_h2(torch.cat((hidden[0], hidden[1]), 1)))
        hidden[2] = self.act(self.h2_to_h3(torch.cat((hidden[1], hidden[2]), 1)))

        output = self.h_to_out(hidden[2])
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        hidden = []
        for i in range(3):
            hidden.append(Variable(torch.zeros(1, self.hidden_size, device = device)))
        return hidden
</code></pre>
Non Blocking copy from CPU to GPU takes long time

username: cpp_for_torch
cooked: <p>Hi guys,</p>
<p>I have been trying to do copy tensor from CPU to GPU with non blocking following tutorial(<a href="https://pytorch.org/cppdocs/notes/tensor_creation.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Creation API — PyTorch main documentation</a>) by doing below:</p>
<blockquote>
<p>#include&lt;torch/torch.h&gt;<br>
<span class="hashtag-raw">#include</span><br>
<span class="hashtag-raw">#include</span><br>
using namespace std::chrono;</p>
<p>int main()<br>
{<br>
auto t0 = high_resolution_clock::now();<br>
torch::Tensor cpu_tensor = torch::randn({128, 3, 24, 24}, torch::kFloat);<br>
torch::Tensor gpu_tensor = cpu_tensor.to(torch::kCUDA,/<em>non_blocking</em>/true);<br>
auto t1 = high_resolution_clock::now();<br>
std::cout&lt;&lt;“Time Taken:”&lt;&lt;(static_cast((duration_cast(t1 - t0)).count()))/1e6&lt;&lt;std::endl;<br>
return 0;<br>
}</p>
</blockquote>
<p>Result: “Time Taken: 0.08”<br>
Which is very slow in comparison to what it would take in python which is 0.0001 seconds. I was wondering if anyone can tell me if I am doing anything/ using the API wrong. Thanks in advance for your answers guys <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: glaringlee
cooked: <p><a class="mention" href="/u/cpp_for_torch">@cpp_for_torch</a><br>
Don’t measure this for the first time you call cuda allocator. It might be slow the first cuda allocator is called. If you want to measure this. Run the to() function several times and then measure its running time, this would be more accurate.</p>

username: colesbury
cooked: <p>In addition to what <a class="mention" href="/u/glaringlee">@glaringlee</a> said, the CPU tensors must also be in pinned memory for the copy to be non-blocking.</p>
<pre><code class="lang-auto">torch::Tensor cpu_tensor = torch::randn({128, 3, 24, 24}, torch::kFloat).pin_memory();
</code></pre>

username: cpp_for_torch
cooked: <p>Thanks Glaringlee, would try that</p>
Method grad returns None for a tensor

username: aktsvigun
cooked: <p>Hi, I am interested in gradients for some parameters after applying a loss function. Hence, I do as follows:</p>
<pre><code class="lang-auto">x = torch.autograd.Variable(torch.Tensor(nna.scalar_dot), requires_grad=True)
w = torch.autograd.Variable(torch.Tensor(nna.W_out), requires_grad=True)
out = torch.matmul(x, w)

loss_fn = nn.BCEWithLogitsLoss()
loss = loss_fn(out, y_tensor)
loss.backward(retain_graph=True)

out.grad --&gt; returns None
</code></pre>
<p>However, if I detach the tensor <code>out</code> and have it as <code>torch.autograd.Variable(torch.Tensor(torch.matmul(x, w)), requires_grad=True) </code>, I get the correct result for it, however, <code>x.grad</code> now turns None (which is understandable as we “unpin” it).</p>
<p>Could someone explain me, why <code>out.grad</code> is <code>None</code> and how can I get it? Thanks in advance!</p>

username: colesbury
cooked: <p>Use <code>out.retain_grad()</code>.</p>
<p>PyTorch by default only saves the gradients for the initial variables <code>x</code> and <code>w</code> (the “leaf” variables) that have <code>requires_grad=True</code> set – not for intermediate outputs like <code>out</code>.</p>
<p>To save the gradient for <code>out</code>, use the <code>retain_grad</code> method</p>
<pre><code class="lang-python">out = torch.matmul(x, w)
out.retain_grad()
</code></pre>

username: aktsvigun
cooked: <p>Sam, thanks a lot! Works perfectly.</p>
How can I convert a Deep Learning project from Tensorflow to PyTorch

username: Samiha
cooked: <p>Please suggest some ideas about how can I convert a pre-existing TensorFlow project to PyTorch.</p>

username: KFrank
cooked: <p>Hi Samiha!</p>
<aside class="quote no-group" data-username="Samiha" data-post="1" data-topic="91607" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ba9def/48.png" class="avatar"> Samiha:</div>
<blockquote>
<p>how can I convert a pre-existing TensorFlow project to PyTorch.</p>
</blockquote>
</aside>
<p>You should first get an overview of the pytorch architecture.  I would<br>
suggest browsing the pytorch documentation, in particular, the <a href="https://pytorch.org/tutorials/" rel="noopener nofollow ugc">tutorials</a>.<br>
The <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" rel="noopener nofollow ugc">60-Minute Blitz</a> might be a good place to start.</p>
<p>You will want to have an overview of the “pytorch way” to construct a<br>
neural network, specify a loss function, specify an optimizer, and run<br>
your training loop.</p>
<p>Then start porting your tensorflow code to pytorch.  If your tensorflow<br>
project uses 'standard" neural-network building blocks and algorithms,<br>
your should be able to find equivalents for (more or less) everything<br>
in pytroch.</p>
<p>If your tensorflow project uses some custom code (e.g., a non-standard<br>
loss function) or happens to use some building block that hasn’t (yet)<br>
been implemented in pytorch, you will have to write custom pytorch<br>
code for that piece.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Samiha
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a>  I need a specific part of the Tensorflow project and integrate that to another Pytorch project. For this purpose, is it necessary to convert the whole Tensorflow project?</p>
Validation loss reduces but accuracy increases

username: dl_enthusiast_real
cooked: <p>I am training a Siamese Network using Resnet50 pre-trained network. I have modified the fully connected layer at the end with a 3 layer classifier and frozen all of the resnet layers before the adaptive average pooling.<br>
The loss function I use is below:</p>
<pre><code>class TripletLoss(nn.Module):

def __init__(self, embedding_size):
    super(TripletLoss, self).__init__()
    self.margin = embedding_size

def forward(self, anchor, positive, negative):
    dp = (anchor - positive).pow(2).sum(1)  
    dn = (anchor - negative).pow(2).sum(1)  
    losses = F.relu(dp - dn + self.margin)
    acc =  (losses == 0.0).type(torch.float32).sum()

    return losses.sum(), acc
</code></pre>
<p>The network is trained on 1400 triplets, and 150 triplets are used for validation.<br>
Below is the plot for the loss and accuracy. I am very confused about how to interpret this as the validation loss increases after epoch 5 but the validation accuracy either remains the same or increases slightly. Would this be a good model as the accuracy is high? Or is something wrong?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32.png" data-download-href="https://discuss.pytorch.org/uploads/default/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32_2_690x245.png" alt="image" data-base62-sha1="acsTYX5P4nYYcn96YdRipn3TCfw" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/7/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32_2_690x245.png, https://discuss.pytorch.org/uploads/default/original/3X/4/7/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/7/477e7d5154a2d5c69c3d44bfab63b601cbdf6b32.png 2x" data-dominant-color="FAF9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">862×307 28.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any comments or suggestions to improve training would be appreciated.</p>
Pytorch object detection false positives

username: Mesut
cooked: <p>Hi, i have been developing an object detection model using pytorch, I have 5.000 images of 1 class, there is actually only 1 class that I want to detect via camera frames.</p>
<p>When I train it 200-300 epoch and the loss stops decreasing, I let it inference with camera, it is very good to detect wanted objects, however It makes false positive predictions with 0.99 confidence on very un-related objects, the objects generally one of the RGB, so they are generally full red or green or blue objects.</p>
<p>I have take photos from this object and augment them with my wanted objects, label them and add to the dataset, however it did not work, same situation continues.</p>
<p>1)The augmented image resolution is 1280p but my camera feed gives 480p, can this be the reason?<br>
2)Some of the slightly blurred images from my dataset can cause this?</p>
<p>Wish to have good days.</p>

username: ptrblck
cooked: <p>How did the validation loss look during training? Did you see some overfitting observing the training and validation loss curves?<br>
Also, does your training and validation set contain similar images with these specific colors?</p>

username: Mesut
cooked: <p>Validation loss is fluctuating, first 20 epochs it is decreasing then fluctuating but decreasing is continuous.</p>
<p>Val loss always a bit lower than training loss, i think it is because of augmentation.</p>
<p>Yes, it is the most interesting part, i have added the photos of the problematic objects to the training dataset, when i show them as a test input, it does not say it positive, but if i show them with my camera it gives false positives also it say positive to the full black area when i close the camera lens , i am really confusing.</p>

username: ptrblck
cooked: <p>If I understand the follow-up correctly, you are seeing a different behavior depending if you are feeding the object images through your “training pipeline” or through your camera?<br>
If that’s the case, then this might be expected, as I’ve worked with models in the past, which were even sensitive to different JPEG decoding setups.</p>
<p>Could you create a (small) dataset captured by your camera, label it, and add to the training and validation set? The difference in the sample domains (training images vs. camera capture) might introduce enough signal, that your model is unable to properly predict the camera images.</p>

username: Mesut
cooked: <p>Thank you sir , i will try to capture images with my camera and create dataset with them.</p>
How to get the hash of the model in libtorch?

username: Shisho_Sama
cooked: <p>Hello everyone,<br>
I was looking for the equivalent of <code>id()</code> in python that would get the objects hash. I noticed libtorch offers a <code>get_hash()</code> function and the documentation says it works with all types!</p>
<blockquote>
<p>Use this function to actually hash multiple things in one line.</p>
</blockquote>
<p>but when I try to use this with a scripted module, it fails :</p>
<pre><code class="lang-auto">Severity	Code	Description	Project	File	Line	Suppression State
Error	C2672	'dispatch_hash': no matching overloaded function found	Model_embedder_test	D:\LibtorchPort\Dependencies\libtorch-release-latest\libtorch\include\torch\csrc\utils\hash.h	78	
Error	C2893	Failed to specialize function template 'unknown-type torch::_hash_detail::dispatch_hash(const T &amp;)'	Model_embedder_test	D:\LibtorchPort\Dependencies\libtorch-release-latest\libtorch\include\torch\csrc\utils\hash.h	78	
</code></pre>
<p>whats wrong here? apart from this whats my other options? I need to use something unique about a module to differentiate it from others.</p>
How to trace a non-forward function

username: shy
cooked: <p>Hi, I have an model class which inherents nn.Module, but the main use case is a member function(not forward function, i hope to trace/script this model, but i have no idea how to do it. For example, the class is kind of like:</p>
<pre><code class="lang-auto">class A(nn.Module):
 def __init__(self):
    self. linear_layer = xxxx.
 def solve():
     pass
a = A()
a.solve()    
</code></pre>
<p>I want to jit the a.solve function using the A’s learnable parameters.</p>

username: ptrblck
cooked: <p>I think you would have to call <code>self.solve()</code> in the <code>forward</code> method to properly script the model.<br>
In eager mode you could still use <code>a.solve()</code>, but if I’m not mistaken, the JIT expects the “standard” implementation using <code>forward</code>.</p>
Should I use optimizer.step() or model.step() function to train my model

username: coyote
cooked: <p>In pytorch, to update the model, should I use optimizer.step or model.step ? My question also valid for zero_grad() method?</p>
<p>Here is a example snippet:</p>
<pre><code class="lang-auto">import torch
import torch nn
class SomeNeuralNet(nn.Module):
    def __init__(self,hs,es,dropout):
          SomeNeuralNet(ClaimRecognizer, self).__init__()
          # Some initialization here
    def forward(x):
          # forward propagation here

model = SomeNeuralNet(es,hs,dp)
optimizer = optim.Adam(model.parameters())
loss_function = nn.NLLLoss()
for epoch in N:
   for x in data:
       # Which one I should call ? optimizer.zero_grad() or model.zero_grad() or both ?       
       model.zero_grad()
       optimizer.zero_grad()
      logp = model(x)
      loss = loss_function(logp,gold_outs)
      loss.backward()
       # Which one I should call ? Optimizer.step() or model.step() or both ?
       optimizer.step()
       model.step()
</code></pre>

username: ptrblck
cooked: <p><code>nn.Module</code> doesn’t have a <code>step</code> method, so you should call <code>optimizer.step()</code>.<br>
The model itself doesn’t know anything about the optimization of its parameters.</p>
<p>In case of calling <code>zero_grad</code> it depends on your use case.<br>
If you pass all parameters to the optimizer, both calls will be identical and clear all gradients.<br>
In case you pass only certain parameters to the optimizer, <code>optimizer.zero_grad()</code> will only clear those, while <code>model.zero_grad()</code> will clear all parameters.</p>

username: coyote
cooked: <p>Thank you for the explanation.</p>

username: prismspecs
cooked: <p>Does the optimizer step() method change/update the optimizer? In other words, should I be saving/loading the optimizer as well as the model during the training process?</p>

username: ptrblck
cooked: <p>Yes, it might change some internal states, such as running estimates of the gradients if these are used, so that you should save the <code>optimizer.stat_dict()</code> additionally to the <code>model.state_dict()</code> to restore the training later.<br>
In case the optimizer doesn’t use these estimates, saving the <code>state_dict()</code> wouldn’t change anything and I would recommend to use it anyway.</p>
Assign gradient manually for adaptive clipping

username: Ruggero_Ragonesi
cooked: <p>I have two loss terms and I want to do an adaptive gradient clipping to prevent one of the two losses to prevail on the other.</p>
<p>By adaptive gradient clipping I mean the gradient among the two that is bigger is normalized by the other one.</p>
<p>after computing the normalized gradient I need to assign it, but then the code breaks</p>
<pre><code class="lang-auto">loss1 = ...
loss2 = ...
loss_tot = loss1 + loss2

model.optimizer.zero_grad()
loss_tot.backward(retain_graph=True)

gg1 = grad(loss1, model.encoder.parameters())
gg2 = grad(loss2, model.encoder.parameters())

# gradient clipping
for i, (g1, g2) in enumerate(zip(gg1, gg2)):

    # compute L2 norms
   g1_ = g1.norm(2)
   g2_ = g2.norm(2)
   g_ = torch.min(g1_, g2_)

   # normalize gradients
   gg2[i] = torch.mul(g_, torch.div(g2, g2_)) # --- NOT WORKING ---


model.optimizer.step()
</code></pre>
<p>I get:<br>
<code>TypeError: 'tuple' object does not support item assignment</code></p>
<p>I am new to Pytorch, so this solution may not be the best one.<br>
Thanks in advance</p>