akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I'm not sure if im correctly able to convert my neural network design (in form of a picture) to pytorch code?

username: Keshav_Sharma
cooked: <p>Hello, I’m new to PyTorch and bit confused about exactly what im building,<br>
so far I’ve never used any ML framework before and my knowledge is based on theory only. I’m learning PyTorch and got the idea of auto-grad and various other stuff however now that its time to build actual stuff, im facing a new issue that is, im not sure how do i exactly build a neural network with specific design, look at the image :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57edfc4012ffb14e662b2e8c4321788835aaadf6.png" data-download-href="https://discuss.pytorch.org/uploads/default/57edfc4012ffb14e662b2e8c4321788835aaadf6" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57edfc4012ffb14e662b2e8c4321788835aaadf6.png" alt="Untitled" data-base62-sha1="cxRq74IJkM5wdOFao1jyaO9vxn8" width="690" height="499" data-dominant-color="ECEEE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">706×511 30.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Okay, so as you can see, suppose i want to build above mentioned neural network (see pic), now my problem is, when i see PyTorch documentation regarding various layers that i can use, i see Linear layer which i believe is what i need to use to build my desired architecture however there is no way to specify non-linear activation function with it,</p>
<p>then i explored more and found “activation functions” (note : NOT LAYERS) and im now confused, sigmoid function doesn’t seem to have same constructor as linear layer where i can specify shape of input and output specifically like in case of Linear Layer constructor<br>
now, what do i do ? does PyTorch want me to use both Linear Layers and Activation function separately , like this ?</p>
<pre><code class="lang-auto">class xyz(Module):
    self.layer1 = torch.nn.Linear(in_features, out_features)
    self.layer1_1 = torch.nn.Sigmoid()
    ..... 
   def forward(self, inp):
      output1 = self.layer1(...) #ignore parameters for now 
      output1_after_activation = self.layer1_1(output1) 
      ...
      ... #above code repeated 2 more times for remaining 2 layers 

</code></pre>
<p>am i correct or i got it all wrong ?</p>

username: G.M
cooked: <p>I think u r correct. Sigmoid is just a function, it is an element-wise operation thus does not have a specific input shape.</p>
<pre><code class="lang-auto">layer = nn.Linear(INPUT_SIZE, OUTPUT_SIZE)
sig = nn.Sigmoid()
inp  = tc.randn(BATCH_SIZE, INPUT_SIZE)
inp = layer(inp)
inp = sig(inp)
print(inp.shape) # (BATCH_SIZE, OUTPUT_SIZE)
</code></pre>
Num_workers tesla v100

username: slavavs
cooked: <p>If I have two tesla v100 graphics cards. Do I need to configure num_workers&gt; 0 in dalaloader or leave 0?</p>

username: Hemantr05
cooked: <p>With respect to Dataloaders:<br>
After going through the pytorch documentation for Dataloader, if you don’t assign the number of subprocesses for num_workers, it assign the task to the main process;<br>
which means, the ‘n’ number of workers assigned calls ‘n’ subprocesses.</p>

username: Hemantr05
cooked: <p><a class="mention" href="/u/slavavs">@slavavs</a> I hope the above answer helps</p>
Mean and std values for transforms.Normalize

username: void32
cooked: <p>Hi,<br>
How do I choose the values for mead and std when using transforms.Normalize(mean, std)?</p>
<p>I have seen both examples where Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])]) are used, but also cases where Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]) are used.<br>
How are these values found; should they be calculated from my data set or are they appropriate constants?</p>
<p>An example from our codebase looks like this:</p>
<pre><code class="lang-auto">def get_transform_eval(input_shape):
    return transforms.Compose(
        [transforms.ToPILImage(),
         transforms.Resize(input_shape),
         transforms.ToTensor(),
         transforms.Normalize(MEAN, STD)])
</code></pre>

username: Hemantr05
cooked: <p><a class="mention" href="/u/void32">@void32</a> the values taken into consideration for mean and std for the transforms:</p>
<blockquote>
<p>Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])]) is used for the ImageNet dataset</p>
</blockquote>
<p>And,</p>
<blockquote>
<p>Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])] is used to convert input image colors to gray scale<br>
You can check <a href="https://discuss.pytorch.org/t/understanding-transform-normalize/21730" class="inline-onebox">Understanding transform.Normalize( )</a> for a better insight</p>
</blockquote>

username: void32
cooked: <p>Hi Hemant,<br>
Thanks for your quick answer.<br>
Okay, it’s simply the mean and std calculated from the ImageNet dataset, so in theory, I should calculate for my own data?</p>

username: Hemantr05
cooked: <p><a class="mention" href="/u/void32">@void32</a> yes, for a custom dataset to get better results</p>
Batch Size issue in MLP

username: AshviniKSharma
cooked: <p>Is there any necessity to have the total number of files to be multiple of batch_size in pytorch? I am having multiple folders containing NumPy files, but each folder is not a multiple of 32 and I am getting size mismatch error while training the network? Is there any fix or tips on loading data so that size mismatch does not occur?</p>

username: Hemantr05
cooked: <p><a class="mention" href="/u/ashviniksharma">@AshviniKSharma</a> could you please share a screenshot of the error, so as to get a clear idea of the size mismatch?</p>

username: AshviniKSharma
cooked: <p>I dont have the screenshot right now, as I have manually set all folder files to be multiple of 32 , but its not working when total files are not multiple of batchsize(32).</p>

username: Hemantr05
cooked: <p><a class="mention" href="/u/ashviniksharma">@AshviniKSharma</a> Try increasing the number of files in each folder (not necessary multiples of 32) and check if it would make any difference.</p>
Mpi4py and PyTorch

username: takafusui
cooked: <p>Hi,</p>
<p>I think it would be a known issue but I cannot find the correct way in the forum…</p>
<p>I have some VMs on the server and want to broadcast <code>torch.tensor</code> product by <code>mpi4py</code> with the following snippet but without success.</p>
<pre><code class="lang-python">if rank == 0:
   x = torch.tensor([1], dtype=torch.float64)
if rank != 0:
    x = None
    x = comm.bcast(x, root=0)
print("Rank {} recieves {}".format(rank, x)))
</code></pre>
<p>How should I broadcast tensors between several ranks?</p>
<p>Thank you in advance.</p>

username: takafusui
cooked: <p>Sorry… I messed up. The correct syntax is</p>
<pre><code class="lang-python">if rank == 0:
    x = torch.tensor([1], dtype=torch.float64)
if rank != 0:
    x = None
x = comm.bcast(x, root=0)
print("Rank {} recieves {}".format(rank, x))
</code></pre>
<p>and now it works as I intend:</p>
<pre><code class="lang-python">Rank 4 recieves tensor([1.], dtype=torch.float64)
Rank 9 recieves tensor([1.], dtype=torch.float64)
Rank 0 recieves tensor([1.], dtype=torch.float64)
Rank 1 recieves tensor([1.], dtype=torch.float64)
Rank 2 recieves tensor([1.], dtype=torch.float64)
Rank 3 recieves tensor([1.], dtype=torch.float64)
Rank 5 recieves tensor([1.], dtype=torch.float64)
Rank 6 recieves tensor([1.], dtype=torch.float64)
Rank 7 recieves tensor([1.], dtype=torch.float64)
Rank 8 recieves tensor([1.], dtype=torch.float64)
</code></pre>
Printing model summary

username: singhvishal0209
cooked: <p>I am trying to print the no. of parameters in my model.</p>
<p>I used</p>
<blockquote>
<p>!pip install pytorch-model-summary<br>
from pytorch_model_summary import summary</p>
<p>batch_size_train = 2<br>
batch_size_validate = 1</p>
<p>num_epochs = 10<br>
learning_rate = 0.000009<br>
img_width = 720<br>
img_height = 720</p>
<p>model = Att_Unet(in_channels =1, out_channels =1)<br>
model.cuda()</p>
<p>print(summary(model, torch.zeros(2, 1, 720, 720), show_input=True))</p>
<p>criterion = Diceloss().cuda()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)</p>
</blockquote>
<p>I get an error in google collab</p>
<blockquote>
<p>RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same</p>
</blockquote>
<p>but when I use</p>
<blockquote>
<p>torch.zeros(2, 1, 720, 720).cuda()</p>
</blockquote>
<p>I get error as</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 252.00 MiB (GPU 0; 7.43 GiB total capacity; 6.76 GiB already allocated; 74.94 MiB free; 6.84 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>but my model still runs without any error of CUDA out of memory, this only happens when I am trying to print model summary</p>
Save image issue after converting 2d to 3d array

username: cbd
cooked: <p>I convert 2d array(arr2d) to 3d array(arr3d) and save as image using below code. arr2d is float64 type.  Why save image is not color image?</p>
<p>arr3d[:, :, 0] = arr3d[:, :, 1] = arr3d[:, :, 2] = arr2d<br>
arr3d=arr3d*255<br>
im=Image.fromarray(np.maximum(np.minimum(arr3d, 255), 0).astype(np.uint8))<br>
im.save(“sample.png”)</p>

username: ptrblck
cooked: <p>This dummy code produces a color image:</p>
<pre><code class="lang-python">x = np.random.randn(224, 224, 3).astype(np.float64)
x = x * 255
x = np.clip(x, 0, 255)
img = Image.fromarray(x.astype(np.uint8))
</code></pre>
<p>Could you check that your color channels have different values?</p>

username: cbd
cooked: <p>Okay. Thats i want. Thank You.</p>
Custom loss function based on NLL + MSE

username: JohnCwok
cooked: <p>Hi !</p>
<p>As implied in the title, I tried to define a custom loss function for a simple LSTM model by combining NLL loss and MSE loss. Since the output of my model is a probability distribution over several classes, I take the argmax of it in order to be able to compute the MSE loss.<br>
But I run into this error :</p>
<p><code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>
<p>It seems that the <code>argmax()</code> function changes the <code>requires_grad</code> attribute of a tensor to <code>False</code>. Is there a way to ensure this doesn’t happen ? Or is this because differentiating max() is mathematically not very defined ?</p>

username: JohnCwok
cooked: <pre><code class="lang-auto">        pred = model(x)[0]
        flat_pred = torch.reshape(pred, (pred.size(0) * pred.size(1), pred.size(2)))
        #flattening for batch loss computation
        loss =  mse(torch.argmax(flat_pred, dim = 1).double(), y.flatten().double()) #+  nll(flat_pred, y.flatten())
        total_loss.append(loss)
</code></pre>
<p>If I print the <code>requires_grad</code>, I see that it is true for the <code>flat_pred</code> tensor, but <code>torch.argmax(flat_pred, dim = 1).double().requires_grad</code> is False</p>

username: JohnCwok
cooked: <p>I have to do it because argmax returns a tensor of type <code>Long</code> and MSE is not defined for these kind of tensors. I will try rewritting the train loop, maybe there another insidious bug that I didn’t spot.</p>

username: G.M
cooked: <p>Sorry, I mixed <code>argmax</code> with <code>max</code>. <a class="mention" href="/u/johncwok">@JohnCwok</a> is right,  the index returned by <code>argmax</code> does not have gradients. In this case, if ur output is onehot, u don’t have to use the <code>argmax</code>, just feed it into the <code>MSELoss</code>.</p>
Time series custom dataset train/val/test split

username: kedar
cooked: <p>Hello. I would like to perform train/val/test split on my time series dataset. The data is stored in a MariaDB database, thus I had to create custom Pytorch dataset, that actually comprises of TWO Dataset classes:<br>
First dataset is responsible for getting chunk of indices of database rows to be read, so that we will load data in chunks to diminish the memory usage.<br>
Example of generating chunk indices:</p>
<pre><code> DB      Table
table   chunks
-----   ------
1.       1.
2.       2.
3.  --&gt;  3.
4.      ------
5.       3.
         4.
         5.
</code></pre>
<p>The second dataset will then create sliding windows within these chunks, and fetch corresponding data from databse.<br>
Example of generating sliding windows within chunks of data:</p>
<pre><code> DB      Table      Sliding
table   chunks      windows
-----   ------      ------
1.       1.         1., 2.
2.       2.     --&gt; ------
3.  --&gt;  3.         2., 3.
4.      ------      ------
5.       3.         3., 4.
         4.     --&gt; ------
         5.         4., 5.
</code></pre>
<p>How to perform train/val/test spliting of such dataset. I heard about <code>torch.utils.data.random_split(dataset, lengths)</code> but i don’t want my data to be split randomly. Is there any tool that can be used with Time series and Pytorch Dataset, so that I don’t have to create sepparate Dataset classes for each train/val/test set?</p>
<p>Any hint will be appreciated.</p>
Is it necessary for model enable an desable in training and testing?

username: rofelia09
cooked: <p>Hi<br>
suppose my model created with name is net.pth. Is is necessary  to make it enable using <code>net.train()    </code> before training and also before evaluation disable using  <code>net.eval() </code> before evaluating.</p>

username: ptrblck
cooked: <p>Yes, this is necessary as <code>model.eval()</code> will set all modules to evaluation mode.<br>
For example batch norm layers will use the running statistics to normalize the current sample instead of the batch statistics and also won’t update the running stats, while dropout layers will be disables during evaluation.</p>
Set learnable parameters to random values

username: science
cooked: <p>Is it possible to set the learnable parameters of a model to random values?  Thanks in advance.</p>

username: ptrblck
cooked: <p>Yes, usually parameters are initialized with a specific initialization method to random values.<br>
E.g. the parameters of <code>nn.Linear</code> are reset in <a href="https://github.com/pytorch/pytorch/blob/bf0951d937bf65eebfacf481257eae7aa3353419/torch/nn/modules/linear.py#L79-L84">these lines of code</a>.</p>
<p>If you want to use an init method, have a look at <code>torch.nn.init</code>.</p>
Sum of squared 2nd-order derivatives

username: mikey
cooked: <p>Hi, I have been stuck on a problem, and I’m wondering if there is a solution. Thank you in advance!</p>
<p>I have two embedding vectors, <code>u</code> and <code>v</code> both of size <code>m</code>, that are inputs to a torch <code>model</code>, and I would like to efficiently compute a form of the model’s summed-hessian <em>between</em> the two vectors. I found that the simple approach is efficient enough for me</p>
<pre><code class="lang-auto">x = torch.stack([u,v]).view(1,2,m) #batchsize of 1, 2 vectors u v, each with size m
y = model(x) #y is a scalar

grads = (autograd.grad(y, x, create_graph=True)[0].squeeze()).sum(1) #sum across emb vectors (not between them)
grad_u = grads[0] #a scalar
grads = (autograd.grad(grad_u, x, retain_graph = True)[0].squeeze()).sum(1) #sum across emb vectors
grad_uv = grads[1] #a scalar
</code></pre>
<p>Note the <code>.sum(1)</code> in each <code>autograd.grad</code> line.<br>
Under the hood, autograd computes <code>grad_uv</code> to be the sum of derivatives: <code>(d^2 y)/(du_1dv_1) + (d^2 y)/(du_1dv_2) + (d^2 y)/(du_2dv_1)  + (d^2 y)/(du_2dv_2) </code> if the sizes of <code>u</code> and <code>v</code> are just <code>m=2</code>.</p>
<p>However, what I would really like is this expression with the sum of squares: <code>((d^2 y)/(du_1dv_1))**2 + ((d^2 y)/(du_1dv_2))**2 + ((d^2 y)/(du_2dv_1))**2  + ((d^2 y)/(du_2dv_2))**2 </code></p>
<p>It seems that chain rule cant give me this. Is it possible to compute this expression without explicitly computing the second <code>autograd.grad</code> line over <em>each</em> element of vector <code>dy/du</code>? I basically want to avoid the extra <code>for</code> loop as is common in higher-order grad computations. I was thinking that the vector-jacobian product within <code>grad()</code> could help, but would I need modify <code>autograd</code> source code?</p>
<p>Thank you so much!</p>
Anaconda3-2019.1 does not load pytorch using recommended conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

username: hadfield75
cooked: <p>Pytorch load via conda gives error</p>

username: ptrblck
cooked: <p>What error are you seeing?</p>

username: hadfield75
cooked: <p>I solved the problem by getting the Github repository of Pytorch and putting it into the Anaconda 3 packages directory.</p>
Decrease in the Speed of Quantization Model

username: mohit7
cooked: <p>I have my base model as Squeezenet after porting it to android the execution time of model is around 160ms and after Quantizing the model generally the time should decrease but the time is increased to 220ms. Is there is something wrong or this can be possible?</p>

username: jerryzh168
cooked: <p>are you sure everything is properly quantized? could you print the model before and after quantization?</p>

username: mohit7
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> I printed the model and everything is properly quantized.<br>
In squeezenet I have to replace torch.cat to nn.FloatFunctional.cat so I think that function is giving issues which is also mentioned in other threads also</p>

username: masahi
cooked: <p><a class="mention" href="/u/mohit7">@mohit7</a> My answer at <a href="https://discuss.pytorch.org/t/quantized-cat-running-time-is-slower-than-fp32-model/68717/4" class="inline-onebox">Quantized::cat running time is slower than fp32 model</a> may help answer your question.</p>
Speed up training when large part of training doesn't require grad?

username: alekhka
cooked: <p>Hello,</p>
<p>I have an atypical training regime - a large part (roughly 80%) of the forward call of the model doesn’t require storing activations. So, I do that by enclosing in no_grad() (there is also parts where this model needs to store activations - that is outside no_grad() context). I was wondering if there is a way I can speed up these computations - since they are almost like inference - so I should be able to use FP16 speedup tricks? (my GPUs do have Tensor cores)</p>
<p>Should I use something like Nvidia Apex (which should switch to FP16 when executed in no_grad() ?)<br>
or should I explicitly convert the model to FP16 (with model.half()) and convert it back to FP32 every time I want to do grad enabled computations?</p>
<p>Thanks!</p>
Expected a Tensor of type torch.cuda.FloatTensor but found a type torch.FloatTensor followed by cuda out of memory error

username: Vishnu_Gopireddy
cooked: <p>Expected a Tensor of type torch.FloatTensor but found a type torch.cuda.FloatTensor for sequence element 5 in sequence argument at position <span class="hashtag">#1</span> ‘tensors’<br>
tensor([13,  0, 63, -1], device=‘cuda:0’)<br>
tensor([17,  0, 69, -1], device=‘cuda:0’)<br>
tensor([20,  0, 74, -1], device=‘cuda:0’)<br>
tensor([25,  0, 78, -1], device=‘cuda:0’)<br>
Expected a Tensor of type torch.cuda.FloatTensor but found a type torch.FloatTensor for sequence element 1 in sequence argument at position <span class="hashtag">#1</span> ‘tensors’<br>
CUDA error: out of memory<br>
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMath.cu line=35 error=2 : out of memory<br>
cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCTensorMath.cu:35<br>
CUDA error: out of memory (allocate at /pytorch/aten/src/THC/THCCachingAllocator.cpp:510)<br>
frame <span class="hashtag">#0:</span> THCStorage_resize + 0x123 (0x7f695dd4a123 in /home/coea/.local/lib/python3.5/site-packages/torch/lib/libcaffe2_gpu.so)<br>
frame <span class="hashtag">#1:</span> THCTensor_resizeNd + 0x293 (0x7f695dd55633 in /home/coea/.local/lib/python3.5/site-packages/torch/lib/libcaffe2_gpu.so)<br>
frame <span class="hashtag">#2:</span> THCudaTensor_catArray + 0x295 (0x7f695de03a35 in /home/coea/.local/lib/python3.5/site-packages/torch/lib/libcaffe2_gpu.so)</p>
<p>Training a model with CNN+LSTM+CTC loss getting this error… Please help me out. Thanks in advance.</p>
Video Dataloading

username: anghst
cooked: <p>I have a dataset of videos, but i want to load them into a CNN frame by frame and analyze them that way (temporal part is being handled by an LSTM). Would I need to make a custom dataset to make that work, or is there another option that works better? If I need to make a custom dataset, how would I handle the conversion from a video to an array of images?</p>
Different weighting of labels using CrossEntropyLoss

username: Bendrien
cooked: <p>Hello there, i am using CrossEntropyLoss for training on sequence data. On my sequence there are sparse single events of which i have labels and their position as indices for the output sequence. I would like the neighbouring samples of a single event to be labeld in the same manner as the event it self but with reduced weighting. This is what i have in mind in pseudo code:</p>
<pre><code class="lang-auto">weight = 0.5
for inputs, labels, label_indices in dataloader:
    outputs = model(inputs)
    normal_loss = criterion(outputs[label_indices], labels)
    left_loss = criterion(outputs[label_indices - 1], labels)
    right_loss = criterion(outputs[label_indices + 1], labels)
    loss = normal_loss + left_loss * weight + right_loss * weight
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>I this approach suitable?</p>

username: ptrblck
cooked: <p>Your approach should work, if you also slice the <code>labels</code> tensor.<br>
However, if might be easier to avoid the loss reduction via <code>reduction='none'</code> and multiply your loss tensor with a weight tensor.<br>
However, I doubt you will see a huge performance difference. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to create Axis-aligned bounding boxes using cv2.Flatten()

username: Samiha
cooked: <p>I am trying to get axis-aligned bounding box where the original algorithm used rotated bounding boxes. corresponding lines are –<br>
if len(contours) != 0 and np.max(cnt_area) &gt; 100:<br>
contour = contours[np.argmax(cnt_area)] # use max area polygon<br>
polygon = contour.reshape(-1, 2)<br>
pbox = cv2.boundingRect(polygon) # Min Max Rectangle<br>
<span class="hashtag">#pbox</span> = cv2.boxPoints(cv2.minAreaRect(polygon))<br>
box_in_img = pbox<br>
state[‘ploygon’] = rbox_in_img if mask_enable else []<br>
location = state[‘ploygon’].flatten()</p>
<p>But getting the following error:  tuple object doesnot have flatten. but flatten() previously worked with rotated bounding boxes.<br>
Then if I delete the flatten() function, I do not get errors but bounding boxes are not correct.<br>
<img src="https://user-images.githubusercontent.com/9627710/74807717-56904180-5313-11ea-82dc-d01fa86acd32.png" alt width="396" height="90"></p>
<p>Can someone give suggestions to solve this?</p>
What is the training image size for Pytorch's own pre-trained image-net

username: JonathanSum
cooked: <p>Hi, every time I use PyTorch torchvison.module, I need to download the pre-trained model. However, I feel it does not work very well for me, and I am thinking to train my own pre-trained. But I guess the image does not need to be very high. If It does not hit the improvement, I can make my dataset like 5 or 3 times smaller. I use my dataset not only for classification.</p>

username: ptrblck
cooked: <p>Most of the classification models were trained with a spatial size of <code>224x224</code>, with the exception of Inception, which was trained with <code>299x299</code>.</p>
<p>What kind of dataset are you using which yields a bad performance?</p>

username: JonathanSum
cooked: <p>Thank You for telling me that. It is a Japanese anime dataset. But the problem is when it is being used for coloring, the image-net won’t work for coloring. I have seem this on my work, and I can color the object with my pre-trained model only but not the image-net.</p>
Implement a meta-trainable step size

username: pinocchio
cooked: <p>I want to implement a (meta) trainable step size. I tried it with this post:</p>
<aside class="quote quote-modified" data-post="17" data-topic="70076">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17">How does one have the parameters of a model NOT BE LEAFS?</a> 
  </div>
  <blockquote>
    thanks for that! 
I’ve been playing around with the library but was wondering if it was possible to have a trainable step-size with that library. Is it possible? 
I tried: 
#
child_model = nn.Sequential(OrderedDict([
        ('conv1', nn.Conv2d(in_channels=3,out_channels=2,kernel_size=5)),
        ('relu1', nn.ReLU()),
        ('Flatten', Flatten()),
        ('fc', nn.Linear(in_features=28*28*2,out_features=10) )
    ]))
eta = nn.Sequential(OrderedDict([
    ('fc', nn.Linear(1,1)),
    ('sigmoid…
  </blockquote>
</aside>

<p>and with the higher library (<a href="https://github.com/facebookresearch/higher" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.</a>) with no luck…</p>
<p>I tried:</p>
<pre><code class="lang-auto">eta = torch.tensor([0.5], requires_grad=True).view(1)
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
#meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_params = itertools.chain(child_model.parameters())
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
# do meta-training/outer training minimize outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) ) 
nb_outer_steps = 10 # note, in this case it's the same as number of meta-train steps (but it's could not be the same depending how you loop through the val set)
for outer_i, (outer_inputs, outer_targets) in enumerate(testloader, 0):
    meta_opt.zero_grad()
    if outer_i &gt;= nb_outer_steps:
        break
    # do inner-training/MAML; minimize innerloop: theta^{T} - eta* Grad L^train(theta^{T}) ~ argmin L^train(theta)
    nb_inner_steps = 3
    with higher.innerloop_ctx(child_model, inner_opt) as (fmodel, diffopt):
</code></pre>
<p>with error:</p>
<pre><code class="lang-auto">Exception has occurred: RuntimeError
Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>which wouldn’t work anyway cuz eta might become negative suddenly so I really want to cap it with a sigmoid function but had to try something…</p>
<hr>
<p>related:</p>
<p>SO: <a href="https://stackoverflow.com/questions/60311609/how-does-one-implement-a-meta-trainable-step-size-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How does one implement a meta-trainable step size in Pytorch? - Stack Overflow</a></p>
1Mb gpu memory can't be relased!

username: zhishen_nie
cooked: <p>While i run the code of FGSM, i find there is  1mb gpu memory can’t br relased in every loop.And available GPU memory is decreasing. My code is here:<br>
def gen_adv(model, device, data, target, epsilon):</p>
<pre><code>data, target = data.to(device), target.to(device)


data.requires_grad = True

# 1067mb  1068mb
output = model(data)
# 1202mb  1203mb

init_pred = output.max(1, keepdim=True)[1]
if init_pred.item() != target.item():
    return
 # 1202mb  1203mb
loss = F.nll_loss(output, target)
 # 1202mb  1203mb
model.zero_grad()
 # 1202mb  1203mb
loss.backward()
 # 1068mb  1069mb
data_grad = data.grad.data
 # 1068mb  1069mb
data = unnormalized_show(data)
 # 1068 mb 1069mb


perturbed_data = fgsm_attack(data, epsilon, data_grad)
perturbed_data = perturbed_data.cpu().detach().numpy()


perturbed_data = perturbed_data.reshape(3, 224, 224)
perturbed_data = np.transpose(perturbed_data, (1, 2, 0))

return perturbed_data
</code></pre>
<p>The meaning of the comment is the used gpu memory. After the first execution of the program, the GPU memory used has been increased from 1067mb to 1068mb. And  after the second execution of the program, the GPU memory used has been increased from 1068mb to 1069mb.() This problem caused my program to run interrupted. GPU memory runs out quickly.QAQ.<br>
How can I resolve this problems? Help me QAQ</p>

username: ptrblck
cooked: <p>Are you storing some tensors in e.g. a list without detaching them?<br>
Also, could you use:</p>
<pre><code class="lang-python">with torch.no_grad():
    data_grad = data.grad
</code></pre>
<p>instead of calling the <code>.data</code> attribute, as it might yield unwanted side effects.</p>
What does task_num mean in the MAML tutorial for higher?

username: pinocchio
cooked: <p>I saw this</p>
<pre><code class="lang-auto">        for i in range(task_num):
</code></pre>
<p>in the MAML tutorial for higher (<a href="https://github.com/facebookresearch/higher/blob/master/examples/maml-omniglot.py" rel="nofollow noopener">https://github.com/facebookresearch/higher/blob/master/examples/maml-omniglot.py</a>) but I thought it was confusing because there is only 1 task and therefor 1 data set. So what does that parameter actuall mean?</p>
How to shrink a tensor while removing specific elements

username: isalirezag
cooked: <p>Given I have a tensor with dimensions<code> bxcxh,w,</code> I like to remove the max value in dimention <code>c</code> and then make a new tensor with dimention <code>bxc-1,w,h</code>.<br>
I can do it in loop, but too slow.</p>
<p>Can anyone suggest what I can use instead of the for loop here  ?</p>
<p>here is an example:</p>
<pre><code class="lang-auto">import torch

mat = torch.rand(1,3,4,4)
b,c,h,w = mat.shape

mat_max = torch.max(mat,dim=1)[0].reshape(h*w,1)
mat_empt =torch.empty((h*w,c-1))

mat_shape = torch.reshape(mat.permute(0,2,3,1),(h*w,c))
A = mat_shape==mat_max
if len(torch.where(torch.sum(A,axis=1)&gt;1)[0]) ==0:
        for indx,k in enumerate(mat_shape):
            mat_empt[indx] =     k[~A[indx]]
# else:
    # need to fix it later
    
new_mat = torch.reshape(mat_empt,(h,w,c-1)).unsqueeze(0).permute(0,3,1,2)

</code></pre>
What could be the cause of big fluctuations in accuracy between epochs?

username: alx
cooked: <p>I was wondering what it mean when there’s &gt;10% difference in accuracy between two epochs. Is that due to the lr? or somehting else,</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ffc6c565693f1101aaa023593f7daca815c848a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ffc6c565693f1101aaa023593f7daca815c848a6" title="Screen Shot 2020-02-16 at 9.52.38 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc6c565693f1101aaa023593f7daca815c848a6_2_690x189.png" alt="Screen Shot 2020-02-16 at 9.52.38 PM" data-base62-sha1="AuHCrq3dTpIkJ8S3Ua2cFDNvRhY" width="690" height="189" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc6c565693f1101aaa023593f7daca815c848a6_2_690x189.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc6c565693f1101aaa023593f7daca815c848a6_2_1035x283.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ffc6c565693f1101aaa023593f7daca815c848a6_2_1380x378.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-16 at 9.52.38 PM</span><span class="informations">2320×638 81.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ziqipang
cooked: <p>There might be various reasons that potentially lead to this. Please update more information.</p>
<p>To diagnose on the problem, more descroptions on your training is necessary, such as the full image of loss curve (both train and validation), the scale of dataset, the policy of dropping learning rate.</p>
<p>Moreover, some deep learning tasks just has this kind of characteristics, such as GAN and NAS, where the proper epoch of training achieves better performance than training more epochs.</p>

username: TinfoilHat0
cooked: <p>Could be due to big LR. Those big up-downs usually mean that your model is overshooting its target. Try reducing it by 10 or so. Also checking schedulers of PyTorch might help.</p>
Stacking LSTM and Linear layers in a time distributed manner

username: hamed
cooked: <p>I want to pass the output from each timestep of an LSTM layer through the same linear layer. meaning that the output from each timestep would itself go through the linear layer and provide a vector of target_size and the same would happen for the following timesteps.<br>
Is the following code the right way to do this?</p>
<pre><code class="lang-auto">class dynamicLayer(nn.Module):

    def __init__(self, input_dim, hidden_dim, num_layers, target_size=7):
        super(dynamicLayer, self).__init__()
        self.hidden_dim = hidden_dim
        self.input_dim = input_dim
        self.num_layers = num_layers
        self.hidden_latent = &lt;random vector from torch.rand&gt;
        
        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers)
        self.hidden2out = nn.Linear(hidden_dim, target_size)
    
    def forward(self, input):
        # input (seq_len, batch, input_size)
        # hidden state (num_layers, batch, hidden_size)
        # output (seq_len, batch, hidden_size)

        lstm_out, (h_n, c_n) = self.lstm(input, self.hidden_latent)
        self.hidden_latent = (h_n, c_n) 
        out = self.hidden2out(lstm_out)
        return out
</code></pre>
Best way to handle a data pipeline where training inputs are randomly sampled and retrieved from a 2nd dataset with torch.utils.data

username: Santosh-Gupta
cooked: <p>Here is an example what I am trying to do. Here’s what my dataset may look like</p>
<pre><code>2   | 34, 64, 2243, 55678, 323, 778, 4454, 23, 4433, 3445, 455, 32
23  | 343 ,56, 2, 5, 675, 34, 232, 677, 7, 54, 436, 77, 85, 33
592 | 343, 54, 4, 6
23  | 34
123 | 2, 4, 54, 38, 6643, 67, 3
...
</code></pre>
<p>Each of these numbers are indexes (not the actual model inputs/target data) which point to the actual data that will be fed into the model. The actual model data is in a separate dataset, organized by indexes (same number used in the first dataset), so it would look something like this</p>
<pre><code>1 | data1
2 | data2
3 | data3
</code></pre>
<p>For the first set, on the left side are indexes to the targets, so 2, 23, 592, 23, 123. On the right side are indexes to the potential inputs, and for each target, 4 inputs are picked at random. For targets which have a number of inputs less than 4, inputs are repeated.</p>
<p>So taking the first training example in the first set, the model input would look like this</p>
<pre><code>target = torch.tensor([data2])
input = torch.mean( torch.tensor([data34]), torch.tensor([data2243]), torch.tensor([data23]), torch.tensor([data32]) )
</code></pre>
<p>Both sets are too big to hold in memory at the sametime, and I don’t have enough diskspace to save targets with every permutation of inputs.</p>
<p>I think the best strategy is to have X number of batches set up ahead of time, using multithreading to setup future batches, so that the data pipeline does’t become the bottle neck in the training, but I am wondering how to go about doing that.</p>
<p>Edit:</p>
<p>I think my description is a bit hazy, so I made an analogy of the type of data pipeline I’m trying to develop.</p>
<p>An analogy would be some types of word embedding trainings, where the number of input words is smaller than the context window. So for a particular target word, the input words need to be chosen at random within the context window of that target word.</p>
<p>For example, take this sentence:</p>
<p><code>There once was a very fast dog who could out run any other animal in the city. </code></p>
<p>Say the target word was dog, and that the context window size was 6, but the input size was 2. So we have a choice of ‘a’, ‘very’, ‘fast’, ‘who’, ‘could’, ‘out’, of which we need to pick two inputs randomly. So an example training example would be the word embedding for ‘dog’ as the target, and the word embeddings of ‘fast’ and ‘out’ for the inputs.</p>
<p>So in word embedding trainings, many different inputs will be used in many different targets. But with word embedding trainings, all the word embeddings are able to be stored in live memory since the vocab is 6 figures. For my case, the inputs can’t all be in live memory.</p>
<p>So an idea of what I’m looking to do is word2vec training but most of the word embeddings have to stay in disk space at any particular instance.</p>
Cuda not available - Cuda 10.0, Driver Version 410.129, Pytorch 1.4.0 

username: Alertbobtum
cooked: <p>Hello. I am having trouble getting Pytorch to detect my GPUs. It was detecting them previously with CUDA 9.0, but after upgrading to CUDA 10.0 it’s not working. From what I’ve read my driver version is high enough for CUDA 10.0. Here is output from a system environment script:</p>
<p>Collecting environment information…<br>
PyTorch version: 1.4.0<br>
Is debug build: No<br>
CUDA used to build PyTorch: 10.0</p>
<p>OS: Ubuntu 16.04.6 LTS<br>
GCC version: (Ubuntu 7.4.0-1ubuntu1<del>16.04</del>ppa1) 7.4.0<br>
CMake version: version 3.17.0-rc1</p>
<p>Python version: 3.7<br>
Is CUDA available: No<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: TITAN Xp<br>
GPU 1: TITAN Xp</p>
<p>Nvidia driver version: 410.129<br>
cuDNN version: Could not collect</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.17.1<br>
[conda] blas 1.0 mkl<br>
[conda] mkl 2020.0 166<br>
[conda] mkl-service 2.3.0 py37he904b0f_0<br>
[conda] mkl_fft 1.0.15 py37ha843d7b_0<br>
[conda] mkl_random 1.1.0 py37hd6b4f25_0<br>
[conda] pytorch 1.4.0 py3.7_cuda10.0.130_cudnn7.6.3_0 pytorch</p>
<p>Process finished with exit code 0</p>
<p>Is there something obviously wrong here?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>Did you restart your machine after updating the driver?<br>
Are other applications able to use your GPU?<br>
How did you install PyTorch? Did you build it from source or install the binaries?</p>

username: Alertbobtum
cooked: <p>Updating to a newer nvidia-driver (430.64) and restarting fixed it. Thanks.</p>
What happens if I replace the last FC layers with a Conv layer

username: banikr
cooked: <p>I am working on a regression problem that takes the global correlation of brain volumes into account. So I have to feed the whole image like 172x220x156 size into the network.<br>
So when I use the FC layer:<br>
<code>self.fc = nn.Linear(30720, num_colEls=256) </code><br>
The cuda out of memory error generates.</p>
<p>I can replace the FC layer with a Conv layer with an exactly the same Kernel size of the input.<br>
<code>self.lastConv = nn.Conv3d(512, 256, kernel_size=(4, 4, 3), stride=1)</code></p>
<p>In terms of, learning, generalization and all how efficient will it be replacing the FC layer with a Conv?</p>
<p>And in case of Conv layer the final output shape looks like:<br>
<code>torch.Size([2, 256, 1, 1, 1])</code></p>
Error in the KL loss of pytorch

username: DidierDeschamps
cooked: <p><a>https://pytorch.org/docs/stable/nn.html#kldivloss</a></p>
<p>the formula used is</p>
<pre><code class="lang-auto">l_n = y_n * (log(y_n) - x_n) 
</code></pre>
<p>while it should be</p>
<pre><code class="lang-auto">l_n = y_n * (log(y_n) - log(x_n)) 
</code></pre>
<p>this could be solved with</p>
<pre><code class="lang-auto">loss(log(x), y)
</code></pre>
<p>instead of</p>
<pre><code class="lang-auto">loss(x, y)
</code></pre>
<p>But this is not pretty …</p>
<p>Did you do it on purpose and if you did why?</p>

username: tom
cooked: <p>As it says on the page</p>
<blockquote>
<p>the input given is expected to contain <em>log-probabilities</em> and is not restricted to a 2D Tensor. The targets are given as <em>probabilities</em> (i.e. without taking the logarithm).</p>
</blockquote>
<p>So it’s not an error.</p>
<p>What happens is that starting from probs and then taking the logarithm is less numerically stable than directly taking the log as an argument.<br>
Also, it’s the same convention as NLLLoss.</p>
<p>Best regards</p>
<p>Thomas</p>
How to train deeplabv3 or fcn model for segmantation?

username: hamzaygrtc
cooked: <p>I labeled my images with <a href="https://github.com/wkentaro/labelme" rel="nofollow noopener">labelme</a>. How could I train deeplabv3 or fcn model with my labeled dataset ? (I will apply model remove to background code that is following)</p>
<p><a href="https://github.com/spmallick/learnopencv/blob/master/app-seperation-semseg/Background-Removal.py" rel="nofollow noopener">Code</a></p>
Error installing pytorch

username: jlselby231
cooked: <p>Hello, Ive been trying to install pytorch and always run into an error. Ive tried every way Ive found from searching the internet for issues but always end with pip inthe same result.</p>
<blockquote>
<p>ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: ‘C:\Users\JonathanSelby\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\caffe2\python\serialized_test\data\operator_test\collect_and_distribute_fpn_rpn_proposals_op_test.test_collect_and_dist.zip’</p>
</blockquote>
<p>Does anyone have a solution for installing this?<br>
Thanks in advanced</p>
Reproducible benchmarks with Pytorch on varied hardware

username: unrahul
cooked: <p>I am looking to do some reproducible benchmarks on number of hardware to measure framework performance. Are there any benchmarking scripts that the community uses / recommends to measure peformance of Pytorch models from the model hub on varied hardware, similar to the TensorFlow benchmarks[1]</p>
<p>[1]. <a href="https://github.com/tensorflow/benchmarks" rel="nofollow noopener">https://github.com/tensorflow/benchmarks</a></p>
How to linearly interpolate tensor of points while preserving the points themselves

username: JWL
cooked: <p>So I have a vector of 2D points of size <code>BxNx2</code>. I want to linearly interpolate the points such that I get the middle point between each point while also preserving the other points, which would effectively double the number of points to <code>Bx2Nx2</code>. Does anyone know how I can do this efficiently in a batch?</p>

username: ptrblck
cooked: <p>Would <a href="https://pytorch.org/docs/stable/nn.functional.html?highlight=grid_sample#torch.nn.functional.grid_sample"><code>grid_sample</code></a> work?<br>
You might have to use a width dimension of size 1 to match the expected shapes.</p>

username: JWL
cooked: <p>This seems to work, but then after implementing it and trying to train my model with it I seem to get this error:</p>
<pre><code class="lang-auto">  File "train.py", line 252, in upsnet_train
    loss.backward()
  File "envs/pytorch1.1/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "envs/pytorch1.1/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: invalid configuration argument
</code></pre>
<p>Note, after I interpolate the points, I compute a loss on top of it. The interpolation code is here:</p>
<pre><code class="lang-auto">B, N_old, N_new = points.size(0), points.size(1), points.size(1) * 2
grid = torch.cat([torch.zeros((B, N_new, 1, 1)),
                  torch.linspace(0, N_old, N_new).unsqueeze(0).repeat((B, 1)).unsqueeze(
                      -1).unsqueeze(-1)], dim=-1).cuda()
grid = (grid - N_old/2.) / N_old * 2.
rows = F.grid_sample(points.unsqueeze(1)[:, :, :, 0].unsqueeze(-1), grid).squeeze(1)
cols = F.grid_sample(points.unsqueeze(1)[:, :, :, 1].unsqueeze(-1), grid).squeeze(1)
points = torch.cat([rows,cols], dim=-1)
</code></pre>

username: ptrblck
cooked: <p>Could you post the shapes and dummy tensors so that we can reproduce this issue, please?</p>
The implementation of COCO metric in the tutorial

username: Michael-J98
cooked: <p>I’m following the mrcnn tutorial on the website but I got confused when I evaluated my dataset. To make the prediction results more “reasonable”, I raised  the “box_score_thresh” in the source code. (Default value is too low, maybe 0.25, that there would be some false detections with such low scores sometimes.) However, the AP value should be lower than before when it was with a lower threshold? I can’t see why, shouldn’t the FP decrease therefore get a higher AP?<br>
I really appreciate it if someone can figure it out. An abstract evaluation process is also welcome. Thanks a lot!</p>
Updating weights using unconventional rule instead of a formal loss function

username: azeeshan
cooked: <p>I want to update the weights (theta) of my neural network using the update rule (t+1:next epoch, A is a constant, f and g are function of theta):</p>
<blockquote>
<p>theta_{t+1} = theta_t + A f(theta) \nabla [ g(theta) + f(theta) ]</p>
</blockquote>
<p>where nabla gives the derivative of the g and f with respect to theta.</p>
<p>Currently, the way I am implementing this update rule is (assume f and g are torch tensors with just one element and which depend of theta, the weights of the neural network):</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = A * f.detach() * (g + f)
loss.backward()
optimizer.step()
</code></pre>
<p>Is this the correct way of implementing this weight update rule in PyTorch?</p>

username: albanD
cooked: <p>That would depend what your optimizer is.</p>
<p>It might be clearer to have a custom optimizer that does the scaling by A * f?</p>
<pre><code class="lang-auto">out = g + f
out.backward() # Compute \nable [ g + f]
optimizer.step(A * f)
</code></pre>
<p>But both should do the same step in the end (if your optimizer above is like SGD and has a lr of 1).</p>
Accessing ChainDataset

username: vr25
cooked: <p>Hi,</p>
<p>I follow that accessing ConcatDataset data (say, for training) can be done as follows:<br>
for i, (x,y) in enumerate(DataLoader(train, batch_size, shuffle=True, num_workers=5)):<br>
pass</p>
<p>I am not able to follow how to follow a similar approach on an IterableDataset joined with ChainDataset. I have an instance of ChainDatset but I am not sure how to iterate over to access the individual elements.</p>
<p>Also, I get the following error:<br>
iter() returned non-iterator of type ‘tuple’</p>
<p>Thanks!</p>

username: zhangguanheng66
cooked: <p>For <code>IterableDataset</code>, you don’t need a <code>DataLoader</code> because it’s iterable. See an example <a href="https://github.com/pytorch/text/blob/master/examples/text_classification/iterable_train.py" rel="nofollow noopener">here</a></p>

username: vr25
cooked: <p><a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a></p>
<p>Thank you for your reply.</p>
<p>Here is a challenge. say, I have 10 PyTorch training tensor files (each ~ 3GB). I cannot load them all at once into the CUDA memory. So, I was thinking of using ChainDataset by creating Iterable Dataset for each of the 10 train datasets. Now the challenge is how to iterate over an object of Chain dataset class?</p>
<p>Thanks!</p>

username: zhangguanheng66
cooked: <p>It’s part of personal choice. One thing I would investigate is <code>itertools.chain.from_iterable([dataset1, dataset2])</code></p>