akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Will there be any difference to transfer tensor to GPU in __getitem__ and in training loop?

username: nivesh_gadipudi
cooked: <p>Which practice is considered good?</p>
<ol>
<li>
<p><code>def __getitem__(self, idx):</code> to use .cuda() while returning features, labels from __get__item or</p>
</li>
<li>
<p>In the training loop<br>
features.cuda(), labels.cuda() = batch. ??</p>
</li>
</ol>

username: ptrblck
cooked: <p>I would recommend the second approach, as it would make sure to create the complete batch using multiprocessing and transferring the batch to the device.<br>
The first approach might work, but you could easily run into multiprocessing errors, if each worker tries to create a new CUDA context.</p>

username: nivesh_gadipudi
cooked: <p>I’m getting the object is not having .cuda() method when using the second one. From <strong>getitem</strong> I’m returning a tuple of (image, label). I have successfully used transformations, changed the dtype to float32. But it’s outputting the error .cuda() method can be used on that.</p>
<p>But works totally fine by 1st method.</p>

username: ptrblck
cooked: <p>The <code>tuple</code> class is a plain Python class and thus doesn’t have the <code>cuda()</code> method.<br>
You would have to unwrap the tensors and call <code>cuda()</code> on each of them.</p>
What is the Pytorch Equivalent of Keras 'K' and 'lambda?

username: Anubhav1107
cooked: <p>I want to know is there any function for<br>
<strong>K.ctc_batch_cost()</strong> for Pytorch.<br>
and also for<br>
<strong>ctc_loss = Lambda(ctc_lambda_func, output_shape=(1,), name=‘ctc’)([y_pred, labels, input_length, label_length])</strong><br>
In this code, ctc_lambda_func is  a function I earlier defined. And y_pred,… are parameteres I have defined. I want the equivalent of lambda.<br>
Thanks !!</p>

username: ptrblck
cooked: <p>I guess <code>nn.CTCLoss</code> might be the equivalent to <code>ctc_batch_loss</code>.<br>
What does the <code>Lambda</code> function do? It seems you are passing a <code>ctc_lambda_func</code> to it and execute it with input data?</p>
<p>Note that you don’t have to define the computation graph beforehand in PyTorch and can just execute the code similar to numpy.<br>
E.g. you can define modules and pass the tensors directly to them. Have a look at the <a href="https://pytorch.org/tutorials/">tutorial section</a> for an introduction. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Anubhav1107
cooked: <p>Great, thanks for explaining !! It does explain itself. I am a beginner,  do you think I should also learn keras for future or only the docs are fine</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Anubhav1107" data-post="3" data-topic="96802">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anubhav1107/48/28776_2.png" class="avatar"> Anubhav1107:</div>
<blockquote>
<p>I am a beginner, do you think I should also learn keras for future or only the docs are fine</p>
</blockquote>
</aside>
<p>I’m not sure, if I misunderstand the question, but if you want to learn PyTorch, the docs and tutorials should be fine. There is no need to learn Keras in order to work with PyTorch. On the other hand, if you would like to learn Keras, I would recommend to look into their tutorial section. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
How to reuse buffers passed through a Queue

username: Axel-Jacobsen
cooked: <p>Hello, I hope you are doing well!</p>
<p>Following the <a href="https://pytorch.org/docs/master/notes/multiprocessing.html#reuse-buffers-passed-through-a-queue" rel="noopener nofollow ugc">multiprocessing Best Practices page</a>, a best practice is stated to</p>
<blockquote>
<p>…make it send the buffers back</p>
</blockquote>
<p>There have been other forum posts about clarifying this point (see <a href="https://discuss.pytorch.org/t/examples-for-asynchronous-rl-impala-ape-x-with-actors-sending-observations-not-gradients-to-a-learners-replay-buffer/38306">this one</a> or <a href="https://discuss.pytorch.org/t/torch-multiprocessing-how-to-reuse-buffers-passed-through-a-queue/16778/10">this one</a>) but those did not get any conclusive answers to the question.</p>
<p>User phizaz suggested this near the bottom of the second post:</p>
<pre><code class="lang-auto">def run(Qr, Qs):
    while True:
        b = Qr.get()
        torch.randn(..., out=b) # reusing the buffer tensor without re-allocating
        Qs.put(b)

def main():
    ...
    for i in range(buffer_size): # allocate buffers
        b = torch.empty(...)
        Qs.put(b)
    ... start processes ...
    while True:
        b = Qr.get() # main's Qr = run's Qs
        ... use b ...
        Qs.put(b) # main's Qs = run's Qr
</code></pre>
<p>Though he mentioned</p>
<blockquote>
<p>Caveat: I see the cost of an additional queue to be huge. It might not be worth the benefit of not reallocating additional shared memory.</p>
</blockquote>
<p>Which I think he is probably correct about. So what exactly was the best practices page alluding to?</p>
<p>Very interested in a response. Thank you!</p>
When the number of output channels is not a multiple of the group in Conv2d

username: Igo312
cooked: <p>As the <code>group</code> in <code>torch.nn.Conv2d</code> said it will split channel into groups, as the example from <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html#torch.nn.Conv2d" rel="noopener nofollow ugc">Conv2d</a></p>
<blockquote>
<p>At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.</p>
</blockquote>
<p>But what will happen if the output channel is not the multiple of the groups? one of the groups will generate more output?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>In the link you have provided, it says that both <code>in_channel</code> and <code>out_channel</code> must be divisible by <code>groups</code>.</p>
<p>Bests</p>

username: Igo312
cooked: <p>sorry, my bad. Thank you for your patience!</p>
Selecting pixel values with a given list of 2xN tensor indices?

username: peepeepoopoo
cooked: <p>I’m trying to extrapolate the pixel value in a WxH img tensor with a given 2xN indices. When I tried to do the old fashioned way, by splitting apart the 2xN indices as a list of x and y values and use img[idx_x,idx_y] to extract the pixel values, I get an error saying that only datatypes of long, byte, or bool are allowed (which I can’t switch to since the locations themselves will be part of backpropagation).  Is there any other way to extract pixel value from a 2D matrix with a given coordinate array beside this? Thanks!</p>
<p>PS:(Actually, it’s a multichannel 2D array (so actually the tensor is WxHxC) with calculated 2D indices)</p>

username: hash-ir
cooked: <p>Have you tried typecasting the indices: <code>img[idx_x.long(), idx_y.long()]</code>?</p>

username: peepeepoopoo
cooked: <p>Oh cool that works! And that doesn’t seem to prevent the autograd function from going through as well! Thanks!</p>
Train two models in Pytorch

username: gnadaf
cooked: <p>Hello,<br>
I am new to PyTorch,</p>
<p>I have created single model training with help of <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>it is working well.</p>
<p>Next, I would like to train() two model parallel, how to create two models and train(), do you have any code reference or resources</p>
<p>Can we use thread or is there any separated method?</p>
Calculate closest tensor in list to another tensor

username: Ethan_P
cooked: <p>Hey I’m kinda a newbie…<br>
How can I calculate closest tensor in list to another tensor?</p>
<p>Example:</p>
<p>Say I have:</p>
<pre><code class="lang-auto">[tensor([1,1,1]),tensor([5,5,5]),tensor([10,10,10])]
</code></pre>
<p>and:</p>
<pre><code class="lang-auto">tensor([2,2,2])
</code></pre>
<p>If would want:</p>
<pre><code class="lang-auto">tensor([1,1,1])
</code></pre>
<p>How can I achieve this? Thank you</p>

username: ptrblck
cooked: <p>You could calculate the distance between the tensors using <code>torch.norm</code> and use <code>argmin</code> to get the index corresponding to the smallest distance:</p>
<pre><code class="lang-python">a = torch.stack([torch.tensor([1,1,1]), torch.tensor([5,5,5]), torch.tensor([10,10,10])]).float()
b = torch.tensor([2,2,2]).float()
min_idx = torch.norm(a - b.unsqueeze(0), dim=1).argmin()
print(a[min_idx])
&gt; tensor([1., 1., 1.])
</code></pre>

username: Ethan_P
cooked: <p>Thank you! I compared this with my old solution, and this is like 10X faster!</p>
Meaning of torch::autograd:: prefix in autograd profiller name

username: Colinou
cooked: <p>Hi,</p>
<p>I have a question about autograd profiler return. I’m trying to link each row of the log to the operations in my python code, and I wonder what is the meaning of the operations tagged with torch::autograd:: ?</p>
<p>Example:</p>
<pre><code class="lang-auto">MseLossBackward
SqueezeBackward0
torch::autograd::CopyBackwards
ViewBackward
AddBackward0
UnsafeViewBackward
BmmBackward0
ExpandBackward
CloneBackward
InverseBackward
MmBackward
TransposeBackward0
SubBackward0
UnsqueezeBackward0
SelectBackward
SliceBackward
MulBackward0
ReluBackward0
NativeBatchNormBackward
SqueezeBackward1
</code></pre>
<p>Thanks !</p>
How average all predictions when using kfold cross validation

username: Ali_Shahbaaz_Khan
cooked: <pre><code class="lang-auto">total_preds = []
for model in models :
    dataloader = DataLoader(test_data,batch_size=1,shuffle=False,num_workers=2)
    preds = []
    temp = []
    i = 0
    for batch in dataloader :
        inputs = batch['input_features'].float()
        with torch.no_grad() :
            p = model(inputs)
            preds.append(torch.squeeze(p))
    
    total_preds.append(preds)
</code></pre>
<p>im using kfold cross validation so i have multiple models and i need to average out the predictions of every model.<br>
In the code above preds if of dimension (730,3) and total_preds is the list of all predictions in every fold. I want to retain the dimensions after averaging.<br>
How do i go about it without using any extra loop?</p>

username: ptrblck
cooked: <p>You could calculate the <code>mean</code> in the “kfold” dimension as given in this code snippet:</p>
<pre><code class="lang-python">total_preds = []
for _ in range(4): # kfold loop
    preds = []
    for _ in range(730): # data loop
        p = torch.randn(3)
        preds.append(p)
    preds = torch.stack(preds)
    total_preds.append(preds)

total_preds = torch.stack(total_preds)
print(total_preds.shape) # [4, 730, 3]
mean_preds = total_preds.mean(0)
print(mean_preds.shape) # [730, 3]
</code></pre>

username: Ali_Shahbaaz_Khan
cooked: <p>That worked!!<br>
Thanks</p>
What kind of network is more suitable to distinguish the fuzzy degree of pictures?

username: ma3252788
cooked: <p>For example, the same image can be divided into different categories according to different fuzzy degrees. What network can be used to effectively classify this data set?</p>

username: ma3252788
cooked: <p>Using ordinary ResNext and resnet is not effective</p>

username: ma3252788
cooked: <p>Anyone can help me?</p>
<p>Or do you have any ideas to tell me</p>
Syntheses two faces through Style-GAN

username: algeriapy
cooked: <p>I am wondering if I can use Style-GAN to synthesis two faces as shown in style-GAN official website  <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3983d03eea58969ed367f78f584c7b671010536.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f3983d03eea58969ed367f78f584c7b671010536" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3983d03eea58969ed367f78f584c7b671010536_2_690x414.jpeg" alt="image" data-base62-sha1="yKWc3vuGCpGiftUNzAtpbfjUxRc" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3983d03eea58969ed367f78f584c7b671010536_2_690x414.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3983d03eea58969ed367f78f584c7b671010536_2_1035x621.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3983d03eea58969ed367f78f584c7b671010536.jpeg 2x" data-dominant-color="7B655A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1280×768 457 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any implementation for that in Pytorch</p>
Uninstalling PyTorch got stuck

username: Mona_Jalal
cooked: <p>I am trying to uninstall PyTorch but it is stuck. What can I do to fix it?</p>
<pre><code class="lang-auto">$ conda uninstall pytorch
Collecting package metadata (repodata.json): done
Solving environment: \ 
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistenc| 

  - conda-forge/linux-64::terminado==0.8.3=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::distributed==1.20.2=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::ipykernel==4.8.0=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::anaconda-project==0.8.2=py36h44fb852_0
  - https://repo.continuum.io/pkgs/main/linux-64/jupyter_console-5.2.0-py36he59e554_1.tar.bz2/linux-64::jupyter_console==5.2.0=py36he59e554_1
  - conda-forge/linux-64::notebook==5.7.8=py36_1
  - https://repo.continuum.io/pkgs/main/linux-64/qtconsole-4.3.1-py36h8f73b5b_0.tar.bz2/linux-64::qtconsole==4.3.1=py36h8f73b5b_0
  - defaults/linux-64::anaconda-navigator==1.8.4=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::jupyterlab_launcher==0.10.2=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::spyder==3.2.6=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::widgetsnbextension==3.1.0=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::ipywidgets==7.1.0=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::jupyterlab==0.31.1=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/_ipyw_jlab_nb_ext_conf-0.1.0-py36he11e457_0.tar.bz2/linux-64::_ipyw_jlab_nb_ext_conf==0.1.0=py36he11e457_0
  - conda-forge/noarch::jupyter==1.0.0=py_2
  - conda-forge/linux-64::matplotlib==3.1.1=py36_0
  - conda-forge/linux-64::matplotlib-base==3.1.1=py36hfd891ef_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::bokeh==0.12.13=py36h2f9c1c0_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::dask==0.16.1=py36_0
  - https://repo.continuum.io/pkgs/main/linux-64/odo-0.5.1-py36h90ed295_0.tar.bz2/linux-64::odo==0.5.1=py36h90ed295_0
  - https://repo.continuum.io/pkgs/main/linux-64/blaze-0.11.3-py36h4e06776_0.tar.bz2/linux-64::blaze==0.11.3=py36h4e06776_0
  - https://repo.continuum.io/pkgs/main/linux-64/linux-64::seaborn==0.8.1=py36hfad7ec4_0
</code></pre>

username: ptrblck
cooked: <p>It seems your current environment might have some conflicts and conda might try to “solve” these:</p>
<pre><code class="lang-auto">The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistenc|
</code></pre>
<p><code>conda</code> isn’t the fastest environment manager and if I run into these kind of issues, they were never solved after waiting for a long time.<br>
If it’s possible, I would just create a new environment and install your packages there.</p>
Imbalance in the dataset for mutli class classification

username: Ashutosh_Mishra
cooked: <p>I am trying to replicate the results of VPGNet ICCV 2017 in which for vanishing point prediction task, the target has 5 channels based on the location of vanishing point. If the VP exists in the image, the first 4 channels get 1 in the respective quadrant locations while the 5th channel is 0. If there doesnt exist a VP, the 5th channel is initalized to be 1. Hence in an image which has VP in it, the output of the model will be a 5D vector and target will also be a 5D vector(kind of one hot encoded). Now, the thing is how should I divide the dataset into train and val as there are total 19,302 images with easy VP, 262 with hard VP and 1,533 with no VP</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> any suggestions from your end!</p>
Low version Pytorch with high version cuda

username: SteinwayCaiZhuo
cooked: <p>If I use a low version PyTorch with a high version CUDA, will I meet any troubles?</p>

username: ptrblck
cooked: <p>Yes, you could run into issues, if you are trying to compile legacy PyTorch versions with more recent CUDA versions, since CUDA methods might have been removed, changed etc.<br>
What’s the reason you would need an older PyTorch version?</p>

username: SteinwayCaiZhuo
cooked: <p>Why I need an older PyTorch version? Reimplementing deep learning methods proposed a few years ago and the codes are not maintained.</p>
Working with too large dataset

username: nzinc
cooked: <p>I have data set in folder <code>data.tar.gz</code>. That archive contains two folder with female face images and male face images. Uncompressed size of this data set is 150 gb (archive size 1.9 gb). I work on Google Colab with GDrive, so i couldn’t allocate this huge data set. There is a binary classification task for training. The question is. Is it possible to somehow use this archive in pytorch, despite the fact that there are two folders with data inside archive and how to put a label for this data (for each folder its own label)?</p>

username: ptrblck
cooked: <p>Based on your description you could most likely use <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#imagefolder"><code>torchvision.datasets.ImageFolder</code></a> to lazily load the data.</p>
Round Tensor to x decimal places

username: eplu
cooked: <p>Hi,</p>
<p>I’m looking for a function to round the values in a tensor to a specific number of decimal places, i.e. some kind of functionality similar to round(…) in base python. Does something like this exist in PyTorch now? What is the best way to do this without losing all gradient history?</p>

username: bananacode
cooked: <p>The following should round to 3 digits:</p>
<pre><code class="lang-auto">n_digits = 3
arr = torch.rand((3,3))
rounded = torch.round(arr * 10^n_digits) / (10^n_digits)
</code></pre>

username: PySimon
cooked: <p>Works, but you have to replace ^ with **:</p>
<pre><code class="lang-auto">n_digits = 3
arr = torch.rand((3,3))
rounded = torch.round(arr * 10**n_digits) / (10**n_digits)
</code></pre>

username: remykarem
cooked: <p><code>torch.Tensor</code> objects have a <code>round()</code> method that does not take any argument. So you could do this:</p>
<pre><code class="lang-auto">n_digits = 3
arr = torch.rand((3,3))
rounded = (arr * 10**n_digits).round() / (10**n_digits)
</code></pre>

username: Lucy_Jackson
cooked: <p>Hi,</p>
<p>As I understand this rounding function is undifferentiable, is this correct? and if so is there a way to use the round function and still allow for backprop?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>No, the rounding function is a step function and almost everywhere differentiable as explained in <a href="https://discuss.pytorch.org/t/torch-round-gradient/28628/4">this post</a>.<br>
However, since it’s a step function the gradient will be zero almost everywhere, so it’s not really usable in training the model.</p>
While running multiple PyTorch scripts Using DataParallel, only the first one is distributed among two gpus, rest are loading 1st GPU memory

username: Sasank_Kottapalli
cooked: <p>while running multiple PyTorch scripts Using DataParallel, only the first one is distributed among two GPUs, rest are loading 1st GPU memory.</p>
<p>Cant all the program distribute themselves like 1st one?<br>
If they can what’s the process?</p>
<p>I am just loading the model and using</p>
<pre><code class="lang-auto">model = ConvTasNet(args.N, args.L, args.B, args.Sk, args.H, args.P, args.X, args.R,
                       args.C, norm_type=args.norm_type, causal=args.causal,
                       mask_nonlinear=args.mask_nonlinear)
print(model)
if args.use_cuda:
    model = torch.nn.DataParallel(model)
    model.cuda()
</code></pre>
<p>Same procedure for all scripts.</p>

username: Sasank_Kottapalli
cooked: <p>Can someone please look into this?</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue and am able to execute the <code>nn.DataParallel</code> script multiple times:</p>
<pre><code class="lang-python"># tmp.py
import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Conv2d(6, 12, 3, 1, 1),
    nn.ReLU())

model = nn.DataParallel(model).cuda()
x = torch.randn(8, 3, 224, 224).cuda()

for _ in range(100):
    out = model(x)
print('done')

# run.sh
python tmp.py

# in terminal
bash run.sh &amp;
bash run.sh &amp;
bash run.sh &amp;
bash run.sh &amp;
bash run.sh &amp;
</code></pre>
<p>Using <code>nvidia-smi</code> I can see that each device will be used by 5 scripts.</p>
I want to change an array of size (1,4,4,4,4) to (1,16,16)

username: skerlet_flandorle
cooked: <p>I would like to change an array of size (1,4,4,4,4) to (1,16,16).</p>
<p>error contents<br>
view size is not compatible with input tensor’s size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape (…) instead.</p>
<p>Until now, this error only appeared when the size was specified incorrectly in the view</p>
<p>I get a mysterious error (1, 4 * 4, 4 * 4) Then why is it impossible?<br>
4 * 4 = 16 ??? Why do you use reshape (…)?</p>
<p>I forgot to put the cause code<br>
inp.size = (1,16,16)<br>
s = inp.view (1,4,4,4,4) .permute (0,1,3,2,4) .view (1,16,16)</p>

username: ptrblck
cooked: <p>The <code>view</code> operation just changes the tensor metadata without triggering a copy.<br>
After the <code>permute</code> call this won’t be possible anymore as explained in the error message and you would either have to use <code>reshape</code> (which copies the data and performs the <code>view</code> op) or call <code>.contiguous()</code> before the <code>view</code> operation on the tensor.</p>
How many hidden layers for a3c with 8 inputs ranging from -2000 to 2000 and 3 discrete actions

username: granth_jain
cooked: <p>Hi,</p>
<p>I am trying to build an A3C with 8 inputs which are normalized and then discretized as int b/w -2000 to 2000.I need to have the past info as well therefore I am using LSTM after first layer.</p>
<p>However I did not have any major success in training the model even after 1Lakh episodes.<br>
May I have some help in what should be the ideal number of hidden layer and size of hidden layer for such a problem.</p>
<p>class ActorCritic(torch.nn.Module):</p>
<pre><code>def __init__(self, params):
    super(ActorCritic, self).__init__()

    self.num_inputs = params.num_inputs
    self.action_space = params.action_dim
    self.hidden_size = params.hidden_size
    num_inputs = params.num_inputs
    self.lstm = nn.LSTM(num_inputs, 8,num_layers = params.lstm_layers)
    self.fc1 = nn.Linear(8, 8)
    self.fc1.apply(init_weights)
    self.fc2 = nn.Linear(8, 8)
    self.fc2.apply(init_weights)
    # self.fc3 = nn.Linear(8, 8)
    # self.fc3.apply(init_weights)
    # self.fc4 = nn.Linear(8, 8)
    # self.fc4.apply(init_weights)
    self.critic_linear = nn.Linear(8, 1)
    self.critic_linear.apply(init_weights)
    self.actor_linear = nn.Linear(8, self.action_space)
    self.actor_linear.apply(init_weights)
    self.train()

def forward(self, inputs):
    inputs, (hx, cx) = inputs
    inputs = inputs.reshape(1,1,-1)
    output, (hx, cx) = self.lstm(inputs, (hx, cx))
    x = torch.tanh(self.fc1(output))
    x = torch.tanh(self.fc2(x))
    # x = torch.tanh(self.fc3(x))
    # x = torch.tanh(self.fc3(x))
    return self.critic_linear(x), self.actor_linear(x), (hx, cx)

def save(self, filename, directory):
    torch.save(self.state_dict(), '%s/%s_actor.pth' % (directory, filename))

def load(self, filename, directory):
        self.load_state_dict(torch.load('%s/%s_actor.pth' % (directory, filename)))
</code></pre>
<p>I have around 2 million rows of data with me. and 1 episode takes around 10 rows on average. The input is raw data therefore no convolutions needed.</p>
<p>Any expert advice on this would be of great help.</p>
<p>Thanks,<br>
Granth</p>
Training while inferencing in Parallel on GPU

username: ZeroMax
cooked: <p>Hi, I have an RL agent which I’d like to both train and do inference at the same time using the gpu. Hiowever using cuda with torch.multiprocessing doesn’t seem to work well, and gives this error upon initialization of the other process:</p>
<p><code>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method</code></p>
<p>However using the ‘spawn’ start method gives me another issue, because this method tries to serialize everything using pickle, and this gives me another mysterious error upon zero_grad(). I have detached everything in my replay buffer…</p>
<pre><code class="lang-auto">  File "env.py", line 60, in &lt;module&gt;
    main()
  File "env.py", line 55, in main
    process_two.start()
  File "/home/`/anaconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/zero/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 136, in reduce_tensor
    raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "
RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).
</code></pre>
<p>So, my question is, is there any way to do this asynchronous training/inference setup in parallel on a GPU? The default start method of torch.multiprocessing seems to work well on my multi-core CPU. but I’d like to move to using my GPU for optimized matmul operations…</p>
Basic: why default method of nn.Linear is forward?

username: jxj362
cooked: <p>Pretty new to python and pytorch. Super basic question…</p>
<p>Per code below</p>
<pre><code class="lang-auto">class Perceptron(nn.Module):
  def __init__(self,input_dim):
    super(Perceptron,self).__init__()
    self.fc1 = nn.Linear(input_dim,1)

  def forward(self, x_in):
    return torch.sigmoid(self.fc1(x_in)).squeeze()

p1 = Perceptron(3)
x = torch.randn((5,3))
torch.equal(p1.fc1.forward(x),p1.fc1(x))
</code></pre>
<p>result is “True”.</p>
<p>Looked at the source code and couldn’t figure out why</p>
<ol>
<li>I can run <code>p1.fc1(x)</code> directly</li>
<li>why it defaults to run forward()</li>
</ol>
<p>Thank you!!!</p>

username: BramVanroy
cooked: <p>Among others hook, <strong>call</strong> also calls forward. When you can directly call an object, you are triggering its magic method <strong>call</strong>…</p>

username: jxj362
cooked: <p>Hi Bram,</p>
<p>Thanks for your reply! Could you elaborate a bit? I don’t see any hooks in the source code -&gt; <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html#Linear</a></p>

username: mariosasko
cooked: <p>When defining a model/layer (or even a loss function, activation) in PyTorch (<code>Perceptron</code> in your example), these models/layers should always subclass <code>nn.Module</code> and the <code>forward</code> method is sort of a special method in that case. Basically, it represents the forward pass of the input data. The reason why you should always do <code>model(some_input)</code> and not <code>model.forward(some_that)</code> is that the first one triggers registered hooks and the second one doesn’t. There are 3 types of hooks: pre-forward hooks, forward hooks and backward hooks. E.g. you can assign a backward hook to the layers to print their gradients during the <code>backward</code> call. You can read more about the hooks (how to register them, …) in the docs. Even if you don’t use them in the code, calling <code>forward</code> implicitly via <code>__call__</code> is the preffered way to do it.</p>
<p>Btw, this is the right <a href="https://github.com/pytorch/pytorch/blob/1c15452703be4e918a2989316f9a1e2fec7a825d/torch/nn/modules/module.py#L751" rel="nofollow noopener">source code</a> you should check.</p>

username: jxj362
cooked: <p>Ahh… now i see. Thank you!</p>
Loss is not decreasing while training LSTM Encoder-Decoder

username: Salah_Belila
cooked: <p>Hi,<br>
I’m new to pytorch and i’m training an encoder-decoder LSTM for abstractive text summarization. The problem is that the loss is almost stable during training.</p>
<p>my configurations are:</p>
<p>VOCAB_SIZE = 50000<br>
BATCH_SIZE = 16<br>
EMBED_DIM = 200<br>
HIDDEN_DIM = 768<br>
DECODER_NUM_LAYERS = 2<br>
MAX_ENCODER_LEN = 350<br>
MAX_DECODER_LEN = 80</p>
<p>loss calculation is as follows:</p>
<pre><code class="lang-auto">loss = 0

'''
the network output is y_hat of shape [batch_size, seq_len, vocab_size]
the target is y of shape [batch_size, target_seq_len]
please note that target_seq_len is equal to seq_len (used right padding)
i'm not sure whether this method is correct or not please correct
me if i'm wrong.
'''

for j in range(y.size(1)):
               
    y_hat_step = y_hat[:, j]
    y_step = y[:, j]
    loss += loss_function(y_hat_step, y_step)
                
    del(y_hat_step)
    del(y_step)

loss.backward()
optimizer.step()
            
</code></pre>
<p>loss_function and optimizer</p>
<pre><code class="lang-auto">loss_function = CrossEntropyLoss(ignore_index=0, reduction='mean')
optimizer = Adam(model.parameters(), lr=0.01)

#also tried lr=0.001 and lr=0.0001
</code></pre>
<p>my model:</p>
<pre><code class="lang-auto">EncoderDecoder(
  (encoder): Encoder(
    (embedding_layer): Embedding(50000, 200, padding_idx=0)
    (dropout): Dropout(p=0.3, inplace=False)
    (lstm_layer): LSTM(200, 768, batch_first=True, dropout=0.3, bidirectional=True)
  )
  (decoder): Decoder(
    (embedding): Embedding(50000, 200, padding_idx=0)
    (dropout): Dropout(p=0.3, inplace=False)
    (lstm): LSTM(200, 768, num_layers=2, batch_first=True, dropout=0.3)
    (output): Linear(in_features=768, out_features=50000, bias=True)
  )
)
</code></pre>
How to penalize cosine similarity using torch.nn.CosineSimilarity?

username: ProGamerGov
cooked: <p>The code below penalizes the cosine similarity between different tensors in batch, but PyTorch has a dedicated CosineSimilarity class that I think might make this code less complex and more efficient. Would it possible to do the same with <code>torch.nn.CosineSimilarity</code>, and if so how?</p>
<pre><code class="lang-auto">batch = input.size(0)
flattened = input.view(batch, input.size(1), -1)
grams = torch.matmul(flattened, torch.transpose(flattened, 1, 2))
grams = F.normalize(grams, p=2, dim=(1, 2), eps=1e-10)
loss = -sum([ sum([ (grams[i]*grams[j]).sum()
           for j in range(batch) if j != i])
           for i in range(batch)]) / batch
</code></pre>
<p>I’ve tried variations of stuff like this, but I can’t get the same result as the above code:</p>
<pre><code class="lang-auto">    list2 = []
    for i in range(input.size(0)):
        list1 = []
        for j in range(input.size(0)):
            similarity = sum(torch.cosine_similarity(input[i].view(1,-1), input[j].view(1,-1)))
            list1.append(similarity)
        list2.append(sum(list1))
    loss = -sum(list2)
</code></pre>

username: ProGamerGov
cooked: <p>I think I’ve got working with this:</p>
<pre><code class="lang-auto">flattened = input.view(input.size(0), input.size(1), -1)
grams = torch.matmul(flattened, torch.transpose(flattened, 1, 2))
list2 = []
for i in range(input.size(0)):
    list1 = []
    for j in range(input.size(0)):
        if j!=i:
            similarity = sum(torch.cosine_similarity(grams[i].view(1,-1), grams[j].view(1,-1)))
            list1.append(similarity)
    list2.append(sum(list1))
loss = -sum(list2) / input.size(0)
</code></pre>
<p>Or the last bit rewritten in list comprehension:</p>
<pre><code class="lang-auto">loss = -sum([ sum([(torch.cosine_similarity(grams[j].view(1,-1), grams[i].view(1,-1))).sum() for i in range(input.size(0)) if i != j]) for j in range(input.size(0))]) / input.size(0)
</code></pre>
How much work should your dataset class do for you?

username: Wesley_Neill
cooked: <p>I’m trying to determine if I should refactor the way may dataset <code>__getitem__</code> method pre-processes.</p>
<p>Are there any heuristics beyond the obvious time complexity and code readability I should consider?</p>
<p>For a given audio file that I load I want to:</p>
<ol>
<li>Pad the length to match a given interval size.</li>
<li>Cut the audio into equal pre-determined interval lengths</li>
<li>Create several types of spectrograms of each piece of this new sequence.</li>
</ol>
<p>It’s a lot. Is there any reason that I shouldn’t do all of these things right inside of <code>__getitem__</code>?</p>

username: hash-ir
cooked: <p>If any of the tasks are independent of the audio file, you can perform them in the <code>__init__</code> method. Else, instead of doing everything in the <code>__getitem__</code> method, you can write custom transforms and use them in the order you want. I am not really sure if this is more efficient than doing everything in the <code>__getitem__</code> method but you can try and check if it is.</p>

username: JuanFMontesinos
cooked: <p><code>__getitem__</code> is the process which is spawn with multiprocess.<br>
The workload should be there (mainly the spectrograms)<br>
As <a class="mention" href="/u/hash-ir">@hash-ir</a> mentions, any other task can be performed in the <code>__init__</code> function (or even outside the dataset class)</p>
<p>It’s all about how many RAM do you have.<br>
Do you need to read audio in <code>__getitem__</code>?<br>
Well, if your dataset fits in your RAM you can preload it in init.<br>
You can use any python function inside <code>__getitem__</code>, therefore it can still be readable.</p>
How to determine anchor sizes for RetinaNet model?

username: ayadav01
cooked: <p>I posted it on stack overflow but got no replies so i am posting it here. I am trying to implement a RetinaNet model in pytorch for my custom dataset, however, i am little confused on how some of the hyper-parameters are chosen. For example, the model uses anchor size of  <code>[32, 64, 128, 256, 512]</code> , anchor ratio of  <code>[0.5, 1, 2]</code>  and anchor scale of  <code>[1, 1.25, 1.58]</code> , but how do we determine these numbers for our own datasets. What are some of the ways we can go about when selecting param for the anchor box generation part ?</p>
Artificially alter batch_size for training a CNN on a sequence

username: Wesley_Neill
cooked: <p>Hi all, I’m looking for a little help sorting out the logistics of an idea I had.</p>
<p><strong>Back story:</strong><br>
I was initially training a CNN to recognize the number of speakers in an audio file that was converted to spectrograms. The way I accomplished this task on varying size spectrograms was to just resize the images.</p>
<p>However, I found this approach to be naive and anticipate problems with it when the audio files vary greatly in length.</p>
<p><strong>New Idea:</strong><br>
My thought now is to take a single audio file, and chunk it into smaller equal sized pieces. Then feed this sequence to my CNN for feature embedding, and pass the feature sequence to an RNN.</p>
<p><strong>Logistics problem:</strong><br>
My idea for a solution was to create a Dataset class that creates the sequence of smaller audio clips from and converts each chunk to a spectrogram image. <strong>Then</strong> I want to somehow trick the CNN into thinking it’s getting a batch of spectrograms.</p>
<p>This seems fairly easy to accomplish… unless I want to use a <code>DataLoader</code> (I do). Then the dimensions of the data coming out of the <code>DataLoader</code> will be wrong for input into the CNN. This is due to the <code>batch_size</code> argument.</p>
<p>Does anybody have any clever ideas that would allow me to use a DataLoader so I can keep the multi-processing capabilities?</p>
Mean Directional Accuracy criterion loss function

username: lorrp1
cooked: <p>Hello, can someone help me create a mean directional accuracy loss function for pytorch?<br>
in numpy it should be this: <a href="https://gist.github.com/bshishov/5dc237f59f019b26145648e2124ca1c9" rel="nofollow noopener">https://gist.github.com/bshishov/5dc237f59f019b26145648e2124ca1c9</a><br>
def mda(actual: np.ndarray, predicted: np.ndarray):<br>
“”" Mean Directional Accuracy “”"<br>
return np.mean((np.sign(actual[1:] - actual[:-1]) == np.sign(predicted[1:] - predicted[:-1])).astype(int))</p>
<p>but i get errors if use it as loss criterion in pytorch</p>

username: tom
cooked: <p>You cannot have sign in the loss function and expect good results and int won’t harmonize with autograd either.</p>

username: lorrp1
cooked: <p>the result of the function there is the mean  as float (not an int)</p>
<blockquote>
<p>You cannot have sign in the loss function and expect good results</p>
</blockquote>
<p>i think i understand the reason but could that be “added” to the result of a normal mse to give importance also to the direction?</p>

username: tom
cooked: <p>Probably multiplying would work better.</p>
<p>And I might add, it could be a good idea to take, say 3-5 elements of your prediction and target vector and visualize the loss you’re getting.</p>
Why does stack stach they way it does?

username: Carsten_Ditzel
cooked: <p>Given a 4-dim tensor X</p>
<p><code>X = torch.tensor([[[[0, 2]], [[1, 3]]]])</code></p>
<p>of shape</p>
<p><code> torch.Size([1, 2, 1, 2])</code></p>
<p>and stack, which stacks along a new dimension (here the last)<br>
<code>Y = torch.stack((X[:, :1], X[:, 1:]), dim= -1)</code></p>
<p>results in a shape</p>
<p><code>Y.shape torch.Size([1, 1, 1, 2, 2])</code></p>
<p>but why are the elements arranged in this way?</p>
<pre><code class="lang-auto">Y tensor([[[[[0, 1],
           [2, 3]]]]])
</code></pre>
<p>I am thinking about explanations as to why this specific order results, but cant seem to find one</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think you should remove all the dimensions of size 1 to make this clearer.<br>
But you basically take each column of the Tensor and then stack them along the last dimension, making them rows.</p>

username: Carsten_Ditzel
cooked: <p>thank you, yes that is what the output says, but I am having trouble understanding why they become rows, especially if I view the initial layout as having two channel, then why would those be restructured in such a way for elements on identical positions across channels to become elements in the same rows?</p>

username: albanD
cooked: <p>This is because usually, the concept of row/column correspond to the last two dimensions of a Tensor.<br>
And when you add a new dimension, what used to be rows become columns. And if you stack stuff on that new dimension, they are columns.</p>

username: Carsten_Ditzel
cooked: <p>thank you, so the logic is basically working your way up from the right hand side always</p>
SyncBatchNorm hangs in this special case

username: soulslicer
cooked: <p>I have a special model where, during the first iterate of a video, one of the components doesnt get used for example:</p>
<pre><code class="lang-auto">if self.iter != 0:
    stuff = self.modelA(input)
    stuff2 = self.modelB(stuff)
    return stuff2
else:
    stuff = self.modelA(input)
    return stuff
</code></pre>
<p>One of the GPU’s / process might hit self.iter = 0, while another GPU may still be running stuff with self.iter = N. When this happens, the model hangs on any batch norm.</p>
<p>How can I get around this?</p>
Different gradient updates when using PackedSequence

username: ss42
cooked: <p>Given a batch of sequences:<br>
[[1,2,3,4],<br>
[1,2]]</p>
<p>I am outputing my hidden layers at each step for a custom lstm in two ways:</p>
<ol>
<li>
</li></ol>
<p>[ 2 x hidden tensor,<br>
2 x hidden tensor,<br>
1 x hidden tensor,<br>
1 x hidden tensor]</p>
<ol start="2">
<li>
</li></ol>
<p>6 x hidden packed sequence</p>
<p>I set the seed and all calculations are the same in the first batch, but in the second batch after calling .backward() on the first batch loss the hidden layers have slightly different values between methods .</p>
<p>Have packed sequences caused issues for anybody when they update the gradients? Anything I should know when using them?</p>