akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom Embedding with Sparse Gradient

username: Thomas_Ahle
cooked: <p>I’m trying to create my own (slightly different) implementation of <code>nn.Embedding</code>.<br>
The simplest form of the code is just</p>
<pre><code class="lang-auto">def forward(self, indices):
    return self.table[indices]
</code></pre>
<p>Which converts a batch of indices to a batch of vectors.</p>
<p>Now I would like the gradient to be sparse, like with <code>nn.Embedding</code>, so I can optimize with SparseAdam.</p>
<p>Is there a way to do this with autograd?<br>
Or do I have to implement my own <code>backward</code> method?</p>
Did I initialize my LSTM correct or do my states reset all the time?

username: Patrick1
cooked: <p>Currently looking to reset my model for every fold in a Cross Validation when I realized I might initialized my model all wrong. I am very confused right now because it looks like every forward pass in my model resets my hidden state and cell state to zeros. I think I added the init_hidden function when I went from only training with batch size =1 to different batch sizes.</p>
<p>Currently rethinking what I did the last week.</p>
<pre><code class="lang-auto">class Model_GRU_1(nn.Module):

    def __init__(self, n_features, n_classes, n_hidden, n_layers,dropout):            
        super().__init__()
        self.gru = nn.GRU(
            input_size=n_features,
            hidden_size=n_hidden,
            num_layers=n_layers,
            batch_first=True,
            dropout=dropout,
            bidirectional=True
        )

        
        self.dense = nn.Linear(n_hidden, n_hidden)
        self.relu = nn.ReLU()
        weight = torch.zeros(n_layers,n_hidden)
        nn.init.kaiming_uniform_(weight)
        self.weight = nn.Parameter(weight)

        self.classifier = nn.Linear(n_hidden, n_classes)


    def init_hidden(self):       
        hidden_state = torch.zeros(self.gru.num_layers,batch_size,self.gru.hidden_size)
        cell_state = torch.zeros(self.gru.num_layers,batch_size,self.gru.hidden_size)
        return (hidden_state, cell_state)

    def forward(self, x):
        self.hidden = self.init_hidden()
        _, (hidden) = self.gru(x)                  
        out=hidden[-1]
        out2 = self.dense(out)
        out3 = self.relu(out2)                                 
        return self.classifier(out3)
</code></pre>
3: RuntimeError Element 0 of tensors does not require grad and does not have a grad_fn

username: Bar_Lerer
cooked: <p>Hi all, I have been doing some reading on this issue online and tried the suggested solutions but with no success,</p>
<p>For some context, I am using a pre-trained model, and I want to use my modified classifier.</p>
<p>This is how I create the model:</p>
<pre><code class="lang-auto">model = torchvision.models.vgg11(pretrained=True)
flag = 0
# We want this to happen only for 2 children
for child in model.children():
   if flag &lt; 2:
      for parameter in child.parameters():
         parameter.requires_grad = False
         flag += 1
model.classifier.add_module('lastfc', nn.Linear(1000, 1))
model.classifier.add_module('lastSigmoid', nn.Sigmoid())

</code></pre>
<p>I want to freeze features and AvgPool so I won’t have to retrain.</p>
<p>This is my loss and optimizer:</p>
<pre><code class="lang-auto">loss_fn = nn.BCELoss()
optimizer = torch.optim.Adam(model.classifier.parameters(), lr=0.001)
</code></pre>
<p>And this is inside of the train function:</p>
<pre><code class="lang-auto">X, y = X.to(device), y.to(device)
# Compute prediction error

pred = model(X)
pred = (pred &gt; 0.5).float().squeeze()
y = torch.tensor(y, dtype=torch.float32, device=device)
loss = loss_fn(pred, y)

# Backpropagation
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>But, when I call</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>I get the error “Element 0 of tensors does not require grad and does not have a grad_fn”</p>
<p>What am I doing wrong?</p>
<p>As a sanity check, I tried to run in debug session:</p>
<pre><code class="lang-auto">model.classifier[0].weight.requires_grad
True
</code></pre>

username: ptrblck
cooked: <p>This line of code:</p>
<pre><code class="lang-python">pred = (pred &gt; 0.5).float().squeeze()
</code></pre>
<p>will detach <code>pred</code> as the comparison is not differentiable.<br>
Pass the probability directly to <code>nn.BCELoss</code> (or better: remove <code>sigmoid</code> and use <code>nn.BCEWithLogitsLoss</code> for better numerical stability) and it should work.</p>
RandomCrop with Condition

username: Malte
cooked: <p>I want to use RandomCrops, but only allow Crops that fulfill a certain condition. Something like the logic of</p>
<pre><code class="lang-auto">
transform = transforms.Compose([while crop condition not fullfilled: do transforms.RandomCrop(crop_size),
                                transforms.ToTensor(),
                                ])
</code></pre>
<p>Is it possible to implement that somehow?</p>

username: my3bikaht
cooked: <aside class="quote no-group" data-username="Malte" data-post="1" data-topic="143309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ce7236/48.png" class="avatar"> Malte:</div>
<blockquote>
<p><code>transforms.RandomCrop(crop_size)</code></p>
</blockquote>
</aside>
<p>create two different transforms and apply one based on condition?</p>
Trying to understand the use of ReLu in a LSTM Network

username: Patrick1
cooked: <p>I am currently trying to optimize a simple NN with Optuna. Besides the Learning Rate, Batch Size etc. I want to optimize different network architecture as well. So up until now I optimize the number of LSTM layers, aswell as the number of Dense layers. But now I was thinking about activation functions. Bare in mind I am very new to NN… but I am constently reading about ReLu and Leaky ReLu and I know LSTM uses tanh and sigmoid internally. So first I thought maybe the internal tanh gets switched with a ReLu function but I think I got that wrong right?</p>
<p>What I have seen is that the nn.ReLu() gets applied in between Layers, so I would think it would only make sense to apply it in between my Dense Layers?</p>
<p>Sorry for the Noob Question. I am having trouble understanding these things as they are so basic that they are nowhere discussed.</p>

username: my3bikaht
cooked: <p>I like explanation from here:</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/444923/activation-function-between-lstm-layers">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/444923/activation-function-between-lstm-layers" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/248735/lsfischer" target="_blank" rel="noopener nofollow ugc">
    <img alt="lsfischer" src="https://i.stack.imgur.com/4ymUA.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/444923/activation-function-between-lstm-layers" target="_blank" rel="noopener nofollow ugc">Activation function between LSTM layers</a>
</h4>

<div class="tags">
  <strong>neural-networks, lstm, tensorflow, rnn</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/248735/lsfischer" target="_blank" rel="noopener nofollow ugc">
    lsfischer
  </a>
  on <a href="https://stats.stackexchange.com/questions/444923/activation-function-between-lstm-layers" target="_blank" rel="noopener nofollow ugc">04:52PM - 15 Jan 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Patrick1
cooked: <p>Ah thats interesting. So that means I should not use any activation function in my whole model? I am still a bit confused since I have seen so many models use ReLu.</p>

username: my3bikaht
cooked: <p>If you have linear layers beside LSTM blocks, adding some activation function between them usually works.  You can test both options, ofc. In my experience ReLU is really efficient for generalization.</p>
Backward step for non-scalar

username: fabian-sp
cooked: <p>Hi,</p>
<p>I want to compute the gradient of a parameter, but for each batch element and output dimension individually.</p>
<p>So assume my model has output <code>y_pred</code> of shape (b,m), then I want to compute <code>y_pred[i,j].backward()</code> for each <code>i,j</code>. I can do this naively looping over i and j. What is the most performant way to do this?</p>
<p>Cheers, Fabian!</p>
Softmax over 3 Tensor

username: Dina_Ahmed
cooked: <p>In a paper, I read that they had a (7<em>7</em>512) tenor and after applying a softmax function they got (1<em>1</em>512) tensor. I can not understand How to implement this in pytorch using softmax function</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/dina_ahmed">@Dina_Ahmed</a> please provide the paper link and more details</p>
Using the TransformerEncoder with padded sequences

username: Haziq
cooked: <p>I am trying to use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html" rel="noopener nofollow ugc">TransformerEncoder</a> with padded sequences where <code>src</code> is the input sequence of batch 2 and padded length 8.</p>
<pre><code class="lang-auto">encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
src = torch.rand(2, 8, 512) # [batch, padded sequence length, d_model]
</code></pre>
<p>I understand that the <code>src_key_padding_mask</code> variable is used to mark the padded timesteps to be ignored in the input where the <code>ones</code> mark the timesteps that should be ignored in the input.</p>
<pre><code class="lang-auto">src_key_padding_mask = torch.tensor([[0,0,0,0,1,1,1,1],[0,0,0,0,0,0,1,1]])
</code></pre>
<p>My question is, do I also have to specify the <code>mask</code> variable to mask out the vectors that were generated by the masked input? Or is it automatically inferred based on <code>src_key_padding_mask</code>?</p>
<pre><code class="lang-auto">out = encoder_layer(src, src_key_padding_mask=src_key_padding_mask, mask=?)
</code></pre>
Truncated normal

username: f3ba
cooked: <p>How to sample from a truncated normal distribution in PyTorch? I could not find any within <a href="https://pytorch.org/docs/stable/distributions.html" rel="nofollow noopener">https://pytorch.org/docs/stable/distributions.html</a>. Is it not implemented? Any workarounds?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/implementing-truncated-normal-initializer/4778/12">This post</a> might be the easiest approach.<br>
In the topic were several methods discussed, so another one might fit your use case better.</p>

username: anton
cooked: <p>I put together a truncated normal distribution class, implementing torch.distributions.Distribution interface: <a href="https://github.com/toshas/torch_truncnorm" rel="noopener nofollow ugc">https://github.com/toshas/torch_truncnorm</a></p>

username: wallyxie
cooked: <p>I wanted to follow up on this as I am looking <code>rsample</code> from a truncated Gaussian in PyTorch and compute <code>log_prob</code> and wanted to see if there were any updated developments. <a class="mention" href="/u/anton">@anton</a>’s module that he shared is very helpful, but unfortunately, I am looking for a solution that is CUDA-capable. Thank you for your help.</p>

username: anton
cooked: <p>It is CUDA-capable now</p>

username: wallyxie
cooked: <p>Yes, made the post before our email exchange. Thanks again for your help!</p>

username: f3ba
cooked: <p>Is it also differentiable in the parameters?<br>
Why not add this to PyTorch?</p>

username: f3ba
cooked: <p>I am looking for a way to sample the truncated normal that is <em>differentiable</em> in the parameters of the distribution. That post doesn’t cover automatic differentiation, as far as I can see.</p>

username: jdhmr
cooked: <p>Has anyone got this working in a similar fashion to Tensorflow’s <code>tensorflow.distributions.TruncatedMultivariateNormal</code>?</p>
<p>See here <a href="https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MultivariateNormalTriL" class="inline-onebox" rel="noopener nofollow ugc">tfp.distributions.MultivariateNormalTriL  |  TensorFlow Probability</a>.</p>
<p>For my application I specifically need to specify the covariance and location of the mean, so something more detailed than a standard unit normal.</p>
<p>I understand it may be possible to sample from an un-truncated multivariate normal, multiply it by the covariance, then truncate, but I would rather a simple and proper way of doing it. Thanks all!</p>
Patch wise agreement of binary images

username: pietz
cooked: <p>Let’s say I have two very sparse binary images (0 is much more frequent than 1) and I want to calculate the per-pixel precision and recall of the two masks agreeing with each other. Basically asking: Where are the masks equal given that a pixel in A is true and the same given for B is true. We could do that along the lines of:</p>
<pre><code class="lang-auto">p = (a == b &amp; a == 1).mean()
r = (a == b &amp; b == 1).mean()
</code></pre>
<p>Now, I want to do this patch wise using a window. For example, given that pixel at i, j in image A is true. Is any pixel equal in image B given a region (KxK) around i, j? I also want to do this batchwise of course.</p>
<p>Is there any way doing this vectorized in pytorch? Currently, I can only come up with solutions using 3 for loops. I feel like this can somehow be done using maxpooling and a custom convolution kernel.</p>

username: KFrank
cooked: <p>Hi Pietz!</p>
<aside class="quote no-group" data-username="pietz" data-post="1" data-topic="143190" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/da6949/48.png" class="avatar"> pietz:</div>
<blockquote>
<p>I feel like this can somehow be done using maxpooling</p>
</blockquote>
</aside>
<p>Apply your <code>a == b</code> test to a max-pooled version of <code>b</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; b = (torch.randn (1, 1, 10, 10) &gt; 1.7).float()
&gt;&gt;&gt; b
tensor([[[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
          [0., 1., 0., 1., 0., 0., 0., 0., 0., 0.]]]])
&gt;&gt;&gt; torch.nn.MaxPool2d (3, stride = 1, padding = 1) (b)
tensor([[[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 1., 1., 1.],
          [0., 0., 0., 0., 0., 0., 0., 1., 1., 1.],
          [0., 0., 0., 0., 0., 0., 0., 1., 1., 1.],
          [1., 1., 1., 0., 0., 0., 0., 0., 0., 0.],
          [1., 1., 1., 0., 1., 1., 1., 0., 0., 0.],
          [1., 1., 1., 0., 1., 1., 1., 0., 0., 0.],
          [0., 0., 0., 0., 1., 1., 1., 1., 1., 0.],
          [1., 1., 1., 1., 1., 0., 1., 1., 1., 0.],
          [1., 1., 1., 1., 1., 0., 1., 1., 1., 0.]]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: pietz
cooked: <p>How didn’t I see that… Anyway, thanks a lot for your time! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
IFFTN imaginary part

username: yegane
cooked: <p>Hi,<br>
I am testing with ifftn, and I am seeing an imaginary part in the value, which I think does not make sense.<br>
In here, we see that we don’t have an imaginary part (it mentions that it ignores it)<br>
<a href="https://pytorch.org/docs/stable/generated/torch.fft.irfft.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.fft.irfft.html</a><br>
, but in the case of ifftn, even in pytorch own example, we have imaginary parts (they are small though).<br>
Should I just ignore them, and use the real values? And what would happen if I use fft and ifft multiple times? Do I lose any information? fft and ifft should neither add or lose any information.<br>
Could someone please elaborate this for me?</p>
Distributed Sampling and Shuffling Samples

username: Ilker_GURCAN
cooked: <p>Hello pytorch devs and users,</p>
<p>My training runs in a distributed environment and I have to ensure that each parameter is initialized to the same value by calling the following instructions before spawning my processes:</p>
<pre><code class="lang-auto">def setRandomSeeds(randomSeed=0):
    torch.manual_seed(randomSeed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark     = False
    np.random.seed(randomSeed)
    random.seed(randomSeed)
</code></pre>
<p>It works perfectly. However; this time “DistributedSampler” fails to shuffle samples at every epoch. What should I turn off/on after weight initialization in every process so that it shuffles samples properly before every epoch starts?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>You might have forgotten to call the <code>sampler.set_epoch()</code> method. From the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler">docs</a>:</p>
<blockquote>
<p>In distributed mode, calling the <code>set_epoch()</code> method at the beginning of each epoch <strong>before</strong> creating the <code>DataLoader</code> iterator is necessary to make shuffling work properly across multiple epochs. Otherwise, the same ordering will be always used.</p>
</blockquote>

username: Ilker_GURCAN
cooked: <p>Sorry for the late reply <a class="mention" href="/u/ptrblck">@ptrblck</a> , but it took some time of mine to give it a try and probably due to different time zone as well.<br>
Yes! It really did the trick <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .<br>
Thank you!</p>
Quantization on avx

username: Jourdelune
cooked: <p>Hi, I tried quantization but the cpu I have uses avx and not avx2. So I get this error:</p>
<pre><code class="lang-auto">RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine
</code></pre>
<p>with this code:</p>
<pre><code class="lang-auto">model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
</code></pre>
<p>Is there any way to use quantization with a cpu that only supports avx?</p>
<p>Sincerely</p>

username: tom
cooked: <p>You could try to use the qnnpack engine with SSE4_1, but “AVX not AVX2” is generally not terribly well-suported.</p>

username: Jourdelune
cooked: <aside class="quote no-group quote-modified" data-username="tom" data-post="2" data-topic="141396">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>qnnpack avec SSE4_1</p>
</blockquote>
</aside>
<p>Hi, thanks for your answer, can you give an example or a link?  Google does not find any result with SSE4_1 <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: tom
cooked: <p><a href="https://github.com/pytorch/pytorch/tree/master/aten/src/ATen/native/quantized/cpu/qnnpack" rel="noopener nofollow ugc">QNNPACK</a> is part of the PyTorch github repo, is also needed on ARM.<br>
I don’t know if SSE support is compiled in by default, but I think it is you best bet.</p>
<p>Best regards</p>
<p>Thomas</p>

username: jerryzh168
cooked: <p>according to the error message, looks like you don’t have qengine set correctly, can you set the qengine with torch.backends.quantized.engine = “fbgemm”</p>
<p>here is the info for supported backend for quantization: <a href="https://pytorch.org/docs/master/quantization.html#natively-supported-backends" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch master documentation</a></p>
Loss funciton of a network with two decoders sharing a same encoder?

username: meshghi
cooked: <p>I have a network that includes one encoder and two decoders for semantic image segmentation. The two decoders share the same encoder. I’m confused about what the loss function would look like. Any help/suggestion/advice would be greatly appreciated.</p>

username: Dazitu616
cooked: <p>You can simply calculate two losses from the two decoder output separately, add the two losses together, and perform backpropagation. PyTorch will automatically accumulate the gradient on the shared encoder. Maybe something like:</p>
<pre><code class="lang-python">feature_map = encoder(image)
out1 = decoder1(feature_map)
out2 = decoder2(feature_map)

loss1 = loss_fn1(out1, label1)
loss2 = loss_fn2(out2, label2)
loss = loss1 + loss2
loss.backward()

optimizer.step()
optimizer.zero_grad()
</code></pre>

username: meshghi
cooked: <p>Thank you <a class="mention" href="/u/dazitu616">@Dazitu616</a>. That helps. I actually have a follow-up question – sorry I wasn’t super clear with my question earlier:</p>
<p>How does your solution change if the two decoders get trained together simultaneously, but each of which on a different dataset? I.e. decoder-1 gets trained on trainset-1 and decoder-2 gets trained on trainset-2.</p>

username: Dazitu616
cooked: <p>Then maybe you should forward data1 and data2 through encoder and decoder1/2 separately? Like:</p>
<pre><code class="lang-python">feat1 = encoder(data1)
feat2 = encoder(data2)
out1 = decoder1(feat1)
out2 = decoder2(feat2)
# the remaining code is the same
</code></pre>
Please give me some advice on model building.

username: tanabe.d.zc32s
cooked: <p>Hi.</p>
<p>I am in the process of building an original model based on Faster-R CNN. Specifically, it is a model that outputs the attributes of a class in addition to outputting the class and boundary box.As an image, I thought it should look like the figure below. I would like some advice. Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15.png" data-download-href="https://discuss.pytorch.org/uploads/default/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15" title="My model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15_2_690x237.png" alt="My model" data-base62-sha1="lceYdvEqzmNGUmU1Bh2URL4FYQR" width="690" height="237" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15_2_690x237.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15_2_1035x355.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/948fe68e3a794383a4a2ecfc0e85cb3e90aa1f15_2_1380x474.png 2x" data-dominant-color="5E7DAF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">My model</span><span class="informations">1935×665 83.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Implement Evolution Strategies on multiple processes or machines with PyTorch (Suggestions)

username: Caden_Miller
cooked: <p>Hello! I am new to using PyTorch and would like some feedback as to the best way to implement an evolution strategies algorithm where I update the parameter of a model myself. Particularly, I would like a way to run the algorithm on multiple processes and possibly multiple machines and share data between them (return UpdatedModel parameters) so that I can update the parameters of my model myself. How would I do this in PyTorch. Please point me towards any useful documentation or suggestions. Thank you.</p>
Pytorch for windows 11

username: arpit_sahni
cooked: <p>Hello kind sir, Actually I can post only 1 time in embedded posts as per discourse guildlines as i’m new.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I’m actually new to pytorch here it says 11.3 is latest CUDA toolkit supported by pytorch</p>
<p>However this toolkit has only windows 10 compatability</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8237ecf3d80ef20360fddb750f7a12aedf5e7220.png" data-download-href="https://discuss.pytorch.org/uploads/default/8237ecf3d80ef20360fddb750f7a12aedf5e7220" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8237ecf3d80ef20360fddb750f7a12aedf5e7220_2_643x500.png" alt="image" data-base62-sha1="izXTfKEhqTvUJcRYOkWm3oK93Ik" width="643" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8237ecf3d80ef20360fddb750f7a12aedf5e7220_2_643x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/8237ecf3d80ef20360fddb750f7a12aedf5e7220_2_964x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/2/8237ecf3d80ef20360fddb750f7a12aedf5e7220.png 2x" data-dominant-color="BABFB0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1221×948 42.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Am I missing something? Which CUDA toolkit to install for your pytorch on windows 11 with GPU support.</p>

username: arpit_sahni
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please throw some shade on this? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: talhaanwarch
cooked: <p>i have upgraded my windows 10 to 11 and cuda 11.0 is working for me. make sure you install the compatible PyTorch version which is think is 1.7 with cuda 11.0</p>

username: MarkerD
cooked: <p>I was in a similar situation. This worked for me:</p>
<p>conda install pytorch torchvision cudatoolkit=11.3 -c pytorch</p>
<p>This downloads the cudatoolkit 11.3 on the machine (this exists in conjunction with 11.6 on my PC) and recognizes my RTX 3070.</p>
<p>Note: If you already have installed a version of pytorch or torchvision, uninstalling it may help - it did in my case.</p>
Torchserve gRPC configuration options for client-side load-balancing

username: phiott256
cooked: <p>We’re experimenting with Torchserve in k8s and would like to use cilent-side load-balancing for an auto-scaling deployment. To trigger re-resolution of DNS A-records, <a href="https://github.com/grpc/grpc/issues/12295#issuecomment-358483266" class="inline-onebox" rel="noopener nofollow ugc">Need more intelligent re-resolution of names · Issue #12295 · grpc/grpc · GitHub</a> suggests setting <code>grpc.max_connection_age_ms</code> and <code>grpc.max_connection_age_grace_ms</code> on the server. Does Torchserve support this level of configuration for gRPC?</p>

username: lxning
cooked: <p>Torchserve does not support configuration for <code>rpc.max_connection_age_ms</code> and <code>grpc.max_connection_age_grace_ms</code>. It uses the GRPC default values.</p>
Java JDK dependencies for TorchServe

username: dhruvsakalley
cooked: <p>Current the doc says:<br>
<a href="https://pytorch.org/serve/install.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/serve/install.html</a><br>
<strong>java 8</strong> : Required. TorchServe use java to serve HTTP requests. You must install java 8 (or later) and make sure java is on available in $PATH environment variable  <em>before</em>  installing TorchServe. If you have multiple java installed, you can use $JAVA_HOME environment variable to control which java to use.</p>
<p>But the instructions refer to Java 11 JDK. Which I think is correct (I tried with just Java8 and it did not work) So the docs need an update. But my real question is… I’m not sure if one needs full JDK or JRE?<br>
From a security perspective JDK is a larger attach surface, having dependency reduced to JRE makes TorchServe better from a production standpoint. Would love some thoughts on this topic from the devs.</p>
<p>Best,<br>
Dhruv</p>

username: lxning
cooked: <p><a href="https://github.com/pytorch/serve/blob/master/README.md#install-torchserve" rel="noopener nofollow ugc">Here</a> is the new link of the installation.</p>
<p>JDK11 is used for Torchserve dev environment. For Prod, we need test if JRE is enough or not.</p>
Accuracy on training in Pytorch

username: shgh
cooked: <p>I would like to do binary classification with softmax. Am I calculating training accuracy in the code below wrong?</p>
<p>I am new to Pytorch and it is very strange I am getting accuracy above 100</p>
<pre><code class="lang-auto">correct = 0
train_loss = []


model.train()

for epoch in range(epochs):
    for xb, yb in train_dl:
        y_pred = model(xb)           
        loss = loss_func(y_pred, yb)  
        optimizer.zero_grad()       
        loss.backward()              
        optimizer.step()              
    train_loss.append(loss.item())
    
    _,y_pred_ = torch.max(y_pred, dim = 1)
    
    correct += (y_pred_ == yb).sum()
    

acc_ = 100 * correct / len(x_tensor)

print(correct)   
print("Accuracy = {}".format(acc_))
    
    
print('Last iteration loss value: '+str(loss.item()))
</code></pre>
<p>thanks in advance</p>

username: eqy
cooked: <p>What is <code>x_tensor</code> here?</p>
<p>You may want to do do a sanity check that <code>len(x_tensor)</code> matches the total number of datapoints seen doing evaluation, e.g., summing the total of <code>y_pred.size(0)</code>.</p>
How to serve 2 model on torchserve?

username: OkabeSan
cooked: <p>Can we Infere 2 model at the same time on a GPU with Torchserve? For example, I want to Infere a model 1 and 2 on the same system at the same time. How can I do that?</p>

username: J_Johnson
cooked: <p>Here is a workflow for multiple models on TorchServe.</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/serve/tree/master/examples/Workflows">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/serve/tree/master/examples/Workflows" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/serve/tree/master/examples/Workflows" target="_blank" rel="noopener nofollow ugc">serve/examples/Workflows at master · pytorch/serve</a></h3>

  <p><a href="https://github.com/pytorch/serve/tree/master/examples/Workflows" target="_blank" rel="noopener nofollow ugc">master/examples/Workflows</a></p>

  <p><span class="label1">Model Serving on PyTorch. Contribute to pytorch/serve development by creating an account on GitHub.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: marksaroufim
cooked: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> is correct in that workflows let you create a DAG of dependent models that you can then run. But if your models are completely independent from each other then you can just have them all be in your <code>model_store</code>. Torchserve was designed with multi model serving in mind from the very beginning</p>
How to make the best use of a single GPU with a small model

username: chtapodi
cooked: <p>I have a small model that uses up very little of the GPUs resources. I am looking for the best method to run data through this model as quickly as possible. The order of the data does not matter so no synchronization needs to occur.</p>
<p>I have looked into multiprocessing, however I have been unable to pass the model/tensors as .cuda(), and converting them every time seems to be a significant slowdown.</p>
<p>What is the best way to approach the problem with getting as much out of a single GPU as possible.</p>

username: J_Johnson
cooked: <p>High batch size should work. Batch size parallelizes compute so as to maximize GPU cores usage. You could think of a batch size of 1,000 as running 1,000 models side by side to compute the results. Granted, that is a subpar comparison because once you maximize your cores usage, higher batch size won’t reduce total job speed.</p>

username: marksaroufim
cooked: <p>Increasing batch size is indeed the easiest way to saturate a GPU. A GPU is asynchronous so you shouldn’t need to parallelize the way you execute jobs on a GPU.</p>
<p>Also if your utlization is very low it could be the case that you’re not using your GPU so using <code>nvidia-smi</code> and making sure that GPU memory allocation is changing to account for the model size and that both model weights and input data are stored on GPU</p>
<p>Other profiling tools like the pytorch profiler will help you root cause your issue more quickly <a href="https://github.com/pytorch/kineto/tree/main/tb_plugin" rel="noopener nofollow ugc">https://github.com/pytorch/kineto/tree/main/tb_plugin</a></p>
Deploying multiple PyTorch models on a single GPU

username: blackberry
cooked: <p>Hello PyTorch community,</p>
<p>Suppose I have 10 different PyTorch models (classification, detection, embedding) and 10 GPUs. I would like to serve real-time image traffic on these models. We can assume a uniform traffic distribution for each model. What is the most efficient (low latency, high throughput) way?</p>
<ol>
<li>Deploy all 10 models onto each and every GPU (10 models on all 10 GPUs). This will probably incur context switch and cache miss costs. Memory management might also be costlier. Latency will be high. (?)</li>
<li>Deploy a single model on each GPU (10 models on 10 GPUs). No context switch or cache miss costs. Can be good for uniform traffic. (?)</li>
</ol>
<p>Any suggestions, insights, experience?</p>
<p>Thanks!</p>

username: J_Johnson
cooked: <p>I can tell that someone was watching Tesla AI day. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>You can deploy as many models as your gpu has memory to hold. However, during training, a model can require 4-5 times more gpu ram and equivalent increase in calculation time as your model is doing optimization. So you might be better allocating GPUs accordingly via a distributed training workflow. But make sure the size of all your GPUs can fit on the deployment GPU. One way to do this would be to test the 10 models on your GPU untrained and in eval mode. Granted, the inference wouldn’t be any good as you haven’t trained them, yet, but it will show if you need to tune model sizes accordingly. Then you can tweak model sizes before training. Make sure you have a memory buffer of 10-15% from the max memory.</p>

username: marksaroufim
cooked: <p>The straightforward answer is have one model per GPU where the main issue will be underutilization and over-utilization of some since traffic will most certainly not be uniform which means you waste money.</p>
<p>Generally in this scenario you want a framework to figure out how to allocate multiple models to multiple GPUs and that’s one of the reasons we built torchserve  a multi model inferencing framework to take care of this for you <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
<p>You would simply need to set some number of workers for each model depending on its importance to you or its speed after doing some profiling</p>
What is the installing file size for torch-serve if I want to deploy a model for inference(image, nlp, or graph)?

username: JonathanSum
cooked: <p>In fact, I want to use it similar to ONNX javascript that allows me to do inference in the bowser without Restful or backend.<br>
There is a way to run C++, python, or other programming languages in javascript today.</p>
<ol>
<li>But, what is the size for torch-serve?</li>
<li>Can I run it with script or code rather than Restful API.</li>
<li>And the size for installing the requirement for torch serve?</li>
</ol>

username: marksaroufim
cooked: <p>Torchserve is not meant to run within a browser but would its own service that you can then call with a RESTful endpoint</p>
<p>If you’re looking for an in browser solution then you can take a look at <a href="https://pytorch.org/live/docs/tutorials/server-model/" class="inline-onebox" rel="noopener nofollow ugc">Server Model | Easy to use set of tools to create on-device ML demos on Android and iOS. Unlock the vast potential of AI innovations.</a> (experimental)</p>
TorchServe for Partially Trained Models

username: A-F-V
cooked: <p>Hi,</p>
<p>I am reading into torchserve and from my understanding it provides a system for clients to send inference requests to a trained model.</p>
<p>Is it possible to use a model which needs to perform gradient ascent to predict values with torchserve, using a partially trained model?</p>
<p>If not, what alternatives could one use?</p>
<p>Thanks</p>

username: marksaroufim
cooked: <p>Could you elaborate what you mean by gradient ascent on a partially trained model? If your intent is to perform an inference on a pretrained model then that should work as is with torchserve,</p>
<p>If your intent is to retrain a model based on the result of some inference then you can deploy pretrained model, get inference, replace via a new model - see <a href="https://github.com/pytorch/serve/blob/master/docs/management_api.md" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/management_api.md</a> for more information</p>
Can i make my pytorch model to docker image?

username: 111335
cooked: <p>I want to pack my model to docker image.<br>
Cause i’m beginner in domain of docker, i just know one solution - make a REST API server with pytorch model and make it as docker image. Then, when use this docker server, send request to train or test model.<br>
Is there other options??</p>
<p>Thank you</p>

username: marksaroufim
cooked: <p>That workflow is similar to how <code>torchserve</code> users have deployed a model in the past, check it out and lemme know if you have any questions <a href="https://github.com/pytorch/serve/tree/master/docker" rel="noopener nofollow ugc">https://github.com/pytorch/serve/tree/master/docker</a> but this should make it easy to take a trained model, serve it from within a container and expose an HTTP endpoint that users can query against</p>
<p>If you’re looking at packaging a model in Docker file for training</p>
<p>Then in your <code>Dockerfile</code> you need to <code>ADD directory_with_data_and_code</code> and then <code>RUN python train.py</code> would be an easy start</p>
AD Sparse KMeans with CSR format?

username: OlaRonning
cooked: <p>Hi,</p>
<p>We are testing AD with sparse representations in PyTorch using a simple <a href="https://gist.github.com/OlaRonning/872a109868daaff058d74a4302dfe6b0#file-sparse_kmeans-py-L15-L42" rel="noopener nofollow ugc">kmeans example</a>. We are currently using the COO format as vjp/vhp throws exceptions with CSR format. We would appreciate it if we could get help with a couple of questions.</p>
<p>Does AD support the CSR format? And if so, are we using the wrong version of matrix multiplication (or another sparse operation)?</p>
<p>edit: (can only have two links, resolved second question)</p>
RuntimeError: 0D or 1D target tensor expected, multi-target not supported in Pytorch

username: shgh
cooked: <p>I would like to do binary classification with softmax in Pytorch. Even though I set the number of output as 2 and use “nn.CrossEntropyLoss()”, I am getting the following error:</p>
<pre><code class="lang-auto">RuntimeError: 0D or 1D target tensor expected, multi-target not supported
</code></pre>
<pre><code class="lang-auto">train_loss = []

for epoch in range(epochs):
    
    y_pred = model(x_train)  
    loss = loss_func(y_pred, y_tensor)  
    optimizer.zero_grad()         
    loss.backward()               
    optimizer.step()             
        
    train_loss.append(loss.item())
</code></pre>
<p>The size I am getting from <code>y_pred</code> and <code>y_tensor</code> is <code>torch.Size([3000, 2]) torch.Size([3000, 1])</code>. How this issue can be solved?</p>
<p>thanks in advance</p>

username: Dazitu616
cooked: <p>I think your GT <code>y_tensor</code> should be of shape <code>[3000]</code> not <code>[3000, 1]</code></p>
<pre><code class="lang-auto">&gt;&gt;&gt; pred = torch.randn(10,2)
&gt;&gt;&gt; gt = torch.randint(0,2,(10,))
&gt;&gt;&gt; nn.CrossEntropyLoss()(pred,gt)
tensor(0.8060)
&gt;&gt;&gt; gt = torch.randint(0,2,(10,1))
&gt;&gt;&gt; nn.CrossEntropyLoss()(pred,gt)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "D:\anaconda3\envs\taichi\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\anaconda3\envs\taichi\lib\site-packages\torch\nn\modules\loss.py", line 1152, in forward
    label_smoothing=self.label_smoothing)
  File "D:\anaconda3\envs\taichi\lib\site-packages\torch\nn\functional.py", line 2846, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: 0D or 1D target tensor expected, multi-target not supported
</code></pre>

username: shgh
cooked: <p>I get the<br>
following error if I make the change to <code>y_tensor</code>:</p>
<pre><code class="lang-auto">IndexError: Target 4 is out of bounds.
</code></pre>
<p>I have two classes. One is labeled as 0 the other labeled as 4. Do I have to change the label to 1 or there is a better way to deal with this error?</p>

username: Dazitu616
cooked: <p>Yes, please change 4 to 1 because the last dimension of your prediction output is 2<br>
You can achieve it by <code>y_tensor[y_tensor == 4] = 1</code> (or modify it in the dataloader)</p>
Pipeline batches during inference

username: adizhol
cooked: <p>Hi,<br>
Google published a paper where they “pipeline” the inference (and backprop) stage (each later at a time).<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.2.7/images/icons/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://arxiv.org/abs/2001.06232" target="_blank" rel="nofollow noopener">arXiv.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="16" height="16">

<h3><a href="https://arxiv.org/abs/2001.06232" target="_blank" rel="nofollow noopener">Sideways: Depth-Parallel Training of Video Models</a></h3>

<p>We propose Sideways, an approximate backpropagation scheme for training video
models. In standard backpropagation, the gradients and activations at every
computation step through the model are temporally synchronized. The forward
activations need to...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I thought that most frameworks do this during the forward pass when inferencing, since you don’t need to wait for the batch to finish calculating all the outputs before loading a new batch.<br>
If that’s not the case, is any work on this planned for pytorch?</p>
<p>Thanks!</p>

username: marksaroufim
cooked: <aside class="quote no-group" data-username="adizhol" data-post="1" data-topic="67769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adizhol/48/12377_2.png" class="avatar"> adizhol:</div>
<blockquote>
<p>I thought that most frameworks do this during the forward pass when inferencing, since you don’t need to wait for the batch to finish calculating all the outputs before loading a new batch.</p>
</blockquote>
</aside>
<p>It’s here <a href="https://pytorch.org/docs/stable/pipeline.html" class="inline-onebox">Pipeline Parallelism — PyTorch 2.1 documentation</a></p>
Does torchserve support batching of requests before inference?

username: reachtarunhere
cooked: <p>I am trying to deploy a DL model where inference time is the biggest bottleneck compared to pre and post-processing. My users might hit my REST API with small-batch numbers. So if I serve each request as its own batch the performance suffers. In the ideal world, I would batch the requests from different users that come with a certain small timeframe and then run the inference on the GPU. Following this separate the results and send each user their response.</p>
<p>Torchserve seems to have what is called “optional request batching” but I would like to know if it is the same thing? If yes on what basis is this batching done?</p>
<p>If not I guess the only solution for me would be to do some dirty async work and roll my own solution.</p>
<p>Thanks!</p>

username: marksaroufim
cooked: <p>Suppose several clients are making requests to torchserve in a batched setting. Torchserve has 2 important configurations the batch size and batch delay.</p>
<p>As requests come in they keep going into the same batch until either the batch size or batch delay is exceeded and then run a single batched inference</p>
<p>You shouldn’t need to build anything from scratch <a href="https://github.com/pytorch/serve/blob/master/docs/batch_inference_with_ts.md" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/batch_inference_with_ts.md</a></p>
Docker container getting crashed while serving Pytorch models

username: atinesh
cooked: <p>I have to deploy some PyTorch models (around 4) in production, for demonstration purpose I have created a minimal code showing how I am deploying the models using <code>Flask</code> and <code>Docker Swarm</code></p>
<p>Problem is that when I load test the <code>API</code> endpoints I found that memory utilization of container/service increases and it crashes after some time, I tried increasing the container memory limit in a <code>docker-compose.yml</code> file but it only extends crashing time</p>
<p>Here is the complete <a href="https://gist.github.com/atinesh-s/e2d1370b4898c2a62ae7af3c1b22e328" rel="noopener nofollow ugc">code</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/4084ae9afaac84fd4a879a715d4632e5a2aff3ad.png" alt="Screenshot from 2021-04-13 11-11-14" data-base62-sha1="9cKPlca98rWl3rnwU6Cx6U7KgXz" width="228" height="218"></p>

username: marksaroufim
cooked: <p>This is a bit tricky to debug so more details would help. I’d suggest taking snapshots of memory and gpu utilization and measuring when the crash happens. And at least to debug I suggest increasing both dramatically to see if the issue disappears</p>