akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Launch.py tool doesn't set local_rank properly

Olivier-CR: <p>Hi,</p>
<p>I’m trying to launch a train.py DDP script to run over a 4-GPU machine.<br>
i’m using the launch.py tool described <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" rel="noopener nofollow ugc">here</a>, (this experience is quite ugly btw, I which there was a clean PyTorch class to do that!) that is supposed to set local_rank properly in each process: <em>“–local_rank: This is passed in via launch.py”</em> as the documentation says.</p>
<pre><code class="lang-auto">python /home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/distributed/launch.py \
    --nnode=1 \
    --node_rank=0 \
    --nproc_per_node=4 \
    train.py \
    --gpu-count 4 \
    --dataset . \
    --cache tmp \
    --height 604 \
    --width 960 \
    --checkpoint-dir . \
    --batch 10 \
    --workers 24 \
    --log-freq 20 \
    --prefetch 2 \
    --bucket $bucket \
    --eval-size 10 \
    --iterations 20 \
    --class-list a2d2_images/camera_lidar_semantic/class_list.json
</code></pre>
<p>However, in each of my processes local_rank = -1 (default value). What is wrong? how to get local_ranks each distinct?</p>

cbalioglu: <p>cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> <a class="mention" href="/u/aivanou">@aivanou</a></p>
Setting some elements of layer parameters to 0 for each training iteration

tsly123: <p>Hi everyone,</p>
<p>I want to fine-tune a pre-trained network with same data as it was trained.<br>
I need help in setting some element of conv2d layer weight (and bias too) to 0 for each training iteration and others are trained as normal. My goal is to have 0 outputs where set as 0.</p>
<p>For example, my layer has shape<br>
<code>conv1.weight.shape # torch.Size([32, 64, 7, 7])</code><br>
and I have a list of unit that need to be set to 0 <code>zeroed_units = [1, 3, 5, 7, 9]</code><br>
and I would like it to have the <code>tensor of conv1.weight at index [1, 3, 5, 7, 9] = 0</code>, i.e. <code>torch([32,zeroed_units,7,7] = 0)</code></p>
<ul>
<li>I have thought about setting these units 0 and set requires_grad=False. However, according to google, requires_grad only allow for entire tensor layer not part of it.</li>
<li>Another way is to apply a non-trainable mask buffer of 1’s and set 0 where needed after this conv2d and does not touch the conv2d layer. This method could give me the desired output. However, when training in this way, the conv2d layer weight distribution would have the same distribution as before. The number might change but the distribution is not changed much. I would like to see the effect of different weight’s distribution on the final results when setting units to 0.</li>
</ul>
<p>Thank you.</p>

tsly123: <p>With a bit of luck, I found some answers that are close to what I want to do.</p>
<p><a href="https://discuss.pytorch.org/t/can-detach-work-for-parts-of-the-layer-weights/38064">Can.detach() work for parts of the layer weights?</a><br>
and<br>
<a href="https://discuss.pytorch.org/t/update-only-sub-elements-of-weights/29101">Update only sub-elements of weights</a></p>
<p>It seems that these are not the optimal solution in terms of saving resources since it uses the gradient mask and sets the gradient to 0 where needed. However, this seems to be the only possible solution for me right now.</p>
PyTorch 1.10 keeps installing in cpu only mode

ChrisHa: <p>I am using anaconda under windows 11.<br>
After fully uninstalling my previous PyTorch 1.9 installation(which worked in CUDA mode),</p>
<p>I tried the following(as per official guidelines):</p>
<p>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</p>
<p>Although this does install the cudatoolkit, pytorch and the other packages installs the CPU version alongside the package “cpuonly”</p>
<p>How do I install the CUDA version of pytorch?</p>

kbt888: <p>You might try downloading the tar file from:<br>
<a href="https://anaconda.org/pytorch/pytorch/files" class="onebox" target="_blank" rel="noopener nofollow ugc">https://anaconda.org/pytorch/pytorch/files</a></p>
<p>and installing via the command line:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/45156080/installing-modules-to-anaconda-from-tar-gz">
  <header class="source">

      <a href="https://stackoverflow.com/questions/45156080/installing-modules-to-anaconda-from-tar-gz" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
    
<h4>
  <a href="https://stackoverflow.com/questions/45156080/installing-modules-to-anaconda-from-tar-gz" target="_blank" rel="noopener nofollow ugc">Installing modules to Anaconda from .tar.gz</a>
</h4>

<div class="tags">
  <strong>python-3.x, anaconda</strong>
</div>

<div class="date">
  asked by
  
  <a href="" rel="noopener">
    user4550164
  </a>
  on <a href="https://stackoverflow.com/questions/45156080/installing-modules-to-anaconda-from-tar-gz" target="_blank" rel="noopener nofollow ugc">01:36AM - 18 Jul 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Can I do modification on a branch of chain in the backward pass?

Chaofei: <p>Hello,  I want to do a kind of custom backpropagation with separate backward chains, but I don’t know how to do it with autograd.</p>
<p>Say I have a set of operations as:</p>
<pre><code>y = f1(x)
u = f2(y)
v = f3(y)
z = f4(u, v)
</code></pre>
<p>Then, the the backward chain should be:</p>
<pre><code>dz/dx = dz/dy *dy/dx
          = (dz/du * du/dy + dz/dv * dv/dy) * dy/dx       (1)
</code></pre>
<p>the above equation (1) can be further parted into two chains as:</p>
<pre><code>dz/dx = (dz/du * du/dy * dy/dx) + (dz/dv *dv/dy * dy/dx)        (2)
</code></pre>
<p>However, I want to modify one branch of the backpropagate chain in equation (2), for example:</p>
<pre><code> dz/dx = (dz/du * du/dy * dy/dx)*mean(dz/du * du/dy * dy/dx)  +  (dz/dv *dv/dy * dy/dx) 
</code></pre>
<p>My understanding is that the autograd compute dz/dx according to equation (1) not equation (2). So the two backpropagate chains may never exist in the autograd computation. How can I implement this gradient modification in autograd?</p>

albanD: <p>If you want to compute these two contributions independently, then you will need to either:</p>
<ul>
<li>create a graph with only that one branch using .detach() in the other</li>
<li>use hooks on the second branch to zero-out its contribution (u.register_hook(lambda x: torch.zeros_like(x))`).</li>
</ul>
<p>That way you can get the two terms by doing two backward passes. And then you can accumulate the gradients from the two backward to get the final value that you want.</p>

Chaofei: <p>Thanks for the reply. But can it have a more elegant solution? By doing above solution, I have to do two forward and two backward computation for the gradient modified sub-network, which is very inefficient. Can I have a way to control how the autograd engine do backward propagation?</p>

albanD: <p>Using the hook version, you can do with a single forward and two backwards.</p>
<blockquote>
<p>Can I have a way to control how the autograd engine do backward propagation?</p>
</blockquote>
<p>There is but what you’re asking for here is a bit weird. It breaks one of the main assumption which is that if you had a Tensor <code>foo</code> during the forward, then the quantity being backpropagated for it is another Tensor of the same size.<br>
In this case, you actually want two values to flow back at once for the top of your graph. So you implicitly do two backward passes. It is just that it is not explicit in your math.</p>
How to use lucent library to do activation maximization

nikiguo93: <p>Hi everyone. I want to use lucent library to do actimation maximization. (<a href="https://pypi.org/project/torch-lucent/" class="inline-onebox" rel="noopener nofollow ugc">torch-lucent · PyPI</a>). But there is no relative tutorial there. Can anybody offer me some good tutorials?<br>
Any relative information would be appreciated.<br>
Thank you!</p>
Fft (shifted) in pytorch for lena image

James_W: <p>I am looking for a minimum example to plot the magnitude and phase of the popular lena image in pytorch. Essentially, I want to replicate this (the mag and phase plots) from matlab:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/13549186/how-to-plot-a-2d-fft-in-matlab">
  <header class="source">

      <a href="https://stackoverflow.com/questions/13549186/how-to-plot-a-2d-fft-in-matlab" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/868331/yoav" target="_blank" rel="noopener nofollow ugc">
    <img alt="Yoav" src="https://www.gravatar.com/avatar/c7c4806aafa8c1f7e8c4ae45731351e4?s=256&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/13549186/how-to-plot-a-2d-fft-in-matlab" target="_blank" rel="noopener nofollow ugc">How to plot a 2D FFT in Matlab?</a>
</h4>

<div class="tags">
  <strong>matlab, image-processing, plot, fft</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/868331/yoav" target="_blank" rel="noopener nofollow ugc">
    Yoav
  </a>
  on <a href="https://stackoverflow.com/questions/13549186/how-to-plot-a-2d-fft-in-matlab" target="_blank" rel="noopener nofollow ugc">07:35AM - 25 Nov 12 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>is there a minimum example somewhere I can borrow for this?</p>
Why run torch.unique twice get different running time?

Skyler_Huang: <p>Code is:<br>
time1 = time.time()<br>
unique, inv =  torch.unique(src, ,sorted=False, return_inverse=True)<br>
time2 = time.time()<br>
src2 = src.clone()<br>
time2 = time.time()<br>
unique, inv =  torch.unique(src2, sorted=True, return_inverse=True)<br>
time3 = time.time()<br>
print(time2- time1, time3-time2)</p>
<p>get the time:  0.015712261199951172 0.000576972961425781<br>
It’s very strange that time-consuming seems to have nothing to do with whether it is sorted or not. The time-consuming to run first is always higher than  later.</p>

ptrblck: <p>In case you are using the GPU, you would have to synchronize the code before starting and stopping the timers since CUDA operations are executed asynchronously. The better way to profile such workloads would be to use <code>torch.utils.benchmark</code>.</p>

Skyler_Huang: <p>Thanks for your reply! after synchronize the code, the time consuming of two operation are in same magnitude.</p>
LRscheduler.stepLR and ReduceLROnPLateau

sairam: <p>For my neural network, I am trying to vary the learning rate using two different approaches - LRscheduler.stepLR and ReduceLROnPlateau. I have tried multiple values for step_size and gamma for LRscheduler.stepLR and factor and patience for ReduceLROnPlateau, but not getting good results compared to a constant learning rate.</p>
<pre><code class="lang-auto">scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)

scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.05, patience=20, verbose=True)
</code></pre>
<p>But the common practice is to schedule the learning rate as it is believed to give better results. So, I wanted to ask is there a common best practice for choosing these values based on the number of epochs or initial learning rate, or some other stuff?</p>

sairam: <p>Any suggestions for the question?</p>
Aggregate tensors in a specific order

ZimoNitrome: <p>I have 10 images that I want to merge into a single image.<br>
These images are RGBA tensors and thus have the shape: <code>[N C H W] = [10, 4, 64, 64]</code></p>
<p>The way images are being merged depends on their order. Later images are being “added” on top of earlier ones; replacing some pixels in the final output image. I want the images to be ordered according to a second tensor <code>z_values</code> of shape <code>[N]</code>.</p>
<p>I could try to sort the images given indices from <code>torch.argsort(z_values)</code> but that would not be differentiable. Any ideas on how I might achieve this?</p>
The error occurred：‘Generator’ object has no attribute ‘translate’

qingye_meng: <p>I am using the mass translation model based on fairseq to apply it to the Django server.Therefore, in the translation phase, I try to use the translate method for reference: <a href="https://github.com/pytorch/fairseq/tree/main/examples/translation#example-usage-torchhub" class="inline-onebox" rel="noopener nofollow ugc">fairseq/examples/translation at main · pytorch/fairseq · GitHub</a></p>
<pre><code class="lang-auto">from fairseq.models.xtransformer import XTransformerModel
zh2en = XTransformerModel.from_pretrained(
'/work/qymeng5/MASS_test/model',
checkpoint_file='model.pt',
data_name_or_path='/work/qymeng5/MASS_test/vocabulary', #Vocabulary path
bpe='subword_nmt',
bpe_codes='/work/qymeng5/MASS_test/bpe/all.zh.bpe.codes' #Chinese BPE path
)

ff=zh2en.translate('你好 世界')
print('ff',ff)
</code></pre>
<p>But running this code will report this error. Is it because MASS doesn’t implement the translate method, or is it my code problem？</p>

ptrblck: <p>Also tracked <a href="https://github.com/pytorch/fairseq/issues/3977">here</a>.</p>
There may be some errors in the `torchvision.transforms.transfroms.RandomCrop`

zswldxb: <p>In the method <code>get_params</code> of <code>torchvision.transforms.transfroms.RandomCrop</code></p>
<p>def get_params(img: Tensor, output_size: Tuple[int, int]) → Tuple[int, int, int, int]:<br>
w, h = F._get_image_size(img)<br>
th, tw = output_size<br>
if h + 1 &lt; th or w + 1 &lt; tw: # <strong>this may be changed to <code>h &lt; th or w &lt; tw</code></strong><br>
raise ValueError(<br>
“Required crop size {} is larger then input image size {}”.format((th, tw), (h, w))<br>
)<br>
if w == tw and h == th:<br>
return 0, 0, h, w<br>
i = torch.randint(0, h - th + 1, size=(1, )).item()<br>
j = torch.randint(0, w - tw + 1, size=(1, )).item()<br>
return i, j, th, tw</p>

ptrblck: <p>Thanks for reporting the issue. It seems the check might need to be updated, as an invalid size by 1 will fail in the sampling step currently instead of the check:</p>
<pre><code class="lang-python">t = transforms.RandomCrop(size=101)
t.get_params(torch.randn(3, 100, 100), (50, 50)) # works
t.get_params(torch.randn(3, 100, 100), (100, 100)) # works
t.get_params(torch.randn(3, 100, 100), (101, 101)) # fails in the sampling step
# &gt; RuntimeError: random_ expects 'from' to be less than 'to', but got from=0 &gt;= to=0
t.get_params(torch.randn(3, 100, 100), (102, 102)) # fails in the check
# &gt; ValueError: Required crop size (102, 102) is larger then input image size (100, 100)
</code></pre>
<p>Would you mind creating an issue in <a href="https://github.com/pytorch/vision/issues">GitHub</a> so that it can be tracked and fixed?</p>
Draw differentiable rectangle?

ZimoNitrome: <p>Is there any way to draw a rectangle on a canvas in a way that is differentiable (i.e. can be backproped through)?</p>
<p>I am guessing something like would not be possible since indexing breaks backprop:</p>
<pre><code class="lang-python">def draw_rect(canvas_tensor, x0, y0, x1, y1, color):
    canvas_tensor[..., y0:y1, x0:x1] = color
</code></pre>
<p>(where all parameters are trainable parameters)</p>

tumble-weed: <p>some hints to help you think:</p>
<ol>
<li>think first in terms of a circle ( it is simpler and has many of the similar requirements as this)</li>
<li>the gaussian function in 2d with a diagonal covariance matrix, and equal x and y variances makes circles. the mean and variance are the parameters for this.</li>
<li>the differentiable color is the easiest part of the problem that i guess you’ll get quickly.</li>
</ol>
<p>if you can solve the circle problem, i think you might get ideas for the rectangle.</p>

ZimoNitrome: <p>Thanks for the pointers!</p>
<p>I will try this out and see if I can extend it to rectangles as well. If nothing else, it might be possible to add circles in a manner similar to a fourier transform.</p>
<p>I will report back during the week.</p>

ZimoNitrome: <p>I am still having some problems with this.<br>
Even in the case of drawing circles, how would I actually put them on a canvas?</p>
<p>Say that I start of with a blank <code>canvas = torch.zeros([1, 1, 256, 256], requires_grad=True)</code><br>
Given a covariance matrix (or a mean and a variance), what torch operation could translate/plot these into pixels on my <code>canvas</code>?</p>

ZimoNitrome: <p>I managed to draw a circle at least!</p>
<pre><code class="lang-auto">s = 256

canvas = torch.zeros([1, 1, s, s])

ss = torch.linspace(0, 1, s)
x, y = torch.meshgrid(ss, ss)

px = 0.5
py = 0.5
w = 0.5
h = 0.5

image = 1 - 2*(((x - px)**2) / w**2 + ((y - py)**2) / h**2)
image2 = image.clamp(0, 1)
canvas[0, 0] = image2
</code></pre>
<p>Tomorrow it’s time for rectangles.</p>

ZimoNitrome: <p>I updated my previous code for a circle to draw a squircle instead.</p>
<p>It is close enough to a square.</p>
<pre><code class="lang-auto">s = 256

canvas = torch.zeros([1, 1, s, s])

ss = torch.linspace(0, 1, s)
x, y = torch.meshgrid(ss, ss)

px = 0.5
py = 0.5
w = 0.5
h = 0.5
exp = 10

image = 1 - 2*(abs((x - px)/w)**exp + abs((y - py)/h)**exp)
image2 = image.clamp(0, 1)
canvas[0, 0] = image2
</code></pre>
Initialize bias using class probabilities

crypdick: <p>In Andrej Karpathy’s famous <em><a href="https://karpathy.github.io/2019/04/25/recipe/" rel="noopener nofollow ugc">“A Recipe for Training Neural Networks”</a></em> post, he recommends:</p>
<blockquote>
<p>Initialize the final layer weights correctly. […] If you have an imbalanced dataset of a ratio 1:10 of positives:negatives, set the bias on your logits such that your network predicts probability of 0.1 at initialization. Setting these correctly will speed up convergence and eliminate “hockey stick” loss curves where in the first few iteration your network is basically just learning the bias.</p>
</blockquote>
<p>How do you do this?</p>
<p>More concretely, say we have the following:</p>
<ul>
<li>a multi-class, multi-label dataset</li>
<li><code>len(dataset)</code> is 1000</li>
<li>dataset sample counts : <code>{A: 10, B: 100, C: 900}</code></li>
<li><code>model.fc = nn.Linear(num_features, 3, bias=True)</code></li>
</ul>
<p>Therefore, I want the model to output probabilities <code>[0.01, 0.1, 0.9]</code> at initialization.</p>
<p>How do I calculate the tensors I need to initialize  <code>model.fc.bias.data</code> and <code>model.fc.weight.data</code> properly?</p>

googlebot: <p>x = torch.tensor([0.01,0.09,0.9]) <span class="hashtag">#sums</span> to 1<br>
log_x = x.log()<br>
log_x.softmax(-1) # == x, so probabilities supplied in x will be recovered<br>
out_layer.bias.data = log_x <span class="hashtag">#what</span> you’re asking for</p>
<p>you don’t have to set weight.data, as it is assumed that matmul(weight, hidden_features) produces zero centered noise initially.</p>
Multivariate data for LSTM training on PyTorch

BigMuffler: <p>I am working with a set of data for training a deep learning LSTM model in PyTorch. I have written a working model with a single variable as input but I was wondering what the convention was for a multi-dimensional input tensor. I already know the LSTM module in PyTorch accepts data of the form <code>(batch_size, sequence_length, input_size)</code> however I’d like to use training data of the form</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Date</th>
<th>x1</th>
<th>x2</th>
<th>x3</th>
</tr>
</thead>
<tbody>
<tr>
<td>date1</td>
<td>data11</td>
<td>data12</td>
<td>data13</td>
</tr>
<tr>
<td>date2</td>
<td>data21</td>
<td>data 22</td>
<td>data 23</td>
</tr>
<tr>
<td>etc</td>
<td>etc</td>
<td>etc</td>
<td>etc</td>
</tr>
</tbody>
</table>
</div><p>I am using a moving window method to get sequences and they are all stored in a large tensor of the form <code>(5,36070,10,1)</code>. So 5 lists (corresponding to the number of variables) of 36070 elements each with length of 10 elements. Due to the nature of the input of the LSTM module my initial approach was to reshape the large tensor to <code>(180350,10,1)</code>, so the 5 lists of 306070 have been stacked. Is this the correct approach? Or can the input of the LSTM module accept tensor shapes instead of a scalar representing the size of a unidimensional vector?</p>

googlebot: <p>the logic is the same as with a simple Linear layer - different data should be fed through different vector positions, with vectors occupying the last dimension.</p>
<p>so, with 5 variables you need to permute the dataset to (*,5).</p>
<p>if you’re just unrolling univariate windows, your “inflated dataset” approach may work. there are some potential issues with mini-batch random sampling, but I think this won’t play a big role with your dimension sizes.</p>

BigMuffler: <p>Hi, what do you mean by “different data should be fed through different vector positions, with vectors occupying the last dimension.” what should the resulting shape be of the input vector be then?</p>

googlebot: <p>if your data is tabular, with 5 explanatory factors, input shape can be either (batch,time,5) or (time,batch,5). if you’re unrolling values from a single column dataset, ignore this, last dimension size 1 is correct.</p>
Correct way copy batch of tensors to another single tensor

Ella: <p>I tried to copy several tensors (a batch of tensors) to one single tensor, but I got the next error:<br>
raise RuntimeError("Only Tensors created explicitly by the user "<br>
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment</p>
<p>Any idea how to copy/transfer a batch of tensors to the single 4D tensor?</p>
<p>I have attached my code:</p>
<p>class GLr(nn.Module):</p>
<pre><code>def __init__(self, shape):
    super(GL, self).__init__()

    assert len(shape) == 4, " Invalid argument: shape must be tuple of fourth"
    self.layers = nn.Parameter(data=torch.Tensor(shape), requires_grad=True)
    self.temp_holder = torch.rand(shape)

def forward(self, images):

    for i, image_ in enumerate(images):
        for j, image in enumerate(image_):
            self.temp_holder[i][j] = self.kernel_fn( .... )
    self.layers.data = copy.deepcopy(self.temp_holder)
    return self.layers</code></pre>

ptrblck: <p>You could <code>detach()</code> the tensors to create a new leaf tensor, which would support the <code>deepcopy</code> again.<br>
A few issues besides that:</p>
<ul>
<li>Don’t use <code>torch.Tensor</code> as it would be using uninitialized memory. If that’s your use case, use <code>torch.empy</code> instead.</li>
<li>Don’t use the <code>.data</code> attribute as it could yield unwanted side effects. Instead wrap your code into a <code>with torch.no_grad()</code> block and <code>copy_</code> the new data into the parameter, if needed.</li>
</ul>

Ella: <p>Thanks for your answer!</p>
<p>Generally, I try to implement something very similar to the f.conv2d layer, i.e., f.conv2d gets a 4D tensor with several layers, iterate over them one by one, and calculates convolution for each layer.</p>
<p>So, my question is how to iterate over 4D tensor one by one?</p>
<p>As you can see in my code, I got a 4D tensor of  layers (images) and iterate over the tensor with ‘double for,’ then I tried to copy results into a new 4D tensor (self.layers).</p>
<p>But I am pretty sure that is the wrong implementation and runs very slow. (The layers suppose to be with trainable parameters for gradients calculation.)</p>

ptrblck: <p>The nested loop would be slow, but should generally work as a first experiment.<br>
In your code it seems you are copying the result to an <code>nn.Parameter</code>, which sounds wrong and you should create a new output tensor instead.<br>
A faster approach would be to use <code>unfold</code> and to apply a matrix multiplication to calculate the convolution. The <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html"><code>nn.Unfold</code> docs</a> give you an example.</p>
How to monitor object detection model in production?

iamexperimentingnow: <p>Hi Everyone, I have a doubt, can someone tell me how to monitor the object detection model in production, relevant articles would be helpful.</p>
<p>I have build FasterRCNN using PyTorch, I’m moving my model to production I would like to know which statistical and which parameter we consider to monitor the model or retrain the model.</p>
<p>Earlier, I have productionized regression model into production and I have used statistical test to monitor the data drift and model drift.</p>
<p>can someone tell me how to monitor the object detection model?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
[ONNX] How to gather on a tensor with two-dim indexing?

Yasser_Khalil: <p>Hi,</p>
<p>How can I perform the following <strong>without</strong> getting a Gather node in my onnx graph? As the Gather node gives me an error in TensorRT 7.</p>
<pre><code class="lang-auto">x = data[:, x_indices, y_indices]
</code></pre>
<p>data is tensor of size[32, 64, 1024]<br>
x_indices is tensor of size [50000,] → range of indices 0 to 31<br>
y_indices is tensor of size [50000,] → range of indices 0 to 1023</p>
<p>The size of tensor x would be [32,50000]</p>
<p>Thanks in advance.</p>
Conv2d in parallel

milan_kalkenings: <p>conv2d exprects an input of size [batch_size, num_ch, width, height].</p>
<p>i have a sequence of 16 regions of interest per observation (roi pooled to same size) and thus my input is of size<br>
[batch_size, num_rois, num_ch, width, height].</p>
<p>How do I perform the same convolution per roi in parallel? my sequential version is:</p>
<p>convoluted_rois = []<br>
for roi in range(rois.size(1)):<br>
____convoluted_rois.append(self.conv(rois[:, roi, :, :]))<br>
convoluted_rois = torch.cat(convoluted_rois, dim=1)</p>

my3bikaht: <aside class="quote no-group" data-username="milan_kalkenings" data-post="1" data-topic="134991">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/milan_kalkenings/48/42748_2.png" class="avatar"> milan_kalkenings:</div>
<blockquote>
<p>batch_size, num_rois</p>
</blockquote>
</aside>
<p>Why not just .view(bs*roi, num_ch, w, h) before conv2d and .view back after?</p>
Cast error details: Unable to cast Python instance to C++ type

Vrushank98: <p>I want to convert .png files to .jpeg files with random quality factor. But it returns this error:</p>
<blockquote>
<p>image::write_file() Expected a value of type ‘str’ for argument ‘_0’ but instead found type ‘BufferedWriter’.<br>
Position: 0<br>
Value: &lt;_io.BufferedWriter name=‘/content/Results0.jpeg’&gt;<br>
Declaration: image::write_file(str _0, Tensor _1) → ()<br>
Cast error details: Unable to cast Python instance to C++ type (compile in debug mode for details)</p>
</blockquote>
<p>Here’s code:</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as T
import torchvision.io as io
import os
from PIL import Image
import random


PATH = '/content/input'
SAVE_PATH = '/content/Results'

img_files = [os.path.join(PATH, imgs) for imgs in os.listdir(PATH)]
j = 0
for i in img_files:
    
    nums = []
    image = Image.open(i)
    image = image.convert('RGB')
    image = T.ToTensor()(image)
    image = image.type(torch.uint8)
    
    r_num = random.randint(10, 96)
    nums.append(r_num)
  
    io.write_jpeg(input = image, filename = open(SAVE_PATH + f'{j}.jpeg', 'wb'), quality = r_num)
    j += 1
</code></pre>
LSTM sentence probability, how to get the sentence probability at each word and then make a sum of those?

PatricYan: <p>Hi:<br>
LSTM  output values<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8be6275260c3fc50b1954f7bf1ac422981f31ccd.png" data-download-href="https://discuss.pytorch.org/uploads/default/8be6275260c3fc50b1954f7bf1ac422981f31ccd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8be6275260c3fc50b1954f7bf1ac422981f31ccd_2_690x320.png" alt="image" data-base62-sha1="jXBt9tH1jM8xxTa5R6uydDyD3ad" width="690" height="320" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8be6275260c3fc50b1954f7bf1ac422981f31ccd_2_690x320.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8be6275260c3fc50b1954f7bf1ac422981f31ccd_2_1035x480.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8be6275260c3fc50b1954f7bf1ac422981f31ccd_2_1380x640.png 2x" data-dominant-color="F1EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1412×656 81.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
how to get the sentence probability at each word and then make a sum of those?</p>
Are there any problems with using "if isinstance(x, torch.fx.proxy.Proxy):" to supply custom hardcoded values to torch.fx for classes & functions with Dynamic Control Flow?

ProGamerGov: <p>Simplified example show below:</p>
<pre><code class="lang-auto">import torch

def test_func(h: int, w: int) -&gt; int:
    return h * w

class TestModule(torch.nn.Module):
    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        height, width = x.shape[2:]

        # Handle torch.fx
        if isinstance(x, torch.fx.proxy.Proxy):
            height, width = 224, 224
        
        
        hxw = test_func(height, width)
        
        return x * hxw


# Seems to work okay
torch.fx.symbolic_trace(TestModule())
</code></pre>
<p>Will this solution cause any issues? It seemed to work when I tested it, but I may be unaware of potential issues with this fix.</p>
<p>I want to deploy this solution in custom GoogleNet and Inception models, so I can use <code>torchvision.models.feature_extraction.get_graph_node_names</code> &amp; <code>torchvision.models.feature_extraction.create_feature_extractor</code> on them.</p>
Torchtext.vocab build_vocab_from_iterator

dimamillion: <p>torchtext.vocab build_vocab_from_iterator</p>
<p>hello, help please, gives an error</p>
<p>vocab = torchtext.vocab.build_vocab_from_iterator(iterate_corpus(all_sentenses), specials=["", “”],special_first=True)<br>
TypeError: build_vocab_from_iterator() got an unexpected keyword argument ‘specials’</p>
<p>torch version 1.7.0<br>
torchtext version 0.8.0<br>
python 3.8</p>
<pre><code class="lang-auto">tokenizer = torchtext.data.get_tokenizer("spacy", language="ru_core_news_sm")

def iterate_corpus(corpus: List[str]):
        for sentense in corpus:
         yield tokenizer(sentense)


    corpus = construct_dataset("paraphrases.xml")
    all_sentenses = corpus["text_1"].tolist() + corpus["text_2"].tolist()
    vocab = torchtext.vocab.build_vocab_from_iterator(iterate_corpus(all_sentenses), 
    specials=["&lt;unk&gt;", "&lt;pad&gt;"],
    special_first=True
    )
</code></pre>

ptrblck: <p>You would have to update <code>torchtext</code>, as this the <code>specials</code> argument was added in <a href="https://pytorch.org/text/0.10.0/vocab.html#torchtext.vocab.build_vocab_from_iterator"><code>torchtext==0.10.0</code></a>.</p>
Wierd performance of a pytorch DNN

Mohamed_Nabih: <p>I have a question and need an explanation. I trained a DNN on a dataset of speech embeddings. The validation accuracy during the training is 99.5% then I stopped the training and save the best model. When I did the inference on the test dataset the accuracy is 91% which is acceptable. The problem is when did the inference on the validation set the accuracy should be approximately the same but in my case, it is 84%. Can someone give me an explanation for this?</p>

ptrblck: <p>Make sure to use the same setup during your validation run in the training and inference script.<br>
I.e. in particular check that both approaches call <code>model.eval()</code>, use the same data preprocessing etc.</p>
Symbol lookup error: /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch.so.1: undefined symbol: __atomic_fetch_add_8

danielmanu93: <p>HI everyone, I have installed pytorch on my raspberry pi but when I try to execute my code on raspberry pi, I get the following error “symbol lookup error: /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch.so.1: undefined symbol: __atomic_fetch_add_8”. I have tried a couple of commands but seems not to get it to work. Any ideas to solve this issue is welcomed. Thanks</p>
<p>/bin/python3: symbol lookup error: /usr/local/lib/python3.7/dist-packages/torch/lib/libtorch.so.1: undefined symbol: __atomic_fetch_add_8</p>

ptrblck: <p>I think you might be hitting <a href="https://github.com/pytorch/pytorch/issues/22898#issuecomment-511692921">this issue</a> and could take a look at the solution.</p>
'NoneType' object cannot be interpreted as an integer

Omama: <p>I want to train linear classifier with RNN encoder and decoder<br>
train_fn(model, dataloaders)<br>
model = LinearClassifier(encoder=encoder, enc_type=‘rnn’, H=H, C=C)</p>
<p>but I’m getting error here while training:<br>
pbar = tqdm(enumerate(train_dataloader),total=len(train_dataloader))</p>
<p>The error is:<br>
TypeError: ‘NoneType’ object cannot be interpreted as an integer</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5bedbc8062f53b589eea602704451d6fbbcd17d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d5bedbc8062f53b589eea602704451d6fbbcd17d" title="Screen Shot 2021-10-23 at 4.09.50 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5bedbc8062f53b589eea602704451d6fbbcd17d.png" alt="Screen Shot 2021-10-23 at 4.09.50 PM" data-base62-sha1="uuSC344mF9SeImzeq13UQcmGpeB" width="690" height="99" data-dominant-color="343434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-10-23 at 4.09.50 PM</span><span class="informations">1636×236 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Check the creation of the <code>Dataset</code>, sampler, and <code>DataLoader</code> as it seems as if the <code>DataLoader</code> got a <code>None</code> input as the dataset or sampler.</p>
Using Tensor Instead of numpy array in reinforcement learning tasks

Arta_A: <p>Hi, I am trying to make an RL agent and environment, in this process I am using numpy array basically for action space which is a shape of [3]. So the main input and output of the neural net agent class which extends the torch.nn module, is a numpy ndarray. And the reward and other primary variables are numpy array too. But, the input of agent should be torch tensor. For example:</p>
<pre><code class="lang-auto">action, prediction = agent.act(state)
next_state, reward, done = env.step(action)
states.append(np.expand_dims(state, axis=0))
next_states.append(np.expand_dims(next_state, axis=0))
action_onehot = np.zeros(3)
</code></pre>
<p>In the code shown that is a function is based on numpy array like expand. And the challeng is transforming the numpy array to torch tensor and vice versa.<br>
After all, I came up with three options and now I want to know which one is better to use(or any other option you may give):<br>
1.Use the from_numpy function and convert the array to tensor <em>before inputting it to the agent</em> and then again transform it to numpy <em>after prediction</em>?<br>
2.Use the transformers(from numpy and to numpy) <em>in the forward function of the agent class</em>?<br>
3.Use <em>torch tensor instead of numpy array</em> in the whole process(which may not possible)?<br>
And I believe keeping the grad of torch tensor is crucial, So please consider this.<br>
Please explain how should I do the task.<br>
Your answer much appreciated in regard.</p>

Arta_A: <p>Would you please help <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I don’t fully understand your use case, but in case you are not using custom <code>autograd.Function</code>s, you are already breaking the computation graph, so</p>
<aside class="quote no-group" data-username="Arta_A" data-post="1" data-topic="134675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arta_a/48/42683_2.png" class="avatar"> Arta_A:</div>
<blockquote>
<p>And I believe keeping the grad of torch tensor is crucial, So please consider this.</p>
</blockquote>
</aside>
<p>might not be important (assuming your current code works fine)?</p>
<p>In any case, I would try to use PyTorch tensors and operations only, as it would also allow you to use the GPU without moving the data back and forth between the GPU and CPU.</p>
Storing activation of only one hidden layer in a NN

hema_rathore: <p>Though this is in the concatenation of my previous <a href="https://discuss.pytorch.org/t/how-to-freeze-the-vector-at-the-second-last-layer-of-shallow-model/134834/4">question</a>, but as the concept is different so I feel like asking another one here.</p>
<p>For better suggestions for my question let me rewrite the query again. I am using a Shallow neural network (a dense NN with a single hidden layer) and after the model, training needs to store the activations of the hidden layer. I got that answer in the previous question as you all can see. But now my next question is how to save the final outputs as I was training in batches. Actually, I want to store these output layer activations for all the samples in my dataset (2000). I think that is quite impossible as the model validation gives the classification results on the basis of trained coefficients only, but can we store their output activations too for that only hidden layer I am using in the network <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> sir I know you can help here.</p>
<p>Please guide.</p>

ptrblck: <aside class="quote no-group" data-username="hema_rathore" data-post="1" data-topic="134927">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hema_rathore/48/33850_2.png" class="avatar"> hema_rathore:</div>
<blockquote>
<p>Actually, I want to store these output layer activations for all the samples in my dataset (2000).</p>
</blockquote>
</aside>
<p>You could iterate the dataset and append all outputs in e.g. a list to store them. Make sure to call <code>detach()</code> on the output tensor to avoid storing the entire computation graph in this list, which would otherwise increase the memory usage.</p>
Implementation of CTCLoss

ThomasGk: <p>Hi everyone. I am studying about CTC from this wonderful article <a href="https://distill.pub/2017/ctc/" class="inline-onebox" rel="noopener nofollow ugc">Sequence Modeling with CTC</a>, and I’d like to ask something regarding PyTorch’s way of computing CTC loss.</p>
<p>In the article it says that to compute the loss, a typical way would be to sum the probability of all valid alignments. However, this might become too costly, as the valid alignments may be too many.</p>
<p>It then goes on to explain an alternative way to compute the loss, which approximates the above mentioned calculation.</p>
<p>Reading PyTorch’s <a href="https://pytorch.org/docs/stable/generated/torch.nn.CTCLoss.html#torch.nn.CTCLoss" rel="noopener nofollow ugc">documentation</a>, I think that it computes the loss uses the first way, that is summing over <em>all</em> possible alignments. Is my understanding correct, and if yes, is there a way to use the second approach described in the article, which uses a dynamic programming algorithm?</p>
<p>Thanks in advance for any insights!</p>
Reduce GPU use (flops, not memory)

OverLordGoldDragon: <p>Is it possible to reduce GPU compute use, as measured in “3D” by Task Manager? <code>scipy</code> allows specifying <code>workers</code> (CPU cores) for <code>fft</code>, for example. Memory isn’t of concern.</p>
<p>I have an intense processing pipeline that maxes out 3D, and it is to run for days, which doesn’t let me use the GPU for anything else. Input size can’t reduce any further.</p>
PIL GaussianBlur vs tfv.gaussian_blur

Separius: <p>Hi, I want to use torchvision’s gaussian_blur instead of PIL’s gaussian blur; in pil you have one sigma input; how can I translate that sigma into kernel_size and sigma of torchvision, also are the paddings the same?</p>
<p>It seems like an easy question but so far I couldn’t figure out the exact parameters even with visualization (btw, I only care about sigma when it’s between 0.1 and 2.0 (self-supervised data augmentation))<br>
I also found this <a href="https://stackoverflow.com/questions/62968174/for-pil-imagefilter-gaussianblur-how-what-kernel-is-used-and-does-the-radius-par" rel="noopener nofollow ugc">question</a> on StackOverflow.</p>
<p>The reason I’m asking is that I used PIL to train a MoCov2 on imagenet for 35 epochs and it’s performing almost 2% (absolute) better than the one with torchvision</p>
<p>and yes, I’m using different blurs for different images of the minibatch, so the only difference is the blur function</p>

tom: <p>The question is more what is the (effective) kernel size for PIL’s GaussianBlur.<br>
For any given radius, you can check experimentally:</p>
<pre><code class="lang-auto">img = torch.zeros(3, 20, 20)
img[:, 10, 10] = 1

img_pil = torchvision.transforms.functional.to_pil_image(img)
#?? torchvision.transforms.functional.gaussian_blur

img_blur_pil = img_pil.filter(PIL.ImageFilter.GaussianBlur(radius=2))
img_blur_pil_t = torchvision.transforms.functional.to_tensor(img_blur_pil)

print((img_blur_pil_t[0]&gt;0).any(dim=1).sum())

img_blur = torchvision.transforms.functional.gaussian_blur(img, 11, 2)
print((img_blur[0]&gt;0).any(dim=1).sum())
</code></pre>
<p>In general, this is a complex topic, the <a href="https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#PIL.ImageFilter.GaussianBlur" rel="noopener nofollow ugc">PIL Documentation</a> references a 12-page article (<a href="https://www.mia.uni-saarland.de/Publications/gwosdek-ssvm11.pdf" rel="noopener nofollow ugc">Gwosdek et al: Theoretical Foundations of Gaussian Convolution by Extended Box Filtering</a>) for their implementation accuracy.</p>
<p>Best regards</p>
<p>Thomas</p>
DenseNet backbone batch normalization

azer: <p>Hi,</p>
<p>I would like to change the backbone of Mask R-NN  from Resnet to DenseNet. I have followed the tutorial but I am not sure if I should freeze batch normalization. I am not using fpn.</p>
<p>Please help!</p>
<pre><code class="lang-auto">backbone= torchvision.models.densenet121(pretrained= True).features
backbone.out_channels =1024

for name, param in backbone.named_parameters():
    if 'norm' in name:
        param.requires_grad = False

anchor_generator = AnchorGenerator(sizes=((8, 16, 32, 128, 256),),
                           aspect_ratios=((0.25, 0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                output_size=7,
                                                sampling_ratio=2)

mask_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                output_size=14,
                                                sampling_ratio=2)


model = MaskRCNN(backbone, num_classes, rpn_anchor_generator=anchor_generator, 
                 box_roi_pool=roi_pooler, mask_roi_pool = mask_pooler)

</code></pre>
Should I remove Dropout layer when testing my trained model?

Reza_Akbari_Movahed: <p>Hi Everyone.<br>
I Have trained a convolutional neural network with below structure.</p>
<pre><code class="lang-auto">### Defining Network
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1 = nn.Conv2d(3,64,kernel_size=7, stride=1, padding=2)
        self.drop1 = nn.Dropout2d(p=0.1)
        self.maxpool1 = nn.MaxPool2d(5,stride=2)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(64,128,kernel_size=5, stride=1, padding=2)
        self.drop2 = nn.Dropout2d(p=0.2)
        ......
        self.fc1 = nn.Linear(3*3*64, 64)
        self.drop4 = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(64,2)
        self.drop5 = nn.Dropout(p=0.5)
        
    def forward(self,x):
        x = self.conv1(x)
        x = self.drop1(x)
        x = self.maxpool1(x)
        x = self.relu1(x)
        x = self.conv2(x)
        x = self.drop2(x)
       .....
        x = self.fc1(x)
        x = self.drop4(x)
        x = self.fc2(x)
        x = self.drop5(x)
        return x 
        
</code></pre>
<p>In this model, also, I used dropout layer to prevent overfitting problem. Actually, when testing the trained model, should I remove the Dropout layer?</p>

Latope2-150: <p>You do not need to remove the Dropout layers in testing but you need to call model.eval() before testing. Calling this will change the behavior of layers such as Dropout, BatchNorm, etc. so that Dropout layers, for example, will not affect the result.</p>

Reza_Akbari_Movahed: <p>Thanks a lot for your help dear friend.</p>

pfren: <p>Does removing the dropout layer have the same result with using model.eval()?<br>
Thanks!</p>

my3bikaht: <p>Removing layers from network just for the inference may introduce you to additional errors.<br>
It is much simpler to do model.eval()</p>