akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Returning the indices of max elements in 1D tensor

username: enesduran
cooked: <p>Hi,</p>
<p>I want a function that returns the indices of the max elements of a 1D tensor. There will not be only<br>
single max value. torch.max() method seems to return only one index, but I want multiple indexes.<br>
if a=tensor([1,4,56,6,7,56,7]) I want to get [2,5]</p>
<p>Thanks</p>
Benefit and usage of torch.nn.utils.prune

username: Oktai15
cooked: <p>What the benefit can we get from torch.nn.utils.prune now?<br>
Will you plan to replace simple tensor with sparse tensor for speedup of inference and memory size of model?</p>

username: Alexander_Shekhovtso
cooked: <p>Join the questions. Not sure if it possible to speedup multiplication of sparse tensors for now. Global structured pruning could reduce multiplication count.<br>
Who knows something about it, please check my related questions here <a href="https://discuss.pytorch.org/t/global-structured-pruning/67263" class="inline-onebox">Global structured pruning</a>.</p>

username: Oktai15
cooked: <p>I agree with you, will wait answer from PyTorch Dev <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: mmisiur
cooked: <p>I also join the question. I’ve checked the forward pass time even with drastic pruning (=0.9) and I have noticed no speedup nor model size change. Is there a way to obtain any of this?</p>

username: ani0075
cooked: <p>Check my answer here.</p><aside class="quote" data-post="4" data-topic="78633">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ani0075/48/19643_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-improve-inference-time-of-pruned-model-using-torch-nn-utils-prune/78633/4">How to improve inference time of pruned model using torch.nn.utils.prune?</a> 
  </div>
  <blockquote>
    As <a class="mention" href="/u/ypsoh">@ypsoh</a> mentioned, the torch.sparse module is still in beta. So you won’t get any speed up for a model pruned in an unstructured way unless you make use of sparse matrix operations. Hopefully torch.sparse will be part of a stable release soon.
  </blockquote>
</aside>


username: Michela
cooked: <p><a class="mention" href="/u/ani0075">@ani0075</a> is right, you’d need to rely on a specific sparse matrix algebra library for speedups.<br>
You can find my old answer to a similar question here: <a href="https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076" rel="nofollow noopener">https://github.com/pytorch/tutorials/pull/605#issuecomment-585994076</a></p>
Batching Inferencing on Images of varying size

username: abhiksark
cooked: <p>This code is working</p>
<pre><code class="lang-auto">for k, image_path in enumerate(image_list):
    image = imgproc.loadImage(image_path)
    print(image.shape)
    with torch.no_grad():
        y, feature = net(x)        
    result = image.cuda()
</code></pre>
<p>However, when I try to use data loader for the same having batch size more than 1, the code is failing as the size of images are different. I know I can resize the image but not doing it as it might change the aspect ratio of the image and might impact my results.</p>
<p>What alternative solution am i having?</p>
<p>Can we use any bucketing technique here?</p>

username: BTNug
cooked: <p>Hi <a class="mention" href="/u/abhiksark">@abhiksark</a>,</p>
<p>You are almost there!</p>
<p>According to my pass projects, I still haven’t been able to train batch cnn with different image size.<br>
What I did is to suggest you to do the following:</p>
<p><em>Step 1</em>. <strong>Resize the image into at least one same dimension</strong> (aka, you could <strong>try to make the width/height into the same size</strong> like width=800, while leaving the other w.r.t their resized-aspect ratio size).<br>
<em>Step 2</em>. <strong>Pick the highest aspect ratio</strong> that might be possible for your case (in my case it is w:h -&gt; 16:9). Hence, w:h -&gt; 800:450).<br>
<em>Step 3</em>. While doing step. 1, you will find that there is a little bit of missing pixels to match them into your highest aspect ratio. Then, we will <strong>fill it with zero padding for the rest</strong>.</p>
<p>Note:</p>
<ul>
<li>Some hack for step 3, I will create a new zero image with w:h according to your aspect ratio, which in this case is 800:450, <strong>fill the 0:800 width and 0:X height</strong> with your image, then leave the rest as zero.</li>
<li>Don’t forget to adjust your mask or annotations w.r.t. your changes.</li>
</ul>
<p>Hope it helps, cheers~</p>
When will THC_GENERIC_FILE defined?

username: tgsmdww
cooked: <p>In file torch/csrc/generic/serialization.cpp,</p>
<pre><code class="lang-auto">#ifndef TH_GENERIC_FILE
#define TH_GENERIC_FILE "torch/csrc/generic/serialization.cpp"
#else

#ifdef THC_GENERIC_FILE
#include &lt;c10/cuda/CUDAGuard.h&gt;
#endif
</code></pre>
<p>I want to know when I use cuda, how THC_GENERIC_FILE defined?</p>
Downgrade PyTorch Version or Intall PyTorch 1.5.0

username: Nick_ishere
cooked: <p>Hi! I’m trying to install PyTorch on my Ubuntu 18.04. However, I found the <code>1.5.1(stable)</code> is the only download version on the <a href="https://pytorch.org/" rel="nofollow noopener">website</a>. Then I jumped into the <a href="https://pytorch.org/get-started/previous-versions/" rel="nofollow noopener">previous version</a> page but still cannot find this one (the latest is <code>V1.2.0</code>). May I ask is there any way I can install PyTorch <code>1.5.0</code> or downgrade to it? My system details are:</p>
<ul>
<li>OS: Ubuntu 18.04</li>
<li>Python version: 3.7.7</li>
<li>PyTorch version now: 1.5.1</li>
<li>CUDA/cuDNN version: 10.1</li>
<li>GCC version: 7.5.0</li>
<li>Driver version: Nvidia 440</li>
</ul>
<p>Thank you a lot for your time in advance!</p>
<hr>
<p>(Here is a little bit more details about my motivation for downgrading. <code>1.5.1</code> version works perfectly on my system. However, when I try to install an open-source library <a href="https://pytorch-geometric.readthedocs.io/en/latest/" rel="nofollow noopener">PyTorch Geometric</a>, it showed the error:</p>
<p><code>Failed to build wheel for torch-scatter: ERROR: Command errored out with exit status 1</code>.</p>
<p>The authors say it may cause by PyTorch version and GCC version. <a href="https://github.com/rusty1s/pytorch_geometric/issues/1419" rel="nofollow noopener">ISSUE here</a>. Hence I plan to use the PyTorch <code>1.5.0</code> to match it. If you have any idea why this problem happens please feel free to let me know. Thank you a lot!)</p>

username: ptrblck
cooked: <p>Seems to have been <a href="https://github.com/rusty1s/pytorch_geometric/issues/1419#issuecomment-657263751">solved</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Nick_ishere
cooked: <p>Yes Yes. Thank you for your help, ptrblck! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Strange behavior of mean on CPU

username: czotti
cooked: <p>I was doing some test and I came across a strange behavior/“bug” ?<br>
Here is the repository to reproduce, <a href="https://github.com/czotti/pytorch_mean_test" rel="nofollow noopener">https://github.com/czotti/pytorch_mean_test</a></p>
<p>I get different results when I apply the mean function on a tensor.</p>
<pre><code class="lang-auto">&gt; features.shape
 torch.Size([5, 15, 64000])

&gt; features[0].mean(dim=-1).numpy()
 [0.45403323 0.08670517 0.02846369 0.02786237 0.02710164 0.02582995
 0.02611523 0.0251685  0.02441121 0.02369287 0.02208564 0.02194764
 0.02004337 0.03743855 0.14910352]

&gt; features[0].mean(dim=-1).numpy()[0]
 0.454033225774765

&gt; features[0, 0].mean(dim=-1).numpy()
 0.4540286958217621
</code></pre>
<p>On GPU it’s consistent I get the same output (as you can check on the repository) for the last two operation. I expect the same behavior for the CPU. Did I miss something ?</p>

username: googlebot
cooked: <p>avx, kahan summation, openmp are possible explanations</p>
Interpreting the output of torch.autograd.grad

username: alsuhr
cooked: <p>The network I am optimizing has an internal bottleneck where all activations are interpretable distributions over a map (e.g., like a heatmap). I am trying to visualize the gradients of these distributions wrt. some loss that I am optimizing. Right now, I am doing it like this:</p>
<pre><code class="lang-auto"># [1] Run a forward pass on the network, 
# getting the output logits and the internal distribution activations 
# (which are Tensors)
logits, distributions = network(inputs)

# [2] Compute the loss for this graph
loss = compute_loss(logits, targets) 

# [3] Get the gradients for the distribution tensors wrt. the loss being minimized
dist_gradients = torch.autograd.grad(loss, distributions) 

# [4] Call my own visualization code which displays the original distributions and the
# gradients layered on top to see how the distributions should change when optimizing
# the loss. Both tensors are the same size (the size of the heatmap, so in my case 25x25)
visualize_grads(distributions, dist_gradients)

# [5] Call a backward pass and apply gradients; this will only update the parameters
# used to compute the distribution tensors (and not the parameters that get from
# distributions to the logits), but that is intentional
torch.autograd.backward(distributions, dist_gradients)
optimizer.step()

# [6] Visualize the new distributions after applying the step and running another forward pass
# on the same inputs
optimizer.zero_grad()
_, new_distributions = network(inputs)
visualize_heatmap(new_distributions)
</code></pre>
<p>The problem I am facing is that the gradients coming out of <code>torch.autograd.grad</code> look like they have the opposite directionality that they should have. I verify this by comparing the difference between the visualized distributions in steps [4] (before optimization) and [6] (after optimization), and also looking at the <code>dist_gradients</code> used to optimize. For example, let’s say position [0, 0] in the distributions had a value of 0.2 before optimization, and a value of 0.4 after optimization. However, <code>dist_gradients[0][0] &lt; 0</code>, which to me, seems like it should not be increasing the value after optimization, but rather decreasing it.</p>
<p>Is it possible that I should negate the gradients before visualizing them this way? After discussing with a few others, it seems like gradients within a graph are actually opposite the intuitive (at least to me) directionality. This might be supported by looking at the code for SGD, where <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py#L112" rel="nofollow noopener">gradients for each parameter are multiplied by negative the learning rate</a> in the optimizer step.</p>
<p>Thanks!<br>
Alane</p>

username: albanD
cooked: <p>Hi,</p>
<p>Gradients give you the direction of steepest ascent. So if you are looking for a descent direction, yes you should flip the signs.<br>
As you saw, in SGD, you do <code>x = x - lr * grad</code> (with a negative sign).</p>

username: alsuhr
cooked: <p>Interesting. Thanks!! This clears up a lot.</p>
Faster-Rcnn fine tuning using wider face dataset

username: Loay_Sharaky
cooked: <p>when I try to finetune the pytorcfaster-rcnn following the official tutorial: <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a> with the Wider-Face dataset.</p>
<p>I got: RuntimeError: input and output sizes should be greater than 0, but got input (H: 1024, W: 1) output (H: 800, W: 0)</p>
<p>here is the Dataset getitem function I used:</p>
<pre><code class="lang-auto">def __getitem__(self,idx):
        img_path = self.imgs[idx]
        ann_path = self.anns[idx]
        img_id = self.images_id[idx]

        img = Image.open(img_path).convert('RGB')
        img = np.asarray(img)
        img = img/255.0
        img = np.moveaxis(img,2,0)
        img = torch.from_numpy(img).float()

        boxes ,w,h= self.__extract_boxes__(ann_path)
        masks = np.zeros([h,w,len(boxes)],dtype='uint8')
        labels=list()
        for i in range (len(boxes)):
            box = boxes[i]
            
            row_s,row_e = box[1],box[3]
            col_s,col_e = box[0],box[2]

            masks[row_s:row_e,col_s:col_e,i]=1
            labels.append(1)
        

        boxes    = torch.as_tensor(boxes,dtype=torch.float32)
        masks    = torch.as_tensor(masks,dtype=torch.uint8)
        labels   = torch.ones((len(labels),),dtype=torch.int64)
        image_id = torch.tensor([idx])
        area     = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        
        target = {}
        target['boxes']=boxes
        target['labels'] = labels
        target['masks']=masks
        target['image_id']=image_id
        target['area']=area
        
        return img,target
</code></pre>

username: ptrblck
cooked: <p>Could you print the shapes of all tensors you are using to compute the prediction as well as to calculate the loss?<br>
While the output width seems to be 0, the input width also looks suspicious as it’s only a single row.</p>

username: Loay_Sharaky
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your reply. I think I solved this problem, and the error is in the mask tensor<br>
mask shape has to be in [N,H,W], but from the above code it arranged in [H,W,N].</p>
How does one store and restore models for AutoML?

username: pinocchio
cooked: <p>I saw this warning:</p>
<pre><code class="lang-auto">warnings.warn("pickle support for Storage will be removed in 1.5. Use `torch.save` instead", FutureWarning)
</code></pre>
<p>that came from pytorch.</p>
<p>I am generating models on the fly so I do not have a class for them as the recommended suggest (<a href="https://pytorch.org/docs/master/notes/serialization.html#recommend-saving-models" rel="nofollow noopener">https://pytorch.org/docs/master/notes/serialization.html#recommend-saving-models</a>).</p>
<p>If my pickled code won’t work later, how am I suppose to save the model that is agnostic to model class code? (i.e. I do NOT have a way to do:</p>
<pre><code class="lang-auto">the_model = TheModelClass(*args, **kwargs)
</code></pre>
<p>)</p>
<p>Pickling is really a fantastic solution for my case since I just do:</p>
<pre><code class="lang-auto">child_model = db['child_model']
</code></pre>
<p>Can we keep the pickling code avaible for us please?</p>
<hr>
<p>gitissue: <a href="https://github.com/pytorch/pytorch/issues/41341" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/41341</a></p>
Element-wise batch matrix multiplication of each row of matrix with entire matrix

username: Remorax
cooked: <p>Hi,</p>
<p>I have two matrices of sizes (30, 24, 512) respectively where 30 is the batch size. Let us call them A and B.</p>
<p>Now what I need to do is this:</p>
<p>For every batch in A, I want to compute element-wise batch matrix multiplication of each row in a single batch of A with each row in a single batch of B and sum them.<br>
In other words, for every batch, I have a (24, 512) matrix on left-hand side (A) and on right-hand side (B). Now for each row (1 * 512) in A, I want to compute element-wise matrix multiplication of that row with each of the (24 * 512) rows in B, one by one, and sum them.<br>
This operation would thus result in a (1 * 512) sized vector.<br>
Done for each row in A, it would result in a (24 * 512) sized matrix and then, done for each batch it would result in (30, 24, 512) sized matrix, which is the preferred dimension of my output.</p>
<p>How do I go about doing this?<br>
I do not want to use for loops, since that would be inefficient. I also do not want to use “repeat” method for the same reason.<br>
Is there any other way of doing the same?</p>
<p>Any help will be appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thank you!</p>
Numerical error cholesky solvers

username: jmaronas
cooked: <p>Hello.</p>
<p>I am running some Gaussian processes models. When computing the posterior mean I am facing some numerical errors in a noise free scenario even using float64 precision. I would like to now if pytorch does any approximation when calling the cholesky solver method.</p>
<p>Basically if <code>X</code> are my training locations and  <code>Y</code> my observations. The posterior mean at locations <code>X*=X</code> should be equal to <code>Y</code> in a noise free scenario.</p>
<p>This posterior mean is given by:  <code>pos_mean  = K(x*,x) @ K(x,x)^{-1}Y</code> for a zero mean GP. In pytorch this is coded by:</p>
<pre><code class="lang-auto"># X represent training points and X* points where the posterior wants to be computed.
# This post asks for the case in which X*=X but the code presented is general.
# K_xsx_t represents the kernel K(X*,X). 
# K_xx represents the kernel K(X,X) 

Lxx_t  = psd_safe_cholesky(Kxx.transpose(1,2),upper = False)
K_xsx_t = K_xsx.transpose(1,2)

# compute the solver and transpose
lhs = torch.cholesky_solve(Kxsx_t, Lxx_t, upper = False).transpose(1,2)

pos_mean = torch.matmul(lhs,Y)

torch.sqrt((pos_mean-Y)**2).sum() is different from zero or from a small value</code></pre>
Manually initialize parameters?

username: jex
cooked: <p>Hi, I am newbie in pytorch.<br>
Is there any way to initialize model parameters to all zero at first?<br>
Say, if I have 2 input and 1 output linear regression, I will have 2 weight and 1 bias.<br>
I want to make all weights and bias zero at first.<br>
I couldn’t find other posts that deal with this issue.</p>

username: jpeg729
cooked: <p>All at once…</p>
<pre><code class="lang-auto">for p in model.parameters():
    p.data.fill_(0)
</code></pre>
<p>Per layer</p>
<pre><code class="lang-auto">model.layer1.weight.data.fill_(0)
model.layer1.bias.data.fill_(0)</code></pre>

username: jex
cooked: <aside class="quote no-group" data-username="jpeg729" data-post="2" data-topic="14337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<p>for p in model.parameters():<br>
p.data.fill_(0)</p>
</blockquote>
</aside>
<p>It works well, thank you!</p>
<aside class="quote no-group" data-username="jpeg729" data-post="2" data-topic="14337">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<p>model.layer1.weight.data.fill_(0)<br>
model.layer1.bias.data.fill_(0)</p>
</blockquote>
</aside>
<p>But this one doesn’t.<br>
It says my model has no layer1 attribute.</p>

username: jpeg729
cooked: <p>I was assuming your model class declared a layer1 submodule.</p>

username: jex
cooked: <p>Oh I didn’t.<br>
Actually It is very as simple as below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a181c7faf760da29b25e683c67f13d0bf378231f.png" alt="21" data-base62-sha1="n2KVvMEjB3Sq9HRSZ3YZkbXsHev" width="428" height="130"></p>

username: jpeg729
cooked: <p>Aha. You called it <code>.linear</code> rather than <code>.layer1</code>, so this will work…</p>
<pre><code class="lang-auto">model.linear.weight.data.fill_(0)
model.linear.bias.data.fill_(0)</code></pre>

username: jex
cooked: <p>It is great!<br>
Got some intuition about pytorch too.<br>
Thank you!</p>

username: Brando_Miranda
cooked: <p>does the answer for this question need to be updated because I think I read somewhere that .data was going to be removed…is that correct?</p>
<p>For example the following code gives an error:</p>
<pre><code>param.normal_(mean=mu,std=s)
RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
</code></pre>
<p>I know that doing .data works but if I recall thats going to be removed</p>

username: Brando_Miranda
cooked: <p><a class="mention" href="/u/alband">@albanD</a></p>
<p>Sorry to bug you, but what is the right way to <code>_fill</code> parameters given that the <code>.data</code> field might be going away (or is bad practice) or something like that I heard/read somewhere in the forum. What’s your take for a safe way to answer this question (i.e. for custom initialization of NNs)?</p>
Best practice for handling huge amounts of data

username: tueboesen
cooked: <p>I’m looking for the best practice for handling large amounts of data. Right now I have a single large datafile with about 250 M samples taking up around 120 GB in ASCII format, and the data is stored in a horrible way so I will need to go through the data anyway and transform it.</p>
<p>But I don’t have any experience working with such large datasets, so I don’t know what the best approach is. Should I split the data into smaller files, (will that be easier to work with/ handle for the dataloader?). Should I convert the data into a binary format? or will that be irrelevant?<br>
What is the best way of storing the data, such that I can read in only parts of the data?</p>
How to check if tensor is in collection of tensors?

username: pyscorcher
cooked: <p>Consider following:</p>
<pre><code class="lang-auto">x = torch.zeros((1,))
l = [torch.zeros((2,)), torch.zeros((2,))]
</code></pre>
<p>Now I want to check whether <code>x</code> is contained in <code>l</code>. The straightforward way would be</p>
<pre><code class="lang-auto">if x in l:
    ...
</code></pre>
<p>This does not work since <code>in</code> for lists uses <code>==</code> to check equality, but <code>==</code> is overridden in pytorch to check for entry-wise equality. We can still do it with a “manual” loop and use <code>torch.equal()</code> or <code>is</code> depending on what we want. But that doesn’t seem to be very elegant.</p>
<p>So I’m wondering, what is the recommended way to check for containedness (both in the sense of <code>==</code> and <code>is</code>) in pytorch?</p>

username: albanD
cooked: <p>Hi,</p>
<p>No I don’t think there is.<br>
Mostly because, as you mentioned,<code> equal()</code> vs <code>is</code> are two distinct things.</p>

username: pyscorcher
cooked: <p>Thank you! For the object-identity case it seems we can use</p>
<pre><code class="lang-auto">if x in set(l):
    ...
</code></pre>
<p>as this uses <code>id()</code> (contrary to lists where <code>.__eq__()</code> is used).</p>
Image Classification-training loss

username: sn07
cooked: <p>Hello,</p>
<p>I am working on image classification problem with 3 classes. There are some issues with my dataset viz, the first class is quite unbalanced ,i.,e the number of images in my whole corpus is in this proportion - 201:1000:1000 . Hence, in the training dataset which is 80% (1760 images), I used weighted random sampling to oversample the 1st class and uses some image augmentation techniques(rotation, horizontal flip, color jitter). Then I fit the Resnet-50 model and Resnet -18 model and only train the last layer. After 10 epochs, (batch size -16 and learning rate-0.0001 ), the loss and accuracy plot on Resnet-50 model are shown below.</p>
<p>My question is should I train for more epoch, there is quite overfitting here. How to solve this problem?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/507d787612fbfdac49590bab7d7a3cc682a1d12e.png" data-download-href="https://discuss.pytorch.org/uploads/default/507d787612fbfdac49590bab7d7a3cc682a1d12e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/507d787612fbfdac49590bab7d7a3cc682a1d12e_2_690x270.png" alt="image" data-base62-sha1="bu313yedV94DeClm7H295ImheN0" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/507d787612fbfdac49590bab7d7a3cc682a1d12e_2_690x270.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/507d787612fbfdac49590bab7d7a3cc682a1d12e_2_1035x405.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/507d787612fbfdac49590bab7d7a3cc682a1d12e.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1055×414 22.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Learn preprocessing parameters and use them in dataloader

username: maze09
cooked: <p>Hi all,</p>
<p>I try to implement a neural network, which also learns preprocessing parameters. The learnable parameters should be adapted based on the input and, therefore, the loading and preprocessing in the Dataloader need to use the parameters. Is there a fast and efficient way to share these parameters and update them during backprob?. Thank you in advance.</p>
How to create a self-made distribution

username: Aque
cooked: <p>Although such as Normal distribution, Bernoulli distribution and Beta distribution are already prepared by PyTorch, how can we create a self-made distribution?</p>
<p>In other words, I’d like to turn some functions which I made into a distribution instance and use .sample() method for example.</p>
<p>Does anyone know how to do this?</p>

username: googlebot
cooked: <p>If that’s coding question - you derive a class from Distribution abstract class and implement some methods of its interface. Just look at source code of some distribution and write yours similarly.</p>
<p>If you ask about .[r]sample(), that has to be implemented manually, normally that is either some transformation of samples from other distributions, or quantile based sampling.</p>
<p>There is also TransformedDistribution that may be useful.</p>
Requires grad == False doesn't save ime

username: Frida
cooked: <p>Hi<br>
I would like to train a network with fixed convolution kernels. Those that come from the initalization. However, requires_grad == False or optimizing over the specific layers I do optimize - doesn’t help in saving training time. It doesn’t make sense since the derivatives are not being changed and not need to be calculated.</p>

username: JuanFMontesinos
cooked: <p>You still need the derivatives to compute backprop.<br>
Imagine I do have layers A–&gt;B–&gt;C<br>
in order to backprop to A i do need B grads even if they are frozen.</p>
Healthcare startup in radiology is looking for a Senior Deep Learning Team Lead and a Senior Deep Learning Expert

username: MicPie
cooked: <p>Hello PyTorch community,</p>
<p><a href="https://contextflow.com/" rel="nofollow noopener">https://contextflow.com/</a>, a startup developing artificial intelligence-based software applications to support radiologists during their clinical routine, is looking for a:</p>
<p>Senior Deep Learning Team Lead<br>
<a href="https://contextflow.com/jobs/senior-deep-learning-team-lead/" class="onebox" target="_blank" rel="nofollow noopener">https://contextflow.com/jobs/senior-deep-learning-team-lead/</a></p>
<p>Senior Deep Learning Expert<br>
<a href="https://contextflow.com/jobs/senior-deep-learning-expert/" class="onebox" target="_blank" rel="nofollow noopener">https://contextflow.com/jobs/senior-deep-learning-expert/</a></p>
<p>If you you are interested send an email to <a href="mailto:jobs@contextflow.com">jobs@contextflow.com</a>.</p>
<p>Kind regards<br>
Michael</p>
Storing data on disk for training

username: Christopher_Hendra
cooked: <p>Hi everyone, I am wondering if anyone has some suggestion on how best to store the data that I am about to describe.</p>
<p>I am working with data that can be partitioned into roughly 30,000 groups, and each of these groups has between 100 to 1000 subgroups. Now, within each subgroup we have variable number of data points (i.e, group A has 100 subgroups and the first subgroup has 10 data points associated with it, group B has 200 subgroups and the second subgroup has 100 data points, etc).</p>
<p>My dataset <strong>getitem</strong> method should be able to retrieve two random elements from a particular subgroup and so currently I store each group as a separate folder, each subgroup as a separate folder within the corresponding group folder and each data point as a separate file within the corresponding subgroup folder. (I ended up with hundreds of thousands folders in the end)</p>
<p>This is easy to access from the dataset point of view but I am sure this is not the best way to represent this data. I am wondering if anyone knows what is the best practice to structure hierarchical data like this <strong>so that it is convenient to store but at the same time we can access this during training time concurrently</strong> (I tried storing this as one single hdf file but has problem accessing the file concurrently when I set the num_workers for dataloader &gt; 1)</p>
Terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 Aborted (core dumped)

username: scutych
cooked: <p>I want build some preprocess tensor in my c++ code， example as</p>
<pre><code># include "common/libtorch/include/torch/script.h"
# include "common/libtorch/include/torch/csrc/api/include/torch/torch.h"
# include &lt;iostream&gt;
# include &lt;vector&gt;
using namespace std;
int main(int argc, char *argv[]) {
    // 构建示例输入
    // std::vector&lt;torch::jit::IValue&gt; inputs;
    std::vector&lt;int64_t&gt; res_data;
    res_data.resize(1 * 3* 16 * 16);
    cout &lt;&lt; res_data.size() &lt;&lt; endl;
    torch::Tensor res_tensor = torch::from_blob(res_data.data(),{1, 3, 16, 16}, torch::kInt64);
    cout &lt;&lt; "OK" &lt;&lt; endl;
}
</code></pre>
<p>but i got the error:</p>
<pre><code>terminate called after throwing an instance of 'std::system_error'
what():  Unknown error -1
Aborted (core dumped)
</code></pre>
<p>my g++ version=5.5，using std=c++14 option，my libtorch version=1.5. Is someone know about this error ? thank you very much !!</p>
Resnext50_32_4d for imbalanced dataset to get better accuracy?

username: Santhosh_Kumar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Can i use resnext50_32_4d for imbalanced dataset to get better accuracy?<br>
is it better than resnet with less samples?,<br>
someone used the resnext without data augmentation , imbalanced data but they got good auc score with simple cross validation technique.</p>

username: ptrblck
cooked: <p>I’m not aware of any model architectures, which would be beneficial for a classification use case using an imbalanced dataset. Usually, you would use a weighted loss, add over/undersampling or manipulate the training procedure in another way to tackle the issue of an imbalanced dataset.</p>
<p>As usually, I would recommend to try out different architectures for your use case, as I don’t think there is a simple answer which model would work best for an unknown task.</p>
<p>PS: please don’t tag certain people, as this might discourage others to answer, and you might tag someone like me, who cannot give a good answer to the question. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Santhosh_Kumar
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank u so much yar for ur reply, Am working on skin cancer dataset.</p>
<p>Am new to tis forum, expected immediate answer from someone that point of time(running out of time) so mentioned u yar. <img src="https://discuss.pytorch.org/images/emoji/apple/yum.png?v=9" title=":yum:" class="emoji" alt=":yum:"></p>
<p>hey guys any idea about resnext? expecting answer from all of u too😋</p>
<p>dont mistaken me😉</p>
How to use (torch.utils.data.DataLoader) in android?

username: Micla-SHL
cooked: <p>Its testcode need （torch.utils.data.DataLoader）。 you can look PSENet Project .&gt; test_ic15.py 72 lines<br>
I have torch==1.4.0 change PSENet.pth ==&gt; PSENet.pt and model load in Android is OK。But，<br>
next I don’t know what to do.<br>
I want a little alittle translation the PSENet testcode in Android 。<br>
Sorry，my English is very poor, if you can, give me some android advice</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/mobile/android/">This Android tutorial</a> should give you a quick overview of the basic functionality.<br>
I’m currently unsure where you are stuck, so the tutorial(s) might be helpful.</p>

username: Micla-SHL
cooked: <p>hi, I have looked the android document when I asked。 What I want to say is ： I trained a text detection model and I wanted it to work on android. I’ve successfully loaded it on android, but the follow-up work (read pictures, output test results) I don’t have a clue. The original code used the (torch.utils.data.dataLoader) interface, so how I used it in android (torch.utils.data.data.DataLoader), I wanted to do nothing more than turn the test code step by step to android runable</p>
<p>Above from Bing Translation</p>

username: ptrblck
cooked: <p>You wouldn’t need to use a <code>DataLoader</code>, but could stick to the <a href="https://pytorch.org/mobile/android/#6-preparing-input">example</a> of loading a single sample.<br>
I’m not sure, if the <code>DataLoader</code> with the complete functionality is ported to the mobile API (and if not, if it’s planned to do so).</p>

username: Micla-SHL
cooked: <p>OK，thanks.  I will try something</p>
Sharing dataset between subprocess

username: mzimmer
cooked: <p>Suppose I am using a quite big Dataset which I want to load once. Using multiprocessing.pool I spawn several different processes which train different models with different hyperparameters, so this is not a distributed learning problem but rather a parallelization of different models. What I want is that each subprocess has access to the once loaded dataset and can build its own DataLoader to receive batches from.</p>
<p>How would one do this in Pytorch in the best way? I can’t figure out a way that makes the Dataset accessible to all subprocesses (except defining it in every subprocess, which is costly).<br>
Thanks!</p>

username: ptrblck
cooked: <p>A while ago I’ve created <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">this example</a> using shared arrays for multiple workers of a <code>DataLoader</code>, which might be useful for you. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: mzimmer
cooked: <p>Thanks for the fast reply, but I am not sure how this can help me. I do not want to share a dataset between multiple workers of a DataLoader. What I would need is to share a Dataset between subprocesses where each subprocess has its own dataloader, independent of the others.</p>
<p>Could I somehow make the Dataset accessible in shared memory and then build my dataloader inside the subprocess routine?</p>
ImageFolder not getting the complete dataset

username: ayrts
cooked: <p>Hello,</p>
<p>When I use <code>torchvision.datasets.ImageFolder</code> on a large dataset (~120 000 images) with two folders (fake &amp; real), the size of the total dataset pytorch uses is equal to the size of the real folder (exactly 68 850).</p>
<p>Since I output a csv file after training, I found that the number of fake images is exactly 30 000 (and no. of real images is 38 850). So <code>ImageFolder</code> only uses a subset of the actual training dataset. Anyone has similar experiences, or advice to give to debug this? I’m using <code>len(ImageFolder)</code>.</p>

username: ptrblck
cooked: <p>Could you check the file extensions of all files and make sure that they are using the <a href="https://github.com/pytorch/vision/blob/cf534fdaee048a7edce90a5577508985c58c3602/torchvision/datasets/folder.py#L149">supported formats</a>?</p>
<p>Also, could you post the folder structure here, please?</p>

username: ayrts
cooked: <p>Hi Patrick,</p>
<p>I checked the file extensions, and it seems to have been modified when it was moved (I was using <code>cp --backup=numbered</code>, so some .png files changed to .png.~1~). I only had .png files that corresponded exactly to what <code>ImageFolder</code> was able to read.</p>
<p>This was a problem on my end.</p>
<p>Thanks for the help.</p>
Pytorch DDP basic help needed

username: bfeeny
cooked: <p>I am trying to learn to use DDP, so I can move my stuff over from DataParallel to DDP.</p>
<p>I am doing this in a Jupyter notebook.  I am trying to just get something basic to run, no model, just print that its even alive.  I have 1 node, with 4 GPU’s.</p>
<pre><code class="lang-auto">def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def train(rank, param):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, param['world_size'])
    
    cleanup() 

if __name__ == '__main__':
    param['world_size'] = param['num_gpus']                                   
    mp.spawn(train, args=(param,), nprocs=param['world_size'], join=True)         
</code></pre>
<p>When I execute this, it just hangs, I get no output.  As you can see I try to print from my train function and I don’t see that at all either.</p>
Loading dataset into GPU

username: icysapphire
cooked: <p>I’m using a dataset of audio feature vectors (each sample is represented with 40 features). At the moment I create a matrix in the <em>init</em>_ method of my custom dataset and push it to GPU:</p>
<pre><code class="lang-auto">self.frame_array_device = torch.from_numpy(self.frame_array)
self.frame_array_device.to(device)
</code></pre>
<p>the dataset getitem method is defined as follows:</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
        return self.frame_array_device[idx, :]
</code></pre>
<p>Samples are drawn by the dataloader:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=batch_size,shuffle=False)
</code></pre>
<p>in the training loop I double check that the features are on the GPU:<br>
<code>batch_features = batch_features.to(device)</code></p>
<p>However, training is embarassingly slow and CPU usage is at 100%…<br>
I think that there is something wrong, could you give me some hint?<br>
Thanks a lot</p>

username: David_Alford
cooked: <p><em>I am newer to ML so take what I say with a grain</em></p>
<p>Loading the data is what I think is the slowest part.  I notice that between epochs my CPU is always 100-200% even thought my device prints as <code>Device: Tesla P100-PCIE-16GB</code>. Then, during a short portion of every epoch my GPU also goes to 100%.</p>
<p>Subscribed to see what experts have to say.</p>

username: ptrblck
cooked: <p>This line of code:</p>
<pre><code class="lang-python">self.frame_array_device.to(device)
</code></pre>
<p>won’t push the data to the <code>device</code>, if you don’t reassign the result via:</p>
<pre><code class="lang-python">self.frame_array_device = self.frame_array_device.to(device)
</code></pre>
<p>That being said, you could try to profile the data loading using the <code>data_time</code> object from the <a href="https://github.com/pytorch/examples/blob/e7870c1fd4706174f52a796521382c9342d4373f/imagenet/main.py#L282">ImageNet example</a> or alternatively you could create a random <code>CUDATensor</code> and just execute the training without the data loading at all to check the GPU utilization.</p>
How to get Sample from dataset with a certain percentage of each category

username: roy.mustang
cooked: <p>I’m quite new to PyTorch. I have a dataset which is an ImageFolder. my dataset contains some folders which are the classes and each folder has some images.<br>
I want to split the data into the train_set and test set. but I want to pick 20 percent of each class randomly and put them into test_set. I have the flowing code.</p>
<pre><code class="lang-auto">dataset = ImageFolder( './data'
                      , transform=transform)

validation_split = 0.2

indices = list(range(len(dataset)))

#split the dataset into train and test sets randomly

train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=64, sampler=train_sampler, num_workers=16)

test_loader = torch.utils.data.DataLoader(dataset, batch_size=64, sampler=test_sampler, num_workers=16)
</code></pre>
<p>How can I do that?</p>

username: ptrblck
cooked: <p>You could use <code>sklearn.model_selection.train_test_split</code> with the <code>stratify</code> argument to create the training and testing indices.</p>
How to convert images stored in torch table in the form of tensors to image dataset

username: rasharma
cooked: <p>Good Evening,</p>
<p>I have an image dataset in the form of torch table stored on my computer (dataset.data file).</p>
<p>The dataset is a torch table. It include two tensor, such as {data,label}. Where image is stored in data tensor and image name in label.</p>
<p>Can you please guide me how can I read the dataset.data file and convert the data into images.</p>
<p>or any tool which can convert</p>
<p>I am new to PyTorch.</p>
<p>I have been trying hard to convert but no success.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Could you post an example of this data format and what a “torch table” is?<br>
I’m currently unsure, if you are referring to a specific format or what the “table” is.</p>
Different performance of model on simulated and experimental data

username: bolt25
cooked: <p>I’m working on image denoising task. And I have trained 2-3 models till now. All of them perform great while denoising simulated noisy images but when it comes to images from experimental setup, it fails to achieve the results.<br>
Can anyone help me understand why this might be the case?<br>
P.S I preprocess both the images using the exact same pipeline.</p>

username: ptrblck
cooked: <p>Often “real world” cases fail, if the image statistics are not equal, e.g. if the modality to capture these images changes in some way (e.g. web images for training vs. smartphone images for the test case).</p>
<p>I’m not sure, if I understand the use case correctly, but I assume you’ve added the noise to the training images manually, while the test images are already noisy?<br>
If that’s the case, did you measure the noise distribution/statistics of the test images and tried to apply the same noise on the training images?</p>
Replacing / mapping values of tensor to be contigous integers

username: Khanh
cooked: <p>Hello,<br>
I have a tensor with values, say,<br>
tensor([ 0,  0,  1,  1,  3,  3,  7])<br>
and would like to have a tensor that replaces these numbers with contiguous integers.<br>
I want the above to become<br>
tensor([ 0,  0,  1,  1,  2,  2,  3])</p>
<p>How do I do that as effectively as possible?</p>

username: ptrblck
cooked: <p>If your original tensor is sorted (or if you could sort it), the reverse indices of <code>unique</code> would give you the desired tensor:</p>
<pre><code class="lang-python">x = torch.tensor([ 0, 0, 1, 1, 3, 3, 7])
_, idx = x.unique(sorted=True, return_inverse=True)
print(idx)
&gt; tensor([0, 0, 1, 1, 2, 2, 3])
</code></pre>
Network structure of theano's network in pytorch

username: ahmadreza9
cooked: <p>hello,<br>
Could you please help me to write below Convolution neural network in pytorch? <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/thomasjo">@thomasjo</a></p>
<pre><code class="lang-auto">l_in = lasagne.layers.InputLayer(
        shape=(None, 1, input_height, input_width),
    )

    l_hid1 = lasagne.layers.Conv2DLayer(
        l_in,
        num_filters=8,
        filter_size=4,
        stride=(2, 2),
        pad=2,
        # untie_biases=False,
        W=lasagne.init.GlorotUniform(.01),
        b=lasagne.init.Constant(0.),
        nonlinearity=lasagne.nonlinearities.rectify,
        convolution=theano.tensor.nnet.conv2d
    )

    l_hid2 = lasagne.layers.Conv2DLayer(
        l_hid1,
        num_filters=16,
        filter_size=2,
        stride=(1, 1),
        pad=1,
        # untie_biases=False,
        W=lasagne.init.GlorotUniform(.01),
        b=lasagne.init.Constant(0.),
        nonlinearity=lasagne.nonlinearities.rectify,
        convolution=theano.tensor.nnet.conv2d
    )

    l_hid3 = lasagne.layers.DenseLayer(
        l_hid2,
        num_units=20,
        # nonlinearity=lasagne.nonlinearities.tanh,
        nonlinearity=lasagne.nonlinearities.rectify,
        # W=lasagne.init.Normal(.0201),
        W=lasagne.init.Normal(.01), # std = 0.01
        b=lasagne.init.Constant(0)
    )

    l_out = lasagne.layers.DenseLayer(
        l_hid3,
        num_units=output_length,
        nonlinearity=lasagne.nonlinearities.softmax,
        # W=lasagne.init.Normal(.0001),
        W=lasagne.init.Normal(.01),
        b=lasagne.init.Constant(0)
    )
</code></pre>
<p>the special part which makes me confused is in l_hid1 and  l_hid2. (filter_size, what is its equivalent in pythorch?kernel size! )</p>

username: ptrblck
cooked: <p>Yes, <code>filter_size</code> corresponds to the <code>kernel_size</code>.<br>
From the <code>lasagne</code> docs:</p>
<blockquote>
<p><strong>filter_size</strong> :int or iterable of int<br>
An integer or a 1-element tuple specifying the size of the filters.</p>
</blockquote>
Equation of a line/linear boundaries: Udacity course

username: Shinster
cooked: <p>I’ve just started the PyTorch course on Udacity and have stumbled already at the first video! The section is linear boundaries and uses the example of university admissions to show the distribution of students who were admitted on one side of the line, and those who failed to be admitted on the other.</p>
<p>I just don’t have any idea where they came up with the equation for the line as being 2x1 + x2 - 18=0.<br>
Is this the standard equation of a line? It’s been ages since I did maths and a quick google doesn’t turn this up as the standard equation.</p>
<p>I’ve just attached an image of the distribution from the course in case this is helpful.<br>
Please be as brutal as need be if you think my maths needs to be refreshed before I continue and if this is something I should already know.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79089c53c04560035b8e9aefacf17a186a830f75.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/79089c53c04560035b8e9aefacf17a186a830f75" title="Screen Shot 2020-07-12 at 8.51.55 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089c53c04560035b8e9aefacf17a186a830f75_2_690x345.jpeg" alt="Screen Shot 2020-07-12 at 8.51.55 PM" data-base62-sha1="hgIdLZHJYAQ4zvRvTCti8eRjCRf" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089c53c04560035b8e9aefacf17a186a830f75_2_690x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089c53c04560035b8e9aefacf17a186a830f75_2_1035x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/79089c53c04560035b8e9aefacf17a186a830f75_2_1380x690.jpeg 2x" data-dominant-color="E4E5E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-07-12 at 8.51.55 PM</span><span class="informations">1504×754 157 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: tom
cooked: <p>If you look for “Linear Equation”, you’ll find <a href="https://en.wikipedia.org/wiki/Linear_equation" rel="nofollow noopener">this Wikipeida article</a> that has the same form (a_1 x_1 + a_2 x_2 + b = 0) if you set a_1 = 2, a_2 = 1, b= -18.<br>
There are various interpretations there that might help.</p>