akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Stochastic Weight Averaging examples for Computer Vision Tasks

mohit117: <p>Hello,<br>
I recently came across <a href="https://pytorch.org/docs/1.8.1/optim.html#putting-it-all-together" rel="noopener nofollow ugc">Stochastic Weight Averaging</a> in Pytorch. I found some codes which use it for Image classification and NLP tasks. But are there some example codes which use it for CV tasks involving image generation such as Image Segmentation, Denoising, Object Detection, Super Resolution or deblurring?<br>
Thanks.</p>

ptrblck: <p>I would expect the usage would be the same for the mentioned use cases or are you concerned about a specific issue which would need some adoption in the SWA usage?</p>

mohit117: <p>Thanks for reply.<br>
Not an adoption request. Just that using SWA I am getting 2dB inferior results, so wanted to go through PyTorch code of someone who has gotten good results with SWA. So far I see SWA used in image classification and NLP tasks only. Just wondering it works well for CV tasks like denoising, segmentation, super resolution, etc.</p>
<p>One last thing, sadly PyTorch does not allow SWA to be used with inbuilt Weight Normalization found in nn.utils. But i am sure this wont be the reason for 2dB lower PSNR.</p>

ptrblck: <p>Ah OK, thanks for clarifying. I haven’t used it in these setup, so let’s wait for some experts who might have experience with it.</p>
Using DataLoader when all data is stored in memory

Jendker: <p>Hello! This is my first post on this forum.</p>
<p>I wanted to ask you if you see any use of utilizing DataLoader for parallel data loading into batches if all the data is already stored in the memory? Is the data actually prepared in advance or is it split into batches still done in sequence with the main loop with only data loading split into separate loaders?</p>

ptrblck: <p>Each worker in the <code>DataLoader</code> will create a batch by calling into <code>Dataset.__getitem__</code> with the corresponding index and will add the batch to the queue.<br>
It depends on your use case, if using a <code>DataLoader</code> (with multiple workers) would yield a speedup.<br>
E.g. if you are preloading the entire dataset, are just returning the sample in <code>__getitem__</code>, and don’t need a custom <code>collate_fn</code> or a specific sampler, the <code>DataLoader</code>'s overhead might be visible and your use case might be faster by directly indexing the <code>Dataset</code> with (random) indices.<br>
However, if you are adding some transformations to each preloaded sample etc. the <code>DataLoader</code> might still be beneficial, especially if the augmentation is done in the background, so you would need to profile both approaches.</p>
Keras vs pytorch

sushmit_roy: <p>Probably not the right forum but I was trying to convert a keras to pytorch lightning. Here is the issue that I am facing</p>
<p><a href="https://github.com/PyTorchLightning/pytorch-lightning/discussions/8647" class="inline-onebox" rel="noopener nofollow ugc">Keras vs Pytorch_lightning execution time · Discussion #8647 · PyTorchLightning/pytorch-lightning · GitHub</a> any idea?</p>
Feature Vector Extraction from Densenet121

RBY: <p>Hi, I have a CNN model that classifies images of an x-rays dataset. Now, what I want is to extract the feature vectors from any convolution layer and save them so that I can use them somewhere else.</p>
<p>I saw some posts mentioning changing sub-classing the model, and overriding the <code>forward</code> method but I wasn’t successful doing that.</p>
<p>This is a part of my code:</p>
<p><strong>class DenseNet121(nn.Module):</strong></p>
<p>**    def <strong>init</strong>(self, out_size):**<br>
**        super(DenseNet121, self).<strong>init</strong>()**<br>
**        self.densenet121 = torchvision.models.densenet121(pretrained = True)**<br>
**        num_ftrs = self.densenet121.classifier.in_features**<br>
**        self.densenet121.classifier = nn.Sequential(**<br>
**            nn.Linear(num_ftrs, out_size),**<br>
**            nn.Sigmoid()**<br>
**        )**</p>
<p>**    def forward(self, x):**<br>
**        x = self.densenet121(x)**<br>
**        return x**</p>
<p>I’d appreciate a detailed explanation of what to do since I’m still new to this field.<br>
Thanks.</p>

ptrblck: <p>If overriding the <code>forward</code> method is too cumbersome, you could use forward hooks instead as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>
Loss of single batch optimization

ericlormul: <p>When debugging a neural network model, do we expect to get 0 loss on a single batch? For example, running 1000 times optimization on the same single batch input.</p>
<p>If the loss is close to 0, we can conclude the model is able to overfit on that batch.<br>
if the loss is far from 0, e.g. say start at 5.5  and end at 4.7, what does it tell us?</p>
<p>For example:</p>
<ul>
<li>it needs more training time on a single batch</li>
<li>model is not able to overfit a single batch, its design is not working</li>
<li>model has way too many parameters for a single batch data to tune</li>
<li>any other possible explanations are welcomed~</li>
</ul>
<p>Thanks!!!</p>

ptrblck: <p>The relative loss values depend on the used criterion, but overfitting a single batch should converge the loss to zero for a “standard” criterion.<br>
If your model isn’t able to overfit a single sample, the overall training is ill-defined. I.e. the model architecture, hyperparameters, etc. could be “bad” and it’s hard to give a general statement as it depends on the use case.</p>

ericlormul: <p>Thanks for replying to this “vague” question. Yes, it feels like there could be any kind of causes depending on use case.</p>
<p>If the model contains dropout and layenorm, do we expect to see zero loss for a single example during training? In this case, we should set model.eval() to expect zero loss, right?</p>
<p>what is a “standard” criterion and what’s not? Thanks.</p>

ptrblck: <p>I would not expect a perfect <code>0</code> as the final loss in any case alone due to the limited floating point precision (but a small value instead).<br>
By “standard” criterion I meant the loss functions included in <code>torch.nn</code> assuming you are not inverting them or add any constant etc.<br>
E.g. depending on your use case, you might define a “zero” loss converging towards e.g. <code>-100</code> if it fits your use case (mathematically it wouldn’t make a difference if you just subtract a constant from the loss).</p>
How to randomly change some pixels to black in any image within Pytorch?

r112: <p>How to randomly change some pixels to black in any image within Pytorch? Also, how to create a U-Net for inpainting?</p>

ptrblck: <ol>
<li>You could index the pixel location directly and set the desired pixel value (<code>0</code> for a black pixel).</li>
<li>
<a href="https://github.com/search?q=inpainting+pytorch">GitHub</a> seems to find some repositories for inpainting in PyTorch.</li>
</ol>

r112: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="128851">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ul>
<li>You could index the pixel location directly and set the desired pixel value (<code>0</code> for a black pixel).</li>
</ul>
</blockquote>
</aside>
<p>Any examples on how to do it on a regular JPG image?</p>

ptrblck: <p>Assuming your image is a numpy array (or you could transform it to one), this would work:</p>
<pre><code class="lang-python">image_array = np.random.randint(0, 255, (3, 24, 24)).astype(np.uint8)
image_array[:, 17, 23] = 0
</code></pre>

r112: <p>Can I use numpy within pytorch?</p>

ptrblck: <p>Yes, you can transform the tensor to a numpy array via <code>tensor.numpy()</code>, but would also be able to index the PyTorch tensor in the same way.</p>

r112: <p>Thanks for the help! It worked.</p>
<p>How would I change a U-Net that segments to a U-Net that inpaints?</p>
<p>Best,</p>

ptrblck: <p>I don’t know the necessary changes and would take a look at some of the linked repositories.</p>
How to batch a stereo image dataset?

sergioleno: <p>Hello guys. I’ve watched many youtube tutorials, read many posts, but can’t figure how to load my data properly. I’m working with Driving Stereo Dataset. This is the structure of the dataset folder.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1fb343166a95489c8da2ceacae72590f014e5931.jpeg" alt="estrutura" data-base62-sha1="4wqRRA5PprWrDTb6PeSEpkPGzVT" width="238" height="175"></p>
<p>When i load it with ImageFolder, it come as an “array” with 11104 positions.</p>
<pre><code class="lang-auto">train_path = './train'

transformations = transforms.Compose([
                                transforms.Resize((400,800)),
                                transforms.ToTensor(),
                                transforms.Grayscale(1)])

train_dataset = torchvision.datasets.ImageFolder(root=train_path,transform=transformations)
</code></pre>
<p>When i print some info about the data, I get these results</p>
<pre><code class="lang-auto">print(len(train_dataset))
print(len(train_dataset[0]))
print(len(train_dataset[0][0]))
print(train_dataset[0][0].type())
print(train_dataset[0][0].shape)

11104
2
1
torch.FloatTensor
torch.Size([1, 400, 800])
</code></pre>
<p>And the related images are in that positions…</p>
<pre><code class="lang-auto">left_image = train_dataset[0][0].clone().detach()
plt.figure(figsize=(5,10))
plt.imshow(left_image[0],cmap='gray')


right_image = train_dataset[2776][0].clone().detach()
plt.figure(figsize=(5,10))
plt.imshow(right_image[0],cmap='gray')

disparity_map = train_dataset[5552][0].clone().detach()
plt.figure(figsize=(5,10))
plt.imshow(disparity_map[0],cmap='gray')

depth_map = train_dataset[8328][0].clone().detach()
plt.figure(figsize=(5,10))
plt.imshow(depth_map[0],cmap='gray')
</code></pre>
<p>how can i batch this data, using disparity map or depth map as the “label”, to use this as the expected output of a network that receives left and right images as input?</p>
<p>here is the link of my notebook</p>
<p><a href="https://colab.research.google.com/drive/107f8365tHXOZDPp6z6O7QxZF-oVCRShn?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/107f8365tHXOZDPp6z6O7QxZF-oVCRShn?usp=sharing</a></p>
<p>thank you so much!</p>
Spatial Transformer Networks for 3D image

Borg: <p>Hi there,</p>
<p>I know there is an official <a href="https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html" rel="nofollow noopener">pytorch STN tutorial</a>, but that doesn’t support 3D volumetric image data since <a href="https://pytorch.org/docs/stable/nn.html#affine-grid" rel="nofollow noopener">torch.nn.functional.affine_grid</a> doesn’t support Depth. Am I correct? Do you guys know if there is any 3D STN pytorch implementation, or will pytorch support that later on?</p>
<p>Thanks!</p>

Kaushik47: <p>I am also looking for this. Were you able to find a solution? Please share if you did. Thanks.</p>

glupol: <p>Hmm, I believe the 3D STN is implemented in the new 1.9.0 verstion of pytorch.<br>
Look up the descriptions of<br>
torch.nn.functional.affine_grid<br>
and<br>
torch.nn.functional.grid_sample</p>
Copying model output to a torch.Tensor where requires_grad is True

gorjanradevski: <p>I have a model that outputs a sequence of vectors for each element in the batch, e.g., <code>[Batch size, Sequence Length, Hidden size]</code>. Then, I want to select a variable number of vectors for each element in the batch, and copy these vectors to a tensor where <code>requires_grad = True</code>. A sample code is bellow:</p>
<pre><code class="lang-python">
from torch import nn
from typing import List

class MyModel(nn.Module):
    
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc = nn.Linear(8,8)
    
    def forward(self, x: torch.Tensor, indices: List[torch.Tensor]):
        # Example indices: [torch.tensor([0,1]), torch.tensor([2,3,4])]
        out = self.fc(x)
        batch_size, _, hidden_size = out.size()
        max_num_hidden_states = max([ind.size(0) for ind in indices])
        selected_hidden_states = torch.zeros(batch_size, max_num_hidden_states, hidden_size, requires_grad=True)
        for i in range(batch_size):
            selected_hidden_states.data[i, :indices[i].size(0)] = out[i, indices[i]]
        return selected_hidden_states
    
model = MyModel()
with torch.no_grad():
    output = model(torch.rand(2, 5, 8), [torch.tensor([0,1]), torch.tensor([2,3,4])])
     
</code></pre>
<p>The questions I have w.r.t. this are:</p>
<ol>
<li>If I train such model, would the gradients be backpropagated in the rest of the model parameters?</li>
<li>Why does <code>output.requires_grad = True</code>, when I explicitly state <code>torch.no_grad()</code>?</li>
<li>The way I’m doing this (which it doesn’t seem to work as expected as of now) seems too hacky and wrong. What is the proper way to achieve what I want?</li>
</ol>
<p>I’m aware of <a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442/2">this answer</a>, which approves my way doing it (at least it seems like it), but still it looks hacky to me.</p>
<p>As this is my first question in the forum, please let me know if I should provide further details <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Cheers!</p>

tom: <p>That is from a time long gone and anwering a different question.</p>
<ol>
<li>No. Between creating a new tensor requiring grad and using <code>.data</code>, which you never should these days, you created a new leaf which will accumulate <code>.grad</code>.</li>
<li>Because you requested it. <code>no_grad</code> signals that you do not need the grad, it does not include guarantees about the <code>requires_grad</code> of the result.</li>
<li>If the utility function does not work for you, dropping the <code>requires_grad</code> and the <code>.data</code> should do the trick.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>

gorjanradevski: <p>Hi Tom! Thanks, I actually made things more complicated then they are <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Ignore old GPU when building Pytorch?

yahado: <p>I’ve a old which isn’t supported<br>
(nvcc fatal   : Unsupported gpu architecture ‘compute_30’),<br>
and a new one which is fine.<br>
Building CUDA it complains about the unsupported GPU, is there a way to ignore the old one?</p>

JosueCom: <p>When choosing CUDA device, you can detrmine the GPU by doing <code>torch.cuda.set_device([device])</code>. Read more here; <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_device.html#torch.cuda.set_device" rel="noopener nofollow ugc">link</a>.</p>
<p>Feel free to read <a href="https://discuss.pytorch.org/t/how-to-change-the-default-device-of-gpu-device-ids-0/1041">this thread</a> too for alternative options such as setting <code>CUDA_VISIBLE_DEVICES</code></p>
<aside class="quote" data-post="1" data-topic="1041">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/51bf81/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-change-the-default-device-of-gpu-device-ids-0/1041">How to change the default device of GPU? device_ids[0]</a> 
  </div>
  <blockquote>
    How to change the default device of GPU? 
for some reason ,I can not use the device_ids[0] of GPU, 
I change the following code:（in data_parallel.py） 
if output_device is None:
output_device =device_ids[0]

to 
if output_device is None:
output_device =device_ids[1]

but it still seem to used the device_ids[0] 
all tensors must be on devices[0]？ 
How to change it？
  </blockquote>
</aside>


yahado: <p>Using <code>CUDA_VISIBLE_DEVICES</code> worked.</p>
How to visualization for heat map for the input image and reconstruction image

sripathisony_sony: <p>How to visualization for heat map for the input image and reconstruction image</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please suggest me</p>

ptrblck: <p>What should this heatmap show exactly? Would you like to visualize some gradient/update information?<br>
If so, then I think <a href="https://captum.ai/">Captum</a> might be useful.</p>

sripathisony_sony: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> …thanks , i am trying to visualize reconstruction image using heat map. i am facing below error:<br>
‘’’<br>
new_imgs = decoder(resnet_vae, device, zz)</p>
<p>fig = plt.figure(figsize=(10, 10))</p>
<p>plt.subplot(1, 2, 1)</p>
<p>plt.imshow(X)</p>
<p>plt.title(‘original’)</p>
<p>plt.axis(‘off’)</p>
<p>plt.subplot(1, 2, 2)</p>
<p>plt.imshow(new_imgs)</p>
<p>plt.title(‘reconstructed’)</p>
<p>plt.axis(‘off’)</p>
<p>plt.savefig("./reconstruction_{}.png".format(exp), bbox_inches=‘tight’, dpi=600)</p>
<p>plt.show()</p>
<p>imgplot = sns.heatmap(new_imgs, xticklabels=False, yticklabels=False) ‘’’</p>
<p>error:ValueError: Must pass 2-d input. shape=(224, 224, 3)</p>

ptrblck: <p><code>seaborn.heatmap</code> expects a 2D array as described in the <a href="https://seaborn.pydata.org/generated/seaborn.heatmap.html">docs</a> so you won’t be able to pass an image in the shape <code>[224, 224, 3]</code> to it.</p>
The confidence loss of no object is surprisingly increasing

sunshine1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dd2e1b243d8c63647c49e71434ffefb57217883.png" data-download-href="https://discuss.pytorch.org/uploads/default/4dd2e1b243d8c63647c49e71434ffefb57217883" title="Screenshot 2021-08-11 at 22-05-29 TensorBoard"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd2e1b243d8c63647c49e71434ffefb57217883_2_690x392.png" alt="Screenshot 2021-08-11 at 22-05-29 TensorBoard" data-base62-sha1="b6szM5hXcYZrMqoxJ4krKDIsUQb" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd2e1b243d8c63647c49e71434ffefb57217883_2_690x392.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd2e1b243d8c63647c49e71434ffefb57217883_2_1035x588.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd2e1b243d8c63647c49e71434ffefb57217883_2_1380x784.png 2x" data-dominant-color="F1E4E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-08-11 at 22-05-29 TensorBoard</span><span class="informations">1444×822 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It is clear that the confidence loss of no object is increasing,and the other loss is decreasing. I don’t think that this is caused by learning rate, because I have set it enough small.</p>
Same initialization values for wight and bias for several trainings

Farhadjavan: <p>Hi All,</p>
<p>I trained a CNN model with my original data and now I want to train it again with my original data + augmented data. I am using pytorch. I really appreciate if you let me know how can I get same initialization values for wight and bias when training the CNN model with original data + augmented data.</p>

gphilip: <p>Did you try setting the random seed to a fixed number? You can set this seed as follows:</p>
<p><code>torch.manual_seed(0)</code></p>
<p>See also: <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility</a>.</p>

Farhadjavan: <p>Thank you very much.</p>

gphilip: <p>Did it work? It would help others if you report what happened.</p>

Farhadjavan: <p>Yes, when I check model.stat_dict(), I see that I get same values for weight and bias.</p>

gphilip: <p>Cool. It will help others if you mark my reply which helped you, as the solution for your question.</p>
Save and load model.state_dict() successfully, but with the same input, the outputs ars differnt

crane: <p>hi, i save the quantization model with</p>
<pre><code class="lang-auto">torch.save(model.state_dict(),"quanted_model.pkl"
</code></pre>
<p>and load it with</p>
<pre><code class="lang-auto">model.fuse_model()
model.encoder.qconfig = torch.quantization.get_default_qconfig('fbgemm')
torch.quantization.prepare(model.encoder, inplace=True)
torch.quantization.convert(model.encoder, inplace=True)
model.load_state_dict(torch.load(modelfile,map_location='cpu'))

</code></pre>
<p>the quantization step is same.</p>
<p>when i pass the calibrate step, i can get the same output from the load model, however , after i do the calibrate, the output is different betwen the reloaded model and the quantization model.</p>
<p>I compare the state_dict() and the metadata, but, they are same.</p>
<p>my model is transformer.</p>

crane: <p>ok, this is because the new parameter of QuantizedLayerNorm didn’t save.</p>

Leroy: <p>hello, crane. How do you quantize the layer_norm module.  I try to use this code to convert the encoder-decoder transformer from fp32 to int8, but it seems the layer_norm is still fp32.</p>
<blockquote>
<p>from torch.quantization import QuantStub, DeQuantStub, float_qparams_weight_only_qconfig, default_qconfig</p>
<h1>eager mode</h1>
<p>backend = “fbgemm”<br>
<span class="hashtag-raw">#model</span>.qconfig = torch.quantization.get_default_qconfig(backend)<br>
model.qconfig = default_qconfig<br>
model.encoder.embeddings.qconfig = float_qparams_weight_only_qconfig<br>
model.decoder.embeddings.qconfig = float_qparams_weight_only_qconfig</p>
<p>model_fp32_prepared = torch.quantization.prepare(model)<br>
model_int8 = torch.quantization.convert(model_fp32_prepared)</p>
</blockquote>
<p>And when I try to inference the model, it raise error about <code>Could not run 'quantized::layer_norm' with arguments from the 'CPU' backend.</code></p>

crane: <p>sorry,  I think u can upgeade your pytorch</p>
Passing models, optimizers and schedulers to functions

char-t: <p>Hi!</p>
<p>Bit confused about passing models, optimizers and schedulers to functions.</p>
<p>I have a function called train_epoch(model, optimizer, scheduler, …). Should the model in the calling function get updated by this function without me returning the model and reassigning it?</p>
<p>My (imperfect) understanding of python is that the function would create an instance of the objects and changes would only effect this scope (not the calling function). However my code currently does not return the model ect and training does appear to progress as expected.</p>
<p>Any insights much appreciated!</p>

gphilip: <p>Try writing a function that accepts a list as argument and changes some elements of this list. Now initialize a list and pass it to this function. Print the list after the function returns. What do you observe?</p>

char-t: <p>Ah yes, make sense now! Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Why divide noise with batch size?

Jiankai_JIN: <pre><code class="lang-auto">for p, clip_value in zip(params, clip_values):
    noise = self._generate_noise(clip_value, p)
    if self.loss_reduction == "mean":
        noise /= batch_size
    if self.rank == 0:
        p.grad += noise
</code></pre>
<p>the noise is added to the averaged gradient, why should it be divided by the batch size?<br>
since the output is the final grad, should it be added with noise (without being divided) directly?</p>

ilyamironov: <p>Compare two cases. If there’s no reduction (self.loss_reduction == “sum”), then we want to add noise calibrated to the clipping norm X. Indeed, let the gradients be g_1,…,g_B. The sum is g_1 + … + g_B, and to make it private the additive noise is sampled from the Gaussian distribution N(0, sigma^2 * C^2) so that it masks presence or absence of any one gradient vector.</p>
<p>If the reduction function is mean, then the output is (g_1 + … + g_B) / B. What should the additive noise be in this case? I think it’s pretty obvious that it is must be a scaled down (by a factor of B) noise from before. The only difference between the two cases is the scaling parameter, and it should be applied equally both to the sensitive inputs and the noise. Right?</p>

Jiankai_JIN: <p>Cool, great thanks Ilya</p>
Getting Subgroup Ranks Info in Custom Backend

esaliya: <p>Hi All,</p>
<p>Looking at the creation of a sub group using <code>new_group(ranks, backend)</code> construct, I see it goes through <a href="https://github.com/pytorch/pytorch/blob/b9f02778b29474ad47bc4f44d727a922bc469853/torch/distributed/distributed_c10d.py#L596" rel="noopener nofollow ugc">_new_process_group_helper()</a>.</p>
<p>I am interested in getting this call to a custom backend, which I see is delegated to the registered process group creation method in the same method at <a href="https://github.com/pytorch/pytorch/blob/b9f02778b29474ad47bc4f44d727a922bc469853/torch/distributed/distributed_c10d.py#L724" rel="noopener nofollow ugc">L724</a></p>
<p>However, it doesn’t seem to forward the participating ranks list to the method. Any help on how to get this information in the custom backend during new_group() creation?</p>
<p>Thanks,</p>

H-Huang: <p>Perhaps I am misunderstanding the question, but the rank is passed into the method and the rank argument is there in the code snippet you sent. For a third party backend you need to register it and perform init process group on all the ranks (<a href="https://pytorch.org/docs/master/distributed.html#third-party-backends" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a>). Then you can use the collectives as usual.</p>

esaliya: <aside class="quote no-group" data-username="H-Huang" data-post="2" data-topic="128067">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h-huang/48/35598_2.png" class="avatar"> H-Huang:</div>
<blockquote>
<p>is there in the code snippet you sent. For a third party backend you need to register it and perform init process group on</p>
</blockquote>
</aside>
<p>Thanks, yes, I am registering a third-party backend as in the document you’ve shared.</p>
<p>What I am referring to is the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc">new_group()</a> method in a ProcessGroup.</p>
<p>This allows to create a subgroup from the default process group by taking on a list of ranks. This list is referred to as the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="noopener nofollow ugc">group_ranks</a> in the _new_process_group_helper() method I pointed above.</p>
<p>My question was that this group_ranks list is not passed to the custom backend method. Instead, only the rank and world_size are passed.</p>
<p>Sees like a missing feature here.</p>

esaliya: <p>Confirmed, this is a missing feature <a href="https://github.com/pytorch/pytorch/pull/28068#issuecomment-893100091" class="inline-onebox" rel="noopener nofollow ugc">add c10d dynamic loading mechanism and unit test by ftian1 · Pull Request #28068 · pytorch/pytorch · GitHub</a></p>
ValueError: Expected input batch_size (1) to match target batch_size (160)

Mukesh1729: <p>Dataset: CAMVID<br>
Input: torch.Size([1, 3, 160, 160])<br>
Target: torch.Size([160, 160, 3])<br>
Predictions torch.Size([1, 3, 160, 160])</p>
<p>Train code:</p>
<p>model.train()<br>
for idx in tqdm(range(EPOCHS)):<br>
train_images, train_masks = torch.tensor(train_img[idx]), torch.tensor(train_lbl[idx])</p>
<pre><code>inputs = Variable((train_images).unsqueeze(0), requires_grad=True).float().permute(0, 3, 1, 2)
targets = Variable((train_masks), requires_grad=True).long()

print('Input:',inputs.shape)
print('Target:',targets.shape)

predictions = model(inputs)
print('Predictions',predictions.shape)
loss = criterion(predictions, targets)
</code></pre>
<p>Network:<br>
super(UNet, self).<strong>init</strong>()<br>
self.pool = torch.nn.MaxPool2d((2,2))<br>
self.up_samp = torch.nn.Upsample(scale_factor=2, mode=‘nearest’)<br>
self.down_conv1 = down_block(3, 32)<br>
self.down_conv2 = down_block(32, 64)<br>
self.down_conv3 = down_block(64, 128)<br>
self.down_conv4 = down_block(128, 256)<br>
self.down_conv5 = down_block(256, 512)<br>
self.down_conv6 = down_block(512, 1024)<br>
self.up_conv1 = up_block(1024, 512)<br>
self.up_conv2 = up_block(512, 256)<br>
self.up_conv3 = up_block(256, 128)<br>
self.up_conv4 = up_block(128, 64)<br>
self.up_conv5 = up_block(64, 32)<br>
self.out = out(32, 32)</p>
<p>What should my targets looks like?<br>
How should i build the evaluation part for multi class classification?<br>
Thanks</p>

ptrblck: <p>A multi-class classification would usually contain a single class index for each sample.<br>
For the posted input shape with a batch size of <code>1</code>, your target would have the shape <code>[batch_size=1]</code> and contain a value in <code>[0, nb_classes-1]</code> assuming you are using <code>nn.CrossentropyLoss</code>.</p>

Mukesh1729: <p>Thanks for getting back. I tried to remove the the 3 from the target and replaced it with a 1 since it was in NumPy before passing it into a tensor and it worked . Would you consider it to be the right way to do it?</p>
[c++] how to show data image from batch

111547: <p>Hi, I am trying to make a classification with custom image data.</p>
<p>I made a dataloader like as below a input data part.</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; process_images(std::vector&lt;std::string&gt; list_images)
{
	std::vector&lt;torch::Tensor&gt; tensor_images;
	for (auto image : list_images)
	{
		cv::Mat img = cv::imread(image);
		cv::resize(img, img, cv::Size(224, 224));

		torch::Tensor tensor_image = torch::from_blob(img.data, { img.rows, img.cols,3 }, at::kByte);
		tensor_image = tensor_image.toType(at::kFloat);
		tensor_image = tensor_image.div_(255);

		tensor_image = tensor_image.permute({ 2, 0, 1 });
		tensor_images.push_back(tensor_image);
	}
	return tensor_images;
}

</code></pre>
<p>and in the evaluation part, I wrote code like this.</p>
<pre><code class="lang-auto">for (torch::data::Example&lt;torch::Tensor, torch::Tensor&gt;&amp; batch : *data_loader_val) {
	auto data = batch.data.to(device);
	auto targets = batch.target.to(device);

	torch::Tensor outputs = resnet.forward(data);
	data = data.cpu().contiguous();
	data = data.permute({ 0, 2, 3, 1 });
	float* gt = data.data_ptr&lt;float&gt;();

	char name[80];
	std::cout &lt;&lt; outputs.sizes()[0] &lt;&lt; std::endl;
	for (auto i = 0; i &lt; outputs.sizes()[0]; i++)
	{
		auto gttmp = gt + i * 224 * 224 * 3;
		cv::Mat gtmat(224, 224, CV_32FC3, gttmp);
		cv::cvtColor(gtmat, gtmat, cv::COLOR_BGR2RGB);
		gtmat.convertTo(gtmat, CV_8UC3, 255);

		sprintf_s(name, "gt%d", i);
		cv::imshow(name, gtmat);

		std::cout &lt;&lt; targets[i].item&lt;int64_t&gt;() &lt;&lt; " ";
	}
	cv::waitkey(0);
}
</code></pre>
<p>But I’ve got the image not the color and not the original one. They are cloned one to 9 images.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0795aba0cfd142964734fc7f483d27bbd1a20cd.jpeg" alt="image" data-base62-sha1="pba2c5ZjOW3i1zMwgi0vNkh3Mwl" width="228" height="258"></p>
<p>How can I see that properly?</p>

ptrblck: <p>The output looks interleaved, so maybe you need to call <code>contiguous()</code> after <code>permuting</code> the image to channels-last and before creating the OpenCV mats.</p>
Epoch and iteration

sunshine1: <p>Should It output loss to tensorboard each epoch or each iteration? Why doesn’t the loss drop continuously?</p>

ptrblck: <ol>
<li>It depends on your use case and how many loss values you would like to plot.</li>
<li>Some minibatches might contain “hard” samples, which could increase the loss for this iteration. Generally there is no guarantee to decrease the loss in each iteration using (mini-batch) SGD.</li>
</ol>
Transfer Learning train two last layer (Bloc)

Chamroukhi: <p>Hello,<br>
it is possible to set params in layer before fully-connected layer to train params like case 2 in the table ?<br>
thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97f4b2c7365007472198f6cd9ab638f86720b6e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/97f4b2c7365007472198f6cd9ab638f86720b6e5" title="Capture d’écran 2021-08-09 à 12.48.56"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f4b2c7365007472198f6cd9ab638f86720b6e5_2_690x245.png" alt="Capture d’écran 2021-08-09 à 12.48.56" data-base62-sha1="lGgmqUvquT5cEKfKwNiPhEYSvLT" width="690" height="245" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97f4b2c7365007472198f6cd9ab638f86720b6e5_2_690x245.png, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97f4b2c7365007472198f6cd9ab638f86720b6e5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/97f4b2c7365007472198f6cd9ab638f86720b6e5.png 2x" data-dominant-color="EDEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d’écran 2021-08-09 à 12.48.56</span><span class="informations">893×318 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Yes, you can freeze specific layers and train others by manipulating the <code>.requires_grad</code> attribute of each parameter (setting it to <code>False</code> would freeze the parameter). The <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">Finetuning tutorial</a> would give you an example.</p>
How to ensure augmentation applied to each class

Aayushktyagi: <p>Hi,</p>
<p>I am applying edge operation on 10% of images on the dataset. Suppose these are 6 classes.</p>
<p><code> How do I ensure that edge operation is applied to samples from each class?</code><br>
(each class should have at least 1 images with edge operation applied to it)<br>
Sample code below:</p>
<pre><code class="lang-python">class cannyedge(object):
    def __init__(self, threshold1 = 100, threshold2 = 200):
        self.threshold1 = threshold1
        self.threshold2 = threshold2
        
    def __call__(self, sample):
        prob = np.random.random_sample()
        if prob &lt; 0.1:
            sample = np.array(sample)
            edges = cv2.Canny(image=sample, threshold1=self.threshold1, threshold2=self.threshold2) 
            edge3c = np.zeros_like(sample)
            edge3c[:,:,0] = edges
            edge3c[:,:,1] = edges
            edge3c[:,:,2] = edges
            invedge = cv2.bitwise_not(edge3c)

            return invedge
        else:
            return sample

color_jitter = transforms.ColorJitter(0.8, 0.8, 0.8, 0.2)
data_transforms = transforms.Compose([transforms.RandomResizedCrop(size=IMAGE_SIZE),
                                              transforms.RandomHorizontalFlip(),
                                              transforms.RandomApply([color_jitter], p=0.5),
                                              transforms.RandomGrayscale(p=0.5),
                                              transforms.RandomRotation(30),
                                              cannyedge(),
                                              transforms.ToTensor(),
                                              
] )
</code></pre>

ptrblck: <p>Since you are randomly applying the transformations you won’t have a guarantee to apply it, but could estimate the likelihood of getting a transformed sample for each class.<br>
Based on your description you could e.g. apply the transformations separately to at least one sample per class and let the other samples use the random process.</p>
Network not training for image alignment task

stuffedfrog27: <p>Hello,</p>
<p>I am currently trying to use deep learning to align pairs of images together (also known as registration). I am using a model similar to that of an autoencoder. The input is 2 times the size of one image flattened (since a pair of images need to be inputted, the pair is simply concatenated and then inputted into the network during training per sample). The output is the size of one image which is what the intended output should be (since the task of registration is to output a registered image).</p>
<p>The problem I am having is that my network is not “learning” in that, during each iteration of the train loop, the exact same tensor is outputted when evaluated by the network. This leads me to believe my network is not learning. How can I go about fixing this? I have posted my training loop below.</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=0.001)

loss_values = [] 
outputs_list = []
for epoch in range(3): 
    for i, data in enumerate(trainloader, 0):
        fixed_sample = data[0][0]
        moving_sample = data[1][0]

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        concat_np = np.concatenate((fixed_sample, moving_sample), axis=None)
        outputs = net(torch.tensor(concat_np, dtype=torch.float32))
        outputs_list.append(outputs)
        loss = criterion(outputs, torch.tensor(torch.flatten(fixed_sample), dtype=torch.float32))
        loss.backward()
        optimizer.step()
        loss_values.append(loss)
</code></pre>
<p>Whenever I evaluate the inputs of <code>net</code> (stored in <code>outputs</code>), it is the same for each iteration of the loop.</p>

stuffedfrog27: <p>Bumping this post here.</p>

ptrblck: <p>You could try to overfit a small dataset first (e.g. just 10 examples) by playing around with some hyperparameters and make sure your model is able to do so. Once this is done, you could try to scale up the use case again. In case the overfitting approach is not working, your training script might have some bugs (which are not shown in the posted code snippet) or the general approach might not work with the proposed architecture etc.</p>
Only one element tensors can be converted to Python scalars? I am getting this error while preparing my dataset for DataLoader

K172388_Tariq_Hussai: <aside class="onebox stackexchange" data-onebox-src="https://datascience.stackexchange.com/questions/99928/value-error-only-one-element-tensors-can-be-converted-to-python-scalars-how-ca">
  <header class="source">

      <a href="https://datascience.stackexchange.com/questions/99928/value-error-only-one-element-tensors-can-be-converted-to-python-scalars-how-ca" target="_blank" rel="noopener nofollow ugc">datascience.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/112583/tariq-hussain" target="_blank" rel="noopener nofollow ugc">
    <img alt="Tariq Hussain" src="https://lh4.googleusercontent.com/-jqzIXxZU5wY/AAAAAAAAAAI/AAAAAAAAAAA/AMZuucmEhtCxnJrTaxjt3AzKfFONr3AUiQ/s96-c/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://datascience.stackexchange.com/questions/99928/value-error-only-one-element-tensors-can-be-converted-to-python-scalars-how-ca" target="_blank" rel="noopener nofollow ugc">Value Error: only one element tensors can be converted to Python scalars? How can I solve it?</a>
</h4>

<div class="tags">
  <strong>python, deep-learning, tensorflow, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://datascience.stackexchange.com/users/112583/tariq-hussain" target="_blank" rel="noopener nofollow ugc">
    Tariq Hussain
  </a>
  on <a href="https://datascience.stackexchange.com/questions/99928/value-error-only-one-element-tensors-can-be-converted-to-python-scalars-how-ca" target="_blank" rel="noopener nofollow ugc">04:18AM - 11 Aug 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


JosueCom: <p>Not sure if this is the root of the issue but <code>__len__()</code> should return an integer, not a string or anything else</p>
Pixel Weight Map for CrossEntropyLoss

John_Doe: <p>Hello,</p>
<p>I am trying to implement a weighted CrossEntropyLoss with different weight for each pixel in the input image.</p>
<p>My code currently looks like this :</p>
<pre><code class="lang-auto">import torch

batch_size = 8
out_channels = 3
W = 128
H = 128

# Initialize logits etc. with random
logits = torch.FloatTensor(batch_size, out_channels, H, W).normal_()
target = torch.FloatTensor(batch_size, H, W).random_(0, out_channels)

target = target.long()

# for exemple only, weight will have other values but keep this size
weights = torch.ones(batch_size, H, W)

loss_fn = torch.nn.CrossEntropyLoss(reduction="none")
raw_loss = loss_fn(logits, target)
weighted_loss = weights * raw_loss

loss = torch.sum(weighted_loss.flatten(start_dim=1), axis=0)
loss = torch.mean(loss)
</code></pre>
<p>I wanted to be sure that there aren’t any mistakes here.</p>
<p>Thank you.</p>

ptrblck: <p>The code looks generally alright. Some minor points:</p>
<ul>
<li>Don’t use <code>torch.FloatTensor</code>, but the factory methods (e.g. <code>torch.empty(...).normal_()</code>, <code>torch.randn(...)</code> etc.)</li>
<li>I’m unsure about the logic to sum the loss in <code>dim0</code> and calculating the <code>mean</code> afterwards, but assume it fits your use case (in e.g. weighted <code>nn.CrossEntropyLoss</code> you would normalize with the weights if <code>reduction='mean'</code> is used)</li>
</ul>
The input data sometimes becomes NaN after .cuda()

63days: <p>Hi, all. I’m struggling a NaN issue. Any thoughts would be appreciated.<br>
When I feed the point cloud data to GPU by calling .cuda(), it becomes NaN very rarely. Code used for debugging is below.</p>
<pre><code class="lang-auto">import torch
import h5py
from torch.utils.data import DataLoader, Dataset

class PCDataset(Dataset):
    def __init__(self):
        self.pc_data = h5py.File("shapenet.h5", "r")["data"][:, :2048]

    def __getitem__(self, index):
        return self.pc_data[index]

    def __len__(self):
        return len(self.pc_data)

def main():
    dataset = PCDataset()
    data_loader = DataLoader(dataset, batch_size=64, shuffle=False, num_workers=4, pin_memory=True, drop_last=True)

    while True:
        for batch_idx, pc in enumerate(data_loader):

            if torch.any(~torch.isfinite(pc)):
                print("cpu contains NaN")
                nan_index = torch.nonzero(~torch.isfinite(pc), as_tuple=True)
                print("cpu:", pc[nan_index])

            pc_cuda = pc.cuda()
            if torch.any(~torch.isfinite(pc_cuda)):
                print("cuda contains NaN")
                nan_index = torch.nonzero(~torch.isfinite(pc_cuda), as_tuple=True)
                print("cuda:", pc_cuda[nan_index])
                print("cpu:", pc[nan_index])
                pc_from_dl = list(iter(data_loader))[batch_idx]
                print("from dl cpu:", pc_from_dl[nan_index])
                print("from dl cuda:", pc_from_dl[nan_index].cuda())
                return

if __name__ == "__main__":
    main()
</code></pre>
<p>Output:<br>
cuda contains NaN<br>
cuda: tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],<br>
device=‘cuda:0’)<br>
cpu: tensor([ 0.2349, -0.2397, -0.1102,  0.0039,  0.1992,  0.3378,  0.0317, -0.3546,<br>
0.2329, -0.1955, -0.3463,  0.2202,  0.2179,  0.0039,  0.0975,  0.2006,<br>
0.0394,  0.0548, -0.2069,  0.1202, -0.2385])<br>
from dl cpu: tensor([ 0.2349, -0.2397, -0.1102,  0.0039,  0.1992,  0.3378,  0.0317, -0.3546,<br>
0.2329, -0.1955, -0.3463,  0.2202,  0.2179,  0.0039,  0.0975,  0.2006,<br>
0.0394,  0.0548, -0.2069,  0.1202, -0.2385])<br>
from dl cuda: tensor([ 0.2349, -0.2397, -0.1102,  0.0039,  0.1992,  0.3378,  0.0317, -0.3546,<br>
0.2329, -0.1955, -0.3463,  0.2202,  0.2179,  0.0039,  0.0975,  0.2006,<br>
0.0394,  0.0548, -0.2069,  0.1202, -0.2385], device=‘cuda:0’)</p>
<p>It’s really strange that the data in cpu doesn’t have any problem but the data just after .cuda() becomes NaN from time to time and even the data reloaded from a dataloader has no problem. My environment is the following.</p>
<p>–Environment–<br>
python 3.8<br>
torch 1.7.1+cu110<br>
NVIDIA-SMI: 460.91.03<br>
CUDA Version: 11.2<br>
GPU: RTX 3090</p>

ptrblck: <p>Could you update to the latest stable PyTorch release (or the nightly) and rerun your script, please?</p>
About nn.Linear after nn.LSTM

last: <p>I have one question about the input of nn.Linear after nn.LSTM.<br>
I have the two different cases about the input of nn.Linear after nn.LSTM using same dataset in time-series, one using output of nn.LSTM as the input of the following nn.Linear, and the other using the hidden state of nn.LSTM as the input of the following nn.Linear.<br>
are there any reasons for these cases?</p>
<p>the case with the output of nn.LSTM as the input of the following nn.Linear,</p>
<p>class LSTM(nn.Module):<br>
def <strong>init</strong>(self, input_size=1, hidden_layer_size=100, output_size=1):<br>
super().<strong>init</strong>()<br>
self.hidden_layer_size = hidden_layer_size</p>
<pre><code>    self.lstm = nn.LSTM(input_size, hidden_layer_size)

    self.linear = nn.Linear(hidden_layer_size, output_size)

    self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size),
                        torch.zeros(1,1,self.hidden_layer_size))

def forward(self, input_seq):
    lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq) ,1, -1), self.hidden_cell)
    predictions = self.linear(lstm_out.view(len(input_seq), -1))
    return predictions[-1]
</code></pre>
<p>the case with the hidden state of nn.LSTM as the input of the following nn.Linear.</p>
<p>class LSTM(nn.Module):</p>
<pre><code>def __init__(self, num_classes, input_size, hidden_size, num_layers):
    super(LSTM, self).__init__()
    
    self.num_classes = num_classes
    self.num_layers = num_layers
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.seq_length = seq_length
    
    self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                        num_layers=num_layers, batch_first=True)
    
    self.fc = nn.Linear(hidden_size, num_classes)

def forward(self, x):
    h_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size))
    
    c_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size))
    
    # Propagate input through LSTM
    ula, (h_out, _) = self.lstm(x, (h_0, c_0))
    
    h_out = h_out.view(-1, self.hidden_size)
    
    out = self.fc(h_out)
    
    return out</code></pre>