akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Distributed Training with docker Container by splitting gpus resource for two instances

username: VISHAL_GHORPADE1
cooked: <p>Hello,<br>
I have a 8gpu server for training and use docker to run my experiments. I wanted to use first 4-gpu with one container for setting 1 of the experiment and the last 4-gpus with another container for a different setting of the experiment. But when I try to do that I get ‘Runtime error: Address already in use’  error. I tried giving different port for each of the runs but still the same issue. Any pointers to help me resolve it?</p>

username: H-Huang
cooked: <p>Hi, if you are using different ports for each run it will be fine, not sure why you are hitting this error. Can you provide more information about the initialization? Are you using <code>init_process_group</code>, what are you setting for initialization. Additionally, can you post the backtrace of the error?</p>

username: VISHAL_GHORPADE1
cooked: <p>Hi,<br>
I’m currently setting different ports…below is the trace:</p>
<pre><code class="lang-auto">
File "deepspeedtrain_pyDL.py", line 345, in __init__
    torch.distributed.init_process_group(backend="nccl", **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 576, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 229, in _env_rendezvous_handler
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout)
  File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 158, in _create_c10d_store
    hostname, port, world_size, start_daemon, timeout, multi_tenant=True
RuntimeError: Address already in use
</code></pre>
<p>I had initialised the global vairables for port, local rank, rank as below:</p>
<pre><code class="lang-auto">os.environ["MASTER_ADDR"] = "localhost"
os.environ["MASTER_PORT"] = "5653"  
os.environ["RANK"] = "0"
os.environ["LOCAL_RANK"] = "0"
</code></pre>
<p>I’m not sure why i’m getting this error…Is there a github example I can refer to for setting up the initialization?</p>

username: H-Huang
cooked: <p>From what I can tell, it is attempting to create a <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.TCPStore" rel="noopener nofollow ugc">TCPStore</a> instance internally. You need to find out what is currently conflicting with the address. As a test, you can try to create the store manually and this should also fail:</p>
<pre><code class="lang-auto">import torch.distributed as dist
# Create a single store instance
server_store = dist.TCPStore("127.0.0.1", 5653, 1, True)
</code></pre>
<p>Perhaps you can put print statements before <code>"deepspeedtrain_pyDL.py", line 345, in __init__     torch.distributed.init_process_group(backend="nccl", **kwargs)</code> to print out the environment variables MASTER_ADDR, MASTER_PORT just to confirm.</p>
<p>What version of PyTorch are you using? I believe this error message has been updated in the newer versions to be more descriptive of where the port/address conflict is (try pytorch 1.10+)</p>
Image impainting

username: leo2k
cooked: <p>Hi all, I’m currently implementing some CV operation on image impainting,<br>
For example I have an image (pytorch) with size (C, H, W) (C may greater than 3) and I want to keep non-zeros pixel and fill the 0-value pixel with background, how can I do this. I do not need to do backpropagation so you can use CV or numpy operations.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d4b8966fd03f7a3513deaae552ff04055b62b67.png" alt="image" data-base62-sha1="fARSCse8D4zln0At6R0VpLli8Yv" width="278" height="310"></p>
<p>Thank you in advance</p>

username: leo2k
cooked: <p>An example of the 64x32x32 image (I calculated mean of the first dimension then plot it)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0e1056a43ad53366dfb9fc12395cb6463e51482.png" alt="image" data-base62-sha1="peK8HG7OFq5oXqvuhhfZTaAKf62" width="376" height="353"></p>

username: ptrblck
cooked: <p>You could create a mask based on the zero pixels, index the tensor/array with it, and assign the new values to it.</p>

username: leo2k
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, but we still have to fill them with appropriate values, right? I need a proper way to calculate the value of these pixel</p>

username: ptrblck
cooked: <p>Wouldn’t the mask be the same for the background image?</p>

username: leo2k
cooked: <p>I cannot calculate the value of the background image pixel <a class="mention" href="/u/ptrblck">@ptrblck</a>, I think that we need some heuristic method</p>

username: Andrei_Cristea
cooked: <p>What do you want the background to be?</p>

username: leo2k
cooked: <p>I have tried filling the zero-valued pixel with the top left pixel of the original image for example but it look not natural at all, my point is can we do some heuristic methods like apply gaussian CNN kernels or stuff like that with the aim of creating natural images</p>

username: leo2k
cooked: <p>anybody can gimme a help on this, please</p>

username: Andrei_Cristea
cooked: <p>I think you’ll have to be a bit more specific about what you’re looking for. Creating natural images is a whole field and it can get really challenging (google GANs). You asked about some simple heuristic method, which sounds like a good idea, but without knowing what your domain is or more details it’s very difficult to blindly suggest a reasonable heuristics. Typically good heuristics require intuition about the domain.</p>
Latest dataset and loading script for any text task

username: Stephen_Fernandes
cooked: <p>hey there, for almost a year ive been using legacy torchtext methods to load custom datasets, preprocess them add a custom pretrained embedding and build the dataloader.</p>
<p>in the recent progress in torchtext, can someone please help me out with the latest torchtext preprocessing loading script that work generically for almost any NLP tasks<br>
that contain all of these functionalities, adding custom datasets, adding custom pretrainined embeddings, adding an iterator that sorts sequences based on their length that reduces to minimum padding.</p>
GNN feature matrix initialisation, torch_geometric

username: oakwheel
cooked: <p>Hi everyone, I am new to torch_geometric and the GNN world.<br>
If my network doesn’t have a feature matrix X but I want my model to rely only on the topology of the network what are the best ways to initialise this feature matrix X.</p>
<p>I have tried: <code>torch.eye(N,N)</code> and <code>torch.rand(N,N)</code>. I noticed that the model performs better when I have the identity matrix as the feature matrix rather than giving random entries.<br>
However my network is pretty big (about 30,000 nodes) and having that amount of features convoluted into a say 128 latent space makes it impossible to converge (the model is not learning anything).</p>
<p>Do you have any ideas on what would be the best solution here, or any papers that I could read regarding this problem?</p>
<p>Thank you!</p>
Help! Supervised approach for interpolating a sparse vector field

username: Kytes
cooked: <p>I am working on an Air Traffic Control problem where the path a plane should take based on its location is given by a vector field. We have a 25x25 grid as seen below, the vectors in red represent input, and the rest of the field is filled out using an optimization approach.</p>
<p>That is we have 5 given vectors (10 values, 1 angle 1 magnitude each) and we interpolate the remaining 625 vectors (1250 values). The optimization approach takes too long however, and we are trying to figure out if we can solve this using a supervised learning approach instead as we already have 10,000 solved samples.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed7da265766a9b60218f8181c68c7677cdb7d128.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed7da265766a9b60218f8181c68c7677cdb7d128" title=""><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed7da265766a9b60218f8181c68c7677cdb7d128_2_499x375.jpeg" alt="" data-base62-sha1="xSWkvSfTV85YXIGdtGyLYiqoFCM" width="499" height="375" role="presentation" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed7da265766a9b60218f8181c68c7677cdb7d128_2_499x375.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed7da265766a9b60218f8181c68c7677cdb7d128.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed7da265766a9b60218f8181c68c7677cdb7d128.jpeg 2x" data-dominant-color="F4F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">700×525 54.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So, each sample is [2, 25, 25] matrix where the first channel holds magnitude and second angle. The input is sparse, that is only 5 spots in each of the channels holds values, the rest are 0’s to be replaced by the network. Here are my questions, I’d appreciate any help on this,</p>
<p><strong>– Is something like this even possible with a supervised approach?</strong><br>
We are trying to learn a predictor which essentially takes 10 values and returns 1250. Is that unreasonable to ask of the network?</p>
<p><strong>– If so, which architecture should I be looking at?</strong><br>
So far I have tried a CNN with an AutoEncoder like structure, several convolutions followed by deconvolutions, with leakyRelu activations to account for all the 0s in my input. The network’s loss goes from 1.6 to around 1.3 and converges, not giving me any useful predictions.</p>
<p>Thank you in advance!</p>
What are alternative implementation for tensor norm and interpolation?

username: JimW
cooked: <p>I am using our internal compiler to convert an existing Pytorch model.  However, some operations are not supported on our side, so we need to use some alternative implementation for the same operations.</p>
<p>For example,  the original pytorch implementation of one layer is like this:</p>
<pre><code class="lang-auto">x = x / x.norm(dim=-1, keepdim=True)
</code></pre>
<p>I changed it to</p>
<pre><code class="lang-auto">x = x / torch.sqrt(torch.sum(x**2, dim=-1, keepdim=True))
</code></pre>
<p>the original implementation of the finally layer (it is a segmentation network like unet) is like this:</p>
<pre><code class="lang-auto">x = F.interpolation(x, size=(256, 256), mode='bilinear')
</code></pre>
<p>and I changed it into multiple less aggressive interpolation and stacked them like this</p>
<pre><code class="lang-auto">x = F.interpolation(x, size=(64, 64), mode='bilinear')
x = F.interpolation(x, size=(128, 128), mode='bilinear')
x = F.interpolation(x, size=(256, 256), mode='bilinear')
</code></pre>
<p>It kind of worked because I can perform training and got some decent result. However, the new result seems to be slightly worse than earlier, and I am not sure if it is due to the above modifications.<br>
Could someone please provide some comments whether the above modifications are correct?  Thanks.</p>

username: tom
cooked: <p>I’ll be very blunt, sorry about this. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  Take all of it with a lot of salt because you’ll know your setup and challenges much better than I will.<br>
That said, and in all honesty, I think you should ask yourself why you need these modifications. We all know that training networks can be finicky but  at some point it stops being a sound engineering exercise. At the very least you should know what the exact difference in computation (both forward and backward) is. Sometimes there are differences and then one makes more sense than the other. But really, to evaluate the performance impact, you should probably do a stochastic trial (i.e. run both options with 10 or 50 or so different initializations and do the statistics to see if the difference you are seeing can be attributed to noise).<br>
The first change might affect have some stabilization term the works differently for norm very close to 0. The second I don’t have that much intuition on, but it seems very strange, too.</p>
<p>Best regards</p>
<p>Thomas</p>
Is there a simple way to prevent zero gradients from being averaged when certain batched input produces zero gradients?

username: kmaeng
cooked: <p>Hi, experts. I have a model where some layers are only used in some datapoints.<br>
For example, let’s say only two out of 128 datapoints in a batch (bs=128) uses that particular layer.<br>
My problem is, I want to train this layer, but the gradient becomes so tiny as 126 out of 128 inputs in the batch produces 0 gradient for that layer, and the gradient is averaged across the 128 inputs.</p>
<p>I want to know if there’s a way for me to ignore zero gradients for irrelevant inputs in the batch (126) and only average gradient for the two relevant samples (2) for these particular layers, while averaging across all 128 samples for other layers.</p>
<p>Thank you for the help.</p>

username: tom
cooked: <p>This is not easily possible, but it should be possible to use some normalization technique to avoid that the gradients vanish (e.g. for the adaptive optimizers like Adam or LAMB, it would not matter that much unless the number of “nonzero gradient examples” varies starkly from one to the other).<br>
At which point do you know if you will get zero gradients for an example?<br>
If you can tell by the (unreduced) loss, the easiest might be to scale the loss appropriately. Or you could try to find the first backpropagation edge when you can tell and scale there.</p>
<p>Of course, the other thing to keep in mind is that if you’re only getting a training signal from 2% of your data, you are not using the data that efficiently, which may be a problem in its own.</p>
<p>Best regards</p>
<p>Thomas</p>
Automatically check whether a module has different behavior for train() and eval() modes

username: jayz
cooked: <p>Hey,</p>
<p>so, is there a simple attribute / method or function that one could use to return a bool whether a module contains code that changes depending on the .train()-state or .eval()-state.</p>
<p>For instance, for batchnorm with running stats, there would be different behavior. In that case, I would like to have a function:</p>
<pre><code class="lang-auto">check_train_eval_difference(batchnorm()) -&gt; True
</code></pre>
<p>For conv, which does not change:</p>
<pre><code class="lang-auto">check_train_eval_difference(conv()) -&gt; False
</code></pre>
<p>Also, I forgot how it is typically called when the model shows different behavior. I read a specific-term to name this somewhere. Does anyone know?</p>
<p>Thanks,</p>
<p>Best, JZ</p>

username: Matias_Vasquez
cooked: <p>You can define your own function.</p>
<p>Maybe this can help you.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

def check_train_eval_difference(module, input):
    with torch.no_grad():
        module.train()
        a = module(input)

        module.eval()
        b = module(input)

    return (a!=b).all()

batch_norm = nn.BatchNorm2d(3)
conv = nn.Conv2d(3, 2, 2)


input = torch.randn(20, 3, 35, 45)

print(check_train_eval_difference(batch_norm, input))
print(check_train_eval_difference(conv, input))

# Output: 
# tensor(True)
# tensor(False)
</code></pre>

username: jayz
cooked: <p>Hello Matias,</p>
<p>I was more thinking of a version of this function that does not require sending input through the layers, but can simply analyse a network in advance (before running) whether it contains layers with different train eval behaviour. Ideally, such layers should have a fixed boolean attribute that indicates this. But if there is no such thing, I will go with the function you provided. Thanks!</p>
<p>Best, JZ</p>

username: ptrblck
cooked: <p>You can check the internal <code>.training</code> flag:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
print(bn.training)
# True

bn.eval()
print(bn.training)
# False
</code></pre>
Is there performance difference when using different ways to create models?

username: seer_mer
cooked: <p>Will there be any performance/memory/other difference if I do the following to create a model:</p>
<ol>
<li>using nested modules (for example, if I make Conv-BN-ReLU a module, which is nested in a Residual Module, which is nested in a stage module, which is nested in a body module, which is nested in the module of the whole model) vs. writing all layers inside a single module without nesting</li>
<li>using Sequential vs Functional api (assuming no branching)</li>
<li>putting simple operations like add, scaling, etc. into a custom module and using the custom module as a layer vs directly adding the tensor in forward</li>
</ol>
<p>Any help would be great. Thanks in advance</p>

username: ptrblck
cooked: <p>All your points seem to be concerned about the Python call overhead by nesting objects.<br>
While I think you might see more overhead in the actual function calls, I would claim they might be tiny for “real” workloads where you are able to keep the GPU busy and would thus probably stick to the “cleanest” coding approach.<br>
If you are really concerned you could profile a “flat” vs. “nested” example model and check if you would be able to see any difference.</p>
Augment image and bounding box lazily

username: unofficial-Jona
cooked: <p>I want to perform image augmentations lazily (during model training). This however affects only the images itself, not the bounding box (defined by 2 coordinates). Imgaug could do just that, but it only works preemptively and therefore needs massive amounts of memory. Any idea how to integrate the bounding box augmentation into the forward pass?</p>

username: ptrblck
cooked: <p>The detection models in <code>torchvision</code> use the <code>resize_keypoints</code> and <code>resize_boxes</code> method as seen <a href="https://github.com/pytorch/vision/blob/316c38257127964c39866638716e64f6929617be/torchvision/models/detection/transform.py#L278-L309">here</a> so you might be able to reuse these (and potentially other detection transformations).</p>
Output of the model depends on the shape of the weights tensor

username: Sergei_Shumilin
cooked: <p>I want to train the model to sum the three inputs. So it is as simple as possible.</p>
<p>Firstly the weights are initialized randomly. It produces bad error estimate (approx. 0.5)</p>
<p>Then I initialize the weights with zeros. There are two options:</p>
<ol>
<li>the shape of the weights tensor is [1, 3]</li>
<li>the shape of the weights tensor is [3]</li>
</ol>
<p>When I choose the 1st option the model still works bad and can’t learn this simple formula.</p>
<p>When I choose the 2nd option it works perfect with the error of 10e-12.</p>
<p>Why the result depends on the shape of the weights? Why do I need to initialize the model with zeros to solve this simple problem?</p>
<pre><code class="lang-auto">    import torch
    from torch.nn import Sequential as Seq, Linear as Lin
    from torch.optim.lr_scheduler import ReduceLROnPlateau
    
    X = torch.rand((1024, 3))
    y = (X[:,0] + X[:,1] + X[:,2])
    m = Seq(Lin(3, 1, bias=False))
    
    # 1 option
    m[0].weight = torch.nn.parameter.Parameter(torch.tensor([[0, 0, 0]], dtype=torch.float))
    
    # 2 option
    #m[0].weight = torch.nn.parameter.Parameter(torch.tensor([0, 0, 0], dtype=torch.float))
    
    optim = torch.optim.SGD(m.parameters(), lr=10e-2)
    scheduler = ReduceLROnPlateau(optim, 'min', factor=0.5, patience=20, verbose=True)
    mse = torch.nn.MSELoss()
    for epoch in range(500):
        optim.zero_grad()
        out = m(X)
        loss = mse(out, y)
        loss.backward()
        optim.step()
        if epoch % 20 == 0:
            print(loss.item())
        scheduler.step(loss)

</code></pre>

username: ptrblck
cooked: <p>In your first approach you will get a warning, which you seem to ignore:</p>
<pre><code class="lang-python">Using a target size (torch.Size([1024])) that is different to the input size (torch.Size([1024, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code></pre>
<p>After fixing this issue via:</p>
<pre><code class="lang-python">loss = mse(out, y.unsqueeze(1))
</code></pre>
<p>the losses behave equal.</p>
Are there any PyTorch dataset attributes which I could use for shape detection?

username: Pytorch_developer
cooked: <p>I want to make a shape detection AI so I want to know if there are any PyTorch dataset attributes like MNIST which I could use for making it. Therefore, the dataset attribute must<br>
have shapes for my AI.</p>

username: Pytorch_developer
cooked: <p>I do not need an answer anymore so please do not reply to this question.</p>
Split dataset into training and validation without applying training transform

username: codeknight11
cooked: <p>I need to split the CIFAR10 dataset into training and validation set. The problem is that I wish to apply augmentations to training data. These are applied while loading the data. But if I split the data into validation set it also contains the augmentations which I obviously don’t want</p>
<pre><code class="lang-auto">train_transform = transforms.Compose([transforms.RandomRotation(10), 
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_data = datasets.CIFAR10('data', train=True, download=True, transform=train_transform)

# Generate validation set - https://stackoverflow.com/a/51768651
train_size = int(len(train_data) * 0.8) # 80% training data
valid_size = len(train_data) - train_size # 20% validation data
train_data, valid_data = random_split(train_data, [train_size, valid_size])
</code></pre>
<p>This snippet above applies augmentations to validation data (since it is just a part of the originally loaded data). Can I split the data before applying transforms?</p>

username: ptrblck
cooked: <p>You could create two different datasets (one for training and the other for validation) with different transformations. Afterwards you could sample the split indices and either use <code>Subet</code> of both datasets or a <code>SubsetRandomSampler</code>.</p>

username: codeknight11
cooked: <p>Yes, but how do I split the dataset before loading the data?<br>
I have to supply the transform while loading the CIFAR dataset.</p>

username: ptrblck
cooked: <p>Usually you would implement a dataset with lazy loading, so that the actual data samples are loading in each call into <code>__getitem__</code>, and thus creating multiple datasets would be cheap (as no data would be loaded in the <code>__init__</code>).<br>
Since <code>CIFAR10</code> is considered small (<code>50000 samples in the train set * 3 channels * 32 height * 32 width ~= 150MB</code>) it’s loaded directly in the <code>__init__</code>.<br>
If you are concerned about wasting these 150MB, you could derive a custom dataset using <code>CIFAR10</code> as the parent class and split the datasets internally.</p>

username: AM.MO
cooked: <p>what should I do, if I want to split only 500 for training and 500 for validation, I tried different ways, and still give me this error.</p>
<p>ValueError: Sum of input lengths does not equal the length of the input dataset!</p>

username: ptrblck
cooked: <p><code>random_split</code> expects to assign all samples of the dataset to the splits.<br>
If you want to use a subset only, use <code>torch.utils.data.Subset</code> or a <code>SubsetRandomSampler</code> and pass the indices to them.</p>
Strange pair of errors using CrossEntropyLoss

username: Nathan_Goedeke
cooked: <p>I’m running into a very strange pair of errors when trying to calculate the Cross Entropy Loss. First I got this error message:</p>
<pre><code class="lang-auto">D:\anaconda3\envs\downgrade\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Long'
</code></pre>
<p>Now, it seems obvious that of course the softmax function isn’t going to work on a long, it should be a float. So I converted the values to float, and I got this error message:</p>
<pre><code class="lang-auto">D:\anaconda3\envs\downgrade\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847 
   2848 

RuntimeError: expected scalar type Long but found Float
</code></pre>
<p>“Expected long but found float”? So should it be a long or a float? this trace log is pretty deep in the library, but points to the exact same line of code both times. How could these two errors, on the same line of code, simultaneously require me to be both a Long and a Float? Why can’t it just implicitly convert the values?<br>
Nathan</p>

username: Nathan_Goedeke
cooked: <p>Turns out they should both be Long, but the two trace logs I showed are pointing to two (very similar) lines of code.</p>

username: Sai_Dheeraj
cooked: <p>Can you please explain how you resolved this issue? We will still get the first Runtime Error if we leave it as long, right?</p>

username: Nathan_Goedeke
cooked: <p>Hello,</p>
<p>In my particular case, the <em>inputs</em> should be float while the <em>targets</em> should be converted to long.</p>
<p>Nathan</p>
How to get the input size of the pretrained model in torchvision.models

username: songyuc
cooked: <p>Hi, guys!<br>
I met a problem today. I wanted to used the pretrained model resnet18 in torchvision.models to solve my problem and the size of the input image is 28x28. So I am wondering whether I need to resize my input image before I send the input into the resnet18.<br>
Any suggestion will be appreciated!</p>

username: Kushaj
cooked: <p>torchvision resnet18 has an averagepool layer at the end. So the input size does not matter much provided the feature map size is greater than kernel size.</p>
<p>You can use 28x28 images.</p>

username: songyuc
cooked: <p>OK, I get it.<br>
Thank sincerely for your answer and explanation!</p>
Siamese network and graph convolution

username: tanvir_14
cooked: <p>I’m trying to predict something with the help of the Siamese network and graph convolution. In my network, my inputs are x1 and x2 but I’m not using the output that was generated by x2 for calculating the loss. When x1 and x2 are different, I’m getting completely different output but when x1 and x2 are same I’m getting the expected output. I’m attaching the code below.</p>
<pre><code class="lang-auto">class MyGCN(nn.Module):
    def __init__(self, adj, hid_dim, coords_dim=(2, 3), num_layers=4, nodes_group=None, p_dropout=None):
        super(MyGCN, self).__init__()
        _gconv_input = [_GraphConv(adj, coords_dim[0], hid_dim, p_dropout=p_dropout)]
        _gconv_layers = []

        for i in range(num_layers):
                _gconv_layers.append(_ResGraphConv(adj, hid_dim, hid_dim, hid_dim, p_dropout=p_dropout))

        self.gconv_input = nn.Sequential(*_gconv_input)
        self.gconv_layers = nn.Sequential(*_gconv_layers)
        self.gconv_output = ModulatedGraphConv(hid_dim, coords_dim[1], adj) 
    
    # Siamese style net
    def forward_once(self, x):
        out = self.gconv_input(x)
        out = self.gconv_layers(out)
        out = self.gconv_output(out)
        return out

    def forward(self, x1, x2):
        out1 = self.forward_once(x1)
        out2 = self.forward_once(x2)    
        return out1,out2
</code></pre>
<p>I’m calling the forward function like below:</p>
<pre><code class="lang-auto">outputs,_ = model_pos(x1, x2)
</code></pre>
<p>And I’m calculating the loss like this:</p>
<pre><code class="lang-auto">loss = (1-lamda)*criterion(outputs, targets) + lamda*criterionL1(outputs, targets)
</code></pre>
<p>Can anyone say what could be the possible reason for not getting the expected output when x1 and x2 are different even though I’m not using the output of x2 for calculating the loss?</p>

username: tanvir_14
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could u please help?</p>
Finding better threshold for the unsupervised problem

username: specialized_boy
cooked: <p>I am working on a unsupervised image reconstruction task. so i need to verify whether the reconstructed image by the model is good are not. I cant do that visually.</p>
<p>so i was calculating the sum of mean and standard deviation of all the predictions of the dataset and finding a threshold.</p>
<p>So that prediction loss greater than the threshold is not good and less than threshold is good for a given reconstructed image.</p>
<pre><code class="lang-auto">threshold = np.mean(losses_train_all)+np.std(losses_train_all)
</code></pre>
<p>I was trying to improve this threshold. are their any existence methods that suit my problem?</p>
Exporting compiled pytorch exec

username: EKami
cooked: <p>Hello,<br>
I following the steps in te Readme to compile pytorch from source with:</p>
<pre><code class="lang-auto">python setup.py install
</code></pre>
<p>The problem is that this command installs pytorch in the current env. What I’d like to have is a binary that I can export and install on other machines, pretty much like in Tensorflow where a <code>.whl</code> is created once the project is compiled.<br>
Is there anything similar in Pytorch? Thanks!</p>

username: ptrblck
cooked: <p>You could try to build the wheels via <code>python setup.pt bdist_wheel</code>.</p>

username: EKami
cooked: <p>Awesome, thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Any runtime configuration to disable TORCH_WARN temporally?

username: yzs
cooked: <p>Purpose: temporally disable some annoying <code>TORCH_WARN</code> at runtime level without upgrading to a newer version that removes the <code>TORCH_WARN</code>. Is it possible and how?</p>
<p>TORCH_WARN in libtorch: <a href="https://github.com/pytorch/pytorch/blob/631f0351313da2edeeb72cf8bb963035b11425d9/c10/util/Exception.h#L518-L530" class="inline-onebox" rel="noopener nofollow ugc">pytorch/Exception.h at 631f0351313da2edeeb72cf8bb963035b11425d9 · pytorch/pytorch · GitHub</a></p>
<p>Some others have the same question in <a href="https://discuss.pytorch.org/t/how-do-i-disable-libtorch-warning/63439/3" class="inline-onebox">How do I disable libtorch warning - #3 by ptrblck</a></p>
How to pass categorical data to neural network

username: aleemsidra
cooked: <p>Hi! I am working on an image classification problem. Each of the images has sex and age categorical data associated with it. I want to use CNN for extracting features from images. After that, I want to concatenate the metadata with these features and pass them to the final layer as shown below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d158d8c890545068307407407718423595937b6e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d158d8c890545068307407407718423595937b6e" title="Screenshot 2022-04-02 at 5.22.36 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d158d8c890545068307407407718423595937b6e_2_689x335.png" alt="Screenshot 2022-04-02 at 5.22.36 PM" data-base62-sha1="tRY8YmUK8FD30RrotwCeB7k1YJ0" width="689" height="335" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d158d8c890545068307407407718423595937b6e_2_689x335.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d158d8c890545068307407407718423595937b6e_2_1033x502.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d158d8c890545068307407407718423595937b6e.png 2x" data-dominant-color="F2F3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-04-02 at 5.22.36 PM</span><span class="informations">1106×538 43.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now the problem is since the neural network takes only numeric values, how can I pass the categorical data.</p>
Cosine similarity of model weights of two different checkpoints

username: meshghi
cooked: <p>Hello, I want to analyze the changes in model weights over the training phase. For doing so, I want to use cosine similarity between model weights in two different checkpoints. I was wondering how I could do that? Thanks a lot in advance.</p>

username: Matias_Vasquez
cooked: <p>You can try something like this. You only need to define a variable to hold the old values and compare to the new ones and after the comparison update the “new” old values.</p>
<pre><code class="lang-python">cos = torch.nn.CosineSimilarity(dim=0)
params = []

for param in model.parameters():
    params.append(param.view(-1))

params = torch.cat(params)

with torch.no_grad():
    # Same input twice --&gt; returns 1
    print(cos(params, params))

    # Small deviation from the original --&gt; close to 1, but smaller
    print(cos(params, params+(torch.rand(params.shape)/100) ))

    # Some random value --&gt; close to 0
    print(cos(params, torch.rand(params.shape)))
</code></pre>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
PyTorch works in Mac terminal but not visual studio code

username: Pytorch_developer
cooked: <p>I downloaded PyTorch with pip in two Conda environments and one base environment but PyTorch only works in my Mac terminal and not in visual studio code. How will I be able to get PyTorch working in visual studio code?</p>

username: ptrblck
cooked: <p>I guess your Visual Studio might be using the wrong Python environment.<br>
<a href="https://code.visualstudio.com/docs/python/environments">These docs</a> might help.</p>

username: Pytorch_developer
cooked: <p>How might I be able to change my python environment then.</p>

username: Pytorch_developer
cooked: <p>I figured out how to get it working in vs code by changing my interpreter path thank you.</p>
Cuda.synchronize()/ .cuda() is extremely slow

username: panzf98
cooked: <p>Hi<br>
I am using Torch 1.7.1 and Cuda 10.1 in  Titan XP.<br>
but when i use .cuda() command,it always takes  more than 10 mins.<br>
According to the same problem answered before,i try to use torch.cuda.synchronize()  before the .cuda() command，but synchronize still needs more than 10mins.<br>
Is there anyway to accelerate this?<br>
Here’s my code and result, thanks in advance.</p>
<pre><code class="lang-auto">import torch
from datetime import datetime
torch.cuda.set_device(2)
t1 = datetime.now()
torch.cuda.synchronize()
print(datetime.now() - t1)
for i in range(10):
    x = torch.randn(10, 10, 10, 10) # similar timings regardless of the tensor size
    t1 = datetime.now()
    x.cuda()
    print(i, datetime.now() - t1)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7c1a1de55c33d4e405f92cd5896fb893c34f088.png" alt="1" data-base62-sha1="nW2AKVcYdYjNdFyL5oKNWr63CgE" width="181" height="203"></p>

username: ptrblck
cooked: <p>Could you update to the latest release and check if you would still be running into the same issue?</p>

username: panzf98
cooked: <p>Thanks,but I’m not authorized to update gpu driver,thus i can only use cuda 10.1.<br>
I tried to update pytorch to 1.8.1,which is the latest release i can get with cuda 10.1,but the problem remains.<br>
Besides,the performance used to be great using pytorch 1.7.1,so i assume it is not the version problem.</p>

username: ptrblck
cooked: <p>The issue sounds a bit as if you are JIT compiling for your architecture.<br>
Since you are using a Titan Xp, it should have a supported compute capability of 6.1. Do you have any other devices in this machine which might be older?<br>
If so, could you mask the Titan Xp via <code>export CUDA_VISIBLE_DEVICES=ID</code> where <code>ID</code> denotes the GPU id and see if you are hitting the same issue again?</p>

username: panzf98
cooked: <p>Sorry for the late response.Actually I only  got another  gpu 2080Ti ,but it is even slower than Titan xp.Anyway, thanks for help.</p>
TLS Cert for download.pytorch.org Expired

username: FrederickGeek8
cooked: <p>Hey, it looks like the TLS certificate for <a href="http://download.pytorch.org" rel="noopener nofollow ugc">download.pytorch.org</a> is expired.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b" title="Screen Shot 2022-04-02 at 5.20.46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b_2_506x500.png" alt="Screen Shot 2022-04-02 at 5.20.46 PM" data-base62-sha1="wA2jxj93AhwCwudsJQasWHr6CNl" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b_2_506x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b_2_759x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4586d0f0b039514e5cffc2b5ddb8a415b00ff2b_2_1012x1000.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-04-02 at 5.20.46 PM</span><span class="informations">1678×1656 231 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Thank you for letting us know! We’ll look into it.</p>

username: seemethere
cooked: <p>Should be resolved now! Thanks for the heads up!</p>
Building a resnet based decoder

username: Naman-ntc
cooked: <p>I want to to build an encoder decoder network, with transposed convolutions in decoder.<br>
Is their a standard approach on building a decoder. Suppose I have and input image and pass it through a resnet50 based encoder (removing last avg pool and fully connected), what would be a nice way to build decoder.<br>
I understand that encoder decoder correspondence doesnt signify anything but how would you build the decoder. Would you try build corresponding transposed convolution layer for every convolution in encoder? And would you use stride of transposed convolution to perform inverse of max pooling.</p>
<p>What are the good/standard design choices???</p>

username: franciscocms
cooked: <p>Hi <a class="mention" href="/u/naman-ntc">@Naman-ntc</a>! I am working on a side project and have the same issue. How did you end up building that decoder?? Thank you</p>

username: Nakkhatra
cooked: <p><a href="https://discuss.pytorch.org/u/Naman-ntc">Naman-ntc</a> Hi, did you build the network? @ <a href="https://discuss.pytorch.org/u/franciscocms">franciscocms</a> I have imported a unet decoder from Timm segmentation library. Though I have not finished working with it yet.</p>
<p>Currently I am facing the following problems:<br>
-I want to take the output from resnet 18 before the last average pool layer and send it to the decoder. I  will use the decoder output and calculate a L1 loss comparing it with the input image.<br>
-I want to remove only the last linear layer and replace it with linear layer for binary classification as my problem requires a binary classification. Then I want to use this output for calculating BCE loss.</p>
<p>Finally, these two losses will be added in my final model. What is the best way to write the model in code?</p>
Overfitting a segmenation network. Validation performs better than train

username: einrone
cooked: <p>Hi!</p>
<p>I am trying to train various types of segmentation network. Currently I am trying to overfit a dataset, consisting of 3 tensors. Two of them are torch.ones of dim 3x192x192 and the last one is torch.zero of dim 3x192x192. For 200 epochs i am getting the following:</p>
<p>current learning rate : 0.0001<br>
Phase: train. Metrics for current epoch: 199:<br>
current epoch 199 – recall : 0.9502495659722897<br>
current epoch 199 – precision : 0.5000000000043909<br>
current epoch 199 – accuracy : 0.8979017469620487<br>
current epoch 199 – dice_score : 0.47386653059730577<br>
current epoch 199 – loss : 0.6017277367443301<br>
current learning rate : 0.0001<br>
Phase: validation. Metrics for current epoch: 199:<br>
current epoch 199 – recall : 0.9998101128472228<br>
current epoch 199 – precision : 1.0<br>
current epoch 199 – accuracy : 0.9999050564236114<br>
current epoch 199 – dice_score : 0.9999050486964542<br>
current epoch 199 – loss : 0.05479495529160455</p>
<p>As you can see, train perform way poorer than validation. I am using the same tensor for validation and train.</p>

username: Andrei_Cristea
cooked: <p>A long shot, but do you have Dropout layers in your architecture? If so, perhaps those are messing up your training loss. Dropout gets turned off in model eval mode (for validation) so that’s a candidate for something that could cause a model to do better in validation than in training.</p>
Calculating training loss - sum of avg batch losses or avg loss across all samples

username: YASJAY
cooked: <p>Hi,</p>
<p>how is loss calculated during training,</p>
<p>is it batchwise average or overall data average?</p>

username: Andrei_Cristea
cooked: <p>Typically the loss is calculated for each batch separately.</p>
<p>In other words, inside your training loop that iterates over batches, you have these commands:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = calculate_loss(*args)
loss.backward()
</code></pre>
<p>So <code>loss</code> is getting overwritten for each batch, which is the same cadence with which the optimizer steps in the gradient direction.</p>
Deploy Pruned Models

username: jordipuig
cooked: <p>Hi, I want to deploy a pruned model but from what I’ve seen, when you prune a neural net, the library creates a mask and sets some weights to zero. What has to be done to actually reduce model size and inference time of the deployed pruned model?</p>
<p>Thanks in advance!</p>

username: marksaroufim
cooked: <aside class="quote no-group" data-username="jordipuig" data-post="1" data-topic="147916">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jordipuig/48/39115_2.png" class="avatar"> jordipuig:</div>
<blockquote>
<p>Hi, I want to deploy a pruned model but from what I’ve seen, when you prune a neural net, the library creates a mask and sets some weights to zero. What has to be done to actually reduce model size and inference time of the deployed pruned model?</p>
</blockquote>
</aside>
<p>You’re correct in that pruning won’t actually reduce your model size and that whether a pruned model runs faster or not depends on the support you’re getting from your hardware.</p>
<p>If you’re more interested in reducing model size and decreasing latency you’re better off using quantization, distillation or exporting to some runtime like tensorRT or IPEX</p>
<p>Examples on each are here <a href="https://github.com/pytorch/serve/tree/master/experimental/torchprep">https://github.com/pytorch/serve/tree/master/experimental/torchprep</a> and <a href="https://github.com/pytorch/serve" class="inline-onebox">GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production</a></p>

username: Andrei_Cristea
cooked: <p>I’m guessing you could prune as a first step and then inspect your (pruned) model to get a feel for which parts of your pipeline could be shrunk down without loss in performance. Meaning, if a bunch of the parameters at a certain layer are getting zeroed out by pruning, you can probably create an alternative net where that layer is smaller, and it should perform just as well (and that one would be of smaller size and training / inference time). I wonder if anyone has built a utility to do this.</p>
How can I deploy multiple models on torchserve and use hot update?

username: iot-rookie
cooked: <p>As the title，I deployed multiple models with Tensorflow Serving,but now , I want to deploy them on torchserve,how can I do that ? Is there any tutorial to solve multi model and model hot update ?Thanks a lot !</p>

username: marksaroufim
cooked: <p>Yep torchserve used to be called multi model server before it got rebranded so you should be able to easily deploy and scale various models with a REST API <a href="https://github.com/pytorch/serve/blob/master/docs/management_api.md" class="inline-onebox" rel="noopener nofollow ugc">serve/management_api.md at master · pytorch/serve · GitHub</a> - if you have any more specific questions feel free to ask a question in a github issue</p>
Apply `torch.autograd.Function` on layer weights without losing gradients

username: SamirMoustafa
cooked: <p>Hello,<br>
I was trying to wrap the layers with a generic wrapper and then perform the layer operation whatever it was, like here:</p>
<pre><code class="lang-auto">class TorchModuleWrapper(Module):
    def __init__(self, torch_object: Module):
        super().__init__()
        # save the layer in the _wrapped_object
        self._wrapped_object = torch_object

    def __call__(self, x):
        # feedforwad the input on the layer with the new weights
        return self._wrapped_object(x)
</code></pre>
<p>The problem happened when I tried to apply the <code>torch.autograd.Function</code> on the layer weights inside the forward function since the function that I have to requires learnable parameters, these learnable parameters lost their gradients.</p>
<p>There’s one solution that I commented on in the example here that could be applying the operation between the input and the weights but it would not make the wrapper anymore.<br>
So, is there any solution to apply the function on the weights without losing the gradient for the learnable parameters located outside the layer?</p>
<pre><code class="lang-auto">class PACT(Function):
    @staticmethod
    def forward(ctx, x, s):
        return torch.round(x * s) / s

    @staticmethod
    def backward(ctx, grad_output):
        dLdx = grad_output
        dLds = grad_output
        return dLdx, dLds

class TorchModuleWrapper(Module):
    def __init__(self, torch_object: Module):
        super().__init__()
        self.bits = 8
        torch_object.bias = None
        self._wrapped_object = torch_object
        self.scale = Parameter(Tensor(1))

    def __call__(self, x):
        weight = self._wrapped_object.weight
        quantized_weight = PACT.apply(weight, self.scale)
        self._wrapped_object.weight = Parameter(quantized_weight)
        return self._wrapped_object(x)
    
    def __repr__(self):
        return "Wrapped" + self._wrapped_object.__repr__()
</code></pre>
<p>Collab example with the unneeded solution:<br>
<a href="https://colab.research.google.com/drive/1QfmSisFYmzOMPpDqOFDwT_1ptJc8Zpqx?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1QfmSisFYmzOMPpDqOFDwT_1ptJc8Zpqx?usp=sharing</a></p>