akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can't find cuda version for pytorch 2.0.0

username: miraboreasu
cooked: <p>Hello, here is my pytorch version,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c16f0ca46271058f1eafcb036b077b88cdfa960.png" alt="image" data-base62-sha1="40uwbCy03Eax259kP4WQLND0EzS" width="354" height="40"><br>
and I am trying to install CUDA, so that I can use GPU for my pytorch, but I got an error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a8143f94faaf877e8660dd3e505e0930b6ae89e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a8143f94faaf877e8660dd3e505e0930b6ae89e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a8143f94faaf877e8660dd3e505e0930b6ae89e.png" alt="image" data-base62-sha1="m2OtNfEHfr5YUzFBJQU0BZzFPlQ" width="690" height="23" data-dominant-color="202020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">743×25 1.36 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>ERROR: Could not find a version that satisfies the requirement torch==2.0.0+cu111 (from versions: 1.7.1, 1.8.0, 1.8.0+cu111, 1.8.1, 1.8.1+cu111, 1.9.0, 1.9.0+cu111, 1.9.1, 1.9.1+cu111, 1.10.0, 1.10.0+cu111, 1.10.1, 1.10.1+cu111, 1.10.2, 1.10.2+cu111, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0)</p>
<p>Thanks!</p>

username: eqy
cooked: <p>You are trying to install a CUDA 11.1 wheel, which doesn’t exist as that CUDA version would be too old for a 2.0.0 build to be available.</p>
<p>Could you try this install command from get started <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">page</a>? It should install the stable 2.0.0 version of PyTorch.<br>
<code>pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu118</code></p>
Reshape some axes but leave all remaining axes unchanged

username: richardbaxter
cooked: <p>Is there a method to reshape some axes of a tensor but leave all the remaining axes unchanged, without specifying the number of remaining axes (should be dynamically inferred)?</p>
<p>Currently the user must specify the number of remaining axes, for every possible number of tensor dimensions; e.g. <code>if(len(x.shape) == 4): pt.reshape(x, (x.shape[0]*x.shape[1], -1, -1)) elif(len..</code>.</p>
<p>For example;</p>
<ul>
<li>
<code>pt.reshape(x, (x.shape[0]*x.shape[1], -2))</code>, where <code>-2</code> will dynamically infer the number of unchanged remaining dimensions.</li>
<li><code>pt.mergeAxes(x, (0, 1))</code></li>
</ul>

username: hrbigelow
cooked: <p>Yes, it is odd that PyTorch doesn’t provide better tensor shape manipulation functions natively.  Maybe there is a good reason for this, I’m not sure.</p>
<p>There is <a href="https://github.com/arogozhnikov/einops#API" rel="noopener nofollow ugc">einops</a> which sort of does what you want, though I don’t think exactly:</p>
<pre><code class="lang-python">from einops import rearrange, reduce, repeat
# rearrange elements according to the pattern
output_tensor = rearrange(input_tensor, 't b c -&gt; b c t')
# combine rearrangement and reduction
output_tensor = reduce(input_tensor, 'b c (h h2) (w w2) -&gt; b h w c', 'mean', h2=2, w2=2)
# copy along a new axis
output_tensor = repeat(input_tensor, 'h w -&gt; h w c', c=3)
</code></pre>
Building for iOS with torch:nn features

username: StuartAbercrombie
cooked: <p>I’m building an iOS application linking in libtorch that uses more than just the low level ops, and it seems build_ios.sh and the cmake build are not set up to accommodate this.  Am I missing something?</p>
<p>We need stuff in the torch::nn namespace, e.g. torch::nn::Module::register_parameter.  It seems I have to modify CMakeLists.txt to have NO_API=0 to get this.  Then I can try building with</p>
<p>BUILD_LITE_INTERPRETER=0 BUILD_PYTORCH_MOBILE=1 IOS_ARCH=arm64 ./scripts/build_ios.sh</p>
<p>That almost works, except there are link errors relating to torch::jit::ExportModule, that meant I had to comment out calls to this in csrc/api/src/serialize/output-archive.cpp.</p>
<p>CMakeLists.txt only compiles ${TORCH_SRC_DIR}/csrc/jit/serialization/export_module.cpp, where this method is defined, if INTERN_BUILD_MOBILE is false, which of course it isn’t for an iOS build.  However, references to this method are compiled in if NO_API is false.  So my build configuration apparently entails incompatible flags.  Various hacks to the build to get around this resulted in other problems.</p>
<p>So as things stand I have to modify the source and CMakeLists.txt to get things running on iOS.  Is there a better way?  Is the mobile build intended to support these features, now or in the future?</p>
<p>Separately from this, I found I had to manually invoke</p>
<p>python -m tools.autograd.gen_autograd <br>
aten/src/ATen/native/native_functions.yaml <br>
aten/src/ATen/native/tags.yaml <br>
torch/csrc/autograd/generated/ <br>
tools/autograd</p>
<p>to get necessary headers like tools/autograd/templates/VariableType.h.</p>
<p>Any insights appreciated.</p>

username: tdomhan
cooked: <p>I have the same problem. I’m trying to follow the changes here <a href="https://github.com/ankurtibrewal/pytorch/commit/77c2e10690623e96d5744abb79baa07658c353b7#diff-c5ee05f1e918772792ff6f2a3f579fc2f182e57b1709fd786ef6dc711fd68b27L441" class="inline-onebox" rel="noopener nofollow ugc">Android Adaptation · ankurtibrewal/pytorch@77c2e10 · GitHub</a><br>
Mentioned in this issue: <a href="https://github.com/pytorch/pytorch/issues/42665" class="inline-onebox" rel="noopener nofollow ugc">Optimizer support via Libtorch C++ on Android · Issue #42665 · pytorch/pytorch · GitHub</a></p>

username: tdomhan
cooked: <p>In the end I was able to get something running: <a href="https://github.com/pytorch/pytorch/compare/main...tdomhan:pytorch:ios_build" class="inline-onebox" rel="noopener nofollow ugc">Comparing pytorch:main...tdomhan:ios_build · pytorch/pytorch · GitHub</a><br>
Need to test more, but at first glance things look ok. (I needed to additionally add the Accelerate framework in XCode under “Link Binary with Libraries”).</p>
Need help to figure out the dimensions

username: Devipriya_Raju
cooked: <p>I have x of shape [32,16,16,16]. I am trying to use nn.UpSample() to make x of shape [32,32,32,32]. But i keep getting [32,16,32,32]. Can anybody please help on this?</p>

username: ptrblck
cooked: <p><code>nn.Upsample</code> interpolates the spatial dimensions, not the channel or batch dims.<br>
Since you explicitly also want to upsample the channel dimension (<code>dim1</code>) you could <code>unsqueeze</code> an additional dim before upsampling the input and <code>squeeze</code>it afterwards:</p>
<pre><code class="lang-python">x = torch.randn(32, 16, 16, 16)
up = nn.Upsample(scale_factor=2)

out = up(x)
print(out.shape)
# torch.Size([32, 16, 32, 32])

out = up(x.unsqueeze(1)).squeeze(1)
print(out.shape)
# torch.Size([32, 32, 32, 32])
</code></pre>
Extracting RBG channels from tensor

username: CSstudent
cooked: <p>I have the following tensor and am attempting to extract the RGB channels. The torch size is ([3, 3, 3]). How do I go about doing this?</p>
<pre><code class="lang-auto">tensor([[[ 0.0968, -0.1558,  0.1071],
         [ 0.0619,  0.0805, -0.1376],
         [ 0.1503,  0.0461, -0.0443]],

        [[ 0.1112, -0.1045, -0.1144],
         [ 0.1630, -0.1243, -0.1621],
         [-0.1156,  0.1858, -0.0574]],

        [[-0.1317,  0.1630,  0.0421],
         [-0.0513,  0.0113,  0.0913],
         [-0.1606,  0.1905,  0.0819]]])</code></pre>

username: ptrblck
cooked: <p><code>unbind</code> should work:</p>
<pre><code class="lang-python">r, g, b = x.unbind(0)
</code></pre>

username: SalCometa
cooked: <p>Hello Piotr,<br>
I have a tensor torch.Size([5, 3, 256, 256]). I am trying to get a vector with the highest values of each RGB channel so I am doing this:<br>
rgb = torch.argmax(mytensor, dim=1, keepdims=True)<br>
and I am having a new tensor with torch size torch.Size([5, 1, 256, 256]).<br>
Is this the correct approach? Thank you</p>

username: ptrblck
cooked: <p>I’m not sure if you want to keep the highest <em>value</em>, as in pixel intensity, or the corresponding index of the color channel.<br>
Currently, <code>rgb</code> will contain the <code>argmax</code>, i.e. the channel index corresponding to the <code>max</code> value in the range <code>[0, 1, 2]</code>. If you want to keep the actual pixel intensity, use <code>rgb = torch.max(x, dim=1)</code> and index the values afterwards via <code>rgb[0]</code> or <code>rgx.values</code>.</p>
How to delete more than one sequential layer in a model

username: xiew
cooked: <p>I write one class for my model. I want to use into another model, and want to delete three last sequential of that.</p>
<pre><code class="lang-auto">class SpermModel(nn.Module):
    def __init__(self):
        super(SpermModel, self).__init__()
        self.cnn_layers = model
        self.modi = torch.nn.Sequential(*(list(self.cnn_layers.children())[:-1]))
        self.fc = nn.Sequential(
            nn.Flatten(),
            nn.Linear(25088, 1024),
            nn.ReLU(),
            nn.Linear(1024, 2)
        )
        self.fc1 = nn.Sequential(
            nn.Flatten(),
            nn.Linear(25088, 1024),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(1024, 2)
        )
        self.fc2 = nn.Sequential(
            nn.Flatten(),
            nn.Linear(25088, 1024),
            nn.ReLU(),
            nn.Linear(1024, 2)
        )
    def forward(self, item):
        out = self.modi(item)
        out1 = self.fc(out)
        out2 = self.fc1(out)
        out3 = self.fc2(out)
        return out1, out2, out3
</code></pre>
<p>I used <code>self.modi = torch.nn.Sequential(*(list(self.cnn_layers.children())[:-1]))</code> for that, but it just delete one sequential layer.</p>
Need a help for channel wise encoding

username: ShanakaRG
cooked: <p>I have an input tesor (2,5,10), batch size 2, channels 5, temporal dimension 10.so i need to encode this signal into 512 dimension channel-wise. that means after the encoding, each channel should have 512 dimensions and the output will look like (2, 5*512,1) as the encoded signal. how can i do it using a 1Dconv in Pytorch? the encoder weights should be the same for all 5 channels.</p>
Find weights with PyTorch

username: Duke_Richards
cooked: <p>I have a ground truth data (<em>G</em>) that is a 3D numeric matrix of shape (<em>1000, 25, 25</em>) where 1000 refers to the number of time steps and (<em>25, 25</em>) is the x and y dimensions of each time frame in 2D.</p>
<p>I also have 5 surrogate datasets (<em>S1, S2, S3, S4, S5</em>) which are approximations of the above ground truth data (<em>G</em>).</p>
<p>I need to learn weights (<em>w1, w2, w3, w4, w5</em>) such that the linear combination of the surrogate datasets  give the best possible approximation to the ground truth dataset, i.e.:</p>
<p><code>w1*S1 + w2*S2 + w3*S3 + w4*S4 + w5*S5 ≈ G</code></p>
<p>Is there some way to do this with PyTorch? Thanks!</p>

username: J_Johnson
cooked: <p>This is a pretty simple use case for Pytorch.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as opt

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.w1 = nn.Parameter(torch.ones((1000,25,25), dtype=torch.float))
        self.w2 = nn.Parameter(torch.ones((1000,25,25), dtype=torch.float))
        self.w3 = nn.Parameter(torch.ones((1000,25,25), dtype=torch.float))
        self.w4 = nn.Parameter(torch.ones((1000,25,25), dtype=torch.float))
        self.w5 = nn.Parameter(torch.ones((1000,25,25), dtype=torch.float))
    def forward(self, s1, s2, s3, s4, s5):
        return self.w1*s1 + self.w2*s2 + self.w3*s3 + self.w4*s4 + self.w5*s5

ground_truth=torch.rand((1, 1000, 25, 25))

s1 = torch.rand(1, 1000, 25, 25)
s2 = torch.rand(1, 1000, 25, 25)
s3 = torch.rand(1, 1000, 25, 25)
s4 = torch.rand(1, 1000, 25, 25)
s5 = torch.rand(1, 1000, 25, 25)

model = Model()

epochs = 1000
criterion = nn.MSELoss()
optimizer = opt.Adam(model.parameters(), lr=0.01)

for i in range(epochs):
    model.zero_grad()
    out = model(s1, s2, s3, s4, s5)
    loss=criterion(out, ground_truth)
    loss.backward()
    optimizer.step()

print(loss.item())
</code></pre>
<p>Edit: Just noticed your equation at the bottom. I’ve updated the above to reflect that. If you’d prefer scalar values, you can just change each weight to <code>nn.Parameter(torch.ones((1), dtype=torch.float))</code>.</p>

username: Duke_Richards
cooked: <p>Hi <a class="mention" href="/u/j_johnson">@J_Johnson</a> thanks a lot for the detailed explanation and solution. I am trying this on my dataset and will let you know. Thanks again!</p>
Why do I only get an output?

username: wenbo_shang
cooked: <p>I was writing codes in Jupyter Lab, and I thought there would be two outputs generated by the code. But in order to get two outputs, I must explicitly use print(). I am confused about it, can someone help me?</p>
<p>code :</p>
<p>import torch</p>
<p>a = torch.tensor([1, 2, 3])<br>
a.dtype  # print(a.dtype)</p>
<p>a = a.type(torch.int32)<br>
a.dtype  # print(a.dtype)</p>

username: InnovArul
cooked: <p>It is not some issue related to torch itself.<br>
Maybe something like <a href="https://stackoverflow.com/questions/36786722/how-to-display-full-output-in-jupyter-not-only-last-result">this</a> might help (if you place it at the top of your notebook and execute it first)?</p>
<pre><code class="lang-auto">from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
</code></pre>

username: wenbo_shang
cooked: <p>Thank you! It works!</p>
Incorrect grad when a function applies partially to a tensor/parameter

username: pylearn
cooked: <p>In my network, one computation only applies partially to a tensor/parameter. For example, square the 1st element of parameter w.</p>
<p>I used two methods. The forward computations of these two methods are correct, but the gradients are different. Method 2 should be correct since it is just regular computation. Hence, Method 1 is incorrect. I reckon this is because the computation node is NOT properly connected when <code>torch.tensor()</code> is used in method 1?</p>
<p>I want to know why they are different and how the computational graph works when a function applies partially to a tensor.</p>
<p>Method 1</p>
<pre><code class="lang-auto"># square the 1st element of parameter w with `torch.tensor()`
self.v = torch.tensor((self.w[0]**2, self.w[1]), requires_grad=True)   
</code></pre>
<p>Method 2</p>
<pre><code class="lang-auto"># square the 1st element of parameter w with a mask
self.mask = torch.tensor([True, False])
self.v = self.w**2 * self.mask + self.w * torch.logical_not(self.mask)
</code></pre>

username: ptrblck
cooked: <p>Recreating a new tensor detaches the passed tensors from the computation graph and creates a new leaf tensor without a gradient history. Operations on <code>self.v</code> will thus not be backpropagated to <code>self.w</code>.</p>

username: pylearn
cooked: <p>Thanks a lot. That makes sense. That is why I use method 2. However, as you can see, Method 2 involves more computation.</p>
<p>It might not differ much in the example mentioned here. However, in my model, the function is not simple as square and it applies to a larger proportion of a high-dimensional parameter.</p>
<p>Is there a more efficient way to do this?</p>

username: ptrblck
cooked: <p>You could <code>torch.stack</code> the tensor parts as seen here:</p>
<pre><code class="lang-python">w = torch.randn(2, requires_grad=True)

# square the 1st element of parameter w with a mask
mask = torch.tensor([True, False])
v = w**2 * mask + w * torch.logical_not(mask)

out = torch.stack((w[0]**2, w[1]))
print((out == v).all())
# tensor(True)

w.mean().backward()
print(w.grad)
# tensor([0.5000, 0.5000])
</code></pre>

username: pylearn
cooked: <p>Thanks. Both of your replies solve my questions.</p>
<p>Since the first one is related directly to the title (Incorrect grad …), I will mark the first one as the Solution.</p>
How do I dequantize a state dictionary?

username: Zowie_Tay
cooked: <p>is it possible to dequantize the entire state dictionary as a whole?</p>
<p>currently my plan is to dequantize layer by layer.<br>
for the conv2d layer, i can just dequantize the “.weight” and use the “.bias”  as is but im having issues with the Linear Layer looking through the keys i see</p>
<blockquote>
<p>‘fc1.scale’, ‘fc1.zero_point’, ‘fc1._packed_params.dtype’, ‘fc1._packed_params._packed_params’</p>
</blockquote>
<p>but i do not know how to convert them into just</p>
<blockquote>
<p>“fc1.weight”, “fc1.bias”</p>
</blockquote>

username: Zowie_Tay
cooked: <pre><code>        sdict = self.model.state_dict()

        sdict['conv1.weight'] = torch.dequantize(state_dict['conv1.weight'])
        sdict['conv1.bias'] = torch.dequantize(state_dict['conv1.bias'])
        sdict['conv2.weight'] = torch.dequantize(state_dict['conv2.weight'])
        sdict['conv2.bias'] = torch.dequantize(state_dict['conv2.bias'])
        sdict['fc1.weight'] = torch.dequantize(state_dict['fc1._packed_params._packed_params'][0])
        sdict['fc1.bias'] = torch.dequantize(state_dict['fc1._packed_params._packed_params'][1])
        sdict['fc2.weight'] = torch.dequantize(state_dict['fc2._packed_params._packed_params'][0])
        sdict['fc2.bias'] = torch.dequantize(state_dict['fc2._packed_params._packed_params'][1])
</code></pre>
<p>well i managed to solve it by doing this since apparently the bias and weight for a linear layer is combined together</p>
Decentralized asynchronous stochastic gradient descent implementation

username: ethan01
cooked: <p>Hi, I want to try out decentralized stochastic gradient descent in an asynchronous setting (ad-psgd). What I want to happen is that for example, if I have some distributed worker nodes, I want to split a dataset across them and these workers are connected in communication to each other in some sort of network (examples from papers are often ring networks). The training algorithm is as follows:</p>
<ol>
<li>
<p>each worker will asynchronously compute the gradient with their dataset while at the same time gathering the weights of the model from one random worker which it is connected to and averaging it along with the worker’s own original weights. Call this W_avg</p>
</li>
<li>
<p>after computing the gradients, the worker will calculate a new weight like so: W_new = W_avg - (learning rate) * (gradient)</p>
</li>
<li>
<p>Set the new weight of the worker to be W_new</p>
</li>
<li>
<p>repeat 1-3 for whole dataset</p>
</li>
</ol>
<p>And this should be done asynchronously meaning that these workers will not wait at all, they will keep on repeating the algorithm until finished and it doesn’t matter if their neighbor weights are outdated or anything like that.</p>
<p>I would also like to try regular d-psgd which is where the worker averages all their neighbor’s weights.</p>
<p>Is there anything already like that in torch fsdp or ddp? If not, what can I look into to implement this? I am not very familiar with torch distributed implementations.</p>
<p>The algorithm comes from: <a href="https://arxiv.org/pdf/1710.06952v3.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1710.06952v3.pdf</a></p>
Add "requires_grad=False" item to the loss

username: ElijahZh
cooked: <p>Hi,<br>
I am trying to add an item to the loss function. The item’s required_grad = False.<br>
Here is the structure:</p>
<p>pred, x = net(input)<br>
loss = nn.MSELoss()(pred, target) + custom_function(x)</p>
<p>The item: custom_function(x) is “required_grad=False”<br>
I know the during the grad process, this item is not going to help with gradient. But it can help decide the value of the loss.item(). Would this item play any role in decreasing the loss?</p>

username: ptrblck
cooked: <p>No, adding a constant to a function will not change it’s derivative and thus gradients:</p>
<pre><code class="lang-python">x = torch.randn(1)
w = torch.randn(1, requires_grad=True)
target = torch.ones(1)

out = x * w
loss = F.mse_loss(out, target)
print(loss)
# tensor(1.0523, grad_fn=&lt;MseLossBackward0&gt;)
loss.backward()
print(w.grad)
#tensor([-0.3013])

w.grad = None
out = x * w
loss = F.mse_loss(out, target)
loss = loss + 1000.
print(loss)
# tensor(1001.0523, grad_fn=&lt;AddBackward0&gt;)
loss.backward()
print(w.grad)
# tensor([-0.3013])
</code></pre>

username: ElijahZh
cooked: <p>Thank you so your response! This is what I need.</p>
Function ‘EigBackward0’ returned nan values in its 0th output

username: S_M
cooked: <p>Hi all, I use mixed-precision for my training when I call tensor.pinverse() to compute eigen vector and values of my tensor  to compute loss. So, I have to change fp16 to fp32 because of sensitivity of pinverse() function to half dtype. Unfortunatly when I use scaler.scale(loss).backward() to backward loss in autocast block I get this error. note that the loss changed to the half dtype before calling backward.</p>
<p>RuntimeError: Function ‘EigBackward0’ returned nan values in its 0th output.</p>
<p>Any help would be really appreciated.</p>

username: ptrblck
cooked: <p>Double post with follow up from <a href="https://discuss.pytorch.org/t/how-can-i-computes-the-pseudoinverse-of-a-matrix-with-mixed-precision-in-pytorch/177871/2">here</a>.</p>
Am I using Huggingface correctly

username: vg224
cooked: <p>Hello,</p>
<p>I am working on a project to see how some models/architectures perform with my custom dataset for semantic segmentation. I want to train my models <strong>from scratch</strong> with no pre trained weights. I am comparing models like ResNet50, Segfromer and Mask2Former. I load all my images and masks using DataLoader. For ResNet50, I use the ResNet50 FCN provided by torchvision: <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.segmentation.fcn_resnet50.html" rel="noopener nofollow ugc">fcn_resnet50 — Torchvision main documentation</a>.</p>
<p>For Segformer, I found that huggingface provides a Segformer model. So I am just using that. The performance isn’t that great so I am wondering if I am doing something wrong. Is it fine if we use huggingface models without using any other huggingface methods like AutoImageProcessor?<br>
I load the model using:</p>
<pre><code class="lang-auto">configuration = SegformerConfig(**dict(arch['args'])) 
configuration.num_labels = data['num_classes']
model = SegformerForSemanticSegmentation(configuration)
</code></pre>
<p>Then I get the results using:</p>
<pre><code class="lang-auto">for _, images, masks in dataloader:
    images = images.to(self.device, non_blocking=True)
    masks = masks.to(self.device, non_blocking=True)
    outputs = model(pixel_values=images, labels=masks).logits
    outputs = nn.functional.interpolate(outputs, size=masks.shape[-2:], mode="bilinear", align_corners=False)
</code></pre>
<p>I am also trying to use Mask2Former to train from scratch, but for post processing I need to use Mask2FormerImageProcessor to get the semantic segmentation. I already have processed my images in my DataLoader. What do I do here to just use Mask2Former with my own data?</p>
<p>So for Mask2Former. I am doing this</p>
<pre><code class="lang-auto">    configuration = Mask2FormerConfig(**dict(arch['args']))
    configuration.num_queries = data['num_classes']
    model = Mask2FormerForUniversalSegmentation(configuration)
</code></pre>
<pre><code class="lang-auto">for _, images, masks in dataloader:
    images = images.to(self.device, non_blocking=True)
    masks = masks.to(self.device, non_blocking=True)
    outputs = model(images,
                mask_labels=masks)
        outputs = outputs.masks_queries_logits
        outputs = nn.functional.interpolate(outputs, size=masks.shape[-2:], mode="bilinear", align_corners=False)
align_corners=False)

</code></pre>
<p>Is this the correct way to use Mask2Former (model only) for semantic segmentation? Its not performing that well…</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>You might want to post this question in the <a href="https://discuss.huggingface.co">HuggingFace forum</a> as it seems to be quite HF-specific.</p>
Forward pass on GPU efficiently

username: fabiola
cooked: <p>I am trying to pass some inputs on a network. I can not simply use <code>net(input_)</code> because I can not push all the inputs at once to the GPU due to memory limits.<br>
I thus wrote a function to pass the inputs by batches:</p>
<pre><code class="lang-auto">def batch_pass(net, input_, batch, device):
    print('cuda memory before batch pass', torch.cuda.memory_allocated(device=device))
    l = 0 if len(input_) % batch == 0 else 1
    r = []
    for i in range(len(input_)//batch + l):
        in_ = input_[i*batch:(i+1)*batch].to(device)
        r += [net(in_)]
        del in_
        torch.cuda.empty_cache()
    print('cuda memory after batch pass', torch.cuda.memory_allocated(device=device))
    return torch.vstack(r)
</code></pre>
<p>However, the cuda memory allocated keeps on growing, and it seems that doing <code>del in_</code> and <code>torch.cuda.empty_cache()</code> has no effect.</p>
<p>The output is:</p>
<pre><code class="lang-auto">cuda memory before batch pass 26404352
cuda memory during batch pass 65019904
cuda memory during batch pass 103635456
cuda memory during batch pass 142251008
cuda memory during batch pass 180866560
...
cuda memory during batch pass 13232923136
cuda memory during batch pass 13271538688
cuda memory during batch pass 13310154240
cuda memory during batch pass 13348769792
cuda memory during batch pass 13387385344
</code></pre>
<p>And the function cannot finish due to error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.90 GiB total capacity; 12.48 GiB already allocated; 9.44 MiB free; 15.05 GiB reserved in total by PyTorch) 
</code></pre>
<p>How can I pass <code>input_</code> without using too much cuda memory?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="fabiola" data-post="1" data-topic="178180">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fabiola/48/30499_2.png" class="avatar"> fabiola:</div>
<blockquote>
<p>However, the cuda memory allocated keeps on growing, and it seems that doing <code>del in_</code> and <code>torch.cuda.empty_cache()</code> has no effect.</p>
</blockquote>
</aside>
<p>This is expected since you are appending the model outputs to the <code>r</code> list including the entire computation graph of each iteration.<br>
If you don’t want to train the model and only attach the actual output tensors, call <code>.detach()</code> on these tensors before appending them to the <code>list</code>.</p>
Fixed learning rate and momentum of SGD in Pytorch?

username: Loona
cooked: <p>Hello,</p>
<p>How do I fix the learning rate and momentum of torch.optim.SGD in Pytorch? Does the momentum in torch.optim.SGD increase overtime?</p>
<p>Thanks!</p>

username: srishti-git1110
cooked: <p>Hey, please refer to the official docs <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html" rel="noopener nofollow ugc">here</a>.<br>
Learning rate shall be kept fixed if you aren’t using schedulers, and momentum is optional in SGD as stated in the docs.</p>

username: Loona
cooked: <p>So momentum is also fixed? How do you adjust momentum then?</p>
Which is the first pytorch version that has the pre_backward-hook?

username: Vinayaka_Hegde
cooked: <p>Hi, I have a quick question. I just wanted to know which is the first version of pytorch that has the <code>pre_backward_hook</code> feature enabled? I’ve tried using <code>v1.12.0</code> but that doesn’t have the <code>pre_backward_hook</code> feature.</p>
<p>FYI : <a href="https://github.com/pytorch/pytorch/pull/86700" class="inline-onebox" rel="noopener nofollow ugc">[nn] module: full_backward_pre_hook by kshitij12345 · Pull Request #86700 · pytorch/pytorch · GitHub</a> was the PR for implementing <code>pre_backward_hook</code></p>
<p>I wanted this for my research purpose, as I’ve done all experiments using <code>v1.12.0</code> and don’t want to use the latest versions like <code>2.0</code>.</p>

username: ptrblck
cooked: <p>You would need to use <code>2.0.0</code> as the linked PR was merged on Oct 13, 2022 while PyTorch <code>1.13.0</code> was released Oct 28, 2022 and thus won’t contain this PR.</p>
Am I using WebDataSet correctly

username: JAllsopp
cooked: <p>Hi,<br>
I’ve taken my data (~70,000 images) and split it into roughly 6:1 training to validation sets, and create tar shards of 500 images, so I’ve 0 to 131 shards. Here’s the code I’m trying to use to load in the shards</p>
<pre><code class="lang-auto">preproc = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Grayscale(num_output_channels=1),
    transforms.Resize(120),
    transforms.ToTensor())
])

def identity(x):
    return x

dataset = (wds.WebDataset("training/training-out-{000001..000131}.tar").shuffle(100).decode("pil").to_tuple("png","json").map_tuple(preproc,identity))
batch_size = 50
dataset1=dataset.with_length(batch_size) 
train_loader = torch.utils.data.DataLoader(dataset1.batched(batch_size), num_workers=1, batch_size=None) 
</code></pre>
<p>However, this doesn’t seem to be shuffling the images and when I try increasing the number of workers from 1 I run into errors. Even when I run the code as above, after the first epoch I get errors like this, here’s the first few of many;</p>
<blockquote>
<p>/home/allsoppj/.conda/envs/pytorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py:640: UserWarning: Length of IterableDataset &lt;webdataset.pipeline.WebDataset_Length object at 0x7fc2d91f12d0&gt; was reported to be 50 (when accessing len(dataloader)), but 51 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.1 documentation</a> for examples.<br>
warnings.warn(warn_msg)<br>
/home/allsoppj/.conda/envs/pytorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py:640: UserWarning: Length of IterableDataset &lt;webdataset.pipeline.WebDataset_Length object at 0x7fc2d91f12d0&gt; was reported to be 50 (when accessing len(dataloader)), but 52 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker.<br>
warnings.warn(warn_msg)<br>
/home/allsoppj/.conda/envs/pytorch/lib/python3.10/site-packages/torch/utils/data/dataloader.py:640: UserWarning: Length of IterableDataset &lt;webdataset.pipeline.WebDataset_Length object at 0x7fc2d91f12d0&gt; was reported to be 50 (when accessing len(dataloader)), but 53 samples have been fetched. For multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker.</p>
</blockquote>
<p>Want I really want to know is, am I doing this correctly, and is there a way of doing it better?  I’m calling train_loader  using for imgs, labels in train_loader inside each epoch loop.</p>
<p>Thanks</p>
The validation result are difference when train and test

username: Jianzhong_He
cooked: <p>Hi,<br>
Has anybody encountered that the validation results are wrong completely while the training is correct and while it’s no problem when place the validation process in the training process.<br>
To clearly, the process is below,<br>
1 i training segmentation and validation, the two process was separate that the training will store a model and val load the model, it’s very bad<br>
2 i val on the process of training, which i place the val process in the process of training, it was no problem. And i use the model stored by training running the validation, it’s still be worst.<br>
Thanks a lot for any help.</p>

username: wangskyone
cooked: <p>I have the same problem. do you make it?</p>
Loss Backward Taking 30x More Time than Forward Pass

username: Debanjan_Mondal
cooked: <p>I am building Graph Neural Network Based NER Code. For this task I am using GNN-BERT-CRF Architecture. While training Forward Pass taking few milliseconds but Backward Pass takes on an average 30seconds to Execute.</p>
<pre><code class="lang-auto">class GATBertCRF(torch.nn.Module):
    def __init__(self, num_classes, input_dim, hidden_dim, output_dim, edge_input_dim, edge_output_dim, multihead_num = 1, dropout = 0.6):
        super(GATBertCRF, self).__init__()

        self.gat = GATLayer(input_dim, hidden_dim, output_dim, edge_input_dim, edge_output_dim, multihead_num, dropout)
        self.mlp = torch.nn.Linear(output_dim+EMBEDDING_SHAPE, num_classes)
        self.crf = CRF(num_classes).to(device)
        self.num_classes = num_classes
        self.input_dim = input_dim
  
    def neg_log_likelihood(self, g):
        h = self.gat(g)
                
        max_seq_len = NODE_TEXT_MAX_LENGTH
        num_nodes   = g.nodes().shape[0]       
        num_tags    = self.num_classes
        
        embeddings = g.ndata['embedding']
        h = torch.unsqueeze(h, dim = 1)
        h = h.repeat(1, embeddings.shape[1], 1)
        emissions = self.mlp(torch.cat([embeddings, h], dim = 2))
        emissions = torch.permute(emissions, (1,0,2))
        
        # For training model
        tags = g.ndata['tags']
        masks = g.ndata['masks']
#         if tags is not None:
        emissions   = emissions.to(device)
        labels      = tags.T.to(device)
        mask        = masks.T.to(device)  

        log_likelihood = self.crf(emissions, labels, mask)            
        return -1 * log_likelihood
        
    def forward(self, g):
        h = self.gat(g)
                
        max_seq_len = NODE_TEXT_MAX_LENGTH
        num_nodes   = g.nodes().shape[0]       
        num_tags    = self.num_classes
        
        embeddings = g.ndata['embedding']
        h = torch.unsqueeze(h, dim = 1)
        h = h.repeat(1, embeddings.shape[1], 1)
        emissions = self.mlp(torch.cat([embeddings, h], dim = 2))
        emissions = torch.permute(emissions, (1,0,2))

        # For prediction
        sequence_of_tags = self.crf.decode(emissions)
        return sequence_of_tags
</code></pre>
<p>This is the Model. Can you tell me why Loss Backward taking Time?</p>
How to handle eval() with BatchNorm layers?

username: dannepai
cooked: <p>Hello!</p>
<p>We’re running a UNet-like segmentation network with a few blocks of Conv2d-&gt;BatchNorm2d-&gt;ReLU. Inference with the network in training mode is acceptable but when we change the mode to eval the output is more or less static and does not change with input image.</p>
<p>I’ve googled this problem and as far as I understand, the problem lies with the BatchNorm2d layers which in eval-mode get its values not from the batch but some running average of the training data. Although there are lot of questions about this (both here and on other forums) I can’t really find an answer on how to solve it.</p>
<p>What is the proper way to solve this? I want to understand how to deal with the problem. More training data? More epochs? Other batch sizes?</p>
Is it possible to run FSDP on a module that is not part of the optimizer?

username: Andras_Iani
cooked: <p>I have a teacher and a student, both submodules of the same model.</p>
<p>Currently I add whole model to the optimizer, but I only wrap teacher transformer blocks in FSDP.<br>
I assume this means that even if my teacher runs with torch.nograd in forward pass, it will have fp16 weights (2bytes), fp16 gradients (2 bytes)+optimizer states(fp32 weights , adam states - 12 bytes) stored. Using mixed precision this is a total of 16 bytes/parameter. These get sharded accross 6 gpus so 16/6 bytes/gpu. (in these calculations I assume that deepspeed stage 3 documentation and way of computing memory usage also applies to FSDP)</p>
<p>I wonder is it possible to get rid of gradients and optimizer states for my teacher and still apply FSDP to it? I found that if I don’t include it into the optimizer I get and error with FSDP…</p>
<p>Old method to reduce memory in teacher: What I did as an alternative to FSDP is that I set my teacher to fp16 with requires_grad=False, no FSDP, and I have only fp16 weights stored - 2 bytes/param in teacher.</p>
<p>My question: would FSDP be more efficient than my old method in reducing memory (is there a way to shard the remaining 2bytes/param across 6 gpus using fsdp)?</p>
Indexing a multidimensional array "with radius" in batch

username: yccyenchicheng
cooked: <p>I have a high dimensional tensor with shape <code>x</code><br>
and the index i want to take values <code>idx</code>:</p>
<pre><code class="lang-auto">B, N = 2, 10
x = torch.zeros(B, 1, 128, 128)
idx = torch.randint(128, size=(B, N, 2))`
</code></pre>
<p>where <code>idx[:, :, 0], idx[:, :, 1]</code> are where i want to take values from <code>x</code>.<br>
Also, I want to specify a <code>radius</code> around the index I want to take values.<br>
For instance, <code>radius = 6</code></p>
<p>I tried to do</p>
<pre><code class="lang-auto">i = idx[:, :, 0]
j = idx[:, :, 1]
x[:, :, i-radius:i+radius, j-radius:j+radius]
</code></pre>
<p>but got <code>TypeError: only integer tensors of a single element can be converted to an index</code>.</p>
<p>Is there any way I can do this without looping over the first and second dimension of <code>idx</code>?</p>
<p>Thanks!</p>
Parameters of model

username: sbht_sbht
cooked: <p>Hello there! I am trying to train a model for segmentation for the first time. I found this way to load the model and the following loss function:</p>
<pre><code class="lang-auto">from torchvision.models.segmentation.deeplabv3 import DeepLabHead

def createDeepLabv3(outputchannels=2):
    """DeepLabv3 class with custom head
    Args:
        outputchannels (int, optional): The number of output channels
        in your dataset masks. Defaults to 1.
    Returns:
        model: Returns the DeepLabv3 model with the ResNet101 backbone.
    """
    model = models.segmentation.deeplabv3_resnet101(pretrained=True)
    model.classifier = DeepLabHead(2048, outputchannels)
    # Set the model in training mode
    model.train()
    return model
</code></pre>
<p>Loss function:</p>
<pre><code class="lang-auto">import torch.nn.functional as F

class DiceBCELoss(nn.Module):
    def __init__(self, weight=None, size_average=True):
        super(DiceBCELoss, self).__init__()

    def forward(self, inputs, targets, smooth=1):
        
        #comment out if your model contains a sigmoid or equivalent activation layer
        inputs = F.sigmoid(inputs)       
        
        #flatten label and prediction tensors
        print(inputs[0])
        print(targets[0])
        inputs = inputs.view(-1)
        targets = targets.view(-1)
        
        intersection = (inputs * targets).sum()                            
        dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
        BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')
        Dice_BCE = BCE + dice_loss
        
        return Dice_BCE
</code></pre>
<p>I end up getting the following error:</p>
<blockquote>
<p>AttributeError: ‘collections.OrderedDict’ object has no attribute ‘sigmoid’</p>
</blockquote>
<blockquote>
<p>AttributeError: ‘collections.OrderedDict’ object has no attribute ‘view’<br>
I assume that I loaded the model incorrectly.</p>
</blockquote>
<p>As I understand it, the classifier of the pre-trained model is simply replaced here with a classifier with random weights. I already tried transfer learning and there I replaced only the last layer. So, I have a question, how to change the classifier correctly and how many classes should I specify, one or two, given that I am segmenting stones. And ofcource explain me please, how to load model correctly?</p>

username: ptrblck
cooked: <p>Based on the error messages it seems the <code>input</code> to <code>DiceBCELoss.forward</code> is a <code>dict</code> while a tensor is expected.<br>
Note that <code>deeplabv3_resnet101</code> will indeed return a <code>dict</code> containing <code>out</code> and <code>aux</code> keys so you might want to index the <code>out</code> tensor before passing it to the loss function.</p>

username: sbht_sbht
cooked: <p>Thank you a lot! Didn’t know this and when read about deeplabv3 did not pay attention to this detail. Sorry for my inattention</p>
Autograd isn't working on iOS

username: 111410
cooked: <p>I have this sample code in file.m (compiled by Objective-C++ compiler) in Xcode iOS project (LibTorch is loaded via Cocoapods):</p>
<pre><code class="lang-auto">auto param = torch::tensor({1.}, torch::requires_grad());
auto result = 1 + param;
result.backward();
</code></pre>
<p>Executing gives this error:<br>
<strong>libc++abi.dylib: terminating with uncaught exception of type c10::Error: Could not run ‘aten::_backward’ with arguments from the ‘CPU’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" rel="noopener nofollow ugc">https://fburl.com/ptmfixes</a> for possible resolutions. ‘aten::_backward’ is only available for these backends: [BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].</strong></p>
<p>I searched a lot and still have no answer, please give any suggestions how can to fix it.</p>

username: ptrblck
cooked: <p>I think Autograd is disabled by default in the mobile backend. You might be able to build it from source and reenable it, but I haven’t tried it and am not sure if this would still be possible (it was in the past, if I’m not mistaken).</p>

username: 111410
cooked: <p>Thanks! I’m sure that’s the only option I have. Can you please give some advice how to do it? What changes can I make in <code>CMakeLists.txt</code>? Can I still use <code>python tools/build_libtorch.py</code>? Seems there’s no instructions for mobile on GitHub page.</p>

username: Jesse_Stone
cooked: <p>try part on aarch64-linux-gnu-gcc</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://fatalfeel.blogspot.com/2020/06/building-libtorch-and-numpy-in-c.html">
  <header class="source">
      <img src="https://fatalfeel.blogspot.com/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://fatalfeel.blogspot.com/2020/06/building-libtorch-and-numpy-in-c.html" target="_blank" rel="noopener nofollow ugc">fatalfeel.blogspot.com</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://fatalfeel.blogspot.com/2020/06/building-libtorch-and-numpy-in-c.html" target="_blank" rel="noopener nofollow ugc">libtorch cross compile on x86_64 &amp; Android &amp; aarch64-linux-gnu-gcc</a></h3>

  <p>Refer to:  https://github.com/pytorch/pytorch#get-the-pytorch-source  https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst  http...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
build all dependencies by manual can use backward</p>

username: tdomhan
cooked: <p>Did you find out what the right flag was to enable backward on mobile?</p>
When to use Variable?

username: Nadav_Potasman
cooked: <p>Lately I read about the autogard.Variable and now I’m little bit confuse.<br>
When should I use Variable during training?<br>
Until now I just load batch from DataLoader, sent it to device and then run my model on it, now I saw that sometimes people warp the batch’s data with Variable before sent it to model’s forward pass, why is that? Why we need that and how this is different from what I was doing till now?</p>

username: ptrblck
cooked: <p>You shouldn’t use <code>autograd.Variable</code> anymore as it’s deprecated since PyTorch <code>0.4</code>.<br>
Use tensors directly instead.</p>

username: Nadav_Potasman
cooked: <p>Oh, wow, so it’s very old code…<br>
So if i’m reading images, I only need to use the .toTensor() on the ndarray?</p>

username: ptrblck
cooked: <p>Yes, you would only need to transform your input to a tensor e.g. images or image-like numpy arrays via <code>transforms.ToTensor()</code>. There is no need to wrap the input tensor to a model into <code>Variable</code>s anymore.</p>
Binary Segmentation Question

username: Ali_Youssef
cooked: <p>Hello everyone!</p>
<p>I had a question regarding binary segmentation as I’m a little confused.</p>
<p>I’m try to implement binary segmentation using semi-sueprvised learning. Each batch has an image [64,2,64,64], and a label [64,1,64,64] which can be binary if labaled or a tensor of -1s if unlaballed.</p>
<p>When I do a prediction, and I’m using BCEWithLogitsLoss with an ignore index of -1. My question is, when passing the prediction to the loss function, should I argmax it or sigmoid it with a threshold or just slice to [:,1,:,:] or should i directly pass it to the loss function along with the labels?</p>
<ul>
<li>Sorry I forgot to clarify, I’m trying to predict the segmentation mask, which i labale as 1 and everything else as 0.</li>
</ul>
<p>Thank you!</p>

username: eqy
cooked: <p>Why does your prediction have more than one value in the “channel” dimension, as this seems unexpected?</p>

username: Ali_Youssef
cooked: <p>I’m using deeplabv3resnet50 with input as 3 and output as 2, since there is two classes background and foreground, is that not correct?</p>

username: eqy
cooked: <p>Passing the logits directly to the loss function is the expected behavior, but note that if you are using BCEWithLogitsLoss anyway you should just resize the output layer to a single “channel.” Otherwise if you cannot change the output shape you should use CrossEntropyLoss as you have a “channel” for each class.</p>
<p>Either way I would not expect an accuracy difference between the approaches.</p>

username: Ali_Youssef
cooked: <p>What I did is I sliced the prediction output taking the second channel preds[:,1,:,:] and used it in BCEWithLogitsloss. However, I’m not sure if that is correct, I assumed the second channel corresponds to the second class which I’m predicting.</p>

username: eqy
cooked: <p>If you do that during training then you are effectively throwing out half of the results of the final layer, although the model should be able “learn around” the strange quirk and ultimately produce results that should be similar to using all of the channels + CrossEntropyLoss. If you do that on a pretrained model without training then it would be incorrect as you would considering the logits of a class without any relation to the other class. As you are considering loss functions here I assume you are doing training which means the model would “work” although with wasted computation being done in a nonstandard way.</p>
<p>In summary:<br>
Indexing one out of two channels + BCE (strange, but “works” if trained this way)<br>
Last layer with single output channel + BCE (expected standard practice)<br>
Last layer with two output channels + CE (should be equivalent to former but somewhat nonstandard)</p>
How to invert affine matrix?

username: Arkun
cooked: <p>Can I invert the affine matrix and apply it to my landmarks while keeping the autograd?</p>

username: AlphaBetaGamma96
cooked: <p>Do you have a minimal reproducible example?</p>

username: Arkun
cooked: <p>I can’t public code for some reason. The following is my problem. I have list of landmarks (location of pixels on image) which transformed by affine matrix M, and now i want to invert M and inverse transform the list of landmarks. How can i do that?</p>