akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Distributed file write

username: adityamnk
cooked: <p>Hi, does pytorch provide routines or support for parallel writes to the same single file (ex. binary file) at the same time from multiple distributed processes? I am looking for functionality similar to the link below, but using only pytorch. <a href="https://mpi4py.readthedocs.io/en/stable/tutorial.html#mpi-io" class="inline-onebox" rel="noopener nofollow ugc">Tutorial — MPI for Python 3.1.4 documentation</a>.</p>
<p>I heavily use torch.distributed. It has many functions for communication between distributed processes. But, I couldn’t find any functions for writing of a binary or similar data file by multiple processes simultaneously.</p>

username: kumpera
cooked: <p>No, PyTorch provides no support for parallel writes to a single file.</p>
<p>This is not an easy task given controlling which process can write is not portable and quite complex. It’s even worse if you’re doing over the network.</p>
<p>You could use object collectives to negotiate which process can write to which part of the file and then use regular file writes.</p>

username: adityamnk
cooked: <p>Hi <a class="mention" href="/u/kumpera">@kumpera</a>, thank you very much for your response. Can you please elaborate on your solution? What do you mean by use of object primitives. Sorry about my lack of object oriented knowledge.</p>
<p>I am willing to change my software architecture. For now, I am writing a separate file per process. It will be so much better to just write one file for all processes.</p>

username: kumpera
cooked: <p>You can use something like this: <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.all_gather_object" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a></p>
<p>So all ranks tell which items they need to write and where.</p>
Install pytorch and use the code in both mac and linux

username: Yuval_Nardi
cooked: <p>I have a python repository with a script that uses torch. I am dockerizing it (installing all packages in requirements.txt) and want to run it:</p>
<ol>
<li>on Mac, which does not use GPU</li>
<li>on Linux, which does use GPU.</li>
</ol>
<p>According to the website, in order to install it on a mac I should do:<br>
<code>pip3 install torch</code><br>
In order to install it on a Linux machine I should do:<br>
<code>pip3 install torch --index-url https://download.pytorch.org/whl/cu118</code></p>
<p>My question: how can I install it (what should I specify in the requirements.txt - I am actually using pyenv locally when developing on a Mac) such that when I dockerize it on either Mac or Linux it will use cpu only (if on a Mac) and GPU if on the Linux?</p>
<p>Thanks,<br>
Yuval</p>
After LSTM, too much zero

username: Gorgen
cooked: <p>hi all:</p>
<p>when use LSTM layer to extract high-level deep learning features based on pre-trained Roberta features  as follows:“self.lstm = nn.LSTM(1024, 256, 2, batch_first= False, bidirectional=True)”,  a lot of zero producing. why? how to solve ?</p>
<p>'text tensor([[[0.0000, 0.0000, 0.2246,  …, 0.0000, 0.0966, 0.1180],<br>
[0.0000, 0.0000, 0.0475,  …, 0.0442, 0.0149, 0.0588]],</p>
<pre><code>    [[0.0000, 0.0000, 0.3146,  ..., 0.0000, 0.0822, 0.2287],
     [0.0000, 0.0000, 0.1793,  ..., 0.0000, 0.0000, 0.2323]],

    [[0.0000, 0.0000, 0.2645,  ..., 0.0000, 0.0585, 0.2474],
     [0.0000, 0.0000, 0.2497,  ..., 0.0000, 0.0686, 0.2068]],

    ...,

    [[0.0000, 0.0000, 0.0288,  ..., 0.0000, 0.0358, 0.0000],
     [0.0000, 0.0000, 0.0535,  ..., 0.0000, 0.0000, 0.0470]],

    [[0.0000, 0.0000, 0.0260,  ..., 0.0000, 0.0000, 0.0591],
     [0.0000, 0.0000, 0.0458,  ..., 0.0000, 0.0000, 0.0327]],

    [[0.1995, 0.0000, 0.3716,  ..., 0.0000, 0.0908, 0.3951],
     [0.0000, 0.0028, 0.0427,  ..., 0.0000, 0.0000, 0.0246]]],
   device='cuda:0', grad_fn=&lt;ReluBackward0&gt;) torch. Size([11, 2, 64])'
</code></pre>

username: J_Johnson
cooked: <p>Nothing to do with LSTM. You’re using a ReLU function. It’s job is to turn negative values into zeroes.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html</a></p>
Gradient tracking in custom loss functions

username: pseudo1
cooked: <p><strong>Errors</strong><br>
I am trying to define a loss function for model training where the loss should be zero in many cases. I cannot simply return 0 in these cases because it returns an AttributeError, but if I try to return torch.Tensor(0) or the part of the function that sets it to zero, I get a Runtime Error that says <code>element 0 of tensors does not require grad and does not have a grad_fn</code>.</p>
<p>Any advice on how to resolve these errors to get the loss function working would be greatly appreciated!</p>
<p><strong>Objective</strong><br>
The code below takes in two discrete distributions with the same support, and a weight vector of equal length to the distributions. It checks a binary condition <code>if torch.all(diffs &gt;= 0)</code>, and if that condtion holds then it returns a type of weighted distance between the distributions. If the condition fails the loss should be zero. Neither returning 0 nor returning torch.all(diffs &gt;= 0) works for the failure case.</p>
<pre><code class="lang-auto">class MyLoss (torch.nn.Module):
    def __init__(self):
        super(MyLoss, self).__init__()
        
    def forward(self, predictions:torch.Tensor, target_dists:torch.Tensor, weights:torch.Tensor):
        pred_sums, target_sums = torch.cumsum(predictions,dim=1), torch.cumsum(target_dists,dim=1)
        diffs = torch.subtract(pred_sums, target_sums)
        if torch.all(diffs &gt;= 0):
            return torch.dot(torch.subtract(predictions,target_dists),weights)
        else:
            return 0
</code></pre>
<p>I suspect the problem is that torch.all prevents gradient tracking, but I’m not sure how to fix it.</p>

username: eqy
cooked: <p>Could you simply skip the backwards step and gradient update for iterations/batches that meet this condition  by checking if the returned loss is 0?</p>

username: pseudo1
cooked: <p>That should work! Thanks. I probably should have thought of that <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
[CUDA RPC] Incorrect results of GPU Tensor transferring using RPC when parallelized with other GPU programs

username: codecaution
cooked: <h3>
<a name="issue-summary-1" class="anchor" href="#issue-summary-1"></a>Issue Summary</h3>
<p>I transfer a GPU tensor between two GPUs using PyTorch RPC, and find incorrect results when parallelized with other GPU programs, such as a pure computation task in the backend using PyTorch. When the other GPU task is stopped, the results become right.</p>
<p>This suggests that other GPU programs could be interfering with the CUDA Support RPC, leading to incorrect message transfers.</p>
<p>Could anyone provide some insight or guidance on how to prevent other GPU programs from interfering with the CUDA Support RPC and ensure correct message transfers? Any help or suggestions would be greatly appreciated.</p>
<p>More details to reproduce my results can be found in <a href="https://github.com/pytorch/pytorch/issues/100784" rel="noopener nofollow ugc">Issue</a></p>
How to modify the nn:Sequential type parameter

username: geekfuns
cooked: <p>Hi everyone ,<br>
I want modify a nn:Sequential type in my project , In python ,I write the following code and it works:</p>
<pre><code class="lang-auto">temp_model = models.vgg19(pretrained=True)
feature_list = temp_model.features
features = list(feature_list)[:33]  # get vgg features
</code></pre>
<p>I mean i want use some layers of the pretrained model,How can i implement in libtorch?<br>
I have already write this code:</p>
<pre><code class="lang-auto">    vision::models::VGG19 models = vision::models::VGG19(1000, false);
    auto features = models-&gt;features;
</code></pre>
<p>Thanks a lot!</p>
the perfect layers to choose to unfreeze for efficientnetv2

username: ManSa
cooked: <p>I am working on creating a classification model, and I chose tf_efficientnetv2_b0 from timm library in pytorch with my images but I don’t know if the image size 224 is suitable for this model, also I want to ask about what is the perfect layers to choose to unfreeze for efficientnetv2</p>
<p>I don’t find any good source about the image size issue in the Internet, but about the unfreezing, I print the modules of the model and find this: 1-conv_stem 2-bn1 3-blocks 4-conv_head 5-bn2 6-global_pool 7-classifier I chose only for fine-tuning and unfreezing these layers (3,4,7) since the bn is needed to be frozen and layer 1 is very initial and contains global features from Imagenet, what I want to ask is if my implementation is correct.</p>
GPU memory consumption increases and run oom in training

username: bornexmachina
cooked: <p>The memory consumption starts with around 4GB but keeps increasing. In the mid of the training it OOMs.</p>
<p>I use VGG16 to extract features.</p>
<pre><code class="lang-auto">def get_features(data, N, body, scaler):
    data = data[0]
    MIN, MAX = min_max_lists(data, N)

    for i in range(len(data)):
            data[i] = scale_minmax(data[i], MIN[i], MAX[i], min=0, max=1)

    loader = DataLoader(data, shuffle=False, batch_size=32)

    features = []

    # the picture has to be upscaled to 224x224 
    # as the VGG16 needs this size
    # and I extract the features of the avgpool layer
    for batch in tqdm(loader):
        # in [0] the raw vectors of n_mels * n_frames are stored
        # and I reshape those to a 2D form
        batch = batch.reshape(batch.shape[0], 64, 64)
        # data in vectors needs to be transposed to actually resemble the slice of a 
        # mel spectrogram
        batch = np.transpose(batch, (0, 2, 1))
        #batch = batch / 255
        # now VGG takes RGB = 3 channels tensors,
        # so I have to copy channels 
        batch = np.repeat(batch[..., np.newaxis], 3, -1)
        # now a transpose, such that the tensor is of shape batchsize, channels, width, height
        batch = np.transpose(batch, (0, 3, 1, 2))
        batch = scaler(batch).float()
        #batch = torch.from_numpy(batch)
        inp = torch.nn.functional.interpolate(batch, size=(224, 224), mode='bilinear')
        inp = inp.float().to(device)
        with torch.no_grad():
            out = body(inp)
        out = torch.flatten(out['avgpool'], 1)    
        features.append(out.detach())
        del out

    # from here I concatenate the tensors to 
    # one big tensor
    features = torch.cat(features)
    features = features.detach()

    return features
</code></pre>
<pre><code class="lang-auto">m = vgg16().to(device)
body = create_feature_extractor(m, return_nodes={'avgpool':'avgpool'})
</code></pre>

username: ptrblck
cooked: <p>How many samples/features are you trying to store and how large is each one? I’m unsure if the OOM is unexpected or not depending on the size of each feature.</p>

username: bornexmachina
cooked: <p>I have something like 1000 samples. Each sample is 224 x 224 after upscaling. So shouldn’t be too huge.</p>
<p>I take the last layer (AVGPOOL), ResNet18 and 34 have an output of 512, but I am not sure for VGG16</p>
How to make sure which masking arguments I need to provide for calling torch.nn.Transformer model?

username: Cyber_punk
cooked: <p>The forward function of <a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" rel="noopener nofollow ugc">PyTorch’s Transformer implementation</a> <code>torch.nn.Transformer</code> have a number of masking arguments that are all <strong>optional</strong> :</p>
<p><code>forward(src, tgt, src_mask=None, tgt_mask=None, memory_mask=None, src_key_padding_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)</code></p>
<p>Parameters:</p>
<ul>
<li>src (Tensor) – the sequence to the encoder (required).</li>
<li>tgt (Tensor) – the sequence to the decoder (required).</li>
<li>src_mask (Optional[Tensor]) – the additive mask for the src sequence<br>
(optional).</li>
<li>tgt_mask (Optional[Tensor]) – the additive mask for the tgt sequence<br>
(optional).</li>
<li>memory_mask (Optional[Tensor]) – the additive mask for the encoder<br>
output (optional).</li>
<li>src_key_padding_mask (Optional[Tensor]) – the Tensor mask for src<br>
keys per batch (optional).</li>
<li>tgt_key_padding_mask (Optional[Tensor]) – the Tensor mask for tgt<br>
keys per batch (optional).</li>
<li>memory_key_padding_mask (Optional[Tensor]) – the Tensor mask for<br>
memory keys per batch (optional).</li>
</ul>
<p><strong>If I just want to use a vanilla Transformer, what masking arguments should I provide?</strong> <strong>What are the effects if I do not provide them?</strong></p>
<p>This is confusing to me because I found people selectively provide masking arguments.</p>
<p>For example, <a href="https://datascience.stackexchange.com/questions/97067/using-subsequent-mask-in-transformer-leads-to-nan-outputs" rel="noopener nofollow ugc">this one</a> provided everything except <code>memory_mask</code> in TransformerDecoder layer. How can I know whether I should provide it or not?</p>
<p>Another example, <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">this one</a> uses a look-ahead mask (upper triangular) in the encoder part.</p>
<p>And in <a href="https://buomsoo-kim.github.io/attention/2020/04/21/Attention-mechanism-19.md/" rel="noopener nofollow ugc">this example</a>, the author didn’t input any mask.</p>
Sparse Matrices and Broadcasting

username: skip
cooked: <p>Is there an obvious trick to working around the limitation of broadcasting on Sparse matrices?</p>
<p>Take for example:</p>
<pre><code class="lang-auto">a = torch.rand((4)).to_sparse()
b = torch.rand((4, 4)).to_sparse()
a * b
</code></pre>
<p>which throws:</p>
<pre><code class="lang-auto">RuntimeError: sparse_binary_op_intersection_cpu(): expects sparse inputs with equal dimensionality, number of sparse dimensions, and shape of sparse dimensions
</code></pre>
<p>I can’t use <code>to_dense()</code> as these are massive tensors, and I also do need <code>require_grad</code> with respect to <code>a</code>. Any workaround?</p>
<p>Thanks!</p>
Get logits with generation method

username: VePaKo
cooked: <p>I want to stack additional layers to pretrained T5 model, so they could use T5 output logits. But as I see bound method GeerationMixin? which provade method generate, returns only predicted tokens.<br>
Can I somehow get logits using generation method? Maybe there are more convinients method to stack layers? Or maybe I can somehow change generation method? so i can return logits?</p>
<p>Now the constructed model with forward method looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af8854879d9ed30af912858380d9bc9569c50ea9.png" data-download-href="https://discuss.pytorch.org/uploads/default/af8854879d9ed30af912858380d9bc9569c50ea9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af8854879d9ed30af912858380d9bc9569c50ea9_2_690x323.png" alt="image" data-base62-sha1="p2PDTjGgfmXUX0BuGAXgTQBrhI5" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af8854879d9ed30af912858380d9bc9569c50ea9_2_690x323.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af8854879d9ed30af912858380d9bc9569c50ea9_2_1035x484.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af8854879d9ed30af912858380d9bc9569c50ea9_2_1380x646.png 2x" data-dominant-color="EBEAEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1692×794 271 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Custom optimizer: how to access loss error after gradients are computed?

username: danbst
cooked: <p>Hello. I want to write a custom optimizer, which requires input activations and output error (regarding loss function) for each parameter.</p>
<p>I’ll try to explain, which “output error” I refer to. Consider linear regression without bias.<br>
<code>xW = y</code></p>
<p>and MSE loss function<br>
<code>loss = (xW - y).square().mean() / 2</code></p>
<p>Gradients are computed like<br>
<code>dW = (xW - y) * x = error * x</code></p>
<p>So autograd can compute <code>dW</code>, but I also want <code>error</code>. For this simple case and MSE loss function I can compute it manually alongside autograd, but for multilayer network this becomes complicated. Can torch autograd preserve this error matrix during backward pass for each parameter?</p>
<p>(another question is for input activations, <code>x</code>, is it possible to access those for each parameter using torch builtin features?)</p>
Good resource to learn TorchText

username: Abdelkareem_Elkhateb
cooked: <p>I searced for any books about nlp with pytorch but found only one and it’s not suitable with the updated for torchtext.<br>
I open the documents and followed the classification tutorial and want to apply it into an arabic dataset but when i get into the tokenizer = get_tokenizer(“”) I don’t know what to apply spacy is for english as they see in torchtext and theres is only basic_english I know that i am missing some steps but couldn’t figure it !</p>

username: J_Johnson
cooked: <p>Try sentencepiece. It’s a good way to turn a sampling of your text into a fixed number of tokens. You may need some trial and error, but it’s pretty straight forward.</p>
<p><a href="https://pytorch.org/text/stable/data_functional.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/text/stable/data_functional.html</a></p>
Multiclass classification without changing image sizes

username: volesteni
cooked: <p>hi all,<br>
my problem is quite difficult:<br>
I have multiclass classification problem.<br>
my images are:</p>
<ol>
<li>very different shapes and sizes, which I want to keep</li>
<li>the data is unbalanced</li>
</ol>
<p>different image sizes work when I fintune with “resnet18” for example, but I cannot use batches.<br>
so my idea was to use “virtual” batches. meaning, train 1 by 1 but calculate the gradients every 16 iterations, which will create the effect of training batched. (in this case I am deactivating batchnorm btw).</p>
<p>My training works ok (meaning Loss curves behave normal), but my results are poor.<br>
the other thing I tried doing is adding weights to my cross entropy loss to deal with the unbalanced data, but the results are EXACTLY the same as training without any weights at all.<br>
any ideas?</p>
<p>any ideas?</p>
Question about compiled model with FX graph op replacement

username: cokespace2
cooked: <p>Following this <a href="https://colab.research.google.com/drive/1Zh-Uo3TcTH8yYJF-LLo5rjlHVMtqvMdf#scrollTo=UklMVs56u9j7" rel="noopener nofollow ugc">colab tutorial</a>, I can get GraphModules of a compiled model, but during op replacement, I found something strange. Here’s what I found:</p>
<p>First, we have a very simple ResNet18 model:</p>
<pre><code class="lang-python">from torchvision.models import resnet18
model = resnet18()
model.eval()
</code></pre>
<p>and then we can compile this model and get its FX graph:</p>
<pre><code class="lang-auto">graph = []
def toy_backend(gm, inputs):
  graph.append(gm)
  return gm.forward
fn = torch.compile(model=mdoel, backend=toy_backend)
# run for compilation
inputs = torch.randn(1, 3, 224, 224)
output = fn(inputs)
</code></pre>
<p>we can print some modules and output shape:</p>
<pre><code class="lang-python">&gt;&gt;&gt; graph[0].self_fc
Linear(in_features=512, out_features=1000, bias=True)
&gt;&gt;&gt; fn.fc
Linear(in_features=512, out_features=1000, bias=True)
&gt;&gt;&gt; output.shape
torch.Size([10, 1000])
</code></pre>
<p>Ok, here we can replace the last <code>fc</code> layer with different parameters from FX graph, then run again:</p>
<pre><code class="lang-python">graph[0].self_fc = torch.nn.Linear(512, 10)
output = fn(inputs)
</code></pre>
<p>print again:</p>
<pre><code class="lang-python">&gt;&gt;&gt; graph[0].self_fc
Linear(in_features=512, out_features=10, bias=True)
&gt;&gt;&gt; fn.fc
Linear(in_features=512, out_features=1000, bias=True)
# see, the fn.fc still a 512x1000 Linear layer
&gt;&gt;&gt; output.shape
torch.Size([10, 10])
</code></pre>
<p>as one can see, after replacing older Linear layer with new one through FX graph, the compiled model still has the older version of module, but the output is what we like it to generate…</p>

username: marksaroufim
cooked: <p>Changing attributes of a compiled model is not entirely supported, you’re better off just recompiling after making changes. Also note that a compiled model isn’t meant to be changed as much as a typical nn module</p>
<p>Could you describe your use case a bit more?</p>

username: cokespace2
cooked: <p>Thanks, Mark.</p>
<p>I was trying to implement a quantization toolkit that can handle different types of models, which takes a nn module in, trace with torch fx and replace some modules to nnq modules, but fx fais to trace models with *args **kwags inputs, so I thought maybe I can use torch compile to do so.</p>
<p>Bwt in my case above, the modified compiled model does outputs what I hope it does, but it’s not by design?</p>
Forward function for object detection or instance segmentation

username: Jawad
cooked: <p>The forward function expects batched input, accordingly, we do not add a for loop for each sample inside it. However, in object detection or instance segmentation, each sample in the batch might have several instances. In this latter case, should I add a for loop for each instance inside my forward function and then concatenate the result?</p>
<p>For simplicity, Imagine the case of object detection. After the region proposal, we applied the ROI. Now for each image in the batch, I have the corresponding ROI for each proposal. The forward function will take the batch of images, so should I loop each ROI? how usually this issue is tackled. I tried to check the code for faster RCNN but I got a bit confused.</p>
Installing pytorch on Raspberry Pie

username: yuriisthebestsuyc
cooked: <p>I am still learning about Raspberry Pi and Pytorch, but I want to complete a project to run a vehicle detection project created by Yolov5 on Raspberry Pi. However, when I installed Pytorch&gt;=1.7.0 on Raspberry Pi, an error occurred. Is there any way I can install Pytorch on Raspberry Pie?</p>
<p>My raspberry Pi is 3B+, ARM71, Python 3.9.3</p>

username: juacamgo
cooked: <p>You must provide more info a out the problem you are facing in order to get help from someone.</p>
Output values of Module are incorrect if sub-module instance variable is lower-cased

username: rlaphoenix
cooked: <p>I have the following small RRDB module network:</p>
<pre data-code-wrap="python"><code class="lang-python">class RRDB(nn.Module):
    """Residual in Residual Dense Block."""

    def __init__(
        self,
        nc: int,
        kernel_size: int = 3,
        gc: int = 32,
        stride: int = 1,
        bias: bool = True,
        pad_type: Optional[blocks.PAD_TYPES_T] = "zero",
        norm_type: Optional[blocks.NORM_TYPES_T] = None,
        act_type: Optional[blocks.ACT_TYPES_T] = "leakyrelu",
        mode: blocks.CONV_MODE_T = "CNA",
        plus: bool = False
    ):
        super().__init__()
        self.RDB1 = ResidualDenseBlock5C(nc, kernel_size, gc, stride, bias, pad_type, norm_type, act_type, mode, plus)
        self.RDB2 = ResidualDenseBlock5C(nc, kernel_size, gc, stride, bias, pad_type, norm_type, act_type, mode, plus)
        self.RDB3 = ResidualDenseBlock5C(nc, kernel_size, gc, stride, bias, pad_type, norm_type, act_type, mode, plus)

    def forward(self, x):
        out = self.RDB1(x)
        out = self.RDB2(out)
        out = self.RDB3(out)
        # Empirically, we use 0.2 to scale the residual for better performance
        return out.mul(0.2) + x
</code></pre>
<p>Within it, you see I create three RDB block instances within it. If I keep them as self.RDBn it works correctly. If I lowercase them as self.rdbn, it returns incorrect output data noticeably.</p>
<p>I have absolutely no idea what could be causing this. I looked and the initialized structure of the instances matches both ways, but the output in forward does not.</p>
<p>Here’s a comparison of both outputs. I’m consistently getting an increase in green, even on sources that are mute of color.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e00073fa1a1a4fe264a4e7e5eb35257068d5e99a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e00073fa1a1a4fe264a4e7e5eb35257068d5e99a" title="diff"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e00073fa1a1a4fe264a4e7e5eb35257068d5e99a.jpeg" alt="diff" data-base62-sha1="vXBVhRVJVncMGnQ2gpzMmJc59oK" width="690" height="460" data-dominant-color="354815"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">diff</span><span class="informations">720×480 60.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: rlaphoenix
cooked: <p>The reason this occurred is because the PyTorch state/model pth file had the param name as uppercase, and it had to match. Therefore, when it ran the RRDB layer it did not have the weights applied.</p>
'Tensor' object is not callable Error is comingg

username: Ashman_Singh
cooked: <p>class LinearRegressionModel(nn.Module):</p>
<pre><code>def __init__(self):
    
    super().__init__()

    self.weight=nn.Parameter(torch.randn(1,requires_grad=True,dtype=torch.float))
    self.bias=nn.Parameter(torch.randn(1,requires_grad=True,dtype=torch.float))
def forward(self, x: torch.Tensor)-&gt;torch.Tensor:#"x is the input data"
    return self.weights*x+self.bias#y=weight*x+bias
</code></pre>
<p>a=LinearRegressionModel()<br>
Trying to run this basic code Getting This Error<br>
TypeError                                 Traceback (most recent call last)<br>
Cell In[18], line 1<br>
----&gt; 1 a=LinearRegressionModel()</p>
<p>Cell In[17], line 8, in LinearRegressionModel.<strong>init</strong>(self)<br>
4 def <strong>init</strong>(self):<br>
6     super().<strong>init</strong>()<br>
----&gt; 8     self.weight=nn.Parameter(torch.randn(1,requires_grad=True,dtype=torch.float))<br>
9     self.bias=nn.Parameter(torch.randn(1,requires_grad=True,dtype=torch.float))</p>
<p>TypeError: ‘Tensor’ object is not callable</p>

username: TzviNoy
cooked: <p>I think you put an extra equal sign in this line:</p>
<p><code>    self.bias=nn.Parameter=(torch.randn(1,requires_grad=True,dtype=torch.float))</code></p>

username: Ashman_Singh
cooked: <p>Yes thanks But still after that its giving the same error the error is in previous line</p>
Difference between AutogradNonFunctional and CompositeExplicitAutograd?

username: ryanc
cooked: <p>what is the difference between CompositeExplicitAutogradNonFunctional and CompositeExplicitAutograd?</p>
<p>I have read the documentation <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native#dispatch" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native at main · pytorch/pytorch · GitHub</a>. And it looks like the biggest difference is whether it calls internally into an aliasing operator.</p>
<p>How does pytorch know that the operator calls an aliasing operator internally and register the op(like mul.Tensor) to dispatch key CompositeExplicitAutogradNonFunctional? At the same time, why the index_put operator is registered to CompositeExplicitAutograd?</p>
The detected CUDA version (12.0) mismatches the version that was used to compile PyTorch (11.7)

username: IsakFoss
cooked: <p>Hi!<br>
I get this runtimeerror when trying to run my code:</p>
<p>File “C:\Users\isakf\anaconda3\envs\fastreg\lib\site-packages\torch\utils\cpp_extension.py”, line 386, in _check_cuda_version<br>
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))<br>
RuntimeError:<br>
The detected CUDA version (12.0) mismatches the version that was used to compile</p>
<p>This message is from the torch.utils.cpp_extension module.</p>
<p>Im struggling to see where I can download the torch version with CUDA 12 enabled.</p>
<p>Do you have any tips?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="IsakFoss" data-post="1" data-topic="169586">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isakfoss/48/56141_2.png" class="avatar"> IsakFoss:</div>
<blockquote>
<p>Im struggling to see where I can download the torch version with CUDA 12 enabled.</p>
</blockquote>
</aside>
<p>The PyTorch binaries do not support CUDA 12 yet and you would have to build from source using a locally installed CUDA 12.0 toolkit.</p>

username: Lucas1
cooked: <p>Hello, just curious if there is any update on pip torch supporting xformers with cuda 12. Thank you!</p>

username: ptrblck
cooked: <p>We are already building nightly PyTorch binaries with CUDA 12.1 but I don’t know which CUDA versions are supported in xformers.</p>

username: compascafe
cooked: <p>same problem!!, thanks. The problem with xformers, is only in the installation.</p>
M1 AssertionError: Torch not compiled with CUDA enabled

username: gcr
cooked: <p>Q: How can I get PyTorch 2.1.0 to use the M1 gpu instead of looking for CUDA?</p>
<p>“AssertionError: Torch not compiled with CUDA enabled”</p>
<p>I have an M1. I have installed PyTorch.</p>
<p><code>pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu</code></p>
<p>I am trying to run safetensors safe_open on a safe tensors file.<br>
Googling, I read that PyTorch now supports M1.</p>
<p>Thanks in advance</p>

username: arunppsg
cooked: <p>Try using <a href="https://pytorch.org/docs/stable/notes/mps.html" rel="noopener nofollow ugc">MPS backend</a> for accessing M1 gpu.</p>
Using flop_counter with torch.compile

username: Hileamlak_Yitayew
cooked: <p>Are compiled methods, <code>__torch_dispatch__</code> overrideable? I wanted to get the flop counts for inductor compiled model, and was trying to do something like,</p>
<p>import torchvision.models as models<br>
import torch</p>
<p>inp = torch.randn(1, 3, 224, 224, device=‘cuda’)<br>
mod = torch.compile(models.resnet18().to(‘cuda’))</p>
<p>flop_counter = FlopCounterMode(mod, depth=4)<br>
with flop_counter:<br>
res = mod(inp).sum().backward()<br>
But this causes all sorts of errors, was wondering if there were any work around it.</p>
Error "Torch is not able to use GPU" when installing Stable Diffusion WebUI

username: luca127
cooked: <p>I have a problem with Torch that occurs when installing AUTOMATIC1111’s Stable Diffusion WebUI.</p>
<p>The following error occurs every time: <code>Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check</code>.</p>
<p>I have tried all the solutions I found on the web without success. My problem seems to be similar to <a href="https://discuss.pytorch.org/t/torch-cant-use-gpu-but-it-could-before/170940">this one</a>. Unfortunately there is no problem with the CUDA_VISIBLE_DEVICES environment variable that could cause this behavior.</p>
<p>Further information:</p>
<ul>
<li>Win10, GeForce GTX 1060</li>
<li>nvidia-smi gives 12.1</li>
<li>
<code>print(torch.version.cuda)</code> returns 11.7</li>
<li>Executing <code>python -c "import torch; print(torch.randn(1).cuda())"</code> returns a blank line</li>
<li>Executing <code>python -c "import torch; print(torch.cuda.is_available())" </code> returns a blank line</li>
<li>Executing <code>python -m torch.utils.collect_env</code> throws this error:<br>
<code>AttributeError: 'NoneType' object has no attribute 'splitlines'</code>
</li>
<li>
<a href="https://www.reddit.com/r/StableDiffusion/comments/z6nkh0/comment/jb96mu3/?rdt=55322" rel="noopener nofollow ugc">This answer from reddit</a> couldn’t fix my problem either</li>
</ul>
<p>I really don’t know what else to try. Maybe one of you has some idea what is causing the problem?<br>
Thanks in advance</p>

username: ptrblck
cooked: <p>Maybe you could try to install PyTorch manually using our install instructions in another virtual environment to check if it can detect and use your GPU. Once this is done you could check if the mention application could reuse your PyTorch installation instead of installing a new one.</p>

username: luca127
cooked: <p>Installed torch in a new virtual enviroment as you suggested. Unfortunately the same thing happens again with <code>torch.cuda.is_available()</code>:<br>
It takes a moment and then an empty line is returned and python exits.</p>

username: ptrblck
cooked: <p>If the Python executable exits unexpectedly I would assume it crashes and you might want to debug it e.g. via <code>gdb</code>.</p>
RuntimeError: both arguments to matmul need to be at least 1D, but they are 0D and 2D

username: hizen
cooked: <p><a href="https://colab.research.google.com/drive/15WvxSutYHZOtPFiRBknkHE5Jzm7bCXLR?usp=sharing" rel="noopener nofollow ugc">Notebook LInk </a></p>
<p>Can anyone help me solved this problem</p>
How to convert smiles (.smi) to pickle (.pkl)

username: peony
cooked: <p>I converted my .smi files to .pkl but I am getting this error when I load the picke file. I think the way I converted it is wrong hence the error:</p>
<pre><code class="lang-auto">smi2pkl.py:16 in &lt;module&gt;
    qm9_smiles = pkl.load(f)

UnpicklingError: invalid load key, '['.
</code></pre>
<p>Here is a snippet of code I used for the conversion:</p>
<pre><code class="lang-auto">with open(out_path, "wb") as f:
    pkl.dump(file_path, f, protocol=pkl.HIGHEST_PROTOCOL)
</code></pre>
<p>Any suggestions for me is welcome. Thank you in advance</p>
FSDP with Apex Tensor Parallel and Sequence Parallel

username: EnricoShippole
cooked: <p>Hi all,</p>
<p>I was wondering if you could confirm that PyTorch FSDP, <code>from torch.distributed.fsdp import FullyShardedDataParallel as FSDP</code>, was compatible with Apex’s Tensor and Sequence Parallelism. So that one could use Column and Row Linear with Sequence Parallelism enabled in conjunction with the PyTorch FSDP model wrapper.</p>
<p>Thank you,</p>
<p>Enrico</p>

username: eqy
cooked: <p>Conceptually I’m not aware of any fundamental incompatibilities between FSDP and tensor/sequence parallelism, but given the sheer complexity and additional communication requirements of FSDP I’d be skeptical if there was a performance benefit from combining the two. Have you observed any issues (compatibility/performance/convergence)?</p>

username: EnricoShippole
cooked: <p>Thank you for the response.</p>
<p>I am testing models a few different models of sizes 150m, 410m, 1B, and 3B with TP, SP, and Flash on C4. I plan on scaling to larger models after. Before the next runs with FSDP, I wanted to survey and see if there were any potential incompatibilities. I will document the training runs with FSDP + TP + SP + Flash and provide all of the hyperparameters and results once they finish.</p>
<p>Best,</p>
<p>Enrico</p>
Reset CUDA_VISIBLE_DEVICES?

username: nlp_8172
cooked: <p>I am trying to use deepspeed to do distillation; I initialize the student model using the deepspeed parameters, and I want a separate GPU to load the teacher model (this model is sufficiently big that I can’t fit both student and teacher in one GPU). The issue is I can’t place the teacher model on a GPU deepspeed is already using, because it has its own process to split the models.</p>
<p>I want to limit deepspeed to a set of GPUs by setting CUDA_VISIBLE_DEVICES=0,1,2, for example, and then reset os.environ[‘CUDA_VISIBLE_DEVICES’]=0,1,2,3 later in the code to allow the teacher model to be placed on a fresh, empty GPU3. But this gives me an error: " <strong>DeferredCudaCallError:</strong> CUDA call failed lazily at initialization with error: device &gt;= <strong>0</strong> &amp;&amp; device <strong>&lt;</strong> num_gpus INTERNAL ASSERT FAILED at" or “CUDA invalid device ordinal”</p>
<p>So is it possible to reset CUDA_VISIBLE_DEVICES later in the code?</p>

username: eqy
cooked: <p>I don’t think it’s safe/supported to reset <code>CUDA_VISIBLE_DEVICES</code> in the middle of script execution (e.g., after CUDA has been initialized): <a href="https://discuss.pytorch.org/t/os-environ-cuda-visible-devices-not-functioning/105545/4" class="inline-onebox">Os.environ ["CUDA_VISIBLE_DEVICES"] not functioning - #4 by albanD</a></p>
<p>If you want to restrict the number of GPUs deepspeed is using, I would check if e.g., setting the <code>world_size</code> parameter or something like <code>nproc_per_node</code> could achieve this.</p>
Build pytorch for H1000

username: funtuan
cooked: <p>Hi, I just got hands on H1000, could someone guide me on how to correctly build pytorch from source?</p>

username: eqy
cooked: <p>Have you taken a look at the build from source instructions: <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration (github.com)</a>?</p>
<p>Note that if you are building from source in order to have H100/sm90 support this is not needed as any PyTorch built with CUDA 11.8 or higher would have it (e.g., the current stable release which can be installed via <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code>).</p>

username: funtuan
cooked: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="179249">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>Note that if you are building from source in order to have H100/sm90 support this is not needed as any PyTorch built with CUDA 11.8 or higher would have it (e.g., the current stable release which can be installed via <code>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118</code>).</p>
</blockquote>
</aside>
<p>I found this post: <a href="https://gist.github.com/Birch-san/211f31f8d901dadd1025398fa1a603b8" class="inline-onebox" rel="noopener nofollow ugc">Installing CUDA 12.1.1 + PyTorch nightly + Python 3.10 on Ubuntu 22.10 · GitHub</a> which is very helpful, However, it shows</p>
<blockquote>
<p>NVIDIA H100 PCIe with CUDA capability sm_90 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_89.<br>
If you want to use the NVIDIA H100 PCIe GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
</blockquote>

username: eqy
cooked: <p>Right, you are getting that error message because that guide is specific to sm89/Ada, as there is <code>TORCH_CUDA_ARCH_LIST=8.9</code> in the build command. If you change that to <code>TORCH_CUDA_ARCH_LIST=9.0</code>, then it would work with sm90/H100. The reason the start locally instructions are linked (same as the install command I posted) is that the prebuilt binaries are compiled with support for a wide range of compute capabilities, including both sm89 and sm90.</p>

username: funtuan
cooked: <aside class="quote no-group" data-username="eqy" data-post="4" data-topic="179249">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>9.0</p>
</blockquote>
</aside>
<p>awesome thanks a lot!</p>
Using two schedulers?

username: Fabi
cooked: <p>Hi Guys,</p>
<p>im currently trying to train my net with the adam optimizer.<br>
For this i want to start with a learning rate of 1e-6 and slowly increase it to 1e-4 after 10,000 steps.<br>
After that i want to decrease the learing rate by 0.5 every 100k steps.</p>
<p>I tried using two MultiStepLR schedulers but stepping both right after another renders the first one useless. Also the base learning rate of both schedulers are automatically set to the same value.</p>
<p>So to my question:<br>
Is there a elegant way of including this scenario into one scheduler or am I bound to step each exclusivly and fixing the base learning rate of one them to make it work?</p>
<p>Thanks a lot in advance!<br>
Fabi</p>

username: ptrblck
cooked: <p>I think the easiest way would be to use the <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR">LambdaLR</a> and to define your learning rate manipulation in a separate function.</p>

username: vincentqb
cooked: <p>By the way, you could also <a href="https://github.com/pytorch/pytorch/pull/26423#issue-318970152" rel="noopener nofollow ugc">chain schedulers</a>, where an if statement triggers the second one at the epoch you want.</p>

username: Mihai_George_Mehedin
cooked: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.SequentialLR.html#torch.optim.lr_scheduler.SequentialLR" rel="noopener nofollow ugc">SequentialLR</a> is applicable too.</p>
Parallel inference in Pytorch

username: oracal
cooked: <p>When I perform deep learning inference in PyTorch, some models have branch structures in their architecture. In this case, does PyTorch consider dispatching the operators on these branches to different streams for parallel execution? Based on my testing results, it seems that there is no such mechanism. So I want to know why this is the case, or how I can achieve this parallel inference.<br>
Thank you so much！</p>

username: marksaroufim
cooked: <p>AFAIK Python doesn’t have a mechanism for branch prediction because in python and pytorch code is executed eagerly as in line by line</p>
<p>So to make code run faster you can use a jit like <code>torch.jit.script</code> or <code>torch.compile</code> which will specialize on or another branch and there’s some active discussions about explicitly adding control flow ops in torch <a href="https://github.com/pytorch/pytorch/issues/99598" class="inline-onebox" rel="noopener nofollow ugc">Add support for dynamic control flow in torch.fx · Issue #99598 · pytorch/pytorch · GitHub</a></p>
Question about the usage of DistributedDataParallel

username: Ford
cooked: <p>I try to improve the performance of my model by introducing a pre-trained teacher model. I have tried many ways to implement it. Unfortunately, none of them is successful. So here I come to request for help.</p>
<p>I incorporate two models. One is a pre-trained teacher model <strong>T</strong>, and the other is a student model <strong>S</strong>. The features generated by the intermediate layers of the student model <strong>S</strong>, are supervised by the features generated by the corresponding layers of the teacher model <strong>T</strong>. In other words, the loss function is the distance between the feature maps of <strong>S</strong> and <strong>T</strong>. The teacher model is frozen, while the student model is trained on 8 cards using DistributedDataParallel, <em>i.e,</em></p>
<blockquote>
<p>torch.nn.parallel.DistributedDataParallel(<strong>S</strong>)</p>
</blockquote>
<p>. The question is how to process the teacher <strong>T</strong> to make sure that the student models distributed at every cards are trained using the same teacher model? Do I need to apply torch.nn.parallel.DistributedDataParallel() on <strong>T</strong>? Can the trained teacher model in evaluation mode <em>i.e.</em> <strong>T</strong>.eval(), be distributed to multi cards？Thanks.</p>

username: pritamdamania87
cooked: <p>Since the teacher model is frozen, you don’t need to use <code>torch.nn.parallel.DistributedDataParallel</code> for it. You can do the following on each process:</p>
<pre><code class="lang-auto">T = torch.load(...) # load the teacher model
S = torch.load(...) # load the student model
ddp_S = torch.nn.parallel.DistributedDataParallel(S)

# Now you can use T and ddp_S for your training.
</code></pre>
Penalize some classes more than others in cross entropy (but only for a specific class!)

username: steeltrooper17
cooked: <p>I have three classes: small drone, large drone and bird.</p>
<p>I want my loss function to penalize small drone and large drone predictions equally when classifying a bird sample.<br>
However, if the sample is actually a small drone – it should penalize large drone misclassification (small drone sample misclassified as large drone) less compared to a bird misclassification (small drone sample misclassified as bird).<br>
Similarly, if the sample is actually a large drone – it should penalize small drone misclassification (large drone sample mis classified as small drone) less compared to a bird misclassification (large drone sample mis classified as bird).</p>
<p>How would I achieve this? Do I need to make a custom loss function or is this supported in pytorch?</p>

username: zshn25
cooked: <p>You could do multi-stage binary classification.</p>
<ol>
<li>Stage: 0 for bird, 1 for drone</li>
<li>Stage: only if drone in 1st stage.</li>
</ol>
<p>Final loss would be combination of the above with additional weighting if required</p>

username: KFrank
cooked: <p>Hi Rohit!</p>
<aside class="quote no-group" data-username="steeltrooper17" data-post="1" data-topic="179168" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/steeltrooper17/48/621_2.png" class="avatar"> steeltrooper17:</div>
<blockquote>
<p>How would I achieve this?</p>
</blockquote>
</aside>
<p>If I understand your question, you will need a custom loss function.</p>
<p>I believe that your use case is addressed by this post (and its entire thread):</p>
<aside class="quote quote-modified" data-post="4" data-topic="157369">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/create-a-variation-of-cross-entropy-loss-with-per-pair-of-classes-weights/157369/4">Create a Variation of Cross Entropy Loss with Per Pair of Classes Weights</a> 
  </div>
  <blockquote>
    Hi David! 

Cross entropy (by definition) doesn’t work this way.  But you can write 
some other loss function that does have “per-pair” penalties. 
I give a couple of possibilities in the example script, below.  misclassA() 
just weights the probability for an incorrect prediction by the per-pair 
weight given in your matrix D. 
My guess is that such a loss function won’t train very well.  In my experience, 
the logarithmic divergence for bad predictions in cross entropy seems to 
be very helpfu…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>
Does the pytorch celebA dataloader has anything to do with with celebaHQ?

username: jS5t3r
cooked: <p>I want to load celeba in HQ.</p>
<p>The pytorch celebA dataloader has nothing to do with with celebaHQ, right?</p>
<p><a href="https://pytorch.org/vision/main/_modules/torchvision/datasets/celeba.html#:~:text=%23%20right%20now.-,file_list,-%3D%20%5B%0A%20%20%20%20%20%20%20%20%23%20File" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/main/_modules/torchvision/datasets/celeba.html#:~:text=%23%20right%20now.-,file_list,-%3D%20[
%20%20%20%20%20%20%20%20%23%20File</a></p>