akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I could not find inf module in torch

username: wasifbaloch
cooked: <p>someone is importing this <code>from torch import inf</code> but it is giving error and when I searched also could not find this module in torch documention. can anybody explain the reason</p>

username: ptrblck
cooked: <p>I’m not aware of such a module so do you know what it’s supposed to contain? Or do you mean <code>torch.isinf</code>?</p>

username: wasifbaloch
cooked: <p>not <code>torch.isinf</code>. It was importing like I stated above</p>

username: ptrblck
cooked: <p>In that case it might not be a module but the infinity definition. What kind of error are you seeing?</p>

username: wasifbaloch
cooked: <p>importerror: cannot import name ‘inf’ from ‘torch’</p>

username: ptrblck
cooked: <p>It works for me:</p>
<pre><code class="lang-python">&gt;&gt;&gt; from torch import inf
&gt;&gt;&gt; inf
inf
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.tensor(inf)
tensor(inf)
&gt;&gt;&gt; 1. / torch.tensor(inf)
tensor(0.)
&gt;&gt;&gt; torch.tensor(inf) / torch.tensor(inf)
tensor(nan)
&gt;&gt;&gt; inf / inf
nan
</code></pre>
<p>so you might need to update your PyTorch version.</p>

username: wasifbaloch
cooked: <p>Thank you. I changed to <code>np.inf</code> that works for me because I could not update Pytorch due CUDA version.</p>
Latency of inference with multithreads and one thread on cpu

username: teenwangjie
cooked: <p>hello, i use libtorch version=1.10.0 to inference my model on cpu(cenos, x86) with one thread, the latency is ~30ms while the latency is ~70ms when setting threads=64(system has 64 cores). Note: each thread has independent computing resources,and only using one thread for inference. eg. at::set_num_threads(1), is that normal? Subconsciously think this is normal because the CPU becomes busy, but I wonder if there is a more detailed explanation？thanks</p>
How to write if condition on a certain output

username: peony
cooked: <p>I want to do backward pass when the output is the logit 4. This is the relevant part of my code:</p>
<p>if output[:, 4] == True:<br>
output[:, 4].backward()</p>
<p>The if condition is not working, the rest is ok. Any ideas for me how to write this condition? Thank you in advance</p>

username: ptrblck
cooked: <p>I don’t quite understand</p>
<aside class="quote no-group" data-username="peony" data-post="1" data-topic="182053">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e5b9ba/48.png" class="avatar"> peony:</div>
<blockquote>
<p>when the output is the logit 4</p>
</blockquote>
</aside>
<p>Do you want to compare the actual logit value against <code>4</code> or slice the output?</p>

username: peony
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am not sure I understand what slicing the output is. But I have up to 20 classes. The output is from 1 to 20</p>
<p>If the output is 4, then I want to do backward pass</p>

username: ptrblck
cooked: <p>Do you want to check if the logit for <code>class4</code> has the largest value and thus <code>class4</code> would be predicted?<br>
If so, use <code>torch.argmax</code> to check for the predicted class and use it then as a condition.</p>
RuntimeError: CUDA error: invalid device ordinal

username: Rongpengju
cooked: <p>import torch<br>
import time<br>
print(torch.<strong>version</strong>)<br>
print(torch.cuda.is_available())</p>
<p>a = torch.randn(10000,1000)<br>
b = torch.randn(1000,2000)</p>
<p>device = torch.device(‘cuda:1’)<br>
a = a.to(device)<br>
b = b.to(device)</p>
<p>start_time = time.time()<br>
c = torch.matmul(a,b)<br>
end_time2 = time.time()<br>
print(a.device,end_time2-start_time,c.norm(2))</p>
<p>Why does this code generate such an error? When I call is available, the output is true, my CUDA version is 10.1, and pytorch version is 1.3.1. I hope to get your help</p>

username: crowsonkb
cooked: <p>CUDA devices start counting at 0. You need to use <code>torch.device('cuda:0')</code> or better yet <code>torch.device('cuda')</code>, which uses the currently selected device instead of hard-coding it.</p>

username: GoingMyWay
cooked: <p>Hi, what if there are 2 GPUs and I want to use the second one?</p>

username: ptrblck
cooked: <p>Use <code>'cuda:1'</code> if you want to select the second GPU while both are visible or mask the second one via <code>CUDA_VISIBLE_DEVICES=1</code> and index it via <code>'cuda:0'</code> inside your script.</p>

username: Anmol_Varshney
cooked: <p>I have Number of available CUDA devices: 1 in google colab but it is showing the error<br>
RuntimeError: CUDA error: invalid device ordinal<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
Different inference results across x86 and ARM64

username: move47
cooked: <p>Hi,</p>
<p>I am using pre-trained GPT-2 from Hugging Face. I am interesting in getting the same inference results across both x86 and ARM64.</p>
<p>Currently, I am getting different results across the CPUs. I would appreciate if anyone can suggest a way to make it possible to have same inference results?</p>
<p>Thanks.</p>
Can't install PyTorch 2.0

username: Marcus_Young
cooked: <p>I’m trying to install PyTorch 2.0 on a GCP VM instance with pip but there the latest version isn’t showing. I’m using Python 3.9 and have pip 23.1.2.</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch&gt;=2.0 (from versions: 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.0+cu117, 1.13.1, 1.13.1+cu117)
ERROR: No matching distribution found for torch&gt;=2.0
</code></pre>
<p>Any help would be appreciated!</p>

username: Marcus_Young
cooked: <p>Resolved. The issue was I was running the script with the python version from my bin rather than the virtual env.</p>
`groups` parameter for `nn.Linear`

username: moi90
cooked: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#conv2d" rel="noopener nofollow ugc"><code>nn.Conv2d</code></a> has a parameter <code>groups</code>:</p>
<blockquote>
<p><code>groups</code> controls the connections between inputs and outputs. <code>in_channels</code> and <code>out_channels</code> must both be divisible by <code>groups</code>. For example,<br>
- At groups=1, all inputs are convolved to all outputs.<br>
- At groups=2, the operation becomes equivalent to having two conv layers side by side</p>
</blockquote>
<p>I think, something similar makes sense for <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear" rel="noopener nofollow ugc"><code>nn.Linear</code></a>. For example, it would have the advantage of reducing the number of parameters.</p>
<p>This is what I came up with:</p>
<pre data-code-wrap="python"><code class="lang-python">class GroupedLinear(nn.Module):
    def __init__(
        self,
        in_features: int,
        out_features: int,
        bias: bool = True,
        groups=1,
        device=None,
        dtype=None,
    ) -&gt; None:
        super().__init__()

        assert in_features % groups == 0
        assert out_features % groups == 0

        self.in_features = in_features
        self.out_features = out_features
        self.groups = groups

        self._linear_layers = nn.ModuleList(
            [
                nn.Linear(
                    in_features // groups,
                    out_features // groups,
                    bias=bias,
                    device=device,
                    dtype=dtype,
                )
                for _ in range(groups)
            ]
        )

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = x.view((x.shape[0], self.groups, -1))
        result = [
            l(x[:, i])
            for i, l in enumerate(self._linear_layers)
        ]
        return torch.cat(result, dim=1)
</code></pre>
<p>Is there a better way?</p>
Distributed Training on PyTorch

username: Jaeseon
cooked: <p>I’m currently immersed in a project where I’m leveraging PyTorch to develop an object detection model using satellite imagery. My immediate objective is to perform distributed training on this model using PySpark.<br>
While I have found several tutorials and examples on image classification, I’m having trouble translating these resources to suit my needs.<br>
Specifically, to the best of my knowledge,  I think that I need to load images and annotation files using PySpark, and subsequently convert or transform these files into a format that’s compatible with PyTorch for the purpose of object detection model building. I’m eagerly seeking advice or any pointers towards helpful tutorials or examples that can aid me in refining and constructing my model.<br>
During my search, I have stumbled upon some resources related to using SparkTorch or pyspark.ml.torch.distributor, and horovod. However, I’ve been encountering difficulties in successfully installing horovod. I appreciate any guidance on this issue as well.</p>

username: tiramisuNcustard
cooked: <p>One alternative to PySpark is Ray. You can do a little research online to see if it is something you want to use. Ray is used a lot in distributed learning so there are plenty of guides online.</p>

username: Jaeseon
cooked: <p>I have experience building object detection models using PyTorch or TensorFlow, but I’m new to the concepts of data parallelization and distributed training, which has left me feeling confused. As a result, I have a few questions to help clarify my understanding.</p>
<p>I currently have my image and annotation files stored in HDFS (Hadoop Distributed File System) directories. Initially, I assumed that I would need to use PySpark to load the data for the purpose of data parallelization and distributed training in deep learning models. However, after reading about examples of ‘Ray,’ I realized that it does not rely on Spark.</p>
<p>Here are my questions:</p>
<ol>
<li>Does Ray fulfill a similar role to Spark?</li>
<li>Do you believe it’s unnecessary to use PySpark even if our files are stored in HDFS?</li>
<li>When using Ray, is it necessary for me to have stored my files in HDFS?</li>
</ol>

username: tiramisuNcustard
cooked: <p>I don’t use Spark and HDFS, so my input will be very limited.</p>
<ol>
<li>
<p>Ray is lower level compared to Spark. You can run Spark on top of Ray. Spark is recommended when database operations are required. See this discussion: <a href="https://discuss.ray.io/t/what-is-the-difference-between-ray-and-spark/1578" rel="noopener nofollow ugc">https://discuss.ray.io/t/what-is-the-difference-between-ray-and-spark/1578</a></p>
</li>
<li>
<p>You can load from HDFS when using Ray. See this discussion: <a href="https://discuss.ray.io/t/ray-distributed-load-from-hdfs/6043" rel="noopener nofollow ugc">https://discuss.ray.io/t/ray-distributed-load-from-hdfs/6043</a></p>
</li>
<li>
<p>It is not necessary. You can use dataloaders, imagefolders etc.</p>
</li>
</ol>
Pretrained model for face detection

username: Surekha_Gaikwad
cooked: <p>Is there any pre-trained model for face detection in pytorch using deep learning approach?<br>
Note:- Not using haar cascade or viola jones using openCV</p>

username: Naveen_Kumar
cooked: <p>Here you go, <a href="https://github.com/timesler/facenet-pytorch" rel="nofollow noopener">https://github.com/timesler/facenet-pytorch</a>  I have not used this personally. you might face issues or might not but you need to take forward from here.</p>

username: Surekha_Gaikwad
cooked: <p><a class="mention" href="/u/naveen_kumar">@Naveen_Kumar</a> thanks for ur input but facenet is for face recognition and I want for face detection…but anyways I got it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: timesler
cooked: <p>The repo linked above by <a class="mention" href="/u/naveen_kumar">@Naveen_Kumar</a> includes both face detection and recognition (as do many facial recognition libraries). It uses a pretrained MTCNN network for the detection.</p>
<p>Disclaimer: <a href="https://github.com/timesler/facenet-pytorch" rel="nofollow noopener">https://github.com/timesler/facenet-pytorch</a> is my repo</p>

username: Kamal
cooked: <p>I’m newbie so not sure what’s the right way to get help about this <a class="mention" href="/u/timesler">@timesler</a> . I tried using your package but I got the following error:<br>
moduleNotFoundError: No module named ‘torchvision.ops’</p>
<p>I’m using miniconda and installed pytorch using conda but your package using pip. Not sure if that’s causing an issue. I’m using<br>
python 3.7.11<br>
pytorch 1.9.0<br>
torchvision 0.2.2<br>
facenet-pytorch 2.5.2</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kamal" data-post="5" data-topic="62362">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kamal/48/53088_2.png" class="avatar"> Kamal:</div>
<blockquote>
<p>torchvision 0.2.2</p>
</blockquote>
</aside>
<p>This release is quite old as the latest <code>torchvision</code> release is <code>0.13.1</code> so you might want to update it.</p>

username: kartikpodugu
cooked: <p><a class="mention" href="/u/timesler">@timesler</a><br>
I am unable to export the mtcnn model from facenet-pytorch repo into ONNX model as it does lot of operations outside the 3 stage models(p,o,r nets) and uses numpy arrays instead of torch arrays. Any ideas of making conversion to ONNX model ?</p>
Attention mask post STFT

username: Peter_Featherstone
cooked: <p>I have model which consumes a 1D time series, i.e. a tensor with shape [B,T], and an attention mask, i.e. a boolean tensor with shape [B,T].<br>
The model first performs an STFT on the input time series, yielding a tensor of shape [B,F,N] where F is the FFT size and N is the number of time frames.<br>
I want to calculate the attention mask at the output of the STFT, i.e. another boolean attention tensor with shape [B,F,N].<br>
How do I do this?</p>
Feature extraction from log-mel spectrograms using CNNs

username: rynczakd
cooked: <p>I am currently working on an ASR-related project in which I would like to combine Convolutional Neural Network with GRU Network and CTC loss function.<br>
The idea is to use the CNN to extract representative features from log-mel spectrograms, and then pass the resulting feature maps as input to the GRU network, after first reducing the dimensions of the maps to (Batch Size x C = 1 x H x W).</p>
<p>I have read the publications on this topic, but I have not found information on whether such a solution makes sense. I know that log-mel spectrograms contain features extracted from audio files, but I would like to make the features more meaningful.</p>
<p>I was planning to use the feature extraction module from the VGG-16 network, but my spectrograms have dimensions of 1x23xW, where W ranges from 600 to 1500 (I made the padding up to the maximum width in the batch), and I cannot then use this module to fine-tune the model and apply it to the GRU because of the discrepancy in the dimensions of the images.</p>
<p>Can you tell me what kind of CNN to use to extract features from spectrograms of such dimensions? I would also appreciate knowing if it makes sense to use CNNs in such a case.</p>

username: bloos
cooked: <p>have a look at the wav2vec2 paper by facebook ai. They are not using melspectograms but use convolutions directly on the raw audio data, but the idea of combining a convolutional network for feature extraction and a transformer/recurrent type network for processing these features is used there as well.</p>

username: rynczakd
cooked: <p>As far as I remember Wav2Vec 2.0 uses 1D convolutional layer applied for each time frame of raw audio data. As a result there is created a matrix which rows correspond to vectors extracted with CNN and columns represent time steps. In general - the idea is the same, but I would like to apply CNN to log-mel spectrograms and then pass the output into GRU network. The main question is - Will the use of CNNs produce a better representation of features from spectrograms? And what information can I extract by applying convolutional layers in terms of speech representation?</p>
Why is my CAM always predicting the same area?

username: peony
cooked: <p>Here is the part of my code relevant to the issue:</p>
<pre><code class="lang-auto">    def forward_hook(module,input,output):
        activation.append(output)

    def backward_hook(module,grad_in,grad_out):
        grad.append(grad_out[0])


    model.layer4[-1].register_forward_hook(forward_hook)
    model.layer4[-1].register_backward_hook(backward_hook)
    grad=[]
    activation=[]


 loader_iter = iter(dataloader_test)
    for _ in range(50):
        data, target, meta = next(loader_iter)       
      
        for d, t, m in zip(data, target, meta):

            hm_dogs = []
            heatmap = []
            d, t = map(lambda x: x.to(device), (d, t))
            
            #remove batch size
            d = d.unsqueeze(0)
            output = model(d)

            output[:, 4].backward()
            #get the gradients and activations collected in the hook
            grads=grad[0].cpu().data.numpy().squeeze()
            fmap=activation[0].cpu().data.numpy().squeeze()
</code></pre>
<p>Basically, only the first sample is getting the right heatmap properly.</p>
<p>Edit:<br>
I tried to iterate but basically the gradients and activations that are being hooked are all the same values for all samples.</p>
<p>Any ideas for me? Thank you in advance</p>
Why does data augmentation decrease the performance of efficientnet_v2_s on ILSVRC2012?

username: a2108870
cooked: <p>I have been trying to train the efficientnet_v2_s classification model on ILSVRC2012 using the official code provided by PyTorch. However, I have noticed that when I apply the automatic augmentation strategy, the model’s classification performance is lower compared to when no data augmentation strategy is applied, by approximately 2%. I have also tested the model’s performance with color jitter augmentation and found that it is still lower than when no data augmentation strategy is used. Could you please explain why this is the case?</p>
LSTM vs GRU performance

username: noahw31
cooked: <p>Hello,</p>
<p>I am implementing approaches to model transfer behavior of a device feeding sequential input data with temporal dependencies. Therefore, the obvious approach are RNNs, more specifically I am using LSTM, GRU and JANET, and it works pretty good as expected. Now I have a couple of questions which came up while comparing the performance of these approaches.</p>
<ol>
<li>
<p>I have the exact same implementation for LSTM and GRU, i.e., same data procedure, exact same architecture except using <code>nn.GRU()</code> instead of <code>nn.LSTM()</code>, and exact same training, validation etc…<br>
I do the training using input dimensions of <code>(5, 100, 60)</code> where <code>(batch_size, sequence_length, input_size)</code> as described in <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" class="inline-onebox" rel="noopener nofollow ugc">LSTM — PyTorch 2.0 documentation</a> .<br>
For validation i want to feed my whole sequential input data (not split up into sequences) of length <code>300e3</code>. While this works very well using the LSTM, it seems to be extremely slow for the GRU.<br>
In numbers, the LSTM takes e.g., 48 seconds for 5 epochs while the GRU takes 12 minutes! Especially the part where i feed a long input to the GRU, it becomes very slow. The training speed itself is comparable to the LSTM.<br>
There is literally not a single line of code different from the LSTM except for the initialization of the network as mentioned before. Maybe someone has an idea what the problem could be, and i am happy to provide code if needed.</p>
</li>
<li>
<p>This is more a general question on data preparation. The NN should model the transfer behavior of a device that introduces nonlinearity and amplification to the input stream, meaning the relation between input data and output data (which should be modeled by the NN) is e.g.,</p>
</li>
</ol>
<pre><code class="lang-auto">x_in = [0.2, 0.3, 0.5]
x_out = [0.4, 0.57, 0.94]
</code></pre>
<p>These are just example values and the range of the numbers can be very large (very small to large). Now i was wondering if i could help the network to learn the nonlinear behavior by doing normalization or something similar, but i am not sure about this thought, and would be happy for advice.</p>
<p>Thanks in advance!</p>
Swin Transformer for image classification

username: Swapnil_Baad
cooked: <p>I am performing transfer learning on the pretrained ‘swin_tiny_patch4_window7_224’ model, but my input image size is 100x100x3, whereas the model input size is 224x224x3.</p>
<p>How can I modify the architecture so I can use it for 100x100x3 input images?</p>
Predicting a list of values

username: gyamfi
cooked: <p>Is it possible to create a model which predicts a list of values?<br>
The problem is that the list of values is ambiguous.<br>
The input data is always one numpy array (shape: (1023,256) )<br>
The label data is stored as csv. For each input array there is a csv with rows of those values. But in one case there are 100 rows and in another there are only 90 rows.</p>
<p>So the network gets one array and predicts a list of values.</p>
<p>Thanks in advance.</p>

username: J_Johnson
cooked: <p>Sure. There are different types of models that can accomplish this, and the ideal one may depend on whether the data is spatial-temporally related or not. Is the data a time or spatial dependent sequence or is each value orthogonal?</p>
<p>Either way, you should normalize your model inputs and targets.</p>
<p>See: <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.normalize.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.normalize — PyTorch 2.0 documentation</a></p>
<p>When calculating your loss, you can make use of CrossEntropyLoss, MSELoss, or L1Loss functions.</p>

username: gyamfi
cooked: <p>Thanks. The Input Arrays store information about the distance and velocity of objects.</p>
How to apply CNN to live/variable-length audio?

username: anon_1101
cooked: <p>I have a simple convolutional model capable of classifying 8 spoken words from ~1 second long audio clips at a <code>sample_rate</code> of 16000. Training/test accuracy is good enough to where I want to start testing it on live microphone audio recorded directly on the device. How would I write an inference function that takes a variable-length audio input (microphone) instead of the 1-sec long audio it was trained on?</p>
<p>I assume I have to use sliding windows with a width equal to my sample rate or something similar, however, I am not too confident in this approach.</p>
<p>Feedback and advice are greatly appreciated, thank you.</p>

username: ElsebaiyMohamed
cooked: <p>i think this may be valid for you</p>
<pre><code class="lang-auto">all_audio = [] # intial empty
while True:
    mic_audio = #get from mic
    all_audio.extend(mic_audio )
    if len(all_audio) &gt; segment_size:
        segment = all_audio[:segment_size]
        all_audio = all_audio[stride_size:]
        result_of_segment = my_model(segment)
</code></pre>

username: bloos
cooked: <p>I’ve run some experiments with 1d convolutions on audio data of variable length too and have discovered some problems that can occur:</p>
<ol>
<li>the network might learn the signal length indirectly and turn it into a feature. Probably not what you want</li>
<li>the approach forces you to use a batch size of 1, which means that you cannot use batch normalization techniques. Other normalization approaches were even worse in my experiments as layer norm led to divergence.</li>
</ol>
<p>What I did to solve the issues during training:</p>
<ol>
<li>apply different padding techniques like zero padding or repetition, to feed a whole batch into one tensor</li>
<li>use batch normalization</li>
</ol>
<p>I’ve read your post again and as it seems you are talking only about inferencing an already existing network. In this case you can slice your data to match the desired input.</p>
Shape error in target

username: Anirudh_G
cooked: <p>I was attempting to take a shot at the SETI ET 2021 competition in Kaggle now, as a practice of course. However, I am running into a shape error which I don’t seem to understand.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4326860d6d97e5d2b79649414bd0b9672246fcc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/4326860d6d97e5d2b79649414bd0b9672246fcc8" title="Screenshot 2023-06-12 at 8.58.42 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4326860d6d97e5d2b79649414bd0b9672246fcc8_2_690x304.png" alt="Screenshot 2023-06-12 at 8.58.42 PM" data-base62-sha1="9A2wFXaoyh6Gi8LYH4Qqq1p2hzG" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4326860d6d97e5d2b79649414bd0b9672246fcc8_2_690x304.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4326860d6d97e5d2b79649414bd0b9672246fcc8_2_1035x456.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/4326860d6d97e5d2b79649414bd0b9672246fcc8_2_1380x608.png 2x" data-dominant-color="E9D2C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-06-12 at 8.58.42 PM</span><span class="informations">2380×1050 475 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The notebook is here for reference - <a href="https://www.kaggle.com/code/anirudhg15/seti-et-2023-w-pytorch" rel="noopener nofollow ugc">https://www.kaggle.com/code/anirudhg15/seti-et-2023-w-pytorch</a></p>

username: ptrblck
cooked: <p><code>nn.BCEWithLogitsLoss</code> expects model outputs and targets in the same shape while your model output is 4-dimensional. In case you are working on a binary classification use case make sure the model output has the shape <code>[batch_size, 1]</code>.</p>

username: Anirudh_G
cooked: <p>Thanks for the response.</p>
<p>It is a binary classification problem. I have tried a few different variations but not able to figure out what is causing this shape error. I am new to PyTorch and hence having a hard time debugging it (I am missing something obvious but not able to figure it out)</p>
<p>I am using the swintranformer from timm. Is there a way where I can check the shapes layer by layer and squish it to the required shape at the end?</p>
<p>Reference - <a href="https://www.kaggle.com/code/anirudhg15/seti-et-2023-w-pytorch" class="inline-onebox" rel="noopener nofollow ugc">SETI ET 2023 w/ PyTorch | Kaggle</a></p>
Is this an effective way to compute gradient norm after each epoch?

username: pytorchuser101
cooked: <p>Is this code an effective way to compute the global L2 gradient norm of the model after each training epoch : -</p>
<pre><code class="lang-auto">current_gradient_norm = nn.utils.clip_grad_norm_(model.parameters(), max_norm=float('inf'), norm_type=2.0)
</code></pre>
<p>I do not want to clip the gradients, I only want to find and store the entire model’s gradient norms after each epoch</p>
Keep track of variable while function optimization using autograd

username: jocelynn
cooked: <p>Optimizing this function using torch.autograd</p>
<pre><code class="lang-auto">def f(z):
    return (z*z).sum()
</code></pre>
<p>using the following code starting at some initial <code>z</code>.</p>
<pre><code class="lang-auto">z = torch.empty(3)
torch.nn.init.uniform_(z,-5,5)
z.requires_grad = True
optimizer = torch.optim.Adam([z], lr=0.1)
losses= []
zs =[] #to keep track of the variable as it gets optimized
print(z) #initial value of z
for i in range(50):
    optimizer.zero_grad()
    loss = f(z)
    loss.backward(retain_graph = True)
    optimizer.step()
    losses.append(loss.detach().numpy())
    zs.append(z.detach().data.numpy())
print(z)   #final value of z
</code></pre>
<p>which gives output:</p>
<pre><code class="lang-auto">tensor([-1.4410,  4.9476,  3.3615], requires_grad=True) 
tensor([-0.0837,  0.8619, -0.0701], requires_grad=True)
</code></pre>
<p>but the list <code>zs</code> has the value</p>
<pre><code class="lang-auto">[array([-0.08373591,  0.8618752 , -0.07005788], dtype=float32),
 array([-0.08373591,  0.8618752 , -0.07005788], dtype=float32),
...
 array([-0.08373591,  0.8618752 , -0.07005788], dtype=float32)]
</code></pre>
<p>which is the final value repeated. Why does this happen and how can I append values of z for each epoch into a list.</p>
<p>I’m new to PyTorch and learning the basics.</p>

username: eqy
cooked: <p>Note that calling <code>.numpy()</code> doesn’t actually create a copy as the data is shared if the tensor is on CPU (see <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.numpy.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.numpy — PyTorch 2.0 documentation</a>). You probably want something like <code>.clone()</code> instead:</p>
<pre><code class="lang-auto">import torch

def f(z):
    return (z*z).sum()

z = torch.empty(3)
torch.nn.init.uniform_(z,-5,5)
z.requires_grad = True
optimizer = torch.optim.Adam([z], lr=0.1)
losses= []
zs =[] #to keep track of the variable as it gets optimized
print(z) #initial value of z
for i in range(5):
    optimizer.zero_grad()
    loss = f(z)
    loss.backward(retain_graph = True)
    optimizer.step()
    losses.append(loss.detach().clone())
    zs.append(z.detach().data.clone())
print(z)   #final value of z
print(zs)
</code></pre>
<pre><code class="lang-auto">tensor([-1.8938, -3.1788, -4.9571], requires_grad=True)
tensor([-1.3970, -2.6804, -4.4581], requires_grad=True)
[tensor([-1.7938, -3.0788, -4.8571]), tensor([-1.6940, -2.9789, -4.7572]), tensor([-1.5945, -2.8791, -4.6573]), tensor([-1.4954, -2.7796, -4.5576]), tensor([-1.3970, -2.6804, -4.4581])]
</code></pre>

username: jocelynn
cooked: <aside class="quote no-group" data-username="jocelynn" data-post="1" data-topic="181568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/5e9695/48.png" class="avatar"> jocelynn:</div>
<blockquote>
<p>Optimizing this function using torch.autograd</p>
</blockquote>
</aside>
<p>thanks eqy, it is the solution I was looking for.</p>
TypeError: step() missing 1 required positional argument: 'metrics'

username: sbht_sbht
cooked: <p>I’m trying to add scheduler ReduceLRonPlateau, following instructions from documentation, but I get error “TypeError: step() missing 1 required positional argument: ‘metrics’” when I run it<br>
Can you please tell me what should I pass to the optimizer?</p>
<pre><code class="lang-auto">learning_rate = 0.0001
num_epochs = 300
n_total_steps = len(train_loader)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=15, threshold=0.00001)

for epoch in range(num_epochs):
    for i, (images, shfl_labels, rot_labels) in enumerate(train_loader):
        # Origin shape: image [batch size, 3, 224, 224]
        #               shfl_labels [batch size, 224, 224]
        #               rot_labels [batch size, 4, 4]


        # Load data in GPU
        images = images.to(device)
        shfl_labels = shfl_labels.to(device)
        rot_labels = rot_labels.to(device)

        # Forward pass
        shfl_output, rot_output = model(images)
        loss_shfl = criterion(shfl_output, shfl_labels.long())
        loss_rot = criterion(rot_output, rot_labels.long())
        loss = loss_shfl + loss_rot

        #Backward pass and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        scheduler.step()
</code></pre>

username: ptrblck
cooked: <p>The <code>step()</code> function of <code>ReduceLROnPlateau</code> expects a metric value as described in the <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#torch.optim.lr_scheduler.ReduceLROnPlateau">docs</a>, which is used to determine if the learning rate should be decreased or not.</p>

username: sbht_sbht
cooked: <p>Oh you are right. Sorry for my inattention. Thanks for your help</p>

username: NikhilMank
cooked: <p>I am also getting the same error, I saw the documentation but couldn’t identify the issue here. Please help me know what exactly should I look for in the documentation. should I mention a loss value in the step()?</p>

username: ptrblck
cooked: <p>Yes, the <code>step()</code> method expects a metric value as explained in my previous post.</p>
Balancing GPU load with DataParallel

username: Surya_Narayanan
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1e122bfb514a763a8ebe694c2adfcbf09667315.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1e122bfb514a763a8ebe694c2adfcbf09667315" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1e122bfb514a763a8ebe694c2adfcbf09667315_2_601x500.jpeg" alt="image" data-base62-sha1="wedMtNh3Pf9Ey8KRmiSTI8K3xu5" width="601" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1e122bfb514a763a8ebe694c2adfcbf09667315_2_601x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1e122bfb514a763a8ebe694c2adfcbf09667315_2_901x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1e122bfb514a763a8ebe694c2adfcbf09667315.jpeg 2x" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">960×798 174 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It seems that wrapping my classes in <code>DataParallel(model)</code>, I see there’s an imbalance between the allocated memory on each GPU. Is there a way to balance them more? It seems this imbalance is making it more likely to cause OOM errors</p>

username: ptrblck
cooked: <p>This is expected as it’s a known drawback of <code>nn.DataParallel</code> which is one of the reasons why we recommend using <code>DistributedDataParallel</code> instead.</p>
Curious why accuracy is causing OOM error

username: Surya_Narayanan
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba6b80dabfc6dd105b75386da133bb610d79afa7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ba6b80dabfc6dd105b75386da133bb610d79afa7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b80dabfc6dd105b75386da133bb610d79afa7_2_690x343.jpeg" alt="image" data-base62-sha1="qB98kpBBhLHY1qCvvxcpzJ0FdFJ" width="690" height="343" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b80dabfc6dd105b75386da133bb610d79afa7_2_690x343.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b80dabfc6dd105b75386da133bb610d79afa7_2_1035x514.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba6b80dabfc6dd105b75386da133bb610d79afa7_2_1380x686.jpeg 2x" data-dominant-color="1B1718"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×956 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Every time I run my script, the script trains fine, but during validation, runs OOM after the same number of batches. Why would this be happening in the middle of the loop?</p>

username: ptrblck
cooked: <p>Check if the memory usage increases during the training or validation and if so make sure to avoid storing tensors to e.g. a <code>list</code> which are still attached to a computation graph.</p>
<p>PS: it’s always better to post code snippets instead of screenshots.</p>
SSIM score using pytorch_msssim

username: shrbrh
cooked: <p>Hello!<br>
I am trying to calculate the SSIM score between two images using the following code:</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms.functional as F
from PIL import Image
from pytorch_msssim import ssim

content_img_path = "/home/input/Version2/Contents/img1.jpg"
stylized_img_path = "/home/input/Version2/Styles/img2.jpg"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 

content_img = Image.open(content_img_path).convert("RGB")
stylized_img = Image.open(stylized_img_path).convert("RGB")

stylized_img = stylized_img.resize(content_img.size, Image.BILINEAR)

content_tensor = F.to_tensor(content_img).unsqueeze(0).to(device)
stylized_tensor = F.to_tensor(stylized_img).unsqueeze(0).to(device)

ssim_value = ssim(content_tensor, stylized_tensor).item()
print("SSIM score:", ssim_value)
</code></pre>
<p>If I take the same image then I get a score of 1.0 which is expected. However, even for two completely different images, I get a very high value around 0.98-0.99.<br>
Can someone please let me know whether I am doing it right or if there is any issue with the code?</p>
Is DataLoader num_workers the number of parallel __getitem__ calls?

username: Olivier-CR
cooked: <p>Hi,</p>
<p>I have a situation with a vision training DataLoader taking a long time (T) and memory (M) to load at the beginning of each epoch. I notice that when we reduce num_workers by a factor of 4, T’=T/4 and M’=M/4.</p>
<p>I’d like to better understand how num_workers and getitem are connected: is num_workers the number of parallel process that individually each call getitem ?</p>
<p>What happens if num_workers &gt; batch_size: does the DataLoader only call getitem the right amount of times? or will the DL do num_workers getitems and discard the useless records? or keep it for next batch? What happens then if prefetching is set to 1?</p>

username: ptrblck
cooked: <p><code>num_workers</code> specifies the number of processes used to load and process the data.<br>
Each process will call into <code>Dataset.__getitem__</code> to create a full batch and depending on the <code>prefetch_factor</code> more samples will be preloaded and put into the queue.</p>
<p>When creating the <code>iterator</code>, the underlying <code>Dataset</code> is copied to each worker and the processes start to create the batches. Depending what <code>Dataset.__init__</code> contains, these copies might be expensive, but since lazy loading is usually used it shouldn’t be a huge slowdown.<br>
In any case, you could also use <code>persistent_workers=True</code> to avoid recreating the workers in each epoch.</p>

username: Olivier-CR
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I’m still not sure I understand - is it:</p>
<ol>
<li>A worker does a for loop calling <code>__getitem__</code> b times to create a batch of size b on its own?</li>
<li>or a worker calls <code>__getitem__</code> once, and the results of the N parallel <code>__getitem__</code> done by N workers are concatenated into batch of size b</li>
</ol>
<p>or something else?</p>

username: ptrblck
cooked: <p>Approach 1 is used currently. I know there was a feature request for 2 but I don’t know what the status of it is.</p>

username: tkella
cooked: <p>Let’s say I have a dataloader with num workers of 2 creating batches of size 1. I can see that <strong>getitem</strong> is called twice (once by each worker). If the output of the loader is viewed, I can only see 1 batch. Where does the other work put its batch?</p>

username: ptrblck
cooked: <p>Each worker collects its loaded batches in a queue and the <code>DataLoader</code> will return the next batch from it.</p>
Ensemble of pretrained models

username: Aishuvenkat09
cooked: <p>Hi,</p>
<p>I have 3 models trained on internal data. I want to ensemble them by calculating mean of their output.</p>
<pre><code class="lang-auto">build network():
  model1 = InceptionV3()
  model1.load_state_dict(torch.load('model1_weights.pth'))
  model1.eval()
  
  model2 = ResNet50()
  model2.load_state_dict(torch.load('model2_weights.pth'))
  model2.eval()
  
  model3 = AnotherModel()
  model3.load_state_dict(torch.load('model3_weights.pth'))
  model3.eval()
  
  for param in model1.parameters():    
      param.requires_grad = False
  for param in model2.parameters():
      param.requires_grad = False
  for param in model3.parameters():
      param.requires_grad = False
  
  model= Ensemble([model1, model2, model3])

return model

class Ensemble(nn.Module):

    def __init__(self, model_list):
        super().__init__()
        self.model_list = nn.ModuleList(model_list)
    
    def forward(self, x,y):
        return torch.stack([model(x,y) for model in self.model_list]).mean(0)


</code></pre>
<p>Now when i train the model using typical pytorch training process, I am getting error<br>
<code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn[1,0]</code></p>
<p>Am i using a right approach ?</p>

username: ptrblck
cooked: <p>The error you are seeing will be raised if you try to call <code>backward</code> on the output of any of the 3 models since you have frozen all trainable parameters by setting their <code>.requires_grad</code> attribute to <code>False</code>.<br>
If you want to compute the gradients and thus need to call <code>backward</code> on any output/loss, you would have to make sure some parameters are still trainable.</p>

username: Aishuvenkat09
cooked: <p>I’ve set for model3 require_grad = True and model3.train() instead of eval(). Just for my understanding, when training ensembles of pretrained models, do they have to be set in the evaluation mode ?</p>

username: Surya_Narayanan
cooked: <p>when you train models, they should be in train mode (even if they are in an ensemble)</p>
Issue with using ramdisk and map_location w/ torch.load()

username: mackamann
cooked: <p>I’m using a colab notebook to generate samples from OpenAI’s Jukebox model.  The notebook has some code to reduce the amount of system RAM when loading the model.  Using the default model provided by OpenAI, it generates audio just fine.  I have a finetuned model, which when using the same notebook/code, generates “garbage” audio.  However, if I remove this code it works fine.</p>
<p>So, given the same notebook and code, modulo the function below it will either work correctly or not.<br>
It works if loaded via:</p>
<p><code>checkpoint = t.load(restore, map_location=t.device('cpu'))</code></p>
<p>and doesn’t with this code:</p>
<p><code>checkpoint = t.load(restore)</code><br>
In the non-working the model is loaded via</p>
<p><code>checkpoint = t.load(restore, map_location=memory_map)</code></p>
<pre><code class="lang-auto">memory_map_idx = 0
def memory_map(storage, location):
    global memory_map_idx
    memory_map_idx += 1
    s = 'disk_tensors/' + str(memory_map_idx) + '.bint'
    f = open(s,"wb")
    f.seek(storage.size()*storage.element_size()-1)
    f.write(b"\0")
    f.close()
    new_storage = storage.__class__.from_file(s, 1, storage.size())
    del storage
    return new_storage
</code></pre>
<p>I’ve tried a bunch of different things, with the same results, I’m hoping someone might have some ideas on what I could try, or what might be the issue.</p>
<p>Things I’ve tried:</p>
<ul>
<li>Tons of different versions of pytorch</li>
<li>I’m not running in a parallel env, but I did try handling concurrency just in case, no difference.</li>
<li>Removed logic that deleted the on disk files after torch.load(), no difference.  I would have thought this was the issue, but no.</li>
</ul>
Neural Network Approximation and Derivative Estimation with additional data

username: froot_loop
cooked: <p>I am currently engaged in training a neural network to learn from a set of data points that contain information about the derivatives, represented as</p>
<blockquote>
<p>Train Data = {x_i, f’(x_i)} with i=100</p>
</blockquote>
<p>Additionally, I have supplementary data available in the form of</p>
<blockquote>
<p>{(f_k}, where k=100^2.</p>
</blockquote>
<p>For the latter set I do not have the corresponding $x_k$. My goal is now to find a approximate the function using a neural network.</p>
<p>My proposed approach involves two local loss function defined as follows:</p>
<blockquote>
<p>L_1 = Sum_i || N’(x_i) - f’(x_i)|| ^2<br>
L_2 = Sum_i (argmin_k ||N(x_i) - f_k||^2</p>
</blockquote>
<p>Is it then possible to define the loss function as</p>
<blockquote>
<p>L = L_1 + L_2</p>
</blockquote>
<p>and is it legit to follow this appropach? I am not sure since L_2 contains a closest point search for each epoch.</p>
How to enumerate over multiple time-series datasets?

username: Bastulli
cooked: <p>If I had a bunch of different image datasets I would just concatenate using “ConcatDataset” from torch.utils.data.  For multiple times-series datasets, I cant concatenate because they need a rolling window and cant just continuously index from one datasets to another.</p>
<p>Is it possible to come up with a solution using custom dataloader or should I just iterate through a different pandas dataframe after each training episode?</p>
<p>Here is my custom dataloader that works with a 1D CNN with one time-series datasets at a time.</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, window):
        self.data = data
        self.window = window
        print(data.tail())
        self.xData = torch.FloatTensor(data[['data_1','data_2','data_3']].values.astype('float'))
        self.yData = torch.FloatTensor(data['labels'].values.astype('int'))

    def __len__(self):
        return len(self.data) - self.window

    def __getitem__(self, index):
        target = self.yData[index+self.window]
        data_val = self.xData[index:index+self.window].reshape(in_channel, window_size)
        return data_val, target

# split data into train test set
df_train, df_test = train_test_split(df, test_size=0.3, shuffle=False)

# send our pandas dataframe with data to our custom data class
train_dataset = MyDataset(df_train, window_size)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = MyDataset(df_test, window_size)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>A method that currently works but I don’t like doing.</p>
<pre><code class="lang-auto"># dictionary full of multiple pandas dataframes
count += 1
str_txt = list(dictionary.data.keys())[count]
data = dictionary.data[str_txt]
if count &gt;= len(list(dictionary.data.keys()):
    count = 0
</code></pre>

username: semaj
cooked: <p>Did you ever find a solution to this?</p>

username: Mark_Pinches
cooked: <p>Ive had to just do something similar.<br>
I have a bunch of individual timeseries arrays in a dictionary called item_dic</p>
<p>class TimeseriesDataset_from_Dict(torch.utils.data.Dataset):</p>
<pre><code>def __init__(self, item_dic, n_steps = 10, seq_len=100):
    self.item_dic = item_dic
    self.dic_items = len(item_dic.keys())
    self.X_shape = item_dic[0].shape[0]
    self.seq_len = seq_len
    self.X_len = self.X_shape - (self.seq_len-1)
    self.n_steps = n_steps

def __len__(self):
    return int(self.X_len*self.dic_items/self.n_steps)

def __getitem__(self, index):
    dic_ind = (index*self.n_steps)//self.X_len
    X = self.item_dic[dic_ind]
    index = (index*self.n_steps)%self.X_len
    print (dic_ind, index)
    return X[index:index+self.seq_len]</code></pre>

username: dorien
cooked: <p>Interesting solution Mark_Pinches!</p>
<p>I am modifying it to include scaler and Y.</p>
<p>I am a bit confused what n_steps is here?</p>