akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to find the source code of fft2 in pytorch

XINGYU_ZHU: <h1>
<a name="background-1" class="anchor" href="#background-1"></a>Background</h1>
<p>Hi, I’ve been diving into the source code of <code>torch.fft.fft2</code>, but I got some problem.</p>
<p>What I have done:</p>
<ul>
<li>Step 1: I called the code <code>torch.fft.fft2(img)</code> and I found the source of it in <code>torch/csrc/autograd/generated/python_fft_functions.cpp</code>
</li>
<li>Step 2: In <code>torch/csrc/autograd/generated/python_fft_functions.cpp</code> the following code calls <code>fft_fft2</code>, which is located in <code>aten/src/ATen/native/SpectralOps.cpp</code><pre><code class="lang-auto">auto dispatch_fft_fft2 = [](const at::Tensor &amp; self, at::OptionalIntArrayRef s, at::IntArrayRef dim, c10::optional&lt;c10::string_view&gt; norm) -&gt; at::Tensor {
    pybind11::gil_scoped_release no_gil;
    std::cout &lt;&lt; "self.sizes()=" &lt;&lt; self.sizes() &lt;&lt; std::endl;
    return at::fft_fft2(self, s, dim, norm);
  };
</code></pre>
</li>
<li>Step 3: In <code>aten/src/ATen/native/SpectralOps.cpp</code>, the following code calls <code>_fft_c2c</code>, which is implemented in <code>torch/csrc/autograd/generated/VariableType_1.cpp</code><pre><code class="lang-auto">Tensor fft_c2c_maybe_out(
  c10::string_view fname, const Tensor&amp; out, const Tensor&amp; input,
  IntArrayRef dim, int64_t norm, bool forward) {
  if (out.defined()) {
    TORCH_CHECK(out.is_complex(), fname,
                " expects a complex output tensor, but got ", out.scalar_type());
    auto out_mut = out;
    return at::_fft_c2c_outf(input, dim, norm, forward, out_mut);
  }
  return at::_fft_c2c(input, dim, norm, forward);
}
</code></pre>
</li>
<li>Step 4: In <code>torch/csrc/autograd/generated/VariableType_1.cpp</code>, the following code calls <code>_fft_c2c_symint </code><pre><code class="lang-auto">auto _tmp = ([&amp;]() {
  at::AutoDispatchBelowADInplaceOrView guard;
  return at::redispatch::_fft_c2c_symint(ks &amp; c10::after_autograd_keyset, self_, dim, normalization, forward);
})();
</code></pre>
</li>
</ul>
<p>Then I failed to find the implement of this call <code>at::redispatch::_fft_c2c_symint</code>.</p>
<h1>
<a name="question-2" class="anchor" href="#question-2"></a>Question</h1>
<p>My first question is where is the core logic of <code>torch.fft.fft2</code>?<br>
If you cannot solve this question directly, then my second question is where is the redispatch function of <code>_fft_c2c_symint </code>?</p>
<p>Here’s my third question.<br>
It seems like there’s a <strong>singleton dispatcher registry</strong> in <code>aten/src/ATen/core/dispatch/Dispatcher.h</code>, where they register &amp; load the redispatch function through mapping:</p>
<pre><code class="lang-auto">const KernelFunction&amp; kernel = op.operatorDef_-&gt;op.lookup(currentDispatchKeySet);
</code></pre>
<p>And the acquired redispatch function is of class <code>KernelFunction</code>. How can I get the name of that redispatch function given that <code>KernelFunction</code>?</p>
Post-Training Quantization using test data?

afaso: <p>I am setting up a PTQ (static) workload and trying to understand what is the best practice to calibrate the quantizers.</p>
<p>I am following the new FX Graph mode tutorial <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" rel="noopener nofollow ugc">here</a>. The Calibration section mentions that in order to initialize the quantizers, some samples “representative of the workload” are used. The text says: “for example a sample of the training data set”. On the other hand, the code snippet loads data from the validation set:</p>
<blockquote>
<p>calibrate(prepared_model, data_loader_test)</p>
</blockquote>
<p>I would have thought using validation/test data for calibration to be a no-no. However, I found this practice of using validation data for calibration reported elsewhere too, in PyTorch documentation, such as <a href="https://pytorch.org/blog/quantization-in-practice/#calibration" rel="noopener nofollow ugc">here</a></p>
<blockquote>
<p>“”"Calibrate</p>
<ul>
<li>This example uses random data for convenience. Use representative (validation) data instead.<br>
“”"</li>
</ul>
</blockquote>
<p>What is the rationale for setting up the quantizers using information from the validation set? Even under the assumption that only a tiny fraction (few examples) of the validation set is used as calibration, wouldn’t this contaminate the evaluation? What about the case where a larger calibration sets is to be used (some PTQ papers report using 1000s of examples)?</p>

Vasiliy_Kuznetsov: <p>Hi <a class="mention" href="/u/afaso">@afaso</a> , you are right, and we need to update the tutorials to reflect this.  If you want a true eval score of a quantized model, you’d have to use a slice of a training dataset (or some other held-out data) for PTQ calibration.</p>
How to create a dataloader for sequence of images as an input to CNN+LSTM?

tejasri19: <p>Hello all,</p>
<p>I have to implement CNN+LSTM for crop classification on image sequence data. I need to generate a dataloader to return list of sequence of images. Lets say, if my sequence length is 10 and batch size is 8, it should return 8 lists of 10 images.</p>
<p>Need help!</p>
<p>Many thanks in advance!</p>

ummadiviany: <p>Hi <a class="mention" href="/u/tejasri19">@tejasri19</a>, I hope you have already figured out the solution, but here is my solution.</p>
<p>I am assuming the following</p>
<ol>
<li>You have the order images</li>
<li>You are trying to learn a spatio-temporal model for classification using ConvLSTM</li>
</ol>
<p>What’s the format of the labels? Do you need labels for all images in a batch?<br>
If you have 10 images in a batch and do you need label for each corresponding image?</p>
<p>I have worked on a similar problem, and I am attaching my code below. The below code produces the labels for all images in a batch, but you can easily modify it as per requirements.</p>
<pre><code class="lang-python">import torch, torch, torchvision
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset
from glob import glob
torch.manual_seed(2000)
import torchvision.transforms.functional as TF
import pandas as pd


class SeqImageDataset(Dataset):
    def __init__(self, dataset_path, label_path, img_transforms=None, sequence_length=3):
        
        self.img_transforms = img_transforms
        self.sequence_length = sequence_length
        self.labels = pd.read_csv(label_path)
        self.files = sorted(glob(f'{dataset_path}/*.png'))
        print(f'Loaded {len(self.files)} images from {dataset_path} dataset')
        
    def __len__(self):
        return len(self.files) - self.sequence_length
    

    def __getitem__(self, idx):
        images = [
            torchvision.io.read_image(self.files[idx+i], mode=torchvision.io.ImageReadMode.GRAY) for i in range(self.sequence_length)
        ]
        images = torch.stack(images, dim=0)
        labels = self.labels.iloc[idx : idx + self.sequence_length]            

        if self.img_transforms:
            images = self.img_transforms(images)
            
        return images, labels
    


trainset = SeqImageDataset('data/train', 'data/train.csv')
train_loader = DataLoader(trainset, batch_size=8, shuffle=True)
    
    
</code></pre>
How to Apply a Hamming/Hanning Window?

BanBot2: <p>Hello everyone,</p>
<p>I’m currently working on a deep neural network that tries to locate musical onsets in audio samples. It returns a 1D tensor containing the probability of an onset at each timestamp, from 0 to 1. It’s working pretty well, but sometimes it returns a double onset where there should only be one.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0758e868edf7dace26923bdce885041180696996.jpeg" alt="Capture" data-base62-sha1="12ZPf0FbcAj87vn7fsQl4HVOuFg" width="365" height="167"><br>
(The double peak on the left should only be a single peak)</p>
<p>While increased training remedies the problem somewhat, it seems to be a natural consequence of the model I’m using. To fix this, I believe that I need to apply a hamming/hanning window across the outputs to smooth out these double peaks, like what was done in this paper: <a href="https://arxiv.org/pdf/1703.06891.pdf" rel="noopener nofollow ugc">Dance Dance Convolution (arxiv.org)</a>.</p>
<p>How would I do this? I have read the PyTorch docs on the hamming window function (<a href="https://pytorch.org/docs/stable/generated/torch.hamming_window.html" rel="noopener nofollow ugc">torch.hamming_window — PyTorch 1.13 documentation</a>), but for some reason the explanation there isn’t very enlightening for me. Any help is much appreciated.</p>
<p>Thank you for your time,</p>
<p>-BanBot2</p>

ptrblck: <p>I haven’t read the details of this paper, but something like this might work:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.zeros(1024)
x[512] = 1.
x[525] = 1.
# add noise
x = x + torch.randn(1024) * 0.01
plt.plot(x)

filt = torch.hamming_window(32)
out = F.conv1d(x[None, :], filt[None, None, :], padding=filt.size(0)//2) / filt.sum()
plt.plot(out.squeeze(0) / out.max())
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7739daea97a7a5541d3fd12fc0407eee21d7d113.png" alt="image" data-base62-sha1="h0ILVClkGF2lF1wusZQPzOcCUM3" width="372" height="248"></p>

BanBot2: <p>That looks like just the ticket! Thanks for the help. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Conv1d for two matrix column by column

FrankiePoon: <p>I have two matrices; I need to do convolution between each column of matrix A and the correspondent column of matrix B and store the results in matrix y. This can be done using a for loop:</p>
<p>for i in range(5):<br>
y[:,i]= F.conv1d(A[:, i], B[:, i])</p>
<p>I am wondering if there is any more efficient way to implement this process? For example, avoid using a for loop.</p>

ptrblck: <p>I don’t know if I understand your use case correctly since you did not post an executable code snippet, but assuming you want to use a “column” of <code>B</code> as the conv kernel, you could avoid the for loop by using a grouped convolution with a permutation in the kernel dimensions as seen here:</p>
<pre><code class="lang-python">A = torch.randn(1, 10, 7)
B = torch.randn(1, 10, 5)
y = torch.zeros(1, 10, 3)

for i in range(A.size(1)):
    y[:, i]= F.conv1d(A[:, i:i+1], B[:, i:i+1])
    
out = F.conv1d(A, B.permute(1, 0, 2), groups=10)
print((out - y).abs().max())
# tensor(3.5763e-07)
</code></pre>

FrankiePoon: <p>Thanks, that is really helpful.</p>
.to(device) inside Model's class

0x97: <p>I am instantiating a model, and I would like to pass the device as an arg from a configuration dataclass, and then inside the model class perform:</p>
<pre><code class="lang-auto">    self.to(config.device)
</code></pre>
<p>but after a check, the model is not on ‘cuda’.<br>
Is there any way of doing this?</p>

ptrblck: <p>It works for me:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, device):
        super().__init__()
        self.fc1 = nn.Linear(1, 1)
        self.fc2 = nn.Linear(1, 2)
        self.to(device)
        
model = MyModel("cuda")
print(model.fc1.weight.device)
# cuda:0
print(model.fc2.weight.device)
# cuda:0
</code></pre>
<p>although I’m unsure why you want to call it inside the model’s <code>__init__</code> instead of after creating the object.</p>
Obtain intermediate outputs from pre-trained models

YosemiteSam: <p>Given that I have a pre-trained pytorch model, what is the “right” way to  obtain <em>several</em> intermediate results of the network?</p>
<p>For example, suppose I obtain an ResNet-50 pre-trained from imagenet.  Now, I want to look at the outputs for each named layer of Table 1 of the original paper (<a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1512.03385.pdf</a>).  What is the “right” way to the intermediate results of the 5 blocks for a fixed dataset?</p>

ptrblck: <p>You could use forward hooks as described in <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">this post</a> and register the desired layers using their <code>name</code>.</p>
Grad_fn for a Classifier's Output

Ibrahim_Shaer: <p>I am trying to obtain the results of a classifier written in sklearn and not available on torch. The outputs of this classifier will be used to create a loss function that alters the weights of a pytorch nn.Module. Since the outputs are not obtained using a pytorch operation, the weights of the pytorch module are not updated. Adding (requires_grad) to the resultant lossand removing detach() function did not change the output. A snippet of this function is as follows. Any help would be greatly appreciated.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1a6fcb3ffc46b810a6141031dbc545eacb47dcb.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1a6fcb3ffc46b810a6141031dbc545eacb47dcb" title="pytroch_forums"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1a6fcb3ffc46b810a6141031dbc545eacb47dcb.png" alt="pytroch_forums" data-base62-sha1="rD7V2GMajhYYpEOBYzONShNduZ5" width="690" height="298" data-dominant-color="242626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytroch_forums</span><span class="informations">1055×456 26.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Since you are using numpy for the loss calculation, you are detaching this tensor from the computation graph and Autograd won’t be able to calculate the gradients for the used parameters in your model.<br>
You could either try to reimplement the numpy method in PyTorch or you could write a custom <code>autograd.Function</code> defining the <code>forward</code> and <code>backward</code> methods as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> using the numpy operations.</p>
Seeking frames in video vs separate frames speed

filby: <p>Hey all,<br>
I am working with some pretty large video datasets and till now I was extracting the different frames and loading the sequences of the video by loading the individual frames. Because this is a bit inefficient in terms of disk space, I would like to know how this approach fares against using videoreader and seeking the separate frames (or use opencv seek), regarding the loading speed.</p>

ptrblck: <p>I would assume the actual performance depends on the libraries you are using, your system, and use case. E.g. you could try to use <a href="https://docs.nvidia.com/deeplearning/dali/user-guide/docs/examples/sequence_processing/video/video_reader_label_example.html">DALI’s video reader</a> which would allow you to load and process the frames on the GPU, or e.g. the <a href="https://pytorch.org/vision/stable/generated/torchvision.io.VideoReader.html"><code>VideoReader</code></a> class from `torchvision, which would also allow GPU decoding of the frames.</p>
Best practice for complex/float operations

Ronan_Gautier: <p>If I want to operate on both float and complex-valued tensors, what is the best practice? Should everything be in complex format throughout computation? Or is it fine and efficient to multiply float and complex tensors together? Assuming that they are of the same precision.</p>
<p>A simple exemple is for solving an ODE <code>y'(t) = f(t, y, θ)</code>, I may have a forward function that looks like</p>
<pre><code class="lang-auto">def forward(self, t, y):
    return torch.exp(1j * self.theta * t) * y
</code></pre>
<p>where <code>self.theta</code> is a real-valued 1-D tensor of parameters, <code>t</code> is the real-valued time, and <code>y</code> is a complex-valued 1-D tensor. Should I store <code>self.theta</code> and <code>t</code> as float or complex objects?</p>
Take what of many medians?

Caipi: <p>Hello, a more formal question, sorry.</p>
<p>I have a CNN and receive as output an image of the shape [1, 512, 512] and supervise it with a ground truth of the same shape. Furthermore, I calculate MAE, MSE, SSIM, … and also the MedAE (Median-Absolute-Error). Formula (N pixels in each tile):</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/098d7a245460b8851adb8c8df555331e157b9d0d.png" alt="Capture" data-base62-sha1="1mvpsbGQc3ZYEPwYrZXSj2DmPkF" width="311" height="44"></p>
<p>The question is for the latter index: When I then have an array with all my median errors, do I take the median from this array for result presentation or the mean of all median errors?</p>
How is np.gradient computed in numpy

ays: <p>I’m going through the <a href="https://numpy.org/doc/stable/reference/generated/numpy.gradient.html" rel="noopener nofollow ugc">np.gradient</a> documentation, one thing I dont understand is how the gradient computed, when we dont pass in a function like (f = x^2). Isn’t the computation supposed to be like this:<br>
f = x^2 (some function which we pass into np.gradient)<br>
df/dx = 2x (this will computed by np.gradient)<br>
Then, if x = 2; we have 2*2 = 4</p>
<p>Please kindly explain so I can I have better understanding.</p>
<pre><code class="lang-auto">f = np.array([1, 2, 4, 7, 11, 16], dtype=float)
&gt;&gt;&gt; np.gradient(f)
array([1. , 1.5, 2.5, 3.5, 4.5, 5. ])
&gt;&gt;&gt; np.gradient(f, 2)
array([0.5 ,  0.75,  1.25,  1.75,  2.25,  2.5 ])
</code></pre>
Declare torch.compile compatible inplace operator

nlgranger: <p>I developed an C++ operator which replaces index_add with a few modifications. Its first argument is modified in-place. It has the following signature:</p>
<pre><code class="lang-auto">TORCH_LIBRARY(ops, m) {
  m.def("index_add(Tensor tensor1, int dim, Tensor indices, Tensor values) -&gt; Tensor");
}
</code></pre>
<p>Where <code>tensor1</code> is modified in-place and returned for convenience.</p>
<p>When using this operator, torch.compile complains that the in-place modified argument is not returned as an output. In reality it is, but it can’t know that from the signature.</p>
<pre><code class="lang-auto">RuntimeError: Some elements marked as dirty during the forward method were not returned as output. The inputs that are modified inplace must all be outputs of the Function.
</code></pre>
<p>How do I declare that the modified tensor is indeed an output as well?</p>
<p>I tried to blindly use something I saw in the pytorch source on the op signature declaration:</p>
<pre><code class="lang-auto">m.def("index_add(Tensor tensor1(a!), int dim, Tensor indices, Tensor values) -&gt; Tensor(a!)");
</code></pre>
<p>but it doesn’t seem to fix the problem.</p>
ElasticTransform is gone

Alexis: <p>Hi! I can’t find old torchvision.transforms.ElasticTransform. Where is it? How to get elastic augmentation?</p>
<pre><code class="lang-auto">elastic_transformer = T.ElasticTransform(alpha=250.0)
transformed_imgs = [elastic_transformer(orig_img) for _ in range(2)]
plot(transformed_imgs)
</code></pre>
<p><a href="https://pytorch.org/vision/stable/auto_examples/plot_transforms.html?highlight=transform" rel="noopener nofollow ugc">link</a></p>

ptrblck: <p><code>transformsElasticTransform</code> was added in <a href="https://github.com/pytorch/vision/commit/9430be7612f6f2c1bd363cf1c18b5e6d81b95fe0">this PR</a> in June 2022 and is still available in the current release:</p>
<pre><code class="lang-python">transforms.ElasticTransform
# torchvision.transforms.transforms.ElasticTransform
</code></pre>

Alexis: <p>Thank you very much!</p>
Which function a generator should maximize during GAN training?

shrbrh: <p>I have a GAN loss function for image style transfer. Let ‘x’ here be the content image that needs to be stylized and ‘y’ be the style image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c26d44a575cee8cf35a6f4618e04f9f5dcaefcf9.png" alt="GANLossFunction" data-base62-sha1="rJYJA6jN2XGCE7WYn1E8QIOS5AZ" width="364" height="72"></p>
<p>Now, the discriminator D() is trying to minimize this function while at the same time the generator G() is trying to maximize it. So while implementing the code for this loss function, will the discriminator function take both the terms into consideration and try to minimize the sum? And the generator function will be concerned with only the second part of the equation that involves the generated image and try to maximize it?<br>
Did I get it right?</p>
How to make a copy of a gpu model on the cpu?

sicroci: <p>Hi All,</p>
<p>I am training a model on the gpu and after each epoch I would like to store the best models on the cpu. Does model.to(‘cpu’) create a copy on the cpu and does it keep the original model on the gpu?</p>
<p>At the moment, I have this code:<br>
best_model = copy.deepcopy(model)<br>
best_model = best_model.to(‘cpu’)<br>
The problem is that this code makes a copy first on the gpu, and then it transfers the copy to the cpu. I would like to avoid making a copy on the gpu, since I don’t have enough gpu memory.</p>
<p>Simone</p>

amalshaji: <p>I think the best way is to save the model using <strong>torch.save(‘modelname.pt’)</strong>. You can now load the saved model for inference. To run inference on CPU, specify <strong>map_location=‘cpu’</strong> on torch.load()</p>

sicroci: <p>Thank you for the reply. The solution that you propose consists of saving the best model to the disk. Do you know an alternative solution that doesn’t save the model to the disk. I just need to make a copy on the cpu of a model on the gpu, without touching the model on the gpu.</p>

Kajaos: <p>Hello,</p>
<p>I am a bit late but I found a in-memory solution to the problem and thought to share it with others. The method does not seem very fast, but I don’t know any other at the moment.</p>
<pre><code class="lang-auto">from io import BytesIO

# Save the model to memory from GPU
model_data_in_memory = BytesIO()
torch.save(original_model, model_data_in_memory)
model_data_in_memory.seek(0)

# Load the model from memory to CPU
model_in_cpu = torch.load(model_data_in_memory, map_location="cpu")
model_data_in_memory.close()
</code></pre>
<p>The seek is needed to seek the in-memory file to the beginning. After that just load the model to cpu assuming that the original model was on the GPU. Be sure to also close the file to free the memory so that the buffer doesn’t leave open.</p>
<p>Edit: I found that if you add to “pickle_protocol=-1” to the save command it speeds up about 10% (in pytorch 1.13).</p>
<pre><code class="lang-auto">torch.save(original_model, model_data_in_memory, pickle_protocol=-1)
</code></pre>
DistributedSampler

ginobilinie: <p>How does the DistributedSampler (together with ddp) split the dataset to different gpus? I know it will split the dataset to num_gpus chunks and each chunk will go to one of the gpus. Is it randomly sampled or sequentially?</p>

seungjun: <p>First, it checks if the dataset size is divisible by <code>num_replicas</code>. If not, extra samples are added.</p>
<p>If <code>shuffle</code> is turned on, it performs random permutation before subsampling.<br>
You should use <code>set_epoch</code> function to modify the random seed for that.</p>
<p>Then the DistributedSampler simply subsamples the data among the whole dataset.<br>
<a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/distributed.py#L68" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/utils/data/distributed.py#L68</a></p>
<pre><code class="lang-auto"># subsample
indices = indices[self.rank:self.total_size:self.num_replicas]
</code></pre>
<p>Note that adding extra data could cause at evaluation time due to the duplicated data.<br>
I personally use a custom sampler (<a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/data/sampler.py" rel="nofollow noopener">DistributedEvalSampler</a>) when testing my models.</p>

ginobilinie: <p>Thank you so much for your answer. I’m now clear about it.</p>

lyt201: <p>I understand that the distributed sampler chunks the dataset for each GPU. However, when using DDP, it loads the entire Dataset on N GPUs N times. Is this how it works?</p>

seungjun: <p>It depends on how you write your dataset.<br>
In most cases, you have the list of file paths and <code>__getitem__</code> function loads the data to memory.<br>
But yes, each process has the full list.<br>
If you only want to load the partial list for each process, you should write your custom dataset, sampler, etc.</p>
Problem with torch.distributed.launch

b.merainani: <p>Good morning eveyone<br>
I am trying to use torch.distributed.launch to train a neural network<br>
python -m torch.distributed.launch --nproc_per_node=2<br>
–master_addr=“127.0.0.1” --master_port=427 train.py …</p>
<p>But i got the followin error. to select an open port, i have used “netstat -a”</p>
<p>[W C:\cb\pytorch_1000000000000\work\torch\csrc\distributed\c10d\socket.cpp:401] [c10d] The server socket has failed to bind to [<a href="http://FNAN-20-107.ad.ifsttar.fr" rel="noopener nofollow ugc">FNAN-20-107.ad.ifsttar.fr</a>]:427 (system error: 10048 - Une seule utilisation de chaque adresse de socket (protocole/adresse rÚseau/port) est habituellement autorisÚe.).<br>
[W C:\cb\pytorch_1000000000000\work\torch\csrc\distributed\c10d\socket.cpp:401] [c10d] The server socket has failed to bind to <a href="http://FNAN-20-107.ad.ifsttar.fr:427" rel="noopener nofollow ugc">FNAN-20-107.ad.ifsttar.fr:427</a> (system error: 10048 - Une seule utilisation de chaque adresse de socket (protocole/adresse rÚseau/port) est habituellement autorisÚe.).</p>
<p>the translation of <strong>: Une seule utilisation de chaque adresse de socket (protocole/adresse rÚseau/port) est habituellement autorisÚe.).</strong> is  <strong>only one usage of each socket address (protocol/network address/port) is normally allowed</strong></p>
<p>Thank you in advance for your help !</p>
Audio data, True/False Large Data Set; Best Methods?

paw: <p><img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=12" title=":wave:" class="emoji only-emoji" alt=":wave:" loading="lazy" width="20" height="20"></p>
<p>hello all.<br>
I need to construct a ML nn to find a boolian answer to large amounts of data.</p>
<p>each 20 hour block only has 1 answer True or False</p>
<p>there will be access to over 6,480 blocks of 20 hours each with labels of True or Flase (currently working on prepping that data now)</p>
<p>will running it similar to the examples I found of the yes, no problem show me all the relationships possible between all the True’s and all the False’s data or will I need to add more plug-ins/functions to find the relationships?</p>
<p>here is the type of example I’m starting with below, but please let me know if a different idea/method is better to build on.</p>
<p>thanks so much all, look forward to going on this journey all together.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji only-emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f_2_690x351.jpeg" alt="image" data-base62-sha1="wG4IduW6Y2cbaKhFSLktHrbQYd9" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f_2_690x351.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f_2_1035x526.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e5072ccf8b1bc365dfc5fbbbad77dbee644dd60f_2_1380x702.jpeg 2x" data-dominant-color="F2F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×979 88.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

shivammehta007: <p>This looks like a good starting point but, maybe you would want to play around with some sort of downsampled representation of the audio signal. As much as I understand you have a 20-hour audio and a binary label for that audio. It looks like a very computation-heavy task you can try some other pretrained representations of the audio, like wav2vec2 vectors or even whisper’s embeddings ( depending on your problem) and do some sort of pooling on them or something to downsample the large audio waveform to something shorter that you can compute and train with.</p>
Issue with AutoAugment when using half precision float (torch.float16)

Phos_phophylitte: <p>Hi,<br>
I’ve run into an issue when trying to use the <code>torchvision.transforms.autoaugment.AutoAugment()</code> module while having set the default d_type to float16 :<br>
The minimum code to reproduce my error is the following :</p>
<pre><code class="lang-python">import torch
import torchvision.transforms as transforms

torch.set_default_dtype(torch.float16)

image = torch.randn(10,3,32,32).cuda()

transform = transforms.autoaugment.AutoAugment()
transform = transform.cuda()

result = transform(image)
</code></pre>
<p>Giving the error :<br>
<code>"round_vml_cpu" not implemented for 'Half'</code></p>
<p>My understanding is that it’s a bug, AutoAugment() does not cast the tensor created in the <code>_augmentation_space</code> function to it’s device.</p>
<p>I’ve currently fixed it by replacing this function with one creating the Tensors on the GPU.</p>
<p>I’m posting this here to check if it is a bug / not yet implemented or if I’ve misunderstood how to use the module.</p>
<p>Thanks in advance.</p>

ptrblck: <p>The failing function is defined in <a href="https://github.com/pytorch/vision/blob/135a0f9ea9841b6324b4fe8974e2543cbb95709a/torchvision/transforms/autoaugment.py#L235">this line of code</a> and does not use the input tensor, but creates a new one based in the passed <code>num_bits</code> argument:</p>
<pre><code class="lang-python">(8 - (torch.arange(num_bins) / ((num_bins - 1) / 4)).round().int(), False)
</code></pre>
<p>Since you have set the default type to <code>float16</code> this newly created tensor will be created in the <code>float16</code> <code>dtype</code> and the <code>round</code> operation will fail.</p>
<p>In my opinion using <code>torch.set_default_dtype</code> is dangerous as it has many side effects such as this issue.</p>
Change worker seed during runtime

mzimmerman: <p>In <code>DataLoader</code>, we could specify the <code>worker_init_fn</code> argument to change the seed accordingly. This however requires the worker to call the init function, effectively only during initialization of the worker. Is there any way I could change the seed of worker during runtime? For example, changes at every n minibatches?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Yes, you can set the seed via <code>torch.manual_seed</code> inside the <code>Dataset.__getitem__</code> and could also use the worker info as seen here:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, idx):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is not None:
            print(worker_info)
            worker_id = worker_info.id
            torch.manual_seed(worker_id)
        return torch.randn(1)

    def __len__(self):
        return(20)

    
dataset = MyDataset()
dataloader = DataLoader(dataset,
                        batch_size=5,
                        shuffle=False, 
                        num_workers=8)

for data in dataloader:
    print(data)
</code></pre>

mzimmerman: <p>Huh, never cross my mind hijacking the <code>__getitem__</code>. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, the Twitter famous celebrity <img src="https://discuss.pytorch.org/images/emoji/apple/crazy_face.png?v=12" title=":crazy_face:" class="emoji" alt=":crazy_face:" loading="lazy" width="20" height="20"></p>
Error with transform in pytorch version 1.8

s_n: <p>Hi,</p>
<p>I am trying to use CenterCrop transform in pytorch version 1.8.0</p>
<p>But I am getting the below error. From the error it seems that the line that is causing the error is in transforms/functional.py file “w,h=img.size”. I found that changing this to   “w,h=img.size()” solves the error. Is this really the error or its possible I am doing something wrong. Also, I am not sure if it is recommended to change any of the pytorch files.</p>
<p>TypeError: Caught TypeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py”, line 202, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “dataloader.py”, line 261, in <strong>getitem</strong><br>
transformed_event_repr[bin_idx, pol_idx] = self.transform(event_repr[bin_idx, pol_idx])<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py”, line 60, in <strong>call</strong><br>
img = t(img)<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py”, line 235, in <strong>call</strong><br>
return F.center_crop(img, self.size)<br>
File “/home/jrstevens/.local/lib/python3.6/site-packages/torchvision/transforms/functional.py”, line 365, in center_crop<br>
w, h = img.size<br>
TypeError: ‘builtin_function_or_method’ object is not iterable</p>
<p>######### Center crop from pytorch version 1.8.0<br>
def center_crop(img, output_size):<br>
if isinstance(output_size, numbers.Number):<br>
output_size = (int(output_size), int(output_size))<br>
w, h = img.size<br>
th, tw = output_size<br>
i = int(round((h - th) / 2.))<br>
j = int(round((w - tw) / 2.))<br>
return crop(img, i, j, th, tw)</p>
<p>Thanks</p>

eqy: <p>This might be a valid bug but it seems like it may have already been fixed in recent versions of <code>torchvision</code> (e.g., <a href="https://github.com/pytorch/vision/commit/095437aad6cfd25b01fa01961e5774904d738347" class="inline-onebox" rel="noopener nofollow ugc">Replace get_image_size/num_channels with get_dimensions (#5487) · pytorch/vision@095437a · GitHub</a>). Could you check if upgrading to a newer version (&gt;= <code>0.14.1</code>) of <code>torchvision</code> resolves the issue?</p>
Loss.backward() error in MultiGPU Training on a single node

Saleh: <p>When I am trying to train my model on 4 GPUs I face the bellow error.<br>
"CUDA_VISIBLE_DEVICES=0,1,2,3  python my_script.py "  and the error is related to “loss.backward()”</p>
<p>Here is the error:<br>
"/opt/conda/envs/Rohit/lib/python3.7/site-packages/torch/nn/modules/module.py:1082: UserWarning: Using non-full backward hooks on a Module that does not return a single Tensor or a tuple of Tensors is deprecated and will be removed in future versions. This hook will be missing some of the grad_output. Please use register_full_backward_hook to get the documented behavior.<br>
warnings.warn("Using non-full backward hooks on a Module that does not return a "<br>
Error executing job with overrides: [‘model=default’]<br>
Traceback (most recent call last):<br>
File “experiments/mobileRobot/mobile_robot_hiprssm.py”, line 87, in my_app<br>
exp = Experiment(model_cfg)<br>
File “experiments/mobileRobot/mobile_robot_hiprssm.py”, line 93, in <strong>init</strong><br>
self._experiment()<br>
File “experiments/mobileRobot/mobile_robot_hiprssm.py”, line 193, in _experiment<br>
transformer_learn.train(train_obs, train_act, train_targets, train_task_idx, cfg.learn.epochs, cfg.learn.batch_size, test_obs, test_act,test_targets, test_task_idx)<br>
File “./learning/transformer_trainer.py”, line 583, in train<br>
batch_size)<br>
File “./learning/transformer_trainer.py”, line 244, in train_step<br>
loss.backward()<br>
File “/opt/conda/envs/Rohit/lib/python3.7/site-packages/torch/_tensor.py”, line 489, in backward<br>
self, gradient, retain_graph, create_graph, inputs=inputs<br>
File “/opt/conda/envs/Rohit/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 199, in backward<br>
allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass<br>
File “/opt/conda/envs/Rohit/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 61, in <strong>call</strong><br>
raise RuntimeError(“You are trying to call the hook of a dead Module!”)<br>
RuntimeError: You are trying to call the hook of a dead Module!<br>
"<br>
I can train exactly the same code on CPU and on a single GPU properly , i.e,<br>
“CUDA_VISIBLE_DEVICES=0 python my_script.py”,  works fine.</p>
<p>as a reference I used (<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 1.13.1+cu117 documentation</a> and <a href="https://glassboxmedicine.com/2020/03/04/multi-gpu-training-in-pytorch-data-and-model-parallelism/" class="inline-onebox" rel="noopener nofollow ugc">Multi-GPU Training in Pytorch: Data and Model Parallelism – Glass Box</a>) for dataparallelism.</p>

eqy: <p>This is a bit difficult to debug without any visible code. Could you post a minimal version of your script (e.g., with data loading replaced with randomly generated values) that reproduces your issue?</p>
Torch grid_sample with 'wrap' padding mode

Naufal_Suryanto: <p>Hello, I am trying to port my code from TensorFlow to PyTorch.<br>
I was using <code>tfa.image.transform</code> for transforming my image based on the given transformation matrix and found that <code>torch.nn.functional.grid_sample</code> is the equivalent function.</p>
<p>However, in my previous code, I use <code>wrap</code> mode for filling the outside boundary.<br>
As a review, wrap mode will extend the input by wrapping around to the opposite edge <code>(a b c d | a b c d | a b c d)</code>.</p>
<p>And on the PyTorch <code>grid_sample</code>, I can only find the supported <code>padding_mode</code> for <code>zeros</code>, and <code>border</code>. and <code>reflection</code>.</p>
<p>Is there any alternative for implementing the <code>wrap</code> padding mode?</p>
<p>Thank you</p>
Finding the first occurrence of an element in a 2D tensor

Remorax: <p>How do I find <em>only</em> the first occurrence of an element in each row of a 2D tensor?</p>
<p>For instance, if the input is:</p>
<pre><code class="lang-auto">t = torch.Tensor([[10, 1, 2, 3, 4, -10, -10,-10],
[10, 4, 2, 3, -10, -10, -10, -10], 
[10, 1, 2, 3, 4, 5, 6, -10]])
</code></pre>
<p>And I wish to find the first occurrence of <code>value=-10</code> in every row of the tensor. Therefore, I would like the output to be:</p>
<p><code>tensor([5, 4, 7])</code></p>
<p>How would I go about this without using for loops and using torch-native operations?</p>

sami: <p>I’m looking for something similar, did you find a solution to this?</p>

KFrank: <p>Hi Sami (and Vivek)!</p>
<aside class="quote no-group" data-username="sami" data-post="2" data-topic="132423" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/35a633/48.png" class="avatar"> sami:</div>
<blockquote>
<p>I’m looking for something similar, did you find a solution to this?</p>
</blockquote>
</aside>
<p><code>((t == -10).cumsum (1).cumsum (1) == 1).argsort (1)[:, -1]</code><br>
should work.</p>
<p>Best.</p>
<p>K. Frank</p>
Computational stability of -torch.log(torch.exp(a)+torch.exp(b))

qiminchen: <p>Suppose <code>a</code> is a tensor of size <code>(1, 1, 256, 256, 256)</code>, it seems like<br>
<code>-torch.log(torch.exp(-|a|) + torch.exp(-|1-a|))</code> as the loss has computational stability issues like <code>nan</code>, are there any other implementations that can avoid this?</p>

eqy: <p>Would this native function be applicable to your use case and would it offer better numerical stability?<br>
<a href="https://pytorch.org/docs/stable/generated/torch.logaddexp.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.logaddexp.html</a></p>

qiminchen: <p>thanks, let me try this one</p>
Regarding the timing of synchronization of model's weights when trained using DDP

tanabe.d.zc32s: <p>Hello.<br>
I have recently been trying my hand at learning with Data Parallel using Distributed Data Parallel (DDP).</p>
<p>I understand that learning with DDP is done by creating replicas of the model on multiple devices (e.g. GPUs), splitting the data and training them, and synchronizing the weights. My question is, when are the weights synchronized between replicas? At each epoch?</p>
<p>I would appreciate it if you could enlighten me.<br>
Best Regards.</p>

ptrblck: <p>DDP will broadcast the <code>state_dict</code> during the construction of the DDP object as described in the <a href="https://pytorch.org/docs/master/notes/ddp.html#internal-design">DDP - Internal Design docs</a>.</p>

tanabe.d.zc32s: <p>Thank you for reply.<br>
I understand about broadcasting.</p>
What is resolving 'RuntimeError: Size Mismatch in Tensor Dimension' in Signal Generation using Pre-trained RoBERTa?

YuhskeHujisaki: <p>I want to learn the obtained signal with pre-trained RoBERTa and generate the signal, but I get a RuntimeError: The size of tensor a (768) must match the size of tensor b (374125) at non-singleton dimension 1 error. What can I do to improve it?</p>
<pre><code class="lang-auto">import torch
from transformers import RobertaTokenizer, RobertaModel
from torch.nn import Linear
from torch.optim import Adam

# Load RoBERTa tokenizer and model
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaModel.from_pretrained('roberta-base')

# Add a linear layer on top of RoBERTa to generate signal
signal_size = 2993 * 125
model.classifier = Linear(model.config.hidden_size, 374125)
# Prepare your signal data as input sequences and target signals
input_sequences = [tokenizer.encode("signal {}".format(i), return_tensors='pt') for i in range(16)]
target_signals = [train_all_data.transpose(1,3,0,2).reshape(16, signal_size)[i] for i in range(16)]
target_signals = [torch.tensor(signal) for signal in target_signals]

# Define loss function and optimizer
criterion = torch.nn.MSELoss()
optimizer = Adam(model.parameters(), lr=0.001)

# Train the model on your signal data
for epoch in range(1000):
    for input_ids, target in zip(input_sequences, target_signals):
        optimizer.zero_grad()
        outputs = model(input_ids)[0].mean(dim=1)
        loss = criterion(outputs, target)
        loss.backward()
        optimizer.step()
        
# Use the fine-tuned model to generate a new signal
input_ids = torch.tensor(tokenizer.encode("Generate a new signal", return_tensors='pt')).unsqueeze(0)
outputs = model(input_ids)[0].mean(dim=1)
new_signal = outputs.detach().numpy()
</code></pre>

eqy: <p>It’s not clear what the issue is without more context on where the error is coming from. It is a shape mismatch and it looks like it is originating from some step past the classifier layer (perhaps that output is used somehow with the original hidden dim size before being returned), but it is difficult to say without more detailed information about what part of the model is producing the error.</p>

YuhskeHujisaki: <p>This is done with loss = criterion(outputs, target).<br>
What we want to do is first run the RoBERTa model with 1,000 training steps on 20 channels of EEG data for each class. After that, there is a paper that uses n RoBERTa to generate n classes of synthetic data.</p>

eqy: <p>Right, I mean that it’s difficult to debug this without a runnable example script (e.g., the definition of <code>train_all_data</code>is missing).</p>
Tracing based selective build for cuda kernels

SimonRelu: <p>Hey, I’m trying to reduce the pytorch installation for running inference. I have a limited set of models that I’d like to run. I saw that there exists a way to trace models to create a selective build for cpu (<a href="https://pytorch.org/blog/pytorchs-tracing-based-selective-build/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch’s Tracing Based Selective Build | PyTorch</a>). I was wondering if that would also be possible to do something similar to reduce the size of <code>libtorch_cuda_cpp.so</code> and <code>libtorch_cuda_cu.so</code> by only including the kernels that are used by these models?</p>

suraj.pt: <p>Good timing! There’s a discussion posted today <a href="https://github.com/pytorch/pytorch/issues/93955" class="inline-onebox">Enable Link Time Optimization in PyTorch 2.0 Release Binaries - Smaller, Faster, Better Binaries · Issue #93955 · pytorch/pytorch · GitHub</a> that you can support</p>
Quick Job - Text Generation

tmptmp: <p>Have some data in the following format. I am paying $250 for someone to write a python program to generate replies to posts. Should be able to take a given input and provide a generated output. If your interested include your telegram username.</p>
<pre><code class="lang-auto">{
"username":"example",
"comment":"example comment",
"replies": [
{
"username":"example2",
"comment":"example comment2",
"replies": []
},
{
"username":"example3",
"comment":"example comment3",
"replies": []
}]
}
</code></pre>