akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
NLP torchtext multiple cuda - the best strategy for a large imbalanced dataset

mark_eu: <p>Hi,</p>
<p>I am working on the problem of <strong>multiclass classification</strong> on a large imbalanced dataset. Next few weeks, I will have the opportunity to use 4 Cuda devices. The structure of the dataset is simple: TEXT, LABEL.<br>
What is the best strategy in using <strong>torchtext</strong> for such a situation? Unfortunately, I could not find an example for processing on <strong>multiple cuda devices</strong> and with such a dataset (<strong>large imbalanced dataset</strong> - 20 Gb).</p>
<p>Thanks to everyone for the help and links to some working examples</p>

ptrblck: <p>You should be able to use <a href="https://pytorch.org/tutorials/beginner/dist_overview.html"><code>DistributedDataParallel</code></a> to utilize all GPUs in your model training. I’m not aware of any torchtext-specific limitations for this.</p>

mark_eu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply.<br>
I was trying to find some example for multiclass classification on multiple cudas but without success. Do you know any of such examples?</p>
<p>Kind regards</p>

ptrblck: <p>You could start by creating a training script for the multi-class classification on a single device and then add the distributed training on top of it using e.g. <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">this tutorial</a>.</p>
Make image to inputable to model

Jake_Eum: <p>0</p>
<p>I tried to load the image and make it inputable to the model.</p>
<p>cracks_guess = []</p>
<pre><code class="lang-auto">def check(arr_img):
    print('length-&gt;',len(arr_img))
    with torch.no_grad():
        for idx,img in enumerate(arr_img):
            img = torch.tensor(img).type(torch.FloatTensor).cuda()
            img = img.permute(2,0,1)
            img = img.unsqueeze(0)
            #print('shape-&gt;',img.shape)
            output = net(img)
            output = torch.sigmoid(output)
            pt = torch.round(output)
            if pt==1:
            #if output &gt; 0.3:
                cracks_guess.append(img)
                print(idx,'output-&gt;',output)
</code></pre>
<p>is this right way to change the img shape to inputable to model?</p>

bsridatta: <p>Hey, what exactly is the error you are facing. Looks like you have the input images as an array. Dont see any problem there. If your data is too big to have it loaded into one array. You could loops over the path and load one at a time.</p>
<p>This works and you are predicting one at a time.</p>
<ul>
<li>You could also just pass the whole array of dim [n_samples, n_channels, height, width] if data fits into memory or iterate over few samples at one.</li>
<li>or just write a dataset and dataloader, and change batchsize to load and compute many samples at once efficiently.</li>
</ul>
<p>I hope this answers</p>

Jake_Eum: <p>I’m not facing some error but just make sure this is the right way to put input image.</p>
<p>afterl I collect my data I try to visualize the image like</p>
<pre><code class="lang-auto">imgA = cracks_guess[0]

imgA = imgA.detach().cpu()

imgA = imgA.squeeze(0)

print(imgA.shape)

imgA = imgA.permute(1,2,0)

plt.imshow(imgA)

#plt.show()
</code></pre>
<p>but can not see the image</p>

bsridatta: <p>Why do you squeeze? Is the output shape CHW? If so plot should would</p>

puranjay_mishra: <p>why did you pass the images indiviually to the network? You can pass all the images in together and get the result. If you cannot see any image please check if your cracks_guess array is empty.<br>
I personally would pass in all my images together, get output of a binary classifier network and then loop through the outputs to see if thy are 1 or not(append the respective image having +ve prediction in cracks_guess)</p>
Libtorch's Interoperability with OpenGL/Cuda

BiPaulEr: <p>I have difficulties to copy the data from at::Tensor(to::kCuda) to an OpenGL Texture directly on the<br>
GPU. Is there a way to avoid going to the Cpu to fill OpenGL texture directly from the GPU with a at::Tensor(to::kCuda) ?<br>
Thanks.</p>
Passing https://github.com/lsrock1/maskrcnn_benchmark.cpp in VS2019

Sylvain_Ard: <p>Hi all !<br>
I found the code <a href="https://github.com/lsrock1/maskrcnn_benchmark.cpp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lsrock1/maskrcnn_benchmark.cpp: Implementation maskrcnn-benchmark, pytorch c++ frontend</a> to implements maskrcnn in C++ but this code is for Ubuntu 16.04 and is old.<br>
I want to upgrade it to work in VS2019 to do inference in Windows.<br>
Some of you could be agree to help me in the task ?<br>
We will exchange our skype and work together at volunteer because it is for a volunteer work.<br>
Thank you<br>
Best regards<br>
PS :<br>
In case there will be noone I will try myself, for example could have I help to this error :</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Gravité</th>
<th>Code</th>
<th>Description</th>
<th>Projet</th>
<th>Fichier</th>
<th>Ligne</th>
<th>État de la suppression</th>
</tr>
</thead>
<tbody>
<tr>
<td>Erreur</td>
<td>C2893</td>
<td>La spécialisation du modèle de fonction ‘enable_if&lt;std::is_same&lt;X,T&gt;::value,unknown-type&gt;::type torch::autograd::Function::apply(Args &amp;&amp;…)’ a échoué</td>
<td>layers</td>
<td>C:\Users\Sylvain ARD\Dropbox\Documents Sylvain\travail avec Boubacar\maskrcnn C++\include\rcnn\layers\conv2d.h</td>
<td>20</td>
<td></td>
</tr>
</tbody>
</table>
</div><p>thank you<br>
best regards</p>

Sylvain_Ard: <p>nobody can help me ?</p>
Intersection between 2D tensors

James_Lee: <p>Hi there, is there a way for PyTorch to calculate the intersection of two 2D tensors? The post here  <a href="https://discuss.pytorch.org/t/intersection-between-to-vectors-tensors/50364/3">Intersection between to vectors/tensors</a>  only provides the method for the 1D tensor. Many thanks!</p>

bsridatta: <p>I think the link answers 2D as well. Does this answer?<br>
<code>a = torch.FloatTensor([[1,2,3],[4,5,6]])</code><br>
<code>b = torch.FloatTensor([[1,2,9],[8,7,6]]) </code><br>
<code>(a==b).nonzero()</code><br>
 → <code>tensor([[0, 0],         [0, 1],         [1, 2]])</code></p>

James_Lee: <p>Thanks a lot. But it seemed only to return the element position. Can I get the tensor that indicates the intersection between 2D tensors?</p>

James_Lee: <p><code>(1) (a==b)</code></p>
<pre><code class="lang-auto">tensor([[ True,  True, False],
        [False, False,  True]])
</code></pre>
<p><code>(2) c = torch.FloatTensor([[1,2,0],[0,0,6]])</code></p>
<p>Can I get the value of  FloatTensor intersection(c in case (2)) rather than the bool value in case (1)?</p>
<p>Thanks!</p>

bsridatta: <p>Check (a==b) *a . This should give elements where intersecting and zeros if not. Use result.nonzeor() if you don’t want zeros</p>
Why does my Pytorch tensor size change and contain NaNs after some batches?

Dakopen: <p>I am Training a Pytorch model. After some time, even if on shuffle, the model contains, besides a few finite tensorrows only NaN values:</p>
<pre><code class="lang-auto">tensor([[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         ...,
         [ 1.4641,  0.0360, -1.1528,  ..., -2.3592, -2.6310,  6.3893],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan]]],
       device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>The detect_anomaly functions return:</p>
<blockquote>
<p>RuntimeError: Function ‘LogSoftmaxBackward’ returned nan values in its 0th output.</p>
</blockquote>
<p>in reference to the next line <code>output = F.log_softmax(output, dim=2)</code></p>
<p>A normal tensor should look like this:</p>
<pre><code class="lang-auto">tensor([[[-3.3904, -3.4340, -3.3703,  ..., -3.3613, -3.5098, -3.4344]],

        [[-3.3760, -3.2948, -3.2673,  ..., -3.4039, -3.3827, -3.3919]],

        [[-3.3857, -3.3358, -3.3901,  ..., -3.4686, -3.4749, -3.3826]],

        ...,

        [[-3.3568, -3.3502, -3.4416,  ..., -3.4463, -3.4921, -3.3769]],

        [[-3.4379, -3.3508, -3.3610,  ..., -3.3707, -3.4030, -3.4244]],

        [[-3.3919, -3.4513, -3.3565,  ..., -3.2714, -3.3984, -3.3643]]],
       device='cuda:0', grad_fn=&lt;TransposeBackward0&gt;)
</code></pre>
<p><em>Please notice the double brackets, if they are import.</em></p>
<p>Code:</p>
<pre><code class="lang-auto">spectrograms, labels, input_lengths, label_lengths = _data
spectrograms, labels = spectrograms.to(device), labels.to(device)
optimizer.zero_grad()

output = model(spectrograms)
</code></pre>
<p>Additionally, I tried to run it with a bigger batch size (current batch size:1, bigger batch size: 6) and it run without errors until 40% of the first epoch in which I got this error.</p>
<blockquote>
<p>Cuda run out of memory</p>
</blockquote>
<p>Also, I tried to normalize the data <code>torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_mels=128, normalized=True)</code></p>
<p>And reducing the learning rate from 5e-4 to 5e-5 did not help either.</p>
<p>Additional information: My dataset contains nearly 300000 .wav files and the error came at 3-10% runtime in the first epoch.</p>
<p>I appreciate any hints and I will gladly submit further information.</p>
<p>Also asked on Stackoverflow: <a href="https://stackoverflow.com/questions/67983039/why-does-my-pytorch-tensor-size-change-and-contain-nans-after-some-batches" class="inline-onebox" rel="noopener nofollow ugc">python - Why does my Pytorch tensor size change and contain NaNs after some batches? - Stack Overflow</a></p>

Dakopen: <p>Something was wrong with my labels. See Stackoverflow for more</p>
How to initialize pretrained weights to my CNN?

HarshRangwala: <p>I am attempting to build a convolutional neural network from scratch for object recognition. My input image size is (3, 256, 256)</p>
<p>Here’s my architecture for CNN.</p>
<pre><code class="lang-auto">class NutSnackClassication(MultiLabelImageClassificationBase):
    def __init__(self):
        super().__init__()
        
        self.network = nn.Sequential(
            nn.Conv2d(3, 32, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            nn.Conv2d(32, 32, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(2,2),
            
            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.MaxPool2d(2, 2),
            nn.Conv2d(128, 256, kernel_size = 3, stride = 1, padding = 1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size = 3, stride = 1, padding = 1), 
            nn.ReLU(),
            nn.MaxPool2d(2, 2),
            
            nn.Flatten(),
            nn.Linear(7*7*256, 512),
            nn.ReLU(),
            nn.Linear(512, 258),
            nn.LogSoftmax(dim = 1),
        )
    
    def forward(self, xb):
        return self.network(xb)
</code></pre>
<p>Instead of using pretrained model I want to initialize pretrained weights from ResNet34 and then use it for predictions. A lot of examples show how to download the model and then use it which doesn’t solve my query. So how do I approach this?</p>

ptrblck: <p>You could use the <code>state_dict</code> of the pretrained resnet, manipulate its keys to match the layer names of your new model (assuming all parameters have the same shape), and load it to your model.<br>
Something like this should work:</p>
<pre><code class="lang-python">model = NutSnackClassication()
reference = models.resnet34(pretrained=True)
sd = reference.state_dict()

# change the keys of sd here

model.load_state_dict(sd)
</code></pre>
Iterating through permutations of DaataLoader with zip() gives error

jwillette: <p>I am trying to iterate through different permutations of a dataset with zip() like this…</p>
<pre><code class="lang-auto">for ((x1, y1), (x2, y2), ..., (xn, yn)) in zip([dataloader for dataloader in range(n)]):
    ...
</code></pre>
<p>I am getting the following error which doesn’t crash, but seems to cause everything to run very slow and hang for a long time…</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/jeff/.venv/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1324, in __del__
    self._shutdown_workers()
  File "/home/jeff/.venv/env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1316, in _shutdown_workers
    if w.is_alive():
  File "/home/jeff/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__ at 0x7fcbdcacce50&gt;
</code></pre>
<p>Is using zip a valid way to accomplish this? Is there a better way to accomplish this without having to load multiple instantiations of the same dataloader?</p>
Torch transformation using numpy

talhaanwarch: <p>I want to create torch code for transform image using numpy. I am not sure whether I am doing it correctly?<br>
Here is pytorch code</p>
<pre><code>transform=torchvision.transforms.Compose([
torchvision.transforms.Resize((224, 224)),
torchvision.transforms.ToTensor(),
torchvision.transforms.Normalize([0.5820, 0.4512, 0.4023], [0.2217, 0.1858, 0.1705])])
img = Image.open("skin_cancer.jpg").convert('RGB')
img=transform(img)
print(torch.max(img[0,:,:]),torch.min(img[0,:,:]),torch.mean(img[0,:,:]))
</code></pre>
<p>I want to reproduce it in numpy</p>
<pre><code>img = Image.open("skin_cancer.jpg").convert('RGB')
img = img.resize((224,224))
img=np.array(img)/255
img=np.moveaxis(img,2,0)
img [0,:,:] = (img [0,:,:]  * 0.5820) / 255 * 0.2217
img [1,:,:] = (img [1,:,:] * 0.4512) / 255 * 0.1858
img [2,:,:] = (img [2,:,:]  * 0.4023) / 255 * 0.1

print(np.max(img[0,:,:]),np.min(img[0,:,:]),np.mean(img[0,:,:]))
</code></pre>
<p>I have two queries, Is <code>moveaxis</code> the correct approach to change image dimensions from <code>(224,224,3)</code> to <code>(3,224,224)</code> and is normalization the correct way to replicate as in pytorch.<br>
What is a more elegant way to do this in numpy. Actually, I am using onnx and I don’t want to install torch.</p>
<p>Output produced in numpy<br>
<code>2.563400461673167 1.245668497439571 2.0800457636539056</code><br>
Output produced in pytorch<br>
<code>tensor(1.6909) tensor(-0.3080) tensor(0.9488)</code></p>

ptrblck: <p>Yes, I think you could use <code>moveaxis</code> as it’s giving the same results here:</p>
<pre><code class="lang-python">arr = np.random.randn(224, 224, 3)
x = torch.from_numpy(arr).permute(2, 0, 1)
arr_new = np.moveaxis(arr, 2, 0)

print((x.numpy() == arr_new).all())
&gt; True
</code></pre>
<p>Your normalization is wrong, since you would need to subtract the mean and divide by the stddev.</p>
DataLoader hangs with custom DataSet

Ipsedo: <p>Hello,</p>
<p>I face a problem with DataLoader and custom DataSet.</p>
<p>Here is my custom DataSet :</p>
<pre><code class="lang-python">class AudioDataset(Dataset):

    def __init__(self, dataset_path: str) -&gt; None:
        super().__init__()

        assert isdir(dataset_path)

        all_magn = [
            f for f in tqdm(listdir(dataset_path))
            if isfile(join(dataset_path, f)) and
               f.startswith("magn")
        ]

        all_phase = [
            f for f in tqdm(listdir(dataset_path))
            if isfile(join(dataset_path, f)) and
               f.startswith("phase")
        ]

        assert len(all_magn) == len(all_phase)

        self.__all_magn = sorted(all_magn)
        self.__all_phase = sorted(all_phase)

        self.__dataset_path = dataset_path

    def __getitem__(self, index: int):
        magn = th.load(join(
            self.__dataset_path,
            self.__all_magn[index]
        ))

        phase = th.load(join(
            self.__dataset_path,
            self.__all_phase[index]
        ))

        return th.stack([magn, phase], dim=0)

    def __len__(self):
        return len(self.__all_magn)
</code></pre>
<p>wich is loaded with :</p>
<pre><code class="lang-python">if __name__ == "__main__":
    audio_dataset = audio.AudioDataset("/path/to/tensor/dir")

    data_loader = DataLoader(
        audio_dataset,
        batch_size=8,
        shuffle=True,
        num_workers=10,
        drop_last=True
    )
</code></pre>
<p>The data is well loaded but in fact the DataLoader hangs when iterating. It seems that the “speed” of loading is not constant (my dataset is +60k tensors of size = (512, 512) ) : it varies from 20min to 1h to make an epoch.</p>
<p>I precise that the “speed” of iteration is constant when I specify num_workers = 0.</p>
<p>I’ve seen that this issue is quite common, how remediate to those hang ?</p>
<p>Python = problem with both 3.6 or 3.8<br>
Pytorch = 1.9.0<br>
CUDA = 11.1<br>
Nvidia driver = 460.84<br>
Ubuntu 20.04</p>
<p>Best regards</p>

ptrblck: <p>You could profile the <code>DataLoader</code> (with <code>num_workers&gt;0</code>) and check, if you are seeing spikes in the data loading time. If so, it would point towards a data loading bottleneck, which would cause the training loop to wait for the next available batch.<br>
<a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> explains common bottlenecks and proposes some workarounds, in case you are indeed seeing this issue.</p>
How to resize a tensor?

Shrabani_Ghosh: <p>Lets say, I have a tensor<br>
a = torch.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])<br>
the size of it is torch.Size([3, 2])</p>
<p>I want to resize it to torch.Size([3, 3]) by adding a 0 to each row. Like this,<br>
torch.tensor([[0.1, 1.2, 0], [2.2, 3.1, 0], [4.9, 5.2, 0]])</p>
<p>How can I resize and add a 0?</p>

eqy: <aside class="quote no-group" data-username="Shrabani_Ghosh" data-post="1" data-topic="124483">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shrabani_ghosh/48/38079_2.png" class="avatar"> Shrabani_Ghosh:</div>
<blockquote>
<p>a = torch.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])</p>
</blockquote>
</aside>
<p>Does simply allocating a new tensor work?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])
&gt;&gt;&gt; b = torch.zeros([3, 3])
&gt;&gt;&gt; b[:3, :2] = a
&gt;&gt;&gt; b
tensor([[0.1000, 1.2000, 0.0000],
        [2.2000, 3.1000, 0.0000],
        [4.9000, 5.2000, 0.0000]])
</code></pre>
<p>My suspicion is that even if a native “resize” function were available the implementation would essentially do the same thing here.</p>
Object Detector(s) implementation w/o compilation

namangoyal: <p>I’m new here, please feel to make suggestions.</p>
<p>Looking for official object detector(s) implementation which works w/o compilation in latest stable Pytorch.<br>
Few examples</p>
<ul>
<li>MaskR-CNN (possibility to train without mask ground truth?)</li>
<li>Faster R-CNN</li>
<li>R-FCN<br>
OR single stage detectors</li>
<li>YoloV3</li>
<li>SSD</li>
</ul>
<p>Would require to ‘train’ and ‘infer’ on new datasets &amp; classes.<br>
It seems the <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="nofollow noopener">https://github.com/facebookresearch/maskrcnn-benchmark</a> requires compilation (referred as installation) and works with Pytorch nightly.</p>
What if one does not remove the forward_hook

phantom90: <p>Hi there,</p>
<p>Suppose we are using a forward hook to analyze a mid layer:</p>
<pre><code class="lang-auto">class Hook():
    def __init__(self,m):
        self.hook = m.register_forward_hook(self.hook_func)
    def hook_func(self,m,i,o):
        self.stored = o.detach().clone()
    def remove():
        self.hook.remove()
</code></pre>
<p>The Hook class is created for multiple cases. If I do not remove the hook after use, what negative effect will occur.</p>
<p>Thanks!</p>

soulitzer: <p>Looks fine to me. Its good that you detached and cloned, otherwise you would be keeping the graph or storage of the output alive.</p>
<p>In general having the hook registered on the module in itself won’t keep anything alive, if you were worried about that. Its what you do as a side effect in your hook, i.e., what you store that could be a problem, but that seems to be okay here.</p>
Downsampling odenet for resnet

Ashish_b: <p>following is my code snippet I want to classify images of CIFAR10 dataset with a ODENET can someone help with the downsampling I am getting the following error<br>
RuntimeError: Given groups=1, weight of size [64, 1, 3, 3], expected input[128, 3, 110, 110] to have 1 channels, but got 3 channels instead</p>
<p>if args.downsampling_method == ‘conv’:<br>
downsampling_layers = [<br>
nn.Conv2d(1, 64, 3, 1),<br>
norm(64),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(64, 64, 4, 2, 1),<br>
norm(64),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(64, 64, 4, 2, 1),<br>
]<br>
elif args.downsampling_method == ‘res’:<br>
downsampling_layers = [<br>
nn.Conv2d(1, 64, 3, 1),<br>
ResBlock(64, 64, stride=2, downsample=conv1x1(64, 64, 2)),<br>
ResBlock(64, 64, stride=2, downsample=conv1x1(64, 64, 2)),<br>
]</p>

ptrblck: <p>Based on the error message the input to the conv layer has 3 channels (I guess it could be the image tensor) while the conv layer expects an input with a single input channel. You could change <code>in_channels</code> to 3 in: <code>nn.Conv2d(3, 64, 3, 1)</code> to fix this.</p>

Ashish_b: <p>Thanks a ton for the reply sir</p>
Why is float tensor addition on CPU slower for AVX2 than the default ATen CPU capability?

imaginary: <p>With <code>ATEN_CPU_CAPABILITY=avx2</code>, adding two float tensors is slower than <code>ATEN_CPU_CAPABILITY=default</code>, assuming a high value for the number of threads.<br>
eg. On a machine with 32 physical cores &amp; 64 logical cores, with 16 threads, <code>ATEN_CPU_CAPABILITY=avx2</code> is slower than <code>ATEN_CPU_CAPABILITY=default</code>.</p>
<p>For just one thread, adding two float tensors is only around 10% faster with <code>ATEN_CPU_CAPABILITY=avx2</code>.</p>
<p>What’s the reason behind this? Both of them are memory-bound.<br>
Thanks!</p>

ptrblck: <p>Could you create an issue on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a> for this so that the code owners could have a look, please?</p>

imaginary: <p>Resolved at <a href="https://github.com/pytorch/pytorch/issues/60202" class="inline-onebox" rel="noopener nofollow ugc">On CPU, vectorized float tensor addition might be slower than unvectorized float tensor addition · Issue #60202 · pytorch/pytorch · GitHub</a>.</p>
<p>Basically, memory allocation &amp; zero-filling costs are worse for AVX2.</p>
Np.pi equivalent in Pytorch

veeresh_d: <p>Hi all,</p>
<p>Could someone tell the <a href="https://docs.scipy.org/doc/numpy/reference/constants.html" rel="nofollow noopener">np.pi</a> equivalent in Pytorch? Is there any built-in way to access pi value from torch library? Thanks in advance.</p>

LeviViana: <p>No there isn’t a pretty built-in way.</p>
<p>There are however a lot of ugly ways of doing so. You can, for instance, define <code>torch.pi</code> at run-time:</p>
<pre><code class="lang-auto">import torch
torch.pi = torch.acos(torch.zeros(1)).item() * 2 # which is 3.1415927410125732
</code></pre>
<p>You can even modify the <code>__init__.py</code> file (which is located in <code>print(torch.__file__)</code>) in your environment by adding a line with <code>pi = 3.14 etc</code>, so that you will always be able to use the <code>torch.pi</code> constant.</p>

veeresh_d: <p>Thank for your response.</p>

Alexander_Soare: <p>Hi! Just stumbled across this. Why would you do this instead of <code>torch.tensor(math.pi)</code>. Is it because OP said “built-in”? Is there any other practical advantage to your approach?</p>
How to insert after a node with a set of ops at one time?

Nick_Young: <p>for example, for a computation flow:</p>
<pre><code class="lang-auto">z = a + b
</code></pre>
<p>i want to change it to:</p>
<pre><code class="lang-auto">z = torch.sum(torch.abs(a)) + b
</code></pre>
<p>for now, i only figure out a way to do it like this:</p>
<pre><code class="lang-auto">with gm.graph.inseting_after(node_a):
    abs_node = gm.graph.call_function(torch.abs)
    abs_node.args = (node_a,)

# without this second with statement, sum_node will be added before abs_node
with gm.graph.inserting_after(abs_node):
    sum_node = gm.graph.call_function(torch.sum)
    sum_node.args = (abs_node,)

node_a.replace_all_uses_with(sum_node)
</code></pre>
<p>the question is, how can i do it faster and more elegant if i have 10 nodes to insert after node_a?</p>

Nick_Young: <p>solved.<br>
pack all operations in a Module, then use call_module instead.</p>
Convex combination of different model probabilities

ritvikgupta199: <p>Hi all,</p>
<p>I want to take a convex combination of the final probabilities from 2 different models in a classification problem.<br>
I want to do something like this</p>
<pre><code class="lang-auto">logits1 = model1(input)
logits2 = model2(input)

prob1 = F.softmax(logits1, dim=1)
prob2 = F.softmax(logits2, dim=1)

final_prob = beta*prob1+(1-beta)*prob2
log_prob = torch.log(final_prob)

loss = torch.nn.NLLLoss(log_prob, target)
</code></pre>
<p>Is there any faster or a computationally more stable way to do this?</p>
PyTorch How to optimize only part of a DataParallel model?

CHENSY: <p>How can one optimize only part of a Dataparallel model, while perserving the data-parallel behaviour in an appropriate way.</p>
<p>For example:</p>
<pre><code class="lang-auto">parallel_model  = DataParallel(model)
optimizer = torch.optim.SGD(parallel_model.last_conv_layer.parameter(),lr=0.01)
</code></pre>
<pre><code class="lang-auto">error：
torch.nn.modules.module.ModuleAttributeError: 'DataParallel' object has no attribute 'last_conv_layer'
</code></pre>
<p>And, if one use <code>parallel_model.module.last_conv_layer.parameter()</code> to obtain trainable weights, would DataParallel still function as expected?</p>
<p>Thanks in advance.</p>

ptrblck: <p>Yes, accessing the underlying layers via the <code>.module</code> attribute will work. You could alternatively create the optimizer before wrapping the model into <code>nn.DataParalllel</code>.</p>
Minibatch size and multiple GPUs

Yolkandwhite: <p>Does the minibatch size divides into number of GPUs when using nn.dataparallel?</p>
<p>I had the minibatch size of 16 and when I use nn.dataparallel with 4 GPUs,<br>
the actual initial input shape which goes through the model (printed it in the beginning of foward function) was like this<br>
[4, channel, width, height]<br>
[4, channel, width, height]<br>
[4, channel, width, height]<br>
[4, channel, width, height]</p>
<p>Is this because of the minibatch size is divided into each 4 GPUs and compute parallel?</p>

ptrblck: <p>Yes, <code>nn.DataParallel</code> splits the input batch in <code>dim0</code> into <code>n_gpu</code> chunks and sends each one to the corresponding device.</p>
Extending Pytorch: Python vs. C++ vs. CUDA

SimlaBurcu: <p>Hello,</p>
<p>I have been trying to implement a custom Conv2d module where grad_input (dx) and grad_weight (dw) are calculated by using different grad_output (dy) values. I implemented this by extending torch.autograd as in Pytorch tutorials.</p>
<p>However I am confused by the information <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#backward-pass" rel="noopener nofollow ugc">in this link</a>.</p>
<ul>
<li>Is extending the autograd.Function not enough?</li>
<li>What is the difference between writing a new autograd function in Python vs C++?</li>
<li>How about the CUDA implementations in <code>/torch/nn/blob/master/lib/THNN/generic/SpatialConvolutionMM.c</code> where dx and dw calculated? Should I change them too?</li>
</ul>
<p>Here is my custom function:</p>
<pre><code class="lang-auto">class myCustomConv2d(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, w, bias=None, stride=1, padding=0, dilation=1, groups=1):
        ctx.save_for_backward(x, w, bias)
        ctx.stride = stride
        ctx.padding = padding
        ctx.dilation = dilation
        ctx.groups = groups
        out = F.conv2d(x, w, bias, stride, padding, dilation, groups)
        return out

    @staticmethod
    def backward(ctx, grad_output):
        input, weight, bias = ctx.saved_tensors
        stride = ctx.stride
        padding = ctx.padding
        dilation = ctx.dilation
        groups = ctx.groups
        grad_input = grad_weight = grad_bias = None

        dy_for_inputs = myspecialfunction1(grad_output)
        dy_for_weights = myspecialfunction2(grad_output)

        grad_input = torch.nn.grad.conv2d_input(input.shape, weight, dy_for_inputs , stride, padding, dilation, groups)
        grad_weight = torch.nn.grad.conv2d_weight(input, weight.shape, dy_for_weights , stride, padding, dilation, groups)

        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = dy_for_weights .sum((0,2,3)).squeeze(0)

        return grad_input, grad_weight, grad_bias, None, None, None, None
</code></pre>

ptrblck: <p>You don’t need to write the C++ extensions and your current custom <code>autograd.Function</code> should work.<br>
Implementing the custom methods in C++ could yield a speedup as shown in the tutorial.</p>

SimlaBurcu: <p>Thank you very much!<br>
And is it the same as C++ for changing the CUDA implementations in /torch/nn/blob/master/lib/THNN/generic/SpatialConvolutionMM.c where dx and dw calculated? I mean is it only for speedup?</p>

ptrblck: <p>Yes, writing a custom CUDA operation could also yield a speedup, but is also optional.</p>
How to get the result from torch.multiprocessing

two_Two: <p>It’s suggested using pytorch multiprocessing insted of multiprocessing from python ,However in python3 I can use something like</p>
<pre><code class="lang-auto">def toy_fun(boxes):
    boxes += 20
    return boxes

with concurrent.futures.ProcessPoolExecutor as executor : 
    results = [ executor.submit(toy_fun,boxes) for _ in range(10) ] 
    
    for f in concurrent.futures.as_completed(results):
        #do something with f.result()
</code></pre>
<p>However I do not know how can I process or get the result  get from method using in  <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">MULTIPROCESSING BEST PRACTICES</a></p>
<p>How I can I do this ? Thanks!</p>
Why my SVD algorithm in cuda is slower than it in cpu?

sunocean: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/351123526c018a9f2e0c49968daca9d2a1fc959f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/351123526c018a9f2e0c49968daca9d2a1fc959f" title="IMG_20210618_152258"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/351123526c018a9f2e0c49968daca9d2a1fc959f_2_690x474.jpeg" alt="IMG_20210618_152258" data-base62-sha1="7zs1tN0QxcvX5ENUnibfFD214DB" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/351123526c018a9f2e0c49968daca9d2a1fc959f_2_690x474.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/351123526c018a9f2e0c49968daca9d2a1fc959f_2_1035x711.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/351123526c018a9f2e0c49968daca9d2a1fc959f_2_1380x948.jpeg 2x" data-dominant-color="63505A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20210618_152258</span><span class="informations">3583×2463 1.27 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this because the version of cuda is 11.2 which is not compatible with torch 1.8.1+cuda11.1, or the SVD algorithm cannot be accelerated by GPU?</p>
Is there any efficient way to load MS1M dataset?

ruka: <p>I try to train <a href="https://arxiv.org/abs/1804.07573" rel="noopener nofollow ugc">MobileFaceNet</a> using <a href="https://github.com/deepinsight/insightface/wiki/Dataset-Zoo" rel="noopener nofollow ugc">MS1M-IBUG</a> (85K ids/3.8M images)</p>
<p>But I am facing low CPU and GPU utilization. CPU keeps a low util like 30%, GPU goes to 70% for one second and keeps 0% most of the time.<br>
Following is one of my implementations for dataloader. My another implementation is converting mxnet record to normal jpeg files beforehand and use PIL/opencv to read it, but it’s also very slow.<br>
I tried num_worker, pin_memory, none of them significantly speed up the data loading. But if I skip the loading part and generate some random tensor, the GPU can reach 90% utilization.<br>
So any suggestions?</p>
<pre><code class="lang-auto">import torch
import torchvision.transforms as transforms
import mxnet as mx
from mxnet import recordio
class MyDataset(torch.utils.data.Dataset):
    def __init__(self, mxnet_record = 'train.rec', mxnet_idx = 'train.idx'):
        self.data = recordio.MXIndexedRecordIO(mxnet_idx, mxnet_record,'r')
        self.transform = transforms.Compose([transforms.RandomHorizontalFlip(),
                                             transforms.ToTensor()
                                             ])

    def __len__(self):
        return 3804846
    
    def __getitem__(self, index):
        header, s = recordio.unpack(self.data.read_idx(index+1))
        image = mx.image.imdecode(s).asnumpy()
        label = int(header.label)
        
        image = self.transform(image)
        
        return image, torch.tensor(label, dtype = torch.long)
</code></pre>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> explains some common data loading bottlenecks and proposes workarounds as well, so you might want to take a look at it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How can I describe whats my problem, Can not use "view"

alyeasin: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2f7d85077014a1cf6fb42cb87d944d937d4d539.png" alt="Screenshot 2021-06-17 at 21-13-43 Google Colaboratory" data-base62-sha1="wnQUEpdugwPlN7K9LvL8qtO9MYV" width="483" height="200"></p>
<p><strong>TypeError: ‘NoneType’ object is not subscriptable</strong><br>
even before that code, When I used “view” like this =&gt;<br>
x = x.view(-1, self._to_linear)<br>
I got error , It said - AttributeError: ‘NoneType’ object has no attribute ‘view’<br>
but just passed it with Try &amp; Exception,<br>
but now again, what’s my problem?</p>

Aditya_Ranganath: <p>What is X_test ? Also what is the ouput of the network ? Can you share what ‘X_test’ and ‘net’ is ? It seems one of them is ‘None’</p>
Torch.load() raise an error

jokerWRN: <p>Hi, when i run the following code, an error raises. Can anybody help? Thanks!</p>
<p>my code:</p>
<pre><code>if len(args.test_model) != 0:
    assert os.path.exists(args.test_model), \
        "file not found: {}".format(args.test_model)

    checkpoint_ = torch.load(args.test_model)
    # FIXME
    key_m, key_u = net.load_state_dict(checkpoint_['model'], strict=False)

    if key_u:
        print('Unexpected keys :')
        print(key_u)

    if key_m:
        print('Missing keys :')
        print(key_m)
        raise KeyError
</code></pre>
<p>error:<br>
Traceback (most recent call last):<br>
File “/home4/user_from_home1/wangyufei/dc/dsd/test.py”, line 625, in <br>
main(config)<br>
File “/home4/user_from_home1/wangyufei/dc/dsd/test.py”, line 621, in main<br>
test(args)<br>
File “/home4/user_from_home1/wangyufei/dc/dsd/test.py”, line 516, in test<br>
checkpoint_ = torch.load(args.test_model)<br>
File “/home1/wangyufei/anaconda3/envs/py3.8bindtr/lib/python3.8/site-packages/torch/serialization.py”, line 585, in load<br>
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)<br>
File “/home1/wangyufei/anaconda3/envs/py3.8bindtr/lib/python3.8/site-packages/torch/serialization.py”, line 765, in _legacy_load<br>
result = unpickler.load()<br>
ModuleNotFoundError: No module named ‘metrics’</p>

ptrblck: <p>The recommended way to serialize a model would be to store its <code>state_dict</code> as described <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html">here</a>. Based on the error you are seeing it seems you’ve stored the entire model via <code>torch.save(model, path)</code>. If that’s the case, you would have to make sure the code structure while loading the model is equal to the one used during the saving.</p>

jokerWRN: <p>Thanks for your reply!<br>
The point in your reply may be a case that caused the problem. But I found that I got the above error because I defined a metrics package that has same name with built-in package. Add from <strong>future</strong> import absolute_import can solve it.</p>
Modify BERT encoder layers?

calusbr: <p>Hello I would like to modify the encoder layers of the BERT model, to insert FC and ReLu layers.</p>
<p>This idea allows you to reproduce the use of <a href="https://arxiv.org/abs/1709.01507" rel="noopener nofollow ugc">Squeeze-and-Excitation Networks</a></p>
<p>How to use an nn.module class to handle encoder outputs?</p>
<p>Example:</p>
<pre><code class="lang-auto">import torch.nn as nn
from transformers import BertModel

class CustomBERTModel(nn.Module):
    def __init__(self):
          super(CustomBERTModel, self).__init__()
          self.bert = BertModel.from_pretrained("bert-base-uncased")
          # add your additional layers here, for example a dropout layer followed by a linear classification head
          self.dropout = nn.Dropout(0.3)
          self.out = nn.Linear(768, 2)

    def forward(self, ids, mask, token_type_ids):
          sequence_output, pooled_output = self.bert(
               ids, 
               attention_mask=mask,
               token_type_ids=token_type_ids
          )

          # we apply dropout to the sequence output, tensor has shape (batch_size, sequence_length, 768)
          sequence_output = self.dropout(sequence_output)
    
          # next, we apply the linear layer. The linear layer (which applies a linear transformation)
          # takes as input the hidden states of all tokens (so seq_len times a vector of size 768, each corresponding to
          # a single token in the input sequence) and outputs 2 numbers (scores, or logits) for every token
          # so the logits are of shape (batch_size, sequence_length, 2)
          logits = self.out(sequence_output)

          return logits
</code></pre>
Gradient decent steps in a batch?

sj27182: <p>Hello,</p>
<p>I am quite new to pytorch, my background is more mathematical.<br>
In the tutorials I’ve been following we use gradient decent as our optimization function. Recently we began using the DataLoader class, and from what I can tell, after taking one batch of observations, and differentiating the resulting weighted sum (inside the cost function), we call .step() on the optimizer, before immediately looping back around for a new batch of observations.</p>
<p>I am confused because it feels like we are taking only one step with the gradient vector, per a batch of observations.</p>
<p>Is it the case that .step() is actually taking many gradient steps?</p>
<p>Or is it possible that taking one step with a given batch then immediately receiving a new cost-function with different terms (different observations) is a good practice?</p>

eqy: <p>No, only one gradient step is taken. You can view this step as somewhere inbetween gradient descent and pure stochastic gradient descent: <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent" rel="noopener nofollow ugc">Stochastic gradient descent - Wikipedia</a>. The reason the typical approach is to do a single step per batch rather than a single step per example is a balance of algorithmic (the ideal would be to take a single step per example) and computational efficiency (batching increases throughput on most hardware).</p>

sj27182: <p>Brilliant, thank-you!</p>

sj27182: <p>For future reference I want to point out that I was just a bit confused by the batches. After using a batch size the same size as the observations in the dataset, I got the “continuous” gradient descent values I was looking for. For some reason I thought each step generated by a batch in the epoch wasn’t immediately applied to the parameters, and instead some sort of “sequence of steps” were applied at the end of the epoch.<br>
SORRY!</p>
SigKill when installing wheels for torch 1.7.1

brutal_logic: <p>Hello,</p>
<p>Our team has torch==1.7.1 as a dependency in requirements.txt. When running bazel build on a target, I’m getting a SigKill action happening when wheels is being built for torch. It looks like at some point, bazel runs:</p>
<p><code>python3 -m pip --isolated wheel torch==1.7.1</code> which results in:</p>
<pre><code class="lang-auto">Collecting torch==1.7.1
Killed
</code></pre>
<p>If I run it as a part of my full bazel build, I get:</p>
<pre><code class="lang-auto">Collecting torch==1.7.1
 (Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vagrant/.cache/bazel/_bazel_vagrant/f9910d98673307a31f928c448bd4acd0/external/rules_python/python/pip_install/extract_wheels/__main__.py", line 5, in &lt;module&gt;
    main()
  File "/home/vagrant/.cache/bazel/_bazel_vagrant/f9910d98673307a31f928c448bd4acd0/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 87, in main
    subprocess.run(pip_args, check=True)
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/vagrant/venv/bin/python3', '-m', 'pip', '--isolated', 'wheel', '-r', '/home/vagrant/tech-backends/requirements.txt']' died with &lt;Signals.SIGKILL: 9&gt;.
)

</code></pre>
<p>Does anyone know how to address this? I have python 3.8.5.</p>