akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Getting correct output dimensions from an autoencoder taking variable input dimensions

username: alee023
cooked: <p>I am attempting to write a variational autoencoder, using PyTorch, that takes in 3D images of varying dimensions. My code right now works for even input dimensions which will face no issues when going through the pooling layer. For odd dimensions, however, the pooling layer will round down (for example, 15-&gt;7). Consequentially, one of my decoder layers that doubles the dimensions will give the incorrect output dimension (7-&gt;14, rather than 15).</p>
<p>One of the workarounds I’ve come across is resizing the inputs to a compatible, even dimension before putting them through the network. Are there any other fixes to this issue – such as by using padding, etc)? I really appreciate any help.</p>

username: ptrblck
cooked: <p>Padding the inputs with shape checks would certainly be another valid approach.<br>
In case you are using transposed conv layers for the upsampling, you could also use the <code>output_size</code> argument (in its <code>forward</code> call) to specify the desired output shape in case it’s ambiguous.</p>
The backward realized by myself is inconsistent with autograd under fp16

username: nobody
cooked: <p>I implement a backward function of pos embedding lookup, and I call torch.allclose to verify it with autograd result. My implementation is very simple, as follows:</p>
<pre><code class="lang-auto"># pos_matrix: [seq_len, seq_len]
# pos_weight: [max_pos_size, hidden_size]
# pos_embed: [seq_len, seq_len, hidden_size]
grad_weight = torch.zeros((max_pos_size, hidden_size))
for i in range(seq_len):
    for j in range(seq_len):
        pos = pos_matrix[i, j]
        grad_weight[pos] += grad_embed[i, j]
</code></pre>
<p>It is consistency under FP32(difference &lt; 1e-5), but there is a big difference on FP16(Max absolute difference: 0.0625), and this difference increases as the length of the sequence increases.</p>
<p>What could cause this difference? The difference of calculation order in cuda code? or fp16 overflow/rounding error? or is there a bug in my implementation?</p>

username: ptrblck
cooked: <p>A larger error using <code>float16</code> would be expected and for your particular use case you could check the errors in FP16, FP32, and FP64 to check, if the relative (and abs.) errors decrease in the expected ranges.</p>
Higher derivatives

username: amaleki
cooked: <p>Consider function <code>f(x, y) = sin(x) sinh(x)</code>, we can easily show: <code>f_xx(x,y) + f_yy(x,y) == 0</code><br>
How come then this code gives me something else</p>
<pre><code class="lang-auto">import torch
N = 10  # number of points
p = torch.rand(10, 2).requires_grad_()  #  `x` and `y` in the example above are the first and second columns of p.
f = torch.sin(p[:, 0]) * torch.sinh(p[:, 1])
df_dp = torch.autograd.grad(f, p, grad_outputs=torch.ones_like(f), create_graph=True, only_inputs=True)[0]
df_dpp = torch.autograd.grad(df_dp, p, grad_outputs=torch.ones_like(df_dp), create_graph=True, only_inputs=True)[0]
print(df_dpp.sum(dim=1))
</code></pre>
<p>I can confirm <code>df_dp</code> columns contain the true <code>_x</code> and <code>_y</code> gradients.</p>
How to concatenate all the predicted labels in XLA?

username: gabrielwong1991
cooked: <p>Hi does anyone knows how to get all predicted labels from all 8 cores of XLA and concatenate them together?</p>
<p>Say I have a model:<br>
<code>outputs = model(ids, mask, token_type_ids)</code><br>
<code>  _, pred_label = torch.max(outputs.data, dim = 1)</code></p>
<p>If I do<br>
<code>all_predictions_np = pred_label.cpu().detach().numpy().tolist()</code></p>
<p>apparently, this only sends the result to CPU from TPU core:0. How can I get all 8 cores and concatenate them together in the same list? I am not sure if <code>xm.all_gather()</code> is used in this case?</p>
How to calculate the entropy of an image?(my softmax results doesn't sum up to 1)

username: jjcopl
cooked: <p>hi, I am very new to ML and this is my first post here, sorry if something doesn’t match the standard of the site.<br>
I am trying to calculate the entropy of an image, and before that, I should use the softmax function to get the probabilities, there is no error in this process but the softmax result doesn’t sum up to be 1, it should add up to 1 right?<br>
here is the part of the code</p>
<pre><code class="lang-auto">    transform = Compose([ToTensor(), Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])
    x = transform(image)
    x = x.unsqueeze_(0).cuda()
    with torch.no_grad():
        y = model(x)
    logits = y.data.cpu()
    logits = torch.squeeze(logits)
    smax = nn.Softmax(dim=1)
    smax = smax(x)
    print("SOFTMAX SUM:",smax.sum())
    # SOFTMAX SUM: tensor(2073600., device='cuda:0')
    # this is the shapes of the variables: SHAPE X, Y, LOGITS torch.Size([1, 3, 1080, 1920]) 
    #torch.Size([1, 19, 1080, 1920]) torch.Size([19, 1080, 1920])
</code></pre>
<p>after this, I should calculate the entropy (E = -sum(p_i * log2(p_i))of the image(or some pieces of it.)<br>
can someone please help? seems like the torch doesn’t have an entropy method.</p>

username: ptrblck
cooked: <p>The output of the <code>nn.Softmax(dim=1)</code> layer will sum to <code>1</code> in the specified dimension as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, 10, 10)
smax = nn.Softmax(dim=1)
out = smax(x)
print(out.sum())
&gt; tensor(1000.)
print(out.sum(dim=1)) # ones
</code></pre>

username: jjcopl
cooked: <p>so for calculating the entropy I should get the softmax for every dimension?<br>
and can you help me to calculate the entropy, please?</p>
Segmentation fault problem while inferring images for larger batch size on CPU

username: morou
cooked: <p>Hi ,<br>
I have a segmentation fault problem while inferring images for larger batch size on CPU.<br>
Environment:</p>
<ul>
<li>CPU name: AMD EPYC 7763 64-Core Processor x 2</li>
<li>Docker image: <a href="http://nvcr.io/nvidia/pytorch:20.10-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:20.10-py3</a>
</li>
<li>torch: 1.7.0a0+7036e91</li>
<li>torchvision: 0.8.0a0<br>
Setting:</li>
<li>Number of works:64</li>
<li>Batch size: [32, 64, 128, 256, 512]<br>
Model: torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)<br>
Image: <a href="https://cocodataset.org/#explore?id=342322" rel="noopener nofollow ugc">https://cocodataset.org/#explore?id=342322</a><br>
To Reproduce simply:</li>
</ul>
<pre><code class="lang-auto">import torch
import torchvision
import os
import cv2
device = torch.device('cpu')
batchsize = 258

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
model.eval() 
model.to(device)

img_path = os.path.join('test_one_img', 'test.jpg')  
img = cv2.imread(img_path, -1)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = torchvision.transforms.functional.to_tensor(img)
img = list(img.to(device) for _ in range(batchsize))

output = model(img)
</code></pre>
<p>Result:<br>
Batch size &lt;128: work fine.<br>
Batch size &gt;256: segmentation fault<br>
gdb debug infos:</p>
<pre><code class="lang-auto">Thread 1 "python" received signal SIGSEGV, Segmentation fault.
ROIAlignForward&lt;float&gt; (nthreads=nthreads@entry=90617856, input=0x7f5d13b50040, spatial_scale=@0x7ffcd15a953c: 0.25,
    channels=channels@entry=256, height=height@entry=200, width=width@entry=272, pooled_height=14, pooled_width=14, sampling_ratio=2,
    aligned=false, rois=0x562a6c532300, output=0x7f6e507aa040) at /tmp/pip-req-build-6hs294b4/torchvision/csrc/cpu/ROIAlign_cpu.cpp:199
199     /tmp/pip-req-build-6hs294b4/torchvision/csrc/cpu/ROIAlign_cpu.cpp: No such file or directory.
</code></pre>
<p>Any help is appreciated.<br>
Thanks!</p>

username: ptrblck
cooked: <p>Could you update PyTorch and torchvison to the latest stable release (or nightly) and check, if you are still seeing the issue, as the <code>1.7.0</code> pre-release is a bit older by now?</p>
1D Convolutional Autoencoder

username: Mehdi
cooked: <p>Hello,</p>
<p>I’m studying some biological trajectories with autoencoders. The trajectories are described using x,y position of a particle every delta t. Given the shape of these trajectories (3000 points for each trajectories) , I thought it would be appropriate to use convolutional networks.</p>
<p>So, given input data as a tensor of (batch_size, 2, 3000), it goes the following layers:</p>
<pre><code>            # encoding part
	self.c1 = nn.Conv1d(2,4,16, stride = 4, padding = 4)
	self.c2 = nn.Conv1d(4,8,16, stride = 4, padding = 1)
	self.c3 = nn.Conv1d(8,1,8, stride = 3, padding = 1)
	self.l1 = nn.Linear(60,20)

	# decoding part
	self.l2 = nn.Linear(50,20)
	self.d1 = nn.ConvTranspose1d(1,8,8,stride = 3, output_padding = 1, padding = 1)
	self.d2 = nn.ConvTranspose1d(8,4,16, stride = 4, output_padding = 1, padding = 1)
	self.d3 = nn.ConvTranspose1d(4,2,16, stride = 4, output_padding = 0)
</code></pre>
<p>It appears that this architecture doesn’t work very well, since I can’t get the loss below 1.9.<br>
Could anyone shed some light on me as how to handle this case ? I believe that I’m using the convolution parameters (kernel size, padding and stride) in a bad way.</p>
<p>Thanks !</p>

username: Amirata_Ghorbani
cooked: <p>You don’t have any nonlinearities.</p>
Custom data loader for big data

username: hossein
cooked: <p>Hi everyone,</p>
<p>I’m going to load a dataset that is about 16G in colab, but as you know, due to low RAM space it is not possible. The best solution (in my opinion!) would be to load data in batches (my data is located in my Google Drive). All in all, I cannot write a custom data loader to do it for me. (load data from my Google Drive and in specified batches). Can any one help me?<br>
Note: My data are in .npy form</p>
<p>Thank you,</p>

username: arya47
cooked: <p>The best approach is to write a custom Dataset, something like:</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader

# custom dataset
class MRIData(Dataset):
    def __init__(self, test=False):
        # List of all image files
        self.patients = os.listdir('./data')
        
    def __len__(self):
        return len(self.patients)
    
    def __getitem__(self, x):
        patient = self.patients[x]
        filename = './data/'+ patient
        image = np.load(filename, allow_pickle=True)

        images = image[0][0]
        images = torch.from_numpy(images)
        label = image[0][1].item()
        label = torch.tensor(abs(label), dtype=torch.float)
        
        return images, label

train_ds = MRIData()
train_dl = DataLoader(train_ds, batch_size=1..) # add whatever args you need
</code></pre>
<p>You can try this by changing path to data</p>

username: hossein
cooked: <p>Thanks for your answer.<br>
I couldn’t understand that how can I load them in batches? (in other words, what parameter defines the batch size?).<br>
Moreover, I’ve converted my data and its labels into a single tensor file using <code>TensorDataset</code> and then save it in my google drive (using <code>torch.save</code>) . Is it possible to load it in bathces??</p>
Using `multiprocessing' in PyTorch on Windows10 got error-RuntimeError: Couldn't open shared file mapping: <torch_13684_4004974554>, error code: <0>

username: Dee1
cooked: <p>Hi,</p>
<p>I am currently running a PyTorch code on Windows10 using PyCharm. This code firstly utilised <code>DataLoader' function (</code>num_workers’=4) to load training data:</p>
<pre><code class="lang-auto">train_loader = DataLoader(train_dset, batch_size, shuffle=True,
                                  num_workers=4, collate_fn=trim_collate)
</code></pre>
<p>Then, in training process, it utilised a `for’ loop to load training data and train the model:</p>
<pre><code class="lang-auto">for i, (v, norm_bb, q, target, _, _, bb, spa_adj_matrix,
                        sem_adj_matrix) in enumerate(train_loader):
</code></pre>
<p><strong>Error:</strong> I got the following error messages when running above `for’ loop:</p>
<pre><code class="lang-auto">0%|          | 0/6934 [00:00&lt;?, ?it/s]Traceback (most recent call last):
  File "E:\PyTorch_env\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "E:\PyTorch_env\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "E:\PyTorch_env\lib\site-packages\torch\multiprocessing\reductions.py", line 286, in reduce_storage
    metadata = storage._share_filename_()
RuntimeError: Couldn't open shared file mapping: &lt;torch_13684_4004974554&gt;, error code: &lt;0&gt;
python-BaseException
Traceback (most recent call last):
  File "E:\PyTorch_env\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
python-BaseException
  0%|          | 0/6934 [00:07&lt;?, ?it/s]
</code></pre>
<p>It seems that there are some issues with the `multiprocessing’ function on Windows system. <strong>Could you please let me know if there are any possible solutions for this?</strong></p>
<p>The environment settings:</p>
<ol>
<li>Windows10, PyCharm</li>
<li>PyTorch v1.0.1, torchvision v0.2.2, Python 3.7.11</li>
<li>One GPU node</li>
</ol>
<p>Many thanks!</p>

username: ptrblck
cooked: <p>For the sake of completeness: issue is also tracked <a href="https://github.com/pytorch/pytorch/issues/63331">here</a>.</p>

username: Dee1
cooked: <p>Many thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
Fairseq RoBERTa Pre-Trained Model Loading Error: "TypeError: expected str, bytes or os.PathLike object, not NoneType"

username: rohanath
cooked: <h3>
<a name="_issue_-1" class="anchor" href="#_issue_-1"></a><strong><em>Issue:</em></strong>
</h3>
<p>Model trained using <strong>Fairseq</strong>, specifically <a href="https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md" rel="noopener nofollow ugc">this</a> example and dataset, i.e. RoBERTa Pre-training, results with checkpoints saved in such a way that they cannot be loaded programmatically using model loading code snippet provided at the bottom of their page, or as the following script:</p>
<pre><code class="lang-auto">
import sys
from fairseq.models.roberta import RobertaModel 

ckpt_path = sys.argv[1]

roberta = RobertaModel.from_pretrained("checkpoints", "checkpoint_best.pt", ckpt_path)

assert isinstance(roberta.model, torch.nn.Module)

</code></pre>
<p>Where the path to the saved checkpoint (saved in a <strong><em>protobuf</em></strong> format during training with the commands from the example) is passed as a command line argument.</p>
<h3>
<a name="_error-stack_-2" class="anchor" href="#_error-stack_-2"></a><strong><em>Error Stack:</em></strong>
</h3>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test_ckpt.py", line 6, in &lt;module&gt;
    roberta = RobertaModel.from_pretrained("checkpoints", "checkpoint_best.pt", ckpt_path)
  File "fairseq/fairseq/models/roberta/model.py", line 284, in from_pretrained
    **kwargs,
  File "/fairseq/fairseq/hub_utils.py", line 66, in from_pretrained
    path = os.path.join(model_path, file)
  File "/opt/anaconda3/envs/robertapretrain0/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

</code></pre>
<p>The issue seems to be related to how the checkpoints are saved during training.<br>
The structure of the checkpoint_best.pt file is as such:</p>
<pre><code class="lang-auto">Checkpoint_best.pt
├── archive
│   ├── data
│   │   ├── 0
│   │   ├── 1
│   │   ├── 10
│   │   ├── 100
│   │   ├── 101
│   │   ├── 102
│   │   ├── 103
│   │   ├── 104
│   │   ├── 105
│   │   ├── 106
│   │   ├── 107
│   │   ├── 108
│   │   ├── 109
│   │   ├── 11
│   │   ├── 110
│   │   ├── 111
│   │   ├── 112
│   │   ├── 113
│   │   ├── 114
│   │   ├── 115
│   │   ├── ...243
│   ├── data.pkl
│   └── version
└── out
2 directories, 247 files
</code></pre>
<p>I’d like to understand whether there seems to be an issue in the way the model is being loaded, or whether there’s an issue with the way the checkpoint is being saved.</p>
<h3>
<a name="_environment-details_-3" class="anchor" href="#_environment-details_-3"></a><strong><em>Environment Details</em></strong>
</h3>
<ul>
<li>fairseq Version: <em>0.10.2</em>
</li>
<li>PyTorch Version: <em>1.9.0+cu102</em>
</li>
<li>OS:<br>
<em>NAME=“Ubuntu”<br>
VERSION=“20.04.2 LTS (Focal Fossa)”<br>
ID=ubuntu<br>
ID_LIKE=debian<br>
PRETTY_NAME=“Ubuntu 20.04.2 LTS”<br>
VERSION_ID=“20.04”<br>
VERSION_CODENAME=focal<br>
UBUNTU_CODENAME=focal</em>
</li>
<li>Python version: <em>Python 3.8.8</em>
</li>
</ul>
Creating A Batch of Sparse Tensors

username: Kevin96
cooked: <p>Let say I want to create a 3-dimensional tensor, where the first dimension is the batch size,</p>
<pre><code class="lang-auto">a[batch_index][sparse_dim1][sparse_dim2].
</code></pre>
<p>Thus I need a hybrid sparse tensor representation. The first dimension is a dense dimension. The second and the third dimensions are sparse dimensions.</p>
<p>However, from the <a href="https://pytorch.org/docs/stable/sparse.html" rel="nofollow noopener">documentation</a>, it seems that the dense dimension must be after the sparse dimension? Then how I represent the hybrid tensor?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can consider <code>batch</code> as a sparse dimension. it just happens to have an entry for each of them.<br>
So considering this as a completely sparse tensor will work.</p>

username: GabbyChan
cooked: <p>Could you provide an example?</p>
<p>At the moment, I have index vector i, such as follows;</p>
<pre><code class="lang-auto">i = torch.tensor([[1,2,3],[0,1,2]])
</code></pre>
<p>The value is True on the location of <code>i</code> in matrix of size 10x10.</p>
<p>However, the problem is that in each batch the size of i is different.<br>
For example, in one batch, it is 2x3. Another maybe 2x5.</p>
<p>Could you recommend a proper tools for this ?</p>
Segmentation fault (core dumped) in torch1.5.1

username: eja
cooked: <p>I tried to train my model and it worked a few days before. However, when I tried to train again now I received this error. i have not changed any codes from the last one.</p>
<pre><code>segmentation fault (core dumped)
</code></pre>
<p>then I did the faulthandler and it showed this:</p>
<pre><code class="lang-auto">Fatal Python error: Segmentation fault

Current thread 0x00007f6ac0982740 (most recent call first):
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/comm.py", line 39 in broadcast_coalesced
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/_functions.py", line 21 in forward
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py", line 72 in _broadcast_coalesced_reshape
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/replicate.py", line 89 in replicate
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 159 in replicate
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 154 in forward
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550 in __call__
  File "train_2.py", line 79 in run
  File "train_2.py", line 120 in &lt;module&gt;
Segmentation fault (core dumped)

</code></pre>
<p>the line 81 in train.py is</p>
<pre><code>predicts = net(image)
</code></pre>
<p>line 122-123 is</p>
<pre><code>if __name__ == '__main__':
run()
</code></pre>
<p>i already reinstalled the python3.6.9 and it still gave me the same error.</p>
<p>this is my dataparallel code:</p>
<pre><code class="lang-auto">cuda_available = torch.cuda.is_available()
device_ids = [0,1] #number of gpu available
torch.cuda.set_device(device_ids[0])

if cuda_available:
    net = net.cuda()
    net = nn.DataParallel(net, device_ids=device_ids)
</code></pre>

username: ptrblck
cooked: <p>Could you update to the latest stable release (<code>1.9.0</code>) or the nightly binary and rerun your script, please?</p>

username: eja
cooked: <p>i tried torch 1.9.0 before but it gave me nccl error.</p>
Model.cuda() takes larger memory usage than the model size

username: helloybz
cooked: <p>Hi,</p>
<p>I’ve just run nothing but below,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt;m = torch.Linear(10,10)
&gt;&gt;&gt;m.cuda()
</code></pre>
<p>and I’ve checked the model’s size in MiB by</p>
<pre><code class="lang-auto">&gt;&gt;&gt;sum([p.numel() * p.element_size() for p in m.parameters()]) / 1024 / 1024
0.00041961669921875
</code></pre>
<p>but when <code>nvidia-smi</code> says,</p>
<pre><code class="lang-auto">Wed Aug 18 00:57:01 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.142.00   Driver Version: 450.142.00   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   67C    P0    41W /  70W |   1072MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     28553      C   python                           1069MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>Why the model takes more than its size, when it moved to cuda?</p>

username: helloybz
cooked: <p>Oh, am I supposed to use <code> torch.cuda.memory_allocated()</code> ?</p>
<p>The result shows 1024bytes.</p>

username: ptrblck
cooked: <p>The reported memory by <code>nvidia-smi</code> shows the allocated memory by PyTorch as well as the CUDA context, which could use between ~600-1000MB depending on the GPU, CUDA version etc.</p>
Multiclass-multioutput classification with different number of classes

username: ljj7975
cooked: <p>As described in this <a href="https://scikit-learn.org/stable/modules/multiclass.html#multiclass-multioutput-classification" rel="noopener nofollow ugc">documentation</a>, multiclass-multioutput classification is a classification task which labels each sample with a set of <strong>non-binary</strong> properties.</p>
<p>In the example provided, each sample has two labels, one for type of fruit (label 1) and the other for the color of the fruit (label 2).</p>
<p>Based on the description, this classification must cover the case where label 1 has x possibilities and label 2 has y possibilities where x and y are different.</p>
<p>If I were to set this up with PyTorch,</p>
<ol>
<li>How should the output/labels be formatted?</li>
<li>What would be proper loss function</li>
</ol>
<p>Thanks</p>

username: ptrblck
cooked: <p>I don’t think a combined loss function is necessarily needed and would probably see both tasks independently using different model outputs (with a shared “root” model) and with an own criterion.<br>
E.g. you could create two different “heads” (e.g. classification layers) while reusing the same feature extractor, return the logits for both tasks separately and train the model using both losses.</p>
How could I configure/optimize a loss function?

username: Z_Huang
cooked: <p>I am working with my CNN model right now and currently fine-tuning it. I choose nn.CrossEntropyLoss() as my loss function due to its strong ability to deal with multiple-class classification tasks. I read the loss function description and see the CrossEntropyLoss() has the following parameters: torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction=‘mean’).</p>
<p>I was wondering that is there any way to optimize these parameters inside of nn.CrossEntropyLoss() function? Further, could I optimize any other loss functions?</p>

username: ptrblck
cooked: <p>The only argument which could potentially be optimized would be the <code>weight</code> argument, as the others are optional arguments to change the internal reduction.<br>
However, I’m unsure how valuable it would be to optimize the <code>weight</code>, as e.g. a zero weight would create a “perfect” but potentially useless loss value of 0, so your model might be able to cheat through optimizing it. In any case, since I’m not familiar with your use case, it could still be valid if you make sure the aforementioned issue cannot happen.</p>
How to find input corresponding to m smallest outputs?

username: samarendra109
cooked: <p>As a continuation of this question,</p>
<aside class="quote" data-post="1" data-topic="129355">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarendra109/48/32204_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-find-input-corresponding-to-argmin-of-output/129355">How to find input corresponding to argmin of output?</a> 
  </div>
  <blockquote>
    I have a basic scenario like this, 
import torch

def someFn(w):
    
    return w.sum(dim=1)

w = torch.rand(10,3,2)
fW = someFn(w)

fMinW, fIndex = fW.min(dim=1)

# Select wmin of size 10,3

Now I want to select the 10 (1x3) vectors that correspond to the fMinW output. How can I achieve this ?
  </blockquote>
</aside>

<p>I have a scenario like this,</p>
<pre><code class="lang-auto">import torch

def someFn(w):
    
    return w.sum(dim=1)

w = torch.rand(10,3,6)
fW = someFn(w)

fIndex = fW.argsort(dim=-1)

# Select wmin of size 10,3,2
</code></pre>
<p>I want to select the minimum m=2 w that correspond to the two minimum output.</p>
<p>I tried</p>
<pre><code class="lang-auto">wmin = w[torch.arange(w.size(0)), :, fIndex[:, :m]]
</code></pre>
<p>But it throws error.</p>
Why dose aten::gt operation takes more time launching kernel?

username: ProHuper
cooked: <p>Here’s the problem, I found in torch profiler timeline file that compare operations takes more time launching cuda kernel.<br>
Here’s a pic comparing aten::gt kernel launching time with  another operation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d89193408c91e7eb255597a0ca9243a6da5b9141.png" data-download-href="https://discuss.pytorch.org/uploads/default/d89193408c91e7eb255597a0ca9243a6da5b9141" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d89193408c91e7eb255597a0ca9243a6da5b9141_2_690x132.png" alt="image" data-base62-sha1="uTR1FZmkNEirKnNkpCojnYCfmdr" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d89193408c91e7eb255597a0ca9243a6da5b9141_2_690x132.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d89193408c91e7eb255597a0ca9243a6da5b9141_2_1035x198.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d89193408c91e7eb255597a0ca9243a6da5b9141_2_1380x264.png 2x" data-dominant-color="E0CDC6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2206×424 57.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>is this normal ? if so, why?</p>
Pytorch some questions about YOLOV5

username: GTX3000
cooked: <p>Every time I run the yolov5 training program on colab it pops up with this error, I don’t feel like it’s a warning issue and every time it pops up underneath as if it’s a copy like symbol for keyboard input, I swear I didn’t move the keyboard, does anyone know why if, you want to see this warning I’ll post it.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92bea1421c9e6b7a6362394d9d3d7c224d6dfb92.png" data-download-href="https://discuss.pytorch.org/uploads/default/92bea1421c9e6b7a6362394d9d3d7c224d6dfb92" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92bea1421c9e6b7a6362394d9d3d7c224d6dfb92.png" alt="image" data-base62-sha1="kWa8gLJSf1pAjnKAbawZuvzdHNw" width="690" height="63" data-dominant-color="494949"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1476×136 4.96 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here’s that warning：</p>
<p>0% 0/6 [00:00&lt;?, ?it/s]/usr/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 6 leaked semaphores to clean up at shutdown<br>
len(cache))</p>
Speeding up a dataset

username: ZeroMax
cooked: <p>Hello,</p>
<p>I have a dataset of video frames in a folder with the directory structure looking like this:</p>
<pre><code class="lang-auto">- parent_dir
 -video_id
  -frame0
  -frame1
  -...
</code></pre>
<p>My code, gathering frames into a Pytorch dataset and splitting them up into patches is quite slow in the <code>__init__</code> method/phase when scaled up to quite a lot of frames/videos. Might there be a more efficient way to create the dataset?</p>
<p>Code:</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
    def __init__(self, directory='../data/*', get_subdirs=True, size=(16,16), max_ctx_length=4096, dry_run=False):
        print("Loading dataset...")
        self.data = glob.glob(directory)
        if get_subdirs:
            data_temp = []
            for p, i in enumerate(self.data):
                print("Loading data from {0}. {1} more to go...".format(i, len(self.data)-p))
                file_data = glob.glob(i+"/*")
                file_data.sort(key=lambda r: int(''.join(x for x in r if (x.isdigit())))) #This sort...
                data_temp.extend(file_data)
                if dry_run:
                    break
        self.data = data_temp
        self.max_ctx_length = max_ctx_length
        self.size = size
    def __len__(self):
        return len(self.data)*self.size[0]-self.max_ctx_length-1
    def __getitem__(self, key):
        frame_start = int(np.floor(key / self.size[0]))
        patch_start = int(np.mod(key, self.size[0]))
        
        patches = []
        i_frame = frame_start

        while len(patches) &lt;= self.max_ctx_length+1:
            frame = (Tvio.read_image(self.data[i_frame], mode=Tvio.ImageReadMode.RGB).float() / 255).unsqueeze(0)
            if len(patches) == 0:
                patches.extend(F.unfold(frame, self.size, stride=self.size).transpose(1,2).split(1,1)[patch_start:])
            else:
                patches.extend(F.unfold(frame, self.size, stride=self.size).transpose(1,2).split(1, 1))
            i_frame += 1
        patches = patches[:self.max_ctx_length+1]

        data_x = patches[0:-1]
        data_y = patches[1:]

        return torch.cat(data_x, dim=1).squeeze(0), torch.cat(data_y, dim=1).squeeze(0)
</code></pre>

username: eqy
cooked: <p>As it is written, it looks like the init function could be parallelized across each item in <code>self.data</code>. Would using something like multiprocessing ( <a href="https://docs.python.org/3/library/multiprocessing.html" rel="noopener nofollow ugc">multiprocessing — Process-based parallelism — Python 3.9.6 documentation</a> ) speed this up?</p>

username: ZeroMax
cooked: <p>Yes, I think so. Probably could create a cache file to store the number of frames in the current video directory as well…</p>
Seed setup for random number generator to ensure reproducibility

username: Huaiyang_Gongzi
cooked: <p>According to the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">reproducibility</a> session of Pytorch Doc, we can set seed for random number generator in Pytorch, python, and numpy-related libraries respectively.</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)

import random
random.seed(0)

import numpy as np
np.random.seed(0)
</code></pre>
<p>For  a generic ML workflow, involving numpy-based libraries, like scikit-learn; and pytorch, etc. The best practice to ensure reproducibility is to set the seed using all the above three ways. Is that right?</p>

username: ptrblck
cooked: <p>Yes, that’s right, but the linked docs also mention more needed steps to ensure reproducibility between runs, so you would also need to check these points.</p>
Generating random vectors in PyTorch

username: Arna_Ghosh
cooked: <p>I am trying to generate several random vectors (of dimensionality d) such that</p>
<ol>
<li>All elements are drawn from a standard normal distribution (mean=0, std=1)</li>
<li>The vector norms are 1</li>
<li>The expectation of Sum of elements in each vector is 0 (expectation taken over the number of vectors generated)</li>
<li>The sum of the generated vector yields a 0 vector, i.e. sum of elements in each dimension is 0.</li>
</ol>
<p>I tried using the <code>torch.randn()</code> function to generate the whole tensor (of size N x d, where N is number of vectors) and use it to test if this set of vectors satisfies the above 4 conditions. As it turns out, condition 2 is satisfied pretty well but conditions 3 and 4 are not matched (it gets slightly better when N and d are increased but the error is often higher than I would want, e.g. setting N=d=1000 gives an error of order 1e-2 in both conditions).<br>
Is there a better way to sample random vectors to match these conditions in pytorch? Also, is there a difference in using <code>torch.randn()</code> and <code>torch.normal()</code>?<br>
Thanks in advance. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
What is the effect of criterion.to(device)

username: seer_mer
cooked: <p>Usually, when we want to train on a certain device, we do xxx.to(device), a typical procedure is</p>
<pre><code class="lang-auto">x = x.to(device)
y = y.to(device)
model = model.to(device)
</code></pre>
<p>where you add data and model to the desired device.</p>
<p>However, I noticed that some people also add criterion (loss function) to device:<br>
<code>criterion = criterion.to(device)</code></p>
<p>Is this necessary? Will the whole training process still occur on GPU if I <strong>only</strong> add data and model to GPU as shown in the first three lines? What’s the benefit of moving criterion to the given device?<br>
Thanks.</p>

username: ptrblck
cooked: <p>The necessity of calling <code>criterion.to(device)</code> would depend on the used criterion and in particular if it’s stateful, i.e. if it contains internal tensors etc.<br>
In the latter case, you should get a device mismatch error in case this line is missing and needed, so in case you are not getting any errors, you could skip this line of code (but it would also be a no-op otherwise).</p>
Batch predict big dataset

username: psonneveld
cooked: <p>Hello all, I am pretty new to PyTorch, so I hope this is not too dumb a question, but I am running into a problem with prediction on a dataset using my trained PyTorch model. I have a trained model for sentiment analysis, that all went well. But I need to apply it to a dataset of about 1.1 million unlabeled texts. I have tried doing so a few times, but I tend to end up with too high RAM usage, causing either my Colab session or my own computer to crash.<br>
Someone suggested doing the prediction in batches, but I don’t understand how I would do so. Could someone please point me in the right direction for how I am supposed to predict new labels in this use case? Any help would be greatly appreciated!</p>

username: ptrblck
cooked: <p>I’m not sure if your use case has specific requirements, but the common approach would be to use a <code>Dataset</code>, wrap it in a <code>DataLoader</code> and process each batch using the model.<br>
The <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">dataloading tutorial</a> might be a good starter.</p>

username: psonneveld
cooked: <p>Thank you for the suggestion! I am using the method suggested here for training the model: <a href="https://github.com/aramakus/ML-and-Data-Analysis/blob/master/RoBERTa%20for%20text%20classification.ipynb" class="inline-onebox" rel="noopener nofollow ugc">ML-and-Data-Analysis/RoBERTa for text classification.ipynb at master · aramakus/ML-and-Data-Analysis · GitHub</a><br>
It saves the model as two files, a model.pkl and a metrics.pkl file.<br>
Basically, I am trying to predict new labels using this model. Is the dataloader then the right approach?</p>

username: ptrblck
cooked: <p>In case you would like to create the predictions for a dataset, using a <code>DataLoader</code> sounds like a good approach. On the other hand, if you would like to get a prediction of a single sample, you could directly pass it to the model, so the “best” approach depends a bit on your use case.</p>
Newbie question ; loading in custom lexicon in torchtext dataset - no legacy

username: suegi
cooked: <p>It’s probably very simple, but I just want to play around with RNN/LSTM/etc… with this text-&gt;regression problem. My dataset is just a t c g without spaces, of variable length. I’m confused by all the examples out there that use the legacy torchtext, spacy, and default tokenizers (e.g. the english corpus). How should I adapt this dataset?</p>
<p>This is what I have so far. It’s a big dataset &gt;1million so I would like to do it in a sensible way. Without packing (to better understand pytorch API), then with packing (as I saw this is the proper way)</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset


#simple pandas dataframe w/ 2 columns for sequence and target
input_data = pd.read_csv('./data/GSE135464_means_nextseq_GPD.csv')

class atcgDataset(Dataset):
    def __init__(self,table):
        self.seq = np.array(table['Seqs'])
        self.values = np.array(table['values'],dtype=np.float32)
    def __len__(self):
        return len(self.seq)
    def __getitem__(self,index):
        return self.values[index], self.seq[index]
</code></pre>
<p>update; I hacked the tutorial here to get it to work</p>
<p><a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html</a></p>
<p>But I am still curious about other ways to do it. If you know of any meaningfully better ways, please tell! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Working with RNG states on CPU and GPU

username: mdlockyer
cooked: <p>I have a bit of confusion surrounding PyTorch’s RNGs and how/when the CPU or GPU/CUDA generator is used. I have a model that saves the RNG state internally and essentially replicates the behavior of <code>torch.random.fork_rng()</code>. I do this as new layers may be added during the training process and I need to deterministically initialize their weights(the whole thing is painfully complex but that kinda sums it up). At a high level, my code stashes the main RNG state from <code>torch.get_rng_state()</code> in a local variable and then swaps it out with the state that it keeps. E.g.,</p>
<pre><code>master_rng_state = torch.get_rng_state()  # Get the "master" RNG state
torch.set_rng_state(self._rng_state)  # This is my "forked" RNG state

... do some initialization stuff here

self._rng_state = torch.get_rng_state()  # Update the forked RNG state
torch.set_rng_state(master_rng_state)  # Restore the master RNG state
</code></pre>
<p>In this example, my forked state <code>self._rng_state</code> is taken from <code>torch.get_rng_state()</code> when the model is first initialized.</p>
<p>This works quite well when running on the CPU, but I have a problem when I run this on the GPU.<br>
When I try to set the RNG state with my forked state: <code>torch.set_rng_state(self._rng_state)</code> it turns out the forked state has been moved to the GPU. Torch gives me a <code>TypeError: expected a torch.ByteTensor, but got torch.cuda.ByteTensor</code>. Does anyone know why this happens? Cuda is never called on the forked state and all the models are moved to CPU memory before any of this happens.</p>

username: Stone
cooked: <p>Did you register the <code>self._rng_state</code> as a <code>buffer</code> or <code>parameters</code>?</p>

username: mdlockyer
cooked: <p>I actually figured out what the issue was. I am appending the RNG state to the models state dict when saving a checkpoint. So when loading from a checkpoint, the <code>map_location</code> was sending it to the GPU. Silly oversight on my part.</p>
<p>Thanks for responding anyway though! the state tensor being a parameter would have been a good explanation <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: alirezakazemipour
cooked: <p>That saved me too!</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=10" title=":+1:" class="emoji" alt=":+1:"></p>
I am plotting three plots per audio file using subplots. How to made axis invisible?

username: krishna511
cooked: <pre><code class="lang-auto">path=glob.glob('............./*.wav') 

fig, ax =  plt.subplots(nrows=4, ncols=3, sharex=True)

    
for i in range(4) :
   
    y, sr = librosa.load(path[i], sr=16000)
    plt.axis('off')
    
    librosa.display.waveplot(y, sr, ax=ax[i, 0])  # put wave in row i, column 0
    
    
   
    mfcc=librosa.feature.mfcc(y) 
    plt.axis('off')
    librosa.display.specshow(mfcc, x_axis='time', ax=ax[i, 1]) # mfcc in row i, column 1
    

    S = librosa.feature.melspectrogram(y, sr)
    plt.axis('off')
    librosa.display.specshow(librosa.power_to_db(S), x_axis='time', y_axis='log', ax=ax[i, 2])  # spectrogram in row i, column 2
    
</code></pre>
<p>I tried different positions for <code>plt.axis(off)</code> But none is working only on the last plot it’s working. any suggestions <a class="mention" href="/u/ptrblck">@ptrblck</a> sir plz help, I know this is out of forum discussion but if you can guide me,<br>
Regards<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/266aa9b5e8450875481266846dc6c473cb18fd5e.png" alt="Fig" data-base62-sha1="5tQFqnZIdYnYhPeTSnxBCm7XMnQ" width="370" height="264"></p>

username: krishna511
cooked: <p>Just to close the conversation and help others , I got the answer . After defining subplot axis off is used like this:</p>
<pre><code class="lang-auto">fig, ax =  plt.subplots(nrows=4, ncols=3, sharex=True)
[axi.set_axis_off() for axi in ax.ravel()]
</code></pre>
<p>and problem is solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54f4485b6c389de0293e13a3cc2ae0a16e4f4866.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/54f4485b6c389de0293e13a3cc2ae0a16e4f4866" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54f4485b6c389de0293e13a3cc2ae0a16e4f4866.jpeg" alt="image" data-base62-sha1="c7xtNHkave0BrgS2XcjC2Pmtfn0" width="690" height="455" data-dominant-color="9DCEB4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">946×625 51.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to sync the training in different processes?

username: gudiandian
cooked: <p>I am training models with DDP. I started several processes manually and each process is responsible for the training on one certain GPU. When the model is trained in the processes, I send messages to these processes. I would like these processes would stop training the current model, release GPU memory of the old model, and start training another model after receiving such a message.<br>
However, if the training of these processes are not “synchronized”, I can never successfully release the memory of the “old model” in GPU.</p>
<p>I am releasing the memory like this:</p>
<pre><code class="lang-auto">def destruction(self):
        torch.cuda.synchronize()
        del self.optimizer
        del self.ddp_model
        del self.train_loader
        torch.cuda.empty_cache()
</code></pre>
<p>For example, when process A  receives the message, it is still in the ith iteration of training; when process B receives the message, it has already enters the (i+1)th iteration. Then, both processes enters <code>destruction()</code> (because they both received the message). Then, the processes will hangs there.</p>
<p>Is there any way to make sure that the training on each process are “synchronized” before they try to release the memory of the “old model” Thanks!</p>

username: cbalioglu
cooked: <p>Which data parallel API are you using? DataParallel or Distributed Data Parallel (aka DDP)? With DDP you can use <code>torch.distributed.barrier()</code> function to explicitly synchronize your workers. Check out <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">our docs</a> for further info.</p>

username: gudiandian
cooked: <p>Thanks for such a quick reply!</p>