akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: Expected object of scalar type Byte but got scalar type Float for argument #3 'mat1' in call to _th_addmm_

username: kdesai
cooked: <p>Hi, I am quite new to PyTorch and FastAI, and I am getting this error while using fastai’s unet_learner and according to the traceback, the error is generated at <code>learn.lr_find()</code>. Can you please let me know how I can fix this? Thanks in advance.</p>
<p>Here’s a part of my code:</p>
<pre><code>class UNIZDataset(Dataset):
 
  def __init__(self, folderlist):

    self.c = 5
    count = 0
    for folder in tqdm(os.listdir(data_path)):
        images, labels = self.extract_data(os.path.join(data_path,folder))
        tensor_x = torch.from_numpy(np.array(images))
        tensor_y = torch.from_numpy(np.array(labels))
        if count == 0:
            self.x = tensor_x
            self.y = tensor_y
            count+=1
        else:
            self.x = torch.cat((self.x,tensor_x), dim = 0)
            self.y = torch.cat((self.y,tensor_y), dim = 0)

 def extract_data(self, path):
    x = []
    y = []
    data_x = os.path.join(path,'DATA_X')
    data_y = os.path.join(path,'DATA_Y')
    fileList = glob.glob(os.path.join(data_x,'*.tif'))
    for file in sorted(fileList):
        x.append(cv2.imread(os.path.join(data_x,file)))
        y.append(cv2.imread(os.path.join(data_y,file)))
    return (x,y)

def __getitem__(self, index):
    image = self.x[index,:,:]
    image = image.permute(2,0,1)
    label = self.y[index,:,:]
    label = label.permute(2,0,1)
    return (image,label)

def __len__(self):
    return self.x.size()[0]

train_dataset = UNIZDataset(train_files)
valid_dataset = UNIZDataset(valid_files)

databnch = DataBunch.create(train_dataset,valid_dataset, bs = 18)
learn = unet_learner(data=databnch,  arch=models.resnet34)
learn.lr_find()</code></pre>

username: user_123454321
cooked: <p>Maybe you need to convert the images to float32…can you replace</p>
<pre><code class="lang-auto">        tensor_x = torch.from_numpy(np.array(images))
</code></pre>
<p>with</p>
<pre><code class="lang-auto">        tensor_x = torch.from_numpy(np.array(images)).float()
</code></pre>

username: kdesai
cooked: <p>Yes, that worked! Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: user_123454321
cooked: <p>You might want to divide by 255 like this</p>
<pre><code class="lang-auto">        tensor_x = torch.from_numpy(np.array(images)).float() / 255.
</code></pre>
<p>otherwise the model may take long time to train</p>

username: kdesai
cooked: <p>Thanks! I will do that <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:">. I am also running into another issue “IndexError: Target 20 is out of bounds.” when it starts to train. Can you please tell me what’s causing this error? Thanks again in advance!</p>
Finding Histogram for DICOM images

username: MonIA
cooked: <p>Is there a way to calculate the histogram of DICOM images? such as the one in OpenCV “cv2.calcHist”</p>

username: ptrblck
cooked: <p><code>cv2.calcHist</code> might work, if you would like to apply it on a single slice.<br>
Are you seeing any errors using it or are you looking for a specific implementation?</p>
How to iterate over a batch?

username: Stanley_C
cooked: <p>I’m currently training with this loop</p>
<pre><code class="lang-auto">for epoch in range(EPOCH):
	for step, (x, y) in enumerate(train_loader):
</code></pre>
<p>However, x and y have the shape of (num_batchs, width, height), where width and height are the number of dimensions in the image. So, how do I iterate over the x and y to get a 3d tensor with the dimensions (1, width, height), for both the x and y, and how do I convert the 3d tensor to a 4d tensor, for a 2d CNN?</p>

username: user_123454321
cooked: <p>Hey, can you try this…</p>
<pre><code class="lang-auto">for epoch in range(EPOCH):
    for step, (x, y) in enumerate(train_loader):
        for i in range(len(x)):
             xi = x[i, ...][None, ...]
             yi = y[i, ...][None, ...]

</code></pre>
Cross Entropy Loss per Example

username: maralm
cooked: <p>I have a cross entropy loss defined as below:</p>
<pre><code class="lang-auto">self.loss_fn = torch.nn.CrossEntropyLoss(ignore_index=-1, reduction = 'mean')
masked_lm_loss = self.loss_fn(prediction_scores.view(-1, self.vocab_size), masked_lm_labels.view(-1))
</code></pre>
<p>where prediction_scores is 64x128x30000 and masked_lm_labels is 64x128 (64 is the batch size).<br>
I need to get per example loss too. So I tried to change the reduction to ‘none’ to get the per sample loss and used</p>
<pre><code class="lang-auto">torch.mean(masked_lm_loss_per_example.view(64, 128))
</code></pre>
<p>to get the total loss. However, the result is different from when I set the reduction to ‘mean’. Per exaqmple loss also doesn’t seem to be correct. Is there anything wrong in my calculation?</p>

username: ptrblck
cooked: <p>Could you post “small” tensors, which would reproduce these wrong results?<br>
<a href="https://discuss.pytorch.org/t/userwarning-size-average-and-reduce-args-will-be-deprecated-please-use-reduction-sum-instead/24629/6">Here</a> is a small example using different <code>reduction</code> settings with and without using <code>ignore_index</code>.</p>
Randomly Sampling Windows from Time Series Dataset

username: calvinh
cooked: <p>I have a large dataset (I have split it into 2000 files, each with an hour of 256hz time series data, too large for memory), and my model takes 30 seconds of this data as input. I want to sample these 30 second windows randomly from the data. I was worried that using a map style dataset would be too slow considering it would have to load a whole file in order to get a  30 second window from it, which turned out to be true. I made a custom batch sampler and a custom dataset, where <strong>getitem</strong> calls the pandas function read_csv with skiprows and nrows parameters to try and only load what I need, but it is still incredibly slow due to the read_csv call on every sample to a file with ~900k rows, even with batching and multiple workers.</p>
<p>First, if I just saved a standard dataset with predefined windows rather than taking random intervals from the data and used a pytorch random sampler on it, would this be an efficient solution? I don’t really know how the stock dataset classes work or if they would be efficient with randomly sampling a dataset of this size. (I would be fine with this if the sampling was well-distributed, worked quickly, and didn’t require storing the data many times over)</p>
<p>I’m open to any suggestions about how to approach this. Thanks.</p>

username: googlebot
cooked: <p>I would preprocess csv files to numpy files. Maybe with numpy.memmap, if no pre-loading method is used.</p>
Convert Tensor to CUDA in Pytorch C++, But the content changed

username: HeyLynne
cooked: <p>Hi, everyone, I’m facing a problem that I need to copy a tensor to CUDA. The code are as follows:</p>
<pre><code class="lang-auto">torch::Tensor r_data1 = torch::from_blob(raw_data.data(), {batch_size, arr_size}).to(torch::kInt64);
auto tmp_data = static_cast&lt;size_t*&gt;(r_data1.storage().data());
size_t tmp_size = r_data1.numel();
cout &lt;&lt; "r data1 tensor ";
for (size_t i = 0; i &lt; tmp_size; i++) {
    cout &lt;&lt; tmp_data[i] &lt;&lt; "\t";
}
cout &lt;&lt; endl;

torch::Tensor r_data2 = r_data1.cuda();
tmp_data = static_cast&lt;size_t*&gt;(r_data2.cpu().storage().data());
tmp_size = r_data2.numel();
cout &lt;&lt; "r data tensor ";
for (size_t i = 0; i &lt; tmp_size; i++) {
    cout &lt;&lt; tmp_data[i] &lt;&lt; "\t";
}
cout &lt;&lt; endl;
</code></pre>
<p>But the results are:</p>
<p><strong>r data1 tensor</strong>  101 4886 2456 4689 3787 2336 2356 1298 2128 2356 4886 2456 4689 1298 2128 2356 736 6963 7252 5401 3215 3333 1921 3441 8110 1384<br>
<strong>r data tensor</strong>  139682015587936 139682014923568 2456 4689 3787 2336 2356 1298 2128 2356 4886 2456 4689 1298 2128 2356 736 6963 7252 5401 3215 3333 1921 3441 8110 1384</p>
<p>The ids have been changed. Why? Have you come into this problem?</p>
Why are all threads bound to the same core?

username: khalil
cooked: <p>I want to use pytorch DDP module to do the distributed training and I use the OpenBLAS as the BLAS. When I execute the following benchmark</p>
<pre><code class="lang-auto">import timeit
runtimes = []
threads = [1] + [t for t in range(2, 49, 2)]
for t in threads:
    torch.set_num_threads(t)
    r = timeit.timeit(setup = "import torch; x = torch.randn(1024, 1024); y = torch.randn(1024, 1024)", stmt="torch.mm(x, y)", number=100)
    runtimes.append(r)
</code></pre>
<p>I found that different threads were running on different cores.<br>
However when I execute my training script, I found all threads are bound to the same core.<br>
script:</p>
<pre><code class="lang-auto">export GLOO_SOCKET_IFNAME=ib0
export NUM_CORES=64
export OMP_NUM_THREADS=$NUM_CORES
NPROC_PER_NODE=1
COMMAND="$HOME/deepnet_mpi/CosmoFlow.py --epochs=120 --backend=gloo --workers=0 --batch-size=1 --print-freq=50 --data=$HOME/Nbody/datasets/v6"

python3 -m torch.distributed.launch \
--nproc_per_node=$NPROC_PER_NODE \
$COMMAND
</code></pre>
<p>What is the reason of this problem? And this is my environment:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.6.0a0+b31f58d
Is debug build: No
CUDA used to build PyTorch: None

OS: CentOS Linux release 7.6.1810 (AltArch)
GCC version: (GCC) 9.2.0
CMake version: version 3.16.5

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.6.0a0+b31f58d
[conda] Could not collect
</code></pre>
<p>I found that when multiple processes are set up, different processes also use the same CPU core.</p>

username: mrshenli
cooked: <p>hmm, I am not aware of any DDP code that would change the threading behavior.</p>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> do you know what might lead to this behavior?</p>

username: khalil
cooked: <p>Thanks for your reply, now I have solved this problem. The reason is that I do not set an openmp environment variable</p>
<pre><code class="lang-auto">export GOMP_CPU_AFFINITY=0-127
</code></pre>
Handling oom error during DDP backward

username: T_Qri
cooked: <p>Hi, I’m wondering how to deal with occasional OOM error happend during DDP backward.</p>
<p>For <code>forward</code>, oom can be captured simply by a try-catch statement. For backward, however, <code>loss.backward()</code> performs gradient calculation and the registered hooks perform gradient reduction at the same time.</p>
<p>Is it possible to hang due to oom errors during backward in several process so that the other successful processes keep waiting for them? If so, is there a nice way to recover from this problem?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="T_Qri" data-post="1" data-topic="92266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/t_qri/48/24891_2.png" class="avatar"> T_Qri:</div>
<blockquote>
<p>Is it possible to hang due to oom errors during backward in several process so that the other successful processes keep waiting for them?</p>
</blockquote>
</aside>
<p>Yes, it is. If one process hit OOM and skipped/reran the the backward pass, it would cause de-synchronization across processes in the same group, which would lead to hang or crash.</p>
<blockquote>
<p>If so, is there a nice way to recover from this problem?</p>
</blockquote>
<p>Yep, <a href="https://pytorch.org/elastic" rel="noopener nofollow ugc"><code>TorchElastic</code></a> is built to solve this issue. cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>

username: Kiuk_Chung
cooked: <p>Have a look here at:</p>
<ol>
<li>
<a href="https://pytorch.org/elastic/0.2.0/train_script.html" rel="nofollow noopener">https://pytorch.org/elastic/0.2.0/train_script.html</a>  - for instructions on how to write a “torchelastic compliant” train script</li>
<li>
<a href="https://pytorch.org/elastic/0.2.0/quickstart.html" rel="nofollow noopener">https://pytorch.org/elastic/0.2.0/quickstart.html</a> - for a quickstart on launching your script with torchelastic</li>
</ol>

username: T_Qri
cooked: <p>Thank you, I will try it out.</p>

username: T_Qri
cooked: <p>Thank you, I will have a try.</p>
Segment errors always occur when doing back propagation

username: khalil
cooked: <p>I compile the source code in the aarch64 machine. When I use Pytorch for distributed training, there are always segment errors, just like this：</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/GPUFS/nsccgz_xliao_lds/local/python3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/GPUFS/nsccgz_xliao_lds/local/python3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/GPUFS/nsccgz_xliao_lds/local/torch/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in &lt;module&gt;
    main()
  File "/GPUFS/nsccgz_xliao_lds/local/torch/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/GPUFS/nsccgz_xliao_lds/local/python3.7/bin/python3', '-u', '/GPUFS/nsccgz_xliao_lds/deepnet_mpi/CosmoFlow.py', '--local_rank=0', '--epochs=120', '--backend=gloo', '--workers=0', '--batch-size=1', '--print-freq=50', '--data=/GPUFS/nsccgz_xliao_lds/Nbody/datasets/v6']' died with &lt;Signals.SIGSEGV: 11&gt;.
</code></pre>
<p>I make a simple test and then I found the error occurs during back propagation.</p>
<pre><code class="lang-auto">loss.sum.backward()
</code></pre>
<p>collect_env:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.5.0a0+4ff3872
Is debug build: No
CUDA used to build PyTorch: None

OS: CentOS Linux release 7.6.1810 (AltArch)
GCC version: (GCC) 9.2.0
CMake version: version 2.8.12.2

Python version: 3.7
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.5.0a0+4ff3872
[conda] Could not collect
</code></pre>
<p>uname -a</p>
<pre><code class="lang-auto">Linux aln220 4.14.0-115.el7a.0.1.aarch64 #1 SMP Sun Nov 25 20:54:21 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
</code></pre>
<p>And I use the OpenBLAS as the BLAS backend.And I found if I set the environment variable OMP_NUM_THREADS to 1, this error will not occur.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Can you give a python stack trace of where it happens? in particular, how do you use distributed: via DDP or rpc? Do you use the distributed optimizer?</p>
<p>Also can you use gdb to see where the segfault comes from?</p>

username: khalil
cooked: <p>Sorry to reply you so late, I recompile the OpenBLAS and now the problem is solved.Thanks for your reply</p>
Accuracy decreases on validation set with smaller batch sizes

username: bPangolin
cooked: <p>I am training the following model to classify is something is in one state or another. What I’m struggling with is when using smaller batch sizes, my models validation score sits at 50%, so no better than guessing. If I increase the batch size to even just 4, I start getting 85+% accuracy. I’m using BCELoss.</p>
<pre><code class="lang-auto">import torch.nn as nn

class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        
        self.conv_layers = nn.Sequential(
            nn.Conv1d(3, 6, 5),
            nn.ReLU(),
            nn.Conv1d(6,16,5),
            nn.ReLU(),
            nn.Conv1d(16,32,5),
            nn.ReLU(),
        )
        self.linear_layers = nn.Sequential(
            nn.Linear(32*3,16),
            nn.ReLU(),
            nn.Linear(16,2),
            nn.Softmax(dim=0)
        )
        
        
    def forward(self, x):
        x = self.conv_layers(x)
        x = x.view(-1, 32*3)
        return self.linear_layers(x)
</code></pre>
<p>Accuracy is being calculated as follows:</p>
<pre><code class="lang-auto">running_acc += torch.eq(torch.argmax(lb, axis=1), torch.argmax(out, axis=1)).sum()
...
acc = running_acc / total_ds_length
</code></pre>
<p>Since the label is one hot encoded.</p>
<p>My model is in eval() mode.</p>

username: bPangolin
cooked: <p>I was actually wrong to use BCELoss.<br>
CrossEntropyLoss better suited my application and now provides an accuracy increase of 4% as well over both the validation and test sets with all kinds of batch sizes.</p>
<p>A lesson in understanding the right approach.</p>
Unexpected F1 score result compare to BLUE score for NMT task

username: Aiman_Mutasem-bellh
cooked: <p>Dear <span class="mention">@all</span>,</p>
<p>I’m trying to calculate F1 score for NMT task. I found some good resources online:</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/latest/metrics.html" rel="nofollow noopener">pytorch-lightning</a><br>
<a href="http://scikit-learn.org/stable/modules/model_evaluation.html#from-binary-to-multiclass-and-multilabel" rel="nofollow noopener">scikit-learn metrics</a><br>
<a href="https://pytorch.org/ignite/metrics.html#" rel="nofollow noopener">ignite.metrics</a></p>
<p>I have used <code>ignite.metrics</code> as follow:</p>
<pre><code class="lang-auto">def eph_precision(output, trg):

    output = torch.tensor(output) #predicted
    trg = torch.tensor(trg) #output
                         
    precision = Precision(output_transform=thresholded_output_transform,average=True)

    precision.update((output, trg))
    epoch_precision = precision.compute()
        
    return epoch_precision

def eph_recall(output, trg):

    output = torch.tensor(output) #predicted
    trg = torch.tensor(trg) #output
                         
    recall = Recall(output_transform=thresholded_output_transform,average=True)

    recall.update((output, trg))
    epoch_recall = recall.compute()
        
    return epoch_recall

def evaluate(model, iterator, criterion):
    
    model.eval()
    
    epoch_loss = 0
    epoch_acc = 0 
    epoch_precision = 0 
    epoch_recall = 0 
    
    with torch.no_grad():
    
        for i, batch in enumerate(iterator):

            src = batch.src
            trg = batch.trg

            output, _ = model(src, trg[:,:-1])
            #output, _ = model(src, trg[:,:-1])
        
            output = output.contiguous().view(-1, output.shape[-1])
            trg = trg[:,1:].contiguous().view(-1)

            loss = criterion(output, trg)
            precision = eph_precision(output, trg)
            recall = eph_recall(output, trg)
  
            epoch_loss += loss.item()
            epoch_precision += precision
            epoch_recall += recall 
        
        
    return epoch_loss / len(iterator), epoch_precision / len(iterator), epoch_recall / len(iterator)
</code></pre>
<p>Print epoch output :</p>
<p><code>valid_loss, valid_prce, valid_recall  = evaluate(model, vali_iterator, criterion)</code><br>
<code>vali_f1score = valid_prce * valid_recall  * 2 / (valid_prce+ valid_recall   + 1e-20)</code></p>
<p>I got an unexpected F1 score <code>0.1245</code> compare to BLEU score = <code>49.31</code>.</p>
<p>Any suggestions to fix this issue, also what is the optimal way to apply the F1 score for NMT task?</p>
<p>Kind regards,</p>
Volatile removed no effect

username: bql2007
cooked: <p>Hi!</p>
<p>I need help. Taking a training course but looks like the code needs to be updated. Since volatile was removed… here is the code and warning:</p>
<pre><code>def select_action(self, state):
    probs = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
    action = probs.multinomial()
    return action.data[0,0]
</code></pre>
<p>/home/ben/Self_Driving_Car/ai.py:63: UserWarning: volatile was removed and now has no effect. Use <code>with torch.no_grad():</code> instead.<br>
probs = F.softmax(self.model<em>100) # T=100<br>
/home/ben/Self_Driving_Car/ai.py:63: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.<br>
probs = F.softmax(self.model</em>100) # T=100</p>
<p>Any help will be highly appreciated.</p>
<p>Regards,<br>
Ben</p>

username: albanD
cooked: <p>Hi,</p>
<p>A new version of this code would be:</p>
<pre><code class="lang-python">@torch.no_grad()
def select_action(self, state):
    probs = F.softmax(self.model(state)*100) # T=100
    action = probs.multinomial()
    return action[0,0]
</code></pre>

username: bql2007
cooked: <p>Thank you very much for your help!</p>
<p>Best regards,</p>

username: ambrish
cooked: <p>can you help me why here in self.model(…) we have passed argument because it is an instance of network class . And if we can do so how it is associated or utilized inside and instance of a class.</p>
Different Input Shapes for Triplet Loss

username: MiloKnell
cooked: <p>I’m using triplet loss on a siamese network where each half receives a different sized input (128 and 512), how would you recommend that I compute the loss? Would expanding the 128 by copying it 4 times mess with autograd? For now I am just padding the input but I was wondering if there was a more efficient way</p>

username: albanD
cooked: <p>Hi,</p>
<p>From the autograd point of view, anything that you do (as long as it is differentiable) will be fine.</p>
<p>It then depends on your application what you want to do.<br>
I guess the 3 options I can see are interpolate to scale up (or down) one to fit the size of the other. pad or crop. And replicate as you mentioned.<br>
I think it will depend a lot on your application which one will work best so you want to test them and see how it goes.</p>

username: MiloKnell
cooked: <p>I’m sorry if I wasn’t clear enough. I’m using an encoder/decoder and I’m performing triplet loss on the output of the encoder. The anchor is differently sized than the positive and negative. Currently I am padding the anchor before it’s encoded, but that wastes memory and I’m very tight on my gpu memory. So I’m wondering if I can pad the encoder output or copy the existing 128 tokens 4 times to create a 512 input for the loss. So will replication or padding of the encoder output make autograd unable to accurately update weights in my encoder?</p>

username: albanD
cooked: <p>Both replication and padding are differentiable so no issue on the autograd side <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: MiloKnell
cooked: <p>Awesome, thank you so much</p>
Is it possible to only store the total gradient of a custom module

username: ProkaryMonster
cooked: <p>I’ve pretrained a CNN module, which I want to insert to object detection deep neural networks.<br>
However, the CNN module I’ve designed will first largely increase the number of feature channels, and then decrease it as the input number of feature channels, which may need to store large feature maps during the forward process or large gradient maps during the backward process and occupies too much CUDA memory, while I just need the total gradient of the module wrt the input feature map.<br>
Since I don’t need the module parameters to be  updated, I’ve set</p>
<pre><code class="lang-auto">for name, p in ce.module.named_parameters():
    p.requires_grad = False
</code></pre>
<p>But as it shows in <a href="https://discuss.pytorch.org/t/will-freezing-an-intermediate-block-influence-the-gradient-flow/88859">my previous post</a>, to ensure the module is differentiable, pytorch just doesn’t calculate the gradients wrt layer weights and the data size it stores might not be changed.<br>
So I’m wondering if it is possible to multiply the gradient of module layer by layer during the forward process and during the backward process we just need to multiply it with the rest parts’ gradients,  instead of saving inputs, outputs, and gradients per layer.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/251699b882e2f4e343e8b9889d3ddc94afcfdb27.png" data-download-href="https://discuss.pytorch.org/uploads/default/251699b882e2f4e343e8b9889d3ddc94afcfdb27" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251699b882e2f4e343e8b9889d3ddc94afcfdb27_2_517x30.png" alt="image" data-base62-sha1="5i65r5TLHIVbKcZ8joYbtSo72MT" width="517" height="30" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251699b882e2f4e343e8b9889d3ddc94afcfdb27_2_517x30.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251699b882e2f4e343e8b9889d3ddc94afcfdb27_2_775x45.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/5/251699b882e2f4e343e8b9889d3ddc94afcfdb27.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">780×46 4.91 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is it possible to do such a thing in pytorch or is there a more elegant way to do it?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The way to do this is to implement a new custom Function (instructions here <a href="https://pytorch.org/docs/stable/notes/extending.html">https://pytorch.org/docs/stable/notes/extending.html</a>) that will encapsulate both these ops.<br>
You will then have the control over what is saved for the backward and you will need to implement the backward yourself to match this new computation that you want to do.</p>
Grad is None in some case

username: 7rick03ligh7
cooked: <p>What is the difference? Why is tensor “a” in the first case not a leaf, but in the second case it is a leaf?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd1cbcba1a5edd024c7c236f7ab198f963761629.png" data-download-href="https://discuss.pytorch.org/uploads/default/fd1cbcba1a5edd024c7c236f7ab198f963761629" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd1cbcba1a5edd024c7c236f7ab198f963761629.png" alt="image" data-base62-sha1="A78mThYLL8aK4MJxRz17F65EDz3" width="690" height="494" data-dominant-color="F1EBEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">856×613 30.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mariosasko
cooked: <p>In the fist example, the <code>reshape</code> op is applied on the leaf node <code>torch.arange(60, dtype=torch.float32, requires_grad=True)</code>, so <code>a</code> is no longer a leaf node (instead it represents the <code>reshape</code> op).</p>
<p>In the second example, changing the flag <code>requires_grad</code> of the node <code>a</code> doesn’t add an additional op on top it (<code>a</code> is still a leaf) so after the <code>backward</code> call, <code>grad</code> is properly assigned.</p>
<p>Note: to get a grad of the non-leaf node, use <code>register_hook</code>.</p>

username: albanD
cooked: <aside class="quote no-group" data-username="mariosasko" data-post="2" data-topic="92171">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosasko/48/27520_2.png" class="avatar"> mariosasko:</div>
<blockquote>
<p>Note: to get a grad of the non-leaf node, use <code>register_hook</code> .</p>
</blockquote>
</aside>
<p>Note that you don’t need to have a full hook. You can just call <code>a.retain_grad()</code> and the <code>.grad</code> field will be populated automatically when you call <code>.backward()</code>.It is very similar to the hook, just slightly simpler and you don’t have to define the hook yourself.</p>
Multi_head_attention_forward() and batch dimension index

username: ychnh
cooked: <p>In multi_head_attention_forward under torch.functional, there is a check to make sure the batch is the second index in the tensor. However, it calls linear() proceeding this, which requires the batch to be the first index of the input tensor. This does not seem to be correct from my understanding, please correct me if I am wrong. Thank you.</p>
<blockquote>
<p>def multi_head_attention_forward(…):<br>
assert list(query.size()) == [tgt_len, bsz, embed_dim]<br>
…<br>
q = linear(query, q_proj_weight_non_opt, in_proj_bias[0:embed_dim])</p>
</blockquote>

username: albanD
cooked: <p>Hi,</p>
<p>Actually, <code>Linear</code> consider anything that is not the last dimension as batch. So this is not an issue here.</p>
Difference between TORCH_LIBRARY() and RegisterOperators::op()

username: Ssapphire
cooked: <p>I have been studying pytorch’s custom op registration. I noticed that there are these two classes used for registration. I am wondering why there isn’t a README for using TORCH_LIBRARY(), and what is the difference betweening using TORCH_LIBRARY(my_ops,m) and static auto registry = torch::RegisterOperators()<br>
.op(“my_ops::myrelu”,  torch::RegisterOperators::options()<br>
.kernel&lt;decltype(myrelu), &amp;myrelu&gt;(c10::DispatchKey::CPU));</p>

username: ezyang
cooked: <p>Check out <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html">https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html</a> and <a href="https://pytorch.org/tutorials/advanced/dispatcher.html">https://pytorch.org/tutorials/advanced/dispatcher.html</a></p>
<p>Reference C++ docs at <a href="https://pytorch.org/cppdocs/api/file_torch_library.h.html">https://pytorch.org/cppdocs/api/file_torch_library.h.html</a></p>
<p>The APIs go into the same internal machinery, but the TORCH_LIBRARY API is the newer one and you should use it instead.</p>
Back propagation through Dataloader

username: tflite
cooked: <p>I was wondering if you put an embeddings Neural network in your dataloader without freezing weights, will back-propagation modify the weights of that embeddings network?</p>

username: mariosasko
cooked: <p>As long as these embeddings are part of the computation graph on which <code>backward</code> was called, they will get their gradient assigned (check <code>embeddings.weight.grad</code> after the <code>backward</code> call). To modify these weights, either you have to manually update them or you can pass them to the optimizer that will do the update for you (<code>embeddings.weight</code> is an instance of <code>torch.nn.Parameter</code> so <code>model.parameters()</code> returns them as well)</p>

username: albanD
cooked: <p>Note though that the Dataloader uses multiprocessing for loading when you use &gt;0 workers. And autograd won’t track across these.<br>
So in general, this might be tricky to get working if you use multiple workers for loading.</p>
Conv1D much slower than linear layers on CPU?

username: urkatz
cooked: <p>Hi everyone,</p>
<p>I had a simple autoencoder implemented with fully-connected linear layers and I tried switching them to Conv1d layers. The number of parameters went down by two orders of magnitude, but the compute time on the CPU went up by more than 10x. Anyone know why this would be the case?  Why are conv layers so slow, at least on the CPU? Am I doing this correctly?</p>
Where to get a professional to help with my notebook?

username: kristerv
cooked: <p>I have tried all the tricks I know on my fairly simple classification problem and it still always predicts 1 (there are only two classes). Perhaps my data is not good enough, but I want to rule out stupid mistakes. I’m out of questions and I don’t expect anyone to read my code (although <a href="https://www.kaggle.com/kristerv/subtitle-timing-sharpener" rel="nofollow noopener">here it is on Kaggle</a>).</p>
<p>So where/who can I turn to to hire someone to read my notebook?</p>

username: ptrblck
cooked: <p>You could create a post in the “Jobs” category here with your search for an engineer.</p>
<p>That being said, I would generally recommend to run a quick test by overfitting the model to a small dataset (e.g. just 10 samples) and make sure your model is able to perfectly fit it.<br>
If that’s not the case, you can quickly iterate and check the code for any issues.</p>
<p>Just by skimming through the code it seems that your model is not using any activation functions:</p>
<pre><code class="lang-python">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(201, 400),
            nn.Linear(400, 200),
            nn.Linear(200, 50),
            nn.Linear(50, 2)
        )

    def forward(self, x):
        return self.model(x)
</code></pre>
<p>and can thus be seen as a single linear layer.</p>

username: kristerv
cooked: <p>thanks for checkout it out.</p>
<p>I have tried activation functions, but it doesn’t help.</p>
<p>Overfitting a small dataset works. Does this mean the problem is data?</p>

username: ptrblck
cooked: <p>This would at least mean that the current training code seems to properly update the model.<br>
Once you were able to overfit the data, you could scale up the problem slowly (i.e. add more data stepwise) and tune the hyperparameters of your model with it.</p>

username: kristerv
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="92195">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>tune the hyperparameters</p>
</blockquote>
</aside>
<p>If I understand correctly this just means learning rate, loss function, layers and so on? If so, this is what I’m doing. Maybe I’ll get to a result eventually.</p>
Is it possible to vectorize training process?

username: yanzhou01
cooked: <p>I have 50 respondents x 120 items and ratings given by them. I train my model with 3 for-loops:</p>
<pre><code class="lang-auto">for each epoch:
 for each respondent of 50
   for each item of 120
      # the label is rating of respondent i item j.
      # the label dimension would be [1, 1]
</code></pre>
<p>I was wondering if I can train the model like the following way:</p>
<pre><code class="lang-auto">for each epoch:
  for each item of 120:
   # the labels are ratings of 50 respondents item j.
   # the label dimension would be [50, 1]
</code></pre>
<p>Can pytorch realize this and optimize the neural network model? However, in this case, I am not sure how the model parameters will be updated.</p>

username: ptrblck
cooked: <p>It depends on your overall workflow and how the data is handled at the moment.<br>
Generally, PyTorch modules accept a batched input (usually the batch dimension is in <code>dim0</code>).<br>
If the “50 respondents” would be seen as a batch of 50 observations, you should be able to directly pass this input tensor to the model and calculate the loss.<br>
By default the criterion will calculate the loss as the average of the sample losses.</p>
GAN's training is not working, loss value is unchanged

username: costa98
cooked: <p>I have set the classes and optimizers as such:</p>
<pre><code>def _ones_target(size):
    return torch.ones(size, 1)

def _zeros_target(size):
    return torch.zeros(size, 1)

disS_optim = optim.Adam(ds.parameters(),lr=0.0005)
disT_optim = optim.Adam(dt.parameters(),lr=0.0005)
gen_optim = optim.Adam(g.parameters(),lr=0.0001)
loss = nn.BCELoss()
g = GeneratorNet()
ds = SpatialDiscriminatorNet()
dt = TemporalDiscriminatorNet()
</code></pre>
<p>And the training is:</p>
<pre><code>lossDs_values = []
lossG_values = []
lossDt_values = []
num_epochs = 3
batch_size = 16
for epoch in trange(num_epochs):
    for i, (video, label) in enumerate(tqdm(data_loader)):
        #video shape ([batch_size, n_frames, 3, 64, 64])
       

        # 1) train discriminators
        disS_optim.zero_grad()
        disT_optim.zero_grad()

        real_data = video.permute(1,0,2,3,4)
        fake_data = g(_generator_noise_input(bs=batch_size, seqLen=n_frames)).detach()
        fake_data2 = g(_generator_noise_input(bs=batch_size, seqLen=n_frames)).detach()

        prediction_real_Ds = ds(real_data)
        error_real_Ds = loss(prediction_real_Ds, _ones_target(prediction_real_Ds.shape[0]))
        
        prediction_real_Dt = dt(real_data)
        error_real_Dt = loss(prediction_real_Dt, _ones_target(prediction_real_Dt.shape[0]))

        prediction_fake_Ds = ds(fake_data)
        error_fake_Ds = loss(prediction_fake_Ds, _zeros_target(prediction_fake_Ds.shape[0]))
        
        prediction_fake_Dt = dt(fake_data)
        error_fake_Dt = loss(prediction_fake_Dt, _zeros_target(prediction_fake_Dt.shape[0]))
        
        error_Ds = error_real_Ds + error_fake_Ds
        error_Dt = error_real_Dt + error_fake_Dt
        print(error_Ds, error_Dt)
 
        lossDs_values.append(error_Ds)
        lossDt_values.append(error_Dt)
        error_Ds.backward()
        error_Dt.backward()

        disS_optim.step()
        disT_optim.step()


        # 2) train generator
        gen_optim.zero_grad()

        prediction_G_Ds = ds(fake_data2)
        prediction_G_Dt = dt(fake_data2)
        
        error_G_Ds = loss(prediction_G_Ds, _ones_target(prediction_G_Ds.shape[0]))
        error_G_Dt = loss(prediction_G_Dt, _ones_target(prediction_G_Dt.shape[0]))
        error_G = error_G_Ds + error_G_Dt
        lossG_values.append(error_G)
        error_G.backward()
        print(error_G)
        
        gen_optim.step()
        
plt.plot(lossG_values,label='G error')
plt.plot(lossDs_values,label='Ds error')
plt.plot(lossDt_values,label='Dt error')
plt.legend(['G error','Ds error','Dt error'])
plt.show()
</code></pre>
<p>The problem is that the values of the graph remain basically still, or they change very slightly<br>
<a href="https://i.stack.imgur.com/C2oPV.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/C2oPV.png" alt="enter image description here" width="425" height="255"></a></p>

username: user_123454321
cooked: <p>Why is fake_data2 detached ? It is used to train generator right ?</p>

username: costa98
cooked: <p>yes, fake_data2 is for the generator</p>
Guys, what's wrong in this? Pl. Check

username: Blizzard_boi2020
cooked: <p>Here- is the directory(+= folder contents)</p>
<p>D:/Data/…/Train/</p>
<p>PX1_0000- T2W.nii.gz + seg.nii.gz</p>
<p>PX1_0001- T2W.nii.gz+ seg.nii.gz<br>
.<br>
.<br>
… PX1_0200- T2W.nii.gz+ seg.nii</p>
<p>Same follows for 200 patients.</p>
<p>def get_list_of_files(base_dir):<br>
list_of_lists = []<br>
for phase_type in [‘train’]:<br>
current_directory = join(base_dir, phase_type)<br>
patients = subfolders(current_directory, join=True)<br>
for p in patients:<br>
patient_directory = join(current_directory, p)<br>
t2_file = join(patient_directory, p + “T2W.nii.gz”)<br>
seg_file = join(patient_directory, p + “seg.nii.gz”)<br>
this_case = [t2_file, seg_file]<br>
assert all((isfile(i) for i in this_case)), "some file is missing for patient %s; make sure the following "<br>
“files are there: %s” % (p, str(this_case))<br>
list_of_lists.append(this_case)<br>
print(“Found %d patients” % len(list_of_lists))<br>
return list_of_lists</p>
<p>CAN YOU PLEASE HELP ME IN GETTING?</p>

username: mathematics
cooked: <p>Proper formatting might be great to review problems for others.<br>
Please explain problem briefly.</p>

username: Blizzard_boi2020
cooked: <p>Hi. Thanks!<br>
My directory structure is like the above… follows for 200 folders.</p>
<p>I want to read in the contents of the file, but it’s not working. Where’s it going wrong?</p>

username: ptrblck
cooked: <p>Code snippets can be added by wrapping them into three backticks ```, which makes debugging easier.<br>
Are you seeing an error or unexpected results when running the script?<br>
This questions doesn’t seem to be PyTorch-specific, so you might also get a better and faster answer e.g. on StackOverflow.</p>
How to fix my learning rate

username: gamebo5000
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3abe9320478d65bb7640efc18189b5c960187395.png" alt="learning rate" data-base62-sha1="8nFZRf4htnzsZHPNOwBQK3AreQZ" width="580" height="362"></p>
<p>I am using cosine annealing scheduler, but it is adjusting and such a small rate.</p>
<p>This is how it is initialized:</p>
<p>lrs = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max = len(train_loader))</p>
<p>and this is how it is being called:</p>
<pre><code class="lang-auto">`for i in range(epoch):`

`trn_corr = 0`

`tst_corr = 0`

`#lrs.step()`

`#adjust the learning rate after 30 epochs`

`#adjust_learning_rate(optimizer, i, learning_rate)`

`#Run training for one epoch`

`train(train_loader, MobileNet, criterion, optimizer, i,trn_corr)`

`#evaluate the validation/test`

`test(test_loader, MobileNet, criterion, i, epoch,tst_corr)`

`lrs.step()`
</code></pre>
<p>inside the training loop the optimizer is being adjusted like so:</p>
<pre><code class="lang-auto">`# Update parameters`

`optimizer.zero_grad()`

`loss.backward()`

`optimizer.step()`
</code></pre>

username: ptrblck
cooked: <p>You can call <code>lrs.step()</code> for each iteration (or after a certain number of itertations) instead of once per epoch.</p>

username: gamebo5000
cooked: <p>How would the code look if I did it for each iteration?</p>

username: ptrblck
cooked: <p>You would have to call <code>lrs.step()</code> inside the <code>DataLoader</code> loop, which is most likely used inside your <code>train</code> method.<br>
In my <a href="https://discuss.pytorch.org/t/learning-rate-not-adjusting-properly/91650/2">other post</a> the <code>range(nb_batches)</code> loop would correspond to the <code>DataLoader</code> loop.</p>
Training time difference in transfer learning

username: DDavid
cooked: <p>Let’s say <em>I</em> is a domain of images and f : <em>I</em> -&gt; <em>E</em> and g : <em>E</em> -&gt; <em>F</em> where <em>E</em> and <em>F</em> are embedding spaces.</p>
<p>There are two cases</p>
<ol>
<li>freeze f</li>
<li>unfreeze f</li>
</ol>
<p>Question is : when training g(f(<em>I</em>)), I expected that training time of the case 1 is more faster than that of the case 2. But from my experience, the training time difference is around 1:1.5 even though the number of parameters of f is way larger(100x) than g. I wanna save the time when freezing f. Any suggestion?</p>
<p>If there’s a way to save f(A) for all A in <em>I</em>, and I can only forward/backward propagate them, please let me know. Thanks in advance.</p>

username: Kushaj
cooked: <p>I don’t think parameter count is a good metric for how fast a model should run. You can benchmark both the cases independently and see if the result what you expected.</p>
<p>Take a batch, run through f (time it few times)<br>
Do it for g, and you will get a rough estimate of the time taken for forward, backward cases.</p>
<aside class="quote no-group" data-username="DDavid" data-post="1" data-topic="91284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ddavid/48/25459_2.png" class="avatar"> DDavid:</div>
<blockquote>
<p>If there’s a way to save f(A) for all A in <em>I</em> , and I can only forward/backward propagate them, please let me know.</p>
</blockquote>
</aside>
<p>You can store the results as files and then have the dataloader load these instead of the images for <code>g</code>.</p>

username: DDavid
cooked: <p>Hi Kushaj. Thanks for paying attention to my question.</p>
<p>usually the output of <code>f(A)</code> are really big like (number of images) x channels x height x width = 50000 x 256 x 128 x 128. So saving these all images is not a good idea.</p>
<p>I agree that the parameter count is not a good metric. But the training time matters.</p>
How to Improving model's performance

username: gamebo5000
cooked: <p>I am using a pre-trained mobilenetv2 model running experiments for 30 epochs with a cosine annealing learning rate scheduler and these hyperparameters:</p>
<p>learning rate = 0.001<br>
weight decay = 4e-5<br>
momentum = 0.9<br>
batch size = 64<br>
optimizer = SGD<br>
dropout = 0.2</p>
<p><strong>My data augmentation techniques include:</strong></p>
<pre><code class="lang-auto">train_transform = transforms.Compose([
            transforms.RandomResizedCrop(224,scale = (0.2,1.0)),
            #transforms.RandomRotation(15),      # rotate +/- 10 degrees
            transforms.RandomHorizontalFlip(),  # reverse 50% of images        
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406],
                                 [0.229, 0.224, 0.225])
        ])
</code></pre>
<pre><code>test_transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])
</code></pre>
<p>I am using class_weights combined with a data sampler( see here: <a href="https://github.com/ufoym/imbalanced-dataset-sampler" rel="nofollow noopener">https://github.com/ufoym/imbalanced-dataset-sampler</a> ) since my dataset is imbalanced and the formula for determining each weight is minority class size/class size. I am getting good results as my confusion report is showing that each class is generating accuracies in the 80s and 90s. How can I push my model’s performance to be in the high 90s?.</p>
<p>What I have tried so far:</p>
<p>Adjusting the class weights</p>
<p>Changing the initial learning rate</p>
Conflict when using Tensorboard with matplotlib

username: wall-e
cooked: <p>Hi everyone, My work runs on Window. I used anaconda to create an environment. I also install Pyqt5. But when I want to use Tensorboard to visualize the training process, I got errors:</p>
<ul>
<li>if I use matplotlib to show any image result, I got an error:</li>
</ul>
<pre><code class="lang-auto">Cannot mix incompatible Qt library (5.9.7) with this library (5.15.0)
</code></pre>
<ul>
<li>If I only use tensorboard, no problem occur.</li>
<li>If I uninstall tensorboard, I can show my image results.</li>
</ul>
<p>Could anyone help me to solve it?<br>
Thank you</p>

username: ptrblck
cooked: <p>Based on the error message I guess you might have some library conflicts with <code>PyQt</code>, i.e. <code>matplotlib</code> seems to need another version than the installed one.<br>
You could try to update <code>matplotlib</code> (and hope it can use the newer <code>PyQt</code> version) or switch to another <a href="https://matplotlib.org/faq/usage_faq.html#what-is-a-backend">backend</a> in <code>matplotlib</code>.</p>

username: wall-e
cooked: <p>Thank you for your reply.<br>
I have tried to update <code>matplotlib</code>, but it cannot solve the problem: when I add:</p>
<blockquote>
<p>from torch.utils.tensorboard import SummaryWriter<br>
writer = SummaryWriter(‘./runs/Classification’)<br>
writer.add_scalar(“Loss/Train”,loss,epoch)</p>
</blockquote>
<p>and using matplotlib to display any graph, it will get error:</p>
<blockquote>
<p>Cannot mix incompatible Qt library (5.9.7) with this library (5.15.0)</p>
</blockquote>
<p>I want to try uninstall Qt library 5.9.7, but I don’t know what command line can help me to do that (with pip or conda)?</p>

username: ptrblck
cooked: <p><code>pip/conda uninstall ...</code> should work and you would have to use the package manager, which was used to install it in the first place.</p>