akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Iterating over dataloader: not enough values to unpack

username: Merwa_Mehmood
cooked: <p>loop = tqdm(loader[0], leave=True)</p>
<pre><code>for batch_idx, (real,_) in enumerate(loop):

    real = real.to(DEVICE)

    cur_batch_size = real.shape[0]
</code></pre>
<h3>
<a name="when-i-iterate-over-dataloader-the-output-is-1" class="anchor" href="#when-i-iterate-over-dataloader-the-output-is-1"></a>When i iterate over dataloader the output is</h3>
<h2>
<a name="my-total-items-are-49190-2" class="anchor" href="#my-total-items-are-49190-2"></a>My total items are 49190</h2>
<p>loader =  DataLoader(</p>
<pre><code>    tupled_dataset_with_label,

    batch_size=64,

    shuffle=True,

    num_workers=NUM_WORKERS,

    pin_memory=True

) 
</code></pre>
<p>it = iter(loader)</p>
<p>first = next(it)</p>
<p>second = next(it)</p>
<p>print(first.shape)</p>
<p>print(second.shape)</p>
<p>Output:<br>
torch.Size([64, 1, 64, 64])<br>
torch.Size([64, 1, 64, 64])</p>

username: eqy
cooked: <p>If <code>iter(loader)</code> only gives you a single tensor back, can you see if only unpacking a single variable fixes this issue? (e.g., <code>for batch_idx, real in enumerate(loop):</code>)</p>

username: evanbung
cooked: <p>During a multiple value assignment, the ValueError: not enough values to unpack occurs when either you have fewer objects to assign than variables, or you have more variables than objects. This error caused by the mismatch between the number of values returned and the number of variables in the assignment statement. This error happened mostly in the case of using <a href="http://net-informations.com/python/file/split.htm" rel="noopener nofollow ugc">python split</a>  function. Verify the assignment variables. If the number of assignment variables is greater than the total number of variables, delete the excess variable from the assignment operator. The number of objects returned, as well as the number of variables available are the same.</p>
<p>To see what line is causing the issue, you could add some debug statements like this:</p>
<pre><code class="lang-auto">if len(line.split()) != "xx":
    print line
</code></pre>
<p>This will resolve the value error.</p>
Out of XX is a view and its base or another view of its base has been modified inplace

username: Einsturing
cooked: <p>When I change my model to DataParallel, there are some quesitons like this:</p>
<p>RuntimeError: Output 186 of BroadcastBackward is a view and its base or another view of its base has been modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.</p>
<p>How should I deal with this problem？</p>
Problem with BI-LSTM CRF model for Punctuation restoration

username: dlindvai
cooked: <p>Hello everyone,</p>
<p>I changed the code in this <a href="https://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html" rel="noopener nofollow ugc">tutorial</a> so it would work for Punctuation restoration (only Periods and Commas for now) instead of NER.</p>
<p>The issue is: After the training, I get good results (Precision, Recall and F1-score are all nearly 1), what should mean that the model is trained well. But after the testing, I get 0.00’s in all 3 metrics. I trained the model with datasets of various sizes (from 2000 lines to 3000 lines)</p>
<p>I would be very happy if you could give me any advice on what should should I change. I wonder if I am even testing the model the right way or if I am computing the metrics right. thank you very much in advance.</p>
<pre><code class="lang-python">EMBEDDING_DIM = 5
HIDDEN_DIM = 4

training_data = [( train.split() , tags_train.split() )]
testing_data = [( test.split() , tags_test.split() )]

word_to_ix = {}
for sentence, tags in training_data:
        for word in sentence:
                if word not in word_to_ix:
                        word_to_ix[word] = len(word_to_ix)

tag_to_ix = {"W": 0, "P": 1, "C": 2, START_TAG: 3, STOP_TAG: 4}
model = BiLSTM_CRF(len(word_to_ix), tag_to_ix, EMBEDDING_DIM, HIDDEN_DIM)
optimizer = optim.SGD(model.parameters(), lr=0.02, weight_decay=0.1)

for epoch in range(20):
  for sentence, tags in training_data:
    model.zero_grad()
    sentence_in = prepare_sequence(sentence, word_to_ix)
    targets = torch.tensor([tag_to_ix[t] for t in tags], dtype=torch.long)
    loss = model.neg_log_likelihood(sentence_in, targets)
    loss.backward()
    optimizer.step()

word_to_ix = {}
for sentence, tags in testing_data:
	for word in sentence:
		if word not in word_to_ix:
			word_to_ix[word] = len(word_to_ix)

for sentence, tags in testing_data:
	test_targets = torch.tensor([tag_to_ix[t] for t in tags], dtype=torch.long)
with torch.no_grad():
	test_precheck_sent = prepare_sequence(testing_data[0][0], word_to_ix)

var = model(test_precheck_sent)
y_true = np.array(test_targets)
y_pred = np.array(var[1])
print(metrics.confusion_matrix(y_true, y_pred), "\n")
print(metrics.classification_report(y_true, y_pred, digits=3))
</code></pre>
<p>PS: I do not have much experience in both python and pytorch so that is why my code is quite messy and I know it could be writted much more effective, but for now I just want the code to work somehow.</p>

username: Abhilash_Srivastava
cooked: <p>You should have just one <code>word_to_ix</code>, shared by both train and test data. <code>word_to_ix</code> should not be modified.</p>

username: dlindvai
cooked: <p>Thank you very much, it helped!</p>

username: NAVEEN_SINGH
cooked: <p>Can you please explain how to use this for a text file in which we have to do punctuation restoration</p>
Model.forward() automatically converts complex128 [a+bj] to stacked complex form [a,b]

username: C_J
cooked: <p>Just before the return, the output format is complex128,<br>
but in the forloop during the training the output is automatically converted to stacked complex form. Just as below:</p>
<pre><code class="lang-auto">    def forward(self, input):
        output=custom_function(input)
        print(output)
        return output
</code></pre>
<p>result: a+bj</p>
<pre><code class="lang-auto">----main----
output=model.forward(input)
print(output)
</code></pre>
<p>result: [a, b]</p>
<p>I don’t understand how this is happening and how to fix it.</p>
<p>Can I get some advice?<br>
Thanks for reading!</p>

username: C_J
cooked: <p>FYI, I have wrapped the model with Dataparallel.<br>
Maybe this causes the issue?</p>
<pre><code class="lang-auto">class DataParallelPassthrough(torch.nn.DataParallel):
    def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)


model=DataParallelPassthrough(model)
model.cuda()
</code></pre>
Clear tensorboard data log

username: Ivanita
cooked: <p>Hi. I am pretty new to programming. May I ask if there is any way to clear the previous log in tensorboard?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d7192c32a63526ee9fa07a04e2c502d3b7d2f06.png" alt="image" data-base62-sha1="hTJ0PJBuW7P6amTUTab9Fckw7Xw" width="328" height="196"></p>
<p>Currently the previous log still appears every time I run. Thanks.</p>

username: ptrblck
cooked: <p>If I’m not mistaken, you would have to delete the log files from the tensorboard logging folder or create a new folder for the current logs.</p>
Accessing non-parameter module member of torchscript model in C++

username: Amit_Gupta
cooked: <p>Suppose I have a simple torch model, and add an additional class member for saving some information, as shown below (<code>extra_info</code>).</p>
<pre><code class="lang-python">import torch

class model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(1, 1)

    def forward(self, x):
        return self.linear(x)


m = model()
m.extra_info = "ExtraString"
m_ts = torch.jit.script(m)
m_ts.save("m.pt")
exit()
</code></pre>
<p>Now when I reload the Torchscript module in python I can access the saved string easily, indicating that it has indeed been saved with the Torchscript model:</p>
<pre><code class="lang-python">import torch
m = torch.jit.load("m.pt")
m.extra_info
#&gt; 'ExtraString'
</code></pre>
<p>But when I try to access it in C++ API, I obviously get error of no such member. How can I access <code>extra_info</code> field in C++?</p>
<p>I can see that the value is saved, and print it as:</p>
<pre><code class="lang-cpp">for (auto p: model.attributes())
        {cout &lt;&lt; p &lt;&lt; endl;}
</code></pre>
<p>which prints the attributes as</p>
<pre><code class="lang-auto">True
None
ExtraString
&lt;__torch__.torch.nn.modules.linear.Linear object at 0x5608d053eb20&gt;
 0.9345
[ CPUFloatType{1,1} ]
0.01 *
</code></pre>
<p>But I cant figure out hot to fetch it specifically.</p>

username: Amit_Gupta
cooked: <p>Figured it out</p>
<pre><code class="lang-cpp">        for (auto p: model.named_attributes())
        {
            if(p.name == "extra_info"){
                cout&lt;&lt;p.value&lt;&lt;endl;
            }
        }
</code></pre>
<p>This yields required output. If anyone has better way to do it, I am all ears.</p>
Imbalanced multi class classification

username: Gorgen
cooked: <p>hello everyone:</p>
<p>my task is emotion recognition. due to unbalance, try to use focal loss to solve it.<br>
Here is coding as follows:<br>
is that true? Thanks, best wishes<br>
class FocalLoss(nn.Module):</p>
<pre><code>def __init__(self, weight=None, alpha=None,
             gamma=2., reduction='none'):
    nn.Module.__init__(self)
    self.weight = weight
    #self.alpha = torch.tensor([alpha,1-alpha]).cuda()
    self.alpha = torch.tensor([4.0/47, 15.0/47, 15.0/47, 3.0/47, 1.0/47, 6.0/47, 3.0/47]).cuda()
    self.gamma = gamma
    self.reduction = reduction
    
def forward(self, input_tensor, target_tensor):
    
    at = self.alpha
    log_prob = F.log_softmax(input_tensor, dim=-1)
    prob = torch.exp(log_prob)
    loss = F.nll_loss(((1 - prob) ** self.gamma) * log_prob*at, target_tensor, weight=self.weight, reduction = self.reduction)

    #self.reduction == 'sum'
    loss = loss.sum()
    return loss</code></pre>
[CTC Loss] CTC Loss not support float16?

username: 111114
cooked: <p>Hello. I’m trying to train ASR model by CTC loss.<br>
But when I apply mixed precision training, CTC Loss does not descend and model predicts only Blank for some Epochs in spite of using wav2vec2 pretrained model.</p>
<p>I’m not sure which part disturbs training, but I think covering optimizer and backward by scaler is the critical one.<br>
How should I fix the code to train CTC loss for mixed precision training? Or is there any other problem in my code?</p>
<p>CTC loss used like below. I copied huggingface’s usage.</p>
<pre><code class="lang-auto">log_probs = F.log_softmax(logits, dim=-1, dtype=torch.float32).transpose(0, 1)

with torch.backends.cudnn.flags(enabled=False):
    loss = nn.functional.ctc_loss(
        log_probs,
        flattened_targets,
        feature_lengths,
        target_lengths,
        blank=self.config.blank_idx,
        reduction=self.config.ctc_loss_reduction,
        zero_infinity=self.config.ctc_zero_infinity,
    )
</code></pre>
<p>And I covered forward and backward by scaler</p>
<pre><code class="lang-auto">from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler(enabled=config.fp16_run)

with autocast(enabled=config.fp16_run):

  predictions = model(inputs)

scaler.scale(predictions['loss']).backward()
scaler.step(optimizer)
scaler.update()
optimizer.zero_grad()

</code></pre>

username: ptrblck
cooked: <p>The <code>log_probs</code> tensor should be created by <code>F.log_softmax</code> which will use <code>float32</code> for the needed numerical stability. <code>ctc_loss</code> should then work on these <code>float32</code> input tensors and not use <code>float16</code>. Could you check the <code>dtype</code> of <code>log_probs</code> and verify it?</p>

username: tom
cooked: <p>Typically, CTC loss is one of the functions that are more demanding wrt numerical accuracy. This shows for fp32 at times and I imagine fp16 to not be a terribly good option for CTC.<br>
Also, it would be nontrivial to get an increase in compute performance.</p>
<p>Best regards</p>
<p>Thomas</p>
Getting losses while predicting inputs

username: ori
cooked: <p>Hi, i am trying to get loss for every input prediction of model. So i have a model after training i am pass inputs to prediction.</p>
<pre><code class="lang-auto">output = model(inputs)
</code></pre>
<p>to get loss, i try</p>
<pre><code class="lang-auto">loss = output.loss
</code></pre>
<p>This does not return any value, but it return generator type. Is there any way to convert generator and get value out of this.</p>

username: ptrblck
cooked: <p>You might be able to get the values by passing the <code>generator</code> to e.g. a <code>list</code>, but it usually depends on your use case. I.e. how is the loss calculated, why is it a <code>generator</code> etc.</p>

username: ori
cooked: <p>I didn’t get how to pass as list, or is it like this list(loss)</p>

username: ptrblck
cooked: <p>Yes, you could try to use <code>list(loss)</code>, but I would also strongly recommend to check why it’s a generator as it seems you don’t expect it:</p>
<pre><code class="lang-python">loss = (torch.randn(1) for _ in range(10))
print(type(loss))
# &lt;class 'generator'&gt;

print(list(loss))
# [tensor([-1.6251]), tensor([-2.1271]), tensor([0.6758]), tensor([0.5699]), tensor([0.1286]), tensor([-1.4132]), tensor([0.1561]), tensor([1.1689]), tensor([-0.4857]), tensor([-1.3968])]
</code></pre>

username: ori
cooked: <p>I tried compute, list(loss) and its {list:0} [] . I am not sure how its emtpy. Any ideas to debug?</p>

username: ori
cooked: <p>I am computing distillation process as in this link <a href="https://github.com/subhasisj/Model-Compression-Techniques/blob/main/01-Modelling-KD.ipynb" rel="noopener nofollow ugc">https://github.com/subhasisj/Model-Compression-Techniques/blob/main/01-Modelling-KD.ipynb</a></p>
<p>Cell 75 can give more idea about the process.</p>
Train and test with the same data

username: nafise
cooked: <p>I trained Resnet18 pre-trained with a 2-class data.<br>
And I got a good accuracy , but to evaluate my model, I also used the data I used for training to test., but unfortunately the accuracy went down. I do not know where the problem comes from. While the data are the same for testing and training.<br>
please guide me.</p>

username: InnovArul
cooked: <p>You may need to provide more information with the post such as what’s the training accuracy, test accuracy, the method used to test, etc.<br>
An obvious answer would be, that the model did not learn anything over the training.<br>
However, if you are sure that the model learns during training, it might be an issue with the testing code?</p>

username: Andrei_Cristea
cooked: <p>Just to add one point, make sure you’re putting your module in .eval() mode before testing. Otherwise, you could see weird results.</p>
Hip device says only has 4096 bytes global memory, but pytorch wants to allocate 2097152 bytes. Why?

username: hughperkins
cooked: <p>Hi, I’m designing a GPU in verilog, <a href="https://github.com/hughperkins/VeriGPU" class="inline-onebox" rel="noopener nofollow ugc">GitHub - hughperkins/VeriGPU: OpenSource GPU, in Verilog, loosely based on RISC-V ISA</a>, and I started dabbling in running it, in simulation, from pytorch, using the HIP API.</p>
<p>I’ve started to therefore implement the HIP API, <a href="https://github.com/hughperkins/VeriGPU/blob/f4279efa308638e7218dd7865d987adaf6c2cc77/src/gpu_runtime/hip_api.cpp#L16-L67" class="inline-onebox" rel="noopener nofollow ugc">VeriGPU/hip_api.cpp at f4279efa308638e7218dd7865d987adaf6c2cc77 · hughperkins/VeriGPU · GitHub</a> , and when I run a single pytorch script like:</p>
<pre><code>import torch

a = torch.rand(3)
a.cuda()
</code></pre>
<p>And then the output when I run with my api so far looks like:</p>
<pre><code>hipInit
hipGetDeviceCount
hipGetDeviceCount
hipGetDeviceProperties
hipGetDeviceCount
hipGetDevice
hipGetDevice
hipGetDevice
hipGetDevice
hipGetLastError
hipMalloc size=2097152
hipSetDevice
hipSetDevice
Traceback (most recent call last):
  File "/home/ubuntu/git/verigpu/verigpu/test_hip.py", line 4, in &lt;module&gt;
    a.cuda()
RuntimeError: Not enough free space
</code></pre>
<p>So, pytorch is trying to allocate 2MB of memory, even though in hipGetDeviceProperties, I’m clearly stating my device only has 4096 bytes of memory, <a href="https://github.com/hughperkins/VeriGPU/blob/f4279efa308638e7218dd7865d987adaf6c2cc77/src/gpu_runtime/hip_api.cpp#L22" class="inline-onebox" rel="noopener nofollow ugc">VeriGPU/hip_api.cpp at f4279efa308638e7218dd7865d987adaf6c2cc77 · hughperkins/VeriGPU · GitHub</a> .</p>
<pre><code>    prop-&gt;totalGlobalMem = 4096;
</code></pre>
<p>Why is pytorch asking for 2 meg of memory, when I clearly state I only have 4096 bytes?</p>
Visualizing LSTM Model with Tensorboard

username: noah14
cooked: <p>I am trying to visualize my LSTM model, which looks like this:</p>
<pre><code class="lang-auto">class SequenceModel(nn.Module):

  def __init__(self, n_features, n_classes, n_hidden=100, n_layers=3):
    super().__init__()

    self.lstm = nn.LSTM( 
        input_size = n_features,
        hidden_size = n_hidden,
        num_layers = n_layers,
        batch_first = True,
        dropout = 0.945
    )
    self.classifier = nn.Linear(n_hidden, n_classes)

  def forward(self,x):
    self.lstm.flatten_parameters()
    _, (hidden,_) = self.lstm(x)
    

    out = hidden [-1] 
    return self.classifier(out) 

</code></pre>
<pre><code class="lang-auto"> (model): SequenceModel(
    (lstm): LSTM(3, 100, num_layers=3, batch_first=True, dropout=0.945)
    (classifier): Linear(in_features=100, out_features=2, bias=True)
  )
  (criterion): CrossEntropyLoss()
)
</code></pre>
<p>I know usually you do this like this:</p>
<blockquote>
<p>To write the computational graph we will be using add_graph() method. add_graph requires two arguments</p>
<ol>
<li>The <strong>model</strong></li>
<li>A <strong>sample image</strong> for the same shape as that of the input to track how it changes as it passes through the network</li>
</ol>
</blockquote>
<p>So i tried it with this code:</p>
<pre><code class="lang-auto">from torch.utils.tensorboard import SummaryWriter
sample=torch.rand((3,2,3))
writer = SummaryWriter()
writer.add_graph(model,  sample)
writer.close()
</code></pre>
<p>But I am getting this error:</p>
<pre><code class="lang-auto">Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
Error occurs, No graph saved
</code></pre>
<p>I can´t figure out what the problem is, has anyone an idea? Thanks in advance.</p>
[Dataparallel] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

username: C_J
cooked: <p>Hi,<br>
I’m trying to use dataparallel, and I encounter this runtime error:</p>
<p>RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!</p>
<p>I use some custom parameters in my model (some are training parameters, some are constant),<br>
and when I initialize the module, I save some parameters in gpu using .cuda, while some are initialized using torch.nn.Parameter. The model is written before I try to use dataparallel. SO I suspect that when I wrap my model, it gets confused that some of the parameters are located in a fixed gpu.</p>
<p>Can I get some advice on how I should appropriately initialize my model to use dataparallel ?</p>
<p>Thanks for reading and have a nice day!</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="C_J" data-post="1" data-topic="149361">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/c_j/48/44891_2.png" class="avatar"> C_J:</div>
<blockquote>
<p>I save some parameters in gpu using .cuda,</p>
</blockquote>
</aside>
<p>Try not to use <code>cuda()</code> and let the <code>DataParallel</code> handle movement of tensors/params on its own.<br>
If the issue still persists, pl. try to share a code snippet to debug and point out the issues if any.</p>
[DataLoader] What if entire dataset is larger than memory size?

username: C_J
cooked: <p>Let’s say I have 10000 datafiles x 100MB size.<br>
In total, dataset size is ~1TB, which is much larger than memory.<br>
To my understanding, in the most of the examples of DataLoader,<br>
“dataset” is considered to be pre-allocated in the memory, and then feeded to DataLoader during the training:</p>
<p>dataloader = DataLoader(dataset, batch_size=20, shuffle=True)</p>
<p>If dataset is larger than memory, should I re-load some portion of dataset and re-define dataloader multiple times during the training iteration?</p>
<p>Specifically, I’m having trouble to implement DataLoader in DistributedDataParallel,<br>
because I don’t seem to have much freedom in DDP coding (e.g. re-load dataset from harddisk during the parallel process).<br>
But I’m also suspecting that my understanding of DataLoader not deep enough.</p>
<p>Please give me some advice,<br>
Thanks!</p>

username: ejguan
cooked: <p>If you are using <code>Dataset</code> class, try to move your loading logic into <code>__getitem__</code> rather than in <code>__init__</code> to make sure each data is fetched lazily.</p>
<p>With the similar idea, we create a new library called <a href="https://github.com/pytorch/data" rel="noopener nofollow ugc">torchdata</a> to make <code>Iterable-style</code> as the first citizen for large dataset. It basically creates multiple data operations to support lazy execution. Feel free to take a look and checkout our repo.</p>
<p>Here is the doc <a href="https://pytorch.org/data" class="inline-onebox" rel="noopener nofollow ugc">TorchData</a></p>

username: C_J
cooked: <p>Thanks, your answer was definitely helpful.<br>
Although my issue haven’t solved fully yet, I will try to play with it more.</p>
How to do constrained minimization in Pytorch?

username: Jeet
cooked: <p>I want to minimize an equation. The equation consists of elements which are all tensors.</p>
<pre><code class="lang-auto">f=alpha + (vnorm/2)

where, vnorm=norm(v)*norm(v)
v is a tensor vector of n*1 and alpha is a tensor of 1*1
 
</code></pre>
<p>Now I need to minimize f with respect to a contraint, that is–</p>
<pre><code class="lang-auto">(A @ v)+alpha&lt;=0

</code></pre>
<p>where A is a tensor of 2*n.</p>
<p>How should I formulate the above equation and the the constraint to minimize the same ?</p>
Define loss function

username: 29233
cooked: <p>I’m trying to customize a loss function which include a weight matrix.<br>
I’d like to use it in this way:<br>
I’m doing a work of 8-class classification.The output of my model with is a batch of 8-dimension tensor and each value of the tensor is the probability.My labels are another batch of one-hot vectors.  I’d like to define my loss function like output.dot(weight_matrix.dot(labels.T)).I did it in a normal way but I got errors telling me that object has no attribute ‘backward’.I have no idea about how to deaal with ‘backward’.</p>
<p>Can anyone give me some advice?</p>
<p>Thanks in advance!</p>
<p>Here’s my code↓</p>
<pre><code class="lang-auto">class emtional_damage(nn.Module):
    def __init__(self, emo_mat, predict, label):
        super(emtional_damage).__init__()
        self.emo_mat = emo_mat
        self.predict = predict
        self.label = label

    def forward(self):
        loss0 = torch.mm(self.predict, self.emo_mat)
        loss = torch.mm(loss0, self.label.t())
        loss = loss.sum()
        return loss
</code></pre>

username: thecho7
cooked: <p>I guess you don’t need to define class to do that.<br>
How about define function instead of class?</p>
<pre><code class="lang-auto">def custom_loss(emo_mat, predict, label):
    loss0 = torch.mm(predict, emo_mat)
    loss = torch.mm(loss0, label.t())
    loss = loss.sum()
    return loss
</code></pre>
<p>You can use it as below,</p>
<pre><code class="lang-auto">model = YourModel()
out = model(input)
loss = custom_loss(emo_mat, out, label)
...
</code></pre>

username: mxahan
cooked: <aside class="quote no-group" data-username="29233" data-post="1" data-topic="149314">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/29233/48/48252_2.png" class="avatar"> 29233:</div>
<blockquote>
<pre><code class="lang-auto">class emtional_damage(nn.Module):
    def __init__(self, emo_mat, predict, label):
        super(emtional_damage).__init__()
        self.emo_mat = emo_mat
        self.predict = predict
        self.label = label

    def forward(self):
        loss0 = torch.mm(self.predict, self.emo_mat)
        loss = torch.mm(loss0, self.label.t())
        loss = loss.sum()
        return loss
</code></pre>
</blockquote>
</aside>
<p>You don’t need a class for this. However, if you really want to use class then</p>
<pre><code class="lang-auto">class emtional_damage(nn.Module):
    def __init__(self):
        super(emtional_damage).__init__()

    def forward(self, emo_mat, predict, label):
        loss0 = torch.mm(predict, emo_mat)
        loss = torch.mm(loss0, label.t())
        loss = loss.sum()
        return loss
</code></pre>
<p>You can use it as follow.</p>
<pre><code class="lang-auto">L = emtional_damage()
.....
loss = L(emo, pred, labes)
</code></pre>
Roi align maxpooling version

username: jhp
cooked: <p>Hi I have a question about pytorch implementation of roi align.<br>
It seems that only<code> average pooling</code> is supported for roi align in pytorch, but actually original roi align was to use<code> max pooling</code> in each interpolated bin.<br>
I wonder if pytorch has a plan to provide max pooling or if there is an unofficial code related to it. (Cant not find:( )</p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [2750]] is at version 2; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compu

username: AD_2000
cooked: <p>Hi everyone. I keep getting this runtime error every time I run the code about inplace operation. However, the operation is slicing a tensor in this case and since the tensor adj is not leaf slicing should not affect backprop but it does. I cant figure out why this happens.</p>
<p>If anyone can help me that would be great<br>
Thank you</p>
<pre><code class="lang-auto">def convert_adj_to_coo_batch(new_node, new_edge):
#     batch = Batch(x=new_node, adj=new_edge)
    adj = new_edge.clone()
    offset, row, col = (adj &gt; 0).nonzero().t()
    **edge_weight = adj[offset, row, col]**
    row += offset * new_node.shape[1]
    col += offset * new_node.shape[1]
    edge_index = torch.stack([row, col], dim=0)
    x = new_node.view(new_node.shape[0] * new_node.shape[1], new_node.shape[2])
    return x, edge_weight, edge_index
</code></pre>
<p>The error is in edge_weight. The code where I do an inplace operation however is here</p>
<pre><code class="lang-auto">def introduce_interventions(input_gnn):
    index = torch.randint(1, input_gnn.shape[1], (1,))
    input_gnn[:, index , :] = 0
    input_gnn_cl = input_gnn.clone()
    return input_gnn_cl
</code></pre>
<p>This function is called before the erroneous function.</p>

username: KFrank
cooked: <p>Hi AD!</p>
<aside class="quote no-group" data-username="AD_2000" data-post="1" data-topic="149222" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ad_2000/48/48671_2.png" class="avatar"> AD_2000:</div>
<blockquote>
<pre><code class="lang-auto">    **edge_weight = adj[offset, row, col]**
    row += offset * new_node.shape[1]
    col += offset * new_node.shape[1]
</code></pre>
</blockquote>
</aside>
<p>Try:</p>
<pre data-code-wrap="python"><code class="lang-python">    row = row + offset * new_node.shape[1]
    col = col + offset * new_node.shape[1]
</code></pre>
<p><code>+=</code> is also an inplace operation for pytorch tensors.</p>
<p>(For pytorch tensors, <code>t += u</code> and <code>t = t + u</code> are not synonymous.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: AD_2000
cooked: <p>Thank you so much <a class="mention" href="/u/kfrank">@KFrank</a>. It worked perfectly!</p>
RuntimeError: expected scalar type Long but found BFloat16

username: silence_cee
cooked: <p>File “/opt/anaconda3/envs/my-rdkit-env/lib/python3.9/site-packages/torch/nn/modules/linear.py”, line 103, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
RuntimeError: expected scalar type Long but found BFloat16</p>

username: ptrblck
cooked: <p>Without having more information than this error message, I can only recommend to check the <code>dtype</code>s of the input data and the model, and make sure they are matching. If you are using <code>torch.cuda.amp.autocast</code>, the transformations should be done internally for you.</p>
Cloning learning rate scheduler

username: thyeros
cooked: <p>Hi, I’m looking for a way to clone a learning rate scheduler without re-instantiating an object. I need to use the same scheduler for a different set of parameters, but don’t want to check the type of existing one and create a new one manually. I just want to clone it, but even after trying various methods, no luck yet. Is there such a way?</p>
<pre><code class="lang-auto">import torch
import torch.optim as optim
import pickle
import copy

lr = 0.1
opt = optim.SGD([torch.rand((2,2), requires_grad=True)], lr=lr)
sch = optim.lr_scheduler.MultiStepLR(opt, milestones=[1,4,8], gamma=0.5)

sch2 = pickle.loads(pickle.dumps(sch))
sch3 = copy.deepcopy(sch)

torch.save(sch,'/tmp/sch.pt')
sch4 = torch.load('/tmp/sch.pt')

test = 3 # change into 1, 2, 3, 4
for epoch in range(1, 10):
    if test==2:        
        sch2.step()
    elif test==3:
        sch3.step()
    elif test==4:
        sch4.step()
    else:
        sch.step()
        
    print('Epoch-{0} lr: {1}'.format(epoch, opt.param_groups[0]['lr']))
    if epoch % 5 == 0:print()
</code></pre>
<p>This is from test=1</p>
<pre><code class="lang-auto">Epoch-1 lr: 0.05
Epoch-2 lr: 0.05
Epoch-3 lr: 0.05
Epoch-4 lr: 0.025
Epoch-5 lr: 0.025

Epoch-6 lr: 0.025
Epoch-7 lr: 0.025
Epoch-8 lr: 0.0125
Epoch-9 lr: 0.0125
</code></pre>
<p>But, all other test=2,3,4 yield this</p>
<pre><code class="lang-auto">Epoch-1 lr: 0.1
Epoch-2 lr: 0.1
Epoch-3 lr: 0.1
Epoch-4 lr: 0.1
Epoch-5 lr: 0.1

Epoch-6 lr: 0.1
Epoch-7 lr: 0.1
Epoch-8 lr: 0.1
Epoch-9 lr: 0.1
</code></pre>
Check value of gradient

username: LEONARDO_MUTTI
cooked: <p>Hi, I am training a NN with PyTorch and I want to check whether the gradients of the loss <code>total_loss</code> (on which I call <code>total_loss.backward()</code>), are constant with respect to the weights contained in an <code>nn.Sequential</code> block, call it <code>block</code>.<br>
How can I easily achieve this? Thanks!</p>

username: JuanFMontesinos
cooked: <p>You can iterate over the model parameters:<br>
<code>for name,param in model.named_parameters()</code> and then inspect <code>param.grad</code><br>
Which should be either a torch.Tensor or None (if your network is breaking backprop due to any bug).</p>
Methods that support generalization

username: seller_basti
cooked: <p>Hello,</p>
<p>I have a problem implementing a deep reinforcement learning agent. Specifically, I am using double deep-Q-learning. I want to use it in order to solve a power control problem in telecommunications. I have a nodeA that walks in linear direction and sends a signal to a Node B with fixed position. The goal is to control the reicie<br>
My input state is as followed:<br>
I have a input tensor that consists out of two categories:<br>
a) Current SNR and several last SNRs (int values between -20dB and +20dB)<br>
b) actions that have been taken inbetween, to perform the controlling.<br>
My training data contains several run of the running nodeA with random initialization of the starting position and the running direction, nodeB is always at the same position (for each training run).<br>
So far I am just putting a tensor of the form [a, b] into the network. My output layer has 4 neurons that correspond to the number of actions the agent can take.<br>
Before going to the hidden layers, my input vector goes through an embedding in order to get a better representation of the data.<br>
My training works fine but still there occur some outliers, I cant get rid of (whether its by increasing the training data or increasing the number of neurons in the hidden layers).<br>
My net looks as followed:</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.emb1= nn.Embedding(45,4).to(device)<br>
self.fc1 = nn.Linear(num_state<em>4, 100).to(device)<br>
self.fc11 = nn.Linear(100,100).to(device)<br>
self.fc2 = nn.Linear(100, 100).to(device)<br>
self.fc3 = nn.Linear(100,num_action).to(device)<br>
def forward(self, x):<br>
x = DoubleDQN.putInEmbeddingForm(x)<br>
x = self.emb1(x).view((-1, num_state</em>4))<br>
x = F.relu(self.fc1(x))<br>
x = F.relu(self.fc11(x))<br>
x = F.relu(self.fc2(x))<br>
action_value = self.fc3(x)<br>
return action_value</p>
<p>I have the general question, is there a common way to improve the performance of such a net? I read a lot of stuff, like batch normalization or layer normalization and also tried some things which did not work. I just don’t have the experience in order to determine which techniques could help.<br>
I someone could propose me some techniques that I can try.</p>
<p>regards,<br>
seller_basti</p>
Dataloader adds batch number to torch.shape

username: mmandayam
cooked: <p>I am using torch.utils.data.DataLoader to load my data in batches. It loads all data fine. But the batch number is the first parameter in the shape of the torch variable. So when I use GRU or LSTM model they complain that the torch dataset has an extra dimension. Is there a way to avoid this problem? For example my dataset is of shape [4, 1, 6]. But dataloader returns shape as [1, 4, 1, 6] which LSTM or GRU do not like.</p>
<p>Even if say batch_size = 1 it still adds a 1 to the shape like [1, 4, 1, 6].</p>
<p>How can I get the size of my dataset to be [4, 1, 6] instead of [1, 4, 1, 6]?</p>

username: nivek
cooked: <ol>
<li>
<p>In the dataset that you are passing into <code>DataLoader</code>, can you check what the output of its <code>__getitem__</code> is (or <code>__iter__</code> for Iterable-style dataset)? Specifically, check the dimension of that output.</p>
</li>
<li>
<p>If the dimension of the above is correct, can you check if auto collation is being applied to your output? Is that step changing the dimension?</p>
<ul>
<li>Within the <code>DataLoader</code>, the <code>collate_fn</code> is applied to your batch (by default, <a href="https://github.com/pytorch/pytorch/blob/4afe2db641c526800fb8b33532a920be625f3cf4/torch/utils/data/_utils/collate.py#L84" rel="noopener nofollow ugc"><code>default_collate</code></a>) as described by <a href="https://pytorch.org/docs/stable/data.html#automatic-batching-default" rel="noopener nofollow ugc">the documentation here</a>. You can choose to override that.</li>
</ul>
</li>
</ol>

username: huahuanZ
cooked: <p>set batch_size=None in the DataLoader</p>

username: mmandayam
cooked: <p>If I set batch_size to None will it read the entire dataset defeating the purpose of splitting the training/validating dataset into batches?</p>

username: InnovArul
cooked: <aside class="quote no-group" data-username="mmandayam" data-post="1" data-topic="148979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e99b99/48.png" class="avatar"> mmandayam:</div>
<blockquote>
<p>How can I get the size of my dataset to be [4, 1, 6] instead of [1, 4, 1, 6]?</p>
</blockquote>
</aside>
<p>If all you need is to get data of size [4,1,6] from the  [1, 4, 1, 6] sized data, you can write <code>data = torch.squeeze(0)</code> (or) <code>data = data[0]</code> after getting the data.</p>

username: mmandayam
cooked: <p>Actually, setting batch_size = None worked. Thanks David.</p>
Replace all values close to zero

username: Samue1
cooked: <p>I would like to know how to set values that are close to zero to a certain value. Let’s say I have a tensor <code>T</code>. During the forward pass, I would like to set all values in <code>T</code> in the range [-1e-4, 1e-4] to -1e-4 if the value is negative and 1e-4 in case of a small but positive value.</p>
<p>I wanted to use <code>torch.where</code> and <code>torch.clamp</code> but couldn’t manage to find a solution that works for me. Any ideas?</p>

username: InnovArul
cooked: <p>Something like this might work:</p>
<pre><code class="lang-auto">import torch
x = torch.randn(5,5)

# to handle 0 &lt;= x &lt;= 1e-4
x[torch.logical_and(x&gt;=0, x&lt;=1e-4)] = 1e-4 

# to handle -1e-4 &lt;= x &lt; 0
x[torch.logical_and(x&lt;0, x&gt;=-1e-4)] = -1e-4 
</code></pre>

username: Samue1
cooked: <p>Thank you. I didn’t know about <code>torch.logical_and()</code>. However, when I use this approach I get the following error in my <code>forward()</code> method:</p>
<blockquote>
<p>RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</p>
</blockquote>

username: InnovArul
cooked: <p>I am not sure about your use case. But when you assign a hard-coded value for certain elements in a tensor, the gradients will be zero for those elements. I hope you are aware of this.</p>
<p>You can overcome the <code>in-pace operation</code> runtime error by cloning <code>x</code> before modifying it.</p>
<pre><code class="lang-auto">x = x.clone()

# to handle 0 &lt;= x &lt;= 1e-4
x[torch.logical_and(x&gt;=0, x&lt;=1e-4)] = 1e-4 

# to handle -1e-4 &lt;= x &lt; 0
x[torch.logical_and(x&lt;0, x&gt;=-1e-4)] = -1e-4 
</code></pre>
Pytorch with DDP throws error with multi-GPU

username: Rini
cooked: <p>Hello,</p>
<p>I am trying to train a multi-task BERT model with PyTorch on a single node with 8GPUs. I am able to train the model if I use a single GPU, however if I switch to multiple GPUs I get an error with a mismatch in batch size. I checked in debug mode that in the forward function (from the validation step) if I specify a batch size of 4 with 2 GPUs, I get 2 <code>input_ids</code> in the <code>batch</code>. If I specify a batch size of 4 with 1 GPU, I get 4 <code>input_ids</code> in batch which right.<br>
My forward function looks like pasted below</p>
<pre data-code-wrap="def"><code class="lang-nohighlight">
        batch = x

        bert_output = self.bert(
            batch["input_ids"],
            position_ids=batch["position_ids"],
            attention_mask=batch["attention_mask"],
            token_type_ids=batch["token_type_ids"],
        )
        sequence_output = bert_output["last_hidden_state"]
        pooled_output = bert_output["pooler_output"]

        masked_seq_indices = np.where([x for x in batch["token_tasks"].values()][0])[0]
        chunk_permuted_seq_indices = np.where([x for x in batch["sequence_tasks"].values()][0])[0]

        mlm_scores = self.lm(sequence_output[masked_seq_indices])
        mlm_loss = self.mlm_loss(mlm_scores.view(-1, self.vocab_size), batch["mlm_labels"].view(-1))

        chunk_permute_scores = self.sent["chunk_perm"](pooled_output[chunk_permuted_seq_indices])
        chunk_perm_loss = self.chunk_perm_loss(chunk_permute_scores, batch["chunk_permuted_labels"])

        return (mlm_loss, chunk_perm_loss)````

What parameters I need to add in my forward function or elsewhere to get the right batch when evaluating the validation step (and training step as well)?

Thank you!</code></pre>
Using {N,H,W,C} format in customized operation

username: IwakuraRein
cooked: <p>I’m thinking about upgrading some of my customized PyTorch operations to support {N,H,W,C} format. However, I’m still confused about using channel-last-format tensors in PyTorch.</p>
<p>First, what does <code>.contiguous(memory_format=torch.channels_last)</code> do?</p>
<pre><code class="lang-python">a=torch.randn((1, 3, 128, 128)).contiguous(memory_format=torch.channels_last)
b=torch.randn((1, 128, 128, 3))
c=torch.randn((1, 128, 128, 3)).contiguous(memory_format=torch.channels_last)
d=torch.randn((1, 128, 128, 3)).permute([0,3,1,2]).contiguous(memory_format=torch.channels_last)
</code></pre>
<p>Which tensor is truly stored with {N,H,W,C} (e.g., {1, 128, 128, 3}) in memory? a, b, c, or d? Btw when I inputed them to convolution layers, both b and c raised errors.</p>
<p>Second, how to tell which memory format a tensor is using in CPP? In the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="noopener nofollow ugc">documentation</a>, the sample codes of creating customized operations with CPP are:</p>
<pre><code class="lang-cpp">#define CHECK_CUDA(x) TORCH_CHECK(x.device().is_cuda(), #x " must be a CUDA tensor")
#define CHECK_CONTIGUOUS(x) TORCH_CHECK(x.is_contiguous(), #x " must be contiguous")
#define CHECK_INPUT_TENSOR(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)

std::vector&lt;torch::Tensor&gt; lltm_forward(
    torch::Tensor input,
    torch::Tensor weights,
    torch::Tensor bias,
    torch::Tensor old_h,
    torch::Tensor old_cell) {
  CHECK_INPUT(input);
  CHECK_INPUT(weights);
  CHECK_INPUT(bias);
  CHECK_INPUT(old_h);
  CHECK_INPUT(old_cell);

  return lltm_cuda_forward(input, weights, bias, old_h, old_cell);
}
</code></pre>
<p>I supposed the <code>x.is_contiguous()</code> called by <code>CHECK_CONTIGUOUS(x)</code> confirms <code>x</code> is {N,C,H,W} but how do I know if it is {N,H,W,C}? If <code>x.is_contiguous()</code> returns <code>false</code>, is the tensor definitely {N,H,W,C}?</p>
<pre><code class="lang-cpp">torch::Tensor my_op(const torch::Tensor &amp;input,
                    const string &amp;data_format) {
    if (data_format == "NCHW") {
        CHECK_INPUT_TENSOR(input)
    }
    else if (data_format == "NHWC") {
        // How do I make sure that the tensor is NHWC?
    }
    else
        throw std::invalid_argument("Error: Input data_format should be \"NCHW\" or \"NHWC\".");
        
    torch::Tensor output = torch::zeros_like(input);          
    if (data_format == "NCHW") {
        // call cuda function for NCHW input
    }
    else if (data_format == "NHWC") {
        // call cuda function for NHWC input
    }
    return output;
</code></pre>
Cuda Out Of Memory in simple operation

username: AleTL
cooked: <p>Hello!</p>
<p>I’ve been reading previous solved issues with CUDA Out of memory but I haven’t found anything that could help me, that’s why I open this thread.</p>
<p>My code is the following: <code>(y - x).pow(2).div(sigma.mul(2))</code>, and the error I receive is: <code>RuntimeError: CUDA out of memory. Tried to allocate 129.21 GiB (GPU 0; 31.75 GiB total capacity; 230.09 MiB already allocated; 30.26 GiB free; 280.00 MiB reserved in total by PyTorch)</code>.</p>
<p>I don’t understand how this operation could try to allocate around 130 GiB. The size of the tensors, obtained using <code>a.element_size()*a.nelement()</code> is 744960 bytes. The simple operation of <code>(y-x)</code> already provokes this error.</p>
<p>Does anyone have any idea about how could this be happening with this tensor sizes? I honestly don’t get it.</p>

username: ptrblck
cooked: <p>Could you post the shapes of both tensors as I would guess that broadcasting is used, which would increase the size?</p>
Why inference time increases after few iteration?

username: jhp
cooked: <p>Hi I faced the problem of increasing inference time after few iteration, specifically when the batch size is large.</p>
<pre><code class="lang-auto">import time
import torch
a=torch.randn((1600,3,model.visual.input_resolution,model.visual.input_resolution)).to(device=device)
st=time.time()
for i in range(20):
    st=time.time()
    with torch.no_grad():
        model.encode_image(a)
    print(time.time()-st)
</code></pre>
<p>When I ran this code, the inference time is printed out as below</p>
<pre><code class="lang-auto">0.0164792537689209
0.015082120895385742
0.24733448028564453
0.4809727668762207
0.48241543769836426
0.4811861515045166
0.4822506904602051
0.48279619216918945
0.4839320182800293
0.48270678520202637
0.4844379425048828
0.4843134880065918
0.48419904708862305
0.48407626152038574
0.4829099178314209
0.48367929458618164
0.48392295837402344
0.4842257499694824
0.4861021041870117
0.4838073253631592
</code></pre>
<p>But it happens when the batch size (in my case 1600) is large. The memory usage seems not heavy, but I wonder why the inference time suddenly increases. In fact, it seems normal to take 0.48 seconds for batch size 1600, but why did it become so quickly in the beginning?</p>

username: AlphaBetaGamma96
cooked: <p>If you’re running this on a GPU you need to call <code>torch.cuda.synchronize()</code> before each call of <code>time.time()</code> as GPU calls are asynchronous. Add <code>torch.cuda.synchronize()</code> and measure the time again and see if the time changes?</p>
Split learning/ problem with trainig with different datasets

username: vojkan7
cooked: <p>Hi everyone<br>
I have 5 clients, one server and one cut layer between them (vanilla split learning)</p>
<p>In the first case, each client becomes a whole cifar10 dataset and each client have one optimizer. Training is running perfectly.</p>
<p>In another case, I divided the dataset into five pieces(each has 2 classes) and each client becomes one piece.<br>
On the training, only the first client will be trained and another four clients not (accuarcy : 0.0).<br>
If I skip the first client at the beginning of training and train client 2,3,4,5, only client nr 2 will be trained.<br>
In the second, third etc epoch all clients have accuracy 0.<br>
I already tried many stuffs, like to make for each client-model optimizer on the server side, but didn’t help</p>
<p>Any hint,help?</p>
Conversion of detectron2 code from Python to C++

username: Sylvain_Ard
cooked: <p>Hello all !<br>
I want to query funds to convert detectron2 code from Python to C++, how much euros do you think would be necessary for a such task and how many time ? And in case I will not find funds to pay an expert have you tracks to advise to me please ? Sorry for my bad english because I’m french.<br>
Thank you<br>
Kind regards</p>