akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Expected tensors to be on one device, but found two: cpu and cuda:0

nlp_8172: <p>Hey guys. I am running code from another repository and ran into the above error. I have looked into it online, and none of the solutions I have found work for me. I have already verified that the model is on cuda:0, so it is the data objects that are lingering on the CPU.</p>
<p>The dataloader is obtained from a dataset as follows:</p>
<pre><code class="lang-auto">eval_dataloader = self.get_eval_dataloader(eval_dataset) #in a class inheriting from Seq2SeqTrainer
</code></pre>
<p>Later, the code is run as follows:</p>
<pre><code class="lang-auto">output = eval_loop(
        eval_dataloader,
        description="Evaluation",
        prediction_loss_only=True if compute_metrics is None else None,
        ignore_keys=ignore_keys,
)
</code></pre>
<p>Eval loop here is get_evaluation_loop from the transformers trainer class. So the point is, because of the way this is run I don’t think I can move the batches to the GPU just before training on them; it is run through the eval_loop method rather than through a modifiable for loop.</p>
<p>From here, I have tried many methods to move the dataloader to the GPU, including:</p>
<pre><code class="lang-auto">for batch in eval_dataloader:
for k, v in batch.items():
        v = v.to(device)
        batch[k] = v
</code></pre>
<p>But, for whatever reason, I still get this error. Any suggestions as to how I can fix this? Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="nlp_8172" data-post="1" data-topic="172801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nlp_8172/48/56774_2.png" class="avatar"> nlp_8172:</div>
<blockquote>
<p>Eval loop here is get_evaluation_loop from the transformers trainer class. So the point is, because of the way this is run I don’t think I can move the batches to the GPU just before training on them;</p>
</blockquote>
</aside>
<p>In this case I would recommend checking the trainer class code to see where the data is supposedly moved to the device, as the error seems to come from the higher-level API you are using.</p>
Is there any way to find the seed used for initialization?

mahsa: <p>Hi,<br>
I implemented my code and got good results. Is there any way to find out which seed is used for my results on server code? I didn’t set the seed manually and want to make sure I can reproduce the results.<br>
Thanks</p>

albanD: <p>Hey,</p>
<p>You can use <code>torch.seed()</code> to get the current seed.<br>
You might want to check the reproducibility part of the doc though: <a href="https://pytorch.org/docs/stable/notes/randomness.html">https://pytorch.org/docs/stable/notes/randomness.html</a> as having the seed most likely won’t allow you to reproduce the result if you’re using a different machine or using ops that are not deterministic.</p>

mahsa: <p>Can you say that what exactly manual.seed() shows? When I set seed at the start of the initialization function, and print torch.seed() for two similar networks, I give bunch of different numbers, but I think since it’s set at the start of the function, any initialization with that seed must be unique. Am I right?</p>

albanD: <p>You can check the doc <a href="https://pytorch.org/docs/stable/torch.html#torch.seed">here</a>. It returns the value used to initialize the random number generator.<br>
You can also use get/set rng state (<a href="https://pytorch.org/docs/stable/torch.html#torch.get_rng_state">https://pytorch.org/docs/stable/torch.html#torch.get_rng_state</a>).</p>

SpandanMadan: <p>I just read the reproducibility, and it seems torch.seed() is to set the seed randomly, not to get the current seed.</p>

infinitylogesh: <p><code>torch.random.initial_seed()</code> has provision to provide the seed value. Got it from the <a href="https://github.com/pytorch/pytorch/blob/45d775cedb72132343eab2422c61d7eb69cc5843/torch/random.py#L66" rel="noopener nofollow ugc">torch repo</a></p>
How to modify forward hook output correctly?

gggg111: <p>Hi, I know this question has already been answered previously, but in my case I go a bit deeper as my output type is not a tensor but a tuple.</p>
<p>Let my hook be:</p>
<pre><code class="lang-auto">def __inout_hook(self, layer, input, output):
    outc = output
    outc[0] += 2

    return outc
</code></pre>
<p>where output is a tuple. In order to modify this output, I use the code above. Note how I create a new variable as I cannot directly modify the first element of the tuple directly.</p>
<p>The thing is, when calculating the gradients by using .backward(), the following error jumps:</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 256, 75, 17]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>I wonder how would I correctly modify the output of a layer’s hook accordingly.</p>
<p>Thank you beforehand.</p>

nkdatascientist: <p>Hi <a class="mention" href="/u/gggg111">@gggg111</a>,</p>
<p>The error message is indicating that there is an in-place modification of a tensor in the input or output of a layer, which is preventing the calculation of gradients during backpropagation.<br>
To fix this issue, you can create a copy of the output tensor and modify the copy instead of modifying the original tensor in-place. Can you try below code?  or  <a href="https://github.com/NVlabs/FUNIT/issues/23#issuecomment-551164653" rel="noopener nofollow ugc">Resolved reference link</a></p>
<pre data-code-wrap="py"><code class="lang-plaintext">def __inout_hook(self, layer, input, output):
    outc = output.clone()
    outc[0] += 2

    return outc
</code></pre>

gggg111: <p>Hi, <a class="mention" href="/u/nkdatascientist">@nkdatascientist</a>.</p>
<p>Thank you very much for your answer!</p>
<p>Because the <code>output</code> argument is of type tuple (and not tensor), I cannot use the <code>.clone()</code> method directly. So, what I just did, while seeming rudimentary, works by simply making the following intermediate steps like so:</p>
<pre><code class="lang-auto">def __inout_hook(self, layer, input, output):
    output0 = output[0].clone()
    output0 += 2

    self.output = (output0, output[1])

    return self.output
</code></pre>
<p>If there is a better way to to this, don’t hesitate to comment!</p>
<p>Much appreciated.</p>
Dataset load is too slow

pkr7098: <p>I’m using my custom numpy dataset (.npy).<br>
Here is my brief Dataset class</p>
<pre><code class="lang-python">class MyDataset(Dataset):
   def __init__(self, ~~~):
      super(MyDataset, self).__init__()
      self.PATH = PATH
      self.x = np.load(PATH['x'])
      self.y = np.load(PATH['y'])
      self.x = [FloatTensor(tensor) for tensor in self.x]
      self.y = [FloatTensor(tensor) for tensor in self.y]
      self.len = len(self.x)

   def __getitem__(self, index):
      x = self.x[index]
      y = self.y[index]
      return x, y
   def __len__(self):
      return self.len
</code></pre>
<p>When I training my model, the interval between epochs is too slow. It takes 10 seconds (training epoch itself takes 6 seconds)</p>
<pre><code class="lang-auto">batch_size=64
batch shape: (64, 3, 2048)
num_workers=6
pin_memory=True
</code></pre>
<p>How to resolve it?</p>

tataganesh: <p>In your case, loading the dataset involves loading two numpy arrays and storing them in <code>self.x</code> and <code>self.y</code> respectively. Once the dataset is loaded, retrieving the elements is a fast operation. Hence, the reason for slow training could be something else in your training process (e.g. model forward pass) . Check each step in your training loop, or post the training loop and the model here so that we can help you further.</p>

pkr7098: <p>Thank you for reply.<br>
Actually, I’m using pytorch-lightning framework so it maybe not good question for here but the code is similiar to pytorch so the pytorch user can give me some advice.</p>
<p>Here is my training code (validation and test code is same):</p>
<pre><code class="lang-python">
    def on_train_epoch_start(self):
        self.train_epoch_loss = 0
        self.train_epoch_loss_1 = 0
        self.train_epoch_loss_2 = 0
        self.train_step_count = 0
        
    def training_step(self, batch, batch_idx):        
        data1, data2, trg = batch
        data_gen1, data_gen2= self.model(trg)

        loss_gen1= self.loss(data_gen1, sp)
        loss_gen2 = self.loss(data_gen2, src)
        loss = loss_gen1 + loss_gen2
        
        self.train_step_count+=trg.shape[0]
        return {
            'loss_gen1': loss_gen1,
            'loss_gen2 ': loss_gen2,
            'loss': loss
        }
    
    def training_step_end(self, loss):
        self.train_epoch_loss += loss['loss'].item()
        self.train_epoch_loss_1 += loss['loss_gen1'].item()
        self.train_epoch_loss_2  += loss['loss_gen2 '].item()
        return super().training_step_end(loss)
    
    def training_epoch_end(self, outputs):
        self.log_dict({
            'train_epoch_loss ': self.train_epoch_loss / self.train_step_count,
            'train_epoch_loss_1 ': self.train_epoch_loss_1 / self.train_step_count,
            'train_epoch_loss_2': self.train_epoch_loss_2/ self.train_step_count
        }, sync_dist=True)
</code></pre>
<p>batch_size: 64<br>
log_every_n_steps: 10<br>
logger: wandb</p>
Why do we need to set local_rank = int(os.environ["LOCAL_RANK"]) in torchrun?

Brando_Miranda: <p>I saw that we need to do:</p>
<pre><code class="lang-auto">local_rank = int(os.environ["LOCAL_RANK"])
</code></pre>
<p>but we never set that env variable ourselfs which seems odd. Does torchrun or torch.distributed.launch set it by itself?</p>

xksteven: <p>Pytorch will create a new process and pass in the local_rank as an argument or set the LOCAL_RANK environment variable before it creates the process then set it again before creating the next process for the next GPU and so on.</p>
<p>You can see the error and pytorch passing it in if you omit the <code>--local_rank</code> as parse_args argument.</p>
Can't update a tensor by assignment

ramenspazz: <p>I am trying to add two tensors and save the result into a third tensor that I update every loop. Adding tensors seems hit or miss, as the velocity calculation works no problem and updates but for some reason my position does not change at all and refuses to update. For context, this snippet is a part of a function inside a class, but the variables y_0 and y_1. dy_0 and dy_1 are local variables defined at the top of this function. The output of self.eval_func.evaluate(*self.eval_func.tRP_tensor)).data does give me the proper tensor so this is not the issue.</p>
<p>Here is the relevant snippet:</p>
<pre><code class="lang-auto">for i, itt in enumerate(range(self.sim_time)):
    # if t is needed, it is calculated here
    cur_real_time = self.start_val + itt * self.dt

    # Calclate velocity from force
    temp1 = torch.mul(self.dt, self.eval_func.evaluate(*self.eval_func.tRP_tensor)).data
    # print(temp.data, dy_0.data)
    dy_1.data = dy_0.data + temp1.data  # noqa

    # calculate new position from velocity
    temp2 = torch.mul(self.dt, dy_1.data).data
    y_1.data = y_0.data + temp2.data
    print(y_0.data, temp2.data, y_1.data)

    # update variables this iterations initial and final values
    # for the next iteration
    y_0.data = y_1.data
    dy_0.data = dy_1.data
    self.update_args(cur_real_time, y_1, dy_1)

    if cur_real_time - last_output_time &gt;= self.output_step:
        # this block is option status output with ETA if check_val
        # is passed as True to the functuion
        if check_val:
            ratio = i / self.sim_time
            cur_time = datetime.now() - start
            ETA = cur_time.seconds / ratio
            print('\rCurrently {:.2f}%. Running for {} seconds : E.T.A {:.2f} Seconds. T - done = {:.2f}'.format(100 * ratio, cur_time, ETA, ETA - cur_time.seconds), end='')  # noqa

        # put the final result for this iteration in the output
        out_list.append(y_1.tolist()[0])
        last_output_time = cur_real_time

print('\rCompleted in {}!'.format(datetime.now()-start))
print(out_list)
return np.array(out_list)
</code></pre>
<p>In this code, y_1 (and y_0) refuses to update with the value of y_1 = y_0 + dt * dy_1. I have this issue sometimes with dy_0 in another function and believe it to the be same issue in a Runge Kutta 4 function but have no idea why it is happening.</p>
<p>The temp1 and temp2 variables exist for debugging purposes so I could see what was actually being assigned when I preformed operations. It has the same behavior if I substitute the definition of temp1 and temp2 into the code and remove the variables.</p>
<p>Thank you very much.</p>

ramenspazz: <p>Solved. The error for reference was that I didn’t make each tensor of the same dtype and instead left y_1 without explicitly being declared as dtype=float.</p>
Have a plan but need some guidance

Mah_Neh: <p>I am planning to train tiny yolo v4-7 for recognizing barcodes in images.</p>
<p>However, all the implementations I find in github are outdated and crash.</p>
<p>Is there any way to just load it using pytorch, in a simple way so that I do not have to implement it from scratch (which seems complex.) ?</p>
Trying to write equivalent Poisson loss function using torch distribution

EvanZ: <p>I’m currently using <code>PoissonNLLLoss</code> (well actually <code>F.poisson_nll_loss</code>) but I wanted to check if I can write my own custom loss using the poisson distribution from <code>torch.distributions</code>:</p>
<pre><code class="lang-auto">def poisson_nll(obs, lambd):
    poisson_dist = dist.Poisson(lambd)
    poisson_prob = poisson_dist.log_prob(obs)
    nll = -poisson_prob.mean()
    return nll
</code></pre>
<p>Does this look corect?</p>
From embedding to lstm

Vadimcg: <p>I need some clarity on how to correctly connect embedding layer and lstm.  For example, if i have only one feature i will send  to embedding layer such vector (batch size, length of sequence, 1 ) and after embedding lstm get  (batch size, length of sequence, size of embedding ) right? (of cource if we set batch_first=True)<br>
I can’t understand what view of vector i need to send to lstm if i have some features and one embedding layer for every feature</p>
Loaded TemporalFusionTransformer model from S3 bucket cannot make predictions

petartushev: <p>I used a <code>S3</code> bucket to store artifacts after training a <code>TemporalFusionTransformer </code> model, and the model itself. Later I needed to load the best model from <code>S3</code> and make predictions. The model is loaded correctly (the returned object is a <code>TemporalFusionTransformer</code> object) but when I try to make a prediction on that model I get the following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/petartushev/Production script/production_script.py", line 185, in &lt;module&gt;
    preds = model.predict(test_dataloader)
  File "/home/petartushev/Production script/production_venv/lib/python3.7/site-packages/pytorch_forecasting/models/base_model.py", line 1059, in predict
    out = self(x, **kwargs)  # raw output is dictionary
  File "/home/petartushev/Production script/production_venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/petartushev/Production script/production_venv/lib/python3.7/site-packages/pytorch_forecasting/models/temporal_fusion_transformer/__init__.py", line 507, in forward
    encoder_lengths=encoder_lengths,
  File "/home/petartushev/Production script/production_venv/lib/python3.7/site-packages/pytorch_forecasting/models/base_model.py", line 546, in to_network_output
    return self._output_class(**results)
TypeError: __new__() got an unexpected keyword argument 'attention'
</code></pre>
<p>When trying to fix this myself, I re-created the model in the same script where I am loading the model, with the same hyperparameters and that re-created model instead of the error above when calling <code>predict</code> with the same test dataloader returned a <code>Tensor</code> as an output.<br>
Any help towards resolving this issue is welcomed.</p>
Reduce number of Minibatches

Raphael_Olipitz: <p>Hey,</p>
<p>hope you are doing well.</p>
<p>I am programming a 3D U-Net, and got output tensors of shape [4,4,128,128,128]. As I assume the minibatch size comes from the torch.cat() function. But now I got an error for the softmax classification with the 4 classes to predict:</p>
<p>ValueError: Expected input batch_size (4) to match target batch_size (1).</p>
<p>I think this is because the mask data have the chape [1,1,128,128,128] but I do not know how to overcome this error. Would someone help me?</p>

ptrblck: <aside class="quote no-group" data-username="Raphael_Olipitz" data-post="1" data-topic="172630">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raphael_olipitz/48/47628_2.png" class="avatar"> Raphael_Olipitz:</div>
<blockquote>
<p>As I assume the minibatch size comes from the torch.cat() function.</p>
</blockquote>
</aside>
<p>I don’t know where the <code>torch.cat</code> function is used, but note that your model should not change the batch size. Since your target contains a single sample I would assume your model input was also originally a single sample?</p>

Raphael_Olipitz: <p>Yes I guess I had a wrong init in the test/train split. thx!</p>
Is there an implementation of the Enhanced Deep Reccurent Visual Attention model?

ADONIS_HAMMOUTENE: <p>Hey everyone, I want to know if there’s a pytorch or pytorch_lightning implementation of EDRAM or DRAM [ (Enhanced) Deep Recurrent Attention Model] .<br>
Here’s the paper discussing this topic: <a href="https://arxiv.org/abs/1706.03581" class="inline-onebox" rel="noopener nofollow ugc">[1706.03581] Enriched Deep Recurrent Visual Attention Model for Multiple Object Recognition</a>.</p>
<p>Thanks for your Attention.</p>
Error from PyTorch Lightning + Google Colab + TPU

PyDavid: <p>I built a translation model using Transformer and PyTorch Lightning. I am running it in Google Colab with TPU. However, I couldn’t get it to work due to the following error,</p>
<p>ModuleNotFoundError: ContextualVersionConflict: (google-api-python-client 2.70.0 (/usr/local/lib/python3.8/dist-packages), Requirement.parse(‘google-api-python-client==1.8.0’), {‘cloud-tpu-client’}). HINT: Try running <code>pip install -U 'torch_xla'</code></p>
<p>I can successfully import torch_xla.</p>
<p>The error happens when I try to create the Trainer</p>
<p>runner = Trainer(logger=tb_logger,<br>
callbacks=[<br>
LearningRateMonitor(),<br>
ModelCheckpoint(save_top_k=2,<br>
dirpath=os.path.join(tb_logger.log_dir, “checkpoints”),<br>
monitor=“val_loss”,<br>
save_last=True),<br>
],<br>
# strategy=DDPStrategy(find_unused_parameters=False),<br>
strategy=None,<br>
**config[“trainer_params”],<br>
)</p>
<p>Here is my google colab notebook: <a href="https://colab.research.google.com/drive/1Or3YcDI9wyHM6v_3WxvBiceTiZD07_Yj?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>At the beginning of the notebook, I was trying to install all the required packages.</p>
Why roi_align returns only zeros? (roi_align, vgg16 - feature map)

MikeTensor: <p>Hello together!</p>
<p>The <code>roi_align</code>-Function in torchvision.ops only returns zeros.<br>
Any clue why? Or where I can find help?<br>
any help or hint would be fantastic and appreciated.</p>
<p>When using the whole image as a ROI I get around 4 numbers per each of the 512 roi_align-Output layer. When using real roi_coordinates its over: Just zeros.</p>
<p>I used <strong>spatial_scale = 0.03</strong> (since 11/375 and 38/1242 are almost equal to 0.03).</p>
<p>####################################################################<br>
####################################################################</p>
<p><strong>Setup:</strong><br>
image [3, 375, 1242] → vgg16(feature_layers)* → <code>roi_align</code>** = y = [n x 512 x 11 x 38] = 0.<br>
Where n is the index of the roi.</p>
<p>*<strong>vgg16:</strong><br>
<code>model = models.vgg16(weights='IMAGENET1K_V1')</code><br>
<code>## remove the classifier completly</code><br>
<code>model.classifier = nn.Sequential(*list(model.classifier.children())[:-7])</code><br>
<code>## remove the avgpooling</code><br>
<code>model.avgpool = nn.Sequential(*list(model.avgpool.children())[:-1])</code></p>
<p>→ Output shape: [1, 512, 11, 38] (but flattend<br>
I do not resize the input images for the vgg16.</p>
<p>**<strong>roi_align</strong><br>
<code>output_size2 = (11, 38)</code><br>
<code>spatial_scale = 0.03</code></p>
<p><code>rois_test = torch.FloatTensor([</code><br>
<code>[0.0000, 1056.1600,  190.5000, 1241.0000,  374.0000],</code><br>
<code>[0.0000,  359.4300,  179.3000,  516.3000,  270.9700],</code><br>
<code>[0.0000,  600.1900,  163.5600,  633.5700,  188.6000]])</code></p>
<p><code># Test; take whole image as one ROI</code><br>
<code>rois_whole_image = torch.FloatTensor([     [   0,  0,  0,  1242,  375] ])</code></p>
<p><code>rois_x = rois.cuda()</code><br>
<code>z = img_feature.to("cuda").view(-1,512,11,38) # since the Tensor is flatten; reshape it.</code><br>
<code>y = torchvision.ops.roi_pool(z, rois_x, output_size2)</code><br>
`</p>
Torch.distributed collectives call logging

vadimkantorov: <p>Is there any way to enable logging of collectives calls?</p>
<p>This is useful for debugging deadlocks.</p>
<p>I could think of monkey-patching all distributed methods to print rank/process group/called collective, but I wonder if there is already an existing option / env. variable</p>
<p>And would such Python-level monkey-patching work for C++ level collectives’ calls? Probably not, any more reliable way / hooks?</p>
<p>There’s <a href="https://pytorch.org/docs/master/distributed.html#debugging-torch-distributed-applications" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a>, but from this page it’s unclear whether any level of <code>TORCH_DISTRIBUTED_DEBUG</code> enables very simple tracing of collectives’ calls.</p>
Multi branch input CNN to LSTM for 1D data

Learner1: <p>i’m trying to implement multi branch Network (CNN) where each branch is inserted with 1D data of different size and later concatenate outputs together and pass to LSTM model for prediction. I’m kind of beginner, and there is not much helpful material available related to PyTorch, so kindly help me.</p>
<pre><code class="lang-auto">train_load_1 = DataLoader(dataset=train_dataset_1, batch_size=100, shuffle=False)
train_load_2 = DataLoader(dataset=train_dataset_2, batch_size=100, shuffle=False)
train_load_3 = DataLoader(dataset=train_dataset_3, batch_size=100, shuffle=False)

test_load_1 = DataLoader(dataset=test_dataset_1, batch_size=100, shuffle=True)
test_load_2 = DataLoader(dataset=test_dataset_2, batch_size=100, shuffle=True)
test_load_3 = DataLoader(dataset=test_dataset_3, batch_size=100, shuffle=True)

class Net(nn.Module): 
   def __init__(self):
      super(Net, self).__init__()
      self.conv = nn.Conv1d( ... )  
      self.fc1 = nn.Linear( ... )  
      self.fc2 = nn.Linear( ... )  

   def forward(self, x1, x2, x3): 
      o1 = self.conv(x1)
      o2 = self.conv(x2)
      o3 = self.conv(x3)
      combined = torch.cat((o1.view(c.size(0), -1),
                            o2.view(c.size(0), -1),
                            o3.view(c.size(0), -1)), dim=1)
      out = self.fc1(combined)
      out = self.fc2(out)
      return F.softmax(x, dim=1)

model = Net().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.01)

for epoch in epochs: 
   model.train()</code></pre>
Custom_collate_fn is not called in dataloader

alireza_samadi: <p>I have written a collate function to pad and stack data in the batch. when I call the custom_collate_function simply as a function passing a list of inputs , it performs well and outputs what I expect ( it can also stack the entire dataset all together ). However, when it is called in the data loader, it says it cannot stack entries. here is my custom_collate_fn:</p>
<pre><code class="lang-auto">
from torch_geometric.data import Batch

def collate_my_dataset(samples):
    
    batch_seq_lens = [data[0].size(0) for data in samples ]
    max_seq_len = max(batch_seq_lens)

    token_list = [data[0] for data in samples ]
    labels = [data[1] for data in samples ]

    tokens_padded = []
    for t in token_list:
        if t.shape[0] &lt; max_seq_len :
            padding = torch.zeros((max_seq_len - t.shape[0], t.shape[1])).to(device)
            tokens_padded.append(torch.cat([t, padding], 0))
        else:
            tokens_padded.append(t)
    tokens = torch.stack(tokens_padded, dim=0)

    return Batch(tokens = tokens, labels= torch.tensor(labels).unsqueeze(-1))

</code></pre>
<p>and here is how I call the dataloader :</p>
<pre><code class="lang-auto">from  torch_geometric import loader
batch_size = 8

div_threshold = int(tu_dataset.__len__()*0.8)
train_dataset = tu_dataset[: div_threshold ]
test_dataset = tu_dataset[div_threshold:]

train_loader = loader.DataLoader(train_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_my_dataset)
batch = next(iter(test_loader))
</code></pre>
<p>the error is something like :<br>
stack expects each tensor to be equal size, but got [31, 64] at entry 0 and [36, 64] at entry 1</p>
<p>PS1: as mentioned what the collate function outputs is what I expect.<br>
PS2 I had the similar question in another topic , but it was in PyG which by nature I couldn’t use PyG dataloader anymore and I have to use native Pytorch dataloader.</p>

ptrblck: <p>Isn’t this the same issue as described in your <a href="https://discuss.pytorch.org/t/collate-function-is-not-called-in-dataloader/171726/2">previous post</a>?<br>
In your code snippet it seems you are still using the PyG <code>DataLoader</code>:</p>
<pre><code class="lang-python">from  torch_geometric import loader
...
train_loader = loader.DataLoader(train_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_my_dataset)
</code></pre>

alireza_samadi: <p>yes, i noticed it immediately after posting the issue. I should not have used PyG. thank you</p>
How to modify my training data (augmentation) while the model is training?

stv: <p>Hi. I want to know if there is a way in which I can train a model in Pytorch, look at the metrics of the model in each image (since my dataset is small), and at a given specific epoch, modify my training data (augmenting the problematic cases specifically), and continue the training? I would really appreciate any kind of help or advice on how to achieve this.</p>

nkdatascientist: <p>Hi <a class="mention" href="/u/stv">@stv</a>,<br>
To augment the dataset while training</p>
<ol>
<li>First technique <a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">torchvision.transforms</a> like flipping, adding noise,…,etc.</li>
<li>Second technique  is <a href="https://github.com/albumentations-team/albumentations" rel="noopener nofollow ugc">Alubumentation</a>. This is my favorite one.</li>
</ol>
<p>Example Implementation: <a href="https://towardsdatascience.com/getting-started-with-albumentation-winning-deep-learning-image-augmentation-technique-in-pytorch-47aaba0ee3f8" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Albumentation: Deep Learning Image Augmentation Technique in PyTorch | by Kevin Bok | Towards Data Science</a><br>
Thanks.</p>

stv: <p>Thank you. However, in the example you indicate the augmentations are all done before training, so this does not answer to my question.</p>
Multiprocessing with shared cuda stream

Alex_Klein: <p>Hello, I am trying to generate multiple Processes that each have their own cuda stream and are able to sync to a main process.<br>
The goal is to have each stream receive camera images, shove it on the gpu, do some preprocessing and then provide the data to the main process for a machine learning application.<br>
I found an example here that works with multiple streams and multiprocessing here:</p>
<aside class="quote quote-modified" data-post="1" data-topic="108363">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mamdouh_aljoud/48/29609_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-run-multiprocessing-with-cuda-streams/108363">How to run multiprocessing with cuda streams</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I am trying to run two cuda streams in parallel, I initiate the streams then use them to run computations in the processes. The problem I have is that the processes are not firing. i.e., thecode is not executed inside the processes. 
Please refer to the code below. 


from torch.multiprocessing import Process, set_start_method
import torch
import time


stream1 = torch.cuda.Stream()
stream2 = torch.cuda.Stream()
torch.cuda.synchronize()
def process1():
    global stream1, stream2
    with torch.…
  </blockquote>
</aside>

<p>The example just creates new streams for every process, due to the fact that <code>spawn</code> start method reinitialize the Stream on each process.</p>
<p>What I need is shared stream that I can sync, so I know when I can acces the data after calculations are finished to avoid race conditions.</p>
<p>I tried to use <code>Stream.cuda_stream</code> to pass the pointer to an <code>ExternalStream</code> as seen in the example below but get the error:<br>
<code>RuntimeError: CUDA error: invalid resource handle</code></p>
<p>Here is my toy example:</p>
<pre><code class="lang-auto">import os
import time
import torch
from torch.multiprocessing import Process, set_start_method

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
set_start_method("spawn", True)

def process(stream_ptr, data):
    stream = torch.cuda.ExternalStream(stream_ptr, "cuda:0")
    with torch.cuda.stream(stream):
        data[:] = data * 2

if __name__ == "__main__":
    stream = torch.cuda.Stream(device='cuda:0')
    with torch.cuda.stream(stream):
        data = torch.ones((2, 2), device='cuda:0')
        data.share_memory_()

    p1 = Process(target=process, args=(stream.cuda_stream, data))
    p1.start()
    timeout_start = time.perf_counter()
    while time.perf_counter() - timeout_start &lt; 10:
        print(data)
        time.sleep(1)
    p1.join()
</code></pre>
gpytorch performs very poorly when the data has different distributions

Kai1: <p>Hi everyone, before I delve deeper into my problem, I want to clarify that I’m using a simple RBF Kernel and a Gaussian likelihood function for the gpytorch model. We also have an embedding network that transforms a data triplet (explained later) into a latent version, which then goes into the Gaussian process. For simplicity, my current dataset consists of discounts where each discount was offered to customer (identified by customerID) that bought a product from a specific product group (PG). Each PG includes discounts offered to customers who bought specific products. The Gaussian Process is used to predict multiple features based on the CustomerId PG combination, such as discounts. To achieve this, I provide the embeddings with a triplet consisting of the CustomerId PG combination and an index for the feature that needs to be predicted. For example, the feature index 18 predicts the discount. So, when I give the GP the triplet (2102, XO, 18), the GP should predict the optimal discount for the customer 2102 who buys a product from the product group XO.</p>
<p>We now encountered a problem with gpytorch. Although the model performed well with GPflow (which is strange), the performance with gpytorch was poor. Gpytorch performs well when the distribution of discounts for PGs is very similar. For instance, when we use a subset of PGs where the discount is distributed ONLY as two Gaussian models (a Gaussian mixture) with similiar variance and mean values, gpytorch performs well. But if we have other PGs that have discount distributions that show only one Gaussian process, a strong variety in variances or mean values, or distributions that look more like log-gamma distributions, the Gaussian process performs very poorly.</p>
<p>Gpytorch’s Gaussian process struggles to handle different distribution types. We observed much better results when we trained individual models for each PG based on its distribution type. However, we face the challenge of combining all PGs into one dataset while still achieving good results. How can we optimize gpytorch to handle more complex tasks? Are there any specific techniques or parameters that we should focus on? I would greatly appreciate any ideas or suggestions.</p>
<p>Regards Kai</p>
Cuda installation error

MOHD_SADIK: <p>while using pip to install pytorch with cuda it shows all reaquirment satisfied but in jupyter while running command torch.cuda.is_available it shows False</p>

ptrblck: <p>Check if your Jupyter environment uses a different PyTorch binary and make sure <code>pip list | grep torch</code> shows a PyTorch version with a CUDA tag.</p>
Transfer Learning With Swint

jagac: <p>I’m trying to do transfer learning with swin_t. I ran the following code to hopefully freeze the layers and add an output one like this:</p>
<pre><code>    weights = Swin_T_Weights.DEFAULT
    model = swin_t(weights=weights)

    for param in model.parameters():
        param.requires_grad = False

     model.fc = nn.Linear(768, 37)
     model = model.to(device)
</code></pre>
<p>However when I try training I get the following error<br>
“Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”</p>
<p>Not sure what’s wrong. Resnet50 worked fine using the same code.</p>
<p>Would the following work?</p>
<pre><code>      for m in model.modules():
          if isinstance(m, nn.Linear):
              m.requires_grad_ = False
</code></pre>

ptrblck: <p><code>swin_t</code> does not have a <code>.fc</code> attribute and you are thus creating it as a new and unused attribute:</p>
<pre><code class="lang-python">model = torchvision.models.swin_t()
model.fc
AttributeError: 'SwinTransformer' object has no attribute 'fc'

model.fc = nn.Linear(768, 37)
model.fc
# Linear(in_features=768, out_features=37, bias=True)
</code></pre>
<p>I guess you might want to replace <code>model.head</code>.</p>
How i step into a hook function when debugging my code

sqLin: <p>The code adds hooks to the backward pass. When i debugging the code, it just ignore the breakpoints i set in the hook functions.<br>
My IDE is pycharm.<br>
I want to know if there’s any method to get into the hook function when debugging?<br>
Any suggestions will greatly appreciated.</p>

sqLin: <p>Solved the problem by using pdb(The Python Debugger) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
How to change Conv2d.weights.data shape?

divided-by-7: <p><strong>code:</strong><br>
class cnn(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(cnn, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 5, 3, 1, 1)<br>
def forward(self, x):<br>
x = self.conv1(x)<br>
return x</p>
<p>model = cnn()</p>
<p>model.conv1.weight.data = torch.cat([model.conv1.weight.data, torch.rand([1, 3, 3, 3])], dim=0)</p>
<p>model.conv1.bias.data = torch.cat([model.conv1.bias.data, torch.ones(1, dtype=torch.float32)], dim=0)<br>
model.conv1.out_channels = 6</p>
<blockquote>
<blockquote>
<blockquote></blockquote>
</blockquote>
</blockquote>
<p>I can get the correct output shape by y=model(x)，<br>
but I meet some error in backward:RuntimeError: Function ThnnConv2DBackward returned an invalid gradient at index 1 - got [6, 3, 3, 3] but expected shape compatible with [5, 3, 3, 3]<br>
I also find some relative Q&amp;A, it’s seem like autograd can’t get the modify in “weights.data” and “bias.data”, but I want to keep the original information, only add convolution core.What should I do?</p>

J_Johnson: <p>Try this:</p>
<pre><code class="lang-auto">cnn=nn.Conv2d(3,5,3,1,1)
print(cnn.weight.data.size())
with torch.no_grad():
    cnn.weight.data =torch.rand(6,3,3,3)
    cnn.bias.data=torch.rand(6)

x=cnn(torch.rand(32,3,32,32))
criterion=nn.MSELoss()
opt=torch.optim.Adam(cnn.parameters())
opt.zero_grad()

loss=criterion(x, torch.rand_like(x))

loss.backward()
opt.step()
</code></pre>
<p>You can also set the output size, but it doesn’t check for that.</p>
<p>Note that you need to reinitialize the optimizer.</p>
GPU MinMaxScaler, training and inference?

eleven: <p>The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html" rel="noopener nofollow ugc">sklearn MinMaxScaler</a> works great. After it is fit() to the data, transform() can be called to normalize data before passing to my torch model for inference, and inverse_transform() after the inference call to get back to original (unscaled) data range.</p>
<p>This does not work with GPU however.  What is the best way to use MinMaxScaler backed by GPU calcluations for fit(), transform(), and inverse_transform() without messing up training? I am using an Adam optimizer and an unsure about including or not including MinMaxScaling in backward() for adjusting weights.</p>

ptrblck: <p>You could re-implement the <code>MixMaxScaler</code> in pure PyTorch and use it as a transformation in your <code>Dataset</code>.</p>
<aside class="quote no-group" data-username="eleven" data-post="1" data-topic="172647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eleven/48/48695_2.png" class="avatar"> eleven:</div>
<blockquote>
<p>I am using an Adam optimizer and an unsure about including or not including MinMaxScaling in backward() for adjusting weights.</p>
</blockquote>
</aside>
<p>I don’t understand this point, as the <code>MinMaxScaler</code> is used to normalize the input data and does neither change any weights nor is it used in the backward pass.</p>
Using a target size (torch.Size([100, 1])) that is different to the input size (torch.Size([100, 100])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size

Mathedelic: <p>I am running the following code for a simple problem, and the issue is that my outputs shape depends on the batch size which is causing issues and giving the warning. I am predicting a single scalar using a vector of dimension 26<br>
Please can you guide me where am I making a mistake in this issue :</p>
<pre><code class="lang-auto">df = pd.read_csv(r'D:\SAA_prime\saa\new_dataset1.csv')
df.shape # [40964,26]
scr = pd.read_csv('D:\SAA_prime\saa\scr_new_data1.csv')
scr.shape #(40964, 1)
inputs = torch.from_numpy(np.array(df.iloc[:, :-1])).type(torch.float32)
inputs.shape #torch.Size([40964, 25])
targets = np.array(scr).reshape(40964,1)
targets = torch.from_numpy(targets).type(torch.float32)
targets.shape#torch.Size([40964, 1])
dataset = TensorDataset(inputs, targets)
train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])
batch_size = 10
trainloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
class scr_model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.W = torch.nn.Parameter(torch.randn((25,25)))
        self.m = torch.nn.Parameter(torch.randn((25,1)))
        self.b = torch.nn.Parameter(torch.randn((1)))

    def forward(self, x):
        return (self.b + x @ self.m  +  x @ self.W @ x.t())  
    
model = scr_model()

for x,y in trainloader:
    preds = model(x)
    print("Prediction is :\n",preds.shape)
    print("\nActual targets is :\n",y.shape)
    break
</code></pre>
<p>Prediction is :<br>
torch.Size([10, 10])</p>
<p>Actual targets is :<br>
torch.Size([10, 1])</p>
<p>I have tried to debug this code many times from different angles but I am unable to find the cause of the error. Any help will be much appreciated!</p>

ptrblck: <p>Cross-post from <a href="https://discuss.pytorch.org/t/custom-linear-regression-layer/169512/9">here</a> with a clarifying question.<br>
It’s currently unclear how you are interpreting the used operation as you are explicitly creating the batch size - dependent output in your model.</p>
Training cascade model

mangoxb: <p>Hello, I want to train cascade separate model.</p>
<ol>
<li>train model A (no problem)</li>
<li>train cascade model<br>
(a) input-&gt;model A  → output of A  → model B → output of B<br>
(b) some layer weights of A and B are shared.<br>
(c) loss(output of A, target) + loss(output of B, target)<br>
(d) the gradient should have flowed from B to A.<br>
it results in gradient explosion.</li>
</ol>
<p>I couldn’t find what causes this problem. I also found similar question. However, I didn’t get the answer. (<a href="https://discuss.pytorch.org/t/strategies-to-debug-exploding-gradients-in-pytorch/51975/7" class="inline-onebox">Strategies to debug exploding gradients in pytorch - #7 by mangoxb</a>)<br>
I need your help or any advice.</p>
Conv1d custom dilation value based on list (non-constant dilation)

alexmehta: <p>Let’s say I have a tensor <code>[1,2,3,4,5,6]</code> and I want to apply kernel of <code>[0,1]</code> to it using conv1d. The slight twist, I want my dilation value to not be constant. In other words, dilation might be equal to <code>[0,1,2]</code> where they are dilation values.</p>
<p>This would yield the following steps (with a stride of 1 and ignoring padding):</p>
<pre><code class="lang-auto">Step 1: [*0,2*,3,4,5]
Step 2: [0,*0*,3,*4*,5,6]
Step 3: [0,0,*0*,4,5,*6*]
</code></pre>
<p>Would I have to modify nn.Conv1d to make this possible? If so, how do I add this functionality without a for loop? Would I have to write external CUDA code to make it run in similar speed to dilation convs usually do?</p>
Gpytorch-- want to modify the forward function from a source code

Huzaifa_Capricorn: <p>I am trying to use RFF kernel from the gpytorch library &lt; <a href="https://gpytorch.ai" rel="noopener nofollow ugc">https://gpytorch.ai</a> &gt;.</p>
<p>The source code can be found below:</p>
<p><a href="https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/rff_kernel.py" rel="noopener nofollow ugc">https://github.com/cornellius-gp/gpytorch/blob/master/gpytorch/kernels/rff_kernel.py</a></p>
<p>This is basically an approximation to the RBF kernel.</p>
<p>The forward function looks like this:</p>
<pre><code>    if last_dim_is_batch:
        x1 = x1.transpose(-1, -2).unsqueeze(-1)
        x2 = x2.transpose(-1, -2).unsqueeze(-1)
    num_dims = x1.size(-1)
    if not hasattr(self, "randn_weights"):
        self._init_weights(num_dims, self.num_samples)
    x1_eq_x2 = torch.equal(x1, x2)
    z1 = self._featurize(x1, normalize=False)
    if not x1_eq_x2:
        z2 = self._featurize(x2, normalize=False)
    else:
        z2 = z1
    D = float(self.num_samples)
    if diag:
        return (z1 * z2).sum(-1) / D
    if x1_eq_x2:
        # Exploit low rank structure, if there are fewer features than data points
        if z1.size(-1) &lt; z2.size(-2):
            return LowRankRootLinearOperator(z1 / math.sqrt(D))
        else:
            return RootLinearOperator(z1 / math.sqrt(D))
    else:
        return MatmulLinearOperator(z1 / D, z2.transpose(-1, -2))
</code></pre>
<p>I only want access to  z1  but a simple modification of this file to return z1 like:</p>
<p>def forward(self, x1: Tensor, x2: Tensor, diag: bool = False, last_dim_is_batch: bool = False, **kwargs) → Tensor:<br>
if last_dim_is_batch:<br>
x1 = x1.transpose(-1, -2).unsqueeze(-1)<br>
x2 = x2.transpose(-1, -2).unsqueeze(-1)<br>
num_dims = x1.size(-1)<br>
if not hasattr(self, “randn_weights”):<br>
self._init_weights(num_dims, self.num_samples)<br>
x1_eq_x2 = torch.equal(x1, x2)<br>
z1 = self._featurize(x1, normalize=False)<br>
return z1</p>
<p>This gives error b/c it is expecting output of dimension torch.matmul(z1.T,z1) and NOT z1.</p>
<p>Please can someone help</p>
CVAT SDK PyTorch adapter: using CVAT datasets in ML pipeline

mdacoca: <p>Hi</p>
<p>We created a tutorial about the Pytorch Adapter and how to use it in CVAT:</p>
<p><a href="https://cvat.webflow.io/post/cvat-sdk-pytorch-adapter" class="onebox" target="_blank" rel="noopener nofollow ugc">https://cvat.webflow.io/post/cvat-sdk-pytorch-adapter</a></p>
<p>A couple of words about what is it:</p>
<p>The PyTorch adapter, also known as the cvat_sdk.pytorch module allows using a CVAT project or task directly as a PyTorch-compatible dataset, eliminating the need for intermediate representation.</p>
<p>The article provides an overview of the PyTorch adapter and its functionality, followed up by some real-life use cases and examples for a better understanding of how it works.</p>
Gradient not passing through sparsity regularization

kkladny: <p>I would like to add a sparsity regularization to the encodings of my VAE. This means I set an activation vector to 1 for all indices where the encoding is not 0 and then take the mean over the batch dimension to get a distribution. Then I can take a pointwise kullback leibler divergence to the desired sparsity probability.</p>
<p>However, the output of the function does not possess a <code>grad_fn</code> attribute. So I guess it does not propagate the gradient back. Here is a minimal working example with the function I use to regularize.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

def sparsity_regularizer(enc, sparsity=.05):
    activations = torch.zeros(size=enc.size())
    # mean activation
    activations[torch.nonzero(enc, as_tuple=True)] = 1
    # take mean along batch dimension
    mean = torch.mean(activations, dim=0)
    reg = -F.kl_div(mean.log(), sparsity*torch.ones(size=mean.size()), reduction="sum")
    return reg

enc = torch.Tensor([0, 2])
enc.requires_grad = True

out = sparsity_regularizer(enc)

print(out)
</code></pre>
<p>What do I need to change?</p>

Grant_Norman: <p>The zero “norm” function: counting up the number of zeros (or nonzeros), is not differentiable. It has a very discontinuous behavior at the point where the entry is zero, so even if you get the PyTorch working, it likely won’t behave as you expect. Maybe try looking at the l1-norm, which is related to sparsity, but is differentiable (almost everywhere).</p>