akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Batchnorm with multiple features and multiple channels

jame: <p>I want to perform a multi-input, multi-output regression using convolution layers. For this the input and output data has the form [N, F_{in/out}] with N: number of samples, F_{in/out}: number of input / output features.</p>
<p>Consequently, within my network I have activations of the form [B, C, F] with B: batch_size, C: channels, F: features to which I want to apply a Batchnorm layer.</p>
<p>I guess the correct way of applying a Batchnorm layer in this setting is to normalize each feature for each channel individually.</p>
<p>However neither torch.nn.Batchnorm1d nor torch.nn.Batchnorm2d fit these needs.</p>
<p>I am wondering how this could be achieved most intuitively, as I guess this is a standard application of the Batchnorm layer.</p>

LambdaWill: <p>Using InstanceNorm gives what you what.</p>
Why we right shift the decoder input for time series forecasting with transformers

Spencerfonbuena: <p>I have been making an effort to implement a vanilla transformer to do time series forecasting. In the normal transformer, it seems like the usual to input the right shifted encoder input into the decoder. For time series however, it seems to me like it would make sense to make the input of the decoder, the last input of the encoder plus your forecasting horizon. For example</p>
<p>Encoder Input: 1, 2, 3, 4<br>
Decoder Input: 4, 5, 6, 7, 8, 9, 10, …</p>
<p>However, I’m not sure if this is the way to do it, or if I would be better of making the decoder input be 2, 3, 4, 5, 6, …etc. (right shifted)</p>
<p>I guess my question comes from a place of ignorance on how the decoder actually works. if the encoder-decoder attention is masked, then it makes sense to me to right shift, as it would be using just the previous time series to predict the next, and only have access to limited encoder information.</p>
<p>However, if the decoder has access to the entire encoder output, it makes sense to me to do it the first way. In this manner, the encoder decoder attention mechanism has access to the entire previous series, as it attempts to predict from time t + 1(the end of the encoder sequence) to time t + p. (p being the prediction window). I apologize if this question is unclear. If it is, let me know how and I will try to edit it in a better way</p>
Empty tensors from NTXentloss

Anesh_Muthiah_A: <p>I am using NTXentloss from pytorch-metric-learning library. I am getting empty tensor when passing throught NTXentloss and getting this error <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>. I am trying to use Supervised contrastive loss.</p>
<pre><code class="lang-auto">class SupervisedContrastiveLoss(nn.Module):
    def __init__(self, temperature=0.1):
        super(SupervisedContrastiveLoss, self).__init__()
        self.temperature = temperature

    def forward(self, feature_vectors, labels):
        # Normalize feature vectors
        #feature_vectors.requires_grad=True
        feature_vectors_normalized = F.normalize(feature_vectors, p=2, dim=1)
        print(feature_vectors_normalized.shape)
        # Compute logits
        logits = torch.div(
            torch.matmul(
                feature_vectors_normalized, torch.transpose(feature_vectors_normalized, 0, 1)
            ),
            self.temperature,
        )
        labels=torch.reshape(labels,(-1,))
        print("logits",logits.shape)
        return losses.NTXentLoss(temperature=0.07)(logits,labels )
</code></pre>
Very low GPU power usage

nibtehaz: <p>Hi, I am training a custom Vision Transformer model on ImageNet. I am using 4 V100 GPUs. Although my GPU utilization is high the power being consumed by the GPU is quite small. Is this normal?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db4c063e5f50fc90499ee35f7b54e5d7b0124f1b.png" data-download-href="https://discuss.pytorch.org/uploads/default/db4c063e5f50fc90499ee35f7b54e5d7b0124f1b" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db4c063e5f50fc90499ee35f7b54e5d7b0124f1b.png" alt="Capture" data-base62-sha1="vhZrKjbhRRNqcmE2PHajUIRSwgr" width="690" height="440" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">903×577 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Initially the training is quite fast, but with time the GPU power usage falls and the training becomes slow.</p>
<p>Is this a problem with my hardware setup? Thank you for your kind help.</p>

ptrblck: <p>You could check if your GPUs are reducing their clocks as they might otherwise overheat. If so, you might want to check the cooling solution and improve it.<br>
Also, you could profile your use case (e.g. via <code>Nsight Systems</code>) to see if the GPUs might be bottlenecked by other code parts and might be waiting (e.g. in a NCCL call).</p>
Pytorch CUDA. Why it's loading CPU?

Dionysus: <p>Why the code is loading my CPU at 17% ? I thought the code in “with” block is executing on the GPU.</p>
<pre><code class="lang-auto">device = torch.device("cuda:0")

size = (50, 50)

x = torch.rand(size, dtype=torch.float32, device=device)

y = torch.rand(size, dtype=torch.float32, device=device)

with torch.cuda.device(device):
    torch.cuda.synchronize()
    for _ in range(1000000):
        result = torch.add(x, y)
</code></pre>
<p>There was no such problem if “for” iterations is much more less but tensors are much more bigger.</p>
<p>GPU RTX 2070</p>

ptrblck: <p>Your Python code is still being executed on the CPU, which will dispatch to the CUDA kernel and launch it, so a CPU utilization is expected.</p>
Parallel Training of Multiple Models

colinlaganier: <p>I am trying to train N independant models using M GPUs in parallel on one machine. What I currently want to achieve is training the N models, M at a time in parallel for given number of epochs, store the intermediate return output of each model until all are done, process the stored outputs and repeat for a number of rounds.</p>
<p>Each <code>client</code> has a <code>device</code> property with a GPU id, and model parameters are assigned to the device before training. The <code>device_dict</code> dictionary has one key for each gpu containing a list of client ids assigned to the device. Here is what I have implemented so far (untested) and am unsure if that is the best way of doing this.</p>
<pre><code class="lang-auto">def train_mp(self, num_rounds, train_epochs):

    # Initialize logit queue for server update after each round
    logit_queue = Queue()

    for _ in num_rounds:
        self.round += 1

        diffusion_seed = self.server.generate_seed()
        server_logit = self.server.get_logit()
        processes = [] 

        # Start processes for each client on each device
        for i in range(math.ceil(self.num_clients / self.num_devices)):
            for device, client_ids in self.device_dict.items():
                if i &lt; len(client_ids):
                    process = mp.Process(target=self.client_update, args=(self.clients[client_ids[i]], server_logit, diffusion_seed, logit_queue))
                    process.start()
                    processes.append(process)

        # Wait for all processes to finish
        for process in processes:
            process.join()

        # Update server model with client logit queue
        self.server.knowledge_distillation(logit_queue)
</code></pre>
<p>I currently do not have access to a multi-GPU machine to test anything so am unsure what the best way of doing this would be. Any help would be appreciated.</p>
PyTorch breaks problem symmetry due to numeric precision

dherrera1911: <p>I have a symmetric matrix <strong>B</strong>, and I am applying a quadratic operation to the matrix with a second matrix <strong>f</strong>, of the form <strong>C</strong> = <strong>f’</strong> <strong>B</strong> <strong>f</strong>. The result of this operation should also be a symmetric matrix. However, due to numeric precision, my implementation gives a result that is not symmetric, but deviates from symmetry very slightly. This is an issue for later processes that require that the resulting matrix is symmetric (actually, symmetric positive definite).</p>
<p>I wonder if there’s an elegant way to have the result stay symmetric, i.e. not having to copy-paste the lower-triangular matrix into the upper triangular part of the matrix.</p>
<p>Code that replicates this problem:</p>
<pre><code class="lang-auto">
# Generate a SPDM
import torch
n = 400
d = 5
A = torch.randn(n, n)
B = torch.matmul(A, A.transpose(0,1)) * 0.01  # Original symmetric matrix
f = torch.randn(d, n) * 0.1  # Random filters matrix
C = torch.einsum('kd,db,mb-&gt;km', f, B, f)  # Compute the quadratic form 

print(f'Symmetric residue of original matrix: \n {B - B.transpose(0,1)}')
print(f'Symmetric residue of quadratic formula result: \n {C - C.transpose(0,1)}')

</code></pre>
<pre><code class="lang-auto">Symmetric residue of original matrix: 
 tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]])
Symmetric residue of quadratic formula result: 
 tensor([[ 0.0000e+00,  3.5763e-07, -1.1921e-07,  2.3842e-07, -4.7684e-07],
        [-3.5763e-07,  0.0000e+00, -2.9802e-07, -4.7684e-07, -1.1921e-07],
        [ 1.1921e-07,  2.9802e-07,  0.0000e+00, -1.7881e-07,  7.1526e-07],
        [-2.3842e-07,  4.7684e-07,  1.7881e-07,  0.0000e+00, -4.7684e-07],
        [ 4.7684e-07,  1.1921e-07, -7.1526e-07,  4.7684e-07,  0.0000e+00]])

</code></pre>
Optimizing concatenation of tensors

PabloVD: <p>I’m dealing with concatenation of tensors in cpu and gpu, and I’m trying to optimize the performance of the process.</p>
<p>Which option should be faster/preferred (if any)?</p>
<pre><code class="lang-python">torch.cat([x.view(1,3) for x in array],dim=0).to(device)
</code></pre>
<p>or</p>
<pre><code class="lang-python">torch.cat([x.view(1,3).to(device) for x in array],dim=0)
</code></pre>
<p>where <code>device="cuda"</code>.</p>
<p>In other words, is faster to employ <code>torch.cat</code> with tensors in cuda, or should I rather move to cpu after the concatenation? Does it depend on the specific case?</p>
<p>Many thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="PabloVD" data-post="1" data-topic="183438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pablovd/48/46511_2.png" class="avatar"> PabloVD:</div>
<blockquote>
<p>Does it depend on the specific case?</p>
</blockquote>
</aside>
<p>Yes, as usual.</p>
<p>Triggering a single copy kernel should be faster than calling them in a loop. However, in your use case it would also depend on the available GPU memory as the <code>torch.cat</code> call would increase it assuming <code>x</code> resides on the GPU and <code>device</code> is set to <code>"cpu"</code>.</p>
<p>In the end it would also depend on the tensor shapes, number of tensors, etc., so you might want to profile your actual workload.</p>

PabloVD: <p>Thanks for the reply, thus I will perform a detailed profiling to check which suits better to my use case.<br>
Many thanks!</p>
Getting attention scores for each head of transformer model

amjass: <p>Hi all,<br>
I am trying to get attention weights for input sentences for a transformer model I have built with my own corpus of text. I have used the PyTorch <a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Language Translation with nn.Transformer and torchtext — PyTorch Tutorials 2.0.1+cu117 documentation</a> as a guide and the model trains and predicts well.</p>
<p>I now need to access attention weights of the encoder but am struggling and would appreciate any guidance. I need the attention scores for an input sentence for each head of the encoder, and the final attention weights of the last layer of the encoder block.<br>
I have managed to access each layers attention score as follows:</p>
<pre><code class="lang-auto">src = text_transform['main'](src).view(-1, 1)
num_tokens = src.shape[0]
src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool)
input_embeddings = transformer.encode(src, src_mask)

for layer in transformer.transformer.encoder.layers:
     emb, att = layer.self_attn(input_embeddings, input_embeddings, input_embeddings, need_weights=True)
     input_embeddings = layer(input_embeddings)
</code></pre>
<p>but this only provides attention scores for each layer of the encoder block and presumably only for the first head as i only get 6 sets of attention (unless they are averaged across the heads which is still not what i need)</p>
<p>I would like the final attention score for each encoder block and for each head. Any advice would be much appreciated.</p>
<p>many thanks!</p>
New Overview section in documentation

davidradl: <p>I am new to PyTorch and what would have been useful to me was an overview of the project, at level higher than the code.</p>
<p>I suggest we have a new Overview / concepts section in the documentation. That would include:</p>
<ol>
<li>An overview of a typical PyTorch application, main and training data, model creation, loss and optimizer , training loop and test. I know these are also standard ML concepts, but I think it would help as they tie to specific PyTorch named concepts…</li>
<li>An overview of the PyTorch project in terms of python modules, models, C++ libraries, overview of the generation</li>
<li>An overview of the packaging and how the C++ and python libraries sit together.</li>
<li>API overviews.</li>
<li>Compiler overview contrasting the newer,  older and recommended approaches.</li>
<li>GPU acceleration overview</li>
<li>Testing overview and approaches.</li>
<li>Any other overview perspective that might be useful</li>
<li>A starter Glossary for main terms to be defined and linked to as the term is used</li>
</ol>
<p>The above should contain diagrams of the concepts and also link down to the existing doc. I hope that such an overview would make it more approachable to beginners and paint the big picture.</p>
<p>Please feedback if you think this would be useful. I am happy to put up initial content for it to be reviewed.</p>

davidradl: <p>I have raised this as an issue <a href="https://github.com/pytorch/pytorch/issues/104623" class="inline-onebox" rel="noopener nofollow ugc">I propose a new overview section in the documentation · Issue #104623 · pytorch/pytorch · GitHub</a> so it does not get missed</p>
Custom transform for augmenting image using corresponding mask in detectron2

ravijoshi: <p>In the process of data augmentation in detectron2, I am trying to modify the image based on the corresponding mask. Basically, I need to get the background from the image, which requires knowing the foreground (mask) in advance.</p>
<p>Therefore, I am looking for a <code>Transform</code> that can provide image and mask as input to my function. For example, previously, I used <a href="https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.ColorTransform" rel="noopener nofollow ugc">ColorTransform</a>, which takes a callable and provides an image as input to this callable. See an example below:</p>
<pre><code class="lang-auto">def apply_blur(img):
  blur_img = cv2.GaussianBlur(img, (3, 3), 0)
  return blur_img

class MyTrainer(DefaultTrainer):
  @classmethod
  def build_train_loader(cls, cfg):
    augmentations = [
        T.RandomFlip(),
        T.ColorTransform(apply_blur),
    ]
    mapper = DatasetMapper(cfg, is_train=True, augmentations=augmentations)
    return build_detection_train_loader(cfg, mapper=mapper)
</code></pre>
<p>I found <a href="https://detectron2.readthedocs.io/en/latest/modules/data_transforms.html#detectron2.data.transforms.AugInput" rel="noopener nofollow ugc">AugInput</a>, but it expects image, mask, etc. as input. Instead, is there something like the following exists?</p>
<pre><code class="lang-auto">def my_transform(img, mask):
  # do something on image using mask and return image
  return img

  ...
  def build_train_loader(cls, cfg):
    augmentations = [
        T.RandomFlip(),
        T.CustomTransform(my_transform),
    ]
  ...
</code></pre>
<p>The question is cross-posted on <a href="https://github.com/facebookresearch/detectron2/discussions/5000" rel="noopener nofollow ugc">GitHub</a>, but the discussions on the  <a href="https://github.com/facebookresearch/detectron2/discussions" rel="noopener nofollow ugc">GitHub page</a> seem dead. I apologize for the inconvenience.</p>
<p>Thank you very much.</p>

ravijoshi: <p>Any hints, please? Thank you very much.</p>
Momentum causes NAN in batchnorm

OMnia: <p>Sorry if this isn’t right forum, I couldn’t find a topic that fits my question.</p>
<p>I created a simple NN class for fully connected layers with the option of adding batch normalization:</p>
<pre><code class="lang-auto">class FC_batchnorm_net(nn.Module):
    def __init__(self, input_sizes, output_size, batchnorm = False, activation = 'ReLU'):
        super().__init__()
        self.n_layers = len(input_sizes)
        self.fc_layers = nn.ModuleDict({'fc'+str(i): nn.Linear(input_sizes[i-1], input_sizes[i])
                          for i in range(1,self.n_layers)})
        self.fc_layers['fc'+str(self.n_layers)] = nn.Linear(input_sizes[self.n_layers-1], output_size)
        if batchnorm == False:
            self.batchnorm = False
        else:
            if batchnorm == True:
                self.batchnorm = nn.ModuleDict({'bn'+str(i): nn.BatchNorm1d(input_sizes[i])
                          for i in range(1,self.n_layers)})
            else:
                if len(batchnorm) != len(input_sizes):
                    print("If bacthnorm isn't set to True of False, it must be a boolean array of the same length as input_size!")
                    return
                self.batchnorm = nn.ModuleDict()
                for i in range(1,self.n_layers):
                    if batchnorm[i]:
                        self.batchnorm['bn'+str(i)] = nn.BatchNorm1d(input_sizes[i])
        if activation == 'ReLU':
            self.activation = nn.ReLU()

    def forward(self, x):
        out = x
        out = self.fc_layers['fc1'](out)
        if self.batchnorm != False:
            if 'bn1' in self.batchnorm.keys():
                out = self.batchnorm['bn1'](out)
        out = self.activation(out)

        for i in range(2, self.n_layers):
            out = self.fc_layers['fc'+str(i)](out)
            if self.batchnorm != False:
                if 'bn'+str(i) in self.batchnorm.keys():
                    out = self.batchnorm['bn'+str(i)](out)
            out = self.activation(out)
        out = self.fc_layers['fc'+str(self.n_layers)](out)
        return out
</code></pre>
<p>I created a basic network using this, with 3 FC layers and a BN layer for the two inner layers. However, when I try to train the network with a momentum higher than 0.1, the parameters of the second BN layers get NANs after a few batches, and I cannot figure out why. The inputs are simply vectors of normalized floats. Any ideas?</p>

ptrblck: <p>I haven’t seen such an effect before, but you could check the input activation stats during training and see if these are exploding.<br>
Using a higher <code>momentum</code> value would increase the importance of the current activation stats while updating the running stats, so it seems these values might be large.</p>

OMnia: <p>Thanks. For now, I bypassed the issue by doing a hyperparameter search from scratch after adding the batchnorm layers, but I’ll keep that in ind if I run into the issue again.</p>
Various quantized/gemm backends scopes

vadimkantorov: <p>There a bunch of backends / dependencies supported by PyTorch for different cases:</p>
<ul>
<li>fbgemm</li>
<li>fbgemm_gpu</li>
<li>nnpack</li>
<li>qnnpack</li>
<li>xnnpack</li>
<li>mkldnn / onednn</li>
<li>onnxruntime</li>
<li>tensorrt</li>
<li>cudnn</li>
<li>etc?</li>
</ul>
<p>Is there anywhere a single docs summary page listing them all compactly and explaining for which usecases are these needed?</p>
<p>Thanks!</p>
How to ensure that tensor data_ptr is aligned to 16 bytes

tridao: <p>I have a tensor that I want to have data_ptr being aligned to 16 bytes as I’m passing it to some CUDA extension that uses vectorized load.<br>
<code>.contiguous()</code> doesn’t always work. Is <code>.clone()</code> guarantee to work (e.g. clone of a tensor of size 16 will always have memory aligned to 16 bytes)?</p>
<pre><code class="lang-auto">import torch
a = torch.randn(32, dtype=torch.bfloat16, device='cuda')
print(a.data_ptr() % 16)  # 0
b = a[1:17]  # tensor has size 16
print(b.data_ptr() % 16)  # 2
print(b.contiguous().data_ptr() % 16)  # 2
print(b.clone().data_ptr() % 16)  # 0
</code></pre>
Dice learning loss unchanging after first epoch

yi-fan-li: <p>I’ve implemented a custom multi-class 2D dice loss function to train a Unet for image segmentation referenced from online code, but when used in training the test loss never changes, and the training loss just oscillates. the Unet works when using Cross-Entropy to train, so I believe the problem is with the custom function, but I cannot find why it behaves this way. From looking into the training, I saw that the loss does change during training, but it rapidly converges, and when validating the loss remains a constant, 0.1604 in the most recent case, and just never changes from it over any number of epochs.</p>
<p>I have also taken other implementations of multi-class dice loss functions, and have faced the same problem. Any clue what could be causing this issue?</p>
<p>class DiceLoss2D(nn.Module):<br>
def <strong>init</strong>(self, classes, epsilon=1e-5, sigmoid_normalization=True):<br>
super(DiceLoss2D, self).<strong>init</strong>()<br>
self.epsilon = epsilon<br>
self.classes = classes</p>
<pre><code>    if sigmoid_normalization:
        self.normalization = nn.Sigmoid()
    else:
        self.normalization = nn.Softmax(dim=1)  # TODO test ?

def flatten(self, tensor):
    return tensor.view(self.classes, -1)

def expand_as_one_hot(self, target):
    """
    Converts label image to CxHxW, where each label gets converted to
    its corresponding one-hot vector
    :param target is of shape  (1xHxW)
    :return: 3D output tensor (CxHxW) where C is the classes
    """
    # target = target.squeeze()
    shape = target.size()
    shape = list(shape)
    shape.insert(1, self.classes)
    shape = tuple(shape)
    # expand the input tensor to Nx1xHxW
    src = target.unsqueeze(1)
    srcshape = src.size()
    return torch.zeros(shape).to(target.device).scatter_(1, src, 1).squeeze(0)

def compute_per_channel_dice(self, input, target):
    epsilon = 1e-5
    One_hot_target = self.expand_as_one_hot(target.long())
    # target: BxCxWxH

    input = input.squeeze()
    assert input.size() == One_hot_target.size(), "input' and 'target' must have the same shape"+ str(input.size()) + " and " + str(target.size())

    input = self.flatten(input)
    One_hot_target = self.flatten(One_hot_target).float()

    # Compute per channel Dice Coefficient
    intersect = (input * One_hot_target).sum(-1)
    denominator = (input + One_hot_target).sum(-1)
    return 2. * intersect / denominator.clamp(min=epsilon)

def forward(self, input, target):
    # input: BxCxWxH
    # target: BxWxH
    probinput = self.normalization(input)
    per_channel_dice = self.compute_per_channel_dice(probinput, target)
    # Average the Dice score across all channels/classes
    return torch.mean(1. - per_channel_dice)
</code></pre>
Transformer Decoder Peeking into the Future Despite Subsequent Mask?

Bartek_Roszak: <p>Hey all,</p>
<p>I’ve been playing around with the Transformer model in PyTorch and bumped into something a bit weird.</p>
<p>I set up a transformer with a proper subsequent mask for the decoder to make sure it’s not peeking into the future when generating the output. But when I feed the whole target sequence to the model and compare it to when I feed only the first part of the sequence, I get different results for that first part.</p>
<p>Here’s what I’m doing in code:</p>
<pre><code class="lang-auto">transformer_model = nn.Transformer(
    nhead=1, 
    d_model=4,
    num_encoder_layers=1,
    num_decoder_layers=1,
    batch_first=True,
    dropout=0.0, 
)
src = torch.rand((1, 10, 4))
tgt = torch.rand((1, 20, 4))

transformer_model.eval()

out = transformer_model(
    src, 
    tgt,
    tgt_mask=transformer_model.generate_square_subsequent_mask(tgt.size(1)),
)

out_limited = transformer_model(
    src, 
    tgt[:, :10, :],
    tgt_mask=transformer_model.generate_square_subsequent_mask(tgt[:, :10, :].size(1)),
)

print((out[:, :10]-out_limited).mean())
</code></pre>
<p>From my understanding of how Transformers and the subsequent mask work, I’d expect the output of the first 10 positions in out to be identical to out_limited. But it looks like the future tokens in tgt are somehow influencing the output of the initial tokens, even with the mask in place.</p>
<p>I’ve got a hunch that this could be because of how the mask is applied in the decoder - after the attentions are calculated but before they’re summed for the output. This might let the future tokens kinda “leak” into the past tokens through the softmax normalization in the attention function.</p>
<p>Just wondering if anyone else has noticed this or if I’m missing something here? Any ideas for a workaround?</p>
<p>Cheers!</p>
Encoder decoder model

Padmaksha_Roy: <p>Hi there! I have a simple encoder-decoder model which inputs a 9 * 8 * 4 matrix and outputs a similar size matrix from the decoder.</p>
<blockquote>
<p>import torch<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
import numpy as np</p>
<p>criterion = nn.MSELoss()<br>
domains = 9<br>
batch = 8</p>
<p>class MyModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>    self.encoder = nn.Sequential(
    nn.Linear(4 , 8),
    nn.ReLU(),
    #nn.Linear(8,4)
    )

    #self.decoder = nn.Linear(batch_size,feats)

    self.decoders = nn.ModuleList()
    for _ in range(9):
        self.decoders.append(nn.Linear(8, 4))
    
def forward(self, inputs):

    z = []
    # for idx, enc in enumerate(self.encoders):
    #     out.append(enc(inputs[idx])) # 1 * 64 * 4 ## each domain

    for x_ in inputs: # assuming x is a list of tensors
      z.append(self.encoder(x_))

    z = torch.stack(z)
    
    outs = []
    for idx, dec in enumerate(self.decoders):
        outs.append(dec(z[idx]))

    outs = torch.stack(outs)
    print ("outs shape:", ((outs.shape)))
    return outs 
</code></pre>
<p>model = MyModel()</p>
<p>inputs = [torch.randn(8, 4) for _ in range(9)]</p>
<p>inputs_shape = torch.stack(inputs) # to check the shape of inputs</p>
<p>print (“INPUTS:”, inputs_shape.shape)</p>
<p>outs = model(inputs)</p>
</blockquote>
<p>But, I have a doubt if the encoder implementation is correct. Since the purpose of encoder is to project to a lower dimension, in my case when I am multiplying with a linear layer 4 * 8, with input 8 * 4, the result is actually an 8 * 8 dimension matrix. Can you plz suggest how should I correct this?</p>
Mixed complex and non-complex convolutional network

spacemeerkat: <h3><a name="tldr-1" class="anchor" href="#tldr-1"></a>TLDR…</h3>
<p>Is it possible to have a network which performs complex convolutions on complex tensors, as well as real convolutions on non-complex tensors, in the same model?</p>
<h3><a name="deeper-dive-2" class="anchor" href="#deeper-dive-2"></a>Deeper dive…</h3>
<p>Say we have the following model:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class ComplexNet(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.fc1 = nn.Linear(64 * 64, 4096)
        self.fc2 = nn.Linear(4096, 64 * 64)
        
        self.Hardtanh = nn.Hardtanh()

    def forward(self, x):
        x = torch.fft.ifft2(torch.fft.fftshift(x))
        x_real = x.real
        x_imag = x.imag
        ### will this work? --&gt; x = torch.sqrt(x_real**2 + x_imag**2)
        x = self.conv1(x)
        x = self.Hardtanh(x)
        x = self.conv2(x)
        x = self.Hardtanh(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.Hardtanh(x)
        x = self.fc2(x)
        x = self.Hardtanh(x)
        x = x.view(x.size(0), 1, 64, 64)
        x = torch.fft.fftshift(torch.fft.fft2(x, dim=(-2, -1)))
        return x
</code></pre>
<p>The input is a complex tensor, and to handle that fact, we can explicitly cast the model to be of complex dtype, prior to training:</p>
<pre><code class="lang-auto">model = model.type(torch.complex64)
</code></pre>
<p>However, the output of <code>x = torch.fft.ifft2(torch.fft.fftshift(x))</code> (i.e. moving from the complex to the <em>image plane</em>) is also complex. The subsequent convolutions are also complex thanks to our model casting above.</p>
<p>Is there a way to set the convolutions in the <em>image plane</em> to be real convolutions only, while still allowing gradients to pass through the initial <code>x = torch.fft.ifft2(torch.fft.fftshift(x))</code> and final <code>x = torch.fft.fftshift(torch.fft.fft2(x, dim=(-2, -1)))</code> operations thanks to the use of a complex cast model?</p>
<p>One way I’ve thought about is to force the output of <code>x = torch.fft.ifft2(torch.fft.fftshift(x))</code> to be a <em>real image</em> using <code>x = torch.sqrt(x_real**2 + x_imag**2)</code>. But the convolution weights and biases will still be complex thanks to our model casting so it will throw an error for mismatched dtypes. Perhaps you can also perform the same magnitude operation on these parameter attributes?</p>
<p>Or is there a much better way of doing this and I’m sort of butchering PyTorch with this example? <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks for any advice!</p>
Unable to install `torchaudio==0.13.1` via conda for Python 3.10

amiasato: <p>Hello everyone, I’m having a weird issue on installing the <code>torchaudio</code> package from the <code>pytorch</code> conda channel for Pytrhon 3.10. In my current environment.</p>
<p>Using miniconda, I create a new environment with Python 3.10 and try to install the 0.13.1 version of <code>torchaudio</code>:</p>
<pre><code class="lang-auto">conda create python=3.10 -n test
conda activate test
conda install -c pytorch torchaudio=0.13.1=py310_cu117
</code></pre>
<p>The output then follows:</p>
<pre><code class="lang-auto">Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - torchaudio==0.13.1=py310_cu117 -&gt; python[version='&gt;=3.11,&lt;3.12.0a0|&gt;=3.9,&lt;3.10.0a0|&gt;=3.8,&lt;3.9.0a0|&gt;=3.7,&lt;3.8.0a0']

Your python: python=3.10

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__cuda==12.0=0
  - feature:/linux-64::__glibc==2.35=0
  - feature:|@/linux-64::__cuda==12.0=0
  - feature:|@/linux-64::__glibc==2.35=0
  - torchaudio==0.13.1=py310_cu117 -&gt; pytorch==1.13.1 -&gt; __cuda
  - torchaudio==0.13.1=py310_cu117 -&gt; pytorch==1.13.1 -&gt; __glibc[version='&gt;=2.17|&gt;=2.17,&lt;3.0.a0']

Your installed version is: 2.35
</code></pre>
<p>Maybe there’s something missing in the package’s version support list? (E.g. an extra <code>|&gt;=3.10,&lt;3.11.0a0</code>)</p>

ptrblck: <p>You should stick to the install instructions from <a href="https://pytorch.org/get-started/previous-versions/#v1131">here</a>, which shows the install command as:</p>
<pre><code class="lang-auto">conda install torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia
</code></pre>
<p>which works properly for me:</p>
<pre><code class="lang-python">The following packages will be downloaded:
...
    torchaudio-0.13.1          |      py310_cu117         6.5 MB  pytorch
...

The following NEW packages will be INSTALLED:

...
  torchaudio         pytorch/linux-64::torchaudio-0.13.1-py310_cu117 
</code></pre>

amiasato: <p>Thanks, it’s working now! The issue was the lack of the <code>nvidia</code> channel (simply running <code>conda install torchaudio==0.13.1 -c pytorch -c nvidia</code> works as well).</p>
How to use ZLUDA to make CUDA works on Intel GPU

XHG78999: <p>How about using ZLUDA on windows to run Intel Corel GPUs? I tested it on linux and it works, but look likes in windows it doesn’t work well.<br>
The computer is HP Elitebook 620, Intel Graphics HD 520, Windows 10.</p>

XHG78999: <p>Can’t work. It cannot detect device.</p>

shanzi: <p>I’ll try on linux ,because linux is faster</p>
Does the dtype used for autograd’s backpropagation depend on the dtype used for forward function?

nobody: <p>I meet fp16 overflow in forward, so i change some module(not all) to fp32, it works well but slow. I try to modify fp32 to bf16 in order to speedup it, but it cause the fp16 overflow in backward.<br>
I don’t quick understand autograd mechnism, why I change the dtype used for forward function will cause the fp16 overflow in backward? Does the dtype used for autograd’s backpropagation depend on the dtype used for forward function? Or just the precision change make the loss and grad change for backward?</p>

ptrblck: <p>The backward pass will match the <code>dtypes</code> used in the forward pass. I.e. if you are using <code>bfloat16</code> in the forward, the backward will also use <code>bfloat16</code> and not <code>float16</code>.</p>

nobody: <p>Thanks for your answer, which solved my confusion.</p>
Machine Learning Inference in the cloud

Srinivasan_Subramani: <p>Hello,</p>
<p>I would like to inquire about the feasibility of performing machine learning inference in a distributed manner, specifically by assigning different layers of the neural network to run on separate GPUs. Is it possible to execute distributed inference in this fashion, where each layer is processed on a different GPU? Furthermore, I am interested in learning about any existing implementations or frameworks that support this type of distributed inference.</p>
Torchrun DistributedDataParallel training exitcode -7

nmamie: <p>Following this excellent tutorial (<a href="https://pytorch.org/tutorials/intermediate/ddp_series_multinode.html" class="inline-onebox" rel="noopener nofollow ugc">Multinode Training — PyTorch Tutorials 2.0.1+cu117 documentation</a>) I attempted to train a model on several devices. The ML pipeline seems to work since training for smaller models (less data) seems to complete correctly. However, the biggest model I am training requires a lot of training data and is therefore very resource intensive. Still, while monitoring the resources during training I can rule out that a lack of resources was the issue for the following error I received:</p>
<pre><code class="lang-auto">ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 280966) of binary...
</code></pre>
<p>Unfortunately I was unable to detect what exactly is causing this issue since I didn’t find any comprehensive docs. What I already tried:</p>
<ul>
<li>set <code>num_workers=0</code> in dataloader</li>
<li>decrease batch size</li>
<li>limit <code>OMP_NUM_THREADS</code></li>
</ul>
<p>The training script can be found <a href="https://gitlab.ethz.ch/raox/science-clf/-/blob/gnn/script/torch_ddp/helpers/multi_trainer.py" rel="noopener nofollow ugc">here</a>.</p>
<p>Any leads on this would be appreciated!</p>
Custom forward pass, but just for one epoch

sergey1: <p>I am looking for a way to run custom forward pass in an arbitrary model in Linear and Conv layers but for first epoch only.<br>
Any ideas? modules() only work for a single sequence (doesn’t work if I have extra ops in model’s forward pass or multiple sequences running in parallel). Couldn’t find any way to use hooks (maybe missed something?) and the only idea I have right now is to unwrap grad_fn backwards to build forward sequence but it seems way too complicated.</p>

witl: <p>I gues you should provide more information for me to help you. But it seems like you could solve your problem by implementing a custom forward pass within your model module, and then, use it given a condition within your forward pass.</p>

sergey1: <p>Thank u!<br>
Given is arbitrary model made of Linear and Conv layers.<br>
During first epoch I want to compute new set of weights in each layer <em><strong>without backprop</strong></em>. Single forward pass is enough for this task (each layer requires values computed at layer before).</p>
<p>Problem is, list of modules does not provide correct order of execution (for example Sequential layer will appear in list of modules altogether with all layers from inside of it). And I’m not sure if it is possible to use forward hooks to modify weights.</p>

AlphaBetaGamma96: <p>Why don’t you just apply the forward hook to just <code>Linear</code> and <code>Conv</code> objectss?</p>
<pre><code class="lang-auto">accepted_modules = ['Linear', 'Conv']
for name, module in net.named_modules():
  if(name is in accepted_modules):
    module.register_forward_pre_hook(forward_pre_hook_fn)
</code></pre>
Preprocessing for Faster RCNN

syaffers: <p>I’m trying to use the pre-trained Faster RCNN in PyTorch. I found that the <code>torchvision</code> package has the Faster R-CNN ResNet-50 FPN pre-trained network. Seeing that it uses ResNet as its feature extractor, I assumed the preprocessing was the same as if I used the ResNet models in <code>torchvision</code>: which is to use the ImageNet preprocessing below</p>
<pre><code class="lang-auto">normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
</code></pre>
<p>However, I’m getting very poor predictions from the ResNet, very few cars are being detected and many of the obvious elements in the scene are not captured. What is the correct preprocessing for the Faster RCNN?</p>

ptrblck: <p>Detection models will normalize the images internally as seen in <a href="https://github.com/pytorch/vision/blob/d2c763e14efe57e4bf3ebf916ec243ce8ce3315c/torchvision/models/detection/faster_rcnn.py#L227">this line of code</a>.<br>
You could pass your own calculated stats or just use the default ImageNet stats.<br>
Could you remove your normalization and rerun the code?</p>

syaffers: <p>Removing the normalization did the trick, more cars and traffic lights are being detected now. Thanks!</p>

chhaya_kumar_das: <p>Hi. Were you able to get the correct results out of the model? Because as it seems that the targets are always modified with simple forward pass. <a href="https://discuss.pytorch.org/t/torchvision-fasterrcnn-bounding-box-error/80985/6">Check this .</a> If possible can you tell me am I doing something wrong or is it a bug? You can comment on that post. Please excuse me for any mistakes. I am new to this.</p>

syaffers: <p><a class="mention" href="/u/chhaya_kumar_das">@chhaya_kumar_das</a> I didn’t do any training, only evaluation so I didn’t explore how the targets are being resized. Playing around with the model on colab suggests that it might be bug because my targets are also increasing rapidly in size (instead of decreasing). I used random values though so it might be the bogus data throwing off the training. I can’t be sure.</p>

chhaya_kumar_das: <p>Thanks for the feedback</p>

chhaya_kumar_das: <p>I have found a workaround. <a href="https://discuss.pytorch.org/t/torchvision-fasterrcnn-bounding-box-error/80985/7">Check this</a>. See if it works</p>

GeorgePearse: <p>Just to double check, I’ve just been plucking the values of std and mean from other code. Should it be calculated for a given dataset?</p>
Reduce LR on plateau based on training loss or validation?

DickyA: <p>as the tittle says, in what loss phase should i use the scheduler.step()? is it the training or the validation?</p>

witl: <p>Deciding whether to reduce the LR based on the training loss or validation loss is a matter of experimentation and depends on the specific problem and dataset. Both approaches can be valid, but basing the LR reduction on the validation loss is common.</p>
<p>By monitoring the validation loss, you can better estimate the model’s performance on unseen data. Reducing the LR when the validation loss plateaus can help the model converge better and avoid overfitting.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim.lr_scheduler import ReduceLROnPlateau

# Define your model and loss function
model = MyModel()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Define the learning rate scheduler
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=3)

# Training loop
for epoch in range(num_epochs):
    train_loss = 0.0
    valid_loss = 0.0

    # Training phase
    model.train()
    for batch in train_data:
        inputs, labels = batch
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()

    # Validation phase
    model.eval()
    with torch.no_grad():
        for batch in valid_data:
            inputs, labels = batch
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            valid_loss += loss.item()

    train_loss /= len(train_data)
    valid_loss /= len(valid_data)

    # Step the learning rate scheduler based on the validation loss
    scheduler.step(valid_loss)

    # Rest of the training loop... 
</code></pre>

DickyA: <p>thankyou for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
Removing datapoints from the Dataset

Shrey_Mukund: <p>I want to remove some particular train images from the CIFAR10 train_set. I have the indices of all the train_images I want get rid of. How can I do that?</p>

ptrblck: <p>You could <code>remove</code> these indices from all indices and create a <code>Subset</code> with these:</p>
<pre><code class="lang-python">indices = list(range(0, 60000))
indices_to_remove = [0, 4, 10, 5000]
[indices.remove(x) for x in indices_to_remove]

print(indices[0:10])
# [1, 2, 3, 5, 6, 7, 8, 9, 11, 12]
print(indices[4990:5000])
# [4993, 4994, 4995, 4996, 4997, 4998, 4999, 5001, 5002, 5003]
</code></pre>
Transfer weight from one architecture to the other

Duc_Banh_Nguyen: <p>Hi everyone !<br>
Currently I have 2 network architecture in two different repo, and I want to load and use the weight in .pth file of one network as pretrain weight for another network. I have copied the .pth file from one repo to the other, but when I called method torch.load(), it show this error “no module name ” . I think this problem come from the .pth file are not in it previous repo so there are some interaction that cannot happen. Could you please give me some advice ? thank you so much !</p>
Printing all columns of a tensor

sroyyors: <p>Hi,<br>
I have a 200 X 200 matrix stored in a tensor and I want to print all the rows and columns. When I use cout or output to file using “&lt;&lt;”, I get outputs in sets of 6 columns. That is all 200 rows of the columns 1-6 are printed, followed by all 200 rows of the columns 7-12 are printed and so on.</p>
<p>Is it possible to have all 200 columns printed in one shot so I could easily visualize the matrix?</p>
<p>thanks,<br>
Sushmita</p>

ptrblck: <p>You could try to call directly <a href="https://github.com/pytorch/pytorch/blob/707d265db2dc1eeb8a1b95bc70def5a1e5af7752/aten/src/ATen/core/Formatting.cpp#L260"><code>std::ostream&amp; print(std::ostream&amp; stream, const Tensor &amp; tensor_, int64_t linesize)</code></a> with the specified <code>linesize</code> argument.</p>

sroyyors: <p>thank you, that worked! It would be nice to know upfront what linesize should be for different sized tensors. But setting it to a big number works.</p>
Need help with Optimizer Error!

xie_huanyi: <p>I am finetuning a gpt-like model with Opacus. But got an error with the optimizer</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 50, in &lt;module&gt;
    main()
  File "main.py", line 47, in main
    args=args
  File "/code/src/finetune.py", line 19, in finetune
    optimizer.step()
  File "/opt/conda/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 513, in step
    if self.pre_step():
  File "/opt/conda/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 494, in pre_step
    self.clip_and_accumulate()
  File "/opt/conda/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 404, in clip_and_accumulate
    per_sample_norms = torch.stack(per_param_norms, dim=1).norm(2, dim=1)
RuntimeError: stack expects each tensor to be equal size, but got [32] at entry 0 and [1] at entry 234
</code></pre>
<p>But it works fine when I don’t use Opacus. Could anyone help me?</p>
Build errors: undefined ref to cuptiXXX -- SOLVED

Jim_Lloyd: <p>I have come back to a project that was building fine a few months ago. I had to reinstall NVidia drivers so I have completely reinstalled the drivers, cuda, cudnn and am now trying to rebuild my c++ project using the most recent nightly build.</p>
<p>I am using cuda-12-1, cudnn 8.9.2.26 on unbuntu 22.04. In the past I built exclusively with g++11.3.0. Building with it now results in the undefined references to cupti errors but also has <code>internal compiler error: Segmentation fault</code>. I can start a separate thread about the seg faults.</p>
<p>When I build instead with clang 13.0.1 I do not see the segfaults but I see numerous linker undefined reference to <code>cuptiXXX</code> symbols. The references are all from <code>libkineto::CuptiActivityApi</code> methods.</p>
<p>After some experimentation I discovered that including all of the following stanza in the root <code>CMakeLists.txt file</code> fixes the problem:</p>
<pre><code class="lang-auto">set(TorchDIR /usr/local/libtorch)
list(APPEND CMAKE_PREFIX_PATH ${TorchDIR})
find_package(Torch REQUIRED)
list(APPEND TORCH_LIBRARIES "/usr/local/cuda/lib64/libcupti.so")
</code></pre>
<p>My project has a subdirectory where I create a library that wraps all of the libtorch usage behind my own API. Putting the above stanza in the CMakeLists.txt for that subdirectory does NOT work.</p>
<p>I’m posting this here 1) in case anyone else has this problem and 2) to suggest that the libtorch team figure out the correct solution (e.g. including <code>libcupti.so</code> in <code>TORCH_LIBRARIES</code> in the first place).</p>