akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.eq for multiple values

Fst_Msn: <p>Given a 2d tensor A, and a 1d tensor B of length &gt; 1:</p>
<pre><code class="lang-auto">A = [ [1, 2], [1, 4], [4, 3]]
B = [1, 4]
</code></pre>
<p>I would like to write a function that returns (in this case) two masks of the same shape as A, where the mask is 1 if the value of B is the same as the value of A:</p>
<pre><code class="lang-auto">res = [
[[1., 0.],[1., 0.],[0., 0.]], 
[[0., 0.],[0., 1.],[1., 0.]]
]
</code></pre>
<p>a naive solution is to use a for loop on B, which performs a torch.eq between A and every element of B.</p>
<p>But is there a smarter (and faster) way to do this?</p>

tom: <p>You want broadcasting, best described here: <a href="https://pytorch.org/docs/stable/notes/broadcasting.html" class="inline-onebox" rel="noopener nofollow ugc">Broadcasting semantics — PyTorch 1.12 documentation</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
Is F.gumbel_softmax correct?

Pasquale_Minervini: <p>In <a href="https://github.com/pytorch/pytorch/blob/b4ed13ea0ff091328c6d0dfdf5d751d4280fb67f/torch/nn/functional.py#L1885" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at b4ed13ea0ff091328c6d0dfdf5d751d4280fb67f · pytorch/pytorch · GitHub</a> you can see that, in F.gumbel_softmax, samples from Gumbel(0, 1) are drawn in the following way:</p>
<pre><code class="lang-python">    gumbels = (
        -torch.empty_like(logits, memory_format=torch.legacy_contiguous_format).exponential_().log()
    )  # ~Gumbel(0,1)
</code></pre>
<p>Is this correct? <code>torch.empty_like</code> returns an uninitialised tensor, which is not initialized in any way here.</p>

ptrblck: <p>Yes, it’s correct as you are right that <code>torch.empty_like</code> will create a tensor with uninitialized memory, but the following <code>.exponential_()</code> operation would fill the values from an exponential distribution as described in the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.exponential_.html#torch.Tensor.exponential_">docs</a>.</p>

Pasquale_Minervini: <p>I see, thank you! I thought .exponential_() was f(x) = e^x!</p>

ptrblck: <p>Yeah, I thought so but this is done by <code>tensor.exp()</code>.</p>

Asthestarsfalll: <p>Hi, I find that in many other implementations of gumbel softmax, they use the formule $G_i = -log(-log(U_i)), U_i \in U(0, 1)$ to generate gumbel noise. Why torch implementation samples from exponential distribution?</p>

ptrblck: <p>Both approaches should be valid as described in <a href="https://arxiv.org/pdf/1611.00712.pdf">THE CONCRETE DISTRIBUTION: A CONTINUOUS RELAXATION OF DISCRETE RANDOM VARIABLES - https://arxiv.org/pdf/1611.00712.pdf</a>:</p>
<blockquote>
<p>The apparently arbitrary choice of noise gives the trick its name, as − log(− log U ) has a Gumbel<br>
distribution. This distribution features in extreme value theory (Gumbel, 1954) where it plays a<br>
central role similar to the Normal distribution: the Gumbel distribution is stable under max opera-<br>
tions, and for some distributions, the order statistics (suitably normalized) of i.i.d. draws approach<br>
the Gumbel in distribution. The Gumbel can also be recognized as a − log-transformed exponen-<br>
tial random variable. So, the correctness of (9) also reduces to a well known result regarding the<br>
argmin of exponential random variables. See (Hazan et al., 2016) for a collection of related work,<br>
and particularly the chapter (Maddison, 2016) for a proof and generalization of this trick.</p>
</blockquote>

Asthestarsfalll: <p>Thank you very much!</p>
1D Gaussian Kernel

MUnique: <p>Hello,</p>
<p>Is there any equivalent for “scipy.ndimage.gaussian_filter1d” in Pytorch? I have a tensor of shape [T, H, W] that I would like to apply a 1d gaussian kernel on its first dimension (T). How can I do so?</p>
<p>Thanks</p>

ptrblck: <p>You could create a gaussian kernel as described e.g. in <a href="https://discuss.pytorch.org/t/is-there-anyway-to-do-gaussian-filtering-for-an-image-2d-3d-in-pytorch/12351/19">this post</a> or <a href="https://discuss.pytorch.org/t/gaussian-tensor-generation/89397">this one</a> and apply it to your input via <code>F.conv2d</code>.</p>

tom: <p>To add to <a class="mention" href="/u/ptrblck">@ptrblck</a>’s comments, here is what I like to do for images:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/TorchDrift/TorchDrift/blob/3d590e8f25185e68d2749d16372d5c4715e543f5/torchdrift/data/functional/corruption_functions.py#L114-L123">
  <header class="source">

      <a href="https://github.com/TorchDrift/TorchDrift/blob/3d590e8f25185e68d2749d16372d5c4715e543f5/torchdrift/data/functional/corruption_functions.py#L114-L123" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/TorchDrift/TorchDrift/blob/3d590e8f25185e68d2749d16372d5c4715e543f5/torchdrift/data/functional/corruption_functions.py#L114-L123" target="_blank" rel="noopener nofollow ugc">TorchDrift/TorchDrift/blob/3d590e8f25185e68d2749d16372d5c4715e543f5/torchdrift/data/functional/corruption_functions.py#L114-L123</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="114" style="counter-reset: li-counter 113 ;">
          <li>distance = torch.arange(</li>
          <li>    -(width // 2), width // 2 + 1, dtype=torch.float, device=x.device</li>
          <li>)</li>
          <li>gaussian = torch.exp(</li>
          <li>    -(distance[:, None] ** 2 + distance[None] ** 2) / (2 * sigma ** 2)</li>
          <li>)</li>
          <li>gaussian /= gaussian.sum()</li>
          <li>channels = x.size(1)</li>
          <li>kernel = gaussian[None, None].expand(channels, -1, -1, -1)</li>
          <li>return torch.nn.functional.conv2d(x, kernel, padding=width // 2, groups=channels)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>To take this to 1d, you save a lot by not needing the funny <code>None</code>-indexing and broadcasting, but you will likely want to rearrange the input tensor to have the folding dimension last to use conv1d.</p>
<p>Best regards</p>
<p>Thomas</p>
TypeError: Unwrapping the module did not yield a `LightningModule`, got <class 'models.resnet.ResNet'> instead

KanZa: <p>I am using Pytorch-Lightning for finding the optimal LR but it is giving this error</p>
<pre><code class="lang-auto">for cycle in range(CYCLES):            
            random.shuffle(unlabeled_set)
            subset = unlabeled_set[:SUBSET]
            # Model - create new instance for every cycle so that it resets
            with torch.cuda.device(CUDA_VISIBLE_DEVICES):            
                resnet18    = resnet.ResNet18(num_classes=NO_CLASSES).cuda()
            models = resnet18
            
            torch.backends.cudnn.benchmark = True
            models = torch.nn.DataParallel(models, device_ids=[0])
            
            # Loss, criterion and scheduler (re)initialization
            criterion      = nn.CrossEntropyLoss(reduction='none')
            #optim_backbone = optim.SGD(models.parameters(), lr=LR, 
            #    momentum=MOMENTUM, weight_decay=WDECAY)
            optim_backbone = optim.SGD(models.parameters(), lr=LR, weight_decay=WDECAY)
 
            #sched_backbone = lr_scheduler.MultiStepLR(optim_backbone, milestones=MILESTONES)
            optimizers =  optim_backbone
            #schedulers = sched_backbone
            
            train(models, criterion, optimizers, dataloaders, Epochs) #schedulers,
            acc = test(models, dataloaders, mode='test')
            logger_directory= 'logs/without_auto_lr'
            version_of_log = 1.1 
            
            logger  = TensorBoardLogger(save_dir=logger_directory,version=version_of_log)
            
            trainer = pl.Trainer(gpus=1, max_epochs=Epochs, logger=logger, auto_lr_find=False, val_check_interval=0.5)

            lr_finder = trainer.tuner.lr_find(models, dataloaders)
            
            models.hparams.lr = lr_finder.suggestion()
            
            print(f'Auto-find model LR: {models.hparams.lr}')

            fig = lr_finder.plot(suggest=True)


</code></pre>

tom: <p>If you want to use PyTorch Lightning, you want to write your model as a subclass of <code>pl.LightningModule</code> instead of <code>torch.nn.Module</code>, like here: <a href="https://pytorch-lightning.readthedocs.io/en/1.7.5/model/train_model_basic.html#define-a-lightningmodule" class="inline-onebox" rel="noopener nofollow ugc">Train a model (basic) — PyTorch Lightning 1.7.5 documentation</a> .</p>
<p>Best regards</p>
<p>Thomas</p>
Convert a LSTM model from Keras to PyTorch

Ahmed_Gaber: <p>I have a model developed in Keras that I wish to port over to PyTorch. The model is</p>
<pre><code class="lang-auto">from keras.models import Sequential

from keras.layers import Dropout, Dense, LSTM, Bidirectional,Embedding, GlobalMaxPool1D

vocabulary_size = vocab_size

seq_len = 40

embed_len=20

model = Sequential()

model.add(Embedding(vocabulary_size+1, embed_len, input_length=seq_len))

model.add(LSTM(units = 80, return_sequences=True))

model.add(Dropout(0.5))

model.add(LSTM(units = 40))

model.add(Dropout(0.5))

model.add(Dense(units=20, activation='softmax'))

model.summary()
</code></pre>

tom: <p>In my experience, the best approach is to do it bit by bit and compare the intermediate results when you put in the same inputs. Typically, you have to rearrange axes a bit to make things work.</p>
How to use StandardScaler in continuous training?

xiguahuoshan: <p>I want continuously training my model as data is  coming online. As I couldn’t know the future’s data, how could I use StandardScaler to cover the coming data?</p>
<p>two solutions could be selected:</p>
<ol>
<li>Make a basic model and do the StandardScaler with its data, do not update the StandardScaler in the coming data. Retrain the basic model with StandardScaler at fixed period.</li>
<li>Make a basic model and do the StandardScaler with its data, update the StandardScaler’s mean and std using flowing mean/std algo with the coming data.</li>
</ol>
<p>Which one is right for online StandardScaler? Or is there any better solution?<br>
Thanks for your time.</p>

xiguahuoshan: <p>What if I use BN layer in my model?  Does it means  I don’t have to do the normalization in the continuous training?</p>