akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
transforms.PILToTensor vs transforms.ToTensor

Greatmiralis: <p>I have recently started using pytorch and have written a few working models. Now i have 2 models that did not work at all and in inference only output something that looked like noise. I have now fixxed the problem by changing my data transform to Tensor from transform.PILToTensor to transforms.ToTensor and now i wanted to know what the actual difference between these two functions is as i do not see the difference in the documentation.</p>

ptrblck: <p><code>PILToTensor</code> transforms a <code>PIL.Image</code> to a PyTorch tensor of the same <code>dtype</code> while <code>ToTensor</code> normalizes the tensor to <code>[0, 1]</code> and will return a <code>FloatTensor</code>.<br>
Usually normalized/standardized tensors are preferred while training a model as the convergence often benefits from it.</p>
I am trying to run the learn.fine_tune in colab but it is showing tensor size mismatch

teja_teja: <p>RuntimeError: Caught RuntimeError in DataLoader worker process 0.<br>
Original Traceback (most recent call last):<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py”, line 302, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py”, line 39, in fetch<br>
data = next(self.dataset_iter)<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 140, in create_batches<br>
yield from map(self.do_batch, self.chunkify(res))<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 170, in do_batch<br>
def do_batch(self, b): return self.retain(self.create_batch(self.before_batch(b)), b)<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 168, in create_batch<br>
if not self.prebatched: collate_error(e,b)<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 166, in create_batch<br>
try: return (fa_collate,fa_convert)<a>self.prebatched</a><br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 52, in fa_collate<br>
else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 52, in <br>
else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/data/load.py”, line 51, in fa_collate<br>
return (default_collate(t) if isinstance(b, _collate_types)<br>
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/<em>utils/collate.py", line 140, in default_collate<br>
out = elem.new(storage).resize</em>(len(batch), *list(elem.size()))<br>
File “/usr/local/lib/python3.7/dist-packages/fastai/torch_core.py”, line 376, in <strong>torch_function</strong><br>
res = super().<strong>torch_function</strong>(func, types, args, ifnone(kwargs, {}))<br>
File “/usr/local/lib/python3.7/dist-packages/torch/_tensor.py”, line 1121, in <strong>torch_function</strong><br>
ret = func(*args, **kwargs)<br>
RuntimeError: Error when trying to collate the data into batches with fa_collate, at least two tensors in the batch are not the same size.</p>
<p>Mismatch found on axis 0 of the batch and is of type <code>TensorImage</code>:<br>
Item at index 0 has shape: torch.Size([3, 4000, 6000])<br>
Item at index 2 has shape: torch.Size([3, 256, 256])</p>
<p>Please include a transform in <code>after_item</code> that ensures all data of type TensorImage is the same size</p>

ptrblck: <p>Based on the error message it seems you are trying to yield image tensors using different spatial sizes, so you might want to add a <code>Resize</code> transformation to create images in the same size.</p>
I've saved a model in Libtorch, now how do I load it?

Alio: <p>I’ve used torch::serialized to save the model’s parameters but can’t seem to load them.<br>
My model below has the component to save without error, and the part to load it are commented out (because they don’t work).</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;

using namespace torch::indexing;
torch::Device device(torch::kCPU);


struct Critic_Net : torch::nn::Module {
    torch::Tensor next_state_batch__sampled_action;
    public:
    Critic_Net() {
        // Construct and register two Linear submodules.
        lin1 = register_module("lin1", torch::nn::Linear(427, 42));
        lin2 = register_module("lin2", torch::nn::Linear(42, 286));
        lin3 = register_module("lin3", torch::nn::Linear(286, 42));
        lin1-&gt;to(device);
        lin2-&gt;to(device);
        lin3-&gt;to(device);
    }
    torch::Tensor forward(torch::Tensor next_state_batch__sampled_action) {
        auto h = next_state_batch__sampled_action;
        h = torch::relu(lin1-&gt;forward(h));
        h = torch::tanh(lin2-&gt;forward(h));
        h = lin3-&gt;forward(h);
        return h;
    }
    torch::nn::Linear lin1{nullptr}, lin2{nullptr}, lin3{nullptr};
};

auto one = torch::rand({42, 427}).to(device);

auto target = torch::rand({42, 42}).to(device);
// auto net_optimizer = torch::optim::Adam(net.parameters(),  1e-3);

torch::Tensor y_hat;
at::Tensor loss;

auto net = Critic_Net();

    //   torch::jit::script::Module net;
    //   std::string module_path = "model.pt";
    //   auto net = torch::jit::load(module_path);  
    //   net.to(device);

int main() {
    for (int e = 0; e&lt;2000; e++) {
        net_optimizer.zero_grad();
        y_hat = net.forward(one);

        loss = torch::smooth_l1_loss(target, y_hat);
        loss.backward();
        net_optimizer.step();

        if (e % 50 == 0) {
            std::cout &lt;&lt; loss.item() &lt;&lt; "  " &lt;&lt; "\n";
        }
    }

    std::string model_path = "model.pt";
    torch::serialize::OutputArchive output_archive;
    net.save(output_archive);
    output_archive.save_to(model_path);
}
</code></pre>
High fake class accuracy when training an ACGAN

lule: <p>I’m training an ACGAN on cifar-10. The network recognizes fake image classes with an accuracy of 100% after only 50 steps which indicates that something goes wrong. I am however not sure what my mistake is.</p>
<p>training log:</p>
<pre><code class="lang-auto">epoch: 1/100 batch: 50/500 G loss: 0.8187 D loss: 4.1763 Loss cls fake: 0.0740 Loss cls real: 1.5676 fake acc: 100.0% real acc: 40.0%
epoch: 1/100 batch: 100/500 G loss: 0.9943 D loss: 3.8245 Loss cls fake: 0.1135 Loss cls real: 1.4479 fake acc: 100.0% real acc: 45.0%
</code></pre>
<p>training loop:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for batch_idx, (image, target) in enumerate(train_loader):
        images = image.to(device)
        target = target.to(device)

        current_batchSize = images.size()[0]

        realLabel = torch.ones(current_batchSize).to(device)
        fakeLabel = torch.zeros(current_batchSize).to(device)

        ###########
        # TRAIN D #
        ###########
        optimizerD.zero_grad()

        # On real data
        predictR, predictRLabel = D(images)

        loss_real_adv = criterion_adv(predictR, realLabel)
        loss_real_aux = criterion_aux(predictRLabel, target)
        real_cls_acc = compute_cls_acc(predictRLabel, target)
        real_score = predictR

        # On fake data
        latent_value = torch.randn(current_batchSize, latent_size).to(device)

        cls_one_hot = torch.zeros(current_batchSize, n_classes, device=device)
        cls_one_hot[torch.arange(current_batchSize), target] = 1.0

        latent = torch.cat((latent_value, cls_one_hot), dim=1)

        fake_images = G(latent)
        predictF, predictFLabel = D(fake_images.detach())
        loss_fake_adv = criterion_adv(predictF, fakeLabel)
        loss_fake_aux = criterion_aux(predictFLabel, target)

        fake_cls_acc = compute_cls_acc(predictFLabel, target)
        fake_score = predictF

        loss_adv = loss_real_adv + loss_fake_adv
        loss_aux = 1.8*loss_real_aux + 0.2*loss_fake_aux

        lossD = loss_adv + loss_aux

        lossD.backward()
        optimizerD.step()

        ###########
        # TRAIN G #
        ###########
        optimizerG.zero_grad()

        predictG, predictLabel = D(fake_images)
        lossG_adv = criterion_adv(predictG, realLabel)
        lossG_aux = criterion_aux(predictLabel, target)
        lossG = lossG_adv + lossG_aux

        lossG.backward()
        optimizerG.step()

        if (batch_idx+1) % 50 == 0:
            print('epoch: {}/{} batch: {}/{} G loss: {:.4f} D loss: {:.4f} Loss cls fake: {:.4f} Loss cls real: {:.4f} fake acc: {}% real acc: {}%'.format(epoch+1, num_epochs, batch_idx+1, total_step, lossG.item(), lossD.item(), loss_fake_aux, loss_real_aux, fake_cls_acc, real_cls_acc))

</code></pre>
<p>Thankful for any help!</p>
Data Augmentation in Pre-trained ResNet

taesiri: <p>Hello,</p>
<p>I am wondering whether the current ResNet family of models from the Torch Vision have been trained with data augmentation or not. In the <a href="https://github.com/pytorch/vision/tree/a35be97a6e6725c83032315a8f5e5f6911c9ef41/references/classification#resnet" rel="noopener nofollow ugc">reference script here</a>, there is no mention of data augmentation for the ResNet family, but how can I make sure there was no augmentation (without training a new model myself)?</p>
<p>Based on the accuracy gap between the <a href="https://pytorch.org/vision/stable/models.html#initializing-pre-trained-models" rel="noopener nofollow ugc">two weight versions</a> of the ResNet-50 model (<code>ResNet50_Weights.IMAGENET1K_V1</code> and <code>ResNet50_Weights.IMAGENET1K_V2</code>), I assume the V1 version has no augmentation, but the V2 version <a href="https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/" rel="noopener nofollow ugc">has several augmentation policies</a>. Am I right?</p>
<p>Thank you.</p>
Access to gradient after call loss.backward()

Yoo: <p>Suppose I have loss = w1<em>loss1 + w2</em>loss2, where I defined two learnable weights for each loss</p>
<pre><code class="lang-auto">Weightloss1 = torch.FloatTensor([1]).clone().detach().requires_grad_(True)
Weightloss2 = torch.FloatTensor([1]).clone().detach().requires_grad_(True)

opt1 = torch.optim.Adam(model.parameters(), ...)
opt2 = torch.optim.Adam([Weightloss1, Weightloss2], ...)

# training
While True:
    model.train()
    for X, Y in train_set:
        pred_Y = model(X)
        loss_1 = Weightloss1 * model.loss_fn_1(pred_Y, Y)
        loss_2 = Weightloss2 * model.loss_fn_2(pred_Y, Y)
        loss = torch.div(torch.add(loss_1,loss_2), 2)

        opt_1.zero_grad()
        opt_2.zero_grad()
        loss.backward(retain_graph=True)

        opt_1.step()
        opt_2.step()
</code></pre>
<p>My question is, when I call loss.backward(retain_graph=True), will pytorch calculate gradients w.r.t. w1 and w2 in addition to model parameters? If so how can I get access to them?</p>
<p>Also, does the order of updating step() call matter? I do not believe so though</p>

ptrblck: <p>Yes, all trainable parameters will receive gradients which you could access via their <code>.grad</code> attribute. Using <code>retain_graph=True</code> is not needed in this case.</p>

Yoo: <p>Thanks, for the previous code, if I additionally add an another loss, loss_3, for params [Weightloss1, Weightloss2] which I set to update in optimizer2, something like</p>
<blockquote>
<pre><code>    pred_Y = model(X)
    loss_1 = Weightloss1 * model.loss_fn_1(pred_Y, Y)
    loss_2 = Weightloss2 * model.loss_fn_2(pred_Y, Y)
    loss = torch.div(torch.add(loss_1,loss_2), 2)
    loss_3 = loss_fn_3(Weightloss1 + Weightloss2, target_Weightloss)
    opt_1.zero_grad()
    opt_2.zero_grad()
    loss.backward()
    loss_3.backward()
    opt_1.step()
    opt_2.step()
</code></pre>
</blockquote>
<p>If I call opt_2.step(), will it use both collected gradients w.r.t. Weightloss from loss and loss_3 to update tunable parameter [Weightloss1, Weightloss2]?</p>

ptrblck: <p>Gradients are accumulated into the <code>.grad</code> attribute of trainable and used parameters. The optimizer’s <code>step()</code> method will thus use the already accumulated gradients in their corresponding parameters.</p>
Cuda Memory Use Batched Matrix Multiplication

buddit: <p>I’m trying to perform a batched matrix multiplication on a GPU and I get a CUDA out of memory error that doesn’t make a lot of sense to me. I’m hoping someone can shed some light on it.</p>
<p>Specifically, I have a tensor <strong>A</strong> of size [128, 1024, 1, 1, 27] and another tensor <strong>B</strong> of size [1, 1, 160, 27, 27]</p>
<p>When I try to multiply them I get a CUDA out of memory error that says “Tried to allocate 56.95 GiB”<br>
I don’t understand why it would require so much space? It’s not just the size of the output tensor because that’s be [128, 1024, 160, 1, 27] which is more on the order of 2 GiB.</p>
<p>My main question is whether there’s any method I can use to compute this batched matrix multiply efficiently? Any advice is greatly appreciated.</p>

Saswat: <p>Is this the only operation that you’re doing or is it a part of several steps?</p>
<p>Try using the following method to compute the actual memory footprint of your matrices.<br>
<code>a.element_size * a.nelement()</code></p>
<p>Since the matrices are itself 5D, I’m guessing a significant amount of memory would be spent in storing the intermediate values as well.</p>

buddit: <p>Hi, thanks for the quick response!</p>
<p>As far as your questions, the same error occurs when the operation is done in isolation with no gradient tracking.</p>
<p>The memory footprint of A is 0.01416 GiB,<br>
B is 0.00047 GiB<br>
and a matrix with the output shape has a memory footprint of 2.2649 GiB.</p>
<p>I can understand that some amount of extra memory is used for intermediate values during the computation but I can’t imagine how 54 GB is needed.</p>
<p>Thanks again for your response, I hope this is helpful context.</p>

ngimel: <p>Unfortunately when doing batched matrix multiply, and batched dimensions cannot be flattened (as in your case) full matrices are materialized. In your case, B will turn into (128, 1024, 160, 27, 27) matrix and will require ~60GB memory. This is a limitation of underlying cublas function that we are using, that assumes batch elements are at regular positions in memory.</p>

buddit: <p>Thanks for the explanation! Am I right in thinking that the only way to solve this on my end is to use a for loop or avoid the multiplication entirely?</p>

ngimel: <p>You can try representing your op as (A*B.transpose(-1, -2)).sum(-1) and then use keops <a href="https://www.kernel-operations.io/keops/index.html" class="inline-onebox" rel="noopener nofollow ugc">Kernel Operations on the GPU, with autodiff, without memory overflows — KeOps</a> to avoid materializing large intermediate tensor. In future versions of pytorch you will also be able to codegen a kernel for this operation without materializing large intermediate), but in the current version your easiest option is a loop (or keops).</p>

buddit: <p>I’ll take a look at that library. Thanks a lot for your help.</p>
Should we always place the sharding filter as early as possible?

Vedant_Roy: <p>When using datapipes, it seems like you want to apply the <code>sharding_filter</code> as early as possible, in order to prevent data loader workers from doing duplicate work.</p>
<p>However, I’m struggling to understand how to use the <code>sharding_filter</code> in the following scenario:</p>
<pre><code class="lang-auto">def create_datapipe(s3_urls):
    pipe = dp.IterableWrapper(s3_urls)
    # a fake function that downloads from s3
    pipe = pipe.map(download_url)
    pipe = pipe.flatmap(chunk: [item for item in chunk])
    pipe = pipe.batch(batch_size=256)
    pipe = pipe.map(do_expensive_computation)     
</code></pre>
<p>Ideally, I would place the sharding filter before <code>do_expensive_computation</code>, or before <code>batch</code>. The issue is; this parallelizes downloading from S3, which I don’t necessarily want.</p>
<p>Ideally, I’d have the downloading from S3 be on a single thread that is accessed from all workers, and then start parallelization after the <code>flatmap</code> stage.</p>
<p>The reason this matters is–imagine my chunks are big. (E.g, each chunk contains 5K samples), then if the number of chunks is not divisible by the number of workers, one worker will have to process an extra chunk. This worker, will take a very long time.</p>

nivek: <ul>
<li>The generally recommendation is that you should shuffle before you shard (for randomness/determinism reasons).</li>
<li>Based on how DataLoader is currently implemented, each worker will duplicate the work of the DataPipe prior to sharding. In the future, when we update the implementation of DataLoader, we may be able to run certain operations on one process only and then other workers will read from it.</li>
</ul>
<p>I suspect for your case, you still want to shard early to avoid duplicate downloading, but it is hard to know for sure without knowing the exact setup and running it.</p>
How to debug S3 errors with torchdata?

Vedant_Roy: <p>I’m getting the following error:</p>
<pre><code class="lang-auto">ValueError: curlCode: 77, Problem with the SSL CA cert (path? access rights?)
</code></pre>
<p>for the code:</p>
<pre><code class="lang-auto">    handler = torchdata._torchdata.S3Handler(1000, "us-west-2")
    obj = handler.s3_read("s3://datasets-v2/150GBv2/fb644bd37250431ab67eb5b4a312bf22_000000.parquet")
</code></pre>
<p>What’s confusing is that, both <code>boto3</code> and the AWS CLI are working fine. So, I’m not sure how my credentials could be wrong.</p>

nivek: <p>That seems to be an issue with AWS SDK which is hard for us to debug.</p>
<p>As a workaround, is it possible for you to use <a href="https://pytorch.org/data/main/tutorial.html#accessing-aws-s3-with-fsspec-datapipes" rel="noopener nofollow ugc"><code>fsspec</code> DataPipes</a>?</p>
AssertionError: For batched (3-D) `query`, expected `key` and `value` to be 3-D but found 2-D and 2-D tensors respectively

Shazinho_tech: <p>So I was developing the Se12Seq architecture for language translation task, and I am testing this for random tensors before using for the proper data, the encoder is working absolutely fine but I am facing this issue with the decoder which I am unable to understand because the code runs fine when not placed in the nn.Module class</p>
<p>class Encoder(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, num_heads):<br>
super().<strong>init</strong>()<br>
self.embedding = nn.Embedding(vocab_size, embedding_dim)<br>
self.encoder_layer = nn.TransformerEncoderLayer(d_model= embedding_dim, nhead= num_heads)</p>
<p>def forward(self, x):<br>
encoder_stack = nn.TransformerEncoder(self.encoder_layer, num_layers=6)<br>
x = self.embedding(x)<br>
x = encoder_stack(x)<br>
return x</p>
<p>class Decoder(nn.Module):<br>
def <strong>init</strong>(self, vocab_size, embedding_dim, num_heads):<br>
super().<strong>init</strong>()<br>
self.embedding = nn.Embedding(vocab_size, embedding_dim)<br>
decoder_layer = nn.TransformerDecoderLayer(d_model=embedding_dim, nhead=num_heads)<br>
self.decoder = nn.TransformerDecoder(decoder_layer, num_layers=5)</p>
<p>def forward(self, memory, trg):<br>
embeddings = self.embedding(trg)<br>
output = self.decoder(memory, trg)<br>
return output</p>
<p>src = torch.arange(0,100).unsqueeze(0)<br>
encoder = Encoder(5000, 100, 5)<br>
mem = encoder(src)<br>
mem.shape</p>
<p>embedding = nn.Embedding(5000, 100)<br>
tgt = torch.arange(100, 200).unsqueeze(0)</p>
<p>encoder = Encoder(5000, 100, 5)<br>
encoder(src)</p>
<p>decoder = Decoder(5000, 100, 5)<br>
decoder.forward(mem, tgt)</p>
Is it possible to have different dimension shapes for index and destination tensor in scatter_add_?

Zevrap-81: <p>The problem i am considering is as follows:<br>
I have a index tensor which essentially is an element connectivity matrix from fem with shape [N_e, 4]<br>
and a src tensor which is a tensor describing the nodal components of each element with shape [N_e, 4]<br>
what i want is to get a out tensor which is essentially the sum of all nodal components for each node with shape [N_n]. I thought this should be possible with scatter_add but this is producing a RuntimeError: Index tensor must have the same number of dimensions as self tensor.</p>
<p>for example we have</p>
<pre><code class="lang-auto">import torch 
elements= torch.tensor([[0,1,2,3],[1,4,5,2]]) #N_e=2
src= torch.tensor([[8, 6, 4, 9],
                   [5, 3, 0, 7]])
#then i want out tensor to look like 
out= torch.tensor([8, 6+5, 4+7, 9, 3, 0])

#my approach
out = torch.zeros(6) #N_n=6
out.scatter_add_(0, elements, src)
</code></pre>
<p>can you please tell me if i am in the right direction or i should simply add all the individual components seperately?</p>

Zevrap-81: <aside class="quote no-group" data-username="Zevrap-81" data-post="1" data-topic="164256">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zevrap-81/48/45981_2.png" class="avatar"> Zevrap-81:</div>
<blockquote>
<pre><code class="lang-auto">src= torch.tensor([[8, 6, 4, 9],
                   [5, 3, 0, 7]])
</code></pre>
</blockquote>
</aside>
<p>I solved the problem using index_add</p>
Loading checkpoint from state_dict does NOT restore progress

neel_g: <p>Hi, I’m using DDP and attempting to resume training progress for my model.<br>
However, Even after I restore the <code>state_dict</code> for the model it still trains from scratch rather than initializing from the checkpoint.</p>
<p>Code: <a href="https://rentry.co/cqn2a" class="inline-onebox" rel="noopener nofollow ugc">Checkpoint Loading process</a></p>
<p>This is absolutely maddening me for quite some time now <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> I can confirm from the above specified <code>print</code> statement that the <code>chkp_file</code> is <em>indeed</em> the latest checkpoint available…</p>
<p>As you can see above, everything is quite nominal here - the only difference is that I use <code>DDP</code> (and <code>fp16</code>) but ensure I follow methods as outlined in the <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models" rel="noopener nofollow ugc">docs</a> about saving/restoring checkpoints with DDP.</p>
<p>Does anyone have any idea why that may be an issue? Cheers.</p>

agu: <p>As a guess, could you try:</p>
<pre><code class="lang-auto">model.module.load_state_dict(checkpoint['model_state_dict'], strict=True)
</code></pre>
<p>To explain, you are saving the state dict from the module wrapped by <code>DDP</code> (i.e. <code>model.module</code>), but you are loading it to the <code>DDP</code> instance itself (i.e. <code>model</code>), which is why my suggestion is to try loading it again to the wrapped module <code>model.module</code>.</p>
<p>Also, I wonder what happens if you only change <code>strict=True</code> in your current code. With <code>strict=False</code>, it may be that none of the parameters are actually being loaded, but this happens silently. The reason none would be loaded would be that none of the state dict keys match due to the model being loaded into having an additional <code>module</code> prefix.</p>

neel_g: <p>Thanks a ton mate! <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=12" title=":tada:" class="emoji" alt=":tada:" loading="lazy" width="20" height="20"> I thought I’d put the <code>model.module</code> (sorry for opening an issue with such a silly typo). I removed the <code>strict=True</code> override too, and it all works perfectly. Again, many thanks and sincerely hope you have an amazing week!!</p>
Building computational graph

artoftime: <p>Hello,</p>
<p>I am currently trying to build computational graphs that are then used to build<br>
symbolic expressions for SMT solving. However, some things are unclear to me. Below you see a picture of one of my example graphs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54bccfc61f0b6161e97592a632348b964917da53.png" data-download-href="https://discuss.pytorch.org/uploads/default/54bccfc61f0b6161e97592a632348b964917da53" title="example"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54bccfc61f0b6161e97592a632348b964917da53_2_506x500.png" alt="example" data-base62-sha1="c5CDlaV8f4Wbwl0RBUHPm3fIfZx" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54bccfc61f0b6161e97592a632348b964917da53_2_506x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54bccfc61f0b6161e97592a632348b964917da53_2_759x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54bccfc61f0b6161e97592a632348b964917da53_2_1012x1000.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example</span><span class="informations">1758×1734 390 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><em>(For clarification: The numbers next to “…Backward”, such as _9, only indicate row number in the graph, and numbers below each String indicate duplicates for each row. This is only a detail related to drawing (unique IDs) and is not relevant to my questions)</em></p>
<p><strong>My questions:</strong></p>
<ol>
<li>(Red) Why is here a double Transpose?</li>
<li>(Red) What does “None” do here in MmBackward?</li>
<li>(Green) What does “None” do here in WhereBackward?</li>
<li>(Blue) What does “None” do here in MulBackward?</li>
</ol>
<p>I know that, in the case of PowPackward, if the number to which the base tensor is raised to is not tracked by autograd then that will also appear as “None” in the graph. Now I am wondering what “None” is referring two in the above questions.</p>
<p>Lastly:<br>
5) (Orange) Does AccumulateGrad do something special with [4.0]?</p>
<p><strong>Note</strong>: The graph was created as follows:</p>
<pre><code class="lang-auto">L = torch.nn.Linear(2, 1, bias=False)
four = torch.tensor([4])
x = torch.tensor([2.034, 3.119])
x.requires_grad = True
four.requires_grad = True #otherwise this would appear as a None node in the graph
with torch.no_grad():
    L.weight[0] = t.tensor([-0.9, 0.5])
y = L(x).pow(four)

#First order derivative
fd = torch.autograd.grad(y, x, create_graph=True)[0]

#Second order derivative
sd = torch.autograd.grad(fd[0], x, create_graph=True)[0]

</code></pre>
<p>Now, <code>sd.grad_fn</code> contains the graph depicted above.</p>
<p>Thank you.</p>
Removing padding from tensor

AfonsoSalgadoSousa: <p>I have a tensor like:</p>
<pre><code class="lang-auto">tensor([[ 9, -1, -1],
        [ 7, -1, -1],
        [ 6,  4, -1]])
</code></pre>
<p>What is the most efficient way to remove the padding and get something like:</p>
<pre><code class="lang-auto">[[9], [7], [ 6,  4]])
</code></pre>
<p>Thanks in advance for any help you can provide.</p>
Masked LogSumExp

conv8d: <p><code>torch.logsumexp</code> doesn’t provide support for masking. Is there an efficient way to implement it with masking?<br>
I came up with the below function but I’m not sure if it’s the best way.</p>
<pre><code class="lang-auto">def masked_logsumexp(x, mask, eps=1e-8):
    mask_1 = mask.sum(dim=-1)
    m = torch.max(x*mask - ((1-mask)/self.eps), dim=-1, keepdim=True).values
    lse = m.squeeze(-1)*mask_1 + torch.log(torch.sum((torch.exp(x-m)*mask), dim=-1) + self.eps)
    return lse
</code></pre>
Is there any way to print a tensor content while debugging in Visual studio?

Anders_G: <p>It seems like such an obvious and simple need. To be able to inspect the content of a tensor while debugging. Still I cannot find a way. Expanding the object in Visual studio just gives useless things. And trying to execute any code in the immediate window either doesn’t give any useful result or isn’t allowed. So how can I view the tensor?</p>

sardanian: <p>Did you ever find a solution to this or a way to work around this issue?</p>

Anders_G: <p>No unfortunately not. I mainly call torch from c# and there I can inspect the content. Would have been nice though.</p>
Visualizing PyTorch images and getting a session management error

papoo13: <p>I am trying to visualize PyTorch image tensors per loader (I have multiple data loaders) as follows:</p>
<pre><code>validation_loaders = []

if len(validation_datasets)==5:
    for validation_dataset in validation_datasets:
        # print one image to sample the labels 
        dict=validation_dataset.__getitem__(0)
        im = dict['input1']
        plt.imshow(im.permute(1, 2, 0))

        validation_loaders.append(DataLoader(
                    validation_dataset,
                    batch_size=args.batch_size,
                    shuffle=False,
                    drop_last=False,
                    **gpuargs))
</code></pre>
<p>but I get this error:</p>
<blockquote>
<p>Qt: Session management error: None of the authentication protocols specified are supported</p>
</blockquote>
<p>I am running the code via ubuntu shell. I also tried to run it via visual studio terminal an in an ubuntu system. The problem is that I do not know where the problem stems from so I can debug it. Is this the correct way to visulize images or I need to do it differently? anyone faced a similar issue?<br>
I appreciate the help.</p>
Apply arbitrary mask to RNN input series

freddym: <p>I’m looking for a way to ignore certain steps of a time series based on a boolean vector. In Keras something like this can be achieved using the mask parameter but I could not figure out how to do this in pytorch:</p>
<p>tensorflow keras example:</p>
<pre><code class="lang-auto">from keras.layers import Input, LSTM, Reshape
from keras.models import Model
import numpy as np
import tensorflow as tf

words = Input(shape=(30, 256), dtype='float32', name='words')

# Create a dummy mask selecting every second step of the time series
np_mask = np.array([x % 2 == 0 for x in range(0,5*30)])
tf_mask = tf.reshape(tf.constant(np_mask), (5, 30))
lstm1 = LSTM(5)(words, mask=tf_mask)
model = Model(inputs=[words], outputs=lstm1)
</code></pre>
<p>I’ve found several posts about using PackedSequence for right-padded series, but I couldn’t find anything for skipping steps somewhere in the middle.</p>
How to understand a large result of nn.NLLLoss() with correct predicts?

evilroach: <p>I’m learning the usage of torch.nn.NLLLoss() and torch.nn.LogSoftmax(), and I’m confused about the results of them.</p>
<p>For example:</p>
<pre><code>lsm = torch.nn.LogSoftmax(dim=-1)
nll = nn.NLLLoss()
grnd_truth = torch.tensor([1])
# let's say it predicted correctly!!!
raw_logits = torch.tensor([[0.3665, 0.5542, -1.0306]])
logsoftmax = lsm(raw_logits)
final_loss = nll(logsoftmax, grnd_truth)

logsoftmax:
&gt;&gt;&gt; tensor([[-0.8976, -0.7099, -2.2947]])
final_loss:
&gt;&gt;&gt; tensor(0.7099)    # Is it normal getting so large loss when predicted correctly?

# let's try a incorrect prediction
grnd_truth = torch.tensor([0])
nll(lsm(raw_logits), grnd_truth)
&gt;&gt;&gt; tensor(0.8976)    # Is this not enough large?
</code></pre>
<p>Obviously, we can see that the mis-classified loss is not large enough, comparing to the loss of a correctly classifing that is near to former.</p>
<p>I read the docs of NLLLoss, and I know its formula. I guessed that the designers must have their reasons to use the formula, but I didn’t have gotten it.</p>
<p>In the fact, the reason of I posting this question, is that I encountered a difficulty to train a triple-classification module with NLLLoss() and LogSoftmax().</p>
<p>Is the goal of a lossing func that make the loss as small as possible when the prediction is right, and vice versa?</p>
Ignore dimension 0 when calculating mean

Blaze: <p>Is there any way I can calculate the mean of all elements in a 4d tensor except along dim 0 - i.e. the equivalent of <code>tensor.mean(1).mean(1).mean(1)</code>?</p>

alexis-jacq: <p>if you know the size of the first dimension you can:</p>
<p><code>tensor.view(size,-1).mean(1)</code></p>

ptrblck: <p>And if you don’t know it, you could use:</p>
<pre><code class="lang-auto">tensor.view(tensor.size(0), -1).mean(1)
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

David_Gilbertson: <p>You can also do:</p>
<pre data-code-wrap="py"><code class="lang-nohighlight">tensor.flatten(1).mean(1)
</code></pre>
Why isn't this network training?

Alio: <p>I’ve been staring at this simple script and don’t know why it isn’t training. The loss is does not budge.</p>
<p><span class="hashtag">#include</span> &lt;torch/torch.h&gt;<br>
using namespace torch::indexing;<br>
torch::Device device(torch::kCUDA);</p>
<p>struct Critic_Net : torch::nn::Module {<br>
torch::Tensor next_state_batch__sampled_action;<br>
public:<br>
Critic_Net() {<br>
// Construct and register two Linear submodules.<br>
lin1 = torch::nn::Linear(427, 42);<br>
lin2 = torch::nn::Linear(42, 286);<br>
lin3 = torch::nn::Linear(286, 42);<br>
lin1-&gt;to(device);<br>
lin2-&gt;to(device);<br>
lin3-&gt;to(device);<br>
}<br>
torch::Tensor forward(torch::Tensor next_state_batch__sampled_action) {<br>
auto h = next_state_batch__sampled_action;<br>
h = torch::relu(lin1-&gt;forward(h));<br>
h = torch::tanh(lin2-&gt;forward(h));<br>
h = lin3-&gt;forward(h);<br>
return torch::nan_to_num(h);<br>
}<br>
torch::nn::Linear lin1{nullptr}, lin2{nullptr}, lin3{nullptr};<br>
};</p>
<p>auto one = torch::rand({42, 427}).to(device);</p>
<p>auto target = torch::rand({42, 42}).to(device);</p>
<p>torch::Tensor y_hat;<br>
at::Tensor loss;</p>
<p>auto net = Critic_Net();<br>
auto net_optimizer = torch::optim::Adam(net.parameters(),  1e-3);</p>
<p>int main() {<br>
for (int e = 0; e&lt;2000; e++) {<br>
net_optimizer.zero_grad();<br>
y_hat = net.forward(one);</p>
<pre><code>    loss = torch::mse_loss(target, y_hat);
    loss.backward();
    net_optimizer.step();

    if (e % 50 == 0) {
        std::cout &lt;&lt; loss.item() &lt;&lt; "  " &lt;&lt; "\n";
    }
}
</code></pre>
<p>}</p>

Matej_Kompanek: <p>Torch is not aware of your parameters if there is no ‘register_module’ in your module, e.g. if you do this, you’ll see there are no parameters</p>
<pre><code class="lang-auto">std::cout &lt;&lt; net.parameters() &lt;&lt; std::endl;
</code></pre>
Small clarification about FSDP docs

Brasilino: <p>Hello!</p>
<p>On the following post below, Figure 1 says that for each layer N, all parameters are gathered (all-gather), forward happens, then all weights/full weights are freed. However, all weights <em><strong>besides layer N weights</strong></em> are freed, right ? Otherwise, in my understanding, they won’t be all-gathered on the backward pass.</p>
<p>I wonder if the “besides layer N weight” are implicit or not in the post. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>Thanks!<br>
Lucas</p>

agu: <p>Could you link the figure?</p>
<p>I may be misunderstanding what you are saying, but why do you think that if layer <code>N</code>’s parameters are freed, then they will not be all-gathered in the backward pass?</p>
<p>The backward pass has an additional all-gather for the parameters that is separate from the one in the forward pass. It would not make sense to say that all parameters besides layer <code>N</code>’s parameters are freed because the parameters besides layer <code>N</code>’s parameters have already been freed / are not in memory – that is how FSDP is able to save memory.</p>

Brasilino: <p>Hello <a class="mention" href="/u/agu">@agu</a></p>
<p>I’m sorry, I’ve posted without the link.This is the Figure: <a href="https://pytorch.org/assets/images/fsdp_workflow.png" rel="noopener nofollow ugc">https://pytorch.org/assets/images/fsdp_workflow.png</a><br>
From: <a href="https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api/" class="inline-onebox" rel="noopener nofollow ugc">Introducing PyTorch Fully Sharded Data Parallel (FSDP) API | PyTorch</a></p>
<p>I’m trying to have a global understanding of the operation by reasoning about that Figure and this one: <a href="https://engineering.fb.com/wp-content/uploads/2021/07/FSDP-graph-2a.png" rel="noopener nofollow ugc">https://engineering.fb.com/wp-content/uploads/2021/07/FSDP-graph-2a.png</a></p>
<blockquote>
<p>I may be misunderstanding what you are saying, but why do you think that if layer <code>N</code> ’s parameters are freed, then they will not be all-gathered in the backward pass?</p>
</blockquote>
<p>I’m getting into the details how PyTorch uses collective communication, but for the allgather to happen (in OpenMPI, for instance) each worker (FSDP instance here) must have a valid (i.e. not freed) sending buffer, right ? Am I overlooking/misunderstanding some point ? (I probably am <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> )</p>
<p>Thanks!<br>
Lucas</p>

agu: <p>I think I understand the confusion.</p>
<p>Each worker keeps a local shard of the model parameters for the layer so that it can contribute that shard in the all-gather. When the diagram says that the model parameters are freed after the forward pass, everything except for the rank’s local shard is freed.</p>
Is there an equivalent to numpy.max with axis option in torch

Roee_Sfaradi: <p>Hi,<br>
I want to do maximum along specific axis for example like in numpy:</p>
<pre><code class="lang-python">x = np.random.rand(3,4,5,6)
x.shape
-----------
(3, 4, 5, 6)
</code></pre>
<p>Followed by:</p>
<pre><code class="lang-python">y=x.min(axis=(1,2))
y.shape
----------
(3, 6)
</code></pre>
<p>That is - the min function reduced the (1,2) axis in the above example.</p>
<p>The min function in torch has the <code>dim</code> parameter - and it can include only one axis, and when used, it doesn’t return a reduced tensor.</p>
<p>Is there a convenient way to overcome this?</p>
<p>Thanks!<br>
Roee</p>