akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training with a random fraction data from data loader every epoch

Hasan_Sait_ARSLAN1: <p>Hi,</p>
<p>I have a very large dataset to train my model. I would like to get n different random batches from the DataLoader at every different epoch. How can I do it?</p>
<p>Thanks,</p>

ptrblck: <p>You could create an iterator via <code>loader_iter = iter(loader)</code> and get the next batch via <code>batch = next(iter)</code>. If you’ve set <code>shuffle=True</code> in the <code>DataLoader</code>, these batches will contain random samples.</p>

DavidManga: <p>Hello i cannot quite understand your answer could you please expand on it</p>

ptrblck: <p>Could you describe what exactly is unclear in my post and where you are currently stuck?</p>

DavidManga: <p>Well honestly speaking am quite new to pytorch.<br>
So I have a very huge dataset that it is taking a lot of time during the training process. I just need the dataloader to use a fraction of the dataset for every iteration.<br>
Please help make your answer friendly for a newbie like me</p>

ptrblck: <p>If you want to load a specific subset of the <code>Dataset</code>, use <a href="https://pytorch.org/docs/stable/data.html?highlight=subset#torch.utils.data.Subset"><code>torch.utils.data.Subset</code></a> and pass your <code>Dataset</code> object as well as the desired indices to it.<br>
My code snippet is manually creating an <code>iterator</code> and is getting a single batch via the <code>next</code> call, which doesn’t seem to match your use case.</p>
Default number of Inter and Intra-ops threads during inference

Hakudoshi: <p>Hi everyone,<br>
I’m trying to use a Roberta model for inference on CPU in production environment. The model is trained in python and then exported to a TorchScript model for inference in Java (using the libtorch library).<br>
During Inference I see that multiple threads and cores are being utilized. when running on a machine with 40 cores and running htop I noticed that 20 threads are running and the CPU utilization is at around 1700%. Also running on a 16 core machine I get around 450% CPU utilization.<br>
According to this doc: <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html#runtime-api" class="inline-onebox" rel="noopener nofollow ugc">CPU threading and TorchScript inference — PyTorch 1.12 documentation</a> the default number of threads used by pytorch for intra-op parallelism is the number of CPU cores.<br>
Since I don’t observe these numbers using htop (and other monitoring mechanisms) I wonder whether scripted models use a different default configuration then the one mentioned in the doc?</p>
<p>Can someone please shed light on this for me? Also any suggestion on how to speed up inference on CPU under libtorch would be greatly appreciated!</p>
Convert learning model .onnx to pytorch and use it by matlab engine

li_James: <p>Hi all, I meet a problem. There is a learning model .onnx. It can be used in Matlab and it can be loaded in python. For some reasons, I need to use matlab engine in python. The learning model .onnx can not be used in python by matlab engine. The matlab function call is ‘predictAndUpdateState’(required deep learning tool box),<br>
The python script looks like:<br>
import onnx<br>
import matlab.engine</p>
<p>matlab_engine = matlab.engine.start_matlab()<br>
learning_model = onnx.load(‘XXX.onnx’)<br>
[learning_model, ans ] = matlab_engine.predictAndUpdateState(learning_model, input)</p>
<p>the error is 'unsupported Python data type: onnx.onnx_ml_pb2.ModelProto"<br>
So, what should I do?<br>
Thanks</p>

ptrblck: <p>This issue doesn’t seem to be PyTorch-related so I would recommend to post the question in a MATLAB or ONNX - specific discussion board.</p>

li_James: <p>thank you, Do you know that does the Pytorch has the same function as matlab learning function ‘predictAndUpdateState’ ?</p>

ptrblck: <p>No, PyTorch does not have a <code>predictAndUpdateState</code> function, but based on its name I would guess it triggers a full training iteration.<br>
In this case you could write the actual training loop as seen <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network">here</a> or wrap it additionally in another function.</p>

li_James: <p>Okay, thank you. I will try</p>
Is there a PyTorch equivalent to tf.reduce_max()?

ToruOwO: <p>If so, would really appreciate it if someone could point me to the definition. Thanks!</p>

Tony-Y: <p>How about <a href="https://pytorch.org/docs/stable/torch.html#torch.max" rel="nofollow noopener">torch.max</a>?</p>
<pre><code class="lang-auto">torch.max(input, dim, keepdim=False, out=None) -&gt; (Tensor, LongTensor)
</code></pre>
<p>Returns the maximum value of each row of the input tensor in the given dimension dim.</p>

manoj_batra: <p>If the input tensor becomes empty torch.max(), will give an error vs tf.reduce_max will give -inf.</p>
<p>Is there someway we can retain the same behavior as tf.</p>
<p>Example:<br>
torch.max(torch.tensor([]))<br>
RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the ‘dim’ argument.</p>
<p>tf.reduce_max(tf.constant([]))<br>
&lt;tf.Tensor: shape=(), dtype=float32, numpy=-inf&gt;</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/convert-tensorflow-code-to-pytorch/69996/5">here</a> with follow up.</p>
How to retrieve the class names of a trained ResNet?

Omar_Zayed: <p>I have a model that I trained around a year ago, and now when I try to make predictions, I can’t retrieve the names of the classes so I don’t know which class does it refer to.</p>
<pre><code class="lang-auto">def predict_image(img, model):
    # Convert to a batch of 1
    device = torch.device('cpu')
    xb = to_device(img.unsqueeze(0), device)
    # Get predictions from model
    yb = model(xb)
    # Pick index with highest probability
    _, preds  = torch.max(yb, dim=1)
    # Retrieve the class label
    return train_ds.classes[preds[0].item()]
</code></pre>
<p>I have no access to train_ds</p>
How to trace module inputs and outputs?

Jan_Pohanka: <p>I want to export my pytorch model (similar to yolo) for our own FPGA accelerator. I understand how to get out the modules (eg. weights/biases of convolutional modules etc.). What I do not know is how to trace inputs and outputs.<br>
The model is not ‘linear’, there are some residual connections. Is there any standard way how this can be described?</p>
<p>I’d like to get a graph depicting that eg. output from first layer is used by the layer two and five…</p>
<p>thanks</p>
Customed connection

Omega_Ma: <p>Hi,<br>
I want to custom a layer with the following structure:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b934f2e56d81cc38e839bb19d4702fc5e2610fc.png" alt="layer" data-base62-sha1="3VWv1CWZzBlfwGQeeHYGgtFqdzK" width="216" height="296"><br>
Here is the code for the customer layer:</p>
<pre data-code-wrap="pythoscripts"><code class="lang-pythoscripts">class MaskedLinear(nn.Module):
    def __init__(self, in_dim, out_dim):
        super(MaskedLinear, self).__init__()

        self.weight = nn.Parameter(torch.randn([in_dim, out_dim]))
        # print(f'0 weight is {self.weight}')
        self.bias = nn.Parameter(torch.randn([1, out_dim]))
        # print(f'0 bias is {self.bias}')
        self.mask = torch.eye(out_dim) # create mask
        # print(f'mask is {self.mask}')

    def forward(self, input):
        self.weight = nn.Parameter(self.weight * self.mask)
        # print(f'1 weight is {self.weight}')
        return torch.matmul(input, self.weight) + self.bias
</code></pre>
<p>I got a  problem. I check the weights after backpropagation. The weights are not updated and always be the initial value. I don’t know what the reason is. Any suggestions?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="Omega_Ma" data-post="1" data-topic="159299">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omega_ma/48/52185_2.png" class="avatar"> Omega_Ma:</div>
<blockquote>
<pre><code class="lang-auto">        self.weight = nn.Parameter(self.weight * self.mask)
</code></pre>
</blockquote>
</aside>
<p>Don’t re-wrap your <code>self.weight</code> with <code>nn.Parameter</code> as that will break your graph, just re-define the weight as the mask times the weight.</p>
<p>Also, if you have <code>N</code> inputs and <code>N</code> outputs and you want to basically want to do an element-wise scaling, why don’t you just define the weight as a vector as the same size as the input then do <code>out = weight * in + bias</code>? (so you have something that scales in linear time rather than cubic time, e.g. matmul)</p>

Omega_Ma: <p>yes,  you are right, thanks for the reminder</p>
Multiple forward passes on the same input - detach or clone?

thomasbbrunner: <p>I have some inputs which I want to pass multiple times through my model. This model is stochastic, i.e. the outputs are different for each pass. Something like this:</p>
<pre><code class="lang-auto"># multiple forward passes
x = [...]
outputs = []
for i in range(num_passes):
    input = x  # TODO clone? detach?
    outputs.append(model(input))

# backpropagation through outputs
(...)
</code></pre>
<p>I want the gradient graph for each of the passes to be from the input to the corresponding output.</p>
<p>Do I have to call clone() or detach() in the inputs to make sure that the gradients flow back properly?</p>
<p>Here’s an example script that can be run:</p>
<details>
<summary>
Script</summary>
<pre><code class="lang-auto">import numpy as np
import torch

device = "cuda"
num_passes = 5

# generate dataset
X = torch.Tensor(np.linspace(-10, 10, 100)[..., None]).to(device)
Y = torch.sin(X).to(device)

class MLP(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = torch.nn.Sequential(
            torch.nn.Linear(1, 64),
            torch.nn.LeakyReLU(),
            torch.nn.Dropout(0.1),
            torch.nn.Linear(64, 64),
            torch.nn.LeakyReLU(),
            torch.nn.Dropout(0.1),
            torch.nn.Linear(64, 1)).to(device)

    def forward(self, x):
        self.train()
        predictions = torch.zeros((num_passes, *x.shape), device=device)
        for i in range(num_passes):
            input = x  # TODO clone? detach?
            predictions[i] = self.model(input)
        return predictions


# repeat desired output to match shape of predictions
# (num_passes, ...)
Y_repeated = Y.unsqueeze(0).repeat_interleave(num_passes, 0)
model = MLP()
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
for epoch in range(100):
    optimizer.zero_grad()
    preds = model(X)
    loss = loss_fn(preds, Y_repeated)
    loss.backward()
    optimizer.step()
</code></pre>
</details>
Constants consume too much gpu memory in torch.fx

gpzlx1: <p>I have the following code</p>
<pre><code class="lang-python">class MyModule(torch.nn.Module):

    def __init__(self, param):
        super().__init__()
        # To simplify, self.param is a large scale tensor
        # In my scenario, it will be a custom class, which is large and consumes 3-10GB cuda memory
        self.param = param 
 
    def forward(self):
        one = self.param
        two = one + one
        three = two + one
        return one, two, three

param = torch.ones((1024, 1024, 768)).cuda()
f = MyModule(param)

torch.cuda.reset_peak_memory_stats()
print(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024)

gm = fx.symbolic_trace(f)

torch.cuda.reset_peak_memory_stats()
print(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024)

gm()

torch.cuda.reset_peak_memory_stats()
print(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024)
</code></pre>
<p>The output is</p>
<pre data-code-wrap="shell"><code class="lang-nohighlight">3.0 GB
9.0 GB
9.0 GB
</code></pre>
<p>For self.param, torch.fx treats it as a constant and then caches the results of all calculations of that constant, resulting in a very large memory overhead. Is there any way to stop this behavior?</p>

gpzlx1: <p>In short, how to trace a global var in <code>torch.fx</code>?</p>

gpzlx1: <p>It seems that the only solution is set <code>param</code> as forward args rather module attr.</p>
PyTorch's share_memory() release strategy

Eyal: <p>Hello,<br>
I have a system with uses multiple threads and processes, which communicate with one another using queues. For optimization I’m storing the Tensor’s which are passing between them in shared memory.</p>
<p>I’m experiencing a memory leak, and I suspect it relates to the fact that the completed tensors are not released from share memory. It would be helpful to get input on how pytorch’s shared memory objects release is performed / how to handle it.<br>
Thanks!</p>
Multiple CUDAGraphs for single model with different shape inputs

MHueting: <p>I am loving the new CUDAGraph functionality in PyTorch. I am trying to graph a transformer-based model, and if I fix the shapes to always use the maximum sequence length, then everything works great.</p>
<p>However, my training data comes in a few different sequence lengths. Let’s say for example’s sake I have 4 different sequence lengths: 16, 24, 32, and 48.</p>
<p>Would it be possible for me to capture multiple CUDAGraph for the same module, so that I have one replayable graph for that module for each shape/sequence length I can expect to give it?</p>
<p>I’m imagining something like the following:</p>
<pre><code class="lang-auto">module = MyModule()

# create each graph based on shapes
expected_shapes = [[16, 16], [16, 24], [16, 32], [16, 48]]
rand_inputs = [torch.randn(shape) for shape in expected_shapes]
graphs = [torch.cuda.make_graphed_callable(module, (input,)) for input in rand_inputs]

# replay correct graph during training
for sample in dataloader:
    # find correct graph idx based on shape
    graph_idx = get_graph_idx(sample.shape)
    output = graphs[graph_idx](sample)
</code></pre>
<p>So far this doesn’t seem to work, as the second call to <code>make_graphed_callable</code> complains about mismatching shapes. Anything I can do here?</p>

MHueting: <p>A minimum working example of what I’d hope would work, but doesn’t, is printed here. Instead of varying sequence length I’m using varying batch sizes here, but the result is the same:</p>
<pre><code class="lang-auto">import torch

class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()

        self.linear = torch.nn.Linear(5, 1)

    def forward(self, x):
        return self.linear(x)

if __name__ == "__main__":
    module = MyModule()

    expected_shapes = [[2, 5], [4, 5], [6, 5]]
    rand_inputs = [torch.randn(shape) for shape in expected_shapes]

    graphed_callables = []
    for input in rand_inputs:
        graphed_module = torch.cuda.make_graphed_callables(module, (input,))
        graphed_callables.append(graphed_module)

    new_inputs = [torch.randn(shape) for shape in expected_shapes]
    for graphed_callable, input in zip(graphed_callables, new_inputs):
        graphed_callable(input)
</code></pre>
<p>When running this, the second call to <code>torch.cuda.make_graphed_callables</code> throws a RuntimeError:</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (2) must match the size of tensor b (4) at non-singleton dimension 0
</code></pre>
About pack_padded_sequence, pad_packed_sequence

Nguyen_Duy: <p>Hello everyone,<br>
I have a question as follows: Can I use pack_padded_sequence and pad_packed_sequence functions when working with Transformer and MultiHeadAttention classes?<br>
The shape of my input data is [batch_size, num_sequences, max_sequence_length]. At dim=1, there are some sequences consisting entirely of pad values.<br>
Example:<br>
Input data :<br>
[[[3, 2, 4, 5],<br>
[2, 3, 1, 1],<br>
[4, 3, 9, 1]],<br>
[[6, 3, 2, 5],<br>
[1, 1, 1, 1],<br>
[1, 1, 1, 1]]]<br>
(batch_size = 2, num_sequences = 3, max_sequence_length=4, padding value = 1)</p>
<p>I want to remove those sequences before doing the forwarding of the above classes.<br>
Thanks for all the support.</p>
What is the parameter "gradient_as_bucket_view" in DDP

coaksidenc: <p>Hello</p>
<p>Among the parameters available in DDP, what does “gradient_as_bucket_view” do?<br>
If I use this parameter as True, the memory of the GPU is reduced. What is the reason?</p>

rvarm1: <p><code>gradient_as_bucket_view</code> enables DDP’s internal implementation to avoid a copy for each parameter gradient, thereby reducing memory.</p>

coaksidenc: <p>Oh, thnks.<br>
As an additional question, why need to copy gradients within DDP?</p>
Data Type error Cannot solve

ANIL_PUDASAINI: <p>TypeError: expected TensorOptions(dtype=__int64, device=cpu, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)) (got TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)))</p>

ANIL_PUDASAINI: <pre><code>        loss = criterion(output,targets.view(batch_size*seq_length).type(torch.LongTensor))
        loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        opt.step()
</code></pre>

ANIL_PUDASAINI: <p>changing the data type to torch.LongTensor does not help. (i am working via CPU only)</p>

ptrblck: <p>Using a <code>LongTensor</code> for <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code> should work and the error points to a <code>dtype</code> mismatch as a <code>FloatTensor</code> seems to be passed.<br>
Could you post a minimal, executable code snippet which would reproduce the issue, please?</p>
How long dose train the model with a pernal computer with cuda

wter26: <p>My computer is CPU i5 4590,gtx750,RAM 8G,only has 640 Stream processors.<br>
I want to train a model of ocr which the datasets has 3.6 million images.<br>
There are 3.6 million images in total, with the image resolution of 280*32 and the file size of about 8.6GB.<br>
If I trained with my pc.How long would it take? One year ?I just dont konw.</p>

ptrblck: <p>You should write the training routine using your desired model, profile a few iterations, and try to estimate how long the entire training might take. Everything else would be just a wild guess as we don’t even know which kind of model you are using, what preprocessing is applied etc.</p>
If I don't use `torch.chunk` will I have backprop errors?

Vedant_Roy: <p>I currently have the following tensor:</p>
<pre><code class="lang-auto">torch.Size([2, 2, 128, 1, 1])
</code></pre>
<p>Normally the batch dimension is at the start, but here I’ve split the tensor on the number of channels. I.e.,</p>
<pre><code class="lang-auto">(2, 512) =&gt; (2, 2, 128, 1, 1)
</code></pre>
<ul>
<li>1st dimension is the “split” dimension</li>
<li>2nd dimension is batch dimension</li>
<li>3rd dimension is # of channels</li>
<li>Last 2 dimensions are 2 dimensions I added for broadcasting</li>
</ul>
<p>Right now, I do the following:</p>
<pre><code class="lang-auto">        cond_w, cond_b = self.emb_layers(emb)
        assert cond_w.shape == (N, self.out_channels, 1, 1)
        x = self.out_norm(x) * (1 + cond_w) + cond_b
</code></pre>
<p>so I just index into the returned tensor and split it into 2 child tensors.<br>
I’m wondering if this will effect backprop, as I never see people do this.</p>
<p>The alternative is:</p>
<pre><code class="lang-auto">        scale_shift = self.emb_layers(emb)
        cond_w, cond_b = th.chunk(scale_shift, 2, dim=1)
        cond_w = cond_w.squeeze(1)
        cond_b = cond_b.squeeze(1)
        assert cond_w.shape == cond_b.shape
        assert cond_w.shape == (N, self.out_channels, 1, 1)
</code></pre>
<p>is one method preferable to the alternative?</p>

ptrblck: <p><code>torch.chunk</code> will not detach the tensors from the computation graph and will yield the same results as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 3, 224, 224, requires_grad=True)
model = models.resnet18().eval()

# full batch
x.grad = None
out = model(x)
out.mean().backward()
print(x.grad.abs().sum())
# tensor(1.1471)

# chunk
x.grad = None
x1, x2 = torch.chunk(x, 2, 0)

out1 = model(x1)
out2 = model(x2)
out12 = torch.cat((out1, out2), 0)
out12.mean().backward()

print(x.grad.abs().sum())
# tensor(1.1471)
</code></pre>
<p>(assuming all other operations are equal).</p>
Time series forecasting in year intervals

Christian_Short: <p>I am pretty new to ML and creating my own prediction models. My case involves a lot of data that continues to grow every hour such as solar data that has yearly/monthly/daily trends. There are many different cases for the data that can vary from system to system and I want to be able to create a model that can use the years worth of data in the past and use that model to predict the estimated solar production for any given time range by learning trends over the previous years.</p>
<p>For example If I have data for a solar system from May 25, 2021 - Today and want to predict what the production could look like in November of 2023 I would want to train the model on the existing data and input my predicted time range as November 10, 2023 - November 12, 2023.</p>
<p>Is there any good tutorials on my case or something I should read into to get started on this kind of prediction method?</p>
"TypeError: __init__() takes 1 positional argument but 2 were given"

Hazrat: <p><strong>I am unable to find a solution to this error:</strong></p>
<p>—&gt; 92         image = T.ToTensor(image)<br>
93         if not self.long_mask:<br>
94             mask = T.ToTensor(mask)</p>
<p>TypeError: <strong>init</strong>() takes 1 positional argument but 2 were given</p>

ptrblck: <p>You have to create the transformation object first before calling it or you could use the functional API directly:</p>
<pre><code class="lang-python">transforms.ToTensor(image) # wrong
# TypeError: __init__() takes 1 positional argument but 2 were given

# create the object first
transform = transforms.ToTensor()
out = transform(image)

# functional API
out = torchvision.transforms.functional.to_tensor(image)
</code></pre>

Hazrat: <pre><code>   Thank you for your reply. The first solution works. But, the second one is not working. Can you explain it?
</code></pre>
<p>—&gt; 92         image = F.to_tensor(image)<br>
93         if not self.long_mask:<br>
94             mask = F.to_tensor(mask)</p>
<p>AttributeError: module ‘torch.nn.functional’ has no attribute ‘to_tensor’</p>
<pre><code>   70     def __call__(self, image, mask):
   71         # transforming to PIL image
</code></pre>
<p>—&gt; 72         image, mask = F.to_pil_image(image), F.to_pil_image(mask)<br>
73<br>
74         # random crop</p>
<p>AttributeError: module ‘torch.nn.functional’ has no attribute ‘to_pil_image’</p>

ptrblck: <p>In my code snippet you can see the proper namespace as <code>torchvision.transforms.functional</code> so use this instead of <code>torch.nn.functional</code>.</p>
What's the difference between torch.reshape vs. torch.view

maxrivera: <p>It seems like torch.reshape and torch.view do pretty much the same thing, am I missing something? Is there a situation where you would use one and not the other?</p>

ptrblck: <p><code>reshape</code> will return a view if possible and will trigger a copy otherwise as explained in the <a href="https://pytorch.org/docs/stable/generated/torch.reshape.html?highlight=reshape#torch.reshape">docs</a>. If in doubt, you can use <code>reshape</code> if you do not explicitly expect a view of the tensor.</p>

maxrivera: <p>I’m having a hard time understanding what a view is… it’s like a reshaped vector with constraints?</p>

ptrblck: <p>A <code>view</code> points to the same data stored in memory using a changed meta-data such as its shape and stride.<br>
Here is a small example:</p>
<pre><code class="lang-python">x = torch.randn(2, 4)
print(x.size(), x.stride())
# torch.Size([2, 4]) (4, 1)
print(x.is_contiguous())
# True

y = x.view(-1)
print(y.size(), y.stride())
# torch.Size([8]) (1,)
print(y.is_contiguous())
# True

x = torch.randn(2, 4, 8)
z = x[:, ::2]
print(z.size(), z.stride())
# torch.Size([2, 2, 8]) (32, 16, 1)
print(z.is_contiguous())
# False

z1 = z.view(-1)
# RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

z2 = z.reshape(-1)
print(z2.is_contiguous())
# True
print(z2.size(), z2.stride())
# torch.Size([32]) (1,)
</code></pre>
I miss the warning that grad and param do not obey the gradient layout contract

Tnak: <p>I have located the line of code that caused this warning, but I do not know how to fix it. I have tried the solutions provided by others, which is to add ‘.contiguous()’, but no matter how I modify it, I can not improve the warning. The code caused this warning is as follows:<br>
x = x + self.drop_path(self.mlp(self.norm(x), H, W))</p>
<p>However, there is also a strange phenomenon that using “x = x + self.drop_path(self.mlp(x, H, W))”, this warning will be solved.</p>
<p>Please help me!</p>

Tnak: <p>I resolved the warning! The reason for it is not <code>norm</code>. Because I added <code>DWConv</code> to self.mlp, which contains <code>transpose().view()</code>, which is actually the culprit that caused the warning.</p>
<p>The modified code is <code>x = x.transpose(1, 2).contiguous().view(N, C, H, W)</code>.</p>
How am I supposed to apply spectral norm in my model?

Andrew_Legend: <p>I am not sure if I apply spectral norm in my model properly.<br>
More specifically I am applying it like this with the parametrization module</p>
<pre><code class="lang-auto">class TestModel(nn.Module):
    def __init__(self):
        super(TestModel, self).__init__()
        self.input_conv= nn.utils.parametrizations.spectral_norm(nn.Conv2d(3, 64, 5, padding=2), n_power_iterations=5)
        self.i_activation= nn.LeakyReLU()
        
        self.downconv1= nn.utils.parametrizations.spectral_norm(nn.Conv2d(64, 64, 3, padding=1), n_power_iterations=5)
        self.d_activation1= nn.LeakyReLU()
    def forward(self, x):
        x= self.input_conv(x)
        x= self.i_activation(x)
        
        x= self.downconv1(x)
        x= self.d_activation1(x)
        return x
</code></pre>
<p>But my critic appears to train like spectral norm does not exist at all. I have done a lot of debugging to see what is going on. So what I am not (yet) sure is if I am doing this properly or not.</p>
Iterate through loader until a condition is met

saandeep_aathreya: <p>I need to build a data loader which iterates through the loader until a certain sample size is met. For example, if my loader looks like below.</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader, Dataset


class CustomLoader(Dataset):
  def __init__(self) -&gt; None:
    super().__init__()

    self.all_files = torch.randn((250))

  
  def __len__(self):
    return len(self.all_files)

  def __getitem__(self, idx):
    rnd = self.all_files[idx]
    x = torch.randn((21, 20, 4)) * rnd
    target = idx

    return x, target

if __name__ == "__main__":
  dataset = CustomLoader()
  loader = DataLoader(dataset, batch_size=8, shuffle=True)

  for b, (x, target) in enumerate(loader, 0):
    bat, peopl, tim, feat = x.size() # 8 * 21 * 20 * 4

    # Flatten x dimension
    x = x.view(bat * peopl, tim, feat) # 168 * 20 * 4

    # Some Condition. Output size varies
    rand_el = torch.randint(0, 22, size=(1,)).item()
    x = x[:rand_el]

    print(x.size()) # 13 * 20 * 4
</code></pre>
<p>My <code>x</code> is of dim <code>8 X 21 X 20 X 4</code>, where <code>8</code> is my batch size. Once I have the data, I combine the <code>batch</code> and <code>21</code> dimension which then becomes <code>168 X 20 X 4</code>. I apply some condition and filter out the data. So my final dimensions can then be <code>13 X 20 X 4</code>. I need to iterate through next batch until my data is append to <code>32</code> batch size.</p>
<p>What is the best way to go about this problem. Applying this filter in dataset class itself outputs variable length input which cannot collate my data effectively. A custom collate function does not allow me to iterate through next batch as far as I know. Please advise.</p>
Dictionary in Data Louder

Ab_is: <p>Hey folks ,<br>
I have a dictionary as dataset , I am curious  to know how to use this as a data in training Pytorch model ?</p>
<p>Any one can help me please</p>

nivek: <p>You can create a custom <code>Dataset</code> with a <code>__getitem__</code> method that access items within the dictionary when given an index. Then you can pass that <code>Dataset</code> into <code>DataLoader</code>. Please have a look at the <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html" rel="noopener nofollow ugc">tutorial</a>.</p>
Unexpected result when operating tensor in mps

Lucass: <p>The result of following code seems not right:</p>
<pre><code class="lang-auto">device = 'mps'
list_ = torch.zeros(3, 2)
print('init:\n{}'.format(list_))

for i in range(3):
    print('---------------%i---------------' %i)
    ten = torch.tensor([i,i], device=device)
    print('tensor:\n{}'.format(ten))
    list_[i] = ten
    print("ten:\n{}\nlist:\n{}".format(ten, list_), end='\n\n')
</code></pre>
<p><strong>result:</strong></p>
<pre><code class="lang-auto">init:
tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])
---------------0---------------
tensor:
tensor([0, 0], device='mps:0')
ten:
tensor([0, 0], device='mps:0')
list:
tensor([[0., 0.],
        [0., 0.],
        [0., 0.]])

---------------1---------------
tensor:
tensor([1, 1], device='mps:0')
ten:
tensor([4575657222473777152,                   1], device='mps:0')
list:
tensor([[1.0000e+00, 1.0000e+00],
        [1.4013e-45, 0.0000e+00],
        [0.0000e+00, 0.0000e+00]])

---------------2---------------
tensor:
tensor([2, 2], device='mps:0')
ten:
tensor([4611686019501129728,                   2], device='mps:0')
list:
tensor([[2.0000e+00, 2.0000e+00],
        [2.8026e-45, 0.0000e+00],
        [0.0000e+00, 0.0000e+00]])
</code></pre>
<hr>
<p>move <strong>list_</strong> to mps:</p>
<pre><code class="lang-auto">device = 'mps'
list_ = torch.zeros(3, 2).to(device)
print('init:\n{}'.format(list_))

for i in range(3):
    print('---------------%i---------------' %i)
    ten = torch.tensor([i,i], device=device)
    print('tensor:\n{}'.format(ten))
    list_[i] = ten
    print("ten:\n{}\nlist:\n{}".format(ten, list_), end='\n\n')
</code></pre>
<p><strong>result:</strong></p>
<pre><code class="lang-auto">init:
tensor([[0., 0.],
        [0., 0.],
        [0., 0.]], device='mps:0')
---------------0---------------
tensor:
tensor([0, 0], device='mps:0')
ten:
tensor([0, 0], device='mps:0')
list:
tensor([[0., 0.],
        [0., 0.],
        [0., 0.]], device='mps:0')

---------------1---------------
tensor:
tensor([1, 1], device='mps:0')
ten:
tensor([1, 1], device='mps:0')
list:
tensor([[1., 1.],
        [0., 0.],
        [0., 0.]], device='mps:0')

---------------2---------------
tensor:
tensor([2, 2], device='mps:0')
ten:
tensor([2, 2], device='mps:0')
list:
tensor([[2., 2.],
        [0., 0.],
        [0., 0.]], device='mps:0')
</code></pre>