akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training slower on GPU than on CPU

CherubimHD: <p>Hi,</p>
<p>I have a regular DDQN algorithm with a custom environment. Interestingly, the training is slower on the cloud machine I have access to through my university than on my own laptop. The cloud machine has a GPU and my laptop does not.<br>
At first I thought that might be due to constantly moving the newest state from the environment to the GPU in order to run the model with that state as input. However, I have now completely rewritten the environment to use torch and execute basically all computations directly on the GPU, i.e. there is virtually no place anymore where I have to move a tensor from CPU to GPU (I searched my whole project for any moving of tensors and there is no such operation as I also initialize most tensors directly on GPU). When I check the device of the most important tensors in my code I get that they are on ‚cuda‘.</p>
<p>I really dont know where the bottleneck of my code on the cloud machine could be. Is it possible that the GPU execution is slower than the CPU execution because during simulation I basically have a batchsize of 1 (and only during replay I have higher batchsizes)?</p>
<p>I‘m grateful for any suggestions!</p>
<p>PS: I cant really use the torch performance profiler since it would require a built from source on the cloudmachine and I really don‘t want to do this.</p>
<p>EDIT: The difference in speed is quite significant as training on my laptop is twice as fast.</p>

ptrblck: <p>I don’t think you need to rebuild PyTorch from source to be able to use the profiler (via Kineto), but might be wrong. In any case, profiling via e.g. Nsight Systems would also work using the PyTorch binaries, which would help isolate the bottleneck.</p>

raharth: <p>I have seen a similar problem when using very small environments and NNs. In my case it was a 10 node(?) model with a single hidden layer.<br>
In my case it was partially due to the swapping, but also my model was as small that it was not able to benefit from the parallel execution of the GPU, instead it suffered from the much lower clock rate.</p>
<p>So in case that your model is REALLY small this might be helpful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Correct way to use sync batch norm for using apex and DDP

111429: <p>Hi, I am using apex and multi-node multi-gpu training.<br>
I wonder what’s the recommended way to setup sync_bn across nodes/cards.</p>
<p>In <a href="https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py#L145" rel="noopener nofollow ugc">Nvidia’s official apex Imagenet example</a>, it uses <code>apex.parallel.convert_syncbn_model()</code></p>
<p>In <a href="https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L190" rel="noopener nofollow ugc">torchvision’s official video-classificaiton example</a>, it uses <code>torch.nn.SyncBatchNorm.convert_sync_batchnorm</code></p>
<p>What’ the difference, and the recommended usage for distributed training?<br>
Thanks.</p>

ptrblck: <p>We recommend to use the native implementations for mixed-precision training as well as the <code>DistributedDataParallel</code>utilities, as the apex implementations were used as a proff of concept and the native implementations will be improved in the future.</p>

jiale: <p>Hi, does your “native” means that we sholud use “torch.nn.SyncBatchNorm.convert_sync_batchnorm + torch.nn.parallel.DistributedDataParallel” ? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

ptrblck: <p>Yes, by “native” I mean the classes and functions from the <code>torch</code> namespace, not the (older and now deprecated) <code>apex</code> implementations.</p>
Canonical way to achieve variable batch sizing, which works best with PyTorch’s infrastructure?

Vadim: <p>Hi All,</p>
<p>My model’s input examples are sequences of highly-variable length.  If I pack a fixed number of these sequences into a batch tensor (which is allocated to accommodate the longest sequence), I eventually run into a case where too many long sequences end up in the same batch, and my model runs out of memory.<br>
Finding a “safe” batch size is annoyingly time-consuming, because the above situation may arise many epochs into the training.</p>
<p>I would prefer to find a way to assemble my batches such that the sum of lengths of all sequences in them is roughly constant.   How can I achieve this with PyTorch DataLoader?</p>
<p>The only way I could think of, is to create a data-aware batch sampler.   However, in this case I would need to pre-scan all data to find how long each example is…   The solution also needs to work with DistributedDataParallel… can I have a distributed batch sampler?</p>
<p>So, is there a canonical way to achieve variable batch sizing, which works best with PyTorch’s infrastructure?</p>
Min/max in RENDEZVOUS and world_size

HuangLED: <p>Trying to understand the connection between these two concepts, after reading this page: <a href="https://pytorch.org/docs/1.9.0/elastic/rendezvous.html" class="inline-onebox" rel="noopener nofollow ugc">Rendezvous — PyTorch 1.9.0 documentation</a></p>
<p>Do we need this min/max if I already know the exact number machines under my control?</p>
<p>Let us say I have two dedicated machines that I want to use for training. That means I want world_size=2.  Then both min and max should be exactly 2 for the intended allocation, right?   In what kind of scenario one would want to set min/max differently to take advantage of this flexibility?</p>

gcramer23: <blockquote>
<p>Let us say I have two dedicated machines that I want to use for training. That means I want world_size=2. Then both min and max should be exactly 2 for the intended allocation, right? In what kind of scenario one would want to set min/max differently to take advantage of this flexibility?</p>
</blockquote>
<p>If you have 2 machines you can use for training, then set the max to 2. If you are willing to start training with 1 machine, then setting min to 1 makes sense. You can set min to a different value than max when you are willing to start a training job with less machines due to faults in the system.</p>
<p>cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a></p>

cbalioglu: <p>In traditional HPC and ML systems all nodes that are part of a job are started simultaneously (as you implied in your question) and the job gets dispatched (by a scheduler) once all nodes are ready to execute. The set of nodes that are part of a job are usually called a “gang”.</p>
<p>TorchElastic is designed in a way to handle systems that do not form a gang. Those systems are usually meant to run traditional distributed applications where the scheduler has no concept of forming a gang of nodes (e.g. the scheduler simply starts executing the job on a node the moment the node becomes available). Since you have no formal gang you have to have some form of mechanism to simulate a gang (a “pseudo-gang”) and dispatch the job at some point in time after the user requested it. This is where the minimum and maximum number of nodes come into play. You basically tell TorchElastic that it should wait until at least min number of nodes become available to execute a job, but that ideally you want it to have max number of jobs for the execution. Once TorchElastic reaches min number of nodes, it sets an internal “lastcall” timer and continues accepting new nodes until the timer expires or the max number of nodes is reached. At that point TorchElastic dispatches the job on all participant nodes. In summary TorchElastic has a built-in capability to form a gang even when run on systems that has no such concept.</p>
<p>In your particular case (and in most cases) setting minimum and maximum number of nodes to the same value is the way to go. They become relevant in non-traditional execution environments as I described above.</p>
Out-of-vocabulary Embedding Layer and pre trained model

lus.7: <p>Hey!<br>
I’m using a pre-trained fasttext model for the weights of my embedding layer for text classification. Once the model is trained, the embeddings layer only has weights corresponding to the vocabulary of the training data.</p>
<p>When predicting something from the test set when I encounter a word not in the training data, it gets assigned an out of vocabulary index and of course cannot use weights from the pre-trained embeddings. One advantage of fasttext is that it can get a vector for oot words but I’m not sure how this can be implemented for the embedding layer? Is there a clever way to do this? Thank you!</p>
How to split a tensor into different dimension?

novice96: <p>Hi, I’m new to Pytorch .<br>
Let’s say I have a tensor that has this shape  <code>torch.size([1, 25200, 11]</code><br>
I want to split it into 3 smaller tensors , each of 3 smaller tensors has the shape of 1st.<br>
<code> torch.size([1, 3, 80, 80, 11])</code>  and 2nd <code> torch.size([1, 3, 40, 40 , 11])</code>   and  3rd  <code>torch.size([1, 3, 20, 20, 11)].</code><br>
Really appreciate your help.<br>
thanks<br>
Explain those number:</p>
<ul>
<li>80x80x3 = 19200</li>
<li>40x40x3 = 4800</li>
<li>20x20x3=1200 , add these result we have 25200, 1 is batch size, 11 is classes + xywh</li>
</ul>

pascal_notsawo: <p>You can proceed as follows :</p>
<pre><code class="lang-auto">
import torch

torch.manual_seed(0)

x = torch.rand(1, 25200, 11)

i=80*80*3#= 19200

j=40*40*3#= 4800

#k=20*20*3#=1200

x1=x[:,:i,:].reshape(1, 3, 80, 80, 11)

x2=x[:,i:i+j,:].reshape(1, 3, 40, 40 , 11)

x3=x[:,i+j:,:].reshape(1, 3, 20, 20, 11)

"""
x1=x[:,:i].reshape(1, 3, 80, 80, 11)
x2=x[:,i:i+j].reshape(1, 3, 40, 40 , 11)
x3=x[:,i+j:].reshape(1, 3, 20, 20, 11)
"""

</code></pre>

novice96: <p>hey Thank for the answer .</p>
Torch.jit.script slows down gpu

Crispin: <p>Hi</p>
<p>I am finding that although <code>torch.jit.script</code> makes CPU code about 8x faster, it makes GPU code about 5x slower, compared to code on each device that is not compiled with <code>torch.jit.script</code>.</p>
<p>How do I fix this - am I using the JIT wrong somehow? Test code below.</p>
<pre><code class="lang-auto">import torch
import numpy as np

size=1000000 # data size

def test(device,torchscript):
    def torch_call(x,mask,a,b):
        # some simulated workload...
        x[mask]=1
        torch.bucketize(a,b)

    if torchscript:
        torch_call = torch.jit.script(torch_call)

    # create data
    x = torch.zeros(size)
    mask = torch.randint(2,(size,))==1
    a = torch.from_numpy(np.random.random(size))
    b = torch.from_numpy(np.linspace(0,1,1000))

    if device=="cuda":
        x = x.cuda()
        mask = mask.cuda()
        a = a.cuda()
        b = b.cuda()

    # time torch_call
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    torch_call(x,mask,a,b) # warmup call
    start.record()
    torch_call(x,mask,a,b)
    end.record()
    torch.cuda.synchronize()
    print (f"{device} {type(torch_call)=} time={start.elapsed_time(end)}")

test("cuda",False)
test("cuda",True)
test("cpu",False)
test("cpu",True)

Results:
cuda type(torch_call)=&lt;class 'function'&gt; time=0.16105599701404572
cuda type(torch_call)=&lt;class 'torch.jit.ScriptFunction'&gt; time=0.7593600153923035
cpu type(torch_call)=&lt;class 'function'&gt; time=83.50847625732422
cpu type(torch_call)=&lt;class 'torch.jit.ScriptFunction'&gt; time=10.096927642822266
</code></pre>
Dataloader [] operator first indexed by 0

Nathb26: <p>Hi,<br>
I build a dataset MyDataset who herite from torch.utils.Dataset. I’m would like to use Transfer Learning for object detection in videos. But MyDataset[0] is not define because I would like to begin on the middle of my video; so on the 3300th frame for example. But when I’m launching a dataloader, the next(iterator function call MyDataset[0] and I have a key error. How can I set the dataloader to have a first index on my 3300 frame ?<br>
Thanks for your help !</p>

emcp: <p>I think you’d need to split your dataset before you give it to the data loader…  so in psuedo code</p>
<ol>
<li>
<p>have dataset of whole video</p>
</li>
<li>
<p>Split or slice dataset at frame 3300</p>
</li>
<li>
<p>load into launcher as normal and train</p>
</li>
</ol>
<p>you can see this actually in their example where they split the data into a train and test dataset of the images… test dataset is last 50 images</p>
<pre><code class="lang-auto">    # use our dataset and defined transformations
    dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))
    dataset_test = PennFudanDataset('PennFudanPed', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)
</code></pre>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>

Nathb26: <p>Thanks for your answer !<br>
I would not like to use the whole video because all images are not annotated. And also because the preprocessing should be very slow.<br>
Do you have an other solution ?<br>
Than you very much</p>

emcp: <p>What format is the video to be fed in?  frame by frame so images?  You can write your <code>Dataset</code> class to handle annotated and unannotated images and “ignore” those frames… I would keep annotated and unannotated images in totally separate <code>Dataset</code> instances myself… you can see in the example I pasted they seem to just create two different loaders and datasets from the same core data (plus some transforms on the training set)… you could easily just slice off all non-annotated frames using a collate, a transform… or just use some intelligence in the <code>Dataset</code> class itself to “drop” data which you cannot use for training</p>

Nathb26: <p>Thank you very much I will try !</p>
Training lstm 10x faster

Zhihan_Yang: <p>I understand how to use <code>nn.LSTM</code> correctly, and I do have access to GPUs. So it might seem weird that I need to train them faster.</p>
<p>I’m trying to apply recurrent layers to reinforcement learning. On standard RL environments, training takes 200k to 1 million gradient updates for non-recurrent agents. To do the same number of updates with a recurrent agent takes way too long, given a backprop through time of 200 to 1000 timesteps.</p>
<p>Are there ways to potentially speed up LSTM training time (even 2x or 3x would help tremendously), apart from (1) using <code>nn.LSTM</code> instead of <code>nn.LSTMCell</code> and (2) using GPU instead of CPU?</p>
<p>Thanks!</p>

Ayman_Al_Jabri: <p>Do you mind sharing the model you’re using and how you are collecting observations from the environment?</p>
Validation Step fails:RuntimeError: Given groups=1, weight of size 32 4 3 3, expected input[1, 3, 770, 770] to have 4 channels, but got 3 channels instead

bhaktatejas922: <p>I’m trying to train a Unet (efficientnet-b4) with segmentation_models_pytorch and running into this issue</p>
<p>I am using 4 channel input images ( RGB dstack-ed with a Depth image)</p>
<p>train:  81%|▊| 53486/65912 [1:24:38&lt;19:39, 10.53it/<br>
valid:   0%| | 16/16478 [00:02&lt;42:09,  6.51it/s, val_loss_mask: 0.7111, val_loss: 0.7111, val_mask_micr</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bhaktatejas922/internal-geometry-ml/src/train.py", line 199, in &lt;module&gt;
    main(cfg)
  File "/home/bhaktatejas922/internal-geometry-ml/src/train.py", line 187, in main
    **cfg.training.fit,
  File "/home/bhaktatejas922/internal-geometry-ml/src/training/runner.py", line 304, in fit
    verbose=verbose,
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/bhaktatejas922/internal-geometry-ml/src/training/runner.py", line 335, in evaluate
    output = self._feed_batch(batch)
  File "/home/bhaktatejas922/internal-geometry-ml/src/training/runner.py", line 35, in wrapped
    res = f(*args, **kwargs)
  File "/home/bhaktatejas922/internal-geometry-ml/src/training/runner.py", line 153, in _feed_batch
    output = self.model(*input)
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/segmentation_models_pytorch/base/model.py", line 15, in forward
    features = self.encoder(x)
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/segmentation_models_pytorch/encoders/efficientnet.py", line 50, in forward
    x = self._swish(self._bn0(self._conv_stem(x)))
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bhaktatejas922/.local/lib/python3.6/site-packages/efficientnet_pytorch/utils.py", line 271, in forward
    x = F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 32 4 3 3, expected input[1, 3, 770, 770] to have 4 channels, but got 3 channels instead
</code></pre>
CustomDataset give me error

Chukwuemeka_Ekwonu: <p>I created a custom dataset but when I test it give me error:<br>
“obj = super().<strong>new</strong>(cls, *args, **kwds)<br>
TypeError: object.<strong>new</strong>() takes exactly one argument (the type to instantiate)”</p>
<p>Below is my code:</p>
<p>class HRLRDataset(Dataset):</p>
<pre><code>def __int__(self, root_dir, transform=None):
    self.root_dir = root_dir
    self.transform = transform

    lr_path, hr_path = os.path.join(self.root_dir, "lr"), os.path.join(self.root_dir, "hr")
    self.lr_files = sorted(glob.glob(lr_path + "/*.*"))
    self.hr_files = sorted(glob.glob(hr_path + "/*.*"))

def __len__(self):
    return max(len(self.lr_files), len(self.hr_files))

def __getitem__(self, index):
    hr_img = Image.open(self.hr_files[index])
    lr_img = Image.open(self.lr_files[index])
    if self.transform:
        hr_img = self.hr_transform(hr_img)
        lr_img = self.lr_transform(lr_img)

    return {"lr": lr_img, "hr": hr_img}</code></pre>

eqy: <p>Did you mean to write <code>__init__</code> rather than <code>__int__</code>?</p>

Chukwuemeka_Ekwonu: <p>Thank you, I could not find these subtle typo bug <strong>int</strong>. I actually meant <strong>init</strong>. many thanks</p>
ScriptModule `register_parameter`

dfalbel: <p>I wonder if <code>register_parameter</code> is expected to work with <code>ScriptdModule</code>s, for example:</p>
<pre><code class="lang-auto">import torch
module = torch.jit.trace(torch.nn.Linear(10, 10), torch.randn(100, 10))
module.register_parameter("new_parameter", torch.nn.Parameter(torch.randn(10, 10)))
</code></pre>
<blockquote>
<p>AttributeError: cannot assign parameter before Module.<strong>init</strong>() call</p>
</blockquote>
<p>I don’t have an specific use case for this… I am implementing a <code>ScriptModule</code> wrapper for R’s implementation of torch and wanted to know if this is expected to wok.</p>

ptrblck: <p>I don’t think this would be a supported use case, since the already traced model won’t have a chance to use the newly registered parameter afterwards.<br>
In the case that this parameter is indeed used in the <code>forward</code> (but not registered yet), tracing would raise an error due to the usage of undefined parameters. On the other hand, if this parameter is never used in the <code>forward</code>, registering it afterwards won’t change anything, so I assume the error is expected.</p>

dfalbel: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your explanation!</p>
<p>In theory one could use <code>register_parameter</code> to overwrite weights, eg:</p>
<pre><code class="lang-auto">import torch
module = torch.jit.trace(torch.nn.Linear(10, 10), torch.randn(100, 10))
module.register_parameter("weight", torch.nn.Parameter(torch.ones(10, 10)))
</code></pre>
<p>And expect it to be used by the <code>ScriptModule</code>. This seems to work fine in non-scripted modules.<br>
But there are other ways to do it anyway.</p>
Output of a piecewise discrete function as a classification problem

Mika_S: <p>I have synthetic data for a piecewise discrete function that bins a feature into bins. So the following:</p>
<pre><code class="lang-auto">def bin(x, min_val, bin_width, num_classes):
   return min(num_classes - 1, max(0, (x - min_val) // bin_width)))
</code></pre>
<p>I am setting this up as a classification problem with target between <code>0</code> and <code>num_classes - 1</code>. I was expecting that a 2 layer fully connected net would be able to predict the class (which is the bin number) accurately. But I am getting disappointing results.</p>
<p>Any ideas on what I am doing wrong here?</p>
Torch.onnx.export() does not support torch.val_mean() op

zihaog: <p>Hi,</p>
<p>I’m trying to convert my model to onnx format but the val_mean op is not supported.<br>
My pytorch version is 1.2.</p>
<p>I know that upgrading to pytorch 1.8 could solve the issue but I have to use 1.2 for some reason.</p>
<p>How to solve this on pytorch 1.2? Any suggestions will be appreciated.</p>
<p>Best,</p>

zihaog: <p>still looking for help…</p>
Binary classification of each column in an image

Cathal_Corbett: <p>Hi there,</p>
<p>I am performing an anomaly detection problem on images and wish to binary classify each column in an image. Each input image is a standard 500 pixels wide and therefore my output layer will contain 500 neurons with a sigmoid activation. I wish to use PyTorch but am having a problem deciding on a loss function. I am wondering if the following can be cleared up for me:</p>
<ol>
<li>Is BCELoss or BCEWithLogitsLoss (preferable) suitable for this problem?</li>
<li>My dataset is also unbalanced and I wish to assign class weighting to the loss function. For example, if the ratio of positive to negative class is 1:3 how would I assign a class weighting here? I am aware BCEWithLogitsLoss has 2 parameters ‘weight’ and ‘pos_weight’. ‘pos_weight’ seems to be the parameter which assigns weighting to the positive class. What I would like to do is assign the positive class a value of 3. However, documentation states ‘pos_weight’ requires ‘length equal to the number of classes’ which would require pos_weight=[3]*500 in my case. Would this be sufficient in assigning class weighting for my problem?</li>
</ol>
<p>Thank you in advance for any help.</p>
Autoencoder with tied weights using sequential()

hosseinshn: <p>Hi,<br>
I am building an autoencoder like this:</p>
<pre><code class="lang-auto">class autoencoder(torch.nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = torch.nn.Sequential(
            torch.nn.Linear(dim_input, h_dim1),
            torch.nn.ReLU(),
            torch.nn.Dropout(),
            torch.nn.Linear(h_dim1, Z_dim),
            torch.nn.ReLU(),
            torch.nn.Dropout())
        self.decoder = torch.nn.Sequential(
            torch.nn.Linear(Z_dim, h_dim1),
            torch.nn.ReLU(),
            torch.nn.Dropout(),
            torch.nn.Linear(h_dim1, dim_input),
            torch.nn.ReLU(),
            torch.nn.Dropout())

    def forward(self, X):
        Z = self.encoder(X)
        X = self.decoder(Z)
        return X, Z
</code></pre>
<p>But weights in the encoder and the decoder are different, how can I make it tied weights (weights in the decoder should be transpose of the encoder weights–parameters of the model are then only the encoder’s weights)?</p>
<p>Another question, in a tied weight autoencoder, if I use dropout for the encoder part for regularization, how should I apply it in the decoder side? To me, it seems in the above code different nodes are being dropped out in the encoder and the decoder sides.</p>
<p>Thanks!</p>

KSLTPK89: <p>I would be interested in understanding this as well!</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i have the same question and same sequential auto-encoder , trying to achieve the same as original poster posted question here to tied weights between encoder and decoder layers</p>

ptrblck: <p>A clean approach would be to define the desired parameters once (via <code>nn.Parameter</code> in the <code>__init__</code> method of the model) and apply them through the functional API in the <code>forward</code> method (e.g. via <code>F.linear</code>).</p>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please share some simple example?</p>

ptrblck: <p>Sure, here is a simple example of using <code>F.linear</code> with a <code>weight</code> parameter:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(4, 10))
        
    def forward(self, x):
        print(x.shape)
        x = F.linear(x, self.weight.T)
        print(x.shape)
        x = F.linear(x, self.weight)
        print(out.shape)
        return x

model = MyModel()
x = torch.randn(1, 4)
out = model(x)
&gt; torch.Size([1, 4])
  torch.Size([1, 10])
  torch.Size([1, 4])
</code></pre>

TheOraware: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you mark it as solution for other viewer, i am not original poster cant do it</p>
Dataloader returns a list of batch_size

chocolocked: <p>So in my dataset, each sample has the different first dimension, for example x1.shape = (2, 100) and y1.shape = (2, 1), and sample 2 could be x2.shape = (5, 100) and y2.shape = (5,1)</p>
<p>How could I customize my collate_fn so that I could return a loader list of batch_size. For example, if my batch size = 2, then how could I get:<br>
for inputs , labels in train_loader:<br>
# inouts is  a list of length 2, with first element ( 2,100), second element (5, 100)<br>
# labels is a list of length 2, with first element ( 2,1), second element (5, 1)</p>

Ali1234: <p>If I understood your question correctly, then the answer would be: by simply returning <code>x[n]</code> and <code>y[n]</code> from your <code>Dataset</code> class. Something like the following:</p>
<pre><code class="lang-auto">import torch

class Dataset(torch.nn.Module):
    def __init__(self):
        super().__init__()
    def __len__(self):
        return 10
    def __getitem__(self, ind):
        inputs = torch.rand(2)
        labels = torch.rand(2)
        return inputs, labels

dataset = Dataset()
data_loader = torch.utils.data.DataLoader(dataset, batch_size=2)
for inputs, labels in data_loader:
    print("Inputs: ", inputs)
    print("Labels: ", labels)
</code></pre>
Failed to script a module containing torch.nn.utils.parametrize.register_parametrization

geogeo28: <p>Hi, I am trying to jit.script a module containing a parametrization as follow:</p>
<pre><code class="lang-auto">class SWSConv2d(nn.Conv2d):
    r"""
    2D Conv layer with Scaled Weight Standardization.

    Characterizing signal propagation to close the performance gap in unnormalized ResNets
    https://arxiv.org/abs/2101.08692
    """
    def __init__(self,
                 in_channels: int, out_channels: int, kernel_size: int,
                 stride: int=1, padding: int=0, padding_mode: str='zeros', dilation=1, groups: int=1,
                 bias: bool=True):

        super().__init__(in_channels, out_channels, kernel_size, stride=stride, padding=padding, padding_mode=padding_mode, dilation=dilation, groups=groups, bias=bias)

        self.register_parametrization()

    def register_parametrization(self):
        if not TP.is_parametrized(self, 'weight'):
            TP.register_parametrization(self, 'weight', ScaledWeight2DStandardization(out_channels=self.weight.shape[0], use_gain=True, eps=1e-4))

    def remove_parametrization(self):
        if TP.is_parametrized(self, 'weight'):
            TP.remove_parametrizations(self, 'weight', leave_parametrized=True)
</code></pre>
<p>model_jit = torch.jit.script(model) fails with the following error:</p>
<pre><code class="lang-auto">  File "/Users/ganneheim/anaconda3/envs/pytorch1.9/lib/python3.7/site-packages/torch/jit/frontend.py", line 137, in &lt;listcomp&gt;
    stmts = [build_stmt(ctx, s) for s in stmts]
  File "/Users/ganneheim/anaconda3/envs/pytorch1.9/lib/python3.7/site-packages/torch/jit/frontend.py", line 330, in __call__
    raise UnsupportedNodeError(ctx, node)
torch.jit.frontend.UnsupportedNodeError: global variables aren't supported:
  File "/Users/ganneheim/anaconda3/envs/pytorch1.9/lib/python3.7/site-packages/torch/nn/utils/parametrize.py", line 166
    def get_parametrized(self) -&gt; Tensor:
        global _cache
        ~~~~~~ &lt;--- HERE
    
        parametrization = self.parametrizations[tensor_name]
</code></pre>
Does DataLoader prefetch_factor matters when num_workers=0?

avish: <p>Default prefetch_factor=2 and num_workers=0, but documentation says that “there will be a total of prefetch_factor * num_workers samples prefetched”. As num_workers=0 means main thread does fetching of batch, so will prefetch_factor will work in this case?</p>

ptrblck: <p>I don’t think the main thread is able to prefetch the data, so the argument shouldn’t change the behavior.<br>
The default value is most likely set, since usually the <code>num_workers</code> argument is changed while the <code>prefetch_factor</code> is often not touched.</p>
Converting a ScriptModule to Trace

sanjibansg: <p>Is it possible to convert a script module to trace with dummy input tensors. I tried with code</p>
<pre><code class="lang-auto">model=torch.jit.load('model.pt')
dummy=torch.rand((1,3,224,224))
traced=torch.jit.trace(model,dummy)
</code></pre>
<p>Following warning was thrown:</p>
<pre><code class="lang-auto">UserWarning: The input to trace is already a ScriptModule, tracing it is a no-op. Returning the object as is.

</code></pre>
VGG implementation error

randinoo: <pre><code class="lang-auto">import vgg
class VGG(nn.Module):
    '''
    VGG model 
    '''
    def __init__(self, features):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 512),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(512, 512),
            nn.ReLU(True),
            nn.Linear(512, 10),
        )
         # Initialize weights


    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        return x

</code></pre>
<p>ModuleNotFoundError: No module named ‘vgg’</p>
<p>help me plzz</p>

eqy: <p>It doesn’t look like you use the <code>vgg</code> import here. Why is it imported?</p>

randinoo: <p><a class="mention" href="/u/eqy">@eqy</a>  for the code to work</p>

randinoo: <p><a class="mention" href="/u/eqy">@eqy</a>  def forward(self, x):<br>
 → 103         x = self.features(x)<br>
104         x = x.view(x.size(0), -1)<br>
105         return x</p>
<p>TypeError: ‘int’ object is not callable</p>
<p>help me plz</p>

eqy: <p>You might be passing an int intead of a tensor here.</p>
Pack_padded_sequence, pad_packed_sequence

Ximing_Dong11104: <p>I want to use the last hidden state for bilstm, for each batch, I firstly sort each example that is in the same batch. and then use pack_padded_sequence-&gt;lstm →  pad_packed_sequence. then Back to the original position。 but I want to use the last hidden state. is that right in my code<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/695e2e7c570ffc9c267633982c9bdc5e0ec0e406.png" data-download-href="https://discuss.pytorch.org/uploads/default/695e2e7c570ffc9c267633982c9bdc5e0ec0e406" title="Screen Shot 2021-06-21 at 10.47.43 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695e2e7c570ffc9c267633982c9bdc5e0ec0e406_2_690x208.png" alt="Screen Shot 2021-06-21 at 10.47.43 AM" data-base62-sha1="f27USUMvkno4XFURbsIw0Xrtsd8" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695e2e7c570ffc9c267633982c9bdc5e0ec0e406_2_690x208.png, https://discuss.pytorch.org/uploads/default/original/3X/6/9/695e2e7c570ffc9c267633982c9bdc5e0ec0e406.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/9/695e2e7c570ffc9c267633982c9bdc5e0ec0e406.png 2x" data-dominant-color="F7F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-06-21 at 10.47.43 AM</span><span class="informations">961×290 29.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
?</p>