akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Model with dynamic number of layers

matthias.l: <p>In order to do a hyperparameter optimization to find out how much cnn layers work best, my module accepts this number in its <code>__init__</code> function and stores the cnn layers in a python list.<br>
They get chained in the <code>forward</code> function.</p>
<p>Doing this pytorch is ignoring the parameters of that cnn layers, as they are in a list and not a attribute of the module directly.</p>
<p>What is the best way to handle this kind of use case?</p>

sabetai: <p>Can’t you use nn.ModuleList() in this case?</p>
Os.name =='nt' meaning?

hunar: <p>i following the facenet in timesler , there is a line</p>
<p>workers = 0 if os.name == ‘nt’ else 8</p>
<p>what</p>
<blockquote>
<p>os.name ==‘nt’</p>
</blockquote>
<p>meaning</p>

KFrank: <p>Hello Namo!</p>
<aside class="quote no-group" data-username="hunar" data-post="1" data-topic="67632" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/c0e974/48.png" class="avatar"> hunar:</div>
<blockquote>
<p>workers = 0 if os.name == ‘nt’ else 8</p>
</blockquote>
</aside>
<p>It means that the operating system (“os”) is a microsoft<br>
operating system of the NT lineage, such as windows<br>
10 or windows 7 (or windows NT), as distinct from the<br>
older dos and semi-dos windows systems such as<br>
windows 3.1 or windows 95 (and also distinct from other<br>
operating systems such as (old) mac and various flavors<br>
of unix).</p>
<p>(I imagine that <code>workers</code> refers to the number of worker<br>
threads.  Windows NT has it own, rather different, threading<br>
implementation,  so it’s plausible you might want to tweak<br>
your threading approach for NT.)</p>
<p>Best.</p>
<p>K. Frank</p>
tensorboardX: usage with model whose outputs is list of tensors?

tin.alegre: <p>Given the model below that returns a list of tensors (logits).</p>
<pre><code class="lang-auto">class MyNet(nn.Module):
    def __init__(self, n_classes):
        super(MyNet, self).__init__()
       ...

    def forward(self, x):
       ...
       return [side_5, side_6, side_7, side_8]
</code></pre>
<p>How can I properly use the  tensorboardX function <code>writer.add_graph(model, input)</code> ?<br>
I tried something like:</p>
<pre><code class="lang-auto">dummy_input = Variable(torch.zeros([1, 3, im_rows, im_cols]).to(self.device), requires_grad=True)
writer.add_graph(model, dummy_input)
</code></pre>
<p>but it seems that the <code>add_graph</code> expects the model to return a tensor and not a list of tensors. Any ideas?</p>
NLLLoss function wrt to getting a error of "bool value of tensor with more than one value is ambiguous"

Bhargava: <p>In PyTorch nn.NLLLoss() function, when we apply it directly to calculate loss value wrt input and Target value, we get a error mentioning “bool value of Tensor with more than one value is ambiguous”.<br>
When we mention something like criterion=nn.NLLLoss() and then apply criterion (input,Target) we get the loss tensor output. Any idea why this happens?</p>

KFrank: <p>Hi Bhargava!</p>
<p>Without seeing you code, I’m only guessing.  However …</p>
<aside class="quote no-group" data-username="Bhargava" data-post="1" data-topic="67609" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/77aa72/48.png" class="avatar"> Bhargava:</div>
<blockquote>
<p>In PyTorch nn.NLLLoss() function, when we apply it directly to calculate loss value wrt input and Target value, we get a error mentioning “bool value of Tensor with more than one value is ambiguous”.</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.NLLLoss" rel="noopener nofollow ugc">NLLLoss</a> is a class, instances of which are callable function objects.</p>
<p>(<a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.nll_loss" rel="noopener nofollow ugc">nll_loss</a> is a function.)</p>
<p>My guess is that you are calling the <em>constructor</em> of <code>NLLLoss</code> with your<br>
<code>input</code> and <code>Target</code> tensors.  The second argument, <code>size_average</code> to<br>
<code>NLLLoss</code>’s constructor is a <code>bool</code>, hence your error.</p>
<pre data-code-wrap="python"><code class="lang-python">nn.NLLLoss (input, Target)   # bad -- calling constructor incorrectly
nn.NLLLoss() (input, Target)   # good -- calling a default-constructed function object correctly
nn.functional.nll_loss (input, Target)   # good -- calling ordinary function
</code></pre>
<p>Good luck!</p>
<p>K. Frank</p>
Maskrcnn_benchmark

lyh: <p>When I train the maskrcnn_benchmark,and then I follow the tutorial follow the library,but Pycharm returns “no module named maskrcnn_benchmark” .I dont know where is the maskrcnn_benchmark module ?Thanks a lot.</p>
<p>–Windows 10 HOME<br>
Pycharm<br>
Python 3.6<br>
pytorch-1.4.0</p>

peterjc123: <p>Could you please explain what you are doing step by step by introducing screenshots or code snippets?</p>

lyh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6aeea02d54835bf9ead27ede347dd3fb02e1431.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6aeea02d54835bf9ead27ede347dd3fb02e1431" title="1579949135(1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6aeea02d54835bf9ead27ede347dd3fb02e1431_2_690x387.png" alt="1579949135(1)" data-base62-sha1="slDi2bJs1raOMbWo0WU9u3qF7S9" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6aeea02d54835bf9ead27ede347dd3fb02e1431_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6aeea02d54835bf9ead27ede347dd3fb02e1431_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6aeea02d54835bf9ead27ede347dd3fb02e1431_2_1380x774.png 2x" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1579949135(1)</span><span class="informations">1573×883 99.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

peterjc123: <p>What’s the output of <code>conda info -e</code> and <code>where python</code>?</p>

lyh: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96f7cd26f71b7b0e3e26a480a14add2dcf0e21db.png" data-download-href="https://discuss.pytorch.org/uploads/default/96f7cd26f71b7b0e3e26a480a14add2dcf0e21db" title="1579949725(1)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96f7cd26f71b7b0e3e26a480a14add2dcf0e21db.png" alt="1579949725(1)" data-base62-sha1="lxwx1bvuYFh4jXcjtHydROdFtKX" width="690" height="221" data-dominant-color="323232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1579949725(1)</span><span class="informations">1116×359 18.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

lyh: <p>Thank you very much.It’s done.</p>
Sample from GumbelSoftmax in a way that track gradients

Dylan_Yung: <p>I’m working with Pyro right now on using a VAE for seq2seq name generation and have this bit of code</p>
<blockquote>
<p>sampled_indexes = pyro.sample(f"{address}_{index}", pyro.distributions.RelaxedOneHotCategoricalStraightThrough(1, logits=char_dist), obs=observed[index])</p>
</blockquote>
<p>Essentially char_dist is a categorical distribution from a neural network for the character in a name as an index i. “pyro.sample” expects a distribution that samples a value, in this case RelaxedOneHotCategoricalStraightThrough samples a categorical distribution. I’d rather be able to do GumbelSoftmax PyTorch distribution that just samples the value that softmaxes to 1, this is better for Pyro to track the sample, as opposed to sampling a categorical distribution over characters. The reason for this is because if it doesn’t sample from the gumbel softmax an exact value I don’t think it’ll track the gradients during backprop. Any suggestions?</p>
In PyTorch's “MaxPool2D”, is padding added depending on “ceil_mode”?

paul_shuvo: <p>In <code>MaxPool2D</code> the padding is by default set to 0 and the <code>ceil_mode</code> is also set to <code>False</code>. Now, if I have an input of size <code>7x7</code> with <code>kernel=2,stride=2</code> the output shape becomes <code>3x3</code>, but when I use <code>ceil_mode=True</code>, it becomes <code>4x4</code>, which makes sense because (if the following formula is correct), for <code>7x7</code> with <code>output_shape</code> would be <code>3.5x3.5</code> and depending on the <code>ceil_mode</code> it would be either <code>3x3</code> or <code>4x4</code>.</p>
<p><img src="https://i.stack.imgur.com/90ZKl.gif" alt width="478" height="38"></p>
<p>Now, my question is, if the <code>ceil_mode=True</code>, does it change the default <code>padding</code>?</p>
<p>Also, what’s the default value for filling the padded cells?</p>
<p>If it does, then how is it adding the padding i.e. is it adding the padding on left first or right, up first or down?</p>
<p>Also, what’s the default value for filling the padded cells?</p>

G.M: <p>No, <code>ceil_mode=True</code> does not use padding. U can use an example input and print the output to see how does it work.</p>
Attention area when CNN

shirui-japina: <p>Now there are:</p>
<ul>
<li>Attention map (the 2D matrix of important value)</li>
<li>Associated original image</li>
</ul>
<p>Then I want to implement CNN, which pays more attention to the location of original image on the basis of attention map.</p>
<p>How can I achieve this idea in PyTorch?</p>

Surekha_Gaikwad: <p>May be you can check the ICLRV2018 paper “Learn to Pay attention”. It works on the same idea.</p>
How to do mean under mask area in a memory efficent way?

twangnh: <p>suppose I have a mask tensor (1 or 0) M of shape [ N, H, W ], and a value tensor P [H, W, C], I want to use the N mask to get all values at all 1’s locations in value tensor and mean them get a mean value under mask tensor of shape [N, C], for now I can only achieve this by epand M to be MM of shape [N, H , W, 1]  and P to be PP of shape  [1, H, W, C] and then do MM*PP to get a result tensor V of shape[N, H, W, C], and then do</p>
<blockquote>
<p>V.sum(dim=(1,2))/(V!=0).sum(dim=(1,2))</p>
</blockquote>
<p>but this takes to much GPU memory as N is too large, so I can only do it on cpu, but that is extremely slow, can someone help?</p>
Please help me see The custom tripletloss implementation I found online is correct

111220: <p>Please help me check whether the tripletloss  is correct.</p>
<p>This is the code:</p>
<pre><code class="lang-auto">label=np.array([0.,1.,2.,3.,0.,1.])

label=torch.from_numpy(label)
label=label.reshape((6,1))
N=6
is_pos = label.expand(N, N).eq(label.expand(N, N).t())
is_neg = label.expand(N, N).ne(label.expand(N, N).t())
dist=comput_dist(label,label)


list_ap=[]
list_an = []
for i in range(N):
    # Find the maximum distance between the ith label and the positive sample
    list_ap.append(dist[i][is_pos[i]].max().unsqueeze(0))
    #Find the minimum distance between the ith label and its negative sample
    list_an.append(dist[i][is_neg[i]].min().unsqueeze(0))
dist_ap = torch.cat(list_ap)
dist_an = torch.cat(list_an)
y = torch.ones_like(dist_an)
loss =torch.nn.MarginRankingLoss()(dist_an,dist_ap,y)
print(loss)
</code></pre>
Are optimizer per-parameter options supported with DistributedDataParallel?

nalapati: <p>Hi All,</p>
<ol>
<li>Lets suppose I have a model that I want to train using DistributedDataParallel, I wrap my model with DistributedDataParallel as follows:</li>
</ol>
<pre><code class="lang-auto">ddp_model = DDP(model, device_ids=[device])
</code></pre>
<ol start="2">
<li>I init my optimizer as follows:</li>
</ol>
<pre><code class="lang-auto">optim = optim.SGD(ddp_model.parameters(), lr=1e-2)
</code></pre>
<p>Is there a way to modify step 2, to apply per parameter optimizer options? What does the following look like given the ddp model?</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>As on <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#per-parameter-options</a></p>
<p>Thanks!</p>

pritamdamania87: <p>I believe per-parameter options should be supported by DistributedDataParallel. Have you tried it out and seen any issues? If you do see issues/unexpected behavior feel free to open an issue on <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">github</a>.</p>
Good way to calculate element-wise difference between two models of the same structure

XW324: <p>Suppose I have two models of the same structure, what is the best way to calculate the sum of difference between corresponding weights from these two models? (pseudo code: L1_loss(model1,model2))</p>

crowsonkb: <p>If the two models have the exact same structure (i.e. their state dicts have the same layers in the same order) then the following should work:</p>
<pre><code class="lang-python">sum((x - y).abs().sum() for x, y in zip(model1.state_dict().values(), model2.state_dict().values()))
</code></pre>
Is it possible to run hyperparameter optimization on multiple GPUs in parallel?

XYJin: <p>Suppose I have a single node with 4 GPUs. I would like to do model selection w.r.t a specific dataset through random search. A manager will keep track of a grid of such hyperparams. At each iteration a distinct model with a specific setting will be created and trained on an assigned GPU. The training dataset is shared by all such models. A minimal example will be like following</p>
<pre><code class="lang-auto">class HyperSearchManager:
    def __init__(self, 
                 train_dataset: torch.utils.data.Dataset,
                 valid_dataset: torch.utils.data.Dataset,
                 test_dataset: torch.utils.data.Dataset,
                 param_grid: Dict[str, List]):
        self.train_dataset = train_dataset
        self.valid_dataset = valid_dataset
        self.test_dataset = test_dataset
        self.param_grid = param_grid

        self.best = float('inf')
        self.optimal_model = None

    def param_iter(self) -&gt; Dict:
        ...
        yield params

    def train_single_model(self, model: nn.Module, num_epoch: int, device: torch.device):
        # copy model to the respective device
        model = model.to(device)
        # train loops for a single model
        loader = torch.utils.data.DataLoader(self.train_dataset, batch_size, ...)
        optimizer = torch.optim.Adam(model.parameters(), lr, ...)
        for epoch in range(num_epoch):
            for data in loader:
                data = data.to(device)
                train(model, data, optimizer)
                ...
            # Do validation with early stopping, etc.
            valid_loss = validation(model, self.valid_dataset)
        # update optimal model according to valid metrics
        self.update(model.cpu(), valid_loss)

    def update(self, model, valid_loss):
        # if valid_loss is minimal, keep current model
        if valid_loss &lt; self.best:
            self.optimal_model = model

    def search(self):
        for _ in range(MAX_HYPER_OPT_ITER):
            params = next(self.param_iter) # get next hyperparam combination
            model = ModuleClass(**params) # create model for the specific hyperparam

            # if a free gpu is available, create a new subprocess to run the model on the allocated gpu
            device = self.get_available_device()
            proc = multiprocessing.Process(target=self.train_single_model, args=(model, num_epochs, device))
            proc.start()
            # else waiting...

        run_test(self.optimal_model, self.test_dataset)                

</code></pre>
<p>I wonder if it is possible to find a schedule to allocate idle gpu for a pending model. That is, at first 4 models are trained on 4 gpus, respectively. Once a training process is finished, a new model will be assigned to the released GPU.</p>
<p>If that’s not straightforward, is there any easy implementation for such functionalities?</p>

pritamdamania87: <p>As far as I know PyTorch doesn’t provide a framework to do this automatically. You will have to build this scheduling mechanism in your application itself.</p>
Why is there a zero argument when iterating through dataloaders (e.g. enumerate(self.trainloader, 0) )?

pinocchio: <p>I’ve seen:</p>
<pre><code class="lang-auto">       for epoch in range(nb_epochs):
            for i, data in enumerate(trainloader, 0):
                # unpack data
                inputs, labels = data

</code></pre>
<p>but I’ve never understood why the enumerate function needs the <code>0</code> as an argument. Does anyone know why?</p>

albanD: <p>You can check the python doc for the specification of the <code>enumerate()</code> function: <a href="https://docs.python.org/3/library/functions.html#enumerate">https://docs.python.org/3/library/functions.html#enumerate</a></p>
Solving a tensor linear system

kaiseryet: <p>Hey guys, I am trying to solve a tensor linear system using PyTorch. That being said, I’m looking for something similar to torch.solve but can handle tensors. Of course, I can reshape the p<em>p</em>p<em>p tensor to p^2</em>p^2 matrix, and p*p tensor to p^2 vector to solve, but it will slow the code by far I assume? Thanks.</p>

LeviViana: <p>Actually the operation <code>view</code> applied to a tensor is almost free, since it leaves the storage intact:</p>
<pre><code class="lang-python">import torch
x = torch.rand(2, 2, 2, 2)
y = x.view(4, 4)
print(y.data_ptr() == x.data_ptr()) # &lt;- prints True
</code></pre>

kaiseryet: <p>if it’s almost free, it’s GREAT!</p>
C++ Torchscript: Running ResNet18 TorchScript model in GPU

solarflarefx: <p>Hi all,</p>
<p>I am new to the C++ API and was following the tutorial on: <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>I am able to successfully save the model in python, import the serialized script, and run inference on the model.  The example loads from the CPU.  Using some of the other C++ code examples as a reference, I loaded the model onto the GPU:</p>
<pre><code class="lang-auto">torch::DeviceType device_type;
  if (torch::cuda::is_available()) {
      std::cout &lt;&lt; "CUDA available! Training on GPU." &lt;&lt; std::endl;
      device_type = torch::kCUDA;
  }
  else {
      std::cout &lt;&lt; "Training on CPU." &lt;&lt; std::endl;
      device_type = torch::kCPU;
  }
  torch::Device device(device_type);
  torch::jit::script::Module module;
  module = torch::jit::load(path_to_resnet_model);
  module.to(at::kCUDA);
</code></pre>
<p>However, I am having trouble loading the tensors to the GPU.  The example code loads the data onto the CPU:</p>
<pre><code class="lang-auto">std::vector&lt;torch::jit::IValue&gt; inputs;
inputs.push_back(torch::ones({ 1, 3, 224, 224 }));
</code></pre>
<p>What is the best way to put this tensor input to the GPU, and then set up inference so that the output is also a GPU tensor?</p>
<p>Thanks in advance.</p>
Inverse sparse matrix

kaiseryet: <p>Hey guys, so I have a matrix A which is block-diagonal, now I need to calculate the inverse of A. For block diagonal matrix, we can always inverse each block to obtain the inverse of A; so is there such kind of function in PyTorch to carry out the implementation?</p>
<p>p.s., the blocks are of different sizes, so reshape to a three-dimension tensor is not an option…</p>
Fast batched multiplication and sum over last 3 dimensions on arbitrary view without reshape

Sam_Pr: <p>I’m trying to write a funky 3D convolution like operation and running into performance issue. I’ve provided a minimal repro of what I’m doing.</p>
<pre><code class="lang-auto">def foo():
  # Generate some data to test on
  volume_data = torch.rand(1, 2, 33, 33, 33, device = "cuda", requires_grad=True)
  funky_kernel = torch.rand(1, 729, 1, 1, 2, 1, 2, 1, 2, 2, 2, 2, device = "cuda", requires_grad=True)

  # Create the unfolded view of the volume
  volume_view = volume_data.unfold(dimension = 2, size = 2, step = 1)
  volume_view = volume_view.unfold(dimension = 3, size = 2, step = 1)
  volume_view = volume_view.unfold(dimension = 4, size = 2, step = 1)
  volume_view = volume_view.view(1, 1, 2, 16, 2, 16, 2, 16, 2, 2, 2, 2)

  # The actual calculation
  output = funky_kernel * volume_view
  output = output.sum(dim = (-3, -2, -1))
  return output

print(foo().shape)
</code></pre>
<p>The issue is that the main output calculation (the multiply and sum) is slow (~5X) compared to a native convolution. My hypothesis is that that’s because the multiplication manifests a large (8X) tensor before the <code>sum</code> reduction runs. I would like to combine this multiplication and addition into a single operation.</p>
<p>I’ve been trying to use one of the many <code>matmul</code>/<code>bmm</code> variations to do this, but I’m not too familiar with the daunting collection of initialism matrix functions and I haven’t been able to find one that fits my needs.</p>
<p>The promising idea seemed to be using <code>matmul</code> like</p>
<pre><code class="lang-auto">output = torch.matmul(volume_view.view(-1, 1, 1, 8), funky_kernel.view(1, -1, 8, 1))
</code></pre>
<p>However this doesn’t work, as <code>volume_view.view(-1, 1, 1, 8)</code> complains that <code>view size is not compatible with input tensor's size and stride</code>. Using <code>reshape</code> would do a large memory copy which is unacceptable for performance and causes an out of memory error with an 11.4 GiB allocation.</p>
<p>Is there another fast way I can do this?</p>
<p>To state simply what I need: A single fast operation that does pointwise multiplication and summation over the last three dimensions, and that can operate on a view without requiring it to be reshaped in an incompatible way.</p>

Sam_Pr: <p><code>einsum</code> seems like it should be the answer, but I’m finding that it’s a lot slower (~25X!) than the basic multiply and sum.</p>
<p>This is the code I’m using for that.</p>
<pre><code class="lang-auto">(a, b) = torch.broadcast_tensors(a, b)
return torch.einsum("...ijk, ...ijk -&gt; ...", a, b)
</code></pre>
<p>I’ve filed a bug for that at <a href="https://github.com/pytorch/pytorch/issues/32591" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/32591</a></p>
CUDA Outof memory error

shantipriya: <p>Hi,</p>
<p>Getting the below error when trying to run my experiment (loading dataset) in GPU. My training and development data set sizes are (  252000 train.txt, and  28000 devel.txt). Now working on a single GPU.</p>
<p>X_temp = X_temp.cuda()<br>
RuntimeError: CUDA error: out of memory</p>

LeviViana: <p>What is the <code>X_temp.size()</code> ? how much memory do you have in your GPU ? did you check if another program is using the GPU and how much memory do you have left (i.e. <code>nvidia-smi</code> in the terminal) ?</p>
Sample equally-sized chunks from a 1D tensor

ygabuev: <p>I have a tensor X of size (T). What I need is to randomly sample N chunks from X of size t&lt;T, so that the result is a tensor of size (N, t) and each “row” is originally contiguous in X. What is the best way to do that (without using for-loops, allocating memory when not needed, etc.)?</p>

nairbv: <p>if you do something like:</p>
<pre><code class="lang-auto">a=torch.arange(20)
length=5
for x in range(10):
    index=random.randint(0, a.numel()-length)
    sample=a[index:index+length]
    print(sample)
</code></pre>
<p>each <code>sample</code> shares memory with <code>a</code>.</p>
<p>I’m not sure what you mean about no loops though, what do you want to do with each sample? If you need the results to be in a single tensor (e.g. with a batch dimension), it might be harder to share memory.</p>

ygabuev: <p>Hi <a class="mention" href="/u/nairbv">@nairbv</a>, currently I do that as follows:</p>
<pre><code class="lang-auto">source = np.random.randn(T)
samples = torch.empty(N, t)
inds = np.random.randint(0, T-t+1, N)
for i in range(N):
    ind = inds[i]
    samples[i] = torch.Tensor(source[ind:(ind+t)])
</code></pre>
<p>So, I am willing to take N*t space for <code>samples</code> tensor. The problem is that if N is large, the for-loop might take some time to execute (but I really am not sure whether it could be a problem. Maybe i’m too precautious). I also do not want to take O(Nt) memory for indices that I will use to extract samples from <code>source</code>.</p>
<p>I am wondering if there is a neat way to vectorize the operation of extracting the samples from <code>source</code>.</p>
<p>After that, each sample is passed through a neural net.</p>
Algorithim of how Conv2d is implemented in PyTorch

Deathstroke_Twelved: <p>I am working on an inference model of a pytorch onnx model which is why this question is being asked.</p>
<p>Assume, I have a image with dimensions <code>32 x 32 x 3</code> (CIFAR-10 dataset). I pass it through a Conv2d with dimensions : <code>3 x 192 x 5 x 5</code>. The command I used is: <code>Conv2d(3, 192, kernel_size=5, stride=1, padding=2)</code></p>
<p>Using the formula (stated here for reference pg12 <a href="https://arxiv.org/pdf/1603.07285.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1603.07285.pdf</a>) I should be getting an output image with dimensions <code>28 x 28 x 192</code> (<code>input - kernel + 1 = 32 - 5 + 1</code>).</p>
<p>Question is how has PyTorch implemented this 4d tensor <code>3 x 192 x 5 x 5</code> to get me an output of <code>28 x 28 x 192</code> ? The layer is a 4d tensor and the input image is a 2d one.</p>
<p>How is the kernel (<code>5x5</code>) spread in the image matrix <code>32 x 32 x 3</code> ? What does the kernel convolve with first -&gt; <code>3 x 192</code>  or <code>32 x 32</code>?</p>
<p><strong>Note</strong> : I have understood the 2d aspects of things. I am asking the above questions in 3 dimensions or more.</p>

ptrblck: <p>Since you are using a padding of 2 for a kernel size of 5, the spatial output should be constant.<br>
The input is expected to be a 4-dimensional tensor in the shape <code>[batch_size, channels, height, width]</code>, while the kernel has the shape <code>[out_channels, in_channels, height, width]</code>.</p>
<p>In the default setup (not depthwise or grouped convolution) the kernel will be applied to patches of the input in a sliding window manner using all input channels.<br>
Have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv">CS231n</a> which explains the underlying method pretty good.</p>

Deathstroke_Twelved: <p>Thanks for the answer <a class="mention" href="/u/ptrblck">@ptrblck</a>. But my question was more related to PyTorch’s implementation.<br>
When I got the tensor.size from the Conv2d layer it is :- [3 x 192 x 5 x 5]. And 3 x 192 x 5 x 5 = 14400 weights are stored in the pth.tar file (checkpoint file).  How is this tensor arranged ?<br>
I understand 192 is the depth, 5 x 5 is the kernel size. Are there 3 tensors of depth 192 with size 5 x 5 being returned from this layer? What does the 3 stand for here then in the pth.tar file.</p>

ptrblck: <p>As explained, the 3 would stand for the number of input channels.<br>
You have basically 192 kernels each with 3 channels and a spatial size of <code>5x5</code>.<br>
The provided link explains the applied method.</p>
<p>PyTorch dispatches to different backends as seen <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp">here</a> in case you are interested in the implementation details.</p>
Is there a way to create a rolled (shifted and wrapped) view of a tensor?

Sam_Pr: <p>I believe the answer to this question is simply no, but I would like to confirm with somebody with more experience.</p>
<p>Given a tensor <code>[0, 1, 2, 3, 4, 5]</code>, is there a way to create a view <code>[1, 2, 3, 4, 5, 0]</code> without a memory copy?</p>
<p>As an extension, is this possible in more dimensions? That is, could a 2D image be rolled both horizontally and vertically?</p>

nairbv: <p>See <code>torch.roll(input, shifts, dims=None)</code></p>

Sam_Pr: <p>Not sure how I missed that, thanks. By the looks of the kernel implementation <code>torch.roll</code> does a copy, so it seems safe to presume that it can’t be done with a view. My understanding of the stride/offset system also implies that conclusion, so that makes sense. I’m happy to call that a confirmation and solution.</p>
Custom Mask for dropout

riktimmondal: <p>I am trying to provide my own dropout mask in the forward function while defining a feed forward network.<br>
For example</p>
<pre><code class="lang-auto">class Perceptron(torch.nn.Module):
    def __init__(self, mask):
        super(Perceptron, self).__init__()
        self.fc = nn.Linear(5,10)
       self.myDrop = MyDrop(mask)
def forward(self, x):
        output = self.fc(x)
        o1 = self.myDrop(output)
        return o1
</code></pre>
<p>Here mask is a vector consisting of 0’s and 1’s so accordingly the value of o1 with be either 0 or the existing values. In MyDrop() function i am simple just multiply the values of output with mask to get o1 as output. So here the gradient should all be zero for that node, but how does autogradd maintain this since i am not making custom Dropout class but instead using a mask through a simple function. So due backprop the graient will still be flow through those nodes even if the output has been made 0 using a mask?</p>

ptrblck: <p>Yes, Autograd will track the multiplication as a differentiable operation and will calculate the gradients accordingly.<br>
Are you seeing any issues using your approach?</p>

riktimmondal: <p>Sorry for late reply.Yes the backpropagation is working but the first linear layer gradient is always zero(I can’t get the reason behind it, if you can suggest something it will be helpful). I still have some confusion which I want to understand.</p>
<ol>
<li>Do we need to multiply the mask after getting the activation output from linear layer like f1=nn.Linear(), then mask_output = mask * self.Relu(f1(x)) or proceed in different way where the ouput of the previous layer x needs  to be mutipled first with mask like</li>
</ol>
<pre><code class="lang-auto">#x is the output of the previous layer
mask_out = x*mask
f1=nn.Linear()
out1 = self.Relu(f1(mask_out)))
</code></pre>
<ol start="2">
<li>Since we are using our own mask as dropout so at the time of evaluation so are passing a variable to the model for checking whether we are in train or eval state using model.eval(). is this correct?</li>
</ol>
<p>3)Also since pytorch uses dropout as inverted dropout so how do we handle this case for train and eval case using our own mask.</p>

G.M: <ol>
<li>It depends, some apply dropout before activation while others apply it after the activation. However, in the case of ReLU these 2 methods should be equivalent.<br>
2.Yes, u should look for whether the model is in “train” or “eval”. I remember there’s a function/attribute called “training”.<br>
3.U should take a closer look at how inverted dropout work(well I suggest u also take a look at how normal dropout work). It’s quite simple to implement.</li>
</ol>
Adding zeros to dataset

Niki: <p>Hi, I want to add some zeros at the end of CIFAR-10 dataset. I tried this:</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(trainloader):
            if use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()

                input_zero = torch.zeros(batch_size, 3, 5, 32).cuda()
                tragets_zero = torch.zeros(batch_size, 10).cuda()

             targets = torch.cat((targets, tragets_zero))
             inputs_new = torch.cat((inputs, input_zero), 2)
</code></pre>
<p>It shows this error at <code> targets = torch.cat((targets, tragets_zero))</code>:</p>
<pre><code class="lang-auto">RuntimeError: Expected object of scalar type Long but got scalar type Float for sequence element 1 in sequence argument at position #1 'tensors'
</code></pre>
<p>How can I solve this?</p>

G.M: <p><code>torch.zeros()</code> gives tensors of type <code>Float</code>, but in this case it’s not what u want. So u should change it to Long type, like <code>input_zero = torch.zeros(batch_size, 3, 5, 32).long().cuda()</code> or <code>input_zero = torch.zeros(batch_size, 3, 5, 32,dtype=torch.long,device=UR_GPU)</code>.</p>

Niki: <aside class="quote no-group" data-username="G.M" data-post="2" data-topic="67551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/g.m/48/19266_2.png" class="avatar"> G.M:</div>
<blockquote>
<p>.long()</p>
</blockquote>
</aside>
<p>Thank you for your reply, <a class="mention" href="/u/g.m">@G.M</a>. That solved the problem.</p>
<p>It seems there is another problem about <code>targets = torch.cat((targets, tragets_zero))</code>:</p>
<pre><code class="lang-auto">RuntimeError:  Tensors must have same number of dimensions: got 2 and 1 at /opt/conda/conda-bld/pytorch_.....
</code></pre>

G.M: <p>U should go and take a look at the doc for <code>torch.cat</code>. Then try to print out all the shapes of the 2 tensors. Finally, u’ll realize what’s wrong <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Niki: <p>Ok. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

G.M: <p>The issue here is quite straight forward.</p>
Kullbach-Leibler implementation

Dennis_Hugle: <p>Dear Community,</p>
<p>I found that some people use the Kullback leibler divergence loss for training their model when the output layer consists of a one-hot-vector using pytorch.</p>
<p>I actually tried it myself, and it works well. However I do not understand <strong>why</strong> that works. As given in the documentation, the loss over the output layer L (of length n) to my understanding is:</p>
<pre><code class="lang-auto">loss = \sum_{_i}^{n} y_i  log(y_i/x_i)
</code></pre>
<p>with i = 0…n, y being the label and x being the estimated prob of the model. But that assumes that y_i is always &gt; 0. Otherwise the loss would be undefined due to log(0).  Therefore I do not understand how I get good results using the KLdiv loss.</p>
<p>Can someone explain whats going on inside the KLdiv function of pytorch? is the label smoothed such that no zeros occur anymore?</p>
<p>I am thankful for any advice.</p>
<p>Cheers,</p>
<p>Dennis</p>
<p>edit: i don’t know why the latex formatting is not working, maybe someone can fix it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <aside class="quote no-group" data-username="Dennis_Hugle" data-post="1" data-topic="67544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dennis_hugle/48/13706_2.png" class="avatar"> Dennis_Hugle:</div>
<blockquote>
<p>Otherwise the loss would be undefined due to log(0). Therefore I do not understand how I get good results using the KLdiv loss.</p>
</blockquote>
</aside>
<p>Less brief <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Definition" rel="noopener nofollow ugc">definitions</a> of the Kullback-Leibler divergence spell out the that the term is to be considered to be 0 when log y_i is 0.</p>
<p>This is by continuous extension:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56145636b345816037c3302ac7b3c698e413391b.png" alt="image" data-base62-sha1="chuDiinSwSH5sjfgEUk6YH7Jt47" width="382" height="266"></p>
<p>The PyTorch implementation uses <code>torch.where</code> to achieve this.</p>
<p>You cannot get away with x_i being zero when y_i is not, then the KL div is infinite. This is the requirement that y_i must be <em>absolutely continuous</em> w.r.t. x_i.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: One might reflect on the intended use of the formula in the PyTorch documentation: We assume that other ressources (Wikipedia, Textbooks, …) are more useful for learning about KL Divergence. The main goal is for someone who knows the formula to see which term is used how. This is particularly important for things like CrossEntropyLoss or NLLLoss, where the mathematical definition and the actual processing of the inputs to get the outputs are not quite in sync.</p>
I have frozen all my layers, but the output changes

YinYang_Untalan: <p>I trained my ann with all layers unfrozen. Then I froze all all my layers using layer.requires_grad = False.</p>
<p>Then I trained it again. The outputs change. I have taken a look at the state_dict of one of the layers to see if the weights have been changed - they are still the same.</p>
<p>Am I missing something here? How do I properly freeze the layers?</p>
<p>I tried doing for p in layer.parameters(): p.requires_grad = False. I did this for all the layers and when I trained, I got an error that basically says there is no gradients to compute - which is what I expected. Then i proceeded to do this for only specific layers. one set of layers is a decoder: This is what i did in more detail</p>
<p>I have an encoder and a decoder which I trained and froze using above (p.req… = False). Then I wanted to play around so I attached a classifier to the encoding, and in the middle of the classifier I have a layer that has the same dimension as the encoding which means I can put it thru the decoder - and i did. My loss functions are the classification loss and the MSELoss involving the middle of the classifier going thru the decoder. My problem is the decoder (or possible even the encoder) changes. This is how I verify that:</p>
<ul>
<li>train the encoder and decoder</li>
<li>freeze the encoder and decoder</li>
<li>set t = random tensor with same size as input</li>
<li>pass it thru the network and visualize the decoded output (A)</li>
<li>train the classifier and also add loss from the output of passing a middle classifier layer thru the decoder (this shouldn’t affect the decoder and probably would just pass the errors on the classification layers)</li>
<li>pass t again it thru the network and visualize the decoded output (A’)</li>
<li>A and A’ are not the same</li>
</ul>
<p>Thank you for helping.</p>

ptrblck: <p>I guess you might have some dropout or batchnorm layers in your model, which are still active even though the trainable parameters are frozen.<br>
Calling <code>model.eval()</code> will use the running stats in batch norm layers (instead of using the current batch stats and updating the running estimates) and dropout layers will be disabled.<br>
Could this be the issue?</p>

YinYang_Untalan: <p>I saw the issue. I was using the Adam optimizer so setting the grad flag off wasnt enough. I needed to filter the parameters I pass to Adam. Is there a cleaner way to this? Thank you.</p>
SIGSEV when trying to use tensor from_blob

AlconDivino: <p>For loss computing (for a convolutional model) i am trying to convert my data which is stored in a 4D-std::vector of type double to a torch::Tensor using torch::from_blob.</p>
<p>i am using the Gramian Angular Field to convert timeseries to an image. My code looks like this:</p>
<pre><code class="lang-auto">std::vector&lt;std::vector&lt;std::vector&lt;std::vector&lt;double&gt;&gt;&gt;&gt; resallGAF(32);
 
    // Compute GAF batch
    for(unsigned int i = 0; i &lt; batchSize; i++)
    {
       // Creating data here        
    }

    torch::Tensor result = torch::from_blob(resallGAF.data(), {static_cast&lt;long&gt;(resallGAF.size()), static_cast&lt;long&gt;(resallGAF[0].size()), static_cast&lt;long&gt;(resallGAF[0][0].size()), static_cast&lt;long&gt;(resallGAF[0][0][0].size())} ).clone();
</code></pre>
<p>When .clone() is called. i get a exc_bad_access error from xcode. (SIGSEV) resallGAF has the shape <code>{32, 4, 150, 150}</code>.</p>
<p>When i use similar code to convert vector of shape <code>{1, 4, 32, 32}</code>  to a Tensor everything works fine. I cannot print or forward the resulting Tensor either, when removing the <code>.clone()</code>.</p>
<p>I checked the data and the shapes and data are normally valid.<br>
I tired it with libtorch 1.3.1 and 1.4.0 both without CUDA support.</p>
<p>Thanks in advance</p>

Matej_Kompanek: <p>Hi,<br>
I think you need to use continous memory, not vector of vectors. For example:</p>
<pre><code class="lang-auto">std::vector&lt;double&gt; resallGAF;

long long DIM0 = 1;
long long DIM1 = 4;
long long DIM2 = 32;
long long DIM3 = 32;

double someValue = 0.0;

for (long long i = 0; i &lt; DIM0; ++i)
{
   for (long long j = 0; j &lt; DIM1; ++j)
   { 
      for (long long k = 0; k &lt; DIM2; ++k)
      {
        for (long long m = 0; m &lt; DIM3; ++m)
	{
            resallGAF.push_back(someValue++);
	}
}}}

torch::Tensor result = torch::from_blob(resallGAF.data(), { DIM0, DIM1, DIM2, DIM3 }, torch::kDouble);
std::cout &lt;&lt; result &lt;&lt; std::endl;
</code></pre>
<p>Your solution probably works only on small tensors because ‘from_blob’ does not reach beyond memory segment, however data inside the tensor is garbage.</p>

AlconDivino: <p>Thanks for the reply. Ur solution would take my data and put it in a 1D tensor. I need a 4d tensor tho, for feeding it into a Conv2d Model.</p>

Matej_Kompanek: <p>If I call</p>
<pre><code class="lang-auto">std::cout &lt;&lt; result.sizes() &lt;&lt; std::endl;
</code></pre>
<p>I get:<br>
[1, 4, 32, 32]</p>
<p>Isn’t that a 4D tensor ?</p>

AlconDivino: <p>Seems like i was confused there for a second. it is indeed a 4D tensor. thank you for your help it works now <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
Asking for explanation in scale model parameters

yegane: <p>Hi,<br>
I have seen a method to scale model, in PySyft, and I had some guesses, but I would appreciate it if you could please explain what is happening in here.<br>
I mention my questions in the code as comments.</p>
<pre><code class="lang-auto">def scale_model(model, scale):
    """Scale the parameters of a model.
    Args:
        model (torch.nn.Module): the models whose parameters will be scaled.
        scale (float): the scaling factor.
    Returns:
        torch.nn.Module: the module with scaled parameters.
    """
    params = model.named_parameters()
    # in here, does params have references of model parameters? It only makes sense
    # for me only in this, way, but how a method return a pointer?
    dict_params = dict(params)
    # I was assuming that casting params as a dictionary, would change the pointer 
    # assignments, but it seems that it still has its references
    with torch.no_grad():
        for name, param in dict_params.items():
            dict_params[name].set_(dict_params[name].data * scale)
            # I am seeing that in the line above, the scale is applying to each layer separately. Is it 
            # correct? and then we have modified the parameters, and the data is changing in the 
            # model, as well
    return model
</code></pre>
Modify layers parameters in loop

yegane: <p>Hi, I want to systematically, change layers parameters of a model,<br>
I saw this post in 2017,</p><aside class="quote quote-modified" data-post="1" data-topic="11638">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinesexingyili/48/2240_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-i-modify-certain-layers-weight-and-bias/11638">How can I modify certain layer's weight and bias?</a> 
  </div>
  <blockquote>
    I want to duplicate the paper’s work:Incremental network quantization. 
In this paper ,the author quantize 50% parameters in a CNN network to pow of 2 and do not update these parameters(update the other 50% parameters to compensate the inaccuracy caused by quanzation) 
I make a mask to indicate the parameters weather are quantized. 
My code is like this: 
qua_weight = qua_tensor(weight, pos_shreshold, mask_weight, max_ind, 2**3)
net.state_dict()['features.0.weight'].data = qua_weight

I found th…
  </blockquote>
</aside>
<p>
But I am looking for some method to iterated over the layers and change them.<br>
something like this:<br>
FOR i in model.[iterable list of parameters]:<br>
model.[list of parameters][i]=model.[list of parameters][i]*2</p>
<p>so I could iterate over layers and assign them new value based on the previous ones.</p>
Did the default weight initialization method change? (torch1.1 vs torch1.3)

spnova12: <p>I found some differences between the torch 1.1 and torch 1.3 initialization methods during deeplearning experiments.<br>
Are there really different default weight initialization methods for torch1.1 and torch1.3?</p>

ptrblck: <p>The default weight initialization for batch norm layers was updated to 1s instead of sampling from a uniform distribution in <a href="https://github.com/pytorch/pytorch/pull/13774">this PR</a>, which shipped with <code>1.2</code>, if I’m not mistaken.<br>
Besides that I’m unaware of any changes.</p>

spnova12: <p>Thank you very much for your answer!</p>