akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch best practices for serialization of Datasets and using YAML config files

ac1d: <p>Hello everyone!</p>
<p>The tittle is pretty much self explanatory, but I would nonetheless like to elaborate a bit. Namely, I have two questions:</p>
<ol>
<li>
<p>Very often we run multiple experiments on a single dataset and very often we also have a bunch of preprocessing steps that we want to perform on the data we have. I am wondering what is the best practice for serializing data after preprocessing? Is it considered best to serialize the entire torch.utils.data.Dataset object? Is it perhaps better to simply generate a preprocessed text file and read the input from there? Is there any preference when it comes to using pickle/dill? Feel free to elaborate and answer any other connected question that I have not asked, but you believe it’s important.</p>
</li>
<li>
<p>A common practice in Machine Learning is to use YAML config files to specify all sorts of things for your data preprocessing pipeline, your model, your layers, hyperparams etc. Are there any best practices on using YAML config files?</p>
</li>
</ol>
<p>Looking forward to your answers <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>Ha, nothing like a best practice question for some bikeshedding.</p>
<ol>
<li>I think serializing the dataset object is not a good idea.
<ul>
<li>If applicable, I tend to favor some native format for the given datatype. It also comes down to performance considerations (I/O is typically slower than some processing, so e.g. saving a tensor representation instead of an JPEG for image data is not a good idea). I’ve also had similar things with medical imaging.</li>
<li>I would avoid pickling. Typical options are using (dicts/lists of) tensors (using torch.save/load), numpy’s format, or, if you feel fancy, something like hdf5.</li>
</ul>
</li>
<li>I think I’m much too grumpy for that. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> IMO whether or not yaml is a good format (as opposed to command line parameters, a .py or whatnot) depends on how large the number of parameters are and how generalizable you need it to be. For example, I would not encode model architecture in anything but code unless you really need to vary it a lot across your experiments. But then I might not be the best source for advice on this part.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>
PPO agent is not learning anything

Jaredeco: <p>I implemented this from ppo clip pseudocode from openai and code was also inspired by one tutorial but my agent is not learning anything.</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
import gym


class Policy(nn.Module):
    def __init__(self, input_dim, n_actions, pol_type):
        super(Policy, self).__init__()
        self.pol_type = pol_type
        self.fc1 = nn.Linear(input_dim, 256)
        self.fc2 = nn.Linear(256, 256)
        self.fc3 = nn.Linear(256, 256)
        self.ac = nn.Linear(256, n_actions)
        self.cri = nn.Linear(256, 1)
        self.optimizer = optim.Adam(self.parameters())
        self.cuda()

    def forward(self, x):
        x = torch.tensor(x, dtype=torch.float32).cuda()
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        if self.pol_type == "actor":
            x = F.softmax(self.ac(x).detach(), dim=-1)
        elif self.pol_type == "critic":
            x = self.cri(x)
        return x


class PPO:
    def __init__(self, input_dim, n_actions):
        self.gamma = 0.99
        self.epsilon = 0.2
        self.actor = Policy(input_dim, n_actions, "actor")
        self.critic = Policy(input_dim, n_actions, "critic")
        self.log_prob = None
        self.prev_log_prob = None

    def select_action(self, obs):
        probs = self.actor(obs)
        dis = torch.distributions.Categorical(probs)
        action = dis.sample()
        self.log_prob = dis.log_prob(action)
        return action.item()

    def train(self, state, reward, next_state, done):
        if self.prev_log_prob is not None:
            reward = torch.tensor(reward, dtype=torch.float32).cuda()
            critic = self.critic(state)
            next_critic = self.critic(next_state)
            advantage = reward + self.gamma * next_critic * (1 - int(done)) - critic
            now_vs_prev = (self.log_prob - self.prev_log_prob).exp() * advantage
            g = torch.clamp(now_vs_prev, (1 - self.epsilon), (1 + self.epsilon)) * advantage
            actor_loss = -torch.min(now_vs_prev, g)
            critic_loss = torch.mean(advantage ** 2)
            loss = (actor_loss + critic_loss)
            self.actor.optimizer.zero_grad()
            self.critic.optimizer.zero_grad()
            loss.backward()
            self.critic.optimizer.step()
            self.actor.optimizer.step()
        self.prev_log_prob = self.log_prob


num_episodes = 400
env = gym.make("MountainCar-v0")
n_actions = env.action_space.n
input_dim = env.observation_space.shape[0]
agent = PPO(input_dim, n_actions)
for episode in range(num_episodes):
    done = False
    score = 0
    state = env.reset()
    while not done:
        if episode &gt; 370:
            env.render()
        action = agent.select_action(state)
        next_state, reward, done, _ = env.step(action)
        agent.train(state, reward, next_state, done)
        score += reward
    print(episode, score)

</code></pre>
how pytorch implement bilinear algorithm in interpolation

hl0737: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/444e29da3ae73f58305ef40ae8a4933d726005ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/444e29da3ae73f58305ef40ae8a4933d726005ef" title="微信截图_20200831164500"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/444e29da3ae73f58305ef40ae8a4933d726005ef_2_690x498.png" alt="微信截图_20200831164500" data-base62-sha1="9KfVOtQ7Z1rXpbx2A3Iv4sB92Zp" width="690" height="498" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/444e29da3ae73f58305ef40ae8a4933d726005ef_2_690x498.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/444e29da3ae73f58305ef40ae8a4933d726005ef_2_1035x747.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/444e29da3ae73f58305ef40ae8a4933d726005ef_2_1380x996.png 2x" data-dominant-color="EEEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200831164500</span><span class="informations">1487×1075 90.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi, i want to implement my bilinear interpolation algorithm in c++ and i want to know how pytorch do it.</p>
<p>In the example, the source tensor (1, 1) = 4, and the scale_factor = 2, so we can know the destination tensor (2, 2) = 4, but why the result is 3.4?? what the inner calculation process is ?</p>
<p>I know the bilinear interpolation in mathmatics very well, so it must be something i dont know when we implement it, anyone know it ?</p>
Does Deep Learning Model Use Server's CPU?

keremnymn: <p>Hello,<br>
Okay so I’ve trained Faster R-CNN model using Detectron2.</p>
<p>And I’m very satisfied with its results. I want to deploy this model on a CPU optimized hosting like DigitalOcean as a REST API.</p>
<p>And it takes relatively long time to infer a single image (6-7 seconds).</p>
<p>So my question is:</p>
<p>-When I deploy this model, will it use hosting server’s CPU, or user’s cpu?</p>
<p>Because if it uses server’s CPU, I think it’ll cause a bottleneck, won’t it?</p>
<p>Thanks in advance.</p>

Steven_Mugisha: <p>Once you deploy a REST API, users will typically send requests to a server and the server resolves these (server-side) and returns corresponding responses. That said, even model inference will be done on the API server and a user will receive a prediction response…</p>
<blockquote>
<p>Because if it uses server’s CPU, I think it’ll cause a bottleneck, won’t it?</p>
</blockquote>
<p>If you choose the wrong hardware (a weak CPU) it definitely will cause slow response times.</p>
How to make 3d affine_grid in pytorch? Need Help

Sanghyeok: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5.png" data-download-href="https://discuss.pytorch.org/uploads/default/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5_2_690x370.png" alt="image" data-base62-sha1="peIBtU9VHrJdIOs7vs96JhTWAHb" width="690" height="370" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5_2_690x370.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5_2_1035x555.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0e04de6de7ef5f27d71e354dd9b5682b2fe7dc5.png 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1121×602 37.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As you see I can make 2d_affine grid for deep learning, But I want to know how to make<br>
3d_affine. Can you give me some advise?</p>

oldriver-wang: <p>Hi, <a class="mention" href="/u/sanghyeok">@Sanghyeok</a><br>
Maybe you need torch.<strong>version</strong>&gt;=1.3.0</p>
Variant of CosineEmbeddingLoss

Federico_Vasile: <p>Hi,</p>
<p>I’m looking for a variant of <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html" rel="nofollow noopener">CosineEmbeddingLoss</a> in which instead of using the cosine distance I would like to use the squared Euclidean distance.<br>
Is it available in Pytorch?<br>
Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>PS: I tried to implement a very first implementation, but it is not vectorized(hence not computational efficient)</p>

tom: <p>Are you looking for MSELoss?</p>

Federico_Vasile: <p>Not exactly; I would like a version of MSELoss that behaves according to the <code>margin</code> and tensor <code>y</code> of similarities, as described in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineEmbeddingLoss.html" rel="nofollow noopener">CosineEmbeddingLoss</a>(i.e. exactly the same behavior described at this link, but using mean squared error instead of cosine distance to compute distances).<br>
Maybe I can do in this way:</p>
<pre><code>import torch 
import torch.nn as nn
import torch.nn.functional as F

class MarginMSELoss(nn.Module):
    def __init__(self, margin, reduction='mean'):
        super(MarginMSELoss, self).__init__()
        self.margin = margin
        self.reduction = reduction

    def forward(self, x1, x2, y):
        # x1.shape == x2.shape == (batch_size, num_features)
        # y.shape == (batch_size,)

        dist = F.mse_loss(x1, x2, reduction='none').mean(dim=1)      # dist.shape == (batch_size,)
        
        mask = y == 1
        out1 = (1 - dist) * mask     # out1.shape == (batch_size,)
        
        mask = y != 1
        out2 = (torch.max(torch.zeros_like(dist), dist - self.margin)) * mask
        
        out = out1 + out2

        if self.reduction == 'none':
            return out
        elif self.reduction == 'sum':
            return out.sum()
        elif self.reduction == 'mean':
            return out.mean()
        else:
            raise Exception('Reduction ' + self.reduction + ' is not supported')
</code></pre>
<pre><code>&gt;&gt;&gt; loss = MarginMSELoss(1)
&gt;&gt;&gt; x1 = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; x2 = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; y = torch.randint(0, 2, (3,))
&gt;&gt;&gt; output = loss(x1, x2, y)
&gt;&gt;&gt; output.backward()
</code></pre>
<p>Thank you for your patience <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>Right, sorry, I think <a href="https://pytorch.org/docs/stable/generated/torch.nn.HingeEmbeddingLoss.html#torch.nn.HingeEmbeddingLoss" rel="nofollow noopener">HingeEmbeddingLoss</a>, but L1-based rather than L2.</p>
Lstm output magic boundary

Stihl: <p>I use LSTM in RL model training<br>
And in differents runs it has value range between 1 and 0 or between 0 and -1<br>
Why does it have always 0 boundary ? Exploration process checks all values between -1 and 1 but LSTM output doesn’t, it always has 0 boundary<br>
I use 1 layer lstm, with 10 input features, and 1 hid. state<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31.png" data-download-href="https://discuss.pytorch.org/uploads/default/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31_2_690x301.png" alt="1" data-base62-sha1="o8uxjxumUc2hG0u3FaZc0tPGzux" width="690" height="301" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31_2_690x301.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31_2_1035x451.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/9/a929ef4d3e1dc1ddd794d09a6317ae9dfc504c31.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1271×555 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Why change a tensor's value in a function can change the outer tensor?

Igo312: <p>I found that defining a tensor variable and change its value in a function, outer the function the tensor’s value will change though I did not return anything. So what does happen when I did things above?</p>
<pre><code class="lang-auto">from torch import Tensor
a = Tensor([0])
&gt;&gt;&gt; a = tensor(0.)
def b(t):
    t[0]=3
b(a)
print(a)
&gt;&gt;&gt; a=tensor(3.)
</code></pre>
<p>It’s may about the scope of a tensor or torch low-level method. However I want someone can give me a complete reason to explain it.</p>

Steven_Mugisha: <p>The reason for the above scenario is that <strong>tensors are mutable objects</strong> therefore they are changeable <strong>in-place</strong><br>
This implies that when you call <code>b(a)</code>, instead of a new local variable <code>a</code> being created in the function scope, a ‘reference’ to <code>a</code> will be made and <code>a[0]</code> will be assigned the value 3.</p>
<p>However if <code>a</code> wasn’t a mutable object (not changeable in-place) the reverse would happen.i.e a new variable a would be created in the function scope and thus not affecting the outer a variable.<br>
For instance, the function below doesn’t change the outer/global a</p>
<pre><code class="lang-python">a = 0
def my_function(b):
    b = 3
    print(b)

my_function(a)
print(a)
</code></pre>
Predict probabilities with pytorch

Imene.b: <p>was asking the last time, if we can use neural network to predict if each pixel in an image is a corner of a box. I want to know<br>
1-  if this code is  correct or not:<br>
2- whene i got the probabilities, how can i distinct the probability of the top left corner from the probability of bottom right corner or the auther</p>
<pre><code class="lang-auto">image input_size = 512

import torch 
import torch.nn as nn
import torch.nn.functional as F
class corner_predict(nn.Module):
        #declarations........................
        dim_out = 512*3 (3 if the pixel is Top left corner, bottom right corner, auther)
        self.base_conv = nn.conv2D(512, 512, 3, 1, 1, bias=True)
        self.class_pred = nn.conv2D( 512, dim_out, 1, 1, 0)
        self.class_loss = 0
         def forward (image, bbox):
             conv1 =   F.relu(self.base_conv(image), inplace=True)
             conv2 = self.class_pred(conv1)
             prob_corner = F.sigmoid(conv2)
#------- loss calculations
</code></pre>

Imene.b: <p>if we can know witch is the probability of each corner</p>

ptrblck: <p>You can directly access the output values of <code>F.sigmoid(conv2)</code> via indexing to get the corner pixels:</p>
<pre><code class="lang-python">top_left = prob_corner[:, :, 0, 0]
top_right = prob_corner[:, :, 0, -1]
...
</code></pre>
<p>Let me know, if I misunderstood the question.</p>

Imene.b: <p>I’m sorry because I haven’t explain my problem well. there is what i mean:<br>
i want to predict if each pixel of an image is one of the 2 corners of a box (top left and bottom right). I have pixel images and the coordinates of boxes, and i want to use a neural network. i dont know how to introduce the comparaision between the pixels and the coordinate of the box.<br>
i wrote this code but in the result i have numbers between [0, 1] i can’t extract wich are belooging to the bottom right, top left corners?</p>

ptrblck: <p>How did you define your target containing the corners of your bounding box?<br>
Did you define it as a “segmentation map” containing <code>1</code>s at the corner pixels and <code>0</code> elsewhere?<br>
If so, you could try to use the position of the prediction to determine which pixel is “top left” and which is “bottom right”. However, since your prediction can output multiple pixels, you would have to deal somehow with this ambiguous result (maybe filtering out invalid candidates?).</p>
<p>I think the usual workflow to predict a bounding box would be to output the coordinates directly instead of a heat map.</p>

Imene.b: <p>Thanks, i will see with the position im my result</p>
Filtering Torchtext Dataset by Field

eprox: <p>Hi,<br>
Given the MNLI data set with something as as follows:</p>
<pre><code class="lang-auto">TEXT = data.Field(lower=True)
LABEL = data.LabelField(sequential=False)
GENRE = data.LabelField(sequential=False)
train, val, test = datasets.MNLI.splits(TEXT, LABEL, GENRE)
</code></pre>
<p>How do I filter the MNLI dataset to only include a particular genre?  I only want to do training on a single genre, for example “Fiction”.</p>
Logging with Distributed Data Parallel with PyTorch

kleingeo: <p>I am trying to setup a training workflow with PyTorch DistributedDataParallel (DDP). Generally when I train I pass a logger through to track outputs and record useful information. However, I am having trouble using the logger I have with the DDP method. Right now my code is as follows:</p>
<pre><code class="lang-python">import torch
import torch.multiprocessing as mp


class BaseModel:
    def __init__(self, *args, **kwargs):
        ...
        "does things"

    def fit(self, *args, **kwargs):
        ...
        'set up stuff'
        mp.spawn(self.distributed_training, nprocs=self.num_gpus, args=(self.params, training_input, self.logger))
        
    def distributed_training(params, training_input, logger):
        ...
        for e in epochs:
            'trains for an epoch'
            logger.info(print_line)
</code></pre>
<p>I know I am supposed to use the <code>QueueHandler</code> and <code>QueueListener</code> tools from <code>logging</code> with the import, but I have been scouring the internet and still do not have a clear understanding as to how. Any help would be greatly appreciated.</p>

rvarm1: <p>What sort of issues do you encounter when running this code?</p>
<p>You could also consider  creating a per-spawned process logger and no longer passing in the same logger into the spawned processes. This would also allow you to configure your logging on a per-DDP process basis, for example, write the logs to different files depending on the process.</p>

kleingeo: <p>With the way the code is set-up, things passed to the logger in the spawned process just don’t go through (ie, wont print or save). I’m not sure how a spawned process logger would help, as I need to capture things in the training. Mainly, I’m just trying to record loss and metrics per-epoch, nothing I would consider overly special for a training process. I think the issue is more that the logger is initiated on the main process so sending it to some type of other process is causing issues (whether it would be on its own process or in the distributed process).</p>

tengerye: <p>Having the same issue here.</p>
Why W_q matrix in nn.MultiheadAttention is quadratic

aktsvigun: <p>Good afternoon,<br>
I am trying to implement nn.MultiheadAttention in my network. According to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">docs</a>,</p>
<blockquote>
<p><strong>embed_dim</strong>  – total dimension of the model.</p>
</blockquote>
<p>However, according to the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention" rel="noopener nofollow ugc">source file</a>,</p>
<blockquote>
<p>embed_dim must be divisible by num_heads</p>
</blockquote>
<p>and</p>
<blockquote>
<p>self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))</p>
</blockquote>
<p>If I understand properly, this means each head takes <strong>only a part of features of each query</strong>, as the matrix is quadratic. Is it a bug of realization or is my understanding wrong? Thanks in advance!</p>

aktsvigun: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> could you please have a look?</p>
PyTorch model training parameters

alameer: <p>Is there a method available to find the best training parameters such as Epoch, learning rate, and batch size for a deep learning model? How do people go about find those values, please?</p>

ptrblck: <p>I think usually these hyperparameters are determined heuristically, i.e. by testing a lot of values or finetuning “good known” values. There are some approaches to automate this, such as FasiAI’s learning rate finder. Also, the general approach would be “learning to learn” or meta learning and is a current area of research.</p>

alameer: <p>Thanks very much, That really useful.</p>
Joined Optimizers

apsod: <p>Hi,<br>
I’m wondering if there’s a simple way to make a single optimizer product out of several optimizers (over disjoint parameters), such that the resulting optimizer product is a well formed instance of torch.optim.Optimizer.</p>
<p>The straightforward <a href="https://discuss.pytorch.org/t/two-optimizers-for-one-model/11085/7">answer</a> works well in simple cases, but fails when the optimizer product needs to be a well formed instance of torch.optim.Optimizer (e.g. when checkpointing or using frameworks such as pytorch lightning).</p>

matthew_zeng: <p>pytorch lighting natively supports multiple optimizers. you just need return a list of optimizers…</p>

apsod: <p>Yes, but not really. According to the docs, returning a list of optimizers is semantically different: It assumes that the optimizers are optimizing different objectives and should be run independently.</p>
<pre><code class="lang-auto">Lightning will call each optimizer sequentially:
for epoch in epochs:
   for batch in data:
      for opt in optimizers:
         train_step(opt)
         opt.step()

   for scheduler in scheduler:
      scheduler.step()
</code></pre>
<p><a href="https://github.com/PyTorchLightning/pytorch-lightning/blob/1015a0050621828c9e8af2c934e19c5c68d61a5e/pytorch_lightning/trainer/training_loop.py#L522" rel="nofollow noopener">Here</a> is the relevant code (I think).</p>
<p>i.e. the forward pass is running once per optimizer.</p>

JacobJonm: <p>Hi guys,</p>
<p>Regarding two optimizers and two losses:<br>
I need to train two networks to learn the coordinates (translation + rotations). Would this work?</p>
<p>…</p>
<p>optimizer_Trans = optim.Adam(parameters_Trans_to_train, lr)<br>
model_lr_scheduler_Trans = optim.lr_scheduler.StepLR(<br>
optimizer_Trans, scheduler_step_size, 0.1)</p>
<p>optimizer_Rotat = optim.Adam(parameters_Rotat_to_train, lr)<br>
model_lr_scheduler_Rotat = optim.lr_scheduler.StepLR(<br>
optimizer_Rotat, scheduler_step_size, 0.1)</p>
<p>…</p>
<p>model_lr_scheduler_Trans.step()<br>
model_lr_scheduler_Rotat.step()</p>
<p>for batch_idx, inputs in enumerate(train_loader):</p>
<pre><code>loss_Trans, loss_Rotat = process_batch(inputs)

optimizer_Trans.zero_grad()
loss_Trans.backward(retain_graph=True)
optimizer_Trans.step()

optimizer_Rotat.zero_grad()
loss_Rotat.backward(retain_graph=True)
optimizer_Rotat.step()</code></pre>
AttributeError: type object 'torch._C._TensorBase' has no attribute '__idiv__'

xpan: <p>I got an error of “AttributeError: type object ‘torch._C._TensorBase’ has no attribute ‘<strong>idiv</strong>’” when I try to run “import torch”.  It prints out error message like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f5226f127eedf7f8eb91105ebd91f1181ea1522.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9f5226f127eedf7f8eb91105ebd91f1181ea1522" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f5226f127eedf7f8eb91105ebd91f1181ea1522_2_666x500.jpeg" alt="image" data-base62-sha1="mJpVwQKNkvYuEAlaLP0DbY9PkJQ" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f5226f127eedf7f8eb91105ebd91f1181ea1522_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f5226f127eedf7f8eb91105ebd91f1181ea1522_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f5226f127eedf7f8eb91105ebd91f1181ea1522_2_1332x1000.jpeg 2x" data-dominant-color="5E525C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1440×1080 278 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Has anyone ever had the same problem? How to solve this problem?</p>
<p>Thank you.</p>

ptrblck: <p>Could you create a new virtual environment and reinstall PyTorch there?<br>
Alternatively, uninstall all PyTorch installations from the current environment and reinstall it. Make sure to run the uninstall commands multiple times, as e.g. binary installations might have been installed on top of source builds.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier and enables the forum search to index the post. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

xpan: <p>Many thanks for your kind help,  I  have solved this issue by creating  a  new virtual environment.</p>
<p>Best Regards</p>
Can image "channels" be different shapes?

Wesley_Neill: <p>Hi all <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
<p>I want to construct a 3 channel “image” from audio recordings to feed into a CNN. The three channels would be:</p>
<ul>
<li>STFT i.e. standard spectrogram</li>
<li>Mel-spectrogram</li>
<li>“Fingerprint” which is a sparse matrix representing the strongest frequencies in the standard spectrogram. This is modelled after the old Shazam algorithm.</li>
</ul>
<p>The problem is that each of these three visual representations of audio data has different shapes given the same audio.</p>
<p><strong>Question 1</strong><br>
Can I construct a 3 channel image without resizing the channels to equal dimensions?</p>
<p><strong>Question 2</strong><br>
If not, is there any reason to fear resizing the images to match each other? My concern is that the meaning of the data in each will be corrupted.</p>
<p><strong>Question 3:</strong><br>
Would it perhaps be better to create an ensemble of three different CNN’s rather than try squeeze all these different shaped data through a single model?</p>
<p>Thanks in advance to you audio/visual experts!</p>

Lin_Jia: <p><strong>Question 3:</strong><br>
Would it perhaps be better to create an ensemble of three different CNN’s rather than try squeeze all these different shaped data through a single model?</p>
<p>Feel free to try different strategies to see which one is the best for your case. My 2 cents here is that this is largely related to whether or not those three channels are related. If three different CNNs are used, they are independent, thus cannot learn the relationship between the three channels before you tangle them at the end. In a typical image, such as one from a camera, the three channels are high correlated because they are the representation of the light reflected from the same group of objects, thus it is the best practice to train using single CNN. But if you restructure an audio into an image format so you can leverage CNN. I think it can be a good idea to have three different CNNs, given those three channels seem to be different format of the same audio. During testing, if you use 3 CNNs and each of them is applied to one channel, you might find one of those 3 CNNs is enough to achieve your goal, this could mean that the corresponding channel is a great representation of the audio signals for your usage case.</p>

Wesley_Neill: <p>I believe the three channels will be correlated. They contain almost identical information, except that the data is represented on different scales (like logarithmic vs linear scales).</p>
<p>But I think I will try both ways and see what works, as you suggest <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Training pytorch model in Google Colab vs local GPUs results in less accurate models using same hyperparams. How to fix?

Keith72: <p>I recently discovered Google Colab and I uploaded my Pytorch project related to training models for processing audio. I got it training models using google’s TPUs, but I noticed that the models were less accurate than the ones I trained on my local machine. It turns out the the state dict weights and biases have about half the decimal places as the locally trained model. After some searching I read that Colab uses float16 By default instead of float32 precision to increase speed, but since the audio I’m training is in float32, it really needs to train in float32 precision. Is there a way to change this in Colab? Or is there a way to change my pytorch model to ensure float32 precision is kept? My model uses a stack of 1d convolutional layers, if that matters.</p>

ptrblck: <p>PyTorch uses float32 by default on CPU and GPU. I’m not deeply familiar with TPUs, but I guess you might be using <code>bfloat16</code> on them? Could you try to call <code>float()</code> on the model and inputs and check, if the TPU run is forcing you to use this format?</p>

Keith72: <p>I went back and looked at the stat_dict of both the locally trained GPU model and the cloud TPU model, and they do have the same precision, around 4 decimal places, so what I said in the original question was incorrect.</p>
<p>After some more reading it sounds like the built in bfloat16 type is part of what makes TPUs so fast, and I don’t understand all the math but I think it can produce the same range of values, so that might not be my issue.  I also should note that I’m using a pytorch_lightning module, although I wouldn’t think that would matter.  I might try calling float() to see if that changes the output, thanks!</p>

Keith72: <p>Here’s the reference I found on bfloat32:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://cloud.google.com/blog/static/assets/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus" target="_blank" rel="nofollow noopener">Google Cloud Blog</a>
  </header>
  <article class="onebox-body">
    <img src="https://cloud.google.com/blog/static/assets/GCP_Twitter_Card-20001000.png" class="thumbnail" width="" height="">

<h3><a href="https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus" target="_blank" rel="nofollow noopener">BFloat16: The secret to high performance on Cloud TPUs | Google Cloud Blog</a></h3>

<p>How the high performance of Google Cloud TPUs is driven by Brain Floating Point Format, or bfloat16</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>While <code>bflaot16</code> uses the same range as <code>float32</code>, it does not provide the same “step size”.<br>
As I’m not deeply familiar with this numerical format, I don’t know if you would have to adapt your model to this format.</p>
How to calculate cosine similarity of two multi-demensional vectors?

JasonChenhx: <p>How to calculate cosine similarity of two multi-demensional vectors through torch.cosine_similarity?</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.cosine_similarity">docs</a> give you an example:</p>
<pre><code class="lang-python">input1 = torch.randn(100, 128)
input2 = torch.randn(100, 128)
output = F.cosine_similarity(input1, input2)
print(output)
</code></pre>
<p>If you want to use more dimensions, refer to the docs for the shape explanation.<br>
E.g. for a 4-dim tensor, where you would like to compute the distance along <code>dim2</code>, this code should work:</p>
<pre><code class="lang-python">input1 = torch.randn(100, 128, 32, 32)
input2 = torch.randn(100, 128, 32, 32)
output = F.cosine_similarity(input1, input2, dim=2)
print(output)
</code></pre>

JasonChenhx: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="94577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>dim</code></p>
</blockquote>
</aside>
<p>Thanks your reply!<br>
But if I want to calculate cosine similarity between one dimension and the fourth dimension, How should I do?</p>

ptrblck: <p>You could <code>permute</code> one tensor, so that the corresponding axes are in the same dimension.</p>

JasonChenhx: <p>I give you my current example.<br>
I obtain two tensor which are [1,3,512,512] by neural network.<br>
I mean that I want to calculate cosine similarity of four-dim tensor.<br>
Can I use torch.cosine_similarity(input1,input2,dim=3)?<br>
Or I only compute the sum of each dimension first, and then average each dimension.</p>

ptrblck: <p>Yes, you should be able to use <code>dim=3</code>, if both tensors have the printed shape.<br>
Are you seeing any errors using it?</p>
<aside class="quote no-group" data-username="JasonChenhx" data-post="5" data-topic="94577">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jasonchenhx/48/21254_2.png" class="avatar"> JasonChenhx:</div>
<blockquote>
<p>Or I only compute the sum of each dimension first, and then average each dimension.</p>
</blockquote>
</aside>
<p>That might depend on your use case and what you would like to achieve, so I can’t be really helpful here.</p>

JasonChenhx: <p>Thanks your reply!<br>
I mean that the two four-dim tensors are two feature vectors, I want to calculate the similarity of the two feature vectors.</p>
Synchronize some information between processes at the end of each epoch (use case: setting time limit for training)

f10w: <p>I have a training script which I launch using <code>torch.distributed.launch</code> on multiple GPUs. I would like to set a time limit, so that my training will early stop without surpassing this limit. Something like this:</p>
<pre><code class="lang-python"># for storing the running times of the last 3 epochs
epoch_time_queue = deque(maxlen=3)

start_time = time.time()
for epoch in range(start_epoch, args.epochs):
    start_epoch_time = time.time()
    # training
    train_epoch(...)
    # validation
    eval_epoch(...)
    # epoch time in minutes
    epoch_time = (time.time() - start_epoch_time)/60
    # average duration of the last 3 epochs
    epoch_time_queue.append(epoch_time)
    avg_time = sum(epoch_time_queue)/len(epoch_time_queue)
    # if the next epoch will likely surpass the time limit, then stop here
    estimated_next_total_time = (time.time() - start_time)/60 + avg_time
    if args.time_limit &gt; 0 and estimated_next_total_time &gt; args.time_limit:
        break
</code></pre>
<p>The issue is that the elapsed time may be different between processes. For example, at the end of the 5th epoch, the process on GPU1 may think that it will surpass the time limit at the next (6th) epoch by a few seconds, so it stops; while GPU2 thinks that it will be able to finish the 6th epoch a few seconds before the limit, so it will continue, which is not good.</p>
<p>I would like to know if there is a way for the processes to communicate about this. Ideally, a process should wait for all the other processes to finish the current epoch to decide whether to go for the next epoch or not.</p>

mrshenli: <p><a class="mention" href="/u/f10w">@f10w</a> yep, this is possible, you can use the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_gather" rel="nofollow noopener"><code>all_gather</code></a> API to let every process to collect elapsed time from all processes.</p>

f10w: <p>Thanks for the reply!<br>
When we do <code>torch.distributed.all_gather()</code>, does it create some kind of “barrier” between the processes?</p>

mrshenli: <p>Yes, it does, all collective communications (e.g., <code>boradcast</code>, <code>all_reduce</code>, <code>all_gather</code>, etc.) can be considered as a barrier.</p>
Downgrade cuda 10.2 to 10.0

zahra: <p>Hi,</p>
<p>How do  I downgrade cuda version from 10.2 to 10.0?</p>
<p>Regards</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/torch-1-1-0-is-not-compatible-with-cuda-10-2/94552/10">here</a>.<br>
If you want to install the binaries with CUDA10.0, please refer to the other thread (and post the error message, if you cannot install it).<br>
On the other hand, if you want to build from source using CUDA10.0, you would have to install the corresponding CUDA version on your workstation and follow <a href="https://github.com/pytorch/pytorch#from-source">these steps</a> to build PyTorch from source.</p>
Set parameter gradient to zero in C++

Ryan5566: <p>Hi, I am new to C++ Pytorch.<br>
I want to set the parameters’ gradient to zero. My code is like:</p>
<pre><code class="lang-auto">torch::Tensor a; // 2x2 matrix
torch::Tensor b; // 2x1 vector
b.requires_grad_(true);
torch::Tensor c = a.mm(b); // 2x1 vector

c[0] .backward();
b.grad.data.zero_();
</code></pre>
<p>the error is:</p>
<pre><code class="lang-auto">error: ‘b.at::Tensor::grad’ does not have class type
     b.grad.data.zero_();
</code></pre>
<p>I couldn’t find the function which set the gradient to zero in C++.<br>
How can I achieve <code>b.grad.data.zero_()</code> same as in python?</p>
<p>Thank you.</p>

Lin_Jia: <p>If you train with an optimizer in your code, you can zero the gradients through the optimizer by:</p>
<p>optimizer.zero_grad();</p>

Ryan5566: <p>Thank you Lin_Jia.<br>
I did it using an optimizer.</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; para;
para.push_back(b);
torch::optim::Adam optim(para);

c[0].backward();
optim.zero_grad();
</code></pre>
Extracting Feature Vectors Using Pre-Build Models

Luke_Kollmorgen: <p>I’m pretty new to using PyTorch and trying to learn about extracting feature vectors. I have been able to piece together  some code that will extract a feature vector from a single image. See below.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision
import torchvision.models as models
import torchvision.transforms as transforms
from torch.autograd import Variable
from PIL import Image



img=Image.open("Documents/01235.png")

# Load the pretrained model
model = models.resnet18(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('avgpool')

# Set model to evaluation mode
model.eval()

transformations = torchvision.transforms.Compose([
    torchvision.transforms.Resize(256),
    torchvision.transforms.CenterCrop(224),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])


def get_vector(image):
    # Create a PyTorch tensor with the transformed image
    t_img = transformations(image)
    
    # Create a vector of zeros that will hold our feature vector
    # The 'avgpool' layer has an output size of 512
    my_embedding = torch.zeros(512)

    # Define a function that will copy the output of a layer
    def copy_data(m, i, o):
        my_embedding.copy_(o.flatten())                 # &lt;-- flatten

    # Attach that function to our selected layer
    h = layer.register_forward_hook(copy_data)
    # Run the model on our transformed image
    with torch.no_grad():                               # &lt;-- no_grad context
        model(t_img.unsqueeze(0))                       # &lt;-- unsqueeze
    # Detach our copy function from the layer
    h.remove()
    # Return the feature vector
    return my_embedding


pic_vector = get_vector(img)
pic_vector.size()
</code></pre>
<p>What I’m curious about is if there is some documentation somewhere that would talk more about the various layers we are able to extract these vectors from. For example, if I wanted to switch to the Resnet 152 model, how could I learn what all the available layers are I could extract from/what the syntax is for each layer? I’ve been looking around for some more documentation on the “_modules.get” piece and I can’t seem to find what I’m looking for.</p>

ptrblck: <p>The approach of using forward hooks looks alright.<br>
You could generally use all intermediate activations as your “feature” tensors.<br>
To see all modules in a model, you could use <code>print(model)</code>, which would print all modules, or look at the implementation of the model directly (e.g. <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py">resnet implementation</a>).</p>
Please help, torch.cuda.is_available() True but is !nvidia-smi : No running processes found

Stihl: <p>Please help<br>
I put all my tensor to(device) device = torch.device(“cuda:0”)<br>
I add .to(device) for every operation I make with tensors<br>
torch.cuda.get_device_name(0) return Tesla P100-PCIE-16GB<br>
But in !nvidia-smi : no runnung processes found<br>
As I see gpu doesn’t work properly.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aadb95b4e9a0136f32b7cae2701eb278b2808622.png" data-download-href="https://discuss.pytorch.org/uploads/default/aadb95b4e9a0136f32b7cae2701eb278b2808622" title="4"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aadb95b4e9a0136f32b7cae2701eb278b2808622.png" alt="4" data-base62-sha1="ontCWT7n21yQGEJA858eq8sB73Q" width="690" height="355" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">780×402 10.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>nvidia-smi</code> might now show all processes depending on the “environment”. E.g. I see this behavior often in docker containers and don’t know where this issue comes from. <code>ps</code> should show the processes however.<br>
The GPU seems to be filled with ~1GB of data and its utilization is at 15%, so how did you determine that the GPU is not working properly?</p>
Masking the Latent Dimension to make the same size as an image

g33sean: <p>How would would take latent code with dimensions (batch_size,K) and then do some element wise multiplication to turn that matrix into dim (batch_size,channels, height, width) where height and width will be N by N matrix essentially and you fill K dimensions into the height, width dimensions. Since K will be less than N by N matrix or (h,w) there will be n-k zero entries.</p>

ptrblck: <p>An elementwise multiplication won’t change the number of elements in the tensors, so you would have to <code>view</code> or <code>reshape</code> the tensor into the desired shape.<br>
If <code>channels*height*width</code> is larger than <code>K</code>, you could create a new tensor via <code>output = torch.zeros(batch_size, channels, height, width)</code> and copy the reshaped input into it.</p>
Load video frames in stack

Sayeed_Shaiban: <p>Suppose I have converted videos to frames and the folder structure is such</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b93e0925d69b1bd96289721b22606cbe7730b005.png" alt="image" data-base62-sha1="qqJf40MOe6DpHEF3wWIir3zBTjD" width="187" height="336"></p>
<p>So now I can use the ImageFolder to load the frames per the classes. But training models frame-wise seem to perform very poor for me. So I want to stack all the frame per video and assign a single label against it. What would be an easy way to achieve this?</p>

Lin_Jia: <p>It seems your pain point is that training is slow. Are you using batch data to train? Every batch will consist of different training frames, and they can go through your neural network in one pass.</p>

Sayeed_Shaiban: <p>What I meant was I need a way to keep track of the video frames from a certain video when I load in the dataset instead of each of them being a seperate entitiy.</p>
<p>Alternately, is there something like the keras SlidingFrameGenerator in pytorch?</p>

Lin_Jia: <p>You might want to checkout SequentialSampler, which is related to data loader: <a href="https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html" rel="nofollow noopener">https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html</a></p>
<p>My experience with SequentialSampler is that it will keep your training data sequence, so that the video frame sequence can be reserved during training.</p>
How to fine-tune my baseline model with other data?

Peng_Ding: <p>hi, now I have a text classfication baseline model and some other data which is used for data augmentation. How can I use this data to fine-tune my baseline model instead of training it from scratch? Is there some tips how to do it ?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html">finetuning tutorial</a> might be a good starter.<br>
While it uses pretrained <code>torchvision</code> models, you could use your own pretrained model and use the general workflow of the tutorial.<br>
Let me know, if you get stuck. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>