Learnable scalars

Niki: <p>Hi, I am looking for mapping inputs by learnable scalars, this is correct?</p>
<pre><code class="lang-auto">for batch_idx, (inputs, targets) in enumerate(trainloader):
            if use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()
                
            shape = torch.Size((batch_size, 3, 32, 32))
            scalar = torch.cuda.FloatTensor(shape)
            M1 = torch.rand(shape, out=scalar)
          
            inputs_new = inputs * M1 
          
</code></pre>

ptrblck: <p>Could you explain your use case a bit more?<br>
It seems there are no trainable parameters in your current code.</p>

Niki: <p>Thank you for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
I want to do some mappings/transferring over the inputs. I saw this <a href="https://discuss.pytorch.org/t/multiply-feature-map-by-a-learnable-scalar/1520">https://discuss.pytorch.org/t/multiply-feature-map-by-a-learnable-scalar/1520</a> should use variable and requires_grad? not sure how I can apply that.</p>

G.M: <p>You should make that scalar a <code>nn.Parameter</code>.</p>

Niki: <p>Thank you for your reply, <a class="mention" href="/u/g.m">@G.M</a>.<br>
So, in model I should add</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, batch_size):
        super(Model, self).__init__()
        shape = torch.Size((batch_size, 3, 32, 32))
        scalar = torch.FloatTensor(shape)
        self.multp = nn.Parameter(torch.randn(shape, out=scalar))
</code></pre>
<p>and in training should add</p>
<pre><code class="lang-auto">def train(epoch):
     for batch_idx, (inputs, targets) in enumerate(trainloader):
            if use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()
 
            M1 = net.parameters()
            inputs_new = inputs * M1
            outputs = net(inputs_new)
</code></pre>
<p>If this is correct, how I can give batch_size to the network in every epoch ? since I am defining <code>net = models.__dict__[args.model](num_classes)</code> before training function and I am considering batch_size as <code>batch_size = inputs.size(0)</code>. I should bring <code>net</code> inside the training function?</p>

G.M: <ol>
<li>No, you should use the parameter in your model code: <code>scalar = nn.Parameter(torch.randn(shape))</code>.</li>
<li>Pytorch supports scalar multiplication like this: (B,C,H,W)*(C,H,W).</li>
</ol>

Niki: <p>Thank you, <a class="mention" href="/u/g.m">@G.M</a>. How should I apply this in optimizer?<br>
the current one is this</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(net.parameters(), lr=args.lr, betas=(0.9, 0.999), weight_decay=args.decay)
</code></pre>

G.M: <p>Include that scalar into your model, like make it an attribute to your model in the constructor.</p>

Niki: <p>you mean just adding</p>
<pre><code class="lang-auto">self.scalar= nn.Parameter(torch.randn(3,32,32))
</code></pre>
<p>in the model?</p>

G.M: <p>Yeah something like this.</p>

Niki: <p>If I want to find random numbers from uniform distributions between [0,3) using</p>
<pre><code class="lang-auto">torch.empty(3, 32,32).uniform_(0, 3)
</code></pre>
<p>is correct?</p>
<p>and from Gaussian distribution with mean 0 and std 9</p>
<pre><code class="lang-auto">torch.normal(0, 9, size=(3,32,32))
</code></pre>
<p>I mean these will do the same as  <code>torch.rand</code>  and  <code>torch.randn</code>  but in different intervals?</p>
<p>How should define mean and std for <code>torch.normal</code>?</p>
<pre><code class="lang-auto"> self.m = nn.Parameter(torch.normal(torch.tensor(0), torch.tensor(9), size=(3,32,32)))
TypeError: normal() received an invalid combination of arguments - got (Tensor, Tensor, size=tuple), but expected one of:
 * (Tensor mean, Tensor std, torch.Generator generator, Tensor out)
 * (Tensor mean, float std, torch.Generator generator, Tensor out)
 * (float mean, Tensor std, torch.Generator generator, Tensor out)
</code></pre>

G.M: <p>I guess u would have to use <code>torch.empty(3, 32,  32).normal(0, 9)</code>, or <code>torch.normal(0, 9, tc.empty(3, 32, 32))</code></p>