akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Runtime error in custom Autograd function

aevens: <p>Hi,</p>
<p>I’m having an issue with a custom autogrid function that implements a sorting operation on an input x.</p>
<p>The operation sorts the rows of X based on taking the sorted indices of an  input vector s.</p>
<p>The sorting part of the operation is all fine and working outside of the method, but I am having problems implementing it inside the method.</p>
<pre><code class="lang-auto">from torch.autograd import Function

class SortBy(Function): 
  @staticmethod
  def forward(ctx, x, s):
    
    _, indices = torch.sort(s)
    results = x[indices, :]
    ctx.save_for_backward(results)

    return result
  
  @staticmethod
  def backward(ctx, grad_output, s):
    
    results, = ctx.saved_tensors

    _, indices = torch.sort(s)
    _, index2 = torch.sort(indices)

    return (results * grad_output)[index2, :]
</code></pre>
<p>I’m attempting to test my code using the following:</p>
<pre><code class="lang-auto">x = torch.autograd.Variable(torch.randn(4,5),requires_grad=True)

s = torch.randn(4)

Y = SortBy.apply(X, s)

Yhat = torch.sum(Y)

Yhat.backward(s)
</code></pre>
<p>But I get the below error:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-283-b5596230715d&gt; in &lt;module&gt;()
      7 Yhat = torch.sum(Y)
      8 
----&gt; 9 Yhat.backward(s)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in _make_grads(outputs, grads)
     27                                    + str(grad.shape) + " and output["
     28                                    + str(outputs.index(out)) + "] has a shape of "
---&gt; 29                                    + str(out.shape) + ".")
     30             new_grads.append(grad)
     31         elif grad is None:

RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([4]) and output[0] has a shape of torch.Size([]).

</code></pre>
<p>I don’t understand where the dimension mismatch is coming in. I assume that the grad_output from the torch.sum step returns a single scalar gradient which I then broadcast to my matrix <em>results</em> in order to get the results, then I apply my reverse indexing to ‘reroute’ the gradient to where I want it.</p>

albanD: <p>Hi,</p>
<p>You don’t need Variables, anymore, you can simply do: <code>torch.randn(4, 5, requires_grad=True)</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>For your Function. Remember that the backward pass works in reverse from the forward. So the inputs to the backward match the outputs of the forward.<br>
In your case, the backward should only have one input: gradient of the loss wrt result. And should return two things, the gradient wrt x and s.</p>
Unsupervised segmentation for defect detection

spiovesan: <p>I would like to train a network for defect detection, on the same type of object (i.e. a production line for plastic caps); I have an image set good with parts and an image set with defective parts. I know that someone is able to do it using a sliding window of <code>patch_size</code> to process the entire image, with a certain overlap, just using the two datasets. Does someone know if there is anything similar in pytorch or give me some hint for this?</p>
Batch Optimization of "Input Samples"

Kamer_Ali_Yuksel: <p>Hello!</p>
<p>I have made an implementation that optimizes input vector samples for a given loss function. With single or few input vector samples it works perfect with Adam. However, when I place 10K samples in a batch and take the mean of their individual losses as an optimization target, the optimization gets very slow and hence inefficient. (The speed of each iteration does not change much but the input vectors get optimized in much smaller steps. I have tried to increase the learning-rate but that did not help much) I guess that it is simply unable to properly figure out which input samples contributes how much to the increase or reduction in average loss with Auto-grad. I have tried to increase the learning rate but that did not help much. Did anybody else here previously worked in such problem? What would be the best way to tackle it? I would be more than glad if anybody can help. Thank you in advance for your time and consideration.</p>
<p>Note: I would be also fine if the learning goes fast in a minority of samples that are the best candidates by making the optimizer to focus into them most. In that case, would it for example make sense to optimize the loss for a percentile loss, rather than the average loss from all samples? (e.g. optimizing the worse candidates can result in better exploration but the exploitation would be much worse in that case).</p>
<p>Note: I am not working in that space but I would assume that adversarial attack generation to a deep learning model can be one example  use-case where such inputs require to be optimized / generated.</p>
<p>Sincerely,<br>
Kamer</p>

Kamer_Ali_Yuksel: <p>I found an initial solution to the problem, which is re-scaling the gradients of each input sample so that the norm of the smallest sample is one. However, to keep using a regular optimizer, I need to figure out how that effects the momentum of the optimizer, etc. At the moment, I am just updating the input samples by adding them their re-scaled gradients. I am looking forward for any suggestions from more experienced.</p>
How to divide positive samples and negative samples

111220: <p>I want to use the torch. Nn. TripletMarginLoss this API, but I don’t know in the image classification data division method of positive and negative samples, if we now get to the label and the output, how to carry out the positive and negative samples?</p>
<p>Is there a specific code implementation method</p>

111220: <p>Then I am not very clear about the concept of positive sample and negative sample in the image classification, how to be a positive sample, how to be a negative sample</p>
Doing operations on model parameters

yegane: <p>Hi,<br>
I was wondering if there is any reliable method to manipulate model parameters, without any side-effects, so it could be deterministic in both GPU and CPU.<br>
I want to be able add some numbers to a specific node of network or all of them, and even add or take average of two model from the same architecture.</p>

ptrblck: <p>Parameter manipulation can be performed inside a <code>with torch.no_grad()</code> block.<br>
I’m not sure to understand the requirement about deterministic results properly.<br>
Do you want to make a GPU run match a CPU run or just GPU to GPU and CPU to CPU?</p>

yegane: <p>Thank you for the answer.<br>
Yes, I meant for both CPU and GPU, but for the manipulation of model parameters, I was looking for some pytorch method, that could manipulate the model parameters.<br>
for others who are visiting this post:<br>
A sample of manipulation from PySyft:</p>
<pre><code class="lang-auto">    params = model.named_parameters()
    dict_params = dict(params)
    with torch.no_grad():
        for name, param in dict_params.items():
            dict_params[name].set_(dict_params[name].data * scale)
</code></pre>
Is there a pytorch API to remove a layer from forward pass?

Sohil_Newa: <p>Hello all,<br>
I have been trying to remove a layer completely from both forward and backward pass while training the model.<br>
Say below is the model.</p>
<blockquote>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(1, 10, kernel_size=5)<br>
self.conv2 = nn.Conv2d(10, 20, kernel_size=5)<br>
self.fc1 = nn.Linear(320, 50)<br>
self.fc2 = nn.Linear(50, 10)</p>
</blockquote>
<pre><code>def forward(self, x):
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = F.relu(F.max_pool2d(self.conv2(x), 2))
    x = x.view(-1, 320)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return F.log_softmax(x)
</code></pre>
<p>Is there any way i can precompute the layer output , load it into a train loader object and use it in future epochs while completely getting rid of training that particular initial layer.<br>
We can use the detach() method in order to prevent backprop but forward pass still computes the forward pass each time even though the weights parameters are fixed which seems redundant. I will really appreciate if there is a hack of any sort that can help make this possible.</p>

Mazhar_Shaikh: <p>Hi <a class="mention" href="/u/sohil_newa">@Sohil_Newa</a>,<br>
I know this is an old topic, so apologies. But were you able to find any hack for this?<br>
Thanks &amp; Regards</p>

ptrblck: <p>If you want to remove a particular layer, you could replace it with <code>nn.Identity</code>.<br>
I’m not sure what the use case from the original post is, as it seems the output should be precomputed somehow.</p>

Mazhar_Shaikh: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>The use case is to create conditional computation of layers, then define multiple entry points to a network and compute layers only corresponding to the specific entry point. I was trying to apply this to progressive training of CNNs for classification/segmentation, applying the principle from Progressive GANs. The advantage could be that this method could generalize to different standard network architectures.</p>
<p>So what I am looking for is below:</p>
<pre><code class="lang-auto">new_resnet18 = MagicConditionalModifier(resnet18, stages=['layer1', 'layer2'])
stage_1_input #torch.tensor(Bx64xHxW)
output = new_resnet18(stage_1_input, stage=1) #or stage='layer1')
stage_2_input #torch.tensor(Bx128xHxW)
output = new_resnet18(stage_2_input, stage=2)#or stage='layer2')
</code></pre>
<p>We can of course create a new network from the old network and create the conditional computational paths, but it seemed to me that there had to be a better way. (Maybe I’m overthinking this!)</p>
Use cv2 to load images in custom dataset

mlearner: <p>Hello,<br>
I wrote a custom dataset to upload images and it works correctly when I use <code>PIL</code> module. How can i use <code>cv2</code> instead?</p>
<p>If I just try to read the image with</p>
<pre><code class="lang-auto">img = cv2.imread(file_path)
</code></pre>
<p>it throws the error<br>
<code>img should be PIL Image. Got &lt;class 'numpy.ndarray'&gt;</code><br>
produced by the transform function (the PyTorch transformation from <code>torchvision.transforms</code> to be clear).</p>
<p>Any idea on how to solve?</p>

JuanFMontesinos: <p>Transforms are PIL based, thus, it only works with PIL. There exist a library called pillow-simd which is a drop-and-replace PILLOW library. It’s optimized and faster than native PIL.</p>

IliasPap: <p>After loading with opencv you can convert it to PIL image using<br>
<code>PIL.Image.fromarray( ) </code> and then apply torchvision transforms</p>

mlearner: <p>And if I want to upload the images using PIL library and then open them with cv2? Is it possible?<br>
I tried, but I was not able to do that, since they seem to have a different format.</p>

IliasPap: <p>again you can use PIL to upload and convert img to numpy array.<br>
Just check tha RGB format because cv2 loads image in BGR format by default</p>
How to efficiently allocate GPU memory?

Veril: <p>So I have a toy attractor model:</p>
<pre><code class="lang-auto">points = torch.rand((2**16,3), dtype=torch.float32, device=cuda, requires_grad=False)
</code></pre>
<pre><code class="lang-auto">def attractor_func(v, a = 10.0, b = 28.0, c = 8.0 / 3.0):
	dv = torch.zeros(points.shape, dtype=torch.float32, device=cuda, requires_grad=False)

	#lorenz
	dv[:, 0] = a * (v[:, 1] - v[:, 0])
	dv[:, 1] = v[:, 0] * (b - v[:, 2]) - v[:, 1]
	dv[:, 2] = v[:, 0] * v[:, 1] - c * v[:, 2]

	return dv
</code></pre>
<p>This function is called 4 times from an Runge-Kutta function that is called as quickly as the cpu and gpu can manage.</p>
<pre><code class="lang-auto">def rk4(func, h, v):
	k1 = func(v)
	k2 = func(v + (h / 2.0) * k1)
	k3 = func(v + (h / 2.0) * k2)
	k4 = func(v + h * k3)
	return h * (k1 / 6.0 + k2 / 3.0 + k3 / 3.0 + k4 / 6.0)
</code></pre>
<p>with this called in between “rendering”:</p>
<pre><code class="lang-auto">for _ in range(32):
		points += rk4(attractor_func, 0.01, points)
</code></pre>
<p>But something tells me that this is not the most efficient code when it comes to managing GPU memory. I can’t even convert attractor_func to a JIT script, if that is of any use.</p>
<p>Perhaps there is a way to vectorize the dv slices across the second dimension as well?</p>

tom: <p>If you write it as torch.chunk + pointwise + torch.cat, it should work with the JIT fuser.</p>

Veril: <p>It occurred to me that the thomas attractor was too simple in allowing the concentration of point-wise operations, that was not the general case I wished to solve for. I changed it to lorenz.</p>

Veril: <p><a class="mention" href="/u/tom">@tom</a></p>
<p>TorchScript fails:</p>
<pre><code class="lang-auto">  File "./main.py", line 37, in rk4
    k2 = func(v + (h / 2.0) * k1)
RuntimeError: default_program(102): error: identifier "__ldg" is undefined

1 error detected in the compilation of "default_program".
</code></pre>
<p>with:</p>
<pre><code class="lang-auto">@torch.jit.script
def attractor_func(v):
	x, y, z = torch.chunk(v, 3, dim=1)

	#lorenz
	a = 10.0
	b = 28.0
	c = 8.0 / 3.0
	dx = a * (y - x)
	dy = x * (b - z) - y
	dz = x * y - c * z

	return torch.cat([dx, dy, dz], 1)
</code></pre>
<p>Without JIT it works.</p>
<p>My card is compute 3.0 (pytorch compiled from source w/ cuda 10.2)</p>

tom: <p>Ohoh. I think that is an incompatibility of your card which isn’t supported anymore. You could go and delete the __ldg in the source code and re-compile from scratch. I will cost you when fusing broadcasting operations.</p>
<p>Best regards</p>
<p>Thomas</p>
Nll loss only calculating error for one of two output nodes

pytorcher: <p>Not sure if I just dont understand nll loss correctly, but I am running super simple network that just has one layer with two conv nodes taking in a 2x2 input which are either 0’s or 1’s, and then a sigmoid nonliniarity.  I have assigned the layer a register_backward_hook to a function that is just printing out the values for grad_input and grad_output.<br>
When I run this with CrossEntropyLoss grad_output will have values for both nodes e.g.:</p>
<blockquote>
<p>grad_output<br>
(tensor([[[[ 0.1271]],<br>
[[-0.1269]]]], device=‘cuda:0’),)</p>
</blockquote>
<p>but when I run with NLLLoss only one of the nodes will have a value, which seems to be the target node. e.g.:</p>
<blockquote>
<p>grad_output<br>
(tensor([[[[ 0.0000]],<br>
[[-0.2495]]]], device=‘cuda:0’),)</p>
</blockquote>
<p>Is this expected behavior, or am I possibly doing something incorrectly.  Rest of the code is identical besides switching the criterion function before calling loss = criterion(outputs, targets) which in both cases looks like:</p>
<blockquote>
<p>targets<br>
tensor([1], device=‘cuda:0’)<br>
outputs<br>
tensor([[0.5123, 0.4775]], device=‘cuda:0’, grad_fn=&lt;ViewBackward)</p>
</blockquote>

ptrblck: <p>Note that <code>nn.CrossEntropyLoss</code> expects raw logits, which can take any value, while <code>nn.NLLLoss</code> expects log probabilities as the input.<br>
Internally in <code>nn.CrossEntropyLoss</code>, <code>F.log_softmax</code> and <code>nn.NLLLoss</code> will be applied.</p>
<p>I’m not sure how you’ve calculated your output, but it doesn’t seem to contain log probabilities. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
Forward() missing 1 required positional argument

Amil_Dravid: <p>I am running this github code: <a href="https://github.com/xbresson/spectral_graph_convnets/blob/master/02_graph_convnet_lenet5_mnist_pytorch.ipynb" rel="nofollow noopener">https://github.com/xbresson/spectral_graph_convnets/blob/master/02_graph_convnet_lenet5_mnist_pytorch.ipynb</a>.</p>
<p>I had to change the forward functions to static methods because that’s what new Pytorch requires I believe. But I get the following error:</p>
<p>forward() missing 1 required positional argument: ‘lmax’.</p>
<p>Yet, I am passing in lmax. What is going wrong. You can see the code in the github repo.</p>

ptrblck: <p>The linked notebook seems to have created a valid output or are you using another version of the notebook?</p>
<p>PS: Don’t manually call <code>model.forward</code>, but instead the model directly via <code>output = model(x)</code>, as otherwise e.g. registered hooks won’t be called.</p>
Split data for train, test, validation in dataloader

Geoffrey_Payne: <p>I take a dataset and split it into 3 and then configure a dataloader to access each one, as follows;</p>
<blockquote>
<p>full_data_args={‘data_dir’:‘penguin_data/data’, ‘data_file’:‘penguin_csv.csv’,‘stage’:‘full’}<br>
data_batch = dataset.PenguinData(**full_data_args)<br>
train_data_params = {‘batch_size’:512, ‘shuffle’:True, ‘num_workers’:0, ‘pin_memory’: True}<br>
train_dataset = data.DataLoader(data_batch.train_dataset, **train_data_params)<br>
valid_data_params = {‘batch_size’:16, ‘shuffle’:True, ‘num_workers’:0, ‘pin_memory’: True}<br>
test_dataset = data.DataLoader(data_batch.train_dataset, **valid_data_params)<br>
valid_dataset = data.DataLoader(data_batch.val_dataset, **valid_data_params)</p>
</blockquote>
<p>However I understand that a better approach is to attach a dataloader to the whole dataset and use that to access the data for training, testing and validation.<br>
I can’t find an example of how to do this. Can you show me how this can be done? It may be that the batch size is the same.</p>

ptrblck: <aside class="quote no-group" data-username="Geoffrey_Payne" data-post="1" data-topic="67250">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geoffrey_payne/48/18868_2.png" class="avatar"> Geoffrey_Payne:</div>
<blockquote>
<p>However I understand that a better approach is to attach a dataloader to the whole dataset and use that to access the data for training, testing and validation.</p>
</blockquote>
</aside>
<p>I don’t think this is a better approach and I would stick to separate <code>DataLoader</code>s for each split.<br>
Why do you think it might be a better approach?</p>

Geoffrey_Payne: <p>It was what a colleague told me but given your response he must be wrong! Thank you for that.</p>

ptrblck: <p>Haha, no he might be right!<br>
It would be interesting to hear his opinion and I might be completely wrong. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>In this use case, I think splitting the <code>Datasets</code> before wrapping them in <code>DataLoaders</code> would be clearer than using a custom sampler, <code>collate_fn</code> or other methods to get the corresponding samples from a single <code>DataLoader</code>.</p>
Runtime Error:size mismatch

nikhil_rana: <h1>Here is my model but it gives me the runtime error for size mismatch</h1>
<p>self.conv1 = nn.Conv2d(1, 32, 5)<br>
self.bn1 = nn.BatchNorm2d(32)<br>
self.pool_L1 = nn.MaxPool2d(2,2)</p>
<pre><code>    self.conv2 = nn.Conv2d(32, 64, 3)
    self.bn2 = nn.BatchNorm2d(64)
    self.pool_L2 = nn.MaxPool2d(2,2)
    
    self.conv3 = nn.Conv2d(64, 128, 3)
    self.bn3 = nn.BatchNorm2d(128)
    self.pool_L3 = nn.MaxPool2d(2,2)
    
    self.conv4 = nn.Conv2d(128, 256, 3)
    self.bn4 = nn.BatchNorm2d(256)
    self.pool_L4 = nn.MaxPool2d(2,2)
    
    self.conv5 = nn.Conv2d(256, 512, 3)
    self.bn5 = nn.BatchNorm2d(512)
    self.pool_L5 = nn.MaxPool2d(2,2)
    
    
    self.dense1 = nn.Linear( 6*6*512 ,1024 )
    self.dense2 = nn.Linear( 1024, 136)
    
    ## Note that among the layers to add, consider including:
    # maxpooling layers, multiple conv layers, fully-connected layers, and other layers (such as dropout or batch normalization) to avoid overfitting
    
    self.drop1 = nn.Dropout(0.20)
    self.drop2 = nn.Dropout(0.25)
    self.drop3 = nn.Dropout(0.30)
    self.drop4 = nn.Dropout(0.35)
    self.drop5 = nn.Dropout(0.40)
    self.drop6 = nn.Dropout(0.25)
    

    
def forward(self, x):
    ## TODO: Define the feedforward behavior of this model
    ## x is the input image and, as an example, here you may choose to include a pool/conv step:
    ## x = self.pool(F.relu(self.conv1(x)))
    
    x = self.bn1(F.relu(self.conv1(x)))
    x = self.pool_L1(x)
    x = self.drop1(x)
    
    x = self.bn2(F.relu(self.conv2(x)))
    x = self.pool_L2(x)
    x = self.drop2(x)
    
    x = self.bn3(F.relu(self.conv3(x)))
    x = self.pool_L3(x)
    x = self.drop3(x)
    
    x = self.bn4(F.relu(self.conv4(x)))
    x = self.pool_L4(x)
    x = self.drop4(x)
    
    x = self.bn5(F.relu(self.conv5(x)))
    x = self.pool_L5(x)
    x = self.drop5(x)
    
    x = x.view(x.size(0),-1)              # Used to redefine the shape of the x by rows:-x.size(0), just like numpy.
    
    x = F.relu(self.dense1(x))
    x = self.drop6(x)
    
    x = F.relu(self.dense2(x))
</code></pre>
<p>RuntimeError: size mismatch, m1: [10 x 12800], m2: [18432 x 1024]<br>
Here 10 in m1 I have take the batch size</p>

ptrblck: <p>Based on the error message, it seems the input to <code>self.dense1</code> has the shape <code>[10, 12800]</code>, while the number of input features is defined as <code>6*6*512=18432</code>.<br>
The layer expects a matching number of input features, so you should change <code>self.dense1</code> to:</p>
<pre><code class="lang-python">self.dense1 = nn.Linear(12800, 1024)
</code></pre>
Runtime Error: cuDNN error: CUDNN_STATUS_BAD_PARAM

SurajSubramanian: <p>While the second epoch of my CycleGAN model was running I ran into this error<br>
<code>RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM</code></p>
<p>and the error pointed to this function</p>
<pre><code class="lang-auto">def pass_through_discriminator(discriminator, image):
    score, k = 0, Variable(torch.zeros(1)).type(dtype)
    xp, yp = 0, 0
    x, y = 70, 70
    offset = 25
    
    while x &lt; 128:
        while y &lt; 128:
            k += 1
            score += discriminator(image[:, :, xp:x, yp:y])
            yp += offset
            y += offset
            
        xp += offset
        x += offset
        
    return score / k

</code></pre>
<p>I read in other topics that the size of the image has to be increased, but am not very clear.<br>
Any suggestions on how to solve this<br>
Thanks in advance !</p>

ptrblck: <p>Could you post a code snippet to reproduce this error, please?</p>
Deal with saved features (very large)

kl_divergence: <p>Hi,</p>
<p>My model is a transformer and have a single GPU, so naturally I have CUDA memory constraints. I’ve saved Faster RCNN features in a tsv format which is 40 GB in size. Now when I start, all the data from the features is loaded in CPU RAM and I run of out of memory. I’ve set <code>num_workers=1</code> btw. Another option is obtain features from Faster RCNN in real time but both transformer and faster_rcnn can’t sit on my GPU RAM. Is there an efficient way of obtaining features from the tsv file ?<br>
My code looks like:</p>
<pre><code class="lang-auto">with open(fname) as f:
        reader = csv.DictReader(f, FIELDNAMES, delimiter="\t")
        for i, item in tqdm(enumerate(reader)):

            for key in ['img_h', 'img_w', 'num_boxes']:
                item[key] = int(item[key])
            
            boxes = item['num_boxes']
            decode_config = [
                ('objects_id', (boxes, ), np.int64),
                ('objects_conf', (boxes, ), np.float32),
                ('attrs_id', (boxes, ), np.int64),
                ('attrs_conf', (boxes, ), np.float32),
                ('boxes', (boxes, 4), np.float32),
                ('features', (boxes, -1), np.float32),
            ]
            for key, shape, dtype in decode_config:
                item[key] = np.frombuffer(base64.b64decode(item[key]), dtype=dtype)
                item[key] = item[key].reshape(shape)
                item[key].setflags(write=False)

            data.append(item)
    return data
</code></pre>
Compare image and feature vector in the model

Hou_ZeYu: <p>I have a binary classification, there are image and variable in the data set, I have an idea for compare image and variable together.</p>
<p>Every time when I pass conv-layer, I want to multiply a weight scalar to all feature map, where weight scalar is computed from a fc-layer.</p>
<p>For example, suppose batch size is 8, there are two tensor x1 and x2, where x1’s size is (8,3,224,224) and x2’s size is (8,16).</p>
<blockquote>
<pre><code>import torch
from torch.nn import Module, Sequential
from torch.nn import Conv2d, BatchNorm2d, ReLU, MaxPool2d, Softmax, Linear
import numpy
batch_size = 8
x1  = torch.rand(batch_size*3*224*224).view(batch_size,3,224,224)
x2  = torch.rand(batch_size*16).view(batch_size,16)
</code></pre>
</blockquote>
<p>I define conv-layer and fc-layer and compute the output from image and variable.</p>
<blockquote>
<p>conv_01     = Conv2d(in_channels=3, out_channels= 9, kernel_size=3, stride=1, padding=1)<br>
linear_02 = Linear(16, 1)<br>
c1 = conv_01(x1)   ##  torch.Size([8, 9, 224, 224])<br>
c2 = linear_02(x2)  ## torch.Size([8, 1])</p>
</blockquote>
<p>The problem is write the suitable code like blow.</p>
<blockquote>
<p>I want to do like blow<br>
c1[0,:,:,:] = c1[0,:,:,:] * c2[0,0] # 1st data in the mini-batch<br>
c1[1,:,:,:] = c1[1,:,:,:] * c2[0,1] # 2nd data in the mini-batch<br>
c1[2,:,:,:] = c1[2,:,:,:] * c2[0,2]<br>
c1[3,:,:,:] = c1[3,:,:,:] * c2[0,3]<br>
c1[4,:,:,:] = c1[4,:,:,:] * c2[0,4]<br>
c1[5,:,:,:] = c1[5,:,:,:] * c2[0,5]<br>
c1[6,:,:,:] = c1[6,:,:,:] * c2[0,6]<br>
c1[7,:,:,:] = c1[7,:,:,:] * c2[0,7]<br>
output is a (8, 9, 224, 224)<br>
and do more layer like this operation</p>
</blockquote>
<p>I already take a look in the <a href="https://discuss.pytorch.org/t/multiply-feature-map-by-a-learnable-scalar/1520/2" class="inline-onebox">Multiply feature map by a learnable scalar - #2 by fmassa</a>.<br>
But this only support when batch size is 1, but in my case, batch size is more than 1.<br>
How to write a suitable code to the forward function in my case?<br>
Thanks a lot.</p>