How to predict one single example using a pre-trained model with bach normalization layer?

Mengran_Wang: <p>Hi all,<br>
This may be not a difficult question, but it has bothered me a few days! Really need some help.<br>
I have trained a model with following structure:</p>
<pre><code class="lang-auto">RNNModel_GRU(
  (embed): Embedding(6, 1, padding_idx=1)
  (gru): GRU(1, 64, bidirectional=True)
  (linear1): Linear(in_features=128, out_features=32, bias=True)
  (linear2): Linear(in_features=32, out_features=1, bias=True)
  **(b_n): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)**
  (dropout): Dropout(p=0.2)
  (relu): ReLU()
)
</code></pre>
<p>Now I’m going to load this model and predict new examples, <strong>since I’ve trained the model by feeding 32 data per batch, when I try a single new data to the model, an error occurs:</strong></p>
<pre><code class="lang-auto">In [259]: scores = model(preprocess_seq)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-259-3ccf22219850&gt; in &lt;module&gt;()
----&gt; 1 scores = model(preprocess_seq)

/picb/rnomics3/wangmr/INSTALL/miniconda3/envs/GPU/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

&lt;ipython-input-219-9bb382769aa2&gt; in forward(self, text)
     28         linear1 = self.relu(linear1)
     29         linear1 = self.dropout(linear1) # [32 * 32]
---&gt; 30         **b_n = self.b_n(linear1) # 32 * 32**
     31         linear2 = self.linear2(b_n)
     32

/picb/rnomics3/wangmr/INSTALL/miniconda3/envs/GPU/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--&gt; 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

/picb/rnomics3/wangmr/INSTALL/miniconda3/envs/GPU/lib/python2.7/site-packages/torch/nn/modules/batchnorm.pyc in forward(self, input)
     58     @weak_script_method
     59     def forward(self, input):
---&gt; 60         self._check_input_dim(input)
     61
     62         exponential_average_factor = 0.0

/picb/rnomics3/wangmr/INSTALL/miniconda3/envs/GPU/lib/python2.7/site-packages/torch/nn/modules/batchnorm.pyc in _check_input_dim(self, input)
    167         if input.dim() != 2 and input.dim() != 3:
    168             raise ValueError('expected 2D or 3D input (got {}D input)'
--&gt; 169                              .format(input.dim()))
    170
    171

**ValueError: expected 2D or 3D input (got 1D input)**
</code></pre>
<p>The debug info shows maybe some problems on input data size.<br>
Indeed, the size of batch.text is [Length<em>32][32 = Batch Size] during training, but the size of the single new data is [Length</em>1]. I’ve tried to repeat the single data 32 times to match the batch size, but this really looks silly! How can I break the constrains of batch size, in an elegant way?</p>

Nikronic: <p>Hi,</p>
<p>When you have a single data, still it means you have a batch of data, but size of batch is 1. Note that in almost all math libraries, rows correspond to observations(samples) and columns correspond to features, so when you pass a single output with length of <code>L</code>, model considers it as <code>L</code> different samples with 1 feature. To solve this issue, just add another dimension to your input.</p>
<aside class="quote no-group" data-username="Mengran_Wang" data-post="1" data-topic="86086">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mengran_wang/48/24239_2.png" class="avatar"> Mengran_Wang:</div>
<blockquote>
<p><code>scores = model(preprocess_seq)</code></p>
</blockquote>
</aside>
<p>Can be replaced by:</p>
<pre><code class="lang-auto">scores = model(preprocess_seq.unsqeeuze(0))
</code></pre>
<p>Bests</p>

Mengran_Wang: <p>Thanks for your reply! But it seems doesn’t work. The reason is that the shape of my data followed the format of dataset loaded by Bucket Iterator.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4405e8081a102e74bb39b8f8196443d415a5322.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c4405e8081a102e74bb39b8f8196443d415a5322" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4405e8081a102e74bb39b8f8196443d415a5322_2_588x500.jpeg" alt="image" data-base62-sha1="s07uEKBaNAeeQQ1yrZYaXNKdnI6" width="588" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4405e8081a102e74bb39b8f8196443d415a5322_2_588x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4405e8081a102e74bb39b8f8196443d415a5322_2_882x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c4405e8081a102e74bb39b8f8196443d415a5322_2_1176x1000.jpeg 2x" data-dominant-color="131F28"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2016×1714 362 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As annotated in the above, a batch when training is tensor with shape [length_seq * batch number],here batch number equals to 32 ,however, an input for prediction is tensor with shape [length_seq * batch], here batch number equals to 1. The shapes between training and prediction are matched but the specific number are different.<br>
when I use preprocess_seq.unsqueeze(0), the shape is [1 * length_seq * 1], the dimension is not matched with the model input.</p>

chetan_patil: <p>Previously, the size was <code>[length_seq, 1]</code>.<br>
After, unsqueezing it became <code>[1,length_seq,1]</code>.<br>
You should use <code>scores = model(preprocess_seq.reshape(1,-1))</code><br>
This will make <code>preprocess_seq</code> of shape <code>[1, length_seq]</code> .</p>

Mengran_Wang: <p>Thanks for your suggestion! I tried and the output of <strong>model(preprocess_seq.reshape(1,-1))</strong> is <strong>[length_seq, 1]</strong>, The length_seq dim is wrongly recognized as batch dim. as a result, the model computes each character in my seq as a seperated seq. As shown here, <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d114d405bd117537f79f32f7e49fecc961ba08c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/d114d405bd117537f79f32f7e49fecc961ba08c8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d114d405bd117537f79f32f7e49fecc961ba08c8_2_690x424.png" alt="image" data-base62-sha1="tPCpLqpnfpU0LeZxreO382gP5cA" width="690" height="424" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d114d405bd117537f79f32f7e49fecc961ba08c8_2_690x424.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/1/d114d405bd117537f79f32f7e49fecc961ba08c8_2_1035x636.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/1/d114d405bd117537f79f32f7e49fecc961ba08c8.png 2x" data-dominant-color="132028"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1190×732 51.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
still waiting for solution <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

chetan_patil: <p>If previously, the <code>preprocess_seq</code> was of shape <code>[length_seq, 1]</code>,<br>
then <code>preprocess_seq.reshape(1,-1)</code> will make it of shape  <code>[1, length_seq]</code>.<br>
How come it changes back to <code>[lenth_seq,1]</code> ?</p>

Mengran_Wang: <p>Here I post the structure of the model</p>
<pre><code class="lang-auto">class RNNModel_GRU(nn.Module):
    def __init__(self, vocab_size, embedding_size, output_size, pad_idx, hidden_size, dropout):
        super(RNNModel_GRU, self).__init__()
        self.embed = nn.Embedding(vocab_size, embedding_size, padding_idx=pad_idx)
        self.gru = nn.GRU(embedding_size, hidden_size, bidirectional=True, num_layers=1)
        self.linear1 = nn.Linear(hidden_size*2, output_size[0])
        self.linear2 = nn.Linear(output_size[0], output_size[1])
        self.b_n = nn.BatchNorm1d(32)
        self.dropout = nn.Dropout(dropout)
        self.relu = nn.ReLU()
        #self.sigmoid = nn.Sigmoid()
        #self.softmax = nn.Softmax(dim=1)
  
    
    def forward(self, text):
        embedded = self.embed(text.long()) # [seq_len, batch_size, embedding_size]
        embedded  = self.dropout(embedded)
        output, hidden = self.gru(embedded)
        # output is [Seqlen * 32(batchSize) * 64(hiddenSize)]
        # hidden is [4 * 32 * 64]
        # cell is [4* 32 * 64]

        # hidden: 2 * batch_size * hidden_size
        hidden = torch.cat([hidden[0], hidden[1]], dim=1)
        # hidden: 32 * 128
        hidden = self.dropout(hidden.squeeze())
        linear1 = self.linear1(hidden)
        linear1 = self.relu(linear1)
        linear1 = self.dropout(linear1) # [32 * 32]
        b_n = self.b_n(linear1) # 32 * 32
        linear2 = self.linear2(b_n)

        return linear2 # linear2 shape: 32
</code></pre>
<p>the first step of my model is embedding, <strong>I think the shape after embeddding is [seq_len, batch_size, embedding_size], here is [1,14,1]. the original seq_len is recognized as batch_size after embedding layer.</strong></p>
<pre><code class="lang-auto">In [130]: embed(preprocess_seq.reshape(1,-1).long())
Out[130]:
tensor([[[ 0.4355],
         [-0.0817],
         [ 0.2178],
         [ 2.4970],
         [ 2.4970],
         [ 0.2178],
         [-0.0817],
         [ 0.4355],
         [ 0.2178],
         [ 0.2178],
         [-0.0817],
         [-0.0817],
         [-0.0817],
         [-0.0817]]], grad_fn=&lt;EmbeddingBackward&gt;)

In [131]: embed(preprocess_seq.reshape(1,-1).long()).size()
Out[131]: torch.Size([1, 14, 1])
</code></pre>

Andrew_OBrien: <p><a class="mention" href="/u/mengran_wang">@Mengran_Wang</a> Have you found a solution to this issue? I am facing the same problem and it looks like most of the answers here don’t directly give a solution.<br>
A workaround I am using for the moment is to feed my network with the single sample repeated batch_size amount of times… Surely there is a better way!?</p>

Krishna_Singh: <p><a class="mention" href="/u/andrew_obrien">@Andrew_OBrien</a> <a class="mention" href="/u/mengran_wang">@Mengran_Wang</a>  Did either of you find any solution for this?I too have been doing the same thing until now,that is.repeating the same sample batch_size number of times.</p>