Inconsistent prediction result in CNN

sanipanwala: <p>Hello,</p>
<p>I have done training using text classification CNN. I’m getting one issue during prediction.</p>
<p>When I ran prediction with my trained model using 3 sentiments every time I got different prediction results.</p>
<pre><code class="lang-auto">    prediction = model(processed_input)
    probabilities = F.softmax(prediction, dim=1)
    probabilities = probabilities.detach().cpu().numpy()
</code></pre>
<p>Here is the code that predicts the result. Let me know if anything needs to be changed.</p>
<p>Thanks,<br>
Sani</p>

ptrblck: <p>Maybe your model uses random operations, such as <code>nn.Dropout</code> layers in its forward, which you should disable during inference? Call <code>model.eval()</code> and check if this helps getting the same predictions.</p>

sanipanwala: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thanks for your response.</p>
<p>I have already used <code>model.eval()</code> for prediction but got different results.</p>
<p>Thanks,<br>
Sani</p>

ptrblck: <p>Could you post the model definition so that I could try to reproduce it, please?</p>

sanipanwala: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Here is my model definition. Please guide me for better results.</p>
<pre><code class="lang-auto">class CharacterLevelCNN(nn.Module):
    def __init__(self, args, number_of_classes):
        super(CharacterLevelCNN, self).__init__()

        # define conv layers
	# RLJ changed to 1d due to runtime warnings
        self.dropout_input = nn.Dropout1d(args.dropout_input)
        
        #RLJ introduced the large feature size via arg modelfeaturesize
        convlayerfeaturesize = 256
        fclayerfeaturesize=1024
        if args.modelfeaturesize=="large":
            convlayerfeaturesize=1024
            fclayerfeaturesize=2048
        print("Conv Layer Feature Size {}".format(convlayerfeaturesize))
        print("FC Layer Feature Size {}".format(fclayerfeaturesize))
        
        self.conv1 = nn.Sequential(
            nn.Conv1d(
                args.number_of_characters + len(args.extra_characters),
                convlayerfeaturesize,
                kernel_size=7,
                padding=0,
            ),
            nn.ReLU(),
            nn.MaxPool1d(3),
        )

        self.conv2 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=7, padding=0), nn.ReLU(), nn.MaxPool1d(3)
        )

        self.conv3 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv4 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv5 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv6 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU(), nn.MaxPool1d(3)
        )

        # compute the  output shape after forwarding an input to the conv layers

        input_shape = (
            args.batch_size,
            args.max_length,
            args.number_of_characters + len(args.extra_characters),
        )
        print("FC Input Shape {}".format(input_shape))
        self.output_dimension = self._get_conv_output(input_shape)
        print("FC Input Size {}".format(self.output_dimension))

        # define linear layers

        self.fc1 = nn.Sequential(
            nn.Linear(self.output_dimension, fclayerfeaturesize), nn.ReLU(), nn.Dropout(0.5)
        )

        self.fc2 = nn.Sequential(nn.Linear(fclayerfeaturesize, fclayerfeaturesize), nn.ReLU(), nn.Dropout(0.5))

        self.fc3 = nn.Linear(fclayerfeaturesize, number_of_classes)

        # initialize weights
        #RLJ changed from default values of 0.0 and 0.05 to the mean and std per the Yan Lecun paper based on whether it is a large or small model
        #self._create_weights()

        if args.modelfeaturesize=="small":
            self._create_weights(0, 0.05)
        else:
            self._create_weights(0, 0.02)

    # utility private functions

    def _create_weights(self, mean=0.0, std=0.05):
        for module in self.modules():
            if isinstance(module, nn.Conv1d) or isinstance(module, nn.Linear):
                module.weight.data.normal_(mean, std)

    def _get_conv_output(self, shape):
        x = torch.rand(shape)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = x.view(x.size(0), -1)
        output_dimension = x.size(1)
        return output_dimension

    # forward

    def forward(self, x):
        x = self.dropout_input(x)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x
		
		
================================
In the Above logic I have passed the below parameter:
1. dropout= 0.5 
2. model size = small
3. batch_size = 16
4. max_length = 10000
5. number_of_characters = 69
6. extra_characters = ""
</code></pre>
<p>Thanks,<br>
Sani</p>

ptrblck: <p>Thanks for the code.<br>
I cannot reproduce the issue and get the same outputs after calling <code>model.eval()</code>:</p>
<pre><code class="lang-python">class CharacterLevelCNN(nn.Module):
    def __init__(self, number_of_classes):
        super(CharacterLevelCNN, self).__init__()

        # define conv layers
	# RLJ changed to 1d due to runtime warnings
        self.dropout_input = nn.Dropout1d(0.5)
        
        #RLJ introduced the large feature size via arg modelfeaturesize
        convlayerfeaturesize = 256
        fclayerfeaturesize=1024
        modelfeaturesize = "small"
        if modelfeaturesize=="large":
            convlayerfeaturesize=1024
            fclayerfeaturesize=2048
        print("Conv Layer Feature Size {}".format(convlayerfeaturesize))
        print("FC Layer Feature Size {}".format(fclayerfeaturesize))
        
        number_of_characters = 69
        self.conv1 = nn.Sequential(
            nn.Conv1d(
                number_of_characters + len(""),
                convlayerfeaturesize,
                kernel_size=7,
                padding=0,
            ),
            nn.ReLU(),
            nn.MaxPool1d(3),
        )

        self.conv2 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=7, padding=0), nn.ReLU(), nn.MaxPool1d(3)
        )

        self.conv3 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv4 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv5 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU()
        )

        self.conv6 = nn.Sequential(
            nn.Conv1d(convlayerfeaturesize, convlayerfeaturesize, kernel_size=3, padding=0), nn.ReLU(), nn.MaxPool1d(3)
        )

        # compute the  output shape after forwarding an input to the conv layers

        input_shape = (
            16,
            10000,
            number_of_characters + len(""),
        )
        print("FC Input Shape {}".format(input_shape))
        self.output_dimension = self._get_conv_output(input_shape)
        print("FC Input Size {}".format(self.output_dimension))

        # define linear layers

        self.fc1 = nn.Sequential(
            nn.Linear(self.output_dimension, fclayerfeaturesize), nn.ReLU(), nn.Dropout(0.5)
        )

        self.fc2 = nn.Sequential(nn.Linear(fclayerfeaturesize, fclayerfeaturesize), nn.ReLU(), nn.Dropout(0.5))

        self.fc3 = nn.Linear(fclayerfeaturesize, number_of_classes)

        # initialize weights
        #RLJ changed from default values of 0.0 and 0.05 to the mean and std per the Yan Lecun paper based on whether it is a large or small model
        #self._create_weights()

        if modelfeaturesize=="small":
            self._create_weights(0, 0.05)
        else:
            self._create_weights(0, 0.02)

    # utility private functions

    def _create_weights(self, mean=0.0, std=0.05):
        for module in self.modules():
            if isinstance(module, nn.Conv1d) or isinstance(module, nn.Linear):
                module.weight.data.normal_(mean, std)

    def _get_conv_output(self, shape):
        x = torch.rand(shape)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = x.view(x.size(0), -1)
        output_dimension = x.size(1)
        return output_dimension

    # forward

    def forward(self, x):
        x = self.dropout_input(x)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        x = self.conv6(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x
    
model = CharacterLevelCNN(10)
x = torch.randn(16, 10000, 69)

# all are different since dropout is still active
for _ in range(10):
    out = model(x)
    print(out.double().abs().sum())
# tensor(19179.5103, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(19980.1256, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(19489.9541, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(21702.8869, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(19419.2389, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(19746.8942, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(17694.0387, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(19148.0841, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(18543.3931, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(18682.2872, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)    


# now all outputs are equal
model.eval()
for _ in range(10):
    out = model(x)
    print(out.double().abs().sum())
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
# tensor(5934.1699, dtype=torch.float64, grad_fn=&lt;SumBackward0&gt;)
</code></pre>

sanipanwala: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thank you for your response.</p>
<p>I’m still not getting good results when I’m training with a few strings as well as a few random CSV data.<br>
Below is an example of training data and testing data.</p>
<p>MY TRAINING DATA</p>
<pre><code class="lang-auto">sentiment,text
"1","[a]"
"0","[b]"
"1","[c]"
"0","[d]"
"1","[e]"
"0","[f]"
"1","[g]"
"0","[h]"
"1","[i]"
</code></pre>
<p>MY TESTING DATA</p>
<pre><code class="lang-auto">sentiment,text
"0","[A]"
"1","[B]"
"1","[C]"
"0","[D]"
</code></pre>
<pre><code class="lang-auto">tensor(0.0494, device='cuda:0', dtype=torch.float64)
tensor(0.0485, device='cuda:0', dtype=torch.float64)
tensor(0.0474, device='cuda:0', dtype=torch.float64)
tensor(0.0488, device='cuda:0', dtype=torch.float64)
tensor(0.0492, device='cuda:0', dtype=torch.float64)
tensor(0.0484, device='cuda:0', dtype=torch.float64)
tensor(0.0469, device='cuda:0', dtype=torch.float64)
</code></pre>
<p>Thanks,<br>
Sani</p>

ptrblck: <aside class="quote no-group" data-username="sanipanwala" data-post="7" data-topic="169496">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8491ac/48.png" class="avatar"> sanipanwala:</div>
<blockquote>
<p>I’m still not getting good results</p>
</blockquote>
</aside>
<p>I’m unsure what “good” results mean in this context as you’ve described to see “inconsistent” predictions (i.e. different or random outputs) even after calling <code>model.eval()</code>.<br>
Are you still getting these inconsistent results using your own minimal code snippet?</p>

sanipanwala: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Sorry for the confusion.</p>
<p>Your code snippet is providing perfect results but when I provide CSV data as I mentioned above during training I’m getting inconsistent results yet.</p>
<p>Thanks,<br>
Sani</p>