Developing a demo classifier

username: blade
cooked: <p>I’m writing a demo code to predict the 2-class label for a dataset of 10-D inputs. Below, function <code> _data</code> generates the data:</p>
<pre><code class="lang-auto">import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

def _data(dimension, num_examples):
    # Create a simulated 10-dimensional training dataset consisting of 1000 labeled
    # examples, of which 800 are labeled correctly and 200 are mislabeled.
    num_mislabeled_examples = 20
    # We will constrain the recall to be at least 90%.
    recall_lower_bound = 0.9

    # Create random "ground truth" parameters for a linear model.
    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    # Generate a random set of features for each example.
    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    # Compute the labels from these features given the ground truth linear model.
    labels = (np.matmul(features, ground_truth_weights) &gt;
              ground_truth_threshold).astype(np.float32)
    # Add noise by randomly flipping num_mislabeled_examples labels.
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

</code></pre>
<p>The code below shows my attempt, where <code>predictor</code> is the model and the loss function is chosen to be Hinge loss.</p>
<pre><code class="lang-auto">import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

dim = 10
N = 100
target, features = _data(dim, N)

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.f_1 = nn.Linear(10, 1)

    def forward(self, features):
        return self.f_1(features)

model = predictor()

optimizer = optim.Adam(model.parameters(), lr=1e-2)
loss = torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

running_loss = 0
for _ in range(1000):
    optimizer.zero_grad()
    output = model(features)
    objective = loss(output, target)
    objective.backward()
    running_loss += objective.item()
    optimizer.step()

    print(running_loss)
</code></pre>
<p><strong>My questions:</strong></p>
<ol>
<li>I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.</li>
<li>I was trying to implement my predictor without using <code>nn.Linear</code> by defining the computations myself as:</li>
</ol>
<pre><code class="lang-auto">class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)

    def forward(self, features):
        return torch.matmul(self.weights, features) - self.threshold
</code></pre>
<p>but in the optimization process,</p>
<pre><code class="lang-auto">model = predictor()
optimizer = optim.Adam(model.parameters(), lr=1e-3) 
</code></pre>
<p>I get the following error:</p>
<blockquote>
<p>ValueError: optimizer got an empty parameter list</p>
</blockquote>
<p>I would appreciate direction or comments on how to fix these issues. Thanks.</p>

username: harsha_g
cooked: <aside class="quote no-group" data-username="blade" data-post="1" data-topic="86160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<pre><code class="lang-auto">self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)
</code></pre>
</blockquote>
</aside>
<pre><code class="lang-auto">self.weights = torch.nn.Parameter(torch.zeros(dim, dim, requires_grad=True))
self.threshold = torch.nn.Parameter(torch.zeros(1, 1, requires_grad=True))
</code></pre>
<p>should do.</p>

username: harsha_g
cooked: <aside class="quote no-group" data-username="blade" data-post="1" data-topic="86160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blade/48/23861_2.png" class="avatar"> blade:</div>
<blockquote>
<p>I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.</p>
</blockquote>
</aside>
<p>Notice that you are doing <code>running_loss += objective.item()</code> which is essentially adding up the loss from the previous epochs as well. Every iteration (in your code) constitutes an epoch because you are sending in all the features at once. Does it make sense?</p>
<p>Something like:</p>
<pre><code class="lang-auto">running_loss = []
for _ in range(1000):
    ...
    running_loss.append(objective.item())
    ...
print(running_loss)
</code></pre>
<p>should do.</p>

username: blade
cooked: <p>So without adding up, my objective.item() starts from around 0.5 and still goes negative. I was expecting the loss not to fall below zero (Although a <a href="https://discuss.pytorch.org/t/hinge-loss-in-pytorch/86220">self implemented version</a> of this loss function works just fine).</p>

username: harsha_g
cooked: <p>Could it be because  <code>target</code>  has  <code>0</code> s and  <code>1</code> s in it unlike what the documentation suggests, that it should contain  <code>1</code> s and  <code>-1</code> s. Is the behavior when you add this line  <code>target[target == 0] = -1</code>  along expected lines?</p>