MultiLabel Problem

username: logarith
cooked: <p>Hi,</p>
<p>I’m trying to solve a multi label problem<br>
I have a tensor of around 400*2000 values<br>
The 2000 are zeros and ones, but the vectors have in average only 10 of the 2000 values with a one the others are zeros.<br>
A one should have more importance than the zeros.<br>
I standardize the values with a mean square algorithm.<br>
So this is my first question. Is this good in that case?</p>
<p>I also have output tensors with the size of 60 classes, which are not mutual exclusive. There are always 10 classes one and the others zero.</p>
<p>This is my network</p>
<pre><code class="lang-auto">network = torch.nn.Sequential(
            torch.nn.Linear(len(self.getVector()), 250),
            torch.nn.ReLU(),
            torch.nn.Linear(250, 150),
            torch.nn.ReLU(),
            torch.nn.Linear(150, 60),            
        )

       loss_function = torch.nn.MultiLabelSoftMarginLoss()

       optimizer = torch.optim.Adam(network.parameters(), lr=0.0007)
       network.train()
       for i in range(500):
            predicted_value = network(test_input_tensor)
            loss = loss_function(predicted_value, test_output_tensor)
            print(i, loss.item())
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()


        network.eval()
        output = network(prognostic_input_tensor)
</code></pre>
<p>As I have not much experience in machine learning, I want to know<br>
if you have some advice, if this is a good approach for a multi label probelm with the<br>
features mentioned above?<br>
It seems to me that it predicts a lot of negative values, what I dont understand.</p>

username: KFrank
cooked: <p>Hi Log!</p>
<aside class="quote no-group quote-modified" data-username="logarith" data-post="1" data-topic="105173" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9bcb4/48.png" class="avatar"> logarith:</div>
<blockquote>
<p>I also have output tensors with the size of 60 classes, which are not mutual exclusive. There are always 10 classes one and the others zero.</p>
<pre><code class="lang-auto">network = torch.nn.Sequential(
            ...
            torch.nn.Linear(150, 60),            
        )

       loss_function = torch.nn.MultiLabelSoftMarginLoss()
</code></pre>
<p>It seems to me that it predicts a lot of negative values, what I dont understand.</p>
</blockquote>
</aside>
<p>Because the output of your model is the output of your last <code>Linear</code><br>
layer, you are predicting raw-score <em>logits.</em>  A logit value that is less<br>
than zero corresponds to a predicted probability less than one half.<br>
Typically a probability of less than one half for the “1” state when<br>
interpreted as a hard “yes-no” prediction would be taken to be a<br>
“0”-state prediction (and greater than one half would be the “1” state).</p>
<p>Many more of your “output-tensor” <code>target</code> values are <code>0</code>’s than are<br>
<code>1</code>’s, so if you weight each individual <code>target</code> value equally in loss<br>
function, your model can train to do a good job on the loss function<br>
by preferentially predicting <code>1</code>’s (that is, predicting negative logits),<br>
regardless of the input data.</p>
<p>The common approach to addressing this is to weight your<br>
less-frequent <code>1</code> <code>target</code> values more heavily in your loss function.</p>
<p>Note that <a href="https://pytorch.org/docs/1.7.0/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> is essentially the same as<br>
<code>MultiLabelSoftMarginLoss</code> but has a <code>pos_weight</code> argument<br>
that you can pass to its constructor.</p>
<p>You say that you have 60 classes, and that any given sample <code>target</code><br>
has 10 classes in the <code>1</code> state and 50 in the <code>0</code> state.  If all of your<br>
classes are about equally likely to be in the <code>1</code> state, you could use<br>
the same <code>pos_weight</code> for all of them.  A reasonable value would be<br>
<code>pos_weight = n_negative / n_positive</code>.  So:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_function = torch.nn.BCEWithLogitsLoss (pos_weight = torch.tensor ([5.0]))
</code></pre>
<p>If the likelihood of your different classes having <code>target</code> value <code>1</code> are<br>
not all broadly similar, then you would pass in a tensor of length 60<br>
for your <code>pos_weight</code>, that is, a different <code>pos_weight</code> value for each<br>
class.</p>
<p>Best.</p>
<p>K. Frank</p>

username: logarith
cooked: <p>thanks for the answer,<br>
but for the input tensors, do I have to normalize them before passing them to the network,<br>
or can I input tensors consisting of ones and zeros?</p>

username: KFrank
cooked: <p>Hi Log!</p>
<aside class="quote no-group" data-username="logarith" data-post="3" data-topic="105173" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9bcb4/48.png" class="avatar"> logarith:</div>
<blockquote>
<p>do I have to normalize them before passing them to the network,<br>
or can I input tensors consisting of ones and zeros?</p>
</blockquote>
</aside>
<p>Passing in the “raw” tensors should be fine.  Being ones and zeros,<br>
they are already close to being normalized.  Changing them to, say,<br>
<code>-1</code> and <code>1</code> so (if they were fifty-fifty) they would have a mean of 0 and<br>
a standard deviation of 1 wouldn’t affect things much.  (Try it both<br>
ways – I doubt you’ll see any difference.)</p>
<p>(In contrast, think about a 16-bit grayscale image as input to a<br>
network.  The pixel values run from zero to about 65,000, so they<br>
can be rather large.  Normalizing the pixel values so that they are<br>
of order one makes like easier for the network.)</p>
<p>Best.</p>
<p>K. Frank</p>