AdaBoost CNN Weights Update

bryan123: <p>Hi there,</p>
<p>AdaBoost reduces the weight of training samples that are properly classified for each iteration while maintaining the weight of all incorrectly classified data so that the training samples that have been classified correctly have less influence than the not correctly classified training samples. Thus, the model in the next iteration is more adapted to the previously misclassified samples.</p>
<p>Would you please tell me what to do once I have the weights for each data set? Say I have 5 data with the following weights: [0.2, 0.1, 0.2, 0.4, 0.1]. Do I need to change the loss function?</p>
<p>Thank you in advance!</p>

ptrblck: <p>I’m not sure which <code>AdaBoost</code> implementation you are using but I assume you are manually calculating the weights and would then like to apply them to all samples.<br>
In this case, you could calculate the unreduced loss via setting <code>reduction='none'</code> to instantiate the loss function, apply the corresponding sample weights to each sample (make sure you are not losing the correspondence via shuffling), and reduce the loss before calculating the gradients in the backward pass.</p>

bryan123: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your response. Could you give me a piece of code on how to implement it? What confuses me is that in the documentation, weight should be applied to each class rather than to each data. Thank you once again!</p>

ptrblck: <p>I was thinking about multiplying the loss with the weights as seen here:</p>
<pre><code class="lang-python">output = torch.randn(10, 10, requires_grad=True)
target = torch.randint(0, 10, (10,))

criterion = nn.CrossEntropyLoss(reduction='none')

loss = criterion(output, target)
weight = torch.rand(10) # grab your weights for the current samples

loss = loss * weight
loss.mean().backward()
</code></pre>

bryan123: <p>Thanks! so based on your example, I assume the total number of data is 10, right? What if the total amount of data exceeds the batch size?</p>

ptrblck: <p>Yes, 10 is the batch size in my example.<br>
You would need to map the weights to each sample as described before.<br>
I don’t know how you are creating these weights, how your dataset looks like etc., but you could use e.g. the indices for each sample returned by a custom <code>Dataset</code> to index the weights for the current sample, or you could also create as new custom <code>Dataset</code> after creating the weights, which would return the data, target, and weights etc.</p>

bryan123: <p>Actually, I’d like to implement CNN’s incremental learning system based on Adaboost in medical images. Thus, all of the weights are based on the Adaboost technique.</p>
<p>I have already picked weights based on the number of batch sizes for each training loop. However, I discovered the following error when (loss = loss * weight) at the end of the loop.</p>
<p>RuntimeError: The size of tensor a (32) must match the size of tensor b (15) at non-singleton dimension 0</p>
<p>It seems the dimension of the weight (15) is not the same as loss (32).</p>

ptrblck: <p>Check the shapes of the <code>loss</code> and <code>weight</code> tensor and make sure you can multiply both. Based on the error it seems you are using a batch size of 32 on the model output and thus <code>loss</code> while <code>weight</code> contains only 15 values.</p>

bryan123: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, that problem has already been solved. May I know why it should be a loss.mean() before backward()? because I often add all of the loss for all of the batches in one epoch and then calculate the mean, not directly calculating the loss mean in one batch.</p>
<p>Thanks!</p>

ptrblck: <p>The <code>mean</code> reduction would be the common approach, but you are of course free to reduce the loss according to your use case.</p>

mMagmer: <p>hi <a class="mention" href="/u/bryan123">@bryan123</a> , boosting is a additive model for minimizing a loss function.<br>
minimizing loss for CNN model is basically the same  thing and it changes weight for each sample automatically.(lower loss for sample that are classified correctly )<br>
read <a href="https://dl.acm.org/doi/abs/10.1145/2939672.2939785" rel="noopener nofollow ugc">xgboost</a> paper for more insight.</p>

bryan123: <p>hi <a class="mention" href="/u/mmagmer">@mMagmer</a>, could you show me how to implement weights for each sample in a piece of code to minimize loss for the CNN model?</p>