How to calculate per-class-accuracy for each batch?

username: Kalle
cooked: <p>I have two questions that are related. If I have a unbalanced dataset, how do I calculate the per-class-accuracy properly for each batch? And if I add class weight, should I then assume the dataset is balanced and calculate the accuracy the usual way?</p>
<p>In the following code snip I assume that the dataset is balanced:</p>
<pre><code class="lang-auto">for local_batch, local_labels in training_generator:
	# Transfer the data to GPU/CPU
	local_batch, local_labels = local_batch.to(device), local_labels.to(device)
	
	# Initialize gradients, zero the parameter gradients
	optimizer.zero_grad()

	# forward pass: compute predicted outputs by passing inputs to the model
	predictions = model(local_batch)
	
	# calculate training accuracy
	predicted_labels = predictions.argmax(dim=1, keepdim=True).squeeze()
	num_train_correct = (predicted_labels == local_labels).sum().item()
	train_accuracy = num_train_correct / local_batch.size(0)
	...
</code></pre>
<ol>
<li>What if my dataset is unbalanced? How do I then calculate the per-class-accuracy for each batch while training?</li>
</ol>
<pre><code class="lang-auto">	# calculate training accuracy
	predicted_labels = predictions.argmax(dim=1, keepdim=True).squeeze()
	num_train_correct = ????
	train_accuracy = ????
	...
</code></pre>
<ol start="2">
<li>What if I add <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731">class weight</a> to CrossEntropyLoss. Should I then assume the dataset is balanced and calculate the accuracy as below or is there a third way to calculate accuracy when class weight is applied?</li>
</ol>
<pre><code class="lang-auto">	# calculate training accuracy
	predicted_labels = predictions.argmax(dim=1, keepdim=True).squeeze()
	num_train_correct = (predicted_labels == local_labels).sum().item()
	train_accuracy = num_train_correct / local_batch.size(0)
	...
</code></pre>

username: mMagmer
cooked: <p>If you are initializing weight for Cross Entropy with proportion to  1 over class prior <code>(1/p_i)</code> for each class, then you’re minimizing average recall over all class.<br>
and  accuracy is good estimation of average recall if you have plenty of data. If not, you should calculate average recall.<br>
You can use <a href="https://pytorch.org/ignite/generated/ignite.metrics.confusion_matrix.ConfusionMatrix.html" rel="noopener nofollow ugc">ignite</a> confusion matrix to calculate any classification metric you want.</p>

username: Kalle
cooked: <p>Hi <a class="mention" href="/u/mmagmer">@mMagmer</a>, so how would you calculate it in each batch? Could you give an example?</p>
<pre><code class="lang-auto">for epoch in n_epochs:
	for local_batch, local_labels in training_generator:
		# Transfer the data to GPU/CPU
		local_batch, local_labels = local_batch.to(device), local_labels.to(device)
		
		# Initialize gradients, zero the parameter gradients
		optimizer.zero_grad()

		# forward pass: compute predicted outputs by passing inputs to the model
		predictions = model(local_batch)
		
		# calculate training accuracy
		predicted_labels = predictions.argmax(dim=1, keepdim=True).squeeze()
		num_train_correct = ????
		train_accuracy = ????
	...```</code></pre>

username: mMagmer
cooked: <p>suppose <code>cm</code> is your confusion matrix, then</p>
<pre><code class="lang-auto">cm.fill(0); # you can zero it every 10 batch for better estimation.

for i , j in zip(local_labels,predicted_labels):
            cm[i,j] += 1

recall = cm.diagonal() / (cm.sum(axis=1) + 1e-15)
AvgRecall = recall.mean()
</code></pre>

username: Kalle
cooked: <p>Ah yes that makes sense <a class="mention" href="/u/mmagmer">@mMagmer</a>. Thanks.<br>
But why does it give better estimation to zero it after 10 batch? Won’t the batch after it is set to zero be a bad estimation then?</p>
<p>And a second question:</p>
<aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="139052">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>If you are initializing weight for Cross Entropy with proportion to 1 over class prior <code>(1/p_i)</code> for each class, then you’re minimizing average recall over all class.<br>
and accuracy is good estimation of average recall if you have plenty of data. If not, you should calculate average recall.</p>
</blockquote>
</aside>
<p>I initialize it like <a href="https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html" rel="noopener nofollow ugc">this</a> and set balance as parameter. Should I then consider the dataset balanced?</p>

username: mMagmer
cooked: <ol>
<li>more sample you have, better the estimation. And yes, last batch is better off regarding error in recall estimation, but there is no way around.</li>
<li>it really depends on the distribution of the data, but you’re probably closer to being in balance with minimizing weighted Cross Entropy.</li>
</ol>
<p>In my experience, it is good solution for unbalance ratio lower than 10.<br>
Also using weight decay in optimizer helps (and other regularization technique)</p>