Precision doesn’t improve when training on custom dataset

panovr: <SCODE>> creating model 'alexnet'
Epoch: [0][0/29]	Time 11.987 (11.987)	Data 10.121 (10.121)	Loss 6.9067 (6.9067)	Prec@1 0.391 (0.391)	Prec@5 0.781 (0.781)
Epoch: [0][10/29]	Time 0.336 (2.764)	Data 0.266 (2.488)	Loss 6.8902 (6.9003)	Prec@1 0.000 (0.178)	Prec@5 1.172 (1.598)
Epoch: [0][20/29]	Time 7.898 (2.771)	Data 7.827 (2.578)	Loss 6.8422 (6.8640)	Prec@1 0.000 (0.186)	Prec@5 1.953 (1.581)
Test: [0/10]	Time 4.700 (4.700)	Loss 6.7994 (6.7994)	Prec@1 0.000 (0.000)	Prec@5 3.125 (3.125)
 * Prec@1 0.333 Prec@5 1.625
Epoch: [1][0/29]	Time 2.927 (2.927)	Data 2.847 (2.847)	Loss 6.8089 (6.8089)	Prec@1 0.000 (0.000)	Prec@5 2.734 (2.734)
Epoch: [1][10/29]	Time 0.192 (0.822)	Data 0.025 (0.681)	Loss 6.7899 (6.7945)	Prec@1 0.391 (0.320)	Prec@5 1.953 (1.776)
Epoch: [1][20/29]	Time 2.253 (0.824)	Data 2.183 (0.689)	Loss 6.4336 (6.7144)	Prec@1 0.391 (0.316)	Prec@5 3.516 (1.730)
Test: [0/10]	Time 3.146 (3.146)	Loss 6.0892 (6.0892)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)
 * Prec@1 0.333 Prec@5 1.667
Epoch: [2][0/29]	Time 3.009 (3.009)	Data 2.920 (2.920)	Loss 6.0913 (6.0913)	Prec@1 0.391 (0.391)	Prec@5 1.953 (1.953)
Epoch: [2][10/29]	Time 0.189 (0.836)	Data 0.000 (0.681)	Loss 6.0209 (6.0952)	Prec@1 0.391 (0.320)	Prec@5 0.391 (1.562)
Epoch: [2][20/29]	Time 2.251 (0.822)	Data 2.181 (0.680)	Loss 5.9183 (6.0205)	Prec@1 0.000 (0.223)	Prec@5 0.781 (1.302)
Test: [0/10]	Time 3.046 (3.046)	Loss 5.9031 (5.9031)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)
 * Prec@1 0.333 Prec@5 1.667
Epoch: [46][0/29]	Time 2.996 (2.996)	Data 2.915 (2.915)	Loss 5.7088 (5.7088)	Prec@1 0.000 (0.000)	Prec@5 0.781 (0.781)
Epoch: [46][10/29]	Time 0.188 (0.844)	Data 0.000 (0.696)	Loss 5.7168 (5.7085)	Prec@1 0.000 (0.178)	Prec@5 1.562 (1.705)
Epoch: [46][20/29]	Time 2.090 (0.828)	Data 2.011 (0.685)	Loss 5.7267 (5.7122)	Prec@1 0.000 (0.205)	Prec@5 0.781 (1.562)
Test: [0/10]	Time 3.080 (3.080)	Loss 5.7117 (5.7117)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)
 * Prec@1 0.333 Prec@5 1.667
Epoch: [47][0/29]	Time 2.943 (2.943)	Data 2.852 (2.852)	Loss 5.7018 (5.7018)	Prec@1 0.781 (0.781)	Prec@5 3.125 (3.125)
Epoch: [47][10/29]	Time 0.196 (0.852)	Data 0.000 (0.701)	Loss 5.7113 (5.7091)	Prec@1 0.781 (0.355)	Prec@5 1.953 (1.953)
Epoch: [47][20/29]	Time 2.221 (0.838)	Data 2.136 (0.695)	Loss 5.7153 (5.7120)	Prec@1 0.000 (0.260)	Prec@5 1.562 (1.656)
Test: [0/10]	Time 3.071 (3.071)	Loss 5.7107 (5.7107)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)
 * Prec@1 0.333 Prec@5 1.667
Epoch: [48][0/29]	Time 3.054 (3.054)	Data 2.978 (2.978)	Loss 5.7045 (5.7045)	Prec@1 0.391 (0.391)	Prec@5 2.344 (2.344)
Epoch: [48][10/29]	Time 0.182 (0.837)	Data 0.000 (0.689)	Loss 5.7104 (5.7084)	Prec@1 0.391 (0.249)	Prec@5 2.734 (1.847)
Epoch: [48][20/29]	Time 1.824 (0.819)	Data 1.753 (0.700)	Loss 5.7171 (5.7120)	Prec@1 0.391 (0.242)	Prec@5 0.781 (1.488)
Test: [0/10]	Time 3.084 (3.084)	Loss 5.7100 (5.7100)	Prec@1 0.000 (0.000)	Prec@5 3.125 (3.125)
 * Prec@1 0.333 Prec@5 1.667
Epoch: [49][0/29]	Time 3.213 (3.213)	Data 3.137 (3.137)	Loss 5.7120 (5.7120)	Prec@1 0.781 (0.781)	Prec@5 1.172 (1.172)
Epoch: [49][10/29]	Time 0.182 (0.869)	Data 0.000 (0.713)	Loss 5.7154 (5.7094)	Prec@1 0.000 (0.426)	Prec@5 0.781 (1.456)
Epoch: [49][20/29]	Time 2.013 (0.829)	Data 1.931 (0.696)	Loss 5.7096 (5.7113)	Prec@1 0.781 (0.316)	Prec@5 2.734 (1.376)
Test: [0/10]	Time 3.072 (3.072)	Loss 5.7060 (5.7060)	Prec@1 0.000 (0.000)	Prec@5 0.000 (0.000)
 * Prec@1 0.333 Prec@5 1.667
<ECODE>

apaszke: Lack of convergence can be caused by a lot of things. The optimization is not guaranteed to succeed and actually it’s a bit of a miracle that NNs even work. We can’t help you a lot on that, but I think there are some problems with your dataset or transforms that are applied to it.

panovr: <SCODE>Epoch: [89][0/57]	Time 1.645 (1.645)	Data 1.525 (1.525)	Loss 0.3686 (0.3686)	Prec@1 92.969 (92.969)	Prec@5 97.656 (97.656)
Epoch: [89][10/57]	Time 0.364 (0.528)	Data 0.000 (0.244)	Loss 0.5915 (0.5153)	Prec@1 87.500 (87.713)	Prec@5 93.750 (95.455)
Epoch: [89][20/57]	Time 0.510 (0.471)	Data 0.379 (0.190)	Loss 0.6496 (0.5262)	Prec@1 85.938 (87.240)	Prec@5 92.969 (95.126)
Epoch: [89][30/57]	Time 0.355 (0.453)	Data 0.000 (0.171)	Loss 0.4592 (0.5180)	Prec@1 90.625 (87.903)	Prec@5 95.312 (95.186)
Epoch: [89][40/57]	Time 0.536 (0.446)	Data 0.413 (0.165)	Loss 0.3770 (0.5072)	Prec@1 92.188 (88.529)	Prec@5 97.656 (95.332)
Epoch: [89][50/57]	Time 0.369 (0.440)	Data 0.000 (0.166)	Loss 0.4453 (0.5025)	Prec@1 89.844 (88.664)	Prec@5 95.312 (95.374)
Test: [0/19]	Time 1.668 (1.668)	Loss 0.8600 (0.8600)	Prec@1 81.250 (81.250)	Prec@5 94.531 (94.531)
Test: [10/19]	Time 0.104 (0.463)	Loss 1.5666 (1.5452)	Prec@1 67.188 (67.827)	Prec@5 84.375 (84.659)
 * Prec@1 67.375 Prec@5 84.208
<ECODE> <SCODE>Epoch: [89][0/29]	Time 3.110 (3.110)	Data 3.040 (3.040)	Loss 4.7523 (4.7523)	Prec@1 5.469 (5.469)	Prec@5 19.922 (19.922)
Epoch: [89][10/29]	Time 0.189 (0.831)	Data 0.070 (0.700)	Loss 4.7577 (4.8041)	Prec@1 6.250 (5.611)	Prec@5 19.141 (17.685)
Epoch: [89][20/29]	Time 2.163 (0.831)	Data 2.079 (0.705)	Loss 4.8331 (4.8019)	Prec@1 4.688 (5.673)	Prec@5 19.531 (17.839)
Test: [0/10]	Time 3.048 (3.048)	Loss 4.6815 (4.6815)	Prec@1 8.203 (8.203)	Prec@5 23.047 (23.047)
 * Prec@1 7.458 Prec@5 22.833
<ECODE> You can see with alexnet, after 90 epoch, the precision@1 is 7.458; with resnet18, after 90 epoch, the precision@ is 67.375. So it is a bit strange to see that the precision of alexnet grows so slowly.

apaszke: It’s probably because the AlexNet doesn’t use BatchNorm and it stabilizes the training. I’d look into data normalization, you might be doing something wrong there.

haoyu: