Observing strange loss jumps between epochs

username: WilliamWithering
cooked: <p>Hi everyone.</p>
<p><strong>Here is my problem :</strong></p>
<p>I’ve been playing around with some neural networks lately, trying to compare different kinds of normalization layers on a simple problem (namely VGG16 on the CIFAR10 dataset).</p>
<p>However the loss graph for only one model is really noisy, so I decided to plot a mean of the losses for 5 runs of the same architecture. Let L_it be the loss for the i-th model on the minibatch at time t, each of my point p_t is computed by the following :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43f3b85abe37ffde49415f081661179355c17ffe.png" alt="mean" data-base62-sha1="9H89Ph3VvTBeEhOjE2lHKrDvISy" width="112" height="52"></p>
<p>I’m also plotting a rolling loss to smooth everything a bit, computed as follows :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a965c7b5bede03a36b75e8c16e31b095edc9e6e6.png" alt="Rolling" data-base62-sha1="oayKPL9ZIQOyiO0O38xDbfv7hau" width="145" height="82"></p>
<p>And here is the kind of result that I get :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/6/06337e898319a98c54c6d24415e36d803dbbe5c4.png" data-download-href="https://discuss.pytorch.org/uploads/default/06337e898319a98c54c6d24415e36d803dbbe5c4" title="Batch_Norm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06337e898319a98c54c6d24415e36d803dbbe5c4_2_667x500.png" alt="Batch_Norm" data-base62-sha1="SRbPw1cEnylTC8dGrrlx51bjDK" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06337e898319a98c54c6d24415e36d803dbbe5c4_2_667x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/6/06337e898319a98c54c6d24415e36d803dbbe5c4_2_1000x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/6/06337e898319a98c54c6d24415e36d803dbbe5c4.png 2x" data-dominant-color="E8F0F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Batch_Norm</span><span class="informations">1170×876 65.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The steps which are appearing between every epochs shouldn’t be there I guess… I talked to my teachers about it, they all agree that it isn’t normal, but none of them has been able to find the problem.</p>
<p><strong>Concerning the code I use to produce those results :</strong></p>
<p>I really tried to keep it as simple as possible, and it follows really closely the code provided in the Pytorch tutorial for CIFAR10.</p>
<p>My DataLoader is created with this code :</p>
<pre><code class="lang-auto">train_transform = transforms.Compose([transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize((.5, .5, .5), (.5, .5, .5))])

trainset = torchvision.datasets.CIFAR10(root = "../data_CIFAR10", train = True, download = False, transform = train_transform)
trainLoader = torch.utils.data.DataLoader(trainset, batch_size = bs, shuffle = True, drop_last = True)
</code></pre>
<p>My model, the criterion and the optimizer are here :</p>
<pre><code class="lang-auto">model = models.vgg16_bn(num_classes = 10)
model.to(gpu)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.003)
train_losses = train_model(model, trainLoader, criterion, optimizer, nb_epochs, gpu)
</code></pre>
<p>And finally, here is the `train_model function :</p>
<pre><code class="lang-auto">def train_model(model, trainLoader, criterion, optimizer, epochs, gpu):
    train_losses = []
    for epoch in range(epochs):
        for i, data in enumerate(trainLoader, 0):
            inputs, labels = data[0].to(gpu), data[1].to(gpu)

            optimizer.zero_grad()
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            train_losses.append(loss.item())

            if(i%100==99):
                print("[%d, %5d] loss : %.5f" % (epoch + 1, i + 1, loss.item()))

    print("Finished training\n\n")
    return train_losses

</code></pre>
<p>Did I miss anything obvious? I tried to describe my problem as precisely as possible (while keeping it simple), but if you need more information please ask.</p>
<p>Thanks.</p>
<p>(PS : what’s the best way to integrate LaTeX equations in a post?)</p>

username: KFrank
cooked: <p>Hi William!</p>
<aside class="quote no-group" data-username="WilliamWithering" data-post="1" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p><strong>Here is my problem :</strong></p>
<p>I’ve been playing around with some neural networks lately,</p>
</blockquote>
</aside>
<p>Well, there’s your problem!  (Just kidding.)</p>
<aside class="quote no-group">
<blockquote>
<p>I’m also plotting a rolling loss to smooth everything a bit, computed as follows :</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a965c7b5bede03a36b75e8c16e31b095edc9e6e6.png" alt="Rolling" data-base62-sha1="oayKPL9ZIQOyiO0O38xDbfv7hau" width="145" height="82"></p>
<p>And here is the kind of result that I get :</p>
<p>…</p>
<p>The steps which are appearing between every epochs shouldn’t be there I guess…</p>
</blockquote>
</aside>
<p>This does look odd.  I don’t see anything obviously wrong, but my<br>
guess is that you have a gremlin hiding in your code somewhere,<br>
and maybe it is in some part of the code you didn’t post.</p>
<p>But before I indulge in any idle speculation, let me point out that<br>
there is something very fishy about your graph.</p>
<p>The rolling loss at time (batch iteration) <code>t</code> and <code>t + 1</code> are related by<br>
<code>p'_{t+1} = p'_t + (p_{t+101} - p_t) / 100</code>.  The drop in the<br>
rolling loss in your graph at around epoch 15 is about 0.05.  Therefore<br>
the difference between <code>p_{t+101}</code> and <code>p_t</code> would have to be about<br>
5, that is, about twice the entire height of your graph!</p>
<p>(Even if your rolling loss is not averaged over five runs like your mean<br>
loss (blue) is, you can’t realistically have a <code>delta-p</code> of 5 get averaged<br>
down to something like the worst case of perhaps 0.3 that could be<br>
hiding in the envelope of your blue graph.)</p>
<p>I could be misinterpreting your graph, or the numbers could be off,<br>
but either way, I can’t really conjecture much at this point.</p>
<p>(As a side note, as written, your formula for your rolling loss sums<br>
over 101 values of <code>p_i</code>, but divides by 100 – not that this is relevant<br>
to the issue at hand.)</p>
<aside class="quote no-group">
<blockquote>
<p>(PS : what’s the best way to integrate LaTeX equations in a post?)</p>
</blockquote>
</aside>
<p>My preference is not to integrate LaTex into this kind of forum.<br>
Whether it’s some kind of embedded LaTeX (e.g., MathJax) or<br>
an image generated from LaTeX, it’s harder to copy and paste<br>
and to search on.  (But that’s just me …)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: WilliamWithering
cooked: <p>Hi K. Frank !</p>
<p>First of all, thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>The rolling loss at time (batch iteration) <code>t</code> and <code>t + 1</code> are related by <code>p'_{t+1} = p'_t + (p_{t+101} - p_t) /100</code> . The drop in the rolling loss in your graph at around epoch 15 is about 0.05. Therefore the difference between <code>p_{t+101}</code> and <code>p_t</code> would have to be about 5, that is, about twice the entire height of your graph!</p>
</blockquote>
</aside>
<p>Your observation is really interesting, and I thought it would be the solution to my problem. However, I zoomed in on the graph, and the drop doesn’t occur exactly between <code>p_t</code> and <code>p_{t+1}</code>, but over the span of a hundred of batch iterations (I’ve got ~46K points in the entire graph, hence the impression of a sudden drop).</p>
<p>Here is a zoom on the drop at the 15th epoch :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36bee0ebedf9b01fecea92408eee915e6ba61ece.png" data-download-href="https://discuss.pytorch.org/uploads/default/36bee0ebedf9b01fecea92408eee915e6ba61ece" title="Rolling_loss%20drop"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bee0ebedf9b01fecea92408eee915e6ba61ece_2_671x500.png" alt="Rolling_loss%20drop" data-base62-sha1="7OiJOp0u4MGLXmQ0d0TD3mX0ABo" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bee0ebedf9b01fecea92408eee915e6ba61ece_2_671x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bee0ebedf9b01fecea92408eee915e6ba61ece_2_1006x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36bee0ebedf9b01fecea92408eee915e6ba61ece.png 2x" data-dominant-color="FCFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Rolling_loss%20drop</span><span class="informations">1177×876 18.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t think that the rolling loss is the problem, because for me it seems that the drops are visible even in the mean loss (from the 6th epoch to the 14th for example). Just in case, this is how I compute it (<code>train_bn</code> being my list of averaged losses) :</p>
<p><code>rolling = [np.mean(train_bn[i:i+100]) for i in range(len(train_bn) - 100)]</code></p>
<aside class="quote no-group quote-modified" data-username="KFrank" data-post="2" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>My guess is that you have a gremlin hiding in your code somewhere, and maybe it is in some part of the code you didn’t post.</p>
</blockquote>
</aside>
<p>Concerning the part of the code I didn’t post, here it is : <a href="https://pastebin.com/RFNLuLQG" class="inline-onebox" rel="noopener nofollow ugc">Dropping rolling loss - Pastebin.com</a></p>
<p>And just in case, here is how I create the graph : <a href="https://pastebin.com/zmT5efJq" class="inline-onebox" rel="noopener nofollow ugc">Graph - Pastebin.com</a></p>
<p>William</p>

username: KFrank
cooked: <p>Hello William!</p>
<aside class="quote no-group" data-username="WilliamWithering" data-post="3" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p>I don’t think that the rolling loss is the problem, because for me it seems that the drops are visible even in the mean loss (from the 6th epoch to the 14th for example). Just in case, this is how I compute it (<code>train_bn</code> being my list of averaged losses) :</p>
</blockquote>
</aside>
<p>Something still seems inconsistent with your graphs.</p>
<p>Let me explain what my concerns are.  I can’t really back this<br>
up with numbers because I can’t see the details of the blue<br>
data in your first graph – it’s all blurred together.  All I can<br>
really see is the envelope of the blue data.</p>
<p>So two concerns:</p>
<p>In your second graph – the rolling-average loss where I can see<br>
the individual points – over the range of about 100 batches where<br>
the rolling loss decreases nearly steadily, it looks like the next<br>
point is lower than the previous about 80% or 90% of the time.</p>
<p>This means (from my discussion in my first post) that the relevant<br>
<code>p_{t+100}</code> has to be lower than <code>p_t</code> about 80% or 90% of the<br>
time.  But the blue envelope in your first graph makes it look to me<br>
like the <code>p_t</code>s are very noisy, jumping back and forth.  I just can’t<br>
imagine a realistic correlation that would cause such noisy data<br>
to systematically satisfy <code>p_{t+100} &lt; p_t</code> (over the relevant<br>
range).</p>
<p>The second also has to do with the noise in the <code>p_t</code> that I deduce<br>
from the blue envelope.  My understanding is that the blue data<br>
in your first graph comes from averaging over five runs.  But let’s<br>
say, just to make the point more extreme, that it’s a single run.<br>
The spread in the data (which should be proportional to the<br>
standard deviation) is about 0.3.  Averaging over 100 such values<br>
should reduce the spread down by a factor of sqrt (100) = 10, so<br>
down to about 0.03.</p>
<p>If I look at the “noise” wiggles in the orange curve in your first<br>
graph – that is, not the sawtooth shape that tracks the epochs,<br>
but rather the short-time fluctuations on top of the sawtooth –<br>
(and, similarly, at the point-to-point, non-trend fluctuations in<br>
your second graph), the noise looks rather smaller than 0.03.</p>
<p>That is, your rolling loss looks less noisy I would expect from<br>
what the blue envelope seems to be telling me about the noise<br>
in the unaveraged <code>p_t</code>s.</p>
<p>Again, it’s hard for me to make a clear-cut case about this, since I<br>
can’t see the specific <code>p_t</code> values inside the blue envelope, and it’s<br>
hard to prove the negative that there aren’t any weird correlations<br>
that would cause the noise to average away unexpectedly rapidly.</p>
<p>If we take your rolling-loss graph at face value, it suggests that<br>
averaging over 100 <code>p_t</code> values averages away enough noise<br>
to make the effect clearly visible.  Would it be possible to run 100<br>
independent training runs (instead of 5), and average those runs<br>
to make a graph of the (averaged over runs) <code>p_t</code>s, but without<br>
any rolling-loss average?</p>
<p>The rolling-loss average isn’t an illegitimate way to try to dig<br>
some signal out of the noise, but it does complicate the analysis<br>
somewhat (and the results do look fishy to me).  It’s not that I don’t<br>
see the effect in the blue envelope – it’s just that rolling-loss graph<br>
noise doesn’t agree with the simplest batch-to-batch fluctuations<br>
suggested by the blue envelope.</p>
<p>If you do 100 runs, could you plot not only the whole (averaged)<br>
training run, but also a couple of epochs and epoch transition<br>
regions where we can see individual points in the graph?</p>
<aside class="quote no-group">
<blockquote>
<p><code>rolling = [np.mean(train_bn[i:i+100]) for i in range(len(train_bn) - 100)]</code></p>
</blockquote>
</aside>
<p>As an aside, this code for the rolling average looks correct to<br>
me (with the quibble that in your training loop you build the list<br>
<code>train_losses</code>, but here you calculate the rolling-loss from<br>
something called <code>train_bn</code>).</p>
<p>Also, based on assuming that you use the 50,000-sample CIFAR-10<br>
training dataset for your training, I estimate that you batch size is<br>
about 32, and that you have about 1,500 batches per epoch.  Are<br>
these values approximately correct?</p>
<p>Now for some idle speculation:  It’s does look like something is<br>
happening at epoch boundaries, but I can’t think of a straightforward<br>
explanation that fits.</p>
<p>It appears that you use a shuffling dataloader to create your<br>
batches, so this explanation doesn’t apply.  But suppose you didn’t<br>
shuffle, and you simply generated batches in the original order<br>
of the CIFAR data.  (I don’t know what that order is, by the way.)<br>
If the CIFAR had easier samples towards the beginning, and<br>
harder samples toward the end (perhaps ordered by class, and<br>
easier classes happened to come first), we could easily see your<br>
effect.</p>
<p>But you do shuffle.  Now, within an epoch, all the batches contain<br>
different samples, so as you train, you don’t get the benefit of<br>
seeing again a sample you’ve already trained on.  But across<br>
an epoch boundary, the first batches you see could <em>happen to<br>
include</em> samples that were seen recently in the last few batches<br>
of the previous epoch.  This could give your network the advantage<br>
of being asked to classify a sample that it had recent trained on<br>
(from a batch in the previous epoch), thereby having an easier<br>
task and a lower a loss.</p>
<p>However, with a batch size of 32, I just don’t see this happening<br>
enough to produce a dramatic (or even noticeable) effect.</p>
<p>Nonetheless, if this <em>were</em> the case, you could do the following<br>
test.  If you created batches by drawing samples <em>with<br>
replacement</em> (<a class="mention" href="/u/ptrblck">@ptrblck</a> – is there a pre-packaged sampling-<br>
with-replacement dataloader, or a simple strategy to write one?)<br>
then there would be no actual epoch boundaries – an “epoch”<br>
would be a number of training samples equal to the size of<br>
your training set, but some samples would occur (randomly)<br>
more than once in an “epoch,” and some not at all.</p>
<p>If the sawtooth-loss structure persisted with this test, it would<br>
indicate clearly a weird bug somewhere.</p>
<p>Again, idle speculation.  There is something going on at your<br>
epoch boundaries, but it’s hard to guess what it might be with<br>
the data partially hidden.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="KFrank" data-post="4" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>is their a pre-packaged sampling-<br>
with-replacement dataloader, or a simple strategy to write one?</p>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler">RandomSampler</a> with <code>replacement=True</code> should probably do the job, if I understand your suggestion correctly.<br>
Side note: the <code>DataLoader</code> creates this sampler by default, if <code>shuffle=True</code> is specified, but uses <code>replacement=False</code>.</p>

username: KFrank
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler" rel="noopener nofollow ugc">RandomSampler</a> with <code>replacement=True</code> should probably do the job, if I understand your suggestion correctly.</p>
</blockquote>
</aside>
<p>Yes, that looks likes what I had in mind for my suggestion to William.</p>
<p>Thanks.</p>
<p>K. Frank</p>

username: WilliamWithering
cooked: <p>Hi Frank!</p>
<p>Once again, thank you for your reply.</p>
<p>First of all, here is a zoom on the first graph around the drop at epoch 9 so that you can have a better view of the individual points (the 400th mini batch here is the batch 0 of epoch 9):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/5807046072c6451836e7d1d9f975f029601f506f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5807046072c6451836e7d1d9f975f029601f506f" title="RollingZoom"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5807046072c6451836e7d1d9f975f029601f506f_2_667x500.png" alt="RollingZoom" data-base62-sha1="cyJ3bOjHzXyhEngUITGTx0Sldbp" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5807046072c6451836e7d1d9f975f029601f506f_2_667x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/5807046072c6451836e7d1d9f975f029601f506f_2_1000x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/5807046072c6451836e7d1d9f975f029601f506f.png 2x" data-dominant-color="DDE9F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RollingZoom</span><span class="informations">1170×876 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="quote no-group quote-modified" data-username="KFrank" data-post="4" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>This means (…) that the relevant <code>p_{t+100}</code> has to be lower than <code>p_t</code> about 80% or 90% of the time.</p>
</blockquote>
</aside>
<p>Obviously it’s just an example, but to me it seems that over the span of the drop, the most important part of the <code>p_{t+100}</code>s are indeed lower than the corresponding <code>p_t</code>s.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Averaging over 100 such values should reduce the spread down by a factor of sqrt (100) = 10, so down to about 0.03.</p>
</blockquote>
</aside>
<p>Concerning the noise : I don’t know if it really helps, but I computed the std for the mini-batches 0 to 300 (as numbered in the above graph, using <code>np.std</code>), and the result is ~0.07. The std of the rolling loss over the same mini-batches is ~0.009.</p>
<p>The same computation for the mini batches 400 to 800 yields ~0.07 and ~0.005. So it seems that averaging over 100 values does indeed reduce the spread by a factor of ~10?</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Would it be possible to run 100 independent training runs (instead of 5), and average those runs to make a graph of the (averaged over runs) <code>p_t</code> s, but without any rolling-loss average?</p>
</blockquote>
</aside>
<p>Training 100 models would be quite long on my computer, but I can at least try to get more than 5. Here are some <em>temporary</em> results with 25 runs (more are training as I’m writing these lines, I’ll provide more graphs when I’ll have more data). If it’s not exactly the kind of graph you wanted, please tell me.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f17adc65913b04587a45607aa9301ac763ea4a37.png" data-download-href="https://discuss.pytorch.org/uploads/default/f17adc65913b04587a45607aa9301ac763ea4a37" title="Average25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f17adc65913b04587a45607aa9301ac763ea4a37_2_667x500.png" alt="Average25" data-base62-sha1="yseiwbwvjUzJi1BRy1aRwWDfpn9" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f17adc65913b04587a45607aa9301ac763ea4a37_2_667x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f17adc65913b04587a45607aa9301ac763ea4a37_2_1000x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f17adc65913b04587a45607aa9301ac763ea4a37.png 2x" data-dominant-color="F3F6F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Average25</span><span class="informations">1170×876 58.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/75810e9a09a1b2d645922f794cae26176f03a102.png" data-download-href="https://discuss.pytorch.org/uploads/default/75810e9a09a1b2d645922f794cae26176f03a102" title="Drop25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75810e9a09a1b2d645922f794cae26176f03a102_2_671x500.png" alt="Drop25" data-base62-sha1="gLumMyE8eQ0IUsSn3zkswv5B5M6" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75810e9a09a1b2d645922f794cae26176f03a102_2_671x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/75810e9a09a1b2d645922f794cae26176f03a102_2_1006x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/5/75810e9a09a1b2d645922f794cae26176f03a102.png 2x" data-dominant-color="E1EDF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Drop25</span><span class="informations">1177×876 195 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b70e0823939e80101112b1295116bef7bfef80bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/b70e0823939e80101112b1295116bef7bfef80bf" title="Zoom2Epochs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70e0823939e80101112b1295116bef7bfef80bf_2_671x500.png" alt="Zoom2Epochs" data-base62-sha1="q7nrhCW5crl2AFPu5DYEjcPMmLl" width="671" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70e0823939e80101112b1295116bef7bfef80bf_2_671x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b70e0823939e80101112b1295116bef7bfef80bf_2_1006x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/7/b70e0823939e80101112b1295116bef7bfef80bf.png 2x" data-dominant-color="BAD5E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Zoom2Epochs</span><span class="informations">1177×876 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="64066">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are these values approximately correct?</p>
</blockquote>
</aside>
<p>Yes they are. I’m using a batch-size of 32, and there are precisely 1562 batches per epoch (all of them being of size 32 because of the <code>drop_last</code> parameter in the train loader, otherwise the 1563rd batch would be of size 16).</p>
<p>I see your point concerning the fact that the network could have an advantage on recently presented samples. I will try using the RandomSampler as soon as my computer is finished with all the runs for the graph you previously asked for.</p>
<p>Thanks,</p>
<p>William</p>

username: KFrank
cooked: <p>Hello William!</p>
<aside class="quote no-group quote-modified" data-username="WilliamWithering" data-post="7" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p>First of all, here is a zoom on the first graph around the drop at epoch 9 so that you can have a better view of the individual points (the 400th mini batch here is the batch 0 of epoch 9):</p>
<p>…</p>
<p>Training 100 models would be quite long on my computer, but I can at least try to get more than 5. Here are some <em>temporary</em> results with 25 runs (more are training as I’m writing these lines, I’ll provide more graphs when I’ll have more data). If it’s not exactly the kind of graph you wanted, please tell me.</p>
</blockquote>
</aside>
<p>Thanks for the zoomed-in and 25-run graphs.</p>
<ol>
<li>
<p>To my eye, they further demonstrate the epoch-boundary-drop<br>
you showed in your first post.</p>
</li>
<li>
<p>Looking at these, I no longer see evidence for any discrepancy<br>
between the rolling-average and single-batch results.  (My intuition<br>
about how they should be related was clearly off.)</p>
</li>
</ol>
<p>Don’t kill yourself trying to get 100 runs if it’s too slow and painful.<br>
It would be nice, but not necessarily worth it if it’s a lot of trouble.</p>
<aside class="quote no-group">
<blockquote>
<p>I see your point concerning the fact that the network could have an advantage on recently presented samples. I will try using the RandomSampler as soon as my computer is finished with all the runs for the graph you previously asked for.</p>
</blockquote>
</aside>
<p>There is clearly something going on at the epoch boundaries.<br>
Unless there’s just a bug somewhere, the <em>only</em> thing that happens<br>
at the epoch boundaries is the change in the character of the<br>
batches (relative to recent batches in the previous epoch).</p>
<p>I’m leaning towards believing my story about the new epoch having<br>
some samples be recent repeats.  (It’s really all I can think of.)  I’m<br>
somewhat surprised that the effect could be so large, but I don’t<br>
really have any intuition about the CIFAR dataset or the VGG16<br>
network.</p>
<p>I do believe that if you create batches with random drawing with<br>
replacement, there should be <em>no</em> change in batch character at<br>
the epoch boundaries.  (In fact, there won’t really be any epoch<br>
boundaries except those from “arbitrarily” counting the number<br>
of batches.)  So the effect should vanish entirely.</p>
<p>If it doesn’t, I would say look for a bug somewhere.  But you have<br>
a very simple nested loop where you loop over epochs and then<br>
loop over batches, so I just don’t see where an epoch-boundary<br>
bug could sneak in …</p>
<p>Anyway, I think your code is sound (in this respect – I’m not saying<br>
there couldn’t be other little nits here and there …).  I think the test<br>
with batch-sampling-with-replacement is the thing to do.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>This is probably unrelated, but additionally to the suggested experiment, you could also try to drop the last batch, if it is smaller than the rest by using <code>drop_last=True</code> in your <code>DataLoader</code>.</p>
<p>If the last batch is much smaller than the preceding ones, the running estimates might get a “bad update” , which could yield some unexpected loss values in the beginning of the new epoch.</p>
<p>I doubt this effect is that large, but might be a quick experiment.</p>

username: WilliamWithering
cooked: <p>Hi !</p>
<p>I’m trying to train the network using a <code>RandomSampler</code> as suggested. However, if I understood correctly, it only yields a single index at a time, so I have to wrap it with a <code>BatchSampler</code> (?)</p>
<p>I tried something like this :</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import BatchSampler, RandomSampler

def prep_data(bs, epochs):
    train_transform = transforms.Compose([transforms.RandomHorizontalFlip(),
                                          transforms.ToTensor(),
                                          transforms.Normalize((.5, .5, .5), (.5, .5, .5))])

    trainset = torchvision.datasets.CIFAR10(root = "../data_CIFAR10", train = True, download = True, transform = train_transform)
    trainSampler = BatchSampler(RandomSampler(trainset, replacement = True, num_samples = epochs * len(trainset)), batch_size = bs, drop_last = True)
    trainloader = torch.utils.data.DataLoader(trainset, sampler = trainSampler)

    return trainloader
</code></pre>
<p>When I test it, I get a TypeError (the following code is enough to reproduce the error).</p>
<pre><code class="lang-auto">trainloader = prep_data(32, 1)
for i, data in enumerate(trainloader):
    print(i)
</code></pre>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-1-5c625c535dea&gt; in &lt;module&gt;()
     22 
     23 trainloader = prep_data(32, 1)
---&gt; 24 for i, data in enumerate(trainloader):
     25     print(i)

/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--&gt; 346         data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self.pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/anaconda3/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in &lt;listcomp&gt;(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/cifar.py in __getitem__(self, index)
    116         #    tuple: (image, target) where target is index of the target class.
    117         
--&gt; 118         img, target = self.data[index], self.targets[index]
    119 
    120         # doing this so that it is consistent with all other datasets

TypeError: list indices must be integers or slices, not list
</code></pre>
<p>I tried to get a little bit further, and it seems that <code>self._next_index()</code> returns an array of shape (1, 32) instead of (32) (so the list comprehension doesn’t separate the indexes as expected by the dataset). I got it to run by replacing it with <code>self._next_index()[0]</code>…</p>
<p>I had never used samplers before, so I’m not really sure of what I’m doing here. Am I doing well in my <code>prep_data</code> function? And my solution to avoid the error is quite of a messy workaround, so is there a better way to handle it?</p>
<p>Thanks,</p>
<p>William</p>

username: KFrank
cooked: <p>Hi William!</p>
<aside class="quote no-group" data-username="WilliamWithering" data-post="10" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p>I’m trying to train the network using a <code>RandomSampler</code> as suggested. However, if I understood correctly, it only yields a single index at a time, so I have to wrap it with a <code>BatchSampler</code> (?)</p>
</blockquote>
</aside>
<p>I’m not sure, but this seems plausible.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    trainSampler = BatchSampler(RandomSampler(trainset, replacement = True, num_samples = epochs * len(trainset)), batch_size = bs, drop_last = True)
    trainloader = torch.utils.data.DataLoader(trainset, sampler = trainSampler)
</code></pre>
<p>When I test it, I get a TypeError</p>
<p>…</p>
<p>I had never used samplers before, so I’m not really sure of what I’m doing here.</p>
</blockquote>
</aside>
<p>I haven’t used samplers either, so I’m not certain of what is<br>
going on.</p>
<p>However, looking at the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">DataLoader</a> documentation, it appears<br>
that the <code>DataLoader()</code> constructor has two mutually-exclusive<br>
arguments,  <code>sampler</code> and <code>batch_sampler</code>.  Your code is using<br>
<code>sampler</code>.</p>
<p>Is it possible that in order to use <code>trainSampler</code>, the<br>
<code>BatchSampler</code> you’ve constructed, you have to pass it<br>
in as <code>batch_sampler = trainSampler</code>?</p>
<p>Just a thought …</p>
<p>Best.</p>
<p>K. Frank</p>

username: WilliamWithering
cooked: <p>Hi Frank !</p>
<p>It worked ! With the replacement method there are no more drops. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thank you so much.</p>
<p>Concerning your 100 runs graph, I’ll stop at 50 (it takes too much time). There isn’t much difference with 25, just a little bit less noise and more defined drops.</p>
<p>So it seems that the problem was coming from the DataLoader. I’d still like to know why, as I think that my first approach should have worked… I did a quick test, following the <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">Pytorch tutorial</a> (I slightly modified it to get batches of size 32, more epochs, and all the losses) I trained 10 small models. The drops are already visible.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61.png" data-download-href="https://discuss.pytorch.org/uploads/default/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61" title="Unknown"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61_2_681x500.png" alt="Unknown" data-base62-sha1="6YLCURtP79NXOduAQcoHO7eX30J" width="681" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61_2_681x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61_2_1021x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30eb923ea0761b2acd2cc0ce0da29bb1fe232f61.png 2x" data-dominant-color="DFEAF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Unknown</span><span class="informations">1173×860 66.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any known issue with the DataLoader or the Sampler? Maybe a statistical bias or something like that?</p>
<p>Thanks,</p>
<p>William</p>

username: KFrank
cooked: <p>Hi William!</p>
<aside class="quote no-group" data-username="WilliamWithering" data-post="12" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p>It worked ! With the replacement method there are no more drops.</p>
</blockquote>
</aside>
<p>To me, this is strong evidence that there isn’t some weird bug<br>
in your code causing the epoch-boundary drops.</p>
<p>I’m very much believing the theory that the drops are caused<br>
by the fact that the new batches near the start of a new epoch<br>
can contain samples that were recently seen in batches near<br>
the end of the previous epoch.</p>
<aside class="quote no-group">
<blockquote>
<p>Concerning your 100 runs graph, I’ll stop at 50 (it takes too much time). There isn’t much difference with 25, just a little bit less noise and more defined drops.</p>
</blockquote>
</aside>
<p>No worries.  There’s nothing magic about 100 runs (or even 50).<br>
As long as you have enough runs to be able to see through the<br>
noise, that’s enough.</p>
<aside class="quote no-group">
<blockquote>
<p>So it seems that the problem was coming from the DataLoader. I’d still like to know why, as I think that my first approach should have worked…</p>
<p>…</p>
<p>Is there any known issue with the DataLoader or the Sampler? Maybe a statistical bias or something like that?</p>
</blockquote>
</aside>
<p>I’m inclined to doubt that there is anything wrong with the<br>
DataLoader (other than that it’s doing what it was designed<br>
to do – which may or may not be what you want).</p>
<p>The DataLoader gives you an entire epoch of samples, one batch<br>
at a time, without any repeats.  When you go on to a new epoch,<br>
you sometimes will get repeats of recently seen samples.  You<br>
would expect this to give the network an advantage, and lead to<br>
a lower loss.  The only question is whether this effect would be<br>
large enough to be significant and/or noticeable.  (I suppose<br>
you could call this a “statistical bias” of a sort, but it is what the<br>
DataLoader is supposed to do.)</p>
<p>(I think this theory could be tested, but at the cost of writing a<br>
custom Sampler.)</p>
<p>If it’s easy, I would be interested in seeing a head-to-head<br>
comparison of graphs (averaged over the same number of runs,<br>
etc.), where the only difference is using the shuffled, no-repeat<br>
DataLoader vs. the drawing-with-replacement RandomSampler.</p>
<p>Best.</p>
<p>K. Frank</p>

username: WilliamWithering
cooked: <p>Hi Frank !</p>
<p>I see your point concerning the DataLoader, and I’d really like to test your theory. However my final exams are approaching, and I’m afraid I won’t have enough time…</p>
<p>Concerning the comparison of graphs, I made one comparing the rolling averages for both methods (averaging on 5 runs for each) :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c313c816a412b55f149e11444753e078756ff456.png" data-download-href="https://discuss.pytorch.org/uploads/default/c313c816a412b55f149e11444753e078756ff456" title="Comparison"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c313c816a412b55f149e11444753e078756ff456_2_677x500.png" alt="Comparison" data-base62-sha1="rPJuidxu6uKFOOMaJFaanKipvz8" width="677" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c313c816a412b55f149e11444753e078756ff456_2_677x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c313c816a412b55f149e11444753e078756ff456_2_1015x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/3/c313c816a412b55f149e11444753e078756ff456.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Comparison</span><span class="informations">1170×864 50.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks,</p>
<p>William</p>

username: KFrank
cooked: <p>Hello William!</p>
<aside class="quote no-group" data-username="WilliamWithering" data-post="14" data-topic="64066" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/e8c25b/48.png" class="avatar"> WilliamWithering:</div>
<blockquote>
<p>Concerning the comparison of graphs, I made one comparing the rolling averages for both methods (averaging on 5 runs for each) :</p>
</blockquote>
</aside>
<p>Thanks for posting the comparison graph.  It shows the effect<br>
you discovered very clearly.  To my eye, I see absolutely nothing<br>
going on at the epoch boundaries in the blue RandomSampler<br>
curve (in contrast to the strong DataLoader effect).</p>
<p>Best.</p>
<p>K. Frank</p>

username: jxmorris12
cooked: <p>Hi everyone, this happened to me too. Random sampling with replacement amortizes the gap over multiple steps, so the steps get smoothed out.</p>
<p>But the real problem was that the model was aggressively overfitting. This meant that the loss for an example was almost directly proportional to the number of times the model had seen that example. So to fix the step function, I had to reduce the overfitting (in my case, by adding regularization between two large linear layers). That was the real problem (not the sampling issue).</p>