tensorboardX SummaryWriter not working when using gpu

ianben6688: <p>pytorch: 1.1.0<br>
tensorboardX: 2.1</p>
<p>the code is like following:</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.optim import adam
from tensorboardX import SummaryWriter

device = "cuda" if torch.cuda.is_available() else "cpu"

net = Model()
net.to(device)
loss_fn = nn.BCELoss()  # MSELoss()
optimizer = adam.Adam(params=net.parameters(), lr=0.0001, weight_decay=0.5)

writer = SummaryWriter("logs")
for epoch in range(50):
    for i, x_batch, y_batch in enumerate(train_loader):
        y_pred = net(x_batch.to(device))
        loss = loss_fn(y_pred, x_batch)

        writer.add_scalar("loss/train", loss, global_step=epoch * len(train_loader) + i)
        writer.flush()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

writer.close()
</code></pre>
<p>I want to see the loss change during the training rather than after the training.</p>
<p>Firstly I trained the model in Titan-xp. When I check the event file during training, the size is 0.  In the tensorboard webpage, I got ‘No scalar data was found’. When I terminate the training, the event becomes 100K and I can see the loss in the tensorboard webpage.</p>
<p>Then I trainined the model in cpu, and everything works fine. Can anyone tell me what the problem it is?</p>

pfloat: <p>Hello,<br>
I have met such problem before. In my case, <code>writer.flush()</code> solved the problem but sometimes, eventhough several <code>add_scalar</code> have been done, it’s not possible to visualize the result.</p>
<p>Could the problem be in the <code>loss</code> argument in <code>add_scalar</code> since it should be a <code>torch.tensor</code>?<br>
Maybe using <code>loss.item()</code> solves your issue ?</p>
<p>Cheers</p>

ianben6688: <p>Thank you. I tried, it still doesn’t work.</p>

pfloat: <p>How about the <code>max_queue</code> argument set to 1?</p>
<pre><code class="lang-auto">writer = SummaryWriter("logs", maxqueue=1)
</code></pre>
<p><a href="https://tensorboardx.readthedocs.io/en/latest/tensorboard.html" rel="nofollow noopener">https://tensorboardx.readthedocs.io/en/latest/tensorboard.html</a></p>
<p>The default argument is 10 so the writer should at least log 5 times to disk… that’s weird anyway</p>
<p>EDIT: just found out the <code>flush_secs=120</code> default argument in the writer documentation</p>

ianben6688: <aside class="quote no-group" data-username="pfloat" data-post="4" data-topic="95500">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pfloat/48/28529_2.png" class="avatar"> pfloat:</div>
<blockquote>
<p><code>maxqueue=1</code></p>
</blockquote>
</aside>
<p>Thank you. I just tried. Still not work.</p>

pfloat: <p>this ?<br>
<code>writer = SummaryWriter("logs", maxqueue=1, flush_secs=1)</code></p>

ianben6688: <p>Yes, I change it to:</p>
<p><code>writer = SummaryWriter(check_point_dir, flush_secs=1, write_to_disk=True, max_queue=1)</code></p>
<p>Still doesn’t work. When I terminate the training or train it on cpu, the summary can be written to disk. So wired.</p>

ianben6688: <p>I think the summary data is in the memory. Just cannot be written to disk.</p>

pfloat: <p>well, I think we cannot do much about it for now to make this work. It seems that there are a lot of people with the same problems online. See:<br>
<a href="https://github.com/lanpa/tensorboardX/pull/451" rel="nofollow noopener">https://github.com/lanpa/tensorboardX/pull/451</a></p>

ianben6688: <p>Thanks for your patient reply. I changed <code>from tensorboardX import SummaryWriter</code> to <code>from torch.utils.tensorboard import SummaryWriter</code> and it works. That’s so wired.</p>
<p>Thank you so much.</p>