Issue with evaluators on binary dataset at higher epoch iterations

katyapanf: <p>Hi.</p>
<p>I am new to PyTorch (am learning it in Google Colab) and am having an issue I can’t resolve. I am working with the BreastMNIST data set and am processing it as binary data.</p>
<p>All worked fine until I reworked it to use Ignite for the metrics. On a handful of epoch iterations all works ok. However, when I increase the epochs to 10+ the data training works fine but the evaluation throws a “<strong>For binary cases, y_pred must be comprised of 0’s and 1’s</strong>” error.</p>
<p>Any thoughts? I’m particularly confused given the exact same dataset is handled just fine when running with fewer epochs.</p>
<p>Thanks.</p>
<details>
<summary>
Error</summary>
<pre><code class="lang-auto">ERROR:ignite.engine.engine.Engine:Current run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's.
ERROR:ignite.engine.engine.Engine:Engine run is terminating due to exception: For binary cases, y_pred must be comprised of 0's and 1's.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-27-950c571740af&gt; in &lt;cell line: 1&gt;()
----&gt; 1 train_evaluator.run(train_loader_at_eval)
      2 metrics = train_evaluator.state.metrics
      3 print(f"Training Eval Results: AUC: {metrics['auc']:.2f}   Accuracy: {metrics['accuracy']:.2f}")
      4 
      5 val_evaluator.run(val_loader)

/usr/local/lib/python3.10/dist-packages/ignite/metrics/accuracy.py in _check_binary_multilabel_cases(self, output)
     56 
     57         if not torch.equal(y_pred, y_pred**2):
---&gt; 58             raise ValueError("For binary cases, y_pred must be comprised of 0's and 1's.")
     59 
     60     def _check_type(self, output: Sequence[torch.Tensor]) -&gt; None:

ValueError: For binary cases, y_pred must be comprised of 0's and 1's
</code></pre>
</details>

vfdev-5: <p>Hi <a class="mention" href="/u/katyapanf">@katyapanf</a></p>
<blockquote>
<p>However, when I increase the epochs to 10+ the data training works fine but the evaluation throws a “<strong>For binary cases, y_pred must be comprised of 0’s and 1’s</strong>” error.</p>
</blockquote>
<p>Accuracy, Precision, Recall metrics for binary case require binarized predictions:</p>
<pre data-code-wrap="python"><code class="lang-python">def binarize_preds(output):
    y_preds, y_true = output
    return y_preds &gt; 0.5, y_true


precision = Precision(average=False, output_transform=binarize_preds)
recall = Recall(average=False, output_transform=binarize_preds)
accuracy = Accuracy(output_transform=binarize_preds)
</code></pre>
<blockquote>
<p>Any thoughts? I’m particularly confused given the exact same dataset is handled just fine when running with fewer epochs.</p>
</blockquote>
<p>In your original code, evaluation part is running in the end of the training only. Usually, we may want to run model evaluation during the training, for example, every epoch ended. Using ignite, this can be done like this:</p>
<pre data-code-wrap="python"><code class="lang-python">trainer = Engine(training_step)
train_evaluator = Engine(eval_step)
val_evaluator = Engine(eval_step)

# Schedule when to run model validation:
@trainer.on(Events.EPOCH_COMPLETED)
def run_validation():
    def log_metrics(epoch, tag, metrics):
        metrics_output = "\t".join([f"\t{k}: {v}" for k, v in metrics.items()])
        print(f"Epoch[{epoch}] - {tag} metrics: {metrics_output}")

    train_evaluator.run(train_loader_at_eval)
    metrics = train_evaluator.state.metrics
    log_metrics(trainer.state.epoch, "Training", metrics)

    val_evaluator.run(val_loader)
    metrics = val_evaluator.state.metrics
    log_metrics(trainer.state.epoch, "Validation", metrics)

trainer.run(train_loader, max_epochs=NUM_EPOCHS)
</code></pre>
<p>(For a better understanding of how events system is working, you can try this <a href="https://pytorch-ignite.ai/playground" rel="noopener nofollow ugc">playground</a>)</p>
<p>Here is a full training code with few updates in colab:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1eEGj9auO-yemp2g-HagjSfnJIWdcni27?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/53f6608ac7f752e07fbba8c73cc6de30/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1eEGj9auO-yemp2g-HagjSfnJIWdcni27?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1eEGj9auO-yemp2g-HagjSfnJIWdcni27?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Hope this helps!</p>

katyapanf: <aside class="quote no-group" data-username="vfdev-5" data-post="2" data-topic="201825">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vfdev-5/48/3535_2.png" class="avatar"> vfdev-5:</div>
<blockquote>
<p><code>binarize</code></p>
</blockquote>
</aside>
<p>That did the trick, thank you!!</p>