Encountered strange overfitting during training, with `num_worker > 0`, when shuffling is off

melisande-c: <p>Hi,</p>
<h2><a name="p-459833-overview-1" class="anchor" href="#p-459833-overview-1"></a>Overview</h2>
<p>Recently I have encountered some strange overfitting when setting <code>num_workers &gt; 0</code>. It turned out we accidentally had <code>shuffle=False</code>, but our models were still training fine when <code>num_workers=0</code>. What’s strange is when we set <code>num_workers &gt; 0</code>, the models started to overfit – training loss was going down, but validation loss started to go up after a while. We solved it by setting <code>shuffle=True</code>; but I would still be interested in getting some insight into why this happened, because I find it strange that no shuffling worked fine, but then possibly a little bit of shuffling or data duplication introduced by the distributed training caused overfitting, but then fully shuffling the data caused it to train normally again.</p>
<h2><a name="p-459833-some-background-2" class="anchor" href="#p-459833-some-background-2"></a>Some background</h2>
<p>We are training an algorithm called Noise2Void, which is a self-supervised method for denoising images that have uncorrelated noise. The basic idea is that it cannot learn the uncorrelated noise, but it can learn underlying structures present in an image. This means if we train using L2 loss it should learn to predict the expected mean of a pixel given the structure contained in some surrounding pixels.</p>
<p>This is implemented as part of our Python library CAREamics, that aims to make this algorithm (N2V) and others more accessible to the scientific community.</p>
<h2><a name="p-459833-example-result-using-dataloader-parameters-num_workers4-and-shufflefalse-3" class="anchor" href="#p-459833-example-result-using-dataloader-parameters-num_workers4-and-shufflefalse-3"></a>Example result using <code>DataLoader</code> parameters <code>num_workers=4</code> and <code>shuffle=False</code></h2>
<p>(Bad result)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a92b412f4e5991ec0528b44bea73d1764d1f8838.png" data-download-href="https://discuss.pytorch.org/uploads/default/a92b412f4e5991ec0528b44bea73d1764d1f8838" title="381082826-4b7037eb-ea2a-429f-bb44-64657f1e12c0"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a92b412f4e5991ec0528b44bea73d1764d1f8838.png" alt="381082826-4b7037eb-ea2a-429f-bb44-64657f1e12c0" data-base62-sha1="o8xmDw7kQ2Z6hEwjOrkQvRpq3KE" width="481" height="321"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">381082826-4b7037eb-ea2a-429f-bb44-64657f1e12c0</span><span class="informations">481×321 94.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h2><a name="p-459833-side-note-4" class="anchor" href="#p-459833-side-note-4"></a>Side note</h2>
<p>We are using PyTorch Lightning and we accidentally didn’t have <code>shuffle=True</code> because of a misunderstanding that PyTorch Lightning automatically applies shuffling to the train dataloader by using a <code>torch.utils.data.DistributedSampler</code>. After some further investigation it turned out that the data is in fact not shuffled during training unless <code>shuffle=True</code> is explicitly set in the <code>train_dataloader</code>.</p>
<p>I haven’t brought this up in the Lightning forums because I don’t believe it is a result of lightning, but if it turns out it is, I will bring it up there.</p>
<h2><a name="p-459833-more-notes-5" class="anchor" href="#p-459833-more-notes-5"></a>More notes</h2>
<p>This is also raised as an issue on our GitHub repo where you can see good training results examples, and also figures plotting the train loss and validation loss.</p><aside class="onebox githubissue" data-onebox-src="https://github.com/CAREamics/careamics/issues/258">
  <header class="source">

      <a href="https://github.com/CAREamics/careamics/issues/258" target="_blank" rel="noopener nofollow ugc">github.com/CAREamics/careamics</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue" data-github-private-repo="false">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/CAREamics/careamics/issues/258" target="_blank" rel="noopener nofollow ugc">Passing dataloader `num_workers` param causes bad results w/o `shuffle=True`</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-10-24" data-time="13:02:50" data-timezone="UTC">01:02PM - 24 Oct 24 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/melisande-c" target="_blank" rel="noopener nofollow ugc">
          <img alt="melisande-c" src="https://avatars.githubusercontent.com/u/63270704?v=4" class="onebox-avatar-inline" width="20" height="20">
          melisande-c
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          bug
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">When passing parameters to the dataloader in the `TrainDataModule` it may preven<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">t the dataloader from shuffling the data. A fix is to explicitly pass `shuffle=True`. After some further investigation an issue should likely be raised on Pytorch Lightning.

Examples initialising the `TrainDataModule` that will give bad results because the shuffling is prevented somehow.

```python
config.data_config.dataloader_params = {"num_workers": 4}
train_data_module = TrainDataModule(
    config.data_config, train_data=train_path, val_data=val_path
)
```

or 

```python
train_data_module = create_train_datamodule(
    train_data=train_path, 
    val_data= val_path, 
    data_type="tiff",
    patch_size=(64, 64),
    axes="SYX",
    batch_size=16,
    dataloader_params={"num_workers": 4},
)
```
To specify the number of workers and have shuffling use `dataloader_params={"num_workers": 4, "shuffle"=True}`</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thanks!</p>

traincheck-team: <p>Hi, thanks for the report. The problem looks very interesting! We’ve tried a few sanity checks on the data returned by the dataloader but got no luck. Can you share the full script to reproduce the issue?</p>
<p>We initialized a toy dataloader like this:</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
import torch
import pytorch_lightning
from careamics.lightning import (  
    create_careamics_module,
    create_predict_datamodule,
    create_train_datamodule,
)
from careamics.prediction_utils import convert_outputs
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks import (
    ModelCheckpoint,
)

# training data
rng = np.random.default_rng(42)
train_array = rng.integers(0, 255, (64, 64)).astype(np.float32)
val_array = rng.integers(0, 255, (64, 64)).astype(np.float32)

# create lightning module
model = create_careamics_module(  
    algorithm="n2v",
    loss="n2v",
    architecture="UNet",
)

# create data module
data = create_train_datamodule(
    train_data=train_array,
    val_data=val_array,
    data_type="array",
    patch_size=(16, 16),
    axes="YX",
    batch_size=2,
    dataloader_params={"num_workers": 4},
    # transforms=[]
)

# create trainer
trainer = Trainer(  
    max_epochs=1,
    default_root_dir=".",
    callbacks=[
        ModelCheckpoint(  
            dirpath="./checkpoints",
            filename="basic_usage_lightning_api",
        )
    ],
)

data.prepare_data()
data.setup()

# check the dataloader 
for batch in data.train_dataloader():
    print(batch)
    print("===")
</code></pre>

melisande-c: <p>Hi,</p>
<p>Of course!</p>
<p>In my python env I first installed PyTorch and then, careamics using <code>pip install "careamics[dev,examples,wandb]==0.0.4.2"</code>.</p>
<p>The following script downloads an example dataset that has artificially added gaussian noise. I use our <code>CAREamist</code> API for convenience but I have tried variations where I instantiate the different lightning modules individually, with the same results.</p>
<p>In another experiment I wrote a lightning callback to save the first <code>n</code> batches each epoch, to see if the data was shuffled, and as far as I can tell, it is only shuffled in the case <code>shuffle</code> is explicitly set to <code>True</code>.</p>
<pre data-code-wrap="python"><code class="lang-python">from pathlib import Path
import glob

import numpy as np
import matplotlib.pyplot as plt
import tifffile
import wandb

from careamics_portfolio import PortfolioManager
from careamics import CAREamist
from careamics.config import create_n2v_configuration

# instantiate data portfolio manage
portfolio = PortfolioManager()

# and download the data
root_path = Path("./data")
files = portfolio.denoising.N2V_BSD68.download(root_path)

# create paths for the data
data_path = Path(root_path / "denoising-N2V_BSD68.unzip/BSD68_reproducibility_data")
train_path = data_path / "train"
val_path = data_path / "val"
test_path = data_path / "test" / "images"
gt_path = data_path / "test" / "gt"

# select single file for displaying results
test_files = glob.glob("*.tif*", root_dir=test_path)
test_files.sort()
test_file = test_files[0]
test_image = tifffile.imread(test_path/test_file)

dataloader_params_combinations = [
    {"shuffle": False, "num_workers": 0},
    {"shuffle": False, "num_workers": 4},
    {"shuffle": True, "num_workers": 4},
]
predictions = {}
for dataloader_params in dataloader_params_combinations:
    experiment_name = (
        f"shuffle-{dataloader_params['shuffle']}"
        f"-num_workers-{dataloader_params['num_workers']}"
    )
    config = create_n2v_configuration(
        experiment_name=experiment_name,
        data_type="tiff",
        axes="SYX",
        patch_size=(64, 64),
        batch_size=64,
        num_epochs=10,
        augmentations=[], # No data augmentation
        logger="wandb",
        dataloader_params=dataloader_params
    )

    careamist = CAREamist(source=config, work_dir=Path(__file__).parent)
    careamist.train(train_source=train_path, val_source=val_path)

    experiment_prediction = careamist.predict(
        source=test_path / test_file,
        data_type=config.data_config.data_type,
        axes= "YX",
        image_means=config.data_config.image_means,
        image_stds=config.data_config.image_stds,
        tile_size=(128, 128),
        tile_overlap=(48, 48),
        tta_transforms=False
    )
    predictions[experiment_name] = experiment_prediction[0]
    wandb.finish()

fig, axes = plt.subplots(2, 2)
fig.set_size_inches(16, 12)
axes.flatten()[0].imshow(test_image, cmap="gray")
axes.flatten()[0].set_title("Input")
for i, dataloader_params in enumerate(dataloader_params_combinations):
    experiment_name = (
        f"shuffle-{dataloader_params['shuffle']}"
        f"-num_workers-{dataloader_params['num_workers']}"
    )
    axes.flatten()[i+1].imshow(np.squeeze(predictions[experiment_name]), cmap="gray")
    axes.flatten()[i+1].set_title(experiment_name)
fig.suptitle("Predictions")
fig.tight_layout()
fig.savefig(Path(__file__).parent/"prediction_figure.png")
plt.show()
</code></pre>
<p>My most recent run of this script has the following training losses, plotted by WandB:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e45d2bb257a33f2c1598278f663701cdf3145ea9.png" data-download-href="https://discuss.pytorch.org/uploads/default/e45d2bb257a33f2c1598278f663701cdf3145ea9" title="W&amp;B Chart 25_11_2024, 11_22_21 (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45d2bb257a33f2c1598278f663701cdf3145ea9_2_615x500.png" alt="W&amp;B Chart 25_11_2024, 11_22_21 (1)" data-base62-sha1="wActNdmPrIAEvxlhsA7By9oIyO5" width="615" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45d2bb257a33f2c1598278f663701cdf3145ea9_2_615x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45d2bb257a33f2c1598278f663701cdf3145ea9_2_922x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45d2bb257a33f2c1598278f663701cdf3145ea9_2_1230x1000.png 2x" data-dominant-color="F5F8F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">W&amp;B Chart 25_11_2024, 11_22_21 (1)</span><span class="informations">2128×1728 299 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the inference results:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/8858874259bb2238a1781a488d0876e69f5e7a0e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8858874259bb2238a1781a488d0876e69f5e7a0e" title="prediction_figure"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8858874259bb2238a1781a488d0876e69f5e7a0e_2_666x500.jpeg" alt="prediction_figure" data-base62-sha1="jsaBDJaV9vKZlPvDb4JuD4Jxvzg" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8858874259bb2238a1781a488d0876e69f5e7a0e_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8858874259bb2238a1781a488d0876e69f5e7a0e_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/8858874259bb2238a1781a488d0876e69f5e7a0e_2_1332x1000.jpeg 2x" data-dominant-color="747474"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">prediction_figure</span><span class="informations">1600×1200 172 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

melisande-c: <p>Hi <a class="mention" href="/u/traincheck-team">@traincheck-team</a>,</p>
<p>Was there ever any insight into what could have caused this problem?</p>
<p>Thanks</p>

traincheck-team: <p>Thanks for the updated information. However I am sorry that I missed the message and will try to dig into this this week.</p>