EarlyStopping BrokenPipeError: [Errno 32] Broken pipe
🐛 Bug
EarlyStopping together with num_workers > 0 yields the following error message for me:
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/user/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/user/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/user/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

I strongly believe this is the case, as setting num_workers=0 or not using the EarlyStopping callbacks removes this error message from showing up.
To Reproduce
This is probably coupled with reading data from a hdf5 file, with an implementation similar to this one:
https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/16
(It seems like h5 does not have good support multiprocessing handling.)
Environment

PyTorch Version (e.g., 1.0): 1.7.0
OS (e.g., Linux): Ubuntu 20.04
How you installed PyTorch (conda, pip, source): poetry
Python version: 3.8.2
CUDA/cuDNN version: nvidia-cuda-toolkit
GPU models and configuration: 2080 ti