Prefetch_factor in Dataloader

mk6: <p>Hi,</p>
<p>According to <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">doc</a>, the <code>prefetch_factor</code> is the number of samples loaded in advance by each worker, and it’s <code>2</code> by default.<br>
I’m wondering what’s the meaning of pre-loading merely <code>2</code> examples, instead of pre-loading, say, <code>2</code> batches of data.<br>
Does pre-loading a few examples really help?</p>
<p>Thanks</p>

eqy: <p>As with many things, the best way to answer a setup-dependent question like that is to instrument a working example. Depending on the speed of model execution, the speed of storage, the number of workers, the OS filesystem caching policy, the “optimal” prefetch factor will vary, so if you find evidence that this isn’t a sane default, please open an upstream issue or PR!</p>

nivek: <p>It does pre-load 2 batches of data (not samples). The docstring on the master branch has been <a href="https://github.com/pytorch/pytorch/commit/cce831c80540e62549812020e2e94d27a6d22e5b" rel="noopener nofollow ugc">corrected</a>.</p>

hellolzc: <p>I was confused.<br>
From the class <code>_MultiProcessingDataLoaderIter</code> in the source file <code>torch\\utils\\data\\dataloader.py</code>, it can be seen from the <code>_reset</code> function that <code>prefetch_factor</code> refers to “Number of samples loaded in advance by each worker.”</p>
<pre data-code-wrap="python"><code class="lang-python">    def _reset(self, loader, first_iter=False):
        super()._reset(loader, first_iter)
        self._send_idx = 0  # idx of the next task to be sent to workers
        self._rcvd_idx = 0  # idx of the next task to be returned in __next__
        # information about data not yet yielded, i.e., tasks w/ indices in range [rcvd_idx, send_idx).
        # map: task idx =&gt; - (worker_id,)        if data isn't fetched (outstanding)
        #                  \ (worker_id, data)   if data is already fetched (out-of-order)
        self._task_info = {}
        self._tasks_outstanding = 0  # always equal to count(v for v in task_info.values() if len(v) == 1)
        # A list of booleans representing whether each worker still has work to
        # do, i.e., not having exhausted its iterable dataset object. It always
        # contains all `True`s if not using an iterable-style dataset
        # (i.e., if kind != Iterable).
        # Not that this indicates that a worker still has work to do *for this epoch*.
        # It does not mean that a worker is dead. In case of `_persistent_workers`,
        # the worker will be reset to available in the next epoch.
        self._workers_status = [True for i in range(self._num_workers)]
        # We resume the prefetching in case it was enabled
        if not first_iter:
            for idx in range(self._num_workers):
                self._index_queues[idx].put(_utils.worker._ResumeIteration())
            resume_iteration_cnt = self._num_workers
            while resume_iteration_cnt &gt; 0:
                return_idx, return_data = self._get_data()
                if isinstance(return_idx, _utils.worker._ResumeIteration):
                    assert return_data is None
                    resume_iteration_cnt -= 1
        # prime the prefetch loop
        for _ in range(self._prefetch_factor * self._num_workers):
            self._try_put_index()
</code></pre>
<p>The function <code>self._try_put_index()</code> puts one index in <code>self._index_queues</code> of a sample at a time.<br>
And <code>self._send_idx += 1</code> in <code>self._try_put_index()</code>.</p>

hellolzc: <p>I see.<br>
<code>prefetch_factor * batch_size * _num_workers</code> is true when using a batch sampler.<br>
If <code>batch_size</code> is <code>None</code>, <code>batch_sampler</code> is not used, so it is <code>prefetch_factor * _num_workers</code> in this condition.<br>
Have I understood correctly?</p>
<p>pytorch/torch/utils/data/dataloader.py : DataLoader</p>
<pre data-code-wrap="python"><code class="lang-python">    def __init__(self, dataset: Dataset[T_co], batch_size: Optional[int] = 1,
                 shuffle: bool = False, sampler: Optional[Sampler[int]] = None,
                 batch_sampler: Optional[Sampler[Sequence[int]]] = None,
                 num_workers: int = 0, collate_fn: Optional[_collate_fn_t] = None,
                 pin_memory: bool = False, drop_last: bool = False,
                 timeout: float = 0, worker_init_fn: Optional[_worker_init_fn_t] = None,
                 multiprocessing_context=None, generator=None,
                 *, prefetch_factor: int = 2,
                 persistent_workers: bool = False):
...
        if batch_size is not None and batch_sampler is None:
            # auto_collation without custom batch_sampler
            batch_sampler = BatchSampler(sampler, batch_size, drop_last)
...

    @property
    def _index_sampler(self):
        # The actual sampler used for generating indices for `_DatasetFetcher`
        # (see _utils/fetch.py) to read data at each time. This would be
        # `.batch_sampler` if in auto-collation mode, and `.sampler` otherwise.
        # We can't change `.sampler` and `.batch_sampler` attributes for BC
        # reasons.
        if self._auto_collation:
            return self.batch_sampler
        else:
            return self.sampler
</code></pre>