_DataLoaderIter vs _BaseDataLoaderIter

Raphikowski: <p>Hi all,</p>
<p>I am confused about the Iterator class of DataLoader. In particular I wanted to ask if the implementation has fundamentally changed between some of the pytorch versions?<br>
Since, in the online documentation<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/utils/data/dataloader.py</a></h4>
<pre><code class="lang-py">r"""Definition of the DataLoader and associated iterators that subclass _BaseDataLoaderIter

To support these two classes, in `./_utils` we define many utility methods and
functions to be run in multiprocessing. E.g., the data loading worker loop is
in `./_utils/worker.py`.
"""

import torch
import multiprocessing as python_multiprocessing
import torch.multiprocessing as multiprocessing
from . import IterableDataset, Sampler, SequentialSampler, RandomSampler, BatchSampler
from . import _utils
from torch._utils import ExceptionWrapper
import threading
import itertools
from torch._six import queue, string_classes


get_worker_info = _utils.worker.get_worker_info

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I can only find the classes _BaseDataLoaderIter(object) and its subclasses _SingleProcessDataLoaderIter(_BaseDataLoaderIter) and _MultiProcessingDataLoaderIter(_BaseDataLoaderIter). However, when I look at the anaconda code on my PC in anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py<br>
these three classes do not exist and instead there is only a class called _DataLoaderIter(object) (which seems somewhat similar to the implementation of _MultiProcessingDataLoaderIter, but they re not exactly the same).<br>
Why cant I find the code of _DataLoaderIter(object) in the documentation? Does it have to do with different pytorch versions? If so, what consequences does that have if I use custom dataset, sampler and collate_fn functions? Will they work for either pytorch version?</p>

albanD: <p>Hi,</p>
<p>All the classes that start with an underscore like <code>_Foo</code> are internal and so are not documented and can change between versions without notice.<br>
The latest big change there I can think of is: <a href="https://github.com/pytorch/pytorch/pull/19228">https://github.com/pytorch/pytorch/pull/19228</a><br>
Which version of pytorch do you currently have installed?</p>

Raphikowski: <p>Hi<br>
thank you for your reply!<br>
I am currently using Version ‘1.0.1.post2’.<br>
In your link indeed is explained that DataLoaderIter was split up into the two classes I have mentioned above. Does this mean that if I now implement a custom collate_fn, sampler and dataset, that these might not work on a newer pytorch version anymore?</p>

albanD: <p>It should.<br>
Your dataset should only touch the Dataset and Sampler classes. Not the <code>_*</code> ones anyway right?</p>

Raphikowski: <p>Well I try to make the <strong>getitem</strong> method expecting two indices as parameters such that I can select data from a 3D tensor. To do so, I at least have to rewrite the collate_fn method, too. Maybe even more, but I havent got so far yet.</p>

albanD: <p>Can you linearize your two indices as a larger 1D tensor? That way you can use the base loader.</p>

SimonW: <p>Only implementation details changed so custom dataset, sampler, and collate_fn that use public APIs will work as is.</p>

Raphikowski: <p>Thanks a lot for your replies both of you.<br>
I have just realized that in both implementations, the <strong>getitem</strong> method is always assumed to only take one argument. Since in __DataLoaderIter class there is line 615:<br>
batch = self.collate_fn([self.dataset[i] for i in indices])<br>
and in the other case, when  _MapDatasetFetcher is used, there is the line:<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
Thus, both implementations demand that the <strong>getitem</strong> method necessarily only takes one argument. And since both of the above are internal methods I guess I should not be changing.<br>
So is there really no option that I adjust the <strong>getitem</strong> method to accept two indices and hence make use of my 3D dataset?</p>