Dataloader length==1 instead of number of images

ntelo007: <p>Hi there,<br>
I am trying to reproduce the results of the following repository:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/7984207?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/anilbatra2185/road_connectivity" target="_blank" rel="nofollow noopener">anilbatra2185/road_connectivity</a></h3>

<p>Improved Road Connectivity by Joint Learning of Orientation and Segmentation (CVPR2019) - anilbatra2185/road_connectivity</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>While training with train_mtl.py file , I noticed that the data loader has length equal to 1, while it should have equal length to the amount of images inside the dataset folder.</p>
<p>Can you help me solve this issue? The contributor used PyTorch 0.3.0 but I am using 0.4.1 because I don’t have Linux and that version is not available fo windows anymore.</p>

ptrblck: <aside class="quote no-group" data-username="ntelo007" data-post="1" data-topic="64058">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ntelo007/48/18761_2.png" class="avatar"> ntelo007:</div>
<blockquote>
<p>while it should have equal length to the amount of images inside the dataset folder</p>
</blockquote>
</aside>
<p>That’s not the case. The length of the <code>DataLoader</code> is defined by the number of batches it can generate from the <code>Dataset</code>. In the default setup it should be <code>ceil(len(dataset)/batch_size)</code>. If the batch size is equal to the length of your dataset, only a single batch will be returned.</p>

ntelo007: <p>Thank you for your reply.</p>
<p>this is the part of the code that created the train_loader variable:</p>
<pre><code class="lang-auto">train_loader = data.DataLoader(
    __dataset__[args.dataset](
        config["train_dataset"],
        seed=config["seed"],
        is_train=True,
        multi_scale_pred=args.multi_scale_pred,
    ),
    batch_size=config["train_batch_size"],
    num_workers=8,
    shuffle=True,
    pin_memory=False,
)
</code></pre>
<p>As you can see batch size is set by a parameter inside a json file. The number of images is 4000 (more or less) and the number of batch is 16.<br>
Here is the part of the json file that provides this kind of info to the code:</p>
<pre><code class="lang-auto">"seed": 7,
    "task1_classes": 2,
    "task2_classes": 37,
    "task1_weight": 1,
    "task2_weight": 1,
    "train_batch_size": 16,
    "val_batch_size": 4,
    "refinement":3,
    "train_dataset": {
        # we are not interested in the spacenet dataset
        "spacenet":{
            "dir": "/data/spacenet/train_crops/",
            "file": "/data/spacenet/train_crops.txt",
            "image_suffix": ".png",
            "gt_suffix": ".png",
            "crop_size": 256
        },
        "deepglobe":{
            "dir": "C:/Users/.../road_connectivity-master/data/deepglobe/train_crops/",
            "file": "C:/Users/.../road_connectivity-master/data/deepglobe/train_crops.txt",
            "image_suffix": ".jpg",
            "gt_suffix": ".png",
            "crop_size": 256
        },
        "crop_size": 256,
        "augmentation": true,
        "mean" : "[70.95016901, 71.16398124, 71.30953645]",
        "std" : "[ 34.00087859, 35.18201658, 36.40463264]",
        "normalize_type": "Mean",
        "thresh": 0.76,
        "angle_theta": 10,
        "angle_bin": 10
    },
</code></pre>
<p>I am not an experienced developer, so please excuse me if I am missing something obvious here.</p>

ntelo007: <p>The train function that uses the train_loader is this:</p>
<pre><code class="lang-auto">def train(epoch):
    ...
    for i, data in enumerate(train_loader, 0):
        inputsBGR, labels, vecmap_angles = data
        inputsBGR = Variable(inputsBGR.float().cuda())
        outputs, pred_vecmaps = model(inputsBGR)
        ...

        viz_util.progress_bar(
            i,
            len(train_loader),
            "Loss: %.6f | VecLoss: %.6f | road miou: %.4f%%(%.4f%%) | angle miou: %.4f%% "
            % (
                train_loss_iou / (i + 1),
                train_loss_vec / (i + 1),
                miou,
                road_iou,
                miou_angle,
            ),
        )

</code></pre>
<p>When I run this part of code to train the algorithm, I get that the length of the train loader is equal to 1, and therefore, it trains only on one image per epoch. Can you help me overcome this issue?</p>

ptrblck: <p>Could you try to narrow down your code to just the data loading, as you shouldn’t get a single batch out of your <code>DataLoader</code> using the specified arguments.</p>
<p>Have a look at this code snippet:</p>
<pre><code class="lang-python">N = 4000
batch_size = 16
dataset = TensorDataset(
    torch.randn(N, 1)
)
loader = DataLoader(
    dataset,
    batch_size=batch_size
)

print('len(dataset) ', len(dataset))
print('len(loader) ', len(loader))
print('expected number of batches {}'.format(
    math.ceil(N / batch_size)))

counter = 0
for data in loader:
    counter += 1

print('loader yields {} batches'.format(counter))
</code></pre>
<p>and try to add the print statements into your code. It would be interesting to see the lengths etc.</p>

ntelo007: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> please check again the previous comment. I have deleted parts of the code that you probably don’t need.</p>

ptrblck: <p>The code looks OK (not sure if the <code>progress_bar</code> is relevant, but I just ignore it), so you would need to add the print statements to your code after creating the <code>Dataset</code> and <code>DataLoader</code>.</p>
<p>In particular, add these lines:</p>
<pre><code class="lang-python">print('len(dataset) ', len(dataset))
print('len(loader) ', len(loader))
print('expected number of batches {}'.format(
    math.ceil(N / batch_size)))
</code></pre>
<p>before executing the training loop and post these numbers here.</p>

ntelo007: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="64058">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>not sure if the <code>progress_bar</code> is relevant, but I just ignore it</p>
</blockquote>
</aside>
<p>The progress bar is where it is shown that the train_loader length is equal to 1, while it should be equal to a bigger number, corresponding to the number of images of the batch.</p>
<p>The result of the previous code snipet is the following:</p>
<pre><code class="lang-auto">len(dataset)  1
len(loader)  1
expected number of batches 2642
</code></pre>
<p>I set N=42264 manually → the number of images withing my training folder.<br>
batch_size=16, set according to the config file</p>

ptrblck: <p>Apparently your <code>Dataset</code> only contains a single sample.<br>
Could you check the image folders and make sure they contain the expected number of files (also make sure they are not in a subfolder etc.).</p>

ntelo007: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f0221a66d63b66c9fbb403b34c19320291bc896f.png" alt="dataset" data-base62-sha1="ygjEPoeksAoDbfSgz5tldY52oaz" width="614" height="281"><br>
As you can see in the above image, the path that I am using is the correct one (it matches the config file).</p>

ntelo007: <p>Solved it. There was a file responsible for the Dataset creation and because I am running on Windows while this code was implemented in Linux, I had to modify some parts of the code that created newline characters and were relevant with the path. Thank you so much for your patience and I am sorry that you spent your time to debug something that didn’t have to do with PyTorch. I wish you a happy remainder of the day.</p>

ptrblck: <p>No worries and I’m glad you’re found the issue. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>