Trying to iterate through my custom dataset

username: dsillman2000
cooked: <p>Hi all,</p>
<p>I’m just starting out with PyTorch and am, unfortunately, a bit confused when it comes to using my own training/testing image dataset for a custom algorithm. For starters, I am making a small “hello world”-esque convolutional shirt/sock/pants classifying network. I’ve only loaded a few images and am just making sure that PyTorch can load them and transform them down properly to 32x32 usable images. My ImageFolder is set up like so:</p>
<blockquote>
<p>imgs/socks/<em>sockimages</em>.jpeg<br>
imgs/pants/<em>pantsimages</em>.jpeg<br>
imgs/shirt/<em>shirtimages</em>.jpeg</p>
</blockquote>
<p>and a similar setup for my testing images folder. According to my current knowledge, the image loader built into PyTorch should read the labels from the subfolder names within the training/test images. However, I’m getting a <code>TypeError</code> complaining that my iterator is not iterable. Here’s my code and the error:</p>
<pre><code>import torch
import torchvision
import torchvision.datasets as dset
import torchvision.transforms as transforms

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Scale((32,32)),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = dset.ImageFolder(root="imgs",transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,shuffle=True,         num_workers=2)

testset = dset.ImageFolder(root='tests',transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,shuffle=True,     num_workers=2)

classes=('shirt','pants','sock')

import matplotlib.pyplot as plt
import numpy as np

# functions to show an image
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
</code></pre>
<p><span class="hashtag-raw">#Error:</span><br>
TypeError: ‘builtin_function_or_method’ object is not iterable<br>
It says that it is in reference to the line containing <code>dataiter.next()</code>, meaning that the compiler believes that I cannot iterate <code>dataiter</code>?</p>
<p>Please help! Thanks in advance,</p>
<p>-David Sillman, new to PyTorch</p>

username: smth
cooked: <p>I just tried this basic sanity check, and it seems to work fine:</p>
<pre><code class="lang-python">import torch
import torchvision
a=torchvision.datasets.MNIST(root='.', download=True, transform=torchvision.transforms.ToTensor())
iter(torch.utils.data.DataLoader(a)).next()
</code></pre>
<p>for some reason <code>dataiter</code> in your case is remaining an inbuilt function, i’m not sure why.</p>

username: dsillman2000
cooked: <aside class="quote no-group quote-modified" data-username="smth" data-post="2" data-topic="1909">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>import torch<br>
import torchvision<br>
a=torchvision.datasets.MNIST(root=‘.’, download=True, transform=torchvision.transforms.ToTensor())<br>
iter(torch.utils.data.DataLoader(a)).next()</p>
</blockquote>
</aside>
<p>Yes, the MNIST example worked for me too. For some reason, though, my custom images are somehow contributing to my dataiter or trainloader or something being uniterable…?</p>
<p>Perhaps someone can try running the code themselves to see if there is just an issue on my side somehow? It’s clear that the compiler thinks that one of my variables is actually a method.</p>
<p>EDIT : Here is my FULL error message:</p>
<pre><code>Traceback (most recent call last):
  File "classifier.py", line 32, in &lt;module&gt;
(images, labels) = DataLoaderIter(trainloader).__next__()
File "//anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 174, in __next__
return self._process_next_batch(batch)
File "//anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 198, in _process_next_batch
 raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
File "//anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 34, in _worker_loop
 samples = collate_fn([dataset[i] for i in batch_indices])
File "//anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 34, in &lt;listcomp&gt;
samples = collate_fn([dataset[i] for i in batch_indices])
File "//anaconda/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 67, in __
getitem__
img = self.transform(img)
File "//anaconda/lib/python3.6/site-packages/torchvision/transforms.py", line 29, in __call__
img = t(img)
File "//anaconda/lib/python3.6/site-packages/torchvision/transforms.py", line 130, in __call__
w, h = img.size
TypeError: 'builtin_function_or_method' object is not iterable
</code></pre>

username: ritchieng
cooked: <p><strong>ERROR DEBUGGING 1</strong><br>
I’m having the same error message when using the official datasets from torchvision such as CIFAR10. It’s because of <code>transforms.Scale((32,32))</code>. <strong>If you remove the scale, it’s fine</strong>. <strong>It seems like a bug.</strong></p>
<p><strong>ERROR DEBUGGING 2</strong><br>
I then went on to understand <code>scale</code> works on PIL. So I went to use <code>ToPILImage</code> first.<br>
So now I’ve</p>
<pre><code class="lang-auto">transform_trf = transforms.Compose([transforms.ToPILImage(),
                                    transforms.Scale(size=[28, 28]),
                                    transforms.ToTensor(),
                                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                                    ])
</code></pre>
<p>Error given <code>AssertionError: pic should be Tensor or ndarray</code>.</p>
<p><strong>ERROR DEBUGGING 3</strong></p>
<p>Naturally I would think there’s a need for Tensor so I added a <code>toTensor()</code> operation.</p>
<pre><code class="lang-auto">transform_trf = transforms.Compose([transforms.ToTensor(),
                                    transforms.ToPILImage(),
                                    transforms.Scale(size=[28, 28]),
                                    transforms.ToTensor(),
                                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                                    ])
</code></pre>
<p>Now there’s this error which is hard to debug <code>TypeError: unsupported operand type(s) for /: 'list' and 'int'</code></p>
<p>I was hoping someone can clarify what’s happening here.</p>
<p><a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/apaszke">@apaszke</a></p>

username: tom
cooked: <p>Hi,</p>
<p>the error message is not terribly intuitive, but the reason is that the <code>size</code> parameter should be an integer that should be the length of the smaller edge (see <a href="http://pytorch.org/docs/torchvision/transforms.html#torchvision.transforms.Scale" rel="nofollow noopener">torchvision.transforms.Scale</a> in the documentation).<br>
It may be worth extending the Scale transform or checking for it if it is a common error case.</p>
<p>Best regards</p>
<p>Thomas</p>

username: tyathalae
cooked: <p>Hello,</p>
<p>I also encounter this error.<br>
My images are 256x256 8bpp PNG files.<br>
Curious thing is, I am running same code on 2 different PCs and it works on the non-CUDA system.<br>
With the CUDA-enabled system,  if I add the scale operation first,<br>
<code>'train': transforms.Compose([ transforms.Scale((64, 64)), transforms.ToTensor() ]),</code></p>
<p>I get the error message: <code>TypeError: unsupported operand type(s) for /: 'tuple' and 'int'</code></p>
<p>If I first convert to tensor and then scale,<br>
<code>'train': transforms.Compose([ transforms.ToTensor(), transforms.Scale((64, 64)) ]),</code><br>
I get the <code>TypeError: 'builtin_function_or_method' object is not iterable</code> error.</p>
<p>Finally, as <a class="mention" href="/u/ritchieng">@ritchieng</a> found out, if I remove the scaling, it works without any problems.</p>
<p>Best regards,<br>
Sel</p>

username: tyathalae
cooked: <p>Hello <a class="mention" href="/u/ritchieng">@ritchieng</a>,<br>
For me this issue was caused by an old (v0.1.8) torchvision package from pip. In that package, scale function was not checking the type of <code>size</code> if it is an int or not.</p>
<p>I have installed it from sources (master branch) and the problem of <code>TypeError: unsupported operand type(s) for /: 'list' and 'int'</code> is resolved.</p>
<p>Regards,<br>
Sel</p>

username: Li_Ting_Tay
cooked: <p>The currently used torchvision package is v0.1.12.<br>
There is still error:<br>
TypeError: unsupported operand type(s) for /: ‘list’ and ‘int’.</p>
<p>May I now any other ways to resolve it?</p>

username: tom
cooked: <aside class="quote no-group" data-username="Li_Ting_Tay" data-post="8" data-topic="1909">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/li_ting_tay/48/940_2.png" class="avatar"> Li_Ting_Tay:</div>
<blockquote>
<p>torchvision package is v0.1.12.</p>
</blockquote>
</aside>
<p>Likely, the torchvision package is still not quite up to date (0.1.12 sounds like a torch version). So you may need the source:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/vision">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/vision" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/a0a99d5b3f2a9568c2df74729b067036859ea62abce589efd4b5e77bb14810f3/pytorch/vision" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/vision" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer...</a></h3>

  <p>Datasets, Transforms and Models specific to Computer Vision - GitHub - pytorch/vision: Datasets, Transforms and Models specific to Computer Vision</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Best regards</p>
<p>Thomas</p>

username: isalirezag
cooked: <p>did you solve this issue?</p>

username: smth
cooked: <p><a class="mention" href="/u/isalirezag">@isalirezag</a> updating your <code>torchvision</code> package should solve the issue.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/pytorch/vision" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/pytorch/vision" target="_blank" rel="nofollow noopener">pytorch/vision</a></h3>

<p>vision - Datasets, Transforms and Models specific to Computer Vision</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Royi
cooked: <p><a class="mention" href="/u/smth">@smth</a>, Is there a way to access subset of the set?</p>
<p>For instance, I defined:</p>
<p><code>hTestLoader = torch.utils.data.DataLoader(datasets.MNIST('../../DataSets/MNIST/', train = False, download = True, transform = transforms.ToTensor()), batch_size = testBatchSize, shuffle = False)</code></p>
<p>Assume its length is 100.<br>
How could one access a random sub set of, let’s say 5, of the batches?</p>

username: QuantScientist
cooked: <pre><code class="lang-auto">class FullTrainningDataset(torch.utils.data.Dataset):
    def __init__(self, full_ds, offset, length):
        self.full_ds = full_ds
        self.offset = offset
        self.length = length
        assert len(full_ds)&gt;=offset+length, Exception("Parent Dataset not long enough")
        super(FullTrainningDataset, self).__init__()
        
    def __len__(self):        
        return self.length
    
    def __getitem__(self, i):
        return self.full_ds[i+self.offset]
    
validationRatio=0.11    

def trainTestSplit(dataset, val_share=validationRatio):
    val_offset = int(len(dataset)*(1-val_share))
    print ("Offest:" + str(val_offset))
    return FullTrainningDataset(dataset, 0, val_offset), FullTrainningDataset(dataset, 
                                                                              val_offset, len(dataset)-val_offset)


# In[25]:


batch_size=128

from torch.utils.data import TensorDataset, DataLoader

# train_imgs = torch.from_numpy(full_img_tr).float()
train_imgs=XnumpyToTensor (full_img)
train_targets = YnumpyToTensor(data['is_iceberg'].values)
dset_train = TensorDataset(train_imgs, train_targets)


train_ds, val_ds = trainTestSplit(dset_train)

train_loader = torch.utils.data.DataLoader(train_ds, batch_size=batch_size, shuffle=False,
                                            num_workers=1)
val_loader = torch.utils.data.DataLoader(val_ds, batch_size=batch_size, shuffle=False, num_workers=1)

print (train_loader)
print (val_loader)

</code></pre>

username: euwern
cooked: <p>I use the most recent version of torchvision (v0.20) and I still get the same error “TypeError: ‘builtin_function_or_method’ object is not iterable”</p>
<pre><code>import torchvision.transforms as t
train_trans = t.Compose([
    t.ToTensor(),
    t.RandomCrop(224),
    t.ColorJitter(
        brightness = 0.4,
        contrast = 0.4,
        saturation = 0.4),
    t.RandomHorizontalFlip(),
    t.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])
</code></pre>
<p>I notice it is the transforms.ToTensor() function that is causing the issue. My workaround is to position ToTensor function just before transforms.Normalize and all the other transform functions before it.</p>
<blockquote>
<pre><code>import torchvision.transforms as t
train_trans = t.Compose([
    t.RandomCrop(224),
    t.ColorJitter(
        brightness = 0.4,
        contrast = 0.4,
        saturation = 0.4),
    t.RandomHorizontalFlip(),
    t.ToTensor(),
    t.Normalize(mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]),
])
</code></pre>
</blockquote>

username: nicolas-gervais
cooked: <p><strong>IT IS NOT A BUG</strong>. As per the <a href="https://pytorch.org/docs/stable/_modules/torchvision/transforms/transforms.html#Scale" rel="nofollow noopener">documentation</a>, <code>scale</code> was a transformation that could be applied to a <code>PIL Image</code>, <em>not</em> a tensor! You will get these errors if you turn your <code>PIL Image</code> into a tensor <strong>before</strong> applying these transformations. See the other thread on this site <a href="https://pytorch.org/docs/stable/_modules/torchvision/transforms/transforms.html#Scale" rel="nofollow noopener">here</a>.</p>
<p>Change the order like this:</p>
<pre><code class="lang-auto">transform = transforms.Compose(
[transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
 transforms.Scale((32,32)),
 transforms.ToTensor()])
</code></pre>

username: Brando_Miranda
cooked: <p>can you not directly loop through the data set object?</p>
<hr>
<p>yes it works:</p>
<pre><code class="lang-auto">    root = Path('~/data/').expanduser()
    import torch
    import torchvision
    mnist = torchvision.datasets.MNIST(root=root, download=True, transform=torchvision.transforms.ToTensor())
    # iter(torch.utils.data.DataLoader(mnist)).next()
    for x, y in mnist:
        assert isinstance(x, torch.Tensor)
        assert isinstance(y, int)
        pass
    for x, y in iter(mnist):
        assert isinstance(x, torch.Tensor)
        assert isinstance(y, int)
        pass
</code></pre>