Using IterableDataset, resulting AttributeError: 'list' object has no attribute 'dim'

CyrilTT: <p>I am new to pytorch and have some issue to with my first build.<br>
I am trying to build a simple ANN by feeding in data through IterableDataset.<br>
However, I get an error with [AttributeError: ‘list’ object has no attribute “dim”]<br>
I guess I must have done something wrong with the output of IterableDataset.<br>
How can I correct the code?<br>
I have to use some method to not load my csv in one go as it is too large for the memory.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">class CustomIterableDatasetv1(IterableDataset):
    
    def __init__(self, filename):
        self.filename = filename

    def line_mapper(self, line):
        #Splits the line into text and label
        tmp = line.strip('\n').split(',')
        feature = tmp[:353]
        label  = tmp[-1]
        return feature, label


    def __iter__(self):
        file_itr = open(self.filename)
        mapped_itr = map(self.line_mapper,file_itr)
        
        return mapped_itr
</code></pre>
<pre><code class="lang-auto">params = {'batch_size': 64,
          'num_workers': 6}
max_epochs = 20

dataset = CustomIterableDatasetv1('pipeline2.csv')
dataloader = DataLoader(dataset, **params)

model = ANNnetwork()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
</code></pre>
<pre><code class="lang-auto">for epoch in range(max_epochs): 
      for (features, label) in dataloader:
        
        optimizer.zero_grad()

        outputs = model(features)
        loss = criterion(outputs, label.squeeze(1))
        
        loss.backward()
        optimizer.step()
        cost = loss.item()
        if i % 100 == 0:
            print('Epoch:' + str(epoch) + ", Iteration: " + str(i) 
                  + ", training cost = " + str(cost))
</code></pre>

nivek: <p>It looks like you are providing a Python <code>list</code> when it is expecting a <code>Tensor</code>, can you check the line where the error is raised to see what the inputs’ types are?</p>
<p>You should also check what your <code>__iter__</code> function is returning. Automatic batching will take 64 (your <code>batch_size</code>) of those and pass them into the <code>default_collate</code> function. Make sure that output has the type that your model expects. You can override <code>collate_fn</code> as well if the default function is not doing what you’d like. You can read more about <a href="https://pytorch.org/docs/stable/data.html#automatic-batching-default" rel="noopener nofollow ugc">automatic batching here</a>.</p>

gilbertcane: <aside class="quote no-group" data-username="CyrilTT" data-post="1" data-topic="152426">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/53a042/48.png" class="avatar"> CyrilTT:</div>
<blockquote>
<p>AttributeError: ‘list’ object has no attribute “dim”</p>
</blockquote>
</aside>
<p>Python lists cannot be divided into separate lists based on characters that appear in the values of a list. This is unlike strings which values can be separated into a list. The AttributeError is an exception thrown when an object does not have the attribute you tried to access. The ‘list’ object has no attribute ‘split’ and you’re trying to call <a href="http://net-informations.com/python/file/split.htm" rel="noopener nofollow ugc">python split()</a> function on the whole list of lines, and you can’t split a list of strings, only a string. So, you need to split each line, not the whole thing.</p>
<p>To solve the above problem, you need to iterate over the strings in the list to get individual strings; then, you can call the split() function.</p>