How to create a Dataloader for a txt file containing pixel values of dimension 28 by 28

sam101: <p>I have many txt files, each containing the pixels values of dimensions 28 by 28. How can I create a Dataloader for them that I will feed to a GAN model</p>

pfloat: <p>Hello,<br>
can you show the content or formatting inside one text file ?</p>

sam101: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e53e9f05ae8f71407c2209290fb717d01207807b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e53e9f05ae8f71407c2209290fb717d01207807b" title="Screen Shot 2020-09-07 at 3.27.15 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e53e9f05ae8f71407c2209290fb717d01207807b_2_690x166.png" alt="Screen Shot 2020-09-07 at 3.27.15 PM" data-base62-sha1="wHZvn2lfvaAuIMOAwS0NJ74tQLh" width="690" height="166" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e53e9f05ae8f71407c2209290fb717d01207807b_2_690x166.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e53e9f05ae8f71407c2209290fb717d01207807b_2_1035x249.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/5/e53e9f05ae8f71407c2209290fb717d01207807b_2_1380x332.png 2x" data-dominant-color="D4D4D4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-09-07 at 3.27.15 PM</span><span class="informations">1502×362 218 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the screenshot of the content of the file</p>

sam101: <p>I have attached the screen shot</p>

pfloat: <p>In order to extract the pixels values from the file, we need to know how to extract one pixel value.<br>
which color domain is used? is it a color pixel rgb or bgr ? or grayscale ? or …?</p>
<p>In the mean time, the float values (that are between -1 and 1?) can be extracted using a whitespace separator. The idea would be to do something like this:</p>
<pre><code class="lang-auto">with open("Im_1.txt", "r") as fd:
    content = fd.readlines().strip()
    # each row is split into a list of string representing float numbers
    # then each string is converted to a float with map(float, ...)
    pixel_values = [map(float, row.split(" ")) for row in content]
</code></pre>
<p>Now, IF the image is grayscale, the list <code>pixel_values</code> and contains 28 rows of 28 elements, we obtain the tensor:</p>
<pre><code class="lang-auto">image = torch.tensor(pixel_values)
</code></pre>

sam101: <p>Thank you for the answer and these are rgb values and how can I make a Dataloader for list of txt files like we have for images in a folder</p>

pfloat: <p>Hello,<br>
ok then it means, it is possible to group the float values by 3 to get color pixels assuming the floats are gathered in a list and that this is the <strong>original intended format</strong> for the file.</p>
<p>Now a solution would be to subclass the <code>torch.utils.data.Dataset</code> class:</p>
<pre><code class="lang-auto">import itertools
from torch.utils.data import Dataset, DataLoader


def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

class TextDataset(Dataset):
    def __init__(self, text_file_paths, other_arguments):
        self.paths = text_file_paths
        # other processing you may want
        ...

    def __len__(self):
        return len(self.paths)
    
    def __getitem__(self, idx):
        path = self.paths[idx]
        with open(path, "r") as fd: # EDIT 
            content = fd.readlines().strip()
        pixel_values = [map(float, row.split(" ")) for row in content]
        # all float values in one list
        all_float_values = list(itertools.chain.from_iterable(pixel_values)))
        # group the float values by 3
        color_pixels = list(chunks(all_float_values, 3))
        # again, split this color_pixels into a list of 28 elements
        image = list(chunks(color_pixels, 28))
        return torch.tensor(image)
</code></pre>
<p>StackOverflow links:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks" rel="nofollow noopener">how-do-you-split-a-list-into-evenly-sized-chunks</a></li>
<li><a href="https://stackoverflow.com/questions/716477/join-list-of-lists-in-python" rel="nofollow noopener">join-list-of-lists-in-python</a></li>
</ul>
<p>Finally</p>
<pre><code class="lang-auto">text_ds = TextDataset(your_text_file_paths_list, ...)
mydataloader = DataLoader(text_ds, batch_size=16, shuffle=True)
</code></pre>
<p>Hope it helps</p>

sam101: <aside class="quote no-group" data-username="pfloat" data-post="5" data-topic="95437">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pfloat/48/28529_2.png" class="avatar"> pfloat:</div>
<blockquote>
<p><code>image = torch.tensor(pixel_values)</code></p>
</blockquote>
</aside>
<p>Thank you so much. I really appreciate it.</p>