akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Data reading from text files

Istvan_Ferenc_Toth: <p>Hi, I am new to Pytorch. I have images stored in many text files and labels (which are vectors) stored in separate text files. I want to extract from these images some physical properties, which are buried in them, that is why the labels are vectors. I want to make a dataset to be able to use dataloaders on these files. What is the best way to load these data? Should I write a custom Dataset class or there is a standard way to read these data. One way, I imagine is to convert the text files into some image format and then use ImageLoader. What is your opinion ? Many thanks.</p>

ptrblck: <p>Based on your description I think writing a custom <code>Dataset</code> sounds like a valid approach.<br>
You can take a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a> for an example.</p>

Istvan_Ferenc_Toth: <p>Thanks. I transformed the txt files to grayscale images using matplotlib’s imsave(cmap=plt.cm.gray). When I use a ToTensor() transform on such an image opened with PIL’s open method I get a tensor shape of [4, 256, 256]. Since the images are grayscale, shouldn’t I get [1, 256, 256]? Many thanks.</p>

ptrblck: <p><code>ToTensor</code> shouldn’t change the number of channels. Could you check the shape of the loaded <code>PIL.Image</code> (or its underlying numpy array)? I would guess that <code>PIL</code> is already loading it in this shape.</p>

Istvan_Ferenc_Toth: <p>The PIL image is of 256x256 size, but as I printed out the image object it said that it has an image mode RGBA. I guess this could be  the problem. I still don’t understand why imsave() does not save it as grayscale with a single channel. I also tried PIL’s fromarray() function to save to image format, but my arrays contain float values (some of the very small values, therefore a direct conversion to int values results in zeros for the whole array) and fromarray expects uint8. Is there any way to save as an image a float valued numpy array ? Many thanks. You are very kind to answer all these questions.</p>

ptrblck: <p>You could try to specify the desired image format in <code>imsave</code>, but would need to make sure that <code>PIL</code> is not transforming the array under the hood and could thus quantize the data (if I’m not mistaken you don’t want that).</p>
<aside class="quote no-group" data-username="Istvan_Ferenc_Toth" data-post="5" data-topic="127211">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/istvan_ferenc_toth/48/40463_2.png" class="avatar"> Istvan_Ferenc_Toth:</div>
<blockquote>
<p>Is there any way to save as an image a float valued numpy array ?</p>
</blockquote>
</aside>
<p>You could store the numpy array directly, in case you want to save the exact values. I don’t know if this would fit your use case and what kind of image format your current data would represent.</p>