Reading a csv file for the prediction of model

Murad_Ali: <p>I am reading the 1st 15 columns of a csv file for the prediction of a model. The 15th column has the scientific name of the plant that I am using in the task. When I run the code it shows an error “no such file or directory “Alliaria petiolata”” which is the scientific name of the plant. I don’t know how to solve this problem as it is not a file or directory but just an entry in a columns. The code is</p>
<pre><code class="lang-auto"> with torch.no_grad():
    for data in testloader:
        images = data
        output = model(images)
        class_probs_batch = [F.softmax(el, dim=0) for el in output]
        _, class_preds_batch = torch.max(output, 1)

        class_probs.append(class_probs_batch)
        class_preds.append(class_preds_batch)
</code></pre>
<p>and a custom Dataset class</p>
<pre><code class="lang-auto">class CustomDataSet(Dataset):
def __init__(self, csv_file, root_dir, transform):
    self.root_dir = root_dir
    self.transforms = transform
    self.dataframe = pd.read_csv(csv_file)

def __len__(self):
    return len(self.dataframe)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    img_path = self.dataframe.iloc[idx, 15]
    image = Image.open(img_path).convert("RGB")
    #image = Image.open(img_path)

    tensor_image = self.transforms(image)
    return tensor_image
</code></pre>
<p>The error is:</p>
<pre><code class="lang-auto">Traceback (most recent call last)
Input In [6], in &lt;cell line: 1&gt;()
      1 with torch.no_grad():
----&gt; 2     for data in testloader:
      3         images = data
      4         output = model(images)

File ~/.conda/envs/flowering/lib/python3.10/site-packages/torch/utils/data/dataloader.py:530, in _BaseDataLoaderIter.__next__(self)
    528 if self._sampler_iter is None:
    529     self._reset()
--&gt; 530 data = self._next_data()
    531 self._num_yielded += 1
    532 if self._dataset_kind == _DatasetKind.Iterable and \
    533         self._IterableDataset_len_called is not None and \
    534         self._num_yielded &gt; self._IterableDataset_len_called:

File ~/.conda/envs/flowering/lib/python3.10/site-packages/torch/utils/data/dataloader.py:570, in _SingleProcessDataLoaderIter._next_data(self)
    568 def _next_data(self):
    569     index = self._next_index()  # may raise StopIteration
--&gt; 570     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    571     if self._pin_memory:
    572         data = _utils.pin_memory.pin_memory(data)

File ~/.conda/envs/flowering/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:49, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---&gt; 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

File ~/.conda/envs/flowering/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py:49, in &lt;listcomp&gt;(.0)
     47 def fetch(self, possibly_batched_index):
     48     if self.auto_collation:
---&gt; 49         data = [self.dataset[idx] for idx in possibly_batched_index]
     50     else:
     51         data = self.dataset[possibly_batched_index]

Input In [2], in CustomDataSet.__getitem__(self, idx)
     18     idx = idx.tolist()
     19 img_path = self.dataframe.iloc[idx, 15]
---&gt; 20 image = Image.open(img_path).convert("RGB")
     21 #image = Image.open(img_path)
     23 tensor_image = self.transforms(image)

File ~/.conda/envs/flowering/lib/python3.10/site-packages/PIL/Image.py:3092, in open(fp, mode, formats)
   3089     filename = fp
   3091 if filename:
-&gt; 3092     fp = builtins.open(filename, "rb")
   3093     exclusive_fp = True
   3095 try:

FileNotFoundError: [Errno 2] No such file or directory: 'Alliaria petiolata'
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Murad_Ali" data-post="1" data-topic="159098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/murad_ali/48/45818_2.png" class="avatar"> Murad_Ali:</div>
<blockquote>
<p>The 15th column has the scientific name of the plant that I am using in the task. When I run the code it shows an error “no such file or directory “Alliaria petiolata”” which is the scientific name of the plant. I don’t know how to solve this problem as it is not a file or directory but just an entry in a columns.</p>
</blockquote>
</aside>
<p>This description does not fit the usage of this column in:</p>
<pre data-code-wrap="python"><code class="lang-python">    img_path = self.dataframe.iloc[idx, 15]
    image = Image.open(img_path).convert("RGB")
</code></pre>
<p>since you are indeed using it as an image path.<br>
Based on your description it sounds as if the actual input data is not an image but is stored as features in the columns <code>[0, 14]</code> while column <code>15</code> contains the target name (which you would most likely have to map to a class index).</p>

Murad_Ali: <p>Thank you for your reply. can you tell me what should I do here. I don’t know what to do exactly. How to change it?</p>

Murad_Ali: <p>When I read less or more columns than 15 like:<br>
img_path = self.dataframe.iloc[idx, 10]<br>
it gives me: numpy.int64’ object has no attribute ‘read’</p>

ptrblck: <p>It seems you are still trying to call <code>Image.open</code> on the indexing result from the <code>DataFrame</code> so you would need to check what exactly is stored in your <code>DataFrame</code> and how you would like to process and use it.<br>
From your description it seems the <code>DataFrame</code> contains numerical values as well as strings containing the class name. In this case, why would you want to use <code>Image.open</code>? Is the <code>DataFrame</code> also storing paths to images, which you want to open?</p>

Murad_Ali: <p>how can I check that what is stored in the dataframe? It is not working with show() or head() or print(). I am not getting this point. I am running someone else code and I have to make some changes later once it works.</p>

Murad_Ali: <p>in the datafram I think I am reading this data<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c0c049db46003d9553bcc13f3ac6517ea434ed5.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c0c049db46003d9553bcc13f3ac6517ea434ed5" title="ss_data"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c0c049db46003d9553bcc13f3ac6517ea434ed5.png" alt="ss_data" data-base62-sha1="1IztE2E9i2q5PiTBuNWCZb7YOrj" width="690" height="471" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ss_data</span><span class="informations">832×568 38.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="Murad_Ali" data-post="6" data-topic="159098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/murad_ali/48/45818_2.png" class="avatar"> Murad_Ali:</div>
<blockquote>
<p>I am running someone else code and I have to make some changes later once it works.</p>
</blockquote>
</aside>
<p>I would recommend to check the <code>DataFrame</code> and think how each entry should be used.<br>
The last picture shows that the <code>DataFrame</code> contains mostly strings, which you won’t be able to use directly as inputs to any neural network.<br>
Check how these string values should be remapped to any feature values, what the data, and target should contain, and write a custom <code>Dataset</code> to load these samples.</p>

Murad_Ali: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your help and support. That part of code is now working but I am facing another error. I am pasting the code here and the error. I don’t know how to give it the path of the images and it gives me keyerror.</p>
<pre><code class="lang-auto">#Get model output
preds = test_preds.numpy()
probs = test_probs.numpy()
model_output = np.column_stack((preds, probs))
model_output = pd.DataFrame(model_output, columns = ['Label', 'f_prob', 'nf_prob'])
model_output['imagepaths'] = list(img_list['imagepaths'])
model_output.to_csv('model_predictions_2.csv')```

In this code I don't know how to pass the imagespath. and the error is
</code></pre>
<p>Traceback (most recent call last)<br>
File ~/.conda/envs/flowering/lib/python3.10/site-packages/pandas/core/indexes/base.py:3621, in Index.get_loc(self, key, method, tolerance)<br>
3620 try:<br>
 → 3621     return self._engine.get_loc(casted_key)<br>
3622 except KeyError as err:</p>
<p>File ~/.conda/envs/flowering/lib/python3.10/site-packages/pandas/_libs/index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()</p>
<p>File ~/.conda/envs/flowering/lib/python3.10/site-packages/pandas/_libs/index.pyx:163, in pandas._libs.index.IndexEngine.get_loc()</p>
<p>File pandas/_libs/hashtable_class_helper.pxi:5198, in pandas._libs.hashtable.PyObjectHashTable.get_item()</p>
<p>File pandas/_libs/hashtable_class_helper.pxi:5206, in pandas._libs.hashtable.PyObjectHashTable.get_item()</p>
<p>KeyError: ‘imagepaths’</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>KeyError                                  Traceback (most recent call last)<br>
Input In [29], in &lt;cell line: 6&gt;()<br>
4 model_output = np.column_stack((preds, probs))<br>
5 model_output = pd.DataFrame(model_output, columns = [‘Label’, ‘f_prob’, ‘nf_prob’])<br>
----&gt; 6 model_output[‘imagepaths’] = list(img_list[‘imagepaths’])<br>
7 model_output.to_csv(‘model_predictions_2.csv’)</p>
<p>File ~/.conda/envs/flowering/lib/python3.10/site-packages/pandas/core/frame.py:3505, in DataFrame.<strong>getitem</strong>(self, key)<br>
3503 if self.columns.nlevels &gt; 1:<br>
3504     return self._getitem_multilevel(key)<br>
 → 3505 indexer = self.columns.get_loc(key)<br>
3506 if is_integer(indexer):<br>
3507     indexer = [indexer]</p>
<p>File ~/.conda/envs/flowering/lib/python3.10/site-packages/pandas/core/indexes/base.py:3623, in Index.get_loc(self, key, method, tolerance)<br>
3621     return self._engine.get_loc(casted_key)<br>
3622 except KeyError as err:<br>
 → 3623     raise KeyError(key) from err<br>
3624 except TypeError:<br>
3625     # If we have a listlike key, _check_indexing_error will raise<br>
3626     #  InvalidIndexError. Otherwise we fall through and re-raise<br>
3627     #  the TypeError.<br>
3628     self._check_indexing_error(key)</p>
<p>KeyError: ‘imagepaths’</p>
<pre><code class="lang-auto">
Please help me in this.
Thank you</code></pre>

ptrblck: <p>It seems that <code>list(img_list['imagepaths'])</code> is failing to index the <code>img_list</code>. Make sure this key is valid and you will be able to read it before trying to assign it to <code>model_output</code>.</p>