Getting a "UnpicklingError: invalid load key, '\xff'." error

username: Jordan_Howell
cooked: <p>I’m getting  a weird error trying to feed data into my first pytorch cov-net.  Here is the error traceback.</p>
<blockquote>
<hr>
<p>UnpicklingError                           Traceback (most recent call last)<br>
 in <br>
6<br>
7 for epoch in range(max_epochs):<br>
----&gt; 8     for i in (partition[‘train’], labels) in enumerate(training_generator):<br>
9         # Run the forward pass<br>
10         outputs = model(images)</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in <strong>next</strong>(self)<br>
344     def <strong>next</strong>(self):<br>
345         index = self._next_index()  # may raise StopIteration<br>
 → 346         data = self.dataset_fetcher.fetch(index)  # may raise StopIteration<br>
347         if self.pin_memory:<br>
348             data = _utils.pin_memory.pin_memory(data)</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in (.0)<br>
42     def fetch(self, possibly_batched_index):<br>
43         if self.auto_collation:<br>
—&gt; 44             data = [self.dataset[idx] for idx in possibly_batched_index]<br>
45         else:<br>
46             data = self.dataset[possibly_batched_index]</p>
<p> in <strong>getitem</strong>(self, index)<br>
24<br>
25             # Load data and get label<br>
—&gt; 26             X = torch.load(root_dir + ID)<br>
27             y = self.labels[ID]<br>
28</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)<br>
384         f = f.open(‘rb’)<br>
385     try:<br>
 → 386         return _load(f, map_location, pickle_module, **pickle_load_args)<br>
387     finally:<br>
388         if new_fd:</p>
<p>C:\ProgramData\Anaconda3\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)<br>
561             f.seek(0)<br>
562<br>
 → 563     magic_number = pickle_module.load(f, **pickle_load_args)<br>
564     if magic_number != MAGIC_NUMBER:<br>
565         raise RuntimeError(“Invalid magic number; corrupt file?”)</p>
<p>UnpicklingError: invalid load key, ‘\xff’.</p>
</blockquote>
<p>My image file names and data resides in two dictionaries.  Dictionary 1 called partition has a train and test key with a list of image filenames.</p>
<p>Here is the data class, transform and data loader:</p>
<blockquote>
<p>root_dir = ‘D:\CIS inspection images 0318\train\roof\’</p>
<p>class roof_dataset(Dataset):</p>
<pre><code>'Characterizes a dataset for PyTorch'

def __init__(self, list_IDs, labels, transform):
    'Initialization'
    self.labels = labels
    self.list_IDs = list_IDs
    self.transform = transform

def __len__(self):
        
    'Denotes the total number of samples'
    return len(self.list_IDs)

def __getitem__(self, index):

        'Generates one sample of data'
        # Select sample
        ID = self.list_IDs[index]

        # Load data and get label
        X = torch.load(root_dir + ID)
        y = self.labels[ID]
        
        if self.transform:
            X = self.transform(X)

        return X, y
</code></pre>
<h1>CUDA for PyTorch</h1>
<p>use_cuda = torch.cuda.is_available()<br>
device = torch.device(“cuda:0” if use_cuda else “cpu”)</p>
<h1>Parameters</h1>
<p>params = {‘batch_size’: 1000,<br>
‘shuffle’: True,<br>
‘num_workers’: 0}<br>
max_epochs = 100</p>
<h1>Generators</h1>
<p>training_set = roof_dataset(partition[‘train’], labels, transform = train_transforms)<br>
training_generator = data.DataLoader(training_set, **params)</p>
<p>test_set = roof_dataset(partition[‘test’], labels, transform = train_transforms)<br>
test_generator = data.DataLoader(test_set, **params)</p>
</blockquote>
<p>Finally, here is where I get the error:</p>
<blockquote>
<p>total_step = len(training_generator)<br>
loss_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
acc_list = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for epoch in range(max_epochs):<br>
for i in (partition[‘train’], labels) in enumerate(training_generator):<br>
# Run the forward pass<br>
outputs = model(images)<br>
loss = criterion(outputs, labels)<br>
loss_list.append(loss.item())</p>
<pre><code>    # Backprop and perform Adam optimisation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # Track the accuracy
    total = labels.size(0)
    _, predicted = torch.max(outputs.data, 1)
    correct = (predicted == labels).sum().item()
    acc_list.append(correct / total)

    if (i + 1) % 100 == 0:
        print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                      (correct / total) * 100))
</code></pre>
</blockquote>
<p>I think it’s in the data class but I’m not sure.  Does anyone see what I’m doing wrong?</p>

username: martinr
cooked: <p>This is some kind of data loading issue. <a href="https://docs.python.org/3/library/pickle.html" rel="nofollow noopener">Pickle</a> is a Python de/serialization module used in Dataloader. /xff might refer to hexademical FF. So most probably you are going out of range, perhaps trying to access a resource that does not exist?</p>
<p><code>X = torch.load(root_dir + ID)</code></p>
<p>I suggest to debug or print out <em>root_dir + ID</em> and see if the file really exists (or perhaps it does, but can  not be unpickled, e.g., corrupted, wrong format etc).</p>

username: Brando_Miranda
cooked: <p>related: <a href="https://github.com/learnables/learn2learn/issues/310" rel="noopener nofollow ugc">https://github.com/learnables/learn2learn/issues/310</a></p>