OSError: image file is truncated (28 bytes not processed) during learning

IdeoG: <p>Hello,<br>
I’m working on semantic segmentation project on pytorch framework. I wrote example code for UNet model with n_classes=1 and run it on windows 10 PC. Everything was great but get a lot of time because of poor gpu. Env was created via conda:<br>
python = 3.6.6<br>
PIL=5.4.1<br>
pytorch=1.0.1<br>
So, i gone to dedicated server with 1080 ti ubuntu 18.04 lts. Created the same env and checked package versions - all right. After that i moved source code and dataset to dedicated server and run it. But after first epoch i get following exception:<br>
<a href="https://hastebin.com/medolataxa.sql" rel="nofollow noopener">https://hastebin.com/medolataxa.sql</a><br>
OSError: image file is truncated (28 bytes not processed)<br>
I haven’t use truncated files. Everything was okay on windows 10, but after first epoch everything collapsed. (p.s. use allow truncated_files doesn’t solve the issue).<br>
It is also strange, that PIL.Image.open(&lt;image_path&gt;) recursive running on dataset didn’t throw any exception with files on dedicated server.<br>
Is someone know how to fix it?</p>

ptrblck: <p>Maybe there were some issues moving these files to the server?<br>
Could you try to load all images in a loop, store the index which gives you the error, and have a look at this particular file?<br>
Something like this should give you the index:</p>
<pre><code class="lang-python">for idx, (data, target) in enumerate(dataset):
    print(idx)
</code></pre>
<p>The error should then be related to <code>idx+1</code>. Depending on the <code>Dataset</code> you are using, you could try to get the corresponding image path and check the file manually.</p>

IdeoG: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for yor reply,</p>
<p>I wrote custom Dataset. Here is a code:</p>
<pre><code class="lang-auto">class DatasetLoader(Dataset):

    def __init__(self, X, y, input_transform=None, label_transform=None):

        self.data = X
        self.labels = y
        self.input_transform = input_transform
        self.label_transform = label_transform

    @staticmethod
    def load_dataset(data_dir: str):
        logger.debug(f"load_dataset: Loading dataset from {data_dir}")

        inputs_dir = f'{data_dir}/inputs'
        labels_dir = f'{data_dir}/labels'

        inputs = []
        for image_path in tqdm(glob.glob(inputs_dir + '/*')):
            image = Image.open(image_path)
            inputs.append(image)

        labels = []
        for image_path in tqdm(glob.glob(labels_dir + '/*')):
            label = Image.open(image_path).convert('L')
            labels.append(label)

        return inputs, labels

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        data = self.data[idx]
        if self.input_transform is not None:
            data = self.input_transform(data)

        label = self.labels[idx]
        if self.label_transform is not None:
            label = self.label_transform(label)
        return data, label
</code></pre>
<p>Also i zipped dataset folder, moved to dedicated server via scp and check sha256sum. Everything was good.</p>

ptrblck: <p>Thanks for the code!<br>
You could add a <code>try ... expect</code> block around the loop where <code>Image.open</code> is called and see, which image makes problems.<br>
Maybe you are using different <code>PIL</code> versions, such that this issue was fixed on your local machine?</p>

IdeoG: <p>Actually, image loading works good. I checked it in jupyter. Also, code using shown below:</p>
<pre><code class="lang-auto">def load_datasets(data_dir, input_size, test_pct=0.2, eval_size=10):
    train_transform, test_transform, label_transform = create_transforms(input_size)

    X, y = DatasetLoader.load_dataset(data_dir)
    train_slice = round((1 - test_pct) * len(X))

    train_data = DatasetLoader(X[:train_slice], y[:train_slice],
                               input_transform=train_transform, label_transform=label_transform)
    test_data = DatasetLoader(X[train_slice:], y[train_slice:],
                              input_transform=test_transform, label_transform=label_transform)
    eval_data = DatasetLoader(X[-eval_size:], y[-eval_size:],
                              input_transform=test_transform, label_transform=label_transform)

    logger.debug(f"load_datasets: (train_data, test_data, eval_data) sizes = "
                 f"{len(train_data), len(test_data), len(eval_data)}")
    return train_data, test_data, eval_data
</code></pre>
<p>So, images get loaded before iteration within trainloader</p>

ptrblck: <p>The error message point to <code>PIL.ImageFile.load</code>, which is weird if image loading is not an issue.<br>
Could you create a (small) code snippet to reproduce this error?</p>

IdeoG: <p>Wait a minute, i’ll provide sample code for train and exact line where exception throws from.</p>

IdeoG: <p>Here is my sample code for training:</p>
<pre><code class="lang-auto">with tensorboardX.SummaryWriter(log_dir=log_dir) as summary_writer:
        for epoch in range(epochs):
            epoch_train_loss = 0

            model.train()
            logger.debug(f"train: Running epoch {epoch + 1} out of {epochs}")
            for inputs, labels in tqdm(trainloader):
                inputs, labels = inputs.cuda(non_blocking=True), labels.cuda(non_blocking=True)
                outputs = model.forward(inputs)

                loss = criterion(outputs, labels)
                epoch_train_loss += loss.item()

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
</code></pre>

IdeoG: <p>Also some utils:</p>
<pre><code class="lang-auto">def create_transforms(input_size):
    channel_means = [0.485, 0.456, 0.406]
    channel_stds = [0.229, 0.224, 0.225]

    train_tfms = transforms.Compose([transforms.Resize(input_size),
                                     transforms.ToTensor(),
                                     transforms.Normalize(channel_means, channel_stds)])
    test_tfms = transforms.Compose([transforms.Resize(input_size),
                                    transforms.ToTensor(),
                                    transforms.Normalize(channel_means, channel_stds)])
    mask_tfms = transforms.Compose([transforms.Resize(input_size),
                                    transforms.ToTensor()])
    return train_tfms, test_tfms, mask_tfms

def create_dataloaders(data_dir, input_size=256, test_pct=0.2, batch_size=64) -&gt; (DataLoader, DataLoader, DataLoader):
    train_data, test_data, eval_data = load_datasets(data_dir, input_size, test_pct)

    trainloader = DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=6, pin_memory=True)
    testloader = DataLoader(test_data, batch_size=batch_size, shuffle=False, num_workers=6, pin_memory=True)
    evalloader = DataLoader(eval_data, batch_size=1, shuffle=False, num_workers=6, pin_memory=True)
    return trainloader, testloader, evalloader

trainloader, testloader, evalloader = create_dataloaders(data_dir, test_pct=test_pct, batch_size=batch_size)
</code></pre>

IdeoG: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b_2_690x411.png" alt="image" data-base62-sha1="zu6YHbDyBdkhIqSciwIsOH0Fwtt" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b_2_690x411.png, https://discuss.pytorch.org/uploads/default/original/2X/f/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f8b37eb998fcbc07ac266f9cd3abb2104e1aaa7b.png 2x" data-dominant-color="111111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1014×604 200 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As you can see, the exception throws on idx=0 within <code>__iter__() and __next__()</code> func</p>

IdeoG: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I had an idea that jpg image compression is different on win10 and linux machines depends on a lib. So i converted all jpg images to png on win10 and send zipped folder to linux machine via scp. But again the same error. First iteration on trainloader is okay, but it’s look like images get corrupted after read, because then we are trying to read same trainloader at a time, we get truncated image.</p>
<p>Any ideas?</p>

IdeoG: <p>hmm, i tried to do random things and found that in my ubuntu server 6 instances of python file, because num_workers = 6. So, i removed num_workers from DataLoader object creation and it worked.</p>
<p>Could you provide some observations why did it happen?<a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Best regards,<br>
Alex.</p>

IdeoG: <p><a class="mention" href="/u/smth">@smth</a> Any ideas? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>I’m really not sure, why <code>PIL</code> throws an error, if you use multiple workers.<br>
Using</p>
<pre><code class="lang-python">from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True
</code></pre>
<p>did not solve the error, right?<br>
Could you upload the image somewhere? I could try to reproduce this issue on my Ubuntu machine.</p>

IdeoG: <p>I get truncated and corrupted images anyway, that doesn’t allowed for learning.</p>

IdeoG: <p>I use LFW dataset. Downloaded from here <a href="http://vis-www.cs.umass.edu/lfw/part_labels/" rel="nofollow noopener">http://vis-www.cs.umass.edu/lfw/part_labels/</a></p>

ptrblck: <p>Thanks for the link.<br>
I’ve downloaded the <code>lfw_funneled</code> dataset and it’s running fine:</p>
<pre><code class="lang-python">path = './lfw_funneled/'

dataset = datasets.ImageFolder(
    path,
    transform=transforms.ToTensor()
)

# Check Dataset
for image, target in dataset:
    print(target)

# Check DataLoader
loader = DataLoader(
    dataset,
    num_workers=6,
    shuffle=False)

for data, target in loader:
    print(target)
</code></pre>
<p>No <code>PIL</code> errors on my machine with 6 workers.<br>
I’m using Ubuntu 18.04.1 LTS, and <code>PIL 5.4.1</code>.</p>

IdeoG: <p>Try to run more than one epoch. First epoch, as i wrote before, was fine.</p>

smth: <p>it’s a clear case of a corrupted image (I’ve seen this error before).<br>
Print the image filename as well, and inspect / redownload / delete the bad image.</p>

ptrblck: <p>I don’t get any errors for 5 epochs and still think it’s related to your image.<br>
Did you try to redownload or delete the image as suggested?</p>