Repository: scikit-image/scikit-image

Issue:
ImageCollection loads images prematurely
`ImageCollection('*.jpg')` loads all JPG images, even when conserve_memory is set to True.  What used to be an instantaneous operation is now very slow.

/cc @blink1073



Discussion:
How many images are in that folder?  I did a quick check and `conserve_memory` is still being respected, in that the `data` member is only populated when it is False.  However, we are now _opening_ each file to see how many images it contains.  That could be what you're seeing.

Yes, I think we may have to change that--I have 2000 images in my folder,
and it takes forever.
​

Then we'd have no way of knowing which if any files had multiple images in them.  Tough nut here...

@blink1073 
- do we need to know this in advance?
- maybe we can only check tifs?
- when you open each image, I presume it's loading up the whole image. For TIFFs, it should be possible to check the number of pages just by looking at the header. (Though this might require learning more about the format than you care to find out!)

Right now we support TIFFs and GIFs.  This is turning out to be a major source of confusion.  Perhaps image collection should only deal with flat images, and leave it up to imread whether to allow indexing on image number.

Even just turning off checking for any file that's not a TIFF and GIF would fix @stefanv's use case. Gotta keep the boss happy. =P

Works for me.


Resolving patch:
diff --git a/skimage/io/collection.py b/skimage/io/collection.py
--- a/skimage/io/collection.py
+++ b/skimage/io/collection.py
@@ -135,7 +135,7 @@ def imread_convert(f):
       ic = ImageCollection('/tmp/*.png', load_func=imread_convert)
 
     For files with multiple images, the images will be flattened into a list
-    and added to the list of available images.  In this case, ``load_func`` 
+    and added to the list of available images.  In this case, ``load_func``
     should accept the keyword argument ``img_num``.
 
     Examples
@@ -202,26 +202,22 @@ def _find_images(self):
                 img = TiffFile(fname)
                 index += [(fname, i) for i in range(len(img.pages))]
             else:
-                im = Image.open(fname)
                 try:
-                    # this will raise an IOError if the file is not readable
-                    im.getdata()[0]
-                except IOError:
-                    site = "http://pillow.readthedocs.org/en/latest/installation.html#external-libraries"
-                    raise ValueError(
-                        'Could not load "%s"\nPlease see documentation at: %s' % (fname, site))
-                else:
-                    i = 0
-                    while True:
-                        try:
-                            im.seek(i)
-                        except EOFError:
-                            break
-                        index.append((fname, i))
-                        i += 1
+                    im = Image.open(fname)
+                    im.seek(0)
+                except (IOError, OSError):
+                    index.append([fname, i])
+                    continue
+                i = 0
+                while True:
+                    try:
+                        im.seek(i)
+                    except EOFError:
+                        break
+                    index.append((fname, i))
+                    i += 1
                 if hasattr(im, 'fp') and im.fp:
                     im.fp.close()
-
         self._frame_index = index
         return len(index)
 


Test patch: