Repository: keras-team/autokeras

Issue:
Upgrade raw image reading to imageio.imread
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description
<!---
A clear and concise description of what the bug is.
-->

I have folders `data/train` and `data/val` that contain jpg images. I created two `labels.csv` (in `data/train` and `data/val`) file that contains two columns: 'File Name' and 'Label', and filled them with the labels of their respective image sets. I can successfully run the following code:

```
from autokeras.image_supervised import ImageClassifier, load_image_dataset

x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train")
print(x_train.shape)
print(y_train.shape)

x_val, y_val = load_image_dataset(csv_file_path="data/val/labels.csv",
                                    images_path="data/val")
```

with the output being:

```
(13750,)
(13750,)
```

I run into errors when trying to use `x_train`. When executing the following code:

```
if __name__ == '__main__':

    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_val, y_val, retrain=True)
    y = clf.evaluate(x_val, y_val)
    print(y)
```

the following error is raised:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/Software/anaconda3/lib/python3.6/site-packages/autokeras/image_supervised.py in _validate(x_train, y_train)
     24     try:
---> 25         x_train = x_train.astype('float64')
     26     except ValueError:

ValueError: setting an array element with a sequence.

During handling of the above exception, another exception occurred:

[OUTPUT TRUNCATED]
```

The output of `print(x_train)` is:

```
array([array([[[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [187, 187, 179],
        [184, 183, 178],
        [182, 181, 176]],

       [[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [184, 184, 176],
        [182, 181, 176],
        [180, 179, 174]],

       [[168, 172, 171],
        [168, 172, 171],
        [168, 172, 171],
        ...,
        [183, 183, 175],
        [180, 179, 174],
        [179, 178, 173]],

       ...,

[OUTPUT TRUNCATED]

       [[ 42,  83, 111],
        [ 40,  75, 103],
        [ 69,  96, 123],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 45,  86, 114],
        [ 49,  84, 112],
        [ 79, 106, 133],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 40,  81, 109],
        [ 51,  86, 114],
        [ 84, 111, 138],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]]], dtype=uint8)], dtype=object)
```

### Reproducing Steps
Steps to reproduce the behavior:
  * Step 1: Import dataset using `autokeras.image_supervised.load_image_dataset` from a folder with `jpg` images and a `labels.csv` file.
  * Step 2: Execute `clf = ImageClassifier(), clf.fit(x_train, y_train)`.

### Expected Behavior
<!---
A clear and concise description of what you expected to happen.
-->
I expected training on the dataset to start.

### Setup Details
<!--- Include the details about the versions of: -->
 - OS type and version: Ubuntu 18.04.1 LTS
 - Python: 3.6.5
 - autokeras: <!--- e.g. 0.2.1, 0.2.0, master--> 0.2.14
 - scikit-learn: 0.19.2
 - numpy: 1.14.5
 - keras: 2.2.2
 - scipy: 1.1.0
 - tensorflow: 1.10.1
 - pytorch: None

<!--- 
### Additional context 
-->
<!---
Add any other context about the problem here.
-->

Upgrade raw image reading to imageio.imread
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description
<!---
A clear and concise description of what the bug is.
-->

I have folders `data/train` and `data/val` that contain jpg images. I created two `labels.csv` (in `data/train` and `data/val`) file that contains two columns: 'File Name' and 'Label', and filled them with the labels of their respective image sets. I can successfully run the following code:

```
from autokeras.image_supervised import ImageClassifier, load_image_dataset

x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train")
print(x_train.shape)
print(y_train.shape)

x_val, y_val = load_image_dataset(csv_file_path="data/val/labels.csv",
                                    images_path="data/val")
```

with the output being:

```
(13750,)
(13750,)
```

I run into errors when trying to use `x_train`. When executing the following code:

```
if __name__ == '__main__':

    clf = ImageClassifier(verbose=True)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)
    clf.final_fit(x_train, y_train, x_val, y_val, retrain=True)
    y = clf.evaluate(x_val, y_val)
    print(y)
```

the following error is raised:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/Software/anaconda3/lib/python3.6/site-packages/autokeras/image_supervised.py in _validate(x_train, y_train)
     24     try:
---> 25         x_train = x_train.astype('float64')
     26     except ValueError:

ValueError: setting an array element with a sequence.

During handling of the above exception, another exception occurred:

[OUTPUT TRUNCATED]
```

The output of `print(x_train)` is:

```
array([array([[[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [187, 187, 179],
        [184, 183, 178],
        [182, 181, 176]],

       [[169, 173, 174],
        [169, 173, 174],
        [169, 173, 174],
        ...,
        [184, 184, 176],
        [182, 181, 176],
        [180, 179, 174]],

       [[168, 172, 171],
        [168, 172, 171],
        [168, 172, 171],
        ...,
        [183, 183, 175],
        [180, 179, 174],
        [179, 178, 173]],

       ...,

[OUTPUT TRUNCATED]

       [[ 42,  83, 111],
        [ 40,  75, 103],
        [ 69,  96, 123],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 45,  86, 114],
        [ 49,  84, 112],
        [ 79, 106, 133],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]],

       [[ 40,  81, 109],
        [ 51,  86, 114],
        [ 84, 111, 138],
        ...,
        [238, 239, 234],
        [237, 238, 233],
        [238, 239, 234]]], dtype=uint8)], dtype=object)
```

### Reproducing Steps
Steps to reproduce the behavior:
  * Step 1: Import dataset using `autokeras.image_supervised.load_image_dataset` from a folder with `jpg` images and a `labels.csv` file.
  * Step 2: Execute `clf = ImageClassifier(), clf.fit(x_train, y_train)`.

### Expected Behavior
<!---
A clear and concise description of what you expected to happen.
-->
I expected training on the dataset to start.

### Setup Details
<!--- Include the details about the versions of: -->
 - OS type and version: Ubuntu 18.04.1 LTS
 - Python: 3.6.5
 - autokeras: <!--- e.g. 0.2.1, 0.2.0, master--> 0.2.14
 - scikit-learn: 0.19.2
 - numpy: 1.14.5
 - keras: 2.2.2
 - scipy: 1.1.0
 - tensorflow: 1.10.1
 - pytorch: None

<!--- 
### Additional context 
-->
<!---
Add any other context about the problem here.
-->



Discussion:
I'm getting the same problem using autokeras on colab. 
When I brought in the mnist example, after the reshape x_train was (60000, 28, 28, 1), but using load_image_dataset on jpgs is also giving me a 1D array of arrays. I'm unsure how to reshape it properly.
I think I've figured out why I was running into problems. The problem was that my `jpg` images were not of the same resolution. 

### Fix?
I normalized my images to the same resolution by adding the following lines, and a new option `normalized_size`, to `autokeras.image_supervised.read_images`:
```
import cv2
...
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)
```

Executing the following lines results in the desired output:
```
x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train", normalized_size = (100,100))
print(x_train.shape)
print(y_train.shape)
```
Output:
```
(13750, 100, 100, 3)
(13750,)
``` 

Running the training then proceeds without error.

### Pull Request

I think requiring that the images be of the same resolution is good behavior, as I've read that usually speeds up the training. I think at least one of the following functionality could be added to autokeras:
1. I think it is useful having an option in `autokeras.image_supervised.load_image_dataset`, or perhaps a separate function, to normalize an image dataset to the same resolution. However, there are possible problems that could occur. Reducing an image's resolution could cause data loss, and, depending on the way the resolution is changed, the images might be changed too much and it might throw off the training. Note that my knowledge in this area is very limited.
2. `help(autokeras.image_supervised.load_image_dataset)` could include a line indicating that images need to be of the same size.
3. Allow image datasets that have multiple resolutions. This is probably the most desirable, but I foresee that it would require changes to functions outside of `image_supervised.py`.

If one (or more) of the options seems desirable, I could code in the relevant sections and submit a pull request.

### Changed Functions

For reference, the following are the changed functions in `image_supervised.py`:
```
def read_images(img_file_names, images_dir_path, normalized_size = None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                #img = ndimage.imread(fname=img_path)
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)


def load_image_dataset(csv_file_path, images_path, normalized_size = None):
    """Load images from the files and labels from a csv file.

    Second, the dataset is a set of images and the labels are in a CSV file.
    The CSV file should contain two columns whose names are 'File Name' and 'Label'.
    The file names in the first column should match the file names of the images with extensions,
    e.g., .jpg, .png.
    The path to the CSV file should be passed through the `csv_file_path`.
    The path to the directory containing all the images should be passed through `image_path`.

    Args:
        csv_file_path: CSV file path.
        images_path: Path where images exist.

    Returns:
        x: Four dimensional numpy.ndarray. The channel dimension is the last dimension.
        y: The labels.
    """
    img_file_name, y = read_csv_file(csv_file_path)
    x = read_images(img_file_name, images_path, normalized_size = normalized_size)
    return np.array(x), np.array(y)
```


Awesome, great work! I figured it was probably due to the different resolutions. I used this code:
def proc_images():
    """
    Returns two arrays: 
        x is an array of resized images
        y is an array of labels
    """
    
    drone="yes"

    x = [] # images as arrays
    y = [] # labels Infiltration or Not_infiltration
    WIDTH = 128
    HEIGHT = 128

    for img in images:
        base = os.path.basename(img)
        finding = labels["Label"][labels["File Name"] == base].values[0]

        # Read and resize image
        full_size_image = cv2.imread(img)
        x.append(cv2.resize(full_size_image, (WIDTH,HEIGHT), interpolation=cv2.INTER_CUBIC))

        # Labels
        if drone in finding:
            #finding = str(disease)
            finding = 1
            y.append(finding)

        else:
            #finding = "Not_" + str(disease)
            finding = 0
            y.append(finding)

    return x,y

which I borrowed from a kaggle kernel: https://www.kaggle.com/crawford/resize-and-save-images-as-numpy-arrays-128x128

to resize the images, and this worked for autokeras. It made me think if there is any automated, or in principle method to find the optimal resizing parameters for any arbitrary set of images, as you rightly point out that resizing can result in information loss or distortion. I don't have a solution for that yet but if I find one I will report it.
The following changes to image_supervised.py worked for me. Used imageio.imread to bring in images and skimage.transform.resize to make all images the same dimension by giving (x, y, z) where z=channels.

I tried to make a PR but I'm a noob and I think I did it in the wrong place.

```
def read_images(img_file_names, images_dir_path, normalized_size=None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
        normalized_size: tuple specifying resize shape (x, y, z), z = channels
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                # img = ndimage.imread(fname=img_path)
                img = imageio.imread(img_path)
                img = resize(img, output_shape=normalized_size, anti_aliasing=True)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)
```
I'm getting the same problem using autokeras on colab. 
When I brought in the mnist example, after the reshape x_train was (60000, 28, 28, 1), but using load_image_dataset on jpgs is also giving me a 1D array of arrays. I'm unsure how to reshape it properly.
I think I've figured out why I was running into problems. The problem was that my `jpg` images were not of the same resolution. 

### Fix?
I normalized my images to the same resolution by adding the following lines, and a new option `normalized_size`, to `autokeras.image_supervised.read_images`:
```
import cv2
...
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)
```

Executing the following lines results in the desired output:
```
x_train, y_train = load_image_dataset(csv_file_path="data/train/labels.csv",
                                      images_path="data/train", normalized_size = (100,100))
print(x_train.shape)
print(y_train.shape)
```
Output:
```
(13750, 100, 100, 3)
(13750,)
``` 

Running the training then proceeds without error.

### Pull Request

I think requiring that the images be of the same resolution is good behavior, as I've read that usually speeds up the training. I think at least one of the following functionality could be added to autokeras:
1. I think it is useful having an option in `autokeras.image_supervised.load_image_dataset`, or perhaps a separate function, to normalize an image dataset to the same resolution. However, there are possible problems that could occur. Reducing an image's resolution could cause data loss, and, depending on the way the resolution is changed, the images might be changed too much and it might throw off the training. Note that my knowledge in this area is very limited.
2. `help(autokeras.image_supervised.load_image_dataset)` could include a line indicating that images need to be of the same size.
3. Allow image datasets that have multiple resolutions. This is probably the most desirable, but I foresee that it would require changes to functions outside of `image_supervised.py`.

If one (or more) of the options seems desirable, I could code in the relevant sections and submit a pull request.

### Changed Functions

For reference, the following are the changed functions in `image_supervised.py`:
```
def read_images(img_file_names, images_dir_path, normalized_size = None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                #img = ndimage.imread(fname=img_path)
                img = cv2.imread(img_path)
                img = cv2.resize(img, dsize=normalized_size, interpolation=cv2.INTER_CUBIC)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)


def load_image_dataset(csv_file_path, images_path, normalized_size = None):
    """Load images from the files and labels from a csv file.

    Second, the dataset is a set of images and the labels are in a CSV file.
    The CSV file should contain two columns whose names are 'File Name' and 'Label'.
    The file names in the first column should match the file names of the images with extensions,
    e.g., .jpg, .png.
    The path to the CSV file should be passed through the `csv_file_path`.
    The path to the directory containing all the images should be passed through `image_path`.

    Args:
        csv_file_path: CSV file path.
        images_path: Path where images exist.

    Returns:
        x: Four dimensional numpy.ndarray. The channel dimension is the last dimension.
        y: The labels.
    """
    img_file_name, y = read_csv_file(csv_file_path)
    x = read_images(img_file_name, images_path, normalized_size = normalized_size)
    return np.array(x), np.array(y)
```


Awesome, great work! I figured it was probably due to the different resolutions. I used this code:
def proc_images():
    """
    Returns two arrays: 
        x is an array of resized images
        y is an array of labels
    """
    
    drone="yes"

    x = [] # images as arrays
    y = [] # labels Infiltration or Not_infiltration
    WIDTH = 128
    HEIGHT = 128

    for img in images:
        base = os.path.basename(img)
        finding = labels["Label"][labels["File Name"] == base].values[0]

        # Read and resize image
        full_size_image = cv2.imread(img)
        x.append(cv2.resize(full_size_image, (WIDTH,HEIGHT), interpolation=cv2.INTER_CUBIC))

        # Labels
        if drone in finding:
            #finding = str(disease)
            finding = 1
            y.append(finding)

        else:
            #finding = "Not_" + str(disease)
            finding = 0
            y.append(finding)

    return x,y

which I borrowed from a kaggle kernel: https://www.kaggle.com/crawford/resize-and-save-images-as-numpy-arrays-128x128

to resize the images, and this worked for autokeras. It made me think if there is any automated, or in principle method to find the optimal resizing parameters for any arbitrary set of images, as you rightly point out that resizing can result in information loss or distortion. I don't have a solution for that yet but if I find one I will report it.
The following changes to image_supervised.py worked for me. Used imageio.imread to bring in images and skimage.transform.resize to make all images the same dimension by giving (x, y, z) where z=channels.

I tried to make a PR but I'm a noob and I think I did it in the wrong place.

```
def read_images(img_file_names, images_dir_path, normalized_size=None):
    """Read the images from the path and return their numpy.ndarray instance.
        Return a numpy.ndarray instance containing the training data.

    Args:
        img_file_names: List containing images names.
        images_dir_path: Path to the directory containing images.
        normalized_size: tuple specifying resize shape (x, y, z), z = channels
    """
    x_train = []
    if os.path.isdir(images_dir_path):
        for img_file in img_file_names:
            img_path = os.path.join(images_dir_path, img_file)
            if os.path.exists(img_path):
                # img = ndimage.imread(fname=img_path)
                img = imageio.imread(img_path)
                img = resize(img, output_shape=normalized_size, anti_aliasing=True)
                if len(img.shape) < 3:
                    img = img[..., np.newaxis]
                x_train.append(img)
            else:
                raise ValueError("%s image does not exist" % img_file)
    else:
        raise ValueError("Directory containing images does not exist")
    return np.asanyarray(x_train)
```

Resolving patch:
diff --git a/autokeras/utils.py b/autokeras/utils.py
--- a/autokeras/utils.py
+++ b/autokeras/utils.py
@@ -5,9 +5,9 @@
 import tempfile
 import zipfile
 
+import imageio
 import requests
 import torch
-from scipy import ndimage
 
 from autokeras.constant import Constant
 
@@ -195,5 +195,5 @@ def read_csv_file(csv_file_path):
 
 
 def read_image(img_path):
-    img = ndimage.imread(fname=img_path)
-    return img
\ No newline at end of file
+    img = imageio.imread(uri=img_path)
+    return img


Test patch: