Repository: keras-team/autokeras

Issue:
Compress image before feed to NN if image is too large
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description
<!---
A clear and concise description of what the bug is.
-->

I did a cat and dog image classification task, only used 100 pictures to train, and a RuntimeError appeared on a single 8GB system: $ Torch: not enough memory: you tried to allocate 0GB. The image input is 100 * 224 * 224 *3. This problem occurs. The input standard for reducing the image is 100 * 32 * 32 * 3. This problem is not the case, but the accuracy is very low for the training result. The final evaluation value of the second classification. Less than 0.6.
The specific issues are as follows:

```
╒==============================================╕
|               Training model 1               |
╘==============================================╛
Using TensorFlow backend.
Current Epoch:   0%|                             | 0/1 [00:00<?, ? batch/s]Exception ignored in: <bound method tqdm.__del__ of Current Epoch:   0%|                             | 0/1 [07:59<?, ? batch/s]>                                                   
Traceback (most recent call last):
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/tqdm/_tqdm.py", line 885, in __del__
    self.close()
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/tqdm/_tqdm.py", line 1090, in close
    self._decr_instances(self)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/tqdm/_tqdm.py", line 454, in _decr_instances
    cls.monitor.exit()
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/tqdm/_monitor.py", line 52, in exit
    self.join()
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/threading.py", line 1053, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/autokeras/search.py", line 296, in train
    verbose=verbose).train_model(**trainer_args)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/autokeras/model_trainer.py", line 103, in train_model
    self._train()
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/autokeras/model_trainer.py", line 147, in _train
    outputs = self.model(inputs)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/autokeras/graph.py", line 610, in forward
    temp_tensor = torch_layer(edge_input_tensor)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 66, in forward
    exponential_average_factor, self.eps)
  File "/root/anaconda3/envs/automl-formal/lib/python3.6/site-packages/torch/nn/functional.py", line 1254, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: $ Torch: not enough memory: you tried to allocate 0GB. Buy new RAM! at [/pytorch/aten/src/TH/THGeneral.cpp:204]
```

### Reproducing Steps
Steps to reproduce the behavior:
  * Step 1: Choose 1 CPU to use (among 8 CPUs):
  * Step 2: Run the Cat and Dog Wars classification task, select 50 cats and 50 dogs for the classification task, enter the image parameters: 100 * 224 * 224 *3

### Expected Behavior
<!---
A clear and concise description of what you expected to happen.
-->
Succeed in running the classification task without this error
Image classification accuracy increased to an acceptable greater than 80%

### Setup Details
Include the details about the versions of:
 - OS type and version:
 - Python: 3.6
 - autokeras: <!--- e.g. 0.2.1, 0.2.0, master-->0.2.14
 - scikit-learn:0.19.1
 - numpy:1.14.5
 - keras:2.2.2
 - scipy:1.1.0
 - tensorflow:1.10.1
 - pytorch:

### Additional context
<!---
Add any other context about the problem here.
-->



Discussion:
@satyakesav 
First, you may examine whether the data is loaded to GPU memory batch by batch, or all together.
If is loaded all together, try to change it to load batch by batch.
https://github.com/jhfjhfj1/autokeras/blob/master/autokeras/model_trainer.py#L219

Second, if the image is oversize (the total size is larger than 64*64*3), resize the image before feed into the NN, according to their original length/width ratio. Keep the size just right.

Thanks.
@aojue1109 
I was trying to reproduce the error that you reported using the following scenario but unable to do that.
My training set has 2000 images (1000 each of cat and dog images from Cat vs Dog image classification task). The images have arbitrary sizes in the training set and hence I resized all of them to 324 x 324 x 3 since you have mentioned that the images in your scenario are the same. I have verified that I am training my network on a CPU not GPU. 

The following is the code that I used
```
x_train, y_train = load_image_dataset(csv_file_path="labels.csv", images_path="data")

x_temp = []
for x in x_train:
    x_temp.append(scipy.misc.imresize(x, [324, 324, 3]))
x_train = numpy.array(x_temp)

if __name__ == '__main__':
    clf = ImageClassifier(verbose=True, augment=False)
    clf.fit(x_train, y_train, time_limit=12 * 60 * 60)

```
Despite the scenario, my model is training without any failure but it is insanely slow because of the obvious reasons. 

Can you please let me know the exact steps to reproduce the issue. Before doing that, it will be good If you can verify whether the issue is reproducible in your environment with the latest pull request. Please let me know.

Thanks,
Satya

Resolving patch:
diff --git a/autokeras/constant.py b/autokeras/constant.py
--- a/autokeras/constant.py
+++ b/autokeras/constant.py
@@ -52,6 +52,10 @@ class Constant:
     PRE_TRAIN_FILE_LINK = "http://nlp.stanford.edu/data/glove.6B.zip"
     PRE_TRAIN_FILE_NAME = "glove.6B.100d.txt"
 
+    # Image Resize
+
+    MAX_IMAGE_SIZE = 128 * 128
+
     # SYS Constant
 
     SYS_LINUX = 'linux'
diff --git a/autokeras/image/image_supervised.py b/autokeras/image/image_supervised.py
--- a/autokeras/image/image_supervised.py
+++ b/autokeras/image/image_supervised.py
@@ -14,7 +14,7 @@
 from autokeras.preprocessor import OneHotEncoder, ImageDataTransformer
 from autokeras.supervised import Supervised, PortableClass
 from autokeras.utils import has_file, pickle_from_file, pickle_to_file, temp_folder_generator, validate_xy, \
-    read_csv_file, read_image
+    read_csv_file, read_image, compute_image_resize_params, resize_image_data
 
 
 def read_images(img_file_names, images_dir_path):
@@ -116,6 +116,9 @@ def __init__(self, verbose=False, path=None, resume=False, searcher_args=None, a
             self.augment = augment
             self.cnn = CnnModule(self.loss, self.metric, searcher_args, path, verbose)
 
+        self.resize_height = None
+        self.resize_width = None
+
     @property
     @abstractmethod
     def metric(self):
@@ -128,6 +131,13 @@ def loss(self):
 
     def fit(self, x, y, x_test=None, y_test=None, time_limit=None):
         x = np.array(x)
+
+        if len(x.shape) != 0 and len(x[0].shape) == 3:
+            self.resize_height, self.resize_width = compute_image_resize_params(x)
+            x = resize_image_data(x, self.resize_height, self.resize_width)
+            if x_test is not None:
+                x_test = resize_image_data(x_test, self.resize_height, self.resize_width)
+
         y = np.array(y).flatten()
         validate_xy(x, y)
         y = self.transform_y(y)
@@ -192,6 +202,8 @@ def inverse_transform_y(self, output):
 
     def evaluate(self, x_test, y_test):
         """Return the accuracy score between predict value and `y_test`."""
+        if len(x_test.shape) != 0 and len(x_test[0].shape) == 3:
+            x_test = resize_image_data(x_test, self.resize_height, self.resize_width)
         y_predict = self.predict(x_test)
         return self.metric().evaluate(y_test, y_predict)
 
@@ -209,6 +221,11 @@ def final_fit(self, x_train, y_train, x_test, y_test, trainer_args=None, retrain
         if trainer_args is None:
             trainer_args = {'max_no_improvement_num': 30}
 
+        if len(x_train.shape) != 0 and len(x_train[0].shape) == 3:
+            x_train = resize_image_data(x_train, self.resize_height, self.resize_width)
+            if x_test is not None:
+                x_test = resize_image_data(x_test, self.resize_height, self.resize_width)
+
         y_train = self.transform_y(y_train)
         y_test = self.transform_y(y_test)
 
@@ -230,7 +247,8 @@ def export_autokeras_model(self, model_file_name):
                                                  y_encoder=self.y_encoder,
                                                  data_transformer=self.data_transformer,
                                                  metric=self.metric,
-                                                 inverse_transform_y_method=self.inverse_transform_y)
+                                                 inverse_transform_y_method=self.inverse_transform_y,
+                                                 resize_params=(self.resize_height, self.resize_width))
         pickle_to_file(portable_model, model_file_name)
 
 
@@ -302,7 +320,7 @@ def __init__(self, **kwargs):
 
 
 class PortableImageSupervised(PortableClass):
-    def __init__(self, graph, data_transformer, y_encoder, metric, inverse_transform_y_method):
+    def __init__(self, graph, data_transformer, y_encoder, metric, inverse_transform_y_method, resize_params):
         """Initialize the instance.
         Args:
             graph: The graph form of the learned model
@@ -312,6 +330,8 @@ def __init__(self, graph, data_transformer, y_encoder, metric, inverse_transform
         self.y_encoder = y_encoder
         self.metric = metric
         self.inverse_transform_y_method = inverse_transform_y_method
+        self.resize_height = resize_params[0]
+        self.resize_width = resize_params[1]
 
     def predict(self, x_test):
         """Return predict results for the testing data.
@@ -324,6 +344,7 @@ def predict(self, x_test):
         """
         if Constant.LIMIT_MEMORY:
             pass
+
         test_loader = self.data_transformer.transform_test(x_test)
         model = self.graph.produce_model()
         model.eval()
@@ -340,5 +361,7 @@ def inverse_transform_y(self, output):
 
     def evaluate(self, x_test, y_test):
         """Return the accuracy score between predict value and `y_test`."""
+        if len(x_test.shape) != 0 and len(x_test.shape) == 3:
+            x_test = resize_image_data(x_test, self.resize_height, self.resize_width)
         y_predict = self.predict(x_test)
         return self.metric().evaluate(y_test, y_predict)
diff --git a/autokeras/utils.py b/autokeras/utils.py
--- a/autokeras/utils.py
+++ b/autokeras/utils.py
@@ -7,9 +7,12 @@
 
 import warnings
 import imageio
+import numpy
 import requests
+from skimage.transform import resize
 import torch
 import subprocess
+
 from autokeras.constant import Constant
 
 
@@ -171,6 +174,55 @@ def read_image(img_path):
     return img
 
 
+def compute_image_resize_params(data):
+    """Compute median height and width of all images in data. These
+    values are used to resize the images at later point. Number of
+    channels do not change from the original images. Currently, only
+    2-D images are supported.
+
+    Args:
+        data: 2-D Image data with shape N x H x W x C.
+
+    Returns:
+        median height: Median height of all images in the data.
+        median width: Median width of all images in the data.
+    """
+    median_height, median_width = numpy.median(numpy.array(list(map(lambda x: x.shape, data))), axis=0)[:2]
+
+    if median_height * median_width > Constant.MAX_IMAGE_SIZE:
+        reduction_factor = numpy.sqrt(median_height * median_width / Constant.MAX_IMAGE_SIZE)
+        median_height = median_height / reduction_factor
+        median_width = median_width / reduction_factor
+
+    return int(median_height), int(median_width)
+
+
+def resize_image_data(data, h, w):
+    """Resize all images in data to size h x w x c, where h is the height,
+    w is the width and c is the number of channels. The number of channels
+    c does not change from data. The function supports only 2-D image data.
+
+    Args:
+        data: 2-D Image data with shape N x H x W x C.
+        h: Image resize height.
+        w: Image resize width.
+
+    Returns:
+        data: Resize data.
+    """
+
+    output_data = []
+    for im in data:
+        if len(im.shape) != 3:
+            return data
+        output_data.append(resize(image=im,
+                                  output_shape=(h, w, im.shape[-1]),
+                                  mode='edge',
+                                  preserve_range=True))
+
+    return numpy.array(output_data)
+
+
 def get_system():
     """
     Get the current system environment. If the current system is not supported,


Test patch:
diff --git a/tests/image/temp_test.py b/tests/image/temp_test.py
new file mode 100644
--- /dev/null
+++ b/tests/image/temp_test.py
@@ -0,0 +1,51 @@
+from unittest.mock import patch
+
+import pytest
+
+from autokeras.image.image_supervised import *
+from tests.common import clean_dir, MockProcess, simple_transform, mock_train, TEST_TEMP_DIR
+
+
+@patch('torch.multiprocessing.get_context', side_effect=MockProcess)
+@patch('autokeras.search.ModelTrainer.train_model', side_effect=mock_train)
+def test_fit_predict(_, _1):
+    Constant.MAX_ITER_NUM = 1
+    Constant.MAX_MODEL_NUM = 4
+    Constant.SEARCH_MAX_ITER = 1
+    Constant.T_MIN = 0.8
+    Constant.DATA_AUGMENTATION = False
+	
+    clf = ImageClassifier(path=TEST_TEMP_DIR, verbose=True)
+    train_x = np.random.rand(100, 25, 25, 1)
+    train_y = np.random.randint(0, 5, 100)
+    clf.fit(train_x, train_y)
+    results = clf.predict(train_x)
+    assert all(map(lambda result: result in train_y, results))
+
+    clf = ImageClassifier1D(path=TEST_TEMP_DIR, verbose=True)
+    train_x = np.random.rand(100, 25, 1)
+    train_y = np.random.randint(0, 5, 100)
+    clf.fit(train_x, train_y)
+    results = clf.predict(train_x)
+    assert all(map(lambda result: result in train_y, results))
+
+    clf = ImageClassifier3D(path=TEST_TEMP_DIR, verbose=True)
+    train_x = np.random.rand(100, 25, 25, 25, 1)
+    train_y = np.random.randint(0, 5, 100)
+    clf.fit(train_x, train_y)
+    results = clf.predict(train_x)
+    assert all(map(lambda result: result in train_y, results))
+
+    clf = ImageRegressor1D(path=TEST_TEMP_DIR, verbose=True)
+    train_x = np.random.rand(100, 25, 1)
+    train_y = np.random.randint(0, 5, 100)
+    clf.fit(train_x, train_y)
+    results = clf.predict(train_x)
+    assert len(results) == len(train_y)
+
+    clf = ImageRegressor3D(path=TEST_TEMP_DIR, verbose=True)
+    train_x = np.random.rand(100, 25, 25, 25, 1)
+    train_y = np.random.randint(0, 5, 100)
+    clf.fit(train_x, train_y)
+    results = clf.predict(train_x)
+    assert len(results) == len(train_y)
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,8 +1,10 @@
+import numpy
 import subprocess
 from unittest.mock import patch
 
 from autokeras.constant import Constant
-from autokeras.utils import temp_folder_generator, download_file, get_system, get_device
+from autokeras.utils import temp_folder_generator, download_file, get_system, get_device, compute_image_resize_params, \
+    resize_image_data
 from tests.common import clean_dir, TEST_TEMP_DIR, mock_nvidia_smi_output
 
 
@@ -47,6 +49,33 @@ def test_fetch(_):
     clean_dir(TEST_TEMP_DIR)
 
 
+def test_compute_image_resize_params():
+    # Case-1: Compute median height and width for smaller images.
+    data = numpy.array([numpy.random.randint(256, size=(10, 10, 3)),
+                        numpy.random.randint(256, size=(20, 20, 3)),
+                        numpy.random.randint(256, size=(30, 30, 3)),
+                        numpy.random.randint(256, size=(40, 40, 3))])
+    resize_height, resize_width = compute_image_resize_params(data)
+    assert resize_height == 25
+    assert resize_width == 25
+
+    modified_data = resize_image_data(data, resize_height, resize_width)
+    for image in modified_data:
+        assert image.shape == (25, 25, 3)
+
+    # Case-2: Resize to max size for larger images.
+    data = numpy.array([numpy.random.randint(256, size=(int(numpy.sqrt(Constant.MAX_IMAGE_SIZE)+1),
+                                                        int(numpy.sqrt(Constant.MAX_IMAGE_SIZE)+1),
+                                                        3))])
+    resize_height, resize_width = compute_image_resize_params(data)
+    assert resize_height == int(numpy.sqrt(Constant.MAX_IMAGE_SIZE))
+    assert resize_width == int(numpy.sqrt(Constant.MAX_IMAGE_SIZE))
+
+    modified_data = resize_image_data(data, resize_height, resize_width)
+    for image in modified_data:
+        assert image.shape == (resize_height, resize_width, 3)
+
+
 def test_get_system():
     sys_name = get_system()
     assert \