Repository: keras-team/autokeras

Issue:
Model evaluation on test set yields different results on the same model
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description
Test set evaluation yields different results if re-run.

### Reproducing Steps
Steps to reproduce the behavior:
re-run multiple times:
```python
y_pred = clf.evaluate(X_test, y_test)
print(y_pred)
```
will yield different results:
```
0.0992
0.1032
0.101
0.0989
```

Further, using manual evaluation:
```python
y_prediction = clf.predict(x_test=X_test)
from sklearn.metrics import accuracy_score
accuracy_score(y_pred=y_prediction, y_true=y_test)
```
leads to different results as well.  It looks like the model either uses some random function (AFAIK all: ```random_states=42``` in the source code) or there is some major error in the pipeline that causes different predictions of the test set all the time.

### Expected Behavior
Final evaluation on a test set should not lead to different results using the same model on the same data.

### Setup Details
Include the details about the versions of:
 - OS type and version: Linux
 - Python: 3.6.5
 - autokeras: 0.2.11
 - scikit-learn:0.19.1
 - numpy:1.14.5
 - keras: 2.2.2
 - scipy:1.1.0
 - tensorflow: 1.10.0
 - pytorch:0.4.1

### Additional context
This error is verified on MNIST and Fashion-MNIST.


Discussion:

Resolving patch:
diff --git a/autokeras/preprocessor.py b/autokeras/preprocessor.py
--- a/autokeras/preprocessor.py
+++ b/autokeras/preprocessor.py
@@ -71,23 +71,31 @@ def transform_train(self, data, targets=None, batch_size=None):
         common_list = [Normalize(torch.Tensor(self.mean), torch.Tensor(self.std))]
         compose_list = augment_list + common_list
 
-        return self._transform(batch_size, compose_list, data, targets)
+        dataset = self._transform(compose_list, data, targets)
+
+        if batch_size is None:
+            batch_size = Constant.MAX_BATCH_SIZE
+        batch_size = min(len(data), batch_size)
+
+        return DataLoader(dataset, batch_size=batch_size, shuffle=True)
 
     def transform_test(self, data, targets=None, batch_size=None):
         common_list = [Normalize(torch.Tensor(self.mean), torch.Tensor(self.std))]
         compose_list = common_list
 
-        return self._transform(batch_size, compose_list, data, targets)
+        dataset = self._transform(compose_list, data, targets)
 
-    def _transform(self, batch_size, compose_list, data, targets):
         if batch_size is None:
             batch_size = Constant.MAX_BATCH_SIZE
         batch_size = min(len(data), batch_size)
+
+        return DataLoader(dataset, batch_size=batch_size, shuffle=False)
+
+    def _transform(self, compose_list, data, targets):
         data = data / self.max_val
         data = torch.Tensor(data.transpose(0, 3, 1, 2))
         data_transforms = Compose(compose_list)
-        dataset = MultiTransformDataset(data, targets, data_transforms)
-        return DataLoader(dataset, batch_size=batch_size, shuffle=True)
+        return MultiTransformDataset(data, targets, data_transforms)
 
 
 class MultiTransformDataset(Dataset):


Test patch: