Repository: keras-team/autokeras

Issue:
Model evaluation - part 3
<!---
**If you are reporting a bug:**
*  Verify that your issue is not being currently addressed by other issues or pull requests.
*  Please note that Auto-Keras is only compatible with **Python 3.6**.
*  Tag the issue with the `bug report` tag.
-->

### Bug Description
<!---
A clear and concise description of what the bug is.
-->
The issue remains that the test accuracy is 1.0.

It seems like the following two lines in ```metrics.py```
```python
        prediction = list(map(lambda x: x.argmax(), prediction))
        target = list(map(lambda x: x.argmax(), target))
```
mess it up and therefore have to be removed. If they are removed, then prediction and target are passed to ```accuracy_score``` directly  which is then yields the same as if I use it manually.

*I have not submitted a pull request since I am not entirely sure if that will mess up some other part of autokeras. As of now I would say: No. I am not sure if ```def _test``` in ```utils.py``` is used ever (for something relevant). But you certainly know the source code much better*.


### Reproducing Steps
Steps to reproduce the behavior:
  * Step 1: ...
  * Step 2: ...

### Expected Behavior
<!---
A clear and concise description of what you expected to happen.
-->
Show the real accuracy.


### Setup Details
Include the details about the versions of:
 - OS type and version:
 - Python: 
 - autokeras: 0.2.12, master, develop
 - scikit-learn:
 - numpy:
 - keras:
 - scipy:
 - tensorflow:
 - pytorch:

### Additional context
<!---
Add any other context about the problem here.
-->



Discussion:
@swenkel thank you for letting us know. That line should not be changed since it is for the model trainer mainly. We are investigating the issue. Thanks.

<sub>Sent with <a href="http://githawk.com">GitHawk</a></sub>

Resolving patch:
diff --git a/autokeras/image_supervised.py b/autokeras/image_supervised.py
--- a/autokeras/image_supervised.py
+++ b/autokeras/image_supervised.py
@@ -285,7 +285,7 @@ def inverse_transform_y(self, output):
     def evaluate(self, x_test, y_test):
         """Return the accuracy score between predict value and `y_test`."""
         y_predict = self.predict(x_test)
-        return self.metric().compute(y_test, y_predict)
+        return self.metric().evaluate(y_test, y_predict)
 
     def save_searcher(self, searcher):
         pickle.dump(searcher, open(os.path.join(self.path, 'searcher'), 'wb'))
@@ -421,4 +421,4 @@ def inverse_transform_y(self, output):
     def evaluate(self, x_test, y_test):
         """Return the accuracy score between predict value and `y_test`."""
         y_predict = self.predict(x_test)
-        return self.metric().compute(y_test, y_predict)
+        return self.metric().evaluate(y_test, y_predict)
diff --git a/autokeras/metric.py b/autokeras/metric.py
--- a/autokeras/metric.py
+++ b/autokeras/metric.py
@@ -15,6 +15,11 @@ def higher_better(cls):
     def compute(cls, prediction, target):
         pass
 
+    @classmethod
+    @abstractmethod
+    def evaluate(cls, prediction, target):
+        pass
+
 
 class Accuracy(Metric):
     @classmethod
@@ -25,6 +30,10 @@ def higher_better(cls):
     def compute(cls, prediction, target):
         prediction = list(map(lambda x: x.argmax(), prediction))
         target = list(map(lambda x: x.argmax(), target))
+        return cls.evaluate(prediction, target)
+
+    @classmethod
+    def evaluate(cls, prediction, target):
         return accuracy_score(prediction, target)
 
 
@@ -35,5 +44,8 @@ def higher_better(cls):
 
     @classmethod
     def compute(cls, prediction, target):
-        return mean_squared_error(prediction, target)
+        return cls.evaluate(prediction, target)
 
+    @classmethod
+    def evaluate(cls, prediction, target):
+        return mean_squared_error(prediction, target)


Test patch: