Repository: keras-team/keras

Issue:
Keras fails silently if the name of the output is not correct when using metrics.
Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on [StackOverflow](http://stackoverflow.com/questions/tagged/keras) or [join the Keras Slack channel](https://keras-slack-autojoin.herokuapp.com/) and ask there instead of filing a GitHub issue.

Thank you!

- [x] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

- [x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found [here](https://www.tensorflow.org/get_started/os_setup).

- [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

- [x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

```python
from keras import Model
from keras.layers import Input, Dense

input_tensor = Input((8,))
x = Dense(4)(input_tensor)
model = Model(input_tensor, x)

model.compile('sgd', 'mse', metrics={'random_inexistant_output_name': 'mae'})
# the previous line should throw an error listing the output names available and 
# printing the output name that  the user provided. It currently fails silently.
```

We should have an error message. Currently, the training works and the metric is just ignored.


Discussion:

Resolving patch:
diff --git a/keras/engine/training_utils.py b/keras/engine/training_utils.py
--- a/keras/engine/training_utils.py
+++ b/keras/engine/training_utils.py
@@ -322,6 +322,11 @@ def collect_metrics(metrics, output_names):
         return [copy.copy(metrics) for _ in output_names]
     elif isinstance(metrics, dict):
         nested_metrics = []
+        if not set(metrics.keys()).issubset(set(output_names)):
+            unknown_output_names = list(set(metrics.keys()) - set(output_names))
+            warnings.warn('Invalid layer name for metric computations: '
+                          '{}. Available names are {}.'
+                          .format(unknown_output_names, output_names))
         for name in output_names:
             output_metrics = metrics.get(name, [])
             output_metrics = to_list(output_metrics)


Test patch:
diff --git a/tests/keras/engine/test_training.py b/tests/keras/engine/test_training.py
--- a/tests/keras/engine/test_training.py
+++ b/tests/keras/engine/test_training.py
@@ -734,6 +734,32 @@ def test_check_bad_shape():
     assert 'targets to have the same shape' in str(exc)
 
 
+@pytest.mark.parametrize('input_metrics,expected_output', [
+    (None, [[], []]),
+    (['mse', 'mae'], [['mse', 'mae'], ['mse', 'mae']]),
+    ({'layer_1': 'mae', 'layer_2': 'mse'}, [['mae'], ['mse']]),
+])
+def test_collect_metrics(input_metrics, expected_output):
+    output_names = ['layer_1', 'layer_2']
+
+    output_metrics = training_utils.collect_metrics(input_metrics,
+                                                    output_names)
+    assert output_metrics == expected_output
+
+
+def test_collect_metrics_with_invalid_metrics_format():
+    with pytest.raises(TypeError):
+        training_utils.collect_metrics({'a', 'set', 'type'}, [])
+
+
+def test_collect_metrics_with_invalid_layer_name():
+    with pytest.warns(Warning) as w:
+        training_utils.collect_metrics({'unknown_layer': 'mse'}, ['layer_1'])
+
+    warning_raised = all(['unknown_layer' in str(w_.message) for w_ in w])
+    assert warning_raised, 'Warning was raised for unknown_layer'
+
+
 @pytest.mark.skipif(K.backend() != 'tensorflow',
                     reason='Requires TensorFlow backend')
 def test_model_with_input_feed_tensor():