Repository: OpenNMT/OpenNMT-tf

Issue:
Question about evaluation metrics
Hello,
I use opennmt-tf to train models for different tasks, and I want to evaluate the effectiveness of a model to know how long should I take for training. But I’m not sure in how to evaluate sequence tagging model or other sequence to sequence model using validation data to measure accuracy, recall, precision, and F1-score. I have tried to train on a supported tagging scheme (BIOES) by adding 
```
train: 
  tagging_scheme: BIOES
```
to parameters in YMAL file, but I could’t find where the additional evaluation metrics are computed. Could you provide some tutorial for me?

Thanks.


Discussion:
Hi,

Thanks for the question. Currently, the tagging scheme should be configured in the model definition. If you used the `SeqTagger` model from the catalog, you should copy its definition to another file and customize it, for example:

```python
import opennmt as onmt

def model():
  return onmt.models.SequenceTagger(
        inputter=onmt.inputters.MixedInputter([
            onmt.inputters.WordEmbedder(
                vocabulary_file_key="words_vocabulary",
                embedding_size=None,
                embedding_file_key="words_embedding",
                trainable=True),
            onmt.inputters.CharConvEmbedder(
                vocabulary_file_key="chars_vocabulary",
                embedding_size=30,
                num_outputs=30,
                kernel_size=3,
                stride=1,
                dropout=0.5)],
            dropout=0.5),
        encoder=onmt.encoders.BidirectionalRNNEncoder(
            num_layers=1,
            num_units=400,
            reducer=onmt.layers.ConcatReducer(),
            dropout=0.5,
            residual_connections=False),
        labels_vocabulary_file_key="tags_vocabulary",
        tagging_scheme="bioes",
        crf_decoding=True)
```

The metrics will then be automatically computed.

But you are correct, this parameter should be configurable in the YAML configuration as it is related to the data, not the model structure. I add this change to my list.
If the tagging scheme is successfully set, would the evaluation report be printed every 100 training steps along with loss, or written in a text file after an epoch of training?
Yes, like this:

```text
INFO:tensorflow:Finished evaluation at 2019-03-04-09:43:49
INFO:tensorflow:Saving dict for global step 6800: accuracy = 0.9654079, f1 = 0.9654079, global_step = 6800, loss = 2.480431, precision = 0.9654079, recall = 0.9654079
```

They also appear in TensorBoard.

Resolving patch:
diff --git a/opennmt/models/sequence_tagger.py b/opennmt/models/sequence_tagger.py
--- a/opennmt/models/sequence_tagger.py
+++ b/opennmt/models/sequence_tagger.py
@@ -49,6 +49,10 @@ def __init__(self,
     else:
       self.tagging_scheme = None
 
+  def initialize(self, metadata):
+    self.tagging_scheme = metadata.get("tagging_scheme", self.tagging_scheme)
+    super(SequenceTagger, self).initialize(metadata)
+
   def _call(self, features, labels, params, mode):
     training = mode == tf.estimator.ModeKeys.TRAIN
     length = self.features_inputter.get_length(features)


Test patch: