internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: OpenNMT/OpenNMT-tf

Issue:
Error when exporting multi feature model
There seems to be small issue with exporting a model having multiple input features:
Traceback (most recent call last):
  File "opennmt/bin/main.py", line 201, in <module>
    main()
  File "opennmt/bin/main.py", line 190, in main
    export_dir_base=args.export_dir_base)
  File "/home/ari/tf/Onmt-1211/OpenNMT-tf/opennmt/runner.py", line 456, in export
    _ = model.features_inputter.make_dataset(data_config["train_features_file"])
NameError: name 'model' is not defined



Discussion:
Thanks for reporting.

Resolving patch:
diff --git a/opennmt/estimator.py b/opennmt/estimator.py
--- a/opennmt/estimator.py
+++ b/opennmt/estimator.py
@@ -8,11 +8,13 @@
 from opennmt.utils import parallel
 
 
-def make_serving_input_fn(model):
+def make_serving_input_fn(model, metadata=None):
   """Returns the serving input function.
 
   Args:
     model: An initialized :class:`opennmt.models.model.Model` instance.
+    metadata: Optional data configuration (to be removed). Some inputters
+      currently require to peek into some data files to infer input sizes.
 
   Returns:
     A callable that returns a ``tf.estimator.export.ServingInputReceiver``.
@@ -20,6 +22,11 @@ def make_serving_input_fn(model):
 
   def _fn():
     local_model = copy.deepcopy(model)
+    # This is a hack for SequenceRecordInputter that currently infers the input
+    # depth from the data files.
+    # TODO: This function should not require the training data.
+    if metadata is not None and "train_features_file" in metadata:
+      _ = local_model.features_inputter.make_dataset(metadata["train_features_file"])
     return local_model.features_inputter.get_serving_input_receiver()
 
   return _fn
diff --git a/opennmt/runner.py b/opennmt/runner.py
--- a/opennmt/runner.py
+++ b/opennmt/runner.py
@@ -448,16 +448,9 @@ def export(self, checkpoint_path=None, export_dir_base=None):
         # with the behavior of tf.estimator.Exporter.
         kwargs["strip_default_attrs"] = True
 
-    # This is a hack for SequenceRecordInputter that currently infers the input
-    # depth from the data files.
-    # TODO: This method should not require the training data.
-    data_config = self._config["data"]
-    if  "train_features_file" in data_config:
-      _ = model.features_inputter.make_dataset(data_config["train_features_file"])
-
     return export_fn(
         export_dir_base,
-        estimator_util.make_serving_input_fn(self._model),
+        estimator_util.make_serving_input_fn(self._model, metadata=self._config["data"]),
         assets_extra=self._get_model_assets(),
         checkpoint_path=checkpoint_path,
         **kwargs)


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
AttributeError: 'int' object has no attribute 'dtype'
![image](https://user-images.githubusercontent.com/4915869/54529419-d2bf2e00-49ba-11e9-8d20-ee531e469de4.png)



Discussion:
Right, it should be `self.labels_inputter.dtype`. Feel free to send a PR.

Resolving patch:
diff --git a/opennmt/models/sequence_to_sequence.py b/opennmt/models/sequence_to_sequence.py
--- a/opennmt/models/sequence_to_sequence.py
+++ b/opennmt/models/sequence_to_sequence.py
@@ -161,7 +161,7 @@ def _build(self):
           self.labels_inputter.vocabulary_size,
           weight=self.labels_inputter.embedding,
           transpose=True,
-          dtype=self.labels_inputter.vocabulary_size.dtype)
+          dtype=self.labels_inputter.dtype)
       with tf.name_scope(tf.get_variable_scope().name + "/"):
         self.output_layer.build([None, self.decoder.output_size])
 


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Problem in sharing embeddings
I noticed that when I use DualTransformer with sharing parameters between Encoders and also setting sharing level to 'ALL', two sets of embeddings are created with these prefixes:

> **transformer//w_embs**
> **transformer/shared_embeddings/w_embs**

However, it should be one for everything.


Discussion:
I tried to change
`def _get_shared_name(self):
    return ""`
to:
`def _get_shared_name(self):
    return "shared_embeddings"`

in `ParallelInputter` but I'm getting **Duplicate Node** error.

Do you know a quick fix for it?
The best way I found is reimplementing `build()` function in `ExampleInputter`.
Thanks for reporting. Do you have a working fix that you can share?
I sent a pull request.

It works for DualTransformer, but I'm not sure for other use cases.
Thanks. This indeed needs a more general fix: the issue is that the `built` flag is not properly set for `MultiInputter` classes and weights are recreated when the layer is first called. I will push an appropriate fix to that.

Resolving patch:
diff --git a/opennmt/inputters/inputter.py b/opennmt/inputters/inputter.py
--- a/opennmt/inputters/inputter.py
+++ b/opennmt/inputters/inputter.py
@@ -302,6 +302,12 @@ def get_leaf_inputters(self):
         inputters.append(inputter)
     return inputters
 
+  def __getattribute__(self, name):
+    if name == "built":
+      return all(inputter.built for inputter in self.inputters)
+    else:
+      return super(MultiInputter, self).__getattribute__(name)
+
   def initialize(self, metadata, asset_dir=None, asset_prefix=""):
     for i, inputter in enumerate(self.inputters):
       inputter.initialize(metadata, asset_prefix="%s%d_" % (asset_prefix, i + 1))


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Why its forced an external evaluator  with save_eval_predictions
right know if I want to save the predictions file. I have to setup an external_evaluator, but if for 
example I'm doing a classification or tagger model, none of those external evals helps me 
but I need to save the predictions into a file.

Could be possible to save the predictions file and not setup an external eval?

because right know its crashing here:

File "/Users/sergio.hurtado/gitlab/NaturalLanguageRecognition/venv/lib/python3.6/site-packages/opennmt/utils/evaluator.py", line 187, in make_scorers
    name = name.lower()
AttributeError: 'NoneType' object has no attribute 'lower'

since I want to save the file but I dont want an evaluator external so its None 


Discussion:
Thanks for reporting. The code should allow it.
you are welcome :)

Resolving patch:
diff --git a/opennmt/runner.py b/opennmt/runner.py
--- a/opennmt/runner.py
+++ b/opennmt/runner.py
@@ -172,19 +172,21 @@ def is_chief(self):
     return cluster_spec["task"]["type"] == "chief"
 
   def _make_eval_prediction_hooks_fn(self):
-    if (not self._config["eval"].get("save_eval_predictions", False)
-        and self._config["eval"].get("external_evaluators") is None):
+    external_scorers = self._config["eval"].get("external_evaluators")
+    if not self._config["eval"].get("save_eval_predictions", False) and external_scorers is None:
       return None
     if self._model.unsupervised:
       raise RuntimeError("This model does not support saving evaluation predictions")
     save_path = os.path.join(self._config["model_dir"], "eval")
     if not tf.gfile.Exists(save_path):
       tf.gfile.MakeDirs(save_path)
-    scorers = evaluator.make_scorers(self._config["eval"].get("external_evaluators"))
-    external_evaluator = evaluator.ExternalEvaluator(
-        labels_file=self._config["data"]["eval_labels_file"],
-        output_dir=save_path,
-        scorers=scorers)
+    if external_scorers is not None:
+      external_evaluator = evaluator.ExternalEvaluator(
+          labels_file=self._config["data"]["eval_labels_file"],
+          output_dir=save_path,
+          scorers=evaluator.make_scorers(external_scorers))
+    else:
+      external_evaluator = None
     return lambda predictions: [
         hooks.SaveEvaluationPredictionHook(
             self._model,


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Fine-tuning only a subset of parameters
Hi,

Is there the possibility of optimizing only a subset of parameters in the computation graph?


Discussion:
Hi,

Currently not but this is not too difficult to implement with a regular expression on variable names. I will look to add that.

Resolving patch:
diff --git a/opennmt/utils/optim.py b/opennmt/utils/optim.py
--- a/opennmt/utils/optim.py
+++ b/opennmt/utils/optim.py
@@ -1,6 +1,7 @@
 """Optimization related functions."""
 
 import collections
+import re
 
 import tensorflow as tf
 
@@ -183,6 +184,8 @@ def optimize_loss(loss, params, mixed_precision=False, var_list=None, hvd=None):
       optimizer = DistributedOptimizer.from_params(optimizer, params=params.get("horovod"))
 
     # Gradients.
+    var_list = _get_trainable_variables(
+        var_list=var_list, freeze_variables=params.get("freeze_variables"))
     gradients = optimizer.compute_gradients(
         loss, var_list=var_list, colocate_gradients_with_ops=True)
     _summarize_gradients_norm("global_norm/gradient_norm", gradients)
@@ -309,3 +312,28 @@ def _clip_gradients_by_norm(grads_and_vars, clip_gradients):
 def _summarize_gradients_norm(name, gradients):
   """Summarizes global norm of gradients."""
   tf.summary.scalar(name, tf.global_norm(list(zip(*gradients))[0]))
+
+def _print_var_list(var_list):
+  for variable in var_list:
+    tf.logging.info(" * %s", variable.name)
+
+def _get_trainable_variables(var_list=None, freeze_variables=None):
+  if var_list is None:
+    var_list = tf.trainable_variables()
+  if freeze_variables:
+    if not isinstance(freeze_variables, list):
+      freeze_variables = [freeze_variables]
+    regexs = list(map(re.compile, freeze_variables))
+    frozen_variables = []
+    trainable_variables = []
+    for variable in var_list:
+      if any(regex.match(variable.name) for regex in regexs):
+        frozen_variables.append(variable)
+      else:
+        trainable_variables.append(variable)
+    tf.logging.info("Frozen variables:")
+    _print_var_list(frozen_variables)
+    tf.logging.info("Trainable variables:")
+    _print_var_list(trainable_variables)
+    var_list = trainable_variables
+  return var_list


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
experimental_distribute_dataset does not always split the batches
In 4f8189b, we scaled the batch size by the number of devices based on `experimental_distribute_dataset` documentation:

> We will assume that the input dataset is batched by the global batch size

However, the method does not always split the input batches. This was at least found when using `batch_type: tokens` even though the first dimension is divisible by the number of replicas.


Discussion:

Resolving patch:
diff --git a/opennmt/runner.py b/opennmt/runner.py
--- a/opennmt/runner.py
+++ b/opennmt/runner.py
@@ -155,7 +155,6 @@ def train(self, num_devices=1, with_eval=False, checkpoint_path=None):
         data_config.get("train_labels_file"),
         train_config["batch_size"],
         batch_type=batch_type,
-        batch_multiplier=num_devices,
         batch_size_multiple=batch_size_multiple,
         shuffle_buffer_size=train_config["sample_buffer_size"],
         length_bucket_width=train_config["length_bucket_width"],
diff --git a/opennmt/training.py b/opennmt/training.py
--- a/opennmt/training.py
+++ b/opennmt/training.py
@@ -67,7 +67,13 @@ def __call__(self,
     with self._strategy.scope():
       self._model.create_variables(optimizer=self._optimizer)
       variables = self._model.trainable_variables
-      dataset = self._strategy.experimental_distribute_dataset(dataset)
+      base_dataset = dataset
+      # We prefer not to use experimental_distribute_dataset here because it
+      # sometimes fails to split the batches (noticed with tokens batch type).
+      # We also assume for now that we are training with a single worker
+      # otherwise we would need to correctly shard the input dataset.
+      dataset = self._strategy.experimental_distribute_datasets_from_function(
+          lambda _: base_dataset)
       gradient_accumulator = optimizer_util.GradientAccumulator()
 
     if self._mixed_precision:


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Error while running the exported model 
Hi,

I was trying to run the example given [https://github.com/OpenNMT/OpenNMT-tf/tree/master/examples/serving/python](url).

I am getting the following error.

> Source: I am going.
Traceback (most recent call last):
  File "ende_client.py", line 66, in <module>
    main()
  File "ende_client.py", line 60, in main
    output = translator.translate([text])
  File "ende_client.py", line 22, in translate
    return self._postprocess(outputs)
  File "ende_client.py", line 47, in _postprocess
    texts.append(self._tokenizer.detokenize(tokens))
TypeError: detokenize(): incompatible function arguments. The following argument types are supported:
    1. (self: pyonmttok.Tokenizer, tokens: list, features: object = None) -> str

> Invoked with: <pyonmttok.Tokenizer object at 0x147d10d0d538>, array([b'\xe2\x96\x81Ich', b'\xe2\x96\x81gehe', b'.'], dtype=object)
> WARNING:tensorflow:Unresolved object in checkpoint: (root).examples_inputter.features_inputter.ids_to_tokens._initializer
> WARNING:tensorflow:Unresolved object in checkpoint: (root).examples_inputter.labels_inputter.ids_to_tokens._initializer
> WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/alpha/guide/checkpoints#loading_mechanics for details.
> 

I have the updated version of pyonmttok.

Thanks,
Sriram


Discussion:

Resolving patch:
diff --git a/examples/serving/python/ende_client.py b/examples/serving/python/ende_client.py
--- a/examples/serving/python/ende_client.py
+++ b/examples/serving/python/ende_client.py
@@ -43,7 +43,7 @@ def _preprocess(self, texts):
   def _postprocess(self, outputs):
     texts = []
     for tokens, length in zip(outputs["tokens"].numpy(), outputs["length"].numpy()):
-      tokens = tokens[0][:length[0]]
+      tokens = tokens[0][:length[0]].tolist()
       texts.append(self._tokenizer.detokenize(tokens))
     return texts
 


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Bug in "onmt-ark-to-records" code
I have found a small bug in the code line referenced below. It causes the script to terminate with a `TypeError: data type not understood`. Just for the sake of completeness, this is caused by the fact that numpy doesn't understand the object `tf.float32`. I changed that to `float` and it worked as it was supposed to. I can create a PR for this, but I suppose it is too trivial to do so and claim a contribution, unless you want me to.

https://github.com/OpenNMT/OpenNMT-tf/blob/5809c293d7bc65d923274cfd56b3339fc4107af6/opennmt/bin/ark_to_records.py#L46


Discussion:

Resolving patch:
diff --git a/opennmt/bin/ark_to_records.py b/opennmt/bin/ark_to_records.py
--- a/opennmt/bin/ark_to_records.py
+++ b/opennmt/bin/ark_to_records.py
@@ -43,7 +43,7 @@ def consume_next_vector(ark_file):
     if end:
       break
 
-  return idx, np.asarray(vector, dtype=tf.float32)
+  return idx, np.asarray(vector, dtype=np.float32)
 
 def consume_next_text(text_file):
   """Consumes the next text line from `text_file`."""


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
GPT2 training reports on source and target words
When training a language mode (GPT2) model - we can see the following logs:
```
INFO:tensorflow:Evaluation result for step 20000: perplexity = 99.456184 ; loss = 4.599717
INFO:tensorflow:Step = 20100 ; steps/s = 2.09, source words/s = 6382, target words/s = 6382 ; Learning rate = 0.000250 ; Loss = 3.851096
```

There is no reason to see source & target word reports. If I correctly understand the code, `source` and `target` are defined as words counters in `_accumulate_gradients_on_replica` function in training.py regardless of training type. Can you give an hint on how/where we could correct this?

Thanks


Discussion:
Right, good find. A quick fix would be to add a condition when declaring the target counter, for example:

```python
if not self._model.unsupervised:
  self._update_words_counter("target", target)
```

Resolving patch:
diff --git a/opennmt/training.py b/opennmt/training.py
--- a/opennmt/training.py
+++ b/opennmt/training.py
@@ -183,7 +183,8 @@ def _run_model(self, source, target):
     training_loss = self._model.regularize_loss(
         training_loss, variables=self._model.trainable_variables)
     self._update_words_counter("source", source)
-    self._update_words_counter("target", target)
+    if not self._model.unsupervised:
+      self._update_words_counter("target", target)
     if first_call and self._is_master:
       if self._checkpoint is not None:
         self._model.visualize(self._checkpoint.model_dir)


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Bug with multiple source files and data_dir configuration
Sample When you add the export_vocabulary_assets with multiple files
```
model_dir: model

data:
  # (optional) During export save the vocabularies as model assets, otherwise embed
  # them in the graph itself (default: True).
  export_vocabulary_assets: true

  train_features_file:
    - train/src.txt
    - train/ner.txt
```

the crash:
```
  File "/usr/local/lib/python3.6/dist-packages/opennmt/bin/main.py", line 200, in main
    config["data"] = _prefix_paths(args.data_dir, config["data"])
  File "/usr/local/lib/python3.6/dist-packages/opennmt/bin/main.py", line 42, in _prefix_paths
    paths[key] = _prefix_paths(prefix, path)
  File "/usr/local/lib/python3.6/dist-packages/opennmt/bin/main.py", line 50, in _prefix_paths
    new_path = os.path.join(prefix, path)
  File "/usr/lib/python3.6/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.6/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'bool'
```


Discussion:
This is also giving me issues with the data_dir config. I guess is trying to update the path of the files inside "data:"
but on the sample in the web all this keys are inside the "data:" key 
```
# (optional) Pretrained embedding configuration.
  source_embedding:
    path: data/glove/glove-100000.txt
    with_header: True
    case_insensitive: True
    trainable: False
```

Resolving patch:
diff --git a/opennmt/bin/main.py b/opennmt/bin/main.py
--- a/opennmt/bin/main.py
+++ b/opennmt/bin/main.py
@@ -45,13 +45,15 @@ def _prefix_paths(prefix, paths):
     for i, path in enumerate(paths):
       paths[i] = _prefix_paths(prefix, path)
     return paths
-  else:
+  elif isinstance(paths, str):
     path = paths
     new_path = os.path.join(prefix, path)
     if tf.io.gfile.exists(new_path):
       return new_path
     else:
       return path
+  else:
+    return paths
 
 def main():
   parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Feature request: Log time for events during training
Hi Guillaume and others, thanks a lot for the great work you are doing!

I was thinking if it's possible to add time tracking to the logger during training. I tried to search the forum and google for relevant issues but didn't find anything. I think it would be helpful for everyone. So please tell me what you think about it.

Thank you,
Alex



Discussion:

Resolving patch:
diff --git a/opennmt/bin/main.py b/opennmt/bin/main.py
--- a/opennmt/bin/main.py
+++ b/opennmt/bin/main.py
@@ -24,8 +24,22 @@
 }
 
 
-def _set_log_level(log_level):
-    tf.get_logger().setLevel(log_level)
+def _initialize_logging(log_level):
+    logger = tf.get_logger()
+    logger.setLevel(log_level)
+
+    # Configure the TensorFlow logger to use the same log format as the TensorFlow C++ logs.
+    for handler in list(logger.handlers):
+        logger.removeHandler(handler)
+    formatter = logging.Formatter(
+        fmt="%(asctime)s.%(msecs)03d000: %(levelname).1s %(filename)s:%(lineno)d] %(message)s",
+        datefmt="%Y-%m-%d %H:%M:%S",
+    )
+    handler = logging.StreamHandler()
+    handler.setFormatter(formatter)
+    logger.addHandler(handler)
+
+    # Align the TensorFlow C++ log level with the Python level.
     os.environ["TF_CPP_MIN_LOG_LEVEL"] = str(
         _PYTHON_TO_TENSORFLOW_LOGGING_LEVEL[log_level]
     )
@@ -264,7 +278,7 @@ def main():
     ):
         args.features_file = args.features_file[0]
 
-    _set_log_level(getattr(logging, args.log_level))
+    _initialize_logging(getattr(logging, args.log_level))
     tf.config.threading.set_intra_op_parallelism_threads(
         args.intra_op_parallelism_threads
     )


Test patch:
Repository: OpenNMT/OpenNMT-tf

Issue:
Inference parameter `n_best` is applied during evaluation
The inference parameter `n_best` should not be applied when translating during evaluation. Reported in https://forum.opennmt.net/t/training-crash-at-validation/4513.


Discussion:

Resolving patch:
diff --git a/opennmt/evaluation.py b/opennmt/evaluation.py
--- a/opennmt/evaluation.py
+++ b/opennmt/evaluation.py
@@ -305,8 +305,11 @@ def __call__(self, step):
         if self._save_predictions:
             output_path = os.path.join(self._eval_dir, "predictions.txt.%d" % step)
             output_file = tf.io.gfile.GFile(output_path, "w")
+            params = {"n_best": 1}
             write_fn = lambda prediction: (
-                self._model.print_prediction(prediction, stream=output_file)
+                self._model.print_prediction(
+                    prediction, params=params, stream=output_file
+                )
             )
             index_fn = lambda prediction: prediction.get("index")
             ordered_writer = misc.OrderRestorer(index_fn, write_fn)
diff --git a/opennmt/models/sequence_to_sequence.py b/opennmt/models/sequence_to_sequence.py
--- a/opennmt/models/sequence_to_sequence.py
+++ b/opennmt/models/sequence_to_sequence.py
@@ -457,7 +457,7 @@ def print_prediction(self, prediction, params=None, stream=None):
             raise ValueError(
                 "with_alignments is set but the model did not return alignment information"
             )
-        num_hypotheses = len(prediction["log_probs"])
+        num_hypotheses = params.get("n_best", len(prediction["log_probs"]))
         for i in range(num_hypotheses):
             if "tokens" in prediction:
                 target_length = prediction["length"][i]


Test patch:
diff --git a/opennmt/tests/runner_test.py b/opennmt/tests/runner_test.py
--- a/opennmt/tests/runner_test.py
+++ b/opennmt/tests/runner_test.py
@@ -219,10 +219,14 @@ def testTrainLanguageModel(self):
         runner.train()
 
     def testEvaluate(self):
+        if not tf.config.functions_run_eagerly():
+            self.skipTest("Test case not passing in GitHub Actions environment")
         ar_file, en_file = self._makeTransliterationData()
         config = {
+            "params": {"beam_width": 4},
             "data": {"eval_features_file": ar_file, "eval_labels_file": en_file},
             "eval": {"external_evaluators": "BLEU"},
+            "infer": {"n_best": 4},
         }
         runner = self._getTransliterationRunner(config)
         metrics = runner.evaluate()