Repository: OpenNMT/OpenNMT-tf

Issue:
Tagger with CRF fails on sequences of length 1
```
File "/opt/anaconda3/bin/onmt-main", line 10, in <module>
    sys.exit(main())
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/bin/main.py", line 189, in main
    checkpoint_path=args.checkpoint_path)
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/runner.py", line 187, in train
    mixed_precision=self._mixed_precision)
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/training.py", line 27, in __init__
    raise ValueError("No optimizer is defined")
ValueError: No optimizer is defined
```

my command is:  onmt-main  --config tag22.yml --model_type LstmCnnCrfTagger train
and with OpenNMT-tf 2.2.1

tag22.yml

```
model_dir: tag4/
data:
  train_features_file: tag_train_q.txt
  train_labels_file:  tag_train_l.txt
  eval_features_file: tag_test_q.txt
  eval_labels_file: tag_test_l.txt
  source_1_vocabulary: tag_src.txt
  source_2_vocabulary: char_vocab.txt
  target_vocabulary: tag.txt
train:
  batch_size: 32
  effective_batch_size: 320
  sample_buffer_size: 10000000
  bucket_width: 5
  train_steps: 1000000000

infer:
  n_best: 30
  batch_size: 1
  with_scores: false
  with_alignments: null

```



Discussion:
You should define an optimizer in your configuration, e.g.:

```yaml
params:
  optimizer: Adam
  learning_rate: 0.001
```
after enable this, and got new error

````
tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Invalid argument: Tried to stack elements of an empty list with non-fully-defined element_shape: [?,2]
         [[{{node cond_2/then/_35/scan/TensorArrayV2Stack/TensorListStack}}]]
Traceback (most recent call last):
  File "/opt/anaconda3/bin/onmt-main", line 10, in <module>
    sys.exit(main())
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/bin/main.py", line 189, in main
    checkpoint_path=args.checkpoint_path)
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/runner.py", line 196, in train
    export_on_best=eval_config.get("export_on_best"))
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/training.py", line 175, in __call__
    for i, (loss, num_words, skipped) in enumerate(_forward()):  # pylint: disable=no-value-for-parameter
  File "/opt/anaconda3/lib/python3.7/site-packages/opennmt/data/dataset.py", line 433, in _fun
    outputs = _tf_fun()
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 487, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1823, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1141, in _filtered_call
    self.captured_inputs)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1224, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 511, in call
    ctx=ctx)
  File "/opt/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError:  Tried to stack elements of an empty list with non-fully-defined element_shape: [?,2]
         [[{{node cond_2/then/_35/scan/TensorArrayV2Stack/TensorListStack}}]] [Op:__inference__tf_fun_8432]

Function call stack:
_tf_fun
```

BTW, should be super(LstmCnnCrfTagger .....

https://github.com/OpenNMT/OpenNMT-tf/blob/master/opennmt/models/catalog.py#L125
> after enable this, and got new error

Thanks, I was able to reproduce. I will have a look.

> BTW, should be super(LstmCnnCrfTagger .....

This one is fixed by the PR linked above.
The error is in the CRF module from TensorFlow Addons. I opened an issue there: https://github.com/tensorflow/addons/issues/694.

In the meantime, you could disable `crf_decoding`.
Let's keep this open to track when the fix will be included in OpenNMT-tf.
Fixed in TFA by commit  https://github.com/tensorflow/addons/commit/7f68da6c717c232bc0760ca1ab9ad13ce26f221a
Thanks for the fix @Squadrick!

Resolving patch:
diff --git a/opennmt/layers/rnn.py b/opennmt/layers/rnn.py
--- a/opennmt/layers/rnn.py
+++ b/opennmt/layers/rnn.py
@@ -49,19 +49,6 @@ def get_initial_state(self, inputs=None, batch_size=None, dtype=None):
     return self.cell.get_initial_state(
         inputs=inputs, batch_size=batch_size, dtype=dtype)
 
-
-class _StackedRNNCells(tf.keras.layers.StackedRNNCells):
-
-  # To pass the training flag to the cell, tf.keras.layers.RNN checks that the
-  # cell call method explicitly takes the "training" argument, which
-  # tf.keras.layers.StackedRNNCells do not.
-  # TODO: remove this when this change is released:
-  # https://github.com/tensorflow/tensorflow/commit/df2b252fa380994cd9236cc56b06557bcf12a9d3
-  def call(self, inputs, states, constants=None, training=None, **kwargs):
-    kwargs["training"] = training
-    return super(_StackedRNNCells, self).call(inputs, states, constants=constants, **kwargs)
-
-
 def make_rnn_cell(num_layers,
                   num_units,
                   dropout=0,
@@ -94,7 +81,7 @@ def make_rnn_cell(num_layers,
       cell = RNNCellWrapper(
           cell, output_dropout=dropout, residual_connection=residual_connections)
     cells.append(cell)
-  return _StackedRNNCells(cells)
+  return tf.keras.layers.StackedRNNCells(cells)
 
 
 class _RNNWrapper(tf.keras.layers.Layer):
diff --git a/opennmt/utils/misc.py b/opennmt/utils/misc.py
--- a/opennmt/utils/misc.py
+++ b/opennmt/utils/misc.py
@@ -2,12 +2,10 @@
 
 import collections
 import copy
-import copyreg
 import sys
 import inspect
 import heapq
 import os
-import threading
 
 import numpy as np
 import tensorflow as tf
@@ -183,9 +181,6 @@ def index_structure(structure, path):
 
 def clone_layer(layer):
   """Clones a layer."""
-  # TODO: clean this up when this change is released:
-  # https://github.com/tensorflow/tensorflow/commit/4fd10c487c7e287f99b9a1831316add453dcba04
-  copyreg.pickle(threading.local, lambda _: (threading.local, []))
   return copy.deepcopy(layer)
 
 def gather_all_layers(layer):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@
         "pyyaml==5.1.*",
         "rouge==0.3.1",
         "sacrebleu>=1.4.1,<2",
-        "tensorflow-addons>=0.6,<0.7"
+        "tensorflow-addons>=0.7,<0.8"
     ],
     extras_require={
         "tests": tests_require,


Test patch: