Does the CRFModelWrapper in the TFA support the serialization for the continuous training?

Question:
Hi everyone,
I am using the CRFModelWrapper method following the tutorial as addons/layers_crf.ipynb at add_crf_tutorial · howl-anderson/addons · GitHub to implement a Bi-LSTM -CRF neural-network for a multi-classes NER problem. The model I built (codes are shown below) can be trained with multiple GPUs and it can be saved and load with the tf.keras.model functions. However, when I I saved the model, a warning shows below which I am not sure if it really matters. After that, I loaded the trained model, it shows many warnings related inconsistent output shape? These warnings are posted below.  want to train it again using the fit function, it shows no
Saving warning:
WARNING:absl:Found untraced functions such as embedding_layer_call_and_return_conditional_losses, embedding_layer_call_fn, embedding_1_layer_call_and_return_conditional_losses, embedding_1_layer_call_fn, multi_head_attention_layer_call_and_return_conditional_losses while saving (showing 5 of 65). These functions will not be directly callable after loading.

Loading warnings:
2021-11-21 00:28:34.222763: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:34.374067: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:35.553642: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:37.286608: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:37.556591: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:37.645775: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:37.742369: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:38.758195: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:38.892746: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:38.905252: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:39.369396: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:39.426702: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:39.439491: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:39.562408: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:39.574280: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:40.102410: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:40.363686: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:40.375926: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:40.706863: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:40.765348: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.099845: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.111754: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.127138: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.139674: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.959556: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:41.971252: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.012887: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.025740: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.284787: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.403613: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.551636: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.773302: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.786014: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:42.993502: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.005655: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.019730: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.031896: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.045523: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.581028: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:43.593377: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.286465: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.298637: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.319105: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.331097: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.446932: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.864126: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:44.995524: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.084922: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.097600: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.134094: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.243815: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.264186: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.599941: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.621283: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:45.633535: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:46.106656: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond/while' has 14 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:46.119119: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:46.141522: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:46.165137: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.
2021-11-21 00:28:46.266813: W tensorflow/core/common_runtime/graph_constructor.cc:803] Node 'cond' has 5 outputs but the _output_shapes attribute specifies shapes for 48 outputs. Output shapes may be inaccurate.

With these warnings, the model can still be saved, loaded, and used for the prediction, the loaded model cannot be re-trained again. The error message is posted below and it seems that the loss function inside the CRFModelWrapper cannot be called again so that the gradient calculation cannot be done. So, I am wondering if the CRFModelWrapper doesn’t support the serialization (save->load->training) or it’s because of some mistakes I have made. If so, is there any way that I can workaround to retrain the model?
Thank you very much.
Error message when re-training the trained model:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py", line 1184, in fit
    tmp_logs = self.train_function(iterator)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\def_function.py", line 933, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\def_function.py", line 759, in _initialize
    self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\function.py", line 3066, in _get_concrete_function_internal_garbage_collected
    graph_function, _ = self._maybe_define_function(args, kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\function.py", line 3463, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\function.py", line 3298, in _create_graph_function
    func_graph_module.func_graph_from_py_func(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\framework\func_graph.py", line 1007, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\eager\def_function.py", line 668, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\framework\func_graph.py", line 994, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py:853 train_function  *
        return step_function(self, iterator)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py:842 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1286 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2849 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3632 _call_for_each_replica
        return fn(*args, **kwargs)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py:835 run_step  **
        outputs = model.train_step(data)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py:791 train_step
        self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\optimizer_v2\optimizer_v2.py:522 minimize
        return self.apply_gradients(grads_and_vars, name=name)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\optimizer_v2\optimizer_v2.py:622 apply_gradients
        grads_and_vars = optimizer_utils.filter_empty_gradients(grads_and_vars)
    C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\optimizer_v2\utils.py:72 filter_empty_gradients
        raise ValueError("No gradients provided for any variable: %s." %
    ValueError: No gradients provided for any variable: ['chain_kernel:0', 'left_boundary:0', 'right_boundary:0', 'crf_model_wrapper/crf/dense/kernel:0', 'crf_model_wrapper/crf/dense/bias:0', 'embedding/embeddings:0', 'bidirectional/forward_bilstm/lstm_cell_1/kernel:0', 'bidirectional/forward_bilstm/lstm_cell_1/recurrent_kernel:0', 'bidirectional/forward_bilstm/lstm_cell_1/bias:0', 'bidirectional/backward_bilstm/lstm_cell_2/kernel:0', 'bidirectional/backward_bilstm/lstm_cell_2/recurrent_kernel:0', 'bidirectional/backward_bilstm/lstm_cell_2/bias:0', 'time_distributed/kernel:0', 'time_distributed/bias:0'].

Below shows my codes except for the data-preprocessing.
#%% Build the base model_1
def build_bilstm_crf_model(
        lstm_unit,
        fc_unit
) -> tf.keras.Model:
    x = tf.keras.layers.Input(shape=(None,), dtype=tf.float32, name="inn")
    y = tf.keras.layers.Embedding(1, 1, mask_zero=True)(x)
    y = tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(lstm_unit, return_sequences=True,name="bilstm")
    )(y)
    y = tf.keras.layers.TimeDistributed(
        tf.keras.layers.Dense(fc_unit,name="fc")
    )(y)
    return tf.keras.Model(
        inputs=x, outputs=y
    )

# CFR Wrapper Model
class CRFModelWrapper(tf.keras.Model):
    def __init__(
            self,
            model: tf.keras.Model,
            units: int,
            chain_initializer="orthogonal",
            use_boundary: bool = True,
            boundary_initializer="zeros",
            use_kernel: bool = True,
            **kwargs
    ):
        super().__init__()

        self.crf_layer = tfa.layers.CRF(
            units=units,
            chain_initializer=chain_initializer,
            use_boundary=use_boundary,
            boundary_initializer=boundary_initializer,
            use_kernel=use_kernel,
            **kwargs
        )

        self.base_model = model

    def unpack_training_data(self, data):
        # override me, if this is not suit for your task
        if len(data) == 3:
            x, y, sample_weight = data
        else:
            x, y = data
            sample_weight = None
        return x, y, sample_weight

    def call(self, inputs, training=None, mask=None, return_crf_internal=False):
        base_model_outputs = self.base_model(inputs, training, mask)

        # change next line, if your model has more outputs
        crf_input = base_model_outputs

        decode_sequence, potentials, sequence_length, kernel = self.crf_layer(crf_input)
        ### potentials =predicted y during training
        # change next line, if your base model has more outputs
        # Always keep `(potentials, sequence_length, kernel), decode_sequence, `
        # as first two outputs of model.
        # current `self.train_step()` expected such settings
        outputs = (potentials, sequence_length, kernel), decode_sequence

        if return_crf_internal:
            return outputs
        else:
            # outputs[0] is the crf internal, skip it
            output_without_crf_internal = outputs[1:]

            # it is nicer to return a tensor instead of an one tensor list
            if len(output_without_crf_internal) == 1:
                return output_without_crf_internal[0]
            else:
                return output_without_crf_internal

    def compute_crf_loss(self, potentials, sequence_length, kernel, y, sample_weight=None):
        ### Added to reshape labels(y)
        shape = y.shape
        if len(shape) > 2:
            y_1 = tf.argmax(y, -1, output_type=tf.int32)
        ################################################
        crf_likelihood, _ = tfa.text.crf_log_likelihood(
            potentials, y_1, sequence_length, kernel
        )
        # convert likelihood to loss
        flat_crf_loss = -1 * crf_likelihood
        if sample_weight is not None:
            flat_crf_loss = flat_crf_loss * sample_weight
        crf_loss = tf.reduce_mean(flat_crf_loss)

        return crf_loss

    def train_step(self, data):
        x, y, sample_weight = self.unpack_training_data(data)
        with tf.GradientTape() as tape:
            (potentials, sequence_length, kernel), decoded_sequence, *_ = self(
                x, training=True, return_crf_internal=True
            )
            crf_loss = self.compute_crf_loss(
                potentials, sequence_length, kernel, y, sample_weight
            )
            loss = crf_loss + tf.reduce_sum(self.losses)
        gradients = tape.gradient(loss, self.trainable_variables)
        self.optimizer.apply_gradients(zip(gradients, self.trainable_variables))

        # Update metrics (includes the metric that tracks the loss)
        self.compiled_metrics.update_state(y, decoded_sequence)
        # Return a dict mapping metric names to current value
        results = {m.name: m.result() for m in self.metrics}
        results.update({"loss": loss, "crf_loss": crf_loss})  # append loss
        return results

    def test_step(self, data):
        x, y, sample_weight = self.unpack_training_data(data)
        (potentials, sequence_length, kernel), decode_sequence, *_ = self(
            x, training=False, return_crf_internal=True
        )
        crf_loss = self.compute_crf_loss(
            potentials, sequence_length, kernel, y, sample_weight
        )
        loss = crf_loss + tf.reduce_sum(self.losses)
        # Update metrics (includes the metric that tracks the loss)
        self.compiled_metrics.update_state(y, decode_sequence)
        # Return a dict mapping metric names to current value
        results = {m.name: m.result() for m in self.metrics}
        results.update({"loss": loss, "crf_loss": crf_loss})  # append loss
        return results

When training the model:
strategy = tf.distribute.MirroredStrategy()
print('Number of devices: {}'.format(strategy.num_replicas_in_sync))
with strategy.scope():
    base_model = build_bilstm_crf_model(units_lstm,TAG_SIZE)
    model = CRFModelWrapper(base_model, TAG_SIZE)
    model.compile(optimizer=tf.keras.optimizers.Adam(lr))
num_epochs = 10
name  = 'BLD_CRF_Lut{}_lr{}_{}'.format(units_lstm,lrr,int(time.time()))
name_csv = 'BLD_CRF_Lut{}_lr{}'.format(units_lstm,lrr)
mc = tf.keras.callbacks.ModelCheckpoint(filepath=os.path.join(name,'{epoch:02d}'), verbose=1,
                     save_best_only=False,save_weights_only=False)
csv_log = tf.keras.callbacks.CSVLogger('{}.csv'.format(name_csv),append=True)
model.fit(tr_gen, epochs=num_epochs, validation_data = val_gen,verbose=2,batch_size = bt_sz,callbacks=[mc,csv_log])

When re-training the model:
#%%
strategy = tf.distribute.MirroredStrategy()
print('Number of devices: {}'.format(strategy.num_replicas_in_sync))
with strategy.scope():
# base_model = build_bilstm_crf_model(units_lstm,TAG_SIZE)
# model = CRFModelWrapper(base_model, TAG_SIZE)
# model.compile(optimizer=tf.keras.optimizers.Adam(lr))
    model = load_model(FILE_PATH4)
num_epochs = 10
name  = 'BLD_CRF_Lut{}_lr{}_{}'.format(units_lstm,lrr,int(time.time()))
name_csv = 'BLD_CRF_Lut{}_lr{}'.format(units_lstm,lrr)
mc = tf.keras.callbacks.ModelCheckpoint(filepath=os.path.join(name,'{epoch:02d}'),verbose=1,save_best_only=False,save_weights_only=False)
csv_log = tf.keras.callbacks.CSVLogger('{}.csv'.format(name_csv),append=True)
model.fit(tr_gen, epochs=num_epochs, validation_data = val_gen,verbose=2,batch_size = bt_sz,callbacks=[mc,csv_log])

Context: /cc @XiaoquanKong can you check this?


All zero values after tensor_scatter_nd_update in a Quantum CNN

Question:
Hi everyone,
We are trying to build a trainable Quantum Convolutional Neural Network.
To this end we are trying to create a new subclass QuantumConvolutionalLayer of the class keras.Layer.
How we are trying this now:
In the initialization we use the add_weight function to add the trainable_parameters as a weight. Then we create (still in the initialization) a quantum circuit (on 4 qubits) with pennylane that takes 4 classical inputs and the trainable weights that will be called upon.
In the call function (for the forward pass):
a 2x2 grid is moved over the entire image (mnist in our case) and everytime the 2x2 grid is processed using the quantum circuit defined in the initialization.
The outcome of the processing with this quantum circuit of such a 2x2 grid [measurement_results] is stored in a tensorflow tensor object and we aim to then store all these in one big tensor [out]. To do this we use the tensor_scatter_nd_update. Unfortunately when we look at the resulting out tensor it has all zero values, even though when we print the measurement_results of the small grids we get non zero values. Any ideas on how this can be solved?
Many thanks in advance for your help!
Odiel
PS: Below you can find our code so far:
# Library installation
!pip install qiskit
!pip install pennylane

# General imports
import pennylane as qml
import numpy as np
from numpy import pi, sqrt
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import backend
from tensorflow.keras.layers import Layer
import matplotlib.pyplot as plt
from qiskit import QuantumCircuit, Aer, assemble, visualization

# Dataset
from keras.datasets import mnist

# Embedding imports
from pennylane.templates import QAOAEmbedding

# Build function to load train and test dataset
def load_dataset():
    # load dataset
    (trainX, trainY), (testX, testY) = mnist.load_data()
    # reshape dataset to have a single channel
    trainX = trainX.reshape((trainX.shape[0], 28, 28, 1))
    testX = testX.reshape((testX.shape[0], 28, 28, 1))
    # one hot encode target values
    trainY = tf.keras.utils.to_categorical(trainY)
    testY = tf.keras.utils.to_categorical(testY)
    return trainX, trainY, testX, testY

# Load train and test dataset
X_train, Y_train, X_test, Y_test = load_dataset()
num_images = 10 # set to -1 in order to keep all images
X_train = X_train[0:num_images]
X_test = X_test[0:num_images]
Y_train = Y_train[0:num_images]
Y_test = Y_test[0:num_images]

# Build a class for the trainable quantum convolutional layer that is a subclass of the keras.Layer class

class QuantumConvolutionalLayer(Layer):
  def __init__(self, device = "default.qubit", stride = 2, wires = 4, layers = 1, n_measurements = 4):

    # Inherits the initialization of the keras.Layer class
    super(QuantumConvolutionalLayer, self).__init__()

    # Initialize the device
    self.wires = wires
    self.dev = qml.device(device, wires = self.wires)

    # Initialize the quantum circuit
    self.layers = layers
    self.stride = stride
    self.n_measurements = n_measurements
    self.trainable_parameters = self.add_weight("trainable_parameters", shape = QAOAEmbedding.shape(n_layers=layers, n_wires=wires), initializer = tf.keras.initializers.RandomNormal())

    # To this end, build the quantum circuit (for 1 square of stride x stride)
    @qml.qnode(device = self.dev, interface = "tf")
    def quantum_circuit(inputs, trainable_parameters = self.trainable_parameters):
      QAOAEmbedding(features = inputs, weights = trainable_parameters, wires = range(wires))
      return [qml.expval(qml.PauliZ(j)) for j in range(n_measurements)]

    #weight_shapes = {"trainable_parameters": QAOAEmbedding.shape(n_layers=self.layers, n_wires=self.wires)}
    #self.quantum_circuit = qml.qnn.KerasLayer(quantum_circuit, weight_shapes = weight_shapes, output_dim = self.n_measurements)
    self.quantum_circuit = quantum_circuit

    dtype = tf.float32 if tf.keras.backend.floatx() == tf.float32 else tf.float64

    if self.quantum_circuit.diff_method != "backprop" or self.quantum_circuit.diff_method_change:
      self.quantum_circuit.to_tf(dtype=dtype)

  def build(self, input_shape):
    super().build(input_shape)

  def call(self, inputs):
    # define forward pass
    num_images=inputs.shape[0]
    h_in, w_in, ch_in = inputs.shape[1:]  # inputs.shape  (28, 28, 1) for MNIST
    h_out, w_out, ch_out = h_in // self.stride, w_in // self.stride, ch_in * self.n_measurements # (14, 14, 4) for MNIST and our quantum circuit filter
    out = tf.zeros((num_images, h_out, w_out, ch_out))
    for img_idx in range(num_images):
      # print(tf.rank(out))
      for j in range(0, h_in, self.stride):
        for k in range(0, w_in, self.stride):
          grid = [inputs[img_idx, j, k, 0], inputs[img_idx, j, k + 1, 0], inputs[img_idx, j + 1, k, 0], inputs[img_idx, j + 1, k + 1, 0]]
          measurement_results = self.quantum_circuit(inputs = grid, trainable_parameters = self.trainable_parameters)
          print(measurement_results)
          for ch in range(self.n_measurements):
              tf.tensor_scatter_nd_update(out[img_idx], tf.constant([[j//2, k//2, ch]]), [measurement_results[ch]])
    return out

quanv = QuantumConvolutionalLayer()
quanv(np.array([X_train[0]]))

Context:
Dear all,
We have found the issue.
The tf.tensor_scatter_nd_update function replaces the entries at the given coordinates by the right values. It just doesn’t store them automatically. so you still have to assign it yourself.
Replacing the following line in the code:
tf.tensor_scatter_nd_update(out[img_idx], tf.constant([[j//2, k//2, ch]]), [measurement_results[ch]])

by this line solves the problem:
out = tf.tensor_scatter_nd_update(out, tf.constant([[img_idx, j//2, k//2, ch]]), [measurement_results[ch]])


Compatibility of the CRF layers in TF-addons and TF2.6

Question:
Hi everyone,
I am trying to use the CRFModelWrapper method following the tutorial as addons/layers_crf.ipynb at add_crf_tutorial · howl-anderson/addons · GitHub 1 to implement a Bi-LSTM -CRF neural-network for a multi-classes time-series NER problem, and it works in TF 2.7 in my PC. However, when I used the same code and same data runing in TF 2.6 environment, it pops out some errors(as shown below) regarding to the tfa-CRF layer.  So, could someone please let me know if the addons CRF layer is only compatible for the TF 2.7 version or this is because I made some mistakes? Thank you very much.
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\training.py", line 1134, in fit
    data_handler = data_adapter.get_data_handler(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 1383, in get_data_handler
    return DataHandler(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 1138, in __init__
    self._adapter = adapter_cls(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 917, in __init__
    super(KerasSequenceAdapter, self).__init__(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 801, in __init__
    model.distribute_strategy.run(
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1286, in run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 2849, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\distribute\distribute_lib.py", line 3632, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 597, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\data_adapter.py", line 802, in <lambda>
    lambda x: model(x, training=False), args=(concrete_x,))
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\base_layer.py", line 1037, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "<input>", line 47, in call
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\keras\engine\base_layer.py", line 1037, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "C:\Users\YenPangLai\anaconda3\envs\tf2p6\lib\site-packages\tensorflow_addons\layers\crf.py", line 131, in call
    raise NotImplementedError(
NotImplementedError: Currently, CRF layer do not support left padding

Context: /cc @XiaoquanKong can you check this?


Question:
Hi there,
I’m trying to quickly load a model to make predictions in a REST API. The tf.keras.models.load_model method takes ~1s to load so it’s too slow for what I’m trying to do.
What is the fastest way to load a model for inference only?
I know there is a TFX serving server to just do this efficiently but I’ve already have a REST API for doing other things. Setting up a specialised server just for predictions feels like an overkill. How is the TFX server handling this?
Thanks in advance,
Joan

Context: Hey Robert, thanks for your reply. Yes, I ended up loading the model in memory on server initialisation. It’s a small model so works well this way.


Factorized_top_k using tensorflow recommenders

Question:
I followed the tensorflow recommenders movie ranking tutorial 1 and built the model. Now I would like to get top_k recommendations using the model. This is what I tried:
layer = tfrs.layers.factorized_top_k.Streaming(model.ranking_model)
layer.index(movies.map(model.ranking_model.movie_embeddings), movies)
tracks = layer.query_with_exclusions(
    queries=np.array([["42", "52"]]),
    exclusions= np.array([[]])
)

But it throws the error “iterating over tf.Tensor is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.”
How to invoke the query_with_exclusions() function correctly?

Context: Also interested in the answer to this, and how one could use this function to remove all previously interacted with items for each user.


Question:
Hi there,
I am really interested in Tensorflow MLIR, and hope to create custom dialects from Tensorflow’s existing dialects.
I found that recently there is a TensorFlow Graph IR 6 , it seems quite exciting. May I ask what is its def status by now?
Does it support basic runtime? I mean, could I use it for actually run by now? I am going to use it for optimizing ResNet training, is it ok now?
Besides, Is there any dev discussion group for now? Like PyTorch’s developer group in slack.

Context:
Hi,
The Graph IR is fairly recent, and enabled by default in Grappler for a bit more than a month now.
We’re ramping up on this this quarter to provide more natural extension point and helpers for transformations on this.


Question: I notice that the tensorflow repo doesn’t have a .clang-tidy file anywhere in its tree. Does anyone have a .clang-tidy file that conforms or nearly conforms to the TF coding style? I specifically need it for the mlir-hlo sub-tree (tensorflow/compiler/mlir/).

Context:
Note that the mlir-hlo subtree follows the LLVM coding style in theory (in practice it is a bit of a mix).
I don’t know about .clang-tidy, but for .clang-format TF just uses the Google style otherwise
.


Question:
@petewarden
I use  the dual core 160 pin microcontroller  Arduino Portenta to make TensorFlowLite/Micro models. Recently I found out about an extra 8 MB of SDRAM on the board. I have managed to use the SDRAM to load the 320x320 Grayscale Camera Buffer when using EdgeImpulse.com models, draft code here.

.
.
I would like to put the entire TensorflowMicro model_tflite array into the 8 MB SDRAM, the board normally only has 1 MB of RAM for each core, so using SDRAM would be a fair improvement.
When using SDRAM for the camera frame buffer I was just changing one pointer for another so it was not that hard.
#include <SDRAM.h>

SDRAMClass mySDRAM;

uint8_t *sdram_frame_buffer;

// in the setup

  mySDRAM.begin(SDRAM_START_ADDRESS);   // for camera 320x320

  sdram_frame_buffer = (uint8_t *)mySDRAM.malloc(320 * 320 * sizeof(uint8_t));

// in the main loop
   int myCamResult =  myCam.grab(sdram_frame_buffer); // myCamResult should be zero

but for TFLITE I need to work with an array, and that is a bit different when calling the array. Anyone got any suggestions to get me started?
Here is the basic TFLITE code.
const unsigned char model_tflite[] = { ... }

unsigned int model_tflite_len = 3048;

// which is then called using

   model = tflite::GetModel(model_tflite);

// I have tried passing the array as a pointer (which works in some situations)
// but doesn't seem to work here
// Not really sure what the GetModel method does with the array.

.
.
using TFLITE tutorial here
.
More info posted on the Arduino Fourm  Portenta - Usage of SDRAM.h library - #3 by jerteach - Portenta - Arduino Forum

Context: Just as a quick followup, I chatted to Jeremy elsewhere and think we reached a solution: SDRAM Examples · Issue #38 · arduino/ArduinoCore-mbed · GitHub 1


Timeseries_dataset_from_array returns future samples instead of previous

Question:
I have a dataframe in my ML project it is in the following format initially.
        Feature 1 | Feature 2| Feature 3 | Feature 4 | etc.
Time 1
Time 2
Time 3
Etc.

I am trying to change this dataframe to be 3d, where each value in this dataframe has another dimension into the screen, containing the same value for the same feature, but at previous 192 timesteps.
Here i am trying to use the built in function keras.preprocessing.timeseries_dataset_from_array(), but it returns the opposite of what i’m trying to achieve.
I expect it to return
          Feature 1 | Feature 2| Feature 3 | Feature 4 | etc.
Time 192| [1-192]   | [1-192]  | [1-192]   |           |
Time 193|           |          |           |           |
Time 194|           |          |           |           |
Time End|           |          |           |           |

Here it instead returns:
        Feature 1 | Feature 2| Feature 3 | Feature 4 | etc.
Time 1| [192-1]   | [192-1]  | [192-1]   |           |
Time 2|           |          |           |           |
Time 3|           |          |           |           |
Time End-192|     |          |           |           |

Basically every sample contains the future 192 values, instead of the previous 192 values of the dataset. Therefore it ends 192 samples before it should, and starts 192 samples too early.
My code is the following:
#Past is defined as 192
#x_val is the 2-d dataframe
#y_val is one of the columns in the dataframe.

dataset_historic_train = keras.preprocessing.timeseries_dataset_from_array(
    x_val,
    y_val,
    sequence_length=past,
    batch_size=len(x_val),
)

Where x_val is the entirety of my 2-d dataframe indexed from first to last time of sample, and y_val is my target feature, which is Feature 1 in this case.
pythondat

Context:
You pass x and y of equal length to the dataset constructor. When it transforms x using sliding window of size 192, the x becomes shorter, because the first 192 rows of your original DataFrame do not have enough previous values. So it drops the last 192 values of y to pair it with x.
To make it work as expected you should pass x and y[192:]. Then it will drop the first 192 values of y.


TF.js SIG Meeting - Tuesday Nov 2nd from 5-6pm PST

Question:
Greetings TF.js Community,
We are looking forward to catching up with you all today Tuesday November 2nd from 5:00-6:00 P.M. PST.
We will discuss the first RFC to the new sig-tfjs repo “WebNN Delegate for TensorFlow Lite Web” (https://github.com/tensorflow/sig-tfjs/pull/2 2), and also discuss Coral Node support and TFJS Debugger tool. Please find the agenda and the Gmeet link below. Feel free to add any questions or topics for discussion to the agenda.

      docs.google.com

[Public] SIG TF.js Meeting Notes 1

[Public] SIG TF.js Meeting Notes Meeting: 2021-11-02 Tuesday, November 2nd, 2021, 5:00 – 6:00 pm Pacific Time  Meeting Recording - TODO  Please join this link: meet.google.com/dac-ngjq-okc (US) +1 617-675-4444 PIN: 298 636 519 4797#  Shared Drive...

Cheers, and talk soon!
Masoud on behalf of the TF.js team

Context:
Hi everyone!
If you didn’t have the chance to join today’s meeting, here’s the link 2 to rewatch it.
See you next time!


[MLIR]The following operations cannot be legalized: tf.VariableV2

Question:
Hello, this is my first time posting here. Let me know whether this question is suited to be posted to this forum or not.
I have been trying to translate tensorflow models into MLIR-HLO. As such I successfully used tf-mlir-translate , tf-opt with very simple TF models like Conv2D, MatMul,MaxPool…, achieving the expected results.
However, if I try to do the same with the official example tf.Variable  with the three steps below:

tf-mlir-translate -graphdef-to-mlir -tf-enable-shape-inference-on-import=false target.pbtxt -tf-prune-unused-nodes -tf-control-output-arrays=Variable/Assign,AssignAdd -o target.mlir
tf-opt -tf-executor-to-functional-conversion target.mlir -o target-func.mlir
tf-opt --tf-to-hlo-pipeline target-func.mlir -o target-mhlo.mlir

I get the following error message with the 3’rd step:
target-func.mlir:2:3: error: The following operations cannot be legalized: tf.Assign (count: 1); tf.AssignAdd (count: 1); tf.VariableV2 (count: 1). These legalization failure(s) may be due to missing TF to HLO lowerings and/or unsupported attributes, etc.
  func @main() attributes {tf.entry_function = {control_outputs = "Variable/Assign,AssignAdd", inputs = "", outputs = ""}} {
  ^
target-func.mlir:2:3: error: Emitting more detail about one op that failed to legalize...
  func @main() attributes {tf.entry_function = {control_outputs = "Variable/Assign,AssignAdd", inputs = "", outputs = ""}} {
  ^
target-func.mlir:7:10: error: 'tf.Assign' op is not legalizable
    %2 = "tf.Assign"(%0, %cst_0) {_class = ["loc:@Variable"], device = "", use_locking = true, validate_shape = true} : (tensor<!tf.f32ref>, tensor<f32>) -> tensor<*x!tf.f32ref>
         ^
target-func.mlir:7:10: note: see current operation: %4 = "tf.Assign"(%2, %0) {_class = ["loc:@Variable"], device = "", use_locking = true, validate_shape = true} : (tensor<!tf.f32ref>, tensor<f32>) -> tensor<!tf.f32ref>

As of right now, is there a way to lower these tf operations into mhlo-hlo ?

Context:
“HLO ops are intended for the numeric computation side and are mostly pure with state management is handled outside them. For TF this is done via TF ops and the state hoisted/sunk outside stateless parts"
How to continue lowering for TF?  does TF lower the resource variables to other dialect than mhlo-hlo?


How to calculate the kTensorArenaSize value in tflite-micro?

Question: Hello everyone, when I read the examples code of tflite-micro, I saw that different examples use different values of kTensorArenaSize. What is the basis for determining this value? For example, in hello_world, the value of kTensorArenaSize is 2000, and in magic_wand, the value of kTensorArenaSize is 64*1024.

Context:
In the comment, there are such lines, so the only way is to try again and again to get minist value.
// Create an area of memory to use for input, output, and intermediate arrays.
// The size of this will depend on the model you’re using, and may need to be
// determined by experimentation.


How to verify if an object is a tf tensor in javascript?

Question:
Using typeof() on a tf tensor only returns object.
Using isinstancof object, or isinstanceof tf.tensor, will generate syntax error of missing ) after argument list…
So, how to verify an object is a tf tensor on which we can apply tensor related operations?
import * as tf from "@tensorflow/tfjs";
const a = tf.tensor([[1, 2], [3, 4]]);
console.log('type:', typeof(a));
// returns "object"

Context:
You can simply print in the console the variable assigned to console and that would print Tensor if the object is tf object.
For Example
import * as tf from "@tensorflow/tfjs";
console.log(tf.tensor([1, 2, 3]))
Output:


Why does my validation loss increase, but validation accuracy perfectly matches training accuracy?

Question:
I am building a simple 1D convolutional neural network in Keras. Here is the model:
def build_model():

    model = models.Sequential()
    model.add(layers.SeparableConv1D(64, kernel_size=2, activation="relu", input_shape=(64,20)))
    model.add(layers.SeparableConv1D(64, kernel_size=2, activation="relu"))
    model.add(layers.MaxPooling1D(4))
    model.add(layers.Flatten())
    model.add(layers.Dense(128, activation="relu"))
    model.add(layers.Dense(128, activation="relu"))
    model.add(layers.Dropout(0.1))
    model.add(layers.Dense(1, activation="sigmoid"))

    model.compile(
        optimizer='rmsprop',
        loss='binary_crossentropy',
        metrics=[
            keras.metrics.BinaryAccuracy(),
        ],
    )

    #model.summary()

    return model

When I train my model on roughly 1500 samples, I always get my training and validation accuracy completely overlapping and virtually equal, reflected in the graph below. This is making me think there is something fishy going on with my code or in Keras/Tensorflow since the loss is increasing dramatically and you would expect the accuracy to be affected at least somewhat by this. It looks like it is massively overfitting and yet only reporting the accuracy values for the training set or something along those lines. When I then test on a test set, the accuracy is nowhere near the 85 to 90 percent reported on the graph, but rather ~70%.
Any help is greatly appreciated, I have been stuck on this for the longest time. Below is the training code.

image1764×615 116 KB

#Define the number of folds... this will give us an 80/20 split
k = 5
epochs = 100
num_val_samples = len(x_train) // k
scores_binacc = []
scores_precision = []
scores_recall = []
histories = []

#Train the dense model in k iterations
for i in range(k):
    print('Processing fold #', i)
    val_data = x_train[i * num_val_samples : (i + 1) * num_val_samples]
    val_targets = y_train[i * num_val_samples : (i + 1) * num_val_samples]

    print('Validation partition =  ', i * num_val_samples, (i + 1) * num_val_samples)
    print('Training partition 1 = ', 0, i * num_val_samples)
    print('Training partition 2 = ', (i+1) * num_val_samples, len(x_train))

    partial_train_data = np.concatenate(
        [
            x_train[:i * num_val_samples],
            x_train[(i+1) * num_val_samples:]
        ],
        axis=0
    )

    partial_train_targets = np.concatenate(
        [
            y_train[:i * num_val_samples],
            y_train[(i+1) * num_val_samples:]
        ],
        axis=0
    )

    model = build_model()
    h = model.fit(
        partial_train_data,
        partial_train_targets,
        validation_data=(val_data, val_targets),
        epochs=epochs,
        verbose=1
    )

    val_loss, val_binacc = model.evaluate(val_data, val_targets, verbose=0)
    scores_binacc.append(val_binacc)
    #scores_precision.append(val_precision)
    #scores_recall.append(val_recall)
    histories.append(h)

Context:
Maybe you’re overfitting but the underlying relationships are simple so your validation set still has decent accuracy but higher loss.
I feel like the change in accuracy could be caused by shuffling. Are you shuffling your data during training but not on test data? Does order matter for your problem?


TF.js SIG Meeting - Tuesday Oct 19th from 5-6pm PST

Question:
Greetings TF.js Community,
We are looking forward to catching up with you all tomorrow Tuesday October 19th from 5:00-6:00PM PST, after a gap of a couple of months now!
Tomorrow we will share a draft of the RFC process for TF.js contributions. The RFCs will be assigned a sponsor from the Tensforflow team that will help your project be successful. With a more formal contribution to the new sig-tfjs repo, our Tensorflow group can help you connect with partners to collaborate through working groups, and we can offer regular support on working group projects through the SIG. In addition, we will help promote your work and there will be swag involved!
Please find the agenda and the Gmeet link below. Feel free to add any questions or topics for discussion to the agenda.

      docs.google.com

[Public] SIG TF.js Meeting Notes 3

[Public] SIG TF.js Meeting Notes Meeting: 2021-10-19 Tuesday, October 19, 2021, 5:00 – 6:00 pm Pacific Time  Meeting Recording - TODO  Please join this link: meet.google.com/dac-ngjq-okc (US) +1 617-675-4444 PIN: 298 636 519 4797#  Shared Drive...

Cheers, and talk soon!
Masoud on behalf of the TF.js team

Context:
If you missed the SIG TF.js meeting, please see the recording here 1.
We look forward to seeing you at the next meeting!


Question:
We are starting to see this error in some of the unit tests, within recent docker containers we are creating.
2021-07-14 02:13:34.052676: E tensorflow/core/lib/monitoring/collection_registry.cc:77] Cannot register 2 metrics with the same name: /tensorflow/api/keras/optimizers

see this in the tip of develop branch (which is a fork of the upstream/master branch) and in the tip of our r2.6 branch (which is a fork of the upstream/r2.6 branch)
see this in about 10+ of the unit tests…these ones
  21134://tensorflow/python/compiler/xla:xla_test_gpu                            FAILED in 3 out of 3 in 3.6s
  21139://tensorflow/python/keras/benchmarks:eager_microbenchmarks_test_gpu      FAILED in 3 out of 3 in 3.6s
  21144://tensorflow/python/keras/benchmarks:model_components_benchmarks_test_gpu FAILED in 3 out of 3 in 3.5s
  21149://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:antirectifier_benchmark_test_gpu FAILED in 3 out of 3 in 3.4s
  21154://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:bidirectional_lstm_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s
  21159://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:cifar10_cnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.6s
  21164://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_conv_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s
  21169://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_conv_custom_training_benchmark_test_gpu FAILED in 3 out of 3 in 3.7s
  21174://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_hierarchical_rnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s
  21179://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:mnist_irnn_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s
  21184://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:reuters_mlp_benchmark_test_gpu FAILED in 3 out of 3 in 3.5s
  21189://tensorflow/python/keras/benchmarks/keras_examples_benchmarks:text_classification_transformer_benchmark_test_gpu FAILED in 3 out of 3 in 3.7s
  21194://tensorflow/python/ops/numpy_ops:np_interop_test_gpu                    FAILED in 3 out of 3 in 29.1s

Anyone else running into this?
Any insight as to what might be causing this?

Context:
The docker containers we build, use this script to install all the pip packages

      github.com

    ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/tools/ci_build/install/install_pip_packages.sh 82

    #!/usr/bin/env bash
# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

set -e

# Get the latest version of pip so it recognize manylinux2010
wget https://bootstrap.pypa.io/get-pip.py

  This file has been truncated. show original

which in turn seems to install the keras-nightly package

      github.com

    ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/tensorflow/tools/ci_build/install/install_pip_packages.sh#L98 13

# TensorFlow Serving integration tests require the following:
          pip3 install grpcio

# Eager-to-graph execution needs astor, gast and termcolor:
          pip3 install --upgrade astor
          pip3 install --upgrade gast
          pip3 install --upgrade termcolor

# Keras
          pip3 install keras-nightly --no-deps
          pip3 install keras_preprocessing==1.1.0 --no-deps
          pip3 install --upgrade h5py==3.1.0

# Estimator
          pip3 install tf-estimator-nightly --no-deps

# Tensorboard
          pip3 install tb-nightly --no-deps

# Argparse

is that still the correct thing to do? and is that co-related to the error we are getting?
thanks


Postprocess object detection predictions with tfjs-tflite

Question:
I recently created an object detection example 2 using tfjs-tflite 1, which uses the ObjectDetector class 1 to load and use the Object Detector.
Now I wanted to create an object detection without using the ObjectDetector class. I managed to load the model into memory and to prepare the image to make predictions by following the ‘Test model runner’ example, but I’m having problems postprocessing the predictions since the dataSync method that is used in the CodePen example throws an error.
index.js:20 Uncaught (in promise) TypeError: result.dataSync is not a function
    at detect (index.js:20)

Without the dataSync() method I’m getting the following output:
{TFLite_Detection_PostProcess: e, TFLite_Detection_PostProcess:1: e, TFLite_Detection_PostProcess:2: e, TFLite_Detection_PostProcess:3: e}
TFLite_Detection_PostProcess: e {kept: false, isDisposedInternal: false, shape: Array(3), dtype: 'float32', size: 40, …}
TFLite_Detection_PostProcess:1: e {kept: false, isDisposedInternal: false, shape: Array(2), dtype: 'float32', size: 10, …}
TFLite_Detection_PostProcess:2: e {kept: false, isDisposedInternal: false, shape: Array(2), dtype: 'float32', size: 10, …}
TFLite_Detection_PostProcess:3: e {kept: false, isDisposedInternal: false, shape: Array(1), dtype: 'float32', size: 1, …}
[[Prototype]]: Object

Code:
index.js:
const img = document.querySelector("img");
const resultEle = document.querySelector(`.result`);
let objectDetector;

/** Detect objects in image. */
async function detect() {
    resultEle.textContent = "Loading...";
    if (!objectDetector) {
        objectDetector = await tflite.loadTFLiteModel(
            "https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2?lite-format=tflite"
        );
    }

    const start = Date.now();
    let input = tf.image.resizeBilinear(tf.browser.fromPixels(img), [300, 300]);
    input = tf.cast(tf.sub(tf.div(tf.expandDims(input), 127.5), 1), 'int32');

    // Run the inference and get the output tensors.
    let result = objectDetector.predict(input);
    console.log(result)
    const latency = Date.now() - start;
    renderDetectionResult(result);
    resultEle.textContent = `Latency: ${latency}ms`;
}

/** Render detection results. */
function renderDetectionResult(result) {
    const boxesContainer = document.querySelector(".boxes-container");
    boxesContainer.innerHTML = "";
    for (let i = 0; i < result.length; i++) {
        const curObject = result[i];
        const boundingBox = curObject.boundingBox;
        const name = curObject.classes[0].className;
        const score = curObject.classes[0].probability;

        if (score > 0.5) {
            const boxContainer = createDetectionResultBox(
                boundingBox.originX,
                boundingBox.originY,
                boundingBox.width,
                boundingBox.height,
                name,
                score
            );
            boxesContainer.appendChild(boxContainer);
        }
    }
}

/** Create a single detection result box. */
function createDetectionResultBox(left, top, width, height, name, score) {
    const container = document.createElement("div");
    container.classList.add("box-container");

    const box = document.createElement("div");
    box.classList.add("box");
    container.appendChild(box);

    const label = document.createElement("div");
    label.classList.add("label");
    label.textContent = `${name} (${score.toFixed(2)})`;
    container.appendChild(label);

    container.style.left = `${left - 1}px`;
    container.style.top = `${top - 1}px`;
    box.style.width = `${width + 1}px`;
    box.style.height = `${height + 1}px`;

    return container;
}

document.querySelector(".btn").addEventListener("click", () => {
    detect();
});

index.html
<!DOCTYPE html>

    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>TFLITE Web API Example</title>
    <link rel="stylesheet" href="style.css">

TFLITE Web API Object Detection Example

Detect

Any help is highly appreciated. Kind regards,
Gilbert Tanner

Context: @Jason can help here


Error Saving TF-Java Model / Exporter API

Question:
I keep getting the Error No Operation named [StatefulPartitionedCall_2:0] in the Graph
when Using SavedModelBundle.exporter to save the model

Tensorflow Python Version : 2.4.1
Tensorflow Java Version: 0.3.1
Os: Windows 10
GPU/CPU: CPU version

ConcreteFunction serveFunction = savedModel.function("serve_model");
SavedModelBundle.exporter(exportDir)
                .withFunction(serveFunction)
                .export();

To access and inspect Graph operations, i can see the StatefulPartitionedCall_2
But without the : at the end of the operation name.
Iterator<Operation> operationIterator  = serveFunction.graph().operations();
while(operationIterator.hasNext()){
    System.out.println(operationIterator.next().name());
}

code snippet output
Adam/iter
Adam/iter/Read/ReadVariableOp
Adam/beta_1
Adam/beta_1/Read/ReadVariableOp
Adam/beta_2
...
...
...
train_model_labels
StatefulPartitionedCall_1
saver_filename
StatefulPartitionedCall_2
StatefulPartitionedCall_3

Works fine when invoking directly the Op from session.runner()
String checkpointPath = "...";
session.runner()
        .feed("saver_filename:0", checkpointPath)
        .fetch("StatefulPartitionedCall_2:0").run() ;

Error could be reproduced using this scripts which defines than saves the model (credits to Thierry Herrmann)
import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras import layers

def make_model():

    class CustomLayer(keras.layers.Layer):
        def __init__(self, **kwargs):
            super().__init__(**kwargs)
            l2_reg = keras.regularizers.l2(0.1)
            self.dense = layers.Dense(1, kernel_regularizer=l2_reg,
                                      name='my_layer_dense')

        def call(self, data):
            return self.dense(data)
    inputs = keras.Input(shape=(8,))
    x1 = layers.Dense(30, activation="relu", name='my_dense')(inputs)
    outputs = CustomLayer()(x1)
    return keras.Model(inputs=inputs, outputs=outputs)

class CustomModule(tf.Module):

    def __init__(self):
        super(CustomModule, self).__init__()
        self.model = make_model()
        self.opt = keras.optimizers.Adam(learning_rate=0.001)

    @tf.function(input_signature=[tf.TensorSpec([None, 8], tf.float32)])
    def __call__(self, X):
        return self.model(X)

    # the my_train function processes one batch (one step): computes the loss and apply the
    # loss gradient to update the model weights
    @tf.function(input_signature=[tf.TensorSpec([None, 8], tf.float32), tf.TensorSpec([None], tf.float32)])
    def my_train(self, X, y):
        with tf.GradientTape() as tape:
            logits = self.model(X, training=True)
            main_loss = tf.reduce_mean(keras.losses.mean_squared_error(y, logits))
            # self.model.losses contains the reularization loss (see l2_reg above)
            loss_value = tf.add_n([main_loss] + self.model.losses)

        grads = tape.gradient(loss_value, self.model.trainable_weights)
        self.opt.apply_gradients(zip(grads, self.model.trainable_weights))
        return loss_value

# instantiate the module
module = CustomModule()

def save_module(module, model_dir):

    tf.saved_model.save(module, model_dir,
                        signatures={
                            'serve_model' :
                                module.__call__.get_concrete_function(tf.TensorSpec([None, 8], tf.float32)),
                            'train_model' :
                                module.my_train.get_concrete_function(tf.TensorSpec([None, 8], tf.float32),
                                                                      tf.TensorSpec([None], tf.float32))})

MODEL_OUTPUT_DIR ="..."
save_module(module, MODEL_OUTPUT_DIR)
```

Context: For those interested to follow this topic, the discussion is happening on this GitHub issue 2.


Use custom model with tfjs-tflite CORS issue

Question:
The tfjs-tflite library allows you to run TFLite models on the web.
Example:
const tfliteModel = await tflite.loadTFLiteModel('url/to/your/model.tflite');

or
const objectDetector = await tflite.ObjectDetector.create(
    "https://storage.googleapis.com/tfhub-lite-models/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/2.tflite"
);

I’m currently working on a simple Object Detection example 2, which works fine for the example models that are stored on Google Cloud, but I couldn’t get it to work with a custom model stored on Github.
This gives me a CORS error:
Access to fetch at 'https://github.com/TannerGilbert/TFLite-Object-Detection-with-TFLite-Model-Maker/raw/master/model.tflite' from origin 'null' has been blocked by CORS policy: The 'Access-Control-Allow-Origin' header contains multiple values 'https://render.githubusercontent.com https://viewscreen.githubusercontent.com https://viewscreen-lab.githubusercontent.com', but only one is allowed. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Therefore, I wanted to ask what’s the simplest way to solve the error or perhaps what the best platform is to store your model on the web for free.
Any help is highly appreciated. Kind regards,
Gilbert Tanner

Context:
Thanks @Jason for the detailed explanations!
@Gi_T, for your use case, it is possible to use the “raw.githubusercontent.com” link to get the model file which has “access-control-allow-origin” header set to “*”.
Try:
https://raw.githubusercontent.com/TannerGilbert/TFLite-Object-Detection-with-TFLite-Model-Maker/master/model.tflite 5
Thanks!


Question:
Hello,
I am trying to apply the command tf-opt --promote-resources-to-args on the following TF/MLIR code.
func @counter(%arg0: tensor) → tensor {
%1 = “tf.VarHandleOp”() {container = “”, shared_name = “x”} : () → tensor<!tf_type.resource<tensor>>
%2 = “tf.ReadVariableOp”(%1) : (tensor<!tf_type.resource<tensor>>) → tensor
%3 = “tf.Add”(%arg0, %2): (tensor, tensor) → tensor
“tf.AssignVariableOp”(%1, %3) {device = “”} : (tensor<!tf_type.resource<tensor>>, tensor) → ()
%4 = “tf.ReadVariableOp”(%1) : (tensor<!tf_type.resource<tensor>>) → tensor
return %4: tensor
}
It does nothing because the function’s name is not @main. Is there a way to apply this command on every function whatever its name is ?
The result I need is :
func @counter(%arg0: tensor, %arg1: tensor {tf.aliasing_output = 1 : i64, tf.resource_name = “x”}) → (tensor, tensor) {
%0 = “tf.Add”(%arg0, %arg1) : (tensor, tensor) → tensor
return %0, %0 : tensor, tensor
}
Best regards,
HP

Context:
Unfortunately this isn’t possible as-is since it is hard-coded here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/tensorflow/transforms/promote_resources_to_args.cc#L348-L349 6
Feel free to send a patch that would make it an option of the pass to specify the entrypoint name!


Unable to assign element of a 1D array obtained from tensor.data() as value of an object

Question:
[what i want to do]
I created a rank-3 tensor.
Then I got a flattened 1D array of all the values in this tensor by using tensor.data() method.
I want to assign each element of this flattened array as value of an object.

[what is the problem]
I’m unable to obtain the individual elements of the array using a.data().then((data) => { card.value = data[i] }); .
console.log(card.value) returns undefined.
However, using card.value = a.dataSync()[i]; seems to work, instead.

[main.js]
import * as tf from "@tensorflow/tfjs";

import Card from "./Card.js";

// create a rank-3 tensor
const a = tf.randomNormal([4, 3, 2]);
a.print();

// assign values in the tensor to a series of div object
for (let i = 0; i < a.size; i += 1) {
        // create card object
        const card = new Card(i, "card " + String(i), "96px", "96px");

        // assign a value to card
        // [method 1] using synchronous method works
        // card.value = a.dataSync()[i];

        // [method 2] using asynchronous method is not working ...
        a.data().then((data) => { card.value = data[i] });

        console.log(card.value);

[Card.js]
export default class Card {
    // constructor
    constructor(_idx, _name, _width, _height, _posx, _posy, _posz, _value) {
        this.idx = _idx;
        this.name = _name;

        this.width = _width;
        this.height = _height;

        this.posx = _posx;
        this.posy = _posy;
        this.posz = _posz;

        this.value = _value;
    }
}

Context: Could anybody kindly help to take a look of the issue shown above and advise what is missing? Thanks.


SIG Build October Meeting: October 5 @ 2pm

Question: SIG Build’s next meeting will be tomorrow, Tuesday, October 5, at 2pm Pacific time. Find the meeting details at bit.ly/tf-sig-build-notes 5, and feel free to suggest new agenda items.

Context:
Thanks to everyone who attended the meeting yesterday. Here are are a couple of notable points from the discussion:

Our November meeting on Nov. 2 may be affected by Daylight Savings Time if you are in Europe.

The TF DevInfra team is ramping up a couple of new members with projects including manylinux2014 support for our build environment and author notifications when a GitHub PR gets rolled back.

manylinux development continues to be complicated. See the notes for details.

Thanks, and see you in November!


Integrating tf.recommender with TFX pipeline ran on kubeflow

Question:
I am currently working on integrating a tf.recommender 1 model into an existing TFX 1 pipeline to provide on-service recommendations. I am quite new to both TFX and tf.recommender and am not seeing any resources on integrating the two. I want to be sure I am implementing best practices–in particular with TFT 1 and TFMA 1. Does anyone know of existing docs that may help me with this, or better yet, existing example pipelines?
Thanks!

Context: maybe @Robert_Crowe have some links to share


How to separate dataset to validate CNN?

Question:
Hello im working training to train a CNN with two datasets that I label manually with negative and positive.(80x60 depth images in each matrix)
# dimensions of our images.
img_width, img_height = 80, 60
n_positives_img, n_negatives_img = 17874, 26308
n_total_img = 44182

#Imports of datasets inside Drive
ds_negatives = np.loadtxt('/content/drive/MyDrive/Colab Notebooks/negative_depth.txt')
ds_positives = np.loadtxt('/content/drive/MyDrive/Colab Notebooks/positive_depth.txt')

#Labeled arrays for datasets
arrayceros = np.zeros(n_negatives_img)
arrayunos = np.ones(n_positives_img)

#Reshaping of datasets to convert separate them
arraynegativos= ds_negatives.reshape(( n_negatives_img, img_width, img_height))
arraypositivos= ds_positives.reshape((n_positives_img, img_width, img_height))

#Labeling datasets with the arrays
ds_negatives_target = tf.data.Dataset.from_tensor_slices((arraynegativos, arrayceros))
ds_positives_target = tf.data.Dataset.from_tensor_slices((arraypositivos, arrayunos))

#Concatenate 2 datasets and shuffle them
ds_concatenate = ds_negatives_target.concatenate(ds_positives_target)
datasetfinal = ds_concatenate.shuffle(n_total_img)

But when I try to separate my dataset 80/20 to validate my CNN:
trainingdataset, validatedataset = train_test_split(datasetfinal, test_size=0.2, random_state=25)

I get this error:
TypeError: Singleton array arrayshapes: ((80, 60), ()), types: (tf.float64, tf.float64)>, dtype=object) cannot be considered a valid collection.

Any ideas? Thank in advance!!!

Context:
It’s impossible to split tensorflow dataset object by passing it to train_test_split from sklearn. You can choose the number of validation samples, which should be int number, and use the following example:
valid_ds = datasetfinal.take(n_samples)
train_ds = datasetfinal.skip(n_samples)

It does what the method says: takes first n_samples from the dataset and skips all the rest or skips the first n_samples and takes all the rest.


I was working on Mask RCNN so while executing demo.ipynb on jupyter notebook i got an error can anyone please help me out

Question:
AttributeError Traceback (most recent call last) in 14 sys.path.append(ROOT_DIR) # To find local version of the library 15 from mrcnn import utils —> 16 import mrcnn.model as modellib 17 from mrcnn import visualize 18 # Import COCO config
~\ComputerVisionProject\Mask_RCNN_CustomDataset\Mask_RCNN-master\Mask_RCNN-master\mrcnn\model.py in 253 254 → 255 class ProposalLayer(KE.Layer): 256 “”"Receives anchor scores and selects a subset to pass as proposals 257 to the second stage. Filtering is done based on anchor scores and
AttributeError: module ‘keras.engine’ has no attribute 'Layer’

Context:
Are you using this repo? I think its for tf version 1.x

      github.com

matterport/Mask_RCNN 50

  Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow


Inverse kinematic approximation with neural network

Question:
Good morning everyone,
I’ll try to briefly explain the context and then the problem I’m facing:
Context: I am using and testing a collaborative Robot. This Robot has been provided to me with a library in python that allows to acquire signals from the robot (currents, velocity, positions, I/O etc) and to command it, in joint and end-effector (EE) coordinates. There are also available the functions of Direct and Inverse Kinematics (DK and IK).
For my curiosity, I was interested in generating a trajectory (in end-effector coordinates) in order to move it within a conical area [I attach a link to the video that shows the movement in question].
LINK: https://www.youtube.com/watch?v=CExtMfvRabo 4
From the robot, moreover, it is possible to save the .csv file containing the trajectories, in joint coordinates, of the single joints.
Initially, not knowing the “shape” that should have the trajectory (in end-effector coordinates) of the movement that I was interested in reproducing, I was able, manually moving the robot in gravity compensation mode, to acquire the trajectories of the individual joints. At this point, using the Direct Kinematics algorithm, I obtained the movement of the consequent end-effector [I attach photos of 2 3D graphs: the first in which I plot the 3 coordinates x,y,z and the second, in which I plot roll, pitch, yaw].
End Effector Angular Displacement 1
End Effector Position Displacement
Here the problem was born.
Problem: out of curiosity, I tried to use the Inverse Kinematics algorithm on the points obtained from the DK and the algorithm returned the error: “Singular Trajectory”. But the robot was able to move according to that trajectory, the problem should be in the calculation of the IK, which probably finds multiple/infinite solutions.
To overcome this limitation I used a Neural Network developed in Python using Tensorflow (Keras) to try to approximate the IK. I will preface this by saying that I have never used Keras or Tensorflow, so I may have made some conceptual errors. I have consulted the API of Keras and also the guide proposed in this link
LINK: https://machinelearningmastery.com/deep-learning-models-for-multi-output-regression/ 1
In my PC I use:

Visual Studio Code for programming in python;
python 3.9.5
Keras 2.6.0;

I thought of the neural network this way: 6 input nodes (corresponding to the 6 coordinates of the end-effector) and 6 output nodes (the 6 coordinates of the joints). The training set consists of a .csv file containing the 6 coordinates of the end-effector computed via the DK run on a .csv file containing the trajectories of the 6 joints. The file containing the joint coordinates is the Label file.
Below I attach the code of the network implementation.
from numpy import loadtxt
import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
from numpy import array

# Model definition
def get_model(I_N_L_1, I_N_L_2, I_N_L_3, I_N_L_4, I_N_L_5 ,n_inputs, n_outputs):

	model = Sequential()
	model.add(Dense(I_N_L_1, input_dim=n_inputs, kernel_initializer='he_uniform', activation='relu'))
	model.add(Dense(I_N_L_2, activation='relu'))
	model.add(Dense(I_N_L_3, activation='relu'))
	model.add(Dense(I_N_L_4, activation='relu'))
	model.add(Dense(I_N_L_5, activation='relu'))
	model.add(Dense(n_outputs))
	model.compile(loss='mae', optimizer='adam', metrics=["mae"])
	return model

# Load Training set csv
dataset_EF = loadtxt('WeldingProve.csv', delimiter=',')
x_train = dataset_EF[0:1700,0:6]
print('shape: ',x_train.shape)

# Load Label set csv
dataset_joints = loadtxt('EF_from_WeldingProve.csv', delimiter=',')
y_train = dataset_joints[0:1700,0:6]
print('shape: ',y_train.shape)

# Test set definition
x_test = dataset_EF[1701:,0:6]
print('shape: ',x_test.shape)

# Label of the test set definition
y_test = dataset_joints[1701:,0:6]
print('shape: ',y_test.shape)

# Number of nodes in the hidden layers
I_N_L_1 = 192
I_N_L_2 = 36
I_N_L_3 = 6
I_N_L_4 = 36
I_N_L_5 = 192

# Number of nodes in the input and output layers
n_inputs = 6
n_outputs = 6

# calling the "get_model" function
model = get_model(I_N_L_1, I_N_L_2, I_N_L_3, I_N_L_4, I_N_L_5 ,n_inputs, n_outputs)
es = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=5)

# fit model
model.fit(x_train, y_train, verbose=1, epochs=600)

# saving the model
model.save("Test_Model.h5")

# Testing procedure
pred = []

# Computing the Prediction on the Test Set
for i in range(len(x_test)-1):
	b = [x_test[i][0], x_test[i][1], x_test[i][2], x_test[i][3], x_test[i][4], x_test[i][5]]
	ToBePredicted = array([b])
	Prediction = model.predict(ToBePredicted)
	a = [Prediction[0][0], Prediction[0][1], Prediction[0][2], Prediction[0][3], Prediction[0][4], Prediction[0][5]]

# Computing the mean vector of the error for each predicted joint trajectory
average_vector = []
sum = 0
average = 0
for j in range(6): # colonne
	for i in range(len(y_test)-1): #righe
		sum = sum + (pred[i][j] - y_test[i][j])

	average = sum/len(y_test)
	average_vector.append(average)

	average = 0
	sum = 0

print('average_vector: ', average_vector)

# Computing the standard deviation vector of the error for each predicted joint trajectory
sum = 0
std_vector = []
for j in range(6): # colonne
	for i in range(len(y_test)-1): #righe
		sum = sum + ((pred[i][j] - y_test[i][j]) - average_vector[j])**2

	std = (sum/len(y_test))**(0.5)
	std_vector.append(std)

	std = 0
	sum = 0

print('std_vector: ', std_vector)

My questions are the following:

once I have trained the neural network, even using a very large training set, I get predictions that are not good. Can you suggest me how to improve these predictions, perhaps going to act on the parameters of the network,

Is it necessary to pre-process the training data and its labels? If yes, which technique should I apply?

Trying to change the number of nodes in the various layers of the network, I saw that the performance changes, even a lot. Do you have advice on the “shape” to give to the network ?

Are there any other solutions that can be used to estimate the IK of the robot ?

Context:
I’m not an expert in robotics. So my comments are only regarding the model architecture and training.

When you call model.fit() you can pass you test data to the argument “validation_data”, and the model will automatically calculate loss and metrics for both train and validation set. TensorFlow has MeanSquaredError and MeanAbsolutePercentageErrror in addition to MAE that you use.
In the EarlyStopping callback you should define monitor=‘val_loss’ and restore_best_weights=True. In this case the training will be stopped, when validation loss starts worsening, and the model will roll back to the optimal state, when best val_loss was reached. At present you monitor training loss, which does not say anything about overfitting.
Check the scale of the coordinates used as input features. If they are not in range 0-1, input data requires normalization. Keras has Normalization layer, which could be used to ensure that all data passed to the model is normalized identically.
Usually number of units in the dense layers gradually decreases. You defined 5 layers with units decreasing and then increasing like V-shape.
If all this does not improve the result, probably you should add more features like previous positions of the object, or it’s speed, or something else.


How do I build a custom voice recognition model for multiple people?

Question:
Is there a quick and easy import tool for custom voice data?
Is there a free local training Speech Recognition to text tool (including exporting model for tf.js) for custom raw voice data?
Can I run tf.js to automatically learn unknown speech sounds and integrate them into existing model examples?
ps: I don’t want to train my custom data through a cloud-based paid service

Context:
Welcome to the community. If you just need sound recognition you can try Teachable Machine that makes it easy to recognize short form sounds eg 1 second in length. I have not seen a full voice recognition conversion yet as those tend to be quite large in file size, but sound recognition is most certainly possible. check:

      teachablemachine.withgoogle.com

Teachable Machine 16

  Train a computer to recognize your own images, sounds, & poses.
        A fast, easy way to create machine learning models for your sites, apps, and more – no expertise or coding required.

And then select audio project. If you like what it trains in browser you can click download on top right and save the model files generated to your computer. All training is done in browser using TensorFlow.js so no server is used here other than to deliver the initial webpage so your sounds are never sent to a server.
If you want to do voice recognition in JavaScript it actually exists via the WebSpeech API:

      developer.mozilla.org

Using the Web Speech API - Web APIs | MDN 11

  The Web Speech API provides two distinct areas of functionality — speech recognition, and speech synthesis (also known as text to speech, or tts) — which open up interesting new possibilities for accessibility, and control mechanisms. This article...

You do not need TensorFlow.js to use that. It is part of the browser implementation and will use whatever OS level voice recognition exists.
Good luck!


How to load tflite model into chrome extension using tfjs

Question:
Hi tensors,
I have seen chrome extension 6 in the tfjs-examples, I am trying to make it for my model. But for that also, I am not able to get required tfjs.min.js and tfjs.js lib into extension.
I have added lib links to popup.html and tried to load the model in background.js.

Uncaught (in promise) ReferenceError: tflite is not defined

I have added package.json file —
{
“name”: “xxx”,
“version”: “0.0.1”,
“description”: “Use tfjs model.predict in a chrome extension”,
“scripts”: {
“copy”: “copy content.js dist/”,
“build”: “parcel build background.js -d dist/ -o background --no-minify && npm run copy”,
“watch”: “npm run copy && parcel watch background.js --hmr-hostname localhost -d dist/ -o background”
},
“license”: “Apache 2.0”,
“devDependencies”: {
“babel-core”: “^6.26.3”,
“babel-plugin-transform-runtime”: “^6.23.0”,
“babel-polyfill”: “^6.26.0”,
“babel-preset-env”: “^1.6.1”,
“clang-format”: “^1.2.3”,
“parcel-bundler”: “^1.9.4”
},
“resolutions”: {
“is-svg”: “4.3.1”,
“node-fetch”: “2.6.1”,
“vega”: “5.17.3”,
“glob-parent”: “5.1.2”,
“postcss”: “8.2.10”
},
“dependencies”: {
“@tensorflow/tfjs”: “^3.9.0”
}
}
The manifest.json is like this—

{
“name”: “xxxxx”,
“description”: “xxxxxx”,
“version”: “1.0”,
“manifest_version”: 2,
“browser_action”: {
“default_icon”: “icon.png”,
“default_popup”: “popup.html”,
“default_title”: “Chrome Extension”
},
“permissions”: [
“<all_urls>”,
“activeTab”
],
“background”: {
“scripts”: [“background.js”],
“persistent”: true
},
“content_scripts”: [
{
“matches”: [“http:///”, “https:///”],
“js”: [“content.js”],
“all_frames”: true,
“run_at”: “document_start”
}
],
“commands”: {
“_execute_browser_action”: {
“suggested_key”: {
“default”: “Ctrl+Shift+F”,
“mac”: “MacCtrl+Shift+F”
},
“description”: “Opens popup.html”
}
},
“content_security_policy”: “script-src ‘self’ https://cdn.jsdelivr.net ‘unsafe-eval’; object-src ‘self’”
}

Background.js is like —
// Copyright 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
//import * as tf from ‘tfjs’;
/**
Function callled when extension installed into browser.
*/
chrome.runtime.onInstalled.addListener(function() {
// load TFLite model into browser
async function load_tflite_model() {
const tfliteModel = await tflite.loadTFLiteModel(
“https://storage.googleapis.com/tfweb/models/cartoongan_fp16.tflite 2”
);
console.log(“tfliteModel…”,tfliteModel)
}
load_tflite_model();
});
Thank
Neha Soni

Context:
Hi there,
I have seen chrome extension in the tfjs-examples, I am trying to make it for my model. But for that also, I am not able to get required tfjs.min.js and tfjs.js lib into extension.
I have added lib links to popup.html and tried to load the model in background.js.

Uncaught (in promise) ReferenceError: tflite is not defined

I have added package.json file —
{
“name”: “xxx”,
“version”: “0.0.1”,
“description”: “Use tfjs model.predict in a chrome extension”,
“scripts”: {
“copy”: “copy content.js dist/”,
“build”: “parcel build background.js -d dist/ -o background --no-minify && npm run copy”,
“watch”: “npm run copy && parcel watch background.js --hmr-hostname localhost -d dist/ -o background”
},
“license”: “Apache 2.0”,
“devDependencies”: {
“babel-core”: “^6.26.3”,
“babel-plugin-transform-runtime”: “^6.23.0”,
“babel-polyfill”: “^6.26.0”,
“babel-preset-env”: “^1.6.1”,
“clang-format”: “^1.2.3”,
“parcel-bundler”: “^1.9.4”
},
“resolutions”: {
“is-svg”: “4.3.1”,
“node-fetch”: “2.6.1”,
“vega”: “5.17.3”,
“glob-parent”: “5.1.2”,
“postcss”: “8.2.10”
},
“dependencies”: {
“@tensorflow/tfjs”: “^3.9.0”
}
}
The manifest.json is like this—

{
“name”: “xxxxx”,
“description”: “xxxxxx”,
“version”: “1.0”,
“manifest_version”: 2,
“browser_action”: {
“default_icon”: “icon.png”,
“default_popup”: “popup.html”,
“default_title”: “Chrome Extension”
},
“permissions”: [
“<all_urls>”,
“activeTab”
],
“background”: {
“scripts”: [“background.js”],
“persistent”: true
},
“content_scripts”: [
{
“matches”: [“http:///”, “https:///”],
“js”: [“content.js”],
“all_frames”: true,
“run_at”: “document_start”
}
],
“commands”: {
“_execute_browser_action”: {
“suggested_key”: {
“default”: “Ctrl+Shift+F”,
“mac”: “MacCtrl+Shift+F”
},
“description”: “Opens popup.html”
}
},
“content_security_policy”: “script-src ‘self’ https://cdn.jsdelivr.net ‘unsafe-eval’; object-src ‘self’”
}

Background.js is like —
// Copyright 2018 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
//import * as tf from ‘tfjs’;
/**
Function callled when extension installed into browser.
*/
chrome.runtime.onInstalled.addListener(function() {
// load TFLite model into browser
async function load_tflite_model() {
const tfliteModel = await tflite.loadTFLiteModel(
“https://storage.googleapis.com/tfweb/models/cartoongan_fp16.tflite”
);
console.log(“tfliteModel…”,tfliteModel)
}
load_tflite_model();
});
Thank
Neha Soni


Setup your favorite editor to develop Keras

Question:
Here is list of editors that you can use to develop Keras.
The steps to setup each of them is provided.
Welcome to reply to this topic to add more.

GitHub Codespaces
This is the easiest option. It helps you setup the environment with one click.
You can click “Code → new codespace” on your fork’s web page to open it in GitHub Codespaces 14.
You can start coding and running the tests there right away.
However, Codespaces is only available in beta. You need to request early access to use it.

Visual Studio Code
This is also an easy option for beginners.
Clone your fork of the repo to your computer.
Open Visual Studio Code 2.
Install the Remote-Containers extension 8.
Open the cloned folder and click “Reopen in Container” in the popup notification.
You can start coding and running the tests there right away.

Context:
Thank you,
I think we are missing the linting tools in keras/requirements.txt at master · keras-team/keras · GitHub 1
See my comment at https://github.com/keras-team/keras/pull/15006#pullrequestreview-716500818 1
If it is complete you can close my June contribution offer at Vscode/Github codespaces
We are also waiting for the same for TF core at Tensorflow with Gitbub Codespaces 1.
For TF Addons it was rejected more then 1 year ago but probably we could re-valuate it:

      github.com/tensorflow/addons

      Add initial vscode devcontainer support 2

      tensorflow:master ← bhack:vscode_devcontainer

        opened

        Mar 15, 2020

          bhack

          +61
          -3

    Initial support for Vscode devcontainer. See https://github.com/tensorflow/addon…s/issues/1305

Since May 2020 we maintained the .devcontainers for TF, Keras (not standalone) and TF Addons at:

      github.com

vscode-dev-containers/repository-containers/github.com/tensorflow at main ·... 7

  main/repository-containers/github.com/tensorflow

  A repository of development container definitions for the VS Code Remote - Containers extension and GitHub Codespaces - vscode-dev-containers/repository-containers/github.com/tensorflow at main · m...


Question: I have built the model, trained it. Again I was trying to set weights to the model from a file saved in my local machine but I was getting errors. So, what are the things that we should care about while setting weights in the trained models?Thank you

Context:
What methods did you use to save the model weights and use them again?
If you need to reuse only the weights, that’s how you can save and reload them:
model.save_weights(‘name_or_path.h5’)
new_model.load_weights(‘name_or_path.h5’)
Architecture of “new_model” should be identical to “model”.


MobileNetV3: small change in architecture between TF2.5 and 2.6

Question:
For tf.keras.applications.MobileNetV3 (large or small), there’s been a slight change to the architecture from TF <=2.5 to TF 2.6. Specifically the GlobalAveragePooling2D layer happens before “Conv_2” in TF2.6, but after “Conv_2” (and it’s non-linear activation) in TF2.5.
These operations don’t commute, so the architectures are slightly different. Both versions point to the same pre-trained weights, so their architectures ought to be the same.
I haven’t checked if this degrades the performance of the pretrained models.
My interest in this is mostly that it’s a breaking change to the API: MobileNetV3Large(include_top=False) will output a tensor of shape [?, 1, 1, 1280] starting with TF2.6 compared to a tensor of shape [?, 7, 7, 1280] with TF <=2.5 (assuming an input of shape [?, 224, 224, 3]).

Context:
This is a bug fix.
The two ops don’t quite commute but they commute well enough that both versions of the model do well with the weights.
You are correct about the change in the feature vector shape. The new version is the one that is “correct”.


Proper use of Keras ImageDataGenerator: Create Masks for Segmentation and sample_weight parameter

Question:
Hello all,
I want to do Image Data Augmentation for an Semantic Segmentation task. Therefore, I want to use the ImageDataGenerator from Keras, together with the flow() method, because my data is in Numpy arrays and does not need to be loaded from a folder. Since this is a segmentation task, I need to augment the image and the corresponding mask. I do this by following the last example in the API reference (ImageDataGenerator 4 ) and accordingly using two different generators for image and mask with the same data_gen_args. I only want to rotate, flip and move my images, so I want to use the arguments rotation_range, width_shift_range, height_shift_range,horizontal_flip, vertical_flip.
Accordingly, I want to get masks that are 8 bit images of the shape (128,128,1) like the input mask and also contain only the classes of the input mask (all integer values). And this is exactly where the problem lies, the masks I get are 32-bit floats, which do not contain integer values at all. Even when specifying the argument dtype = “uint8” the code always returns only float32 masks. I have not found an example that fixes this problem? Is there a trick that can be used ?
Another problem in connection with the ImageDataGenerator is sample_weight. As my dataset is quite unbalanced, I would like to use them. In a segmentation task, I think the sample_weight parameter in the flow() method would have to correspond to another mask containing the respective class_weight for the class of each pixel in the original mask. If I do it this way, I get sample_weight back as well, but it seems to me that these weights, similar to the mask, are not correct either, as my UNet does not train well with them anymore. In the meantime I use a third ImageDataGenerator only for the sample_weight, so the training works better, but I hardly think this is the right approach. However, I have not found an example for the correct use. Therefore I hope that the community can help me with their experience.
Thank you.
Kind regards,
Jonas

Context:
Hi Jonas
ImageDataGenerator has been superseded by Keras Preprocessing Layers 5 for data preprocessing, to be used together with the tf.data 2 API. However, at this time, you cannot yet do joint preprocessing of the image and mask using Keras Preprocessing Layers so I cannot recommend that route yet.
In my experience, the following data augmentation frameworks support image segmentation use cases directly:

Albumentations 6
ImgAug 6

Your best way for now is to use one of these libraries and then format your dataset as a Python generator (or tf.data.Dataset through tf.data.Dataset.from_generator 4)
The limitations of these approaches is that they do the data transformations in Python rather than TF operations and therefore cannot be saved to SavedModel and deployed in production.
Until we have a segmentation-compatible, Keras Preprocessing Layer implemented with TF ops, I advise you to special-case inference in your model setup. You can use Python libraries for data preprocessing for training and evaluation, but implement the minimal necessary inference-time data transformations (JPEG decompression, size, scale, …) using TF functions and Keras Preprocessing Layers. For example tf.io.decode_image and tf.keras.layers.Resizing.


SIG Build September Meeting: September 7 (today!) @ 2pm

Question:
SIG Build’s next meeting will be today, Tuesday, September 7, at 2pm Pacific time. Find the meeting details at bit.ly/tf-sig-build-notes 6, and feel free to suggest new agenda items.
I hope those of you in the USA had a relaxing Labor Day!

Context:
Thanks to everyone who attended this month’s meeting. Here’s a summary of some of the main points:

About 20 vulnerabilities are expected to be patched on top of TF 2.6; there will be patch releases for previous releases as well.
The DevInfra team is looking at bumping some dependencies, but it’s slow going because of how much testing is necessary.
manylinux2014 or perennial manylinux is a requirement for CUDA 11.4. The DevInfra team is working on this.
I’m looking at Docker containers again (finally!) and will be checking out the PYPA containers to see if we can use those directly instead of using our own custom toolchain, which we needed for Ubuntu previously.

Check out the notes, linked in the first post, for full details.


Locally connected 2D layer without summation over colors

Question:
I want to create a neural network with a locally connected layer but without summation over the 3rd dimension (colors) of the inputs.
I saw in the docs of LocallyConnected2D 2 that there is no “group” argument as in the conv2d.
Is there a way to do that?

Context:
Do you mean something like this?

      github.com/keras-team/keras

      Feature request: In-plane locally-connected 2D convolutions 6

        opened

        Mar 29, 2018

          closed

        Jun 25, 2021

          tsoernes

    I'm requesting a convolution that works on each channel separately using the sam…e filter (as https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/contrib/layers/conv2d_in_plane), and but uses different filters spatially (like LocallyConnected2D).

One approach would be to modify `LocallyConnected2D`:
```
individual_channels = tf.split(inputs, inputs.shape[3], -1)
convs = []
for channel in individual_channels:
    conv = K.local_conv2d(channel, self.kernel, self.kernel_size, self.strides,
       (self.output_row, self.output_col), self.data_format)
    convs.append(conv)
outputs = tf.concat(convs, -1)
```
where
```
self.kernel_shape = (output_row * output_col,
    self.kernel_size[0] * self.kernel_size[1], 1)
```
But the above approach is very slow.