with op Tile must be a compile-time constant.
Author: gumowangfei
Pull request: no
Labels: name: TensorFlow

TPU tensorflow2.6.0

TFLongTransformer

model.fit(
    train_data, tf.gather(train_label, train_idx),
    validation_data=(val_data, tf.gather(train_label, val_idx)),
    epochs=10,
    batch_size=BATCH_SIZE,
    callbacks=[checkpoint_callback],
  )

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-51-69b9068b95d6> in <module>()
     21     epochs=10,
     22     batch_size=BATCH_SIZE,
---> 23     callbacks=[checkpoint_callback],
     24   )
     25   # model.load_weights('/content/ckpt_'+str(fold)+'_model.h5')

13 frames
/usr/local/lib/python3.7/dist-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1187               logs = tmp_logs  # No error, now safe to assign to logs.
   1188               end_step = step + data_handler.step_increment
-> 1189               callbacks.on_train_batch_end(end_step, logs)
   1190               if self.stop_training:
   1191                 break

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in on_train_batch_end(self, batch, logs)
    433     """
    434     if self._should_call_train_batch_hooks:
--> 435       self._call_batch_hook(ModeKeys.TRAIN, 'end', batch, logs=logs)
    436 
    437   def on_test_batch_begin(self, batch, logs=None):

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in _call_batch_hook(self, mode, hook, batch, logs)
    293       self._call_batch_begin_hook(mode, batch, logs)
    294     elif hook == 'end':
--> 295       self._call_batch_end_hook(mode, batch, logs)
    296     else:
    297       raise ValueError('Unrecognized hook: {}'.format(hook))

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in _call_batch_end_hook(self, mode, batch, logs)
    313       self._batch_times.append(batch_time)
    314 
--> 315     self._call_batch_hook_helper(hook_name, batch, logs)
    316 
    317     if len(self._batch_times) >= self._num_batches_for_timing_check:

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in _call_batch_hook_helper(self, hook_name, batch, logs)
    351     for callback in self.callbacks:
    352       hook = getattr(callback, hook_name)
--> 353       hook(batch, logs)
    354 
    355     if self._check_timing:

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in on_train_batch_end(self, batch, logs)
   1026 
   1027   def on_train_batch_end(self, batch, logs=None):
-> 1028     self._batch_update_progbar(batch, logs)
   1029 
   1030   def on_test_batch_end(self, batch, logs=None):

/usr/local/lib/python3.7/dist-packages/keras/callbacks.py in _batch_update_progbar(self, batch, logs)
   1098     if self.verbose == 1:
   1099       # Only block async when verbose = 1.
-> 1100       logs = tf_utils.sync_to_numpy_or_python_type(logs)
   1101       self.progbar.update(self.seen, list(logs.items()), finalize=False)
   1102 

/usr/local/lib/python3.7/dist-packages/keras/utils/tf_utils.py in sync_to_numpy_or_python_type(tensors)
    514     return t  # Don't turn ragged or sparse tensors to NumPy.
    515 
--> 516   return tf.nest.map_structure(_to_single_numpy_or_python_type, tensors)
    517 
    518 

/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, **kwargs)
    867 
    868   return pack_sequence_as(
--> 869       structure[0], [func(*x) for x in entries],
    870       expand_composites=expand_composites)
    871 

/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/nest.py in <listcomp>(.0)
    867 
    868   return pack_sequence_as(
--> 869       structure[0], [func(*x) for x in entries],
    870       expand_composites=expand_composites)
    871 

/usr/local/lib/python3.7/dist-packages/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)
    510   def _to_single_numpy_or_python_type(t):
    511     if isinstance(t, tf.Tensor):
--> 512       x = t.numpy()
    513       return x.item() if np.ndim(x) == 0 else x
    514     return t  # Don't turn ragged or sparse tensors to NumPy.

/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in numpy(self)
   1092     """
   1093     # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
-> 1094     maybe_arr = self._numpy()  # pylint: disable=protected-access
   1095     return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
   1096 

/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)
   1060       return self._numpy_internal()
   1061     except core._NotOkStatusException as e:  # pylint: disable=protected-access
-> 1062       six.raise_from(core._status_to_exception(e.code, e.message), None)  # pylint: disable=protected-access
   1063 
   1064   @property

/usr/local/lib/python3.7/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: 9 root error(s) found.
  (0) Invalid argument: {{function_node __inference_train_function_367359}} Input 1 to node `model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile` with op Tile must be a compile-time constant.

XLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.

	 [[{{node model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile}}]]

	 [[model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1]]
	 [[TPUReplicate/_compile/_13050048924464197292/_9]]
	 [[TPUReplicate/_compile/_13050048924464197292/_9/_438]]
  (1) Invalid argument: {{function_node __inference_train_function_367359}} Input 1 to node `model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile` with op Tile must be a compile-time constant.

XLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.

	 [[{{node model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile}}]]

	 [[model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1]]
	 [[TPUReplicate/_compile/_13050048924464197292/_9]]
	 [[tpu_compile_succeeded_assert/_1014831673988967065/_10/_397]]
  (2) Invalid argument: {{function_node __inference_train_function_367359}} Input 1 to node `model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile` with op Tile must be a compile-time constant.

XLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concrete values at compile time. This error means that a shape or dimension argument could not be evaluated at compile time, usually because the value of the argument depends on a parameter to the computation, on a variable, or on a stateful operation such as a random number generator.

	 [[{{node model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile}}]]

	 [[model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1]]
	 [[TPUReplicate/_compile/_13050048924464197292/_9]]
	 [[TPUReplicate/_compile/_13050048924464197292/_9/_416]]
  (3) Invalid argument: {{function_node __inference_train_function_367359}} Input 1 to node `model/tf_longformer_model/longformer/encoder/layer_._0/attention/self/cond_1/Tile` with op Tile must be a compile-time constant.

XLA compilation requires that operator arguments that represent shapes or dimensions be evaluated to concre ... [truncated]


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I have the same issue. I am guessing it's because of random attention mechanism of Longformer



I have the same issue

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi there! Possibly late for this issue, but an issue with the same error message (on a different model) was fixed [here](https://github.com/huggingface/transformers/issues/18476).

The issue was fixed by setting fixed shapes in the input.