Repository: tensorflow/tensorflow (language: cpp)

Issue:
Executor failed to create kernel. Not found: No registered 'NGraphVariable' OpKernel for GPU devices compatible with node {{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()     .  Registered:  device='CPU'
### **Executor failed to create kernel. Not found: No registered 'NGraphVariable' OpKernel for GPU devices compatible with node {{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()     .  Registered:  device='CPU'**

**System information**
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 18.04
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary):
source
- TensorFlow version (use command below):
Tensorflow GPU 1.11.0-rc1
- Python version:
python 3.6.6
- CUDA/cuDNN version:
cuda 10 cuDNN 7.2
- GPU model and memory:
Quadro M4000
8126MiB

You can collect some of this information using our environment capture [script](https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh)
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

While giving the command for training tensorflow model _(tensorflow models_master/research/object_detection)_ for object detection, it gives an error given below

### **Command:**
python model_main.py --logtostderr --train_dir=training/ --pipeline_config_path=training/pipeline.config

### **Error:**
E tensorflow/core/common_runtime/executor.cc:623] Executor failed to create kernel. Not found: No registered 'NGraphVariable' OpKernel for GPU devices compatible with node {{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()
    .  Registered:  device='CPU'

     [[{{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'NGraphVariable' OpKernel for GPU devices compatible with node {{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()
    .  Registered:  device='CPU'

     [[{{node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta}} = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
    return self.run_local()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1183, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1217, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1408, in _train_with_estimator_spec
    log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 504, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 921, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 643, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1107, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1112, in _create_session
    return self._sess_creator.create_session()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 800, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 566, in create_session
    init_fn=self._scaffold.init_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/session_manager.py", line 287, in prepare_session
    sess.run(init_op, feed_dict=init_feed_dict)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'NGraphVariable' OpKernel for GPU devices compatible with node node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py:277)  = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()
    .  Registered:  device='CPU'

     [[node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py:277)  = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta', defined at:
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
    return self.run_local()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 354, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1183, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1213, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1171, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/model_lib.py", line 284, in model_fn
    features[fields.InputDataFields.true_image_shape])
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 647, in predict
    image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 978, in _extract_rpn_feature_maps
    scope=self.first_stage_feature_extractor_scope))
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 163, in extract_proposal_features
    return self._extract_proposal_features(preprocessed_inputs, scope)
  File "/usr/local/lib/python3.6/dist-packages/object_detection-0.1-py3.6.egg/object_detection/models/faster_rcnn_inception_v2_feature_extractor.py", line 138, in _extract_proposal_features
    scope=scope)
  File "/usr/local/lib/python3.6/dist-packages/slim-0.1-py3.6.egg/nets/inception_v2.py", line 117, in inception_v2_base
    scope=end_point)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 2790, in separable_convolution2d
    outputs = normalizer_fn(outputs, **normalizer_params)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 650, in batch_norm
    outputs = layer.apply(inputs, training=is_training)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 828, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 364, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 759, in __call__
    self.build(input_shapes)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/normalization.py", line 297, in build
    trainable=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 278, in add_weight
    getter=vs.get_variable)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 586, in add_weight
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpointable/base.py", line 639, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 1484, in get_variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 1234, in get_variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 521, in get_variable
    return custom_getter(**custom_getter_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 1922, in wrapped_custom_getter
    *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1749, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1740, in _model_variable_getter
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 350, in model_variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 277, in variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1749, in layer_variable_getter
    return _model_variable_getter(getter, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1740, in _model_variable_getter
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 350, in model_variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 277, in variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 492, in _true_getter
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 920, in _get_single_variable
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 145, in __call__
    return cls._variable_call(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 141, in _variable_call
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 120, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 2441, in default_variable_creator
    expected_shape=expected_shape, import_scope=import_scope)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 147, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 1104, in __init__
    constraint=constraint)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 1218, in _init_from_args
    name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/state_ops.py", line 77, in variable_op_v2
    shared_name=shared_name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 1357, in variable_v2
    shared_name=shared_name, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): No registered 'NGraphVariable' OpKernel for GPU devices compatible with node node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py:277)  = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()
    .  Registered:  device='CPU'

     [[node FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta (defined at /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/framework/python/ops/variables.py:277)  = NGraphVariable[_class=["loc:@Assign"], container="", dtype=DT_FLOAT, just_looking=false, shape=[64], shared_name="", _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'report_uninitialized_variables_1/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
  File "model_main.py", line 109, in <module>
    tf.app.run()  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 610, in run
    return self.run_local()  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 711, in run_local
    saving_listeners=saving_listeners)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 356, in train
    return self  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1183, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1217, in _train_model_default
    saving_listeners)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1408, in _train_with_estimator_spec
    log_step_count_steps=self._config.log_step_count_steps) as mon_sess:  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 504, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 921, in __init__
    stop_grace_period_secs=stop_grace_period_secs)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 643, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1107, in __init__
    _WrappedSession.__init__(self, self._create_session())  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1121, in _create_session
    'the job. Error: %s', e)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 800, in create_session
    self.tf_sess = self._session_creator.create_session()  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 566, in create_session
    init_fn=self._scaffold.init_fn)  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 219, in finalize
    return self  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 268, in get_or_default
    return op  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 199, in default_ready_for_local_init_op
    variables.global_variables())  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))

What can be the solution to this?


Discussion / hints:
The code you are trying is taken from `tensorflow models_master/research/object_detection` repository.
This issue is more on suitable on TensorFlow/models repo. Since its not a bug or feature request on TF side,Please post on TF Models from [here](https://github.com/tensorflow/models/issues). If you think we have misinterpreted the bug please provide a minimal code snippet to reproduce the issue reported here. Thanks!
I have the same problem here, I've trained a model on my local PC and it worked just fine. However, when I use the same model on the cloud, this problem occurs. Why @ymodak thought this is a TF Model's problem, if you don't mind me asking. 
Has anyone solved this? Same issue here.

Fix patch:
diff --git a/third_party/xla/xla/runtime/execution_graph.cc b/third_party/xla/xla/runtime/execution_graph.cc
index 74eb39edbdec44..805ffe376fb188 100644
--- a/third_party/xla/xla/runtime/execution_graph.cc
+++ b/third_party/xla/xla/runtime/execution_graph.cc
@@ -15,6 +15,8 @@ limitations under the License.
 
 #include "xla/runtime/execution_graph.h"
 
+#include <sys/stat.h>
+
 #include <cstddef>
 #include <cstdint>
 #include <limits>
@@ -243,7 +245,9 @@ int64_t ExecutionGraph::EraseEdge(NodeDefBuilder& from, NodeDefBuilder& to,
       in_edges_it != to.in_edges.end() && in_edges_it->id == from.id;
 
   DCHECK(has_in_edge) << "In-edge must exist if out-edge exists";
-  DCHECK_EQ(in_edges_it->kind, out_edges_it->kind) << "Edges kind must match";
+  DCHECK_EQ(static_cast<int>(in_edges_it->kind),
+            static_cast<int>(out_edges_it->kind))
+      << "Edges kind must match";
 
   // At this point we must have exactly one edge between `from` and `to` nodes.
   DCHECK_EQ(absl::c_count_if(from.out_edges, EdgePredicate(to.id)), 1)


Test patch: