opened
nttgithubntt
username_0
Hj, Thanks for creating tfrs. I examinate a recommendation model based on tfrs, after that, I fit, save model and load model again. the summary after loading model:
Model: "ranking_model"

Layer (type)                 Output Shape              Param #   
=================================================================
user_model (UserModel)       multiple                  5248      

card_model (CardModel)       multiple                  957640    

sequential_5 (Sequential)    (None, 4)                 4644      

ranking (Ranking)            multiple                  2         
=================================================================
Total params: 967,534
Trainable params: 967,532
Non-trainable params: 2

None

But when predicting: the loaded model show error:

ValueError: Could not find matching function to call loaded from the SavedModel. Got:
      Positional arguments (3 total):
        * {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float32>}
        * False
        * None
      Keyword arguments: {}
    
    Expected these arguments to match one of the following 4 option(s).....


How to solve this issues? Tks again.
can't predict after saving then loading model
issue
created
maciejkula
username_1
This most commonly happens where the features you pass to `call` in training are different than the features you pass to `predict`. For example, you might accidentally pass labels or example weights to `call` during training, but these are not available in prediction.

To avoid this, make sure you only pass the features you have in prediction into `call` when defining your `compute_loss` method.

Can you provide all the options that the saved model error gives you?
comment
created
nttgithubntt
username_0
Hj @username_1 , thanks you for responding. Here are the ranking model:

class RankingModel(tfrs.models.Model):

    def __init__(self, user_ids, topic_ids, card_ids, card_titles):
        # We take the loss weights in the constructor: this allows us to instantiate
        # several model objects with different loss weights.
        super().__init__()

        self.query_model = UserModel(user_ids, topic_ids)
        self.candidate_model = CardModel(card_ids, card_titles, topic_ids)

        # A small model to take in user and card embedding and predict interactions.
        # We can make this as complicated as we want as long as we output a vector
        # as our prediction. A vector prediction is [Favorites, Shares, Discussions_Started, Press]
        self.interacting_model = tf.keras.Sequential([
            tf.keras.layers.Dense(64, activation="relu"),
            tf.keras.layers.Dense(32, activation="relu"),
            tf.keras.layers.Dense(4),
        ])

        # The ranking tasks.
        self.ranking_task: tf.keras.layers.Layer = tfrs.tasks.Ranking(
            loss=tf.keras.losses.MeanSquaredError(),
            metrics=[tf.keras.metrics.RootMeanSquaredError()]
        )

    def call(self, inputs):
        print("call-ranking model inputs info:")
        print(inputs)
        user_embeddings = self.query_model(inputs)
        card_embeddings = self.candidate_model(inputs)
        return self.interacting_model(tf.concat([user_embeddings, card_embeddings], axis=1))

    def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:
        inputs = features.copy()
        del inputs["favorites"], inputs["share"], inputs["discussions_started"], inputs["press"]
        interacting_predictions = self(inputs)
        interacting = tf.concat(
            [tf.expand_dims(features["favorites"], -1),
             tf.expand_dims(features["share"], -1),
             tf.expand_dims(features["discussions_started"], -1),
             tf.expand_dims(features["press"], -1)
             ], axis=1)

        return self.ranking_task(
            labels=interacting,
            predictions=interacting_predictions
        )

model = RankingModel(user_ids=vocabularies[0], topic_ids=vocabularies[1], card_ids=vocabularies[2],
                                       card_titles=vocabularies[3])
model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))

train, test = self.data_manager.get_train_test_batch_set()
model.fit(train.batch(8192), epochs=3)
print("ranking model summary:")
print(self._model.summary())

print("save ranking model:")
model.save(DEEP_MODEL_PATH)


model = tf.keras.models.load_model(DEEP_MODEL_PATH)
# Check its architecture
print("loaded ranking model ok")
print("loaded ranking model summary:")
print(model.summary())

predict_dataset = DeepDataManager.get_df_score(user_info, df_card).batch(5)
results = model.predict(predict_dataset)


Here are the logs:

/home/ai/Documents/AI/gh-ai/eurekai/bin/python /home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py
2020-10-06 11:58:08.502124: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py:23: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False.
  self.data_manager = DeepDataManager(data_path=data_path)
2020-10-06 11:58:10.596744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-06 11:58:10.662514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-06 11:58:10.663019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-10-06 11:58:10.663046: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-06 11:58:10.663244: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2020-10-06 11:58:10.663287: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-10-06 11:58:10.663326: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-10-06 11:58:10.663365: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-10-06 11:58:10.663403: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-10-06 11:58:10.666135: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-06 11:58:10.666155: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-10-06 11:58:10.666427: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-06 11:58:10.678109: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3299990000 Hz
2020-10-06 11:58:10.679010: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5cb4090 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-06 11:58:10.679039: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-06 11:58:10.680719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-06 11:58:10.680732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      
fitting ranking model:
Epoch 1/3
WARNING:tensorflow:Layer ranking_model is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2.  The layer has dtype float64 because its dtype defaults to floatx.

If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float32 by default, call `tf.keras.backend.set_floatx('float32')`. To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

call-ranking model inputs info:
{'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>}
WARNING:tensorflow:Layer ranking is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2.  The layer has dtype float64 because its dtype defaults to floatx.

If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float32 by default, call `tf.keras.backend.set_floatx('float32')`. To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

call-ranking model inputs info:
{'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>}
1/1 [==============================] - 0s 389us/step - root_mean_squared_error: 12.4154 - loss: 154.1427 - regularization_loss: 0.0000e+00 - total_loss: 154.1427
Epoch 2/3
1/1 [==============================] - 0s 435us/step - root_mean_squared_error: 12.1651 - loss: 147.9898 - regularization_loss: 0.0000e+00 - total_loss: 147.9898
Epoch 3/3
1/1 [==============================] - 0s 442us/step - root_mean_squared_error: 11.2242 - loss: 125.9836 - regularization_loss: 0.0000e+00 - total_loss: 125.9836
ranking model summary:
Model: "ranking_model"

Layer (type)                 Output Shape              Param #   
=================================================================
user_model (UserModel)       multiple                  5248      

card_model (CardModel)       multiple                  957640    

sequential_5 (Sequential)    (None, 4)                 4644      

ranking (Ranking)            multiple                  2         
=================================================================
Total params: 967,534
Trainable params: 967,532
Non-trainable params: 2

None
save ranking model:
call-ranking model inputs info:
{'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
call-ranking model inputs info:
{'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
call-ranking model inputs info:
{'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>}
call-ranking model inputs info:
{'user_id': <tf.Tensor 'inputs_5:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None,) dtype=float64>}
WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.
WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:2309: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.
call-ranking model inputs info:
{'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>}
call-ranking model inputs info:
{'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>}
2020-10-06 11:58:13.487382: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7f4ab8ff99d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
loaded ranking model ok
loaded ranking model summary:
Model: "ranking_model"

Layer (type)                 Output Shape              Param #   
=================================================================
user_model (UserModel)       multiple                  5248      

card_model (CardModel)       multiple                  957640    

sequential_5 (Sequential)    (None, 4)                 4644      

ranking (Ranking)            multiple                  2         
=================================================================
Total params: 967,534
Trainable params: 967,532
Non-trainable params: 2

None
sys:1: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False.
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:69: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_interactions["user_id"] = user_info["user_id"]
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:70: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_interactions["interested_topic_id"] = user_info["interested_topic_id"]
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:71: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_interactions['view_time'] = 0
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:18: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_interactions['updated_at'] = pd.to_datetime(df_interactions['updated_at'])
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:21: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  lambda x: (pd.Timestamp.now() - x['updated_at']).days + 1, axis=1)
/home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:23: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_interactions['card_recency'] = 1 / df_interactions['card_recency']
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:4167: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:3065: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]
Traceback (most recent call last):
  File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 82, in <module>
    score_phase()
  File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 76, in score_phase
    results = recommender_.predict(predict_dataset)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1599, in predict
    tmp_batch_outputs = predict_function(iterator)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1462 predict_function  *
        return step_function(self, iterator)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1452 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1445 run_step  **
        outputs = model.predict_step(data)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1418 predict_step
        return self(x, training=False)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__
        outputs = call_fn(inputs, *args, **kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:71 return_outputs_and_add_losses
        outputs, losses = fn(inputs, *args, **kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 wrap_with_training_arg
        lambda: replace_training_and_call(False))
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py:65 smart_cond
        pred, true_fn=true_fn, false_fn=false_fn, name=name)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py:56 smart_cond
        return false_fn()
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 <lambda>
        lambda: replace_training_and_call(False))
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:165 replace_training_and_call
        return wrapped_call(*args, **kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:780 __call__
        result = self._call(*args, **kwds)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:823 _call
        self._initialize(args, kwds, add_initializers_to=initializers)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:697 _initialize
        *args, **kwds))
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:2855 _get_concrete_function_internal_garbage_collected
        graph_function, _, _ = self._maybe_define_function(args, kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3213 _maybe_define_function
        graph_function = self._create_graph_function(args, kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3075 _create_graph_function
        capture_by_value=self._capture_by_value),
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:600 wrapped_fn
        return weak_wrapped_fn().__wrapped__(*args, **kwds)
    /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/saved_model/function_deserialization.py:257 restored_function_body
        "\n\n".join(signature_descriptions)))

    ValueError: Could not find matching function to call loaded from the SavedModel. Got:
      Positional arguments (2 total):
        * {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None, 1) dtype=int32>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float64>}
        * False
      Keyword arguments: {}
    
    Expected these arguments to match one of the following 4 option(s):
    
    Option 1:
      Positional arguments (2 total):
        * {'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency')}
        * True
      Keyword arguments: {}
    
    Option 2:
      Positional arguments (2 total):
        * {'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id')}
        * False
      Keyword arguments: {}
    
    Option 3:
      Positional arguments (2 total):
        * {'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id')}
        * False
      Keyword arguments: {}
    
    Option 4:
      Positional arguments (2 total):
        * {'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id')}
        * True
      Keyword arguments: {}


Process finished with exit code 1
comment
created
nttgithubntt
username_0
Hj @username_1 , I had find the cause of the problem, due to the mismatch shape between 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, and requirement input 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'). I have fixed it, tks you again :)
comment
closed
nttgithubntt
username_0
issue
created
yoke2c
username_2
Hi @username_0 I encountered the same error as you while trying to call `model.predict` or `model.evaluate` after saving the model using `model.save` and reloading the model again using `tf.keras.models.load_model` and understand that you have managed to fix it. Perhaps could you share or advise where the shape mismatch can be fixed? Thank you.
comment
created
nttgithubntt
username_0
Hj @username_2 , you push the log to check more detail. the common cause of this error is the tensor variable shape in training phase does not match the evaluating or predicting phase.
I my case error:
In training phase, the tensor variable 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 
has shape = shape=(None, 1)
But in predicting phase, we input it with  'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id')
has shape =(None,)
Clearly, shape=(None, 1) not match shape =(None,).
comment
created
yoke2c
username_2
Hi @username_0 thank you for sharing! While I'm still trying to troubleshoot the shape issue, a work-around now is using `model.save_weights()` and then using `model.load_weights()` after re-creating the model.
comment
created
maciejkula
username_1
I suspect it might be related to different batching that you're doing between training/evaluation and prediction.
comment
created
yoke2c
username_2
Thanks for the advice @username_1, shall explore this as well.
comment
created
cory1219
username_3
Hello @username_0, can you explain how you changed tensor variable shape from (None, 1) to (None,) thank!
comment