Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no (https://www.tensorflow.org/alpha/tutorials/keras/feature_columns + model.save + load_model)

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 2.0.0-alpha0

Python version: 3.6.8

CUDA/cuDNN version: 10.0.130/7.3.1

GPU model and memory: GTX 1060 6GB

Describe the current behavior

loading a saved .h5 model which includes a DenseFeatures Layer fails:

ValueError: Unknown layer: DenseFeatures

Describe the expected behavior

model is loading

Code to reproduce the issue

import pandas as pd

# pip install -q tensorflow==2.0.0-alpha0
import tensorflow as tf
from tensorflow import feature_column
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split

URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
dataframe = pd.read_csv(URL)
train, test = train_test_split(dataframe, test_size=0.2)
train, val = train_test_split(train, test_size=0.2)

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
dataframe = dataframe.copy()
labels = dataframe.pop('target')
ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
if shuffle:
ds = ds.shuffle(buffer_size=len(dataframe))
ds = ds.batch(batch_size)
return ds

feature_columns = []

# numeric cols
for header in ['age', 'trestbps', 'chol', 'thalach', 'oldpeak', 'slope', 'ca']:
feature_columns.append(feature_column.numeric_column(header))

feature_layer = tf.keras.layers.DenseFeatures(feature_columns)

batch_size = 32
train_ds = df_to_dataset(train, batch_size=batch_size)
val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

model = tf.keras.Sequential([
feature_layer,
layers.Dense(128, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
loss='binary_crossentropy')

model.fit(train_ds,
validation_data=val_ds,
epochs=5)

model.save('my_model.h5')
from tensorflow import keras
new_model = keras.models.load_model('my_model.h5')

Other info / logs

Traceback (most recent call last):
File "error.py", line 67, in <module>
new_model = keras.models.load_model('my_model.h5')
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 215, in load_model
custom_objects=custom_objects)
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 95, in deserialize
printable_module_name='layer')
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 192, in deserialize_keras_object
list(custom_objects.items())))
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\engine\sequential.py", line 351, in from_config
custom_objects=custom_objects)
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 95, in deserialize
printable_module_name='layer')
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 181, in deserialize_keras_object
config, module_objects, custom_objects, printable_module_name)
File "C:\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 166, in class_and_config_for_serialized_keras_object
raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: DenseFeatures

Issue 2:
Do we have any options to control the number of threads in TF-Slim both in training and evaluation processes?

Specifically, I use this network for my classification problem. I changed the evaluation part in a way that runs train and evaluation in parallel like this code. I can run it on my own CPU without any problem. But I can't execute them on a supercomputer. It seems that it is related to the very large number of threads which are being created by Tensorflow. If the number of threads exceeds the maximum number of threads pre-set in SLURM (= 28) then the job will fail. Since it's unable to create new threads it will end up with error "resource temporarily unavailable".

This error provided when the code tries to restore parameters from checkpoints. If there is no limitation on the number of threads (like on my pc) it works fine:

INFO:tensorflow:Restoring parameters from ./model.ckpt-0
INFO:tensorflow:Starting evaluation at
I tensorflow/core/kernels/logging_ops.cc:79] eval/Accuracy[0]
I tensorflow/core/kernels/logging_ops.cc:79] eval/Recall_5[0]
INFO:tensorflow:Evaluation [1/60]

However, when there is a limitation on the number of threads (like SLURM job submission on supercomputers) we get:

INFO:tensorflow:Restoring parameters from ./model.ckpt-0
terminate called after throwing an instance of 'std::system_error'
what(): Resource temporarily unavailable

I tried to limit the number of CPU threads used by Tensorflow to 1 by creating config like:

FLAGS.num_preprocessing_threads=1

config = tf.ConfigProto()
config.intra_op_parallelism_threads = FLAGS.num_preprocessing_threads
config.inter_op_parallelism_threads = FLAGS.num_preprocessing_threads

slim.evaluation.evaluation_loop(
master=FLAGS.master,
checkpoint_path=each_ckpt,
logdir=FLAGS.eval_dir,
num_evals=num_batches,
eval_op=list(names_to_updates.values()) + print_ops,
variables_to_restore=variables_to_restore,
session_config=config)

But unfortunately, that didn't help. In my opinion, the main problem we are having here is the fact that we are not able to control the number of threads here. Although we set it to 1 with various TF options you can actually see that this job is creating many more threads on the node:

slurm_script─┬─python───128*[{python}]
└─python───8*[{python}]

Training script is creating 128 threads and evaluation script is creating 8 (both numbers vary over time).

P.S. I'm using Python 2.7.13 and Tensorflow 1.3.0.