Issue 1:
I would like to know why at the model's methods which use a generator, has the argument `nb_worker deprecated.

There is any way to do it now correctly to have multiple threads fetching for your data?

Because in my case I work with a dataset made up by videos, it has an increment of speed fetching the data in multiple threads rather in only one.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 and (RHEL-like (custom distro ran by organization))

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): Tensorflow 2.5.0 on Windows 10 , Tensorflow 2.5.0 AND 2.7.0 on RHEL

Python version: Windows 10 -> python 3.7.6 : RHEL -> python 3.7.2+Tensorflow 2.5.0 AND python 3.8.2+Tensorflow 2.7.0

CUDA/cuDNN version: CUDA:11.2.0 cuDNN:8.1.1

GPU model and memory: Windows -> nVidia Quadro RTX 3000 : RHEL -> nVidia P100

Describe the current behavior

When utilizing a custom loss function that has been tested to work, it will break with certain batch sizes by claiming that a (None) datatype was received, or rather, the None could not be converted to tensor. This is despite the fact that a numpy array of dtype 'float32' was passed to it.

This was first noted in a script that trained two separate autoencoders on two separate datasets, 1- 12800x400 and 1-8000x400, with batch sizes of 128 and 80 respectively. The first would train, but the second would break despite the code for the second autoencoder being a duplicate of the first. The first one in that situation utilized the same custom loss function ExplainedVar(). This does not occur for the built-in loss functions. The batch_size in all cases was smaller than the training set.

It should be noted that when it fails, true and pred reach the loss function as Nones . This is not a result of the math inside the loss function.

Below are the two different errors produced by Tensorflow 2.5.0 and 2.7.0 respectively.

Tensorflow 2.5.0

Traceback (most recent call last):
File "divided_models.py", line 218, in <module>
a_sae.fit(rnd_dat,rnd_dat,epochs=10,batch_size=32,verbose=2)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/keras/engine/training.py", line 1184, in fit
tmp_logs = self.train_function(iterator)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
result = self._call(*args, **kwds)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 933, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 760, in _initialize
*args, **kwds))
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3066, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3463, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3308, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 1007, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 668, in wrapped_fn
out = weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 994, in wrapper
raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/keras/engine/training.py:853 train_function *
return step_function(self, iterator)
./custom_loss_funcs.py:117 call *
pred_mean = tf.reshape(tf.math.reduce_mean(pred,axis=1),(pred.shape[0],1))
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:206 wrapper **
return target(*args, **kwargs)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:196 reshape
result = gen_array_ops.reshape(tensor, shape, name)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py:8404 reshape
"Reshape", tensor=tensor, shape=shape, name=name)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:525 _apply_op_helper
raise err
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:515 _apply_op_helper
preferred_dtype=default_dtype)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/profiler/trace.py:163 wrapped
return func(*args, **kwargs)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1566 convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:346 _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:272 constant
allow_broadcast=True)
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:290 _constant_impl
allow_broadcast=allow_broadcast))
/usr/WS2/mvander/py3venv/py3venv/lib/python3.7/site-packages/tensorflow/python/framework/tensor_util.py:553 make_tensor_proto
"supported type." % (type(values), values))

TypeError: Failed to convert object of type <class 'tuple'> to Tensor. Contents: (None, 1). Consider casting elements to a supported type.

Tensorflow 2.7.0

Traceback (most recent call last):
File "divided_models3.py", line 111, in <module>
a_ae.fit(scaled_aData,scaled_aData,epochs=n_epochs,
File "/usr/WS2/mvander/py3_8/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/WS2/mvander/py3_8/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 1129, in autograph_handler
raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

File "/usr/WS2/mvander/py3_8/lib/python3.8/site-packages/keras/engine/training.py", line 878, in train_function *
return step_function(self, iterator)
File "divided_models3.py", line 23, in call *
pred_mean = tf.reshape(tf.math.reduce_mean(pred,axis=1),(pred.shape[0],1))

TypeError: Failed to convert elements of (None, 1) to Tensor. Consider casting elements to a supported type. See https://www.tensorflow.org/api_docs/python/tf/dtypes for supported TF dtypes.

Describe the expected behavior

Model.fit() should work for any batch_size > 0 and <= dataset size. Alternatively, it should work for any dataset size greater than zero where a batch_size larger than the dataset size would default to the size of the dataset. This should work regardless of whether it is an built-in loss function or a custom loss function.

Contributing

Do you want to contribute a PR? (yes/no): No

Standalone code to reproduce the issue

import os
import sys
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
import tensorflow.keras as krs

class ExplainedVar(krs.losses.Loss):
def __init__(self, reduction = krs.losses.Reduction.AUTO, **kwargs):
super( ExplainedVar, self).__init__(reduction = reduction, **kwargs)
# self.counter = 0

def call(self,true,pred):
#print(pred.shape)
#print(true.shape)
#if true.shape[0] is None:
# raise Exception()
# self.counter += 1
# print(self.counter)
pred = tf.convert_to_tensor(pred)
true = tf.cast(true, pred.dtype)

res = tf.math.reduce_sum( tf.math.square( pred - true ), axis=1 )
pred_mean = tf.reshape(tf.math.reduce_mean(pred,axis=1),(pred.shape[0],1))
tot = tf.math.reduce_sum( tf.math.square( pred - pred_mean ),axis=1)

return res/tot

def make_auto(in_dim,string):

enc_in = krs.Input((in_dim,))
x = enc_in

x0_0 = krs.layers.Dense(96,activation='elu',kernel_initializer='lecun_normal')(x)
x1_0 = krs.layers.Dense(48,activation='elu',kernel_initializer='lecun_normal')(x0_0)
x2_0 = krs.layers.Dense(25,activation='elu',kernel_initializer='lecun_normal')(x1_0)
x3_0 = krs.layers.Dense(16,activation='elu',kernel_initializer='lecun_normal')(x2_0)

x0_1 = krs.layers.Dense(12,activation='elu',kernel_initializer='lecun_normal')(x)
x1_1 = krs.layers.Dense(10,activation='elu',kernel_initializer='lecun_normal')(x0_0)
x2_1 = krs.layers.Dense(8,activation='elu',kernel_initializer='lecun_normal')(x1_0)
x3_1 = krs.layers.Dense(6,activation='elu',kernel_initializer='lecun_normal')(x2_0)
x4_1 = krs.layers.Dense(4,activation='elu',kernel_initializer='lecun_normal')(x3_0)

enc_out = krs.layers.Concatenate(axis=1)([x0_1,x1_1,x2_1,x3_1,x4_1])

a_enc = krs.Model( inputs = enc_in, outputs = enc_out, name = "%s_enc"%string)

dec_in = krs.Input((40,))
x = dec_in
x0 = x[:,:4]
x1 = x[:,4:10]
x2 = x[:,10:18]
x3 = x[:,18:28]
x4 = x[:,28:40]

x4 = krs.layers.Dense(16,activation='elu',kernel_initializer='lecun_normal')(x4)

x3 = krs.layers.Concatenate()([x3,x4])
x3 = krs.layers.Dense(25,activation='elu',kernel_initializer='lecun_normal')(x3)

x2 = krs.layers.Concatenate()([x2,x3])
x2 = krs.layers.Dense(48,activation='elu',kernel_initializer='lecun_normal')(x2)

x1 = krs.layers.Concatenate()([x1,x2])
x1 = krs.layers.Dense(96,activation='elu',kernel_initializer='lecun_normal')(x1)

x0 = krs.layers.Concatenate()([x0,x1])
dec_out = krs.layers.Dense(400,activation='elu',kernel_initializer='lecun_normal')(x0)

a_dec = krs.Model( inputs = dec_in, outputs = dec_out, name = "%s_dec"%string)

a_sae = krs.Model( inputs = enc_in, outputs = a_dec( a_enc(enc_in)), name = "%s_sae"%string)

return a_sae, a_enc, a_dec

def samp_minmax_transform(xmin,xmax,data):
data = ((data.T - xmin)/(xmax-xmin)).T
return data

def samp_minmax_inverse_transform(xmin,xmax,data):
data = (data.T*(xmax-xmin)+xmin).T
return data

def scale(x):
return np.power(x,1./30.)

def descale(x):
return np.power(x,30.)

aData = np.random.uniform(0,10.,(5000,400))
bData = np.random.uniform(0,10.,(5000,400))

scaled_aData = scale(aData)
scaled_bData = scale(bData)

print("aData DTYPE :",aData.dtype)
print("bData DTYPE :",bData.dtype)
###############################################

a_ae, a_enc, a_dec = make_auto(400,'a')

loss = ExplainedVar()
# loss = krs.losses.MSLE #### This works with all batch sizes
opt = krs.optimizers.Adam(.0001)
a_ae.compile(optimizer=opt,loss=loss)

n_epochs = 10

batch_size = int(scaled_aData.shape[0]*.8) ### This will break everything with custom loss function
# batch_size = int(scaled_aData.shape[0]*.05) #### This make it work with custom loss function

a_ae.fit(scaled_aData,scaled_aData,epochs=n_epochs,
batch_size=batch_size,verbose=2)

###############################################
## THIS SECTION IS TO VERIFY THAT IT ISN'T THE ARCHITECTURE ITSELF

# enc_in = krs.Input((400,))
# x = krs.layers.Dense(50,activation='elu')(enc_in)
# enc_out = krs.layers.Dense(10,activation='elu')(x)
# enc = krs.Model(inputs=enc_in,outputs=enc_out,name="enc")

# dec_in = krs.Input((10,))
# x = krs.layers.Dense(50,activation='elu')(dec_in)
# dec_out = krs.layers.Dense(400,activation='elu')(x)
# dec = krs.Model(inputs=dec_in,outputs=dec_out,name="dec")

# n_epochs = 10
# batch_size = int(scaled_aData.shape[0]*.8)
# ae = krs.Model(inputs=enc_in, outputs = dec( enc(enc_in)),name='ae')
# loss = ExplainedVar()
# opt = krs.optimizers.Adam(.0001)
# ae.compile(optimizer=opt,loss=loss)
# ae.fit(scaled_aData,scaled_aData,epochs=n_epochs,
# batch_size=batch_size,verbose=2)