Issue 1:
Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

Check that you are up-to-date with the master branch of Keras. You can update with:

pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

I'm training a model on p2.8xlarge, my base model looks like this:

And parallel_model = multi_gpu_model(model, gpus=TOTAL_GPU) looks like this,

Warnings when initiating base model on CPU:

WARNING:tensorflow:From /root/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
INFO:root:Model Instantiate on CPU

Warnings when initiating parallel_model on GPU:

/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))

Model training:

TOTAL_GPU = 8
BATCH_SIZE = 512 * TOTAL_GPU
train_gen = generate_arrays_from_file(TRAIN_FEATURES, TRAIN_LABELS, BATCH_SIZE)
valid_gen = generate_arrays_from_file(VALID_FEATURES, VALID_LABELS, BATCH_SIZE)

train_data_count = 268200000
valid_data_count = 29800000

ST_PR_EP_TR = int(train_data_count // BATCH_SIZE)
ST_PR_EP_VA = int(valid_data_count // BATCH_SIZE)

history = parallel_model.fit_generator(
train_gen,
steps_per_epoch=ST_PR_EP_TR,
epochs=100,
verbose=0,
validation_data=valid_gen,
validation_steps=ST_PR_EP_VA,
callbacks=callback_list,
max_queue_size=80,
use_multiprocessing=True,
workers=8,
shuffle=True,
initial_epoch=0)

Warnings when training model with fit_generator with batches:

WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py:2088: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
UserWarning('Using a generator with `use_multiprocessing=True`'

Does Postponing error-checking until all devices are assigned. is somehow affecting model training ? or I can ignore this? I'm running Keras 2.1.4 and TensorFlow 1.6.0rc1 with Cuda compilation tools, release 9.0, V9.0.176

Thanks!

Issue 2:
We should consider moving the modules preprocessing and applications to separate repositories under keras-team: keras-team/preprocessing and keras-team/applications.

They would be listed as a dependency of keras, and would still be importable from e.g. keras.preprocessing.

Why?

They're not part of "the core Keras API" (which is about model building and training)

They're not really coupled to the rest of the codebase, so unbundling is safe

Faster CI runs, better test coverage

Unbundling makes it easier to develop in an open-source setting

Easier to sync across keras-team/keras and tf.keras (shared dependencies instead of code duplication)

Other projects might have need of Numpy preprocessing utilities

Any comments or concerns? Who would be interested in taking greater ownership of these repos, should they materialize?