Issue 1:
System information

TensorFlow version (you are using): 1.12

Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

Gradient Accumulation(GA) is a workaround to enable big batches on limited memory GPUs which has been supported in Caffe and PyTorch. Instead of back-propagating for every batch feed-forward; gradients across multiple batches are accumulated. After multiple feed forwards, the accumulated gradient is back-propagated through the network layer. It boosts performance with a couple of percentages on our several workload (e.g., XLNet, Transformer) with Distribution Strategy. Unfortunately, I am not aware of any official documentation to use such feature in Tensorflow. tf.contrib.opt.AGNOptimizer has a similar GA implementation but is not common for general distribution job on DistributionStrategy. Besides it will introduces OOM for large embeddings(#31637).

Will this change the current api? How?

Yes. Just need to add one new parameter iter_size to current strategy API, which for example:

# For MirroredStrategy, gradient accumulation is supported using iter_size parameter.
distribution = tf.distribute.MirroredStrategy(num_gpus=2, iter_size=4)

# For MultiWorkerMirroredStrategy:
tf.distribute.experimental.MultiWorkerMirroredStrategy(..., iter_size=4)

Who will benefit with this feature?

Anyone who use distribution strategy and want to enable big batches on limited memory GPUs

Any Other info.

None.

We already have a concise implementation of GA on DistributionStrategy and are willing to contribute. Thanks in advance for any feedback.

Issue 2:
System information

CentOS Linux release 7.4.1708

Source, hash: bf4767c

TensorFlow version: 1.12

Python version: Python 3.4.9

Installed using virtualenv? pip? conda?:

Bazel version (if compiling from source): 0.19.2 (bazel from centos repos)

GCC/Compiler version (if compiling from source): 4.8.5 (gcc from centos repos)

CUDA/cuDNN version: -

GPU model and memory: -

Describe the problem

Cannot import tensorflow due to import 'mock' library from tensorflow.python.platform.googletest failling.

Provide the exact sequence of commands / steps that you executed before running into the problem

Successfully built and installed TensorFlow with the following commands:

bazel build --incompatible_remove_native_http_archive=false --incompatible_package_name_is_a_function=false --config=opt //tensorflow/tools/pip_package:build_pip_package

./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

pip3.4 install --user /tmp/tensorflow_pkg/tensorflow-1.12.0-cp34-cp34m-linux_x86_64.whl

See pip log below for found dependencies

Attempting to import tensorflow i get this:

Python 3.4.9 (default, Aug 14 2018, 21:28:57)

[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import tensorflow

/usr/lib64/python3.4/importlib/_bootstrap.py:321: FutureWarning: Conversion of the second argument of issubdtype from floattonp.floating is deprecated. In future, it will be treated as 'np.float64 == np.dtype(float).type'.

return f(*args, **kwds)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/__init__.py", line 34, in <module>

from tensorflow._api.v1 import compat

File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/__init__.py", line 21, in <module>

from tensorflow._api.v1.compat import v1

File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/v1/__init__.py", line 69, in <module>

from tensorflow._api.v1.compat.v1 import test

File "/home1/private/mavridis/.local/lib/python3.4/site-packages/tensorflow/_api/v1/compat/v1/test/__init__.py", line 24, in <module>

from tensorflow.python.platform.googletest import mock

ImportError: cannot import name 'mock'

Any other info / logs

pip install output:

Processing /tmp/tensorflow_pkg/tensorflow-1.12.0-cp34-cp34m-linux_x86_64.whl

Requirement already satisfied: grpcio>=1.8.6 in /usr/lib64/python3.4/site-packages (from tensorflow==1.12.0) (1.10.0)

Requirement already satisfied: keras-preprocessing>=1.0.5 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.0.5)

Requirement already satisfied: six>=1.10.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.12.0)

Requirement already satisfied: absl-py>=0.1.6 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.1.11)

Requirement already satisfied: google-pasta>=0.1.1 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.1.1)

Requirement already satisfied: numpy>=1.13.3 in /usr/lib64/python3.4/site-packages (from tensorflow==1.12.0) (1.14.1)

Requirement already satisfied: protobuf>=3.6.1 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (3.6.1)

Requirement already satisfied: wheel>=0.26 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.32.3)

Requirement already satisfied: tensorboard<1.13.0,>=1.12.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.12.2)

Requirement already satisfied: astor>=0.6.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.6.2)

Requirement already satisfied: keras-applications>=1.0.6 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.0.6)

Requirement already satisfied: tensorflow-estimator<1.14.0rc0,>=1.13.0rc0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.13.0rc0)

Requirement already satisfied: gast>=0.2.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (0.2.0)

Requirement already satisfied: termcolor>=1.1.0 in /usr/lib/python3.4/site-packages (from tensorflow==1.12.0) (1.1.0)

Requirement already satisfied: setuptools in /home1/private/mavridis/.local/lib/python3.4/site-packages (from protobuf>=3.6.1->tensorflow==1.12.0) (40.6.3)

Requirement already satisfied: werkzeug>=0.11.10 in /usr/lib64/python3.4/site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0) (0.14.1)

Requirement already satisfied: markdown>=2.6.8 in /usr/lib64/python3.4/site-packages (from tensorboard<1.13.0,>=1.12.0->tensorflow==1.12.0) (2.6.11)

Requirement already satisfied: h5py in /usr/lib64/python3.4/site-packages (from keras-applications>=1.0.6->tensorflow==1.12.0) (2.7.1)

Requirement already satisfied: mock>=2.0.0 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0rc0->tensorflow==1.12.0) (2.0.0)

Requirement already satisfied: pbr>=0.11 in /home1/private/mavridis/.local/lib/python3.4/site-packages (from mock>=2.0.0->tensorflow-estimator<1.14.0rc0,>=1.13.0rc0->tensorflow==1.12.0) (5.1.1)

Installing collected packages: tensorflow

Successfully installed tensorflow-1.12.0