Issue 1:
MaxPool gradient of a gradient is not supported as determined in keras-team/keras#4694, which means that examples work in keras with the theano backend but not with the TensorFlow backend. I believe the relevant place where the op is registered is at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/nn_grad.cc#L153.

From the keras issue:

I successfully ran python mnist_cnn.py from keras-team/keras@7e2e7a5 on my GTX1080 with a TensorFlow R0.12rc0 backend. I then ran python mnist_swwae.py and got the following failure:

~/src/keras/examples on master
± python mnist_swwae.py
Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so.8.0 locally
X_train shape: (60000, 1, 28, 28)
60000 train samples
10000 test samples
Traceback (most recent call last):
File "mnist_swwae.py", line 136, in <module>
y = MaxPooling2D(pool_size=(pool_sizes[i], pool_sizes[i]))(y_prepool)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 517, in __call__
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 571, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 155, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/layers/pooling.py", line 158, in call
dim_ordering=self.dim_ordering)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/layers/pooling.py", line 207, in _pooling_function
border_mode, dim_ordering, pool_mode='max')
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/backend/tensorflow_backend.py", line 1853, in pool2d
x = tf.nn.max_pool(x, pool_size, strides, padding=padding)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1617, in max_pool
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1598, in _max_pool
data_format=data_format, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2242, in create_op
set_shapes_for_outputs(ret)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1617, in set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1568, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
debug_python_shape_fn, require_shape_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 675, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Negative dimension size caused by subtracting 2 from 1 for 'MaxPool' (op: 'MaxPool') with input shapes: [?,1,32,8].
-> [1]

Issue 2:
Currently, neither tf.nn.avg_pool or tf.nn.max_pool support automatically taking second derivatives, making it difficult to experiment with second-order methods on neural networks that use any sort of pooling (which unfortunately includes most modern networks). Trying to take a second derivative gives a LookupError, e.g., LookupError: No gradient defined for operation 'gradients_4...AvgPoolGrad' (op type: AvgPoolGrad).

Do you have any plans for implementing second derivatives for pooling operations, or any suggested workarounds like rewriting the pooling operations in terms of elementary functions that have second derivatives defined? Thank you!