Issue 1:
I attempt to implement naive RBM/DBN through tensorflow's python API.

The functionality is easy to implement if you don't care about computational performance.

For now, I've tried many alternatives solutions and find it difficult to implement with high performance since there's no way to cache the intermediate tensor result.

While lacking of the ability to cache the intermediate result, there will be too much memory migrate from GPU to RAM or vice versa (say I'm using GPU) if I want to do something through intermediate result.

Here's part of my code to illustrate my meaning.

import tensorflow as tf
alpha = .01
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
trX, trY, teX, teY = mnist.train.images, mnist.train.labels,\
mnist.test.images, mnist.test.labels
W = tf.Variable(tf.random_normal([784, 100]), name="weights")
hb = tf.Variable(tf.zeros([100]), name="hbias")
vb = tf.Variable(tf.zeros([784]), name="vbias")
v0 = tf.placeholder("float", [None, 784])

h0 = tf.nn.sigmoid(tf.matmul(v0, W) + hb)
h0 = sample_prob(h0)
v1 = tf.nn.sigmoid(tf.matmul(h0, tf.transpose(W)) + vb)
v1 = sample_prob(v1)
h1 = tf.nn.sigmoid(tf.matmul(v1, W) + hb)

pg = tf.matmul(tf.transpose(v0), h0)
ng = tf.matmul(tf.transpose(v1), h1)

dW = (pg - ng) / tf.to_float(tf.shape(v0)[0])
dhb = tf.reduce_mean(h0 - h1, 0)
dvb = tf.reduce_mean(v0 - v1, 0)

update_W_op = W.assign_add(alpha * dW)
update_hb_op = hb.assign_add(alpha * dhb)
update_vb_op = vb.assign_add(alpha * dvb)

with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
for _ in range(10):
for start, end in zip(
range(0, 50000, 100), range(100, 50000, 100)):
X = trX[start:end]
# sess.run([dW, dhb, dvb, h1], feed_dict={v0: X})
# sess.run(h1, feed_dict={v0: X})
# sess.run(dW)
# sess.run(dhb)
# sess.run(dvb)
# _W = sess.run(update_W_op)
# _hb = sess.run(update_hb_op)
# _vb = sess.run(update_vb_op)
[_W, _hb, _vb] = sess.run([update_W_op, update_hb_op, update_vb_op], feed_dict={v0: X})

This is the most efficient way I could think out. But, there's no way to optimize the progress by calculating [_W, _hb, _vb] with the aid of intermediate result pg, ng, h0, h1, v0, v1. It's obvious that here comes so much duplicated but useless extra computation.

If feed v0 with X once, the follower Tensor need the former Tensor.

h0 needs v0, so needs X

v1 needs h0, so needs X

h1 needs v1, so needs X

......

update_W_op needs dW, so needs X

update_hb_op needs dhb, so needs X

update_vb_op needs dvb, so needs X

Since there's no way to cache the intermediate result, you can only get one Tensor result for one computation. There's no way to avoid the duplication computation and X memory migration cost.

I was wondering if there's a property of Tensor say cached = False, we could set it to True manually. And when intermediate cached result is enough to go on, we need not to feed data in, and just using intermediate cache to compute. If you want to flush the cache, just feed in the new data into the placeholder. But for now, if the graph contains placeholder in the path, the data must be feed in specifically.

For example, if the cache is enabled.

h0.cached = True
h1.cached = True
v0.cached = True
v1.cached = True
sess.run(h1, feed_dict={v0: X})
sess.run([update_W_op, update_hb_op, update_vb_op]) # using cache h0,h1,v0,v1 is enough to compute the three update without X fed in.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Fedora 27

TensorFlow installed from (source or binary): Source

TensorFlow version (use command below): v1.8.0-rc1-909-g968addadfd 1.8.0-rc1

Python version: 3.6m

Bazel version (if compiling from source): 0.12.0

GCC/Compiler version (if compiling from source): 6.4

CUDA/cuDNN version: 9.1/7.1

GPU model and memory: GTX 1080 8GB

Exact command to reproduce:

include Makefile.config
TF_INC = `$(PYTHON) -W ignore -c 'import tensorflow as tf; print(tf.sysconfig.get_include())'`
TF_CFLAGS = `$(PYTHON) -W ignore -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'`
TF_LFLAGS = `$(PYTHON) -W ignore -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'`
CUDA_INC = $(CUDA_HOME)/../
GPUCC = nvcc -ccbin=$(CXX)
CFLAGS = -std=c++11 $(TF_CFLAGS) -I. -I$(CUDA_INC) -I$(TF_INC)
GPUCFLAGS = -c
LFLAGS = $(ODFLAGS) -pthread -fopenmp -shared -fPIC $(TF_LFLAGS)
GPULFLAGS = -x cu -Xcompiler "$(CFLAGS) $(LFLAGS)" --expt-relaxed-constexpr
GPUDEF = -DGOOGLE_CUDA=1
CGPUFLAGS = -lcuda
SRC = cart_hex_interpol.cc
GPUSRC = cart_hex_interpol.cu.cc
SRC_O	= cart_hex_interpol.o
GPUSRC_O = cart_hex_interpol.cu.o
LIB = cart_hex_interpol.so
all: gpu
default: gpu
cpu:
$(CXX) $(CFLAGS) $(SRC) $(LFLAGS) -o $(LIB)
gpu:
$(GPUCC) $(GPUDEF) $(CFLAGS) $(GPUCFLAGS) $(GPUSRC) $(GPULFLAGS) -o $(GPUSRC_O)
$(CXX) $(GPUDEF) $(CFLAGS) $(SRC) $(GPUSRC_O) $(LFLAGS) $(CGPUFLAGS) -o $(LIB)
clean:
rm -f $(SRC_O) $(GPUSRC_O) $(LIB)

Describe the problem

When compiling a custom TensorFlow operator using a Makefile and including CUDA/GPU code, the compilation fails due to missing fatal error: tensorflow/compiler/xla/statusor.h: No such file or directory. The files are missing in /usr/local/lib/python3.6/site-packages/tensorflow/ because the headers do not get installed during installation of the wheel package for TensorFlow.

A temporary fix is to copy the headers from source:

sudo mkdir /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/compiler/
sudo mkdir /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/compiler/xla
sudo cp tensorflow/compiler/xla/*.h /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/compiler/xla/

This issue was not present in TensorFlow 1.5, but must have been introduced since then. Compiling with or without XLA makes no difference.

Source code / logs

make gpu -j2
nvcc -ccbin=/usr/local/gcc-6.4/bin/g++-6.4 -DGOOGLE_CUDA=1 -std=c++11 `python3 -W ignore -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'` -I. -I/usr/local/cuda/../ -I`python3 -W ignore -c 'import tensorflow as tf; print(tf.sysconfig.get_include())'` -c cart_hex_interpol.cu.cc -x cu -Xcompiler "-std=c++11 `python3 -W ignore -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'` -I. -I/usr/local/cuda/../ -I`python3 -W ignore -c 'import tensorflow as tf; print(tf.sysconfig.get_include())'` -O3 -pthread -fopenmp -shared -fPIC `python3 -W ignore -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'`" --expt-relaxed-constexpr -o cart_hex_interpol.cu.o
In file included from /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:26:0,
from /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/platform/stream_executor.h:26,
from /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/util/cuda_launch_config.h:27,
from /usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:22,
from cart_hex_interpol.cu.cc:5:
/usr/local/lib/python3.6/site-packages/tensorflow/include/tensorflow/stream_executor/lib/statusor.h:21:46: fatal error: tensorflow/compiler/xla/statusor.h: No such file or directory
#include "tensorflow/compiler/xla/statusor.h"