Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No. Built for iOS using provided scripts. Defined ops to be included in build as documented (using ops_to_register.h).

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Built on macOS 10.12.6 for distribution on iOS

TensorFlow installed from (source or binary): source

TensorFlow version (use command below): Unsure, sorry, my python environment is all messed up right now. My HEAD is at 0d2f691

Python version: 2.7.12, but I believe that's irrelevant for this issue

Bazel version (if compiling from source): 0.5.1-homebrew, but I'm using build_all_ios.sh instead of Bazel to build

CUDA/cuDNN version: Unknown

GPU model and memory: iPhone 7 Plus, but AFAIK GPU is unavailable on iOS

Exact command to reproduce: No particular command. Please see description of issue

Also asked about this on Stack Overflow, where it was suggested I file an issue

Describe the problem

I believe this is a bug in Tensorflow. I am running Tensorflow on iOS, using the C++ API. I'm doing some image classification. I have a long-lived session, and I call Run many times on it, to evaluate different images from a backlog. Once I'm done, the RunQueues (via NonBlockingThreadPools) continue to pin the CPU at near max usage. They appear to be stuck in the Steal loop, presumably with no work to do.

Source code / logs

I tried Close()ing and then deleteing the session, and having read some of the C++ source, this should have shut down the thread pools that belong to the session, but this didn't change the situation:

auto status = session->Close();
delete session;
session = nil;

I tried setting some specific configuration options so I could be sure that the session did in fact own its thread pools instead of using a global thread pool, but this didn't help either:

tensorflow::SessionOptions options;
options.config.clear_session_inter_op_thread_pool();
options.config.set_use_per_session_threads(true);
auto status = tensorflow::NewSession(options, &session);

One thing to note: while my understanding is that this isn't necessary, I did also try using a mutex to ensure that Close and delete would not be called concurrently with any call to Run, but again, no luck.

The only thing that has reduced CPU load to a reasonable level is to set inter_op_parallelism_threads to 1. This doesn't resolve the underlying problem (that the threads are never cleaned up), but it does mean that the Steal loop is avoided, so the thread just blocks forever.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): N/A

TensorFlow version (use command below): 1.10.0

Python version: 3.6.3

Bazel version (if compiling from source): N/A

GCC/Compiler version (if compiling from source): N/A

CUDA/cuDNN version: N/A

GPU model and memory: N/A

Exact command to reproduce: N/A

Describe the problem

When train.batch_sequences_with_states extracts batches from a sequence of samples, what it actually does is duplicating the same segment for batch_size times.

Source code / logs

The code is revised based on batch_sequences_with_states_test.py

from pprint import pprint

import numpy as np
import tensorflow as tf
from tensorflow.contrib.training.python.training import sequence_queueing_state_saver as sqss

batch_size = 3
num_unroll = 2
lstm_size = 4
value_length = 8
input_key = tf.as_string(tf.cast(10000 * tf.random_uniform(()), tf.int32))
input_sequences = {'input': np.random.rand(value_length, 3)}
input_context = {'context_key': [1]}
initial_states = {"lstm_state": np.random.rand(10, lstm_size)}

with tf.Session() as sess:
batch = sqss.batch_sequences_with_states(
input_key=input_key,
input_sequences=input_sequences,
input_context=input_context,
input_length=value_length,
initial_states=initial_states,
num_unroll=num_unroll,
batch_size=batch_size)
state = batch.state('lstm_state')
update_state = batch.save_state('lstm_state', state + 1)
coord = tf.train.Coordinator()
tf.train.start_queue_runners(sess=sess, coord=coord)
input_batch_val = sess.run([
batch.key, batch.next_key, batch.sequences['input'],
batch.context['context_key'], state, batch.length, update_state][2])
pprint(input_batch_val)

Output:

array([[[0.33537843, 0.76504494, 0.368679 ],
[0.47943187, 0.58871135, 0.06263617]],

[[0.33537843, 0.76504494, 0.368679 ],
[0.47943187, 0.58871135, 0.06263617]],

[[0.33537843, 0.76504494, 0.368679 ],
[0.47943187, 0.58871135, 0.06263617]]])

What further justifies my suspect is the expected values of sequences in unit tests batch_sequences_with_states_test.py, which is duplicating the first segment of "seq1".

def _testBasicPadding(self, pad, key=None, make_keys_unique=False):
num_unroll = 2 # Divisor of value_length - so no padding necessary.
expected_seq1_batch1 = np.tile(
self.sequences["seq1"][np.newaxis, 0:num_unroll, :],
(self.batch_size, 1, 1))