Issue 1:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): from conda

TensorFlow version (use command below): 1.10 gpu from conda

Python version: 3.7 from conda

Bazel version (if compiling from source): n/a

GCC/Compiler version (if compiling from source): n/a

CUDA/cuDNN version: 10

GPU model and memory: GX 1070 maxq

You can collect some of this information using our environment capture script

You can also obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the current behavior

Suddenly running update row operations became laggy and never progressed...

Describe the expected behavior

update row operations should not run indefinetly

from mymodule import ex
import gc, resource, multiprocessing

def run_ex(print_usage=False, *args, **kwargs):
ex.run(config_updates=kwargs)
if print_usage:
usage = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print('resources used: {}'.format(usage))
gc.collect()

def pool_run(*args, processes=1, print_usage=False, **kwargs):
with multiprocessing.Pool(processes=processes) as pool:
results = pool.apply(run_ex, (print_usage, *args), kwargs)
gc.collect()
return results

pool_run(processes=1, print_usage=True, **opts)

# ex is a wrapper that calls

def train_fn(
model, # <--- WALSModel
input_tensor, # tf.SparseTensor
iterations:int,
sp_sparse=None, # sp.sparse.coo_matrix
sacred_run=None,
print_progress:bool=True,
logger=None
):
os.environ['KMP_DUPLICATE_LIB_OK']='True'
# extract row and column factors
row_factor = model.row_factors[0]
col_factor = model.col_factors[0]

# update operations
row_update_op = model.update_row_factors(sp_input=input_tensor)#[1]
col_update_op = model.update_col_factors(sp_input=input_tensor)#[1]

with tf.Session() as sess:
# init model
sess.run(model.initialize_op)
sess.run(model.worker_init)

if print_progress:
status = Sil(total=iterations)
for i in range(iterations):
if logger is not None: logger.debug('Training iteration {}/{}'.format(str(i), str(iterations)))
# update rows
if logger is not None: logger.debug('Updating rows') # <--- I see this in logs
sess.run(model.row_update_prep_gramian_op)
sess.run(model.initialize_row_update_op)
_, _, loss, reg, _ = sess.run(row_update_op)
if sacred_run is not None: sacred_run.log_scalar("loss.row", loss, i)

# update cols
if logger is not None: logger.debug('Updating columns') # <--- never makes it here
sess.run(model.col_update_prep_gramian_op)
sess.run(model.initialize_col_update_op)
_, _, loss, reg, _ = sess.run(col_update_op)
if sacred_run is not None: sacred_run.log_scalar("loss.col", loss, i)

# update status
if print_progress:
status.tick(prefix='iteration')

if sacred_run is not None and sp_sparse is not None:
if logger is not None: logger.debug('Calculating rmse.')
rf = row_factor.eval(session=sess)
cf = col_factor.eval(session=sess)
it_rmse = rmse(sp_sparse, rf, cf)
sacred_run.log_scalar("rmse", it_rmse, i)

# eval row / col factors
output_row = row_factor.eval(session=sess)
output_col = col_factor.eval(session=sess)
sess.close()
os.environ['KMP_DUPLICATE_LIB_OK']='False'
return output_row, output_col

Code to reproduce the issue

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Issue 2:
If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

x_nparray = $VERY_LARGE_NP_ARRAY(bigger than 2G)

x = tf.Variable(init_value=x_nparray)

saver=tf.Saver()

with tf.Session() as sess:

saver.save(sess, 'path/to/save')

that would generate very large metagraph binary file, when i try to restore the model, it would failed because of the meta graph file size exceed the protobuf limit size.

I cannot reduce the size of x_nparray, because it's generated by other training system like caffe.

What other attempted solutions have you tried?

set bigger limit byte of protobuf, however, the x_nparray is bigger than 3G, while the largest limit i can set is 2G.