Issue 1:
I am using TF2.0 latest nightly build and I am trying to train LSTM model for text classification on very large dataset of 16455928 sentences. For embedding layer in the model, I have a vocab size of 366856 and I used 1000 as embedding dimension value in it, on which the 2 GPUs(Tesla T4 from Google) ran out of memory.

Since I can not lower the size of vocabulary (maybe there is a way), so I used lower value for embedding dimension (100) on which the model starts training. Now my question is if there is a way I can use higher value of embedding dimension?. Maybe by putting set of layers of my model on different GPUs, if so then what is the way in TF2.0? Also, will using more number GPUs help? Thank you!

Issue 2:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Mac Sierra

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):

binary

TensorFlow version (use command below):

v1.12.0-rc2-3-ga6d8ffae09 1.12.0

Python version:

3.6.5

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

no Cuda

GPU model and memory:

not using GPU

You can collect some of this information using our environment capture script

You can also obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the current behavior

I'm trying Magenta from Google Brain. When I run python onsets_frames_transcription_transcribe.py --acoustic_run_dir /Users/lorenzori/Downloads/maestro_checkpoint ~/Downloads/test_audio.wav the script starts, then it ends with:

/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
/Users/lorenzori/virtualenvs/test-audio/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
return _inspect.getargspec(target)
2019-01-10 17:49:57.458805: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
INFO:tensorflow:Restoring parameters from /Users/lorenzori/Downloads/maestro_checkpoint/train/model.ckpt-maestro
INFO:tensorflow:Starting transcription for /Users/lorenzori/Downloads/test_audio.wav...
INFO:tensorflow:Processing file...
INFO:tensorflow:Running inference...
[1] 20612 bus error python onsets_frames_transcription_transcribe.py --acoustic_run_dir

Using lldb I obtain the following info:

Process 19280 stopped
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn:
-> 0x110a24805 <+1541>: xchgq %rdi, -0x40(%rsi)
0x110a24809 <+1545>: xorl %edi, %edi
0x110a2480b <+1547>: xchgq %rdi, (%rsi)
0x110a2480e <+1550>: addq $0x200, %rsi ; imm = 0x200
Target 0: (python) stopped.
(lldb) bt
* thread #25, stop reason = EXC_BAD_ACCESS (code=2, address=0x700001382000)
* frame #0: 0x0000000110a24805 libopenblasp-r0.3.0.dev.dylib`dgemm_thread_tn + 1541
frame #1: 0x00000001108f3e26 libopenblasp-r0.3.0.dev.dylib`cblas_dgemm + 854
frame #2: 0x0000000105564285 multiarray.cpython-36m-darwin.so`cblas_matrixproduct + 4917
frame #3: 0x0000000105529d37 multiarray.cpython-36m-darwin.so`PyArray_MatrixProduct2 + 215
frame #4: 0x000000010552ed1f multiarray.cpython-36m-darwin.so`array_matrixproduct + 191
frame #5: 0x00000001000d1cbe Python`_PyCFunction_FastCallDict + 463
frame #6: 0x00000001001362d6 Python`call_function + 489
frame #7: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
frame #8: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
frame #9: 0x000000010013713b Python`fast_function + 218
frame #10: 0x00000001001362ad Python`call_function + 448
frame #11: 0x000000010012f224 Python`_PyEval_EvalFrameDefault + 4964
frame #12: 0x00000001001373db Python`_PyFunction_FastCall + 121
frame #13: 0x00000001001362ad Python`call_function + 448
frame #14: 0x000000010012f18b Python`_PyEval_EvalFrameDefault + 4811
frame #15: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
frame #16: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
frame #17: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
frame #18: 0x0000000100182073 Python`partial_call + 258
frame #19: 0x0000000100099da2 Python`PyObject_Call + 101
frame #20: 0x000000010012f3f4 Python`_PyEval_EvalFrameDefault + 5428
frame #21: 0x0000000100136a38 Python`_PyEval_EvalCodeWithName + 1719
frame #22: 0x000000010013730b Python`_PyFunction_FastCallDict + 449
frame #23: 0x0000000100099f21 Python`_PyObject_FastCallDict + 196
frame #24: 0x000000010009a044 Python`_PyObject_Call_Prepend + 156
frame #25: 0x0000000100099da2 Python`PyObject_Call + 101
frame #26: 0x00000001000e4460 Python`slot_tp_call + 50
frame #27: 0x0000000100099da2 Python`PyObject_Call + 101
frame #28: 0x00000001212b078e _pywrap_tensorflow_internal.so`tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*) + 974
frame #29: 0x000000012bd82422 libtensorflow_framework.so`tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) + 6690
frame #30: 0x000000012bd895ba libtensorflow_framework.so`std::__1::__function::__func<std::__1::__bind<void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&, long long&>, std::__1::allocator<std::__1::__bind<void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long), tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode const&, long long&> >, void ()>::operator()() + 58
frame #31: 0x000000012bdde824 libtensorflow_framework.so`Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) + 1876
frame #32: 0x000000012bdddfd4 libtensorflow_framework.so`std::__1::__function::__func<tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'(), std::__1::allocator<tensorflow::thread::EigenEnvironment::CreateThread(std::__1::function<void ()>)::'lambda'()>, void ()>::operator()() + 52
frame #33: 0x000000012be00070 libtensorflow_framework.so`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::function<void ()> > >(void*) + 96
frame #34: 0x00007fffb96b893b libsystem_pthread.dylib`_pthread_body + 180
frame #35: 0x00007fffb96b8887 libsystem_pthread.dylib`_pthread_start + 286
frame #36: 0x00007fffb96b808d libsystem_pthread.dylib`thread_start + 13

Describe the expected behavior

The script should run

Code to reproduce the issue

python onsets_frames_transcription_transcribe.py --acoustic_run_dir <checkpoint_dir> <wav_file>