Q:

Resource exhausted: OOM when allocating tensor with shape[845246,300]

I am working with a sequence to sequence language model, and after changing the code to pass custom word embedding weights to the Embeddings layer, I am receiving a OOM error when I try to train on the gpu. 
Here is the relevant code:
def create_model(word_map, X_train, Y_train, vocab_size, max_length):
    # define model
    model = Sequential()
    # get custom embedding weights as matrix
    embedding_matrix = get_weights_matrix_from_word_map(word_map)
    model.add(Embedding(len(word_map)+1, 300, weights=[embedding_matrix], input_length=max_length-1))
    model.add(LSTM(50))
    model.add(Dense(vocab_size, activation='softmax'))
    print(model.summary())
    # compile network
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    model.fit(X_train, Y_train, epochs=100, verbose=2)
    return model

And here is the full error log from the server:
    File "/home2/slp24/thesis/UpdatedLanguageModel_7_31.py", line 335, in create_model_2
    model.fit(X_train, Y_train, batch_size=32, epochs=1, verbose=2)  ## prev X, y
  File "/opt/python-3.4.1/lib/python3.4/site-packages/keras/models.py", line 963, in fit
    validation_steps=validation_steps)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/keras/engine/training.py", line 1682, in fit
    self._make_train_function()
  File "/opt/python-3.4.1/lib/python3.4/site-packages/keras/engine/training.py", line 990, in _make_train_function
    loss=self.total_loss)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/keras/optimizers.py", line 466, in get_updates
    m_t = (self.beta_1 * m) + (1. -  self.beta_1) * g
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/ops/math_ops.py", line 898, in binary_op_wrapper
    y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y")
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 932, in convert_to_tensor
    as_ref=False)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 1022, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/ops/gradients_impl.py", line 100, in _IndexedSlicesToTensor
    value.values, value.indices, value.dense_shape[0], name=name)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5186, in unsorted_segment_sum
    num_segments=num_segments, name=name)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/opt/python-3.4.1/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[845246,300] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[Node: training/Adam/mul_2/y = UnsortedSegmentSum[T=DT_FLOAT, Tindices=DT_INT32, Tnumsegments=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/embedding_1/Gather_grad/Reshape, training/Adam/gradients/embedding_1/Gather_grad/Reshape_1/_101, training/Adam/mul_2/strided_slice)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. 

Edit:
So far I have tried

Adding batching, started with batch_size=32
I am currently working to decrease the number of output classes from 845,286.  I think something went wrong when I calculated the custom embedding matrix, specifically when I was "connecting" the vocabulary token index's assigned during preprocessing and the y_categorical values assigned by Keras that the model uses...

Any help or guidance is greatly appreciated!  I have searched many similar issued but have not been able to apply those fixes to my code thus far.  Thank you

A:

You're exceeding the memory size of your GPU. 
You can:

Train/Predict with smaller batches    
Or, if even a batch_size=1 is too much, you need a model with less parameters. 

Hint, the length in that tensor (845246) is really really big.  Is that the correct length?