opened
chrisrytting
username_0
I'm trying to train the 3B parameter model from scratch. When I try to do so by executing the following command

```
export CUDA_VISIBLE_DEVICES=3
export MODEL_DIR=models-t5/3Bscratch
export DATA_DIR=data/c/tsvs
t5_mesh_transformer     --model_dir="${MODEL_DIR}"   --t5_tfds_data_dir="${DATA_DIR}"   \
--gin_file="dataset.gin" --gin_param="utils.run.mesh_shape = 'model:1,batch:1'" \
--gin_param="utils.run.mesh_devices = ['gpu:0']"   --gin_file="models/bi_v1_3B.gin"   \
--gin_param="MIXTURE_NAME = 'atr'" --module_import='t5_task_def' 
```

I get an OOM error

```
  (0) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node encoder/block_018/layer_000/SelfAttention/einsum_8/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[scalar_add/parallel_0/add/_5677]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node encoder/block_018/layer_000/SelfAttention/einsum_8/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
```

I add `--gin_param="tokens_per_batch=1"` to the above command, thinking I need to reduce my batch size to fit on the GPU, (which in the first place this error is weird because I'm using a Nvidia Tesla V100-SXM3-32GB), and I still get the OOM error. Is there something wrong with the command I'm running?
OOM error when training even when batch
issue
created
craffel
username_1
I'm not sure the 3B model will fit in memory on a single GPU (even a v100), though it is possible to train the 3B model on a v2-8 TPU which has 64 GB of RAM. I'm also not sure `tokens_per_bath=1` will work if your input sequences have length 512 (which it appears they do based on the error you're seeing). You could try `tokens_per_bath=512` which would correspond to a single sequence in each batch, and/or you could try decreasing the input/output sequence lengths depending on the sequence lengths of the task you are trying.
comment
closed
craffel
username_1
issue
created
chrisrytting
username_0
@username_1 thanks for the quick response. However, I have fine-tuned without a problem on these GPUs. Does training from scratch require more memory for some reason? I understand the pre-training is different than fine-tuning on specific tasks, but I basically want to fine-tune without any pre-training.

Also, if I'm getting OOM error with `tokens_per_batch=1`, there's no way I could go bigger, right?
comment
created
craffel
username_1
If your input sequence has 512 tokens and you set tokens_per_batch=1, I'm not sure what the code will do. You should set tokens_per_batch to be an integer multiple of your input sequence length, so the smallest you could use is 512.
comment
created
adarob
username_2
Can you try installing mesh-tf and t5 both from master on github?

I think this will be fixed by the combination of:
https://github.com/tensorflow/mesh/pull/175
https://github.com/google-research/text-to-text-transfer-transformer/pull/386

If we can confirm that solves your problem, I will update the packages.
comment
reopened
adarob
username_2
I'm trying to train the 3B parameter model from scratch. When I try to do so by executing the following command

```
export CUDA_VISIBLE_DEVICES=3
export MODEL_DIR=models-t5/3Bscratch
export DATA_DIR=data/c/tsvs
t5_mesh_transformer     --model_dir="${MODEL_DIR}"   --t5_tfds_data_dir="${DATA_DIR}"   \
--gin_file="dataset.gin" --gin_param="utils.run.mesh_shape = 'model:1,batch:1'" \
--gin_param="utils.run.mesh_devices = ['gpu:0']"   --gin_file="models/bi_v1_3B.gin"   \
--gin_param="MIXTURE_NAME = 'atr'" --module_import='t5_task_def' 
```

I get an OOM error

```
  (0) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node encoder/block_018/layer_000/SelfAttention/einsum_8/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[scalar_add/parallel_0/add/_5677]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[node encoder/block_018/layer_000/SelfAttention/einsum_8/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
```

I add `--gin_param="tokens_per_batch=1"` to the above command, thinking I need to reduce my batch size to fit on the GPU, (which in the first place this error is weird because I'm using a Nvidia Tesla V100-SXM3-32GB), and I still get the OOM error. Is there something wrong with the command I'm running?
OOM error when training even when batch
issue
created
chrisrytting
username_0
So I'm working out of a docker container, and I hope that doesn't make things too tricky. But basically before I had just pip installed t5 and so I think the place it had the packages `t5` and `mesh_tensorflow` was in `/usr/local/lib/python3.6/dist-packages`. So I just renamed the ones installed by pip to `t5package` and `mesh_tensorflowpackage` and then cloned https://github.com/tensorflow/mesh into `mesh_tensorflow` and https://github.com/google-research/text-to-text-transfer-transformer into `t5` to replace the pip-installed packages with the master branches from github.

But then when I run the same command, I get this error

```
  File "/usr/local/bin/t5_mesh_transformer", line 5, in <module>
    from t5.models.mesh_transformer_main import console_entry_point
ModuleNotFoundError: No module named 't5.models'
```
comment
created
adarob
username_2
You should be able to do `pip install git+
https://github.com/tensorflow/mesh`, etc.
comment
created
chrisrytting
username_0
Sorry I deleted that last comment I thought there was a detail wrong and panicked but there wasn't. So I pip uninstalled t5 and mesh_tensorflow at that point, and pip installed both with git+.

I get nearly the same errors, it just looks like some of the dimensions are changed:

```
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node encoder/block_018/layer_000/SelfAttention/einsum_7/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[scalar_add/parallel_0/add/_5679]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted: OOM when allocating tensor with shape[1,4,32,512,128] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node encoder/block_018/layer_000/SelfAttention/einsum_7/parallel_0/einsum/Einsum (defined at /lib/python3.6/dist-packages/mesh_tensorflow/ops.py:1336) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
```
comment
created
chrisrytting
username_0
So I tried parallelizing across two of the v100s, and I still got those errors above. When I parallelized across 4 (3 wouldn't work because the model doesn't split evenly across 3), it finally worked. 

Still, it seems weird to me that I could fine-tune with a sequence length of 250 on 1 GPU but couldn't train with the same sequence length across 2 GPUs. Aren't training and fine-tuning the same thing, just at different stages of training? Or am I misunderstanding the difference between them? Like when you guys train on MRPC from scratch in the readme, there's no reason that would be more computationally intensive than if you had fine-tuned on MRPC using one of the pre-trained models, right?
comment
closed
craffel
username_1
issue
created
craffel
username_1
Yes, they are exactly the same, the only difference is that before fine-tuning we load variables from a checkpoint.

You can diff the operative config gin files that got written out to your model directories to triple check that there are no other differences (other than sequence length) between your fine-tuning and pre-training runs.
comment
created
chrisrytting
username_0
Oh sorry that finally made sense about the quadratic scaling. Thanks for your help, guys!
comment