T5 base use a lot of memory to train on 
Author: ghost
Pull request: no

Hi
I am using transformers 3.5.1, t5-base uses a lot of memory I cannot even train it on 4 V100 GPUs with batch size of 32 could you elaborate is there is any memory issue with this model? thanks

Comments:
Hi @juliahane 
Could you post the training command ? 
batch size 32 is total batch size or `per_device_batch_size` ?

In my experiments I was able to use max 4 `per_device_batch_size` with `max_input_length` 512 and `max_target_length` 64.

Also, this kind of question should be asked on the[ forum ](https://discuss.huggingface.co/t/t5-finetuning-tips/684) as it's not a bug or issue.

[This](https://discuss.huggingface.co/t/t5-finetuning-tips/684) discussion might help.

this is per device batch size. max_length = 128
to me this is a bug that the model requires this much memory
for small I can run the same thing with batch size =64

On Wed, Jan 6, 2021 at 8:40 AM Suraj Patil <notifications@github.com> wrote:

> Hi @juliahane <https://github.com/juliahane>
> Could you post the training command ?
> batch size 32 is total batch size or per_device_batch_size ?
>
> In my experiments I was able to use max 4 per_device_batch_size with
> max_input_length 512 and max_target_length 64.
>
> Also, this kind of question should be asked on the forum
> <https://discuss.huggingface.co/t/t5-finetuning-tips/684> as it's not a
> bug or issue.
>
> This <https://discuss.huggingface.co/t/t5-finetuning-tips/684> discussion
> might help.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/9430#issuecomment-755164832>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM72EZRVYSEBPLPVBF3SYQOXJANCNFSM4VXEGT3A>
> .
>


coud you tell me if Adafactor save on memory?

On Wed, Jan 6, 2021 at 8:40 AM Suraj Patil <notifications@github.com> wrote:

> Hi @juliahane <https://github.com/juliahane>
> Could you post the training command ?
> batch size 32 is total batch size or per_device_batch_size ?
>
> In my experiments I was able to use max 4 per_device_batch_size with
> max_input_length 512 and max_target_length 64.
>
> Also, this kind of question should be asked on the forum
> <https://discuss.huggingface.co/t/t5-finetuning-tips/684> as it's not a
> bug or issue.
>
> This <https://discuss.huggingface.co/t/t5-finetuning-tips/684> discussion
> might help.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/9430#issuecomment-755164832>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM72EZRVYSEBPLPVBF3SYQOXJANCNFSM4VXEGT3A>
> .
>


Hi @juliahane it is indeed the case that adafactor improves memory usage, which is why the original author uses it. You can check out the [paper](https://arxiv.org/abs/1804.04235) on adafactor for more info, but the abstract says the most. My intuition here is that adafactor (or similar memory-efficient optimizer) is required to train the large t5 models.

thank you, very helpful, I will try it.

On Wed, Jan 6, 2021 at 11:39 AM Kenneth Enevoldsen <notifications@github.com>
wrote:

> Hi @juliahane <https://github.com/juliahane> it is indeed the case that
> adafactor improves memory usage, which is why the original author uses it.
> You can check out the paper <https://arxiv.org/abs/1804.04235> on
> adafactor for more info, but the abstract says the most. My intuition here
> is that adafactor (or similar memory-efficient optimizer) is required to
> train the large t5 models.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/9430#issuecomment-755223491>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AM3GZM7WKUR23L23FVKAMZTSYQ4UZANCNFSM4VXEGT3A>
> .
>


In the thread they say to set autoscaling to off, do you know @kenneth  how
I can do it?
Apart from this, I could not find more suggestion for saving memory on GPU
in that thread
thanks

On Wed, Jan 6, 2021 at 2:13 PM julia hane <juliahane123@gmail.com> wrote:

> thank you, very helpful, I will try it.
>
> On Wed, Jan 6, 2021 at 11:39 AM Kenneth Enevoldsen <
> notifications@github.com> wrote:
>
>> Hi @juliahane <https://github.com/juliahane> it is indeed the case that
>> adafactor improves memory usage, which is why the original author uses it.
>> You can check out the paper <https://arxiv.org/abs/1804.04235> on
>> adafactor for more info, but the abstract says the most. My intuition here
>> is that adafactor (or similar memory-efficient optimizer) is required to
>> train the large t5 models.
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/issues/9430#issuecomment-755223491>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AM3GZM7WKUR23L23FVKAMZTSYQ4UZANCNFSM4VXEGT3A>
>> .
>>
>


I assume it is the:
```
scale_parameter (bool, optional, defaults to True) – If True, learning rate is scaled by root mean square
```
in the adafactor ([documentation](https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adafactor-pytorch))

but maybe @patil-suraj could confirm this?

But as they used the scaling in the original paper I couldn't imagine it to be highly influential.

For reference the optimizer I use is:
```
optimizer = transformers.Adafactor(model.parameters(), lr=0.001,
                                   relative_step=False, warmup_init=False, 
                                   decay_rate=0.0, clip_threshold=1.0)
scheduler = None
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.