noamaanMulla-03/transformers-issues huggingface dataset:
[Albert] Add position ids to allowed uninitialized weights
Author: patrickvonplaten
Pull request: yes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6700 


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=h1) Report
> Merging [#6719](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/625318f52516b413126be1bb1cb6818231d2eca6?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6719/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6719      +/-   ##
==========================================
- Coverage   79.49%   79.48%   -0.01%     
==========================================
  Files         156      156              
  Lines       28405    28406       +1     
==========================================
- Hits        22581    22579       -2     
- Misses       5824     5827       +3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6719/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `83.53% <100.00%> (+0.03%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6719/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.71% <0.00%> (-0.76%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=footer). Last update [625318f...8f67175](https://codecov.io/gh/huggingface/transformers/pull/6719?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Finetune.sh showing killed
Author: mc2259
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I am getting this very strange issue while running the finetuning script and have absolutely no idea what is going wrong.

./finetune.sh: line 14:  9040 Killed                  python finetune.py --learning_rate=3e-5 --fp16 --gpus 1 --do_train --do_predict --n_val 1000 --val_check_interval 0.1 "$@"


<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hi, can you post the full command and your env info

lets all hang out in #6711
TF2 TPU slow?
Author: volker42maru
Pull request: no

## Environment info
- `transformers` version: 3.0.2
- Platform: ubuntu 18.04
- Python version: 3.6
- Tensorflow version (GPU?): TF 2.3.0

### Who can help
@jplu 

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] task: MLM

## To reproduce
I am using the TFTrainer from `transformers` for MLM pretraining. However, it seems that even for TPU training each batch is fed separately to the TPU, while it's usually more common to feed a bunch of batches to the TPU for efficiency (see https://github.com/tensorflow/models/blob/master/official/nlp/bert/model_training_utils.py#L226).

I am not sure that's the only problem, but MLM pretraining BERT is around 3x slower on TPU with the TFTrainer compared to the official implementation (https://github.com/google-research/bert).

For better TPU usage, we probably need sth like here:
https://github.com/tensorflow/models/blob/master/official/nlp/bert/model_training_utils.py#L345-L361

Comments:
Hello!

Training on TPU is slow indeed because the training loop is not optimized for TPU. For two reasons:
1) we don't want to have a version that is specifically optimized for each device which will create to much confusion for maintenance.
2) having a training loop optimized for TPU will limit the possibility of logging, the logging will be at every epoch instead of every X steps. Which is a not wanted behavior.

Nevertheless if you have a solution that respect these two points, I will be happy to review your PR :)

You are right, this will introduce some behavior changes because logging/saving is not possible while the TPU is processing several batches.

I actually played around with this a bit and introduced a variable `steps_per_loop` that I set to `200` when using TPU. I then used an iterator for the dataset to only do batch feeding/optimization during this period. However, this only improved the training speed marginally, so I don't think it's worth a PR.

What gives me a bigger speedup (around 80%) is actually using 'keras compile/fit', where we can set `experimental_steps_per_execution` to different values depending on the device, for the training loop. I would be curious though if we could substitute part of the trainer or even the whole training by a "keras style" implementation. However, this would change the current architecture quite significantly and I am not sure it will retain the generic properties of the TFTrainer class.

Maybe it becomes clearer if I just show you how I implemented an option to use keras for the training loop in the TFTrainer (pretty hacky of course): https://gist.github.com/volker42maru/20641970599c27dc9503161f52aa67c9#file-tf_trainer_train-py-L85-L113


I will close this for now, because it's probably more of a long term prospect.
Huggingface create_optimizer method not working
Author: brand17
Pull request: no

## Environment info
     
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.6
- PyTorch version (GPU?): 1.5.0+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger
@jplu 

## Information

Model I am using (Bert, XLNet ...): Roberta

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run this code:
```
import tensorflow as tf
from transformers import RobertaConfig, TFRobertaForMaskedLM, create_optimizer
config = RobertaConfig()  
optimizer,lr = create_optimizer(1e-4,1000000,10000,0.1,1e-6,0.01)
training_loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model = TFRobertaForMaskedLM(config)
model.compile(optimizer=optimizer, loss=training_loss)
input = tf.random.uniform(shape=[1,25], maxval=100, dtype=tf.int32)
hist = model.fit(input, input, epochs=1, steps_per_epoch=1,verbose=0)

```
2. I am getting an error:

> TypeError: apply_gradients() got an unexpected keyword argument 'experimental_aggregate_gradients'


## Expected behavior
optimizer should be created

Comments:
It seems that keras is passing `experimental_aggregate_gradients` to `apply_gradients`, but the *transformers* TF2 optimizer does not have this argument (see https://github.com/huggingface/transformers/blob/master/src/transformers/optimization_tf.py#L224).

One workaround right now is to set `optimizer._HAS_AGGREGATE_GRAD = False`, which prevents keras from passing this argument.

Thanks for the analysis @volker42maru. @jplu when you're back from vacation, we should fix this optimizer to accept this argument.

Hello!

I was aware of this, and this was on purpose to make the trainer compliant with all the TF 2.X versions. Now that the trainer is fixed to v 2.2 min, I will modify accordingly the method. Thanks @volker42maru to raise this and make me remember I had to update this.

The PR #6717 should fix the problem.
Adding model cards for 5 models
Author: AMontgomerie
Pull request: yes
Labels: name: model card

Adding model cards for:
- iarfmoose/bert-base-cased-qa-evaluator
- iarfmoose/roberta-base-bulgarian-pos
- iarfmoose/roberta-base-bulgarian
- iarfmoose/roberta-small-bulgarian-pos
- iarfmoose/roberta-small-bulgarian

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=h1) Report
> Merging [#6703](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/640550fc7a1e311915ead1bcca6dacea0c503faf?el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6703/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6703      +/-   ##
==========================================
- Coverage   77.85%   77.75%   -0.10%     
==========================================
  Files         146      146              
  Lines       26326    26326              
==========================================
- Hits        20496    20471      -25     
- Misses       5830     5855      +25     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.20% <0.00%> (-0.29%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (+5.26%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/6703/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9ncHQyLnB5) | `95.32% <0.00%> (+29.90%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=footer). Last update [640550f...8293ed6](https://codecov.io/gh/huggingface/transformers/pull/6703?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Model card for kuisailab/albert-xlarge-arabic
Author: alisafaya
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=h1) Report
> Merging [#6731](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e11d923bfc61ed640bc7e696549578361126485e?el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6731/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6731      +/-   ##
==========================================
+ Coverage   79.42%   79.48%   +0.06%     
==========================================
  Files         156      156              
  Lines       28411    28411              
==========================================
+ Hits        22565    22583      +18     
+ Misses       5846     5828      -18     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG5ldC5weQ==) | `21.12% <0.00%> (-71.05%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `77.37% <0.00%> (-19.71%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.64% <0.00%> (-0.66%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.66% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `82.93% <0.00%> (+66.66%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/6731/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hbGJlcnQucHk=) | `90.90% <0.00%> (+69.43%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=footer). Last update [e11d923...bfd52f8](https://codecov.io/gh/huggingface/transformers/pull/6731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[s2s] run_eval.py QOL improvements and cleanup
Author: sshleifer
Pull request: yes


- default to saving metrics to `metrics.json`
- cleanup: delete some dead code.
- save seconds_per_sample, runtime, num_samples


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=h1) Report
> Merging [#6746](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a75c64d80c76c3dc71f735d9197a4a601847e0cd?el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6746/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6746      +/-   ##
==========================================
- Coverage   78.96%   78.93%   -0.04%     
==========================================
  Files         157      157              
  Lines       28486    28486              
==========================================
- Hits        22495    22485      -10     
- Misses       5991     6001      +10     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `80.70% <0.00%> (-2.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6746/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <0.00%> (-0.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=footer). Last update [a75c64d...0b9f1cd](https://codecov.io/gh/huggingface/transformers/pull/6746?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Cached feature files - Naming introduces confusion/model mixing
Author: VictorSanh
Pull request: no
Labels: name: wontfix

# 🐛 Bug

The problem arises when using:
* [X] the official example scripts: glue.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: any example of GLUE
* [ ] my own task or dataset: (give details below)

In `run_glue.py` (and possibly in other run_* scripts interfaced with the new Trainer), the cache naming for the pre-processed features introduces confusion between models.

https://github.com/huggingface/transformers/blob/3d3e605affb792b78c918aac48f6bc82cfbf7e3e/src/transformers/data/datasets/glue.py#L87

`tokenizer.__class__.__name__` is `BertTokenizer`, `DistilBertTokenizer`, etc. which removes the information of which tokenization is used. For instance, `bert-base-uncased` and `bert-base-cased` would end up with the same file name while being potentially two really different files.

In scripts not interfaced yet with the Trainer, the `model_name_or_path` is used which prevents conflicting files to have the same name:
https://github.com/huggingface/transformers/blob/3d3e605affb792b78c918aac48f6bc82cfbf7e3e/examples/text-classification/run_xnli.py#L315

Comments:
Good point. We can do a simple hack for the time being but the cleanest way out will be to use 🤗nlp to process the data (which I'm adding at the moment but probably won't be fully operational in the examples before early next week).

🤗nlp has a lot more reliable and general hash-naming caching scheme for all data processing based on the function and all the inputs/outputs when processing the dataset (using some smart serialization with `dill` and hashing algo).

Ok! I might go back to the old scripts to experiment in the meantime.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can not reproduce article numbers
Author: vladgets
Pull request: no
Labels: name: wontfix

# 🐛 Bug
https://github.com/huggingface/transformers/tree/master/examples/summarization/bertabs

## Information
Bertabs Rouge1/2 F1 evaluation numbers that I am getting are much less than in their article. About twice less!  

Model I am using (Bert, XLNet ...):
Bertabs 

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
Summarization 

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
These are numbers that I am getting when running the examples (in the article or here: https://github.com/nlpyang/PreSumm) they mention much higher numbers: 

****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.195
Precision >> 0.209
Recall    >> 0.186

** ROUGE 2
F1        >> 0.094
Precision >> 0.104
Recall    >> 0.089

** ROUGE L
F1        >> 0.221
Precision >> 0.233
Recall    >> 0.212

same here. Also getting ROUGE scores in the same ball-park

Well, I already switched to Bart model. I do not know what is the reasons for low numbers here, possibly some bugs in the porting, but in general BertAbs model does not seem the best approach for it.

Alex, by the way I see that you are currently working on Summarization topic too. If you like you can write me directly to vgetselevich@nvidia.com (I am currently conversational AI researcher at Nvidia) and we can have more detailed discussion on this topic.

Will do, thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bertabs metrics lower than paper
Author: kyungyunji
Pull request: no
Labels: name: wontfix

name: bertabs

I tested abstractive summarization pre-trained model using the source under transformers/examples/summarization/bertabs/...

My dataset are CNN & Daily mail, which are 30 thousands of docs.

However, the result of rouge score is as follows



****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.275
Precision >> 0.299
Recall    >> 0.260

** ROUGE 2
F1        >> 0.161
Precision >> 0.184
Recall    >> 0.149

** ROUGE L
F1        >> 0.305
Precision >> 0.326
Recall    >> 0.290


why is the result different from that of the article, Text Summarization with Pretrained Encoders ?


Comments:
bump. i have the same question. paper says 5 BeamSize and better accuracy for bert-base. 

same here

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Could you please implement a Adafactor optimizer? :)
Author: christophschuhmann
Pull request: no

## 🚀 Feature

Could you please implement a Adafactor optimizer? :)

( https://arxiv.org/abs/1804.04235 )

## Motivation

In contrast to Adam it requires much less GPU memory. 
I tried to use the FairSeq implementation for the pytorch-transformers, but I'm no expert and I couldn't get it done.

Could you please do that? :)


## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
What didn't work for you with the fairseq implementation?

It seems pretty self-contained: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py#L65-L213

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


FYI @sshleifer -- I was wrong -- able to train T5-large even batch==1 with FP32, no gradient check-pointing and ADAM. Given that T5 team strongly recommends AdaFactor -- giving it a try, other pieces perhaps being more difficult...