(TF) InternalError: Multiple CPU devices 
Author: jmwoloso
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: Linux-5.4.0-90-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.3.4 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed


### Who can help
@patrickvonplaten @Rocketknight1
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. instantiate `TFAutoModelForSequenceClassification.from_pretrained("allenai/longformer-base-4096")`
2. compile the model under `tf.distribute.experimental.MultiWorkerMirroredStrategy` strategy
3. run `fit()`

```
Traceback (most recent call last):
  File "training.py", line 20, in <module>
    classifier.fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/training.py", line 27, in fit
    self._fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 50, in _fit
    self._fit_model()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 430, in _fit_model
    class_weight=self.class_weight_
  File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/miniconda/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError: Multiple CPU devices [/job:worker/replica:0/task:0/device:GPU:7,/job:worker/replica:0/task:0/device:CPU:0,/job:localhost/replica:0/task:0/device:CPU:0] [Op:__inference_train_function_800368]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I would expect the model to begin training however it errors with the traceback above. I have tested this same script with `TFDistilBertForSequenceClassification` and the model successfully trains, so I'm kinda stuck here.

EDIT: This isn't running locally on my machine, but in Azure on the [A100 VM series](https://docs.microsoft.com/en-us/azure/virtual-machines/nda100-v4-series)

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
This is unusual - is this issue unique to Hugging Face models? Does it still occur with a basic Keras model? 

Hi @Rocketknight1! I haven't tried with basic Keras models. I can try that if you think it would help, however, `TFDistilBertForSequenceClassification` trained successfully, so I suspect it is something specific to `Longformer`

So I traced it to [here](https://github.com/tensorflow/tensorflow/blob/b1b76e24284d55156ad4e12fb847bfbc637b9a20/tensorflow/compiler/jit/device_util.cc#L173-L176). Must be some issue with the JIT compilation. I'll test some stuff out.

Still testing stuff, but I didn't pay close attention when I ran the cli tool to get system info. I'm actually using a docker container on the A100 VMs that is based on Debian + NVIDIA (cudnn 8/ cuda 11.2).

The `tensorflow` version is actually 2.7.*
The `torch` version is actually 1.10.0

[this](https://github.com/tensorflow/tensorflow/issues/53192) seems suspiciously related as they're having JIT issues on the A100. I think in my case though, TF isn't sure which device to use.

I was able to train the model successfully in the end by disabling XLA JIT compilation with `ENV TF_XLA_FLAGS="--tf_xla_auto_jit=-1"`

Ok, so it works for a single epoch. When running more than 1 epoch, the first epoch succeeds and upon beginning the second epoch, we get this (first epoch logging included here for completeness).

```
1/232 [..............................] - ETA: 35:39:11 - loss: 0.5014 - Accuracy: 0.8750
  2/232 [..............................] - ETA: 4:16 - loss: 0.4386 - Accuracy: 0.8125    
  3/232 [..............................] - ETA: 4:02 - loss: 0.3492 - Accuracy: 0.6250
  4/232 [..............................] - ETA: 3:51 - loss: 0.3182 - Accuracy: 0.6250
  5/232 [..............................] - ETA: 3:46 - loss: 0.2899 - Accuracy: 0.6750
  6/232 [..............................] - ETA: 3:42 - loss: 0.2548 - Accuracy: 0.7083
  7/232 [..............................] - ETA: 3:38 - loss: 0.2454 - Accuracy: 0.6786
  8/232 [>.............................] - ETA: 3:37 - loss: 0.2363 - Accuracy: 0.6719
  9/232 [>.............................] - ETA: 3:37 - loss: 0.2329 - Accuracy: 0.6389
 10/232 [>.............................] - ETA: 3:36 - loss: 0.2262 - Accuracy: 0.6375
 11/232 [>.............................] - ETA: 3:34 - loss: 0.2139 - Accuracy: 0.6705
 12/232 [>.............................] - ETA: 3:33 - loss: 0.2021 - Accuracy: 0.6875
 13/232 [>.............................] - ETA: 3:32 - loss: 0.1965 - Accuracy: 0.6923
 14/232 [>.............................] - ETA: 3:30 - loss: 0.1912 - Accuracy: 0.7054
 15/232 [>.............................] - ETA: 3:30 - loss: 0.1894 - Accuracy: 0.7000
 16/232 [=>............................] - ETA: 3:29 - loss: 0.1802 - Accuracy: 0.7188
 17/232 [=>............................] - ETA: 3:27 - loss: 0.1793 - Accuracy: 0.7132
 18/232 [=>............................] - ETA: 3:25 - loss: 0.1753 - Accuracy: 0.7153
 19/232 [=>............................] - ETA: 3:24 - loss: 0.1798 - Accuracy: 0.6974
 20/232 [=>............................] - ETA: 3:22 - loss: 0.1771 - Accuracy: 0.7063
 21/232 [=>............................] - ETA: 3:21 - loss: 0.1741 - Accuracy: 0.7083
 22/232 [=>............................] - ETA: 3:20 - loss: 0.1751 - Accuracy: 0.6989
 23/232 [=>............................] - ETA: 3:19 - loss: 0.1753 - Accuracy: 0.6902
 24/232 [==>...........................] - ETA: 3:17 - loss: 0.1702 - Accuracy: 0.7031
 25/232 [==>...........................] - ETA: 3:16 - loss: 0.1677 - Accuracy: 0.7050
 26/232 [==>...........................] - ETA: 3:15 - loss: 0.1656 - Accuracy: 0.7067
 27/232 [==>...........................] - ETA: 3:14 - loss: 0.1619 - Accuracy: 0.7130
 28/232 [==>...........................] - ETA: 3:12 - loss: 0.1616 - Accuracy: 0.7098
 29/232 [==>...........................] - ETA: 3:11 - loss: 0.1619 - Accuracy: 0.7069
 30/232 [==>...........................] - ETA: 3:10 - loss: 0.1644 - Accuracy: 0.6958
 31/232 [===>..........................] - ETA: 3:09 - loss: 0.1616 - Accuracy: 0.7016
 32/232 [===>..........................] - ETA: 3:08 - loss: 0.1605 - Accuracy: 0.7031
 33/232 [===>..........................] - ETA: 3:08 - loss: 0.1612 - Accuracy: 0.7008
 34/232 [===>..........................] - ETA: 3:06 - loss: 0.1588 - Accuracy: 0.7059
 35/232 [===>..........................] - ETA: 3:06 - loss: 0.1563 - Accuracy: 0.7143
 36/232 [===>..........................] - ETA: 3:05 - loss: 0.1550 - Accuracy: 0.7153
 37/232 [===>..........................] - ETA: 3:04 - loss: 0.1549 - Accuracy: 0.7128
 38/232 [===>..........................] - ETA: 3:03 - loss: 0.1535 - Accuracy: 0.7138
 39/232 [====>.........................] - ETA: 3:02 - loss: 0.1527 - Accuracy: 0.7147
 40/232 [====>.........................] - ETA: 3:01 - loss: 0.1533 - Accuracy: 0.7125
 41/232 [====>.........................] - ETA: 2:59 - loss: 0.1500 - Accuracy: 0.7195
 42/232 [====>.........................] - ETA: 2:59 - loss: 0.1524 - Accuracy: 0.7113
 43/232 [====>.........................] - ETA: 2:58 - loss: 0.1523 - Accuracy: 0.7122
 44/232 [====>.........................] - ETA: 2:57 - loss: 0.1526 - Accuracy: 0.7102
 45/232 [====>.........................] - ETA: 2:56 - loss: 0.1528 - Accuracy: 0.7083
 46/232 [====>.........................] - ETA: 2:55 - loss: 0.1509 - Accuracy: 0.7147
 47/232 [=====>........................] - ETA: 2:54 - loss: 0.1496 - Accuracy: 0.7181
 48/232 [=====>........................] - ETA: 2:53 - loss: 0.1507 - Accuracy: 0.7135
 49/232 [=====>........................] - ETA: 2:52 - loss: 0.1507 - Accuracy: 0.7117
 50/232 [=====>........................] - ETA: 2:51 - loss: 0.1508 - Accuracy: 0.7100
 51/232 [=====>........................] - ETA: 2:50 - loss: 0.1508 - Accuracy: 0.7083
 52/232 [=====>........................] - ETA: 2:49 - loss: 0.1522 - Accuracy: 0.7019
 53/232 [=====>........................] - ETA: 2:48 - loss: 0.1504 - Accuracy: 0.7075
 54/232 [=====>........................] - ETA: 2:47 - loss: 0.1500 - Accuracy: 0.7083
 55/232 [======>.......................] - ETA: 2:46 - loss: 0.1494 - Accuracy: 0.7091
 56/232 [======>.......................] - ETA: 2:45 - loss: 0.1487 - Accuracy: 0.7098
 57/232 [======>.......................] - ETA: 2:44 - loss: 0.1491 - Accuracy: 0.7061
 58/232 [======>.......................] - ETA: 2:44 - loss: 0.1485 - Accuracy: 0.7069
 59/232 [======>.......................] - ETA: 2:43 - loss: 0.1473 - Accuracy: 0.7097
 60/232 [======>.......................] - ETA: 2:42 - loss: 0.1467 - Accuracy: 0.7104
 61/232 [======>.......................] - ETA: 2:41 - loss: 0.1458 - Accuracy: 0.7111
 62/232 [=======>......................] - ETA: 2:40 - loss: 0.1447 - Accuracy: 0.7137
 63/232 [=======>......................] - ETA: 2:39 - loss: 0.1449 - Accuracy: 0.7123
 64/232 [=======>......................] - ETA: 2:38 - loss: 0.1446 - Accuracy: 0.7129
 65/232 [=======>......................] - ETA: 2:37 - loss: 0.1470 - Accuracy: 0.7038
 66/232 [=======>......................] - ETA: 2:36 - loss: 0.1468 - Accuracy: 0.7027
 67/232 [=======>......................] - ETA: 2:35 - loss: 0.1472 - Accuracy: 0.6996
 68/232 [=======>......................] - ETA: 2:34 - loss: 0.1472 - Accuracy: 0.6967
 69/232 [=======>......................] - ETA: 2:33 - loss: 0.1481 - Accuracy: 0.6957
 70/232 [========>.....................] - ETA: 2:32 - loss: 0.1482 - Accuracy: 0.7000
 71/232 [========>.....................] - ETA: 2:31 - loss: 0.1488 - Accuracy: 0.6972
 72/232 [========>.....................] - ETA: 2:30 - loss: 0.1481 - Accuracy: 0.6997
 73/232 [========>.....................] - ETA: 2:29 - loss: 0.1485 - Accuracy: 0.6969
 74/232 [========>.....................] - ETA: 2:28 - loss: 0.1496 - Accuracy: 0.6926
 75/232 [========>.....................] - ETA: 2:28 - loss: 0.1515 - Accuracy: 0.6867
 76/232 [========>.....................] - ETA: 2:26 - loss: 0.1505 - Accuracy: 0.6891
 77/232 [========>.....................] - ETA: 2:25 - loss: 0.1511 - Accuracy: 0.6867
 78/232 [=========>....................] - ETA: 2:24 - loss: 0.1506 - Accuracy: 0.6875
 79/232 [=========>....................] - ETA: 2:23 - loss: 0.1513 - Accuracy: 0.6835
 80/232 [=========>....................] - ETA: 2:22 - loss: 0.1516 - Accuracy: 0.6812
 81/232 [=========>....................] - ETA: 2:21 - loss: 0.1514 - Accuracy: 0.6806
 82/232 [=========>....................] - ETA: 2:20 - loss: 0.1513 - Accuracy: 0.6799
 83/232 [=========>....................] - ETA: 2:20 - loss: 0.1507 - Accuracy: 0.6822
 84/232 [=========>....................] - ETA: 2:19 - loss: 0.1512 - Accuracy: 0.6786
 85/232 [=========>....................] - ETA: 2:18 - loss: 0.1515 - Accuracy: 0.6750
 86/232 [==========>...................] - ETA: 2:17 - loss: 0.1512 - Accuracy: 0.6744
 87/232 [==========>...................] - ETA: 2:16 - loss: 0.1509 - Accuracy: 0.6753
 88/232 [==========>...................] - ETA: 2:15 - loss: 0.1506 - Accuracy: 0.6776
 89/232 [==========>...................] - ETA: 2:14 - loss: 0.1505 - Accuracy: 0.6784
 90/232 [==========>...................] - ETA: 2:13 - loss: 0.1501 - Accuracy: 0.6806
 91/232 [==========>...................] - ETA: 2:12 - loss: 0.1496 - Accuracy: 0.6827
 92/232 [==========>...................] - ETA: 2:11 - loss: 0.1501 - Accuracy: 0.6793
 93/232 [===========>..................] - ETA: 2:10 - loss: 0.1495 - Accuracy: 0.6815
 94/232 [===========>..................] - ETA: 2:09 - loss: 0.1497 - Accuracy: 0.6795
 95/232 [===========>..................] - ETA: 2:08 - loss: 0.1493 - Accuracy: 0.6803
 96/232 [===========>..................] - ETA: 2:07 - loss: 0.1489 - Accuracy: 0.6810
 97/232 [===========>..................] - ETA: 2:06 - loss: 0.1494 - Accuracy: 0.6791
 98/232 [===========>..................] - ETA: 2:05 - loss: 0.1486 - Accuracy: 0.6811
 99/232 [===========>..................] - ETA: 2:04 - loss: 0.1478 - Accuracy: 0.6831
100/232 [===========>..................] - ETA: 2:03 - loss: 0.1479 - Accuracy: 0.6825
101/232 [============>.................] - ETA: 2:02 - loss: 0.1483 - Accuracy: 0.6807
102/232 [============>.................] - ETA: 2:01 - loss: 0.1496 - Accuracy: 0.6765
103/232 [============>.................] - ETA: 2:01 - loss: 0.1492 - Accuracy: 0.6772
104/232 [============>.................] - ETA: 2:00 - loss: 0.1493 - Accuracy: 0.6767
105/232 [============>.................] - ETA: 1:59 - loss: 0.1497 - Accuracy: 0.6750
106/232 [============>.................] - ETA: 1:58 - loss: 0.1493 - Accuracy: 0.6757
107/232 [============>.................] - ETA: 1:57 - loss: 0.1503 - Accuracy: 0.6717
108/232 [============>.................] - ETA: 1:56 - loss: 0.1496 - Accuracy: 0.6736
109/232 [=============>................] - ETA: 1:55 - loss: 0.1496 - Accuracy: 0.6732
110/232 [=============>................] - ETA: 1:54 - loss: 0.1500 - Accuracy: 0.6716
111/232 [=============>................] - ETA: 1:53 - loss: 0.1496 - Accuracy: 0.6723
112/232 [=============>................] - ETA: 1:52 - loss: 0.1496 - Accuracy: 0.6719
113/232 [=============>................] - ETA: 1:51 - loss: 0.1496 - Accuracy: 0.6715
114/232 [=============>................] - ETA: 1:50 - loss: 0.1490 - Accuracy: 0.6732
115/232 [=============>................] - ETA: 1:49 - loss: 0.1490 - Accuracy: 0.6728
116/232 [==============>...............] - ETA: 1:48 - loss: 0.1486 - Accuracy: 0.6746
117/232 [==============>...............] - ETA: 1:47 - loss: 0.1484 - Accuracy: 0.6752
118/232 [==============>...............] - ETA: 1:47 - loss: 0.1480 - Accuracy: 0.6769
119/232 [==============>...............] - ETA: 1:46 - loss: 0.1472 - Accuracy: 0.6796
120/232 [==============>...............] - ETA: 1:45 - loss: 0.1469 - Accuracy: 0.6802
121/232 [==============>...............] - ETA: 1:44 - loss: 0.1471 - Accuracy: 0.6787
122/232 [==============>...............] - ETA: 1:43 - loss: 0.1468 - Accuracy: 0.6793
123/232 [==============>...............] - ETA: 1:42 - loss: 0.1462 - Accuracy: 0.6809
124/232 [===============>..............] - ETA: 1:41 - loss: 0.1459 - Accuracy: 0.6815
125/232 [===============>..............] - ETA: 1:40 - loss: 0.1456 - Accuracy: 0.6820
126/232 [===============>..............] - ETA: 1:39 - loss: 0.1450 - Accuracy: 0.6835
127/232 [===============>..............] - ETA: 1:38 - loss: 0.1452 - Accuracy: 0.6831
128/232 [===============>..............] - ETA: 1:37 - loss: 0.1450 - Accuracy: 0.6836
129/232 [===============>..............] - ETA: 1:36 - loss: 0.1454 - Accuracy: 0.6822
130/232 [===============>..............] - ETA: 1:35 - loss: 0.1452 - Accuracy: 0.6827
131/232 [===============>..............] - ETA: 1:34 - loss: 0.1449 - Accuracy: 0.6832
132/232 [================>.............] - ETA: 1:33 - loss: 0.1450 - Accuracy: 0.6828
133/232 [================>.............] - ETA: 1:32 - loss: 0.1448 - Accuracy: 0.6833
134/232 [================>.............] - ETA: 1:32 - loss: 0.1448 - Accuracy: 0.6828
135/232 [================>.............] - ETA: 1:31 - loss: 0.1446 - Accuracy: 0.6833
136/232 [================>.............] - ETA: 1:30 - loss: 0.1440 - Accuracy: 0.6847
137/232 [================>.............] - ETA: 1:29 - loss: 0.1435 - Accuracy: 0.6861
138/232 [================>.............] - ETA: 1:28 - loss: 0.1436 - Accuracy: 0.6857
139/232 [================>.............] - ETA: 1:27 - loss: 0.1433 - Accuracy: 0.6862
140/232 [=================>............] - ETA: 1:26 - loss: 0.1432 - Accuracy: 0.6866
141/232 [=================>............] - ETA: 1:25 - loss: 0.1427 - Accuracy: 0.6879
142/232 [=================>............] - ETA: 1:24 - loss: 0.1425 - Accuracy: 0.6884
143/232 [=================>............] - ETA: 1:23 - loss: 0.1423 - Accuracy: 0.6888
144/232 [=================>............] - ETA: 1:22 - loss: 0.1419 - Accuracy: 0.6901
145/232 [=================>............] - ETA: 1:21 - loss: 0.1419 - Accuracy: 0.6897
146/232 [=================>............] - ETA: 1:20 - loss: 0.1423 - Accuracy: 0.6884
147/232 [==================>...........] - ETA: 1:19 - loss: 0.1418 - Accuracy: 0.6896
148/232 [==================>...........] - ETA: 1:18 - loss: 0.1422 - Accuracy: 0.6883
149/232 [==================>...........] - ETA: 1:17 - loss: 0.1420 - Accuracy: 0.6888
150/232 [==================>...........] - ETA: 1:16 - loss: 0.1423 - Accuracy: 0.6875
151/232 [==================>...........] - ETA: 1:15 - loss: 0.1430 - Accuracy: 0.6846
152/232 [==================>...........] - ETA: 1:15 - loss: 0.1427 - Accuracy: 0.6859
153/232 [==================>...........] - ETA: 1:14 - loss: 0.1431 - Accuracy: 0.6838
154/232 [==================>...........] - ETA: 1:13 - loss: 0.1427 - Accuracy: 0.6851
155/232 [===================>..........] - ETA: 1:12 - loss: 0.1427 - Accuracy: 0.6847
156/232 [===================>..........] - ETA: 1:11 - loss: 0.1429 - Accuracy: 0.6835
157/232 [===================>..........] - ETA: 1:10 - loss: 0.1430 - Accuracy: 0.6823
158/232 [===================>..........] - ETA: 1:09 - loss: 0.1429 - Accuracy: 0.6835
159/232 [===================>..........] - ETA: 1:08 - loss: 0.1428 - Accuracy: 0.6832
160/232 [===================>..........] - ETA: 1:07 - loss: 0.1427 - Accuracy: 0.6836
161/232 [===================>..........] - ETA: 1:06 - loss: 0.1426 - Accuracy: 0.6840
162/232 [===================>..........] - ETA: 1:05 - loss: 0.1425 - Accuracy: 0.6844
163/232 [====================>.........] - ETA: 1:04 - loss: 0.1426 - Accuracy: 0.6840
164/232 [====================>.........] - ETA: 1:03 - loss: 0.1423 - Accuracy: 0.6845
165/232 [====================>.........] - ETA: 1:02 - loss: 0.1427 - Accuracy: 0.6826
166/232 [====================>.........] - ETA: 1:01 - loss: 0.1425 - Accuracy: 0.6830
167/232 [====================>.........] - ETA: 1:00 - loss: 0.1420 - Accuracy: 0.6849
168/232 [====================>.........] - ETA: 59s - loss: 0.1420 - Accuracy: 0.6845 
169/232 [====================>.........] - ETA: 59s - loss: 0.1428 - Accuracy: 0.6812
170/232 [====================>.........] - ETA: 58s - loss: 0.1433 - Accuracy: 0.6794
171/232 [=====================>........] - ETA: 57s - loss: 0.1429 - Accuracy: 0.6806
172/232 [=====================>........] - ETA: 56s - loss: 0.1428 - Accuracy: 0.6802
173/232 [=====================>........] - ETA: 55s - loss: 0.1427 - Accuracy: 0.6806
174/232 [=====================>........] - ETA: 54s - loss: 0.1433 - Accuracy: 0.6782
175/232 [=====================>........] - ETA: 53s - loss: 0.1430 - Accuracy: 0.6793
176/232 [=====================>........] - ETA: 52s - loss: 0.1429 - Accuracy: 0.6790
177/232 [=====================>........] - ETA: 51s - loss: 0.1430 - Accuracy: 0.6787
178/232 [======================>.......] - ETA: 50s - loss: 0.1429 - Accuracy: 0.6791
179/232 [======================>.......] - ETA: 49s - loss: 0.1429 - Accuracy: 0.6788
180/232 [======================>.......] - ETA: 48s - loss: 0.1429 - Accuracy: 0.6785
181/232 [======================>.......] - ETA: 47s - loss: 0.1427 - Accuracy: 0.6789
182/232 [======================>.......] - ETA: 46s - loss: 0.1426 - Accuracy: 0.6793
183/232 [======================>.......] - ETA: 45s - loss: 0.1426 - Accuracy: 0.6790
184/232 [======================>.......] - ETA: 44s - loss: 0.1428 - Accuracy: 0.6780
185/232 [======================>.......] - ETA: 44s - loss: 0.1429 - Accuracy: 0.6777
186/232 [=======================>......] - ETA: 43s - loss: 0.1432 - Accuracy: 0.6761
187/232 [=======================>......] - ETA: 42s - loss: 0.1431 - Accuracy: 0.6765
188/232 [=======================>......] - ETA: 41s - loss: 0.1434 - Accuracy: 0.6749
189/232 [=======================>......] - ETA: 40s - loss: 0.1431 - Accuracy: 0.6759
190/232 [=======================>......] - ETA: 39s - loss: 0.1431 - Accuracy: 0.6757
191/232 [=======================>......] - ETA: 38s - loss: 0.1429 - Accuracy: 0.6767
192/232 [=======================>......] - ETA: 37s - loss: 0.1426 - Accuracy: 0.6777
193/232 [=======================>......] - ETA: 36s - loss: 0.1423 - Accuracy: 0.6794
194/232 [========================>.....] - ETA: 35s - loss: 0.1419 - Accuracy: 0.6811
195/232 [========================>.....] - ETA: 34s - loss: 0.1419 - Accuracy: 0.6808
196/232 [========================>.....] - ETA: 33s - loss: 0.1418 - Accuracy: 0.6811
197/232 [========================>.....] - ETA: 32s - loss: 0.1414 - Accuracy: 0.6821
198/232 [========================>.....] - ETA: 31s - loss: 0.1413 - Accuracy: 0.6824
199/232 [========================>.....] - ETA: 30s - loss: 0.1412 - Accuracy: 0.6828
200/232 [========================>.....] - ETA: 30s - loss: 0.1413 - Accuracy: 0.6825
201/232 [========================>.....] - ETA: 29s - loss: 0.1416 - Accuracy: 0.6816
202/232 [=========================>....] - ETA: 28s - loss: 0.1415 - Accuracy: 0.6819
203/232 [=========================>....] - ETA: 27s - loss: 0.1414 - Accuracy: 0.6823
204/232 [=========================>....] - ETA: 26s - loss: 0.1411 - Accuracy: 0.6832
205/232 [=========================>....] - ETA: 25s - loss: 0.1409 - Accuracy: 0.6835
206/232 [=========================>....] - ETA: 24s - loss: 0.1414 - Accuracy: 0.6820
207/232 [=========================>....] - ETA: 23s - loss: 0.1415 - Accuracy: 0.6818
208/232 [=========================>....] - ETA: 22s - loss: 0.1414 - Accuracy: 0.6821
209/232 [==========================>...] - ETA: 21s - loss: 0.1409 - Accuracy: 0.6836
210/232 [==========================>...] - ETA: 20s - loss: 0.1407 - Accuracy: 0.6839
211/232 [==========================>...] - ETA: 19s - loss: 0.1405 - Accuracy: 0.6848
212/232 [==========================>...] - ETA: 18s - loss: 0.1400 - Accuracy: 0.6863
213/232 [==========================>...] - ETA: 17s - loss: 0.1400 - Accuracy: 0.6860
214/232 [==========================>...] - ETA: 16s - loss: 0.1401 - Accuracy: 0.6857
215/232 [==========================>...] - ETA: 15s - loss: 0.1403 - Accuracy: 0.6849
216/232 [==========================>...] - ETA: 14s - loss: 0.1400 - Accuracy: 0.6858
217/232 [===========================>..] - ETA: 14s - loss: 0.1404 - Accuracy: 0.6843
218/232 [===========================>..] - ETA: 13s - loss: 0.1402 - Accuracy: 0.6846
219/232 [===========================>..] - ETA: 12s - loss: 0.1403 - Accuracy: 0.6844
220/232 [===========================>..] - ETA: 11s - loss: 0.1403 - Accuracy: 0.6841
221/232 [===========================>..] - ETA: 10s - loss: 0.1403 - Accuracy: 0.6844
222/232 [===========================>..] - ETA: 9s - loss: 0.1407 - Accuracy: 0.6824 
223/232 [===========================>..] - ETA: 8s - loss: 0.1406 - Accuracy: 0.6822
224/232 [===========================>..] - ETA: 7s - loss: 0.1404 - Accuracy: 0.6830
225/232 [============================>.] - ETA: 6s - loss: 0.1404 - Accuracy: 0.6828
226/232 [============================>.] - ETA: 5s - loss: 0.1404 - Accuracy: 0.6825
227/232 [============================>.] - ETA: 4s - loss: 0.1404 - Accuracy: 0.6823
228/232 [============================>.] - ETA: 3s - loss: 0.1405 - Accuracy: 0.6815
229/232 [============================>.] - ETA: 2s - loss: 0.1406 - Accuracy: 0.6807
230/232 [============================>.] - ETA: 1s - loss: 0.1408 - Accuracy: 0.6799
231/232 [============================>.] - ETA: 0s - loss: 0.1409 - Accuracy: 0.6802
232/232 [==============================] - ETA: 0s - loss: 0.1408 - Accuracy: 0.68002021-12-08 03:38:06.835804: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_819036"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\021FlatMapDataset:53"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 03:38:06.891410: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.893979: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.907358: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.910977: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.924507: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.927559: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.939629: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:06.941576: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.108656: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.111741: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.125587: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.128333: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.129748: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.143568: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.146326: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.159566: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.162239: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.163579: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.177174: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.179804: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.193266: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.195900: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.197273: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.210907: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.213517: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.226549: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.229243: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.230606: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.244240: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.246853: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.259984: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.262666: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.264019: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.277687: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.280270: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.293432: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.296074: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.297418: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.310926: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.313478: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.326484: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.329043: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.330331: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.344024: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.346536: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.359540: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.362145: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:38:07.363468: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 03:39:50.853125: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 19176/53881 nodes to float16 precision using 992 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 03:40:12.677238: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/50369 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)

232/232 [==============================] - 924s 2s/step - loss: 0.1408 - Accuracy: 0.6800 - val_loss: 0.9255 - val_Accuracy: 0.6700
Epoch 2/5
9 root error(s) found.
  (0) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_7/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Cast_1/_775]]
  (1) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_5/tf_longformer_for_sequence_classification/longformer/embeddings/Cumsum/_524]]
  (2) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_4/tf_longformer_for_sequence_classification/longformer/Greater/_658]]
  (3) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_3/tf_longformer_for_sequence_classification/longformer/Abs/_797]]
  (4) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_2/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Tile_2/_2065]]
  (5) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_1/tf_longformer_for_sequence_classification/longformer/Any/_741]]
  (6) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[div_no_nan_1/ReadVariableOp_2/_12654]]
  (7) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[cond/output/_50/_393]]
  (8) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_817844]

Errors may have originated from an input operation.
Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Function call stack:
train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function



[2021-12-08T03:40:38.420250] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
2 items cleaning up...
Cleanup took 0.1139984130859375 seconds
Traceback (most recent call last):
  File "training.py", line 20, in <module>
    classifier.fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/training.py", line 27, in fit
    self._fit()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 50, in _fit
    self._fit_model()
  File "/opt/miniconda/lib/python3.7/site-packages/pml/training/_mixins.py", line 418, in _fit_model
    class_weight=self.class_weight_
  File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/miniconda/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 9 root error(s) found.
  (0) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_7/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Cast_1/_775]]
  (1) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_5/tf_longformer_for_sequence_classification/longformer/embeddings/Cumsum/_524]]
  (2) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_4/tf_longformer_for_sequence_classification/longformer/Greater/_658]]
  (3) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_3/tf_longformer_for_sequence_classification/longformer/Abs/_797]]
  (4) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_2/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Tile_2/_2065]]
  (5) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[replica_1/tf_longformer_for_sequence_classification/longformer/Any/_741]]
  (6) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[div_no_nan_1/ReadVariableOp_2/_12654]]
  (7) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
	 [[cond/output/_50/_393]]
  (8) INVALID_ARGUMENT:  Requires start <= limit when delta > 0: 0/-2147483648
	 [[node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2
 (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1214)
]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_817844]

Errors may have originated from an input operation.
Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Input Source operations connected to node replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2:
In[0] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/start:	
In[1] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/Max (defined at /opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1208)	
In[2] replica_6/tf_longformer_for_sequence_classification/longformer/encoder/layer_._11/attention/self/range_2/delta:

Operation defined at: (most recent call last)
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 890, in _bootstrap
>>>     self._bootstrap_inner()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
>>>     self.run()
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/training.py", line 860, in run_step
>>>     outputs = model.train_step(data)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 796, in train_step
>>>     y_pred = self(x, training=True)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 2424, in call
>>>     outputs = self.longformer(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1733, in call
>>>     encoder_outputs = self.encoder(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1550, in call
>>>     for idx, layer_module in enumerate(self.layer):
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1555, in call
>>>     layer_outputs = layer_module(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1513, in call
>>>     attention_outputs = self.attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1485, in call
>>>     self_outputs = self.self_attention(
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/engine/base_layer.py", line 1083, in __call__
>>>     outputs = call_fn(inputs, *args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
>>>     return fn(*args, **kwargs)
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 778, in call
>>>     (
>>> 
>>>   File "/opt/miniconda/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py", line 1214, in _get_global_attn_indices
>>>     is_local_index_global_attn = tf.range(max_num_global_attn_indices) < tf.expand_dims(
>>> 

Function call stack:
train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function -> train_function


[2021-12-08T03:40:38.708621] Finished context manager injector with Exception.
```

Any thoughts @Rocketknight1? I was able to get it to successfully train for 2 epochs if I limit the visible devices to 2 GPUs relevant logs below

```
[5 rows x 5 columns]
2021-12-08 21:15:29.463814: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-08 21:15:30.377640: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-12-08 21:15:30.377729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 38414 MB memory:  -> device: 0, name: A100-SXM4-40GB, pci bus id: 0001:00:00.0, compute capability: 8.0
2021-12-08 21:15:30.380280: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-12-08 21:15:30.380309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 38414 MB memory:  -> device: 1, name: A100-SXM4-40GB, pci bus id: 0002:00:00.0, compute capability: 8.0

Downloading:   0%|          | 0.00/694 [00:00<?, ?B/s]
Downloading: 100%|██████████| 694/694 [00:00<00:00, 566kB/s]

Downloading:   0%|          | 0.00/729M [00:00<?, ?B/s]
Downloading:   1%|          | 6.10M/729M [00:00<00:11, 63.9MB/s]
Downloading:   2%|▏         | 13.0M/729M [00:00<00:10, 69.1MB/s]
Downloading:   3%|▎         | 19.9M/729M [00:00<00:10, 70.3MB/s]
Downloading:   4%|▎         | 26.7M/729M [00:00<00:10, 71.0MB/s]
Downloading:   5%|▍         | 33.6M/729M [00:00<00:10, 71.4MB/s]
Downloading:   6%|▌         | 40.5M/729M [00:00<00:10, 71.6MB/s]
Downloading:   6%|▋         | 47.3M/729M [00:00<00:10, 71.3MB/s]
Downloading:   7%|▋         | 54.1M/729M [00:00<00:09, 71.4MB/s]
Downloading:   8%|▊         | 61.1M/729M [00:00<00:09, 71.8MB/s]
Downloading:   9%|▉         | 68.0M/729M [00:01<00:09, 72.0MB/s]
Downloading:  10%|█         | 75.0M/729M [00:01<00:09, 72.6MB/s]
Downloading:  11%|█         | 82.0M/729M [00:01<00:09, 72.7MB/s]
Downloading:  12%|█▏        | 88.9M/729M [00:01<00:09, 72.7MB/s]
Downloading:  13%|█▎        | 95.9M/729M [00:01<00:09, 72.7MB/s]
Downloading:  14%|█▍        | 103M/729M [00:01<00:09, 72.7MB/s] 
Downloading:  15%|█▌        | 110M/729M [00:01<00:08, 72.9MB/s]
Downloading:  16%|█▌        | 117M/729M [00:01<00:08, 72.9MB/s]
Downloading:  17%|█▋        | 124M/729M [00:01<00:08, 72.9MB/s]
Downloading:  18%|█▊        | 131M/729M [00:01<00:08, 72.9MB/s]
Downloading:  19%|█▉        | 138M/729M [00:02<00:08, 72.7MB/s]
Downloading:  20%|█▉        | 145M/729M [00:02<00:08, 72.7MB/s]
Downloading:  21%|██        | 151M/729M [00:02<00:08, 72.7MB/s]
Downloading:  22%|██▏       | 158M/729M [00:02<00:08, 72.9MB/s]
Downloading:  23%|██▎       | 165M/729M [00:02<00:08, 72.8MB/s]
Downloading:  24%|██▎       | 172M/729M [00:02<00:07, 73.0MB/s]
Downloading:  25%|██▍       | 179M/729M [00:02<00:07, 72.8MB/s]
Downloading:  26%|██▌       | 186M/729M [00:02<00:07, 72.7MB/s]
Downloading:  27%|██▋       | 193M/729M [00:02<00:07, 72.7MB/s]
Downloading:  27%|██▋       | 200M/729M [00:02<00:07, 72.7MB/s]
Downloading:  28%|██▊       | 207M/729M [00:03<00:07, 72.7MB/s]
Downloading:  29%|██▉       | 214M/729M [00:03<00:07, 72.7MB/s]
Downloading:  30%|███       | 221M/729M [00:03<00:07, 72.7MB/s]
Downloading:  31%|███▏      | 228M/729M [00:03<00:07, 72.7MB/s]
Downloading:  32%|███▏      | 235M/729M [00:03<00:07, 73.0MB/s]
Downloading:  33%|███▎      | 242M/729M [00:03<00:07, 72.8MB/s]
Downloading:  34%|███▍      | 249M/729M [00:03<00:06, 72.9MB/s]
Downloading:  35%|███▌      | 256M/729M [00:03<00:06, 72.9MB/s]
Downloading:  36%|███▌      | 263M/729M [00:03<00:06, 72.7MB/s]
Downloading:  37%|███▋      | 270M/729M [00:03<00:06, 72.6MB/s]
Downloading:  38%|███▊      | 277M/729M [00:04<00:06, 72.4MB/s]
Downloading:  39%|███▉      | 284M/729M [00:04<00:06, 72.5MB/s]
Downloading:  40%|███▉      | 291M/729M [00:04<00:06, 72.7MB/s]
Downloading:  41%|████      | 298M/729M [00:04<00:06, 72.6MB/s]
Downloading:  42%|████▏     | 304M/729M [00:04<00:06, 72.5MB/s]
Downloading:  43%|████▎     | 311M/729M [00:04<00:06, 72.5MB/s]
Downloading:  44%|████▎     | 318M/729M [00:04<00:05, 72.5MB/s]
Downloading:  45%|████▍     | 325M/729M [00:04<00:05, 72.7MB/s]
Downloading:  46%|████▌     | 332M/729M [00:04<00:05, 72.6MB/s]
Downloading:  47%|████▋     | 339M/729M [00:04<00:05, 72.7MB/s]
Downloading:  47%|████▋     | 346M/729M [00:05<00:05, 72.5MB/s]
Downloading:  48%|████▊     | 353M/729M [00:05<00:05, 72.5MB/s]
Downloading:  49%|████▉     | 360M/729M [00:05<00:05, 72.5MB/s]
Downloading:  50%|█████     | 367M/729M [00:05<00:05, 72.6MB/s]
Downloading:  51%|█████▏    | 374M/729M [00:05<00:05, 72.6MB/s]
Downloading:  52%|█████▏    | 381M/729M [00:05<00:05, 72.3MB/s]
Downloading:  53%|█████▎    | 388M/729M [00:05<00:04, 72.0MB/s]
Downloading:  54%|█████▍    | 395M/729M [00:05<00:04, 72.2MB/s]
Downloading:  55%|█████▌    | 402M/729M [00:05<00:04, 72.4MB/s]
Downloading:  56%|█████▌    | 408M/729M [00:05<00:04, 72.5MB/s]
Downloading:  57%|█████▋    | 415M/729M [00:06<00:04, 72.5MB/s]
Downloading:  58%|█████▊    | 422M/729M [00:06<00:04, 72.6MB/s]
Downloading:  59%|█████▉    | 429M/729M [00:06<00:04, 72.8MB/s]
Downloading:  60%|█████▉    | 436M/729M [00:06<00:04, 72.8MB/s]
Downloading:  61%|██████    | 443M/729M [00:06<00:04, 73.0MB/s]
Downloading:  62%|██████▏   | 450M/729M [00:06<00:04, 72.9MB/s]
Downloading:  63%|██████▎   | 457M/729M [00:06<00:03, 72.9MB/s]
Downloading:  64%|██████▎   | 464M/729M [00:06<00:03, 72.7MB/s]
Downloading:  65%|██████▍   | 471M/729M [00:06<00:03, 72.5MB/s]
Downloading:  66%|██████▌   | 478M/729M [00:06<00:03, 72.7MB/s]
Downloading:  67%|██████▋   | 485M/729M [00:07<00:03, 72.7MB/s]
Downloading:  67%|██████▋   | 492M/729M [00:07<00:03, 72.7MB/s]
Downloading:  68%|██████▊   | 499M/729M [00:07<00:03, 72.6MB/s]
Downloading:  69%|██████▉   | 506M/729M [00:07<00:03, 72.6MB/s]
Downloading:  70%|███████   | 513M/729M [00:07<00:03, 72.9MB/s]
Downloading:  71%|███████▏  | 520M/729M [00:07<00:03, 72.8MB/s]
Downloading:  72%|███████▏  | 527M/729M [00:07<00:02, 72.9MB/s]
Downloading:  73%|███████▎  | 534M/729M [00:07<00:02, 72.8MB/s]
Downloading:  74%|███████▍  | 541M/729M [00:07<00:02, 72.5MB/s]
Downloading:  75%|███████▌  | 548M/729M [00:07<00:02, 72.5MB/s]
Downloading:  76%|███████▌  | 554M/729M [00:08<00:02, 72.1MB/s]
Downloading:  77%|███████▋  | 561M/729M [00:08<00:02, 72.4MB/s]
Downloading:  78%|███████▊  | 568M/729M [00:08<00:02, 72.5MB/s]
Downloading:  79%|███████▉  | 575M/729M [00:08<00:02, 72.6MB/s]
Downloading:  80%|███████▉  | 582M/729M [00:08<00:02, 72.7MB/s]
Downloading:  81%|████████  | 589M/729M [00:08<00:02, 72.7MB/s]
Downloading:  82%|████████▏ | 596M/729M [00:08<00:01, 72.6MB/s]
Downloading:  83%|████████▎ | 603M/729M [00:08<00:01, 72.6MB/s]
Downloading:  84%|████████▎ | 610M/729M [00:08<00:01, 72.6MB/s]
Downloading:  85%|████████▍ | 617M/729M [00:08<00:01, 72.9MB/s]
Downloading:  86%|████████▌ | 624M/729M [00:09<00:01, 72.7MB/s]
Downloading:  87%|████████▋ | 631M/729M [00:09<00:01, 72.6MB/s]
Downloading:  87%|████████▋ | 638M/729M [00:09<00:01, 72.7MB/s]
Downloading:  88%|████████▊ | 645M/729M [00:09<00:01, 72.6MB/s]
Downloading:  89%|████████▉ | 652M/729M [00:09<00:01, 72.8MB/s]
Downloading:  90%|█████████ | 659M/729M [00:09<00:01, 72.6MB/s]
Downloading:  91%|█████████▏| 666M/729M [00:09<00:00, 72.8MB/s]
Downloading:  92%|█████████▏| 673M/729M [00:09<00:00, 72.7MB/s]
Downloading:  93%|█████████▎| 680M/729M [00:09<00:00, 72.7MB/s]
Downloading:  94%|█████████▍| 687M/729M [00:09<00:00, 72.7MB/s]
Downloading:  95%|█████████▌| 693M/729M [00:10<00:00, 72.6MB/s]
Downloading:  96%|█████████▌| 700M/729M [00:10<00:00, 72.7MB/s]
Downloading:  97%|█████████▋| 707M/729M [00:10<00:00, 72.6MB/s]
Downloading:  98%|█████████▊| 714M/729M [00:10<00:00, 72.6MB/s]
Downloading:  99%|█████████▉| 721M/729M [00:10<00:00, 72.5MB/s]
Downloading: 100%|█████████▉| 728M/729M [00:10<00:00, 72.6MB/s]
Downloading: 100%|██████████| 729M/729M [00:10<00:00, 72.5MB/s]
2021-12-08 21:15:42.678981: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
Some layers from the model checkpoint at allenai/longformer-base-4096 were not used when initializing TFLongformerForSequenceClassification: ['lm_head']
- This IS expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFLongformerForSequenceClassification were not initialized from the model checkpoint at allenai/longformer-base-4096 and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some layers from the model checkpoint at allenai/longformer-base-4096 were not used when initializing TFLongformerForSequenceClassification: ['lm_head']
- This IS expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFLongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some layers of TFLongformerForSequenceClassification were not initialized from the model checkpoint at allenai/longformer-base-4096 and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2021-12-08 21:15:47.914719: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_59118"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\020FlatMapDataset:4"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 21:15:47.967533: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.969043: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.978416: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.980483: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.985689: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.987135: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.991900: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.993490: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.997905: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:47.999031: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.038998: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.040759: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.048322: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.049956: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.050807: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.056971: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.058550: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.064085: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.065707: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:15:48.066561: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
Epoch 1/2
WARNING:tensorflow:Efficient allreduce is not supported for 3 IndexedSlices
WARNING:tensorflow:Efficient allreduce is not supported for 3 IndexedSlices
2021-12-08 21:17:31.182628: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 10772/50331 nodes to float16 precision using 1486 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 21:17:49.083818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/42321 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO Bootstrap : Using eth0:10.0.0.5<0>
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO NET/IB : Using [0]mlx5_ib4:1/IB [1]mlx5_ib2:1/IB [2]mlx5_ib0:1/IB [3]mlx5_ib7:1/IB [4]mlx5_ib5:1/IB [5]mlx5_ib3:1/IB [6]mlx5_ib1:1/IB [7]mlx5_ib6:1/IB ; OOB eth0:10.0.0.5<0>
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1172 [1] NCCL INFO Using network IB
NCCL version 2.8.3+cudaCUDA_MAJOR.CUDA_MINOR
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO NCCL_IB_TIMEOUT set by environment to 16.
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/microsoft/ndv4-topo.xml
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/microsoft/ndv4-topo.xml
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Loading unnamed topology
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Loading unnamed topology
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO === System : maxWidth 252.0 totalWidth 252.0 ===
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/0 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF010
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - GPU/100000 (0)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10100000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - GPU/200000 (1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10200000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/1 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF020
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10300000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10400000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/2 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF030
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10500000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10600000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO CPU/3 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO + PCI[24.0] - PCI/FFFFFF040
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10700000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO               + PCI[24.0] - NIC/10800000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO ==========================================
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO GPU/100000 :GPU/100000 (0/5000.000000/LOC) GPU/200000 (2/252.000000/NVL) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO GPU/200000 :GPU/100000 (2/252.000000/NVL) GPU/200000 (0/5000.000000/LOC) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 4, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 42.000000/42.000000, type NVL/PIX, sameChannels 0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  6 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  7 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  8 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO  9 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 10 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 11 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Attribute coll of node net not found
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO === System : maxWidth 252.0 totalWidth 252.0 ===
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/0 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF010
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - GPU/100000 (0)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10100000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - GPU/200000 (1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO                             + NVL[252.0] - NVS/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10200000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/1 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF020
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10300000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10400000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/2 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/3
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF030
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10500000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10600000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO CPU/3 (1/2/-1)
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + SYS[5000.0] - CPU/2
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO + PCI[24.0] - PCI/FFFFFF040
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10700000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO               + PCI[24.0] - NIC/10800000
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO ==========================================
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO GPU/100000 :GPU/100000 (0/5000.000000/LOC) GPU/200000 (2/252.000000/NVL) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO GPU/200000 :GPU/100000 (2/252.000000/NVL) GPU/200000 (0/5000.000000/LOC) CPU/0 (2/24.000000/PHB) CPU/1 (3/24.000000/SYS) CPU/2 (3/24.000000/SYS) CPU/3 (3/24.000000/SYS) 
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 21.000000/21.000000, type NVL/PIX, sameChannels 1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 12, speed 42.000000/42.000000, type NVL/PIX, sameChannels 0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  0 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  1 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  2 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  3 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  4 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  5 : GPU/0 GPU/1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  6 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  7 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  8 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO  9 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 10 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 11 : GPU/1 GPU/0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 12 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 1 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 13 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 2 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 14 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 3 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 15 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 4 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 16 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 5 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 17 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 6 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 18 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 7 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 19 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 8 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 20 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 9 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] -1/-1/-1->1->0 [7] -1/-1/-1->1->0 [8] -1/-1/-1->1->0 [9] -1/-1/-1->1->0 [10] -1/-1/-1->1->0 [11] -1/-1/-1->1->0 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] -1/-1/-1->1->0 [19] -1/-1/-1->1->0 [20] -1/-1/-1->1->0 [21] -1/-1/-1->1->0 [22] -1/-1/-1->1->0 [23] -1/-1/-1->1->0
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 21 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 10 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 22 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 11 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Tree 23 : -1 -> 0 -> 1/-1/-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 00/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 01/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 02/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 03/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 04/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 05/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 06/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 07/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 08/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 09/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 10/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 11/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 12/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 13/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 14/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 15/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 16/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 17/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 18/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 19/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 20/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 21/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 22/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 23/24 :    0   1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 00 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 00 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 01 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 01 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 02 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 02 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 03 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 03 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 04 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 04 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 05 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 05 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 06 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 06 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 07 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 07 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 08 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 08 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 09 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 09 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 10 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 10 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 11 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 11 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 12 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 12 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 13 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 13 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 14 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 14 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 15 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 15 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 16 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 16 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 17 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 17 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 18 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 18 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 19 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 19 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 20 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 20 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 21 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 21 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 22 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 22 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Channel 23 : 0[100000] -> 1[200000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Channel 23 : 1[200000] -> 0[100000] via P2P/direct pointer/read
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Connected all rings
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO Connected all trees
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Connected all rings
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO Connected all trees
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO 24 coll channels, 32 p2p channels, 32 p2p channels per peer
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1337 [0] NCCL INFO comm 0x7fca28002e00 rank 0 nranks 2 cudaDev 0 busId 100000 - Init COMPLETE
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1338 [1] NCCL INFO comm 0x7fc494002dd0 rank 1 nranks 2 cudaDev 1 busId 200000 - Init COMPLETE
0bfd6a2c57ad4d419fa42cd7f69f5160000001:160:1326 [0] NCCL INFO Launch mode Group/CGMD

  1/232 [..............................] - ETA: 8:43:48 - loss: 0.5153 - Accuracy: 0.8750
  2/232 [..............................] - ETA: 6:34 - loss: 0.4137 - Accuracy: 0.8125   
  3/232 [..............................] - ETA: 6:37 - loss: 0.3161 - Accuracy: 0.6250
  4/232 [..............................] - ETA: 6:35 - loss: 0.2721 - Accuracy: 0.6250
  5/232 [..............................] - ETA: 6:38 - loss: 0.2351 - Accuracy: 0.6750
  6/232 [..............................] - ETA: 6:31 - loss: 0.2066 - Accuracy: 0.7083
  7/232 [..............................] - ETA: 6:28 - loss: 0.1930 - Accuracy: 0.6786
  8/232 [>.............................] - ETA: 6:26 - loss: 0.1812 - Accuracy: 0.6719
  9/232 [>.............................] - ETA: 6:26 - loss: 0.1753 - Accuracy: 0.6389
 10/232 [>.............................] - ETA: 6:23 - loss: 0.1670 - Accuracy: 0.6375
 11/232 [>.............................] - ETA: 6:20 - loss: 0.1559 - Accuracy: 0.6705
 12/232 [>.............................] - ETA: 6:18 - loss: 0.1467 - Accuracy: 0.6875
 13/232 [>.............................] - ETA: 6:16 - loss: 0.1405 - Accuracy: 0.6923
 14/232 [>.............................] - ETA: 6:16 - loss: 0.1339 - Accuracy: 0.7054
 15/232 [>.............................] - ETA: 6:14 - loss: 0.1317 - Accuracy: 0.7000
 16/232 [=>............................] - ETA: 6:11 - loss: 0.1250 - Accuracy: 0.7188
 17/232 [=>............................] - ETA: 6:10 - loss: 0.1233 - Accuracy: 0.7132
 18/232 [=>............................] - ETA: 6:07 - loss: 0.1199 - Accuracy: 0.7153
 19/232 [=>............................] - ETA: 6:05 - loss: 0.1210 - Accuracy: 0.6974
 20/232 [=>............................] - ETA: 6:04 - loss: 0.1190 - Accuracy: 0.7063
 21/232 [=>............................] - ETA: 6:01 - loss: 0.1176 - Accuracy: 0.7083
 22/232 [=>............................] - ETA: 5:59 - loss: 0.1176 - Accuracy: 0.6989
 23/232 [=>............................] - ETA: 5:57 - loss: 0.1179 - Accuracy: 0.6902
 24/232 [==>...........................] - ETA: 5:56 - loss: 0.1141 - Accuracy: 0.7031
 25/232 [==>...........................] - ETA: 5:54 - loss: 0.1122 - Accuracy: 0.7050
 26/232 [==>...........................] - ETA: 5:52 - loss: 0.1104 - Accuracy: 0.7067
 27/232 [==>...........................] - ETA: 5:49 - loss: 0.1092 - Accuracy: 0.7130
 28/232 [==>...........................] - ETA: 5:48 - loss: 0.1089 - Accuracy: 0.7098
 29/232 [==>...........................] - ETA: 5:46 - loss: 0.1085 - Accuracy: 0.7069
 30/232 [==>...........................] - ETA: 5:44 - loss: 0.1098 - Accuracy: 0.6958
 31/232 [===>..........................] - ETA: 5:42 - loss: 0.1080 - Accuracy: 0.7016
 32/232 [===>..........................] - ETA: 5:40 - loss: 0.1069 - Accuracy: 0.7031
 33/232 [===>..........................] - ETA: 5:38 - loss: 0.1065 - Accuracy: 0.7008
 34/232 [===>..........................] - ETA: 5:36 - loss: 0.1048 - Accuracy: 0.7059
 35/232 [===>..........................] - ETA: 5:35 - loss: 0.1026 - Accuracy: 0.7143
 36/232 [===>..........................] - ETA: 5:33 - loss: 0.1017 - Accuracy: 0.7153
 37/232 [===>..........................] - ETA: 5:32 - loss: 0.1016 - Accuracy: 0.7128
 38/232 [===>..........................] - ETA: 5:30 - loss: 0.1007 - Accuracy: 0.7138
 39/232 [====>.........................] - ETA: 5:28 - loss: 0.0998 - Accuracy: 0.7147
 40/232 [====>.........................] - ETA: 5:26 - loss: 0.0997 - Accuracy: 0.7125
 41/232 [====>.........................] - ETA: 5:25 - loss: 0.0981 - Accuracy: 0.7195
 42/232 [====>.........................] - ETA: 5:23 - loss: 0.0988 - Accuracy: 0.7113
 43/232 [====>.........................] - ETA: 5:22 - loss: 0.0982 - Accuracy: 0.7122
 44/232 [====>.........................] - ETA: 5:20 - loss: 0.0980 - Accuracy: 0.7102
 45/232 [====>.........................] - ETA: 5:18 - loss: 0.0979 - Accuracy: 0.7083
 46/232 [====>.........................] - ETA: 5:16 - loss: 0.0966 - Accuracy: 0.7147
 47/232 [=====>........................] - ETA: 5:15 - loss: 0.0955 - Accuracy: 0.7181
 48/232 [=====>........................] - ETA: 5:13 - loss: 0.0958 - Accuracy: 0.7135
 49/232 [=====>........................] - ETA: 5:11 - loss: 0.0959 - Accuracy: 0.7117
 50/232 [=====>........................] - ETA: 5:10 - loss: 0.0959 - Accuracy: 0.7100
 51/232 [=====>........................] - ETA: 5:09 - loss: 0.0959 - Accuracy: 0.7083
 52/232 [=====>........................] - ETA: 5:07 - loss: 0.0967 - Accuracy: 0.7019
 53/232 [=====>........................] - ETA: 5:05 - loss: 0.0956 - Accuracy: 0.7075
 54/232 [=====>........................] - ETA: 5:03 - loss: 0.0954 - Accuracy: 0.7083
 55/232 [======>.......................] - ETA: 5:02 - loss: 0.0949 - Accuracy: 0.7091
 56/232 [======>.......................] - ETA: 5:00 - loss: 0.0945 - Accuracy: 0.7098
 57/232 [======>.......................] - ETA: 4:59 - loss: 0.0948 - Accuracy: 0.7061
 58/232 [======>.......................] - ETA: 4:57 - loss: 0.0944 - Accuracy: 0.7069
 59/232 [======>.......................] - ETA: 4:55 - loss: 0.0936 - Accuracy: 0.7097
 60/232 [======>.......................] - ETA: 4:54 - loss: 0.0932 - Accuracy: 0.7104
 61/232 [======>.......................] - ETA: 4:52 - loss: 0.0929 - Accuracy: 0.7111
 62/232 [=======>......................] - ETA: 4:50 - loss: 0.0921 - Accuracy: 0.7137
 63/232 [=======>......................] - ETA: 4:48 - loss: 0.0922 - Accuracy: 0.7123
 64/232 [=======>......................] - ETA: 4:47 - loss: 0.0918 - Accuracy: 0.7129
 65/232 [=======>......................] - ETA: 4:48 - loss: 0.0932 - Accuracy: 0.7038
 66/232 [=======>......................] - ETA: 4:46 - loss: 0.0931 - Accuracy: 0.7027
 67/232 [=======>......................] - ETA: 4:45 - loss: 0.0934 - Accuracy: 0.6996
 68/232 [=======>......................] - ETA: 4:43 - loss: 0.0935 - Accuracy: 0.6967
 69/232 [=======>......................] - ETA: 4:41 - loss: 0.0937 - Accuracy: 0.6957
 70/232 [========>.....................] - ETA: 4:39 - loss: 0.0928 - Accuracy: 0.7000
 71/232 [========>.....................] - ETA: 4:37 - loss: 0.0932 - Accuracy: 0.6972
 72/232 [========>.....................] - ETA: 4:35 - loss: 0.0925 - Accuracy: 0.6997
 73/232 [========>.....................] - ETA: 4:33 - loss: 0.0929 - Accuracy: 0.6969
 74/232 [========>.....................] - ETA: 4:32 - loss: 0.0936 - Accuracy: 0.6926
 75/232 [========>.....................] - ETA: 4:30 - loss: 0.0945 - Accuracy: 0.6867
 76/232 [========>.....................] - ETA: 4:28 - loss: 0.0939 - Accuracy: 0.6891
 77/232 [========>.....................] - ETA: 4:26 - loss: 0.0942 - Accuracy: 0.6867
 78/232 [=========>....................] - ETA: 4:24 - loss: 0.0940 - Accuracy: 0.6875
 79/232 [=========>....................] - ETA: 4:23 - loss: 0.0944 - Accuracy: 0.6835
 80/232 [=========>....................] - ETA: 4:21 - loss: 0.0945 - Accuracy: 0.6812
 81/232 [=========>....................] - ETA: 4:19 - loss: 0.0945 - Accuracy: 0.6806
 82/232 [=========>....................] - ETA: 4:18 - loss: 0.0944 - Accuracy: 0.6799
 83/232 [=========>....................] - ETA: 4:16 - loss: 0.0940 - Accuracy: 0.6822
 84/232 [=========>....................] - ETA: 4:14 - loss: 0.0944 - Accuracy: 0.6786
 85/232 [=========>....................] - ETA: 4:12 - loss: 0.0948 - Accuracy: 0.6750
 86/232 [==========>...................] - ETA: 4:10 - loss: 0.0948 - Accuracy: 0.6744
 87/232 [==========>...................] - ETA: 4:09 - loss: 0.0946 - Accuracy: 0.6753
 88/232 [==========>...................] - ETA: 4:07 - loss: 0.0942 - Accuracy: 0.6776
 89/232 [==========>...................] - ETA: 4:05 - loss: 0.0939 - Accuracy: 0.6784
 90/232 [==========>...................] - ETA: 4:04 - loss: 0.0935 - Accuracy: 0.6806
 91/232 [==========>...................] - ETA: 4:02 - loss: 0.0930 - Accuracy: 0.6827
 92/232 [==========>...................] - ETA: 4:00 - loss: 0.0935 - Accuracy: 0.6793
 93/232 [===========>..................] - ETA: 3:59 - loss: 0.0930 - Accuracy: 0.6815
 94/232 [===========>..................] - ETA: 3:57 - loss: 0.0932 - Accuracy: 0.6795
 95/232 [===========>..................] - ETA: 3:55 - loss: 0.0929 - Accuracy: 0.6803
 96/232 [===========>..................] - ETA: 3:53 - loss: 0.0927 - Accuracy: 0.6810
 97/232 [===========>..................] - ETA: 3:52 - loss: 0.0930 - Accuracy: 0.6791
 98/232 [===========>..................] - ETA: 3:50 - loss: 0.0925 - Accuracy: 0.6811
 99/232 [===========>..................] - ETA: 3:48 - loss: 0.0920 - Accuracy: 0.6831
100/232 [===========>..................] - ETA: 3:46 - loss: 0.0920 - Accuracy: 0.6825
101/232 [============>.................] - ETA: 3:44 - loss: 0.0922 - Accuracy: 0.6807
102/232 [============>.................] - ETA: 3:43 - loss: 0.0929 - Accuracy: 0.6765
103/232 [============>.................] - ETA: 3:41 - loss: 0.0927 - Accuracy: 0.6772
104/232 [============>.................] - ETA: 3:39 - loss: 0.0927 - Accuracy: 0.6767
105/232 [============>.................] - ETA: 3:37 - loss: 0.0928 - Accuracy: 0.6750
106/232 [============>.................] - ETA: 3:36 - loss: 0.0927 - Accuracy: 0.6757
107/232 [============>.................] - ETA: 3:34 - loss: 0.0931 - Accuracy: 0.6717
108/232 [============>.................] - ETA: 3:32 - loss: 0.0929 - Accuracy: 0.6736
109/232 [=============>................] - ETA: 3:31 - loss: 0.0929 - Accuracy: 0.6732
110/232 [=============>................] - ETA: 3:29 - loss: 0.0930 - Accuracy: 0.6716
111/232 [=============>................] - ETA: 3:27 - loss: 0.0928 - Accuracy: 0.6723
112/232 [=============>................] - ETA: 3:25 - loss: 0.0928 - Accuracy: 0.6719
113/232 [=============>................] - ETA: 3:23 - loss: 0.0927 - Accuracy: 0.6715
114/232 [=============>................] - ETA: 3:22 - loss: 0.0924 - Accuracy: 0.6732
115/232 [=============>................] - ETA: 3:20 - loss: 0.0924 - Accuracy: 0.6728
116/232 [==============>...............] - ETA: 3:18 - loss: 0.0920 - Accuracy: 0.6746
117/232 [==============>...............] - ETA: 3:17 - loss: 0.0918 - Accuracy: 0.6752
118/232 [==============>...............] - ETA: 3:15 - loss: 0.0914 - Accuracy: 0.6769
119/232 [==============>...............] - ETA: 3:13 - loss: 0.0908 - Accuracy: 0.6796
120/232 [==============>...............] - ETA: 3:12 - loss: 0.0906 - Accuracy: 0.6802
121/232 [==============>...............] - ETA: 3:10 - loss: 0.0909 - Accuracy: 0.6787
122/232 [==============>...............] - ETA: 3:08 - loss: 0.0907 - Accuracy: 0.6793
123/232 [==============>...............] - ETA: 3:07 - loss: 0.0903 - Accuracy: 0.6809
124/232 [===============>..............] - ETA: 3:05 - loss: 0.0901 - Accuracy: 0.6815
125/232 [===============>..............] - ETA: 3:03 - loss: 0.0899 - Accuracy: 0.6820
126/232 [===============>..............] - ETA: 3:01 - loss: 0.0896 - Accuracy: 0.6835
127/232 [===============>..............] - ETA: 3:00 - loss: 0.0896 - Accuracy: 0.6831
128/232 [===============>..............] - ETA: 2:58 - loss: 0.0895 - Accuracy: 0.6836
129/232 [===============>..............] - ETA: 2:56 - loss: 0.0897 - Accuracy: 0.6822
130/232 [===============>..............] - ETA: 2:54 - loss: 0.0896 - Accuracy: 0.6827
131/232 [===============>..............] - ETA: 2:53 - loss: 0.0894 - Accuracy: 0.6832
132/232 [================>.............] - ETA: 2:51 - loss: 0.0895 - Accuracy: 0.6828
133/232 [================>.............] - ETA: 2:49 - loss: 0.0893 - Accuracy: 0.6833
134/232 [================>.............] - ETA: 2:47 - loss: 0.0893 - Accuracy: 0.6828
135/232 [================>.............] - ETA: 2:46 - loss: 0.0892 - Accuracy: 0.6833
136/232 [================>.............] - ETA: 2:44 - loss: 0.0889 - Accuracy: 0.6847
137/232 [================>.............] - ETA: 2:42 - loss: 0.0886 - Accuracy: 0.6861
138/232 [================>.............] - ETA: 2:41 - loss: 0.0886 - Accuracy: 0.6857
139/232 [================>.............] - ETA: 2:39 - loss: 0.0885 - Accuracy: 0.6862
140/232 [=================>............] - ETA: 2:37 - loss: 0.0884 - Accuracy: 0.6866
141/232 [=================>............] - ETA: 2:35 - loss: 0.0881 - Accuracy: 0.6879
142/232 [=================>............] - ETA: 2:34 - loss: 0.0879 - Accuracy: 0.6884
143/232 [=================>............] - ETA: 2:32 - loss: 0.0878 - Accuracy: 0.6888
144/232 [=================>............] - ETA: 2:30 - loss: 0.0874 - Accuracy: 0.6901
145/232 [=================>............] - ETA: 2:28 - loss: 0.0875 - Accuracy: 0.6897
146/232 [=================>............] - ETA: 2:27 - loss: 0.0878 - Accuracy: 0.6884
147/232 [==================>...........] - ETA: 2:25 - loss: 0.0875 - Accuracy: 0.6896
148/232 [==================>...........] - ETA: 2:23 - loss: 0.0877 - Accuracy: 0.6883
149/232 [==================>...........] - ETA: 2:22 - loss: 0.0875 - Accuracy: 0.6888
150/232 [==================>...........] - ETA: 2:20 - loss: 0.0877 - Accuracy: 0.6875
151/232 [==================>...........] - ETA: 2:18 - loss: 0.0882 - Accuracy: 0.6846
152/232 [==================>...........] - ETA: 2:16 - loss: 0.0879 - Accuracy: 0.6859
153/232 [==================>...........] - ETA: 2:15 - loss: 0.0882 - Accuracy: 0.6838
154/232 [==================>...........] - ETA: 2:13 - loss: 0.0880 - Accuracy: 0.6851
155/232 [===================>..........] - ETA: 2:11 - loss: 0.0880 - Accuracy: 0.6847
156/232 [===================>..........] - ETA: 2:10 - loss: 0.0881 - Accuracy: 0.6835
157/232 [===================>..........] - ETA: 2:08 - loss: 0.0882 - Accuracy: 0.6823
158/232 [===================>..........] - ETA: 2:06 - loss: 0.0881 - Accuracy: 0.6835
159/232 [===================>..........] - ETA: 2:04 - loss: 0.0881 - Accuracy: 0.6832
160/232 [===================>..........] - ETA: 2:03 - loss: 0.0880 - Accuracy: 0.6836
161/232 [===================>..........] - ETA: 2:01 - loss: 0.0880 - Accuracy: 0.6840
162/232 [===================>..........] - ETA: 1:59 - loss: 0.0878 - Accuracy: 0.6844
163/232 [====================>.........] - ETA: 1:57 - loss: 0.0879 - Accuracy: 0.6840
164/232 [====================>.........] - ETA: 1:56 - loss: 0.0877 - Accuracy: 0.6845
165/232 [====================>.........] - ETA: 1:54 - loss: 0.0881 - Accuracy: 0.6826
166/232 [====================>.........] - ETA: 1:52 - loss: 0.0880 - Accuracy: 0.6830
167/232 [====================>.........] - ETA: 1:51 - loss: 0.0875 - Accuracy: 0.6849
168/232 [====================>.........] - ETA: 1:49 - loss: 0.0876 - Accuracy: 0.6845
169/232 [====================>.........] - ETA: 1:47 - loss: 0.0883 - Accuracy: 0.6812
170/232 [====================>.........] - ETA: 1:46 - loss: 0.0886 - Accuracy: 0.6794
171/232 [=====================>........] - ETA: 1:44 - loss: 0.0883 - Accuracy: 0.6806
172/232 [=====================>........] - ETA: 1:42 - loss: 0.0884 - Accuracy: 0.6802
173/232 [=====================>........] - ETA: 1:40 - loss: 0.0882 - Accuracy: 0.6806
174/232 [=====================>........] - ETA: 1:39 - loss: 0.0886 - Accuracy: 0.6782
175/232 [=====================>........] - ETA: 1:37 - loss: 0.0884 - Accuracy: 0.6793
176/232 [=====================>........] - ETA: 1:35 - loss: 0.0884 - Accuracy: 0.6790
177/232 [=====================>........] - ETA: 1:34 - loss: 0.0884 - Accuracy: 0.6787
178/232 [======================>.......] - ETA: 1:32 - loss: 0.0884 - Accuracy: 0.6791
179/232 [======================>.......] - ETA: 1:30 - loss: 0.0884 - Accuracy: 0.6788
180/232 [======================>.......] - ETA: 1:28 - loss: 0.0884 - Accuracy: 0.6785
181/232 [======================>.......] - ETA: 1:27 - loss: 0.0883 - Accuracy: 0.6789
182/232 [======================>.......] - ETA: 1:25 - loss: 0.0882 - Accuracy: 0.6793
183/232 [======================>.......] - ETA: 1:23 - loss: 0.0882 - Accuracy: 0.6790
184/232 [======================>.......] - ETA: 1:22 - loss: 0.0884 - Accuracy: 0.6780
185/232 [======================>.......] - ETA: 1:20 - loss: 0.0884 - Accuracy: 0.6777
186/232 [=======================>......] - ETA: 1:18 - loss: 0.0886 - Accuracy: 0.6761
187/232 [=======================>......] - ETA: 1:16 - loss: 0.0885 - Accuracy: 0.6765
188/232 [=======================>......] - ETA: 1:15 - loss: 0.0888 - Accuracy: 0.6749
189/232 [=======================>......] - ETA: 1:13 - loss: 0.0886 - Accuracy: 0.6759
190/232 [=======================>......] - ETA: 1:11 - loss: 0.0886 - Accuracy: 0.6757
191/232 [=======================>......] - ETA: 1:10 - loss: 0.0884 - Accuracy: 0.6767
192/232 [=======================>......] - ETA: 1:08 - loss: 0.0882 - Accuracy: 0.6777
193/232 [=======================>......] - ETA: 1:06 - loss: 0.0879 - Accuracy: 0.6794
194/232 [========================>.....] - ETA: 1:04 - loss: 0.0876 - Accuracy: 0.6811
195/232 [========================>.....] - ETA: 1:03 - loss: 0.0876 - Accuracy: 0.6808
196/232 [========================>.....] - ETA: 1:01 - loss: 0.0875 - Accuracy: 0.6811
197/232 [========================>.....] - ETA: 59s - loss: 0.0873 - Accuracy: 0.6821 
198/232 [========================>.....] - ETA: 58s - loss: 0.0872 - Accuracy: 0.6824
199/232 [========================>.....] - ETA: 56s - loss: 0.0871 - Accuracy: 0.6828
200/232 [========================>.....] - ETA: 54s - loss: 0.0872 - Accuracy: 0.6825
201/232 [========================>.....] - ETA: 52s - loss: 0.0874 - Accuracy: 0.6816
202/232 [=========================>....] - ETA: 51s - loss: 0.0873 - Accuracy: 0.6819
203/232 [=========================>....] - ETA: 49s - loss: 0.0872 - Accuracy: 0.6823
204/232 [=========================>....] - ETA: 47s - loss: 0.0870 - Accuracy: 0.6832
205/232 [=========================>....] - ETA: 46s - loss: 0.0869 - Accuracy: 0.6835
206/232 [=========================>....] - ETA: 44s - loss: 0.0872 - Accuracy: 0.6820
207/232 [=========================>....] - ETA: 42s - loss: 0.0873 - Accuracy: 0.6818
208/232 [=========================>....] - ETA: 40s - loss: 0.0872 - Accuracy: 0.6821
209/232 [==========================>...] - ETA: 39s - loss: 0.0869 - Accuracy: 0.6836
210/232 [==========================>...] - ETA: 37s - loss: 0.0868 - Accuracy: 0.6839
211/232 [==========================>...] - ETA: 35s - loss: 0.0866 - Accuracy: 0.6848
212/232 [==========================>...] - ETA: 34s - loss: 0.0863 - Accuracy: 0.6863
213/232 [==========================>...] - ETA: 32s - loss: 0.0864 - Accuracy: 0.6860
214/232 [==========================>...] - ETA: 30s - loss: 0.0864 - Accuracy: 0.6857
215/232 [==========================>...] - ETA: 29s - loss: 0.0865 - Accuracy: 0.6849
216/232 [==========================>...] - ETA: 27s - loss: 0.0863 - Accuracy: 0.6858
217/232 [===========================>..] - ETA: 25s - loss: 0.0866 - Accuracy: 0.6843
218/232 [===========================>..] - ETA: 23s - loss: 0.0865 - Accuracy: 0.6846
219/232 [===========================>..] - ETA: 22s - loss: 0.0865 - Accuracy: 0.6844
220/232 [===========================>..] - ETA: 20s - loss: 0.0865 - Accuracy: 0.6841
221/232 [===========================>..] - ETA: 18s - loss: 0.0865 - Accuracy: 0.6844
222/232 [===========================>..] - ETA: 17s - loss: 0.0867 - Accuracy: 0.6824
223/232 [===========================>..] - ETA: 15s - loss: 0.0868 - Accuracy: 0.6822
224/232 [===========================>..] - ETA: 13s - loss: 0.0867 - Accuracy: 0.6830
225/232 [============================>.] - ETA: 11s - loss: 0.0867 - Accuracy: 0.6828
226/232 [============================>.] - ETA: 10s - loss: 0.0867 - Accuracy: 0.6825
227/232 [============================>.] - ETA: 8s - loss: 0.0867 - Accuracy: 0.6823 
228/232 [============================>.] - ETA: 6s - loss: 0.0869 - Accuracy: 0.6815
229/232 [============================>.] - ETA: 5s - loss: 0.0870 - Accuracy: 0.6807
230/232 [============================>.] - ETA: 3s - loss: 0.0871 - Accuracy: 0.6799
231/232 [============================>.] - ETA: 1s - loss: 0.0870 - Accuracy: 0.6802
232/232 [==============================] - ETA: 0s - loss: 0.0871 - Accuracy: 0.68002021-12-08 21:24:38.031434: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:766] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset: name: "FlatMapDataset/_9"
op: "FlatMapDataset"
input: "PrefetchDataset/_8"
attr {
  key: "Targuments"
  value {
    list {
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: -2
  }
}
attr {
  key: "f"
  value {
    func {
      name: "__inference_Dataset_flat_map_slice_batch_indices_244154"
    }
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\021FlatMapDataset:35"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: -1
        }
      }
    }
  }
}
attr {
  key: "output_types"
  value {
    list {
      type: DT_INT64
    }
  }
}
. Consider either turning off auto-sharding or switching the auto_shard_policy to DATA to shard this dataset. You can do this by creating a new `tf.data.Options()` object then setting `options.experimental_distribute.auto_shard_policy = AutoShardPolicy.DATA` before applying the options object to the dataset via `dataset.with_options(options)`.
2021-12-08 21:24:38.065957: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.068157: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.074784: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.077624: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.083942: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.085927: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.090917: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.092183: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.135684: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.138288: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.145706: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.148147: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.149358: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.157175: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.159599: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.166636: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.169011: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:24:38.170183: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:25:02.591269: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 4794/13482 nodes to float16 precision using 248 cast(s) to float16 (excluding Const and Variable casts)
2021-12-08 21:25:06.903502: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1991] Converted 0/12606 nodes to float16 precision using 0 cast(s) to float16 (excluding Const and Variable casts)

232/232 [==============================] - 579s 2s/step - loss: 0.0871 - Accuracy: 0.6800 - val_loss: 1.1778 - val_Accuracy: 0.6700
Epoch 2/2

  1/232 [..............................] - ETA: 4:56 - loss: 0.0820 - Accuracy: 0.6667
  2/232 [..............................] - ETA: 6:36 - loss: 0.0660 - Accuracy: 0.7857
  3/232 [..............................] - ETA: 6:24 - loss: 0.0625 - Accuracy: 0.8182
  4/232 [..............................] - ETA: 6:19 - loss: 0.0699 - Accuracy: 0.7667
  5/232 [..............................] - ETA: 6:20 - loss: 0.0697 - Accuracy: 0.7632
  6/232 [..............................] - ETA: 6:20 - loss: 0.0768 - Accuracy: 0.7174
  7/232 [..............................] - ETA: 6:17 - loss: 0.0756 - Accuracy: 0.7222
  8/232 [>.............................] - ETA: 6:15 - loss: 0.0743 - Accuracy: 0.7258
  9/232 [>.............................] - ETA: 6:13 - loss: 0.0759 - Accuracy: 0.7143
 10/232 [>.............................] - ETA: 6:10 - loss: 0.0751 - Accuracy: 0.7179
 11/232 [>.............................] - ETA: 6:09 - loss: 0.0763 - Accuracy: 0.7093
 12/232 [>.............................] - ETA: 6:07 - loss: 0.0757 - Accuracy: 0.7128
 13/232 [>.............................] - ETA: 6:07 - loss: 0.0800 - Accuracy: 0.6863
 14/232 [>.............................] - ETA: 6:05 - loss: 0.0826 - Accuracy: 0.6727
 15/232 [>.............................] - ETA: 6:03 - loss: 0.0820 - Accuracy: 0.6780
 16/232 [=>............................] - ETA: 6:02 - loss: 0.0838 - Accuracy: 0.6667
 17/232 [=>............................] - ETA: 6:04 - loss: 0.0839 - Accuracy: 0.6716
 18/232 [=>............................] - ETA: 6:02 - loss: 0.0833 - Accuracy: 0.6761
 19/232 [=>............................] - ETA: 6:01 - loss: 0.0836 - Accuracy: 0.6733
 20/232 [=>............................] - ETA: 5:59 - loss: 0.0853 - Accuracy: 0.6646
 21/232 [=>............................] - ETA: 5:58 - loss: 0.0866 - Accuracy: 0.6566
 22/232 [=>............................] - ETA: 5:56 - loss: 0.0869 - Accuracy: 0.6552
 23/232 [=>............................] - ETA: 5:55 - loss: 0.0872 - Accuracy: 0.6538
 24/232 [==>...........................] - ETA: 5:53 - loss: 0.0875 - Accuracy: 0.6526
 25/232 [==>...........................] - ETA: 5:51 - loss: 0.0859 - Accuracy: 0.6616
 26/232 [==>...........................] - ETA: 5:51 - loss: 0.0844 - Accuracy: 0.6699
 27/232 [==>...........................] - ETA: 5:48 - loss: 0.0830 - Accuracy: 0.6776
 28/232 [==>...........................] - ETA: 5:47 - loss: 0.0833 - Accuracy: 0.6757
 29/232 [==>...........................] - ETA: 5:45 - loss: 0.0827 - Accuracy: 0.6783
 30/232 [==>...........................] - ETA: 5:43 - loss: 0.0821 - Accuracy: 0.6807
 31/232 [===>..........................] - ETA: 5:41 - loss: 0.0809 - Accuracy: 0.6870
 32/232 [===>..........................] - ETA: 5:39 - loss: 0.0798 - Accuracy: 0.6929
 33/232 [===>..........................] - ETA: 5:38 - loss: 0.0794 - Accuracy: 0.6947
 34/232 [===>..........................] - ETA: 5:36 - loss: 0.0797 - Accuracy: 0.6926
 35/232 [===>..........................] - ETA: 5:34 - loss: 0.0816 - Accuracy: 0.6835
 36/232 [===>..........................] - ETA: 5:32 - loss: 0.0813 - Accuracy: 0.6853
 37/232 [===>..........................] - ETA: 5:30 - loss: 0.0814 - Accuracy: 0.6837
 38/232 [===>..........................] - ETA: 5:29 - loss: 0.0814 - Accuracy: 0.6854
 39/232 [====>.........................] - ETA: 5:27 - loss: 0.0808 - Accuracy: 0.6903
 40/232 [====>.........................] - ETA: 5:26 - loss: 0.0818 - Accuracy: 0.6855
 41/232 [====>.........................] - ETA: 5:24 - loss: 0.0834 - Accuracy: 0.6779
 42/232 [====>.........................] - ETA: 5:23 - loss: 0.0823 - Accuracy: 0.6826
 43/232 [====>.........................] - ETA: 5:21 - loss: 0.0820 - Accuracy: 0.6842
 44/232 [====>.........................] - ETA: 5:19 - loss: 0.0810 - Accuracy: 0.6886
 45/232 [====>.........................] - ETA: 5:18 - loss: 0.0813 - Accuracy: 0.6872
 46/232 [====>.........................] - ETA: 5:16 - loss: 0.0811 - Accuracy: 0.6885
 47/232 [=====>........................] - ETA: 5:14 - loss: 0.0825 - Accuracy: 0.6818
 48/232 [=====>........................] - ETA: 5:13 - loss: 0.0823 - Accuracy: 0.6832
 49/232 [=====>........................] - ETA: 5:12 - loss: 0.0816 - Accuracy: 0.6872
 50/232 [=====>........................] - ETA: 5:10 - loss: 0.0814 - Accuracy: 0.6884
 51/232 [=====>........................] - ETA: 5:09 - loss: 0.0828 - Accuracy: 0.6798
 52/232 [=====>........................] - ETA: 5:08 - loss: 0.0830 - Accuracy: 0.6787
 53/232 [=====>........................] - ETA: 5:06 - loss: 0.0831 - Accuracy: 0.6777
 54/232 [=====>........................] - ETA: 5:04 - loss: 0.0831 - Accuracy: 0.6791
 55/232 [======>.......................] - ETA: 5:03 - loss: 0.0838 - Accuracy: 0.6735
 56/232 [======>.......................] - ETA: 5:01 - loss: 0.0834 - Accuracy: 0.6771
 57/232 [======>.......................] - ETA: 4:59 - loss: 0.0835 - Accuracy: 0.6762
 58/232 [======>.......................] - ETA: 4:57 - loss: 0.0829 - Accuracy: 0.6797
 59/232 [======>.......................] - ETA: 4:55 - loss: 0.0830 - Accuracy: 0.6787
 60/232 [======>.......................] - ETA: 4:53 - loss: 0.0824 - Accuracy: 0.6820
 61/232 [======>.......................] - ETA: 4:52 - loss: 0.0821 - Accuracy: 0.6831
 62/232 [=======>......................] - ETA: 4:50 - loss: 0.0814 - Accuracy: 0.6862
 63/232 [=======>......................] - ETA: 4:48 - loss: 0.0821 - Accuracy: 0.6833
 64/232 [=======>......................] - ETA: 4:46 - loss: 0.0820 - Accuracy: 0.6843
 65/232 [=======>......................] - ETA: 4:45 - loss: 0.0809 - Accuracy: 0.6892
 66/232 [=======>......................] - ETA: 4:43 - loss: 0.0803 - Accuracy: 0.6920
 67/232 [=======>......................] - ETA: 4:41 - loss: 0.0802 - Accuracy: 0.6929
 68/232 [=======>......................] - ETA: 4:39 - loss: 0.0805 - Accuracy: 0.6919
 69/232 [=======>......................] - ETA: 4:38 - loss: 0.0812 - Accuracy: 0.6891
 70/232 [========>.....................] - ETA: 4:36 - loss: 0.0810 - Accuracy: 0.6900
 71/232 [========>.....................] - ETA: 4:34 - loss: 0.0808 - Accuracy: 0.6908
 72/232 [========>.....................] - ETA: 4:32 - loss: 0.0810 - Accuracy: 0.6899
 73/232 [========>.....................] - ETA: 4:30 - loss: 0.0809 - Accuracy: 0.6907
 74/232 [========>.....................] - ETA: 4:28 - loss: 0.0807 - Accuracy: 0.6915
 75/232 [========>.....................] - ETA: 4:27 - loss: 0.0814 - Accuracy: 0.6873
 76/232 [========>.....................] - ETA: 4:25 - loss: 0.0811 - Accuracy: 0.6898
 77/232 [========>.....................] - ETA: 4:23 - loss: 0.0816 - Accuracy: 0.6857
 78/232 [=========>....................] - ETA: 4:21 - loss: 0.0820 - Accuracy: 0.6849
 79/232 [=========>....................] - ETA: 4:20 - loss: 0.0820 - Accuracy: 0.6857
 80/232 [=========>....................] - ETA: 4:18 - loss: 0.0815 - Accuracy: 0.6881
 81/232 [=========>....................] - ETA: 4:16 - loss: 0.0816 - Accuracy: 0.6873
 82/232 [=========>....................] - ETA: 4:14 - loss: 0.0811 - Accuracy: 0.6896
 83/232 [=========>....................] - ETA: 4:12 - loss: 0.0815 - Accuracy: 0.6873
 84/232 [=========>....................] - ETA: 4:11 - loss: 0.0816 - Accuracy: 0.6866
 85/232 [=========>....................] - ETA: 4:09 - loss: 0.0824 - Accuracy: 0.6829
 86/232 [==========>...................] - ETA: 4:07 - loss: 0.0828 - Accuracy: 0.6808
 87/232 [==========>...................] - ETA: 4:06 - loss: 0.0824 - Accuracy: 0.6830
 88/232 [==========>...................] - ETA: 4:04 - loss: 0.0825 - Accuracy: 0.6823
 89/232 [==========>...................] - ETA: 4:02 - loss: 0.0825 - Accuracy: 0.6817
 90/232 [==========>...................] - ETA: 4:00 - loss: 0.0820 - Accuracy: 0.6852
 91/232 [==========>...................] - ETA: 3:59 - loss: 0.0826 - Accuracy: 0.6818
 92/232 [==========>...................] - ETA: 3:57 - loss: 0.0829 - Accuracy: 0.6798
 93/232 [===========>..................] - ETA: 3:55 - loss: 0.0830 - Accuracy: 0.6792
 94/232 [===========>..................] - ETA: 3:54 - loss: 0.0832 - Accuracy: 0.6773
 95/232 [===========>..................] - ETA: 3:52 - loss: 0.0835 - Accuracy: 0.6755
 96/232 [===========>..................] - ETA: 3:50 - loss: 0.0835 - Accuracy: 0.6762
 97/232 [===========>..................] - ETA: 3:49 - loss: 0.0837 - Accuracy: 0.6757
 98/232 [===========>..................] - ETA: 3:47 - loss: 0.0834 - Accuracy: 0.6777
 99/232 [===========>..................] - ETA: 3:45 - loss: 0.0833 - Accuracy: 0.6785
100/232 [===========>..................] - ETA: 3:44 - loss: 0.0835 - Accuracy: 0.6767
101/232 [============>.................] - ETA: 3:42 - loss: 0.0834 - Accuracy: 0.6774
102/232 [============>.................] - ETA: 3:40 - loss: 0.0837 - Accuracy: 0.6757
103/232 [============>.................] - ETA: 3:38 - loss: 0.0838 - Accuracy: 0.6752
104/232 [============>.................] - ETA: 3:37 - loss: 0.0837 - Accuracy: 0.6759
105/232 [============>.................] - ETA: 3:35 - loss: 0.0835 - Accuracy: 0.6766
106/232 [============>.................] - ETA: 3:33 - loss: 0.0834 - Accuracy: 0.6773
107/232 [============>.................] - ETA: 3:32 - loss: 0.0837 - Accuracy: 0.6756
108/232 [============>.................] - ETA: 3:30 - loss: 0.0840 - Accuracy: 0.6740
109/232 [=============>................] - ETA: 3:28 - loss: 0.0838 - Accuracy: 0.6747
110/232 [=============>................] - ETA: 3:27 - loss: 0.0835 - Accuracy: 0.6765
111/232 [=============>................] - ETA: 3:25 - loss: 0.0834 - Accuracy: 0.6772
112/232 [=============>................] - ETA: 3:23 - loss: 0.0833 - Accuracy: 0.6779
113/232 [=============>................] - ETA: 3:22 - loss: 0.0836 - Accuracy: 0.6763
114/232 [=============>................] - ETA: 3:20 - loss: 0.0836 - Accuracy: 0.6758
115/232 [=============>................] - ETA: 3:18 - loss: 0.0833 - Accuracy: 0.6776
116/232 [==============>...............] - ETA: 3:16 - loss: 0.0834 - Accuracy: 0.6771
117/232 [==============>...............] - ETA: 3:15 - loss: 0.0833 - Accuracy: 0.6777
118/232 [==============>...............] - ETA: 3:13 - loss: 0.0837 - Accuracy: 0.6752
119/232 [==============>...............] - ETA: 3:11 - loss: 0.0836 - Accuracy: 0.6758
120/232 [==============>...............] - ETA: 3:10 - loss: 0.0834 - Accuracy: 0.6775
121/232 [==============>...............] - ETA: 3:08 - loss: 0.0836 - Accuracy: 0.6760
122/232 [==============>...............] - ETA: 3:06 - loss: 0.0831 - Accuracy: 0.6786
123/232 [==============>...............] - ETA: 3:05 - loss: 0.0832 - Accuracy: 0.6782
124/232 [===============>..............] - ETA: 3:03 - loss: 0.0829 - Accuracy: 0.6798
125/232 [===============>..............] - ETA: 3:01 - loss: 0.0828 - Accuracy: 0.6804
126/232 [===============>..............] - ETA: 3:00 - loss: 0.0829 - Accuracy: 0.6799
127/232 [===============>..............] - ETA: 2:58 - loss: 0.0824 - Accuracy: 0.6824
128/232 [===============>..............] - ETA: 2:56 - loss: 0.0822 - Accuracy: 0.6830
129/232 [===============>..............] - ETA: 2:54 - loss: 0.0822 - Accuracy: 0.6835
130/232 [===============>..............] - ETA: 2:53 - loss: 0.0818 - Accuracy: 0.6850
131/232 [===============>..............] - ETA: 2:51 - loss: 0.0817 - Accuracy: 0.6855
132/232 [================>.............] - ETA: 2:49 - loss: 0.0816 - Accuracy: 0.6860
133/232 [================>.............] - ETA: 2:48 - loss: 0.0821 - Accuracy: 0.6836
134/232 [================>.............] - ETA: 2:46 - loss: 0.0820 - Accuracy: 0.6841
135/232 [================>.............] - ETA: 2:44 - loss: 0.0823 - Accuracy: 0.6827
136/232 [================>.............] - ETA: 2:43 - loss: 0.0821 - Accuracy: 0.6842
137/232 [================>.............] - ETA: 2:41 - loss: 0.0821 - Accuracy: 0.6837
138/232 [================>.............] - ETA: 2:39 - loss: 0.0825 - Accuracy: 0.6815
139/232 [================>.............] - ETA: 2:38 - loss: 0.0825 - Accuracy: 0.6811
140/232 [=================>............] - ETA: 2:36 - loss: 0.0825 - Accuracy: 0.6816
141/232 [=================>............] - ETA: 2:34 - loss: 0.0823 - Accuracy: 0.6829
142/232 [=================>............] - ETA: 2:33 - loss: 0.0821 - Accuracy: 0.6843
143/232 [=================>............] - ETA: 2:31 - loss: 0.0818 - Accuracy: 0.6856
144/232 [=================>............] - ETA: 2:29 - loss: 0.0817 - Accuracy: 0.6861
145/232 [=================>............] - ETA: 2:27 - loss: 0.0814 - Accuracy: 0.6874
146/232 [=================>............] - ETA: 2:26 - loss: 0.0816 - Accuracy: 0.6870
147/232 [==================>...........] - ETA: 2:24 - loss: 0.0819 - Accuracy: 0.6857
148/232 [==================>...........] - ETA: 2:22 - loss: 0.0818 - Accuracy: 0.6861
149/232 [==================>...........] - ETA: 2:21 - loss: 0.0819 - Accuracy: 0.6857
150/232 [==================>...........] - ETA: 2:19 - loss: 0.0826 - Accuracy: 0.6828
151/232 [==================>...........] - ETA: 2:17 - loss: 0.0829 - Accuracy: 0.6816
152/232 [==================>...........] - ETA: 2:16 - loss: 0.0829 - Accuracy: 0.6812
153/232 [==================>...........] - ETA: 2:14 - loss: 0.0828 - Accuracy: 0.6817
154/232 [==================>...........] - ETA: 2:12 - loss: 0.0826 - Accuracy: 0.6829
155/232 [===================>..........] - ETA: 2:10 - loss: 0.0827 - Accuracy: 0.6826
156/232 [===================>..........] - ETA: 2:09 - loss: 0.0824 - Accuracy: 0.6846
157/232 [===================>..........] - ETA: 2:07 - loss: 0.0825 - Accuracy: 0.6834
158/232 [===================>..........] - ETA: 2:05 - loss: 0.0829 - Accuracy: 0.6807
159/232 [===================>..........] - ETA: 2:04 - loss: 0.0829 - Accuracy: 0.6803
160/232 [===================>..........] - ETA: 2:02 - loss: 0.0832 - Accuracy: 0.6784
161/232 [===================>..........] - ETA: 2:00 - loss: 0.0833 - Accuracy: 0.6788
162/232 [===================>..........] - ETA: 1:59 - loss: 0.0832 - Accuracy: 0.6793
163/232 [====================>.........] - ETA: 1:57 - loss: 0.0832 - Accuracy: 0.6797
164/232 [====================>.........] - ETA: 1:56 - loss: 0.0833 - Accuracy: 0.6794
165/232 [====================>.........] - ETA: 1:54 - loss: 0.0834 - Accuracy: 0.6783
166/232 [====================>.........] - ETA: 1:52 - loss: 0.0834 - Accuracy: 0.6780
167/232 [====================>.........] - ETA: 1:50 - loss: 0.0835 - Accuracy: 0.6777
168/232 [====================>.........] - ETA: 1:49 - loss: 0.0836 - Accuracy: 0.6766
169/232 [====================>.........] - ETA: 1:47 - loss: 0.0835 - Accuracy: 0.6770
170/232 [====================>.........] - ETA: 1:45 - loss: 0.0835 - Accuracy: 0.6775
171/232 [=====================>........] - ETA: 1:44 - loss: 0.0834 - Accuracy: 0.6779
172/232 [=====================>........] - ETA: 1:42 - loss: 0.0834 - Accuracy: 0.6776
173/232 [=====================>........] - ETA: 1:40 - loss: 0.0835 - Accuracy: 0.6773
174/232 [=====================>........] - ETA: 1:39 - loss: 0.0833 - Accuracy: 0.6784
175/232 [=====================>........] - ETA: 1:37 - loss: 0.0835 - Accuracy: 0.6774
176/232 [=====================>........] - ETA: 1:35 - loss: 0.0835 - Accuracy: 0.6771
177/232 [=====================>........] - ETA: 1:33 - loss: 0.0836 - Accuracy: 0.6768
178/232 [======================>.......] - ETA: 1:32 - loss: 0.0835 - Accuracy: 0.6772
179/232 [======================>.......] - ETA: 1:30 - loss: 0.0836 - Accuracy: 0.6762
180/232 [======================>.......] - ETA: 1:28 - loss: 0.0835 - Accuracy: 0.6766
181/232 [======================>.......] - ETA: 1:27 - loss: 0.0834 - Accuracy: 0.6770
182/232 [======================>.......] - ETA: 1:25 - loss: 0.0834 - Accuracy: 0.6774
183/232 [======================>.......] - ETA: 1:23 - loss: 0.0836 - Accuracy: 0.6765
184/232 [======================>.......] - ETA: 1:21 - loss: 0.0837 - Accuracy: 0.6762
185/232 [======================>.......] - ETA: 1:20 - loss: 0.0837 - Accuracy: 0.6759
186/232 [=======================>......] - ETA: 1:18 - loss: 0.0837 - Accuracy: 0.6763
187/232 [=======================>......] - ETA: 1:16 - loss: 0.0836 - Accuracy: 0.6767
188/232 [=======================>......] - ETA: 1:15 - loss: 0.0834 - Accuracy: 0.6778
189/232 [=======================>......] - ETA: 1:13 - loss: 0.0834 - Accuracy: 0.6781
190/232 [=======================>......] - ETA: 1:11 - loss: 0.0836 - Accuracy: 0.6765
191/232 [=======================>......] - ETA: 1:09 - loss: 0.0839 - Accuracy: 0.6750
192/232 [=======================>......] - ETA: 1:08 - loss: 0.0840 - Accuracy: 0.6747
193/232 [=======================>......] - ETA: 1:06 - loss: 0.0839 - Accuracy: 0.6751
194/232 [========================>.....] - ETA: 1:04 - loss: 0.0839 - Accuracy: 0.6748
195/232 [========================>.....] - ETA: 1:03 - loss: 0.0841 - Accuracy: 0.6739
196/232 [========================>.....] - ETA: 1:01 - loss: 0.0841 - Accuracy: 0.6737
197/232 [========================>.....] - ETA: 59s - loss: 0.0839 - Accuracy: 0.6747 
198/232 [========================>.....] - ETA: 57s - loss: 0.0838 - Accuracy: 0.6751
199/232 [========================>.....] - ETA: 56s - loss: 0.0837 - Accuracy: 0.6761
200/232 [========================>.....] - ETA: 54s - loss: 0.0836 - Accuracy: 0.6765
201/232 [========================>.....] - ETA: 52s - loss: 0.0837 - Accuracy: 0.6762
202/232 [=========================>....] - ETA: 51s - loss: 0.0836 - Accuracy: 0.6766
203/232 [=========================>....] - ETA: 49s - loss: 0.0836 - Accuracy: 0.6763
204/232 [=========================>....] - ETA: 47s - loss: 0.0836 - Accuracy: 0.6767
205/232 [=========================>....] - ETA: 46s - loss: 0.0838 - Accuracy: 0.6758
206/232 [=========================>....] - ETA: 44s - loss: 0.0839 - Accuracy: 0.6750
207/232 [=========================>....] - ETA: 42s - loss: 0.0838 - Accuracy: 0.6753
208/232 [=========================>....] - ETA: 40s - loss: 0.0836 - Accuracy: 0.6769
209/232 [==========================>...] - ETA: 39s - loss: 0.0837 - Accuracy: 0.6760
210/232 [==========================>...] - ETA: 37s - loss: 0.0836 - Accuracy: 0.6770
211/232 [==========================>...] - ETA: 35s - loss: 0.0836 - Accuracy: 0.6767
212/232 [==========================>...] - ETA: 34s - loss: 0.0835 - Accuracy: 0.6771
213/232 [==========================>...] - ETA: 32s - loss: 0.0836 - Accuracy: 0.6769
214/232 [==========================>...] - ETA: 30s - loss: 0.0835 - Accuracy: 0.6772
215/232 [==========================>...] - ETA: 28s - loss: 0.0833 - Accuracy: 0.6781
216/232 [==========================>...] - ETA: 27s - loss: 0.0830 - Accuracy: 0.6796
217/232 [===========================>..] - ETA: 25s - loss: 0.0831 - Accuracy: 0.6794
218/232 [===========================>..] - ETA: 23s - loss: 0.0829 - Accuracy: 0.6803
219/232 [===========================>..] - ETA: 22s - loss: 0.0829 - Accuracy: 0.6800
220/232 [===========================>..] - ETA: 20s - loss: 0.0831 - Accuracy: 0.6792
221/232 [===========================>..] - ETA: 18s - loss: 0.0830 - Accuracy: 0.6795
222/232 [===========================>..] - ETA: 17s - loss: 0.0831 - Accuracy: 0.6793
223/232 [===========================>..] - ETA: 15s - loss: 0.0830 - Accuracy: 0.6796
224/232 [===========================>..] - ETA: 13s - loss: 0.0829 - Accuracy: 0.6799
225/232 [============================>.] - ETA: 11s - loss: 0.0833 - Accuracy: 0.6780
226/232 [============================>.] - ETA: 10s - loss: 0.0832 - Accuracy: 0.6783
227/232 [============================>.] - ETA: 8s - loss: 0.0833 - Accuracy: 0.6781 
228/232 [============================>.] - ETA: 6s - loss: 0.0833 - Accuracy: 0.6778
229/232 [============================>.] - ETA: 5s - loss: 0.0833 - Accuracy: 0.6776
230/232 [============================>.] - ETA: 3s - loss: 0.0833 - Accuracy: 0.6774
231/232 [============================>.] - ETA: 1s - loss: 0.0832 - Accuracy: 0.6782
232/232 [==============================] - ETA: 0s - loss: 0.0831 - Accuracy: 0.67912021-12-08 21:32:02.219818: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.222070: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.228852: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.232040: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.238568: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.240707: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.246288: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.247601: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.294783: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.297749: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.305270: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.307566: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.308656: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.316060: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.318295: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.324953: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.327110: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do
2021-12-08 21:32:02.328203: I tensorflow/core/grappler/optimizers/auto_mixed_precision.cc:1386] No allowlist ops found, nothing to do

232/232 [==============================] - 412s 2s/step - loss: 0.0831 - Accuracy: 0.6791 - val_loss: 1.2342 - val_Accuracy: 0.6700
```

Training also works with 3 GPUs.

Gently pinging @Rocketknight1 here



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Been doing library upgrade testing for our stuff lately (we were still using `transformers==2.11.0`) :sweat_smile: and that is almost complete, so hoping to get back to more testing around this in the next few weeks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I still have planned testing coming up, so let's re-open