2021-11-17 23:21:43,874 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,875 Model: "SequenceTagger( (embeddings): TransformerWordEmbeddings( (model): RobertaModel( (embeddings): RobertaEmbeddings( (word_embeddings): Embedding(50265, 768, padding_idx=1) (position_embeddings): Embedding(514, 768, padding_idx=1) (token_type_embeddings): Embedding(1, 768) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) (encoder): RobertaEncoder( (layer): ModuleList( (0): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (1): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (2): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (3): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (4): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (5): RobertaLayer( (attention): RobertaAttention( (self): RobertaSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): RobertaSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): RobertaIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) ) (output): RobertaOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) ) ) (pooler): RobertaPooler( (dense): Linear(in_features=768, out_features=768, bias=True) (activation): Tanh() ) ) ) (word_dropout): WordDropout(p=0.05) (locked_dropout): LockedDropout(p=0.5) (embedding2nn): Linear(in_features=1536, out_features=1536, bias=True) (linear): Linear(in_features=1536, out_features=16, bias=True) (beta): 1.0 (weights): None (weight_tensor) None )" 2021-11-17 23:21:43,876 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,877 Corpus: "Corpus: 56700 train + 6300 dev + 7000 test sentences" 2021-11-17 23:21:43,877 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,878 Parameters: 2021-11-17 23:21:43,878 - learning_rate: "5e-05" 2021-11-17 23:21:43,879 - mini_batch_size: "64" 2021-11-17 23:21:43,879 - patience: "3" 2021-11-17 23:21:43,879 - anneal_factor: "0.5" 2021-11-17 23:21:43,880 - max_epochs: "8" 2021-11-17 23:21:43,881 - shuffle: "True" 2021-11-17 23:21:43,881 - train_with_dev: "False" 2021-11-17 23:21:43,882 - batch_growth_annealing: "False" 2021-11-17 23:21:43,882 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,883 Model training base path: "training/flair_ner/en/17112021_231902" 2021-11-17 23:21:43,883 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,884 Device: cuda 2021-11-17 23:21:43,885 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:43,885 Embeddings storage mode: cpu 2021-11-17 23:21:43,886 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:21:57,350 epoch 1 - iter 88/886 - loss 0.50060718 - samples/sec: 418.55 - lr: 0.000050 2021-11-17 23:22:10,500 epoch 1 - iter 176/886 - loss 0.32189657 - samples/sec: 428.58 - lr: 0.000050 2021-11-17 23:22:23,215 epoch 1 - iter 264/886 - loss 0.25798771 - samples/sec: 443.41 - lr: 0.000050 2021-11-17 23:22:35,888 epoch 1 - iter 352/886 - loss 0.22669943 - samples/sec: 444.82 - lr: 0.000050 2021-11-17 23:22:48,672 epoch 1 - iter 440/886 - loss 0.20548598 - samples/sec: 440.79 - lr: 0.000050 2021-11-17 23:23:01,458 epoch 1 - iter 528/886 - loss 0.19096343 - samples/sec: 440.79 - lr: 0.000050 2021-11-17 23:23:14,258 epoch 1 - iter 616/886 - loss 0.18023473 - samples/sec: 440.24 - lr: 0.000050 2021-11-17 23:23:27,118 epoch 1 - iter 704/886 - loss 0.17198943 - samples/sec: 438.19 - lr: 0.000050 2021-11-17 23:23:39,791 epoch 1 - iter 792/886 - loss 0.16499517 - samples/sec: 444.63 - lr: 0.000050 2021-11-17 23:23:52,506 epoch 1 - iter 880/886 - loss 0.15942326 - samples/sec: 443.19 - lr: 0.000050 2021-11-17 23:23:53,362 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:23:53,363 EPOCH 1 done: loss 0.1591 - lr 0.0000500 2021-11-17 23:24:06,817 DEV : loss 0.002542673610150814 - f1-score (micro avg) 0.9992 2021-11-17 23:24:06,902 BAD EPOCHS (no improvement): 0 2021-11-17 23:24:06,903 saving best model 2021-11-17 23:24:07,239 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:24:20,356 epoch 2 - iter 88/886 - loss 0.11000766 - samples/sec: 429.70 - lr: 0.000050 2021-11-17 23:24:33,380 epoch 2 - iter 176/886 - loss 0.10909856 - samples/sec: 432.73 - lr: 0.000050 2021-11-17 23:24:46,404 epoch 2 - iter 264/886 - loss 0.10926820 - samples/sec: 432.72 - lr: 0.000050 2021-11-17 23:24:59,233 epoch 2 - iter 352/886 - loss 0.10950969 - samples/sec: 439.32 - lr: 0.000050 2021-11-17 23:25:12,123 epoch 2 - iter 440/886 - loss 0.11018886 - samples/sec: 437.23 - lr: 0.000050 2021-11-17 23:25:25,126 epoch 2 - iter 528/886 - loss 0.10995752 - samples/sec: 433.43 - lr: 0.000050 2021-11-17 23:25:38,072 epoch 2 - iter 616/886 - loss 0.10983300 - samples/sec: 435.34 - lr: 0.000050 2021-11-17 23:25:51,102 epoch 2 - iter 704/886 - loss 0.10978674 - samples/sec: 432.51 - lr: 0.000050 2021-11-17 23:26:05,660 epoch 2 - iter 792/886 - loss 0.10974621 - samples/sec: 387.25 - lr: 0.000050 2021-11-17 23:26:19,108 epoch 2 - iter 880/886 - loss 0.10964924 - samples/sec: 419.09 - lr: 0.000050 2021-11-17 23:26:20,019 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:26:20,020 EPOCH 2 done: loss 0.1098 - lr 0.0000500 2021-11-17 23:26:34,470 DEV : loss 0.0029088123701512814 - f1-score (micro avg) 0.9988 2021-11-17 23:26:34,553 BAD EPOCHS (no improvement): 1 2021-11-17 23:26:34,553 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:26:47,966 epoch 3 - iter 88/886 - loss 0.11118611 - samples/sec: 420.23 - lr: 0.000050 2021-11-17 23:27:01,224 epoch 3 - iter 176/886 - loss 0.11113361 - samples/sec: 425.09 - lr: 0.000050 2021-11-17 23:27:14,454 epoch 3 - iter 264/886 - loss 0.11038604 - samples/sec: 426.17 - lr: 0.000050 2021-11-17 23:27:27,741 epoch 3 - iter 352/886 - loss 0.11138497 - samples/sec: 424.34 - lr: 0.000050 2021-11-17 23:27:40,811 epoch 3 - iter 440/886 - loss 0.11143778 - samples/sec: 431.20 - lr: 0.000050 2021-11-17 23:27:54,062 epoch 3 - iter 528/886 - loss 0.11093105 - samples/sec: 425.34 - lr: 0.000050 2021-11-17 23:28:07,198 epoch 3 - iter 616/886 - loss 0.11050488 - samples/sec: 429.21 - lr: 0.000050 2021-11-17 23:28:20,418 epoch 3 - iter 704/886 - loss 0.11064153 - samples/sec: 426.32 - lr: 0.000050 2021-11-17 23:28:33,690 epoch 3 - iter 792/886 - loss 0.11022304 - samples/sec: 424.79 - lr: 0.000050 2021-11-17 23:28:47,015 epoch 3 - iter 880/886 - loss 0.11054611 - samples/sec: 422.95 - lr: 0.000050 2021-11-17 23:28:47,991 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:28:47,992 EPOCH 3 done: loss 0.1105 - lr 0.0000500 2021-11-17 23:29:04,469 DEV : loss 0.0013118594652041793 - f1-score (micro avg) 0.9994 2021-11-17 23:29:04,549 BAD EPOCHS (no improvement): 0 2021-11-17 23:29:04,550 saving best model 2021-11-17 23:29:05,206 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:29:19,255 epoch 4 - iter 88/886 - loss 0.11101590 - samples/sec: 401.22 - lr: 0.000050 2021-11-17 23:29:33,081 epoch 4 - iter 176/886 - loss 0.10997834 - samples/sec: 407.62 - lr: 0.000050 2021-11-17 23:29:46,787 epoch 4 - iter 264/886 - loss 0.11031061 - samples/sec: 411.18 - lr: 0.000050 2021-11-17 23:30:00,054 epoch 4 - iter 352/886 - loss 0.10969025 - samples/sec: 424.81 - lr: 0.000050 2021-11-17 23:30:13,298 epoch 4 - iter 440/886 - loss 0.11001565 - samples/sec: 425.52 - lr: 0.000050 2021-11-17 23:30:26,545 epoch 4 - iter 528/886 - loss 0.11013209 - samples/sec: 425.45 - lr: 0.000050 2021-11-17 23:30:39,776 epoch 4 - iter 616/886 - loss 0.10980630 - samples/sec: 425.95 - lr: 0.000050 2021-11-17 23:30:52,924 epoch 4 - iter 704/886 - loss 0.10947482 - samples/sec: 428.65 - lr: 0.000050 2021-11-17 23:31:06,186 epoch 4 - iter 792/886 - loss 0.10976788 - samples/sec: 424.94 - lr: 0.000050 2021-11-17 23:31:19,571 epoch 4 - iter 880/886 - loss 0.10976014 - samples/sec: 421.06 - lr: 0.000050 2021-11-17 23:31:20,467 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:31:20,468 EPOCH 4 done: loss 0.1098 - lr 0.0000500 2021-11-17 23:31:36,227 DEV : loss 0.0019321050494909286 - f1-score (micro avg) 0.999 2021-11-17 23:31:36,311 BAD EPOCHS (no improvement): 1 2021-11-17 23:31:36,312 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:31:49,776 epoch 5 - iter 88/886 - loss 0.11196203 - samples/sec: 418.62 - lr: 0.000050 2021-11-17 23:32:03,347 epoch 5 - iter 176/886 - loss 0.11146165 - samples/sec: 415.27 - lr: 0.000050 2021-11-17 23:32:16,869 epoch 5 - iter 264/886 - loss 0.11038997 - samples/sec: 416.80 - lr: 0.000050 2021-11-17 23:32:30,210 epoch 5 - iter 352/886 - loss 0.10969957 - samples/sec: 422.45 - lr: 0.000050 2021-11-17 23:32:43,385 epoch 5 - iter 440/886 - loss 0.10883622 - samples/sec: 427.75 - lr: 0.000050 2021-11-17 23:32:57,014 epoch 5 - iter 528/886 - loss 0.10885199 - samples/sec: 413.52 - lr: 0.000050 2021-11-17 23:33:11,225 epoch 5 - iter 616/886 - loss 0.10919470 - samples/sec: 396.74 - lr: 0.000050 2021-11-17 23:33:25,329 epoch 5 - iter 704/886 - loss 0.10968561 - samples/sec: 399.65 - lr: 0.000050 2021-11-17 23:33:38,569 epoch 5 - iter 792/886 - loss 0.10952831 - samples/sec: 425.68 - lr: 0.000050 2021-11-17 23:33:51,869 epoch 5 - iter 880/886 - loss 0.10925988 - samples/sec: 423.91 - lr: 0.000050 2021-11-17 23:33:52,767 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:33:52,768 EPOCH 5 done: loss 0.1092 - lr 0.0000500 2021-11-17 23:34:08,633 DEV : loss 0.001400615437887609 - f1-score (micro avg) 0.9994 2021-11-17 23:34:08,713 BAD EPOCHS (no improvement): 2 2021-11-17 23:34:08,716 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:34:22,104 epoch 6 - iter 88/886 - loss 0.10971184 - samples/sec: 421.02 - lr: 0.000050 2021-11-17 23:34:35,452 epoch 6 - iter 176/886 - loss 0.10810577 - samples/sec: 422.40 - lr: 0.000050 2021-11-17 23:34:48,789 epoch 6 - iter 264/886 - loss 0.10923295 - samples/sec: 422.58 - lr: 0.000050 2021-11-17 23:35:02,187 epoch 6 - iter 352/886 - loss 0.10832324 - samples/sec: 420.62 - lr: 0.000050 2021-11-17 23:35:15,501 epoch 6 - iter 440/886 - loss 0.10890621 - samples/sec: 423.47 - lr: 0.000050 2021-11-17 23:35:28,932 epoch 6 - iter 528/886 - loss 0.10836666 - samples/sec: 419.60 - lr: 0.000050 2021-11-17 23:35:42,421 epoch 6 - iter 616/886 - loss 0.10866986 - samples/sec: 417.83 - lr: 0.000050 2021-11-17 23:35:56,321 epoch 6 - iter 704/886 - loss 0.10845591 - samples/sec: 405.45 - lr: 0.000050 2021-11-17 23:36:10,189 epoch 6 - iter 792/886 - loss 0.10875052 - samples/sec: 406.44 - lr: 0.000050 2021-11-17 23:36:23,804 epoch 6 - iter 880/886 - loss 0.10904969 - samples/sec: 413.93 - lr: 0.000050 2021-11-17 23:36:24,703 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:36:24,704 EPOCH 6 done: loss 0.1092 - lr 0.0000500 2021-11-17 23:36:40,380 DEV : loss 0.0009049061918631196 - f1-score (micro avg) 0.9992 2021-11-17 23:36:40,463 BAD EPOCHS (no improvement): 3 2021-11-17 23:36:40,463 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:36:54,014 epoch 7 - iter 88/886 - loss 0.11094486 - samples/sec: 415.95 - lr: 0.000050 2021-11-17 23:37:07,422 epoch 7 - iter 176/886 - loss 0.10949810 - samples/sec: 420.52 - lr: 0.000050 2021-11-17 23:37:21,230 epoch 7 - iter 264/886 - loss 0.10970254 - samples/sec: 408.14 - lr: 0.000050 2021-11-17 23:37:34,444 epoch 7 - iter 352/886 - loss 0.11019445 - samples/sec: 426.59 - lr: 0.000050 2021-11-17 23:37:47,833 epoch 7 - iter 440/886 - loss 0.11044571 - samples/sec: 420.94 - lr: 0.000050 2021-11-17 23:38:01,118 epoch 7 - iter 528/886 - loss 0.11022272 - samples/sec: 424.19 - lr: 0.000050 2021-11-17 23:38:14,537 epoch 7 - iter 616/886 - loss 0.10975761 - samples/sec: 420.00 - lr: 0.000050 2021-11-17 23:38:27,909 epoch 7 - iter 704/886 - loss 0.10944174 - samples/sec: 421.63 - lr: 0.000050 2021-11-17 23:38:41,133 epoch 7 - iter 792/886 - loss 0.10960931 - samples/sec: 426.17 - lr: 0.000050 2021-11-17 23:38:54,481 epoch 7 - iter 880/886 - loss 0.10960868 - samples/sec: 422.22 - lr: 0.000050 2021-11-17 23:38:55,367 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:38:55,368 EPOCH 7 done: loss 0.1096 - lr 0.0000500 2021-11-17 23:39:11,689 DEV : loss 0.0013050935231149197 - f1-score (micro avg) 0.9995 2021-11-17 23:39:11,770 BAD EPOCHS (no improvement): 0 2021-11-17 23:39:11,773 saving best model 2021-11-17 23:39:12,423 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:39:26,468 epoch 8 - iter 88/886 - loss 0.11104233 - samples/sec: 401.32 - lr: 0.000050 2021-11-17 23:39:40,269 epoch 8 - iter 176/886 - loss 0.11088406 - samples/sec: 408.36 - lr: 0.000050 2021-11-17 23:39:53,968 epoch 8 - iter 264/886 - loss 0.11062941 - samples/sec: 411.41 - lr: 0.000050 2021-11-17 23:40:07,630 epoch 8 - iter 352/886 - loss 0.11052519 - samples/sec: 412.67 - lr: 0.000050 2021-11-17 23:40:21,700 epoch 8 - iter 440/886 - loss 0.10981883 - samples/sec: 400.57 - lr: 0.000050 2021-11-17 23:40:35,699 epoch 8 - iter 528/886 - loss 0.10959840 - samples/sec: 402.57 - lr: 0.000050 2021-11-17 23:40:49,510 epoch 8 - iter 616/886 - loss 0.10968087 - samples/sec: 408.23 - lr: 0.000050 2021-11-17 23:41:03,430 epoch 8 - iter 704/886 - loss 0.10975513 - samples/sec: 404.86 - lr: 0.000050 2021-11-17 23:41:17,719 epoch 8 - iter 792/886 - loss 0.10979006 - samples/sec: 394.41 - lr: 0.000050 2021-11-17 23:41:32,411 epoch 8 - iter 880/886 - loss 0.10979431 - samples/sec: 383.61 - lr: 0.000050 2021-11-17 23:41:33,357 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:41:33,358 EPOCH 8 done: loss 0.1098 - lr 0.0000500 2021-11-17 23:41:50,962 DEV : loss 0.0015213226433843374 - f1-score (micro avg) 0.9993 2021-11-17 23:41:51,053 BAD EPOCHS (no improvement): 1 2021-11-17 23:41:51,466 ---------------------------------------------------------------------------------------------------- 2021-11-17 23:41:51,467 loading file training/flair_ner/en/17112021_231902/best-model.pt 2021-11-17 23:42:09,058 0.9993 0.9993 0.9993 0.9993 2021-11-17 23:42:09,064 Results: - F-score (micro) 0.9993 - F-score (macro) 0.9992 - Accuracy 0.9993 By class: precision recall f1-score support nb_rounds 0.9999 0.9981 0.9990 6889 duration_wt_sd 1.0000 1.0000 1.0000 3292 duration_br_min 0.9975 1.0000 0.9988 3239 duration_wt_min 1.0000 1.0000 1.0000 2685 duration_br_sd 0.9981 0.9995 0.9988 2068 duration_wt_hr 1.0000 1.0000 1.0000 1023 duration_br_hr 0.9957 1.0000 0.9978 230 micro avg 0.9993 0.9993 0.9993 19426 macro avg 0.9987 0.9997 0.9992 19426 weighted avg 0.9993 0.9993 0.9993 19426 samples avg 0.9993 0.9993 0.9993 19426 2021-11-17 23:42:09,065 ----------------------------------------------------------------------------------------------------