File size: 5,893 Bytes
555a2b0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
---
language: da
tags:
- bert
- masked-lm
license: cc-by-4.0
datasets:
- common_crawl
- wikipedia
pipeline_tag: fill-mask
widget:
- text: "København er [MASK] i Danmark."
---

# Danish BERT (uncased) model 

[BotXO.ai](https://www.botxo.ai/) developed this model. For data and training details see their [GitHub repository](https://github.com/botxo/nordic_bert).  

The original model was trained in TensorFlow then I converted it to Pytorch using [transformers-cli](https://huggingface.co/transformers/converting_tensorflow_models.html?highlight=cli).

For TensorFlow version download here: https://www.dropbox.com/s/19cjaoqvv2jicq9/danish_bert_uncased_v2.zip?dl=1


## Architecture

```python
from transformers import AutoModelForPreTraining

model = AutoModelForPreTraining.from_pretrained("DJSammy/bert-base-danish-uncased_BotXO,ai")

params = list(model.named_parameters())
print('danish_bert_uncased_v2 has {:} different named parameters.\n'.format(len(params)))

print('==== Embedding Layer ====\n')
for p in params[0:5]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

print('\n==== First Transformer ====\n')
for p in params[5:21]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

print('\n==== Last Transformer ====\n')
for p in params[181:197]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

print('\n==== Output Layer ====\n')
for p in params[197:]:
    print("{:<55} {:>12}".format(p[0], str(tuple(p[1].size()))))

# danish_bert_uncased_v2 has 206 different named parameters.

# ==== Embedding Layer ====

# bert.embeddings.word_embeddings.weight                  (32000, 768)
# bert.embeddings.position_embeddings.weight                (512, 768)
# bert.embeddings.token_type_embeddings.weight                (2, 768)
# bert.embeddings.LayerNorm.weight                              (768,)
# bert.embeddings.LayerNorm.bias                                (768,)

# ==== First Transformer ====

# bert.encoder.layer.0.attention.self.query.weight          (768, 768)
# bert.encoder.layer.0.attention.self.query.bias                (768,)
# bert.encoder.layer.0.attention.self.key.weight            (768, 768)
# bert.encoder.layer.0.attention.self.key.bias                  (768,)
# bert.encoder.layer.0.attention.self.value.weight          (768, 768)
# bert.encoder.layer.0.attention.self.value.bias                (768,)
# bert.encoder.layer.0.attention.output.dense.weight        (768, 768)
# bert.encoder.layer.0.attention.output.dense.bias              (768,)
# bert.encoder.layer.0.attention.output.LayerNorm.weight        (768,)
# bert.encoder.layer.0.attention.output.LayerNorm.bias          (768,)
# bert.encoder.layer.0.intermediate.dense.weight           (3072, 768)
# bert.encoder.layer.0.intermediate.dense.bias                 (3072,)
# bert.encoder.layer.0.output.dense.weight                 (768, 3072)
# bert.encoder.layer.0.output.dense.bias                        (768,)
# bert.encoder.layer.0.output.LayerNorm.weight                  (768,)
# bert.encoder.layer.0.output.LayerNorm.bias                    (768,)

# ==== Last Transformer ====

# bert.encoder.layer.11.attention.self.query.weight         (768, 768)
# bert.encoder.layer.11.attention.self.query.bias               (768,)
# bert.encoder.layer.11.attention.self.key.weight           (768, 768)
# bert.encoder.layer.11.attention.self.key.bias                 (768,)
# bert.encoder.layer.11.attention.self.value.weight         (768, 768)
# bert.encoder.layer.11.attention.self.value.bias               (768,)
# bert.encoder.layer.11.attention.output.dense.weight       (768, 768)
# bert.encoder.layer.11.attention.output.dense.bias             (768,)
# bert.encoder.layer.11.attention.output.LayerNorm.weight       (768,)
# bert.encoder.layer.11.attention.output.LayerNorm.bias         (768,)
# bert.encoder.layer.11.intermediate.dense.weight          (3072, 768)
# bert.encoder.layer.11.intermediate.dense.bias                (3072,)
# bert.encoder.layer.11.output.dense.weight                (768, 3072)
# bert.encoder.layer.11.output.dense.bias                       (768,)
# bert.encoder.layer.11.output.LayerNorm.weight                 (768,)
# bert.encoder.layer.11.output.LayerNorm.bias                   (768,)

# ==== Output Layer ====

# bert.pooler.dense.weight                                  (768, 768)
# bert.pooler.dense.bias                                        (768,)
# cls.predictions.bias                                        (32000,)
# cls.predictions.transform.dense.weight                    (768, 768)
# cls.predictions.transform.dense.bias                          (768,)
# cls.predictions.transform.LayerNorm.weight                    (768,)
# cls.predictions.transform.LayerNorm.bias                      (768,)
# cls.seq_relationship.weight                                 (2, 768)
# cls.seq_relationship.bias                                       (2,)
```

## Example Pipeline

```python
from transformers import pipeline
unmasker = pipeline('fill-mask', model='DJSammy/bert-base-danish-uncased_BotXO,ai')

unmasker('København er [MASK] i Danmark.')

# Copenhagen is the [MASK] of Denmark.
# =>

# [{'score': 0.788068950176239,
#  'sequence': '[CLS] københavn er hovedstad i danmark. [SEP]',
#  'token': 12610,
#  'token_str': 'hovedstad'},
# {'score': 0.07606703042984009,
#  'sequence': '[CLS] københavn er hovedstaden i danmark. [SEP]',
#  'token': 8108,
#  'token_str': 'hovedstaden'},
# {'score': 0.04299738258123398,
#  'sequence': '[CLS] københavn er metropol i danmark. [SEP]',
#  'token': 23305,
#  'token_str': 'metropol'},
# {'score': 0.008163209073245525,
#  'sequence': '[CLS] københavn er ikke i danmark. [SEP]',
#  'token': 89,
#  'token_str': 'ikke'},
# {'score': 0.006238455418497324,
#  'sequence': '[CLS] københavn er ogsa i danmark. [SEP]',
#  'token': 25253,
#  'token_str': 'ogsa'}]
```