Modern GBERT 1B

Hello there, this is the super secret wip repo for our Modern GBERT 1B checkpoints. Check out the branches and load the model like this:

tokenizer = AutoTokenizer.from_pretrained('aehrm/modern-gbert-1b-dev', revision='pretrain-ba96117')
model = AutoModelForMaskedLM.from_pretrained('aehrm/modern-gbert-1b-dev', revision='pretrain-ba96117', device_map='cuda', _attn_implementation='sdpa')
# remove _attn_implementation keyword if you have Flash Attention 2
text = "Die Hauptstadt von [MASK] ist Paris."
inputs = tokenizer(text, return_tensors="pt")

outputs = model(**inputs.to('cuda'))


output_token_ids = outputs.logits[0].argmax(axis=-1)
output_tokens = tokenizer.convert_ids_to_tokens(output_token_ids)
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print("Prediction for mask:", output_tokens[masked_index])