bigscience
/

bloomz-mt

Text Generation Transformers PyTorch TensorBoard

46 languages bloom Eval Results Inference Endpoints text-generation-inference

Model card Files Files and versions Metrics Training metrics Community

bloomz-mt / config.json

cakiki's picture

Fix architecture (#2)

4f161c3 about 1 year ago

raw history blame contribute delete

No virus

573 Bytes

	{
	"apply_residual_connection_post_layernorm": false,
	"attention_dropout": 0.0,
	"architectures": [
	"BloomForCausalLM"
	],
	"attention_softmax_in_fp32": true,
	"pad_token_id": 3,
	"bos_token_id": 1,
	"eos_token_id": 2,
	"hidden_dropout": 0.0,
	"initializer_range": 0.02,
	"layer_norm_epsilon": 1e-05,
	"masked_softmax_fusion": true,
	"model_type": "bloom",
	"n_embed": 14336,
	"n_layer": 70,
	"num_attention_heads": 112,
	"pretraining_tp": 4,
	"slow_but_exact": false,
	"transformers_version": "4.21.0",
	"use_cache": true,
	"vocab_size": 250880
	}