bigscience

/

bloom

like 4.57k

I am trying to simply use the smallest bloom model to learn how to train or specialize the model to generate a new language.

I run into this error: AttributeError: 'dict' object has no attribute 'full_determinism'.

I executed pip install transformers so I am hoping it has the latest transformer library. Any help is much appreciated!

Full error message:

AttributeError Traceback (most recent call last)
Cell In[25], line 9
5 t = time.process_time()
6 #model = AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b1", use_cache=True)
7 #tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-1b1")
----> 9 trainer = Trainer(
10 model=AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b1"),
11 args=training_args,
12 train_dataset=dataset["train"]
13 )
14 t2 = time.process_time()
15 print("Chekpoint 1 elapsed time "+t2-t1)

File /opt/homebrew/lib/python3.11/site-packages/transformers/trainer.py:337, in Trainer.init(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
335 self.args = args
336 # Seed must be set before instantiating the model when using model
--> 337 enable_full_determinism(self.args.seed) if self.args.full_determinism else set_seed(self.args.seed)
338 self.hp_name = None
339 self.deepspeed = None

AttributeError: 'dict' object has no attribute 'full_determinism'

Here is my code-------------------------------------:

#Load the smallest Bloom tokenizer:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-1b1")
print("Done loading")

#Load the treaining dataset
dataset = load_dataset("text", data_files={"train": "FirstTrainingSet_clean_v1.txt"})
print(dataset)

#Define the training arguments:

training_args = {
"output_dir": "trained_model",
"num_train_epochs": 10,
"per_device_train_batch_size": 4,
"learning_rate": 1e-5,
"full_determinism":"False",
}

#Train the model:
from transformers import Trainer, AutoModelForCausalLM
import time

t = time.process_time()
#model = AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b1", use_cache=True)
#tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom-1b1")

trainer = Trainer(
model=AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b1"),
args=training_args,
train_dataset=dataset["train"]
)
t2 = time.process_time()
print("Chekpoint 1 elapsed time "+t2-t)

trainer.train()

t3 = time.process_time()
print("Chekpoint 2 elapsed time [trainer.train()] "+t3-t2)

#save the trained model
trainer.save_model()
t4 = time.process_time()
print("Chekpoint 2 elapsed time [trainer.save_model()] "+t4-t3)