bigscience

/

bloomz

like 492

I don't know if any other ever tried bloomz-7b1 (https://huggingface.co/bigscience/bloomz-7b1) and got same feeling with me.
Especially greedy mode, it seems output nothing all the time, or just output one or two tokens.

That sounds surprising, can you provide some examples you tried and the generated tokens?

Thanks @Muennighoff 's feedback

Allow me make some correction statements:
(1) After clarifying again, I found It's good if I do inference through the web page. Even the answers are not totally identical to BLOOMZ-1176b, but I thought it's good enough to zero-shot.
(2) But if I load model downloaded from here, https://huggingface.co/bigscience/bloomz-7b1/tree/main , and do inference by load it in local, then output nothing else than input.
(2.1) 3 samples I tried. "Write a code snippet with O(log(n)) computational complexity.", "Why is sky blue?", "Translate to English: Je t’aime.",
(2.2) greedy mode used by set up sample=False, and max_new_tokens=100
(2.3) by observing the outputs, actually 100 new tokens truly inferenced, but all of them are index of 0, so actually the output text is identical to input (because skip_special_tokens=True)
(3) The model launched following instruction, and so as inference:
"model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto")"
"outputs = model.generate(inputs)" <== here I add one parameter, "sample=False"
(4) The experiment running on docker containers image from dockerhub repo: tag = huggingface/transformers-pytorch-gpu:latest
(5) I guess the checkpoint file used by web page (hosted inference api) may not be identical to the latest version in repos (https://huggingface.co/bigscience/bloomz-7b1/tree/main)

On my side everything works fine:

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "bigscience/bloomz-7b1"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype="auto", device_map="auto")
inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt").to("cuda")
outputs = model.generate(inputs, do_sample=False)
print(tokenizer.decode(outputs[0]))
Translate to English: Je t’aime. I love you.
inputs = tokenizer.encode("Write a code snippet with O(log(n)) computational complexity.", return_tensors="pt").to("cuda")
outputs = model.generate(inputs, do_sample=False)
print(tokenizer.decode(outputs[0]))
Write a code snippet with O(log(n)) computational complexity. def solve(n):
outputs = model.generate(inputs, do_sample=False, max_length=100)
print(tokenizer.decode(outputs[0]))
Write a code snippet with O(log(n)) computational complexity. def solve(n):
if n == 1:
return 1
return solve(n-1) + solve(n-2)

Can you double-chech you're passing do_sample not sample?
Also make sure you use the latest transformers version if you aren't

Thanks. @Muennighoff , it's my typo. Actually what I passed is "do_sample".
Due to our environment is offline, I would download the checkpoints file again to rule out effect from incorrected "pytorch_model.bin".

I have verified sha256 of checkpoint file and also tokenizer.json, but all of them identical to ones shown in https://huggingface.co/bigscience/bloomz-7b1/tree/main
@Muennighoff could I know the version of pytorch and transformers you used in experiment and also the GPU (Nvidia A100? or else?) thanks.

I have verified sha256 of checkpoint file and also tokenizer.json, but all of them identical to ones shown in https://huggingface.co/bigscience/bloomz-7b1/tree/main
@Muennighoff could I know the version of pytorch and transformers you used in experiment and also the GPU (Nvidia A100? or else?) thanks.

I used the latest versions of transformer (4.24.0) & torch. Esp. the transformers version is important as previous ones had bugs in bloom.
Nvidia A100 w/ 80 GB.

@Muennighoff Thanks for your quick feedback. Anyway, problem resolved. The solution is simple, just downloaded again whole model folder. It's also said the root cause not been found because I have checked each sha256sum of each files in model folder, include checkpoint files, configs, and tokenizer files, they are all identical. Now I tried bloom-7b1 and bloomz-7b1-mt, both of them are good. Thanks again.

@Muennighoff Thanks for your quick feedback. Anyway, problem resolved. The solution is simple, just downloaded again whole model folder. It's also said the root cause not been found because I have checked each sha256sum of each files in model folder, include checkpoint files, configs, and tokenizer files, they are all identical. Now I tried bloom-7b1 and bloomz-7b1-mt, both of them are good. Thanks again.

Amazing! 🤗 Closing this for now, feel free to reopen if issues occur.