bigscience/bloom · Unable to load model using `accelerate`

Jul 13, 2022

•

edited Jul 13, 2022

Hi all, I'm trying to load Bloom onto eight A40 GPUs (48 GB of GPU RAM each) using

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")

which fails with

Traceback (most recent call last):

  File "/home/me/bloom/lib/python3.8/site-packages/torch/serialization.py", line 308, in _check_seekable
    f.seek(f.tell())

AttributeError: 'list' object has no attribute 'seek'

Here's the full stack trace.

Here's my system setup:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

$ pip show torch
Name: torch
Version: 1.12.0+cu116

$ pip show transformers
Name: transformers
Version: 4.20.1

$ pip show accelerate
Name: accelerate
Version: 0.10.0

Has anyone encountered this and found a solution?

ybelkada

BigScience Workshop org Jul 13, 2022

Hi,

Thanks for your question ! THis issue is related to that PR: https://github.com/huggingface/transformers/pull/18061
Could you try with this command while the PR gets merged:

from transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)

ricwo

Jul 15, 2022

Thanks so much @ybelkada , it now works as expected

ricwo changed discussion status to closed Jul 15, 2022