RuntimeError: You must initialize the accelerate state by calling either `PartialState()` or `Accelerator()` before using the logging utility.
Author: amarv3142
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@Narsil @ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am using Google colab to run starchat-beta model from here https://huggingface.co/HuggingFaceH4/starchat-beta

Google Colab Link: https://colab.research.google.com/drive/1I1-zAY3AYNEiZ9Lk-35yqNrOncLFUsSo#scrollTo=-vH0ityRx9u4

**Step1**: Installed required library on colab
```
!pip install transformers
!pip install accelerate
!pip install xformers
```

**Step2:** Run below sample code from the model card of starchat-beta model
```
import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="HuggingFaceH4/starchat-beta", torch_dtype=torch.bfloat16, device_map="auto")
prompt_template = "
\n
\n
\n{query}
\n
"
prompt = prompt_template.format(query="How do I sort a list in Python?")
outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.2, top_k=50, top_p=0.95, eos_token_id=49155)
```

**Issue:** The last line is producing the following error 

> RuntimeError: You must initialize the accelerate state by calling either `PartialState()` or `Accelerator()` before using the logging utility.


### Expected behavior

Text output similar to the below one (may or may not be exact)
```
# You can sort a list in Python by using the sort() method. Here's an example:\n\n```\nnumbers = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]\nnumbers.sort()\nprint(numbers)\n```\n\nThis will sort the list in place and print the sorted list.
```

Comments:
Could you share your accelerate version ? Might be outdated: https://github.com/huggingface/accelerate/issues/835

> Could you share your accelerate version ? Might be outdated: [huggingface/accelerate#835](https://github.com/huggingface/accelerate/issues/835)

0.20.3

@sgugger maybe for input ? Seems like `accelerate` issue but I cannot find anything relevant.

Could you provide us with the whole traceback? Also cc @muellerzr 

A full traceback is definitely needed here to know what logger is being init'd wrong 

> Could you provide us with the whole traceback? Also cc @muellerzr

```
 in <cell line: 2>:2                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py:201 in         │
│ __call__                                                                                         │
│                                                                                                  │
│   198 │   │   │   - **generated_token_ids** (`torch.Tensor` or `tf.Tensor`, present when `retu   │
│   199 │   │   │     ids of the generated text.                                                   │
│   200 │   │   """                                                                                │
│ ❱ 201 │   │   return super().__call__(text_inputs, **kwargs)                                     │
│   202 │                                                                                          │
│   203 │   def preprocess(self, prompt_text, prefix="", handle_long_generation=None, **generate   │
│   204 │   │   inputs = self.tokenizer(                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1120 in __call__          │
│                                                                                                  │
│   1117 │   │   │   │   )                                                                         │
│   1118 │   │   │   )                                                                             │
│   1119 │   │   else:                                                                             │
│ ❱ 1120 │   │   │   return self.run_single(inputs, preprocess_params, forward_params, postproces  │
│   1121 │                                                                                         │
│   1122 │   def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params):   │
│   1123 │   │   return [self.run_single(item, preprocess_params, forward_params, postprocess_par  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1127 in run_single        │
│                                                                                                  │
│   1124 │                                                                                         │
│   1125 │   def run_single(self, inputs, preprocess_params, forward_params, postprocess_params):  │
│   1126 │   │   model_inputs = self.preprocess(inputs, **preprocess_params)                       │
│ ❱ 1127 │   │   model_outputs = self.forward(model_inputs, **forward_params)                      │
│   1128 │   │   outputs = self.postprocess(model_outputs, **postprocess_params)                   │
│   1129 │   │   return outputs                                                                    │
│   1130                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py:1026 in forward           │
│                                                                                                  │
│   1023 │   │   │   │   inference_context = self.get_inference_context()                          │
│   1024 │   │   │   │   with inference_context():                                                 │
│   1025 │   │   │   │   │   model_inputs = self._ensure_tensor_on_device(model_inputs, device=se  │
│ ❱ 1026 │   │   │   │   │   model_outputs = self._forward(model_inputs, **forward_params)         │
│   1027 │   │   │   │   │   model_outputs = self._ensure_tensor_on_device(model_outputs, device=  │
│   1028 │   │   │   else:                                                                         │
│   1029 │   │   │   │   raise ValueError(f"Framework {self.framework} is not supported")          │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py:263 in         │
│ _forward                                                                                         │
│                                                                                                  │
│   260 │   │   │   │   generate_kwargs["min_length"] += prefix_length                             │
│   261 │   │                                                                                      │
│   262 │   │   # BS x SL                                                                          │
│ ❱ 263 │   │   generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=att   │
│   264 │   │   out_b = generated_sequence.shape[0]                                                │
│   265 │   │   if self.framework == "pt":                                                         │
│   266 │   │   │   generated_sequence = generated_sequence.reshape(in_b, out_b // in_b, *genera   │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115 in decorate_context       │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1572 in generate        │
│                                                                                                  │
│   1569 │   │   │   )                                                                             │
│   1570 │   │   │                                                                                 │
│   1571 │   │   │   # 13. run sample                                                              │
│ ❱ 1572 │   │   │   return self.sample(                                                           │
│   1573 │   │   │   │   input_ids,                                                                │
│   1574 │   │   │   │   logits_processor=logits_processor,                                        │
│   1575 │   │   │   │   logits_warper=logits_warper,                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:2619 in sample          │
│                                                                                                  │
│   2616 │   │   │   model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)  │
│   2617 │   │   │                                                                                 │
│   2618 │   │   │   # forward pass to get next token                                              │
│ ❱ 2619 │   │   │   outputs = self(                                                               │
│   2620 │   │   │   │   **model_inputs,                                                           │
│   2621 │   │   │   │   return_dict=True,                                                         │
│   2622 │   │   │   │   output_attentions=output_attentions,                                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in _call_impl            │
│                                                                                                  │
│   1498 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1499 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1502 │   │   # Do not call functions when jit is used                                          │
│   1503 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1504 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:165 in new_forward                   │
│                                                                                                  │
│   162 │   │   │   with torch.no_grad():                                                          │
│   163 │   │   │   │   output = old_forward(*args, **kwargs)                                      │
│   164 │   │   else:                                                                              │
│ ❱ 165 │   │   │   output = old_forward(*args, **kwargs)                                          │
│   166 │   │   return module._hf_hook.post_forward(module, output)                                │
│   167 │                                                                                          │
│   168 │   module.forward = new_forward                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py: │
│ 809 in forward                                                                                   │
│                                                                                                  │
│    806 │   │   """                                                                               │
│    807 │   │   return_dict = return_dict if return_dict is not None else self.config.use_return  │
│    808 │   │                                                                                     │
│ ❱  809 │   │   transformer_outputs = self.transformer(                                           │
│    810 │   │   │   input_ids,                                                                    │
│    811 │   │   │   past_key_values=past_key_values,                                              │
│    812 │   │   │   attention_mask=attention_mask,                                                │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in _call_impl            │
│                                                                                                  │
│   1498 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1499 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1502 │   │   # Do not call functions when jit is used                                          │
│   1503 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1504 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py: │
│ 674 in forward                                                                                   │
│                                                                                                  │
│    671 │   │   │   │   │   encoder_attention_mask,                                               │
│    672 │   │   │   │   )                                                                         │
│    673 │   │   │   else:                                                                         │
│ ❱  674 │   │   │   │   outputs = block(                                                          │
│    675 │   │   │   │   │   hidden_states,                                                        │
│    676 │   │   │   │   │   layer_past=layer_past,                                                │
│    677 │   │   │   │   │   attention_mask=attention_mask,                                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in _call_impl            │
│                                                                                                  │
│   1498 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1499 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1502 │   │   # Do not call functions when jit is used                                          │
│   1503 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1504 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:165 in new_forward                   │
│                                                                                                  │
│   162 │   │   │   with torch.no_grad():                                                          │
│   163 │   │   │   │   output = old_forward(*args, **kwargs)                                      │
│   164 │   │   else:                                                                              │
│ ❱ 165 │   │   │   output = old_forward(*args, **kwargs)                                          │
│   166 │   │   return module._hf_hook.post_forward(module, output)                                │
│   167 │                                                                                          │
│   168 │   module.forward = new_forward                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py: │
│ 315 in forward                                                                                   │
│                                                                                                  │
│    312 │   │   Tuple[torch.Tensor], Tuple[torch.Tensor, torch.Tensor], Tuple[torch.Tensor, torc  │
│    313 │   ]:                                                                                    │
│    314 │   │   residual = hidden_states                                                          │
│ ❱  315 │   │   hidden_states = self.ln_1(hidden_states)                                          │
│    316 │   │   attn_outputs = self.attn(                                                         │
│    317 │   │   │   hidden_states,                                                                │
│    318 │   │   │   layer_past=layer_past,                                                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in _call_impl            │
│                                                                                                  │
│   1498 │   │   if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks   │
│   1499 │   │   │   │   or _global_backward_pre_hooks or _global_backward_hooks                   │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1501 │   │   │   return forward_call(*args, **kwargs)                                          │
│   1502 │   │   # Do not call functions when jit is used                                          │
│   1503 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1504 │   │   backward_pre_hooks = []                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:160 in new_forward                   │
│                                                                                                  │
│   157 │                                                                                          │
│   158 │   @functools.wraps(old_forward)                                                          │
│   159 │   def new_forward(*args, **kwargs):                                                      │
│ ❱ 160 │   │   args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)                │
│   161 │   │   if module._hf_hook.no_grad:                                                        │
│   162 │   │   │   with torch.no_grad():                                                          │
│   163 │   │   │   │   output = old_forward(*args, **kwargs)                                      │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:282 in pre_forward                   │
│                                                                                                  │
│   279 │   │   │   for name, _ in named_module_tensors(                                           │
│   280 │   │   │   │   module, include_buffers=self.offload_buffers, recurse=self.place_submodu   │
│   281 │   │   │   ):                                                                             │
│ ❱ 282 │   │   │   │   set_module_tensor_to_device(module, name, self.execution_device, value=s   │
│   283 │   │                                                                                      │
│   284 │   │   return send_to_device(args, self.execution_device), send_to_device(                │
│   285 │   │   │   kwargs, self.execution_device, skip_keys=self.skip_keys                        │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/utils/offload.py:123 in __getitem__           │
│                                                                                                  │
│   120 │   │   self.prefix = prefix                                                               │
│   121 │                                                                                          │
│   122 │   def __getitem__(self, key):                                                            │
│ ❱ 123 │   │   return self.dataset[f"{self.prefix}{key}"]                                         │
│   124 │                                                                                          │
│   125 │   def __iter__(self):                                                                    │
│   126 │   │   return iter([key for key in self.dataset if key.startswith(self.prefix)])          │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/utils/offload.py:176 in __getitem__           │
│                                                                                                  │
│   173 │   │   │   │   raise ImportError("These offloaded weights require the use of safetensor   │
│   174 │   │   │                                                                                  │
│   175 │   │   │   if "SAFETENSORS_FAST_GPU" not in os.environ:                                   │
│ ❱ 176 │   │   │   │   logger.info("Enabling fast loading with safetensors by setting `SAFETENS   │
│   177 │   │   │   │   os.environ["SAFETENSORS_FAST_GPU"] = "1"                                   │
│   178 │   │   │                                                                                  │
│   179 │   │   │   from safetensors import safe_open                                              │
│                                                                                                  │
│ /usr/lib/python3.10/logging/__init__.py:1841 in info                                             │
│                                                                                                  │
│   1838 │   │   """                                                                               │
│   1839 │   │   Delegate an info call to the underlying logger.                                   │
│   1840 │   │   """                                                                               │
│ ❱ 1841 │   │   self.log(INFO, msg, *args, **kwargs)                                              │
│   1842 │                                                                                         │
│   1843 │   def warning(self, msg, *args, **kwargs):                                              │
│   1844 │   │   """                                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/accelerate/logging.py:51 in log                          │
│                                                                                                  │
│    48 │   │   `in_order` is ignored if `main_process_only` is passed.                            │
│    49 │   │   """                                                                                │
│    50 │   │   if PartialState._shared_state == {}:                                               │
│ ❱  51 │   │   │   raise RuntimeError(                                                            │
│    52 │   │   │   │   "You must initialize the accelerate state by calling either `PartialStat   │
│    53 │   │   │   )                                                                              │
│    54 │   │   main_process_only = kwargs.pop("main_process_only", True)                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: You must initialize the accelerate state by calling either `PartialState()` or `Accelerator()` before
using the logging utility.
```

Thanks, I can see where this stems from. As a temporary workaround while we fix the issue, your can set `SAFETENSORS_FAST_GPU=1` in your environment to avoid this error.

@sgugger we can disable that flag for more recent versions, it's not used anymore btw. (safetensors>0.3.0)

> Thanks, I can see where this stems from. As a temporary workaround while we fix the issue, your can set `SAFETENSORS_FAST_GPU=1` in your environment to avoid this error.

I tried this and I am no longer getting the error. However, the below line from the sample code shared earlier takes forever to execute. (waited for 16 minutes before interrupting it)
`outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.2, top_k=50, top_p=0.95, eos_token_id=49155)`

Any suggesstions on how I can debug it.

If you're trying to run this super large model on a small maching, everything will end up being offloading to CPU + DISK, making everything EXTREMELY slow. There's no easy solution on small hardware

Try setting `max_new_tokens=1`  and letting it run several minutes It should give you the next token.

Subsequent tokens are faster to get than the first one, however it should be of the same order of latency.

> If you're trying to run this super large model on a small maching, everything will end up being offloading to CPU + DISK, making everything EXTREMELY slow. There's no easy solution on small hardware
> 
> Try setting `max_new_tokens=1` and letting it run several minutes It should give you the next token.
> 
> Subsequent tokens are faster to get than the first one, however it should be of the same order of latency.

This worked and also helped understand the problem. Thank you so much.