Multi-threaded parallel inferrence problem
Author: zhaotyer
Pull request: no

### System Info

accelerate == 0.19.0
bitsandbytes == 0.37.1
- `transformers` version: 4.29.2
- Platform: Linux-3.10.0-1160.92.1.el7.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I use chatglm2/bloomz-7b for multithreaded parallel inference:

```
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from transformers import AutoTokenizer, AutoModelForCausalLM, LlamaForCausalLM, TextIteratorStreamer
import transformers
import torch
from threading import Thread, currentThread
import time
model = "/workspace/model-files/chatglm2"

tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model, device_map='auto', trust_remote_code=True, load_in_8bit=True)

def infer(prompt):
    inputs = tokenizer(prompt, return_tensors='pt')
    inputs = inputs.to(model.device)
    print('before generate')

    # out = model.generate(**inputs)
    # print('after generate')
    # out_text = tokenizer.decode(out[0])
    # print('out_text is:', out_text)
    t = currentThread()
    streamer = TextIteratorStreamer(tokenizer)
    generation_kwargs = dict(inputs, streamer=streamer, max_length=2048)
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()
    print("------******-------")
    for new_text in streamer:
        print("thread id:", t.ident ,"new text:",new_text)
    print("------******-------")

if __name__ == '__main__':
    prompt1 = '写一篇关于黄鹤楼的800字作文'
    prompt2 = 'Describe each state in the United States in detail'
    t1 = Thread(target=infer, args=(prompt1,))
    t2 = Thread(target=infer, args=(prompt2,))
    t1.start()
    time.sleep(5)
    t2.start()
    t1.join()
    t2.join()
```

I get below error info:
```
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 1515, in generate
    return self.greedy_search(
  File "/usr/local/lib/python3.8/dist-packages/transformers/generation/utils.py", line 2332, in greedy_search
    outputs = self(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 932, in forward
    transformer_outputs = self.transformer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 828, in forward
    hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 638, in forward
    layer_ret = layer(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 563, in forward
    mlp_output = self.mlp(layernorm_output)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/.cache/huggingface/modules/transformers_modules/modeling_chatglm.py", line 499, in forward
    output = self.dense_4h_to_h(intermediate_parallel)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/nn/modules.py", line 242, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
  File "/usr/local/lib/python3.8/dist-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.8/dist-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
    output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x4 and 2x4096)
```

### Expected behavior

able to inference normally

Comments:
Hi @zhaotyer, thanks for opening this issue! 

Could you edit the issue details to make sure that the code and traceback are properly formatted? This will make it easier for us to read and understand what's going on. 

The code examples should be wrapped around three backticks. You can also specify the language to get color coded formatting :) i.e. ` ```python CODE GOES HERE ``` `

Same for the traceback - it should go between a pair of three tickbacks ` ``` error message ``` `

cc @gante as this seems to be related to generate :) 

@zhaotyer that seems to be a bitsandbytes issue -- would you be able to update this library to its latest version and confirm whether the issue persists? :)

> Collaborator

Thanks for the reminder, I have edited it


> @zhaotyer that seems to be a bitsandbytes issue -- would you be able to update this library to its latest version and confirm whether the issue persists? :)
Should have nothing to do with bitsandbytes
The following are the test results of different transformers/accelerate versions
```
1.transformers==4.31.0 accelerate==0.21.0 bitsandbytes==0.37.1
 	1.1 chatglm2 load_in_8bit=true  singlethread normal, multithread normal
 	1.2 chatglm2 load_in_8bit=false singlethread normal, multithread normal
2.transformers==4.29.2 accelerate==0.19.0 bitsandbytes==0.37.1
	1.1 chatglm2 load_in_8bit=true singlethread normal, multithread have error(RuntimeError: mat1 and mat2 shapes cannot be multiplied)
 	1.2 chatglm2 load_in_8bit=false singlethread normal, multithread normal
```
@gante 

@zhaotyer OK, thanks for running with updated versions and reporting. If I've understood correctly, it looks like everything is running as expected on the most recent versions of accelerate and transformers and so the issue has been resolved in the most recent releases. 

> @zhaotyer OK, thanks for running with updated versions and reporting. If I've understood correctly, it looks like everything is running as expected on the most recent versions of accelerate and transformers and so the issue has been resolved in the most recent releases.

Can you explain what changes have been made between the two releases to solve this problem? thks

@zhaotyer The best way to find this is running `git bisect` to identify the commit and PR which resolved this for your example script. 

> @zhaotyer The best way to find this is running `git bisect` to identify the commit and PR which resolved this for your example script.

when i use pip install -U git+https://github.com/huggingface/transformers.git for #25228 , this multiplied problem still exists，use transformer==4.31.0 #25228 problem exists,So is there a version that solves both problems at the same time

@zhaotyer Just to make sure I've understood, could you confirm: 

1. Which versions work and which don't
2. Whether the same issues are being seen for the non-working versions? 

My understanding of the above comment is that: 
* Multi-thread issue exists for 4.29.2
* Multi-thread issue is resolved in 4.31.0
* Multi-thread issue returns when installing from source
* Another, separate, issue (#25228) occurs when using 4.31.0 but is resolved when installing from source

Is my understanding correct? 

> @zhaotyer Just to make sure I've understood, could you confirm:
> 
> 1. Which versions work and which don't
> 2. Whether the same issues are being seen for the non-working versions?
> 
> My understanding of the above comment is that:
> 
> * Multi-thread issue exists for 4.29.2
> * Multi-thread issue is resolved in 4.31.0
> * Multi-thread issue returns when installing from source
> * Another, separate, issue ([chatglm2 load_in_8bit=true can't reduce gpu memory when using transformer==4.31.0 #25228](https://github.com/huggingface/transformers/issues/25228)) occurs when using 4.31.0 but is resolved when installing from source
> 
> Is my understanding correct?

your understanding is correct

cc @younesbelkada  - any ideas on what might have changed on the bitsandbytes side? 

> cc @younesbelkada - any ideas on what might have changed on the bitsandbytes side?

Excuse me, is there any progress?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gentle ping @younesbelkada :)

Hi @zhaotyer 
Apologies for the delay, can you confirm the issue still persists if you install transformers from source now?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> till persists

I have tried almost all version combinations and this problem still persists.
Only when transformers==4.31.0, [Multi-thread issue ](https://github.com/huggingface/transformers/issues/25197)was resolved but the GPU memory was not reduced [# 25228](https://github.com/huggingface/transformers/issues/25228)