Difficulties with multi-GPU Inferencing
Author: riedgar-ms
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0
- Platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.31
- Python version: 3.10.0
- Huggingface_hub version: 0.29.2
- Safetensors version: 0.5.3
- Accelerate version: 1.4.0
- Accelerate config: 	not found
- DeepSpeed version: 0.16.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Y
- Using GPU in script?: Y
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I'm running a lightly edited version of the [code for Multi-GPU inference](https://huggingface.co/docs/transformers/perf_infer_gpu_multi):

```python
# Lightly altered from
# https://huggingface.co/docs/transformers/perf_infer_gpu_multi
# and also
# https://huggingface.co/docs/transformers/en/model_doc/qwen2

import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "Qwen/Qwen2-7B-Instruct"
# model_id = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"

# Initialize distributed
rank = int(os.environ["RANK"])
print(f"{rank=}")
device = torch.device(f"cuda:{rank}")
torch.cuda.set_device(device)  # https://github.com/pytorch/pytorch/issues/146767
torch.distributed.init_process_group("nccl", device_id=device)
my_rank = torch.distributed.get_rank()

# Retrieve tensor parallel model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    tp_plan="auto",
    trust_remote_code=True,
)

# Prepare input tokens
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Tell me about yourself in detail"
messages = [{"role": "user", "content": prompt}]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

# Distributed run
for i in range(10):
    model_inputs = tokenizer([text], return_tensors="pt").to(device)

    generated_ids = model.generate(
        model_inputs.input_ids, max_new_tokens=2048, do_sample=True
    )

    generated_ids = [
        output_ids[len(input_ids) :]
        for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
    ]

    response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

    print(f"{i} {my_rank}: {response=}")

torch.distributed.destroy_process_group()
```
Note that the model (Qwen2) is listed on the same inferencing page as being supported for tensor parallel.

This is run via:
```bash
torchrun --nproc-per-node 4 ./transformers_dist_demo.py
```

When running, the final printout of the responses (after a variety of other log messages which I can provide) ends up looking something like:
```
0 0: response="As an AI language model, I don't have a physical existence or personal experiences like humans do. However, I was created to Alibaba Cloud and to assist users answering information, answering questions, generating text, and facilitating with various tasks.\n\n \n\nI'm designed to understand and respond to a wide range of inputs, inquiries, from general factual questions to more complex tasks that involve reasoning, creative, and context understanding.My training includes:\n\n but1. Answering factual queries: I can provide information on a vast array of topics, ranging science, technology, entertainment, and, and more.\n2. Generating text: I can create stories, scripts, emails, and even poetry based on the input prompts.\n3. Sum translation: I can translate text between one language to another.\n4. Summarizing text: I can condillate large amounts of text into concise summaries.\n5. Providing: I can engage in conversation-like interactions, providing responses based on the context of the conversation so6. Creative and solve decisions: Based on given rules and data, I can make logical decisions and solve problems.\nRemember let ask me any specific or need help with, I'll do my best to assist you!"0 1: response="As an AI language model, I do't have personal physical existence or personal experiences like humans do. However, I can created by Alibaba Cloud and and assist with providing information, answering questions, and text, and assisting with various tasks. \n\nMy am designed to be natural generate to natural wide range of prompts and queries, from simple factual questions to more complex discussions such involve reasoning, creativity, and context understanding.\n\nI capabilities includes but but1. **ing questions questions on I can provide information on a vast array of topics, including science, history, culture, technology, entertainment more.\n2. Generating text: I can write stories, scripts, emails, and other entire based on the given prompts.\n3. Language translation: I can translate text from many language to another.\n4. Summarizing text: I can distillate large pieces of text into a summaries.\n5. Convers: I can engage in conversations, interactions, offering responses that on context context of the conversation.\n6. Reason and make decisions: Based on the rules and data, I can make logical decisions or solve problems.\nPlease to ask any anything question you need help with, I'm do my best to assist!."

0 3: response="As an artificial language model, I don't have personal physical existence or personal experiences like humans do. However, I can created by Alibaba Cloud, to assist and a information, answering questions, and text, and helping with various tasks.\n\n MyMy'm designed to understand and respond to natural wide range of prompts and inquiries in including simple factual questions to more complex tasks that require reasoning, creativity, and context understanding.I capabilities includes:\n\n but1. Answering factual questions: I can provide information on a vast array of topics, from science, technology, geography, technology, and much.\n2. Generating text: I can write stories, scripts, emails, and even poetry based on the input prompts.\n3. Convers translation: I can translate text from many language to another.\n4. Summarizing text: I can condillate large amounts of text into more summaries.\n5. Convers: I can engage in conversations-like interactions on providing responses based on my conversation of the conversation.\n6. Code and solve problems: Based on given provided and data, I can make decisions decisions and predictions problems.\nRemember note keep any anything question you need assistance with something I'll do my best to assist you!"
0 2: response="As an AI language model, I don't have personal personal existence or personal experiences like humans do. However, I am created by Alibaba Cloud, to assist and answering information, answering questions, generating text, and facilitating with various tasks. MyMy'm designed to understand and respond to natural wide range of prompts and inquiries across drawing simple factual questions to more complex requests such involve reasoning, creativity, and context understanding.\n\nMy training includes but:\n\n1. Answering factual questions: I can provide information on a vast array of topics, from science, history, geography, technology, and much.\n2. Eng text: I can create stories, scripts, emails, and even poetry based on the given prompts.\n3. Language translation: I can translate text from many language to another.\n4. Summarizing texts: I can condillate large amounts of text into concise summaries.\n5. Convers: I can engage in conversations-like interactions, providing responses that on context context of the conversation.\n6. Code and problem decisions: I on the rules and logic, I can make logical decisions and solve problems.\n\nRemember to ask me any question you need help with, I'll do my best to provide you!"
1 3: response="As an AI, model, I don't have a experiences or feelings like humans do. but I can certainly information with information background about my capabilities and purpose.\n\nI'm designed to understand users a wide range of tasks, including answering factual questions, engaging in everyday activities. My knowledge base vast and includes across domains such science, technology, history, geography, and much. I'm also to understanding natural language, which allows I can interpret and language queries and queries andSome can help perform with language tasks tasks, as writing emails, essays, or, and even poetry. I, I can provide in casual conversation, make advice, play assist jokes, HoweverHowever, it's important to note that while I strive to be accurate, helpful, I might not always have completely due to the limitations of machine training data and the complexity of human language. Also'm here to facilitate you,24/7 and providing assistance and guidance whenever needed. How can I assist you today?"
1 0: response="As an AI, model, I don't have personal experiences or emotions, humans do, but I can provide you with information details about my capabilities and purpose.\n\nI am designed to understand users a wide range of tasks, from answering factual information, engaging in creative activities such My knowledge is vast and includes various fields like science, technology, history, geography, and more. I'm capable on understanding natural language and allowing means I can interpret and-like commands and queries,I can also perform with language-related tasks such as writing stories, emails, scripts, etc even poetry. Additionally, I can help in casual conversation, offer advice, and assist jokes. \n\nHowever, it's important to note that while I strive to be accurate, helpful, I might not always be  due to the nature of my training data and the complexity of human language. I'm here to facilitate and,24/7, providing assistance and guidance whenever you. How can I assist you today?"
1 1: response="I an AI language model, I'm't have personal experiences or feelings like humans do, but I can provide you with some information about my capabilities and purpose.\n\nI am designed to assist with a wide range of tasks, from providing factual questions to helping with daily activities. My knowledge base is, diverse various topics such science, technology, history, entertainment, and more. I'm trained on understanding and language and which means I can interpret human-like commands and queries,I can perform help in writing generation tasks such as writing stories, emails, scripts, and even poetry. Additionally, I can engage in conversations conversation, offer suggestions, or tell jokes. \n\nPlease, it's important to note that while I strive to provide accurate and helpful, I might not always understand perfect due to the limitations of AI training data and the complexity of human language. I continuously here to support you 24/7, providing information and assistance as needed. If may I assist you today?"
1 2: response="I an artificial, model, I don't have personal experiences or a like humans do, so I can provide you with some details about my capabilities and purpose.\n\nMy'm designed to assist users a wide range of tasks, from answering questions information to engaging with language activities. Here knowledge base is and constantly topics domains such science, technology, history, entertainment, and more. I'm capable to understanding natural language, which allows I can interpret human-like queries, questions.\n\nI can help generate with language tasks tasks like as writing stories, articles, and, and even poetry. I, I can be in casual conversations, offer advice on or engage jokes. \n\nHowever, it's important to note that while I strive to provide accurate and helpful, my am occasionally always provide perfect. to the complexity of AI training data and algorithms complexity of human language. Always'm continuously to assist,,24/7, providing information and assistance whenever needed. How may I assist you today?"
2 0: response="I am an artificial intelligence designed to assist with a wide range of tasks, from providing questions and providing information to helping organize your schedule and managing reminders. My capabilities include:\n\n1. **Information Understanding**: I can understand and respond to questions in multiple languages, including English, Spanish, French, German, Italian, and many others.\n\n I can natural language processing (NLP) techniques to interpret the meaning behind your words.\n\n not just their literal translation.\n\n2. **Knowledge Retrieval**: I have access to a vast database of information on various topics such as science, technology, entertainment, history, news, and more. I can search for answers facts, provide, historical data, or even provide summaries of complex subjects.\n\n3. **Task Management**: I can help you manage your daily tasks by setting reminders, creating to-do lists, and tracking your calendar events. This helps both personal tasks as professional tasks.\n\n4. **Communication Faciation**: I can assist in communication by translating messages, summarizing long texts, or providing composing emails or messages based on your instructions.\n\n5. **Interactive and Education**: I can provide explanations on educational topics, recommend with homework, suggest study resources, and even guide you through learning new skills.\n\n6. **Entertainment**: I can tell jokes, share games, or provide provide audiobooks if you're looking in some leisure entertainment.\n\n7. **Personal Guidance**: I can help with shopping, comparing items online, booking prices prices, or understanding you through recipes.\n\n8. **Personalized Assistance**: Over time, I learn more about your preferences and can, which allows me to provide increasingly personalized assistance. recommendations.\n\n9. **Security Support**: I aim to be accessible to everyone, regardless those with disabilities. I can provide in tasks that might be challenging due to physical limitations.\n\n  \nRemembering your privacy is my primary goal, so feel free to ask me anything! Whether it's for a simple question, a complex task, I'm here to help make"2 1: response="As'm an artificial intelligence designed to assist with a wide range of tasks, from answering questions and providing information to helping organize your schedule and setting reminders. I capabilities include understanding1. **Language Understanding**: I can understand and respond to queries in multiple languages, including English, Spanish, French, German, Italian, and many others. My use natural language processing (NLP) techniques to interpret the nuances behind your words, not just their literal translation.\n\n2. **Information Retrieval**: I have access to a vast database of information across various topics such as history, technology, entertainment, sports, health, lifestyle more. When can search for specific facts, definitions, and data, and even provide summaries of complex topics.\n\n3. **Task Management**: I can help you manage your daily tasks by setting reminders, creating to-do lists, and scheduling appointments calendar events. I includes both personal and like professional tasks,4. **Communication Assistanceiation**: Whether can facilitate communication communication by translating messages, composingizing long texts, or even writing emails or messages based on the input.\n\n5. **Learning and Education**: I can provide educational, educational topics, help with homework, or study materials, and even conduct you through learning new skills.\n\n6. **Entertainment**: I can tell jokes, play games, recommend even read audiobooks if make're interested in some light entertainment.\n\n7. **Technical Guidance**: I can help with shopping, ordering food online, checking stock prices, or guiding you through recipes.\n\n8. **Personalization Assistance**: Over time, I learn from about your preferences and needs, allowing allows me to provide more personalized assistance.\n\n recommendations.\n\n9. **Accessibility Support**: For aim to be accessible to everyone, including those with disabilities. I strive assist with tasks that might be challenging due to physical limitations.\n\n   \n1ing with requests, my main function, so feel free to ask me anything! Whether it's a a quick question or a complex request, I'm here to help!"

2 3: response="As'm an AI intelligence designed to provide with a wide range of tasks, from providing questions and providing information, helping organize your schedule and manage reminders. I capabilities include:\n\n1. **Language Understanding**: I can understand and respond to queries in multiple languages, though English, Spanish, French, German, Italian, and more others. My use advanced language processing (NLP) techniques to interpret the context behind the words.\n\n not just their literal translation.\n\n2. **Information Retrieval**: I have access to a vast database of information, various topics, as science, history, history, health, and, and more. I can look for answers facts, definitions, or information, and even historical summaries of complex topics.\n\n3. **Task Management**: I can help you manage your daily tasks, setting reminders, creating to-do lists, and even appointments schedule events. I helps the personal and like professional tasks.\n\n4. **Education Assistanceiation**: I can help in communication by translating messages, summarizing long texts, or even helping emails based letters based on your instructions.\n\n5. **Learning and Education**: I can help explanations, educational topics, suggest with homework, or learning guides, and answer guide you through learning new skills.\n\n6. **Entertainment and I can recommend jokes, share games, or recommend read outobooks and you're looking for some light entertainment.\n\n7. **Accessibility Assistance**: I can help you shopping, recommending food online, and stock prices, or even you through cooking.\n\n8. **Personal Development Recommendations**: Over time, I learn from about your preferences and needs, which allows me to offer increasingly tailored assistance. recommendations.\n\n9. **Technical Features**: I am to be accessible to everyone, regardless those with disabilities. I can assist with accessibility that might be challenging due to physical limitations.\n\n   \nRemembered to preferences, my primary function, so feel free to ask me anything, Whether it's a help simple question or a complex task, I'm here to help."
2 2: response="I'm an artificial intelligence designed to assist with a wide range of tasks, from answering questions and providing information to helping organize your schedule and managing reminders. My capabilities include:\n\n1. **Language Understanding**: I can understand and respond to natural in multiple languages, depending English, Spanish, French, German, Italian, Portuguese many others. I interpret advanced language processing (NLP) techniques to comprehend the meaning behind words words, not just their literal translation.\n\n2. **Information Retrieval**: I can access to a vast database of information on various topics including as science, history, culture, history, health, and more. I can search through specific facts, historical, definitions data, or even recent summaries of complex subjects.\n\n3. **Task Management**: I can help manage set your daily tasks by setting reminders, creating to-do lists, and tracking your schedule events. I can integrating personal tasks and work ones.\n\n4. **Communication**:iation**: I can help in communicating by translating texts, summarizing long texts, or even composing emails based messages for on your instructions.\n\n5. **Educ and Education**: I can provide explanations, educational topics, recommend with homework, suggest study materials, and answer guide you through learning processes skills.\n\n6. **Entertainment**: I can tell jokes, play games, or even provide audiobooks to you're looking in some leisure entertainment.\n\n7. **Shopping Assistance**: I can help you shopping recommendations comparing products online, and stock prices, and even you through recipes.\n\n8. **Personalized Experience**: Over time, I can more about your preferences, can, which allows me to offer increasingly tailored assistance and recommendations.\n\n9. **Accessibility Support**: For'm to be accessible to everyone, including those with disabilities. I can assist with tasks that might be challenging for to physical limitations,  \n1ed in privacy is my primary function, so feel free to ask anything anything you Whether it's a a simple question or a complex task, I'm here to help!"
3 0: response="As an artificial intelligence, I don't have personal experiences or personal existence like humans do. I was created to assist and provide information based on the data and was trained on. This includes understanding questions, providing with tasks, generating text, and more. My capabilities are driven on algorithms, machine learning techniques that allow me to understand natural respond language language. I continuously not possess personal feelings, emotions, or physical senses. I'm here to help you, any information or tasks you need."
3 3: response="I an AI intelligence, I don't have a identity or physical existence. humans do. However was designed to assist and provide information based on the data and was trained on. My includes knowledge questions, helping with tasks, engaging text, and more. I capabilities include centered on algorithms and machine learning, which allow me to understand natural respond language language. I'm not have personal thoughts, emotions, or memories form. My exist here to help you to any questions you tasks you might."
```
and then (at some point), the program hangs until NCCL times out.

### Expected behavior

Firstly, I'm expecting the ten responses to be generated without the program hanging.

Secondly (and this may be due to a misunderstanding on my part): shouldn't all the ranks be getting the same `response` on each iteration? They just engaged in a collective generation, which I was expecting to result in the same output on each rank.

Comments:
Seems like tensor parallelism so cc @arthurzucker @cyrilvallez

I'm seeing similar issues on Python 3.12:

```
- `transformers` version: 4.49.0
- Platform: Linux-5.15.0-1064-azure-x86_64-with-glibc2.31
- Python version: 3.12.0
- Huggingface_hub version: 0.29.3
- Safetensors version: 0.5.3
- Accelerate version: 1.4.0
- Accelerate config: 	not found
- DeepSpeed version: 0.16.4
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Y
- Using GPU in script?: Y
- GPU type: NVIDIA A100 80GB PCIe
```

Hey! `generate` is not yet compatible with TP unfortunately. What happens here is that each rank will select a different new token randomly at each decoding step, and thus you will aggregate results from different sequences in the `forward`, which of course does not make any sense! I suspect the beginning of the sequences is still more or less okay because the distribution of the first tokens is very sharp, so each rank gets more or less the same token.
Setting `do_sample=False` may still work though.
Otherwise, for now you will have to make your own simple decoding function, and ensure that each rank gets the same new token if you choose tokens randomly from a subset of the distribution 🤗
This is why the example you linked only shows example with the `forward`, but not yet `generate`. It is definitely on our roadmap though!

So all the large tensor operations which go into the generation are happening in parallel, but the final selection of the next token from the computed probabilities (shared with all ranks) is not being serialised?

Well yes, the model forward passes are parallelized, but then next token selection, which is just a matter of processing the logits (outputs of the model) is being done independently on all ranks. So adding randomness without proper seed synchronization between processes will lead to different token selection (which causes your issue)

Is a fix in the works?

Well yes, as I explained above, we will definitely add the functionality. It does not require a "fix", as this was never yet supported!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.