`device_map="auto"` support multi-node
Author: guozhiyao
Pull request: no

### Feature request

`AutoModel.from_pretrained(model_dir, device_map="auto", trust_remote_code=True).half()`
I want to load a huge model in multi-node for inference, such as 4 node with 1 gpu per node. But I do not know how to do it. The `device_map="auto"` seems only work for one node.

### Motivation

I want to test the long-context ppl. When I increase the context, the gpu memory increase too. So I need more node to do the inference.

### Your contribution

not yet.

Comments:
cc @younesbelkada 

Hi @guozhiyao, 

Thanks for raising this issue. At the moment, there isn't enough information for us to be able to help you. Could you specify what you mean by "doesn't work"? 

As a side note, I don't think you can do `device_map=auto"` and then `.half()`, instead you can pass in a flag to `from_pretrained` to specify the precision e.g. `torch_dtype=torch.float16` or `load_in_8bit=True`. @younesbelkada  can confirm :) 





hi @guozhiyao 
thanks for raising this up ! 
firstly as @amyeroberts suggested, the canonical way of loading a model with a specific dtype (in your case `half`=`torch.float16`) is by passing `torch_dtype=torch.float16` thus you avoid any unexpected issue you may encounter

Regarding your second question, I don't think this is supported by `device_map="auto"` for inference. Usually the multi-node paradigm is useful for training, where you have an entire training process running independently on each node. I think accelerate supports multi-node training (you can select mutli node training when running `accelerate config` and we have made some training process work under multi-node regime using accelerate internally). 

However I doubt that you can run multi-node inference out of the box with `device_map='auto'` as this is intended only for single node (single / multi GPU or CPU only). In multi-node setting each process will run independently `AutoModel.from_pretrained(model_dir, device_map="auto", trust_remote_code=True).half()` thus the model will not be shared across both processes.
I am also unsure about the benefits of such protocol - the only case it might be interesting to see this would be if someone wants to fit a model that can't fit in more than a node (more than 8xA100 80GB at most)

Would like also to hear from @sgugger or @muellerzr , in case I missed something I am not aware of.

I can confirm that multi-node is not supported by `device_map="auto"` :-) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not everyone has access to 8xA100, so I think there is benefit to support multi-node inference (e.g. if someone has access to multi-node V100s setup but not A100, then this feature would allow them to try a big model like Llama-2-80B on their V100s. Correct me if I'm wrong, but at present, there doesn't appear to be a straightforward method for running inference with models of the scale of Llama-2-80B on multi-node V100 setups)

@runopti just to be clear, our language here is multinode == multiple *computers*, not multinode == multiple GPUs on the *same computer*, correct?

@muellerzr Yes, what I mean is multinode == multiple computers. 

I'd recommend opening a feature request on the Accelerate github: https://github.com/huggingface/accelerate

Any progress on this? I have access to many nodes with 2 A100s each, but cant fit Llama 3.1 405B. I think I would be able to fit it into 6 of such nodes

Hi I also have the exact same question -- I followed the HF accelerate tutorial and was able to use device_map="auto" for (relatively) smaller models (upto 8B) but for example to run inference on 70B model using A40s I believe I need two nodes -- any help with this would be greatly appreciated !!! 🙏🙏🙏