[OPT/Galactica] Load large `galactica` models
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes a small bug on `OPT`. Before, the `bias` term [was always set to `True`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py#L277) - leading to some external implementations to hardcode it if they wanted to train an OPT model without bias terms. See for example [here](https://github.com/paperswithcode/galai/blob/c1e16979c1748e7e823fe96da941d6df60f1006b/galai/architecture.py#L280). This PR aims to give more control on whether we should use or not `bias` terms on `Linear` layers of OPT.
The PR also fixes the same issue with `nn.LayerNorm`. Some derivatives of OPT does not use learnable parameters for layer norm's weights and biases (ie, set `elementwise_affine` to `False`), therefore avoids having hardcoded hacks in the future.
This PR should not be a breaking change as the default values of these booleans are set to `True` (as we were doing nothing)

This PR should also fix: https://huggingface.co/facebook/galactica-30b/discussions/4 (ofc, after updating the relevant config files)

cc @sgugger @ydshieh @mrm8488 

All slow tests pass


Comments:
_The documentation is not available anymore as the PR was closed or merged._

I am not 100% sure if this is the approach we want to have, despite I can understand the intention. Would like to hear from @sgugger.

For reference, `class OPTDecoderLayer` from `galai` does pass `bias` to `OPTAttention`

https://github.com/paperswithcode/galai/blob/c1e16979c1748e7e823fe96da941d6df60f1006b/galai/architecture.py#L280

Yes, I think it was a mistake from our side. We should either port a new model (with controlable bias and layer norm) and remove the `bias` boolean from `OPTAttention` as it is always set to `True` or go with this fix

Thanks! 
Sorry for the last minute clarification as I just realized that the description and title are not clear, but the main goal of this PR is to support loading and using large `galatica` models that uses `OPT` architecture, initially reported in: https://huggingface.co/facebook/galactica-30b/discussions/4  / therefore the title + description is slightly misleading
The snippet to reproduce: 
```
import torch
from transformers import AutoTokenizer, OPTForCausalLM, AutoModel

tokenizer = AutoTokenizer.from_pretrained("facebook/galactica-30b")
model = OPTForCausalLM.from_pretrained("facebook/galactica-30b", device_map="auto", torch_dtpye=torch.float16)

input_text = "The Transformer architecture [START_REF]"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
```
In case we don't merge this PR we may be want to add `galatica` as a separate new architecture as some `galactica` models (such as `30b`) does not use `bias` on linear layers and don't have any learnable weights on their `LayerNorm`

I understood, and yes, that will be the alternative if this PR is declined :-)

Hi @sgugger and @younesbelkada, it's one of the Galactica authors here. We think that there might be something wrong with the 30bn model specifically on HuggingFace. We're currently migrating our galai library to use the huggingface model without our custom OPT config. There seems to have been a conversion process applied to our models to give null weights to the biases (or something else similar to the OPT models), but specifically not on the 30bn file. Hopefully, this can be resolved without a PR by fixing the model file. See the great investigations done by @Jackmin801  on this ticket https://github.com/paperswithcode/galai/issues/37#issuecomment-1323929437

> For reference, `class OPTDecoderLayer` from `galai` does pass `bias` to `OPTAttention`
> 

Hi @ydshieh, the `bias` flag is passed only so that `Galactica` extension of `OPT` architecture is backward compatible. We set all the additional config parameters to the values used by `OPT` (see https://github.com/paperswithcode/galai/blob/main/galai/config.py#L92-L95) so that `OPT` checkpoints work as before, but we set them accordingly in `Galactica` configs (see f.e., https://huggingface.co/mrm8488/galactica-125m/blob/main/config.json#L18). Whether these changes should be ported back to `modeling_opt` or the `Galactica` should be forked-out from it depends on how much it deviates from the general philosophy of Transformers as @sgugger noted.

Hi @AnthonyHartshorn 
Thanks a lot for your message. Indeed, big kudos to @Jackmin801 for the investigation, his investigation in https://huggingface.co/facebook/galactica-30b/discussions/4#637e90571dbae0919104b582 helped me define the rootcause of the bug. 
I guess it can be also fixed by saving zero bias and ones for the layer norms, updating the weights on the hub with the new ones can do the trick too yes.

As @sgugger said above, this goes very clearly against the foundation of `transformers` to add configurable parameters to a previous model architecture to support a new model architecture.

However, fixing this any other way would result in some setups breaking in the wild; it would require us to update the architecture name to `galactica` instead of `opt` which would break every existing setup that currently uses these models unless the upgrade to the latest version. 

Given that, I'm also inclined to accept this change even if it goes against our design decisions. If we could do it all over again however, I would heavily push for a new model architecture.

Thanks @LysandreJik for approving this PR. I have another related question. As pointed by Jackmin801 in the comment linked above by Anthony (https://github.com/paperswithcode/galai/issues/37#issuecomment-1323929437), almost all of the checkpoints were converted from our float16 checkpoints and uploaded to the hub in full float32 precision (except for 30B which is an exact copy). That's not the best for user experience: download time, disk usage and loading time doubles for no benefit. I wonder if we can fix it, there are couple options I see:

* upload our float16 checkpoints once this PR is merged. This would not be backward compatible as this PR is required,
* do the same conversion that @mrm8488 did, but `.half()` the models before exporting. This would be almost backward compatible except for the case when a user doesn't specify `pytorch_dtype` when loading a model, as after that the models would load as float16 by default,
* keep the existing checkpoints, potentially fix the 30B to be float32 as well for consistency (it wasn't working before this PR anyway). Not the best user experience,
* add new checkpoints galactica-125m-fp16, ..., galactica-120b-fp16. Might be too confusing for users.

What do you think? I'm in favor of the second option as it's the best for backward compatibility and user experience.

PyTorch automatically converts checkpoint weights to the dtype of the model when you load the state_dict, so option 2 is actually 100% backward compatible.

Thanks @sgugger, I missed the fact that `torch_dtype` is part of `config.json`.

This PR is ok for me - galactica is build on top of OPT so one could fine-tune OPT using these two configs => so this PR is def ok for me

Thanks everyone! 
@mkardas @mrm8488  : https://huggingface.co/facebook/galactica-30b/discussions/5 since now this PR has been merged, can you merge this PR to fix the initial issue for `30b` ? 

@younesbelkada I'm not a member of the org yet. I've verified my work email address, but wasn't auto-added. How can I learn who are the admins?

@patrickvonplaten can you add me to https://huggingface.co/facebook (same username)?

I can merge the PR if this is the only thing needed! 🤗 


Thanks @ArthurZucker. I was working on providing float16 weights in the backward compatible way as discussed above. I think it's best to just fix all the checkpoints to make them float16 and keep zero biases for backward compatibility with HF 4.21.0-4.24.0. I'm in the middle of preparing a new HF hub PR for this, I'll let you know in case I won't be able to merge it.

@sgugger
From my tests on backward compatibility, it seems that calling `OPTForCausalLM.from_pretrained` with `torch_dtype=None, device_map=None` results in `float32` weights regardless of what's in the checkpoint bin files and `config.json`. However, `torch_dtype=None, device_map="auto"` results in the same weights type as in the checkpoint bin files, regardless of `config.json`. Is it to be expected?

I think this is expected as if you want to load a model natively without using `accelerate` (i.e. without adding `device_map="auto"`), `transformers` will automatically load the weights in fp32, in this case whenever you want to load a model with its native dtype of the weights you need to use `torch_dtype="auto"`. 

Mmm, no. If it is indeed the case then it's a bug. Do you have a small reproducer/repo ID I could look at?

This is what I used:

```python
import torch
from transformers import OPTForCausalLM

for device_map in [None, "auto"]:
    for dtype in [None, torch.float16, torch.float32]:
        model = OPTForCausalLM.from_pretrained(
            "facebook/galactica-125m",
            revision="refs/pr/6",
            torch_dtype=dtype,
            device_map=device_map
        )
        print(f"[device_map={device_map}]: {dtype} -> {model.lm_head.weight.dtype}")
    print()
```

What I get for `refs/pr/6` (which has `torch_dtype=float32` in config.json and `float16` bin files):

```
[device_map=None]: None -> torch.float32
[device_map=None]: torch.float16 -> torch.float16
[device_map=None]: torch.float32 -> torch.float32

[device_map=auto]: None -> torch.float16
[device_map=auto]: torch.float16 -> torch.float16
[device_map=auto]: torch.float32 -> torch.float32
```

For `facebook/opt-125m` the output is the same, even though `opt-125m` has `float16` both in config.json and bin files.

PRs replacing the existing `float32` checkpoints with `float16` checkpoints:

https://huggingface.co/facebook/galactica-125m/discussions/6
https://huggingface.co/facebook/galactica-1.3b/discussions/6
https://huggingface.co/facebook/galactica-6.7b/discussions/8
https://huggingface.co/facebook/galactica-30b/discussions/6
https://huggingface.co/facebook/galactica-120b/discussions/6

Found the issue. The PR mentioned above should make the result consistent between `device_map=None` and `device_map="auto"`.