AttributeError: Can't get attribute 'SiLUActivation' on <module 'transformers.activations' 
Author: Lokesh-Jatangi
Pull request: no

### System Info

System info - 

- `transformers` version: 4.36.2
- Platform: Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

I am using a custom script which loads LLAMA checkpoint through torch. 
`model_orig = torch.load(checkpoint_path)`

While unpickling  checkpoints in torch "SiLUActivation" class is missing from activations.py.
This PR https://github.com/huggingface/transformers/pull/27136  removed the SiLUActivation class mentioning it was reduntant.

P.S :- With transformers version 4.35.0 , loading a checkpoint through torch containing SiLU activation layer was succesful.

Find the below trace :-  

` line 65, in load_model_from_checkpoint
    model_orig = torch.load(checkpoint_path)
  File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1014, in load
    return _load(opened_zipfile,
  File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1422, in _load
    result = unpickler.load()
  File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1415, in find_class
    return super().find_class(mod_name, name)
AttributeError: Can't get attribute 'SiLUActivation' on <module 'transformers.activations' from '/opt/conda/envs/adapt/lib/python3.10/site-packages/transformers/activations.py'>`

I would happy to add it the SiLU class back to activations.py file and submit it here. Please let me know if i can proceed .

### Who can help?
@amyeroberts 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Any model which has SILU activation function and loaded through "torch.load()" will face this issue.

### Expected behavior

After adding reverting back the changes , the torch should be able identify SiLU activation class.

Comments:
Hi @Lokesh-Jatangi, thanks for raising this issue! 

Is there a reason you're using `torch.load` here? The officially supported way to load checkpoints is through the `from_pretrained` method. 

The checkpoint stores a pruned model whose structure and weights are different and hence I couldnot use `from_pretrained` method.

@Lokesh-Jatangi   Have you solved your problem?

@Lokesh-Jatangi We can't guarantee backwards compatibility for a checkpoint which isn't a transformers architecture and isn't loaded through the officially supported API. In order to be able to maintain the repo, there will be objects which we'll move, rename and delete and so pickling in this way may cause issues. 

I'd suggest loading the model on the most recent compatible version of transformers. Updating the model to use torch's silu activation implementation and then resave the model out. I _think_ this should resolve the issue and allow you to load in the model in more recent transformers versions again. 

> @Lokesh-Jatangi Have you solved your problem?

I add this to "site-packages/transformers/activations.py" ,it works
import torch.nn.functional as F
class SiLUActivation(nn.Module):
    def forward(self, input: Tensor) -> Tensor:
        return F.silu(input)