[generate] Run custom generation code from the Hub
Author: gante
Pull request: yes

# What does this PR do?

This PR expands our modeling code on the hub support: it enables the definition of custom decoding methods on the hub, sharing the same interface as `generate`.

## How does it work?

Assumptions:
📖  custom decoding methods are defined in *model repos* 
✍️  the custom generation code is defined in `custom_generate/generate.py`  
🔎 these repos have a `custom_generate` tag so we can quickly find them  

The custom generation code defined in the repo `"foo/bar"` can be accessed in two ways:
1. If we load `model = AutoXXX.from_pretrained("foo/bar")`: `model.generate` will always call the custom generate defined in `"foo/bar"`;
2. Any other `model`: we can specify the repo to load the custom generate from, i.e. `model.generate(..., custom_generate="foo/bar")`

(for more info, read the added documentation)

## Expected benefits

🧠 Creators of new decoding methods:
- No need to go through a lengthy PR review cycle;
- Day 0 support of their techniques, no need to wait for a `transformers` release;
- Same `model.generate` interface, but much higher implementation freedom 
- Users don't need to install a new Python repo in their environment
- Hub features (custom readme, discussion pages, track downloads, ...)

🔨 `transformers` team:
- No more gatekeeping (technique A gets added, technique B doesn't get added)
- More generation methods, lower expected maintenance 🤞 
- Empower our community 💛 


## Working Example

([generate.py in transformers-community/custom_generate_example](https://huggingface.co/transformers-community/custom_generate_example/blob/main/custom_generate/generate.py))

```py
from transformers import AutoModelForCausalLM, AutoTokenizer

# Qwen/Qwen2.5-0.5B-Instruct copy, but with custom generation code -> `generate` uses the custom code
# note: calling the custom method prints "✨ using a custom generation method ✨"
tokenizer = AutoTokenizer.from_pretrained("transformers-community/custom_generate_example")
model = AutoModelForCausalLM.from_pretrained("transformers-community/custom_generate_example", device_map="auto", trust_remote_code=True)

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

# `generate` with `custom_generate` -> `generate` uses custom code
# note: calling the custom method prints "✨ using a custom generation method ✨"
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, custom_generate="transformers-community/custom_generate_example", trust_remote_code=True)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))
```

Sanity checks, copy paste after the script above:
```py
# Sanity check: `transformers-community/custom_generate_example` contains a simplified greedy decoding method
custom_generated_text = tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]
gen_out = model.generate(**inputs, do_sample=False, repetition_penalty=1.0, max_length=20)
greedy_generated_text = tokenizer.batch_decode(gen_out, skip_special_tokens=True)[0]
assert custom_generated_text == greedy_generated_text

# Sanity check 2: `trust_remote_code` is required
try:
    model.generate(**inputs, custom_generate="transformers-community/custom_generate_example")
except Exception as e:
    print("got exception 👍")
else:
    raise Exception("This should have failed -> no `trust_remote_code`")

try:
    model = AutoModelForCausalLM.from_pretrained("transformers-community/custom_generate_example", device_map="auto")
except Exception as e:
    print("got exception 👍")
else:
    raise Exception("This should have failed -> no `trust_remote_code`")
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36405). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.