Transformers LocalAgent - Error with GPT model, are there any specific limitations for LocalAgent?
Author: gidzr
Pull request: no

### System Info

cpu, latest version transformers, python 3.10

### Who can help?

@ArthurZucker, @younesbelkada and @Narsil

Hey there

I am using the example you provide for LocalAgent at: https://huggingface.co/docs/transformers/main_classes/agent#transformers.LocalAgent

Except I've replaced the example repo with GPT, and asked a general question to see how the model responds.

I get the following warnings and error in CLI
```bash
Token indices sequence length is longer than the specified maximum sequence length for this model (1592 > 1024). Running this sequence through the model will result in indexing errors
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Traceback (most recent call last):
  File "/root/test.py", line 1838, in <module>
    runInferenceCausalLM1(context)
  File "/root/test.py", line 1662, in runInferenceCausalLM1
    agent.run("Tell me what a black hole is made of?.")
  File "/root/projectName11/lib/python3.10/site-packages/transformers/tools/agents.py", line 341, in run
    result = self.generate_one(prompt, stop=["Task:"])
  File "/root/projectName11/lib/python3.10/site-packages/transformers/tools/agents.py", line 740, in generate_one
    outputs = self.model.generate(
  File "/root/projectName11/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/projectName11/lib/python3.10/site-packages/transformers/generation/utils.py", line 1602, in generate
    return self.greedy_search(
  File "/root/projectName11/lib/python3.10/site-packages/transformers/generation/utils.py", line 2450, in greedy_search
    outputs = self(
  File "/root/projectName11/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/projectName11/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1076, in forward
    transformer_outputs = self.transformer(
  File "/root/projectName11/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/projectName11/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 844, in forward
    position_embeds = self.wpe(position_ids)
  File "/root/projectName11/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/projectName11/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "/root/projectName11/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

The code I'm using is shown below, and is a straight copy from the online quickstart - except - modified for a different task and model. However, should still work.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
   import torch
   from transformers import AutoModelForCausalLM, AutoTokenizer, LocalAgent

   model = AutoModelForCausalLM.from_pretrained('gpt2', device_map="auto")
   tokenizer = AutoTokenizer.from_pretrained('gpt2')

   agent = LocalAgent(model, tokenizer)
   agent.run("Tell me what a black hole is made of?.")   
```


I've tried with and without   torch_dtype=torch.bfloat16 - I don't think this is the cause.

### Expected behavior

I thought an agent would return the same result as if I'd run the inference with pipeline or transformer approach. I am testing the agent because I was keen on seeing how the prompting helper works.

Thanks

Comments:
Hey! as the issue mentions, there seems to be an issue with the tokenized inputs. I'll ping @Rocketknight1 here as I won't have time to look into this

Hi @gidzr, the cause here is that GPT-2 simply doesn't have the required sequence length to handle the very long default prompt used by transformers agents. The default prompt describes multiple tools and their arguments, and tokenizes to ~1590 tokens in GPT-2, whereas GPT-2 only has a maximum sequence length of 1024.

In general, agent prompts like these will only work well with models that have a long maximum sequence length and have been fine-tuned with instruction training.

@Rocketknight1 Ahh I see.. Re-reading the error, I can see the issue re token size, but my prompt was simply "Tell me what a black hole is made of?" was only ~15 tokens which really confused me.

You refer to the "default prompt" which describes tools and arguments and behind the scenes. Is there any way for me to 'hack' this in a LocalAgent config? eg. if I add my info for these?
- chat_prompt_template (str, optional
- run_prompt_template (str, optional)
- additional_tools ([Tool] list of tools or dictionary with tool values, optional)

I'm assuming if I enter null or custom values for these params it can reduce the token size for input? However, I'm not sure what I'm overriding and how this would affect the Agent's operation. 

**Is there a view of what the defaults for these look like so I can override with a shorter version of each**?

Hi @gidzr, take a look at the [Custom Tools and Prompts](https://huggingface.co/docs/transformers/custom_tools) guide!

Again though, note that GPT-2 is a pure text-completion model, and has not been trained to follow instructions or use external tools. I expect even if you can make the prompt short enough to work with it, the results will probably not be very good!

Hey @Rocketknight1 - noted. 

Just playing around with the features etc. There's a learning curve so wanted to be cheeky and pick your brains if possible in this forum.

Am I correct in understanding that:
- Tools: just like pipelines, but whereas Pipelines use task as a shorthand to optimise parameters (num_beams, temperature, etc) with a flexible prompt for a given LLM, Tools are locked down on a specific predetermined LLM, prompt, and parameters. Is that right?
- Agents: an LLM tuned to accept a specific type of prompt format, which guides the LLM to generate code to execute tools for a desired outcome? **Or** are Agents emergent from any reasonably sized LLM with the correct type of prompt? (ie. could I copy/paste your prompt into Llama/Phi1_5/GPT4 and get the same results?)

There's a lot in huggingface, it's like drinking from a firehose.. My only remaining confusion for tools and agents is how much is clever prompting vs customised LLM models running in the background vs coding shorthands (like JQuery is to Javascript).

References
https://huggingface.co/docs/transformers/custom_tools
https://huggingface.co/docs/transformers/transformers_agents
https://huggingface.co/docs/transformers/main_classes/agent

Ha, I'll allow it. Tools can be anything - they're often not even ML at all, just some code that exposes an API. A calculator app is a common 'tool' for LLMs. However, they can also be a full model/pipeline that does a specific task (like image recognition).

An agent is an LLM that has been trained in the presence of tools. I suspect some smarter LLMs that have been trained to follow instructions can use tools if they're well-described in the prompt, even if they haven't seen tools during training, but you'll probably get the best results if the model has been specifically trained with them.

The 'agent' LLM is given a description of the tools it has available, as well as the task and input data, and then it tries to solve the task by either writing answers itself, or by making an API call to one of the tools it has available. This process can be iterative - the agent can call multiple tools before emitting a final answer.

In general, agents are quite an advanced and experimental topic, even at the cutting edge. If what you're trying to do can be accomplished with a single pipeline, you'll often get better results than trying to do it with a fully general agent!

EPICNESS!  ..thanks for that..  totally makes sense. Agent-tools are a very cool experimentation. 

I was thinking along the same lines re pipelines. My impression is that agents are best suited for multi-modal uses that bring audio+visual+text together when a task is very open-ended, with pipes best suited for a single LLM and targeted use case, whereas multi-LLM (translation+completion) within a mode (text) will need a bit of trial and error to see which works best. Also, as I'm running CPU and agents use specific LLMs for tools, performance will influence the solution.

Your comments re Langchain surprised me a little. I was initially 100% Langchain and loved it, but the outdated API documentation, blackbox approach, and unclear multiple approaches (eg. chain.run() vs chain.generate() vs chain())  moved me towards a full HF eco-system. Langchain feels like it wants to be an umbrella or integration layer, but doesn't simplify or accelerate the base technology it's drawing on - and ends up becoming bloat (eg. the qa_chain-chroma implementation). Some elements are awesome, but HF/onnx/accelerate/pipelines, etc .. provide better performance and usage for me.

Thanks again!

Refs
https://huggingface.co/docs/transformers/transformers_agents#a-curated-set-of-tools
https://huggingface.co/docs/transformers/custom_tools#future-compatibility-with-langchain