UnboundLocalError: local variable 'tokens' referenced before assignment
Author: pseudotensor
Pull request: no

### System Info

Since transformers==4.32.0


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just try to tokenize an empty string

```
File "/workspace/src/gen.py", line 2064, in evaluate
  context, num_prompt_tokens2 = H2OTextGenerationPipeline.limit_prompt(context, tokenizer)
File "/workspace/src/h2oai_pipeline.py", line 86, in limit_prompt
  prompt_tokens = tokenizer(prompt_text)['input_ids']
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2602, in __call__
  encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2708, in _call_one
  return self.encode_plus(
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2781, in encode_plus
  return self._encode_plus(
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 656, in _encode_plus
  first_ids = get_input_ids(text)
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 623, in get_input_ids
  tokens = self.tokenize(text, **kwargs)
File "/h2ogpt_conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 208, in tokenize
  if tokens[0] == SPIECE_UNDERLINE and tokens[1] in self.all_special_tokens:
UnboundLocalError: local variable 'tokens' referenced before assignment
```

Bug here: 
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/tokenization_llama.py#L203-L216

```
    # Copied from transformers.models.t5.tokenization_t5.T5Tokenizer.tokenize
    def tokenize(self, text: "TextInput", **kwargs) -> List[str]:
        """
        Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the
        first token is special.
        """
        if self.legacy:
            return super().tokenize(text, **kwargs)

        if len(text) > 0:
            tokens = super().tokenize(SPIECE_UNDERLINE + text.replace(SPIECE_UNDERLINE, " "), **kwargs)

        if tokens[0] == SPIECE_UNDERLINE and tokens[1] in self.all_special_tokens:
            tokens = tokens[1:]
        return tokens
```

Should still define tokens as empty list if nothing to tokenize.  A failure is odd result.

### Expected behavior

Not fail

Comments:
Yep, fix is here #25793 
Only happens when the input text is empty 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Closing as https://github.com/huggingface/transformers/pull/25793 was merged