[bug fix]tokenizer init via tokenization_utils_base.py in some case make "Do you accept" emerge even if trust_remote_code=True is setted 
Author: ShadowTeamCN
Pull request: yes

# What does this PR do?

This PR fix a bug for tokenizer from_pretrained method causing manual input issue in tokenizer initialization
if not fixed ,in some rare contidition tokenizer init via tokenization_utils_base would need typed "yes",which caused from inner call AutoConfig.from_pretrained() , but not transfer the trust_remote_code parameter when init tokenizer

Fixes # (issue)

the original behavior of tokenization_utils_base AutoConfig.from_pretrained() works in most cases.But in some rare scene, when the tokenizer initializes, it internally calls AutoConfig.pretrained. During this call, the trust_remote_code parameter was omitted, leading to a situation where the user is always required to manually enter yes, regardless of original parameters or settings.

I encountered this bug when i init tokenizer from local disk file, a model named Baichuan-13B-Chat but tokenizer type equals to LlamaTokenizer (same as sentencepiece),and the from_pretrained-used parameter directory 
has config.json,tokenizer.model (but no other tokenizer related file), this bug triggered

![image](https://github.com/huggingface/transformers/assets/11573254/55568839-f4f2-4118-9e66-ec5a4f8bf4ac)
_this picture is minimal reproduce file structure_

While this might seem benign, it becomes problematic in scenarios like some distribute environment, (eg.  Aliyun PAI platform,Tencent cloud platform Ti-One) where manual input isn't feasible. As a result, the interface flushes in the incorrect input and eventually raises an EOFError

![image](https://github.com/huggingface/transformers/assets/11573254/f08a744d-f62c-431d-bb36-040bc146ff37)
_this log message ascend from bottom to up_

To Reproduce:
1. mkdir test_tokenizer && cd test_tokenizer
2. wget https://huggingface.co/baichuan-inc/Baichuan-13B-Chat/resolve/main/tokenizer.model
 wget https://huggingface.co/baichuan-inc/Baichuan-13B-Chat/resolve/main/config.json
4. cd ..

> from transformers import LlamaTokenizer
> tokenizer=LlamaTokenizer.from_pretrained('test_tokenizer/',trust_remote_code=True)

Then run this code would triggered the bug, this may caused by abnormal used of tokenizer,
but i think add the trust_remote_code parameter in tokenization_utils_base  would be safer


![image](https://github.com/huggingface/transformers/assets/11573254/b2db3d14-4866-4380-81fd-06a08bcf9f05)


## Before submitting
- This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- Did you write any new necessary tests?


## Who can review?
@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25736). All of your documentation changes will be reflected on that endpoint.

> Looks good to me! Thanks. Would be great if you can add a test in `test_tokenization_auto.py` with a dummy repo!

Happy to see that, I'll review the logic in the test_tokenization_auto.py and try to add a dummy test

Hey @ShadowTeamCN, let us know if you need help with the addition of the test!

> Hey @ShadowTeamCN, let us know if you need help with the addition of the test!

Hi @LysandreJik , I was wondering what the test function should verify. Given that this PR forwards user-provided parameters to AutoConfig, should I focus on validating the correct initialization of AutoConfig, or simply ensure that the user-provided parameters are accurately passed through to AutoConfig?

I think making sure that the args are properly propagate is the best way to do this. If you want to load a tokenizer from the hub that has custom code we'll have to check it and set a revision !

Hello @ArthurZucker , I've added a test case to this PR. However, there remains one issue: I employed a mock method to verify that the parameters are passed to AutoConfig.from_pretrained. Yet, I encountered difficulty in utilizing unittest.TestCase's assertEqual within the mock function, as passing the TestCase's self pointer to the external mock function seems challenging. Consequently, I resorted to using 'assert', though ideally, this should be substituted with a more appropriate method.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @ShadowTeamCN I. might have been a bit too harsh on the review! Do you need help to finish it? 🤗 

> Hey @ShadowTeamCN I. might have been a bit too harsh on the review! Do you need help to finish it? 🤗

I'm sorry for delay the PR because of busy doing company-related work. 
Besides I'm little confused of whether I should start a new PR to meet the latest branch.

> Thanks for adding a tests. Let's try to blend it in our current framework of tests, and let's use additional kwargs that are relevant like the `revision` for example. We can use 2 repos were the tokenizer are suppose to be different depending on the version. Simple is best so if you don't really know how to do this you can just remove it I 'll try to think of something else!

And may you offer me some examples of how the `revision` paramter work?


You can probably just do `git pull upstream main`. 
A test might be hard to do, if you can confirme this fixes the issue should just be alright! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

#28854 superseeds this, I think the new solution is more elegant

> #28854 superseeds this, I think the new solution is more elegant

good to see that, and i found it similar to my original implement, so i will close this PR