Issue 1:
Hi, hope all is well :)

It looks like DistilBertTokenizerFast doesn't take do_lower_case into account.

from transformers import DistilBertTokenizerFast, DistilBertTokenizer

PRE_TRAINED_MODEL_NAME = "distilbert-base-uncased"
tokenizer_f = DistilBertTokenizerFast.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)
tokenizer_s = DistilBertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME, do_lower_case=False)

sample = "Hello, world. How are you?"

tokens_f = tokenizer_f.tokenize(sample)
tokens_s = tokenizer_s.tokenize(sample)

print(tokens_f)
print(tokens_s)

output:

['hello', ',', 'world', '.', 'how', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']

expected:

['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']
['[UNK]', ',', 'world', '.', '[UNK]', 'are', 'you', '?']

packages:

argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
decorator==4.4.2
defusedxml==0.7.1
entrypoints==0.3
filelock==3.0.12
idna==2.10
ipykernel==5.5.0
ipython==7.21.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==2.11.3
joblib==1.0.1
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
MarkupSafe==1.1.1
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
numpy==1.20.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.9.0
prompt-toolkit==3.0.16
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pyzmq==22.0.3
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
Send2Trash==1.5.0
six==1.15.0
terminado==0.9.2
testpath==0.4.4
tokenizers==0.10.1
torch==1.8.0+cu111
tornado==6.1
tqdm==4.59.0
traitlets==5.0.5
transformers==4.3.3
typing-extensions==3.7.4.3
urllib3==1.26.3
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1

Python version:

Python 3.8.6

System:

PopOS 20, happy to provide more info on system specs such as hardware if needed

Issue 2:
System Info

2023-05-24 23:09:53.575434: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/transformers/commands/env.py:63: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.config.list_physical_devices('GPU') instead.

2023-05-24 23:10:05.261610: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

transformers version: 4.29.2

Platform: Linux-5.15.107+-x86_64-with-glibc2.31

Python version: 3.10.11

Huggingface_hub version: 0.14.1

Safetensors version: not installed

PyTorch version (GPU?): 2.0.1+cu118 (True)

Tensorflow version (GPU?): 2.12.0 (True)

Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)

Jax version: 0.4.10

JaxLib version: 0.4.10

Using GPU in script?:

Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker

@younesbelkada

Information

The official example scripts

My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)

My own task or dataset (give details below)

Reproduction

I was trying to use the new shiny mpt model from the huggingface hub from a revision :

from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
import torch
import transformers
import accelerate

model_name = 'mosaicml/mpt-7b'

model = AutoModelForCausalLM.from_pretrained(model_name,
trust_remote_code=True,
revision="refs/pr/23",
device_map="auto"
)

But I stumble on this error after the using the above code :

ValueError: MPTForCausalLM does not support device_map='auto' yet.

The "auto" was indeed not supported in the main branch but we add a correction in the PR branch (so the argument revision="refs/pr/23")

I did some investigation and the model was indeed loading the main .py files :

Downloading (…)main/modeling_mpt.py: 100%
17.4k/17.4k [00:00<00:00, 1.12MB/s]
Downloading (…)in/param_init_fns.py: 100%
12.6k/12.6k [00:00<00:00, 971kB/s]
Downloading (…)resolve/main/norm.py: 100%
2.56k/2.56k [00:00<00:00, 131kB/s]
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- norm.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/mosaicml/mpt-7b:
- param_init_fns.py
- norm.py

You can see the main/ here. I did manually check the modeling_mpt.py file it didn't have the PR changes.

So I did try to find where the bug where inside the transformers package ... (first time looking at the code).

I am a bit surprised !

Basicly the code rewrite the config values after having read it (it adds the information about the repo ids (in add_model_info_to_auto_map in generic.py in utils/ from the transformers package) something that seems normal.

"auto_map": {
"AutoConfig": "mosaicml/mpt-7b--configuration_mpt.MPTConfig",
"AutoModelForCausalLM": "mosaicml/mpt-7b--modeling_mpt.MPTForCausalLM"
}

It notably add the "--" string.

then in get_class_from_dynamic_module (in dynamic_module_utils.py) it has :

if "--" in class_reference:
repo_id, class_reference = class_reference.split("--")
# Invalidate revision since it's not relevant for this repo
revision = "main"

So the revision become "main" and from here we are done.

I suppose if i do a PR removing the revision overide some people will not be happy ?

Expected behavior

The expected behaviour is to load the file from the PR branch. (not the main/)