RuntimeError: Failed to import transformers.models.mistral.modeling_mistral because of the following error (look up to see its traceback):
Author: AlexMcgrigor
Pull request: no

**Issue**:
Mistral have just updated their model files on HuggingFace. I am trying to test two things, load the model and load the model from local storage. However I can't even load models with my updated package versions. 

- Running these requirements below created a tokenizer error. Mistral have updated their model files to reflect the change in tokenizer. 

**Requirements:**
delta-spark==3.1.0
mypy==1.10.0
ruff==0.4.4
pre-commit==3.7.0
types-PyYAML==6.0.12.20240311
scikit-learn==1.3.2
py-readability-metrics==1.4.5
nltk==3.8.1
transformers==4.42.3
protobuf==3.20.3
bert-score==0.3.13
rouge==1.0.1
torch==2.2.2
pytest==8.2.0
requests==2.31.0
mlflow==2.13.1
bitsandbytes==0.42.0
accelerate==0.31.0
flash-attn==2.5.8

cuda version 11.8
Also getting the error on CUDA Version: 12.2

**Also ran the below as per flash_attn git page requirements:**
!pip install packaging
!pip uninstall -y ninja && pip install ninja
!pip install flash-attn --no-build-isolation

**Below code to run:**
import os
from abc import ABC, abstractmethod
from pathlib import Path
from typing import TypedDict

import torch
from huggingface_hub import login, snapshot_download
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    pipeline,
) 

from core.src.constructor import Configuration

spark, catalog_name = Configuration().execute()

token = os.getenv("HUGGINGFACEHUB-API-TOKEN")
if token is None:
    msg = "Missing Hugging Face API token."
    raise ValueError(msg)
login(token=token)

model_path = "/Volumes/fms_dev/models/model_store/Mistral-7B-Instruct-v0.1/"

model_name = "mistralai/Mistral-7B-Instruct-v0.1"

config = AutoConfig.from_pretrained(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    config=config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    )
   
   - Running with flash-attn==2.5.8 then throws a cannot import module error for the model I'm trying to load. 
RuntimeError: Failed to import transformers.models.mistral.modeling_mistral because of the following error (look up to see its traceback): /local_disk0/.ephemeral_nfs/envs/pythonEnv-6244686b-385a-4991-9d6c-f7fbe63ef415/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c106SymIntltEl

Can someone help?
    

Comments:
cc @ArthurZucker 

Tried solutions from this page https://github.com/Dao-AILab/flash-attention/issues/853 and building explicit flash_attn wheels is not supported on the databricks platform that I'm building on. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

still facing this issue.. do we have any solution?


> still facing this issue.. do we have any solution?

I met this problem, too. The solutions from this page https://github.com/Dao-AILab/flash-attention/issues/853 works for me. Just need to choose the correct version of whls. Check the version info in the traceback like `flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so`