Failed to import transformers.data.data_collator because of the following error (look up to see its traceback): unhashable type: 'list'
Author: lizhaoan123
Pull request: no
Labels: name: bug

### System Info

I encountered an error while using transformers to run BGEM3EmbeddingFunction in the CUDA environment：

Traceback (most recent call last):
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1778, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/__init__.py", line 28, in <module>
    from .processors import (
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/processors/__init__.py", line 15, in <module>
    from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/processors/glue.py", line 30, in <module>
    import tensorflow as tf
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/__init__.py", line 49, in <module>
    from tensorflow._api.v2 import __internal__
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/_api/v2/__internal__/__init__.py", line 8, in <module>
    from tensorflow._api.v2.__internal__ import autograph
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/_api/v2/__internal__/autograph/__init__.py", line 8, in <module>
    from tensorflow.python.autograph.core.ag_ctx import control_status_ctx # line: 34
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/core/ag_ctx.py", line 21, in <module>
    from tensorflow.python.autograph.utils import ag_logging
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/utils/__init__.py", line 17, in <module>
    from tensorflow.python.autograph.utils.context_managers import control_dependency_on_returns
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/utils/context_managers.py", line 19, in <module>
    from tensorflow.python.framework import ops
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 5925, in <module>
    ) -> Optional[Callable[[Any], message.Message]]:
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 316, in __getitem__
    return self._getitem(self, parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 433, in Optional
    return Union[arg, type(None)]
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 316, in __getitem__
    return self._getitem(self, parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 421, in Union
    parameters = _remove_dups_flatten(parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 215, in _remove_dups_flatten
    all_params = set(params)
TypeError: unhashable type: 'list'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data-ai/adp/label-rag-server/main.py", line 36, in match_concept
    return vector_search(request_param)
  File "/data-ai/adp/label-rag-server/main.py", line 63, in vector_search
    ef = BGEM3EmbeddingFunction(device="cuda:0", model_name='BAAI/bge-m3')
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/hybrid/__init__.py", line 9, in BGEM3EmbeddingFunction
    return bge_m3.BGEM3EmbeddingFunction(*args, **kwargs)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/utils/lazy_import.py", line 18, in __getattr__
    module = self._load()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/utils/lazy_import.py", line 12, in _load
    module = importlib.import_module(self.__name__)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/hybrid/bge_m3.py", line 15, in <module>
    from FlagEmbedding import BGEM3FlagModel
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/FlagEmbedding/__init__.py", line 2, in <module>
    from .bge_m3 import BGEM3FlagModel
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/FlagEmbedding/bge_m3.py", line 7, in <module>
    from transformers import PreTrainedTokenizerFast, BatchEncoding, DataCollatorWithPadding, XLMRobertaForMaskedLM, is_torch_npu_available
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1766, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1780, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.data.data_collator because of the following error (look up to see its traceback):
unhashable type: 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 1079, in emit
    msg = self.format(record)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 923, in format
    return fmt.format(record)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 659, in format
    record.message = record.getMessage()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 363, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 908, in _bootstrap
    self._bootstrap_inner()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 720, in __init__
    self.handle()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 398, in handle
    super().handle()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/http/server.py", line 427, in handle
    self.handle_one_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/http/server.py", line 415, in handle_one_request
    method()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 370, in run_wsgi
    execute(self.server.app)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 331, in execute
    application_iter = app(environ, start_response)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/data-ai/adp/label-rag-server/main.py", line 41, in match_concept
    logging.error("concepts vector search error: {}", e)
Message: 'concepts vector search error: {}'
Arguments: (RuntimeError("Failed to import transformers.data.data_collator because of the following error (look up to see its traceback):\nunhashable type: 'list'"),)

Is it a problem with my transformer version?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

--- Logging error ---
Traceback (most recent call last):
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1778, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/__init__.py", line 28, in <module>
    from .processors import (
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/processors/__init__.py", line 15, in <module>
    from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/data/processors/glue.py", line 30, in <module>
    import tensorflow as tf
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/__init__.py", line 49, in <module>
    from tensorflow._api.v2 import __internal__
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/_api/v2/__internal__/__init__.py", line 8, in <module>
    from tensorflow._api.v2.__internal__ import autograph
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/_api/v2/__internal__/autograph/__init__.py", line 8, in <module>
    from tensorflow.python.autograph.core.ag_ctx import control_status_ctx # line: 34
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/core/ag_ctx.py", line 21, in <module>
    from tensorflow.python.autograph.utils import ag_logging
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/utils/__init__.py", line 17, in <module>
    from tensorflow.python.autograph.utils.context_managers import control_dependency_on_returns
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/autograph/utils/context_managers.py", line 19, in <module>
    from tensorflow.python.framework import ops
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 5925, in <module>
    ) -> Optional[Callable[[Any], message.Message]]:
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 316, in __getitem__
    return self._getitem(self, parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 433, in Optional
    return Union[arg, type(None)]
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 316, in __getitem__
    return self._getitem(self, parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 421, in Union
    parameters = _remove_dups_flatten(parameters)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/typing.py", line 215, in _remove_dups_flatten
    all_params = set(params)
TypeError: unhashable type: 'list'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data-ai/adp/label-rag-server/main.py", line 36, in match_concept
    return vector_search(request_param)
  File "/data-ai/adp/label-rag-server/main.py", line 63, in vector_search
    ef = BGEM3EmbeddingFunction(device="cuda:0", model_name='BAAI/bge-m3')
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/hybrid/__init__.py", line 9, in BGEM3EmbeddingFunction
    return bge_m3.BGEM3EmbeddingFunction(*args, **kwargs)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/utils/lazy_import.py", line 18, in __getattr__
    module = self._load()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/utils/lazy_import.py", line 12, in _load
    module = importlib.import_module(self.__name__)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/milvus_model/hybrid/bge_m3.py", line 15, in <module>
    from FlagEmbedding import BGEM3FlagModel
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/FlagEmbedding/__init__.py", line 2, in <module>
    from .bge_m3 import BGEM3FlagModel
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/FlagEmbedding/bge_m3.py", line 7, in <module>
    from transformers import PreTrainedTokenizerFast, BatchEncoding, DataCollatorWithPadding, XLMRobertaForMaskedLM, is_torch_npu_available
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1766, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1780, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.data.data_collator because of the following error (look up to see its traceback):
unhashable type: 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 1079, in emit
    msg = self.format(record)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 923, in format
    return fmt.format(record)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 659, in format
    record.message = record.getMessage()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/logging/__init__.py", line 363, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 908, in _bootstrap
    self._bootstrap_inner()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 650, in process_request_thread
    self.finish_request(request, client_address)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/socketserver.py", line 720, in __init__
    self.handle()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 398, in handle
    super().handle()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/http/server.py", line 427, in handle
    self.handle_one_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/http/server.py", line 415, in handle_one_request
    method()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 370, in run_wsgi
    execute(self.server.app)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/werkzeug/serving.py", line 331, in execute
    application_iter = app(environ, start_response)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 1536, in __call__
    return self.wsgi_app(environ, start_response)
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
  File "/data-ai/adp/anaconda3/envs/python3.91/lib/python3.9/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/data-ai/adp/label-rag-server/main.py", line 41, in match_concept
    logging.error("concepts vector search error: {}", e)
Message: 'concepts vector search error: {}'
Arguments: (RuntimeError("Failed to import transformers.data.data_collator because of the following error (look up to see its traceback):\nunhashable type: 'list'"),)
140.210.223.35 - - [15/Nov/2024 17:47:04] "POST /api/rag/matchConceptByVector HTTP/1.1" 200 -

### Expected behavior

I hope there are some common ways to solve this problem

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.