When I run topic_model.visualize_documents, AttributeError: 'RobertaModel' object has no attribute 'attn_implementation' error occurs.
Author: kungmo
Pull request: no
Labels: name: bug

### System Info

Hello.

When I run topic_model.visualize_documents(interpretation_list, custom_labels=True) , then an error occurs like below.

AttributeError: 'RobertaModel' object has no attribute 'attn_implementation'

This is my full error codes. How can I solve this problem?

`---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[30], line 1
----> 1 topic_model.visualize_documents(interpretation_list, custom_labels=True)

File [~/python312/lib/python3.12/site-packages/bertopic/_bertopic.py:2524](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/bertopic/_bertopic.py#line=2523), in BERTopic.visualize_documents(self, docs, topics, embeddings, reduced_embeddings, sample, hide_annotations, hide_document_hover, custom_labels, title, width, height)
   2522 check_is_fitted(self)
   2523 check_documents_type(docs)
-> 2524 return plotting.visualize_documents(
   2525     self,
   2526     docs=docs,
   2527     topics=topics,
   2528     embeddings=embeddings,
   2529     reduced_embeddings=reduced_embeddings,
   2530     sample=sample,
   2531     hide_annotations=hide_annotations,
   2532     hide_document_hover=hide_document_hover,
   2533     custom_labels=custom_labels,
   2534     title=title,
   2535     width=width,
   2536     height=height,
   2537 )

File [~/python312/lib/python3.12/site-packages/bertopic/plotting/_documents.py:119](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/bertopic/plotting/_documents.py#line=118), in visualize_documents(topic_model, docs, topics, embeddings, reduced_embeddings, sample, hide_annotations, hide_document_hover, custom_labels, title, width, height)
    117         embeddings_to_reduce = embeddings[indices]
    118     elif embeddings is None and reduced_embeddings is None:
--> 119         embeddings_to_reduce = topic_model._extract_embeddings(df.doc.to_list(), method="document")
    121 # Reduce input embeddings
    122 if reduced_embeddings is None:

File [~/python312/lib/python3.12/site-packages/bertopic/_bertopic.py:3676](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/bertopic/_bertopic.py#line=3675), in BERTopic._extract_embeddings(self, documents, images, method, verbose)
   3674     embeddings = self.embedding_model.embed_words(words=documents, verbose=verbose)
   3675 elif method == "document":
-> 3676     embeddings = self.embedding_model.embed_documents(documents, verbose=verbose)
   3677 elif documents[0] is None and images is None:
   3678     raise ValueError(
   3679         "Make sure to use an embedding model that can either embed documents"
   3680         "or images depending on which you want to embed."
   3681     )

File [~/python312/lib/python3.12/site-packages/bertopic/backend/_base.py:62](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/bertopic/backend/_base.py#line=61), in BaseEmbedder.embed_documents(self, document, verbose)
     50 def embed_documents(self, document: List[str], verbose: bool = False) -> np.ndarray:
     51     """Embed a list of n words into an n-dimensional
     52     matrix of embeddings.
     53 
   (...)
     60         that each have an embeddings size of `m`
     61     """
---> 62     return self.embed(document, verbose)

File [~/python312/lib/python3.12/site-packages/bertopic/backend/_sentencetransformers.py:65](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/bertopic/backend/_sentencetransformers.py#line=64), in SentenceTransformerBackend.embed(self, documents, verbose)
     53 def embed(self, documents: List[str], verbose: bool = False) -> np.ndarray:
     54     """Embed a list of n documents/words into an n-dimensional
     55     matrix of embeddings.
     56 
   (...)
     63         that each have an embeddings size of `m`
     64     """
---> 65     embeddings = self.embedding_model.encode(documents, show_progress_bar=verbose)
     66     return embeddings

File [~/python312/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py:517](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/sentence_transformers/SentenceTransformer.py#line=516), in SentenceTransformer.encode(self, sentences, prompt_name, prompt, batch_size, show_progress_bar, output_value, precision, convert_to_numpy, convert_to_tensor, device, normalize_embeddings)
    514 features.update(extra_features)
    516 with torch.no_grad():
--> 517     out_features = self.forward(features)
    518     if self.device.type == "hpu":
    519         out_features = copy.deepcopy(out_features)

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/container.py:219](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/container.py#line=218), in Sequential.forward(self, input)
    217 def forward(self, input):
    218     for module in self:
--> 219         input = module(input)
    220     return input

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/module.py:1553](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/module.py#line=1552), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/module.py:1562](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/module.py#line=1561), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/python312/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py:118](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/sentence_transformers/models/Transformer.py#line=117), in Transformer.forward(self, features)
    115 if "token_type_ids" in features:
    116     trans_features["token_type_ids"] = features["token_type_ids"]
--> 118 output_states = self.auto_model(**trans_features, return_dict=False)
    119 output_tokens = output_states[0]
    121 features.update({"token_embeddings": output_tokens, "attention_mask": features["attention_mask"]})

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/module.py:1553](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/module.py#line=1552), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/module.py:1562](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/module.py#line=1561), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [~/python312/lib/python3.12/site-packages/transformers/models/roberta/modeling_roberta.py:924](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/transformers/models/roberta/modeling_roberta.py#line=923), in RobertaModel.forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    920 if attention_mask is None:
    921     attention_mask = torch.ones((batch_size, seq_length + past_key_values_length), device=device)
    923 use_sdpa_attention_masks = (
--> 924     self.attn_implementation == "sdpa"
    925     and self.position_embedding_type == "absolute"
    926     and head_mask is None
    927     and not output_attentions
    928 )
    930 # Expand the attention mask
    931 if use_sdpa_attention_masks and attention_mask.dim() == 2:
    932     # Expand the attention mask for SDPA.
    933     # [bsz, seq_len] -> [bsz, 1, seq_len, seq_len]

File [~/python312/lib/python3.12/site-packages/torch/nn/modules/module.py:1729](https://acer2.snu.ac.kr:8443/user/ubuntu/lab/tree/grounded_theory/python312/lib/python3.12/site-packages/torch/nn/modules/module.py#line=1728), in Module.__getattr__(self, name)
   1727     if name in modules:
   1728         return modules[name]
-> 1729 raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")

AttributeError: 'RobertaModel' object has no attribute 'attn_implementation'`

And here's version of installed libraries.
' sentence-transformers                    3.0.1
transformers                             4.45.1
bertopic                                 0.16.3 '

Thank you.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import torch
from sentence_transformers import SentenceTransformer, util
from bertopic import BERTopic
from hdbscan import HDBSCAN
import tqdm as notebook_tqdm

#hdbscan_model = HDBSCAN(min_cluster_size=3, metric='euclidean', prediction_data=True)

topic_model = BERTopic(verbose=True,
                       embedding_model="bespin-global/klue-sroberta-base-continue-learning-by-mnr",
                       min_topic_size=4,
                       nr_topics = 'auto', # 묶을 군집 개수 지정. 기본값으로 두니 140개나 나옴.
                       #hdbscan_model=hdbscan_model
                      )

# df에 sentence-BERT로 임베딩한 열 추가하기

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("bespin-global/klue-sroberta-base-continue-learning-by-mnr")

df['embedding'] = df.apply(lambda row: model.encode(row.interpretation), axis = 1)

# pkl 파일 불러오기
import pickle

with open('interpretation_3rd_embeddings.pkl', 'rb') as f:
    df = pickle.load(f)

interpretation_list = df['interpretation'].to_list()

topic_model = BERTopic.load("topic_model_3rd.model")

# 주제별 이름 정하기
topic_model.set_topic_labels({-1:"미분류", 0: "캐러셀 적용, 웹페이지 디자인",
                              1: "데이터셋 업데이트", 2: "백엔드 기능개선(교사용 기록 확인, 새 서버, 국어사전, 임베딩 모델 변경)",
                              3:"사용 중 오류 수정(장고 static 등) ", 4:"GPU 필요 시 사용량 최적화",
                              5:"Chainlit 사용 관련 시행착오", 6:"학년별 운영",
                              7:"GitHub 연동 관련 시행착오", 8:"보안 강화",
                              9:"구글 검색 등록", 10:"index.html 수정",
                              11:"UUID 추가"})

topic_model.visualize_documents(interpretation_list, custom_labels=True)

### Expected behavior

AttributeError: 'RobertaModel' object has no attribute 'attn_implementation' error occurs.

Comments:
The issue you're encountering (AttributeError: 'RobertaModel' object has no attribute 'attn_implementation') is likely due to an incompatibility between the sentence-transformers model and the transformers version you're using. Here's how I can resolve it:
firstly I wll Upgrade transformers library,
then Upgrade sentence-transformers library,
Downgrade transformers if it is needed or else continue
or i will try an alternative model:



Hi @kungmo, thanks for sharing your code. I've taken a look, and there are a few small discrepancies in the code preventing it from running smoothly. I have listed them as follows:

## **1. `df` referenced before declaration:**
> # df에 sentence-BERT로 임베딩한 열 추가하기
> from sentence_transformers import SentenceTransformer
> 
> model = SentenceTransformer("bespin-global/klue-sroberta-base-continue-learning-by-mnr")
> 
> df['embedding'] = df.apply(lambda row: model.encode(row.interpretation), axis = 1)
> 
> # pkl 파일 불러오기
> import pickle
> 
> with open('interpretation_3rd_embeddings.pkl', 'rb') as f: 
>df = pickle.load(f)
> 
> interpretation_list = df['interpretation'].to_list()
> 
> topic_model = BERTopic.load("topic_model_3rd.model")

## **2. Model and Dataset files are not accessible:** 
Dataset
> with open('interpretation_3rd_embeddings.pkl', 'rb') as f:

Model
> topic_model = BERTopic.load("topic_model_3rd.model")

## **3. Redeclaration of `topic_model`:**
Here
> topic_model = BERTopic(verbose=True,
> embedding_model="bespin-global/klue-sroberta-base-continue-learning-by-mnr",
> min_topic_size=4,
> nr_topics = 'auto', # 묶을 군집 개수 지정. 기본값으로 두니 140개나 나옴.
> #hdbscan_model=hdbscan_model
> )


And Here
> topic_model = BERTopic.load("topic_model_3rd.model")

##
However using the library versions as you have provided:

![image](https://github.com/user-attachments/assets/37672afb-27bc-495e-b407-d07639b52ac8)

With my env-config as follows:

- `transformers` version: 4.45.1
- Platform: Linux-5.15.154+-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.25.0
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): 2.16.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26.dev20240620
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: Tesla T4


And a few minor changes in your code as below:
1. Replacing `interpretation_3rd_embeddings.pkl` with `random_interpretations` array dataset,
2. Removing the second `topic_model` declaration and,
3. Setting nr_topics=12.
```python
nr_topics=12 
# nr_topics='auto' works for a bigger dataset and we have set it to 12 as we are using a very small dataset)
```

I was able to run the following code without any errors or bugs in a Kaggle Notebook.
```python
import torch
from sentence_transformers import SentenceTransformer, util
from bertopic import BERTopic
from hdbscan import HDBSCAN
import tqdm as notebook_tqdm
import pandas as pd
import random

# Generate random Korean interpretations Datset
random_interpretations = [
    "홈페이지에 캐러셀 기능 구현",
    "새로운 사용자 정보로 데이터셋 업데이트",
    "Django 정적 파일 로딩 문제 해결",
    "대규모 계산을 위한 GPU 사용 최적화",
    "개선된 UI를 위한 Chainlit 실험",
    "학년별 콘텐츠 전달 구성",
    "GitHub 통합 충돌 해결",
    "사용자 인증을 위한 보안 프로토콜 강화",
    "Google 검색 콘솔에 웹사이트 등록",
    "SEO 개선을 위한 index.html 수정",
    "고유 사용자 식별을 위한 UUID 통합",
    "데이터베이스 쿼리 성능 최적화",
    "모바일 반응형 디자인 구현",
    "서버 로그 분석 자동화 스크립트 개발",
    "API 엔드포인트 보안 강화",
    "사용자 피드백 시스템 구축",
    "다국어 지원을 위한 i18n 구현",
    "CI/CD 파이프라인 구축 및 최적화",
    "마이크로서비스 아키텍처로 전환",
    "실시간 채팅 기능 추가"
]

# Create a DataFrame with random interpretations
df = pd.DataFrame({
    'interpretation': [random.choice(random_interpretations) for _ in range(100)]
})

# Initialize BERTopic model
topic_model = BERTopic(
    verbose=True,
    embedding_model="bespin-global/klue-sroberta-base-continue-learning-by-mnr",
    min_topic_size=4,
    nr_topics=12  # Set a fixed number of topics
)

# Load pre-trained SentenceTransformer model
model = SentenceTransformer("bespin-global/klue-sroberta-base-continue-learning-by-mnr")

# Add embedding column to df using sentence-BERT
df['embedding'] = df['interpretation'].apply(model.encode)

# Create a list of interpretations
interpretation_list = df['interpretation'].tolist()

# Fit the BERTopic model
topics, probs = topic_model.fit_transform(interpretation_list)

# Set topic labels
topic_model.set_topic_labels({
    -1: "미분류",
    0: "캐러셀 적용, 웹페이지 디자인",
    1: "데이터셋 업데이트",
    2: "백엔드 기능개선(교사용 기록 확인, 새 서버, 국어사전, 임베딩 모델 변경)",
    3: "사용 중 오류 수정(장고 static 등)",
    4: "GPU 필요 시 사용량 최적화",
    5: "Chainlit 사용 관련 시행착오",
    6: "학년별 운영",
    7: "GitHub 연동 관련 시행착오",
    8: "보안 강화",
    9: "구글 검색 등록",
    10: "index.html 수정",
    11: "UUID 추가"
})

# Visualize documents
topic_model.visualize_documents(interpretation_list, custom_labels=True)
```
The output for which is as follows:
![newplot](https://github.com/user-attachments/assets/280c2b54-5546-4b93-a7b9-a9ce439f837a)

To address your error and move forward efficiently, I kindly request that you provide the following:
1. Please run `transformers-cli env` and share the resulting output to provide a full dependency and config list.
2. If possible, upload the Dataset and Model files to 🤗Hugging Face or GitHub, and share a link for these so that we can match a reproducible code with yours.
3. As the current code has a few discrepancies, please provide a clear, reproducible code in the order it should be run, so we can accurately the replicate the bug in the same environment.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.