CohereForAI

/

c4ai-command-r-v01

like 968

Trying to use this model as part of a pipeline (4x A100 80GB)

import torch
from transformers import pipeline, AutoTokenizer, BitsAndBytesConfig


bnb_config = BitsAndBytesConfig(load_in_8bit=True)
model_kwargs = {
       "quantization_config": bnb_config,
}

tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01")
pipe = pipeline(
                "conversational",
                model="CohereForAI/c4ai-command-r-v01",
                tokenizer=tokenizer,
                model_kwargs=model_kwargs,
                trust_remote_code=True,
                device_map="auto",
                torch_dtype=torch.bfloat16,
)

messages = [{"role": "user", "content": "Hello, how are you?"}]
response = pipe(messages)
print(response)

But getting this error in the forward pass:

│ /dodrio/scratch/projects/2023_071/cache/huggingface/modules/transformers_mod │
│ ules/CohereForAI/c4ai-command-r-v01/2a6d259c29bd319c3bdb8dd88b8d59b8c303c318 │
│ /modeling_cohere.py:1060 in _update_causal_mask                              │
│                                                                              │
│   1057 │   │   │   │   # Attend to all tokens in fully masked rows in the ca │
│   1058 │   │   │   │   # using left padding. This is required by F.scaled_do │
│   1059 │   │   │   │   # Details: https://github.com/pytorch/pytorch/issues/ │
│ ❱ 1060 │   │   │   │   causal_mask = AttentionMaskConverter._unmask_unattend │
│   1061 │   │                                                                 │
│   1062 │   │   return causal_mask                                            │
│   1063

TypeError: AttentionMaskConverter._unmask_unattended() missing 1 required positional argument: 'unmasked_value'

Any thoughts on this?