augmxnt

/

shisa-gamma-7b-v1

like 13

I executed the following code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "augmxnt_shisa-gamma-7b-v1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
  model_name,
  torch_dtype="auto",
)
seed = 23  
torch.manual_seed(seed)
model.cuda()
inputs = tokenizer("AI で科学研究を加速するには、", return_tensors="pt").to("cuda")
tokens = model.generate(
  **inputs,
  max_new_tokens=64,
  temperature=0.75,
  top_p=0.95,
  do_sample=True,
  pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))
# AI で科学研究を加速するには、これらの課題に取り組み、最先端の AI を科学研究に適用することが必要です。
# Google Cloud は、次のような科学研究を加速するための AI

The results were as follows.

Traceback (most recent call last):
  File "D:\text-generation-webui\models\_test_augmxnt_shisa-gamma-7b-v1.py", line 13, in <module>
    tokens = model.generate(
  File "D:\text-generation-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\generation\utils.py", line 1764, in generate
    return self.sample(
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\generation\utils.py", line 2861, in sample
    outputs = self(
  File "D:\text-generation-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\text-generation-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1053, in forward
    outputs = self.model(
  File "D:\text-generation-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\text-generation-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 908, in forward
    attention_mask = _prepare_4d_causal_attention_mask(
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\modeling_attn_mask_utils.py", line 306, in _prepare_4d_causal_attention_mask
    attention_mask = attn_mask_converter.to_4d(
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\modeling_attn_mask_utils.py", line 121, in to_4d
    causal_4d_mask = self._make_causal_mask(
  File "D:\text-generation-webui\venv\lib\site-packages\transformers\modeling_attn_mask_utils.py", line 156, in _make_causal_mask
    mask_cond = torch.arange(mask.size(-1), device=device)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

However, it seemed that text was generated when I copied and executed the tokenizer from stabilityai/japanese-stablelm-base-gamma-7b.