KeyError: 'model.layers.0.self_attn.rotary_emb.cos_cached'

tonyaw: <p>I’m trying to use NVIDIA GeForce RTX 2080 SUPER to run gpt4-x-alpaca(<a href="https://huggingface.co/chavinlo/gpt4-x-alpaca/" class="inline-onebox" rel="noopener nofollow ugc">chavinlo/gpt4-x-alpaca · Hugging Face</a>) via koldAI, and got following error. It only happens when I try to use GPU. If I just uses CPU, no error with same pytorch. Any idea/suggestion for this error?<br>
ERROR | <strong>main</strong>:generate:4945 - Traceback (most recent call last):<br>
File “aiserver.py”, line 4934, in generate<br>
genout, already_generated = tpool.execute(_generate, txt, minimum, maximum, found_entries)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/eventlet/tpool.py”, line 132, in execute<br>
six.reraise(c, e, tb)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/six.py”, line 719, in reraise<br>
raise value<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/eventlet/tpool.py”, line 86, in tworker<br>
rv = meth(*args, **kwargs)<br>
File “aiserver.py”, line 4857, in _generate<br>
genout = generator(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/autograd/grad_mode.py”, line 27, in decorate_context<br>
return func(*args, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/generation/utils.py”, line 1485, in generate<br>
return self.sample(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/generation/utils.py”, line 2524, in sample<br>
outputs = self(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/hooks.py”, line 165, in new_forward<br>
output = old_forward(*args, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py”, line 687, in forward<br>
outputs = self.model(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py”, line 577, in forward<br>
layer_outputs = decoder_layer(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/hooks.py”, line 165, in new_forward<br>
output = old_forward(*args, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py”, line 292, in forward<br>
hidden_states, self_attn_weights, present_key_value = self.self_attn(<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/hooks.py”, line 165, in new_forward<br>
output = old_forward(*args, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py”, line 203, in forward<br>
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/hooks.py”, line 160, in new_forward<br>
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/hooks.py”, line 280, in pre_forward<br>
set_module_tensor_to_device(module, name, self.execution_device, value=self.weights_map[name])<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/utils/offload.py”, line 123, in <strong>getitem</strong><br>
return self.dataset[f"{self.prefix}{key}"]<br>
File “/home/tonyaw/sandbox//koboldai-client/runtime/envs/koboldai/lib/python3.8/site-packages/accelerate/utils/offload.py”, line 170, in <strong>getitem</strong><br>
weight_info = self.index[key]<br>
KeyError: ‘model.layers.0.self_attn.rotary_emb.cos_cached’</p>

ptrblck: <p>I’m not familiar with <code>koboldai</code>, but it seems its internal <code>offload</code> function is raising the issue as an indexing operation is failing.<br>
Could you explain what this feature does and if specific requirements are set?</p>

tonyaw: <p>The code is here：</p><aside class="onebox githubblob" data-onebox-src="https://github.com/huggingface/accelerate/blob/v0.18.0/src/accelerate/utils/offload.py#L170">
  <header class="source">

      <a href="https://github.com/huggingface/accelerate/blob/v0.18.0/src/accelerate/utils/offload.py#L170" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/huggingface/accelerate/blob/v0.18.0/src/accelerate/utils/offload.py#L170" target="_blank" rel="noopener nofollow ugc">huggingface/accelerate/blob/v0.18.0/src/accelerate/utils/offload.py#L170</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="160" style="counter-reset: li-counter 159 ;">
          <li>            index = json.load(f)</li>
          <li>    self.index = {} if index is None else index</li>
          <li>    self.all_keys = list(self.state_dict.keys())</li>
          <li>    self.all_keys.extend([key for key in self.index if key not in self.all_keys])</li>
          <li>    self.device = device</li>
          <li>
          </li>
<li>def __getitem__(self, key: str):</li>
          <li>    # State dict gets priority</li>
          <li>    if key in self.state_dict:</li>
          <li>        return self.state_dict[key]</li>
          <li class="selected">    weight_info = self.index[key]</li>
          <li>    if weight_info.get("safetensors_file") is not None:</li>
          <li>        if not is_safetensors_available():</li>
          <li>            raise ImportError("These offloaded weights require the use of safetensors: `pip install safetensors`.")</li>
          <li>
          </li>
<li>        if "SAFETENSORS_FAST_GPU" not in os.environ:</li>
          <li>            logger.info("Enabling fast loading with safetensors by setting `SAFETENSORS_FAST_GPU` to 1.")</li>
          <li>            os.environ["SAFETENSORS_FAST_GPU"] = "1"</li>
          <li>
          </li>
<li>        from safetensors import safe_open</li>
          <li>
      </li>
</ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
It looks like it is a missed layer, so I assume it shall be pytorch issue instead of accelerate issue. <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"><br>
May I ask if “self_attn.rotary_emb.cos_cached” mentioned here is some torch layer?</p>

ptrblck: <aside class="quote no-group" data-username="tonyaw" data-post="3" data-topic="177229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tonyaw/48/59354_2.png" class="avatar"> tonyaw:</div>
<blockquote>
<p>May I ask if “self_attn.rotary_emb.cos_cached” mentioned here is some torch layer?</p>
</blockquote>
</aside>
<p>It could be, but it’s not known since you have only posted an attribute name of the model.</p>