File size: 6,110 Bytes
ea64011
 
 
77fbbc5
ea64011
 
 
 
 
 
 
 
 
 
 
 
9488e4b
e8b80bb
 
ea64011
 
b8fbe09
2e18b8c
 
925543a
 
 
c2fa2d4
5cd3b4e
e8b80bb
ea64011
 
e8b80bb
ea64011
 
 
 
 
 
 
 
 
 
 
 
 
77fbbc5
ea64011
e8b80bb
ea64011
 
 
 
 
 
 
77fbbc5
ea64011
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b06b002
 
 
e8b80bb
 
ea64011
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b06b002
 
e8b80bb
ea64011
 
e8b80bb
 
ea64011
d5feda7
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
---
license: cc-by-sa-3.0
datasets:
- VMware/open-instruct
language:
- en
library_name: transformers
pipeline_tag: text-generation
---

# VMware/open-llama-7B-v2-open-instruct
Instruction-tuned version of the fully trained Open LLama 7B v2  model. The model is open for <b>COMMERCIAL USE</b>. <br>

- This model performs better on code compared to v1 due to the improvements made on the base model by the openlm-research team.
- The instruction model is trained on an improved instruction tuning dataset compared to v1

**NOTE**: The model was trained using the Alpaca prompt template <br>
**NOTE**: Fast tokenizer results in incorrect encoding, set the ```use_fast = False``` parameter, when instantiating the tokenizer


## License
- CC BY-SA-3.0 **(Commercially Viable!)**
- Base Language Model ([openlm-research/open_llama_v2_7b](https://huggingface.co/openlm-research/open_llama_v2_7b)) is under apache-2.0
- Fine-Tuning Dataset ([VMware/open-instruct](https://huggingface.co/datasets/VMware/open-instruct)) is under cc-by-sa-3.0

## Datasets used for Fine-Tuning

### Open-instruct

**Open-instruct-v1**
- Mosaic/Dolly-HHRLHF + filtered  OASST1 - cc by 3.0 

**Subset of COT SUBMIX (FROM FLAN V2) Zeroshot examples**
- ESNLI -  MIT 
- ECQA  - CDLA 1.0 - Sharing
- Strategy  - MIT
- CREAK  - MIT
- gsmk8 - MIT
- aqua  - MIT
- qasc  - Apache 2.0


## Nomenclature 

- Model : Open-llama-v2
- Model Size: 7B parameters
- Dataset: Open-instruct


## Use in Transformers

```
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'VMware/open-llama-7b-v2-open-instruct'


tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map='sequential')

prompt_template = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:"

prompt = """What is attention mechanism of a transformer model? 
 Write a python code to illustrate how attention works within a transformer model using numpy library. Donot use pytorch or tensorflow."""


inputt = prompt_template.format(instruction= prompt)
input_ids = tokenizer(inputt, return_tensors="pt").input_ids.to("cuda")

output1 = model.generate(input_ids, max_length=512)
input_length = input_ids.shape[1]
output1 = output1[:, input_length:]
output = tokenizer.decode(output1[0])

print(output)

```


### Output

Sure, I can help you with that!

Attention mechanisms in transformer models are typically implemented using the attention mechanism in the self-attention layer. Self-attention allows the model to focus on different parts of the input sequence when processing it. This is achieved by computing a set of attention weights, which are used to weigh the contribution of each input element to the output.

Here's an example code using NumPy to illustrate how attention works in a transformer model:

```python
import numpy as np

def attention_weights(query, key, value, mask):
    # Query, key, and value are input tensors. Mask is a tensor of zeros and ones that represents the attention mask.
    # It is used to prevent the model from attending to certain positions in the input sequence if they are not relevant.
    # The attention weights are the element-wise product of the query, key, and mask tensors.
    # The result is a tensor of the same shape as the query tensor.
    
    # Compute the dot product between the query tensor and the key tensor
    dot = np.matmul(query, key)
    
    # Compute the element-wise softmax of the dot product tensor
    exp_dot = np.exp(dot)
    
    # Multiply the dot product and the softmax of the dot product tensors
    weights = dot * exp_dot
    
    # Return the attention weights as a NumPy tensor
    return weights

# Define the input sequence
query = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
key = np.array([[0.1, 0.2], [0.3, 0.4]])
value = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
mask = np.array([[False, True, True], [False, True, True]])

# Compute the attention weights
weights = attention_weights(query, key, value, mask)

# Print the attention weights
print(weights)
```

In this example, the `attention_weights` function takes as input the query tensor, key tensor, value tensor, and mask tensor. It computes the dot product between the query and key tensors using the `np.matmul` function, and then applies a softmax function using the `np.exp` function to the element-wise dot product tensor. It then multiplies the dot product and softmax tensors using the `np.matmul` function, and returns the result as a NumPy tensor.

The `query`, `key`, and `value` tensors represent the input sequence to the transformer model. The `mask` tensor represents the attention mask, which is used to prevent the model from attending to certain positions in the input sequence if they are not relevant.

The output of the `attention_weights` function is a NumPy tensor that represents the attention weights for the input sequence. These weights are used by the transformer model to weigh the contribution of each input element to the output.

I hope this helps!</s>
<hr>


## Finetuning details
The finetuning scripts will be available in our [RAIL Github Repository](https://github.com/vmware-labs/research-and-development-artificial-intelligence-lab/tree/main/instruction-tuning)




# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Vmware__open-llama-7b-v2-open-instruct)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 40.34   |
| ARC (25-shot)         | 39.76          |
| HellaSwag (10-shot)   | 70.31    |
| MMLU (5-shot)         | 35.16         |
| TruthfulQA (0-shot)   | 39.53   |
| Winogrande (5-shot)   | 64.33   |
| GSM8K (5-shot)        | 7.43        |
| DROP (3-shot)         | 25.88         |