File size: 6,941 Bytes
cbb874e
 
 
 
20f3bda
cbb874e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6861d86
 
cbb874e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
---
language:
- en
- ko
license: llama3
library_name: transformers
tags:
- llama-cpp
- gguf-my-repo
base_model:
- meta-llama/Meta-Llama-3-70B
- jeiku/Average_Test_v1
- Bllossom/llama-3-Korean-Bllossom-70B
---


<a href="https://github.com/MLP-Lab/Bllossom">
  <img src="https://github.com/teddysum/bllossom/blob/main//bllossom_icon.png?raw=true" width="40%" height="50%">
</a>

# Bllossom | [Demo]() | [Homepage](https://www.bllossom.ai/) | [Github](https://github.com/MLP-Lab/Bllossom) | [Colab-tutorial](https://colab.research.google.com/drive/1fBOzUVZ6NRKk_ugeoTbAOokWKqSN47IG?usp=sharing) |


```bash
์ €ํฌ BllossomํŒ€ ์—์„œ ํ•œ๊ตญ์–ด-์˜์–ด ์ด์ค‘ ์–ธ์–ด๋ชจ๋ธ์ธ Bllossom์„ ๊ณต๊ฐœํ–ˆ์Šต๋‹ˆ๋‹ค!
์„œ์šธ๊ณผ๊ธฐ๋Œ€ ์Šˆํผ์ปดํ“จํŒ… ์„ผํ„ฐ์˜ ์ง€์›์œผ๋กœ 100GB๊ฐ€๋„˜๋Š” ํ•œ๊ตญ์–ด๋กœ ๋ชจ๋ธ์ „์ฒด๋ฅผ ํ’€ํŠœ๋‹ํ•œ ํ•œ๊ตญ์–ด ๊ฐ•ํ™” ์ด์ค‘์–ธ์–ด ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค!
ํ•œ๊ตญ์–ด ์ž˜ํ•˜๋Š” ๋ชจ๋ธ ์ฐพ๊ณ  ์žˆ์ง€ ์•Š์œผ์…จ๋‚˜์š”?
 - ํ•œ๊ตญ์–ด ์ตœ์ดˆ! ๋ฌด๋ ค 3๋งŒ๊ฐœ๊ฐ€ ๋„˜๋Š” ํ•œ๊ตญ์–ด ์–ดํœ˜ํ™•์žฅ
 - Llama3๋Œ€๋น„ ๋Œ€๋žต 25% ๋” ๊ธด ๊ธธ์ด์˜ ํ•œ๊ตญ์–ด Context ์ฒ˜๋ฆฌ๊ฐ€๋Šฅ
 - ํ•œ๊ตญ์–ด-์˜์–ด Pararell Corpus๋ฅผ ํ™œ์šฉํ•œ ํ•œ๊ตญ์–ด-์˜์–ด ์ง€์‹์—ฐ๊ฒฐ (์‚ฌ์ „ํ•™์Šต)
 - ํ•œ๊ตญ์–ด ๋ฌธํ™”, ์–ธ์–ด๋ฅผ ๊ณ ๋ คํ•ด ์–ธ์–ดํ•™์ž๊ฐ€ ์ œ์ž‘ํ•œ ๋ฐ์ดํ„ฐ๋ฅผ ํ™œ์šฉํ•œ ๋ฏธ์„ธ์กฐ์ •
 - ๊ฐ•ํ™”ํ•™์Šต
์ด ๋ชจ๋“ ๊ฒŒ ํ•œ๊บผ๋ฒˆ์— ์ ์šฉ๋˜๊ณ  ์ƒ์—…์  ์ด์šฉ์ด ๊ฐ€๋Šฅํ•œ Bllossom์„ ์ด์šฉํ•ด ์—ฌ๋Ÿฌ๋ถ„ ๋งŒ์˜ ๋ชจ๋ธ์„ ๋งŒ๋“ค์–ด๋ณด์„ธ์šฅ!
๋ณธ ๋ชจ๋ธ์€ 42GB GPU์—์„œ ๊ตฌ๋™ ๊ฐ€๋Šฅํ•œ ์–‘์žํ™” ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค!

1. Bllossom-8B๋Š” ์„œ์šธ๊ณผ๊ธฐ๋Œ€, ํ…Œ๋””์ธ, ์—ฐ์„ธ๋Œ€ ์–ธ์–ด์ž์› ์—ฐ๊ตฌ์‹ค์˜ ์–ธ์–ดํ•™์ž์™€ ํ˜‘์—…ํ•ด ๋งŒ๋“  ์‹ค์šฉ์ฃผ์˜๊ธฐ๋ฐ˜ ์–ธ์–ด๋ชจ๋ธ์ž…๋‹ˆ๋‹ค! ์•ž์œผ๋กœ ์ง€์†์ ์ธ ์—…๋ฐ์ดํŠธ๋ฅผ ํ†ตํ•ด ๊ด€๋ฆฌํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค ๋งŽ์ด ํ™œ์šฉํ•ด์ฃผ์„ธ์š” ๐Ÿ™‚
2. ์ดˆ ๊ฐ•๋ ฅํ•œ Advanced-Bllossom 8B, 70B๋ชจ๋ธ, ์‹œ๊ฐ-์–ธ์–ด๋ชจ๋ธ์„ ๋ณด์œ ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค! (๊ถ๊ธˆํ•˜์‹ ๋ถ„์€ ๊ฐœ๋ณ„ ์—ฐ๋ฝ์ฃผ์„ธ์š”!!)
3. Bllossom์€ NAACL2024, LREC-COLING2024 (๊ตฌ๋‘) ๋ฐœํ‘œ๋กœ ์ฑ„ํƒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
4. ์ข‹์€ ์–ธ์–ด๋ชจ๋ธ ๊ณ„์† ์—…๋ฐ์ดํŠธ ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค!! ํ•œ๊ตญ์–ด ๊ฐ•ํ™”๋ฅผ์œ„ํ•ด ๊ณต๋™ ์—ฐ๊ตฌํ•˜์‹ค๋ถ„(ํŠนํžˆ๋…ผ๋ฌธ) ์–ธ์ œ๋“  ํ™˜์˜ํ•ฉ๋‹ˆ๋‹ค!! 
   ํŠนํžˆ ์†Œ๋Ÿ‰์˜ GPU๋ผ๋„ ๋Œ€์—ฌ ๊ฐ€๋Šฅํ•œํŒ€์€ ์–ธ์ œ๋“  ์—ฐ๋ฝ์ฃผ์„ธ์š”! ๋งŒ๋“ค๊ณ  ์‹ถ์€๊ฑฐ ๋„์™€๋“œ๋ ค์š”.
```

The Bllossom language model is a Korean-English bilingual language model based on the open-source LLama3. It enhances the connection of knowledge between Korean and English. It has the following features:

* **Knowledge Linking**: Linking Korean and English knowledge through additional training
* **Vocabulary Expansion**: Expansion of Korean vocabulary to enhance Korean expressiveness.
* **Instruction Tuning**: Tuning using custom-made instruction following data specialized for Korean language and Korean culture
* **Human Feedback**: DPO has been applied
* **Vision-Language Alignment**: Aligning the vision transformer with this language model 

**This model developed by [MLPLab at Seoultech](http://mlp.seoultech.ac.kr), [Teddysum](http://teddysum.ai/) and [Yonsei Univ](https://sites.google.com/view/hansaemkim/hansaem-kim).** 
**This model was converted to GGUF format from [`Bllossom/llama-3-Korean-Bllossom-70B`](https://huggingface.co/Bllossom/llama-3-Korean-Bllossom-70B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/Bllossom/llama-3-Korean-Bllossom-70B) for more details on the model.**


## Demo Video

<div style="display: flex; justify-content: space-between;">
  <!-- ์ฒซ ๋ฒˆ์งธ ์ปฌ๋Ÿผ -->
  <div style="width: 49%;">
    <a>
      <img src="https://github.com/lhsstn/lhsstn/blob/main/x-llava_dem.gif?raw=true" style="width: 100%; height: auto;">
    </a>
    <p style="text-align: center;">Bllossom-V Demo</p>
  </div>

  <!-- ๋‘ ๋ฒˆ์งธ ์ปฌ๋Ÿผ (ํ•„์š”ํ•˜๋‹ค๋ฉด) -->
  <div style="width: 49%;">
    <a>
      <img src="https://github.com/lhsstn/lhsstn/blob/main/bllossom_demo_kakao.gif?raw=true" style="width: 70%; height: auto;">
    </a>
    <p style="text-align: center;">Bllossom Demo(Kakao)ใ…คใ…คใ…คใ…คใ…คใ…คใ…คใ…ค</p>
  </div>
</div>



## NEWS
* [2024.05.08] Vocab Expansion Model Update
* [2024.04.25] We released Bllossom v2.0, based on llama-3
* [2023/12] We released Bllossom-Vision v1.0, based on Bllossom
* [2023/08] We released Bllossom v1.0, based on llama-2. 
* [2023/07] We released Bllossom v0.7, based on polyglot-ko.


## Example code


## Use with llama.cpp

Install llama.cpp through brew.

```bash
brew install ggerganov/ggerganov/llama.cpp
```
Invoke the llama.cpp server or the CLI.

CLI:

```bash
llama-cli --hf-repo Bllossom/llama-3-Korean-Bllossom-70B-gguf-Q4_K_M --model bllossom_llama3_70b.Q4_K_M.gguf -p "์„œ์šธ๊ณผํ•™๊ธฐ์ˆ ๋Œ€ํ•™๊ต ์ž„๊ฒฝํƒœ ๊ต์ˆ˜๋Š” ์–ด๋–ค์—ฐ๊ตฌ๋ฅผํ•˜๋‹ˆ?"
```

Server:

```bash
llama-server --hf-repo Bllossom/llama-3-Korean-Bllossom-70B-gguf-Q4_K_M --model bllossom_llama3_70b.Q4_K_M.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

```
git clone https://github.com/ggerganov/llama.cpp &&             cd llama.cpp &&             make &&             ./main -m bllossom_llama3_70b.Q4_K_M.gguf -n 128
```




## Citation
**Language Model**
```text
@misc{bllossom,
  author = {ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim},
  title = {Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean},
  year = {2024},
  journal = {LREC-COLING 2024},
  paperLink = {\url{https://arxiv.org/pdf/2403.10882}},
 },
}
```

**Vision-Language Model**
```text
@misc{bllossom-V,
  author = {Dongjae Shin, Hyunseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae Lim},
  title = {X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment},
  year = {2024},
  publisher = {GitHub},
  journal = {NAACL 2024 findings},
  paperLink = {\url{https://arxiv.org/pdf/2403.11399}},
 },
}
```

## Contact
 - ์ž„๊ฒฝํƒœ(KyungTae Lim), Professor at Seoultech. `ktlim@seoultech.ac.kr`
 - ํ•จ์˜๊ท (Younggyun Hahm), CEO of Teddysum. `hahmyg@teddysum.ai`
 - ๊น€ํ•œ์ƒ˜(Hansaem Kim), Professor at Yonsei. `khss@yonsei.ac.kr`

## Contributor
 - ์ตœ์ฐฝ์ˆ˜(Chansu Choi), choics2623@seoultech.ac.kr
 - ๊น€์ƒ๋ฏผ(Sangmin Kim), sangmin9708@naver.com
 - ์›์ธํ˜ธ(Inho Won), wih1226@seoultech.ac.kr
 - ๊น€๋ฏผ์ค€(Minjun Kim), mjkmain@seoultech.ac.kr 
 - ์†ก์Šน์šฐ(Seungwoo Song), sswoo@seoultech.ac.kr
 - ์‹ ๋™์žฌ(Dongjae Shin), dylan1998@seoultech.ac.kr
 - ์ž„ํ˜„์„(Hyeonseok Lim), gustjrantk@seoultech.ac.kr
 - ์œก์ •ํ›ˆ(Jeonghun Yuk), usually670@gmail.com
 - ์œ ํ•œ๊ฒฐ(Hangyeol Yoo), 21102372@seoultech.ac.kr
 - ์†ก์„œํ˜„(Seohyun Song), alexalex225225@gmail.com