File size: 4,992 Bytes
c1b2943 727fd6c c1b2943 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 |
---
license: cc-by-nc-sa-4.0
language:
- en
- zh
base_model: Krystalan/DRT-o1-7B
tags:
- machine tranlsation
- O1-like model
- Chat
- llama-cpp
- gguf-my-repo
pipeline_tag: text-generation
---
# Triangle104/DRT-o1-7B-Q6_K-GGUF
This model was converted to GGUF format from [`Krystalan/DRT-o1-7B`](https://huggingface.co/Krystalan/DRT-o1-7B) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/Krystalan/DRT-o1-7B) for more details on the model.
---
Model details:
-
This repository contains the resources for our paper "DRT-o1: Optimized Deep Reasoning Translation via Long Chain-of-Thought"
Updates:
2024.12.24: We released our paper. Check it out!
2024.12.23: We released our model checkpoints. 🤗 DRT-o1-7B and 🤗 DRT-o1-14B.
Introduction
In this work, we introduce DRT-o1, an attempt to bring the success of long thought reasoning to neural machine translation (MT). To this end,
🌟 We mine English sentences with similes or metaphors from existing literature books, which are suitable for translation via long thought.
🌟 We propose a designed multi-agent framework with three agents (i.e., a translator, an advisor and an evaluator) to synthesize the MT samples with long thought. There are 22,264 synthesized samples in total.
🌟 We train DRT-o1-7B and DRT-o1-14B using Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct as backbones.
Quickstart
⛷️ Huggingface Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Krystalan/DRT-o1-7B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
prompt = "Please translate the following text from English to Chinese:\nThe mother, with her feet propped up on a stool, seemed to be trying to get to the bottom of that answer, whose feminine profundity had struck her all of a heap."
messages = [
{"role": "system", "content": "You are a philosopher skilled in deep thinking, accustomed to exploring complex problems with profound insight."},
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=2048
)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
⛷️ vllm
Deploying LLMs:
python3 -m vllm.entrypoints.openai.api_server --model [model_ckpt] --served-model-name [model_name]
Calling LLMs:
from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
chat_response = client.chat.completions.create(
model=[model_name],
messages=[
{"role": "system", "content": "You are a philosopher skilled in deep thinking, accustomed to exploring complex problems with profound insight."},
{"role": "user", "content": "Please translate the following text from English to Chinese:\nThe mother, with her feet propped up on a stool, seemed to be trying to get to the bottom of that answer, whose feminine profundity had struck her all of a heap."},
],
temperature=0.7,
top_p=0.8,
max_tokens=2048,
extra_body={
"repetition_penalty": 1.05,
},
)
print("Chat response:", chat_response)
License
This work is licensed under cc-by-nc-sa-4.0
---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
```bash
brew install llama.cpp
```
Invoke the llama.cpp server or the CLI.
### CLI:
```bash
llama-cli --hf-repo Triangle104/DRT-o1-7B-Q6_K-GGUF --hf-file drt-o1-7b-q6_k.gguf -p "The meaning to life and the universe is"
```
### Server:
```bash
llama-server --hf-repo Triangle104/DRT-o1-7B-Q6_K-GGUF --hf-file drt-o1-7b-q6_k.gguf -c 2048
```
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```
Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/DRT-o1-7B-Q6_K-GGUF --hf-file drt-o1-7b-q6_k.gguf -p "The meaning to life and the universe is"
```
or
```
./llama-server --hf-repo Triangle104/DRT-o1-7B-Q6_K-GGUF --hf-file drt-o1-7b-q6_k.gguf -c 2048
```
|