wortegaLM-1b / README.md
AlexWortega's picture
Create README.md
efc5a1b
---
license: apache-2.0
datasets:
- IlyaGusev/rulm
inference:
parameters:
min_length: 20
max_new_tokens: 250
top_k: 50
top_p: 0.9
early_stopping: true
no_repeat_ngram_size: 2
use_cache: true
repetition_penalty: 1.5
length_penalty: 0.8
num_beams: 2
language:
- ru
library_name: transformers
pipeline_tag: text-generation
tags:
- finance
- code
---
<h1 style="font-size: 42px">WortegaLM 109m<h1/>
# Model Summary
> Это GPTneo like модель обученная с нуля на сете в 95gb кода, хабра, пикабу, новостей(порядка 12B токенов) Она умеет решать примитивные задачи, не пригодна для ZS FS, но идеальна как модель для студенческих проектов
# Quick Start
```python
from transformers import AutoTokenizer, AutoModelForCausalLM,
tokenizer = AutoTokenizer.from_pretrained('AlexWortega/wortegaLM',padding_side='left')
device = 'cuda'
model = AutoModelForCausalLM.from_pretrained('AlexWortega/wortegaLM')
model.resize_token_embeddings(len(tokenizer))
model.to(device)
def generate_seqs(q,model, k=2):
gen_kwargs = {
"min_length": 20,
"max_new_tokens": 100,
"top_k": 50,
"top_p": 0.7,
"do_sample": True,
"early_stopping": True,
"no_repeat_ngram_size": 2,
"eos_token_id": tokenizer.eos_token_id,
"pad_token_id": tokenizer.eos_token_id,
"use_cache": True,
"repetition_penalty": 1.5,
"length_penalty": 1.2,
"num_beams": 4,
"num_return_sequences": k
}
t = tokenizer.encode(q, add_special_tokens=False, return_tensors='pt').to(device)
g = model.generate(t, **gen_kwargs)
generated_sequences = tokenizer.batch_decode(g, skip_special_tokens=False)
return generated_sequences
```