---
library_name: transformers
base_model: EleutherAI/pythia-14m
tags:
- generated_from_trainer
model-index:
- name: pythia-14m-finewebedu-sentences
  results: []
---

# pythia-14m-finewebedu-sentences

- Generate half intelligible English sentences using a small GPT like model.
- Will output one sentence at a time.

This model is a fine-tuned version of [EleutherAI/pythia-14m](https://huggingface.co/EleutherAI/pythia-14m) on the [agentlans/finewebedu-sentences dataset](https://huggingface.co/datasets/agentlans/finewebedu-sentences).

## Model description

To generate 10 random sentences starting from an empty string on a CUDA device:

```python
from transformers import pipeline, set_seed

generator = pipeline('text-generation', model='agentlans/pythia-14m-finewebedu-sentences', device='cuda')

set_seed(1234)
results = generator("", max_length=100, num_return_sequences=10, do_sample=True)

for x in results:
	print(x['generated_text'])
```

Output:
```text
The main outcome is that the group is associated with other domains.
If you're planning to get a long-term answer, you can check the link "watch" and see that you can change your website.
They are very difficult to make it easy to understand how it works as a healthy.
In the most cases, the prevalence of DTP is reduced from 5-HT1.
It is a significant difference between the risk of injury, and there is no need to be a single complication.
The time of taking too high on the scale of the region is to begin, with a bit other type of view to the whole system.
The total cost of your daily distribution is $24. The overall number of children is 0.5 times is 50.
The more difficult time is to learn the basics of the work, but it is important to do the same job.
It is now on the other hand, however, in the middle of the 19th century, the country must follow the law of the country and alliances between the countries and communities.
This is why it is hard to do this.
```

## Intended uses & limitations

- For generating short lines of English text
- Could be useful for
  - data augmentation
  - creative inspiration
  - entertainment
  - CAPTCHA
- Can be further finetuned on other data such as:
  - prompts
  - famous quotes
  - news headlines
  - blog post titles

Limitations include:

- Not guaranteed to make sensible, coherent, or grammatically correct sentences
- No regard for accuracy or truthfulness whatsoever
  - It's a bunch of words from a probability model, what do you expect?

## Training and evaluation data

Sentences from [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 15.0

### Framework versions

- Transformers 4.44.2
- Pytorch 2.2.2+cu121
- Datasets 2.18.0
- Tokenizers 0.19.1