README.md

---
license: apache-2.0
datasets:
- pg19
tags:
- mamba
metrics:
- perplexity
pipeline_tag: text-generation
language:
- en
---

Train in 150B Byte. Mode size 972M. Table 3 in [MambaByte](https://arxiv.org/abs/2401.13660)

Traning dataset [PG-19](https://github.com/google-deepmind/pg19)

To use

```
import torch
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel

import numpy as np

model=MambaLMHeadModel.from_pretrained("JunxiongWang/MambaByte_PG19_972M", device='cuda', dtype=torch.float32)

text = "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife,"
text_byte = np.frombuffer(text.encode('utf-8'), dtype=np.uint8)
input_ids = torch.from_numpy(text_byte[None, :].copy()).long().cuda()

sample = model.generate(
    input_ids=input_ids,
    max_length=2048,
    cg=True,
    return_dict_in_generate=True,
    output_scores=True,
    enable_timing=True,
    temperature=1,
    top_k=256,
    top_p=0.9,
)

print(bytes(sample.sequences[0].tolist()).decode('utf-8'))
```

Output

```
It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife, and that a small one must be always spoilt by a bad one, or else it will cease to be so. I can safely affirm, that married women, in general, are the best fortunes for a wife; because their education so near a resemblance to the father's, while it expands their understandings, so enlarges their views, and strengthens their passions, that they give birth to more children than the father, who can only say, _Je meurs, or je mourrai_.

That most of the proposals at present made for the abolition of marriage in this country have been rejected, is proof sufficient of the truth of the remark; but even in that case, I would add, that any plan which proposed to diminish the population in the country would be a proposal incompatible with that feeling of refinement and good order, which the situation of our women demands. In short, let me state the proposal for this purpose as it is at present in Paris. Monsieur, who knows how to write, and is a man of letters, has already translated the translation of an English letter from a periodical work, that I might peruse the whole with more than ordinary attention.


                         "THE CELIBACY OF THE PRIESTHOOD.

"It is proposed to forbid marriage, by making it a sacrament to all those who shall take the vows, and the brothers and sisters of the society are required, under the severest penalties, to promise that they will allow no illicit connections whatsoever to exist between them and those of other orders, nor accept any communication from them.

"In order to give full effect to the obligations of marriage, the obligation is imposed of fulfilling all the points which are enjoined by the Catholic religion on persons of both sexes. Besides this, it is proposed to legislate in the interest of sensual pleasures and of immorality.

"In relation to marriage, the celibacy of the clergy is in particular denounced as a grievance. But this also is a
```