|
--- |
|
language: |
|
- en |
|
- zh |
|
tags: |
|
- GENIUS |
|
- conditional text generation |
|
- sketch-based text generation |
|
- data augmentation |
|
license: apache-2.0 |
|
datasets: |
|
- c4 |
|
- beyond/chinese_clean_passages_80m |
|
|
|
widget: |
|
- text: "<mask> Conference on Empirical Methods <mask> submission of research papers <mask> Deep Learning <mask>" |
|
example_title: "Example 1" |
|
- text: "<mask> machine learning <mask> my research interest <mask> data science <mask>" |
|
example_title: "Example 2" |
|
- text: "<mask> play basketball <mask> a strong team <mask> Shanghai University of Finance and Economics <mask> last Sunday <mask>" |
|
example_title: "Example 3" |
|
- text: "Good news: <mask> the European Union <mask> month by EU <mask> Farm Commissioner Franz <mask>" |
|
example_title: "Example with a prompt 1" |
|
- text: "Bad news: <mask> the European Union <mask> month by EU <mask> Farm Commissioner Franz <mask>" |
|
example_title: "Example with a prompt 2" |
|
|
|
inference: |
|
parameters: |
|
max_length: 200 |
|
num_beams: 3 |
|
do_sample: True |
|
--- |
|
|
|
# GENIUS: generating text using sketches! |
|
|
|
|
|
- **Paper: [GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation](https://arxiv.org/abs/2211.10330)** |
|
- **GitHub: [GENIUS, Pre-training/Data Augmentation Tutorial](https://github.com/beyondguo/genius)** |
|
|
|
|
|
|
|
|
|
|
|
You can use this model directly with a pipeline for masked language modeling: |
|
|
|
```python |
|
from transformers import pipeline |
|
# 1. load the model with the huggingface `pipeline` |
|
genius = pipeline("text2text-generation", model='beyond/genius-large', device=0) |
|
# 2. provide a sketch (joint by <mask> tokens) |
|
sketch = "<mask> Conference on Empirical Methods <mask> submission of research papers <mask> Deep Learning <mask>" |
|
# 3. here we go! |
|
generated_text = genius(sketch, num_beams=3, do_sample=True, max_length=200)[0]['generated_text'] |
|
print(generated_text) |
|
``` |
|
|
|
If you find our paper/code/demo useful, please cite our paper: |
|
``` |
|
@article{guo2022genius, |
|
title={GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation}, |
|
author={Guo, Biyang and Gong, Yeyun and Shen, Yelong and Han, Songqiao and Huang, Hailiang and Duan, Nan and Chen, Weizhu}, |
|
journal={arXiv preprint arXiv:2211.10330}, |
|
year={2022} |
|
} |
|
``` |
