beyond/genius-large-k2t

💡GENIUS – generating text using sketches!

• Paper: GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation

💡GENIUS is a powerful conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large- scale textual corpus with a novel reconstruction from sketch objective using an extreme and selective masking strategy, enabling it to generate diverse and high-quality texts given sketches.

• Models hosted in 🤗 Huggingface:

Model variations:

Model	#params	Language	comment
genius-large	406M	English	The version used in paper (recommend)
genius-large-k2t	406M	English	keywords-to-text
genius-base	139M	English	smaller version
genius-base-ps	139M	English	pre-trained both in paragraphs and short sentences
genius-base-chinese	116M	中文	在一千万纯净中文段落上预训练