Zhaoming213/WordPredit2

Introduce

WordPredit2 (second generation) is now available!!!

Key Update
Key highlights of this update: We have increased the number of model layers and implemented more aggressive data-cleansing strategies to effectively remove noise from the training datasets. Furthermore, we utilized shorter sentences for pre-training, extended the total pre-training duration, and incorporated larger datasets. These enhancements ensure greater model robustness, superior generalization, and an improved capacity for creative generation.

To investigate the divergence characteristics of small models while minimizing risks associated with black-box distillation corpora—such as tendencies toward moralizing, refusal, or accusation—this study employs a large volume of native Chinese web text for pre-training. By utilizing shorter context windows, we avoid unexpected model outputs often associated with long contexts and significantly reduce training costs and time; this approach also allows us to explore methods for enhancing model performance—such as adjusting dimensionality and layer depth—within a short-context framework.

Application Areas

This model is capable of tasks such as short-text prediction, text completion, brainstorming, and creative copywriting. You need not worry about it suddenly lecturing or refusing you; as a pre-trained model—rather than a conversational one—it relies on data that has undergone extensive cleaning to minimize any tendency toward preachiness or refusal.

Previous versions

First generation:
https://huggingface.co/Zhaoming213/WordPredit

Model training configuration

• GPU: T4 16 GB x 2
• Time:
  1. Pre-training：24h+

Model parameters

parameters	hidden_size	hidden_layers	attention_heads	max_seq_len	Parameter volume
ChatBox-unrestricted-normal	768	12	8	34	93.41M

Dialogue Example

Input: 同比增加了
Ouput: 120万美元

Input: 今天我市
Ouput: 对县域经济发展起到较大的作用

Input: 北京市
Ouput: 气象台将继续发布新冠疫情新情况

Input: 北京
Ouput: 市海淀区公路局组织召开新闻发布会

Input: 凯迪拉克汽车
Ouput: 官方宣布

Input: 老板我发现
Ouput: 了这个

Input: 老板我发现
Ouput: 自己也有这样的心理

Input: 我是一名
Ouput: 学生

Input: 我是一名
Ouput: 专职律师

Input: 我是一名
Ouput: 新生

Input: 本台记者
Ouput: 走访了该小区

Input: 企业的关键决策
Ouput: 就是要坚持客户第一的原则

Input: 我今天遇到了
Ouput: 很多人

Input: 什么游戏
Ouput: 最能打

Input: 我的电脑出现了
Ouput: 问题

Input: 中美关系
Ouput: 是日益紧张的

Input: 美元股票
Ouput: 继续上涨

Input: 在烹饪
Ouput: 过程中

Expected results

Based on the output, the model demonstrates a certain degree of divergent thinking and logic.

Datasets Resources

The dataset underwent two rounds of in-depth cleaning, ensuring its quality.

Pre-training:
https://huggingface.co/datasets/CASIA-LM/ChineseWebText2.0

Recommend

If you are looking for a small-scale, portable conversational model without usage limits, you might want to try the ChatBox series.

ChatBox series:
First Generation:
https://huggingface.co/Zhaoming213/ChatBox-unrestricted

Second Generation:
https://huggingface.co/Zhaoming213/ChatBox2-unrestricted

Third Generation:
https://huggingface.co/Zhaoming213/ChatBox3-unrestricted(This model is currently experimental.)

How to Use

Download all model files and place them in the model folder; then, run Predit.py to start using the model.

Disclaimer

The consequences of using this model shall be borne by the user.