init commit

README.md

@@ -1,3 +1,114 @@
 ---
 license: apache-2.0
+language: zh
+tags:
+- Text2Text Generation
+- T5
+- chinese
+- sentencepiece
+inference: true
+widget:
+- text: "新闻分类任务：【微软披露拓扑量子计算机计划！】这篇文章的类别是什么？故事/文化/娱乐/体育/财经/房产/汽车/教育/科技"
+- type: "text-generation"
 ---
+
+# Randeng-T5-784M-MultiTask-Chinese
+
+- Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM)
+- Docs: [Fengshenbang-Docs](https://fengshenbang-doc.readthedocs.io/)
+
+## 简介 Brief Introduction
+
+在Randeng-T5-784M的基础上，收集了100个左右的中文数据集，进行Text2Text统一范式的有监督任务预训练。
+
+On the basis of Randeng-T5-784M, about 100 Chinese datasets were collected and pre-trained for the supervised task of Text2Text unified paradigm.
+
+## 模型分类 Model Taxonomy
+
+|  需求 Demand  | 任务 Task       | 系列 Series      | 模型 Model    | 参数 Parameter | 额外 Extra |
+|  :----:  | :----:  | :----:  | :----:  | :----:  | :----:  |
+| 通用 General | 自然语言转换 NLT | 燃灯 Randeng | MultiTask |      784M      |    多任务-中文 MultiTask-Chinese    |
+
+
+## 模型信息 Model Information
+
+参考论文：[Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](http://jmlr.org/papers/v21/20-074.html)
+
+基于[Randeng-T5-784M](https://huggingface.co/IDEA-CCNL/Randeng-T5-784M)，我们在收集的100+个中文领域的多任务数据集（从中采样了30w+个样本）上微调了它，得到了此多任务版本。这些多任务包括：情感分析，新闻分类，文本分类，意图识别，自然语言推理，多项选择，指代消解，抽取式阅读理解，实体识别，关键词抽取，生成式摘要。
+
+Based on [Randeng-T5-784M](https://huggingface.co/IDEA-CCNL/Randeng-T5-784M), we fine-tuned it on a collection of 100+ multitasking datasets in Chinese domains (from which 30w+ samples were sampled) to obtain this multitasking version. These multitasks include: sentiment analysis, news classification, text classification, intention recognition, natural language inference, multiple choice, denotational disambiguation, extractive reading comprehension, entity recognition, keyword extraction, and generative summarization.
+
+
+## 使用 Usage
+
+```python
+import torch
+from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration
+
+# load tokenizer and model 
+pretrained_model = "IDEA-CCNL/Randeng-T5-784M-MultiTask-Chinese"
+
+special_tokens = ["<extra_id_{}>".format(i) for i in range(100)]
+tokenizer = T5Tokenizer.from_pretrained(
+    args.pretrained_model,
+    do_lower_case=True,
+    max_length=512,
+    truncation=True,
+    additional_special_tokens=special_tokens,
+)
+config = T5Config.from_pretrained(args.pretrained_model)
+model = T5ForConditionalGeneration.from_pretrained(args.pretrained_model, config=config)
+model.resize_token_embeddings(len(tokenizer))
+model.eval()
+
+# tokenize
+text = "新闻分类任务：【微软披露拓扑量子计算机计划！】这篇文章的类别是什么？故事/文化/娱乐/体育/财经/房产/汽车/教育/科技"
+encode_dict = tokenizer(text, max_length=512, padding='max_length',truncation=True)
+
+inputs = {
+  "input_ids": torch.tensor(encode_dict['input_ids']).long(),
+  "attention_mask": torch.tensor(encode_dict['attention_mask']).long(),
+  }
+
+# generate answer
+logits = model.generate(
+  input_ids = inputs['input_ids'],
+  max_length=100, 
+  do_sample= True
+  # early_stopping=True,
+  )
+
+logits=logits[:,1:]
+predict_label = [tokenizer.decode(i,skip_special_tokens=True) for i in logits]
+
+# model Output: 科技
+```
+
+## 引用 Citation
+
+如果您在您的工作中使用了我们的模型，可以引用我们的[论文](https://arxiv.org/abs/2209.02970)：
+
+If you are using the resource for your work, please cite the our [paper](https://arxiv.org/abs/2209.02970):
+
+```text
+@article{fengshenbang,
+  author    = {Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen and Ruyi Gan and Jiaxing Zhang},
+  title     = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
+  journal   = {CoRR},
+  volume    = {abs/2209.02970},
+  year      = {2022}
+}
+```
+
+也可以引用我们的[网站](https://github.com/IDEA-CCNL/Fengshenbang-LM/):
+
+You can also cite our [website](https://github.com/IDEA-CCNL/Fengshenbang-LM/):
+
+```text
+@misc{Fengshenbang-LM,
+  title={Fengshenbang-LM},
+  author={IDEA-CCNL},
+  year={2021},
+  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
+}
+```

config.json

@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "/cognitive_comp/wuxiaojun/pretrained/pytorch/Randeng-T5-784M",
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "d_ff": 2816,
+  "d_kv": 64,
+  "d_model": 1024,
+  "decoder_start_token_id": 0,
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "layer_norm_epsilon": 1e-06,
+  "max_length": 200,
+  "model_type": "t5",
+  "num_decoder_layers": 24,
+  "num_heads": 16,
+  "num_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "T5Tokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.18.0",
+  "use_cache": true,
+  "vocab_size": 32596
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f7231e9482940b12b4e067030c9f25b0aee562a88ab9d1683fc1612febcc30e
+size 3136623589

special_tokens_map.json

@@ -0,0 +1 @@
+{"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c65feffa65ff0378759778193852083d23349cb1b40c906e9463a12f8076ff32
+size 680811

tokenizer_config.json

@@ -0,0 +1 @@
+{"eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>", "extra_ids": 0, "additional_special_tokens": [], "sp_model_kwargs": {}, "name_or_path": "/cognitive_comp/wuxiaojun/pretrained/pytorch/Randeng-T5-784M", "do_lower_case": true, "max_length": 1024, "truncation": true, "special_tokens_map_file": "/cognitive_comp/wuxiaojun/pretrained/pytorch/Randeng-T5-784M/special_tokens_map.json", "tokenizer_class": "T5Tokenizer"}