CausalLM
/

miniG

+# miniG
+A model trained on a synthesis dataset of over 120 million entries, this dataset having been generated through the application of state-of-the-art language models utilizing large context windows, alongside methodologies akin to retrieval-augmented generation and knowledge graph integration, where the data synthesis is conducted within clusters derived from a curated pretraining corpus of 20 billion tokens, with subsequent validation performed by the model itself.
+Despite the absence of thorough alignment with human preferences, the model is under no obligation to cater to poorly constructed prompts or the clichés often found in conventional benchmarks. Bonus: Included is an implementation of a Vision Language Model that has undergone Locked-Image Tuning.
+Model Parameters: LLM - 9B (initialized from THUDM/glm-4-9b-chat-1m); Optional ViT - 5B
+Cautionary Notes: It is strongly recommended to utilize a standardized implementation for inference, such as Hugging Face Transformers, to avoid the significant performance degradation that might occur when using accelerated kernels like vllm or lmdeploy - not to mention the potentially catastrophic effects of model quantization. As of now, these accelerated inference implementations are known to severely compromise effective vision inference, though they have a less pronounced impact on pure text performance.
+Inference Parameters: Our observations suggest that, if one desires to achieve results with fewer hallucinations, it is advisable to employ sampling with top_p=0.8 followed by a temperature setting of 0.3, or alternatively, to use pure temperature sampling with a setting of 0.2. In general, a lower temperature is required compared to similar models, which we tentatively attribute to overfitting on the vast dataset.
+Disclaimer: Please note that the model was trained on unfiltered internet data. Since we do not have the capacity to vet all of it, there may be a substantial amount of objectionable content, pornography, violence, and offensive language present that we are unable to remove. Therefore, you will still need to complete your own checks on the model's safety and filter keywords in the output. Due to computational resource constraints, we are presently unable to implement RLHF for the model's ethics and safety, nor training on SFT samples that refuse to answer certain questions for restrictive fine-tuning.
+# 迷你G
+一个在超过1.2亿条数据合成数据集上训练的模型，这些数据集是通过应用具有大上下文窗口的最先进语言模型生成的，并结合了类似于检索增强生成和知识图谱集成的方法，数据合成是在一个由200亿个标记组成的预训练语料库中提取的聚类内进行的，随后由模型本身进行验证。
+尽管该模型没有完全对齐人类偏好，但它没有义务迎合不良构建的提示或常见基准测试中的陈词滥调。额外内容：包含了经过锁定图像微调的视觉语言模型实现。
+模型参数：LLM - 9B（从THUDM/glm-4-9b-chat-1m初始化）；可选的ViT - 5B。
+注意事项：强烈建议使用标准化的推理实现，例如Hugging Face Transformers，以避免在使用加速内核（如vllm或lmdeploy）时可能发生的显著性能下降——更不用说模型量化可能带来的灾难性影响。目前，这些加速推理实现已知会严重损害视觉推理的有效性，尽管对纯文本性能的影响较小。
+推理参数：我们的观察表明，如果想要减少幻觉结果，建议使用top_p=0.8的采样方式，然后设置temperature为0.3，或者使用纯粹的temperature采样，设置为0.2。总体来说，相比类似的模型，该模型需要较低的temperature，我们暂时将其归因于在庞大数据集上的过拟合。
+免责声明：请注意，该模型是在未经过滤的互联网数据上训练的。由于我们无法对所有数据进行筛选，仍有可能存在大量不适当的内容——包括从露骨的材料到暴力和攻击性语言的内容——我们无法移除。因此，您必须自行对模型进行安全检查，并在输出中实施关键词过滤。由于计算资源的限制，我们目前无法为伦理和安全考虑进行人类反馈的强化学习（RLHF），也不能对SFT样本进行限制性微调，以限制模型回答某些问题的能力。