Orionnnn/hub-stats huggingface dataset:
TeleChat Technical Report
Authors: name: Zihan Wang, name: Xinzhang Liu, name: Shixuan Liu, name: Yitong Yao, name: Yuyao Huang, name: Zhongjiang He, name: Xuelong Li, name: Yongxiang Li, name: Zhonghao Che, name: Zhaoxi Zhang, name: Yan Wang, name: Xin Wang, name: Luwen Pu, name: Huihan Xu, name: Ruiyu Fang, name: Yu Zhao, name: Jie Zhang, name: Xiaomeng Huang, name: Zhilong Lu, name: Jiaxin Peng, name: Wenjun Zheng, name: Shiquan Wang, name: Bingkai Yang, name: Xuewei he, name: Zhuoru Jiang, name: Qiyi Xie, name: Yanhan Zhang, name: Zhongqiu Li, name: Lingling Shi, name: Weiwei Fu, name: Yin Zhang, name: Zilu Huang, name: Sishi Xiong, name: Yuxiang Zhang, name: Chao Wang, name: Shuangyong Song
Published: 2024-01-08 10:43:19

In this technical report, we present TeleChat, a collection of large language
models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It
includes pretrained language models as well as fine-tuned chat models that is
aligned with human preferences. TeleChat is initially pretrained on an
extensive corpus containing a diverse collection of texts from both English and
Chinese languages, including trillions of tokens. Subsequently, the model
undergoes fine-tuning to align with human preferences, following a detailed
methodology that we describe. We evaluate the performance of TeleChat on
various tasks, including language understanding, mathematics, reasoning, code
generation, and knowledge-based question answering. Our findings indicate that
TeleChat achieves comparable performance to other open-source models of similar
size across a wide range of public benchmarks. To support future research and
applications utilizing LLMs, we release the fine-tuned model checkpoints of
TeleChat's 7B and 12B variant, along with code and a portion of our pretraining
data, to the public community.
Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon
Authors: name: Peitian Zhang, name: Zheng Liu, name: Shitao Xiao, name: Ninglu Shao, name: Qiwei Ye, name: Zhicheng Dou
Published: 2024-01-07 11:57:40

The utilization of long contexts poses a big challenge for large language
models due to their limited context window length. Although the context window
can be extended through fine-tuning, it will result in a considerable cost at
both training and inference time, and exert an unfavorable impact to the LLM's
original capabilities. In this work, we propose Activation Beacon, which
condenses LLM's raw activations into more compact forms such that it can
perceive a much longer context with a limited context window. Activation Beacon
is introduced as a plug-and-play module for the LLM. It fully preserves the
LLM's original capability on short contexts while extending the new capability
on processing longer contexts. Besides, it works with short sliding windows to
process the long context, which achieves a competitive memory and time
efficiency in both training and inference. Activation Beacon is learned by the
auto-regression task conditioned on a mixture of beacons with diversified
condensing ratios. Thanks to such a treatment, it can be efficiently trained
purely with short-sequence data in just 10K steps, which consumes less than 9
hours on a single 8xA800 GPU machine. The experimental studies show that
Activation Beacon is able to extend Llama-2-7B's context length by times100
times (from 4K to 400K), meanwhile achieving a superior result on both
long-context generation and understanding tasks. Our model and code will be
available at the BGE repository.
AST-T5: Structure-Aware Pretraining for Code Generation and
  Understanding
Authors: name: Linyuan Gong, name: Mostafa Elhoushi, name: Alvin Cheung
Published: 2024-01-05 06:51:08

Large language models (LLMs) have made significant advancements in
code-related tasks, yet many LLMs treat code as simple sequences, neglecting
its structured nature. We introduce AST-T5, a novel pretraining paradigm that
leverages the Abstract Syntax Tree (AST) for enhanced code generation,
transpilation, and understanding. Using dynamic programming, our AST-Aware
Segmentation retains code structure, while our AST-Aware Span Corruption
objective equips the model to reconstruct various code structures. Unlike other
models, AST-T5 avoids intricate program analyses or architectural changes, so
it integrates seamlessly with any encoder-decoder Transformer. Evaluations show
that AST-T5 consistently outperforms similar-sized LMs across various
code-related tasks. Structure-awareness makes AST-T5 particularly powerful in
code-to-code tasks, surpassing CodeT5 by 2 points in exact match score for the
Bugs2Fix task and by 3 points in exact match score for Java-C# Transpilation in
CodeXGLUE. Our code and model are publicly available at
https://github.com/gonglinyuan/ast_t5.
Has Your Pretrained Model Improved? A Multi-head Posterior Based
  Approach
Authors: name: Prince Aboagye, name: Yan Zheng, name: Junpeng Wang, name: Uday Singh Saini, name: Xin Dai, name: Michael Yeh, name: Yujie Fan, name: Zhongfang Zhuang, name: Shubham Jain, name: Liang Wang, name: Wei Zhang
Published: 2024-01-02 17:08:26

The emergence of pretrained models has significantly impacted from Natural
Language Processing (NLP) and Computer Vision to relational datasets.
Traditionally, these models are assessed through fine-tuned downstream tasks.
However, this raises the question of how to evaluate these models more
efficiently and more effectively. In this study, we explore a novel approach
where we leverage the meta features associated with each entity as a source of
worldly knowledge and employ entity representations from the models. We propose
using the consistency between these representations and the meta features as a
metric for evaluating pretrained models. Our method's effectiveness is
demonstrated across various domains, including models with relational datasets,
large language models and images models.
Blending Is All You Need: Cheaper, Better Alternative to
  Trillion-Parameters LLM
Authors: name: Xiaoding Lu, name: Adian Liusie, name: Vyas Raina, name: Yuwen Zhang, name: William Beauchamp
Published: 2024-01-04 07:45:49

In conversational AI research, there's a noticeable trend towards developing
models with a larger number of parameters, exemplified by models like ChatGPT.
While these expansive models tend to generate increasingly better chat
responses, they demand significant computational resources and memory. This
study explores a pertinent question: Can a combination of smaller models
collaboratively achieve comparable or enhanced performance relative to a
singular large model? We introduce an approach termed "blending", a
straightforward yet effective method of integrating multiple chat AIs. Our
empirical evidence suggests that when specific smaller models are
synergistically blended, they can potentially outperform or match the
capabilities of much larger counterparts. For instance, integrating just three
models of moderate size (6B/13B paramaeters) can rival or even surpass the
performance metrics of a substantially larger model like ChatGPT (175B+
paramaters). This hypothesis is rigorously tested using A/B testing
methodologies with a large user base on the Chai research platform over a span
of thirty days. The findings underscore the potential of the "blending"
strategy as a viable approach for enhancing chat AI efficacy without a
corresponding surge in computational demands.
GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
Authors: name: Tong Wu, name: Guandao Yang, name: Zhibing Li, name: Kai Zhang, name: Ziwei Liu, name: Leonidas Guibas, name: Dahua Lin, name: Gordon Wetzstein
Published: 2024-01-08 18:52:09

Despite recent advances in text-to-3D generative methods, there is a notable
absence of reliable evaluation metrics. Existing metrics usually focus on a
single criterion each, such as how well the asset aligned with the input text.
These metrics lack the flexibility to generalize to different evaluation
criteria and might not align well with human preferences. Conducting user
preference studies is an alternative that offers both adaptability and
human-aligned results. User studies, however, can be very expensive to scale.
This paper presents an automatic, versatile, and human-aligned evaluation
metric for text-to-3D generative models. To this end, we first develop a prompt
generator using GPT-4V to generate evaluating prompts, which serve as input to
compare text-to-3D models. We further design a method instructing GPT-4V to
compare two 3D assets according to user-defined criteria. Finally, we use these
pairwise comparison results to assign these models Elo ratings. Experimental
results suggest our metric strongly align with human preference across
different evaluation criteria.
CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
Authors: name: Alex Gu, name: Baptiste Rozière, name: Hugh Leather, name: Armando Solar-Lezama, name: Gabriel Synnaeve, name: Sida I. Wang
Published: 2024-01-05 20:53:51

We present CRUXEval (Code Reasoning, Understanding, and eXecution
Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
function comes with an input-output pair, leading to two natural tasks: input
prediction and output prediction. First, we propose a generic recipe for
generating our execution benchmark which can be used to create future variation
of the benchmark. Second, we evaluate twenty code models on our benchmark and
discover that many recent high-scoring models on HumanEval do not show the same
improvements on our benchmark. Third, we show that simple CoT and fine-tuning
schemes can improve performance on our benchmark but remain far from solving
it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75%
and 81% on input and output prediction, respectively. In contrast, Code Llama
34B achieves a pass@1 of 50% and 46% on input and output prediction,
highlighting the gap between open and closed source models. As no model is
close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
simple programs as a lens into its code reasoning capabilities and areas for
improvement.
Mixtral of Experts
Authors: name: Albert Q. Jiang, name: Alexandre Sablayrolles, name: Antoine Roux, name: Arthur Mensch, name: Blanche Savary, name: Chris Bamford, name: Devendra Singh Chaplot, name: Diego de las Casas, name: Emma Bou Hanna, name: Florian Bressand, name: Gianna Lengyel, name: Guillaume Bour, name: Guillaume Lample, name: Lélio Renard Lavaud, name: Lucile Saulnier, name: Marie-Anne Lachaux, name: Pierre Stock, name: Sandeep Subramanian, name: Sophia Yang, name: Szymon Antoniak, name: Teven Le Scao, name: Théophile Gervet, name: Thibaut Lavril, name: Thomas Wang, name: Timothée Lacroix, name: William El Sayed
Published: 2024-01-08 18:47:34

We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.
Mixtral has the same architecture as Mistral 7B, with the difference that each
layer is composed of 8 feedforward blocks (i.e. experts). For every token, at
each layer, a router network selects two experts to process the current state
and combine their outputs. Even though each token only sees two experts, the
selected experts can be different at each timestep. As a result, each token has
access to 47B parameters, but only uses 13B active parameters during inference.
Mixtral was trained with a context size of 32k tokens and it outperforms or
matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and
multilingual benchmarks. We also provide a model fine-tuned to follow
instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,
Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both
the base and instruct models are released under the Apache 2.0 license.
DocGraphLM: Documental Graph Language Model for Information Extraction
Authors: name: Dongsheng Wang, name: Zhiqiang Ma, name: Armineh Nourbakhsh, name: Kang Gu, name: Sameena Shah
Published: 2024-01-05 14:15:36

Advances in Visually Rich Document Understanding (VrDU) have enabled
information extraction and question answering over documents with complex
layouts. Two tropes of architectures have emerged -- transformer-based models
inspired by LLMs, and Graph Neural Networks. In this paper, we introduce
DocGraphLM, a novel framework that combines pre-trained language models with
graph semantics. To achieve this, we propose 1) a joint encoder architecture to
represent documents, and 2) a novel link prediction approach to reconstruct
document graphs. DocGraphLM predicts both directions and distances between
nodes using a convergent joint loss function that prioritizes neighborhood
restoration and downweighs distant node detection. Our experiments on three
SotA datasets show consistent improvement on IE and QA tasks with the adoption
of graph features. Moreover, we report that adopting the graph features
accelerates convergence in the learning process during training, despite being
solely constructed through link prediction.
Denoising Vision Transformers
Authors: name: Jiawei Yang, name: Katie Z Luo, name: Jiefeng Li, name: Kilian Q Weinberger, name: Yonglong Tian, name: Yue Wang
Published: 2024-01-05 18:59:52

We delve into a nuanced but significant challenge inherent to Vision
Transformers (ViTs): feature maps of these models exhibit grid-like artifacts,
which detrimentally hurt the performance of ViTs in downstream tasks. Our
investigations trace this fundamental issue down to the positional embeddings
at the input stage. To address this, we propose a novel noise model, which is
universally applicable to all ViTs. Specifically, the noise model dissects ViT
outputs into three components: a semantics term free from noise artifacts and
two artifact-related terms that are conditioned on pixel locations. Such a
decomposition is achieved by enforcing cross-view feature consistency with
neural fields in a per-image basis. This per-image optimization process
extracts artifact-free features from raw ViT outputs, providing clean features
for offline applications. Expanding the scope of our solution to support online
functionality, we introduce a learnable denoiser to predict artifact-free
features directly from unprocessed ViT outputs, which shows remarkable
generalization capabilities to novel data without the need for per-image
optimization. Our two-stage approach, termed Denoising Vision Transformers
(DVT), does not require re-training existing pre-trained ViTs and is
immediately applicable to any Transformer-based architecture. We evaluate our
method on a variety of representative ViTs (DINO, MAE, DeiT-III, EVA02, CLIP,
DINOv2, DINOv2-reg). Extensive evaluations demonstrate that our DVT
consistently and significantly improves existing state-of-the-art
general-purpose models in semantic and geometric tasks across multiple datasets
(e.g., +3.84 mIoU). We hope our study will encourage a re-evaluation of ViT
design, especially regarding the naive use of positional embeddings.
Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer
  Level Loss
Authors: name: Yatharth Gupta, name: Vishnu V. Jaddipal, name: Harish Prabhala, name: Sayak Paul, name: Patrick Von Platen
Published: 2024-01-05 07:21:46

Stable Diffusion XL (SDXL) has become the best open source text-to-image
model (T2I) for its versatility and top-notch image quality. Efficiently
addressing the computational demands of SDXL models is crucial for wider reach
and applicability. In this work, we introduce two scaled-down variants, Segmind
Stable Diffusion (SSD-1B) and Segmind-Vega, with 1.3B and 0.74B parameter
UNets, respectively, achieved through progressive removal using layer-level
losses focusing on reducing the model size while preserving generative quality.
We release these models weights at https://hf.co/Segmind. Our methodology
involves the elimination of residual networks and transformer blocks from the
U-Net structure of SDXL, resulting in significant reductions in parameters, and
latency. Our compact models effectively emulate the original SDXL by
capitalizing on transferred knowledge, achieving competitive results against
larger multi-billion parameter SDXL. Our work underscores the efficacy of
knowledge distillation coupled with layer-level losses in reducing model size
while preserving the high-quality generative capabilities of SDXL, thus
facilitating more accessible deployment in resource-constrained environments.
Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes
  Interactively
Authors: name: Haobo Yuan, name: Xiangtai Li, name: Chong Zhou, name: Yining Li, name: Kai Chen, name: Chen Change Loy
Published: 2024-01-05 18:59:22

The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
models (VFMs). SAM excels in segmentation tasks across diverse domains, while
CLIP is renowned for its zero-shot recognition capabilities. This paper
presents an in-depth exploration of integrating these two models into a unified
framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
model designed for simultaneous interactive segmentation and recognition,
leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
former adapts SAM's knowledge into the CLIP via distillation and learnable
transformer adapters, while the latter transfers CLIP knowledge into SAM,
enhancing its recognition capabilities. Extensive experiments on various
datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
segmentation and recognition tasks, significantly outperforming the naive
baselines of simply combining SAM and CLIP. Furthermore, aided with image
classification data training, our method can segment and recognize
approximately 22,000 classes.
Pheme: Efficient and Conversational Speech Generation
Authors: name: Paweł Budzianowski, name: Taras Sereda, name: Tomasz Cichy, name: Ivan Vulić
Published: 2024-01-05 14:47:20

In recent years, speech generation has seen remarkable progress, now
achieving one-shot generation capability that is often virtually
indistinguishable from real human voice. Integrating such advancements in
speech generation with large language models might revolutionize a wide range
of applications. However, certain applications, such as assistive
conversational systems, require natural and conversational speech generation
tools that also operate efficiently in real time. Current state-of-the-art
models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs,
require large neural components and extensive training data to work well. In
contrast, MQTTS aims to build more compact conversational TTS models while
capitalizing on smaller-scale real-life conversational speech data. However,
its autoregressive nature yields high inference latency and thus limits its
real-time usage. In order to mitigate the current limitations of the
state-of-the-art TTS models while capitalizing on their strengths, in this work
we introduce the Pheme model series that 1) offers compact yet high-performing
models, 2) allows for parallel speech generation of 3) natural conversational
speech, and 4) it can be trained efficiently on smaller-scale conversational
data, cutting data demands by more than 10x but still matching the quality of
the autoregressive TTS models. We also show that through simple teacher-student
distillation we can meet significant improvements in voice quality for
single-speaker setups on top of pretrained Pheme checkpoints, relying solely on
synthetic speech generated by much larger teacher models. Audio samples and
pretrained models are available online.
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
Authors: name: DeepSeek-AI, name: Xiao Bi, name: Deli Chen, name: Guanting Chen, name: Shanhuang Chen, name: Damai Dai, name: Chengqi Deng, name: Honghui Ding, name: Kai Dong, name: Qiushi Du, name: Zhe Fu, name: Huazuo Gao, name: Kaige Gao, name: Wenjun Gao, name: Ruiqi Ge, name: Kang Guan, name: Daya Guo, name: Jianzhong Guo, name: Guangbo Hao, name: Zhewen Hao, name: Ying He, name: Wenjie Hu, name: Panpan Huang, name: Erhang Li, name: Guowei Li, name: Jiashi Li, name: Yao Li, name: Y. K. Li, name: Wenfeng Liang, name: Fangyun Lin, name: A. X. Liu, name: Bo Liu, name: Wen Liu, name: Xiaodong Liu, name: Xin Liu, name: Yiyuan Liu, name: Haoyu Lu, name: Shanghao Lu, name: Fuli Luo, name: Shirong Ma, name: Xiaotao Nie, name: Tian Pei, name: Yishi Piao, name: Junjie Qiu, name: Hui Qu, name: Tongzheng Ren, name: Zehui Ren, name: Chong Ruan, name: Zhangli Sha, name: Zhihong Shao, name: Junxiao Song, name: Xuecheng Su, name: Jingxiang Sun, name: Yaofeng Sun, name: Minghui Tang, name: Bingxuan Wang, name: Peiyi Wang, name: Shiyu Wang, name: Yaohui Wang, name: Yongji Wang, name: Tong Wu, name: Y. Wu, name: Xin Xie, name: Zhenda Xie, name: Ziwei Xie, name: Yiliang Xiong, name: Hanwei Xu, name: R. X. Xu, name: Yanhong Xu, name: Dejian Yang, name: Yuxiang You, name: Shuiping Yu, name: Xingkai Yu, name: B. Zhang, name: Haowei Zhang, name: Lecong Zhang, name: Liyue Zhang, name: Mingchuan Zhang, name: Minghua Zhang, name: Wentao Zhang, name: Yichao Zhang, name: Chenggang Zhao, name: Yao Zhao, name: Shangyan Zhou, name: Shunfeng Zhou, name: Qihao Zhu, name: Yuheng Zou
Published: 2024-01-05 18:59:13

The rapid development of open-source large language models (LLMs) has been
truly remarkable. However, the scaling law described in previous literature
presents varying conclusions, which casts a dark cloud over scaling LLMs. We
delve into the study of scaling laws and present our distinctive findings that
facilitate scaling of large scale models in two commonly used open-source
configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
LLM, a project dedicated to advancing open-source language models with a
long-term perspective. To support the pre-training phase, we have developed a
dataset that currently consists of 2 trillion tokens and is continuously
expanding. We further conduct supervised fine-tuning (SFT) and Direct
Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
creation of DeepSeek Chat models. Our evaluation results demonstrate that
DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
the domains of code, mathematics, and reasoning. Furthermore, open-ended
evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
compared to GPT-3.5.
Infinite-LLM: Efficient LLM Service for Long Context with DistAttention
  and Distributed KVCache
Authors: name: Bin Lin, name: Tao Peng, name: Chen Zhang, name: Minmin Sun, name: Lanbo Li, name: Hanyu Zhao, name: Wencong Xiao, name: Qi Xu, name: Xiafei Qiu, name: Shen Li, name: Zhigang Ji, name: Yong Li, name: Wei Lin
Published: 2024-01-05 06:53:00

The rapid proliferation of Large Language Models (LLMs) has been a driving
force in the growth of cloud-based LLM services, which are now integral to
advancing AI applications. However, the dynamic auto-regressive nature of LLM
service, along with the need to support exceptionally long context lengths,
demands the flexible allocation and release of substantial resources. This
presents considerable challenges in designing cloud-based LLM service systems,
where inefficient management can lead to performance degradation or resource
wastage. In response to these challenges, this work introduces DistAttention, a
novel distributed attention algorithm that segments the KV Cache into smaller,
manageable units, enabling distributed processing and storage of the attention
module. Based on that, we propose DistKV-LLM, a distributed LLM serving system
that dynamically manages KV Cache and effectively orchestrates all accessible
GPU and CPU memories spanning across the data center. This ensures a
high-performance LLM service on the cloud, adaptable to a broad range of
context lengths. Validated in a cloud environment with 32 NVIDIA A100 GPUs in
configurations from 2 to 32 instances, our system exhibited 1.03-2.4x
end-to-end throughput improvements and supported context lengths 2-19x longer
than current state-of-the-art LLM service systems, as evidenced by extensive
testing across 18 datasets with context lengths up to 1,900K.