Orionnnn/hub-stats huggingface dataset:
Alchemy: Amplifying Theorem-Proving Capability through Symbolic Mutation
Authors: name: Shaonan Wu, name: Shuai Lu, name: Yeyun Gong, name: Nan Duan, name: Ping Wei
Published: 2024-10-21 08:04:21

Formal proofs are challenging to write even for experienced experts. Recent
progress in Neural Theorem Proving (NTP) shows promise in expediting this
process. However, the formal corpora available on the Internet are limited
compared to the general text, posing a significant data scarcity challenge for
NTP. To address this issue, this work proposes Alchemy, a general framework for
data synthesis that constructs formal theorems through symbolic mutation.
Specifically, for each candidate theorem in Mathlib, we identify all invocable
theorems that can be used to rewrite or apply to it. Subsequently, we mutate
the candidate theorem by replacing the corresponding term in the statement with
its equivalent form or antecedent. As a result, our method increases the number
of theorems in Mathlib by an order of magnitude, from 110k to 6M. Furthermore,
we perform continual pretraining and supervised finetuning on this augmented
corpus for large language models. Experimental results demonstrate the
effectiveness of our approach, achieving a 5% absolute performance improvement
on Leandojo benchmark. Additionally, our synthetic data achieve a 2.5% absolute
performance gain on the out-of-distribution miniF2F benchmark. To provide
further insights, we conduct a comprehensive analysis of synthetic data
composition and the training paradigm, offering valuable guidance for
developing a strong theorem prover.
Selecting Influential Samples for Long Context Alignment via Homologous
  Models' Guidance and Contextual Awareness Measurement
Authors: name: Shuzheng Si, name: Haozhe Zhao, name: Gang Chen, name: Yunshui Li, name: Kangyang Luo, name: Chuancheng Lv, name: Kaikai An, name: Fanchao Qi, name: Baobao Chang, name: Maosong Sun
Published: 2024-10-21 04:30:53

The expansion of large language models to effectively handle instructions
with extremely long contexts has yet to be fully investigated. The primary
obstacle lies in constructing a high-quality long instruction-following dataset
devised for long context alignment. Existing studies have attempted to scale up
the available data volume by synthesizing long instruction-following samples.
However, indiscriminately increasing the quantity of data without a
well-defined strategy for ensuring data quality may introduce low-quality
samples and restrict the final performance. To bridge this gap, we aim to
address the unique challenge of long-context alignment, i.e., modeling the
long-range dependencies for handling instructions and lengthy input contexts.
We propose GATEAU, a novel framework designed to identify the influential and
high-quality samples enriched with long-range dependency relations by utilizing
crafted Homologous Models' Guidance (HMG) and Contextual Awareness Measurement
(CAM). Specifically, HMG attempts to measure the difficulty of generating
corresponding responses due to the long-range dependencies, using the
perplexity scores of the response from two homologous models with different
context windows. Also, the role of CAM is to measure the difficulty of
understanding the long input contexts due to long-range dependencies by
evaluating whether the model's attention is focused on important segments.
Built upon both proposed methods, we select the most challenging samples as the
influential data to effectively frame the long-range dependencies, thereby
achieving better performance of LLMs. Comprehensive experiments indicate that
GATEAU effectively identifies samples enriched with long-range dependency
relations and the model trained on these selected samples exhibits better
instruction-following and long-context understanding capabilities.
AutoTrain: No-code training for state-of-the-art models
Authors: name: Abhishek Thakur
Published: 2024-10-21 07:53:32

With the advancements in open-source models, training (or finetuning) models
on custom datasets has become a crucial part of developing solutions which are
tailored to specific industrial or open-source applications. Yet, there is no
single tool which simplifies the process of training across different types of
modalities or tasks. We introduce AutoTrain (aka AutoTrain Advanced) -- an
open-source, no code tool/library which can be used to train (or finetune)
models for different kinds of tasks such as: large language model (LLM)
finetuning, text classification/regression, token classification,
sequence-to-sequence task, finetuning of sentence transformers, visual language
model (VLM) finetuning, image classification/regression and even classification
and regression tasks on tabular data. AutoTrain Advanced is an open-source
library providing best practices for training models on custom datasets. The
library is available at https://github.com/huggingface/autotrain-advanced.
AutoTrain can be used in fully local mode or on cloud machines and works with
tens of thousands of models shared on Hugging Face Hub and their variations.
Pre-training Distillation for Large Language Models: A Design Space
  Exploration
Authors: name: Hao Peng, name: Xin Lv, name: Yushi Bai, name: Zijun Yao, name: Jiajie Zhang, name: Lei Hou, name: Juanzi Li
Published: 2024-10-21 17:16:13

Knowledge distillation (KD) aims to transfer knowledge from a large teacher
model to a smaller student model. Previous work applying KD in the field of
large language models (LLMs) typically focused on the post-training phase,
where the student LLM learns directly from instructions and corresponding
responses generated by the teacher model. In this paper, we extend KD to the
pre-training phase of LLMs, named pre-training distillation (PD). We first
conduct a preliminary experiment using GLM-4-9B as the teacher LLM to distill a
1.9B parameter student LLM, validating the effectiveness of PD. Considering the
key impact factors of distillation, we systematically explore the design space
of pre-training distillation across four aspects: logits processing, loss
selection, scaling law, and offline or online logits. We conduct extensive
experiments to explore the design space of pre-training distillation and find
better configurations and interesting conclusions, such as larger student LLMs
generally benefiting more from pre-training distillation, while a larger
teacher LLM does not necessarily guarantee better results. We hope our
exploration of the design space will inform future practices in pre-training
distillation.
SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
  Training-Free Memory Tree
Authors: name: Shuangrui Ding, name: Rui Qian, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Yuwei Guo, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-10-21 17:59:19

The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
model for object segmentation in both images and videos, paving the way for
various downstream video applications. The crucial design of SAM 2 for video
segmentation is its memory module, which prompts object-aware memories from
previous frames for current frame prediction. However, its greedy-selection
memory design suffers from the "error accumulation" problem, where an errored
or missed mask will cascade and influence the segmentation of the subsequent
frames, which limits the performance of SAM 2 toward complex long-term videos.
To this end, we introduce SAM2Long, an improved training-free video object
segmentation strategy, which considers the segmentation uncertainty within each
frame and chooses the video-level optimal results from multiple segmentation
pathways in a constrained tree search manner. In practice, we maintain a fixed
number of segmentation pathways throughout the video. For each frame, multiple
masks are proposed based on the existing pathways, creating various candidate
branches. We then select the same fixed number of branches with higher
cumulative scores as the new pathways for the next frame. After processing the
final frame, the pathway with the highest cumulative score is chosen as the
final segmentation result. Benefiting from its heuristic search design,
SAM2Long is robust toward occlusions and object reappearances, and can
effectively segment and track objects for complex long-term videos. Notably,
SAM2Long achieves an average improvement of 3.0 points across all 24
head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
video object segmentation benchmarks such as SA-V and LVOS. The code is
released at https://github.com/Mark12Ding/SAM2Long.
RM-Bench: Benchmarking Reward Models of Language Models with Subtlety
  and Style
Authors: name: Yantao Liu, name: Zijun Yao, name: Rui Min, name: Yixin Cao, name: Lei Hou, name: Juanzi Li
Published: 2024-10-21 16:48:26

Reward models are critical in techniques like Reinforcement Learning from
Human Feedback (RLHF) and Inference Scaling Laws, where they guide language
model alignment and select optimal responses. Despite their importance,
existing reward model benchmarks often evaluate models by asking them to
distinguish between responses generated by models of varying power. However,
this approach fails to assess reward models on subtle but critical content
changes and variations in style, resulting in a low correlation with policy
model performance. To this end, we introduce RM-Bench, a novel benchmark
designed to evaluate reward models based on their sensitivity to subtle content
differences and resistance to style biases. Extensive experiments demonstrate
that RM-Bench strongly correlates with policy model performance, making it a
reliable reference for selecting reward models to align language models
effectively. We evaluate nearly 40 reward models on RM-Bench. Our results
reveal that even state-of-the-art models achieve an average performance of only
46.6%, which falls short of random-level accuracy (50%) when faced with style
bias interference. These findings highlight the significant room for
improvement in current reward models. Related code and data are available at
https://github.com/THU-KEG/RM-Bench.
FrugalNeRF: Fast Convergence for Few-shot Novel View Synthesis without
  Learned Priors
Authors: name: Chin-Yang Lin, name: Chung-Ho Wu, name: Chang-Han Yeh, name: Shih-Han Yen, name: Cheng Sun, name: Yu-Lun Liu
Published: 2024-10-21 17:59:53

Neural Radiance Fields (NeRF) face significant challenges in few-shot
scenarios, primarily due to overfitting and long training times for
high-fidelity rendering. Existing methods, such as FreeNeRF and SparseNeRF, use
frequency regularization or pre-trained priors but struggle with complex
scheduling and bias. We introduce FrugalNeRF, a novel few-shot NeRF framework
that leverages weight-sharing voxels across multiple scales to efficiently
represent scene details. Our key contribution is a cross-scale geometric
adaptation scheme that selects pseudo ground truth depth based on reprojection
errors across scales. This guides training without relying on externally
learned priors, enabling full utilization of the training data. It can also
integrate pre-trained priors, enhancing quality without slowing convergence.
Experiments on LLFF, DTU, and RealEstate-10K show that FrugalNeRF outperforms
other few-shot NeRF methods while significantly reducing training time, making
it a practical solution for efficient and accurate 3D scene reconstruction.
Meta-Chunking: Learning Efficient Text Segmentation via Logical
  Perception
Authors: name: Jihao Zhao, name: Zhiyuan Ji, name: Pengnian Qi, name: Simin Niu, name: Bo Tang, name: Feiyu Xiong, name: Zhiyu Li
Published: 2024-10-16 17:59:32

Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline, which impacts the quality of knowledge-intensive
tasks. This paper introduces the concept of Meta-Chunking, which refers to a
granularity between sentences and paragraphs, consisting of a collection of
sentences within a paragraph that have deep linguistic logical connections. To
implement Meta-Chunking, we designed two strategies based on LLMs: Margin
Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform
binary classification on whether consecutive sentences need to be segmented,
making decisions based on the probability difference obtained from margin
sampling. The latter precisely identifies text chunk boundaries by analyzing
the characteristics of perplexity distribution. Additionally, considering the
inherent complexity of different texts, we propose a strategy that combines
Meta-Chunking with dynamic merging to achieve a balance between fine-grained
and coarse-grained text chunking. Experiments conducted on eleven datasets
demonstrate that Meta-Chunking can more efficiently improve the performance of
single-hop and multi-hop question answering based on RAG. For instance, on the
2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only
consuming 45.8% of the time. Our code is available at
https://github.com/IAAR-Shanghai/Meta-Chunking.
Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
Authors: name: Xiang Yue, name: Yueqi Song, name: Akari Asai, name: Seungone Kim, name: Jean de Dieu Nyandwi, name: Simran Khanuja, name: Anjali Kantharuban, name: Lintang Sutawika, name: Sathyanarayanan Ramamoorthy, name: Graham Neubig
Published: 2024-10-21 16:19:41

Despite recent advances in multimodal large language models (MLLMs), their
development has predominantly focused on English- and western-centric datasets
and tasks, leaving most of the world's languages and diverse cultural contexts
underrepresented. This paper introduces Pangea, a multilingual multimodal LLM
trained on PangeaIns, a diverse 6M instruction dataset spanning 39 languages.
PangeaIns features: 1) high-quality English instructions, 2) carefully
machine-translated instructions, and 3) culturally relevant multimodal tasks to
ensure cross-cultural coverage. To rigorously assess models' capabilities, we
introduce PangeaBench, a holistic evaluation suite encompassing 14 datasets
covering 47 languages. Results show that Pangea significantly outperforms
existing open-source models in multilingual settings and diverse cultural
contexts. Ablation studies further reveal the importance of English data
proportions, language popularity, and the number of multimodal training samples
on overall performance. We fully open-source our data, code, and trained
checkpoints, to facilitate the development of inclusive and robust multilingual
MLLMs, promoting equity and accessibility across a broader linguistic and
cultural spectrum.
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
Authors: name: Sumanth Doddapaneni, name: Mohammed Safi Ur Rahman Khan, name: Dilip Venkatesh, name: Raj Dabre, name: Anoop Kunchukuttan, name: Mitesh M. Khapra
Published: 2024-10-17 09:45:32

Evaluating machine-generated text remains a significant challenge in NLP,
especially for non-English languages. Current methodologies, including
automated metrics, human assessments, and LLM-based evaluations, predominantly
focus on English, revealing a significant gap in multilingual evaluation
frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
extensible framework that includes evaluator LLMs (Hercule) and a novel test
set (Recon) specifically designed for multilingual evaluation. Our test set
features 500 human-annotated instructions spanning various task capabilities
along with human judgment scores across six languages. This would enable
benchmarking of general-purpose multilingual LLMs and facilitate
meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
cross-lingual evaluation model that addresses the scarcity of reference answers
in the target language by learning to assign scores to responses based on
easily available reference answers in English. Our experiments demonstrate that
Hercule aligns more closely with human judgments compared to proprietary
models, demonstrating the effectiveness of such cross-lingual evaluation in low
resource scenarios. Further, it is also effective in zero-shot evaluation on
unseen languages. This study is the first comprehensive examination of
cross-lingual evaluation using LLMs, presenting a scalable and effective
approach for multilingual assessment. All code, datasets, and models will be
publicly available to enable further research in this important area.
PUMA: Empowering Unified MLLM with Multi-granular Visual Generation
Authors: name: Rongyao Fang, name: Chengqi Duan, name: Kun Wang, name: Hao Li, name: Hao Tian, name: Xingyu Zeng, name: Rui Zhao, name: Jifeng Dai, name: Hongsheng Li, name: Xihui Liu
Published: 2024-10-17 17:59:57

Recent advancements in multimodal foundation models have yielded significant
progress in vision-language understanding. Initial attempts have also explored
the potential of multimodal large language models (MLLMs) for visual content
generation. However, existing works have insufficiently addressed the varying
granularity demands of different image generation tasks within a unified MLLM
paradigm - from the diversity required in text-to-image generation to the
precise controllability needed in image manipulation. In this work, we propose
PUMA, emPowering Unified MLLM with Multi-grAnular visual generation. PUMA
unifies multi-granular visual features as both inputs and outputs of MLLMs,
elegantly addressing the different granularity requirements of various image
generation tasks within a unified MLLM framework. Following multimodal
pretraining and task-specific instruction tuning, PUMA demonstrates proficiency
in a wide range of multimodal tasks. This work represents a significant step
towards a truly unified MLLM capable of adapting to the granularity demands of
various visual tasks. The code and model will be released in
https://github.com/rongyaofang/PUMA.
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and
  Evolution
Authors: name: Maosong Cao, name: Alexander Lam, name: Haodong Duan, name: Hongwei Liu, name: Songyang Zhang, name: Kai Chen
Published: 2024-10-21 17:56:51

Efficient and accurate evaluation is crucial for the continuous improvement
of large language models (LLMs). Among various assessment methods, subjective
evaluation has garnered significant attention due to its superior alignment
with real-world usage scenarios and human preferences. However, human-based
evaluations are costly and lack reproducibility, making precise automated
evaluators (judgers) vital in this process. In this report, we introduce
CompassJudger-1, the first open-source all-in-one judge LLM.
CompassJudger-1 is a general-purpose LLM that demonstrates remarkable
versatility. It is capable of: 1. Performing unitary scoring and two-model
comparisons as a reward model; 2. Conducting evaluations according to specified
formats; 3. Generating critiques; 4. Executing diverse tasks like a general
LLM. To assess the evaluation capabilities of different judge models under a
unified setting, we have also established JudgerBench, a new benchmark
that encompasses various subjective evaluation tasks and covers a wide range of
topics. CompassJudger-1 offers a comprehensive solution for various evaluation
tasks while maintaining the flexibility to adapt to diverse requirements. Both
CompassJudger and JudgerBench are released and available to the research
community athttps://github.com/open-compass/CompassJudger. We believe that by
open-sourcing these tools, we can foster collaboration and accelerate progress
in LLM evaluation methodologies.
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Authors: name: Md Mubtasim Ahasan, name: Md Fahim, name: Tasnim Mohiuddin, name: A K M Mahbubur Rahman, name: Aman Chadha, name: Tariq Iqbal, name: M Ashraful Amin, name: Md Mofijul Islam, name: Amin Ahsan Ali
Published: 2024-10-19 07:14:14

Recent advancements in speech-language models have yielded significant
improvements in speech tokenization and synthesis. However, effectively mapping
the complex, multidimensional attributes of speech into discrete tokens remains
challenging. This process demands acoustic, semantic, and contextual
information for precise speech representations. Existing speech representations
generally fall into two categories: acoustic tokens from audio codecs and
semantic tokens from speech self-supervised learning models. Although recent
efforts have unified acoustic and semantic tokens for improved performance,
they overlook the crucial role of contextual representation in comprehensive
speech modeling. Our empirical investigations reveal that the absence of
contextual representations results in elevated Word Error Rate (WER) and Word
Information Lost (WIL) scores in speech transcriptions. To address these
limitations, we propose two novel distillation approaches: (1) a language model
(LM)-guided distillation method that incorporates contextual information, and
(2) a combined LM and self-supervised speech model (SM)-guided distillation
technique that effectively distills multimodal representations (acoustic,
semantic, and contextual) into a comprehensive speech tokenizer, termed
DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder
framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM
during the training process. Experiments show DM-Codec significantly
outperforms state-of-the-art speech tokenization models, reducing WER by up to
13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility
by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model
checkpoints are available at https://github.com/mubtasimahasan/DM-Codec.
Baichuan Alignment Technical Report
Authors: name: Mingan Lin, name: Fan Yang, name: Yanjun Shen, name: Haoze Sun, name: Tianpeng Li, name: Tao Zhang, name: Chenzheng Zhu, name: Tao Zhang, name: Miao Zheng, name: Xu Li, name: Yijie Zhou, name: Mingyang Chen, name: Yanzhao Qin, name: Youquan Li, name: Hao Liang, name: Fei Li, name: Yadong Li, name: Mang Wang, name: Guosheng Dong, name: Kun Fang, name: Jianhua Xu, name: Bin Cui, name: Wentao Zhang, name: Zenan Zhou, name: Weipeng Chen
Published: 2024-10-19 02:07:33

We introduce Baichuan Alignment, a detailed analysis of the alignment
techniques employed in the Baichuan series of models. This represents the
industry's first comprehensive account of alignment methodologies, offering
valuable insights for advancing AI research. We investigate the critical
components that enhance model performance during the alignment process,
including optimization methods, data strategies, capability enhancements, and
evaluation processes. The process spans three key stages: Prompt Augmentation
System (PAS), Supervised Fine-Tuning (SFT), and Preference Alignment. The
problems encountered, the solutions applied, and the improvements made are
thoroughly recorded.
  Through comparisons across well-established benchmarks, we highlight the
technological advancements enabled by Baichuan Alignment. Baichuan-Instruct is
an internal model, while Qwen2-Nova-72B and Llama3-PBM-Nova-70B are instruct
versions of the Qwen2-72B and Llama-3-70B base models, optimized through
Baichuan Alignment. Baichuan-Instruct demonstrates significant improvements in
core capabilities, with user experience gains ranging from 17% to 28%, and
performs exceptionally well on specialized benchmarks. In open-source benchmark
evaluations, both Qwen2-Nova-72B and Llama3-PBM-Nova-70B consistently
outperform their respective official instruct versions across nearly all
datasets. This report aims to clarify the key technologies behind the alignment
process, fostering a deeper understanding within the community.
Llama3-PBM-Nova-70B model is available at
https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B.