Orionnnn/hub-stats huggingface dataset:
LLMtimesMapReduce: Simplified Long-Sequence Processing using Large
  Language Models
Authors: name: Zihan Zhou, name: Chong Li, name: Xinyi Chen, name: Shuo Wang, name: Yu Chao, name: Zhili Li, name: Haoyu Wang, name: Rongqiao An, name: Qi Shi, name: Zhixing Tan, name: Xu Han, name: Xiaodong Shi, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-10-12 03:13:44

Enlarging the context window of large language models (LLMs) has become a
crucial research area, particularly for applications involving extremely long
texts. In this work, we propose a novel training-free framework for processing
long texts, utilizing a divide-and-conquer strategy to achieve comprehensive
document understanding. The proposed LLMtimesMapReduce framework splits the
entire document into several chunks for LLMs to read and then aggregates the
intermediate answers to produce the final output. The main challenge for
divide-and-conquer long text processing frameworks lies in the risk of losing
essential long-range information when splitting the document, which can lead
the model to produce incomplete or incorrect answers based on the segmented
texts. Disrupted long-range information can be classified into two categories:
inter-chunk dependency and inter-chunk conflict. We design a structured
information protocol to better cope with inter-chunk dependency and an
in-context confidence calibration mechanism to resolve inter-chunk conflicts.
Experimental results demonstrate that LLMtimesMapReduce can outperform
representative open-source and commercial long-context LLMs, and is applicable
to several different models.
Empirical Study of Mutual Reinforcement Effect and Application in
  Few-shot Text Classification Tasks via Prompt
Authors: name: Chengguang Gan, name: Tatsunori Mori
Published: 2024-10-13 06:43:54

The Mutual Reinforcement Effect (MRE) investigates the synergistic
relationship between word-level and text-level classifications in text
classification tasks. It posits that the performance of both classification
levels can be mutually enhanced. However, this mechanism has not been
adequately demonstrated or explained in prior research. To address this gap, we
employ empirical experiment to observe and substantiate the MRE theory. Our
experiments on 21 MRE mix datasets revealed the presence of MRE in the model
and its impact. Specifically, we conducted compare experiments use fine-tune.
The results of findings from comparison experiments corroborates the existence
of MRE. Furthermore, we extended the application of MRE to prompt learning,
utilizing word-level information as a verbalizer to bolster the model's
prediction of text-level classification labels. In our final experiment, the
F1-score significantly surpassed the baseline in 18 out of 21 MRE Mix datasets,
further validating the notion that word-level information enhances the language
model's comprehension of the text as a whole.
SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement
  Learning
Authors: name: Hojoon Lee, name: Dongyoon Hwang, name: Donghu Kim, name: Hyunseung Kim, name: Jun Jet Tai, name: Kaushik Subramanian, name: Peter R. Wurman, name: Jaegul Choo, name: Peter Stone, name: Takuma Seno
Published: 2024-10-13 07:20:53

Recent advances in CV and NLP have been largely driven by scaling up the
number of network parameters, despite traditional theories suggesting that
larger networks are prone to overfitting. These large networks avoid
overfitting by integrating components that induce a simplicity bias, guiding
models toward simple and generalizable solutions. However, in deep RL,
designing and scaling up networks have been less explored. Motivated by this
opportunity, we present SimBa, an architecture designed to scale up parameters
in deep RL by injecting a simplicity bias. SimBa consists of three components:
(i) an observation normalization layer that standardizes inputs with running
statistics, (ii) a residual feedforward block to provide a linear pathway from
the input to output, and (iii) a layer normalization to control feature
magnitudes. By scaling up parameters with SimBa, the sample efficiency of
various deep RL algorithms-including off-policy, on-policy, and unsupervised
methods-is consistently improved. Moreover, solely by integrating SimBa
architecture into SAC, it matches or surpasses state-of-the-art deep RL methods
with high computational efficiency across DMC, MyoSuite, and HumanoidBench.
These results demonstrate SimBa's broad applicability and effectiveness across
diverse RL algorithms and environments.
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
Authors: name: Ziyue Li, name: Tianyi Zhou
Published: 2024-10-14 17:59:44

While large language models (LLMs) excel on generation tasks, their
decoder-only architecture often limits their potential as embedding models if
no further representation finetuning is applied. Does this contradict their
claim of generalists? To answer the question, we take a closer look at
Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE
LLMs can serve as an off-the-shelf embedding model with promising performance
on a diverse class of embedding-focused tasks, without requiring any
finetuning. Moreover, our extensive analysis shows that the MoE routing weights
(RW) is complementary to the hidden state (HS) of LLMs, a widely-used
embedding. Compared to HS, we find that RW is more robust to the choice of
prompts and focuses on high-level semantics. Motivated by the analysis, we
propose MoEE combining RW and HS, which achieves better performance than using
either separately. Our exploration of their combination and prompting strategy
shed several novel insights, e.g., a weighted sum of RW and HS similarities
outperforms the similarity on their concatenation. Our experiments are
conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding
Benchmark (MTEB). The results demonstrate the significant improvement brought
by MoEE to LLM-based embedding without further finetuning.
Efficient Diffusion Models: A Comprehensive Survey from Principles to
  Practices
Authors: name: Zhiyuan Ma, name: Yuzhu Zhang, name: Guoli Jia, name: Liangliang Zhao, name: Yichao Ma, name: Mingjie Ma, name: Gaofeng Liu, name: Kaiyan Zhang, name: Jianjun Li, name: Bowen Zhou
Published: 2024-10-15 17:19:46

As one of the most popular and sought-after generative models in the recent
years, diffusion models have sparked the interests of many researchers and
steadily shown excellent advantage in various generative tasks such as image
synthesis, video generation, molecule design, 3D scene rendering and multimodal
generation, relying on their dense theoretical principles and reliable
application practices. The remarkable success of these recent efforts on
diffusion models comes largely from progressive design principles and efficient
architecture, training, inference, and deployment methodologies. However, there
has not been a comprehensive and in-depth review to summarize these principles
and practices to help the rapid understanding and application of diffusion
models. In this survey, we provide a new efficiency-oriented perspective on
these existing efforts, which mainly focuses on the profound principles and
efficient practices in architecture designs, model training, fast inference and
reliable deployment, to guide further theoretical research, algorithm migration
and model application for new scenarios in a reader-friendly way.
https://github.com/ponyzym/Efficient-DMs-Survey
LVD-2M: A Long-take Video Dataset with Temporally Dense Captions
Authors: name: Tianwei Xiong, name: Yuqing Wang, name: Daquan Zhou, name: Zhijie Lin, name: Jiashi Feng, name: Xihui Liu
Published: 2024-10-14 17:59:56

The efficacy of video generation models heavily depends on the quality of
their training datasets. Most previous video generation models are trained on
short video clips, while recently there has been increasing interest in
training long video generation models directly on longer videos. However, the
lack of such high-quality long videos impedes the advancement of long video
generation. To promote research in long video generation, we desire a new
dataset with four key features essential for training long video generation
models: (1) long videos covering at least 10 seconds, (2) long-take videos
without cuts, (3) large motion and diverse contents, and (4) temporally dense
captions. To achieve this, we introduce a new pipeline for selecting
high-quality long-take videos and generating temporally dense captions.
Specifically, we define a set of metrics to quantitatively assess video quality
including scene cuts, dynamic degrees, and semantic-level quality, enabling us
to filter high-quality long-take videos from a large amount of source videos.
Subsequently, we develop a hierarchical video captioning pipeline to annotate
long videos with temporally-dense captions. With this pipeline, we curate the
first long-take video dataset, LVD-2M, comprising 2 million long-take videos,
each covering more than 10 seconds and annotated with temporally dense
captions. We further validate the effectiveness of LVD-2M by fine-tuning video
generation models to generate long videos with dynamic motions. We believe our
work will significantly contribute to future research in long video generation.
Towards Synergistic, Generalized, and Efficient Dual-System for Robotic
  Manipulation
Authors: name: Qingwen Bu, name: Hongyang Li, name: Li Chen, name: Jisong Cai, name: Jia Zeng, name: Heming Cui, name: Maoqing Yao, name: Yu Qiao
Published: 2024-10-10 14:57:51

The increasing demand for versatile robotic systems to operate in diverse and
dynamic environments has emphasized the importance of a generalist policy,
which leverages a large cross-embodiment data corpus to facilitate broad
adaptability and high-level reasoning. However, the generalist would struggle
with inefficient inference and cost-expensive training. The specialist policy,
instead, is curated for specific domain data and excels at task-level precision
with efficiency. Yet, it lacks the generalization capacity for a wide range of
applications. Inspired by these observations, we introduce RoboDual, a
synergistic dual-system that supplements the merits of both generalist and
specialist policy. A diffusion transformer-based specialist is devised for
multi-step action rollouts, exquisitely conditioned on the high-level task
understanding and discretized action output of a vision-language-action (VLA)
based generalist. Compared to OpenVLA, RoboDual achieves 26.7% improvement in
real-world setting and 12% gain on CALVIN by introducing a specialist policy
with merely 20M trainable parameters. It maintains strong performance with 5%
of demonstration data only, and enables a 3.8 times higher control frequency in
real-world deployment. Code would be made publicly available. Our project page
is hosted at: https://opendrivelab.com/RoboDual/
What Matters in Transformers? Not All Attention is Needed
Authors: name: Shwai He, name: Guoheng Sun, name: Zheyu Shen, name: Ang Li
Published: 2024-06-22 08:41:48

While scaling Transformer-based large language models (LLMs) has demonstrated
promising performance across various tasks, it also introduces redundant
architectures, posing efficiency challenges for real-world deployment. Despite
some recognition of redundancy in LLMs, the variability of redundancy across
different architectures in transformers, such as MLP and Attention layers, is
under-explored. In this work, we investigate redundancy across different
modules within Transformers, including Blocks, MLP, and Attention layers, using
a similarity-based metric. Surprisingly, despite the critical role of attention
layers in distinguishing transformers from other architectures, we found that a
large portion of these layers exhibit excessively high similarity and can be
pruned without degrading performance. For instance, Llama-2-70B achieved a
48.4\% speedup with only a 2.4\% performance drop by pruning half of the
attention layers. Furthermore, by tracing model checkpoints throughout the
training process, we observed that attention layer redundancy is inherent and
consistent across training stages. Additionally, we further propose a method
that jointly drops Attention and MLP layers, allowing us to more aggressively
drop additional layers. For instance, when dropping 31 layers (Attention +
MLP), Llama-2-13B still retains 90\% of the performance on the MMLU task. Our
work provides valuable insights for future network architecture design. The
code is released at: https://github.com/Shwai-He/LLM-Drop.
MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation
Authors: name: Chenxi Wang, name: Xiang Chen, name: Ningyu Zhang, name: Bozhong Tian, name: Haoming Xu, name: Shumin Deng, name: Huajun Chen
Published: 2024-10-15 16:57:44

Multimodal Large Language Models (MLLMs) frequently exhibit hallucination
phenomena, but the underlying reasons remain poorly understood. In this paper,
we present an empirical analysis and find that, although MLLMs incorrectly
generate the objects in the final output, they are actually able to recognize
visual objects in the preceding layers. We speculate that this may be due to
the strong knowledge priors of the language model suppressing the visual
information, leading to hallucinations. Motivated by this, we propose a novel
dynamic correction decoding method for MLLMs (DeCo), which adaptively selects
the appropriate preceding layers and proportionally integrates knowledge into
the final layer to adjust the output logits. Note that DeCo is model agnostic
and can be seamlessly incorporated with various classic decoding strategies and
applied to different MLLMs. We evaluate DeCo on widely-used benchmarks,
demonstrating that it can reduce hallucination rates by a large margin compared
to baselines, highlighting its potential to mitigate hallucinations. Code is
available at https://github.com/zjunlp/DeCo.
MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language
  Models
Authors: name: Pei Wang, name: Yanan Wu, name: Zekun Wang, name: Jiaheng Liu, name: Xiaoshuai Song, name: Zhongyuan Peng, name: Ken Deng, name: Chenchen Zhang, name: Jiakai Wang, name: Junran Peng, name: Ge Zhang, name: Hangyu Guo, name: Zhaoxiang Zhang, name: Wenbo Su, name: Bo Zheng
Published: 2024-10-15 15:46:17

Large Language Models (LLMs) have displayed massive improvements in reasoning
and decision-making skills and can hold natural conversations with users.
Recently, many tool-use benchmark datasets have been proposed. However,
existing datasets have the following limitations: (1). Insufficient evaluation
scenarios (e.g., only cover limited tool-use scenes). (2). Extensive evaluation
costs (e.g., GPT API costs). To address these limitations, in this work, we
propose a multi-granularity tool-use benchmark for large language models called
MTU-Bench. For the "multi-granularity" property, our MTU-Bench covers five tool
usage scenes (i.e., single-turn and single-tool, single-turn and multiple-tool,
multiple-turn and single-tool, multiple-turn and multiple-tool, and
out-of-distribution tasks). Besides, all evaluation metrics of our MTU-Bench
are based on the prediction results and the ground truth without using any GPT
or human evaluation metrics. Moreover, our MTU-Bench is collected by
transforming existing high-quality datasets to simulate real-world tool usage
scenarios, and we also propose an instruction dataset called MTU-Instruct data
to enhance the tool-use abilities of existing LLMs. Comprehensive experimental
results demonstrate the effectiveness of our MTU-Bench. Code and data will be
released at https: //github.com/MTU-Bench-Team/MTU-Bench.git.
SecCodePLT: A Unified Platform for Evaluating the Security of Code GenAI
Authors: name: Yu Yang, name: Yuzhou Nie, name: Zhun Wang, name: Yuheng Tang, name: Wenbo Guo, name: Bo Li, name: Dawn Song
Published: 2024-10-14 21:17:22

Existing works have established multiple benchmarks to highlight the security
risks associated with Code GenAI. These risks are primarily reflected in two
areas: a model potential to generate insecure code (insecure coding) and its
utility in cyberattacks (cyberattack helpfulness). While these benchmarks have
made significant strides, there remain opportunities for further improvement.
For instance, many current benchmarks tend to focus more on a model ability to
provide attack suggestions rather than its capacity to generate executable
attacks. Additionally, most benchmarks rely heavily on static evaluation
metrics, which may not be as precise as dynamic metrics such as passing test
cases. Conversely, expert-verified benchmarks, while offering high-quality
data, often operate at a smaller scale. To address these gaps, we develop
SecCodePLT, a unified and comprehensive evaluation platform for code GenAIs'
risks. For insecure code, we introduce a new methodology for data creation that
combines experts with automatic generation. Our methodology ensures the data
quality while enabling large-scale generation. We also associate samples with
test cases to conduct code-related dynamic evaluation. For cyberattack
helpfulness, we set up a real environment and construct samples to prompt a
model to generate actual attacks, along with dynamic metrics in our
environment. We conduct extensive experiments and show that SecCodePLT
outperforms the state-of-the-art (SOTA) benchmark CyberSecEval in security
relevance. Furthermore, it better identifies the security risks of SOTA models
in insecure coding and cyberattack helpfulness. Finally, we apply SecCodePLT to
the SOTA code agent, Cursor, and, for the first time, identify non-trivial
security risks in this advanced coding agent.
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
  Language Family Experts
Authors: name: Guorui Zheng, name: Xidong Wang, name: Juhao Liang, name: Nuo Chen, name: Yuping Zheng, name: Benyou Wang
Published: 2024-10-14 15:31:54

Adapting medical Large Language Models to local languages can reduce barriers
to accessing healthcare services, but data scarcity remains a significant
challenge, particularly for low-resource languages. To address this, we first
construct a high-quality medical dataset and conduct analysis to ensure its
quality. In order to leverage the generalization capability of multilingual
LLMs to efficiently scale to more resource-constrained languages, we explore
the internal information flow of LLMs from a multilingual perspective using
Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
routing method that employs language-specific experts and cross-lingual
routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
in the End information flow mechanism: while earlier layers concentrate
cross-lingual information flow, the later layers exhibit language-specific
divergence. This insight directly led to the development of the Post-MoE
architecture, which applies sparse routing only in the later layers while
maintaining dense others. Experimental results demonstrate that this approach
enhances the generalization of multilingual models to other languages while
preserving interpretability. Finally, to efficiently scale the model to 50
languages, we introduce the concept of language family experts, drawing on
linguistic priors, which enables scaling the number of languages without adding
additional parameters.
GS^3: Efficient Relighting with Triple Gaussian Splatting
Authors: name: Zoubin Bi, name: Yixin Zeng, name: Chong Zeng, name: Fan Pei, name: Xiang Feng, name: Kun Zhou, name: Hongzhi Wu
Published: 2024-10-15 09:11:30

We present a spatial and angular Gaussian based representation and a triple
splatting process, for real-time, high-quality novel lighting-and-view
synthesis from multi-view point-lit input images. To describe complex
appearance, we employ a Lambertian plus a mixture of angular Gaussians as an
effective reflectance function for each spatial Gaussian. To generate
self-shadow, we splat all spatial Gaussians towards the light source to obtain
shadow values, which are further refined by a small multi-layer perceptron. To
compensate for other effects like global illumination, another network is
trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness
of our representation is demonstrated on 30 samples with a wide variation in
geometry (from solid to fluffy) and appearance (from translucent to
anisotropic), as well as using different forms of input data, including
rendered images of synthetic/reconstructed objects, photographs captured with a
handheld camera and a flash, or from a professional lightstage. We achieve a
training time of 40-70 minutes and a rendering speed of 90 fps on a single
commodity GPU. Our results compare favorably with state-of-the-art techniques
in terms of quality/performance. Our code and data are publicly available at
https://GSrelight.github.io/.
Latent Action Pretraining from Videos
Authors: name: Seonghyeon Ye, name: Joel Jang, name: Byeongguk Jeon, name: Sejune Joo, name: Jianwei Yang, name: Baolin Peng, name: Ajay Mandlekar, name: Reuben Tan, name: Yu-Wei Chao, name: Bill Yuchen Lin, name: Lars Liden, name: Kimin Lee, name: Jianfeng Gao, name: Luke Zettlemoyer, name: Dieter Fox, name: Minjoon Seo
Published: 2024-10-15 16:28:09

We introduce Latent Action Pretraining for general Action models (LAPA), an
unsupervised method for pretraining Vision-Language-Action (VLA) models without
ground-truth robot action labels. Existing Vision-Language-Action models
require action labels typically collected by human teleoperators during
pretraining, which significantly limits possible data sources and scale. In
this work, we propose a method to learn from internet-scale videos that do not
have robot action labels. We first train an action quantization model
leveraging VQ-VAE-based objective to learn discrete latent actions between
image frames, then pretrain a latent VLA model to predict these latent actions
from observations and task descriptions, and finally finetune the VLA on
small-scale robot manipulation data to map from latent to robot actions.
Experimental results demonstrate that our method significantly outperforms
existing techniques that train robot manipulation policies from large-scale
videos. Furthermore, it outperforms the state-of-the-art VLA model trained with
robotic action labels on real-world manipulation tasks that require language
conditioning, generalization to unseen objects, and semantic generalization to
unseen instructions. Training only on human manipulation videos also shows
positive transfer, opening up the potential for leveraging web-scale data for
robotics foundation model.