Orionnnn/hub-stats huggingface dataset:
Adaptive Decoding via Latent Preference Optimization
Authors: name: Shehzaad Dhuliawala, name: Ilia Kulikov, name: Ping Yu, name: Asli Celikyilmaz, name: Jason Weston, name: Sainbayar Sukhbaatar, name: Jack Lanchantin
Published: 2024-11-14 18:31:39

During language model decoding, it is known that using higher temperature
sampling gives more creative responses, while lower temperatures are more
factually accurate. However, such models are commonly applied to general
instruction following, which involves both creative and fact seeking tasks,
using a single fixed temperature across all examples and tokens. In this work,
we introduce Adaptive Decoding, a layer added to the model to select the
sampling temperature dynamically at inference time, at either the token or
example level, in order to optimize performance. To learn its parameters we
introduce Latent Preference Optimization (LPO) a general approach to train
discrete latent variables such as choices of temperature. Our method
outperforms all fixed decoding temperatures across a range of tasks that
require different temperatures, including UltraFeedback, Creative Story
Writing, and GSM8K.
StableV2V: Stablizing Shape Consistency in Video-to-Video Editing
Authors: name: Chang Liu, name: Rui Li, name: Kaidong Zhang, name: Yunwei Lan, name: Dong Liu
Published: 2024-11-17 11:48:01

Recent advancements of generative AI have significantly promoted content
creation and editing, where prevailing studies further extend this exciting
progress to video editing. In doing so, these studies mainly transfer the
inherent motion patterns from the source videos to the edited ones, where
results with inferior consistency to user prompts are often observed, due to
the lack of particular alignments between the delivered motions and edited
contents. To address this limitation, we present a shape-consistent video
editing method, namely StableV2V, in this paper. Our method decomposes the
entire editing pipeline into several sequential procedures, where it edits the
first video frame, then establishes an alignment between the delivered motions
and user prompts, and eventually propagates the edited contents to all other
frames based on such alignment. Furthermore, we curate a testing benchmark,
namely DAVIS-Edit, for a comprehensive evaluation of video editing, considering
various types of prompts and difficulties. Experimental results and analyses
illustrate the outperforming performance, visual consistency, and inference
efficiency of our method compared to existing state-of-the-art studies.
LLäMmlein: Compact and Competitive German-Only Language Models from
  Scratch
Authors: name: Jan Pfister, name: Julia Wunderle, name: Andreas Hotho
Published: 2024-11-17 20:44:34

We create two German-only decoder models, LL\"aMmlein 120M and 1B,
transparently from scratch and publish them, along with the training data, for
the German NLP research community to use. The model training involved several
key steps, including extensive data preprocessing, the creation of a custom
German tokenizer, the training itself, as well as the evaluation of the final
models on various benchmarks. Throughout the training process, multiple
checkpoints were saved and analyzed using the SuperGLEBer benchmark to monitor
the models' learning dynamics. Compared to state-of-the-art models on the
SuperGLEBer benchmark, both LL\"aMmlein models performed competitively,
consistently matching or surpassing models with similar parameter sizes. The
results show that the models' quality scales with size as expected, but
performance improvements on some tasks plateaued early, offering valuable
insights into resource allocation for future model development.
Generative World Explorer
Authors: name: Taiming Lu, name: Tianmin Shu, name: Alan Yuille, name: Daniel Khashabi, name: Jieneng Chen
Published: 2024-11-18 18:59:31

Planning with partial observation is a central challenge in embodied AI. A
majority of prior works have tackled this challenge by developing agents that
physically explore their environment to update their beliefs about the world
state.In contrast, humans can imagine unseen parts of the world
through a mental exploration and revise their beliefs with imagined
observations. Such updated beliefs can allow them to make more informed
decisions, without necessitating the physical exploration of the world at all
times. To achieve this human-like ability, we introduce the Generative
World Explorer (Genex), an egocentric world exploration framework that allows
an agent to mentally explore a large-scale 3D world (e.g., urban scenes) and
acquire imagined observations to update its belief. This updated belief will
then help the agent to make a more informed decision at the current step. To
train Genex, we create a synthetic urban scene dataset, Genex-DB.
Our experimental results demonstrate that (1) Genex can generate
high-quality and consistent observations during long-horizon exploration of a
large virtual physical world and (2) the beliefs updated with the generated
observations can inform an existing decision-making model (e.g., an LLM agent)
to make better plans.
Comprehensive and Practical Evaluation of Retrieval-Augmented Generation
  Systems for Medical Question Answering
Authors: name: Nghia Trung Ngo, name: Chien Van Nguyen, name: Franck Dernoncourt, name: Thien Huu Nguyen
Published: 2024-11-14 06:19:18

Retrieval-augmented generation (RAG) has emerged as a promising approach to
enhance the performance of large language models (LLMs) in knowledge-intensive
tasks such as those from medical domain. However, the sensitive nature of the
medical domain necessitates a completely accurate and trustworthy system. While
existing RAG benchmarks primarily focus on the standard retrieve-answer
setting, they overlook many practical scenarios that measure crucial aspects of
a reliable medical system. This paper addresses this gap by providing a
comprehensive evaluation framework for medical question-answering (QA) systems
in a RAG setting for these situations, including sufficiency, integration, and
robustness. We introduce Medical Retrieval-Augmented Generation Benchmark
(MedRGB) that provides various supplementary elements to four medical QA
datasets for testing LLMs' ability to handle these specific scenarios.
Utilizing MedRGB, we conduct extensive evaluations of both state-of-the-art
commercial LLMs and open-source models across multiple retrieval conditions.
Our experimental results reveals current models' limited ability to handle
noise and misinformation in the retrieved documents. We further analyze the
LLMs' reasoning processes to provides valuable insights and future directions
for developing RAG systems in this critical medical domain.
SlimLM: An Efficient Small Language Model for On-Device Document
  Assistance
Authors: name: Thang M. Pham, name: Phat T. Nguyen, name: Seunghyun Yoon, name: Viet Dac Lai, name: Franck Dernoncourt, name: Trung Bui
Published: 2024-11-15 04:44:34

While small language models (SLMs) show promises for mobile deployment, their
real-world performance and applications on smartphones remains underexplored.
We present SlimLM, a series of SLMs optimized for document assistance tasks on
mobile devices. Through extensive experiments on a Samsung Galaxy S24, we
identify the optimal trade-offs between model size (ranging from 125M to 7B
parameters), context length, and inference time for efficient on-device
processing. SlimLM is pre-trained on SlimPajama-627B and fine-tuned on
DocAssist, our constructed dataset for summarization, question answering and
suggestion tasks. Our smallest model demonstrates efficient performance on S24,
while larger variants offer enhanced capabilities within mobile constraints. We
evaluate SlimLM against existing SLMs, showing comparable or superior
performance and offering a benchmark for future research in on-device language
models. We also provide an Android application, offering practical insights
into SLM deployment. Our findings provide valuable insights and illuminate the
capabilities of running advanced language models on high-end smartphones,
potentially reducing server costs and enhancing privacy through on-device
processing.
AnimateAnything: Consistent and Controllable Animation for Video
  Generation
Authors: name: Guojun Lei, name: Chi Wang, name: Hong Li, name: Rong Zhang, name: Yikai Wang, name: Weiwei Xu
Published: 2024-11-16 16:36:49

We present a unified controllable video generation approach AnimateAnything
that facilitates precise and consistent video manipulation across various
conditions, including camera trajectories, text prompts, and user motion
annotations. Specifically, we carefully design a multi-scale control feature
fusion network to construct a common motion representation for different
conditions. It explicitly converts all control information into frame-by-frame
optical flows. Then we incorporate the optical flows as motion priors to guide
final video generation. In addition, to reduce the flickering issues caused by
large-scale motion, we propose a frequency-based stabilization module. It can
enhance temporal coherence by ensuring the video's frequency domain
consistency. Experiments demonstrate that our method outperforms the
state-of-the-art approaches. For more details and videos, please refer to the
webpage: https://yu-shaonian.github.io/Animate_Anything/.
Search, Verify and Feedback: Towards Next Generation Post-training
  Paradigm of Foundation Models via Verifier Engineering
Authors: name: Xinyan Guan, name: Yanjiang Liu, name: Xinyu Lu, name: Boxi Cao, name: Ben He, name: Xianpei Han, name: Le Sun, name: Jie Lou, name: Bowen Yu, name: Yaojie Lu, name: Hongyu Lin
Published: 2024-11-18 12:04:52

The evolution of machine learning has increasingly prioritized the
development of powerful models and more scalable supervision signals. However,
the emergence of foundation models presents significant challenges in providing
effective supervision signals necessary for further enhancing their
capabilities. Consequently, there is an urgent need to explore novel
supervision signals and technical approaches. In this paper, we propose
verifier engineering, a novel post-training paradigm specifically designed for
the era of foundation models. The core of verifier engineering involves
leveraging a suite of automated verifiers to perform verification tasks and
deliver meaningful feedback to foundation models. We systematically categorize
the verifier engineering process into three essential stages: search, verify,
and feedback, and provide a comprehensive review of state-of-the-art research
developments within each stage. We believe that verifier engineering
constitutes a fundamental pathway toward achieving Artificial General
Intelligence.
Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
  Experts
Authors: name: Jinqiang Long, name: Yanqi Dai, name: Guoxing Yang, name: Hongpeng Lin, name: Nanyi Fei, name: Yizhao Gao, name: Zhiwu Lu
Published: 2024-11-16 02:10:14

As the research of Multimodal Large Language Models (MLLMs) becomes popular,
an advancing MLLM model is typically required to handle various textual and
visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
real-world applications. However, due to the significant differences in
representation and distribution among data from various tasks, simply mixing
data of all tasks together leads to the well-known``multi-task conflict" issue,
resulting in performance degradation across various tasks. To address this
issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
suitable for MLLM, which acquires the multi-task capabilities through multiple
sparsely activated experts. To speed up the training and inference of
Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
(LoRA) structure. Extensive experiments on multiple latest benchmarks
demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
released in our Project Page: https://github.com/MetabrainAGI/Awaker.
SmoothCache: A Universal Inference Acceleration Technique for Diffusion
  Transformers
Authors: name: Joseph Liu, name: Joshua Geddes, name: Ziyu Guo, name: Haomiao Jiang, name: Mahesh Kumar Nandwana
Published: 2024-11-15 16:24:02

Diffusion Transformers (DiT) have emerged as powerful generative models for
various tasks, including image, video, and speech synthesis. However, their
inference process remains computationally expensive due to the repeated
evaluation of resource-intensive attention and feed-forward modules. To address
this, we introduce SmoothCache, a model-agnostic inference acceleration
technique for DiT architectures. SmoothCache leverages the observed high
similarity between layer outputs across adjacent diffusion timesteps. By
analyzing layer-wise representation errors from a small calibration set,
SmoothCache adaptively caches and reuses key features during inference. Our
experiments demonstrate that SmoothCache achieves 8% to 71% speed up while
maintaining or even improving generation quality across diverse modalities. We
showcase its effectiveness on DiT-XL for image generation, Open-Sora for
text-to-video, and Stable Audio Open for text-to-audio, highlighting its
potential to enable real-time applications and broaden the accessibility of
powerful DiT models.
FitDiT: Advancing the Authentic Garment Details for High-fidelity
  Virtual Try-on
Authors: name: Boyuan Jiang, name: Xiaobin Hu, name: Donghao Luo, name: Qingdong He, name: Chengming Xu, name: Jinlong Peng, name: Jiangning Zhang, name: Chengjie Wang, name: Yunsheng Wu, name: Yanwei Fu
Published: 2024-11-15 11:02:23

Although image-based virtual try-on has made considerable progress, emerging
approaches still encounter challenges in producing high-fidelity and robust
fitting images across diverse scenarios. These methods often struggle with
issues such as texture-aware maintenance and size-aware fitting, which hinder
their overall effectiveness. To address these limitations, we propose a novel
garment perception enhancement technique, termed FitDiT, designed for
high-fidelity virtual try-on using Diffusion Transformers (DiT) allocating more
parameters and attention to high-resolution features. First, to further improve
texture-aware maintenance, we introduce a garment texture extractor that
incorporates garment priors evolution to fine-tune garment feature,
facilitating to better capture rich details such as stripes, patterns, and
text. Additionally, we introduce frequency-domain learning by customizing a
frequency distance loss to enhance high-frequency garment details. To tackle
the size-aware fitting issue, we employ a dilated-relaxed mask strategy that
adapts to the correct length of garments, preventing the generation of garments
that fill the entire mask area during cross-category try-on. Equipped with the
above design, FitDiT surpasses all baselines in both qualitative and
quantitative evaluations. It excels in producing well-fitting garments with
photorealistic and intricate details, while also achieving competitive
inference times of 4.57 seconds for a single 1024x768 image after DiT structure
slimming, outperforming existing methods.
BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large
  Language Models on Mobile Devices
Authors: name: Xudong Lu, name: Yinghao Chen, name: Cheng Chen, name: Hui Tan, name: Boheng Chen, name: Yina Xie, name: Rui Hu, name: Guanxin Tan, name: Renshou Wu, name: Yan Hu, name: Yi Zeng, name: Lei Wu, name: Liuyang Bian, name: Zhaoxiong Wang, name: Long Liu, name: Yanzhou Yang, name: Han Xiao, name: Aojun Zhou, name: Yafei Wen, name: Xiaoxin Chen, name: Shuai Ren, name: Hongsheng Li
Published: 2024-11-16 00:14:51

The emergence and growing popularity of multimodal large language models
(MLLMs) have significant potential to enhance various aspects of daily life,
from improving communication to facilitating learning and problem-solving.
Mobile phones, as essential daily companions, represent the most effective and
accessible deployment platform for MLLMs, enabling seamless integration into
everyday tasks. However, deploying MLLMs on mobile phones presents challenges
due to limitations in memory size and computational capability, making it
difficult to achieve smooth and real-time processing without extensive
optimization. In this paper, we present BlueLM-V-3B, an algorithm and system
co-design approach specifically tailored for the efficient deployment of MLLMs
on mobile platforms. To be specific, we redesign the dynamic resolution scheme
adopted by mainstream MLLMs and implement system optimization for
hardware-aware deployment to optimize model inference on mobile phones.
BlueLM-V-3B boasts the following key highlights: (1) Small Size: BlueLM-V-3B
features a language model with 2.7B parameters and a vision encoder with 400M
parameters. (2) Fast Speed: BlueLM-V-3B achieves a generation speed of 24.4
token/s on the MediaTek Dimensity 9300 processor with 4-bit LLM weight
quantization. (3) Strong Performance: BlueLM-V-3B has attained the highest
average score of 66.1 on the OpenCompass benchmark among models with leq 4B
parameters and surpassed a series of models with much larger parameter sizes
(e.g., MiniCPM-V-2.6, InternVL2-8B).
Drowning in Documents: Consequences of Scaling Reranker Inference
Authors: name: Mathew Jacob, name: Erik Lindgren, name: Matei Zaharia, name: Michael Carbin, name: Omar Khattab, name: Andrew Drozdov
Published: 2024-11-18 17:46:32

Rerankers, typically cross-encoders, are often used to re-score the documents
retrieved by cheaper initial IR systems. This is because, though expensive,
rerankers are assumed to be more effective. We challenge this assumption by
measuring reranker performance for full retrieval, not just re-scoring
first-stage retrieval. Our experiments reveal a surprising trend: the best
existing rerankers provide diminishing returns when scoring progressively more
documents and actually degrade quality beyond a certain limit. In fact, in this
setting, rerankers can frequently assign high scores to documents with no
lexical or semantic overlap with the query. We hope that our findings will spur
future research to improve reranking.
Top-nσ: Not All Logits Are You Need
Authors: name: Chenxia Tang, name: Jianchun Liu, name: Hongli Xu, name: Liusheng Huang
Published: 2024-11-12 08:46:43

Large language models (LLMs) typically employ greedy decoding or
low-temperature sampling for reasoning tasks, reflecting a perceived trade-off
between diversity and accuracy. We challenge this convention by introducing
top-nsigma, a novel sampling method that operates directly on pre-softmax
logits by leveraging a statistical threshold. Our key insight is that logits
naturally separate into a Gaussian-distributed noisy region and a distinct
informative region, enabling efficient token filtering without complex
probability manipulations. Unlike existing methods (e.g., top-p, min-p)
that inadvertently include more noise tokens at higher temperatures,
top-nsigma maintains a stable sampling space regardless of temperature
scaling. We also provide a theoretical analysis of top-nsigma to better
understand its behavior. The extensive experimental results across four
reasoning-focused datasets demonstrate that our method not only outperforms
existing sampling approaches but also surpasses greedy decoding, while
maintaining consistent performance even at high temperatures.
MARS: Unleashing the Power of Variance Reduction for Training Large
  Models
Authors: name: Huizhuo Yuan, name: Yifeng Liu, name: Shuang Wu, name: Xun Zhou, name: Quanquan Gu
Published: 2024-11-15 18:57:39

Training deep neural networks--and more recently, large models--demands
efficient and scalable optimizers. Adaptive gradient algorithms like Adam,
AdamW, and their variants have been central to this task. Despite the
development of numerous variance reduction algorithms in the past decade aimed
at accelerating stochastic optimization in both convex and nonconvex settings,
variance reduction has not found widespread success in training deep neural
networks or large language models. Consequently, it has remained a less favored
approach in modern AI. In this paper, to unleash the power of variance
reduction for efficient training of large models, we propose a unified
optimization framework, MARS (Make vAriance Reduction Shine), which reconciles
preconditioned gradient methods with variance reduction via a scaled stochastic
recursive momentum technique. Within our framework, we introduce three
instances of MARS that leverage preconditioned gradient updates based on AdamW,
Lion, and Shampoo, respectively. We also draw a connection between our
algorithms and existing optimizers. Experimental results on training GPT-2
models indicate that MARS consistently outperforms AdamW by a large margin.
Region-Aware Text-to-Image Generation via Hard Binding and Soft
  Refinement
Authors: name: Zhennan Chen, name: Yajie Li, name: Haofan Wang, name: Zhibo Chen, name: Zhengkai Jiang, name: Jun Li, name: Qian Wang, name: Jian Yang, name: Ying Tai
Published: 2024-11-10 18:45:41

In this paper, we present RAG, a Regional-Aware text-to-image Generation
method conditioned on regional descriptions for precise layout composition.
Regional prompting, or compositional generation, which enables fine-grained
spatial control, has gained increasing attention for its practicality in
real-world applications. However, previous methods either introduce additional
trainable modules, thus only applicable to specific models, or manipulate on
score maps within cross-attention layers using attention masks, resulting in
limited control strength when the number of regions increases. To handle these
limitations, we decouple the multi-region generation into two sub-tasks, the
construction of individual region (Regional Hard Binding) that ensures the
regional prompt is properly executed, and the overall detail refinement
(Regional Soft Refinement) over regions that dismiss the visual boundaries and
enhance adjacent interactions. Furthermore, RAG novelly makes repainting
feasible, where users can modify specific unsatisfied regions in the last
generation while keeping all other regions unchanged, without relying on
additional inpainting models. Our approach is tuning-free and applicable to
other frameworks as an enhancement to the prompt following property.
Quantitative and qualitative experiments demonstrate that RAG achieves superior
performance over attribute binding and object relationship than previous
tuning-free methods.
Xmodel-1.5: An 1B-scale Multilingual LLM
Authors: name: Wang Qun, name: Liu Yang, name: Lin Qingquan, name: Jiang Ling
Published: 2024-11-15 10:01:52

We introduce Xmodel-1.5, a novel 1-billion-parameter multilingual large model
pretrained on approximately 2 trillion tokens. The model demonstrates strong
performance across several languages, with particularly notable results in
Thai, Arabic, and French, alongside its effectiveness in Chinese and English.
In addition, we contribute to the research community by releasing a Thai
evaluation dataset, which includes hundreds of questions annotated by students
from Chulalongkorn University's School of Integrated Innovation. While the
results are promising, we acknowledge that there is still room for improvement.
We hope this work advances ongoing efforts in multilingual AI research and
promotes better cross-linguistic understanding in various natural language
processing tasks. Our models and code are publicly available on GitHub at
https://github.com/XiaoduoAILab/XmodelLM.