Orionnnn/hub-stats huggingface dataset:
ALPINE: Unveiling the Planning Capability of Autoregressive Learning in
  Language Models
Authors: name: Siwei Wang, name: Yifei Shen, name: Shi Feng, name: Haoran Sun, name: Shang-Hua Teng, name: Wei Chen
Published: 2024-05-15 09:59:37

In this paper, we present the findings of our Project ALPINE which stands for
``Autoregressive Learning for Planning In NEtworks." Project ALPINE initiates a
theoretical investigation into the development of planning capabilities in
Transformer-based language models through their autoregressive learning
mechanisms, aiming to identify any potential limitations in their planning
abilities. We abstract planning as a network path-finding task where the
objective is to generate a valid path from a specified source node to a
designated target node. In terms of expressiveness, we show that the
Transformer is capable of executing path-finding by embedding the adjacency and
reachability matrices within its weights. Our theoretical analysis of the
gradient-based learning dynamic of the Transformer reveals that the Transformer
is capable of learning both the adjacency matrix and a limited form of the
reachability matrix. These theoretical insights are then validated through
experiments, which demonstrate that the Transformer indeed learns the adjacency
matrix and an incomplete reachability matrix, which aligns with the predictions
made in our theoretical analysis. Additionally, when applying our methodology
to a real-world planning benchmark, called Blocksworld, our observations remain
consistent. Our theoretical and empirical analyses further unveil a potential
limitation of Transformer in path-finding: it cannot identify reachability
relationships through transitivity, and thus would fail when path concatenation
is needed to generate a path. In summary, our findings shed new light on how
the internal mechanisms of autoregressive learning enable planning in networks.
This study may contribute to our understanding of the general planning
capabilities in other related domains.
Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model
Authors: name: Wanting Xu, name: Yang Liu, name: Langping He, name: Xucheng Huang, name: Ling Jiang
Published: 2024-05-15 09:47:59

We introduce Xmodel-VLM, a cutting-edge multimodal vision language model. It
is designed for efficient deployment on consumer GPU servers. Our work directly
confronts a pivotal industry issue by grappling with the prohibitive service
costs that hinder the broad adoption of large-scale multimodal systems. Through
rigorous training, we have developed a 1B-scale language model from the ground
up, employing the LLaVA paradigm for modal alignment. The result, which we call
Xmodel-VLM, is a lightweight yet powerful multimodal vision language model.
Extensive testing across numerous classic multimodal benchmarks has revealed
that despite its smaller size and faster execution, Xmodel-VLM delivers
performance comparable to that of larger models. Our model checkpoints and code
are publicly available on GitHub at https://github.com/XiaoduoAILab/XmodelVLM.
No Time to Waste: Squeeze Time into Channel for Mobile Video
  Understanding
Authors: name: Yingjie Zhai, name: Wenshuo Li, name: Yehui Tang, name: Xinghao Chen, name: Yunhe Wang
Published: 2024-05-14 06:32:40

Current architectures for video understanding mainly build upon 3D
convolutional blocks or 2D convolutions with additional operations for temporal
modeling. However, these methods all regard the temporal axis as a separate
dimension of the video sequence, which requires large computation and memory
budgets and thus limits their usage on mobile devices. In this paper, we
propose to squeeze the time axis of a video sequence into the channel dimension
and present a lightweight video recognition network, term as
SqueezeTime, for mobile video understanding. To enhance the temporal
modeling capability of the proposed network, we design a Channel-Time Learning
(CTL) Block to capture temporal dynamics of the sequence. This module has two
complementary branches, in which one branch is for temporal importance learning
and another branch with temporal position restoring capability is to enhance
inter-temporal object modeling ability. The proposed SqueezeTime is much
lightweight and fast with high accuracies for mobile video understanding.
Extensive experiments on various video recognition and action detection
benchmarks, i.e., Kinetics400, Kinetics600, HMDB51, AVA2.1 and THUMOS14,
demonstrate the superiority of our model. For example, our SqueezeTime achieves
+1.2% accuracy and +80% GPU throughput gain on Kinetics400 than prior
methods. Codes are publicly available at
https://github.com/xinghaochen/SqueezeTime and
https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SqueezeTime.
Beyond Scaling Laws: Understanding Transformer Performance with
  Associative Memory
Authors: name: Xueyan Niu, name: Bo Bai, name: Lei Deng, name: Wei Han
Published: 2024-05-14 15:48:36

Increasing the size of a Transformer model does not always lead to enhanced
performance. This phenomenon cannot be explained by the empirical scaling laws.
Furthermore, improved generalization ability occurs as the model memorizes the
training samples. We present a theoretical framework that sheds light on the
memorization process and performance dynamics of transformer-based language
models. We model the behavior of Transformers with associative memories using
Hopfield networks, such that each transformer block effectively conducts an
approximate nearest-neighbor search. Based on this, we design an energy
function analogous to that in the modern continuous Hopfield network which
provides an insightful explanation for the attention mechanism. Using the
majorization-minimization technique, we construct a global energy function that
captures the layered architecture of the Transformer. Under specific
conditions, we show that the minimum achievable cross-entropy loss is bounded
from below by a constant approximately equal to 1. We substantiate our
theoretical results by conducting experiments with GPT-2 on various data sizes,
as well as training vanilla Transformers on a dataset of 2M tokens.
Understanding the performance gap between online and offline alignment
  algorithms
Authors: name: Yunhao Tang, name: Daniel Zhaohan Guo, name: Zeyu Zheng, name: Daniele Calandriello, name: Yuan Cao, name: Eugene Tarassov, name: Rémi Munos, name: Bernardo Ávila Pires, name: Michal Valko, name: Yong Cheng, name: Will Dabney
Published: 2024-05-14 09:12:30

Reinforcement learning from human feedback (RLHF) is the canonical framework
for large language model alignment. However, rising popularity in offline
alignment algorithms challenge the need for on-policy sampling in RLHF. Within
the context of reward over-optimization, we start with an opening set of
experiments that demonstrate the clear advantage of online methods over offline
methods. This prompts us to investigate the causes to the performance
discrepancy through a series of carefully designed experimental ablations. We
show empirically that hypotheses such as offline data coverage and data quality
by itself cannot convincingly explain the performance difference. We also find
that while offline algorithms train policy to become good at pairwise
classification, it is worse at generations; in the meantime the policies
trained by online algorithms are good at generations while worse at pairwise
classification. This hints at a unique interplay between discriminative and
generative capabilities, which is greatly impacted by the sampling process.
Lastly, we observe that the performance discrepancy persists for both
contrastive and non-contrastive loss functions, and appears not to be addressed
by simply scaling up policy networks. Taken together, our study sheds light on
the pivotal role of on-policy sampling in AI alignment, and hints at certain
fundamental challenges of offline alignment algorithms.
SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large
  Language Models
Authors: name: Raghuveer Peri, name: Sai Muralidhar Jayanthi, name: Srikanth Ronanki, name: Anshu Bhatia, name: Karel Mundnich, name: Saket Dingliwal, name: Nilaksh Das, name: Zejiang Hou, name: Goeric Huybrechts, name: Srikanth Vishnubhotla, name: Daniel Garcia-Romero, name: Sundararajan Srinivasan, name: Kyu J Han, name: Katrin Kirchhoff
Published: 2024-05-14 04:51:23

Integrated Speech and Large Language Models (SLMs) that can follow speech
instructions and generate relevant text responses have gained popularity
lately. However, the safety and robustness of these models remains largely
unclear. In this work, we investigate the potential vulnerabilities of such
instruction-following speech-language models to adversarial attacks and
jailbreaking. Specifically, we design algorithms that can generate adversarial
examples to jailbreak SLMs in both white-box and black-box attack settings
without human involvement. Additionally, we propose countermeasures to thwart
such jailbreaking attacks. Our models, trained on dialog data with speech
instructions, achieve state-of-the-art performance on spoken question-answering
task, scoring over 80% on both safety and helpfulness metrics. Despite safety
guardrails, experiments on jailbreaking demonstrate the vulnerability of SLMs
to adversarial perturbations and transfer attacks, with average attack success
rates of 90% and 10% respectively when evaluated on a dataset of carefully
designed harmful questions spanning 12 different toxic categories. However, we
demonstrate that our proposed countermeasures reduce the attack success
significantly.
SpeechVerse: A Large-scale Generalizable Audio Language Model
Authors: name: Nilaksh Das, name: Saket Dingliwal, name: Srikanth Ronanki, name: Rohit Paturi, name: David Huang, name: Prashant Mathur, name: Jie Yuan, name: Dhanush Bekal, name: Xing Niu, name: Sai Muralidhar Jayanthi, name: Xilai Li, name: Karel Mundnich, name: Monica Sunkara, name: Sundararajan Srinivasan, name: Kyu J Han, name: Katrin Kirchhoff
Published: 2024-05-14 03:33:31

Large language models (LLMs) have shown incredible proficiency in performing
tasks that require semantic understanding of natural language instructions.
Recently, many works have further expanded this capability to perceive
multimodal audio and text inputs, but their capabilities are often limited to
specific fine-tuned tasks such as automatic speech recognition and translation.
We therefore develop SpeechVerse, a robust multi-task training and curriculum
learning framework that combines pre-trained speech and text foundation models
via a small set of learnable parameters, while keeping the pre-trained models
frozen during training. The models are instruction finetuned using continuous
latent representations extracted from the speech foundation model to achieve
optimal zero-shot performance on a diverse range of speech processing tasks
using natural language instructions. We perform extensive benchmarking that
includes comparing our model performance against traditional baselines across
several datasets and tasks. Furthermore, we evaluate the model's capability for
generalized instruction following by testing on out-of-domain datasets, novel
prompts, and unseen tasks. Our empirical experiments reveal that our multi-task
SpeechVerse model is even superior to conventional task-specific baselines on 9
out of the 11 tasks.
Coin3D: Controllable and Interactive 3D Assets Generation with
  Proxy-Guided Conditioning
Authors: name: Wenqi Dong, name: Bangbang Yang, name: Lin Ma, name: Xiao Liu, name: Liyuan Cui, name: Hujun Bao, name: Yuewen Ma, name: Zhaopeng Cui
Published: 2024-05-13 17:56:13

As humans, we aspire to create media content that is both freely willed and
readily controlled. Thanks to the prominent development of generative
techniques, we now can easily utilize 2D diffusion methods to synthesize images
controlled by raw sketch or designated human poses, and even progressively
edit/regenerate local regions with masked inpainting. However, similar
workflows in 3D modeling tasks are still unavailable due to the lack of
controllability and efficiency in 3D generation. In this paper, we present a
novel controllable and interactive 3D assets modeling framework, named Coin3D.
Coin3D allows users to control the 3D generation using a coarse geometry proxy
assembled from basic shapes, and introduces an interactive generation workflow
to support seamless local part editing while delivering responsive 3D object
previewing within a few seconds. To this end, we develop several techniques,
including the 3D adapter that applies volumetric coarse shape control to the
diffusion model, proxy-bounded editing strategy for precise part editing,
progressive volume cache to support responsive preview, and volume-SDS to
ensure consistent mesh reconstruction. Extensive experiments of interactive
generation and editing on diverse shape proxies demonstrate that our method
achieves superior controllability and flexibility in the 3D assets generation
task.
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with
  Fine-Grained Chinese Understanding
Authors: name: Zhimin Li, name: Jianwei Zhang, name: Qin Lin, name: Jiangfeng Xiong, name: Yanxin Long, name: Xinchi Deng, name: Yingfang Zhang, name: Xingchao Liu, name: Minbin Huang, name: Zedong Xiao, name: Dayou Chen, name: Jiajun He, name: Jiahao Li, name: Wenyue Li, name: Chen Zhang, name: Rongwei Quan, name: Jianxiang Lu, name: Jiabin Huang, name: Xiaoyan Yuan, name: Xiaoxiao Zheng, name: Yixuan Li, name: Jihong Zhang, name: Chao Zhang, name: Meng Chen, name: Jie Liu, name: Zheng Fang, name: Weiyan Wang, name: Jinbao Xue, name: Yangyu Tao, name: Jianchen Zhu, name: Kai Liu, name: Sihuan Lin, name: Yifu Sun, name: Yun Li, name: Dongdong Wang, name: Mingtao Chen, name: Zhichao Hu, name: Xiao Xiao, name: Yan Chen, name: Yuhong Liu, name: Wei Liu, name: Di Wang, name: Yong Yang, name: Jie Jiang, name: Qinglin Lu
Published: 2024-05-14 16:33:25

We present Hunyuan-DiT, a text-to-image diffusion transformer with
fine-grained understanding of both English and Chinese. To construct
Hunyuan-DiT, we carefully design the transformer structure, text encoder, and
positional encoding. We also build from scratch a whole data pipeline to update
and evaluate data for iterative model optimization. For fine-grained language
understanding, we train a Multimodal Large Language Model to refine the
captions of the images. Finally, Hunyuan-DiT can perform multi-turn multimodal
dialogue with users, generating and refining images according to the context.
Through our holistic human evaluation protocol with more than 50 professional
human evaluators, Hunyuan-DiT sets a new state-of-the-art in Chinese-to-image
generation compared with other open-source models. Code and pretrained models
are publicly available at github.com/Tencent/HunyuanDiT
Compositional Text-to-Image Generation with Dense Blob Representations
Authors: name: Weili Nie, name: Sifei Liu, name: Morteza Mardani, name: Chao Liu, name: Benjamin Eckart, name: Arash Vahdat
Published: 2024-05-14 00:22:06

Existing text-to-image models struggle to follow complex text prompts,
raising the need for extra grounding inputs for better controllability. In this
work, we propose to decompose a scene into visual primitives - denoted as dense
blob representations - that contain fine-grained details of the scene while
being modular, human-interpretable, and easy-to-construct. Based on blob
representations, we develop a blob-grounded text-to-image diffusion model,
termed BlobGEN, for compositional generation. Particularly, we introduce a new
masked cross-attention module to disentangle the fusion between blob
representations and visual features. To leverage the compositionality of large
language models (LLMs), we introduce a new in-context learning approach to
generate blob representations from text prompts. Our extensive experiments show
that BlobGEN achieves superior zero-shot generation quality and better
layout-guided controllability on MS-COCO. When augmented by LLMs, our method
exhibits superior numerical and spatial correctness on compositional image
generation benchmarks. Project page: https://blobgen-2d.github.io.
What matters when building vision-language models?
Authors: name: Hugo Laurençon, name: Léo Tronchon, name: Matthieu Cord, name: Victor Sanh
Published: 2024-05-03 17:00:00

The growing interest in vision-language models (VLMs) has been driven by
improvements in large language models and vision transformers. Despite the
abundance of literature on this subject, we observe that critical decisions
regarding the design of VLMs are often not justified. We argue that these
unsupported decisions impede progress in the field by making it difficult to
identify which choices improve model performance. To address this issue, we
conduct extensive experiments around pre-trained models, architecture choice,
data, and training methods. Our consolidation of findings includes the
development of Idefics2, an efficient foundational VLM of 8 billion parameters.
Idefics2 achieves state-of-the-art performance within its size category across
various multimodal benchmarks, and is often on par with models four times its
size. We release the model (base, instructed, and chat) along with the datasets
created for its training.
Piccolo2: General Text Embedding with Multi-task Hybrid Loss Training
Authors: name: Junqin Huang, name: Zhongjie Hu, name: Zihao Jing, name: Mengya Gao, name: Yichao Wu
Published: 2024-05-11 06:32:08

In this report, we introduce Piccolo2, an embedding model that surpasses
other models in the comprehensive evaluation over 6 tasks on CMTEB benchmark,
setting a new state-of-the-art. Piccolo2 primarily leverages an efficient
multi-task hybrid loss training approach, effectively harnessing textual data
and labels from diverse downstream tasks. In addition, Piccolo2 scales up the
embedding dimension and uses MRL training to support more flexible vector
dimensions. The latest information of piccolo models can be accessed via:
https://huggingface.co/sensenova/
MS MARCO Web Search: a Large-scale Information-rich Web Dataset with
  Millions of Real Click Labels
Authors: name: Qi Chen, name: Xiubo Geng, name: Corby Rosset, name: Carolyn Buractaon, name: Jingwen Lu, name: Tao Shen, name: Kun Zhou, name: Chenyan Xiong, name: Yeyun Gong, name: Paul Bennett, name: Nick Craswell, name: Xing Xie, name: Fan Yang, name: Bryan Tower, name: Nikhil Rao, name: Anlei Dong, name: Wenqi Jiang, name: Zheng Liu, name: Mingqin Li, name: Chuanjie Liu, name: Zengzhong Li, name: Rangan Majumder, name: Jennifer Neville, name: Andy Oakley, name: Knut Magne Risvik, name: Harsha Vardhan Simhadri, name: Manik Varma, name: Yujing Wang, name: Linjun Yang, name: Mao Yang, name: Ce Zhang
Published: 2024-05-13 07:46:44

Recent breakthroughs in large models have highlighted the critical
significance of data scale, labels and modals. In this paper, we introduce MS
MARCO Web Search, the first large-scale information-rich web dataset, featuring
millions of real clicked query-document labels. This dataset closely mimics
real-world web document and query distribution, provides rich information for
various kinds of downstream tasks and encourages research in various areas,
such as generic end-to-end neural indexer models, generic embedding models, and
next generation information access system with large language models. MS MARCO
Web Search offers a retrieval benchmark with three web retrieval challenge
tasks that demand innovations in both machine learning and information
retrieval system research domains. As the first dataset that meets large, real
and rich data requirements, MS MARCO Web Search paves the way for future
advancements in AI and system research. MS MARCO Web Search dataset is
available at: https://github.com/microsoft/MS-MARCO-Web-Search.
LogoMotion: Visually Grounded Code Generation for Content-Aware
  Animation
Authors: name: Vivian Liu, name: Rubaiat Habib Kazi, name: Li-Yi Wei, name: Matthew Fisher, name: Timothy Langlois, name: Seth Walker, name: Lydia Chilton
Published: 2024-05-11 18:20:40

Animated logos are a compelling and ubiquitous way individuals and brands
represent themselves online. Manually authoring these logos can require
significant artistic skill and effort. To help novice designers animate logos,
design tools currently offer templates and animation presets. However, these
solutions can be limited in their expressive range. Large language models have
the potential to help novice designers create animated logos by generating
animation code that is tailored to their content. In this paper, we introduce
LogoMotion, an LLM-based system that takes in a layered document and generates
animated logos through visually-grounded program synthesis. We introduce
techniques to create an HTML representation of a canvas, identify primary and
secondary elements, synthesize animation code, and visually debug animation
errors. When compared with an industry standard tool, we find that LogoMotion
produces animations that are more content-aware and are on par in terms of
quality. We conclude with a discussion of the implications of LLM-generated
animation for motion design.
Large Language Models as Planning Domain Generators
Authors: name: James Oswald, name: Kavitha Srinivas, name: Harsha Kokel, name: Junkyu Lee, name: Michael Katz, name: Shirin Sohrabi
Published: 2024-04-02 19:39:23

Developing domain models is one of the few remaining places that require
manual human labor in AI planning. Thus, in order to make planning more
accessible, it is desirable to automate the process of domain model generation.
To this end, we investigate if large language models (LLMs) can be used to
generate planning domain models from simple textual descriptions. Specifically,
we introduce a framework for automated evaluation of LLM-generated domains by
comparing the sets of plans for domain instances. Finally, we perform an
empirical analysis of 7 large language models, including coding and chat models
across 9 different planning domains, and under three classes of natural
language domain descriptions. Our results indicate that LLMs, particularly
those with high parameter counts, exhibit a moderate level of proficiency in
generating correct planning domains from natural language descriptions. Our
code is available at https://github.com/IBM/NL2PDDL.
RLHF Workflow: From Reward Modeling to Online RLHF
Authors: name: Hanze Dong, name: Wei Xiong, name: Bo Pang, name: Haoxiang Wang, name: Han Zhao, name: Yingbo Zhou, name: Nan Jiang, name: Doyen Sahoo, name: Caiming Xiong, name: Tong Zhang
Published: 2024-05-13 15:50:39

We present the workflow of Online Iterative Reinforcement Learning from Human
Feedback (RLHF) in this technical report, which is widely reported to
outperform its offline counterpart by a large margin in the recent large
language model (LLM) literature. However, existing open-source RLHF projects
are still largely confined to the offline learning setting. In this technical
report, we aim to fill in this gap and provide a detailed recipe that is easy
to reproduce for online iterative RLHF. In particular, since online human
feedback is usually infeasible for open-source communities with limited
resources, we start by constructing preference models using a diverse set of
open-source datasets and use the constructed proxy preference model to
approximate human feedback. Then, we discuss the theoretical insights and
algorithmic principles behind online iterative RLHF, followed by a detailed
practical implementation. Our trained LLM, SFR-Iterative-DPO-LLaMA-3-8B-R,
achieves impressive performance on LLM chatbot benchmarks, including
AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks
such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning
(SFT) and iterative RLHF can obtain state-of-the-art performance with fully
open-source datasets. Further, we have made our models, curated datasets, and
comprehensive step-by-step code guidebooks publicly available. Please refer to
https://github.com/RLHFlow/RLHF-Reward-Modeling and
https://github.com/RLHFlow/Online-RLHF for more detailed information.