Orionnnn/hub-stats huggingface dataset:
RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting
  for Text-to-Speech Synthesis
Authors: name: Detai Xin, name: Xu Tan, name: Kai Shen, name: Zeqian Ju, name: Dongchao Yang, name: Yuancheng Wang, name: Shinnosuke Takamichi, name: Hiroshi Saruwatari, name: Shujie Liu, name: Jinyu Li, name: Sheng Zhao
Published: 2024-04-04 05:15:07

We present RALL-E, a robust language modeling method for text-to-speech (TTS)
synthesis. While previous work based on large language models (LLMs) shows
impressive performance on zero-shot TTS, such methods often suffer from poor
robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
high word error rate (WER), due to the autoregressive prediction style of
language models. The core idea behind RALL-E is chain-of-thought (CoT)
prompting, which decomposes the task into simpler steps to enhance the
robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
prosody features (pitch and duration) of the input text and uses them as
intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E
utilizes the predicted duration prompt to guide the computing of self-attention
weights in Transformer to enforce the model to focus on the corresponding
phonemes and prosody features when predicting speech tokens. Results of
comprehensive objective and subjective evaluations demonstrate that, compared
to a powerful baseline method VALL-E, RALL-E significantly improves the WER of
zero-shot TTS from 6.3% (without reranking) and 2.1% (with reranking) to
2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E
correctly synthesizes sentences that are hard for VALL-E and reduces the error
rate from 68% to 4%.
MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with
  Interleaved Visual-Textual Tokens
Authors: name: Kirolos Ataallah, name: Xiaoqian Shen, name: Eslam Abdelrahman, name: Essam Sleiman, name: Deyao Zhu, name: Jian Ding, name: Mohamed Elhoseiny
Published: 2024-04-04 12:46:01

This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
designed specifically for video understanding. The model is capable of
processing both temporal visual and textual data, making it adept at
understanding the complexities of videos. Building upon the success of
MiniGPT-v2, which excelled in translating visual features into the LLM space
for single images and achieved impressive results on various image-text
benchmarks, this paper extends the model's capabilities to process a sequence
of frames, enabling it to comprehend videos. MiniGPT4-video does not only
consider visual content but also incorporates textual conversations, allowing
the model to effectively answer queries involving both visual and text
components. The proposed model outperforms existing state-of-the-art methods,
registering gains of 4.22%, 1.13%, 20.82%, and 13.1% on the MSVD, MSRVTT, TGIF,
and TVQA benchmarks respectively. Our models and code have been made publicly
available here https://vision-cair.github.io/MiniGPT4-video/
LVLM-Intrepret: An Interpretability Tool for Large Vision-Language
  Models
Authors: name: Gabriela Ben Melech Stan, name: Raanan Yehezkel Rohekar, name: Yaniv Gurwicz, name: Matthew Lyle Olson, name: Anahita Bhiwandiwalla, name: Estelle Aflalo, name: Chenfei Wu, name: Nan Duan, name: Shao-Yen Tseng, name: Vasudev Lal
Published: 2024-04-03 23:57:34

In the rapidly evolving landscape of artificial intelligence, multi-modal
large language models are emerging as a significant area of interest. These
models, which combine various forms of data input, are becoming increasingly
popular. However, understanding their internal mechanisms remains a complex
task. Numerous advancements have been made in the field of explainability tools
and mechanisms, yet there is still much to explore. In this work, we present a
novel interactive application aimed towards understanding the internal
mechanisms of large vision-language models. Our interface is designed to
enhance the interpretability of the image patches, which are instrumental in
generating an answer, and assess the efficacy of the language model in
grounding its output in the image. With our application, a user can
systematically investigate the model and uncover system limitations, paving the
way for enhancements in system capabilities. Finally, we present a case study
of how our application can aid in understanding failure mechanisms in a popular
large multi-modal model: LLaVA.
PointInfinity: Resolution-Invariant Point Diffusion Models
Authors: name: Zixuan Huang, name: Justin Johnson, name: Shoubhik Debnath, name: James M. Rehg, name: Chao-Yuan Wu
Published: 2024-04-04 16:24:32

We present PointInfinity, an efficient family of point cloud diffusion
models. Our core idea is to use a transformer-based architecture with a
fixed-size, resolution-invariant latent representation. This enables efficient
training with low-resolution point clouds, while allowing high-resolution point
clouds to be generated during inference. More importantly, we show that scaling
the test-time resolution beyond the training resolution improves the fidelity
of generated point clouds and surfaces. We analyze this phenomenon and draw a
link to classifier-free guidance commonly used in diffusion models,
demonstrating that both allow trading off fidelity and variability during
inference. Experiments on CO3D show that PointInfinity can efficiently generate
high-resolution point clouds (up to 131k points, 31 times more than Point-E)
with state-of-the-art quality.
ReFT: Representation Finetuning for Language Models
Authors: name: Zhengxuan Wu, name: Aryaman Arora, name: Zheng Wang, name: Atticus Geiger, name: Dan Jurafsky, name: Christopher D. Manning, name: Christopher Potts
Published: 2024-04-04 17:00:37

Parameter-efficient fine-tuning (PEFT) methods seek to adapt large models via
updates to a small number of weights. However, much prior interpretability work
has shown that representations encode rich semantic information, suggesting
that editing representations might be a more powerful alternative. Here, we
pursue this hypothesis by developing a family of Representation
Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and
learn task-specific interventions on hidden representations. We define a strong
instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT). LoReFT is
a drop-in replacement for existing PEFTs and learns interventions that are
10x-50x more parameter-efficient than prior state-of-the-art PEFTs. We showcase
LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks,
Alpaca-Eval v1.0, and GLUE. In all these evaluations, LoReFT delivers the best
balance of efficiency and performance, and almost always outperforms
state-of-the-art PEFTs. We release a generic ReFT training library publicly at
https://github.com/stanfordnlp/pyreft.
Training LLMs over Neurally Compressed Text
Authors: name: Brian Lester, name: Jaehoon Lee, name: Alex Alemi, name: Jeffrey Pennington, name: Adam Roberts, name: Jascha Sohl-Dickstein, name: Noah Constant
Published: 2024-04-04 17:48:28

In this paper, we explore the idea of training large language models (LLMs)
over highly compressed text. While standard subword tokenizers compress text by
a small factor, neural text compressors can achieve much higher rates of
compression. If it were possible to train LLMs directly over neurally
compressed text, this would confer advantages in training and serving
efficiency, as well as easier handling of long text spans. The main obstacle to
this goal is that strong compression tends to produce opaque outputs that are
not well-suited for learning. In particular, we find that text na\"ively
compressed via Arithmetic Coding is not readily learnable by LLMs. To overcome
this, we propose Equal-Info Windows, a novel compression technique whereby text
is segmented into blocks that each compress to the same bit length. Using this
method, we demonstrate effective learning over neurally compressed text that
improves with scale, and outperforms byte-level baselines by a wide margin on
perplexity and inference speed benchmarks. While our method delivers worse
perplexity than subword tokenizers for models trained with the same parameter
count, it has the benefit of shorter sequence lengths. Shorter sequence lengths
require fewer autoregressive generation steps, and reduce latency. Finally, we
provide extensive analysis of the properties that contribute to learnability,
and offer concrete suggestions for how to further improve the performance of
high-compression tokenizers.
Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak
  Attacks?
Authors: name: Shuo Chen, name: Zhen Han, name: Bailan He, name: Zifeng Ding, name: Wenqian Yu, name: Philip Torr, name: Volker Tresp, name: Jindong Gu
Published: 2024-04-04 12:38:14

Various jailbreak attacks have been proposed to red-team Large Language
Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some
methods are not limited to the textual modality and extend the jailbreak attack
to Multimodal Large Language Models (MLLMs) by perturbing the visual input.
However, the absence of a universal evaluation benchmark complicates the
performance reproduction and fair comparison. Besides, there is a lack of
comprehensive evaluation of closed-source state-of-the-art (SOTA) models,
especially MLLMs, such as GPT-4V. To address these issues, this work first
builds a comprehensive jailbreak evaluation dataset with 1445 harmful questions
covering 11 different safety policies. Based on this dataset, extensive
red-teaming experiments are conducted on 11 different LLMs and MLLMs, including
both SOTA proprietary models and open-source models. We then conduct a deep
analysis of the evaluated results and find that (1) GPT4 and GPT-4V demonstrate
better robustness against jailbreak attacks compared to open-source LLMs and
MLLMs. (2) Llama2 and Qwen-VL-Chat are more robust compared to other
open-source models. (3) The transferability of visual jailbreak methods is
relatively limited compared to textual jailbreak methods. The dataset and code
can be found here
https://anonymous.4open.science/r/red_teaming_gpt4-C1CE/README.md .
CodeEditorBench: Evaluating Code Editing Capability of Large Language
  Models
Authors: name: Jiawei Guo, name: Ziming Li, name: Xueling Liu, name: Kaijing Ma, name: Tianyu Zheng, name: Zhouliang Yu, name: Ding Pan, name: Yizhi LI, name: Ruibo Liu, name: Yue Wang, name: Shuyue Guo, name: Xingwei Qu, name: Xiang Yue, name: Ge Zhang, name: Wenhu Chen, name: Jie Fu
Published: 2024-04-04 15:49:49

Large Language Models (LLMs) for code are rapidly evolving, with code editing
emerging as a critical capability. We introduce CodeEditorBench, an evaluation
framework designed to rigorously assess the performance of LLMs in code editing
tasks, including debugging, translating, polishing, and requirement switching.
Unlike existing benchmarks focusing solely on code generation, CodeEditorBench
emphasizes real-world scenarios and practical aspects of software development.
We curate diverse coding challenges and scenarios from five sources, covering
various programming languages, complexity levels, and editing tasks. Evaluation
of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and
GPT-4), outperform open-source models in CodeEditorBench, highlighting
differences in model performance based on problem types and prompt
sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by
providing a robust platform for assessing code editing capabilities. We will
release all prompts and datasets to enable the community to expand the dataset
and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to
the advancement of LLMs in code editing and provide a valuable resource for
researchers and practitioners.
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept
  Matching
Authors: name: Dongzhi Jiang, name: Guanglu Song, name: Xiaoshi Wu, name: Renrui Zhang, name: Dazhong Shen, name: Zhuofan Zong, name: Yu Liu, name: Hongsheng Li
Published: 2024-04-04 17:59:46

Diffusion models have demonstrated great success in the field of
text-to-image generation. However, alleviating the misalignment between the
text prompts and images is still challenging. The root reason behind the
misalignment has not been extensively investigated. We observe that the
misalignment is caused by inadequate token attention activation. We further
attribute this phenomenon to the diffusion model's insufficient condition
utilization, which is caused by its training paradigm. To address the issue, we
propose CoMat, an end-to-end diffusion model fine-tuning strategy with an
image-to-text concept matching mechanism. We leverage an image captioning model
to measure image-to-text alignment and guide the diffusion model to revisit
ignored tokens. A novel attribute concentration module is also proposed to
address the attribute binding problem. Without any image or human preference
data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL.
Extensive experiments show that CoMat-SDXL significantly outperforms the
baseline model SDXL in two text-to-image alignment benchmarks and achieves
start-of-the-art performance.
Freditor: High-Fidelity and Transferable NeRF Editing by Frequency
  Decomposition
Authors: name: Yisheng He, name: Weihao Yuan, name: Siyu Zhu, name: Zilong Dong, name: Liefeng Bo, name: Qixing Huang
Published: 2024-04-03 07:07:02

This paper enables high-fidelity, transferable NeRF editing by frequency
decomposition. Recent NeRF editing pipelines lift 2D stylization results to 3D
scenes while suffering from blurry results, and fail to capture detailed
structures caused by the inconsistency between 2D editings. Our critical
insight is that low-frequency components of images are more
multiview-consistent after editing compared with their high-frequency parts.
Moreover, the appearance style is mainly exhibited on the low-frequency
components, and the content details especially reside in high-frequency parts.
This motivates us to perform editing on low-frequency components, which results
in high-fidelity edited scenes. In addition, the editing is performed in the
low-frequency feature space, enabling stable intensity control and novel scene
transfer. Comprehensive experiments conducted on photorealistic datasets
demonstrate the superior performance of high-fidelity and transferable NeRF
editing. The project page is at https://aigc3d.github.io/freditor.
Language Models as Compilers: Simulating Pseudocode Execution Improves
  Algorithmic Reasoning in Language Models
Authors: name: Hyungjoo Chae, name: Yeonghyeon Kim, name: Seungone Kim, name: Kai Tzu-iunn Ong, name: Beong-woo Kwak, name: Moohyeon Kim, name: Seonghwan Kim, name: Taeyoon Kwon, name: Jiwan Chung, name: Youngjae Yu, name: Jinyoung Yeo
Published: 2024-04-03 08:49:11

Algorithmic reasoning refers to the ability to understand the complex
patterns behind the problem and decompose them into a sequence of reasoning
steps towards the solution. Such nature of algorithmic reasoning makes it a
challenge for large language models (LLMs), even though they have demonstrated
promising performance in other reasoning tasks. Within this context, some
recent studies use programming languages (e.g., Python) to express the
necessary logic for solving a given instance/question (e.g.,
Program-of-Thought) as inspired by their strict and precise syntaxes. However,
it is non-trivial to write an executable code that expresses the correct logic
on the fly within a single inference call. Also, the code generated
specifically for an instance cannot be reused for others, even if they are from
the same task and might require identical logic to solve. This paper presents
Think-and-Execute, a novel framework that decomposes the reasoning process of
language models into two steps. (1) In Think, we discover a task-level logic
that is shared across all instances for solving a given task and then express
the logic with pseudocode; (2) In Execute, we further tailor the generated
pseudocode to each instance and simulate the execution of the code. With
extensive experiments on seven algorithmic reasoning tasks, we demonstrate the
effectiveness of Think-and-Execute. Our approach better improves LMs' reasoning
compared to several strong baselines performing instance-specific reasoning
(e.g., CoT and PoT), suggesting the helpfulness of discovering task-level
logic. Also, we show that compared to natural language, pseudocode can better
guide the reasoning of LMs, even though they are trained to follow natural
language instructions.
Mixture-of-Depths: Dynamically allocating compute in transformer-based
  language models
Authors: name: David Raposo, name: Sam Ritter, name: Blake Richards, name: Timothy Lillicrap, name: Peter Conway Humphreys, name: Adam Santoro
Published: 2024-04-02 19:28:11

Transformer-based language models spread FLOPs uniformly across input
sequences. In this work we demonstrate that transformers can instead learn to
dynamically allocate FLOPs (or compute) to specific positions in a sequence,
optimising the allocation along the sequence for different layers across the
model depth. Our method enforces a total compute budget by capping the number
of tokens (k) that can participate in the self-attention and MLP computations
at a given layer. The tokens to be processed are determined by the network
using a top-k routing mechanism. Since k is defined a priori, this simple
procedure uses a static computation graph with known tensor sizes, unlike other
conditional computation techniques. Nevertheless, since the identities of the
k tokens are fluid, this method can expend FLOPs non-uniformly across the
time and model depth dimensions. Thus, compute expenditure is entirely
predictable in sum total, but dynamic and context-sensitive at the token-level.
Not only do models trained in this way learn to dynamically allocate compute,
they do so efficiently. These models match baseline performance for equivalent
FLOPS and wall-clock times to train, but require a fraction of the FLOPs per
forward pass, and can be upwards of 50\% faster to step during post-training
sampling.
ChatGLM-Math: Improving Math Problem-Solving in Large Language Models
  with a Self-Critique Pipeline
Authors: name: Yifan Xu, name: Xiao Liu, name: Xinghan Liu, name: Zhenyu Hou, name: Yueyan Li, name: Xiaohan Zhang, name: Zihan Wang, name: Aohan Zeng, name: Zhengxiao Du, name: Wenyi Zhao, name: Jie Tang, name: Yuxiao Dong
Published: 2024-04-03 17:51:18

Large language models (LLMs) have shown excellent mastering of human
language, but still struggle in real-world applications that require
mathematical problem-solving. While many strategies and datasets to enhance
LLMs' mathematics are developed, it remains a challenge to simultaneously
maintain and improve both language and mathematical capabilities in deployed
LLM systems.In this work, we tailor the Self-Critique pipeline, which addresses
the challenge in the feedback learning stage of LLM alignment. We first train a
general Math-Critique model from the LLM itself to provide feedback signals.
Then, we sequentially employ rejective fine-tuning and direct preference
optimization over the LLM's own generations for data collection. Based on
ChatGLM3-32B, we conduct a series of experiments on both academic and our newly
created challenging dataset, MathUserEval. Results show that our pipeline
significantly enhances the LLM's mathematical problem-solving while still
improving its language ability, outperforming LLMs that could be two times
larger. Related techniques have been deployed to
ChatGLM\url{https://chatglm.cn}, an online serving LLM. Related
evaluation dataset and scripts are released at
https://github.com/THUDM/ChatGLM-Math.
Cross-Attention Makes Inference Cumbersome in Text-to-Image Diffusion
  Models
Authors: name: Wentian Zhang, name: Haozhe Liu, name: Jinheng Xie, name: Francesco Faccio, name: Mike Zheng Shou, name: Jürgen Schmidhuber
Published: 2024-04-03 13:44:41

This study explores the role of cross-attention during inference in
text-conditional diffusion models. We find that cross-attention outputs
converge to a fixed point after few inference steps. Accordingly, the time
point of convergence naturally divides the entire inference process into two
stages: an initial semantics-planning stage, during which, the model relies on
cross-attention to plan text-oriented visual semantics, and a subsequent
fidelity-improving stage, during which the model tries to generate images from
previously planned semantics. Surprisingly, ignoring text conditions in the
fidelity-improving stage not only reduces computation complexity, but also
maintains model performance. This yields a simple and training-free method
called TGATE for efficient generation, which caches the cross-attention output
once it converges and keeps it fixed during the remaining inference steps. Our
empirical study on the MS-COCO validation set confirms its effectiveness. The
source code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale
  Prediction
Authors: name: Keyu Tian, name: Yi Jiang, name: Zehuan Yuan, name: Bingyue Peng, name: Liwei Wang
Published: 2024-04-03 17:59:53

We present Visual AutoRegressive modeling (VAR), a new generation paradigm
that redefines the autoregressive learning on images as coarse-to-fine
"next-scale prediction" or "next-resolution prediction", diverging from the
standard raster-scan "next-token prediction". This simple, intuitive
methodology allows autoregressive (AR) transformers to learn visual
distributions fast and generalize well: VAR, for the first time, makes AR
models surpass diffusion transformers in image generation. On ImageNet 256x256
benchmark, VAR significantly improve AR baseline by improving Frechet inception
distance (FID) from 18.65 to 1.80, inception score (IS) from 80.4 to 356.4,
with around 20x faster inference speed. It is also empirically verified that
VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions
including image quality, inference speed, data efficiency, and scalability.
Scaling up VAR models exhibits clear power-law scaling laws similar to those
observed in LLMs, with linear correlation coefficients near -0.998 as solid
evidence. VAR further showcases zero-shot generalization ability in downstream
tasks including image in-painting, out-painting, and editing. These results
suggest VAR has initially emulated the two important properties of LLMs:
Scaling Laws and zero-shot task generalization. We have released all models and
codes to promote the exploration of AR/VAR models for visual generation and
unified learning.