Orionnnn/hub-stats huggingface dataset:
LLMs + Persona-Plug = Personalized LLMs
Authors: name: Jiongnan Liu, name: Yutao Zhu, name: Shuting Wang, name: Xiaochi Wei, name: Erxue Min, name: Yu Lu, name: Shuaiqiang Wang, name: Dawei Yin, name: Zhicheng Dou
Published: 2024-09-18 11:54:45

Personalization plays a critical role in numerous language tasks and
applications, since users with the same requirements may prefer diverse outputs
based on their individual interests. This has led to the development of various
personalized approaches aimed at adapting large language models (LLMs) to
generate customized outputs aligned with user preferences. Some of them involve
fine-tuning a unique personalized LLM for each user, which is too expensive for
widespread application. Alternative approaches introduce personalization
information in a plug-and-play manner by retrieving the user's relevant
historical texts as demonstrations. However, this retrieval-based strategy may
break the continuity of the user history and fail to capture the user's overall
styles and patterns, hence leading to sub-optimal performance. To address these
challenges, we propose a novel personalized LLM model, . It constructs a
user-specific embedding for each individual by modeling all her historical
contexts through a lightweight plug-in user embedder module. By attaching this
embedding to the task input, LLMs can better understand and capture user habits
and preferences, thereby producing more personalized outputs without tuning
their own parameters. Extensive experiments on various tasks in the language
model personalization (LaMP) benchmark demonstrate that the proposed model
significantly outperforms existing personalized LLM approaches.
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at
  Any Resolution
Authors: name: Peng Wang, name: Shuai Bai, name: Sinan Tan, name: Shijie Wang, name: Zhihao Fan, name: Jinze Bai, name: Keqin Chen, name: Xuejing Liu, name: Jialin Wang, name: Wenbin Ge, name: Yang Fan, name: Kai Dang, name: Mengfei Du, name: Xuancheng Ren, name: Rui Men, name: Dayiheng Liu, name: Chang Zhou, name: Jingren Zhou, name: Junyang Lin
Published: 2024-09-18 17:59:32

We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL
models that redefines the conventional predetermined-resolution approach in
visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism,
which enables the model to dynamically process images of varying resolutions
into different numbers of visual tokens. This approach allows the model to
generate more efficient and accurate visual representations, closely aligning
with human perceptual processes. The model also integrates Multimodal Rotary
Position Embedding (M-RoPE), facilitating the effective fusion of positional
information across text, images, and videos. We employ a unified paradigm for
processing both images and videos, enhancing the model's visual perception
capabilities. To explore the potential of large multimodal models, Qwen2-VL
investigates the scaling laws for large vision-language models (LVLMs). By
scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the
amount of training data, the Qwen2-VL Series achieves highly competitive
performance. Notably, the Qwen2-VL-72B model achieves results comparable to
leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal
benchmarks, outperforming other generalist models. Code is available at
https://github.com/QwenLM/Qwen2-VL.
Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models
Authors: name: EverestAI, name: Sijin Chen, name: Yuan Feng, name: Laipeng He, name: Tianwei He, name: Wendi He, name: Yanni Hu, name: Bin Lin, name: Yiting Lin, name: Pengfei Tan, name: Chengwei Tian, name: Chen Wang, name: Zhicheng Wang, name: Ruoye Xie, name: Jingjing Yin, name: Jianhao Ye, name: Jixun Yao, name: Quanlei Yan, name: Yuguang Yang
Published: 2024-09-18 17:03:12

With the advent of the big data and large language model era, zero-shot
personalized rapid customization has emerged as a significant trend. In this
report, we introduce Takin AudioLLM, a series of techniques and models, mainly
including Takin TTS, Takin VC, and Takin Morphing, specifically designed for
audiobook production. These models are capable of zero-shot speech production,
generating high-quality speech that is nearly indistinguishable from real human
speech and facilitating individuals to customize the speech content according
to their own needs. Specifically, we first introduce Takin TTS, a neural codec
language model that builds upon an enhanced neural speech codec and a
multi-task training framework, capable of generating high-fidelity natural
speech in a zero-shot way. For Takin VC, we advocate an effective content and
timbre joint modeling approach to improve the speaker similarity, while
advocating for a conditional flow matching based decoder to further enhance its
naturalness and expressiveness. Last, we propose the Takin Morphing system with
highly decoupled and advanced timbre and prosody modeling approaches, which
enables individuals to customize speech production with their preferred timbre
and prosody in a precise and controllable manner. Extensive experiments
validate the effectiveness and robustness of our Takin AudioLLM series models.
For detailed demos, please refer to https://takinaudiollm.github.io.
Human-like Affective Cognition in Foundation Models
Authors: name: Kanishk Gandhi, name: Zoe Lynch, name: Jan-Philipp Fränken, name: Kayla Patterson, name: Sharon Wambu, name: Tobias Gerstenberg, name: Desmond C. Ong, name: Noah D. Goodman
Published: 2024-09-18 06:42:13

Understanding emotions is fundamental to human interaction and experience.
Humans easily infer emotions from situations or facial expressions, situations
from emotions, and do a variety of other affective cognition. How adept
is modern AI at these inferences? We introduce an evaluation framework for
testing affective cognition in foundation models. Starting from psychological
theory, we generate 1,280 diverse scenarios exploring relationships between
appraisals, emotions, expressions, and outcomes. We evaluate the abilities of
foundation models (GPT-4, Claude-3, Gemini-1.5-Pro) and humans (N = 567) across
carefully selected conditions. Our results show foundation models tend to agree
with human intuitions, matching or exceeding interparticipant agreement. In
some conditions, models are ``superhuman'' -- they better predict modal human
judgements than the average human. All models benefit from chain-of-thought
reasoning. This suggests foundation models have acquired a human-like
understanding of emotions and their influence on beliefs and behavior.
Promptriever: Instruction-Trained Retrievers Can Be Prompted Like
  Language Models
Authors: name: Orion Weller, name: Benjamin Van Durme, name: Dawn Lawrie, name: Ashwin Paranjape, name: Yuhao Zhang, name: Jack Hessel
Published: 2024-09-17 12:42:55

Instruction-tuned language models (LM) are able to respond to imperative
commands, providing a more natural user interface compared to their base
counterparts. In this work, we present Promptriever, the first retrieval model
able to be prompted like an LM. To train Promptriever, we curate and release a
new instance-level instruction training set from MS MARCO, spanning nearly 500k
instances. Promptriever not only achieves strong performance on standard
retrieval tasks, but also follows instructions. We observe: (1) large gains
(reaching SoTA) on following detailed relevance instructions (+14.3 p-MRR /
+3.1 nDCG on FollowIR), (2) significantly increased robustness to lexical
choices/phrasing in the query+instruction (+12.9 Robustness@10 on InstructIR),
and (3) the ability to perform hyperparameter search via prompting to reliably
improve retrieval performance (+1.4 average increase on BEIR). Promptriever
demonstrates that retrieval models can be controlled with prompts on a
per-query basis, setting the stage for future work aligning LM prompting
techniques with information retrieval.
Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded
  Attributions and Learning to Refuse
Authors: name: Maojia Song, name: Shang Hong Sim, name: Rishabh Bhardwaj, name: Hai Leong Chieu, name: Navonil Majumder, name: Soujanya Poria
Published: 2024-09-17 14:47:33

LLMs are an integral part of retrieval-augmented generation (RAG) systems.
While many studies focus on evaluating the quality of end-to-end RAG systems,
there is a lack of research on understanding the appropriateness of an LLM for
the RAG task. Thus, we introduce a new metric, Trust-Score, that provides a
holistic evaluation of the trustworthiness of LLMs in an RAG framework. We show
that various prompting methods, such as in-context learning, fail to adapt LLMs
effectively to the RAG task. Thus, we propose Trust-Align, a framework to align
LLMs for higher Trust-Score. LLaMA-3-8b, aligned with our method, significantly
outperforms open-source LLMs of comparable sizes on ASQA (up 10.7), QAMPARI (up
29.2) and ELI5 (up 14.9). We release our code at:
https://github.com/declare-lab/trust-align.
A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language
  Models: An Experimental Analysis up to 405B
Authors: name: Jemin Lee, name: Sihyeong Park, name: Jinse Kwon, name: Jihun Oh, name: Yongin Kwon
Published: 2024-09-17 10:31:37

Prior research works have evaluated quantized LLMs using limited metrics such
as perplexity or a few basic knowledge tasks and old datasets. Additionally,
recent large-scale models such as Llama 3.1 with up to 405B have not been
thoroughly examined. This paper evaluates the performance of instruction-tuned
LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on
models ranging from 7B to 405B. Using 13 benchmarks, we assess performance
across six task types: commonsense Q\&A, knowledge and language understanding,
instruction following, hallucination detection, mathematics, and dialogue. Our
key findings reveal that (1) quantizing a larger LLM to a similar size as a
smaller FP16 LLM generally performs better across most benchmarks, except for
hallucination detection and instruction following; (2) performance varies
significantly with different quantization methods, model size, and bit-width,
with weight-only methods often yielding better results in larger models; (3)
task difficulty does not significantly impact accuracy degradation due to
quantization; and (4) the MT-Bench evaluation method has limited discriminatory
power among recent high-performing LLMs.
Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think
Authors: name: Gonzalo Martin Garcia, name: Karim Abou Zeid, name: Christian Schmidt, name: Daan de Geus, name: Alexander Hermans, name: Bastian Leibe
Published: 2024-09-17 16:58:52

Recent work showed that large diffusion models can be reused as highly
precise monocular depth estimators by casting depth estimation as an
image-conditional image generation task. While the proposed model achieved
state-of-the-art results, high computational demands due to multi-step
inference limited its use in many scenarios. In this paper, we show that the
perceived inefficiency was caused by a flaw in the inference pipeline that has
so far gone unnoticed. The fixed model performs comparably to the best
previously reported configuration while being more than 200times faster. To
optimize for downstream task performance, we perform end-to-end fine-tuning on
top of the single-step model with task-specific losses and get a deterministic
model that outperforms all other diffusion-based depth and normal estimation
models on common zero-shot benchmarks. We surprisingly find that this
fine-tuning protocol also works directly on Stable Diffusion and achieves
comparable performance to current state-of-the-art diffusion-based depth and
normal estimation models, calling into question some of the conclusions drawn
from prior works.
Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks
Authors: name: Ali Mehrabian, name: Parsa Mojarad Adi, name: Moein Heidari, name: Ilker Hacihaliloglu
Published: 2024-09-14 05:53:33

Implicit neural representations (INRs) use neural networks to provide
continuous and resolution-independent representations of complex signals with a
small number of parameters. However, existing INR models often fail to capture
important frequency components specific to each task. To address this issue, in
this paper, we propose a Fourier Kolmogorov Arnold network (FKAN) for INRs. The
proposed FKAN utilizes learnable activation functions modeled as Fourier series
in the first layer to effectively control and learn the task-specific frequency
components. In addition, the activation functions with learnable Fourier
coefficients improve the ability of the network to capture complex patterns and
details, which is beneficial for high-resolution and high-dimensional data.
Experimental results show that our proposed FKAN model outperforms three
state-of-the-art baseline schemes, and improves the peak signal-to-noise ratio
(PSNR) and structural similarity index measure (SSIM) for the image
representation task and intersection over union (IoU) for the 3D occupancy
volume representation task, respectively.
Single-Layer Learnable Activation for Implicit Neural Representation
  (SL^{2}A-INR)
Authors: name: Moein Heidari, name: Reza Rezaeian, name: Reza Azad, name: Dorit Merhof, name: Hamid Soltanian-Zadeh, name: Ilker Hacihaliloglu
Published: 2024-09-17 02:02:15

Implicit Neural Representation (INR), leveraging a neural network to
transform coordinate input into corresponding attributes, has recently driven
significant advances in several vision-related domains. However, the
performance of INR is heavily influenced by the choice of the nonlinear
activation function used in its multilayer perceptron (MLP) architecture.
Multiple nonlinearities have been investigated; yet, current INRs face
limitations in capturing high-frequency components, diverse signal types, and
handling inverse problems. We have identified that these problems can be
greatly alleviated by introducing a paradigm shift in INRs. We find that an
architecture with learnable activations in initial layers can represent fine
details in the underlying signals. Specifically, we propose SL^{2}A-INR, a
hybrid network for INR with a single-layer learnable activation function,
prompting the effectiveness of traditional ReLU-based MLPs. Our method performs
superior across diverse tasks, including image representation, 3D shape
reconstructions, inpainting, single image super-resolution, CT reconstruction,
and novel view synthesis. Through comprehensive experiments, SL^{2}A-INR sets
new benchmarks in accuracy, quality, and convergence rates for INR.
EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion
  Transformer
Authors: name: Jiarui Hai, name: Yong Xu, name: Hao Zhang, name: Chenxing Li, name: Helin Wang, name: Mounya Elhilali, name: Dong Yu
Published: 2024-09-17 01:27:28

Latent diffusion models have shown promising results in text-to-audio (T2A)
generation tasks, yet previous models have encountered difficulties in
generation quality, computational cost, diffusion sampling, and data
preparation. In this paper, we introduce EzAudio, a transformer-based T2A
diffusion model, to handle these challenges. Our approach includes several key
innovations: (1) We build the T2A model on the latent space of a 1D waveform
Variational Autoencoder (VAE), avoiding the complexities of handling 2D
spectrogram representations and using an additional neural vocoder. (2) We
design an optimized diffusion transformer architecture specifically tailored
for audio latent representations and diffusion modeling, which enhances
convergence speed, training stability, and memory usage, making the training
process easier and more efficient. (3) To tackle data scarcity, we adopt a
data-efficient training strategy that leverages unlabeled data for learning
acoustic dependencies, audio caption data annotated by audio-language models
for text-to-audio alignment learning, and human-labeled data for fine-tuning.
(4) We introduce a classifier-free guidance (CFG) rescaling method that
simplifies EzAudio by achieving strong prompt alignment while preserving great
audio quality when using larger CFG scores, eliminating the need to struggle
with finding the optimal CFG score to balance this trade-off. EzAudio surpasses
existing open-source models in both objective metrics and subjective
evaluations, delivering realistic listening experiences while maintaining a
streamlined model structure, low training costs, and an easy-to-follow training
pipeline. Code, data, and pre-trained models are released at:
https://haidog-yaqub.github.io/EzAudio-Page/.
Phidias: A Generative Model for Creating 3D Content from Text, Image,
  and 3D Conditions with Reference-Augmented Diffusion
Authors: name: Zhenwei Wang, name: Tengfei Wang, name: Zexin He, name: Gerhard Hancke, name: Ziwei Liu, name: Rynson W. H. Lau
Published: 2024-09-17 17:59:33

In 3D modeling, designers often use an existing 3D model as a reference to
create new ones. This practice has inspired the development of Phidias, a novel
generative model that uses diffusion for reference-augmented 3D generation.
Given an image, our method leverages a retrieved or user-provided 3D reference
model to guide the generation process, thereby enhancing the generation
quality, generalization ability, and controllability. Our model integrates
three key components: 1) meta-ControlNet that dynamically modulates the
conditioning strength, 2) dynamic reference routing that mitigates misalignment
between the input image and 3D reference, and 3) self-reference augmentations
that enable self-supervised training with a progressive curriculum.
Collectively, these designs result in a clear improvement over existing
methods. Phidias establishes a unified framework for 3D generation using text,
image, and 3D conditions with versatile applications.
PDMX: A Large-Scale Public Domain MusicXML Dataset for Symbolic Music
  Processing
Authors: name: Phillip Long, name: Zachary Novack, name: Taylor Berg-Kirkpatrick, name: Julian McAuley
Published: 2024-09-17 01:48:42

The recent explosion of generative AI-Music systems has raised numerous
concerns over data copyright, licensing music from musicians, and the conflict
between open-source AI and large prestige companies. Such issues highlight the
need for publicly available, copyright-free musical data, in which there is a
large shortage, particularly for symbolic music data. To alleviate this issue,
we present PDMX: a large-scale open-source dataset of over 250K public domain
MusicXML scores collected from the score-sharing forum MuseScore, making it the
largest available copyright-free symbolic music dataset to our knowledge. PDMX
additionally includes a wealth of both tag and user interaction metadata,
allowing us to efficiently analyze the dataset and filter for high quality
user-generated scores. Given the additional metadata afforded by our data
collection process, we conduct multitrack music generation experiments
evaluating how different representative subsets of PDMX lead to different
behaviors in downstream models, and how user-rating statistics can be used as
an effective measure of data quality. Examples can be found at
https://pnlong.github.io/PDMX.demo/.
NVLM: Open Frontier-Class Multimodal LLMs
Authors: name: Wenliang Dai, name: Nayeon Lee, name: Boxin Wang, name: Zhuoling Yang, name: Zihan Liu, name: Jon Barker, name: Tuomas Rintamaki, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Wei Ping
Published: 2024-09-17 17:59:06

We introduce NVLM 1.0, a family of frontier-class multimodal large language
models (LLMs) that achieve state-of-the-art results on vision-language tasks,
rivaling the leading proprietary models (e.g., GPT-4o) and open-access models
(e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved
text-only performance over its LLM backbone after multimodal training. In terms
of model design, we perform a comprehensive comparison between decoder-only
multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g.,
Flamingo). Based on the strengths and weaknesses of both approaches, we propose
a novel architecture that enhances both training efficiency and multimodal
reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for
tile-based dynamic high-resolution images, which significantly boosts
performance on multimodal reasoning and OCR-related tasks. Regarding training
data, we meticulously curate and provide detailed information on our multimodal
pretraining and supervised fine-tuning datasets. Our findings indicate that
dataset quality and task diversity are more important than scale, even during
the pretraining phase, across all architectures. Notably, we develop
production-grade multimodality for the NVLM-1.0 models, enabling them to excel
in vision-language tasks while maintaining and even improving text-only
performance compared to their LLM backbones. To achieve this, we craft and
integrate a high-quality text-only dataset into multimodal training, alongside
a substantial amount of multimodal math and reasoning data, leading to enhanced
math and coding capabilities across modalities. To advance research in the
field, we are releasing the model weights and will open-source the code for the
community: https://nvlm-project.github.io/.
OmniGen: Unified Image Generation
Authors: name: Shitao Xiao, name: Yueze Wang, name: Junjie Zhou, name: Huaying Yuan, name: Xingrun Xing, name: Ruiran Yan, name: Shuting Wang, name: Tiejun Huang, name: Zheng Liu
Published: 2024-09-17 16:42:46

In this work, we introduce OmniGen, a new diffusion model for unified image
generation. Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen
no longer requires additional modules such as ControlNet or IP-Adapter to
process diverse control conditions. OmniGenis characterized by the following
features: 1) Unification: OmniGen not only demonstrates text-to-image
generation capabilities but also inherently supports other downstream tasks,
such as image editing, subject-driven generation, and visual-conditional
generation. Additionally, OmniGen can handle classical computer vision tasks by
transforming them into image generation tasks, such as edge detection and human
pose recognition. 2) Simplicity: The architecture of OmniGen is highly
simplified, eliminating the need for additional text encoders. Moreover, it is
more user-friendly compared to existing diffusion models, enabling complex
tasks to be accomplished through instructions without the need for extra
preprocessing steps (e.g., human pose estimation), thereby significantly
simplifying the workflow of image generation. 3) Knowledge Transfer: Through
learning in a unified format, OmniGen effectively transfers knowledge across
different tasks, manages unseen tasks and domains, and exhibits novel
capabilities. We also explore the model's reasoning capabilities and potential
applications of chain-of-thought mechanism. This work represents the first
attempt at a general-purpose image generation model, and there remain several
unresolved issues. We will open-source the related resources at
https://github.com/VectorSpaceLab/OmniGen to foster advancements in this field.