Orionnnn/hub-stats huggingface dataset:
Parrot: Efficient Serving of LLM-based Applications with Semantic
  Variable
Authors: name: Chaofan Lin, name: Zhenhua Han, name: Chengruidong Zhang, name: Yuqing Yang, name: Fan Yang, name: Chen Chen, name: Lili Qiu
Published: 2024-05-30 09:46:36

The rise of large language models (LLMs) has enabled LLM-based applications
(a.k.a. AI agents or co-pilots), a new software paradigm that combines the
strength of LLM and conventional software. Diverse LLM applications from
different tenants could design complex workflows using multiple LLM requests to
accomplish one task. However, they have to use the over-simplified
request-level API provided by today's public LLM services, losing essential
application-level information. Public LLM services have to blindly optimize
individual LLM requests, leading to sub-optimal end-to-end performance of LLM
applications.
  This paper introduces Parrot, an LLM service system that focuses on the
end-to-end experience of LLM-based applications. Parrot proposes Semantic
Variable, a unified abstraction to expose application-level knowledge to public
LLM services. A Semantic Variable annotates an input/output variable in the
prompt of a request, and creates the data pipeline when connecting multiple LLM
requests, providing a natural way to program LLM applications. Exposing
Semantic Variables to the public LLM service allows it to perform conventional
data flow analysis to uncover the correlation across multiple LLM requests.
This correlation opens a brand-new optimization space for the end-to-end
performance of LLM-based applications. Extensive evaluations demonstrate that
Parrot can achieve up to an order-of-magnitude improvement for popular and
practical use cases of LLM applications.
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
  Real-World Code Repositories
Authors: name: Jia Li, name: Ge Li, name: Yunfei Zhao, name: Yongmin Li, name: Huanyu Liu, name: Hao Zhu, name: Lecheng Wang, name: Kaibo Liu, name: Zheng Fang, name: Lanshen Wang, name: Jiazheng Ding, name: Xuanming Zhang, name: Yuqi Zhu, name: Yihong Dong, name: Zhi Jin, name: Binhua Li, name: Fei Huang, name: Yongbin Li
Published: 2024-05-30 09:03:42

How to evaluate the coding abilities of Large Language Models (LLMs) remains
an open question. We find that existing benchmarks are poorly aligned with
real-world code repositories and are insufficient to evaluate the coding
abilities of LLMs.
  To address the knowledge gap, we propose a new benchmark named DevEval, which
has three advances. (1) DevEval aligns with real-world repositories in multiple
dimensions, e.g., code distributions and dependency distributions. (2) DevEval
is annotated by 13 developers and contains comprehensive annotations (e.g.,
requirements, original repositories, reference code, and reference
dependencies). (3) DevEval comprises 1,874 testing samples from 117
repositories, covering 10 popular domains (e.g., Internet, Database). Based on
DevEval, we propose repository-level code generation and evaluate 8 popular
LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
Our experiments reveal these LLMs' coding abilities in real-world code
repositories. For example, in our experiments, the highest Pass@1 of
gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
their shortcomings. We hope DevEval can facilitate the development of LLMs in
real code repositories. DevEval, prompts, and LLMs' predictions have been
released.
DeMamba: AI-Generated Video Detection on Million-Scale GenVideo
  Benchmark
Authors: name: Haoxing Chen, name: Yan Hong, name: Zizheng Huang, name: Zhuoer Xu, name: Zhangxuan Gu, name: Yaohui Li, name: Jun Lan, name: Huijia Zhu, name: Jianfu Zhang, name: Weiqiang Wang, name: Huaxiong Li
Published: 2024-05-30 05:36:12

Recently, video generation techniques have advanced rapidly. Given the
popularity of video content on social media platforms, these models intensify
concerns about the spread of fake information. Therefore, there is a growing
demand for detectors capable of distinguishing between fake AI-generated videos
and mitigating the potential harm caused by fake information. However, the lack
of large-scale datasets from the most advanced video generators poses a barrier
to the development of such detectors. To address this gap, we introduce the
first AI-generated video detection dataset, GenVideo. It features the following
characteristics: (1) a large volume of videos, including over one million
AI-generated and real videos collected; (2) a rich diversity of generated
content and methodologies, covering a broad spectrum of video categories and
generation techniques. We conducted extensive studies of the dataset and
proposed two evaluation methods tailored for real-world-like scenarios to
assess the detectors' performance: the cross-generator video classification
task assesses the generalizability of trained detectors on generators; the
degraded video classification task evaluates the robustness of detectors to
handle videos that have degraded in quality during dissemination. Moreover, we
introduced a plug-and-play module, named Detail Mamba (DeMamba), designed to
enhance the detectors by identifying AI-generated videos through the analysis
of inconsistencies in temporal and spatial dimensions. Our extensive
experiments demonstrate DeMamba's superior generalizability and robustness on
GenVideo compared to existing detectors. We believe that the GenVideo dataset
and the DeMamba module will significantly advance the field of AI-generated
video detection. Our code and dataset will be aviliable at
https://github.com/chenhaoxing/DeMamba.
Xwin-LM: Strong and Scalable Alignment Practice for LLMs
Authors: name: Bolin Ni, name: JingCheng Hu, name: Yixuan Wei, name: Houwen Peng, name: Zheng Zhang, name: Gaofeng Meng, name: Han Hu
Published: 2024-05-30 17:59:31

In this work, we present Xwin-LM, a comprehensive suite of alignment
methodologies for large language models (LLMs). This suite encompasses several
key techniques, including supervised finetuning (SFT), reward modeling (RM),
rejection sampling finetuning (RS), and direct preference optimization (DPO).
The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned
with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn
preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward
models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B
parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt
is linked to 64 unique responses generated by Xwin-LM-SFT and scored by
Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses
from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the
DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate
consistent and significant improvements across the pipeline, demonstrating the
strength and scalability of Xwin-LM. The repository
https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster
community research.
PLA4D: Pixel-Level Alignments for Text-to-4D Gaussian Splatting
Authors: name: Qiaowei Miao, name: Yawei Luo, name: Yi Yang
Published: 2024-05-30 11:23:01

As text-conditioned diffusion models (DMs) achieve breakthroughs in image,
video, and 3D generation, the research community's focus has shifted to the
more challenging task of text-to-4D synthesis, which introduces a temporal
dimension to generate dynamic 3D objects. In this context, we identify Score
Distillation Sampling (SDS), a widely used technique for text-to-3D synthesis,
as a significant hindrance to text-to-4D performance due to its Janus-faced and
texture-unrealistic problems coupled with high computational costs. In this
paper, we propose Pixel-Level Alignments for
Text-to-4D Gaussian Splatting (PLA4D), a novel method that
utilizes text-to-video frames as explicit pixel alignment targets to generate
static 3D objects and inject motion into them. Specifically, we introduce Focal
Alignment to calibrate camera poses for rendering and GS-Mesh Contrastive
Learning to distill geometry priors from rendered image contrasts at the pixel
level. Additionally, we develop Motion Alignment using a deformation network to
drive changes in Gaussians and implement Reference Refinement for smooth 4D
object surfaces. These techniques enable 4D Gaussian Splatting to align
geometry, texture, and motion with generated videos at the pixel level.
Compared to previous methods, PLA4D produces synthesized outputs with better
texture details in less time and effectively mitigates the Janus-faced problem.
PLA4D is fully implemented using open-source models, offering an accessible,
user-friendly, and promising direction for 4D digital content creation. Our
project page:
https://github.com/MiaoQiaowei/PLA4D.github.io{https://github.com/MiaoQiaowei/PLA4D.github.io}.
Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Authors: name: Andreas Koukounas, name: Georgios Mastrapas, name: Michael Günther, name: Bo Wang, name: Scott Martens, name: Isabelle Mohr, name: Saba Sturua, name: Mohammad Kalim Akram, name: Joan Fontanals Martínez, name: Saahil Ognawala, name: Susana Guzman, name: Maximilian Werk, name: Nan Wang, name: Han Xiao
Published: 2024-05-30 16:07:54

Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.
Similarity is Not All You Need: Endowing Retrieval Augmented Generation
  with Multi Layered Thoughts
Authors: name: Chunjing Gan, name: Dan Yang, name: Binbin Hu, name: Hanxiao Zhang, name: Siyuan Li, name: Ziqi Liu, name: Yue Shen, name: Lin Ju, name: Zhiqiang Zhang, name: Jinjie Gu, name: Lei Liang, name: Jun Zhou
Published: 2024-05-30 09:50:38

In recent years, large language models (LLMs) have made remarkable
achievements in various domains. However, the untimeliness and cost of
knowledge updates coupled with hallucination issues of LLMs have curtailed
their applications in knowledge intensive tasks, where retrieval augmented
generation (RAG) can be of help. Nevertheless, existing retrieval augmented
models typically use similarity as a bridge between queries and documents and
follow a retrieve then read procedure. In this work, we argue that similarity
is not always the panacea and totally relying on similarity would sometimes
degrade the performance of retrieval augmented generation. To this end, we
propose MetRag, a Multi layEred Thoughts enhanced Retrieval Augmented
Generation framework. To begin with, beyond existing similarity oriented
thought, we embrace a small scale utility model that draws supervision from an
LLM for utility oriented thought and further come up with a smarter model by
comprehensively combining the similarity and utility oriented thoughts.
Furthermore, given the fact that the retrieved document set tends to be huge
and using them in isolation makes it difficult to capture the commonalities and
characteristics among them, we propose to make an LLM as a task adaptive
summarizer to endow retrieval augmented generation with compactness-oriented
thought. Finally, with multi layered thoughts from the precedent stages, an LLM
is called for knowledge augmented generation. Extensive experiments on
knowledge-intensive tasks have demonstrated the superiority of MetRag.
DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music
  Generation
Authors: name: Zachary Novack, name: Julian McAuley, name: Taylor Berg-Kirkpatrick, name: Nicholas Bryan
Published: 2024-05-30 17:40:11

Controllable music generation methods are critical for human-centered
AI-based music creation, but are currently limited by speed, quality, and
control design trade-offs. Diffusion Inference-Time T-optimization (DITTO), in
particular, offers state-of-the-art results, but is over 10x slower than
real-time, limiting practical use. We propose Distilled Diffusion
Inference-Time T -Optimization (or DITTO-2), a new method to speed up
inference-time optimization-based control and unlock faster-than-real-time
generation for a wide-variety of applications such as music inpainting,
outpainting, intensity, melody, and musical structure control. Our method works
by (1) distilling a pre-trained diffusion model for fast sampling via an
efficient, modified consistency or consistency trajectory distillation process
(2) performing inference-time optimization using our distilled model with
one-step sampling as an efficient surrogate optimization task and (3) running a
final multi-step sampling generation (decoding) using our estimated noise
latents for best-quality, fast, controllable generation. Through thorough
evaluation, we find our method not only speeds up generation over 10-20x, but
simultaneously improves control adherence and generation quality all at once.
Furthermore, we apply our approach to a new application of maximizing text
adherence (CLAP score) and show we can convert an unconditional diffusion model
without text inputs into a model that yields state-of-the-art text control.
Sound examples can be found at https://ditto-music.github.io/ditto2/.
GECO: Generative Image-to-3D within a SECOnd
Authors: name: Chen Wang, name: Jiatao Gu, name: Xiaoxiao Long, name: Yuan Liu, name: Lingjie Liu
Published: 2024-05-30 17:58:00

3D generation has seen remarkable progress in recent years. Existing
techniques, such as score distillation methods, produce notable results but
require extensive per-scene optimization, impacting time efficiency.
Alternatively, reconstruction-based approaches prioritize efficiency but
compromise quality due to their limited handling of uncertainty. We introduce
GECO, a novel method for high-quality 3D generative modeling that operates
within a second. Our approach addresses the prevalent issues of uncertainty and
inefficiency in current methods through a two-stage approach. In the initial
stage, we train a single-step multi-view generative model with score
distillation. Then, a second-stage distillation is applied to address the
challenge of view inconsistency from the multi-view prediction. This two-stage
process ensures a balanced approach to 3D generation, optimizing both quality
and efficiency. Our comprehensive experiments demonstrate that GECO achieves
high-quality image-to-3D generation with an unprecedented level of efficiency.
MOFA-Video: Controllable Image Animation via Generative Motion Field
  Adaptions in Frozen Image-to-Video Diffusion Model
Authors: name: Muyao Niu, name: Xiaodong Cun, name: Xintao Wang, name: Yong Zhang, name: Ying Shan, name: Yinqiang Zheng
Published: 2024-05-30 16:22:22

We present MOFA-Video, an advanced controllable image animation method that
generates video from the given image using various additional controllable
signals (such as human landmarks reference, manual trajectories, and another
even provided video) or their combinations. This is different from previous
methods which only can work on a specific motion domain or show weak control
abilities with diffusion prior. To achieve our goal, we design several
domain-aware motion field adapters (\ie, MOFA-Adapters) to control the
generated motions in the video generation pipeline. For MOFA-Adapters, we
consider the temporal motion consistency of the video and generate the dense
motion flow from the given sparse control conditions first, and then, the
multi-scale features of the given image are wrapped as a guided feature for
stable video diffusion generation. We naively train two motion adapters for the
manual trajectories and the human landmarks individually since they both
contain sparse information about the control. After training, the MOFA-Adapters
in different domains can also work together for more controllable video
generation.
LLMs achieve adult human performance on higher-order theory of mind
  tasks
Authors: name: Winnie Street, name: John Oliver Siy, name: Geoff Keeling, name: Adrien Baranes, name: Benjamin Barnett, name: Michael McKibben, name: Tatenda Kanyere, name: Alison Lentz, name: Blaise Aguera y Arcas, name: Robin I. M. Dunbar
Published: 2024-05-29 08:31:16

This paper examines the extent to which large language models (LLMs) have
developed higher-order theory of mind (ToM); the human ability to reason about
multiple mental and emotional states in a recursive manner (e.g. I think that
you believe that she knows). This paper builds on prior work by introducing a
handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to
compare the performance of five LLMs to a newly gathered adult human benchmark.
We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level
performance on ToM tasks overall, and that GPT-4 exceeds adult performance on
6th order inferences. Our results suggest that there is an interplay between
model size and finetuning for the realisation of ToM abilities, and that the
best-performing LLMs have developed a generalised capacity for ToM. Given the
role that higher-order ToM plays in a wide range of cooperative and competitive
human behaviours, these findings have significant implications for user-facing
LLM applications.
NPGA: Neural Parametric Gaussian Avatars
Authors: name: Simon Giebenhain, name: Tobias Kirschstein, name: Martin Rünz, name: Lourdes Agapito, name: Matthias Nießner
Published: 2024-05-29 17:58:09

The creation of high-fidelity, digital versions of human heads is an
important stepping stone in the process of further integrating virtual
components into our everyday lives. Constructing such avatars is a challenging
research problem, due to a high demand for photo-realism and real-time
rendering performance. In this work, we propose Neural Parametric Gaussian
Avatars (NPGA), a data-driven approach to create high-fidelity, controllable
avatars from multi-view video recordings. We build our method around 3D
Gaussian Splatting for its highly efficient rendering and to inherit the
topological flexibility of point clouds. In contrast to previous work, we
condition our avatars' dynamics on the rich expression space of neural
parametric head models (NPHM), instead of mesh-based 3DMMs. To this end, we
distill the backward deformation field of our underlying NPHM into forward
deformations which are compatible with rasterization-based rendering. All
remaining fine-scale, expression-dependent details are learned from the
multi-view videos. To increase the representational capacity of our avatars, we
augment the canonical Gaussian point cloud using per-primitive latent features
which govern its dynamic behavior. To regularize this increased dynamic
expressivity, we propose Laplacian terms on the latent features and predicted
dynamics. We evaluate our method on the public NeRSemble dataset, demonstrating
that NPGA significantly outperforms the previous state-of-the-art avatars on
the self-reenactment task by 2.6 PSNR. Furthermore, we demonstrate accurate
animation capabilities from real-world monocular videos.
SoundCTM: Uniting Score-based and Consistency Models for Text-to-Sound
  Generation
Authors: name: Koichi Saito, name: Dongjun Kim, name: Takashi Shibuya, name: Chieh-Hsin Lai, name: Zhi Zhong, name: Yuhta Takida, name: Yuki Mitsufuji
Published: 2024-05-28 18:14:52

Sound content is an indispensable element for multimedia works such as video
games, music, and films. Recent high-quality diffusion-based sound generation
models can serve as valuable tools for the creators. However, despite producing
high-quality sounds, these models often suffer from slow inference speeds. This
drawback burdens creators, who typically refine their sounds through trial and
error to align them with their artistic intentions. To address this issue, we
introduce Sound Consistency Trajectory Models (SoundCTM). Our model enables
flexible transitioning between high-quality 1-step sound generation and
superior sound quality through multi-step generation. This allows creators to
initially control sounds with 1-step samples before refining them through
multi-step generation. While CTM fundamentally achieves flexible 1-step and
multi-step generation, its impressive performance heavily depends on an
additional pretrained feature extractor and an adversarial loss, which are
expensive to train and not always available in other domains. Thus, we reframe
CTM's training framework and introduce a novel feature distance by utilizing
the teacher's network for a distillation loss. Additionally, while distilling
classifier-free guided trajectories, we train conditional and unconditional
student models simultaneously and interpolate between these models during
inference. We also propose training-free controllable frameworks for SoundCTM,
leveraging its flexible sampling capability. SoundCTM achieves both promising
1-step and multi-step real-time sound generation without using any extra
off-the-shelf networks. Furthermore, we demonstrate SoundCTM's capability of
controllable sound generation in a training-free manner.
Atlas3D: Physically Constrained Self-Supporting Text-to-3D for
  Simulation and Fabrication
Authors: name: Yunuo Chen, name: Tianyi Xie, name: Zeshun Zong, name: Xuan Li, name: Feng Gao, name: Yin Yang, name: Ying Nian Wu, name: Chenfanfu Jiang
Published: 2024-05-28 18:33:18

Existing diffusion-based text-to-3D generation methods primarily focus on
producing visually realistic shapes and appearances, often neglecting the
physical constraints necessary for downstream tasks. Generated models
frequently fail to maintain balance when placed in physics-based simulations or
3D printed. This balance is crucial for satisfying user design intentions in
interactive gaming, embodied AI, and robotics, where stable models are needed
for reliable interaction. Additionally, stable models ensure that 3D-printed
objects, such as figurines for home decoration, can stand on their own without
requiring additional supports. To fill this gap, we introduce Atlas3D, an
automatic and easy-to-implement method that enhances existing Score
Distillation Sampling (SDS)-based text-to-3D tools. Atlas3D ensures the
generation of self-supporting 3D models that adhere to physical laws of
stability under gravity, contact, and friction. Our approach combines a novel
differentiable simulation-based loss function with physically inspired
regularization, serving as either a refinement or a post-processing module for
existing frameworks. We verify Atlas3D's efficacy through extensive generation
tasks and validate the resulting 3D models in both simulated and real-world
environments.