Orionnnn/hub-stats huggingface dataset:
TEDRA: Text-based Editing of Dynamic and Photoreal Actors
Authors: name: Basavaraj Sunagad, name: Heming Zhu, name: Mohit Mendiratta, name: Adam Kortylewski, name: Christian Theobalt, name: Marc Habermann
Published: 2024-08-28 17:59:02

Over the past years, significant progress has been made in creating
photorealistic and drivable 3D avatars solely from videos of real humans.
However, a core remaining challenge is the fine-grained and user-friendly
editing of clothing styles by means of textual descriptions. To this end, we
present TEDRA, the first method allowing text-based edits of an avatar, which
maintains the avatar's high fidelity, space-time coherency, as well as
dynamics, and enables skeletal pose and view control. We begin by training a
model to create a controllable and high-fidelity digital replica of the real
actor. Next, we personalize a pretrained generative diffusion model by
fine-tuning it on various frames of the real character captured from different
camera angles, ensuring the digital representation faithfully captures the
dynamics and movements of the real person. This two-stage process lays the
foundation for our approach to dynamic human avatar editing. Utilizing this
personalized diffusion model, we modify the dynamic avatar based on a provided
text prompt using our Personalized Normal Aligned Score Distillation Sampling
(PNA-SDS) within a model-based guidance framework. Additionally, we propose a
time step annealing strategy to ensure high-quality edits. Our results
demonstrate a clear improvement over prior work in functionality and visual
quality.
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
Authors: name: Fangxun Shu, name: Yue Liao, name: Le Zhuo, name: Chenning Xu, name: Guanghao Zhang, name: Haonan Shi, name: Long Chen, name: Tao Zhong, name: Wanggui He, name: Siming Fu, name: Haoyuan Li, name: Bolin Li, name: Zhelun Yu, name: Si Liu, name: Hongsheng Li, name: Hao Jiang
Published: 2024-08-28 15:52:23

We introduce LLaVA-MoD, a novel framework designed to enable the efficient
training of small-scale Multimodal Language Models (s-MLLM) by distilling
knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental
challenges in MLLM distillation. First, we optimize the network structure of
s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the
language model, striking a balance between computational efficiency and model
expressiveness. Second, we propose a progressive knowledge transfer strategy to
ensure comprehensive knowledge migration. This strategy begins with mimic
distillation, where we minimize the Kullback-Leibler (KL) divergence between
output distributions to enable the student model to emulate the teacher
network's understanding. Following this, we introduce preference distillation
via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM
as the reference model. During this phase, the s-MLLM's ability to discriminate
between superior and inferior examples is significantly enhanced beyond l-MLLM,
leading to a better student that surpasses its teacher, particularly in
hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD
outperforms existing models across various multimodal benchmarks while
maintaining a minimal number of activated parameters and low computational
costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses
Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of
the training data and 23% trainable parameters. These results underscore
LLaVA-MoD's ability to effectively distill comprehensive knowledge from its
teacher model, paving the way for the development of more efficient MLLMs. The
code will be available on: https://github.com/shufangxun/LLaVA-MoD.
Towards Realistic Example-based Modeling via 3D Gaussian Stitching
Authors: name: Xinyu Gao, name: Ziyi Yang, name: Bingchen Gong, name: Xiaoguang Han, name: Sipeng Yang, name: Xiaogang Jin
Published: 2024-08-28 11:13:27

Using parts of existing models to rebuild new models, commonly termed as
example-based modeling, is a classical methodology in the realm of computer
graphics. Previous works mostly focus on shape composition, making them very
hard to use for realistic composition of 3D objects captured from real-world
scenes. This leads to combining multiple NeRFs into a single 3D scene to
achieve seamless appearance blending. However, the current SeamlessNeRF method
struggles to achieve interactive editing and harmonious stitching for
real-world scenes due to its gradient-based strategy and grid-based
representation. To this end, we present an example-based modeling method that
combines multiple Gaussian fields in a point-based representation using
sample-guided synthesis. Specifically, as for composition, we create a GUI to
segment and transform multiple fields in real time, easily obtaining a
semantically meaningful composition of models represented by 3D Gaussian
Splatting (3DGS). For texture blending, due to the discrete and irregular
nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF
is not supported. Thus, a novel sampling-based cloning method is proposed to
harmonize the blending while preserving the original rich texture and content.
Our workflow consists of three steps: 1) real-time segmentation and
transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis
to identify boundary points in the intersecting area between the source and
target models, and 3) two-phase optimization of the target model using
sampling-based cloning and gradient constraints. Extensive experimental results
validate that our approach significantly outperforms previous works in terms of
realistic synthesis, demonstrating its practicality. More demos are available
at https://ingra14m.github.io/gs_stitching_website.
Leveraging Open Knowledge for Advancing Task Expertise in Large Language
  Models
Authors: name: Yuncheng Yang, name: Yulei Qin, name: Tong Wu, name: Zihan Xu, name: Gang Li, name: Pengcheng Guo, name: Hang Shao, name: Yucheng Shi, name: Ke Li, name: Xing Sun, name: Jie Yang, name: Yun Gu
Published: 2024-08-28 16:28:07

The cultivation of expertise for large language models (LLMs) to solve tasks
of specific areas often requires special-purpose tuning with calibrated
behaviors on the expected stable outputs. To avoid huge cost brought by manual
preparation of instruction datasets and training resources up to hundreds of
hours, the exploitation of open knowledge including a wealth of low rank
adaptation (LoRA) models and instruction datasets serves as a good starting
point. However, existing methods on model and data selection focus on the
performance of general-purpose capabilities while neglecting the knowledge gap
exposed in domain-specific deployment. In the present study, we propose to
bridge such gap by introducing few human-annotated samples (i.e., K-shot) for
advancing task expertise of LLMs with open knowledge. Specifically, we develop
an efficient and scalable pipeline to cost-efficiently produce task experts
where K-shot data intervene in selecting the most promising expert candidates
and the task-relevant instructions. A mixture-of-expert (MoE) system is built
to make the best use of individual-yet-complementary knowledge between multiple
experts. We unveil the two keys to the success of a MoE system, 1) the abidance
by K-shot, and 2) the insistence on diversity. For the former, we ensure that
models that truly possess problem-solving abilities on K-shot are selected
rather than those blind guessers. Besides, during data selection, instructions
that share task-relevant contexts with K-shot are prioritized. For the latter,
we highlight the diversity of constituting experts and that of the fine-tuning
instructions throughout the model and data selection process. Extensive
experimental results confirm the superiority of our approach over existing
methods on utilization of open knowledge across various tasks. Codes and models
will be released later.
Dolphin: Long Context as a New Modality for Energy-Efficient On-Device
  Language Models
Authors: name: Wei Chen, name: Zhiyuan Li, name: Shuo Xin, name: Yihao Wang
Published: 2024-08-28 04:06:14

This paper presents Dolphin, a novel decoder-decoder architecture for
energy-efficient processing of long contexts in language models. Our approach
addresses the significant energy consumption and latency challenges inherent in
on-device models. Dolphin employs a compact 0.5B parameter decoder to distill
extensive contextual information into a memory embedding, substantially
reducing the input length for the primary 7B parameter decoder model. Inspired
by vision-language models, we repurpose the image embedding projector to encode
long textual contexts, effectively treating extended context as a distinct
modality. This innovative method enables processing of substantially longer
contexts without the typical computational overhead associated with extended
input sequences. Empirical evaluations demonstrate a 10-fold improvement in
energy efficiency and a 5-fold reduction in latency compared to conventional
full-length context processing methods without losing quality of the response.
Our work contributes to the development of more sustainable and scalable
language models for on-device applications, addressing the critical need for
energy-efficient and responsive AI technologies in resource-constrained
environments while maintaining the accuracy to understand long contexts. This
research has implications for the broader field of natural language processing,
particularly in the domain of efficient model design for resource-limited
settings. By enabling more sophisticated AI capabilities on edge devices,
Dolphin paves the way for advanced language processing in a wide range of
applications where computational resources are at a premium. The Dolphin model
is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
LLM-3D Print: Large Language Models To Monitor and Control 3D Printing
Authors: name: Yayati Jadhav, name: Peter Pak, name: Amir Barati Farimani
Published: 2024-08-26 14:38:19

Industry 4.0 has revolutionized manufacturing by driving digitalization and
shifting the paradigm toward additive manufacturing (AM). Fused Deposition
Modeling (FDM), a key AM technology, enables the creation of highly customized,
cost-effective products with minimal material waste through layer-by-layer
extrusion, posing a significant challenge to traditional subtractive methods.
However, the susceptibility of material extrusion techniques to errors often
requires expert intervention to detect and mitigate defects that can severely
compromise product quality. While automated error detection and machine
learning models exist, their generalizability across diverse 3D printer setups,
firmware, and sensors is limited, and deep learning methods require extensive
labeled datasets, hindering scalability and adaptability. To address these
challenges, we present a process monitoring and control framework that
leverages pre-trained Large Language Models (LLMs) alongside 3D printers to
detect and address printing defects. The LLM evaluates print quality by
analyzing images captured after each layer or print segment, identifying
failure modes and querying the printer for relevant parameters. It then
generates and executes a corrective action plan. We validated the effectiveness
of the proposed framework in identifying defects by comparing it against a
control group of engineers with diverse AM expertise. Our evaluation
demonstrated that LLM-based agents not only accurately identify common 3D
printing errors, such as inconsistent extrusion, stringing, warping, and layer
adhesion, but also effectively determine the parameters causing these failures
and autonomously correct them without any need for human intervention.
Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On
  Wikidata using LM probing
Authors: name: Hanna Abi Akl
Published: 2024-08-27 08:01:13

We introduce SHADOW, a fine-tuned language model trained on an intermediate
task using associative deductive reasoning, and measure its performance on a
knowledge base construction task using Wikidata triple completion. We evaluate
SHADOW on the LM-KBC 2024 challenge and show that it outperforms the baseline
solution by 20% with a F1 score of 68.72%.
Temporally-consistent 3D Reconstruction of Birds
Authors: name: Johannes Hägerlind, name: Jonas Hentati-Sundberg, name: Bastian Wandt
Published: 2024-08-24 17:12:36

This paper deals with 3D reconstruction of seabirds which recently came into
focus of environmental scientists as valuable bio-indicators for environmental
change. Such 3D information is beneficial for analyzing the bird's behavior and
physiological shape, for example by tracking motion, shape, and appearance
changes. From a computer vision perspective birds are especially challenging
due to their rapid and oftentimes non-rigid motions. We propose an approach to
reconstruct the 3D pose and shape from monocular videos of a specific breed of
seabird - the common murre. Our approach comprises a full pipeline of
detection, tracking, segmentation, and temporally consistent 3D reconstruction.
Additionally, we propose a temporal loss that extends current single-image 3D
bird pose estimators to the temporal domain. Moreover, we provide a real-world
dataset of 10000 frames of video observations on average capture nine birds
simultaneously, comprising a large variety of motions and interactions,
including a smaller test set with bird-specific keypoint labels. Using our
temporal optimization, we achieve state-of-the-art performance for the
challenging sequences in our dataset.
DSTI at LLMs4OL 2024 Task A: Intrinsic versus extrinsic knowledge for
  type classification
Authors: name: Hanna Abi Akl
Published: 2024-08-26 12:50:27

We introduce semantic towers, an extrinsic knowledge representation method,
and compare it to intrinsic knowledge in large language models for ontology
learning. Our experiments show a trade-off between performance and semantic
grounding for extrinsic knowledge compared to a fine-tuned model intrinsic
knowledge. We report our findings on the Large Language Models for Ontology
Learning (LLMs4OL) 2024 challenge.
Writing in the Margins: Better Inference Pattern for Long Context
  Retrieval
Authors: name: Melisa Russak, name: Umar Jamil, name: Christopher Bryant, name: Kiran Kamble, name: Axel Magnuson, name: Mateusz Russak, name: Waseem AlShikh
Published: 2024-08-27 09:34:38

In this paper, we introduce Writing in the Margins (WiM), a new inference
pattern for Large Language Models designed to optimize the handling of long
input sequences in retrieval-oriented tasks. This approach leverages the
chunked prefill of the key-value cache to perform segment-wise inference, which
enables efficient processing of extensive contexts along with the generation
and classification of intermediate information ("margins") that guide the model
towards specific tasks. This method increases computational overhead marginally
while significantly enhancing the performance of off-the-shelf models without
the need for fine-tuning. Specifically, we observe that WiM provides an average
enhancement of 7.5% in accuracy for reasoning skills (HotpotQA, MultiHop-RAG)
and more than a 30.0% increase in the F1-score for aggregation tasks (CWE).
Additionally, we show how the proposed pattern fits into an interactive
retrieval design that provides end-users with ongoing updates about the
progress of context processing, and pinpoints the integration of relevant
information into the final response. We release our implementation of WiM using
Hugging Face Transformers library at
https://github.com/writer/writing-in-the-margins.
Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
  Interpolation
Authors: name: Xiaojuan Wang, name: Boyang Zhou, name: Brian Curless, name: Ira Kemelmacher-Shlizerman, name: Aleksander Holynski, name: Steven M. Seitz
Published: 2024-08-27 17:57:14

We present a method for generating video sequences with coherent motion
between a pair of input key frames. We adapt a pretrained large-scale
image-to-video diffusion model (originally trained to generate videos moving
forward in time from a single input image) for key frame interpolation, i.e.,
to produce a video in between two input frames. We accomplish this adaptation
through a lightweight fine-tuning technique that produces a version of the
model that instead predicts videos moving backwards in time from a single input
image. This model (along with the original forward-moving model) is
subsequently used in a dual-directional diffusion sampling process that
combines the overlapping model estimates starting from each of the two
keyframes. Our experiments show that our method outperforms both existing
diffusion-based methods and traditional frame interpolation techniques.
The Mamba in the Llama: Distilling and Accelerating Hybrid Models
Authors: name: Junxiong Wang, name: Daniele Paliotta, name: Avner May, name: Alexander M. Rush, name: Tri Dao
Published: 2024-08-27 17:56:11

Linear RNN architectures, like Mamba, can be competitive with Transformer
models in language modeling while having advantageous deployment
characteristics. Given the focus on training large-scale Transformer models, we
consider the challenge of converting these pretrained models for deployment. We
demonstrate that it is feasible to distill large Transformers into linear RNNs
by reusing the linear projection weights from attention layers with academic
GPU resources. The resulting hybrid model, which incorporates a quarter of the
attention layers, achieves performance comparable to the original Transformer
in chat benchmarks and outperforms open-source hybrid Mamba models trained from
scratch with trillions of tokens in both chat benchmarks and general
benchmarks. Moreover, we introduce a hardware-aware speculative decoding
algorithm that accelerates the inference speed of Mamba and hybrid models.
Overall we show how, with limited computation resources, we can remove many of
the original attention layers and generate from the resulting model more
efficiently. Our top-performing model, distilled from Llama3-8B-Instruct,
achieves a 29.61 length-controlled win rate on AlpacaEval 2 against GPT-4 and
7.35 on MT-Bench, surpassing the best instruction-tuned linear RNN model.
Diffusion Models Are Real-Time Game Engines
Authors: name: Dani Valevski, name: Yaniv Leviathan, name: Moab Arar, name: Shlomi Fruchter
Published: 2024-08-27 07:46:07

We present GameNGen, the first game engine powered entirely by a neural model
that enables real-time interaction with a complex environment over long
trajectories at high quality. GameNGen can interactively simulate the classic
game DOOM at over 20 frames per second on a single TPU. Next frame prediction
achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are
only slightly better than random chance at distinguishing short clips of the
game from clips of the simulation. GameNGen is trained in two phases: (1) an
RL-agent learns to play the game and the training sessions are recorded, and
(2) a diffusion model is trained to produce the next frame, conditioned on the
sequence of past frames and actions. Conditioning augmentations enable stable
auto-regressive generation over long trajectories.
Text2SQL is Not Enough: Unifying AI and Databases with TAG
Authors: name: Asim Biswal, name: Liana Patel, name: Siddarth Jha, name: Amog Kamsetty, name: Shu Liu, name: Joseph E. Gonzalez, name: Carlos Guestrin, name: Matei Zaharia
Published: 2024-08-27 00:50:14

AI systems that serve natural language questions over databases promise to
unlock tremendous value. Such systems would allow users to leverage the
powerful reasoning and knowledge capabilities of language models (LMs)
alongside the scalable computational power of data management systems. These
combined capabilities would empower users to ask arbitrary natural language
questions over custom data sources. However, existing methods and benchmarks
insufficiently explore this setting. Text2SQL methods focus solely on natural
language questions that can be expressed in relational algebra, representing a
small subset of the questions real users wish to ask. Likewise,
Retrieval-Augmented Generation (RAG) considers the limited subset of queries
that can be answered with point lookups to one or a few data records within the
database. We propose Table-Augmented Generation (TAG), a unified and
general-purpose paradigm for answering natural language questions over
databases. The TAG model represents a wide range of interactions between the LM
and database that have been previously unexplored and creates exciting research
opportunities for leveraging the world knowledge and reasoning capabilities of
LMs over data. We systematically develop benchmarks to study the TAG problem
and find that standard methods answer no more than 20% of queries correctly,
confirming the need for further research in this area. We release code for the
benchmark at https://github.com/TAG-Research/TAG-Bench.
Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image
  Generation
Authors: name: Abdelrahman Eldesokey, name: Peter Wonka
Published: 2024-08-27 07:01:56

We propose a diffusion-based approach for Text-to-Image (T2I) generation with
interactive 3D layout control. Layout control has been widely studied to
alleviate the shortcomings of T2I diffusion models in understanding objects'
placement and relationships from text descriptions. Nevertheless, existing
approaches for layout control are limited to 2D layouts, require the user to
provide a static layout beforehand, and fail to preserve generated images under
layout changes. This makes these approaches unsuitable for applications that
require 3D object-wise control and iterative refinements, e.g., interior design
and complex scene generation. To this end, we leverage the recent advancements
in depth-conditioned T2I models and propose a novel approach for interactive 3D
layout control. We replace the traditional 2D boxes used in layout control with
3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation
process, where at each stage, the user can insert, change, and move an object
in 3D while preserving objects from earlier stages. We achieve this through our
proposed Dynamic Self-Attention (DSA) module and the consistent 3D object
translation strategy. Experiments show that our approach can generate
complicated scenes based on 3D layouts, boosting the object generation success
rate over the standard depth-conditioned T2I methods by 2x. Moreover, it
outperforms other methods in comparison in preserving objects under layout
changes. Project Page: https://abdo-eldesokey.github.io/build-a-scene/