Orionnnn/hub-stats huggingface dataset:
Expressive Whole-Body 3D Gaussian Avatar
Authors: name: Gyeongsik Moon, name: Takaaki Shiratori, name: Shunsuke Saito
Published: 2024-07-31 15:29:13

Facial expression and hand motions are necessary to express our emotions and
interact with the world. Nevertheless, most of the 3D human avatars modeled
from a casually captured video only support body motions without facial
expressions and hand motions.In this work, we present ExAvatar, an expressive
whole-body 3D human avatar learned from a short monocular video. We design
ExAvatar as a combination of the whole-body parametric mesh model (SMPL-X) and
3D Gaussian Splatting (3DGS). The main challenges are 1) a limited diversity of
facial expressions and poses in the video and 2) the absence of 3D
observations, such as 3D scans and RGBD images. The limited diversity in the
video makes animations with novel facial expressions and poses non-trivial. In
addition, the absence of 3D observations could cause significant ambiguity in
human parts that are not observed in the video, which can result in noticeable
artifacts under novel motions. To address them, we introduce our hybrid
representation of the mesh and 3D Gaussians. Our hybrid representation treats
each 3D Gaussian as a vertex on the surface with pre-defined connectivity
information (i.e., triangle faces) between them following the mesh topology of
SMPL-X. It makes our ExAvatar animatable with novel facial expressions by
driven by the facial expression space of SMPL-X. In addition, by using
connectivity-based regularizers, we significantly reduce artifacts in novel
facial expressions and poses.
Tora: Trajectory-oriented Diffusion Transformer for Video Generation
Authors: name: Zhenghao Zhang, name: Junchao Liao, name: Menghao Li, name: Long Qin, name: Weizhi Wang
Published: 2024-07-31 15:53:20

Recent advancements in Diffusion Transformer (DiT) have demonstrated
remarkable proficiency in producing high-quality video content. Nonetheless,
the potential of transformer-based diffusion models for effectively generating
videos with controllable motion remains an area of limited exploration. This
paper introduces Tora, the first trajectory-oriented DiT framework that
integrates textual, visual, and trajectory conditions concurrently for video
generation. Specifically, Tora consists of a Trajectory Extractor~(TE), a
Spatial-Temporal DiT, and a Motion-guidance Fuser~(MGF). The TE encodes
arbitrary trajectories into hierarchical spacetime motion patches with a 3D
video compression network. The MGF integrates the motion patches into the DiT
blocks to generate consistent videos following trajectories. Our design aligns
seamlessly with DiT's scalability, allowing precise control of video content's
dynamics with diverse durations, aspect ratios, and resolutions. Extensive
experiments demonstrate Tora's excellence in achieving high motion fidelity,
while also meticulously simulating the movement of the physical world. Page can
be found at https://ali-videoai.github.io/tora_video.
MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware
  Experts
Authors: name: Xi Victoria Lin, name: Akshat Shrivastava, name: Liang Luo, name: Srinivasan Iyer, name: Mike Lewis, name: Gargi Gosh, name: Luke Zettlemoyer, name: Armen Aghajanyan
Published: 2024-07-31 17:46:51

We introduce MoMa, a novel modality-aware mixture-of-experts (MoE)
architecture designed for pre-training mixed-modal, early-fusion language
models. MoMa processes images and text in arbitrary sequences by dividing
expert modules into modality-specific groups. These groups exclusively process
designated tokens while employing learned routing within each group to maintain
semantically informed adaptivity. Our empirical results reveal substantial
pre-training efficiency gains through this modality-specific parameter
allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model,
featuring 4 text experts and 4 image experts, achieves impressive FLOPs
savings: 3.7x overall, with 2.6x for text and 5.2x for image processing
compared to a compute-equivalent dense baseline, measured by pre-training loss.
This outperforms the standard expert-choice MoE with 8 mixed-modal experts,
which achieves 3x overall FLOPs savings (3x for text, 2.8x for image).
Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs
savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination
hurts performance in causal inference due to increased sensitivity to router
accuracy. These results demonstrate MoMa's potential to significantly advance
the efficiency of mixed-modal, early-fusion language model pre-training, paving
the way for more resource-efficient and capable multimodal AI systems.
Towards Achieving Human Parity on End-to-end Simultaneous Speech
  Translation via LLM Agent
Authors: name: Shanbo Cheng, name: Zhichao Huang, name: Tom Ko, name: Hang Li, name: Ningxin Peng, name: Lu Xu, name: Qini Zhang
Published: 2024-07-31 14:48:27

In this paper, we present Cross Language Agent -- Simultaneous
Interpretation, CLASI, a high-quality and human-like Simultaneous Speech
Translation (SiST) System. Inspired by professional human interpreters, we
utilize a novel data-driven read-write strategy to balance the translation
quality and latency. To address the challenge of translating in-domain
terminologies, CLASI employs a multi-modal retrieving module to obtain relevant
information to augment the translation. Supported by LLMs, our approach can
generate error-tolerated translation by considering the input audio, historical
context, and retrieved information. Experimental results show that our system
outperforms other systems by significant margins. Aligned with professional
human interpreters, we evaluate CLASI with a better human evaluation metric,
valid information proportion (VIP), which measures the amount of information
that can be successfully conveyed to the listeners. In the real-world
scenarios, where the speeches are often disfluent, informal, and unclear, CLASI
achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese
translation directions, respectively. In contrast, state-of-the-art commercial
or open-source systems only achieve 35.4% and 41.6%. On the extremely hard
dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70%
VIP.
Data Contamination Report from the 2024 CONDA Shared Task
Authors: name: Oscar Sainz, name: Iker García-Ferrero, name: Alon Jacovi, name: Jon Ander Campos, name: Yanai Elazar, name: Eneko Agirre, name: Yoav Goldberg, name: Wei-Lin Chen, name: Jenny Chim, name: Leshem Choshen, name: Luca D'Amico-Wong, name: Melissa Dell, name: Run-Ze Fan, name: Shahriar Golchin, name: Yucheng Li, name: Pengfei Liu, name: Bhavish Pahwa, name: Ameya Prabhu, name: Suryansh Sharma, name: Emily Silcock, name: Kateryna Solonko, name: David Stap, name: Mihai Surdeanu, name: Yu-Min Tseng, name: Vishaal Udandarao, name: Zengzhi Wang, name: Ruijie Xu, name: Jinglin Yang
Published: 2024-07-31 11:26:57

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant
aspects of data contamination in natural language processing, where data
contamination is understood as situations where evaluation data is included in
pre-training corpora used to train large scale models, compromising evaluation
results. The workshop fostered a shared task to collect evidence on data
contamination in current available datasets and models. The goal of the shared
task and associated database is to assist the community in understanding the
extent of the problem and to assist researchers in avoiding reporting
evaluation results on known contaminated resources. The shared task provides a
structured, centralized public database for the collection of contamination
evidence, open to contributions from the community via GitHub pool requests.
This first compilation paper is based on 566 reported entries over 91
contaminated sources from a total of 23 contributors. The details of the
individual contamination events are available in the platform. The platform
continues to be online, open to contributions from the community.
Fine-gained Zero-shot Video Sampling
Authors: name: Dengsheng Chen, name: Jie Hu, name: Xiaoming Wei, name: Enhua Wu
Published: 2024-07-31 09:36:58

Incorporating a temporal dimension into pretrained image diffusion models for
video generation is a prevalent approach. However, this method is
computationally demanding and necessitates large-scale video datasets. More
critically, the heterogeneity between image and video datasets often results in
catastrophic forgetting of the image expertise. Recent attempts to directly
extract video snippets from image diffusion models have somewhat mitigated
these problems. Nevertheless, these methods can only generate brief video clips
with simple movements and fail to capture fine-grained motion or non-grid
deformation. In this paper, we propose a novel Zero-Shot video Sampling
algorithm, denoted as ZS^2, capable of directly sampling
high-quality video clips from existing image synthesis methods, such as Stable
Diffusion, without any training or optimization. Specifically, ZS^2
utilizes the dependency noise model and temporal momentum attention to ensure
content consistency and animation coherence, respectively. This ability enables
it to excel in related tasks, such as conditional and context-specialized video
generation and instruction-guided video editing. Experimental results
demonstrate that ZS^2 achieves state-of-the-art performance in
zero-shot video generation, occasionally outperforming recent supervised
methods.
  Homepage: https://densechen.github.io/zss/.
A Large Encoder-Decoder Family of Foundation Models For Chemical
  Language
Authors: name: Eduardo Soares, name: Victor Shirasuna, name: Emilio Vital Brazil, name: Renato Cerqueira, name: Dmitry Zubarev, name: Kristin Schmidt
Published: 2024-07-24 20:30:39

Large-scale pre-training methodologies for chemical language models represent
a breakthrough in cheminformatics. These methods excel in tasks such as
property prediction and molecule generation by learning contextualized
representations of input tokens through self-supervised learning on large
unlabeled corpora. Typically, this involves pre-training on unlabeled data
followed by fine-tuning on specific tasks, reducing dependence on annotated
datasets and broadening chemical language representation understanding. This
paper introduces a large encoder-decoder chemical foundation models pre-trained
on a curated dataset of 91 million SMILES samples sourced from PubChem, which
is equivalent to 4 billion of molecular tokens. The proposed foundation model
supports different complex tasks, including quantum property prediction, and
offer flexibility with two main variants (289M and 8times289M). Our
experiments across multiple benchmark datasets validate the capacity of the
proposed model in providing state-of-the-art results for different tasks. We
also provide a preliminary assessment of the compositionality of the embedding
space as a prerequisite for the reasoning tasks. We demonstrate that the
produced latent space is separable compared to the state-of-the-art with
few-shot learning capabilities.
Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings
Authors: name: Gili Goldin, name: Shuly Wintner
Published: 2024-07-30 06:29:01

We present Knesset-DictaBERT, a large Hebrew language model fine-tuned on the
Knesset Corpus, which comprises Israeli parliamentary proceedings. The model is
based on the DictaBERT architecture and demonstrates significant improvements
in understanding parliamentary language according to the MLM task. We provide a
detailed evaluation of the model's performance, showing improvements in
perplexity and accuracy over the baseline DictaBERT model.
Harvesting Textual and Structured Data from the HAL Publication
  Repository
Authors: name: Francis Kulumba, name: Wissam Antoun, name: Guillaume Vimont, name: Laurent Romary
Published: 2024-07-30 07:14:04

HAL (Hyper Articles en Ligne) is the French national publication repository,
used by most higher education and research organizations for their open science
policy. As a digital library, it is a rich repository of scholarly documents,
but its potential for advanced research has been underutilized. We present
HALvest, a unique dataset that bridges the gap between citation networks and
the full text of papers submitted on HAL. We craft our dataset by filtering HAL
for scholarly publications, resulting in approximately 700,000 documents,
spanning 34 languages across 13 identified domains, suitable for language model
training, and yielding approximately 16.5 billion tokens (with 8 billion in
French and 7 billion in English, the most represented languages). We transform
the metadata of each paper into a citation network, producing a directed
heterogeneous graph. This graph includes uniquely identified authors on HAL, as
well as all open submitted papers, and their citations. We provide a baseline
for authorship attribution using the dataset, implement a range of
state-of-the-art models in graph representation learning for link prediction,
and discuss the usefulness of our generated knowledge graph structure.
Adapting Safe-for-Work Classifier for Malaysian Language Text: Enhancing
  Alignment in LLM-Ops Framework
Authors: name: Aisyah Razak, name: Ariff Nazhan, name: Kamarul Adha, name: Wan Adzhar Faiq Adzlan, name: Mas Aisyah Ahmad, name: Ammar Azman
Published: 2024-07-30 10:51:51

As large language models (LLMs) become increasingly integrated into
operational workflows (LLM-Ops), there is a pressing need for effective
guardrails to ensure safe and aligned interactions, including the ability to
detect potentially unsafe or inappropriate content across languages. However,
existing safe-for-work classifiers are primarily focused on English text. To
address this gap for the Malaysian language, we present a novel safe-for-work
text classifier tailored specifically for Malaysian language content. By
curating and annotating a first-of-its-kind dataset of Malaysian text spanning
multiple content categories, we trained a classification model capable of
identifying potentially unsafe material using state-of-the-art natural language
processing techniques. This work represents an important step in enabling safer
interactions and content filtering to mitigate potential risks and ensure
responsible deployment of LLMs. To maximize accessibility and promote further
research towards enhancing alignment in LLM-Ops for the Malaysian context, the
model is publicly released at
https://huggingface.co/malaysia-ai/malaysian-sfw-classifier.
Meltemi: The first open Large Language Model for Greek
Authors: name: Leon Voukoutis, name: Dimitris Roussis, name: Georgios Paraskevopoulos, name: Sokratis Sofianopoulos, name: Prokopis Prokopidis, name: Vassilis Papavasileiou, name: Athanasios Katsamanis, name: Stelios Piperidis, name: Vassilis Katsouros
Published: 2024-07-30 11:22:52

We describe the development and capabilities of Meltemi 7B, the first open
Large Language Model for the Greek language. Meltemi 7B has 7 billion
parameters and is trained on a 40 billion token Greek corpus. For the
development of Meltemi 7B, we adapt Mistral, by continuous pretraining on the
Greek Corpus. Meltemi 7B contains up-to-date information up to September 2023.
Furthermore, we have translated and curated a Greek instruction corpus, which
has been used for the instruction-tuning of a chat model, named Meltemi 7B
Instruct. Special care has been given to the alignment and the removal of toxic
content for the Meltemi 7B Instruct. The developed models are evaluated on a
broad set of collected evaluation corpora, and examples of prompts and
responses are presented. Both Meltemi 7B and Meltemi 7B Instruct are available
at https://huggingface.co/ilsp under the Apache 2.0 license.
Diffusion Augmented Agents: A Framework for Efficient Exploration and
  Transfer Learning
Authors: name: Norman Di Palo, name: Leonard Hasenclever, name: Jan Humplik, name: Arunkumar Byravan
Published: 2024-07-30 13:01:31

We introduce Diffusion Augmented Agents (DAAG), a novel framework that
leverages large language models, vision language models, and diffusion models
to improve sample efficiency and transfer learning in reinforcement learning
for embodied agents. DAAG hindsight relabels the agent's past experience by
using diffusion models to transform videos in a temporally and geometrically
consistent way to align with target instructions with a technique we call
Hindsight Experience Augmentation. A large language model orchestrates this
autonomous process without requiring human supervision, making it well-suited
for lifelong learning scenarios. The framework reduces the amount of
reward-labeled data needed to 1) finetune a vision language model that acts as
a reward detector, and 2) train RL agents on new tasks. We demonstrate the
sample efficiency gains of DAAG in simulated robotics environments involving
manipulation and navigation. Our results show that DAAG improves learning of
reward detectors, transferring past experience, and acquiring new tasks - key
abilities for developing efficient lifelong learning agents. Supplementary
material and visualizations are available on our website
https://sites.google.com/view/diffusion-augmented-agents/
JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create
  State-of-the-Art Japanese Retrievers with Constrained Resources
Authors: name: Benjamin Clavié
Published: 2024-07-30 11:42:19

Neural Information Retrieval has advanced rapidly in high-resource languages,
but progress in lower-resource ones such as Japanese has been hindered by data
scarcity, among other challenges. Consequently, multilingual models have
dominated Japanese retrieval, despite their computational inefficiencies and
inability to capture linguistic nuances. While recent multi-vector monolingual
models like JaColBERT have narrowed this gap, they still lag behind
multilingual methods in large-scale evaluations. This work addresses the
suboptimal training methods of multi-vector retrievers in lower-resource
settings, focusing on Japanese. We systematically evaluate and improve key
aspects of the inference and training settings of JaColBERT, and more broadly,
multi-vector models. We further enhance performance through a novel checkpoint
merging step, showcasing it to be an effective way of combining the benefits of
fine-tuning with the generalization capabilities of the original checkpoint.
Building on our analysis, we introduce a novel training recipe, resulting in
the JaColBERTv2.5 model. JaColBERTv2.5, with only 110 million parameters and
trained in under 15 hours on 4 A100 GPUs, significantly outperforms all
existing methods across all common benchmarks, reaching an average score of
0.754, significantly above the previous best of 0.720. To support future
research, we make our final models, intermediate checkpoints and all data used
publicly available.
ThinK: Thinner Key Cache by Query-Driven Pruning
Authors: name: Yuhui Xu, name: Zhanming Jie, name: Hanze Dong, name: Lei Wang, name: Xudong Lu, name: Aojun Zhou, name: Amrita Saha, name: Caiming Xiong, name: Doyen Sahoo
Published: 2024-07-30 17:59:08

Large Language Models (LLMs) have revolutionized the field of natural
language processing, achieving unprecedented performance across a variety of
applications by leveraging increased model sizes and sequence lengths. However,
the associated rise in computational and memory costs poses significant
challenges, particularly in managing long sequences due to the quadratic
complexity of the transformer attention mechanism. This paper focuses on the
long-context scenario, addressing the inefficiencies in KV cache memory
consumption during inference. Unlike existing approaches that optimize the
memory based on the sequence lengths, we uncover that the channel dimension of
the KV cache exhibits significant redundancy, characterized by unbalanced
magnitude distribution and low-rank structure in attention weights. Based on
these observations, we propose ThinK, a novel query-dependent KV cache pruning
method designed to minimize attention weight loss while selectively pruning the
least significant channels. Our approach not only maintains or enhances model
accuracy but also achieves a reduction in memory costs by over 20% compared
with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and
Mistral models across various long-sequence datasets confirm the efficacy of
ThinK, setting a new precedent for efficient LLM deployment without
compromising performance. We also outline the potential of extending our method
to value cache pruning, demonstrating ThinK's versatility and broad
applicability in reducing both memory and computational overheads.
Matting by Generation
Authors: name: Zhixiang Wang, name: Baiang Li, name: Jian Wang, name: Yu-Lun Liu, name: Jinwei Gu, name: Yung-Yu Chuang, name: Shin'ichi Satoh
Published: 2024-07-30 17:58:52

This paper introduces an innovative approach for image matting that redefines
the traditional regression-based task as a generative modeling challenge. Our
method harnesses the capabilities of latent diffusion models, enriched with
extensive pre-trained knowledge, to regularize the matting process. We present
novel architectural innovations that empower our model to produce mattes with
superior resolution and detail. The proposed method is versatile and can
perform both guidance-free and guidance-based image matting, accommodating a
variety of additional cues. Our comprehensive evaluation across three benchmark
datasets demonstrates the superior performance of our approach, both
quantitatively and qualitatively. The results not only reflect our method's
robust effectiveness but also highlight its ability to generate visually
compelling mattes that approach photorealistic quality. The project page for
this paper is available at
https://lightchaserx.github.io/matting-by-generation/
Futga: Towards Fine-grained Music Understanding through
  Temporally-enhanced Generative Augmentation
Authors: name: Junda Wu, name: Zachary Novack, name: Amit Namburi, name: Jiaheng Dai, name: Hao-Wen Dong, name: Zhouhang Xie, name: Carol Chen, name: Julian McAuley
Published: 2024-07-29 22:53:32

Existing music captioning methods are limited to generating concise global
descriptions of short music clips, which fail to capture fine-grained musical
characteristics and time-aware musical changes. To address these limitations,
we propose FUTGA, a model equipped with fined-grained music understanding
capabilities through learning from generative augmentation with temporal
compositions. We leverage existing music caption datasets and large language
models (LLMs) to synthesize fine-grained music captions with structural
descriptions and time boundaries for full-length songs. Augmented by the
proposed synthetic dataset, FUTGA is enabled to identify the music's temporal
changes at key transition points and their musical functions, as well as
generate detailed descriptions for each music segment. We further introduce a
full-length music caption dataset generated by FUTGA, as the augmentation of
the MusicCaps and the Song Describer datasets. We evaluate the automatically
generated captions on several downstream tasks, including music generation and
retrieval. The experiments demonstrate the quality of the generated captions
and the better performance in various downstream tasks achieved by the proposed
music captioning approach. Our code and datasets can be found in
https://huggingface.co/JoshuaW1997/FUTGA{blue{https://huggingface.co/JoshuaW1997/FUTGA}}.