Orionnnn/hub-stats huggingface dataset:
Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond
Authors: name: Beomseok Lee, name: Ioan Calapodescu, name: Marco Gaido, name: Matteo Negri, name: Laurent Besacier
Published: 2024-08-07 16:55:28

We present Speech-MASSIVE, a multilingual Spoken Language Understanding (SLU)
dataset comprising the speech counterpart for a portion of the MASSIVE textual
corpus. Speech-MASSIVE covers 12 languages from different families and inherits
from MASSIVE the annotations for the intent prediction and slot-filling tasks.
Our extension is prompted by the scarcity of massively multilingual SLU
datasets and the growing need for versatile speech datasets to assess
foundation models (LLMs, speech encoders) across languages and tasks. We
provide a multimodal, multitask, multilingual dataset and report SLU baselines
using both cascaded and end-to-end architectures in various training scenarios
(zero-shot, few-shot, and full fine-tune). Furthermore, we demonstrate the
suitability of Speech-MASSIVE for benchmarking other tasks such as speech
transcription, language identification, and speech translation. The dataset,
models, and code are publicly available at:
https://github.com/hlt-mt/Speech-MASSIVE
Compact 3D Gaussian Splatting for Static and Dynamic Radiance Fields
Authors: name: Joo Chan Lee, name: Daniel Rho, name: Xiangyu Sun, name: Jong Hwan Ko, name: Eunbyung Park
Published: 2024-08-07 14:56:34

3D Gaussian splatting (3DGS) has recently emerged as an alternative
representation that leverages a 3D Gaussian-based representation and introduces
an approximated volumetric rendering, achieving very fast rendering speed and
promising image quality. Furthermore, subsequent studies have successfully
extended 3DGS to dynamic 3D scenes, demonstrating its wide range of
applications. However, a significant drawback arises as 3DGS and its following
methods entail a substantial number of Gaussians to maintain the high fidelity
of the rendered images, which requires a large amount of memory and storage. To
address this critical issue, we place a specific emphasis on two key
objectives: reducing the number of Gaussian points without sacrificing
performance and compressing the Gaussian attributes, such as view-dependent
color and covariance. To this end, we propose a learnable mask strategy that
significantly reduces the number of Gaussians while preserving high
performance. In addition, we propose a compact but effective representation of
view-dependent color by employing a grid-based neural field rather than relying
on spherical harmonics. Finally, we learn codebooks to compactly represent the
geometric and temporal attributes by residual vector quantization. With model
compression techniques such as quantization and entropy coding, we consistently
show over 25x reduced storage and enhanced rendering speed compared to 3DGS for
static scenes, while maintaining the quality of the scene representation. For
dynamic scenes, our approach achieves more than 12x storage efficiency and
retains a high-quality reconstruction compared to the existing state-of-the-art
methods. Our work provides a comprehensive framework for 3D scene
representation, achieving high performance, fast training, compactness, and
real-time rendering. Our project page is available at
https://maincold2.github.io/c3dgs/.
Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in
  Long-Horizon Tasks
Authors: name: Zaijing Li, name: Yuquan Xie, name: Rui Shao, name: Gongwei Chen, name: Dongmei Jiang, name: Liqiang Nie
Published: 2024-08-07 08:16:32

Building a general-purpose agent is a long-standing vision in the field of
artificial intelligence. Existing agents have made remarkable progress in many
domains, yet they still struggle to complete long-horizon tasks in an open
world. We attribute this to the lack of necessary world knowledge and
multimodal experience that can guide agents through a variety of long-horizon
tasks. In this paper, we propose a Hybrid Multimodal Memory module to address
the above challenges. It 1) transforms knowledge into Hierarchical Directed
Knowledge Graph that allows agents to explicitly represent and learn world
knowledge, and 2) summarises historical information into Abstracted Multimodal
Experience Pool that provide agents with rich references for in-context
learning. On top of the Hybrid Multimodal Memory module, a multimodal agent,
Optimus-1, is constructed with dedicated Knowledge-guided Planner and
Experience-Driven Reflector, contributing to a better planning and reflection
in the face of long-horizon tasks in Minecraft. Extensive experimental results
show that Optimus-1 significantly outperforms all existing agents on
challenging long-horizon task benchmarks, and exhibits near human-level
performance on many tasks. In addition, we introduce various Multimodal Large
Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show
that Optimus-1 exhibits strong generalization with the help of the Hybrid
Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.
Openstory++: A Large-scale Dataset and Benchmark for Instance-aware
  Open-domain Visual Storytelling
Authors: name: Zilyu Ye, name: Jinxiu Liu, name: Ruotian Peng, name: Jinjin Cao, name: Zhiyang Chen, name: Yiyang Zhang, name: Ziwei Xuan, name: Mingyuan Zhou, name: Xiaoqian Shen, name: Mohamed Elhoseiny, name: Qi Liu, name: Guo-Jun Qi
Published: 2024-08-07 11:20:37

Recent image generation models excel at creating high-quality images from
brief captions. However, they fail to maintain consistency of multiple
instances across images when encountering lengthy contexts. This inconsistency
is largely due to in existing training datasets the absence of granular
instance feature labeling in existing training datasets. To tackle these
issues, we introduce Openstory++, a large-scale dataset combining additional
instance-level annotations with both images and text. Furthermore, we develop a
training methodology that emphasizes entity-centric image-text generation,
ensuring that the models learn to effectively interweave visual and textual
information. Specifically, Openstory++ streamlines the process of keyframe
extraction from open-domain videos, employing vision-language models to
generate captions that are then polished by a large language model for
narrative continuity. It surpasses previous datasets by offering a more
expansive open-domain resource, which incorporates automated captioning,
high-resolution imagery tailored for instance count, and extensive frame
sequences for temporal consistency. Additionally, we present Cohere-Bench, a
pioneering benchmark framework for evaluating the image generation tasks when
long multimodal context is provided, including the ability to keep the
background, style, instances in the given context coherent. Compared to
existing benchmarks, our work fills critical gaps in multi-modal generation,
propelling the development of models that can adeptly generate and interpret
complex narratives in open-domain environments. Experiments conducted within
Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality
visual storytelling models, enhancing their ability to address open-domain
generation tasks. More details can be found at https://openstorypp.github.io/
EXAONE 3.0 7.8B Instruction Tuned Language Model
Authors: name: LG AI Research, name: Soyoung An, name: Kyunghoon Bae, name: Eunbi Choi, name: Stanley Jungkyu Choi, name: Yemuk Choi, name: Seokhee Hong, name: Yeonjung Hong, name: Junwon Hwang, name: Hyojin Jeon, name: Gerrard Jeongwon Jo, name: Hyunjik Jo, name: Jiyeon Jung, name: Yountae Jung, name: Euisoon Kim, name: Hyosang Kim, name: Joonkee Kim, name: Seonghwan Kim, name: Soyeon Kim, name: Sunkyoung Kim, name: Yireun Kim, name: Youchul Kim, name: Edward Hwayoung Lee, name: Haeju Lee, name: Honglak Lee, name: Jinsik Lee, name: Kyungmin Lee, name: Moontae Lee, name: Seungjun Lee, name: Woohyung Lim, name: Sangha Park, name: Sooyoun Park, name: Yongmin Park, name: Boseong Seo, name: Sihoon Yang, name: Heuiyeen Yeen, name: Kyungjae Yoo, name: Hyeongu Yun
Published: 2024-08-07 04:38:38

We introduce EXAONE 3.0 instruction-tuned language model, the first open
model in the family of Large Language Models (LLMs) developed by LG AI
Research. Among different model sizes, we publicly release the 7.8B
instruction-tuned model to promote open research and innovations. Through
extensive evaluations across a wide range of public and in-house benchmarks,
EXAONE 3.0 demonstrates highly competitive real-world performance with
instruction-following capability against other state-of-the-art open models of
similar size. Our comparative analysis shows that EXAONE 3.0 excels
particularly in Korean, while achieving compelling performance across general
tasks and complex reasoning. With its strong real-world effectiveness and
bilingual proficiency, we hope that EXAONE keeps contributing to advancements
in Expert AI. Our EXAONE 3.0 instruction-tuned model is available at
https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct
RayGauss: Volumetric Gaussian-Based Ray Casting for Photorealistic Novel
  View Synthesis
Authors: name: Hugo Blanc, name: Jean-Emmanuel Deschaud, name: Alexis Paljic
Published: 2024-08-06 10:59:58

Differentiable volumetric rendering-based methods made significant progress
in novel view synthesis. On one hand, innovative methods have replaced the
Neural Radiance Fields (NeRF) network with locally parameterized structures,
enabling high-quality renderings in a reasonable time. On the other hand,
approaches have used differentiable splatting instead of NeRF's ray casting to
optimize radiance fields rapidly using Gaussian kernels, allowing for fine
adaptation to the scene. However, differentiable ray casting of irregularly
spaced kernels has been scarcely explored, while splatting, despite enabling
fast rendering times, is susceptible to clearly visible artifacts.
  Our work closes this gap by providing a physically consistent formulation of
the emitted radiance c and density {\sigma}, decomposed with Gaussian functions
associated with Spherical Gaussians/Harmonics for all-frequency colorimetric
representation. We also introduce a method enabling differentiable ray casting
of irregularly distributed Gaussians using an algorithm that integrates
radiance fields slab by slab and leverages a BVH structure. This allows our
approach to finely adapt to the scene while avoiding splatting artifacts. As a
result, we achieve superior rendering quality compared to the state-of-the-art
while maintaining reasonable training times and achieving inference speeds of
25 FPS on the Blender dataset. Project page with videos and code:
https://raygauss.github.io/
Achieving Human Level Competitive Robot Table Tennis
Authors: name: David B. D'Ambrosio, name: Saminda Abeyruwan, name: Laura Graesser, name: Atil Iscen, name: Heni Ben Amor, name: Alex Bewley, name: Barney J. Reed, name: Krista Reymann, name: Leila Takayama, name: Yuval Tassa, name: Krzysztof Choromanski, name: Erwin Coumans, name: Deepali Jain, name: Navdeep Jaitly, name: Natasha Jaques, name: Satoshi Kataoka, name: Yuheng Kuang, name: Nevena Lazic, name: Reza Mahjourian, name: Sherry Moore, name: Kenneth Oslund, name: Anish Shankar, name: Vikas Sindhwani, name: Vincent Vanhoucke, name: Grace Vesom, name: Peng Xu, name: Pannag R. Sanketi
Published: 2024-08-07 17:10:55

Achieving human-level speed and performance on real world tasks is a north
star for the robotics research community. This work takes a step towards that
goal and presents the first learned robot agent that reaches amateur
human-level performance in competitive table tennis. Table tennis is a
physically demanding sport which requires human players to undergo years of
training to achieve an advanced level of proficiency. In this paper, we
contribute (1) a hierarchical and modular policy architecture consisting of (i)
low level controllers with their detailed skill descriptors which model the
agent's capabilities and help to bridge the sim-to-real gap and (ii) a high
level controller that chooses the low level skills, (2) techniques for enabling
zero-shot sim-to-real including an iterative approach to defining the task
distribution that is grounded in the real-world and defines an automatic
curriculum, and (3) real time adaptation to unseen opponents. Policy
performance was assessed through 29 robot vs. human matches of which the robot
won 45% (13/29). All humans were unseen players and their skill level varied
from beginner to tournament level. Whilst the robot lost all matches vs. the
most advanced players it won 100% matches vs. beginners and 55% matches vs.
intermediate players, demonstrating solidly amateur human-level performance.
Videos of the matches can be viewed at
https://sites.google.com/view/competitive-robot-table-tennis
Synthesizing Text-to-SQL Data from Weak and Strong LLMs
Authors: name: Jiaxi Yang, name: Binyuan Hui, name: Min Yang, name: Jian Yang, name: Junyang Lin, name: Chang Zhou
Published: 2024-08-06 15:40:32

The capability gap between open-source and closed-source large language
models (LLMs) remains a challenge in text-to-SQL tasks. In this paper, we
introduce a synthetic data approach that combines data produced by larger, more
powerful models (strong models) with error information data generated by
smaller, not well-aligned models (weak models). The method not only enhances
the domain generalization of text-to-SQL models but also explores the potential
of error data supervision through preference learning. Furthermore, we employ
the synthetic data approach for instruction tuning on open-source LLMs,
resulting SENSE, a specialized text-to-SQL model. The effectiveness of SENSE is
demonstrated through state-of-the-art results on the SPIDER and BIRD
benchmarks, bridging the performance gap between open-source models and methods
prompted by closed-source models.
StructEval: Deepen and Broaden Large Language Model Assessment via
  Structured Evaluation
Authors: name: Boxi Cao, name: Mengjie Ren, name: Hongyu Lin, name: Xianpei Han, name: Feng Zhang, name: Junfeng Zhan, name: Le Sun
Published: 2024-08-06 16:28:30

Evaluation is the baton for the development of large language models. Current
evaluations typically employ a single-item assessment paradigm for each atomic
test objective, which struggles to discern whether a model genuinely possesses
the required capabilities or merely memorizes/guesses the answers to specific
questions. To this end, we propose a novel evaluation framework referred to as
StructEval. Starting from an atomic test objective, StructEval deepens and
broadens the evaluation by conducting a structured assessment across multiple
cognitive levels and critical concepts, and therefore offers a comprehensive,
robust and consistent evaluation for LLMs. Experiments on three widely-used
benchmarks demonstrate that StructEval serves as a reliable tool for resisting
the risk of data contamination and reducing the interference of potential
biases, thereby providing more reliable and consistent conclusions regarding
model capabilities. Our framework also sheds light on the design of future
principled and trustworthy LLM evaluation protocols.
AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual
  Segmentation
Authors: name: Zili Wang, name: Qi Yang, name: Linsu Shi, name: Jiazhong Yu, name: Qinghua Liang, name: Fei Li, name: Shiming Xiang
Published: 2024-08-03 08:25:26

Recently, transformer-based models have demonstrated remarkable performance
on audio-visual segmentation (AVS) tasks. However, their expensive
computational cost makes real-time inference impractical. By characterizing
attention maps of the network, we identify two key obstacles in AVS models: 1)
attention dissipation, corresponding to the over-concentrated attention weights
by Softmax within restricted frames, and 2) inefficient, burdensome transformer
decoder, caused by narrow focus patterns in early stages. In this paper, we
introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation
transformer that achieves fast, efficient and light-weight simultaneously. Our
model leverages an efficient prompt query generator to correct the behaviour of
cross-attention. Additionally, we propose ELF decoder to bring greater
efficiency by facilitating convolutions suitable for local features to reduce
computational burdens. Extensive experiments demonstrate that our AVESFormer
significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3
and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an
excellent trade-off between performance and speed. Code can be found at
https://github.com/MarkXCloud/AVESFormer.git.
LLaVA-OneVision: Easy Visual Task Transfer
Authors: name: Bo Li, name: Yuanhan Zhang, name: Dong Guo, name: Renrui Zhang, name: Feng Li, name: Hao Zhang, name: Kaichen Zhang, name: Yanwei Li, name: Ziwei Liu, name: Chunyuan Li
Published: 2024-08-06 17:59:44

We present LLaVA-OneVision, a family of open large multimodal models (LMMs)
developed by consolidating our insights into data, models, and visual
representations in the LLaVA-NeXT blog series. Our experimental results
demonstrate that LLaVA-OneVision is the first single model that can
simultaneously push the performance boundaries of open LMMs in three important
computer vision scenarios: single-image, multi-image, and video scenarios.
Importantly, the design of LLaVA-OneVision allows strong transfer learning
across different modalities/scenarios, yielding new emerging capabilities. In
particular, strong video understanding and cross-scenario capabilities are
demonstrated through task transfer from images to videos.
ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
  Synced Facial Performer
Authors: name: Jiazhi Guan, name: Zhiliang Xu, name: Hang Zhou, name: Kaisiyuan Wang, name: Shengyi He, name: Zhanwang Zhang, name: Borong Liang, name: Haocheng Feng, name: Errui Ding, name: Jingtuo Liu, name: Jingdong Wang, name: Youjian Zhao, name: Ziwei Liu
Published: 2024-08-06 16:31:45

Lip-syncing videos with given audio is the foundation for various
applications including the creation of virtual presenters or performers. While
recent studies explore high-fidelity lip-sync with different techniques, their
task-orientated models either require long-term videos for clip-specific
training or retain visible artifacts. In this paper, we propose a unified and
effective framework ReSyncer, that synchronizes generalized audio-visual facial
information. The key design is revisiting and rewiring the Style-based
generator to efficiently adopt 3D facial dynamics predicted by a principled
style-injected Transformer. By simply re-configuring the information insertion
mechanisms within the noise and style space, our framework fuses motion and
appearance with unified training. Extensive experiments demonstrate that
ReSyncer not only produces high-fidelity lip-synced videos according to audio,
but also supports multiple appealing properties that are suitable for creating
virtual presenters and performers, including fast personalized fine-tuning,
video-driven lip-syncing, the transfer of speaking styles, and even face
swapping. Resources can be found at
https://guanjz20.github.io/projects/ReSyncer.
IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning
  using Instruct Prompts
Authors: name: Ciara Rowles, name: Shimon Vainer, name: Dante De Nigris, name: Slava Elizarov, name: Konstantin Kutsy, name: Simon Donné
Published: 2024-08-06 14:08:22

Diffusion models continuously push the boundary of state-of-the-art image
generation, but the process is hard to control with any nuance: practice proves
that textual prompts are inadequate for accurately describing image style or
fine structural details (such as faces). ControlNet and IPAdapter address this
shortcoming by conditioning the generative process on imagery instead, but each
individual instance is limited to modeling a single conditional posterior: for
practical use-cases, where multiple different posteriors are desired within the
same workflow, training and using multiple adapters is cumbersome. We propose
IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct''
prompts to swap between interpretations for the same conditioning image: style
transfer, object extraction, both, or something else still? IPAdapterInstruct
efficiently learns multiple tasks with minimal loss in quality compared to
dedicated per-task models.
An Object is Worth 64x64 Pixels: Generating 3D Object via Image
  Diffusion
Authors: name: Xingguang Yan, name: Han-Hung Lee, name: Ziyu Wan, name: Angel X. Chang
Published: 2024-08-06 13:22:51

We introduce a new approach for generating realistic 3D models with UV maps
through a representation termed "Object Images." This approach encapsulates
surface geometry, appearance, and patch structures within a 64x64 pixel image,
effectively converting complex 3D shapes into a more manageable 2D format. By
doing so, we address the challenges of both geometric and semantic irregularity
inherent in polygonal meshes. This method allows us to use image generation
models, such as Diffusion Transformers, directly for 3D shape generation.
Evaluated on the ABO dataset, our generated shapes with patch structures
achieve point cloud FID comparable to recent 3D generative models, while
naturally supporting PBR material generation.
MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular
  Annotations for Medicine
Authors: name: Yunfei Xie, name: Ce Zhou, name: Lang Gao, name: Juncheng Wu, name: Xianhang Li, name: Hong-Yu Zhou, name: Sheng Liu, name: Lei Xing, name: James Zou, name: Cihang Xie, name: Yuyin Zhou
Published: 2024-08-06 02:09:35

This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal
dataset for medicine, covering over 25 million images across 10 modalities,
with multigranular annotations for more than 65 diseases. These enriched
annotations encompass both global textual information, such as disease/lesion
type, modality, region-specific descriptions, and inter-regional relationships,
as well as detailed local annotations for regions of interest (ROIs), including
bounding boxes, segmentation masks. Unlike existing approach which is limited
by the availability of image-text pairs, we have developed the first automated
pipeline that scales up multimodal data by generating multigranular visual and
texual annotations (in the form of image-ROI-description triplets) without the
need for any paired text descriptions. Specifically, data from over 90
different sources have been collected, preprocessed, and grounded using
domain-specific expert models to identify ROIs related to abnormal regions. We
then build a comprehensive knowledge base and prompt multimodal large language
models to perform retrieval-augmented generation with the identified ROIs as
guidance, resulting in multigranular texual descriptions. Compared to existing
datasets, MedTrinity-25M provides the most enriched annotations, supporting a
comprehensive range of multimodal tasks such as captioning and report
generation, as well as vision-centric tasks like classification and
segmentation. Pretraining on MedTrinity-25M, our model achieves
state-of-the-art performance on VQA-RAD and PathVQA, surpassing both multimodal
large language models and other representative SoTA approaches. This dataset
can also be utilized to support large-scale pre-training of multimodal medical
AI models, contributing to the development of future foundation models in the
medical domain.