Orionnnn/hub-stats huggingface dataset:
Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side
  Images
Authors: name: Zhangyang Qi, name: Yunhan Yang, name: Mengchen Zhang, name: Long Xing, name: Xiaoyang Wu, name: Tong Wu, name: Dahua Lin, name: Xihui Liu, name: Jiaqi Wang, name: Hengshuang Zhao
Published: 2024-07-08 17:59:55

Recent advances in 3D AIGC have shown promise in directly creating 3D objects
from text and images, offering significant cost savings in animation and
product design. However, detailed edit and customization of 3D assets remains a
long-standing challenge. Specifically, 3D Generation methods lack the ability
to follow finely detailed instructions as precisely as their 2D image creation
counterparts. Imagine you can get a toy through 3D AIGC but with undesired
accessories and dressing. To tackle this challenge, we propose a novel pipeline
called Tailor3D, which swiftly creates customized 3D assets from editable
dual-side images. We aim to emulate a tailor's ability to locally change
objects or perform overall style transfer. Unlike creating 3D assets from
multiple views, using dual-side images eliminates conflicts on overlapping
areas that occur when editing individual views. Specifically, it begins by
editing the front view, then generates the back view of the object through
multi-view diffusion. Afterward, it proceeds to edit the back views. Finally, a
Dual-sided LRM is proposed to seamlessly stitch together the front and back 3D
features, akin to a tailor sewing together the front and back of a garment. The
Dual-sided LRM rectifies imperfect consistencies between the front and back
views, enhancing editing capabilities and reducing memory burdens while
seamlessly integrating them into a unified 3D representation with the LoRA
Triplane Transformer. Experimental results demonstrate Tailor3D's effectiveness
across various 3D generation and editing tasks, including 3D generative fill
and style transfer. It provides a user-friendly, efficient solution for editing
3D assets, with each editing step taking only seconds to complete.
LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation
  Capabilities Beyond 100 Languages
Authors: name: Yinquan Lu, name: Wenhao Zhu, name: Lei Li, name: Yu Qiao, name: Fei Yuan
Published: 2024-07-08 14:18:28

Large Language Models~(LLMs) demonstrate remarkable translation capabilities
in high-resource language tasks, yet their performance in low-resource
languages is hindered by insufficient multilingual data during pre-training. To
address this, we dedicate 35,000 A100-SXM4-80GB GPU hours in conducting
extensive multilingual continual pre-training on the LLaMA series models,
enabling translation support across more than 100 languages. Through a
comprehensive analysis of training strategies, such as vocabulary expansion and
data augmentation, we develop LLaMAX. Remarkably, without sacrificing its
generalization ability, LLaMAX achieves significantly higher translation
performance compared to existing open-source LLMs~(by more than 10 spBLEU
points) and performs on-par with specialized translation model~(M2M-100-12B) on
the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve
as a robust multilingual foundation model. The
code~\url{https://github.com/CONE-MT/LLaMAX/.} and
models~\url{https://huggingface.co/LLaMAX/.} are publicly available.
RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language
  Models
Authors: name: Peng Xia, name: Kangyu Zhu, name: Haoran Li, name: Hongtu Zhu, name: Yun Li, name: Gang Li, name: Linjun Zhang, name: Huaxiu Yao
Published: 2024-07-06 16:45:07

The recent emergence of Medical Large Vision Language Models (Med-LVLMs) has
enhanced medical diagnosis. However, current Med-LVLMs frequently encounter
factual issues, often generating responses that do not align with established
medical facts. Retrieval-Augmented Generation (RAG), which utilizes external
knowledge, can improve the factual accuracy of these models but introduces two
major challenges. First, limited retrieved contexts might not cover all
necessary information, while excessive retrieval can introduce irrelevant and
inaccurate references, interfering with the model's generation. Second, in
cases where the model originally responds correctly, applying RAG can lead to
an over-reliance on retrieved contexts, resulting in incorrect answers. To
address these issues, we propose RULE, which consists of two components. First,
we introduce a provably effective strategy for controlling factuality risk
through the calibrated selection of the number of retrieved contexts. Second,
based on samples where over-reliance on retrieved contexts led to errors, we
curate a preference dataset to fine-tune the model, balancing its dependence on
inherent knowledge and retrieved contexts for generation. We demonstrate the
effectiveness of RULE on three medical VQA datasets, achieving an average
improvement of 20.8% in factual accuracy. We publicly release our benchmark and
code in https://github.com/richard-peng-xia/RULE.
Granular Privacy Control for Geolocation with Vision Language Models
Authors: name: Ethan Mendes, name: Yang Chen, name: James Hays, name: Sauvik Das, name: Wei Xu, name: Alan Ritter
Published: 2024-07-06 04:06:55

Vision Language Models (VLMs) are rapidly advancing in their capability to
answer information-seeking questions. As these models are widely deployed in
consumer applications, they could lead to new privacy risks due to emergent
abilities to identify people in photos, geolocate images, etc. As we
demonstrate, somewhat surprisingly, current open-source and proprietary VLMs
are very capable image geolocators, making widespread geolocation with VLMs an
immediate privacy risk, rather than merely a theoretical future concern. As a
first step to address this challenge, we develop a new benchmark, GPTGeoChat,
to test the ability of VLMs to moderate geolocation dialogues with users. We
collect a set of 1,000 image geolocation conversations between in-house
annotators and GPT-4v, which are annotated with the granularity of location
information revealed at each turn. Using this new dataset, we evaluate the
ability of various VLMs to moderate GPT-4v geolocation conversations by
determining when too much location information has been revealed. We find that
custom fine-tuned models perform on par with prompted API-based models when
identifying leaked location information at the country or city level; however,
fine-tuning on supervised data appears to be needed to accurately moderate
finer granularities, such as the name of a restaurant or building.
Flash-VStream: Memory-Based Real-Time Understanding for Long Video
  Streams
Authors: name: Haoji Zhang, name: Yiqin Wang, name: Yansong Tang, name: Yong Liu, name: Jiashi Feng, name: Jifeng Dai, name: Xiaojie Jin
Published: 2024-06-12 11:07:55

Benefiting from the advancements in large language models and cross-modal
alignment, existing multi-modal video understanding methods have achieved
prominent performance in offline scenario. However, online video streams, as
one of the most common media forms in the real world, have seldom received
attention. Compared to offline videos, the 'dynamic' nature of online video
streams poses challenges for the direct application of existing models and
introduces new problems, such as the storage of extremely long-term
information, interaction between continuous visual content and 'asynchronous'
user questions. Therefore, in this paper we present Flash-VStream, a
video-language model that simulates the memory mechanism of human. Our model is
able to process extremely long video streams in real-time and respond to user
queries simultaneously. Compared to existing models, Flash-VStream achieves
significant reductions in inference latency and VRAM consumption, which is
intimately related to performing understanding of online streaming video. In
addition, given that existing video understanding benchmarks predominantly
concentrate on offline scenario, we propose VStream-QA, a novel question
answering benchmark specifically designed for online video streaming
understanding. Comparisons with popular existing methods on the proposed
benchmark demonstrate the superiority of our method for such challenging
setting. To verify the generalizability of our approach, we further evaluate it
on existing video understanding benchmarks and achieves state-of-the-art
performance in offline scenarios as well. All code, models, and datasets are
available at the https://invinciblewyq.github.io/vstream-page/
Safe Unlearning: A Surprisingly Effective and Generalizable Solution to
  Defend Against Jailbreak Attacks
Authors: name: Zhexin Zhang, name: Junxiao Yang, name: Pei Ke, name: Shiyao Cui, name: Chujie Zheng, name: Hongning Wang, name: Minlie Huang
Published: 2024-07-03 07:14:05

LLMs are known to be vulnerable to jailbreak attacks, even after safety
alignment. An important observation is that, while different types of jailbreak
attacks can generate significantly different queries, they mostly result in
similar responses that are rooted in the same harmful knowledge (e.g., detailed
steps to make a bomb). Therefore, we conjecture that directly unlearn the
harmful knowledge in the LLM can be a more effective way to defend against
jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
approaches. Our extensive experiments confirmed our insight and suggested
surprising generalizability of our unlearning-based approach: using only 20 raw
harmful questions without any jailbreak prompt during training, our
solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
out-of-distribution (OOD) harmful questions wrapped with various complex
jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
still has an ASR of 21.9\% even under the help of an additional safety system
prompt. Further analysis reveals that the generalization ability of our
solution stems from the intrinsic relatedness among harmful responses across
harmful questions (e.g., response patterns, shared steps and actions, and
similarity among their learned representations in the LLM). Our code is
available at https://github.com/thu-coai/SafeUnlearning.
Stark: Social Long-Term Multi-Modal Conversation with Persona
  Commonsense Knowledge
Authors: name: Young-Jun Lee, name: Dokyong Lee, name: Junyoung Youn, name: Kyeongjin Oh, name: Byungsoo Ko, name: Jonghwan Hyeon, name: Ho-Jin Choi
Published: 2024-07-04 14:26:49

Humans share a wide variety of images related to their personal experiences
within conversations via instant messaging tools. However, existing works focus
on (1) image-sharing behavior in singular sessions, leading to limited
long-term social interaction, and (2) a lack of personalized image-sharing
behavior. In this work, we introduce Stark, a large-scale long-term multi-modal
conversation dataset that covers a wide range of social personas in a
multi-modality format, time intervals, and images. To construct Stark
automatically, we propose a novel multi-modal contextualization framework, Mcu,
that generates long-term multi-modal dialogue distilled from ChatGPT and our
proposed Plan-and-Execute image aligner. Using our Stark, we train a
multi-modal conversation model, Ultron 7B, which demonstrates impressive visual
imagination ability. Furthermore, we demonstrate the effectiveness of our
dataset in human evaluation. We make our source code and dataset publicly
available.
DotaMath: Decomposition of Thought with Code Assistance and
  Self-correction for Mathematical Reasoning
Authors: name: Chengpeng Li, name: Guanting Dong, name: Mingfeng Xue, name: Ru Peng, name: Xiang Wang, name: Dayiheng Liu
Published: 2024-07-04 17:39:16

Large language models (LLMs) have made impressive progress in handling simple
math problems, yet they still struggle with more challenging and complex
mathematical tasks. In this paper, we introduce a series of LLMs that employs
the Decomposition of thought with code assistance and self-correction for
mathematical reasoning, dubbed as DotaMath. DotaMath models tackle complex
mathematical tasks by decomposing them into simpler logical subtasks,
leveraging code to solve these subtasks, obtaining fine-grained feedback from
the code interpreter, and engaging in self-reflection and correction. By
annotating diverse interactive tool-use trajectories and employing query
evolution on GSM8K and MATH datasets, we generate an instruction fine-tuning
dataset called DotaMathQA with 574K query-response pairs. We train a series of
base LLMs using imitation learning on DotaMathQA, resulting in DotaMath models
that achieve remarkable performance compared to open-source LLMs across various
in-domain and out-of-domain benchmarks. Notably, DotaMath-deepseek-7B showcases
an outstanding performance of 64.8% on the competitive MATH dataset and 86.7%
on GSM8K. Besides, DotaMath-deepseek-7B maintains strong competitiveness on a
series of in-domain and out-of-domain benchmarks (Avg. 80.1%). Looking forward,
we anticipate that the DotaMath paradigm will open new pathways for addressing
intricate mathematical problems. Our code is publicly available at
https://github.com/ChengpengLi1003/DotaMath.
AriGraph: Learning Knowledge Graph World Models with Episodic Memory for
  LLM Agents
Authors: name: Petr Anokhin, name: Nikita Semenov, name: Artyom Sorokin, name: Dmitry Evseev, name: Mikhail Burtsev, name: Evgeny Burnaev
Published: 2024-07-05 09:06:47

Advancements in generative AI have broadened the potential applications of
Large Language Models (LLMs) in the development of autonomous agents. Achieving
true autonomy requires accumulating and updating knowledge gained from
interactions with the environment and effectively utilizing it. Current
LLM-based approaches leverage past experiences using a full history of
observations, summarization or retrieval augmentation. However, these
unstructured memory representations do not facilitate the reasoning and
planning essential for complex decision-making. In our study, we introduce
AriGraph, a novel method wherein the agent constructs a memory graph that
integrates semantic and episodic memories while exploring the environment. This
graph structure facilitates efficient associative retrieval of interconnected
concepts, relevant to the agent's current state and goals, thus serving as an
effective environmental model that enhances the agent's exploratory and
planning capabilities. We demonstrate that our Ariadne LLM agent, equipped with
this proposed memory architecture augmented with planning and decision-making,
effectively handles complex tasks on a zero-shot basis in the TextWorld
environment. Our approach markedly outperforms established methods such as
full-history, summarization, and Retrieval-Augmented Generation in various
tasks, including the cooking challenge from the First TextWorld Problems
competition and novel tasks like house cleaning and puzzle Treasure Hunting.
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Authors: name: Ahmed Masry, name: Megh Thakkar, name: Aayush Bajaj, name: Aaryaman Kartha, name: Enamul Hoque, name: Shafiq Joty
Published: 2024-07-04 22:16:40

Given the ubiquity of charts as a data analysis, visualization, and
decision-making tool across industries and sciences, there has been a growing
interest in developing pre-trained foundation models as well as general purpose
instruction-tuned models for chart understanding and reasoning. However,
existing methods suffer crucial drawbacks across two critical axes affecting
the performance of chart representation models: they are trained on data
generated from underlying data tables of the charts, ignoring the visual trends
and patterns in chart images, and use weakly aligned vision-language backbone
models for domain-specific training, limiting their generalizability when
encountering charts in the wild. We address these important drawbacks and
introduce ChartGemma, a novel chart understanding and reasoning model developed
over PaliGemma. Rather than relying on underlying data tables, ChartGemma is
trained on instruction-tuning data generated directly from chart images, thus
capturing both high-level trends and low-level visual information from a
diverse set of charts. Our simple approach achieves state-of-the-art results
across 5 benchmarks spanning chart summarization, question answering, and
fact-checking, and our elaborate qualitative studies on real-world charts show
that ChartGemma generates more realistic and factually correct summaries
compared to its contemporaries. We release the code, model checkpoints,
dataset, and demos at https://github.com/vis-nlp/ChartGemma.
CRiM-GS: Continuous Rigid Motion-Aware Gaussian Splatting from Motion
  Blur Images
Authors: name: Junghe Lee, name: Donghyeong Kim, name: Dogyoon Lee, name: Suhwan Cho, name: Sangyoun Lee
Published: 2024-07-04 13:37:04

Neural radiance fields (NeRFs) have received significant attention due to
their high-quality novel view rendering ability, prompting research to address
various real-world cases. One critical challenge is the camera motion blur
caused by camera movement during exposure time, which prevents accurate 3D
scene reconstruction. In this study, we propose continuous rigid motion-aware
gaussian splatting (CRiM-GS) to reconstruct accurate 3D scene from blurry
images with real-time rendering speed. Considering the actual camera motion
blurring process, which consists of complex motion patterns, we predict the
continuous movement of the camera based on neural ordinary differential
equations (ODEs). Specifically, we leverage rigid body transformations to model
the camera motion with proper regularization, preserving the shape and size of
the object. Furthermore, we introduce a continuous deformable 3D transformation
in the SE(3) field to adapt the rigid body transformation to
real-world problems by ensuring a higher degree of freedom. By revisiting
fundamental camera theory and employing advanced neural network training
techniques, we achieve accurate modeling of continuous camera trajectories. We
conduct extensive experiments, demonstrating state-of-the-art performance both
quantitatively and qualitatively on benchmark datasets.
HEMM: Holistic Evaluation of Multimodal Foundation Models
Authors: name: Paul Pu Liang, name: Akshay Goindani, name: Talha Chafekar, name: Leena Mathur, name: Haofei Yu, name: Ruslan Salakhutdinov, name: Louis-Philippe Morency
Published: 2024-07-03 18:00:48

Multimodal foundation models that can holistically process text alongside
images, video, audio, and other sensory modalities are increasingly used in a
variety of real-world applications. However, it is challenging to characterize
and study progress in multimodal foundation models, given the range of possible
modeling decisions, tasks, and domains. In this paper, we introduce Holistic
Evaluation of Multimodal Models (HEMM) to systematically evaluate the
capabilities of multimodal foundation models across a set of 3 dimensions:
basic skills, information flow, and real-world use cases. Basic multimodal
skills are internal abilities required to solve problems, such as learning
interactions across modalities, fine-grained alignment, multi-step reasoning,
and the ability to handle external knowledge. Information flow studies how
multimodal content changes during a task through querying, translation,
editing, and fusion. Use cases span domain-specific challenges introduced in
real-world multimedia, affective computing, natural sciences, healthcare, and
human-computer interaction applications. Through comprehensive experiments
across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g.,
basic skills, information flows, and use cases) that pose challenges to today's
models, and (2) distill performance trends regarding how different modeling
dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training,
and instruction tuning objectives) influence performance. Our conclusions
regarding challenging multimodal interactions, use cases, and tasks requiring
reasoning and external knowledge, the benefits of data and model scale, and the
impacts of instruction tuning yield actionable insights for future work in
multimodal foundation models.
Unveiling Encoder-Free Vision-Language Models
Authors: name: Haiwen Diao, name: Yufeng Cui, name: Xiaotong Li, name: Yueze Wang, name: Huchuan Lu, name: Xinlong Wang
Published: 2024-06-17 17:59:44

Existing vision-language models (VLMs) mostly rely on vision encoders to
extract visual features followed by large language models (LLMs) for
visual-language tasks. However, the vision encoders set a strong inductive bias
in abstracting visual representation, e.g., resolution, aspect ratio, and
semantic priors, which could impede the flexibility and efficiency of the VLMs.
Training pure VLMs that accept the seamless vision and language inputs, i.e.,
without vision encoders, remains challenging and rarely explored. Empirical
observations reveal that direct training without encoders results in slow
convergence and large performance gaps. In this work, we bridge the gap between
encoder-based and encoder-free models, and present a simple yet effective
training recipe towards pure VLMs. Specifically, we unveil the key aspects of
training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
vision-language representation inside one unified decoder; (2) Enhancing visual
recognition capability via extra supervision. With these strategies, we launch
EVE, an encoder-free vision-language model that can be trained and forwarded
efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
impressively rival the encoder-based VLMs of similar capacities across multiple
vision-language benchmarks. It significantly outperforms the counterpart
Fuyu-8B with mysterious training procedures and undisclosed training data. We
believe that EVE provides a transparent and efficient route for developing a
pure decoder-only architecture across modalities. Our code and models are
publicly available at: https://github.com/baaivision/EVE.
FunAudioLLM: Voice Understanding and Generation Foundation Models for
  Natural Interaction Between Humans and LLMs
Authors: name: Tongyi SpeechTeam
Published: 2024-07-04 16:49:02

This report introduces FunAudioLLM, a model family designed to enhance
natural voice interactions between humans and large language models (LLMs). At
its core are two innovative models: SenseVoice, which handles multilingual
speech recognition, emotion recognition, and audio event detection; and
CosyVoice, which facilitates natural speech generation with control over
multiple languages, timbre, speaking style, and speaker identity.
SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and
SenseVoice-Large supports high-precision ASR for over 50 languages, while
CosyVoice excels in multi-lingual voice generation, zero-shot in-context
learning, cross-lingual voice cloning, and instruction-following capabilities.
The models related to SenseVoice and CosyVoice have been open-sourced on
Modelscope and Huggingface, along with the corresponding training, inference,
and fine-tuning codes released on GitHub. By integrating these models with
LLMs, FunAudioLLM enables applications such as speech-to-speech translation,
emotional voice chat, interactive podcasts, and expressive audiobook narration,
thereby pushing the boundaries of voice interaction technology. Demos are
available at https://fun-audio-llm.github.io, and the code can be accessed at
https://github.com/FunAudioLLM.