Orionnnn/hub-stats huggingface dataset:
Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning
  with LLMs
Authors: name: Ronit Singhal, name: Pransh Patwa, name: Parth Patwa, name: Aman Chadha, name: Amitava Das
Published: 2024-08-22 01:42:34

Given the widespread dissemination of misinformation on social media,
implementing fact-checking mechanisms for online claims is essential. Manually
verifying every claim is highly challenging, underscoring the need for an
automated fact-checking system. This paper presents our system designed to
address this issue. We utilize the Averitec dataset to assess the veracity of
claims. In addition to veracity prediction, our system provides supporting
evidence, which is extracted from the dataset. We develop a Retrieve and
Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge
base, which are then inputted along with the claim into a large language model
(LLM) for classification. We also evaluate the few-shot In-Context Learning
(ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of
0.33, which is a 22% absolute improvement over the baseline. All code will be
made available on All code will be made available on
https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.
The Russian-focused embedders' exploration: ruMTEB benchmark and Russian
  embedding model design
Authors: name: Artem Snegirev, name: Maria Tikhonova, name: Anna Maksimova, name: Alena Fenogenova, name: Alexander Abramov
Published: 2024-08-22 15:53:23

Embedding models play a crucial role in Natural Language Processing (NLP) by
creating text embeddings used in various tasks such as information retrieval
and assessing semantic text similarity. This paper focuses on research related
to embedding models in the Russian language. It introduces a new
Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark,
the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our
benchmark includes seven categories of tasks, such as semantic textual
similarity, text classification, reranking, and retrieval. The research also
assesses a representative set of Russian and multilingual models on the
proposed benchmark. The findings indicate that the new model achieves results
that are on par with state-of-the-art models in Russian. We release the model
ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code,
integration into the original framework and a public leaderboard.
Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese
Authors: name: Khang T. Doan, name: Bao G. Huynh, name: Dung T. Hoang, name: Thuc D. Pham, name: Nhat H. Pham, name: Quan T. M. Nguyen, name: Bang Q. Vo, name: Suong N. Hoang
Published: 2024-08-22 15:15:51

In this report, we introduce Vintern-1B, a reliable 1-billion-parameters
multimodal large language model (MLLM) for Vietnamese language tasks. By
integrating the Qwen2-0.5B-Instruct language model with the
InternViT-300M-448px visual model, Vintern-1B is optimized for a range of
applications, including optical character recognition (OCR), document
extraction, and general question-answering in Vietnamese context. The model is
fine-tuned on an extensive dataset of over 3 million image-question-answer
pairs, achieving robust performance and reliable results across multiple
Vietnamese language benchmarks like OpenViVQA and ViTextVQA. Vintern-1B is
small enough to fit into various on-device applications easily. Additionally,
we have open-sourced several Vietnamese vision question answering (VQA)
datasets for text and diagrams, created with Gemini 1.5 Flash. Our models are
available at: https://huggingface.co/5CD-AI/Vintern-1B-v2.
Anim-Director: A Large Multimodal Model Powered Agent for Controllable
  Animation Video Generation
Authors: name: Yunxin Li, name: Haoyuan Shi, name: Baotian Hu, name: Longyue Wang, name: Jiashun Zhu, name: Jinyi Xu, name: Zhen Zhao, name: Min Zhang
Published: 2024-08-19 08:27:31

Traditional animation generation methods depend on training generative models
with human-labelled data, entailing a sophisticated multi-stage pipeline that
demands substantial human effort and incurs high training costs. Due to limited
prompting plans, these methods typically produce brief, information-poor, and
context-incoherent animations. To overcome these limitations and automate the
animation process, we pioneer the introduction of large multimodal models
(LMMs) as the core processor to build an autonomous animation-making agent,
named Anim-Director. This agent mainly harnesses the advanced understanding and
reasoning capabilities of LMMs and generative AI tools to create animated
videos from concise narratives or simple instructions. Specifically, it
operates in three main stages: Firstly, the Anim-Director generates a coherent
storyline from user inputs, followed by a detailed director's script that
encompasses settings of character profiles and interior/exterior descriptions,
and context-coherent scene descriptions that include appearing characters,
interiors or exteriors, and scene events. Secondly, we employ LMMs with the
image generation tool to produce visual images of settings and scenes. These
images are designed to maintain visual consistency across different scenes
using a visual-language prompting method that combines scene descriptions and
images of the appearing character and setting. Thirdly, scene images serve as
the foundation for producing animated videos, with LMMs generating prompts to
guide this process. The whole process is notably autonomous without manual
intervention, as the LMMs interact seamlessly with generative tools to generate
prompts, evaluate visual quality, and select the best one to optimize the final
output.
ConflictBank: A Benchmark for Evaluating the Influence of Knowledge
  Conflicts in LLM
Authors: name: Zhaochen Su, name: Jun Zhang, name: Xiaoye Qu, name: Tong Zhu, name: Yanshu Li, name: Jiashuo Sun, name: Juntao Li, name: Min Zhang, name: Yu Cheng
Published: 2024-08-22 02:33:13

Large language models (LLMs) have achieved impressive advancements across
numerous disciplines, yet the critical issue of knowledge conflicts, a major
source of hallucinations, has rarely been studied. Only a few research explored
the conflicts between the inherent knowledge of LLMs and the retrieved
contextual knowledge. However, a thorough assessment of knowledge conflict in
LLMs is still missing. Motivated by this research gap, we present ConflictBank,
the first comprehensive benchmark developed to systematically evaluate
knowledge conflicts from three aspects: (i) conflicts encountered in retrieved
knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the
interplay between these conflict forms. Our investigation delves into four
model families and twelve LLM instances, meticulously analyzing conflicts
stemming from misinformation, temporal discrepancies, and semantic divergences.
Based on our proposed novel construction framework, we create 7,453,853
claim-evidence pairs and 553,117 QA pairs. We present numerous findings on
model scale, conflict causes, and conflict types. We hope our ConflictBank
benchmark will help the community better understand model behavior in conflicts
and develop more reliable LLMs.
SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for
  Large-scale Vision-Language Models
Authors: name: Youngjoon Yu, name: Sangyun Chung, name: Byung-Kwan Lee, name: Yong Man Ro
Published: 2024-08-22 03:59:48

Large-scale Vision-Language Models (LVLMs) have significantly advanced with
text-aligned vision inputs. They have made remarkable progress in computer
vision tasks by aligning text modality with vision inputs. There are also
endeavors to incorporate multi-vision sensors beyond RGB, including thermal,
depth, and medical X-ray images. However, we observe that current LVLMs view
images taken from multi-vision sensors as if they were in the same RGB domain
without considering the physical characteristics of multi-vision sensors. They
fail to convey the fundamental multi-vision sensor information from the dataset
and the corresponding contextual knowledge properly. Consequently, alignment
between the information from the actual physical environment and the text is
not achieved correctly, making it difficult to answer complex sensor-related
questions that consider the physical environment. In this paper, we aim to
establish a multi-vision Sensor Perception And Reasoning benchmarK called SPARK
that can reduce the fundamental multi-vision sensor information gap between
images and multi-vision sensors. We generated 6,248 vision-language test
samples automatically to investigate multi-vision sensory perception and
multi-vision sensory reasoning on physical sensor knowledge proficiency across
different formats, covering different types of sensor-related questions. We
utilized these samples to assess ten leading LVLMs. The results showed that
most models displayed deficiencies in multi-vision sensory reasoning to varying
extents. Codes and data are available at https://github.com/top-yun/SPARK
DreamCinema: Cinematic Transfer with Free Camera and 3D Character
Authors: name: Weiliang Chen, name: Fangfu Liu, name: Diankun Wu, name: Haowen Sun, name: Haixu Song, name: Yueqi Duan
Published: 2024-08-22 17:59:44

We are living in a flourishing era of digital media, where everyone has the
potential to become a personal filmmaker. Current research on cinematic
transfer empowers filmmakers to reproduce and manipulate the visual elements
(e.g., cinematography and character behaviors) from classic shots. However,
characters in the reimagined films still rely on manual crafting, which
involves significant technical complexity and high costs, making it
unattainable for ordinary users. Furthermore, their estimated cinematography
lacks smoothness due to inadequate capturing of inter-frame motion and modeling
of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC
has opened up the possibility of efficiently generating characters tailored to
users' needs, diversifying cinematography. In this paper, we propose
DreamCinema, a novel cinematic transfer framework that pioneers generative AI
into the film production paradigm, aiming at facilitating user-friendly film
creation. Specifically, we first extract cinematic elements (i.e., human and
camera pose) and optimize the camera trajectory. Then, we apply a character
generator to efficiently create 3D high-quality characters with a human
structure prior. Finally, we develop a structure-guided motion transfer
strategy to incorporate generated characters into film creation and transfer it
via 3D graphics engines smoothly. Extensive experiments demonstrate the
effectiveness of our method for creating high-quality films with free camera
and 3D characters.
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
  Integration in MLLMs
Authors: name: Yuanyang Yin, name: Yaqi Zhao, name: Yajie Zhang, name: Ke Lin, name: Jiahao Wang, name: Xin Tao, name: Pengfei Wan, name: Di Zhang, name: Baoqun Yin, name: Wentao Zhang
Published: 2024-08-21 17:58:02

Multimodal Large Language Models (MLLMs) have recently demonstrated
remarkable perceptual and reasoning abilities, typically comprising a Vision
Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
the critical bridge between the visual and language components. However,
training adapters with image-level supervision often results in significant
misalignment, undermining the LLMs' capabilities and limiting the potential of
Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
(SEA), a token-level alignment method that leverages vision-language
pre-trained models, such as CLIP, to align visual tokens with the LLM's
embedding space through contrastive learning. This approach ensures a more
coherent integration of visual and language representations, enhancing the
performance and interpretability of multimodal LLMs while preserving their
inherent capabilities. Extensive experiments show that SEA effectively improves
MLLMs, particularly for smaller models, without adding extra data or inference
computation. SEA also lays the groundwork for developing more general and
adaptable solutions to enhance multimodal systems.
Real-Time Video Generation with Pyramid Attention Broadcast
Authors: name: Xuanlei Zhao, name: Xiaolong Jin, name: Kai Wang, name: Yang You
Published: 2024-08-22 17:54:21

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and
training-free approach for DiT-based video generation. Our method is founded on
the observation that attention difference in the diffusion process exhibits a
U-shaped pattern, indicating significant redundancy. We mitigate this by
broadcasting attention outputs to subsequent steps in a pyramid style. It
applies different broadcast strategies to each attention based on their
variance for best efficiency. We further introduce broadcast sequence parallel
for more efficient distributed inference. PAB demonstrates superior results
across three models compared to baselines, achieving real-time generation for
up to 720p videos. We anticipate that our simple yet effective method will
serve as a robust baseline and facilitate future research and application for
video generation.
Open-FinLLMs: Open Multimodal Large Language Models for Financial
  Applications
Authors: name: Qianqian Xie, name: Dong Li, name: Mengxi Xiao, name: Zihao Jiang, name: Ruoyu Xiang, name: Xiao Zhang, name: Zhengyu Chen, name: Yueru He, name: Weiguang Han, name: Yuzhe Yang, name: Shunian Chen, name: Yifei Zhang, name: Lihang Shen, name: Daniel Kim, name: Zhiwei Liu, name: Zheheng Luo, name: Yangyang Yu, name: Yupeng Cao, name: Zhiyang Deng, name: Zhiyuan Yao, name: Haohang Li, name: Duanyu Feng, name: Yongfu Dai, name: VijayaSai Somasundaram, name: Peng Lu, name: Yilun Zhao, name: Yitao Long, name: Guojun Xiong, name: Kaleb Smith, name: Honghai Yu, name: Yanzhao Lai, name: Min Peng, name: Jianyun Nie, name: Jordan W. Suchow, name: Xiao-Yang Liu, name: Benyou Wang, name: Alejandro Lopez-Lira, name: Jimin Huang, name: Sophia Ananiadou
Published: 2024-08-20 16:15:28

Large language models (LLMs) have advanced financial applications, yet they
often lack sufficient financial knowledge and struggle with tasks involving
multi-modal inputs like tables and time series data. To address these
limitations, we introduce Open-FinLLMs, a series of Financial LLMs. We
begin with FinLLaMA, pre-trained on a 52 billion token financial corpus,
incorporating text, tables, and time-series data to embed comprehensive
financial knowledge. FinLLaMA is then instruction fine-tuned with 573K
financial instructions, resulting in FinLLaMA-instruct, which enhances task
performance. Finally, we present FinLLaVA, a multimodal LLM trained with 1.43M
image-text instructions to handle complex financial data types. Extensive
evaluations demonstrate FinLLaMA's superior performance over LLaMA3-8B,
LLaMA3.1-8B, and BloombergGPT in both zero-shot and few-shot settings across 19
and 4 datasets, respectively. FinLLaMA-instruct outperforms GPT-4 and other
Financial LLMs on 15 datasets. FinLLaVA excels in understanding tables and
charts across 4 multimodal tasks. Additionally, FinLLaMA achieves impressive
Sharpe Ratios in trading simulations, highlighting its robust financial
application capabilities. We will continually maintain and improve our models
and benchmarks to support ongoing innovation in academia and industry.
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
Authors: name: Jamba Team, name: Barak Lenz, name: Alan Arazi, name: Amir Bergman, name: Avshalom Manevich, name: Barak Peleg, name: Ben Aviram, name: Chen Almagor, name: Clara Fridman, name: Dan Padnos, name: Daniel Gissin, name: Daniel Jannai, name: Dor Muhlgay, name: Dor Zimberg, name: Edden M Gerber, name: Elad Dolev, name: Eran Krakovsky, name: Erez Safahi, name: Erez Schwartz, name: Gal Cohen, name: Gal Shachaf, name: Haim Rozenblum, name: Hofit Bata, name: Ido Blass, name: Inbal Magar, name: Itay Dalmedigos, name: Jhonathan Osin, name: Julie Fadlon, name: Maria Rozman, name: Matan Danos, name: Michael Gokhman, name: Mor Zusman, name: Naama Gidron, name: Nir Ratner, name: Noam Gat, name: Noam Rozen, name: Oded Fried, name: Ohad Leshno, name: Omer Antverg, name: Omri Abend, name: Opher Lieber, name: Or Dagan, name: Orit Cohavi, name: Raz Alon, name: Ro'i Belson, name: Roi Cohen, name: Rom Gilad, name: Roman Glozman, name: Shahar Lev, name: Shaked Meirom, name: Tal Delbari, name: Tal Ness, name: Tomer Asida, name: Tom Ben Gal, name: Tom Braude, name: Uriya Pumerantz, name: Yehoshua Cohen, name: Yonatan Belinkov, name: Yuval Globerson, name: Yuval Peleg Levy, name: Yoav Shoham
Published: 2024-08-22 17:38:59

We present Jamba-1.5, new instruction-tuned large language models based on
our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts
architecture, providing high throughput and low memory usage across context
lengths, while retaining the same or better quality as Transformer models. We
release two model sizes: Jamba-1.5-Large, with 94B active parameters, and
Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a
variety of conversational and instruction-following capabilties, and have an
effective context length of 256K tokens, the largest amongst open-weight
models. To support cost-effective inference, we introduce ExpertsInt8, a novel
quantization technique that allows fitting Jamba-1.5-Large on a machine with 8
80GB GPUs when processing 256K-token contexts without loss of quality. When
evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models
achieve excellent results while providing high throughput and outperforming
other open-weight models on long-context benchmarks. The model weights for both
sizes are publicly available under the Jamba Open Model License and we release
ExpertsInt8 as open source.
Show-o: One Single Transformer to Unify Multimodal Understanding and
  Generation
Authors: name: Jinheng Xie, name: Weijia Mao, name: Zechen Bai, name: David Junhao Zhang, name: Weihao Wang, name: Kevin Qinghong Lin, name: Yuchao Gu, name: Zhijie Chen, name: Zhenheng Yang, name: Mike Zheng Shou
Published: 2024-08-22 16:32:32

We present a unified transformer, i.e., Show-o, that unifies multimodal
understanding and generation. Unlike fully autoregressive models, Show-o
unifies autoregressive and (discrete) diffusion modeling to adaptively handle
inputs and outputs of various and mixed modalities. The unified model flexibly
supports a wide range of vision-language tasks including visual
question-answering, text-to-image generation, text-guided
inpainting/extrapolation, and mixed-modality generation. Across various
benchmarks, it demonstrates comparable or superior performance to existing
individual models with an equivalent or larger number of parameters tailored
for understanding or generation. This significantly highlights its potential as
a next-generation foundation model. Code and models are released at
https://github.com/showlab/Show-o.
Subsurface Scattering for 3D Gaussian Splatting
Authors: name: Jan-Niklas Dihlmann, name: Arjun Majumdar, name: Andreas Engelhardt, name: Raphael Braun, name: Hendrik P. A. Lensch
Published: 2024-08-22 10:34:01

3D reconstruction and relighting of objects made from scattering materials
present a significant challenge due to the complex light transport beneath the
surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at
real-time speeds. While 3D Gaussians efficiently approximate an object's
surface, they fail to capture the volumetric properties of subsurface
scattering. We propose a framework for optimizing an object's shape together
with the radiance transfer field given multi-view OLAT (one light at a time)
data. Our method decomposes the scene into an explicit surface represented as
3D Gaussians, with a spatially varying BRDF, and an implicit volumetric
representation of the scattering component. A learned incident light field
accounts for shadowing. We optimize all parameters jointly via ray-traced
differentiable rendering. Our approach enables material editing, relighting and
novel view synthesis at interactive rates. We show successful application on
synthetic data and introduce a newly acquired multi-view multi-light dataset of
objects in a light-stage setup. Compared to previous work we achieve comparable
or better results at a fraction of optimization and rendering time while
enabling detailed control over material attributes. Project page
https://sss.jdihlmann.com/
Scalable Autoregressive Image Generation with Mamba
Authors: name: Haopeng Li, name: Jinyue Yang, name: Kexin Wang, name: Xuerui Qiu, name: Yuhong Chou, name: Xin Li, name: Guoqi Li
Published: 2024-08-22 09:27:49

We introduce AiM, an autoregressive (AR) image generative model based on
Mamba architecture. AiM employs Mamba, a novel state-space model characterized
by its exceptional performance for long-sequence modeling with linear time
complexity, to supplant the commonly utilized Transformers in AR image
generation models, aiming to achieve both superior generation quality and
enhanced inference speed. Unlike existing methods that adapt Mamba to handle
two-dimensional signals via multi-directional scan, AiM directly utilizes the
next-token prediction paradigm for autoregressive image generation. This
approach circumvents the need for extensive modifications to enable Mamba to
learn 2D spatial representations. By implementing straightforward yet
strategically targeted modifications for visual generative tasks, we preserve
Mamba's core structure, fully exploiting its efficient long-sequence modeling
capabilities and scalability. We provide AiM models in various scales, with
parameter counts ranging from 148M to 1.3B. On the ImageNet1K 256*256
benchmark, our best AiM model achieves a FID of 2.21, surpassing all existing
AR models of comparable parameter counts and demonstrating significant
competitiveness against diffusion models, with 2 to 10 times faster inference
speed. Code is available at https://github.com/hp-l33/AiM
Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event
  Condition For Foley Sound
Authors: name: Junwon Lee, name: Jaekwon Im, name: Dabin Kim, name: Juhan Nam
Published: 2024-08-21 18:06:15

Foley sound synthesis is crucial for multimedia production, enhancing user
experience by synchronizing audio and video both temporally and semantically.
Recent studies on automating this labor-intensive process through
video-to-sound generation face significant challenges. Systems lacking explicit
temporal features suffer from poor controllability and alignment, while
timestamp-based models require costly and subjective human annotation. We
propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as a
temporal event condition with semantic timbre prompts (audio or text). RMS, a
frame-level intensity envelope feature closely related to audio semantics,
ensures high controllability and synchronization. The annotation-free
self-supervised learning framework consists of two stages, Video2RMS and
RMS2Sound, incorporating novel ideas including RMS discretization and
RMS-ControlNet with a pretrained text-to-audio model. Our extensive evaluation
shows that Video-Foley achieves state-of-the-art performance in audio-visual
alignment and controllability for sound timing, intensity, timbre, and nuance.
Code, model weights, and demonstrations are available on the accompanying
website. (https://jnwnlee.github.io/video-foley-demo)