Orionnnn/hub-stats huggingface dataset:
Instruction-tuned Language Models are Better Knowledge Learners
Authors: name: Zhengbao Jiang, name: Zhiqing Sun, name: Weijia Shi, name: Pedro Rodriguez, name: Chunting Zhou, name: Graham Neubig, name: Xi Victoria Lin, name: Wen-tau Yih, name: Srinivasan Iyer
Published: 2024-02-20 09:20:32

In order for large language model (LLM)-based assistants to effectively adapt
to evolving information needs, it must be possible to update their factual
knowledge through continued training on new data. The standard recipe for doing
so involves continued pre-training on new documents followed by
instruction-tuning on question-answer (QA) pairs. However, we find that LLMs
trained with this recipe struggle to answer questions, even though the
perplexity of documents is minimized. We found that QA pairs are generally
straightforward, while documents are more complex, weaving many factual
statements together in an intricate manner. Therefore, we hypothesize that it
is beneficial to expose LLMs to QA pairs before continued pre-training on
documents so that the process of encoding knowledge from complex documents
takes into account how this knowledge is accessed through questions. Based on
this, we propose pre-instruction-tuning (PIT), a method that instruction-tunes
on questions prior to training on documents. This contrasts with standard
instruction-tuning, which learns how to extract knowledge after training on
documents. Extensive experiments and ablation studies demonstrate that PIT
significantly enhances the ability of LLMs to absorb knowledge from new
documents, outperforming standard instruction-tuning by 17.8%.
Improving Robustness for Joint Optimization of Camera Poses and
  Decomposed Low-Rank Tensorial Radiance Fields
Authors: name: Bo-Yu Cheng, name: Wei-Chen Chiu, name: Yu-Lun Liu
Published: 2024-02-20 18:59:02

In this paper, we propose an algorithm that allows joint refinement of camera
pose and scene geometry represented by decomposed low-rank tensor, using only
2D images as supervision. First, we conduct a pilot study based on a 1D signal
and relate our findings to 3D scenarios, where the naive joint pose
optimization on voxel-based NeRFs can easily lead to sub-optimal solutions.
Moreover, based on the analysis of the frequency spectrum, we propose to apply
convolutional Gaussian filters on 2D and 3D radiance fields for a
coarse-to-fine training schedule that enables joint camera pose optimization.
Leveraging the decomposition property in decomposed low-rank tensor, our method
achieves an equivalent effect to brute-force 3D convolution with only incurring
little computational overhead. To further improve the robustness and stability
of joint optimization, we also propose techniques of smoothed 2D supervision,
randomly scaled kernel parameters, and edge-guided loss mask. Extensive
quantitative and qualitative evaluations demonstrate that our proposed
framework achieves superior performance in novel view synthesis as well as
rapid convergence for optimization.
VideoPrism: A Foundational Visual Encoder for Video Understanding
Authors: name: Long Zhao, name: Nitesh B. Gundavarapu, name: Liangzhe Yuan, name: Hao Zhou, name: Shen Yan, name: Jennifer J. Sun, name: Luke Friedman, name: Rui Qian, name: Tobias Weyand, name: Yue Zhao, name: Rachel Hornung, name: Florian Schroff, name: Ming-Hsuan Yang, name: David A. Ross, name: Huisheng Wang, name: Hartwig Adam, name: Mikhail Sirotenko, name: Ting Liu, name: Boqing Gong
Published: 2024-02-20 18:29:49

We introduce VideoPrism, a general-purpose video encoder that tackles diverse
video understanding tasks with a single frozen model. We pretrain VideoPrism on
a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M
video clips with noisy parallel text (e.g., ASR transcripts). The pretraining
approach improves upon masked autoencoding by global-local distillation of
semantic video embeddings and a token shuffling scheme, enabling VideoPrism to
focus primarily on the video modality while leveraging the invaluable text
associated with videos. We extensively test VideoPrism on four broad groups of
video understanding tasks, from web video question answering to CV for science,
achieving state-of-the-art performance on 30 out of 33 video understanding
benchmarks.
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for
  Language Models
Authors: name: Haoran Li, name: Qingxiu Dong, name: Zhengyang Tang, name: Chaojun Wang, name: Xingxing Zhang, name: Haoyang Huang, name: Shaohan Huang, name: Xiaolong Huang, name: Zeqiang Huang, name: Dongdong Zhang, name: Yuxian Gu, name: Xin Cheng, name: Xun Wang, name: Si-Qing Chen, name: Li Dong, name: Wei Lu, name: Zhifang Sui, name: Benyou Wang, name: Wai Lam, name: Furu Wei
Published: 2024-02-20 15:00:35

We introduce Generalized Instruction Tuning (called GLAN), a general and
scalable method for instruction tuning of Large Language Models (LLMs). Unlike
prior work that relies on seed examples or existing datasets to construct
instruction tuning data, GLAN exclusively utilizes a pre-curated taxonomy of
human knowledge and capabilities as input and generates large-scale synthetic
instruction data across all disciplines. Specifically, inspired by the
systematic structure in human education system, we build the taxonomy by
decomposing human knowledge and capabilities to various fields, sub-fields and
ultimately, distinct disciplines semi-automatically, facilitated by LLMs.
Subsequently, we generate a comprehensive list of subjects for every discipline
and proceed to design a syllabus tailored to each subject, again utilizing
LLMs. With the fine-grained key concepts detailed in every class session of the
syllabus, we are able to generate diverse instructions with a broad coverage
across the entire spectrum of human knowledge and skills. Extensive experiments
on large language models (e.g., Mistral) demonstrate that GLAN excels in
multiple dimensions from mathematical reasoning, coding, academic exams,
logical reasoning to general instruction following without using task-specific
training data of these tasks. In addition, GLAN allows for easy customization
and new fields or skills can be added by simply incorporating a new node into
our taxonomy.
Video ReCap: Recursive Captioning of Hour-Long Videos
Authors: name: Md Mohaiminul Islam, name: Ngan Ho, name: Xitong Yang, name: Tushar Nagarajan, name: Lorenzo Torresani, name: Gedas Bertasius
Published: 2024-02-20 18:58:54

Most video captioning models are designed to process short video clips of few
seconds and output text describing low-level visual concepts (e.g., objects,
scenes, atomic actions). However, most real-world videos last for minutes or
hours and have a complex hierarchical structure spanning different temporal
granularities. We propose Video ReCap, a recursive video captioning model that
can process video inputs of dramatically different lengths (from 1 second to 2
hours) and output video captions at multiple hierarchy levels. The recursive
video-language architecture exploits the synergy between different video
hierarchies and can process hour-long videos efficiently. We utilize a
curriculum learning training scheme to learn the hierarchical structure of
videos, starting from clip-level captions describing atomic actions, then
focusing on segment-level descriptions, and concluding with generating
summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by
augmenting Ego4D with 8,267 manually collected long-range video summaries. Our
recursive model can flexibly generate captions at different hierarchy levels
while also being useful for other complex video understanding tasks, such as
VideoQA on EgoSchema. Data, code, and models are available at:
https://sites.google.com/view/vidrecap
TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue
  Summarization
Authors: name: Liyan Tang, name: Igor Shalyminov, name: Amy Wing-mei Wong, name: Jon Burnsky, name: Jake W. Vincent, name: Yu'an Yang, name: Siffi Singh, name: Song Feng, name: Hwanjun Song, name: Hang Su, name: Lijia Sun, name: Yi Zhang, name: Saab Mansour, name: Kathleen McKeown
Published: 2024-02-20 18:58:49

Single document news summarization has seen substantial progress on
faithfulness in recent years, driven by research on the evaluation of factual
consistency, or hallucinations. We ask whether these advances carry over to
other text summarization domains. We propose a new evaluation benchmark on
topic-focused dialogue summarization, generated by LLMs of varying sizes. We
provide binary sentence-level human annotations of the factual consistency of
these summaries along with detailed explanations of factually inconsistent
sentences. Our analysis shows that existing LLMs hallucinate significant
amounts of factual errors in the dialogue domain, regardless of the model's
size. On the other hand, when LLMs, including GPT-4, serve as binary factual
evaluators, they perform poorly and can be outperformed by prevailing
state-of-the-art specialized factuality evaluation metrics. Finally, we
conducted an analysis of hallucination types with a curated error taxonomy. We
find that there are diverse errors and error distributions in model-generated
summaries and that non-LLM based metrics can capture all error types better
than LLM-based evaluators.
A Touch, Vision, and Language Dataset for Multimodal Alignment
Authors: name: Letian Fu, name: Gaurav Datta, name: Huang Huang, name: William Chung-Ho Panitch, name: Jaimyn Drake, name: Joseph Ortiz, name: Mustafa Mukadam, name: Mike Lambeta, name: Roberto Calandra, name: Ken Goldberg
Published: 2024-02-20 18:47:56

Touch is an important sensing modality for humans, but it has not yet been
incorporated into a multimodal generative language model. This is partially due
to the difficulty of obtaining natural language labels for tactile data and the
complexity of aligning tactile readings with both visual observations and
language descriptions. As a step towards bridging that gap, this work
introduces a new dataset of 44K in-the-wild vision-touch pairs, with English
language labels annotated by humans (10%) and textual pseudo-labels from GPT-4V
(90%). We use this dataset to train a vision-language-aligned tactile encoder
for open-vocabulary classification and a touch-vision-language (TVL) model for
text generation using the trained encoder. Results suggest that by
incorporating touch, the TVL model improves (+29% classification accuracy)
touch-vision-language alignment over existing models trained on any pair of
those modalities. Although only a small fraction of the dataset is
human-labeled, the TVL model demonstrates improved visual-tactile understanding
over GPT-4V (+12%) and open-source vision-language models (+32%) on a new
touch-vision understanding benchmark. Code and data:
https://tactile-vlm.github.io.
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on
  Deceptive Prompts
Authors: name: Yusu Qian, name: Haotian Zhang, name: Yinfei Yang, name: Zhe Gan
Published: 2024-02-20 18:31:27

The remarkable advancements in Multimodal Large Language Models (MLLMs) have
not rendered them immune to challenges, particularly in the context of handling
deceptive information in prompts, thus producing hallucinated responses under
such conditions. To quantitatively assess this vulnerability, we present
MAD-Bench, a carefully curated benchmark that contains 850 test samples divided
into 6 categories, such as non-existent objects, count of objects, spatial
relationship, and visual confusion. We provide a comprehensive analysis of
popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as
LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps
between GPT-4V and other models; and previous robust instruction-tuned models,
such as LRV-Instruction and LLaVA-RLHF, are not effective on this new
benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of
any other model in our experiments ranges from 5% to 35%. We further propose a
remedy that adds an additional paragraph to the deceptive prompts to encourage
models to think twice before answering the question. Surprisingly, this simple
method can even double the accuracy; however, the absolute numbers are still
too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark
to stimulate further research to enhance models' resilience against deceptive
prompts.
The FinBen: An Holistic Financial Benchmark for Large Language Models
Authors: name: Qianqian Xie, name: Weiguang Han, name: Zhengyu Chen, name: Ruoyu Xiang, name: Xiao Zhang, name: Yueru He, name: Mengxi Xiao, name: Dong Li, name: Yongfu Dai, name: Duanyu Feng, name: Yijing Xu, name: Haoqiang Kang, name: Ziyan Kuang, name: Chenhan Yuan, name: Kailai Yang, name: Zheheng Luo, name: Tianlin Zhang, name: Zhiwei Liu, name: Guojun Xiong, name: Zhiyang Deng, name: Yuechen Jiang, name: Zhiyuan Yao, name: Haohang Li, name: Yangyang Yu, name: Gang Hu, name: Jiajia Huang, name: Xiao-Yang Liu, name: Alejandro Lopez-Lira, name: Benyou Wang, name: Yanzhao Lai, name: Hao Wang, name: Min Peng, name: Sophia Ananiadou, name: Jimin Huang
Published: 2024-02-20 02:16:16

LLMs have transformed NLP and shown promise in various fields, yet their
potential in finance is underexplored due to a lack of thorough evaluations and
the complexity of financial tasks. This along with the rapid development of
LLMs, highlights the urgent need for a systematic financial evaluation
benchmark for LLMs. In this paper, we introduce FinBen, the first comprehensive
open-sourced evaluation benchmark, specifically designed to thoroughly assess
the capabilities of LLMs in the financial domain. FinBen encompasses 35
datasets across 23 financial tasks, organized into three spectrums of
difficulty inspired by the Cattell-Horn-Carroll theory, to evaluate LLMs'
cognitive abilities in inductive reasoning, associative memory, quantitative
reasoning, crystallized intelligence, and more. Our evaluation of 15
representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals
insights into their strengths and limitations within the financial domain. The
findings indicate that GPT-4 leads in quantification, extraction, numerical
reasoning, and stock trading, while Gemini shines in generation and
forecasting; however, both struggle with complex extraction and forecasting,
showing a clear need for targeted enhancements. Instruction tuning boosts
simple task performance but falls short in improving complex reasoning and
forecasting abilities. FinBen seeks to continuously evaluate LLMs in finance,
fostering AI development with regular updates of tasks and models.
MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for
  Single or Sparse-view 3D Object Reconstruction
Authors: name: Shitao Tang, name: Jiacheng Chen, name: Dilin Wang, name: Chengzhou Tang, name: Fuyang Zhang, name: Yuchen Fan, name: Vikas Chandra, name: Yasutaka Furukawa, name: Rakesh Ranjan
Published: 2024-02-20 04:25:57

This paper presents a neural architecture MVDiffusion++ for 3D object
reconstruction that synthesizes dense and high-resolution views of an object
given one or a few images without camera poses. MVDiffusion++ achieves superior
flexibility and scalability with two surprisingly simple ideas: 1) A
``pose-free architecture'' where standard self-attention among 2D latent
features learns 3D consistency across an arbitrary number of conditional and
generation views without explicitly using camera pose information; and 2) A
``view dropout strategy'' that discards a substantial number of output views
during training, which reduces the training-time memory footprint and enables
dense and high-resolution view synthesis at test time. We use the Objaverse for
training and the Google Scanned Objects for evaluation with standard novel view
synthesis and 3D reconstruction metrics, where MVDiffusion++ significantly
outperforms the current state of the arts. We also demonstrate a text-to-3D
application example by combining MVDiffusion++ with a text-to-image generative
model.
RealCompo: Dynamic Equilibrium between Realism and Compositionality
  Improves Text-to-Image Diffusion Models
Authors: name: Xinchen Zhang, name: Ling Yang, name: Yaqi Cai, name: Zhaochen Yu, name: Jiake Xie, name: Ye Tian, name: Minkai Xu, name: Yong Tang, name: Yujiu Yang, name: Bin Cui
Published: 2024-02-20 10:56:52

Diffusion models have achieved remarkable advancements in text-to-image
generation. However, existing models still have many difficulties when faced
with multiple-object compositional generation. In this paper, we propose a new
training-free and transferred-friendly text-to-image generation framework,
namely RealCompo, which aims to leverage the advantages of text-to-image and
layout-to-image models to enhance both realism and compositionality of the
generated images. An intuitive and novel balancer is proposed to dynamically
balance the strengths of the two models in denoising process, allowing
plug-and-play use of any model without extra training. Extensive experiments
show that our RealCompo consistently outperforms state-of-the-art text-to-image
models and layout-to-image models in multiple-object compositional generation
while keeping satisfactory realism and compositionality of the generated
images. Code is available at https://github.com/YangLing0818/RealCompo
Neural Network Diffusion
Authors: name: Kai Wang, name: Zhaopan Xu, name: Yukun Zhou, name: Zelin Zang, name: Trevor Darrell, name: Zhuang Liu, name: Yang You
Published: 2024-02-20 16:59:03

Diffusion models have achieved remarkable success in image and video
generation. In this work, we demonstrate that diffusion models can also
generate high-performing neural network parameters. Our approach is
simple, utilizing an autoencoder and a standard latent diffusion model. The
autoencoder extracts latent representations of a subset of the trained network
parameters. A diffusion model is then trained to synthesize these latent
parameter representations from random noise. It then generates new
representations that are passed through the autoencoder's decoder, whose
outputs are ready to use as new subsets of network parameters. Across various
architectures and datasets, our diffusion process consistently generates models
of comparable or improved performance over trained networks, with minimal
additional cost. Notably, we empirically find that the generated models perform
differently with the trained networks. Our results encourage more exploration
on the versatile use of diffusion models.
FiT: Flexible Vision Transformer for Diffusion Model
Authors: name: Zeyu Lu, name: Zidong Wang, name: Di Huang, name: Chengyue Wu, name: Xihui Liu, name: Wanli Ouyang, name: Lei Bai
Published: 2024-02-19 18:59:07

Nature is infinitely resolution-free. In the context of this reality,
existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To overcome this limitation, we present the Flexible Vision Transformer (FiT),
a transformer architecture specifically designed for generating images with
unrestricted resolutions and aspect ratios. Unlike traditional methods that
perceive images as static-resolution grids, FiT conceptualizes images as
sequences of dynamically-sized tokens. This perspective enables a flexible
training strategy that effortlessly adapts to diverse aspect ratios during both
training and inference phases, thus promoting resolution generalization and
eliminating biases induced by image cropping. Enhanced by a meticulously
adjusted network structure and the integration of training-free extrapolation
techniques, FiT exhibits remarkable flexibility in resolution extrapolation
generation. Comprehensive experiments demonstrate the exceptional performance
of FiT across a broad range of resolutions, showcasing its effectiveness both
within and beyond its training resolution distribution. Repository available at
https://github.com/whlzy/FiT.
Binary Opacity Grids: Capturing Fine Geometric Detail for Mesh-Based
  View Synthesis
Authors: name: Christian Reiser, name: Stephan Garbin, name: Pratul P. Srinivasan, name: Dor Verbin, name: Richard Szeliski, name: Ben Mildenhall, name: Jonathan T. Barron, name: Peter Hedman, name: Andreas Geiger
Published: 2024-02-19 18:59:41

While surface-based view synthesis algorithms are appealing due to their low
computational requirements, they often struggle to reproduce thin structures.
In contrast, more expensive methods that model the scene's geometry as a
volumetric density field (e.g. NeRF) excel at reconstructing fine geometric
detail. However, density fields often represent geometry in a "fuzzy" manner,
which hinders exact localization of the surface. In this work, we modify
density fields to encourage them to converge towards surfaces, without
compromising their ability to reconstruct thin structures. First, we employ a
discrete opacity grid representation instead of a continuous density field,
which allows opacity values to discontinuously transition from zero to one at
the surface. Second, we anti-alias by casting multiple rays per pixel, which
allows occlusion boundaries and subpixel structures to be modelled without
using semi-transparent voxels. Third, we minimize the binary entropy of the
opacity values, which facilitates the extraction of surface geometry by
encouraging opacity values to binarize towards the end of training. Lastly, we
develop a fusion-based meshing strategy followed by mesh simplification and
appearance model fitting. The compact meshes produced by our model can be
rendered in real-time on mobile devices and achieve significantly higher view
synthesis quality compared to existing mesh-based approaches.
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Authors: name: Jun Zhan, name: Junqi Dai, name: Jiasheng Ye, name: Yunhua Zhou, name: Dong Zhang, name: Zhigeng Liu, name: Xin Zhang, name: Ruibin Yuan, name: Ge Zhang, name: Linyang Li, name: Hang Yan, name: Jie Fu, name: Tao Gui, name: Tianxiang Sun, name: Yugang Jiang, name: Xipeng Qiu
Published: 2024-02-19 15:33:10

We introduce AnyGPT, an any-to-any multimodal language model that utilizes
discrete representations for the unified processing of various modalities,
including speech, text, images, and music. AnyGPT can be trained stably without
any alterations to the current large language model (LLM) architecture or
training paradigms. Instead, it relies exclusively on data-level preprocessing,
facilitating the seamless integration of new modalities into LLMs, akin to the
incorporation of new languages. We build a multimodal text-centric dataset for
multimodal alignment pre-training. Utilizing generative models, we synthesize
the first large-scale any-to-any multimodal instruction dataset. It consists of
108k samples of multi-turn conversations that intricately interweave various
modalities, thus equipping the model to handle arbitrary combinations of
multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is
capable of facilitating any-to-any multimodal conversation while achieving
performance comparable to specialized models across all modalities, proving
that discrete representations can effectively and conveniently unify multiple
modalities within a language model. Demos are shown in
https://junzhan2000.github.io/AnyGPT.github.io/