Orionnnn/hub-stats huggingface dataset:
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal
  Large Language Model
Authors: name: Yiwei Ma, name: Zhibin Wang, name: Xiaoshuai Sun, name: Weihuang Lin, name: Qiang Zhou, name: Jiayi Ji, name: Rongrong Ji
Published: 2024-07-23 06:02:30

With advancements in data availability and computing resources, Multimodal
Large Language Models (MLLMs) have showcased capabilities across various
fields. However, the quadratic complexity of the vision encoder in MLLMs
constrains the resolution of input images. Most current approaches mitigate
this issue by cropping high-resolution images into smaller sub-images, which
are then processed independently by the vision encoder. Despite capturing
sufficient local details, these sub-images lack global context and fail to
interact with one another. To address this limitation, we propose a novel MLLM,
INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA
incorporates two innovative components. First, we introduce a Dual-perspective
Cropping Module (DCM), which ensures that each sub-image contains continuous
details from a local perspective and comprehensive information from a global
perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to
enable the mutual enhancement of global and local features, allowing INF-LLaVA
to effectively process high-resolution images by simultaneously capturing
detailed local information and comprehensive global context. Extensive ablation
studies validate the effectiveness of these components, and experiments on a
diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing
MLLMs. Code and pretrained model are available at
https://github.com/WeihuangLin/INF-LLaVA.
F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions
Authors: name: Jie Yang, name: Xuesong Niu, name: Nan Jiang, name: Ruimao Zhang, name: Siyuan Huang
Published: 2024-07-17 09:43:58

Existing 3D human object interaction (HOI) datasets and models simply align
global descriptions with the long HOI sequence, while lacking a detailed
understanding of intermediate states and the transitions between states. In
this paper, we argue that fine-grained semantic alignment, which utilizes
state-level descriptions, offers a promising paradigm for learning semantically
rich HOI representations. To achieve this, we introduce Semantic-HOI, a new
dataset comprising over 20K paired HOI states with fine-grained descriptions
for each HOI state and the body movements that happen between two consecutive
states. Leveraging the proposed dataset, we design three state-level HOI tasks
to accomplish fine-grained semantic alignment within the HOI sequence.
Additionally, we propose a unified model called F-HOI, designed to leverage
multimodal instructions and empower the Multi-modal Large Language Model to
efficiently handle diverse HOI tasks. F-HOI offers multiple advantages: (1) It
employs a unified task formulation that supports the use of versatile
multimodal inputs. (2) It maintains consistency in HOI across 2D, 3D, and
linguistic spaces. (3) It utilizes fine-grained textual supervision for direct
optimization, avoiding intricate modeling of HOI states. Extensive experiments
reveal that F-HOI effectively aligns HOI states with fine-grained semantic
descriptions, adeptly tackling understanding, reasoning, generation, and
reconstruction tasks.
MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Authors: name: Canyu Zhao, name: Mingyu Liu, name: Wen Wang, name: Jianlong Yuan, name: Hao Chen, name: Bo Zhang, name: Chunhua Shen
Published: 2024-07-23 17:17:05

Recent advancements in video generation have primarily leveraged diffusion
models for short-duration content. However, these approaches often fall short
in modeling complex narratives and maintaining character consistency over
extended periods, which is essential for long-form video production like
movies. We propose MovieDreamer, a novel hierarchical framework that integrates
the strengths of autoregressive models with diffusion-based rendering to
pioneer long-duration video generation with intricate plot progressions and
high visual fidelity. Our approach utilizes autoregressive models for global
narrative coherence, predicting sequences of visual tokens that are
subsequently transformed into high-quality video frames through diffusion
rendering. This method is akin to traditional movie production processes, where
complex stories are factorized down into manageable scene capturing. Further,
we employ a multimodal script that enriches scene descriptions with detailed
character information and visual style, enhancing continuity and character
identity across scenes. We present extensive experiments across various movie
genres, demonstrating that our approach not only achieves superior visual and
narrative quality but also effectively extends the duration of generated
content significantly beyond current capabilities. Homepage:
https://aim-uofa.github.io/MovieDreamer/.
CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis
Authors: name: Junying Chen, name: Chi Gui, name: Anningzhe Gao, name: Ke Ji, name: Xidong Wang, name: Xiang Wan, name: Benyou Wang
Published: 2024-07-18 09:06:27

The field of medical diagnosis has undergone a significant transformation
with the advent of large language models (LLMs), yet the challenges of
interpretability within these models remain largely unaddressed. This study
introduces Chain-of-Diagnosis (CoD) to enhance the interpretability of
LLM-based medical diagnostics. CoD transforms the diagnostic process into a
diagnostic chain that mirrors a physician's thought process, providing a
transparent reasoning pathway. Additionally, CoD outputs the disease confidence
distribution to ensure transparency in decision-making. This interpretability
makes model diagnostics controllable and aids in identifying critical symptoms
for inquiry through the entropy reduction of confidences. With CoD, we
developed DiagnosisGPT, capable of diagnosing 9604 diseases. Experimental
results demonstrate that DiagnosisGPT outperforms other LLMs on diagnostic
benchmarks. Moreover, DiagnosisGPT provides interpretability while ensuring
controllability in diagnostic rigor.
Compact Language Models via Pruning and Knowledge Distillation
Authors: name: Saurav Muralidharan, name: Sharath Turuvekere Sreenivas, name: Raviraj Joshi, name: Marcin Chochowski, name: Mostofa Patwary, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Jan Kautz, name: Pavlo Molchanov
Published: 2024-07-19 21:47:57

Large language models (LLMs) targeting different deployment scales and sizes
are currently produced by training each variant from scratch; this is extremely
compute-intensive. In this paper, we investigate if pruning an existing LLM and
then re-training it with a fraction (<3%) of the original training data can be
a suitable alternative to repeated, full retraining. To this end, we develop a
set of practical and effective compression best practices for LLMs that combine
depth, width, attention and MLP pruning with knowledge distillation-based
retraining; we arrive at these best practices through a detailed empirical
exploration of pruning strategies for each axis, methods to combine axes,
distillation strategies, and search techniques for arriving at optimal
compressed architectures. We use this guide to compress the Nemotron-4 family
of LLMs by a factor of 2-4x, and compare their performance to similarly-sized
models on a variety of language modeling tasks. Deriving 8B and 4B models from
an already pretrained 15B model using our approach requires up to 40x fewer
training tokens per model compared to training from scratch; this results in
compute cost savings of 1.8x for training the full model family (15B, 8B, and
4B). Minitron models exhibit up to a 16% improvement in MMLU scores compared to
training from scratch, perform comparably to other community models such as
Mistral 7B, Gemma 7B and Llama-3 8B, and outperform state-of-the-art
compression techniques from the literature. We have open-sourced Minitron model
weights on Huggingface, with corresponding supplementary material including
example code available on GitHub.
Visual Haystacks: Answering Harder Questions About Sets of Images
Authors: name: Tsung-Han Wu, name: Giscard Biamby, name: Jerome Quenum, name: Ritwik Gupta, name: Joseph E. Gonzalez, name: Trevor Darrell, name: David M. Chan
Published: 2024-07-18 17:59:30

Recent advancements in Large Multimodal Models (LMMs) have made significant
progress in the field of single-image visual question answering. However, these
models face substantial challenges when tasked with queries that span extensive
collections of images, similar to real-world scenarios like searching through
large photo albums, finding specific information across the internet, or
monitoring environmental changes through satellite imagery. This paper explores
the task of Multi-Image Visual Question Answering (MIQA): given a large set of
images and a natural language query, the task is to generate a relevant and
grounded response. We propose a new public benchmark, dubbed "Visual Haystacks
(VHs)," specifically designed to evaluate LMMs' capabilities in visual
retrieval and reasoning over sets of unrelated images, where we perform
comprehensive evaluations demonstrating that even robust closed-source models
struggle significantly. Towards addressing these shortcomings, we introduce
MIRAGE (Multi-Image Retrieval Augmented Generation), a novel retrieval/QA
framework tailored for LMMs that confronts the challenges of MIQA with marked
efficiency and accuracy improvements over baseline methods. Our evaluation
shows that MIRAGE surpasses closed-source GPT-4o models by up to 11% on the VHs
benchmark and offers up to 3.4x improvements in efficiency over text-focused
multi-stage approaches.
CGB-DM: Content and Graphic Balance Layout Generation with
  Transformer-based Diffusion Model
Authors: name: Yu Li, name: Yifan Chen, name: Gongye Liu, name: Jie Wu, name: Yujiu Yang
Published: 2024-07-21 17:58:21

Layout generation is the foundation task of intelligent design, which
requires the integration of visual aesthetics and harmonious expression of
content delivery. However, existing methods still face challenges in generating
precise and visually appealing layouts, including blocking, overlap, or spatial
misalignment between layouts, which are closely related to the spatial
structure of graphic layouts. We find that these methods overly focus on
content information and lack constraints on layout spatial structure, resulting
in an imbalance of learning content-aware and graphic-aware features. To tackle
this issue, we propose Content and Graphic Balance Layout Generation with
Transformer-based Diffusion Model (CGB-DM). Specifically, we first design a
regulator that balances the predicted content and graphic weight, overcoming
the tendency of paying more attention to the content on canvas. Secondly, we
introduce a graphic constraint of saliency bounding box to further enhance the
alignment of geometric features between layout representations and images. In
addition, we adapt a transformer-based diffusion model as the backbone, whose
powerful generation capability ensures the quality in layout generation.
Extensive experimental results indicate that our method has achieved
state-of-the-art performance in both quantitative and qualitative evaluations.
Our model framework can also be expanded to other graphic design fields.
Discrete Flow Matching
Authors: name: Itai Gat, name: Tal Remez, name: Neta Shaul, name: Felix Kreuk, name: Ricky T. Q. Chen, name: Gabriel Synnaeve, name: Yossi Adi, name: Yaron Lipman
Published: 2024-07-22 12:33:27

Despite Flow Matching and diffusion models having emerged as powerful
generative paradigms for continuous variables such as images and videos, their
application to high-dimensional discrete data, such as language, is still
limited. In this work, we present Discrete Flow Matching, a novel discrete flow
paradigm designed specifically for generating discrete data. Discrete Flow
Matching offers several key contributions: (i) it works with a general family
of probability paths interpolating between source and target distributions;
(ii) it allows for a generic formula for sampling from these probability paths
using learned posteriors such as the probability denoiser (x-prediction) and
noise-prediction (epsilon-prediction); (iii) practically, focusing on
specific probability paths defined with different schedulers considerably
improves generative perplexity compared to previous discrete diffusion and flow
models; and (iv) by scaling Discrete Flow Matching models up to 1.7B
parameters, we reach 6.7% Pass@1 and 13.4% Pass@10 on HumanEval and 6.7% Pass@1
and 20.6% Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of
generating high-quality discrete data in a non-autoregressive fashion,
significantly closing the gap between autoregressive models and discrete flow
models.
Consent in Crisis: The Rapid Decline of the AI Data Commons
Authors: name: Shayne Longpre, name: Robert Mahari, name: Ariel Lee, name: Campbell Lund, name: Hamidah Oderinwale, name: William Brannon, name: Nayan Saxena, name: Naana Obeng-Marnu, name: Tobin South, name: Cole Hunter, name: Kevin Klyman, name: Christopher Klamm, name: Hailey Schoelkopf, name: Nikhil Singh, name: Manuel Cherep, name: Ahmad Anis, name: An Dinh, name: Caroline Chitongo, name: Da Yin, name: Damien Sileo, name: Deividas Mataciunas, name: Diganta Misra, name: Emad Alghamdi, name: Enrico Shippole, name: Jianguo Zhang, name: Joanna Materzynska, name: Kun Qian, name: Kush Tiwary, name: Lester Miranda, name: Manan Dey, name: Minnie Liang, name: Mohammed Hamdy, name: Niklas Muennighoff, name: Seonghyeon Ye, name: Seungone Kim, name: Shrestha Mohanty, name: Vipul Gupta, name: Vivek Sharma, name: Vu Minh Chien, name: Xuhui Zhou, name: Yizhi Li, name: Caiming Xiong, name: Luis Villa, name: Stella Biderman, name: Hanlin Li, name: Daphne Ippolito, name: Sara Hooker, name: Jad Kabbara, name: Sandy Pentland
Published: 2024-07-20 16:50:18

General-purpose artificial intelligence (AI) systems are built on massive
swathes of public web data, assembled into corpora such as C4, RefinedWeb, and
Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit
of the consent protocols for the web domains underlying AI training corpora.
Our audit of 14,000 web domains provides an expansive view of crawlable web
data and how consent preferences to use it are changing over time. We observe a
proliferation of AI-specific clauses to limit use, acute differences in
restrictions on AI developers, as well as general inconsistencies between
websites' expressed intentions in their Terms of Service and their robots.txt.
We diagnose these as symptoms of ineffective web protocols, not designed to
cope with the widespread re-purposing of the internet for AI. Our longitudinal
analyses show that in a single year (2023-2024) there has been a rapid
crescendo of data restrictions from web sources, rendering ~5%+ of all tokens
in C4, or 28%+ of the most actively maintained, critical sources in C4, fully
restricted from use. For Terms of Service crawling restrictions, a full 45% of
C4 is now restricted. If respected or enforced, these restrictions are rapidly
biasing the diversity, freshness, and scaling laws for general-purpose AI
systems. We hope to illustrate the emerging crisis in data consent, foreclosing
much of the open web, not only for commercial AI, but non-commercial AI and
academic purposes.
MIBench: Evaluating Multimodal Large Language Models over Multiple
  Images
Authors: name: Haowei Liu, name: Xi Zhang, name: Haiyang Xu, name: Yaya Shi, name: Chaoya Jiang, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Chunfeng Yuan, name: Bing Li, name: Weiming Hu
Published: 2024-07-21 21:22:58

Built on the power of LLMs, numerous multimodal large language models (MLLMs)
have recently achieved remarkable performance on various vision-language tasks
across multiple benchmarks. However, most existing MLLMs and benchmarks
primarily focus on single-image input scenarios, leaving the performance of
MLLMs when handling realistic multiple images remain underexplored. Although a
few benchmarks consider multiple images, their evaluation dimensions and
samples are very limited. Therefore, in this paper, we propose a new benchmark
MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in
multi-image scenarios. Specifically, MIBench categorizes the multi-image
abilities into three scenarios: multi-image instruction (MII), multimodal
knowledge-seeking (MKS) and multimodal in-context learning (MIC), and
constructs 13 tasks with a total of 13K annotated samples. During data
construction, for MII and MKS, we extract correct options from manual
annotations and create challenging distractors to obtain multiple-choice
questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and
transform the original datasets into in-context learning formats. We evaluate
several open-source MLLMs and close-source MLLMs on the proposed MIBench. The
results reveal that although current models excel in single-image tasks, they
exhibit significant shortcomings when faced with multi-image inputs, such as
confused fine-grained perception, limited multi-image reasoning, and unstable
in-context learning. The annotated data in MIBench is available at
https://huggingface.co/datasets/StarBottle/MIBench.
BOND: Aligning LLMs with Best-of-N Distillation
Authors: name: Pier Giuseppe Sessa, name: Robert Dadashi, name: Léonard Hussenot, name: Johan Ferret, name: Nino Vieillard, name: Alexandre Ramé, name: Bobak Shariari, name: Sarah Perrin, name: Abe Friesen, name: Geoffrey Cideron, name: Sertan Girgin, name: Piotr Stanczyk, name: Andrea Michi, name: Danila Sinopalnikov, name: Sabela Ramos, name: Amélie Héliou, name: Aliaksei Severyn, name: Matt Hoffman, name: Nikola Momchev, name: Olivier Bachem
Published: 2024-07-19 18:38:25

Reinforcement learning from human feedback (RLHF) is a key driver of quality
and safety in state-of-the-art large language models. Yet, a surprisingly
simple and strong inference-time strategy is Best-of-N sampling that selects
the best generation among N candidates. In this paper, we propose Best-of-N
Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
without its significant computational overhead at inference time. Specifically,
BOND is a distribution matching algorithm that forces the distribution of
generations from the policy to get closer to the Best-of-N distribution. We use
the Jeffreys divergence (a linear combination of forward and backward KL) to
balance between mode-covering and mode-seeking behavior, and derive an
iterative formulation that utilizes a moving anchor for efficiency. We
demonstrate the effectiveness of our approach and several design choices
through experiments on abstractive summarization and Gemma models. Aligning
Gemma policies with BOND outperforms other RLHF algorithms by improving results
on several benchmarks.
AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
Authors: name: Ori Yoran, name: Samuel Joseph Amouyal, name: Chaitanya Malaviya, name: Ben Bogin, name: Ofir Press, name: Jonathan Berant
Published: 2024-07-22 15:18:45

Language agents, built on top of language models (LMs), are systems that can
interact with complex environments, such as the open web. In this work, we
examine whether such agents can perform realistic and time-consuming tasks on
the web, e.g., monitoring real-estate markets or locating relevant nearby
businesses. We introduce AssistantBench, a challenging new benchmark consisting
of 214 realistic tasks that can be automatically evaluated, covering different
scenarios and domains. We find that AssistantBench exposes the limitations of
current systems, including language models and retrieval-augmented language
models, as no model reaches an accuracy of more than 25 points. While
closed-book LMs perform well, they exhibit low precision since they tend to
hallucinate facts. State-of-the-art web agents reach a score of near zero.
Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
outperforms previous agents, and an ensemble of SPA and closed-book models
reaches the best overall performance. Moreover, we analyze failures of current
systems and highlight that web navigation remains a major challenge.
BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis
  in Large-scale Scenes
Authors: name: Chih-Hai Su, name: Chih-Yao Hu, name: Shr-Ruei Tsai, name: Jie-Ying Lee, name: Chin-Yang Lin, name: Yu-Lun Liu
Published: 2024-07-22 17:59:46

While Neural Radiance Fields (NeRFs) have demonstrated exceptional quality,
their protracted training duration remains a limitation. Generalizable and
MVS-based NeRFs, although capable of mitigating training time, often incur
tradeoffs in quality. This paper presents a novel approach called BoostMVSNeRFs
to enhance the rendering quality of MVS-based NeRFs in large-scale scenes. We
first identify limitations in MVS-based NeRF methods, such as restricted
viewport coverage and artifacts due to limited input views. Then, we address
these limitations by proposing a new method that selects and combines multiple
cost volumes during volume rendering. Our method does not require training and
can adapt to any MVS-based NeRF methods in a feed-forward fashion to improve
rendering quality. Furthermore, our approach is also end-to-end trainable,
allowing fine-tuning on specific scenes. We demonstrate the effectiveness of
our method through experiments on large-scale datasets, showing significant
rendering quality improvements in large-scale scenes and unbounded outdoor
scenarios. We release the source code of BoostMVSNeRFs at
https://su-terry.github.io/BoostMVSNeRFs/.
POGEMA: A Benchmark Platform for Cooperative Multi-Agent Navigation
Authors: name: Alexey Skrynnik, name: Anton Andreychuk, name: Anatolii Borzilov, name: Alexander Chernyavskiy, name: Konstantin Yakovlev, name: Aleksandr Panov
Published: 2024-07-20 16:37:21

Multi-agent reinforcement learning (MARL) has recently excelled in solving
challenging cooperative and competitive multi-agent problems in various
environments with, mostly, few agents and full observability. Moreover, a range
of crucial robotics-related tasks, such as multi-robot navigation and obstacle
avoidance, that have been conventionally approached with the classical
non-learnable methods (e.g., heuristic search) is currently suggested to be
solved by the learning-based or hybrid methods. Still, in this domain, it is
hard, not to say impossible, to conduct a fair comparison between classical,
learning-based, and hybrid approaches due to the lack of a unified framework
that supports both learning and evaluation. To this end, we introduce POGEMA, a
set of comprehensive tools that includes a fast environment for learning, a
generator of problem instances, the collection of pre-defined ones, a
visualization toolkit, and a benchmarking tool that allows automated
evaluation. We introduce and specify an evaluation protocol defining a range of
domain-related metrics computed on the basics of the primary evaluation
indicators (such as success rate and path length), allowing a fair multi-fold
comparison. The results of such a comparison, which involves a variety of
state-of-the-art MARL, search-based, and hybrid methods, are presented.