Orionnnn/hub-stats huggingface dataset:
Understanding Visual Feature Reliance through the Lens of Complexity
Authors: name: Thomas Fel, name: Louis Bethune, name: Andrew Kyle Lampinen, name: Thomas Serre, name: Katherine Hermann
Published: 2024-07-08 16:21:53

Recent studies suggest that deep learning models inductive bias towards
favoring simpler features may be one of the sources of shortcut learning. Yet,
there has been limited focus on understanding the complexity of the myriad
features that models learn. In this work, we introduce a new metric for
quantifying feature complexity, based on V-information and
capturing whether a feature requires complex computational transformations to
be extracted. Using this V-information metric, we analyze the
complexities of 10,000 features, represented as directions in the penultimate
layer, that were extracted from a standard ImageNet-trained vision model. Our
study addresses four key questions: First, we ask what features look like as a
function of complexity and find a spectrum of simple to complex features
present within the model. Second, we ask when features are learned during
training. We find that simpler features dominate early in training, and more
complex features emerge gradually. Third, we investigate where within the
network simple and complex features flow, and find that simpler features tend
to bypass the visual hierarchy via residual connections. Fourth, we explore the
connection between features complexity and their importance in driving the
networks decision. We find that complex features tend to be less important.
Surprisingly, important features become accessible at earlier layers during
training, like a sedimentation process, allowing the model to build upon these
foundational elements.
Evaluating Language Model Context Windows: A "Working Memory" Test and
  Inference-time Correction
Authors: name: Amanda Dsouza, name: Christopher Glaze, name: Changho Shin, name: Frederic Sala
Published: 2024-07-04 05:46:20

Large language models are prominently used in real-world applications, often
tasked with reasoning over large volumes of documents. An exciting development
in this space is models boasting extended context capabilities, with some
accommodating over 2 million tokens. Such long context model capabilities
remain uncertain in production systems, motivating the need to benchmark their
performance on real world use cases. We address this challenge by proposing
SWiM, an evaluation framework that addresses the limitations of standard tests.
Testing the framework on eight long context models, we find that even strong
models such as GPT-4 and Claude 3 Opus degrade in performance when information
is present in the middle of the context window (lost-in-the-middle effect).
Next, in addition to our benchmark, we propose medoid voting, a simple, but
effective training-free approach that helps alleviate this effect, by
generating responses a few times, each time randomly permuting documents in the
context, and selecting the medoid answer. We evaluate medoid voting on single
document QA tasks, achieving up to a 24% lift in accuracy.
Learning Action and Reasoning-Centric Image Editing from Videos and
  Simulations
Authors: name: Benno Krojer, name: Dheeraj Vattikonda, name: Luis Lara, name: Varun Jampani, name: Eva Portelance, name: Christopher Pal, name: Siva Reddy
Published: 2024-07-03 19:36:33

An image editing model should be able to perform diverse edits, ranging from
object replacement, changing attributes or style, to performing actions or
movement, which require many forms of reasoning. Current general
instruction-guided editing models have significant shortcomings with action and
reasoning-centric edits. Object, attribute or stylistic changes can be learned
from visually static datasets. On the other hand, high-quality data for action
and reasoning-centric edits is scarce and has to come from entirely different
sources that cover e.g. physical dynamics, temporality and spatial reasoning.
To this end, we meticulously curate the AURORA Dataset
(Action-Reasoning-Object-Attribute), a collection of high-quality training
data, human-annotated and curated from videos and simulation engines. We focus
on a key aspect of quality training data: triplets (source image, prompt,
target image) contain a single meaningful visual change described by the
prompt, i.e., truly minimal changes between source and target images. To
demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
Our model significantly outperforms previous editing models as judged by human
raters. For automatic evaluations, we find important flaws in previous metrics
and caution their use for semantically hard editing tasks. Instead, we propose
a new automatic metric that focuses on discriminative understanding. We hope
that our efforts : (1) curating a quality training dataset and an evaluation
benchmark, (2) developing critical evaluations, and (3) releasing a
state-of-the-art model, will fuel further progress on general image editing.
Training Task Experts through Retrieval Based Distillation
Authors: name: Jiaxin Ge, name: Xueying Jia, name: Vijay Viswanathan, name: Hongyin Luo, name: Graham Neubig
Published: 2024-07-07 18:27:59

One of the most reliable ways to create deployable models for specialized
tasks is to obtain an adequate amount of high-quality task-specific data.
However, for specialized tasks, often such datasets do not exist. Existing
methods address this by creating such data from large language models (LLMs)
and then distilling such knowledge into smaller models. However, these methods
are limited by the quality of the LLMs output, and tend to generate repetitive
or incorrect data. In this work, we present Retrieval Based Distillation
(ReBase), a method that first retrieves data from rich online sources and then
transforms them into domain-specific data. This method greatly enhances data
diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills
the reasoning capacity of LLMs. We test our method on 4 benchmarks and results
show that our method significantly improves performance by up to 7.8% on SQuAD,
1.37% on MNLI, and 1.94% on BigBench-Hard.
UltraEdit: Instruction-based Fine-Grained Image Editing at Scale
Authors: name: Haozhe Zhao, name: Xiaojian Ma, name: Liang Chen, name: Shuzheng Si, name: Rujie Wu, name: Kaikai An, name: Peiyu Yu, name: Minjia Zhang, name: Qing Li, name: Baobao Chang
Published: 2024-07-07 06:50:22

This paper presents UltraEdit, a large-scale (approximately 4 million editing
samples), automatically generated dataset for instruction-based image editing.
Our key idea is to address the drawbacks in existing image editing datasets
like InstructPix2Pix and MagicBrush, and provide a systematic approach to
producing massive and high-quality image editing samples. UltraEdit offers
several distinct advantages: 1) It features a broader range of editing
instructions by leveraging the creativity of large language models (LLMs)
alongside in-context editing examples from human raters; 2) Its data sources
are based on real images, including photographs and artworks, which provide
greater diversity and reduced bias compared to datasets solely generated by
text-to-image models; 3) It also supports region-based editing, enhanced by
high-quality, automatically produced region annotations. Our experiments show
that canonical diffusion-based editing baselines trained on UltraEdit set new
records on MagicBrush and Emu-Edit benchmarks. Our analysis further confirms
the crucial role of real image anchors and region-based editing data. The
dataset, code, and models can be found in https://ultra-editing.github.io.
Multi-Object Hallucination in Vision-Language Models
Authors: name: Xuweiyi Chen, name: Ziqiao Ma, name: Xuejun Zhang, name: Sihan Xu, name: Shengyi Qian, name: Jianing Yang, name: David F. Fouhey, name: Joyce Chai
Published: 2024-07-08 17:59:57

Large vision language models (LVLMs) often suffer from object hallucination,
producing objects not present in the given images. While current benchmarks for
object hallucination primarily concentrate on the presence of a single object
class rather than individual entities, this work systematically investigates
multi-object hallucination, examining how models misperceive (e.g., invent
nonexistent objects or become distracted) when tasked with focusing on multiple
objects simultaneously. We introduce Recognition-based Object Probing
Evaluation (ROPE), an automated evaluation protocol that considers the
distribution of object classes within a single image during testing and uses
visual referring prompts to eliminate ambiguity. With comprehensive empirical
studies and analysis of potential factors leading to multi-object
hallucination, we found that (1) LVLMs suffer more hallucinations when focusing
on multiple objects compared to a single object. (2) The tested object class
distribution affects hallucination behaviors, indicating that LVLMs may follow
shortcuts and spurious correlations.(3) Hallucinatory behaviors are influenced
by data-specific factors, salience and frequency, and model intrinsic
behaviors. We hope to enable LVLMs to recognize and reason about multiple
objects that often occur in realistic visual scenes, provide insights, and
quantify our progress towards mitigating the issues.
Associative Recurrent Memory Transformer
Authors: name: Ivan Rodkin, name: Yuri Kuratov, name: Aydar Bulatov, name: Mikhail Burtsev
Published: 2024-07-05 19:57:49

This paper addresses the challenge of creating a neural architecture for very
long sequences that requires constant time for processing new information at
each time step. Our approach, Associative Recurrent Memory Transformer (ARMT),
is based on transformer self-attention for local context and segment-level
recurrence for storage of task specific information distributed over a long
context. We demonstrate that ARMT outperfors existing alternatives in
associative retrieval tasks and sets a new performance record in the recent
BABILong multi-task long-context benchmark by answering single-fact questions
over 50 million tokens with an accuracy of 79.9%. The source code for training
and evaluation is available on github.
Compositional Video Generation as Flow Equalization
Authors: name: Xingyi Yang, name: Xinchao Wang
Published: 2024-06-10 16:27:47

Large-scale Text-to-Video (T2V) diffusion models have recently demonstrated
unprecedented capability to transform natural language descriptions into
stunning and photorealistic videos. Despite the promising results, a
significant challenge remains: these models struggle to fully grasp complex
compositional interactions between multiple concepts and actions. This issue
arises when some words dominantly influence the final video, overshadowing
other concepts.To tackle this problem, we introduce Vico, a generic
framework for compositional video generation that explicitly ensures all
concepts are represented properly. At its core, Vico analyzes how input tokens
influence the generated video, and adjusts the model to prevent any single
concept from dominating. Specifically, Vico extracts attention weights from all
layers to build a spatial-temporal attention graph, and then estimates the
influence as the max-flow from the source text token to the video target
token. Although the direct computation of attention flow in diffusion models is
typically infeasible, we devise an efficient approximation based on subgraph
flows and employ a fast and vectorized implementation, which in turn makes the
flow computation manageable and differentiable. By updating the noisy latent to
balance these flows, Vico captures complex interactions and consequently
produces videos that closely adhere to textual descriptions. We apply our
method to multiple diffusion-based video models for compositional T2V and video
editing. Empirical results demonstrate that our framework significantly
enhances the compositional richness and accuracy of the generated videos. Visit
our website
at~https://adamdad.github.io/vico/{https://adamdad.github.io/vico/}.
PAS: Data-Efficient Plug-and-Play Prompt Augmentation System
Authors: name: Miao Zheng, name: Hao Liang, name: Fan Yang, name: Haoze Sun, name: Tianpeng Li, name: Lingchu Xiong, name: Yan Zhang, name: Yozhen Wu, name: Kun Li, name: Yanjun Sheng, name: Mingan Lin, name: Tao Zhang, name: Guosheng Dong, name: Yujing Qiao, name: Kun Fang, name: Weipeng Chen, name: Bin Cui, name: Wentao Zhang, name: Zenan Zhou
Published: 2024-07-08 15:25:33

In recent years, the rise of Large Language Models (LLMs) has spurred a
growing demand for plug-and-play AI systems. Among the various AI techniques,
prompt engineering stands out as particularly significant. However, users often
face challenges in writing prompts due to the steep learning curve and
significant time investment, and existing automatic prompt engineering (APE)
models can be difficult to use. To address this issue, we propose PAS, an
LLM-based plug-and-play APE system. PAS utilizes LLMs trained on high-quality,
automatically generated prompt complementary datasets, resulting in exceptional
performance. In comprehensive benchmarks, PAS achieves state-of-the-art (SoTA)
results compared to previous APE models, with an average improvement of 6.09
points. Moreover, PAS is highly efficient, achieving SoTA performance with only
9000 data points. Additionally, PAS can autonomously generate prompt
augmentation data without requiring additional human labor. Its flexibility
also allows it to be compatible with all existing LLMs and applicable to a wide
range of tasks. PAS excels in human evaluations, underscoring its suitability
as a plug-in for users. This combination of high performance, efficiency, and
flexibility makes PAS a valuable system for enhancing the usability and
effectiveness of LLMs through improved prompt engineering.
PartCraft: Crafting Creative Objects by Parts
Authors: name: Kam Woh Ng, name: Xiatian Zhu, name: Yi-Zhe Song, name: Tao Xiang
Published: 2024-07-05 15:53:04

This paper propels creative control in generative visual AI by allowing users
to "select". Departing from traditional text or sketch-based methods, we for
the first time allow users to choose visual concepts by parts for their
creative endeavors. The outcome is fine-grained generation that precisely
captures selected visual concepts, ensuring a holistically faithful and
plausible result. To achieve this, we first parse objects into parts through
unsupervised feature clustering. Then, we encode parts into text tokens and
introduce an entropy-based normalized attention loss that operates on them.
This loss design enables our model to learn generic prior topology knowledge
about object's part composition, and further generalize to novel part
compositions to ensure the generation looks holistically faithful. Lastly, we
employ a bottleneck encoder to project the part tokens. This not only enhances
fidelity but also accelerates learning, by leveraging shared knowledge and
facilitating information exchange among instances. Visual results in the paper
and supplementary material showcase the compelling power of PartCraft in
crafting highly customized, innovative creations, exemplified by the "charming"
and creative birds. Code is released at https://github.com/kamwoh/partcraft.
ANOLE: An Open, Autoregressive, Native Large Multimodal Models for
  Interleaved Image-Text Generation
Authors: name: Ethan Chern, name: Jiadi Su, name: Yan Ma, name: Pengfei Liu
Published: 2024-07-08 17:08:02

Previous open-source large multimodal models (LMMs) have faced several
limitations: (1) they often lack native integration, requiring adapters to
align visual representations with pre-trained large language models (LLMs); (2)
many are restricted to single-modal generation; (3) while some support
multimodal generation, they rely on separate diffusion models for visual
modeling and generation. To mitigate these limitations, we present Anole, an
open, autoregressive, native large multimodal model for interleaved image-text
generation. We build Anole from Meta AI's Chameleon, adopting an innovative
fine-tuning strategy that is both data-efficient and parameter-efficient. Anole
demonstrates high-quality, coherent multimodal generation capabilities. We have
open-sourced our model, training framework, and instruction tuning data.
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for
  Text-to-Image Generation?
Authors: name: Zhaorun Chen, name: Yichao Du, name: Zichen Wen, name: Yiyang Zhou, name: Chenhang Cui, name: Zhenzhen Weng, name: Haoqin Tu, name: Chaoqi Wang, name: Zhengwei Tong, name: Qinglan Huang, name: Canyu Chen, name: Qinghao Ye, name: Zhihong Zhu, name: Yuqing Zhang, name: Jiawei Zhou, name: Zhuokai Zhao, name: Rafael Rafailov, name: Chelsea Finn, name: Huaxiu Yao
Published: 2024-07-05 20:03:16

While text-to-image models like DALLE-3 and Stable Diffusion are rapidly
proliferating, they often encounter challenges such as hallucination, bias, and
the production of unsafe, low-quality output. To effectively address these
issues, it is crucial to align these models with desired behaviors based on
feedback from a multimodal judge. Despite their significance, current
multimodal judges frequently undergo inadequate evaluation of their
capabilities and limitations, potentially leading to misalignment and unsafe
fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel
benchmark which incorporates a comprehensive preference dataset to evaluate
multimodal judges in providing feedback for image generation models across four
key perspectives: alignment, safety, image quality, and bias. Specifically, we
evaluate a large variety of multimodal judges including smaller-sized
CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and
close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our
preference dataset. Experiments reveal that close-source VLMs generally provide
better feedback, with GPT-4o outperforming other judges in average. Compared
with open-source VLMs, smaller-sized scoring models can provide better feedback
regarding text-image alignment and image quality, while VLMs provide more
accurate feedback regarding safety and generation bias due to their stronger
reasoning capabilities. Further studies in feedback scale reveal that VLM
judges can generally provide more accurate and stable feedback in natural
language (Likert-scale) than numerical scales. Notably, human evaluations on
end-to-end fine-tuned models using separate feedback from these multimodal
judges provide similar conclusions, further confirming the effectiveness of
MJ-Bench. All data, code, models are available at
https://huggingface.co/MJ-Bench.
LLMAEL: Large Language Models are Good Context Augmenters for Entity
  Linking
Authors: name: Amy Xin, name: Yunjia Qi, name: Zijun Yao, name: Fangwei Zhu, name: Kaisheng Zeng, name: Xu Bin, name: Lei Hou, name: Juanzi Li
Published: 2024-07-04 15:55:13

Entity Linking (EL) models are well-trained at mapping mentions to their
corresponding entities according to a given context. However, EL models
struggle to disambiguate long-tail entities due to their limited training data.
Meanwhile, large language models (LLMs) are more robust at interpreting
uncommon mentions. Yet, due to a lack of specialized training, LLMs suffer at
generating correct entity IDs. Furthermore, training an LLM to perform EL is
cost-intensive. Building upon these insights, we introduce LLM-Augmented Entity
Linking LLMAEL, a plug-and-play approach to enhance entity linking through LLM
data augmentation. We leverage LLMs as knowledgeable context augmenters,
generating mention-centered descriptions as additional input, while preserving
traditional EL models for task specific processing. Experiments on 6 standard
datasets show that the vanilla LLMAEL outperforms baseline EL models in most
cases, while the fine-tuned LLMAEL set the new state-of-the-art results across
all 6 benchmarks.
InverseCoder: Unleashing the Power of Instruction-Tuned Code LLMs with
  Inverse-Instruct
Authors: name: Yutong Wu, name: Di Huang, name: Wenxuan Shi, name: Wei Wang, name: Lingzhe Gao, name: Shihao Liu, name: Ziyuan Nan, name: Kaizhao Yuan, name: Rui Zhang, name: Xishan Zhang, name: Zidong Du, name: Qi Guo, name: Yewen Pu, name: Dawei Yin, name: Xing Hu, name: Yunji Chen
Published: 2024-07-08 08:00:05

Recent advancements in open-source code large language models (LLMs) have
demonstrated remarkable coding abilities by fine-tuning on the data generated
from powerful closed-source LLMs such as GPT-3.5 and GPT-4 for instruction
tuning. This paper explores how to further improve an instruction-tuned code
LLM by generating data from itself rather than querying closed-source LLMs. Our
key observation is the misalignment between the translation of formal and
informal languages: translating formal language (i.e., code) to informal
language (i.e., natural language) is more straightforward than the reverse.
Based on this observation, we propose INVERSE-INSTRUCT, which summarizes
instructions from code snippets instead of the reverse. Specifically, given an
instruction tuning corpus for code and the resulting instruction-tuned code
LLM, we ask the code LLM to generate additional high-quality instructions for
the original corpus through code summarization and self-evaluation. Then, we
fine-tune the base LLM on the combination of the original corpus and the
self-generated one, which yields a stronger instruction-tuned LLM. We present a
series of code LLMs named InverseCoder, which surpasses the performance of the
original code LLMs on a wide range of benchmarks, including Python text-to-code
generation, multilingual coding, and data-science code generation.
ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language
  Models
Authors: name: Yuzhe Gu, name: Ziwei Ji, name: Wenwei Zhang, name: Chengqi Lyu, name: Dahua Lin, name: Kai Chen
Published: 2024-07-05 17:56:38

Large language models (LLMs) exhibit hallucinations in long-form
question-answering tasks across various domains and wide applications. Current
hallucination detection and mitigation datasets are limited in domains and
sizes, which struggle to scale due to prohibitive labor costs and insufficient
reliability of existing hallucination annotators. To facilitate the scalable
oversight of LLM hallucinations, this paper introduces an iterative
self-training framework that simultaneously and progressively scales up the
hallucination annotation dataset and improves the accuracy of the hallucination
annotator. Based on the Expectation Maximization (EM) algorithm, in each
iteration, the framework first applies a hallucination annotation pipeline to
annotate a scaled dataset and then trains a more accurate hallucination
annotator on the dataset. This new hallucination annotator is adopted in the
hallucination annotation pipeline used for the next iteration. Extensive
experimental results demonstrate that the finally obtained hallucination
annotator with only 7B parameters surpasses the performance of GPT-4 and
obtains new state-of-the-art hallucination detection results on HaluEval and
HalluQA by zero-shot inference. Such an annotator can not only evaluate the
hallucination levels of various LLMs on the large-scale dataset but also help
to mitigate the hallucination of LLMs generations, with the Natural Language
Inference (NLI) metric increasing from 25% to 37% on HaluEval.