Orionnnn/hub-stats huggingface dataset:
An Early Evaluation of GPT-4V(ision)
Authors: name: Yang Wu, name: Shilong Wang, name: Hao Yang, name: Tian Zheng, name: Hongbo Zhang, name: Yanyan Zhao, name: Bing Qin
Published: 2023-10-25 10:33:17

In this paper, we evaluate different abilities of GPT-4V including visual
understanding, language understanding, visual puzzle solving, and understanding
of other modalities such as depth, thermal, video, and audio. To estimate
GPT-4V's performance, we manually construct 656 test instances and carefully
evaluate the results of GPT-4V. The highlights of our findings are as follows:
(1) GPT-4V exhibits impressive performance on English visual-centric benchmarks
but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows
inconsistent refusal behavior when answering questions related to sensitive
traits such as gender, race, and age; (3) GPT-4V obtains worse results than
GPT-4 (API) on language understanding tasks including general language
understanding benchmarks and visual commonsense knowledge evaluation
benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both
visual understanding and language understanding; (5) GPT-4V struggles to find
the nuances between two similar images and solve the easy math picture puzzles;
(6) GPT-4V shows non-trivial performance on the tasks of similar modalities to
image, such as video and thermal. Our experimental results reveal the ability
and limitations of GPT-4V and we hope our paper can provide some insights into
the application and research of GPT-4V.
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
  Understanding
Authors: name: Haoxiang Wang, name: Pavan Kumar Anasosalu Vasu, name: Fartash Faghri, name: Raviteja Vemulapalli, name: Mehrdad Farajtabar, name: Sachin Mehta, name: Mohammad Rastegari, name: Oncel Tuzel, name: Hadi Pouransari
Published: 2023-10-23 19:21:57

The landscape of publicly available vision foundation models (VFMs), such as
CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
with distinct capabilities stemming from their pre-training objectives. For
instance, CLIP excels in semantic understanding, while SAM specializes in
spatial understanding for segmentation. In this work, we introduce a simple
recipe to efficiently merge VFMs into a unified model that assimilates their
expertise. Our proposed method integrates multi-task learning, continual
learning techniques, and teacher-student distillation. This strategy entails
significantly less computational cost compared to traditional multi-task
training from scratch. Additionally, it only demands a small fraction of the
pre-training datasets that were initially used to train individual models. By
applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
for edge device applications. We show that SAM-CLIP learns richer visual
representations, equipped with both localization and semantic features,
suitable for a broad range of vision tasks. SAM-CLIP obtains improved
performance on several head probing tasks when compared with SAM and CLIP. We
further show that SAM-CLIP not only retains the foundational strengths of its
precursor models but also introduces synergistic functionalities, most notably
in zero-shot semantic segmentation, where SAM-CLIP establishes new
state-of-the-art results on 5 benchmarks. It outperforms previous models that
are specifically designed for this task by a large margin, including +6.8% and
+5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
Dissecting In-Context Learning of Translations in GPTs
Authors: name: Vikas Raunak, name: Hany Hassan Awadalla, name: Arul Menezes
Published: 2023-10-24 16:37:18

Most of the recent work in leveraging Large Language Models (LLMs) such as
GPT-3 for Machine Translation (MT) has focused on selecting the few-shot
samples for prompting. In this work, we try to better understand the role of
demonstration attributes for the in-context learning of translations through
perturbations of high-quality, in-domain demonstrations. We find that
asymmetric perturbation of the source-target mappings yield vastly different
results. We show that the perturbation of the source side has surprisingly
little impact, while target perturbation can drastically reduce translation
quality, suggesting that it is the output text distribution that provides the
most important learning signal during in-context learning of translations. We
propose a method named Zero-Shot-Context to add this signal automatically in
Zero-Shot prompting. We demonstrate that it improves upon the zero-shot
translation performance of GPT-3, even making it competitive with few-shot
prompted translations.
In-Context Learning Creates Task Vectors
Authors: name: Roee Hendel, name: Mor Geva, name: Amir Globerson
Published: 2023-10-24 15:17:14

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a
powerful new learning paradigm. However, its underlying mechanism is still not
well understood. In particular, it is challenging to map it to the "standard"
machine learning framework, where one uses a training set S to find a
best-fitting function f(x) in some hypothesis class. Here we make progress on
this problem by showing that the functions learned by ICL often have a very
simple structure: they correspond to the transformer LLM whose only inputs are
the query x and a single "task vector" calculated from the training set.
Thus, ICL can be seen as compressing S into a single task vector
theta(S) and then using this task vector to modulate the
transformer to produce the output. We support the above claim via comprehensive
experiments across a range of models and tasks.
KITAB: Evaluating LLMs on Constraint Satisfaction for Information
  Retrieval
Authors: name: Marah I Abdin, name: Suriya Gunasekar, name: Varun Chandrasekaran, name: Jerry Li, name: Mert Yuksekgonul, name: Rahee Ghosh Peshawaria, name: Ranjita Naik, name: Besmira Nushi
Published: 2023-10-24 04:40:38

We study the ability of state-of-the art models to answer constraint
satisfaction queries for information retrieval (e.g., 'a list of ice cream
shops in San Diego'). In the past, such queries were considered to be tasks
that could only be solved via web-search or knowledge bases. More recently,
large language models (LLMs) have demonstrated initial emergent abilities in
this task. However, many current retrieval benchmarks are either saturated or
do not measure constraint satisfaction. Motivated by rising concerns around
factual incorrectness and hallucinations of LLMs, we present KITAB, a new
dataset for measuring constraint satisfaction abilities of language models.
KITAB consists of book-related data across more than 600 authors and 13,000
queries, and also offers an associated dynamic data collection and constraint
verification approach for acquiring similar test data for other authors. Our
extended experiments on GPT4 and GPT3.5 characterize and decouple common
failure modes across dimensions such as information popularity, constraint
types, and context availability. Results show that in the absence of context,
models exhibit severe limitations as measured by irrelevant information,
factual errors, and incompleteness, many of which exacerbate as information
popularity decreases. While context availability mitigates irrelevant
information, it is not helpful for satisfying constraints, identifying
fundamental barriers to constraint satisfaction. We open source our
contributions to foster further research on improving constraint satisfaction
abilities of future models.
TRAMS: Training-free Memory Selection for Long-range Language Modeling
Authors: name: Haofei Yu, name: Cunxiang wang, name: Yue Zhang, name: Wei Bi
Published: 2023-10-24 03:42:49

The Transformer architecture is crucial for numerous AI models, but it still
faces challenges in long-range language modeling. Though several specific
transformer architectures have been designed to tackle issues of long-range
dependencies, existing methods like Transformer-XL are plagued by a high
percentage of ineffective memories. In this study, we present a plug-and-play
strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens
participating in attention calculation based on one simple metric. This
strategy allows us to keep tokens that are likely to have a high attention
score with the current queries and ignore the other ones. We have tested our
approach on the word-level benchmark (WikiText-103) and the character-level
benchmark (enwik8), and the results indicate an improvement without having
additional training or adding additional parameters.
Moral Foundations of Large Language Models
Authors: name: Marwa Abdulhai, name: Gregory Serapio-Garcia, name: Clément Crepy, name: Daria Valter, name: John Canny, name: Natasha Jaques
Published: 2023-10-23 20:05:37

Moral foundations theory (MFT) is a psychological assessment tool that
decomposes human moral reasoning into five factors, including care/harm,
liberty/oppression, and sanctity/degradation (Graham et al., 2009). People vary
in the weight they place on these dimensions when making moral decisions, in
part due to their cultural upbringing and political ideology. As large language
models (LLMs) are trained on datasets collected from the internet, they may
reflect the biases that are present in such corpora. This paper uses MFT as a
lens to analyze whether popular LLMs have acquired a bias towards a particular
set of moral values. We analyze known LLMs and find they exhibit particular
moral foundations, and show how these relate to human moral foundations and
political affiliations. We also measure the consistency of these biases, or
whether they vary strongly depending on the context of how the model is
prompted. Finally, we show that we can adversarially select prompts that
encourage the moral to exhibit a particular set of moral foundations, and that
this can affect the model's behavior on downstream tasks. These findings help
illustrate the potential risks and unintended consequences of LLMs assuming a
particular moral stance.
Inject Semantic Concepts into Image Tagging for Open-Set Recognition
Authors: name: Xinyu Huang, name: Yi-Jie Huang, name: Youcai Zhang, name: Weiwei Tian, name: Rui Feng, name: Yuejie Zhang, name: Yanchun Xie, name: Yaqian Li, name: Lei Zhang
Published: 2023-10-23 08:13:33

In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
fundamental image recognition model with strong open-set recognition
capabilities, by injecting semantic concepts into image tagging training
framework. Previous approaches are either image tagging models constrained by
limited semantics, or vision-language models with shallow interaction for
suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
image-text alignment and image-tagging within a unified fine-grained
interaction framework based on image-tags-text triplets. This design enables
RAM++ not only excel in identifying predefined categories, but also
significantly augment the recognition ability in open-set categories. Moreover,
RAM++ employs large language models~(LLMs) to generate diverse visual tag
descriptions, pioneering the integration of LLM's knowledge into image tagging
training. This approach empowers RAM++ to integrate visual description concepts
for open-set recognition during inference. Evaluations on comprehensive image
recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
(SOTA) fundamental image recognition models on most aspects. Specifically, for
predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
enhancements over CLIP on OpenImages and ImageNet. For open-set categories
beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
and RAM respectively on OpenImages. For diverse human-object interaction
phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
Code, datasets and pre-trained models are available at
https://github.com/xinyu1205/recognize-anything.
Woodpecker: Hallucination Correction for Multimodal Large Language
  Models
Authors: name: Shukang Yin, name: Chaoyou Fu, name: Sirui Zhao, name: Tong Xu, name: Hao Wang, name: Dianbo Sui, name: Yunhang Shen, name: Ke Li, name: Xing Sun, name: Enhong Chen
Published: 2023-10-24 17:58:07

Hallucination is a big shadow hanging over the rapidly evolving Multimodal
Large Language Models (MLLMs), referring to the phenomenon that the generated
text is inconsistent with the image content. In order to mitigate
hallucinations, existing studies mainly resort to an instruction-tuning manner
that requires retraining the models with specific data. In this paper, we pave
a different way, introducing a training-free method named Woodpecker. Like a
woodpecker heals trees, it picks out and corrects hallucinations from the
generated text. Concretely, Woodpecker consists of five stages: key concept
extraction, question formulation, visual knowledge validation, visual claim
generation, and hallucination correction. Implemented in a post-remedy manner,
Woodpecker can easily serve different MLLMs, while being interpretable by
accessing intermediate outputs of the five stages. We evaluate Woodpecker both
quantitatively and qualitatively and show the huge potential of this new
paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement
in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released
at https://github.com/BradyFU/Woodpecker.
Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
Authors: name: Xavier Puig, name: Eric Undersander, name: Andrew Szot, name: Mikael Dallaire Cote, name: Tsung-Yen Yang, name: Ruslan Partsey, name: Ruta Desai, name: Alexander William Clegg, name: Michal Hlavac, name: So Yeon Min, name: Vladimír Vondruš, name: Theophile Gervet, name: Vincent-Pierre Berges, name: John M. Turner, name: Oleksandr Maksymets, name: Zsolt Kira, name: Mrinal Kalakrishnan, name: Jitendra Malik, name: Devendra Singh Chaplot, name: Unnat Jain, name: Dhruv Batra, name: Akshara Rai, name: Roozbeh Mottaghi
Published: 2023-10-19 17:29:17

We present Habitat 3.0: a simulation platform for studying collaborative
human-robot tasks in home environments. Habitat 3.0 offers contributions across
three dimensions: (1) Accurate humanoid simulation: addressing challenges in
modeling complex deformable bodies and diversity in appearance and motion, all
while ensuring high simulation speed. (2) Human-in-the-loop infrastructure:
enabling real human interaction with simulated robots via mouse/keyboard or a
VR interface, facilitating evaluation of robot policies with human input. (3)
Collaborative tasks: studying two collaborative tasks, Social Navigation and
Social Rearrangement. Social Navigation investigates a robot's ability to
locate and follow humanoid avatars in unseen environments, whereas Social
Rearrangement addresses collaboration between a humanoid and robot while
rearranging a scene. These contributions allow us to study end-to-end learned
and heuristic baselines for human-robot collaboration in-depth, as well as
evaluate them with humans in the loop. Our experiments demonstrate that learned
robot policies lead to efficient task completion when collaborating with unseen
humanoid agents and human partners that might exhibit behaviors that the robot
has not seen before. Additionally, we observe emergent behaviors during
collaborative task execution, such as the robot yielding space when obstructing
a humanoid agent, thereby allowing the effective completion of the task by the
humanoid agent. Furthermore, our experiments using the human-in-the-loop tool
demonstrate that our automated evaluation with humanoids can provide an
indication of the relative ordering of different policies when evaluated with
real human collaborators. Habitat 3.0 unlocks interesting new features in
simulators for Embodied AI, and we hope it paves the way for a new frontier of
embodied human-AI interaction capabilities.
TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion
  Models
Authors: name: Tianshi Cao, name: Karsten Kreis, name: Sanja Fidler, name: Nicholas Sharp, name: Kangxue Yin
Published: 2023-10-20 19:15:29

We present TexFusion (Texture Diffusion), a new method to synthesize textures
for given 3D geometries, using large-scale text-guided image diffusion models.
In contrast to recent works that leverage 2D text-to-image diffusion models to
distill 3D objects using a slow and fragile optimization process, TexFusion
introduces a new 3D-consistent generation technique specifically designed for
texture synthesis that employs regular diffusion model sampling on different 2D
rendered views. Specifically, we leverage latent diffusion models, apply the
diffusion model's denoiser on a set of 2D renders of the 3D object, and
aggregate the different denoising predictions on a shared latent texture map.
Final output RGB textures are produced by optimizing an intermediate neural
color field on the decodings of 2D renders of the latent texture. We thoroughly
validate TexFusion and show that we can efficiently generate diverse, high
quality and globally coherent textures. We achieve state-of-the-art text-guided
texture synthesis performance using only image diffusion models, while avoiding
the pitfalls of previous distillation-based methods. The text-conditioning
offers detailed control and we also do not rely on any ground truth 3D textures
for training. This makes our method versatile and applicable to a broad range
of geometry and texture types. We hope that TexFusion will advance AI-based
texturing of 3D assets for applications in virtual reality, game design,
simulation, and more.
Matryoshka Diffusion Models
Authors: name: Jiatao Gu, name: Shuangfei Zhai, name: Yizhe Zhang, name: Josh Susskind, name: Navdeep Jaitly
Published: 2023-10-23 17:20:01

Diffusion models are the de facto approach for generating high-quality images
and videos, but learning high-dimensional models remains a formidable task due
to computational and optimization challenges. Existing methods often resort to
training cascaded models in pixel space or using a downsampled latent space of
a separately trained auto-encoder. In this paper, we introduce Matryoshka
Diffusion Models(MDM), an end-to-end framework for high-resolution image and
video synthesis. We propose a diffusion process that denoises inputs at
multiple resolutions jointly and uses a NestedUNet architecture where features
and parameters for small-scale inputs are nested within those of large scales.
In addition, MDM enables a progressive training schedule from lower to higher
resolutions, which leads to significant improvements in optimization for
high-resolution generation. We demonstrate the effectiveness of our approach on
various benchmarks, including class-conditioned image generation,
high-resolution text-to-image, and text-to-video applications. Remarkably, we
can train a single pixel-space model at resolutions of up to 1024x1024 pixels,
demonstrating strong zero-shot generalization using the CC12M dataset, which
contains only 12 million images.
FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling
Authors: name: Haonan Qiu, name: Menghan Xia, name: Yong Zhang, name: Yingqing He, name: Xintao Wang, name: Ying Shan, name: Ziwei Liu
Published: 2023-10-23 17:59:58

With the availability of large-scale video datasets and the advances of
diffusion models, text-driven video generation has achieved substantial
progress. However, existing video generation models are typically trained on a
limited number of frames, resulting in the inability to generate high-fidelity
long videos during inference. Furthermore, these models only support
single-text conditions, whereas real-life scenarios often require multi-text
conditions as the video content changes over time. To tackle these challenges,
this study explores the potential of extending the text-driven capability to
generate longer videos conditioned on multiple texts. 1) We first analyze the
impact of initial noise in video diffusion models. Then building upon the
observation of noise, we propose FreeNoise, a tuning-free and time-efficient
paradigm to enhance the generative capabilities of pretrained video diffusion
models while preserving content consistency. Specifically, instead of
initializing noises for all frames, we reschedule a sequence of noises for
long-range correlation and perform temporal attention over them by window-based
function. 2) Additionally, we design a novel motion injection method to support
the generation of videos conditioned on multiple text prompts. Extensive
experiments validate the superiority of our paradigm in extending the
generative capabilities of video diffusion models. It is noteworthy that
compared with the previous best-performing method which brought about 255%
extra time cost, our method incurs only negligible time cost of approximately
17%. Generated video samples are available at our website:
http://haonanqiu.com/projects/FreeNoise.html.
Ensemble-Instruct: Generating Instruction-Tuning Data with a
  Heterogeneous Mixture of LMs
Authors: name: Young-Suk Lee, name: Md Arafat Sultan, name: Yousef El-Kurdi, name: Tahira Naseem Asim Munawar, name: Radu Florian, name: Salim Roukos, name: Ramón Fernandez Astudillo
Published: 2023-10-21 10:21:17

Using in-context learning (ICL) for data generation, techniques such as
Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023)
can train strong conversational agents with only a small amount of human
supervision. One limitation of these approaches is that they resort to very
large language models (around 175B parameters) that are also proprietary and
non-public. Here we explore the application of such techniques to language
models that are much smaller (around 10B--40B parameters) and have permissive
licenses. We find the Self-Instruct approach to be less effective at these
sizes and propose new ICL methods that draw on two main ideas: (a)
Categorization and simplification of the ICL templates to make prompt learning
easier for the LM, and (b) Ensembling over multiple LM outputs to help select
high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct
seed tasks and employs separate pipelines for instructions that require an
input and instructions that do not. Empirical investigations with different LMs
show that: (1) Our proposed method yields higher-quality instruction tuning
data than Self-Instruct, (2) It improves performances of both vanilla and
instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned
LMs generate more useful outputs than their larger un-tuned counterparts. Our
codebase is available at https://github.com/IBM/ensemble-instruct.