Orionnnn/hub-stats huggingface dataset:
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on
  Generalizability, Trustworthiness and Causality through Four Modalities
Authors: name: Chaochao Lu, name: Chen Qian, name: Guodong Zheng, name: Hongxing Fan, name: Hongzhi Gao, name: Jie Zhang, name: Jing Shao, name: Jingyi Deng, name: Jinlan Fu, name: Kexin Huang, name: Kunchang Li, name: Lijun Li, name: Limin Wang, name: Lu Sheng, name: Meiqi Chen, name: Ming Zhang, name: Qibing Ren, name: Sirui Chen, name: Tao Gui, name: Wanli Ouyang, name: Yali Wang, name: Yan Teng, name: Yaru Wang, name: Yi Wang, name: Yinan He, name: Yingchun Wang, name: Yixu Wang, name: Yongting Zhang, name: Yu Qiao, name: Yujiong Shen, name: Yurong Mou, name: Yuxi Chen, name: Zaibin Zhang, name: Zhelun Shi, name: Zhenfei Yin, name: Zhipin Wang
Published: 2024-01-26 18:53:03

Multi-modal Large Language Models (MLLMs) have shown impressive abilities in
generating reasonable responses with respect to multi-modal contents. However,
there is still a wide gap between the performance of recent MLLM-based
applications and the expectation of the broad public, even though the most
powerful OpenAI's GPT-4 and Google's Gemini have been deployed. This paper
strives to enhance understanding of the gap through the lens of a qualitative
study on the generalizability, trustworthiness, and causal reasoning
capabilities of recent proprietary and open-source MLLMs across four
modalities: ie, text, code, image, and video, ultimately aiming to improve the
transparency of MLLMs. We believe these properties are several representative
factors that define the reliability of MLLMs, in supporting various downstream
applications. To be specific, we evaluate the closed-source GPT-4 and Gemini
and 6 open-source LLMs and MLLMs. Overall we evaluate 230 manually designed
cases, where the qualitative results are then summarized into 12 scores (ie, 4
modalities times 3 properties). In total, we uncover 14 empirical findings that
are useful to understand the capabilities and limitations of both proprietary
and open-source MLLMs, towards more reliable downstream multi-modal
applications.
Adaptive Mobile Manipulation for Articulated Objects In the Open World
Authors: name: Haoyu Xiong, name: Russell Mendonca, name: Kenneth Shaw, name: Deepak Pathak
Published: 2024-01-25 18:59:44

Deploying robots in open-ended unstructured environments such as homes has
been a long-standing research problem. However, robots are often studied only
in closed-off lab settings, and prior mobile manipulation work is restricted to
pick-move-place, which is arguably just the tip of the iceberg in this area. In
this paper, we introduce Open-World Mobile Manipulation System, a full-stack
approach to tackle realistic articulated object operation, e.g. real-world
doors, cabinets, drawers, and refrigerators in open-ended unstructured
environments. The robot utilizes an adaptive learning framework to initially
learns from a small set of data through behavior cloning, followed by learning
from online practice on novel objects that fall outside the training
distribution. We also develop a low-cost mobile manipulation hardware platform
capable of safe and autonomous online adaptation in unstructured environments
with a cost of around 20,000 USD. In our experiments we utilize 20 articulate
objects across 4 buildings in the CMU campus. With less than an hour of online
learning for each object, the system is able to increase success rate from 50%
of BC pre-training to 95% using online adaptation. Video results at
https://open-world-mobilemanip.github.io/
pix2gestalt: Amodal Segmentation by Synthesizing Wholes
Authors: name: Ege Ozguroglu, name: Ruoshi Liu, name: Dídac Surís, name: Dian Chen, name: Achal Dave, name: Pavel Tokmakov, name: Carl Vondrick
Published: 2024-01-25 18:57:36

We introduce pix2gestalt, a framework for zero-shot amodal segmentation,
which learns to estimate the shape and appearance of whole objects that are
only partially visible behind occlusions. By capitalizing on large-scale
diffusion models and transferring their representations to this task, we learn
a conditional diffusion model for reconstructing whole objects in challenging
zero-shot cases, including examples that break natural and physical priors,
such as art. As training data, we use a synthetically curated dataset
containing occluded objects paired with their whole counterparts. Experiments
show that our approach outperforms supervised baselines on established
benchmarks. Our model can furthermore be used to significantly improve the
performance of existing object recognition and 3D reconstruction methods in the
presence of occlusions.
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other
  Modalities
Authors: name: Yiyuan Zhang, name: Xiaohan Ding, name: Kaixiong Gong, name: Yixiao Ge, name: Ying Shan, name: Xiangyu Yue
Published: 2024-01-25 18:59:58

We propose to improve transformers of a specific modality with irrelevant
data from other modalities, e.g., improve an ImageNet model with audio or point
cloud datasets. We would like to highlight that the data samples of the target
modality are irrelevant to the other modalities, which distinguishes our method
from other works utilizing paired (e.g., CLIP) or interleaved data of different
modalities. We propose a methodology named Multimodal Pathway - given a target
modality and a transformer designed for it, we use an auxiliary transformer
trained with data of another modality and construct pathways to connect
components of the two models so that data of the target modality can be
processed by both models. In this way, we utilize the universal
sequence-to-sequence modeling abilities of transformers obtained from two
modalities. As a concrete implementation, we use a modality-specific tokenizer
and task-specific head as usual but utilize the transformer blocks of the
auxiliary model via a proposed method named Cross-Modal Re-parameterization,
which exploits the auxiliary weights without any inference costs. On the image,
point cloud, video, and audio recognition tasks, we observe significant and
consistent performance improvements with irrelevant data from other modalities.
The code and models are available at https://github.com/AILab-CVC/M2PT.
Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent
  Diffusion Models for Virtual Try-All
Authors: name: Mehmet Saygin Seyfioglu, name: Karim Bouyarmane, name: Suren Kumar, name: Amir Tavanaei, name: Ismail B. Tutar
Published: 2024-01-24 20:25:48

As online shopping is growing, the ability for buyers to virtually visualize
products in their settings-a phenomenon we define as "Virtual Try-All"-has
become crucial. Recent diffusion models inherently contain a world model,
rendering them suitable for this task within an inpainting context. However,
traditional image-conditioned diffusion models often fail to capture the
fine-grained details of products. In contrast, personalization-driven models
such as DreamPaint are good at preserving the item's details but they are not
optimized for real-time applications. We present "Diffuse to Choose," a novel
diffusion-based image-conditioned inpainting model that efficiently balances
fast inference with the retention of high-fidelity details in a given reference
item while ensuring accurate semantic manipulations in the given scene content.
Our approach is based on incorporating fine-grained features from the reference
image directly into the latent feature maps of the main diffusion model,
alongside with a perceptual loss to further preserve the reference item's
details. We conduct extensive testing on both in-house and publicly available
datasets, and show that Diffuse to Choose is superior to existing zero-shot
diffusion inpainting methods as well as few-shot diffusion personalization
algorithms like DreamPaint.
Rethinking Patch Dependence for Masked Autoencoders
Authors: name: Letian Fu, name: Long Lian, name: Renhao Wang, name: Baifeng Shi, name: Xudong Wang, name: Adam Yala, name: Trevor Darrell, name: Alexei A. Efros, name: Ken Goldberg
Published: 2024-01-25 18:49:57

In this work, we re-examine inter-patch dependencies in the decoding
mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
for masked patch reconstruction in MAE into self-attention and cross-attention.
Our investigations suggest that self-attention between mask patches is not
essential for learning good representations. To this end, we propose a novel
pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
CrossMAE's decoder leverages only cross-attention between masked and visible
tokens, with no degradation in downstream performance. This design also enables
decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
each decoder block can now leverage different encoder features, resulting in
improved representation learning. CrossMAE matches MAE in performance with 2.5
to 3.7times less decoding compute. It also surpasses MAE on ImageNet
classification and COCO instance segmentation under the same compute. Code and
models: https://crossmae.github.io
Genie: Achieving Human Parity in Content-Grounded Datasets Generation
Authors: name: Asaf Yehudai, name: Boaz Carmeli, name: Yosi Mass, name: Ofir Arviv, name: Nathaniel Mills, name: Assaf Toledo, name: Eyal Shnarch, name: Leshem Choshen
Published: 2024-01-25 18:14:57

The lack of high-quality data for content-grounded generation tasks has been
identified as a major obstacle to advancing these tasks. To address this gap,
we propose Genie, a novel method for automatically generating high-quality
content-grounded data. It consists of three stages: (a) Content Preparation,
(b) Generation: creating task-specific examples from the content (e.g.,
question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure
the quality and faithfulness of the generated data. We showcase this
methodology by generating three large-scale synthetic data, making wishes, for
Long-Form Question-Answering (LFQA), summarization, and information extraction.
In a human evaluation, our generated data was found to be natural and of high
quality. Furthermore, we compare models trained on our data with models trained
on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for
Summarization. We show that our models are on par with or outperforming models
trained on human-generated data and consistently outperforming them in
faithfulness. Finally, we applied our method to create LFQA data within the
medical domain and compared a model trained on it with models trained on other
domains.
Deconstructing Denoising Diffusion Models for Self-Supervised Learning
Authors: name: Xinlei Chen, name: Zhuang Liu, name: Saining Xie, name: Kaiming He
Published: 2024-01-25 18:59:57

In this study, we examine the representation learning abilities of Denoising
Diffusion Models (DDM) that were originally purposed for image generation. Our
philosophy is to deconstruct a DDM, gradually transforming it into a classical
Denoising Autoencoder (DAE). This deconstructive procedure allows us to explore
how various components of modern DDMs influence self-supervised representation
learning. We observe that only a very few modern components are critical for
learning good representations, while many others are nonessential. Our study
ultimately arrives at an approach that is highly simplified and to a large
extent resembles a classical DAE. We hope our study will rekindle interest in a
family of classical methods within the realm of modern self-supervised
learning.
DeepSeek-Coder: When the Large Language Model Meets Programming -- The
  Rise of Code Intelligence
Authors: name: Daya Guo, name: Qihao Zhu, name: Dejian Yang, name: Zhenda Xie, name: Kai Dong, name: Wentao Zhang, name: Guanting Chen, name: Xiao Bi, name: Y. Wu, name: Y. K. Li, name: Fuli Luo, name: Yingfei Xiong, name: Wenfeng Liang
Published: 2024-01-25 14:17:53

The rapid development of large language models has revolutionized code
intelligence in software development. However, the predominance of
closed-source models has restricted extensive research and development. To
address this, we introduce the DeepSeek-Coder series, a range of open-source
code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion
tokens. These models are pre-trained on a high-quality project-level code
corpus and employ a fill-in-the-blank task with a 16K window to enhance code
generation and infilling. Our extensive evaluations demonstrate that
DeepSeek-Coder not only achieves state-of-the-art performance among open-source
code models across multiple benchmarks but also surpasses existing
closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models
are under a permissive license that allows for both research and unrestricted
commercial use.
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric
  Algorithm-System Co-Design
Authors: name: Haojun Xia, name: Zhen Zheng, name: Xiaoxia Wu, name: Shiyang Chen, name: Zhewei Yao, name: Stephen Youn, name: Arash Bakhtiari, name: Michael Wyatt, name: Donglin Zhuang, name: Zhongzhu Zhou, name: Olatunji Ruwase, name: Yuxiong He, name: Shuaiwen Leon Song
Published: 2024-01-25 11:46:38

Six-bit quantization (FP6) can effectively reduce the size of large language
models (LLMs) and preserve the model quality consistently across varied
applications. However, existing systems do not provide Tensor Core support for
FP6 quantization and struggle to achieve practical performance improvements
during LLM inference. It is challenging to support FP6 quantization on GPUs due
to (1) unfriendly memory access of model weights with irregular bit-width and
(2) high runtime overhead of weight de-quantization. To address these problems,
we propose TC-FPx, the first full-stack GPU kernel design scheme with unified
Tensor Core support of float-point weights for various quantization bit-width.
We integrate TC-FPx kernel into an existing inference system, providing new
end-to-end support (called FP6-LLM) for quantized LLM inference, where better
trade-offs between inference cost and model quality are achieved. Experiments
show that FP6-LLM enables the inference of LLaMA-70b using only a single GPU,
achieving 1.69x-2.65x higher normalized inference throughput than the FP16
baseline. The source code will be publicly available soon.
Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation
  for Generative AI
Authors: name: Elron Bandel, name: Yotam Perlitz, name: Elad Venezian, name: Roni Friedman-Melamed, name: Ofir Arviv, name: Matan Orbach, name: Shachar Don-Yehyia, name: Dafna Sheinwald, name: Ariel Gera, name: Leshem Choshen, name: Michal Shmueli-Scheuer, name: Yoav Katz
Published: 2024-01-25 08:57:33

In the dynamic landscape of generative NLP, traditional text processing
pipelines limit research flexibility and reproducibility, as they are tailored
to specific dataset, task, and model combinations. The escalating complexity,
involving system prompts, model-specific formats, instructions, and more, calls
for a shift to a structured, modular, and customizable solution. Addressing
this need, we present Unitxt, an innovative library for customizable textual
data preparation and evaluation tailored to generative language models. Unitxt
natively integrates with common libraries like HuggingFace and LM-eval-harness
and deconstructs processing flows into modular components, enabling easy
customization and sharing between practitioners. These components encompass
model-specific formats, task prompts, and many other comprehensive dataset
processing definitions. The Unitxt-Catalog centralizes these components,
fostering collaboration and exploration in modern textual data workflows.
Beyond being a tool, Unitxt is a community-driven platform, empowering users to
build, share, and advance their pipelines collaboratively. Join the Unitxt
community at https://github.com/IBM/unitxt!
WebVoyager: Building an End-to-End Web Agent with Large Multimodal
  Models
Authors: name: Hongliang He, name: Wenlin Yao, name: Kaixin Ma, name: Wenhao Yu, name: Yong Dai, name: Hongming Zhang, name: Zhenzhong Lan, name: Dong Yu
Published: 2024-01-25 03:33:18

The advancement of large language models (LLMs) leads to a new era marked by
the development of autonomous applications in the real world, which drives
innovation in the creation of advanced web-based agents. Existing web agents
typically only handle one input modality and are evaluated only in simplified
web simulators or static web snapshots, greatly limiting their applicability in
real-world scenarios. To bridge this gap, we introduce WebVoyager, an
innovative Large Multimodal Model (LMM) powered web agent that can complete
user instructions end-to-end by interacting with real-world websites. Moreover,
we propose a new evaluation protocol for web agents to address the challenges
of automatic evaluation of open-ended web agent tasks, leveraging the robust
multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
gathering real-world tasks from 15 widely used websites to evaluate our agents.
We show that WebVoyager achieves a 55.7% task success rate, significantly
surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
(text-only) setups, underscoring the exceptional capability of WebVoyager in
practical applications. We found that our proposed automatic evaluation
achieves 85.3% agreement with human judgment, paving the way for further
development of web agents in a real-world setting.
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation
Authors: name: Minglin Chen, name: Longguang Wang, name: Weihao Yuan, name: Yukun Wang, name: Zhe Sheng, name: Yisheng He, name: Zilong Dong, name: Liefeng Bo, name: Yulan Guo
Published: 2024-01-25 15:49:12

Recently, text-to-3D approaches have achieved high-fidelity 3D content
generation using text description. However, the generated objects are
stochastic and lack fine-grained control. Sketches provide a cheap approach to
introduce such fine-grained control. Nevertheless, it is challenging to achieve
flexible control from these sketches due to their abstraction and ambiguity. In
this paper, we present a multi-view sketch-guided text-to-3D generation
framework (namely, Sketch2NeRF) to add sketch control to 3D generation.
Specifically, our method leverages pretrained 2D diffusion models (e.g., Stable
Diffusion and ControlNet) to supervise the optimization of a 3D scene
represented by a neural radiance field (NeRF). We propose a novel synchronized
generation and reconstruction method to effectively optimize the NeRF. In the
experiments, we collected two kinds of multi-view sketch datasets to evaluate
the proposed method. We demonstrate that our method can synthesize 3D
consistent contents with fine-grained sketch control while being high-fidelity
to text prompts. Extensive results show that our method achieves
state-of-the-art performance in terms of sketch similarity and text alignment.
CreativeSynth: Creative Blending and Synthesis of Visual Arts based on
  Multimodal Diffusion
Authors: name: Nisha Huang, name: Weiming Dong, name: Yuxin Zhang, name: Fan Tang, name: Ronghui Li, name: Chongyang Ma, name: Xiu Li, name: Changsheng Xu
Published: 2024-01-25 10:42:09

Large-scale text-to-image generative models have made impressive strides,
showcasing their ability to synthesize a vast array of high-quality images.
However, adapting these models for artistic image editing presents two
significant challenges. Firstly, users struggle to craft textual prompts that
meticulously detail visual elements of the input image. Secondly, prevalent
models, when effecting modifications in specific zones, frequently disrupt the
overall artistic style, complicating the attainment of cohesive and
aesthetically unified artworks. To surmount these obstacles, we build the
innovative unified framework CreativeSynth, which is based on a diffusion model
with the ability to coordinate multimodal inputs and multitask in the field of
artistic image generation. By integrating multimodal features with customized
attention mechanisms, CreativeSynth facilitates the importation of real-world
semantic content into the domain of art through inversion and real-time style
transfer. This allows for the precise manipulation of image style and content
while maintaining the integrity of the original model parameters. Rigorous
qualitative and quantitative evaluations underscore that CreativeSynth excels
in enhancing artistic images' fidelity and preserves their innate aesthetic
essence. By bridging the gap between generative models and artistic finesse,
CreativeSynth becomes a custom digital palette.
BootPIG: Bootstrapping Zero-shot Personalized Image Generation
  Capabilities in Pretrained Diffusion Models
Authors: name: Senthil Purushwalkam, name: Akash Gokul, name: Shafiq Joty, name: Nikhil Naik
Published: 2024-01-25 06:18:20

Recent text-to-image generation models have demonstrated incredible success
in generating images that faithfully follow input prompts. However, the
requirement of using words to describe a desired concept provides limited
control over the appearance of the generated concepts. In this work, we address
this shortcoming by proposing an approach to enable personalization
capabilities in existing text-to-image diffusion models. We propose a novel
architecture (BootPIG) that allows a user to provide reference images of an
object in order to guide the appearance of a concept in the generated images.
  The proposed BootPIG architecture makes minimal modifications to a pretrained
text-to-image diffusion model and utilizes a separate UNet model to steer the
generations toward the desired appearance. We introduce a training procedure
that allows us to bootstrap personalization capabilities in the BootPIG
architecture using data generated from pretrained text-to-image models, LLM
chat agents, and image segmentation models. In contrast to existing methods
that require several days of pretraining, the BootPIG architecture can be
trained in approximately 1 hour. Experiments on the DreamBooth dataset
demonstrate that BootPIG outperforms existing zero-shot methods while being
comparable with test-time finetuning approaches. Through a user study, we
validate the preference for BootPIG generations over existing methods both in
maintaining fidelity to the reference object's appearance and aligning with
textual prompts.
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic
  Image Restoration In the Wild
Authors: name: Fanghua Yu, name: Jinjin Gu, name: Zheyuan Li, name: Jinfan Hu, name: Xiangtao Kong, name: Xintao Wang, name: Jingwen He, name: Yu Qiao, name: Chao Dong
Published: 2024-01-24 17:58:07

We introduce SUPIR (Scaling-UP Image Restoration), a groundbreaking image
restoration method that harnesses generative prior and the power of model
scaling up. Leveraging multi-modal techniques and advanced generative prior,
SUPIR marks a significant advance in intelligent and realistic image
restoration. As a pivotal catalyst within SUPIR, model scaling dramatically
enhances its capabilities and demonstrates new potential for image restoration.
We collect a dataset comprising 20 million high-resolution, high-quality images
for model training, each enriched with descriptive text annotations. SUPIR
provides the capability to restore images guided by textual prompts, broadening
its application scope and potential. Moreover, we introduce negative-quality
prompts to further improve perceptual quality. We also develop a
restoration-guided sampling method to suppress the fidelity issue encountered
in generative-based restoration. Experiments demonstrate SUPIR's exceptional
restoration effects and its novel capacity to manipulate restoration through
textual prompts.