Orionnnn/hub-stats huggingface dataset:
InternVL: Scaling up Vision Foundation Models and Aligning for Generic
  Visual-Linguistic Tasks
Authors: name: Zhe Chen, name: Jiannan Wu, name: Wenhai Wang, name: Weijie Su, name: Guo Chen, name: Sen Xing, name: Zhong Muyan, name: Qinglong Zhang, name: Xizhou Zhu, name: Lewei Lu, name: Bin Li, name: Ping Luo, name: Tong Lu, name: Yu Qiao, name: Jifeng Dai
Published: 2023-12-21 18:59:31

The exponential growth of large language models (LLMs) has opened up numerous
possibilities for multi-modal AGI systems. However, the progress in vision and
vision-language foundation models, which are also critical elements of
multi-modal AGI, has not kept pace with LLMs. In this work, we design a
large-scale vision-language foundation model (InternVL), which scales up the
vision foundation model to 6 billion parameters and progressively aligns it
with the large language model, using web-scale image-text data from various
sources. This model can be broadly applied to and achieve state-of-the-art
performance on visual perception tasks such as image-level or pixel-level
recognition, vision-language tasks such as zero-shot image/video
classification, zero-shot image/video-text retrieval, and link with LLMs to
create multi-modal dialogue systems. We hope that our research could contribute
to the development of multi-modal large models. Code and models are available
at https://github.com/OpenGVLab/InternVL.
WaveCoder: Widespread And Versatile Enhanced Instruction Tuning with
  Refined Data Generation
Authors: name: Zhaojian Yu, name: Xin Zhang, name: Ning Shang, name: Yangyu Huang, name: Can Xu, name: Yishujie Zhao, name: Wenxiang Hu, name: Qiufeng Yin
Published: 2023-12-20 09:02:29

Recent work demonstrates that, after being fine-tuned on a high-quality
instruction dataset, the resulting model can obtain impressive capabilities to
address a wide range of tasks. However, existing methods for instruction data
generation often produce duplicate data and are not controllable enough on data
quality. In this paper, we extend the generalization of instruction tuning by
classifying the instruction data to 4 code-related tasks and propose a
LLM-based Generator-Discriminator data process framework to generate diverse,
high-quality instruction data from open source code. Hence, we introduce
CodeOcean, a dataset comprising 20,000 instruction instances across 4 universal
code-related tasks,which is aimed at augmenting the effectiveness of
instruction tuning and improving the generalization ability of fine-tuned
model. Subsequently, we present WaveCoder, a fine-tuned Code LLM with
Widespread And Versatile Enhanced instruction tuning. This model is
specifically designed for enhancing instruction tuning of Code Language Models
(LLMs). Our experiments demonstrate that Wavecoder models outperform other
open-source models in terms of generalization ability across different
code-related tasks at the same level of fine-tuning scale. Moreover, Wavecoder
exhibits high efficiency in previous code generation tasks. This paper thus
offers a significant contribution to the field of instruction data generation
and fine-tuning models, providing new insights and tools for enhancing
performance in code-related tasks.
Shai: A large language model for asset management
Authors: name: Zhongyang Guo, name: Guanran Jiang, name: Zhongdan Zhang, name: Peng Li, name: Zhefeng Wang, name: Yinchun Wang
Published: 2023-12-21 05:08:57

This paper introduces "Shai" a 10B level large language model specifically
designed for the asset management industry, built upon an open-source
foundational model. With continuous pre-training and fine-tuning using a
targeted corpus, Shai demonstrates enhanced performance in tasks relevant to
its domain, outperforming baseline models. Our research includes the
development of an innovative evaluation framework, which integrates
professional qualification exams, tailored tasks, open-ended question
answering, and safety assessments, to comprehensively assess Shai's
capabilities. Furthermore, we discuss the challenges and implications of
utilizing large language models like GPT-4 for performance assessment in asset
management, suggesting a combination of automated evaluation and human
judgment. Shai's development, showcasing the potential and versatility of
10B-level large language models in the financial sector with significant
performance and modest computational requirements, hopes to provide practical
insights and methodologies to assist industry peers in their similar endeavors.
Exploiting Novel GPT-4 APIs
Authors: name: Kellin Pelrine, name: Mohammad Taufeeque, name: Michał Zając, name: Euan McLean, name: Adam Gleave
Published: 2023-12-21 21:22:41

Language model attacks typically assume one of two extreme threat models:
full white-box access to model weights, or black-box access limited to a text
generation API. However, real-world APIs are often more flexible than just text
generation: these APIs expose "gray-box" access leading to new threat vectors.
To explore this, we red-team three new functionalities exposed in the GPT-4
APIs: fine-tuning, function calling and knowledge retrieval. We find that
fine-tuning a model on as few as 15 harmful examples or 100 benign examples can
remove core safeguards from GPT-4, enabling a range of harmful outputs.
Furthermore, we find that GPT-4 Assistants readily divulge the function call
schema and can be made to execute arbitrary function calls. Finally, we find
that knowledge retrieval can be hijacked by injecting instructions into
retrieval documents. These vulnerabilities highlight that any additions to the
functionality exposed by an API can create new vulnerabilities.
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Authors: name: Jitesh Jain, name: Jianwei Yang, name: Humphrey Shi
Published: 2023-12-21 18:49:47

Humans possess the remarkable skill of Visual Perception, the ability to see
and understand the seen, helping them make sense of the visual world and, in
turn, reason. Multimodal Large Language Models (MLLM) have recently achieved
impressive performance on vision-language tasks ranging from visual
question-answering and image captioning to visual reasoning and image
generation. However, when prompted to identify or count (perceive) the entities
in a given image, existing MLLM systems fail. Working towards developing an
accurate MLLM system for perception and reasoning, we propose using Versatile
vision enCoders (VCoder) as perception eyes for Multimodal LLMs. We feed the
VCoder with perception modalities such as segmentation or depth maps, improving
the MLLM's perception abilities. Secondly, we leverage the images from COCO and
outputs from off-the-shelf vision perception models to create our COCO
Segmentation Text (COST) dataset for training and evaluating MLLMs on the
object perception task. Thirdly, we introduce metrics to assess the object
perception abilities in MLLMs on our COST dataset. Lastly, we provide extensive
experimental evidence proving the VCoder's improved object-level perception
skills over existing Multimodal LLMs, including GPT-4V. We open-source our
dataset, code, and models to promote research. We open-source our code at
https://github.com/SHI-Labs/VCoder
LLM4VG: Large Language Models Evaluation for Video Grounding
Authors: name: Wei Feng, name: Xin Wang, name: Hong Chen, name: Zeyang Zhang, name: Zihan Song, name: Yuwei Zhou, name: Wenwu Zhu
Published: 2023-12-21 08:15:02

Recently, researchers have attempted to investigate the capability of LLMs in
handling videos and proposed several video LLM models. However, the ability of
LLMs to handle video grounding (VG), which is an important time-related video
task requiring the model to precisely locate the start and end timestamps of
temporal moments in videos that match the given textual queries, still remains
unclear and unexplored in literature. To fill the gap, in this paper, we
propose the LLM4VG benchmark, which systematically evaluates the performance of
different LLMs on video grounding tasks. Based on our proposed LLM4VG, we
design extensive experiments to examine two groups of video LLM models on video
grounding: (i) the video LLMs trained on the text-video pairs (denoted as
VidLLM), and (ii) the LLMs combined with pretrained visual description models
such as the video/image captioning model. We propose prompt methods to
integrate the instruction of VG and description from different kinds of
generators, including caption-based generators for direct visual description
and VQA-based generators for information enhancement. We also provide
comprehensive comparisons of various VidLLMs and explore the influence of
different choices of visual models, LLMs, prompt designs, etc, as well. Our
experimental evaluations lead to two conclusions: (i) the existing VidLLMs are
still far away from achieving satisfactory video grounding performance, and
more time-related video tasks should be included to further fine-tune these
models, and (ii) the combination of LLMs and visual models shows preliminary
abilities for video grounding with considerable potential for improvement by
resorting to more reliable models and further guidance of prompt instructions.
DyBluRF: Dynamic Deblurring Neural Radiance Fields for Blurry Monocular
  Video
Authors: name: Minh-Quan Viet Bui, name: Jongmin Park, name: Jihyong Oh, name: Munchurl Kim
Published: 2023-12-21 02:01:19

Video view synthesis, allowing for the creation of visually appealing frames
from arbitrary viewpoints and times, offers immersive viewing experiences.
Neural radiance fields, particularly NeRF, initially developed for static
scenes, have spurred the creation of various methods for video view synthesis.
However, the challenge for video view synthesis arises from motion blur, a
consequence of object or camera movement during exposure, which hinders the
precise synthesis of sharp spatio-temporal views. In response, we propose a
novel dynamic deblurring NeRF framework for blurry monocular video, called
DyBluRF, consisting of an Interleave Ray Refinement (IRR) stage and a Motion
Decomposition-based Deblurring (MDD) stage. Our DyBluRF is the first that
addresses and handles the novel view synthesis for blurry monocular video. The
IRR stage jointly reconstructs dynamic 3D scenes and refines the inaccurate
camera pose information to combat imprecise pose information extracted from the
given blurry frames. The MDD stage is a novel incremental latent sharp-rays
prediction (ILSP) approach for the blurry monocular video frames by decomposing
the latent sharp rays into global camera motion and local object motion
components. Extensive experimental results demonstrate that our DyBluRF
outperforms qualitatively and quantitatively the very recent state-of-the-art
methods. Our project page including source codes and pretrained model are
publicly available at https://kaist-viclab.github.io/dyblurf-site/.
ShowRoom3D: Text to High-Quality 3D Room Generation Using 3D Priors
Authors: name: Weijia Mao, name: Yan-Pei Cao, name: Jia-Wei Liu, name: Zhongcong Xu, name: Mike Zheng Shou
Published: 2023-12-20 15:31:38

We introduce ShowRoom3D, a three-stage approach for generating high-quality
3D room-scale scenes from texts. Previous methods using 2D diffusion priors to
optimize neural radiance fields for generating room-scale scenes have shown
unsatisfactory quality. This is primarily attributed to the limitations of 2D
priors lacking 3D awareness and constraints in the training methodology. In
this paper, we utilize a 3D diffusion prior, MVDiffusion, to optimize the 3D
room-scale scene. Our contributions are in two aspects. Firstly, we propose a
progressive view selection process to optimize NeRF. This involves dividing the
training process into three stages, gradually expanding the camera sampling
scope. Secondly, we propose the pose transformation method in the second stage.
It will ensure MVDiffusion provide the accurate view guidance. As a result,
ShowRoom3D enables the generation of rooms with improved structural integrity,
enhanced clarity from any view, reduced content repetition, and higher
consistency across different perspectives. Extensive experiments demonstrate
that our method, significantly outperforms state-of-the-art approaches by a
large margin in terms of user study.
Unlocking Pre-trained Image Backbones for Semantic Image Synthesis
Authors: name: Tariq Berrada, name: Jakob Verbeek, name: Camille Couprie, name: Karteek Alahari
Published: 2023-12-20 09:39:19

Semantic image synthesis, i.e., generating images from user-provided semantic
label maps, is an important conditional image generation task as it allows to
control both the content as well as the spatial layout of generated images.
Although diffusion models have pushed the state of the art in generative image
modeling, the iterative nature of their inference process makes them
computationally demanding. Other approaches such as GANs are more efficient as
they only need a single feed-forward pass for generation, but the image quality
tends to suffer on large and diverse datasets. In this work, we propose a new
class of GAN discriminators for semantic image synthesis that generates highly
realistic images by exploiting feature backbone networks pre-trained for tasks
such as image classification. We also introduce a new generator architecture
with better context modeling and using cross-attention to inject noise into
latent variables, leading to more diverse generated images. Our model, which we
dub DP-SIMS, achieves state-of-the-art results in terms of image quality and
consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes,
surpassing recent diffusion models while requiring two orders of magnitude less
compute for inference.
AppAgent: Multimodal Agents as Smartphone Users
Authors: name: Chi Zhang, name: Zhao Yang, name: Jiaxuan Liu, name: Yucheng Han, name: Xin Chen, name: Zebiao Huang, name: Bin Fu, name: Gang Yu
Published: 2023-12-21 11:52:45

Recent advancements in large language models (LLMs) have led to the creation
of intelligent agents capable of performing complex tasks. This paper
introduces a novel LLM-based multimodal agent framework designed to operate
smartphone applications. Our framework enables the agent to operate smartphone
applications through a simplified action space, mimicking human-like
interactions such as tapping and swiping. This novel approach bypasses the need
for system back-end access, thereby broadening its applicability across diverse
apps. Central to our agent's functionality is its innovative learning method.
The agent learns to navigate and use new apps either through autonomous
exploration or by observing human demonstrations. This process generates a
knowledge base that the agent refers to for executing complex tasks across
different applications. To demonstrate the practicality of our agent, we
conducted extensive testing over 50 tasks in 10 different applications,
including social media, email, maps, shopping, and sophisticated image editing
tools. The results affirm our agent's proficiency in handling a diverse array
of high-level tasks.
Time is Encoded in the Weights of Finetuned Language Models
Authors: name: Kai Nylund, name: Suchin Gururangan, name: Noah A. Smith
Published: 2023-12-20 20:04:45

We present time vectors, a simple tool to customize language models to new
time periods. Time vectors are created by finetuning a language model on data
from a single time (e.g., a year or month), and then subtracting the weights of
the original pretrained model. This vector specifies a direction in weight
space that, as our experiments show, improves performance on text from that
time period. Time vectors specialized to adjacent time periods appear to be
positioned closer together in a manifold. Using this structure, we interpolate
between time vectors to induce new models that perform better on intervening
and future time periods, without any additional training. We demonstrate the
consistency of our findings across different tasks, domains, model sizes, and
time scales. Our results suggest that time is encoded in the weight space of
finetuned models.
HeadCraft: Modeling High-Detail Shape Variations for Animated 3DMMs
Authors: name: Artem Sevastopolsky, name: Philip-William Grassal, name: Simon Giebenhain, name: ShahRukh Athar, name: Luisa Verdoliva, name: Matthias Niessner
Published: 2023-12-21 18:57:52

Current advances in human head modeling allow to generate plausible-looking
3D head models via neural representations. Nevertheless, constructing complete
high-fidelity head models with explicitly controlled animation remains an
issue. Furthermore, completing the head geometry based on a partial
observation, e.g. coming from a depth sensor, while preserving details is often
problematic for the existing methods. We introduce a generative model for
detailed 3D head meshes on top of an articulated 3DMM which allows explicit
animation and high-detail preservation at the same time. Our method is trained
in two stages. First, we register a parametric head model with vertex
displacements to each mesh of the recently introduced NPHM dataset of accurate
3D head scans. The estimated displacements are baked into a hand-crafted UV
layout. Second, we train a StyleGAN model in order to generalize over the UV
maps of displacements. The decomposition of the parametric model and
high-quality vertex displacements allows us to animate the model and modify it
semantically. We demonstrate the results of unconditional generation and
fitting to the full or partial observation. The project page is available at
https://seva100.github.io/headcraft.
HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image
  Inpainting with Diffusion Models
Authors: name: Hayk Manukyan, name: Andranik Sargsyan, name: Barsegh Atanyan, name: Zhangyang Wang, name: Shant Navasardyan, name: Humphrey Shi
Published: 2023-12-21 18:09:30

Recent progress in text-guided image inpainting, based on the unprecedented
success of text-to-image diffusion models, has led to exceptionally realistic
and visually plausible results. However, there is still significant potential
for improvement in current text-to-image inpainting models, particularly in
better aligning the inpainted area with user prompts and performing
high-resolution inpainting. Therefore, in this paper we introduce HD-Painter, a
completely training-free approach that accurately follows to prompts and
coherently scales to high-resolution image inpainting. To this end, we design
the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention
scores by prompt information and resulting in better text alignment
generations. To further improve the prompt coherence we introduce the
Reweighting Attention Score Guidance (RASG) mechanism seamlessly integrating a
post-hoc sampling strategy into general form of DDIM to prevent
out-of-distribution latent shifts. Moreover, HD-Painter allows extension to
larger scales by introducing a specialized super-resolution technique
customized for inpainting, enabling the completion of missing regions in images
of up to 2K resolution. Our experiments demonstrate that HD-Painter surpasses
existing state-of-the-art approaches qualitatively and quantitatively,
achieving an impressive generation accuracy improvement of 61.4% vs 51.9%. We
will make the codes publicly available at:
https://github.com/Picsart-AI-Research/HD-Painter
Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion
  Models with RL Finetuning
Authors: name: Desai Xie, name: Jiahao Li, name: Hao Tan, name: Xin Sun, name: Zhixin Shu, name: Yi Zhou, name: Sai Bi, name: Sören Pirk, name: Arie E. Kaufman
Published: 2023-12-21 16:10:33

Recent advancements in the text-to-3D task leverage finetuned text-to-image
diffusion models to generate multi-view images, followed by NeRF
reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
training longer with SFT improves consistency, it also causes distribution
shift, which reduces diversity and realistic details. We argue that the SFT of
multi-view diffusion models resembles the instruction finetuning stage of the
LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
Essentially, RLFT methods optimize models beyond their SFT data distribution by
using their own outputs, effectively mitigating distribution shift. To this
end, we introduce Carve3D, a RLFT method coupled with the Multi-view
Reconstruction Consistency (MRC) metric, to improve the consistency of
multi-view diffusion models. To compute MRC on a set of multi-view images, we
compare them with their corresponding renderings of the reconstructed NeRF at
the same viewpoints. We validate the robustness of MRC with extensive
experiments conducted under controlled inconsistency levels. We enhance the
base RLFT algorithm to stabilize the training process, reduce distribution
shift, and identify scaling laws. Through qualitative and quantitative
experiments, along with a user study, we demonstrate Carve3D's improved
multi-view consistency, the resulting superior NeRF reconstruction quality, and
minimal distribution shift compared to longer SFT. Project webpage:
https://desaixie.github.io/carve-3d.
PIA: Your Personalized Image Animator via Plug-and-Play Modules in
  Text-to-Image Models
Authors: name: Yiming Zhang, name: Zhening Xing, name: Yanhong Zeng, name: Youqing Fang, name: Kai Chen
Published: 2023-12-21 15:51:12

Recent advancements in personalized text-to-image (T2I) models have
revolutionized content creation, empowering non-experts to generate stunning
images with unique styles. While promising, adding realistic motions into these
personalized images by text poses significant challenges in preserving distinct
styles, high-fidelity details, and achieving motion controllability by text. In
this paper, we present PIA, a Personalized Image Animator that excels in
aligning with condition images, achieving motion controllability by text, and
the compatibility with various personalized T2I models without specific tuning.
To achieve these goals, PIA builds upon a base T2I model with well-trained
temporal alignment layers, allowing for the seamless transformation of any
personalized T2I model into an image animation model. A key component of PIA is
the introduction of the condition module, which utilizes the condition frame
and inter-frame affinity as input to transfer appearance information guided by
the affinity hint for individual frame synthesis in the latent space. This
design mitigates the challenges of appearance-related image alignment within
and allows for a stronger focus on aligning with motion-related guidance.
Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
Authors: name: Xianfang Zeng, name: Xin Chen, name: Zhongqi Qi, name: Wen Liu, name: Zibo Zhao, name: Zhibin Wang, name: BIN FU, name: Yong Liu, name: Gang Yu
Published: 2023-12-21 15:01:47

This paper presents Paint3D, a novel coarse-to-fine generative framework that
is capable of producing high-resolution, lighting-less, and diverse 2K UV
texture maps for untextured 3D meshes conditioned on text or image inputs. The
key challenge addressed is generating high-quality textures without embedded
illumination information, which allows the textures to be re-lighted or
re-edited within modern graphics pipelines. To achieve this, our method first
leverages a pre-trained depth-aware 2D diffusion model to generate
view-conditional images and perform multi-view texture fusion, producing an
initial coarse texture map. However, as 2D models cannot fully represent 3D
shapes and disable lighting effects, the coarse texture map exhibits incomplete
areas and illumination artifacts. To resolve this, we train separate UV
Inpainting and UVHD diffusion models specialized for the shape-aware refinement
of incomplete areas and the removal of illumination artifacts. Through this
coarse-to-fine process, Paint3D can produce high-quality 2K UV textures that
maintain semantic consistency while being lighting-less, significantly
advancing the state-of-the-art in texturing 3D objects.