Orionnnn/hub-stats huggingface dataset:
DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
Authors: name: Shentong Mo, name: Enze Xie, name: Ruihang Chu, name: Lewei Yao, name: Lanqing Hong, name: Matthias Nießner, name: Zhenguo Li
Published: 2023-07-04 17:15:46

Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful
effectiveness in generating high-quality 2D images. However, it is still being
determined whether the Transformer architecture performs equally well in 3D
shape generation, as previous 3D diffusion methods mostly adopted the U-Net
architecture. To bridge this gap, we propose a novel Diffusion Transformer for
3D shape generation, namely DiT-3D, which can directly operate the denoising
process on voxelized point clouds using plain Transformers. Compared to
existing U-Net approaches, our DiT-3D is more scalable in model size and
produces much higher quality generations. Specifically, the DiT-3D adopts the
design philosophy of DiT but modifies it by incorporating 3D positional and
patch embeddings to adaptively aggregate input from voxelized point clouds. To
reduce the computational cost of self-attention in 3D shape generation, we
incorporate 3D window attention into Transformer blocks, as the increased 3D
token length resulting from the additional dimension of voxels can lead to high
computation. Finally, linear and devoxelization layers are used to predict the
denoised point clouds. In addition, our transformer architecture supports
efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on
ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on
the ShapeNet dataset demonstrate that the proposed DiT-3D achieves
state-of-the-art performance in high-fidelity and diverse 3D point cloud
generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy
of the state-of-the-art method by 4.59 and increases the Coverage metric by
3.51 when evaluated on Chamfer Distance.
What Matters in Training a GPT4-Style Language Model with Multimodal
  Inputs?
Authors: name: Yan Zeng, name: Hanbo Zhang, name: Jiani Zheng, name: Jiangnan Xia, name: Guoqiang Wei, name: Yang Wei, name: Yuchen Zhang, name: Tao Kong
Published: 2023-07-05 17:44:28

Recent advancements in Large Language Models (LLMs) such as GPT4 have
displayed exceptional multi-modal capabilities in following open-ended
instructions given images. However, the performance of these models heavily
relies on design choices such as network structures, training data, and
training strategies, and these choices have not been extensively discussed in
the literature, making it difficult to quantify progress in this field. To
address this issue, this paper presents a systematic and comprehensive study,
quantitatively and qualitatively, on training such models. We implement over 20
variants with controlled settings. Concretely, for network structures, we
compare different LLM backbones and model designs. For training data, we
investigate the impact of data and sampling strategies. For instructions, we
explore the influence of diversified prompts on the instruction-following
ability of the trained models. For benchmarks, we contribute the first, to our
best knowledge, comprehensive evaluation set including both image and video
tasks through crowd-sourcing. Based on our findings, we present Lynx, which
performs the most accurate multi-modal understanding while keeping the best
multi-modal generation ability compared to existing open-sourced GPT4-style
models.
Robots That Ask For Help: Uncertainty Alignment for Large Language Model
  Planners
Authors: name: Allen Z. Ren, name: Anushri Dixit, name: Alexandra Bodrova, name: Sumeet Singh, name: Stephen Tu, name: Noah Brown, name: Peng Xu, name: Leila Takayama, name: Fei Xia, name: Jake Varley, name: Zhenjia Xu, name: Dorsa Sadigh, name: Andy Zeng, name: Anirudha Majumdar
Published: 2023-07-04 21:25:12

Large language models (LLMs) exhibit a wide range of promising capabilities
-- from step-by-step planning to commonsense reasoning -- that may provide
utility for robots, but remain prone to confidently hallucinated predictions.
In this work, we present KnowNo, which is a framework for measuring and
aligning the uncertainty of LLM-based planners such that they know when they
don't know and ask for help when needed. KnowNo builds on the theory of
conformal prediction to provide statistical guarantees on task completion while
minimizing human help in complex multi-step planning settings. Experiments
across a variety of simulated and real robot setups that involve tasks with
different modes of ambiguity (e.g., from spatial to numeric uncertainties, from
human preferences to Winograd schemas) show that KnowNo performs favorably over
modern baselines (which may involve ensembles or extensive prompt tuning) in
terms of improving efficiency and autonomy, while providing formal assurances.
KnowNo can be used with LLMs out of the box without model-finetuning, and
suggests a promising lightweight approach to modeling uncertainty that can
complement and scale with the growing capabilities of foundation models.
Website: https://robot-help.github.io
Embodied Task Planning with Large Language Models
Authors: name: Zhenyu Wu, name: Ziwei Wang, name: Xiuwei Xu, name: Jiwen Lu, name: Haibin Yan
Published: 2023-07-04 17:58:25

Equipping embodied agents with commonsense is important for robots to
successfully complete complex human instructions in general environments.
Recent large language models (LLM) can embed rich semantic knowledge for agents
in plan generation of complex tasks, while they lack the information about the
realistic world and usually yield infeasible action sequences. In this paper,
we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
with physical scene constraint, where the agent generates executable plans
according to the existed objects in the scene by aligning LLMs with the visual
perception models. Specifically, we first construct a multimodal dataset
containing triplets of indoor scenes, instructions and action plans, where we
provide the designed prompts and the list of existing objects in the scene for
GPT-3.5 to generate a large number of instructions and corresponding planned
actions. The generated data is leveraged for grounded plan tuning of
pre-trained LLMs. During inference, we discover the objects in the scene by
extending open-vocabulary object detectors to multi-view RGB images collected
in different achievable locations. Experimental results show that the generated
plan from our TaPA framework can achieve higher success rate than LLaVA and
GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
planning in general and complex environments.
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Authors: name: Jiayu Ding, name: Shuming Ma, name: Li Dong, name: Xingxing Zhang, name: Shaohan Huang, name: Wenhui Wang, name: Furu Wei
Published: 2023-07-05 17:59:38

Scaling sequence length has become a critical demand in the era of large
language models. However, existing methods struggle with either computational
complexity or model expressivity, rendering the maximum sequence length
restricted. In this work, we introduce LongNet, a Transformer variant that can
scale sequence length to more than 1 billion tokens, without sacrificing the
performance on shorter sequences. Specifically, we propose dilated attention,
which expands the attentive field exponentially as the distance grows. LongNet
has significant advantages: 1) it has a linear computation complexity and a
logarithm dependency between tokens; 2) it can be served as a distributed
trainer for extremely long sequences; 3) its dilated attention is a drop-in
replacement for standard attention, which can be seamlessly integrated with the
existing Transformer-based optimization. Experiments results demonstrate that
LongNet yields strong performance on both long-sequence modeling and general
language tasks. Our work opens up new possibilities for modeling very long
sequences, e.g., treating a whole corpus or even the entire Internet as a
sequence.
Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks
Authors: name: Meysam Alizadeh, name: Maël Kubli, name: Zeynab Samei, name: Shirin Dehghani, name: Juan Diego Bermeo, name: Maria Korobeynikova, name: Fabrizio Gilardi
Published: 2023-07-05 10:15:07

This study examines the performance of open-source Large Language Models
(LLMs) in text annotation tasks and compares it with proprietary models like
ChatGPT and human-based services such as MTurk. While prior research
demonstrated the high performance of ChatGPT across numerous NLP tasks,
open-source LLMs like HugginChat and FLAN are gaining attention for their
cost-effectiveness, transparency, reproducibility, and superior data
protection. We assess these models using both zero-shot and few-shot approaches
and different temperature parameters across a range of text annotation tasks.
Our findings show that while ChatGPT achieves the best performance in most
tasks, open-source LLMs not only outperform MTurk but also demonstrate
competitive potential against ChatGPT in specific tasks.
Building Cooperative Embodied Agents Modularly with Large Language
  Models
Authors: name: Hongxin Zhang, name: Weihua Du, name: Jiaming Shan, name: Qinhong Zhou, name: Yilun Du, name: Joshua B. Tenenbaum, name: Tianmin Shu, name: Chuang Gan
Published: 2023-07-05 17:59:27

Large Language Models (LLMs) have demonstrated impressive planning abilities
in single-agent embodied tasks across various domains. However, their capacity
for planning and communication in multi-agent cooperation remains unclear, even
though these are crucial skills for intelligent embodied agents. In this paper,
we present a novel framework that utilizes LLMs for multi-agent cooperation and
tests it in various embodied environments. Our framework enables embodied
agents to plan, communicate, and cooperate with other embodied agents or humans
to accomplish long-horizon tasks efficiently. We demonstrate that recent LLMs,
such as GPT-4, can surpass strong planning-based methods and exhibit emergent
effective communication using our framework without requiring fine-tuning or
few-shot prompting. We also discover that LLM-based agents that communicate in
natural language can earn more trust and cooperate more effectively with
humans. Our research underscores the potential of LLMs for embodied AI and lays
the foundation for future research in multi-agent cooperation. Videos can be
found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.
Jailbroken: How Does LLM Safety Training Fail?
Authors: name: Alexander Wei, name: Nika Haghtalab, name: Jacob Steinhardt
Published: 2023-07-05 17:58:10

Large language models trained for safety and harmlessness remain susceptible
to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on
early releases of ChatGPT that elicit undesired behavior. Going beyond
recognition of the issue, we investigate why such attacks succeed and how they
can be created. We hypothesize two failure modes of safety training: competing
objectives and mismatched generalization. Competing objectives arise when a
model's capabilities and safety goals conflict, while mismatched generalization
occurs when safety training fails to generalize to a domain for which
capabilities exist. We use these failure modes to guide jailbreak design and
then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's
Claude v1.3, against both existing and newly designed attacks. We find that
vulnerabilities persist despite the extensive red-teaming and safety-training
efforts behind these models. Notably, new attacks utilizing our failure modes
succeed on every prompt in a collection of unsafe requests from the models'
red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our
analysis emphasizes the need for safety-capability parity -- that safety
mechanisms should be as sophisticated as the underlying model -- and argues
against the idea that scaling alone can resolve these safety failure modes.
Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN
  Fine-Tuning
Authors: name: Deepanway Ghosal, name: Yew Ken Chia, name: Navonil Majumder, name: Soujanya Poria
Published: 2023-07-05 06:36:54

Recently, the release of INSTRUCTEVAL has provided valuable insights into the
performance of large language models (LLMs) that utilize encoder-decoder or
decoder-only architecture. Interestingly, despite being introduced four years
ago, T5-based LLMs, such as FLAN-T5, continue to outperform the latest
decoder-based LLMs, such as LLAMA and VICUNA, on tasks that require general
problem-solving skills. This performance discrepancy can be attributed to three
key factors: (1) Pre-training data, (2) Backbone architecture, and (3)
Instruction dataset. In this technical report, our main focus is on
investigating the impact of the third factor by leveraging VICUNA, a large
language model based on LLAMA, which has undergone fine-tuning on ChatGPT
conversations. To achieve this objective, we fine-tuned VICUNA using a
customized instruction dataset collection called FLANMINI. This collection
includes a subset of the large-scale instruction dataset known as FLAN, as well
as various code-related datasets and conversational datasets derived from
ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand
problem-solving skills. Our experimental findings strongly indicate that the
enhanced problem-solving abilities of our model, FLACUNA, are obtained through
fine-tuning VICUNA on the FLAN dataset, leading to significant improvements
across numerous benchmark datasets in INSTRUCTEVAL. FLACUNA is publicly
available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.
Real-time Monocular Full-body Capture in World Space via Sequential
  Proxy-to-Motion Learning
Authors: name: Yuxiang Zhang, name: Hongwen Zhang, name: Liangxiao Hu, name: Hongwei Yi, name: Shengping Zhang, name: Yebin Liu
Published: 2023-07-03 17:59:45

Learning-based approaches to monocular motion capture have recently shown
promising results by learning to regress in a data-driven manner. However, due
to the challenges in data collection and network designs, it remains
challenging for existing solutions to achieve real-time full-body capture while
being accurate in world space. In this work, we contribute a sequential
proxy-to-motion learning scheme together with a proxy dataset of 2D skeleton
sequences and 3D rotational motions in world space. Such proxy data enables us
to build a learning-based network with accurate full-body supervision while
also mitigating the generalization issues. For more accurate and physically
plausible predictions, a contact-aware neural motion descent module is proposed
in our network so that it can be aware of foot-ground contact and motion
misalignment with the proxy observations. Additionally, we share the body-hand
context information in our network for more compatible wrist poses recovery
with the full-body model. With the proposed learning-based solution, we
demonstrate the first real-time monocular full-body capture system with
plausible foot-ground contact in world space. More video results can be found
at our project page: https://liuyebin.com/proxycap.
Segment Anything Meets Point Tracking
Authors: name: Frano Rajič, name: Lei Ke, name: Yu-Wing Tai, name: Chi-Keung Tang, name: Martin Danelljan, name: Fisher Yu
Published: 2023-07-03 17:58:01

The Segment Anything Model (SAM) has established itself as a powerful
zero-shot image segmentation model, employing interactive prompts such as
points to generate masks. This paper presents SAM-PT, a method extending SAM's
capability to tracking and segmenting anything in dynamic videos. SAM-PT
leverages robust and sparse point selection and propagation techniques for mask
generation, demonstrating that a SAM-based segmentation tracker can yield
strong zero-shot performance across popular video object segmentation
benchmarks, including DAVIS, YouTube-VOS, and MOSE. Compared to traditional
object-centric mask propagation strategies, we uniquely use point propagation
to exploit local structure information that is agnostic to object semantics. We
highlight the merits of point-based tracking through direct evaluation on the
zero-shot open-world Unidentified Video Objects (UVO) benchmark. To further
enhance our approach, we utilize K-Medoids clustering for point initialization
and track both positive and negative points to clearly distinguish the target
object. We also employ multiple mask decoding passes for mask refinement and
devise a point re-initialization strategy to improve tracking accuracy. Our
code integrates different point trackers and video segmentation benchmarks and
will be released at https://github.com/SysCV/sam-pt.
SketchMetaFace: A Learning-based Sketching Interface for High-fidelity
  3D Character Face Modeling
Authors: name: Zhongjin Luo, name: Dong Du, name: Heming Zhu, name: Yizhou Yu, name: Hongbo Fu, name: Xiaoguang Han
Published: 2023-07-03 07:41:07

Modeling 3D avatars benefits various application scenarios such as AR/VR,
gaming, and filming. Character faces contribute significant diversity and
vividity as a vital component of avatars. However, building 3D character face
models usually requires a heavy workload with commercial tools, even for
experienced artists. Various existing sketch-based tools fail to support
amateurs in modeling diverse facial shapes and rich geometric details. In this
paper, we present SketchMetaFace - a sketching system targeting amateur users
to model high-fidelity 3D faces in minutes. We carefully design both the user
interface and the underlying algorithm. First, curvature-aware strokes are
adopted to better support the controllability of carving facial details.
Second, considering the key problem of mapping a 2D sketch map to a 3D model,
we develop a novel learning-based method termed "Implicit and Depth Guided Mesh
Modeling" (IDGMM). It fuses the advantages of mesh, implicit, and depth
representations to achieve high-quality results with high efficiency. In
addition, to further support usability, we present a coarse-to-fine 2D
sketching interface design and a data-driven stroke suggestion tool. User
studies demonstrate the superiority of our system over existing modeling tools
in terms of the ease to use and visual quality of results. Experimental
analyses also show that IDGMM reaches a better trade-off between accuracy and
efficiency. SketchMetaFace are available at
https://zhongjinluo.github.io/SketchMetaFace/.
DisCo: Disentangled Control for Referring Human Dance Generation in Real
  World
Authors: name: Tan Wang, name: Linjie Li, name: Kevin Lin, name: Chung-Ching Lin, name: Zhengyuan Yang, name: Hanwang Zhang, name: Zicheng Liu, name: Lijuan Wang
Published: 2023-06-30 17:37:48

Generative AI has made significant strides in computer vision, particularly
in image/video synthesis conditioned on text descriptions. Despite the
advancements, it remains challenging especially in the generation of
human-centric content such as dance synthesis. Existing dance synthesis methods
struggle with the gap between synthesized content and real-world dance
scenarios. In this paper, we define a new problem setting: Referring Human
Dance Generation, which focuses on real-world dance scenarios with three
important properties: (i) Faithfulness: the synthesis should retain the
appearance of both human subject foreground and background from the reference
image, and precisely follow the target pose; (ii) Generalizability: the model
should generalize to unseen human subjects, backgrounds, and poses; (iii)
Compositionality: it should allow for composition of seen/unseen subjects,
backgrounds, and poses from different sources. To address these challenges, we
introduce a novel approach, DISCO, which includes a novel model architecture
with disentangled control to improve the faithfulness and compositionality of
dance synthesis, and an effective human attribute pre-training for better
generalizability to unseen humans. Extensive qualitative and quantitative
results demonstrate that DISCO can generate high-quality human dance images and
videos with diverse appearances and flexible motions. Code, demo, video and
visualization are available at: https://disco-dance.github.io/.
LEDITS: Real Image Editing with DDPM Inversion and Semantic Guidance
Authors: name: Linoy Tsaban, name: Apolinário Passos
Published: 2023-07-02 09:11:09

Recent large-scale text-guided diffusion models provide powerful
image-generation capabilities. Currently, a significant effort is given to
enable the modification of these images using text only as means to offer
intuitive and versatile editing. However, editing proves to be difficult for
these generative models due to the inherent nature of editing techniques, which
involves preserving certain content from the original image. Conversely, in
text-based models, even minor modifications to the text prompt frequently
result in an entirely distinct result, making attaining one-shot generation
that accurately corresponds to the users intent exceedingly challenging. In
addition, to edit a real image using these state-of-the-art tools, one must
first invert the image into the pre-trained models domain - adding another
factor affecting the edit quality, as well as latency. In this exploratory
report, we propose LEDITS - a combined lightweight approach for real-image
editing, incorporating the Edit Friendly DDPM inversion technique with Semantic
Guidance, thus extending Semantic Guidance to real image editing, while
harnessing the editing capabilities of DDPM inversion as well. This approach
achieves versatile edits, both subtle and extensive as well as alterations in
composition and style, while requiring no optimization nor extensions to the
architecture.
MVDiffusion: Enabling Holistic Multi-view Image Generation with
  Correspondence-Aware Diffusion
Authors: name: Shitao Tang, name: Fuyang Zhang, name: Jiacheng Chen, name: Peng Wang, name: Yasutaka Furukawa
Published: 2023-07-03 15:19:17

This paper introduces MVDiffusion, a simple yet effective multi-view image
generation method for scenarios where pixel-to-pixel correspondences are
available, such as perspective crops from panorama or multi-view images given
geometry (depth maps and poses). Unlike prior models that rely on iterative
image warping and inpainting, MVDiffusion concurrently generates all images
with a global awareness, encompassing high resolution and rich content,
effectively addressing the error accumulation prevalent in preceding models.
MVDiffusion specifically incorporates a correspondence-aware attention
mechanism, enabling effective cross-view interaction. This mechanism underpins
three pivotal modules: 1) a generation module that produces low-resolution
images while maintaining global correspondence, 2) an interpolation module that
densifies spatial coverage between images, and 3) a super-resolution module
that upscales into high-resolution outputs. In terms of panoramic imagery,
MVDiffusion can generate high-resolution photorealistic images up to
1024times1024 pixels. For geometry-conditioned multi-view image generation,
MVDiffusion demonstrates the first method capable of generating a textured map
of a scene mesh. The project page is at https://mvdiffusion.github.io.