Orionnnn/hub-stats huggingface dataset:
SAGS: Structure-Aware 3D Gaussian Splatting
Authors: name: Evangelos Ververas, name: Rolandos Alexandros Potamias, name: Jifei Song, name: Jiankang Deng, name: Stefanos Zafeiriou
Published: 2024-04-29 23:26:30

Following the advent of NeRFs, 3D Gaussian Splatting (3D-GS) has paved the
way to real-time neural rendering overcoming the computational burden of
volumetric methods. Following the pioneering work of 3D-GS, several methods
have attempted to achieve compressible and high-fidelity performance
alternatives. However, by employing a geometry-agnostic optimization scheme,
these methods neglect the inherent 3D structure of the scene, thereby
restricting the expressivity and the quality of the representation, resulting
in various floating points and artifacts. In this work, we propose a
structure-aware Gaussian Splatting method (SAGS) that implicitly encodes the
geometry of the scene, which reflects to state-of-the-art rendering performance
and reduced storage requirements on benchmark novel-view synthesis datasets.
SAGS is founded on a local-global graph representation that facilitates the
learning of complex scenes and enforces meaningful point displacements that
preserve the scene's geometry. Additionally, we introduce a lightweight version
of SAGS, using a simple yet effective mid-point interpolation scheme, which
showcases a compact representation of the scene with up to 24times size
reduction without the reliance on any compression strategies. Extensive
experiments across multiple benchmark datasets demonstrate the superiority of
SAGS compared to state-of-the-art 3D-GS methods under both rendering quality
and model size. Besides, we demonstrate that our structure-aware method can
effectively mitigate floating artifacts and irregular distortions of previous
methods while obtaining precise depth maps. Project page
https://eververas.github.io/SAGS/.
Better & Faster Large Language Models via Multi-token Prediction
Authors: name: Fabian Gloeckle, name: Badr Youbi Idrissi, name: Baptiste Rozière, name: David Lopez-Paz, name: Gabriel Synnaeve
Published: 2024-04-30 17:33:57

Large language models such as GPT and Llama are trained with a next-token
prediction loss. In this work, we suggest that training language models to
predict multiple future tokens at once results in higher sample efficiency.
More specifically, at each position in the training corpus, we ask the model to
predict the following n tokens using n independent output heads, operating on
top of a shared model trunk. Considering multi-token prediction as an auxiliary
training task, we measure improved downstream capabilities with no overhead in
training time for both code and natural language models. The method is
increasingly useful for larger model sizes, and keeps its appeal when training
for multiple epochs. Gains are especially pronounced on generative benchmarks
like coding, where our models consistently outperform strong baselines by
several percentage points. Our 13B parameter models solves 12 % more problems
on HumanEval and 17 % more on MBPP than comparable next-token models.
Experiments on small algorithmic tasks demonstrate that multi-token prediction
is favorable for the development of induction heads and algorithmic reasoning
capabilities. As an additional benefit, models trained with 4-token prediction
are up to 3 times faster at inference, even with large batch sizes.
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
Authors: name: Kai Zhang, name: Sai Bi, name: Hao Tan, name: Yuanbo Xiangli, name: Nanxuan Zhao, name: Kalyan Sunkavalli, name: Zexiang Xu
Published: 2024-04-30 16:47:46

We propose GS-LRM, a scalable large reconstruction model that can predict
high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23
seconds on single A100 GPU. Our model features a very simple transformer-based
architecture; we patchify input posed images, pass the concatenated multi-view
image tokens through a sequence of transformer blocks, and decode final
per-pixel Gaussian parameters directly from these tokens for differentiable
rendering. In contrast to previous LRMs that can only reconstruct objects, by
predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large
variations in scale and complexity. We show that our model can work on both
object and scene captures by training it on Objaverse and RealEstate10K
respectively. In both scenarios, the models outperform state-of-the-art
baselines by a wide margin. We also demonstrate applications of our model in
downstream 3D generation tasks. Our project webpage is available at:
https://sai-bi.github.io/project/gs-lrm/ .
DOCCI: Descriptions of Connected and Contrasting Images
Authors: name: Yasumasa Onoe, name: Sunayana Rane, name: Zachary Berger, name: Yonatan Bitton, name: Jaemin Cho, name: Roopal Garg, name: Alexander Ku, name: Zarana Parekh, name: Jordi Pont-Tuset, name: Garrett Tanzer, name: Su Wang, name: Jason Baldridge
Published: 2024-04-30 17:56:24

Vision-language datasets are vital for both text-to-image (T2I) and
image-to-text (I2T) research. However, current datasets lack descriptions with
fine-grained detail that would allow for richer associations to be learned by
models. To fill the gap, we introduce Descriptions of Connected and Contrasting
Images (DOCCI), a dataset with long, human-annotated English descriptions for
15k images that were taken, curated and donated by a single researcher intent
on capturing key challenges such as spatial relations, counting, text
rendering, world knowledge, and more. We instruct human annotators to create
comprehensive descriptions for each image; these average 136 words in length
and are crafted to clearly distinguish each image from those that are related
or similar. Each description is highly compositional and typically encompasses
multiple challenges. Through both quantitative and qualitative analyses, we
demonstrate that DOCCI serves as an effective training resource for
image-to-text generation -- a PaLI 5B model finetuned on DOCCI shows equal or
superior results compared to highly-performant larger models like LLaVA-1.5 7B
and InstructBLIP 7B. Furthermore, we show that DOCCI is a useful testbed for
text-to-image generation, highlighting the limitations of current text-to-image
models in capturing long descriptions and fine details.
InstantFamily: Masked Attention for Zero-shot Multi-ID Image Generation
Authors: name: Chanran Kim, name: Jeongin Lee, name: Shichang Joung, name: Bongmo Kim, name: Yeul-Min Baek
Published: 2024-04-30 10:16:21

In the field of personalized image generation, the ability to create images
preserving concepts has significantly improved. Creating an image that
naturally integrates multiple concepts in a cohesive and visually appealing
composition can indeed be challenging. This paper introduces "InstantFamily,"
an approach that employs a novel masked cross-attention mechanism and a
multimodal embedding stack to achieve zero-shot multi-ID image generation. Our
method effectively preserves ID as it utilizes global and local features from a
pre-trained face recognition model integrated with text conditions.
Additionally, our masked cross-attention mechanism enables the precise control
of multi-ID and composition in the generated images. We demonstrate the
effectiveness of InstantFamily through experiments showing its dominance in
generating images with multi-ID, while resolving well-known multi-ID generation
problems. Additionally, our model achieves state-of-the-art performance in both
single-ID and multi-ID preservation. Furthermore, our model exhibits remarkable
scalability with a greater number of ID preservation than it was originally
trained with.
Stylus: Automatic Adapter Selection for Diffusion Models
Authors: name: Michael Luo, name: Justin Wong, name: Brandon Trabucco, name: Yanping Huang, name: Joseph E. Gonzalez, name: Zhifeng Chen, name: Ruslan Salakhutdinov, name: Ion Stoica
Published: 2024-04-29 17:59:16

Beyond scaling base models with more data or parameters, fine-tuned adapters
provide an alternative way to generate high fidelity, custom images at reduced
costs. As such, adapters have been widely adopted by open-source communities,
accumulating a database of over 100K adapters-most of which are highly
customized with insufficient descriptions. This paper explores the problem of
matching the prompt to a set of relevant adapters, built on recent work that
highlight the performance gains of composing adapters. We introduce Stylus,
which efficiently selects and automatically composes task-specific adapters
based on a prompt's keywords. Stylus outlines a three-stage approach that first
summarizes adapters with improved descriptions and embeddings, retrieves
relevant adapters, and then further assembles adapters based on prompts'
keywords by checking how well they fit the prompt. To evaluate Stylus, we
developed StylusDocs, a curated dataset featuring 75K adapters with
pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion
checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as
preferred, with humans and multimodal models as evaluators, over the base
model. See stylus-diffusion.github.io for more.
DressCode: Autoregressively Sewing and Generating Garments from Text
  Guidance
Authors: name: Kai He, name: Kaixin Yao, name: Qixuan Zhang, name: Jingyi Yu, name: Lingjie Liu, name: Lan Xu
Published: 2024-01-29 16:24:21

Apparel's significant role in human appearance underscores the importance of
garment digitalization for digital human creation. Recent advances in 3D
content creation are pivotal for digital human creation. Nonetheless, garment
generation from text guidance is still nascent. We introduce a text-driven 3D
garment generation framework, DressCode, which aims to democratize design for
novices and offer immense potential in fashion design, virtual try-on, and
digital human creation. For our framework, we first introduce SewingGPT, a
GPT-based architecture integrating cross-attention with text-conditioned
embedding to generate sewing patterns with text guidance. We also tailored a
pre-trained Stable Diffusion for high-quality, tile-based PBR texture
generation. By leveraging a large language model, our framework generates
CG-friendly garments through natural language interaction. Our method also
facilitates pattern completion and texture editing, simplifying the process for
designers by user-friendly interaction. With comprehensive evaluations and
comparisons with other state-of-the-art methods, our method showcases the best
quality and alignment with input prompts. User studies further validate our
high-quality rendering results, highlighting its practical utility and
potential in production settings.
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
Authors: name: Fangcheng Liu, name: Yehui Tang, name: Zhenhua Liu, name: Yunsheng Ni, name: Kai Han, name: Yunhe Wang
Published: 2024-04-29 17:53:54

Speculative decoding has demonstrated its effectiveness in accelerating the
inference of large language models while maintaining a consistent sampling
distribution. However, the conventional approach of training a separate draft
model to achieve a satisfactory token acceptance rate can be costly. Drawing
inspiration from early exiting, we propose a novel self-speculative decoding
framework Kangaroo, which uses a fixed shallow sub-network as a
self-draft model, with the remaining layers serving as the larger target model.
We train a lightweight and efficient adapter module on top of the sub-network
to bridge the gap between the sub-network and the full model's representation
ability. It is noteworthy that the inference latency of the self-draft model
may no longer be negligible compared to the large model, necessitating
strategies to increase the token acceptance rate while minimizing the drafting
steps of the small model. To address this challenge, we introduce an additional
early exiting mechanism for generating draft tokens. Specifically, we halt the
small model's subsequent prediction during the drafting phase once the
confidence level for the current token falls below a certain threshold.
Extensive experiments on the Spec-Bench demonstrate the effectiveness of
Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to
1.68times on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional
parameters (67M compared to 591M). The code for Kangaroo is available at
https://github.com/Equationliu/Kangaroo.
BlenderAlchemy: Editing 3D Graphics with Vision-Language Models
Authors: name: Ian Huang, name: Guandao Yang, name: Leonidas Guibas
Published: 2024-04-26 19:37:13

Graphics design is important for various applications, including movie
production and game design. To create a high-quality scene, designers usually
need to spend hours in software like Blender, in which they might need to
interleave and repeat operations, such as connecting material nodes, hundreds
of times. Moreover, slightly different design goals may require completely
different sequences, making automation difficult. In this paper, we propose a
system that leverages Vision-Language Models (VLMs), like GPT-4V, to
intelligently search the design action space to arrive at an answer that can
satisfy a user's intent. Specifically, we design a vision-based edit generator
and state evaluator to work together to find the correct sequence of actions to
achieve the goal. Inspired by the role of visual imagination in the human
design process, we supplement the visual reasoning capabilities of VLMs with
"imagined" reference images from image-generation models, providing visual
grounding of abstract language descriptions. In this paper, we provide
empirical evidence suggesting our system can produce simple but tedious Blender
editing sequences for tasks such as editing procedural materials from text
and/or reference images, as well as adjusting lighting configurations for
product renderings in complex scenes.
Replacing Judges with Juries: Evaluating LLM Generations with a Panel of
  Diverse Models
Authors: name: Pat Verga, name: Sebastian Hofstatter, name: Sophia Althammer, name: Yixuan Su, name: Aleksandra Piktus, name: Arkady Arkhangorodsky, name: Minjie Xu, name: Naomi White, name: Patrick Lewis
Published: 2024-04-29 15:33:23

As Large Language Models (LLMs) have become more advanced, they have outpaced
our abilities to accurately evaluate their quality. Not only is finding data to
adequately probe particular model properties difficult, but evaluating the
correctness of a model's freeform generation alone is a challenge. To address
this, many evaluations now rely on using LLMs themselves as judges to score the
quality of outputs from other LLMs. Evaluations most commonly use a single
large model like GPT4. While this method has grown in popularity, it is costly,
has been shown to introduce intramodel bias, and in this work, we find that
very large models are often unnecessary. We propose instead to evaluate models
using a Panel of LLm evaluators (PoLL). Across three distinct judge settings
and spanning six different datasets, we find that using a PoLL composed of a
larger number of smaller models outperforms a single large judge, exhibits less
intra-model bias due to its composition of disjoint model families, and does so
while being over seven times less expensive.
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual
  and Action Representations
Authors: name: Puhao Li, name: Tengyu Liu, name: Yuyang Li, name: Muzhi Han, name: Haoran Geng, name: Shu Wang, name: Yixin Zhu, name: Song-Chun Zhu, name: Siyuan Huang
Published: 2024-04-26 16:40:17

Autonomous robotic systems capable of learning novel manipulation tasks are
poised to transform industries from manufacturing to service automation.
However, modern methods (e.g., VIP and R3M) still face significant hurdles,
notably the domain gap among robotic embodiments and the sparsity of successful
task executions within specific action spaces, resulting in misaligned and
ambiguous task representations. We introduce Ag2Manip (Agent-Agnostic
representations for Manipulation), a framework aimed at surmounting these
challenges through two key innovations: a novel agent-agnostic visual
representation derived from human manipulation videos, with the specifics of
embodiments obscured to enhance generalizability; and an agent-agnostic action
representation abstracting a robot's kinematics to a universal agent proxy,
emphasizing crucial interactions between end-effector and object. Ag2Manip's
empirical validation across simulated benchmarks like FrankaKitchen, ManiSkill,
and PartManip shows a 325% increase in performance, achieved without
domain-specific demonstrations. Ablation studies underline the essential
contributions of the visual and action representations to this success.
Extending our evaluations to the real world, Ag2Manip significantly improves
imitation learning success rates from 50% to 77.5%, demonstrating its
effectiveness and generalizability across both simulated and physical
environments.
LEGENT: Open Platform for Embodied Agents
Authors: name: Zhili Cheng, name: Zhitong Wang, name: Jinyi Hu, name: Shengding Hu, name: An Liu, name: Yuge Tu, name: Pengkai Li, name: Lei Shi, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-04-28 16:50:12

Despite advancements in Large Language Models (LLMs) and Large Multimodal
Models (LMMs), their integration into language-grounded, human-like embodied
agents remains incomplete, hindering complex real-life task performance in
physical environments. Existing integrations often feature limited open
sourcing, challenging collective progress in this field. We introduce LEGENT,
an open, scalable platform for developing embodied agents using LLMs and LMMs.
LEGENT offers a dual approach: a rich, interactive 3D environment with
communicable and actionable agents, paired with a user-friendly interface, and
a sophisticated data generation pipeline utilizing advanced algorithms to
exploit supervision from simulated worlds at scale. In our experiments, an
embryonic vision-language-action model trained on LEGENT-generated data
surpasses GPT-4V in embodied tasks, showcasing promising generalization
capabilities.
MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Authors: name: Shangzhan Zhang, name: Sida Peng, name: Tao Xu, name: Yuanbo Yang, name: Tianrun Chen, name: Nan Xue, name: Yujun Shen, name: Hujun Bao, name: Ruizhen Hu, name: Xiaowei Zhou
Published: 2024-04-26 17:54:38

This paper aims to generate materials for 3D meshes from text descriptions.
Unlike existing methods that synthesize texture maps, we propose to generate
segment-wise procedural material graphs as the appearance representation, which
supports high-quality rendering and provides substantial flexibility in
editing. Instead of relying on extensive paired data, i.e., 3D meshes with
material graphs and corresponding text descriptions, to train a material graph
generative model, we propose to leverage the pre-trained 2D diffusion model as
a bridge to connect the text and material graphs. Specifically, our approach
decomposes a shape into a set of segments and designs a segment-controlled
diffusion model to synthesize 2D images that are aligned with mesh parts. Based
on generated images, we initialize parameters of material graphs and fine-tune
them through the differentiable rendering module to produce materials in
accordance with the textual description. Extensive experiments demonstrate the
superior performance of our framework in photorealism, resolution, and
editability over existing methods. Project page:
https://zhanghe3z.github.io/MaPa/
AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
Authors: name: Anselm Paulus, name: Arman Zharmagambetov, name: Chuan Guo, name: Brandon Amos, name: Yuandong Tian
Published: 2024-04-21 22:18:13

While recently Large Language Models (LLMs) have achieved remarkable
successes, they are vulnerable to certain jailbreaking attacks that lead to
generation of inappropriate or harmful content. Manual red-teaming requires
finding adversarial prompts that cause such jailbreaking, e.g. by appending a
suffix to a given instruction, which is inefficient and time-consuming. On the
other hand, automatic adversarial prompt generation often leads to semantically
meaningless attacks that can easily be detected by perplexity-based filters,
may require gradient information from the TargetLLM, or do not scale well due
to time-consuming discrete optimization processes over the token space. In this
paper, we present a novel method that uses another LLM, called the AdvPrompter,
to generate human-readable adversarial prompts in seconds, sim800times
faster than existing optimization-based approaches. We train the AdvPrompter
using a novel algorithm that does not require access to the gradients of the
TargetLLM. This process alternates between two steps: (1) generating
high-quality target adversarial suffixes by optimizing the AdvPrompter
predictions, and (2) low-rank fine-tuning of the AdvPrompter with the generated
adversarial suffixes. The trained AdvPrompter generates suffixes that veil the
input instruction without changing its meaning, such that the TargetLLM is
lured to give a harmful response. Experimental results on popular open source
TargetLLMs show state-of-the-art results on the AdvBench dataset, that also
transfer to closed-source black-box LLM APIs. Further, we demonstrate that by
fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made
more robust against jailbreaking attacks while maintaining performance, i.e.
high MMLU scores.
HaLo-NeRF: Learning Geometry-Guided Semantics for Exploring
  Unconstrained Photo Collections
Authors: name: Chen Dudai, name: Morris Alper, name: Hana Bezalel, name: Rana Hanocka, name: Itai Lang, name: Hadar Averbuch-Elor
Published: 2024-02-14 14:02:04

Internet image collections containing photos captured by crowds of
photographers show promise for enabling digital exploration of large-scale
tourist landmarks. However, prior works focus primarily on geometric
reconstruction and visualization, neglecting the key role of language in
providing a semantic interface for navigation and fine-grained understanding.
In constrained 3D domains, recent methods have leveraged vision-and-language
models as a strong prior of 2D visual semantics. While these models display an
excellent understanding of broad visual semantics, they struggle with
unconstrained photo collections depicting such tourist landmarks, as they lack
expert knowledge of the architectural domain. In this work, we present a
localization system that connects neural representations of scenes depicting
large-scale landmarks with text describing a semantic region within the scene,
by harnessing the power of SOTA vision-and-language models with adaptations for
understanding landmark scene semantics. To bolster such models with
fine-grained knowledge, we leverage large-scale Internet data containing images
of similar landmarks along with weakly-related textual information. Our
approach is built upon the premise that images physically grounded in space can
provide a powerful supervision signal for localizing new concepts, whose
semantics may be unlocked from Internet textual metadata with large language
models. We use correspondences between views of scenes to bootstrap spatial
understanding of these semantics, providing guidance for 3D-compatible
segmentation that ultimately lifts to a volumetric scene representation. Our
results show that HaLo-NeRF can accurately localize a variety of semantic
concepts related to architectural landmarks, surpassing the results of other 3D
models as well as strong 2D segmentation baselines. Our project page is at
https://tau-vailab.github.io/HaLo-NeRF/.