Orionnnn/hub-stats huggingface dataset:
Efficient 3D Articulated Human Generation with Layered Surface Volumes
Authors: name: Yinghao Xu, name: Wang Yifan, name: Alexander W. Bergman, name: Menglei Chai, name: Bolei Zhou, name: Gordon Wetzstein
Published: 2023-07-11 17:50:02

Access to high-quality and diverse 3D articulated digital human assets is
crucial in various applications, ranging from virtual reality to social
platforms. Generative approaches, such as 3D generative adversarial networks
(GANs), are rapidly replacing laborious manual content creation tools. However,
existing 3D GAN frameworks typically rely on scene representations that
leverage either template meshes, which are fast but offer limited quality, or
volumes, which offer high capacity but are slow to render, thereby limiting the
3D fidelity in GAN settings. In this work, we introduce layered surface volumes
(LSVs) as a new 3D object representation for articulated digital humans. LSVs
represent a human body using multiple textured mesh layers around a
conventional template. These layers are rendered using alpha compositing with
fast differentiable rasterization, and they can be interpreted as a volumetric
representation that allocates its capacity to a manifold of finite thickness
around the template. Unlike conventional single-layer templates that struggle
with representing fine off-surface details like hair or accessories, our
surface volumes naturally capture such details. LSVs can be articulated, and
they exhibit exceptional efficiency in GAN settings, where a 2D generator
learns to synthesize the RGBA textures for the individual layers. Trained on
unstructured, single-view 2D image datasets, our LSV-GAN generates high-quality
and view-consistent 3D articulated digital humans without the need for
view-inconsistent 2D upsampling networks.
Self-Supervised Learning with Lie Symmetries for Partial Differential
  Equations
Authors: name: Grégoire Mialon, name: Quentin Garrido, name: Hannah Lawrence, name: Danyal Rehman, name: Yann LeCun, name: Bobak T. Kiani
Published: 2023-07-11 16:52:22

Machine learning for differential equations paves the way for computationally
efficient alternatives to numerical solvers, with potentially broad impacts in
science and engineering. Though current algorithms typically require simulated
training data tailored to a given setting, one may instead wish to learn useful
information from heterogeneous sources, or from real dynamical systems
observations that are messy or incomplete. In this work, we learn
general-purpose representations of PDEs from heterogeneous data by implementing
joint embedding methods for self-supervised learning (SSL), a framework for
unsupervised representation learning that has had notable success in computer
vision. Our representation outperforms baseline approaches to invariant tasks,
such as regressing the coefficients of a PDE, while also improving the
time-stepping performance of neural solvers. We hope that our proposed
methodology will prove useful in the eventual development of general-purpose
foundation models for PDEs.
Differentiable Blocks World: Qualitative 3D Decomposition by Rendering
  Primitives
Authors: name: Tom Monnier, name: Jake Austin, name: Angjoo Kanazawa, name: Alexei A. Efros, name: Mathieu Aubry
Published: 2023-07-11 17:58:31

Given a set of calibrated images of a scene, we present an approach that
produces a simple, compact, and actionable 3D world representation by means of
3D primitives. While many approaches focus on recovering high-fidelity 3D
scenes, we focus on parsing a scene into mid-level 3D representations made of a
small set of textured primitives. Such representations are interpretable, easy
to manipulate and suited for physics-based simulations. Moreover, unlike
existing primitive decomposition methods that rely on 3D input data, our
approach operates directly on images through differentiable rendering.
Specifically, we model primitives as textured superquadric meshes and optimize
their parameters from scratch with an image rendering loss. We highlight the
importance of modeling transparency for each primitive, which is critical for
optimization and also enables handling varying numbers of primitives. We show
that the resulting textured primitives faithfully reconstruct the input images
and accurately model the visible 3D points, while providing amodal shape
completions of unseen object regions. We compare our approach to the state of
the art on diverse scenes from DTU, and demonstrate its robustness on real-life
captures from BlendedMVS and Nerfstudio. We also showcase how our results can
be used to effortlessly edit a scene or perform physical simulations. Code and
video results are available at https://www.tmonnier.com/DBW .
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the
  Backbone
Authors: name: Shraman Pramanick, name: Yale Song, name: Sayan Nag, name: Kevin Qinghong Lin, name: Hardik Shah, name: Mike Zheng Shou, name: Rama Chellappa, name: Pengchuan Zhang
Published: 2023-07-11 17:50:15

Video-language pre-training (VLP) has become increasingly important due to
its ability to generalize to various vision and language tasks. However,
existing egocentric VLP frameworks utilize separate video and language encoders
and learn task-specific cross-modal information only during fine-tuning,
limiting the development of a unified system. In this work, we introduce the
second generation of egocentric video-language pre-training (EgoVLPv2), a
significant improvement from the previous generation, by incorporating
cross-modal fusion directly into the video and language backbones. EgoVLPv2
learns strong video-text representation during pre-training and reuses the
cross-modal attention modules to support different downstream tasks in a
flexible and efficient manner, reducing fine-tuning costs. Moreover, our
proposed fusion in the backbone strategy is more lightweight and
compute-efficient than stacking additional fusion-specific layers. Extensive
experiments on a wide range of VL tasks demonstrate the effectiveness of
EgoVLPv2 by achieving consistent state-of-the-art performance over strong
baselines across all downstream. Our project page can be found at
https://shramanpramanick.github.io/EgoVLPv2/.
Test-Time Training on Video Streams
Authors: name: Renhao Wang, name: Yu Sun, name: Yossi Gandelsman, name: Xinlei Chen, name: Alexei A. Efros, name: Xiaolong Wang
Published: 2023-07-11 05:17:42

Prior work has established test-time training (TTT) as a general framework to
further improve a trained model at test time. Before making a prediction on
each test instance, the model is trained on the same instance using a
self-supervised task, such as image reconstruction with masked autoencoders. We
extend TTT to the streaming setting, where multiple test instances - video
frames in our case - arrive in temporal order. Our extension is online TTT: The
current model is initialized from the previous model, then trained on the
current frame and a small window of frames immediately before. Online TTT
significantly outperforms the fixed-model baseline for four tasks, on three
real-world datasets. The relative improvement is 45% and 66% for instance and
panoptic segmentation. Surprisingly, online TTT also outperforms its offline
variant that accesses more information, training on all frames from the entire
test video regardless of temporal order. This differs from previous findings
using synthetic videos. We conceptualize locality as the advantage of online
over offline TTT. We analyze the role of locality with ablations and a theory
based on bias-variance trade-off.
AutoDecoding Latent 3D Diffusion Models
Authors: name: Evangelos Ntavelis, name: Aliaksandr Siarohin, name: Kyle Olszewski, name: Chaoyang Wang, name: Luc Van Gool, name: Sergey Tulyakov
Published: 2023-07-07 17:59:14

We present a novel approach to the generation of static and articulated 3D
assets that has a 3D autodecoder at its core. The 3D autodecoder framework
embeds properties learned from the target dataset in the latent space, which
can then be decoded into a volumetric representation for rendering
view-consistent appearance and geometry. We then identify the appropriate
intermediate volumetric latent space, and introduce robust normalization and
de-normalization operations to learn a 3D diffusion from 2D images or monocular
videos of rigid or articulated objects. Our approach is flexible enough to use
either existing camera supervision or no camera information at all -- instead
efficiently learning it during training. Our evaluations demonstrate that our
generation results outperform state-of-the-art alternatives on various
benchmark datasets and metrics, including multi-view image datasets of
synthetic objects, real in-the-wild videos of moving people, and a large-scale,
real video dataset of static objects.
Collaborative Score Distillation for Consistent Visual Synthesis
Authors: name: Subin Kim, name: Kyungmin Lee, name: June Suk Choi, name: Jongheon Jeong, name: Kihyuk Sohn, name: Jinwoo Shin
Published: 2023-07-04 17:31:50

Generative priors of large-scale text-to-image diffusion models enable a wide
range of new generation and editing applications on diverse visual modalities.
However, when adapting these priors to complex visual modalities, often
represented as multiple images (e.g., video), achieving consistency across a
set of images is challenging. In this paper, we address this challenge with a
novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein
Variational Gradient Descent (SVGD). Specifically, we propose to consider
multiple samples as "particles" in the SVGD update and combine their score
functions to distill generative priors over a set of images synchronously.
Thus, CSD facilitates seamless integration of information across 2D images,
leading to a consistent visual synthesis across multiple samples. We show the
effectiveness of CSD in a variety of tasks, encompassing the visual editing of
panorama images, videos, and 3D scenes. Our results underline the competency of
CSD as a versatile method for enhancing inter-sample consistency, thereby
broadening the applicability of text-to-image diffusion models.
Generative Pretraining in Multimodality
Authors: name: Quan Sun, name: Qiying Yu, name: Yufeng Cui, name: Fan Zhang, name: Xiaosong Zhang, name: Yueze Wang, name: Hongcheng Gao, name: Jingjing Liu, name: Tiejun Huang, name: Xinlong Wang
Published: 2023-07-11 12:45:39

We present Emu, a Transformer-based multimodal foundation model, which can
seamlessly generate images and texts in multimodal context. This omnivore model
can take in any single-modality or multimodal data input indiscriminately
(e.g., interleaved image, text and video) through a one-model-for-all
autoregressive training process. First, visual signals are encoded into
embeddings, and together with text tokens form an interleaved input sequence.
Emu is then end-to-end trained with a unified objective of classifying the next
text token or regressing the next visual embedding in the multimodal sequence.
This versatile multimodality empowers the exploration of diverse pretraining
data sources at scale, such as videos with interleaved frames and text,
webpages with interleaved images and text, as well as web-scale image-text
pairs and video-text pairs. Emu can serve as a generalist multimodal interface
for both image-to-text and text-to-image tasks, and supports in-context image
and text generation. Across a broad range of zero-shot/few-shot tasks including
image captioning, visual question answering, video question answering and
text-to-image generation, Emu demonstrates superb performance compared to
state-of-the-art large multimodal models. Extended capabilities such as
multimodal assistants via instruction tuning are also demonstrated with
impressive performance.
Empowering Cross-lingual Behavioral Testing of NLP Models with
  Typological Features
Authors: name: Ester Hlavnova, name: Sebastian Ruder
Published: 2023-07-11 17:33:03

A challenge towards developing NLP systems for the world's languages is
understanding how they generalize to typological differences relevant for
real-world applications. To this end, we propose M2C, a morphologically-aware
framework for behavioral testing of NLP models. We use M2C to generate tests
that probe models' behavior in light of specific linguistic features in 12
typologically diverse languages. We evaluate state-of-the-art language models
on the generated tests. While models excel at most tests in English, we
highlight generalization failures to specific typological characteristics such
as temporal expressions in Swahili and compounding possessives in Finish. Our
findings motivate the development of models that address these blind spots.
Unleashing Cognitive Synergy in Large Language Models: A Task-Solving
  Agent through Multi-Persona Self-Collaboration
Authors: name: Zhenhailong Wang, name: Shaoguang Mao, name: Wenshan Wu, name: Tao Ge, name: Furu Wei, name: Heng Ji
Published: 2023-07-11 14:45:19

Human intelligence thrives on the concept of cognitive synergy, where
collaboration and information integration among different cognitive processes
yield superior outcomes compared to individual cognitive processes in
isolation. Although Large Language Models (LLMs) have demonstrated promising
performance as general task-solving agents, they still struggle with tasks that
require intensive domain knowledge and complex reasoning. In this work, we
propose Solo Performance Prompting (SPP), which transforms a single LLM into a
cognitive synergist by engaging in multi-turn self-collaboration with multiple
personas. A cognitive synergist refers to an intelligent agent that
collaborates with multiple minds, combining their individual strengths and
knowledge, to enhance problem-solving and overall performance in complex tasks.
By dynamically identifying and simulating different personas based on task
inputs, SPP unleashes the potential of cognitive synergy in LLMs. We have
discovered that assigning multiple, fine-grained personas in LLMs elicits
better problem-solving abilities compared to using a single or fixed number of
personas. We evaluate SPP on three challenging tasks: Trivia Creative Writing,
Codenames Collaborative, and Logic Grid Puzzle, encompassing both
knowledge-intensive and reasoning-intensive types. Unlike previous works, such
as Chain-of-Thought, that solely enhance the reasoning abilities in LLMs, SPP
effectively elicits internal knowledge acquisition abilities, reduces
hallucination, and maintains strong reasoning capabilities. Code, data, and
prompts can be found at:
https://github.com/MikeWangWZHL/Solo-Performance-Prompting.git.
Secrets of RLHF in Large Language Models Part I: PPO
Authors: name: Rui Zheng, name: Shihan Dou, name: Songyang Gao, name: Wei Shen, name: Binghai Wang, name: Yan Liu, name: Senjie Jin, name: Qin Liu, name: Limao Xiong, name: Lu Chen, name: Zhiheng Xi, name: Yuhao Zhou, name: Nuo Xu, name: Wenbin Lai, name: Minghao Zhu, name: Rongxiang Weng, name: Wensen Cheng, name: Cheng Chang, name: Zhangyue Yin, name: Yuan Hua, name: Haoran Huang, name: Tianxiang Sun, name: Hang Yan, name: Tao Gui, name: Qi Zhang, name: Xipeng Qiu, name: Xuanjing Huang
Published: 2023-07-11 01:55:24

Large language models (LLMs) have formulated a blueprint for the advancement
of artificial general intelligence. Its primary objective is to function as a
human-centric (helpful, honest, and harmless) assistant. Alignment with humans
assumes paramount significance, and reinforcement learning with human feedback
(RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
Current technical routes usually include reward models to measure
human preferences, Proximal Policy Optimization (PPO) to optimize
policy model outputs, and process supervision to improve step-by-step
reasoning capabilities. However, due to the challenges of reward design,
environment interaction, and agent training, coupled with huge trial and error
cost of large language models, there is a significant barrier for AI
researchers to motivate the development of technical alignment and safe landing
of LLMs. The stable training of RLHF has still been a puzzle. In the first
report, we dissect the framework of RLHF, re-evaluate the inner workings of
PPO, and explore how the parts comprising PPO algorithms impact policy agent
training. We identify policy constraints being the key factor for the effective
implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
advanced version of PPO algorithm, to efficiently improve the training
stability of the policy model. Based on our main results, we perform a
comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
The absence of open-source implementations has posed significant challenges to
the investigation of LLMs alignment. Therefore, we are eager to release
technical reports, reward models and PPO codes
International Institutions for Advanced AI
Authors: name: Lewis Ho, name: Joslyn Barnhart, name: Robert Trager, name: Yoshua Bengio, name: Miles Brundage, name: Allison Carnegie, name: Rumman Chowdhury, name: Allan Dafoe, name: Gillian Hadfield, name: Margaret Levi, name: Duncan Snidal
Published: 2023-07-10 16:55:55

International institutions may have an important role to play in ensuring
advanced AI systems benefit humanity. International collaborations can unlock
AI's ability to further sustainable development, and coordination of regulatory
efforts can reduce obstacles to innovation and the spread of benefits.
Conversely, the potential dangerous capabilities of powerful and
general-purpose AI systems create global externalities in their development and
deployment, and international efforts to further responsible AI practices could
help manage the risks they pose. This paper identifies a set of governance
functions that could be performed at an international level to address these
challenges, ranging from supporting access to frontier AI systems to setting
international safety standards. It groups these functions into four
institutional models that exhibit internal synergies and have precedents in
existing organizations: 1) a Commission on Frontier AI that facilitates expert
consensus on opportunities and risks from advanced AI, 2) an Advanced AI
Governance Organization that sets international standards to manage global
threats from advanced models, supports their implementation, and possibly
monitors compliance with a future governance regime, 3) a Frontier AI
Collaborative that promotes access to cutting-edge AI, and 4) an AI Safety
Project that brings together leading researchers and engineers to further AI
safety research. We explore the utility of these models and identify open
questions about their viability.
Solvent: A Framework for Protein Folding
Authors: name: Jaemyung Lee, name: Jaehoon Kim, name: Hasun Yu, name: Youhan Lee
Published: 2023-07-07 09:01:42

Consistency and reliability are crucial for conducting AI research. Many
famous research fields, such as object detection, have been compared and
validated with solid benchmark frameworks. After AlphaFold2, the protein
folding task has entered a new phase, and many methods are proposed based on
the component of AlphaFold2. The importance of a unified research framework in
protein folding contains implementations and benchmarks to consistently and
fairly compare various approaches. To achieve this, we present Solvent, an
protein folding framework that supports significant components of
state-of-th-arts models in the manner of off-the-shelf interface Solvent
contains different models implemented in a unified codebase and supports
training and evaluation for defined models on the same dataset. We benchmark
well-known algorithms and their components and provide experiments that give
helpful insights into the protein structure modeling field. We hope that
Solvent will increase the reliability and consistency of proposed models and
gives efficiency in both speed and costs, resulting in acceleration on protein
folding modeling research. The code is available at
https://github.com/kakaobrain/solvent, and the project will continue to be
developed.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation
  System
Authors: name: Yuzhe Qin, name: Wei Yang, name: Binghao Huang, name: Karl Van Wyk, name: Hao Su, name: Xiaolong Wang, name: Yu-Wei Chao, name: Dietor Fox
Published: 2023-07-10 14:11:07

Vision-based teleoperation offers the possibility to endow robots with
human-level intelligence to physically interact with the environment, while
only requiring low-cost camera sensors. However, current vision-based
teleoperation systems are designed and engineered towards a particular robot
model and deploy environment, which scales poorly as the pool of the robot
models expands and the variety of the operating environment increases. In this
paper, we propose AnyTeleop, a unified and general teleoperation system to
support multiple different arms, hands, realities, and camera configurations
within a single system. Although being designed to provide great flexibility to
the choice of simulators and real hardware, our system can still achieve great
performance. For real-world experiments, AnyTeleop can outperform a previous
system that was designed for a specific robot hardware with a higher success
rate, using the same robot. For teleoperation in simulation, AnyTeleop leads to
better imitation learning performance, compared with a previous system that is
particularly designed for that simulator. Project page: http://anyteleop.com/.
Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal
  Rearrangement
Authors: name: Anthony Simeonov, name: Ankit Goyal, name: Lucas Manuelli, name: Lin Yen-Chen, name: Alina Sarmiento, name: Alberto Rodriguez, name: Pulkit Agrawal, name: Dieter Fox
Published: 2023-07-10 17:56:06

We propose a system for rearranging objects in a scene to achieve a desired
object-scene placing relationship, such as a book inserted in an open slot of a
bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of
both scenes and objects, and is trained from demonstrations to operate directly
on 3D point clouds. Our system overcomes challenges associated with the
existence of many geometrically-similar rearrangement solutions for a given
scene. By leveraging an iterative pose de-noising training procedure, we can
fit multi-modal demonstration data and produce multi-modal outputs while
remaining precise and accurate. We also show the advantages of conditioning on
relevant local geometric features while ignoring irrelevant global structure
that harms both generalization and precision. We demonstrate our approach on
three distinct rearrangement tasks that require handling multi-modality and
generalization over object shape and pose in both simulation and the real
world. Project website, code, and videos:
https://anthonysimeonov.github.io/rpdiff-multi-modal/