Orionnnn/hub-stats huggingface dataset:
Gen4Gen: Generative Data Pipeline for Generative Multi-Concept
  Composition
Authors: name: Chun-Hsiao Yeh, name: Ta-Ying Cheng, name: He-Yen Hsieh, name: Chuan-En Lin, name: Yi Ma, name: Andrew Markham, name: Niki Trigoni, name: H. T. Kung, name: Yubei Chen
Published: 2024-02-23 18:55:09

Recent text-to-image diffusion models are able to learn and synthesize images
containing novel, personalized concepts (e.g., their own pets or specific
items) with just a few examples for training. This paper tackles two
interconnected issues within this realm of personalizing text-to-image
diffusion models. First, current personalization techniques fail to reliably
extend to multiple concepts -- we hypothesize this to be due to the mismatch
between complex scenes and simple text descriptions in the pre-training dataset
(e.g., LAION). Second, given an image containing multiple personalized
concepts, there lacks a holistic metric that evaluates performance on not just
the degree of resemblance of personalized concepts, but also whether all
concepts are present in the image and whether the image accurately reflects the
overall text description. To address these issues, we introduce Gen4Gen, a
semi-automated dataset creation pipeline utilizing generative models to combine
personalized concepts into complex compositions along with text-descriptions.
Using this, we create a dataset called MyCanvas, that can be used to benchmark
the task of multi-concept personalization. In addition, we design a
comprehensive metric comprising two scores (CP-CLIP and TI-CLIP) for better
quantifying the performance of multi-concept, personalized text-to-image
diffusion methods. We provide a simple baseline built on top of Custom
Diffusion with empirical prompting strategies for future researchers to
evaluate on MyCanvas. We show that by improving data quality and prompting
strategies, we can significantly increase multi-concept personalized image
generation quality, without requiring any modifications to model architecture
or training algorithms.
GeneOH Diffusion: Towards Generalizable Hand-Object Interaction
  Denoising via Denoising Diffusion
Authors: name: Xueyi Liu, name: Li Yi
Published: 2024-02-22 18:59:21

In this work, we tackle the challenging problem of denoising hand-object
interactions (HOI). Given an erroneous interaction sequence, the objective is
to refine the incorrect hand trajectory to remove interaction artifacts for a
perceptually realistic sequence. This challenge involves intricate interaction
noise, including unnatural hand poses and incorrect hand-object relations,
alongside the necessity for robust generalization to new interactions and
diverse noise patterns. We tackle those challenges through a novel approach,
GeneOH Diffusion, incorporating two key designs: an innovative contact-centric
HOI representation named GeneOH and a new domain-generalizable denoising
scheme. The contact-centric representation GeneOH informatively parameterizes
the HOI process, facilitating enhanced generalization across various HOI
scenarios. The new denoising scheme consists of a canonical denoising model
trained to project noisy data samples from a whitened noise space to a clean
data manifold and a "denoising via diffusion" strategy which can handle input
trajectories with various noise patterns by first diffusing them to align with
the whitened noise space and cleaning via the canonical denoiser. Extensive
experiments on four benchmarks with significant domain variations demonstrate
the superior effectiveness of our method. GeneOH Diffusion also shows promise
for various downstream applications. Project website:
https://meowuu7.github.io/GeneOH-Diffusion/.
CyberDemo: Augmenting Simulated Human Demonstration for Real-World
  Dexterous Manipulation
Authors: name: Jun Wang, name: Yuzhe Qin, name: Kaiming Kuang, name: Yigit Korkmaz, name: Akhilan Gurumoorthy, name: Hao Su, name: Xiaolong Wang
Published: 2024-02-22 18:54:32

We introduce CyberDemo, a novel approach to robotic imitation learning that
leverages simulated human demonstrations for real-world tasks. By incorporating
extensive data augmentation in a simulated environment, CyberDemo outperforms
traditional in-domain real-world demonstrations when transferred to the real
world, handling diverse physical and visual conditions. Regardless of its
affordability and convenience in data collection, CyberDemo outperforms
baseline methods in terms of success rates across various tasks and exhibits
generalizability with previously unseen objects. For example, it can rotate
novel tetra-valve and penta-valve, despite human demonstrations only involving
tri-valves. Our research demonstrates the significant potential of simulated
human demonstrations for real-world dexterous manipulation tasks. More details
can be found at https://cyber-demo.github.io
BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human
  Racing Gameplay
Authors: name: Catherine Weaver, name: Chen Tang, name: Ce Hao, name: Kenta Kawamoto, name: Masayoshi Tomizuka, name: Wei Zhan
Published: 2024-02-22 00:38:43

Imitation learning learns a policy from demonstrations without requiring
hand-designed reward functions. In many robotic tasks, such as autonomous
racing, imitated policies must model complex environment dynamics and human
decision-making. Sequence modeling is highly effective in capturing intricate
patterns of motion sequences but struggles to adapt to new environments or
distribution shifts that are common in real-world robotics tasks. In contrast,
Adversarial Imitation Learning (AIL) can mitigate this effect, but struggles
with sample inefficiency and handling complex motion patterns. Thus, we propose
BeTAIL: Behavior Transformer Adversarial Imitation Learning, which combines a
Behavior Transformer (BeT) policy from human demonstrations with online AIL.
BeTAIL adds an AIL residual policy to the BeT policy to model the sequential
decision-making process of human experts and correct for out-of-distribution
states or shifts in environment dynamics. We test BeTAIL on three challenges
with expert-level demonstrations of real human gameplay in Gran Turismo Sport.
Our proposed residual BeTAIL reduces environment interactions and improves
racing performance and stability, even when the BeT is pretrained on different
tracks than downstream learning. Videos and code available at:
https://sites.google.com/berkeley.edu/BeTAIL/home.
Linear Transformers are Versatile In-Context Learners
Authors: name: Max Vladymyrov, name: Johannes von Oswald, name: Mark Sandler, name: Rong Ge
Published: 2024-02-21 23:45:57

Recent research has demonstrated that transformers, particularly linear
attention models, implicitly execute gradient-descent-like algorithms on data
provided in-context during their forward inference step. However, their
capability in handling more complex problems remains unexplored. In this paper,
we prove that any linear transformer maintains an implicit linear model and can
be interpreted as performing a variant of preconditioned gradient descent. We
also investigate the use of linear transformers in a challenging scenario where
the training data is corrupted with different levels of noise. Remarkably, we
demonstrate that for this problem linear transformers discover an intricate and
highly effective optimization algorithm, surpassing or matching in performance
many reasonable baselines. We reverse-engineer this algorithm and show that it
is a novel approach incorporating momentum and adaptive rescaling based on
noise levels. Our findings show that even linear transformers possess the
surprising ability to discover sophisticated optimization strategies.
Beyond A*: Better Planning with Transformers via Search Dynamics
  Bootstrapping
Authors: name: Lucas Lehnert, name: Sainbayar Sukhbaatar, name: Paul Mcvay, name: Michael Rabbat, name: Yuandong Tian
Published: 2024-02-21 19:17:28

While Transformers have enabled tremendous progress in various application
settings, such architectures still lag behind traditional symbolic planners for
solving complex decision making tasks. In this work, we demonstrate how to
train Transformers to solve complex planning tasks and present Searchformer, a
Transformer model that optimally solves previously unseen Sokoban puzzles 93.7%
of the time, while using up to 26.8% fewer search steps than standard A^*
search. Searchformer is an encoder-decoder Transformer model trained to predict
the search dynamics of A^*. This model is then fine-tuned via expert
iterations to perform fewer search steps than A^* search while still
generating an optimal plan. In our training method, A^*'s search dynamics are
expressed as a token sequence outlining when task states are added and removed
into the search tree during symbolic planning. In our ablation studies on maze
navigation, we find that Searchformer significantly outperforms baselines that
predict the optimal plan directly with a 5-10times smaller model size and a
10times smaller training dataset. We also demonstrate how Searchformer
scales to larger and more complex decision making tasks like Sokoban with
improved percentage of solved tasks and shortened search dynamics.
GaussianPro: 3D Gaussian Splatting with Progressive Propagation
Authors: name: Kai Cheng, name: Xiaoxiao Long, name: Kaizhi Yang, name: Yao Yao, name: Wei Yin, name: Yuexin Ma, name: Wenping Wang, name: Xuejin Chen
Published: 2024-02-22 16:00:20

The advent of 3D Gaussian Splatting (3DGS) has recently brought about a
revolution in the field of neural rendering, facilitating high-quality
renderings at real-time speed. However, 3DGS heavily depends on the initialized
point cloud produced by Structure-from-Motion (SfM) techniques. When tackling
with large-scale scenes that unavoidably contain texture-less surfaces, the SfM
techniques always fail to produce enough points in these surfaces and cannot
provide good initialization for 3DGS. As a result, 3DGS suffers from difficult
optimization and low-quality renderings. In this paper, inspired by classical
multi-view stereo (MVS) techniques, we propose GaussianPro, a novel method that
applies a progressive propagation strategy to guide the densification of the 3D
Gaussians. Compared to the simple split and clone strategies used in 3DGS, our
method leverages the priors of the existing reconstructed geometries of the
scene and patch matching techniques to produce new Gaussians with accurate
positions and orientations. Experiments on both large-scale and small-scale
scenes validate the effectiveness of our method, where our method significantly
surpasses 3DGS on the Waymo dataset, exhibiting an improvement of 1.15dB in
terms of PSNR.
Consolidating Attention Features for Multi-view Image Editing
Authors: name: Or Patashnik, name: Rinon Gal, name: Daniel Cohen-Or, name: Jun-Yan Zhu, name: Fernando De la Torre
Published: 2024-02-22 18:50:18

Large-scale text-to-image models enable a wide range of image editing
techniques, using text prompts or even spatial controls. However, applying
these editing methods to multi-view images depicting a single scene leads to
3D-inconsistent results. In this work, we focus on spatial control-based
geometric manipulations and introduce a method to consolidate the editing
process across various views. We build on two insights: (1) maintaining
consistent features throughout the generative process helps attain consistency
in multi-view editing, and (2) the queries in self-attention layers
significantly influence the image structure. Hence, we propose to improve the
geometric consistency of the edited images by enforcing the consistency of the
queries. To do so, we introduce QNeRF, a neural radiance field trained on the
internal query features of the edited images. Once trained, QNeRF can render
3D-consistent queries, which are then softly injected back into the
self-attention layers during generation, greatly improving multi-view
consistency. We refine the process through a progressive, iterative method that
better consolidates queries across the diffusion timesteps. We compare our
method to a range of existing techniques and demonstrate that it can achieve
better multi-view consistency and higher fidelity to the input scene. These
advantages allow us to train NeRFs with fewer visual artifacts, that are better
aligned with the target geometry.
PALO: A Polyglot Large Multimodal Model for 5B People
Authors: name: Muhammad Maaz, name: Hanoona Rasheed, name: Abdelrahman Shaker, name: Salman Khan, name: Hisham Cholakal, name: Rao M. Anwer, name: Tim Baldwin, name: Michael Felsberg, name: Fahad S. Khan
Published: 2024-02-22 18:59:58

In pursuit of more inclusive Vision-Language Models (VLMs), this study
introduces a Large Multilingual Multimodal Model called Palo.
Palo offers visual reasoning capabilities in 10 major languages,
including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian,
Urdu, and Japanese, that span a total of sim5B people (65\% of the world
population). Our approach involves a semi-automated translation approach to
adapt the multimodal instruction dataset from English to the target languages
using a fine-tuned Large Language Model, thereby ensuring high linguistic
fidelity while allowing scalability due to minimal manual effort. The
incorporation of diverse instruction sets helps us boost overall performance
across multiple languages especially those that are underrepresented like
Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three
scales (1.7B, 7B and 13B parameters) to show the generalization and scalability
where we observe substantial improvements compared to strong baselines. We also
propose the first multilingual multimodal benchmark for the forthcoming
approaches to evaluate their vision-language reasoning capabilities across
languages. Code: https://github.com/mbzuai-oryx/PALO.
Scaling Up LLM Reviews for Google Ads Content Moderation
Authors: name: Wei Qiao, name: Tushar Dogra, name: Otilia Stretcu, name: Yu-Han Lyu, name: Tiantian Fang, name: Dongjin Kwon, name: Chun-Ta Lu, name: Enming Luo, name: Yuan Wang, name: Chih-Chun Chia, name: Ariel Fuxman, name: Fangzhou Wang, name: Ranjay Krishna, name: Mehmet Tek
Published: 2024-02-07 23:47:02

Large language models (LLMs) are powerful tools for content moderation, but
their inference costs and latency make them prohibitive for casual use on large
datasets, such as the Google Ads repository. This study proposes a method for
scaling up LLM reviews for content moderation in Google Ads. First, we use
heuristics to select candidates via filtering and duplicate removal, and create
clusters of ads for which we select one representative ad per cluster. We then
use LLMs to review only the representative ads. Finally, we propagate the LLM
decisions for the representative ads back to their clusters. This method
reduces the number of reviews by more than 3 orders of magnitude while
achieving a 2x recall compared to a baseline non-LLM model. The success of this
approach is a strong function of the representations used in clustering and
label propagation; we found that cross-modal similarity representations yield
better results than uni-modal representations.
OmniPred: Language Models as Universal Regressors
Authors: name: Xingyou Song, name: Oscar Li, name: Chansoo Lee, name: Bangding, name: Yang, name: Daiyi Peng, name: Sagi Perel, name: Yutian Chen
Published: 2024-02-22 13:36:53

Over the broad landscape of experimental design, regression has been a
powerful tool to accurately predict the outcome metrics of a system or model
given a set of parameters, but has been traditionally restricted to methods
which are only applicable to a specific task. In this paper, we propose
OmniPred, a framework for training language models as universal end-to-end
regressors over (x,y) evaluation data from diverse real world experiments.
Using data sourced from Google Vizier, one of the largest blackbox optimization
databases in the world, our extensive experiments demonstrate that through only
textual representations of mathematical parameters and values, language models
are capable of very precise numerical regression, and if given the opportunity
to train over multiple tasks, can significantly outperform traditional
regression models.
Subobject-level Image Tokenization
Authors: name: Delong Chen, name: Samuel Cahyawijaya, name: Jianfeng Liu, name: Baoyuan Wang, name: Pascale Fung
Published: 2024-02-22 06:47:44

Transformer-based vision models typically tokenize images into fixed-size
square patches as input units, which lacks the adaptability to image content
and overlooks the inherent pixel grouping structure. Inspired by the subword
tokenization widely adopted in language models, we propose an image tokenizer
at a subobject level, where the subobjects are represented by semantically
meaningful image segments obtained by segmentation models (e.g., segment
anything models). To implement a learning system based on subobject
tokenization, we first introduced a Sequence-to-sequence AutoEncoder (SeqAE) to
compress subobject segments of varying sizes and shapes into compact embedding
vectors, then fed the subobject embeddings into a large language model for
vision language learning. Empirical results demonstrated that our
subobject-level tokenization significantly facilitates efficient learning of
translating images into object and attribute descriptions compared to the
traditional patch-level tokenization. Codes and models will be open-sourced at
https://github.com/ChenDelong1999/subobjects.
TinyLLaVA: A Framework of Small-scale Large Multimodal Models
Authors: name: Baichuan Zhou, name: Ying Hu, name: Xi Weng, name: Junlong Jia, name: Jie Luo, name: Xien Liu, name: Ji Wu, name: Lei Huang
Published: 2024-02-22 05:05:30

We present the TinyLLaVA framework that provides a unified perspective in
designing and analyzing the small-scale Large Multimodal Models (LMMs). We
empirically study the effects of different vision encoders, connection modules,
language models, training data and training recipes. Our extensive experiments
showed that better quality of data combined with better training recipes,
smaller LMMs can consistently achieve on-par performances compared to bigger
LMMs. Under our framework, we train a family of small-scale LMMs. Our best
model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B
models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as
baselines for future research in terms of data scaling, training setups and
model selections. Our model weights and codes will be made public.
Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming
Authors: name: Anisha Agarwal, name: Aaron Chan, name: Shubham Chandel, name: Jinu Jang, name: Shaun Miller, name: Roshanak Zilouchian Moghaddam, name: Yevhen Mohylevskyy, name: Neel Sundaresan, name: Michele Tufano
Published: 2024-02-22 03:51:34

The integration of Large Language Models (LLMs) into Development Environments
(IDEs) has become a focal point in modern software development. LLMs such as
OpenAI GPT-3.5/4 and Code Llama offer the potential to significantly augment
developer productivity by serving as intelligent, chat-driven programming
assistants. However, utilizing LLMs out of the box is unlikely to be optimal
for any given scenario. Rather, each system requires the LLM to be honed to its
set of heuristics to ensure the best performance. In this paper, we introduce
the Copilot evaluation harness: a set of data and tools for evaluating
LLM-guided IDE interactions, covering various programming scenarios and
languages. We propose our metrics as a more robust and information-dense
evaluation than previous state of the art evaluation systems. We design and
compute both static and execution based success metrics for scenarios
encompassing a wide range of developer tasks, including code generation from
natural language (generate), documentation generation from code (doc), test
case generation (test), bug-fixing (fix), and workspace understanding and query
resolution (workspace). These success metrics are designed to evaluate the
performance of LLMs within a given IDE and its respective parameter space. Our
learnings from evaluating three common LLMs using these metrics can inform the
development and validation of future scenarios in LLM guided IDEs.
AgentScope: A Flexible yet Robust Multi-Agent Platform
Authors: name: Dawei Gao, name: Zitao Li, name: Weirui Kuang, name: Xuchen Pan, name: Daoyuan Chen, name: Zhijian Ma, name: Bingchen Qian, name: Liuyi Yao, name: Lin Zhu, name: Chen Cheng, name: Hongzhu Shi, name: Yaliang Li, name: Bolin Ding, name: Jingren Zhou
Published: 2024-02-21 04:11:28

With the rapid advancement of Large Language Models (LLMs), significant
progress has been made in multi-agent applications. However, the complexities
in coordinating agents' cooperation and LLMs' erratic performance pose notable
challenges in developing robust and efficient multi-agent applications. To
tackle these challenges, we propose AgentScope, a developer-centric multi-agent
platform with message exchange as its core communication mechanism. Together
with abundant syntactic tools, built-in resources, and user-friendly
interactions, our communication mechanism significantly reduces the barriers to
both development and understanding. Towards robust and flexible multi-agent
application, AgentScope provides both built-in and customizable fault tolerance
mechanisms while it is also armed with system-level supports for multi-modal
data generation, storage and transmission. Additionally, we design an
actor-based distribution framework, enabling easy conversion between local and
distributed deployments and automatic parallel optimization without extra
effort. With these features, AgentScope empowers developers to build
applications that fully realize the potential of intelligent agents. We have
released AgentScope at https://github.com/modelscope/agentscope, and hope
AgentScope invites wider participation and innovation in this fast-moving
field.
OpenCodeInterpreter: Integrating Code Generation with Execution and
  Refinement
Authors: name: Tianyu Zheng, name: Ge Zhang, name: Tianhao Shen, name: Xueling Liu, name: Bill Yuchen Lin, name: Jie Fu, name: Wenhu Chen, name: Xiang Yue
Published: 2024-02-22 16:06:23

The introduction of large language models has significantly advanced code
generation. However, open-source models often lack the execution capabilities
and iterative refinement of advanced systems like the GPT-4 Code Interpreter.
To address this, we introduce OpenCodeInterpreter, a family of open-source code
systems designed for generating, executing, and iteratively refining code.
Supported by Code-Feedback, a dataset featuring 68K multi-turn interactions,
OpenCodeInterpreter integrates execution and human feedback for dynamic code
refinement. Our comprehensive evaluation of OpenCodeInterpreter across key
benchmarks such as HumanEval, MBPP, and their enhanced versions from EvalPlus
reveals its exceptional performance. Notably, OpenCodeInterpreter-33B achieves
an accuracy of 83.2 (76.4) on the average (and plus versions) of HumanEval and
MBPP, closely rivaling GPT-4's 84.2 (76.2) and further elevates to 91.6 (84.6)
with synthesized human feedback from GPT-4. OpenCodeInterpreter brings the gap
between open-source code generation models and proprietary systems like GPT-4
Code Interpreter.