Orionnnn/hub-stats huggingface dataset:
Learning the 3D Fauna of the Web
Authors: name: Zizhang Li, name: Dor Litvak, name: Ruining Li, name: Yunzhi Zhang, name: Tomas Jakab, name: Christian Rupprecht, name: Shangzhe Wu, name: Andrea Vedaldi, name: Jiajun Wu
Published: 2024-01-04 18:32:48

Learning 3D models of all animals on the Earth requires massively scaling up
existing solutions. With this ultimate goal in mind, we develop 3D-Fauna, an
approach that learns a pan-category deformable 3D animal model for more than
100 animal species jointly. One crucial bottleneck of modeling animals is the
limited availability of training data, which we overcome by simply learning
from 2D Internet images. We show that prior category-specific attempts fail to
generalize to rare species with limited training images. We address this
challenge by introducing the Semantic Bank of Skinned Models (SBSM), which
automatically discovers a small set of base animal shapes by combining
geometric inductive priors with semantic knowledge implicitly captured by an
off-the-shelf self-supervised feature extractor. To train such a model, we also
contribute a new large-scale dataset of diverse animal species. At inference
time, given a single image of any quadruped animal, our model reconstructs an
articulated 3D mesh in a feed-forward fashion within seconds.
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost
  Whole-Body Teleoperation
Authors: name: Zipeng Fu, name: Tony Z. Zhao, name: Chelsea Finn
Published: 2024-01-04 07:55:53

Imitation learning from human demonstrations has shown impressive performance
in robotics. However, most results focus on table-top manipulation, lacking the
mobility and dexterity necessary for generally useful tasks. In this work, we
develop a system for imitating mobile manipulation tasks that are bimanual and
require whole-body control. We first present Mobile ALOHA, a low-cost and
whole-body teleoperation system for data collection. It augments the ALOHA
system with a mobile base, and a whole-body teleoperation interface. Using data
collected with Mobile ALOHA, we then perform supervised behavior cloning and
find that co-training with existing static ALOHA datasets boosts performance on
mobile manipulation tasks. With 50 demonstrations for each task, co-training
can increase success rates by up to 90%, allowing Mobile ALOHA to autonomously
complete complex mobile manipulation tasks such as sauteing and serving a piece
of shrimp, opening a two-door wall cabinet to store heavy cooking pots, calling
and entering an elevator, and lightly rinsing a used pan using a kitchen
faucet. Project website: https://mobile-aloha.github.io
Improving Diffusion-Based Image Synthesis with Context Prediction
Authors: name: Ling Yang, name: Jingwei Liu, name: Shenda Hong, name: Zhilong Zhang, name: Zhilin Huang, name: Zheming Cai, name: Wentao Zhang, name: Bin Cui
Published: 2024-01-04 01:10:56

Diffusion models are a new class of generative models, and have dramatically
promoted image generation with unprecedented quality and diversity. Existing
diffusion models mainly try to reconstruct input image from a corrupted one
with a pixel-wise or feature-wise constraint along spatial axes. However, such
point-based reconstruction may fail to make each predicted pixel/feature fully
preserve its neighborhood context, impairing diffusion-based image synthesis.
As a powerful source of automatic supervisory signal, context has been well
studied for learning representations. Inspired by this, we for the first time
propose ConPreDiff to improve diffusion-based image synthesis with context
prediction. We explicitly reinforce each point to predict its neighborhood
context (i.e., multi-stride features/tokens/pixels) with a context decoder at
the end of diffusion denoising blocks in training stage, and remove the decoder
for inference. In this way, each point can better reconstruct itself by
preserving its semantic connections with neighborhood context. This new
paradigm of ConPreDiff can generalize to arbitrary discrete and continuous
diffusion backbones without introducing extra parameters in sampling procedure.
Extensive experiments are conducted on unconditional image generation,
text-to-image generation and image inpainting tasks. Our ConPreDiff
consistently outperforms previous methods and achieves a new SOTA text-to-image
generation results on MS-COCO, with a zero-shot FID score of 6.21.
LLaMA Pro: Progressive LLaMA with Block Expansion
Authors: name: Chengyue Wu, name: Yukang Gan, name: Yixiao Ge, name: Zeyu Lu, name: Jiahao Wang, name: Ye Feng, name: Ping Luo, name: Ying Shan
Published: 2024-01-04 18:59:12

Humans generally acquire new skills without compromising the old; however,
the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to
CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with
an expansion of Transformer blocks. We tune the expanded blocks using only new
corpus, efficiently and effectively improving the model's knowledge without
catastrophic forgetting. In this paper, we experiment on the corpus of code and
math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from
LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro
and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced
performance among various benchmarks, demonstrating superiority over existing
open models in the LLaMA family and the immense potential of reasoning and
addressing diverse tasks as an intelligent agent. Our findings provide valuable
insights into integrating natural and programming languages, laying a solid
foundation for developing advanced language agents that operate effectively in
various environments.
LLaVA-φ: Efficient Multi-Modal Assistant with Small Language Model
Authors: name: Yichen Zhu, name: Minjie Zhu, name: Ning Liu, name: Zhicai Ou, name: Xiaofeng Mou, name: Jian Tang
Published: 2024-01-04 16:07:43

In this paper, we introduce LLaVA-phi (LLaVA-Phi), an efficient
multi-modal assistant that harnesses the power of the recently advanced small
language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a
notable advancement in the realm of compact multi-modal models. It demonstrates
that even smaller language models, with as few as 2.7B parameters, can
effectively engage in intricate dialogues that integrate both textual and
visual elements, provided they are trained with high-quality corpora. Our model
delivers commendable performance on publicly available benchmarks that
encompass visual comprehension, reasoning, and knowledge-based perception.
Beyond its remarkable performance in multi-modal dialogue tasks, our model
opens new avenues for applications in time-sensitive environments and systems
that require real-time interaction, such as embodied agents. It highlights the
potential of smaller language models to achieve sophisticated levels of
understanding and interaction, while maintaining greater resource
efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as
  Programmers
Authors: name: Aleksandar Stanić, name: Sergi Caelles, name: Michael Tschannen
Published: 2024-01-03 20:48:47

Visual reasoning is dominated by end-to-end neural networks scaled to
billions of model parameters and training examples. However, even the largest
models struggle with compositional reasoning, generalization, fine-grained
spatial and temporal reasoning, and counting. Visual reasoning with large
language models (LLMs) as controllers can, in principle, address these
limitations by decomposing the task and solving subtasks by orchestrating a set
of (visual) tools. Recently, these models achieved great performance on tasks
such as compositional visual question answering, visual grounding, and video
temporal reasoning. Nevertheless, in their current form, these models heavily
rely on human engineering of in-context examples in the prompt, which are often
dataset- and task-specific and require significant labor by highly skilled
programmers. In this work, we present a framework that mitigates these issues
by introducing spatially and temporally abstract routines and by leveraging a
small number of labeled examples to automatically generate in-context examples,
thereby avoiding human-created in-context examples. On a number of visual
reasoning tasks, we show that our framework leads to consistent gains in
performance, makes LLMs as controllers setup more robust, and removes the need
for human engineering of in-context examples.
FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D
  Scene Understanding
Authors: name: Xingxing Zuo, name: Pouya Samangouei, name: Yunwen Zhou, name: Yan Di, name: Mingyang Li
Published: 2024-01-03 20:39:02

Precisely perceiving the geometric and semantic properties of real-world 3D
objects is crucial for the continued evolution of augmented reality and robotic
applications. To this end, we present  (), which
incorporates vision-language embeddings of foundation models into 3D Gaussian
Splatting (GS). The key contribution of this work is an efficient method to
reconstruct and represent 3D vision-language models. This is achieved by
distilling feature maps generated from image-based foundation models into those
rendered from our 3D model. To ensure high-quality rendering and fast training,
we introduce a novel scene representation by integrating strengths from both GS
and multi-resolution hash encodings (MHE). Our effective training procedure
also introduces a pixel alignment loss that makes the rendered feature distance
of same semantic entities close, following the pixel-level semantic boundaries.
Our results demonstrate remarkable multi-view semantic consistency,
facilitating diverse downstream tasks, beating state-of-the-art methods by
10.2 percent on open-vocabulary language-based object detection,
despite that we are 851times faster for inference. This research
explores the intersection of vision, language, and 3D scene representation,
paving the way for enhanced scene understanding in uncontrolled real-world
environments. We plan to release the code upon paper acceptance.
TinyLlama: An Open-Source Small Language Model
Authors: name: Peiyuan Zhang, name: Guangtao Zeng, name: Tianduo Wang, name: Wei Lu
Published: 2024-01-04 17:54:59

We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and
tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
open-source community (e.g., FlashAttention), achieving better computational
efficiency. Despite its relatively small size, TinyLlama demonstrates
remarkable performance in a series of downstream tasks. It significantly
outperforms existing open-source language models with comparable sizes. Our
model checkpoints and code are publicly available on GitHub at
https://github.com/jzhang38/TinyLlama.
What You See is What You GAN: Rendering Every Pixel for High-Fidelity
  Geometry in 3D GANs
Authors: name: Alex Trevithick, name: Matthew Chan, name: Towaki Takikawa, name: Umar Iqbal, name: Shalini De Mello, name: Manmohan Chandraker, name: Ravi Ramamoorthi, name: Koki Nagano
Published: 2024-01-04 18:50:38

3D-aware Generative Adversarial Networks (GANs) have shown remarkable
progress in learning to generate multi-view-consistent images and 3D geometries
of scenes from collections of 2D images via neural volume rendering. Yet, the
significant memory and computational costs of dense sampling in volume
rendering have forced 3D GANs to adopt patch-based training or employ
low-resolution rendering with post-processing 2D super resolution, which
sacrifices multiview consistency and the quality of resolved geometry.
Consequently, 3D GANs have not yet been able to fully resolve the rich 3D
geometry present in 2D images. In this work, we propose techniques to scale
neural volume rendering to the much higher resolution of native 2D images,
thereby resolving fine-grained 3D geometry with unprecedented detail. Our
approach employs learning-based samplers for accelerating neural rendering for
3D GAN training using up to 5 times fewer depth samples. This enables us to
explicitly "render every pixel" of the full-resolution image during training
and inference without post-processing superresolution in 2D. Together with our
strategy to learn high-quality surface geometry, our method synthesizes
high-resolution 3D geometry and strictly view-consistent images while
maintaining image quality on par with baselines relying on post-processing
super resolution. We demonstrate state-of-the-art 3D gemetric quality on FFHQ
and AFHQ, setting a new standard for unsupervised learning of 3D shapes in 3D
GANs.
LLM Augmented LLMs: Expanding Capabilities through Composition
Authors: name: Rachit Bansal, name: Bidisha Samanta, name: Siddharth Dalmia, name: Nitish Gupta, name: Shikhar Vashishth, name: Sriram Ganapathy, name: Abhishek Bapna, name: Prateek Jain, name: Partha Talukdar
Published: 2024-01-04 18:53:01

Foundational models with billions of parameters which have been trained on
large corpora of data have demonstrated non-trivial skills in a variety of
domains. However, due to their monolithic structure, it is challenging and
expensive to augment them or impart new skills. On the other hand, due to their
adaptation abilities, several new instances of these models are being trained
towards new domains and tasks. In this work, we study the problem of efficient
and practical composition of existing foundation models with more specific
models to enable newer capabilities. To this end, we propose CALM --
Composition to Augment Language Models -- which introduces cross-attention
between models to compose their representations and enable new capabilities.
Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using'
existing LLMs along with a few additional parameters and data, (ii) Existing
model weights are kept intact, and hence preserves existing capabilities, and
(iii) Applies to diverse domains and settings. We illustrate that augmenting
PaLM2-S with a smaller model trained on low-resource languages results in an
absolute improvement of up to 13\% on tasks like translation into English and
arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is
augmented with a code-specific model, we see a relative improvement of 40\%
over the base model for code generation and explanation tasks -- on-par with
fully fine-tuned counterparts.
ODIN: A Single Model for 2D and 3D Perception
Authors: name: Ayush Jain, name: Pushkal Katara, name: Nikolaos Gkanatsios, name: Adam W. Harley, name: Gabriel Sarch, name: Kriti Aggarwal, name: Vishrav Chaudhary, name: Katerina Fragkiadaki
Published: 2024-01-04 18:59:25

State-of-the-art models on contemporary 3D perception benchmarks like ScanNet
consume and label dataset-provided 3D point clouds, obtained through post
processing of sensed multiview RGB-D images. They are typically trained
in-domain, forego large-scale 2D pre-training and outperform alternatives that
featurize the posed RGB-D multiview images instead. The gap in performance
between methods that consume posed images versus post-processed 3D point clouds
has fueled the belief that 2D and 3D perception require distinct model
architectures. In this paper, we challenge this view and propose ODIN
(Omni-Dimensional INstance segmentation), a model that can segment and label
both 2D RGB images and 3D point clouds, using a transformer architecture that
alternates between 2D within-view and 3D cross-view information fusion. Our
model differentiates 2D and 3D feature operations through the positional
encodings of the tokens involved, which capture pixel coordinates for 2D patch
tokens and 3D coordinates for 3D feature tokens. ODIN achieves state-of-the-art
performance on ScanNet200, Matterport3D and AI2THOR 3D instance segmentation
benchmarks, and competitive performance on ScanNet, S3DIS and COCO. It
outperforms all previous works by a wide margin when the sensed 3D point cloud
is used in place of the point cloud sampled from 3D mesh. When used as the 3D
perception engine in an instructable embodied agent architecture, it sets a new
state-of-the-art on the TEACh action-from-dialogue benchmark. Our code and
checkpoints can be found at the project website: https://odin-seg.github.io.
Instruct-Imagen: Image Generation with Multi-modal Instruction
Authors: name: Hexiang Hu, name: Kelvin C. K. Chan, name: Yu-Chuan Su, name: Wenhu Chen, name: Yandong Li, name: Kihyuk Sohn, name: Yang Zhao, name: Xue Ben, name: Boqing Gong, name: William Cohen, name: Ming-Wei Chang, name: Xuhui Jia
Published: 2024-01-03 19:31:58

This paper presents instruct-imagen, a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce *multi-modal
instruction* for image generation, a task representation articulating a range
of generation intents with precision. It uses natural language to amalgamate
disparate modalities (e.g., text, edge, style, subject, etc.), such that
abundant generation intents can be standardized in a uniform format.
  We then build instruct-imagen by fine-tuning a pre-trained text-to-image
diffusion model with a two-stage framework. First, we adapt the model using the
retrieval-augmented training, to enhance model's capabilities to ground its
generation on external multimodal context. Subsequently, we fine-tune the
adapted model on diverse image generation tasks that requires vision-language
understanding (e.g., subject-driven generation, etc.), each paired with a
multi-modal instruction encapsulating the task's essence. Human evaluation on
various image generation datasets reveals that instruct-imagen matches or
surpasses prior task-specific models in-domain and demonstrates promising
generalization to unseen and more complex tasks.
ICE-GRT: Instruction Context Enhancement by Generative Reinforcement
  based Transformers
Authors: name: Chen Zheng, name: Ke Sun, name: Da Tang, name: Yukun Ma, name: Yuyu Zhang, name: Chenguang Xi, name: Xun Zhou
Published: 2024-01-04 05:47:41

The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA
encounter limitations in domain-specific tasks, with these models often lacking
depth and accuracy in specialized areas, and exhibiting a decrease in general
capabilities when fine-tuned, particularly analysis ability in small sized
models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement
Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization
(PPO), demonstrating remarkable ability in in-domain scenarios without
compromising general task performance. Our exploration of ICE-GRT highlights
its understanding and reasoning ability to not only generate robust answers but
also to provide detailed analyses of the reasons behind the answer. This
capability marks a significant progression beyond the scope of Supervised
Fine-Tuning models. The success of ICE-GRT is dependent on several crucial
factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage
Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in
domain-specific tasks and across 12 general Language tasks against equivalent
size and even larger size LLMs, highlighting the effectiveness of our approach.
We provide a comprehensive analysis of the ICE-GRT, underscoring the
significant advancements it brings to the field of LLM.
Understanding LLMs: A Comprehensive Overview from Training to Inference
Authors: name: Yiheng Liu, name: Hao He, name: Tianle Han, name: Xu Zhang, name: Mengyuan Liu, name: Jiaming Tian, name: Yutong Zhang, name: Jiaqi Wang, name: Xiaohui Gao, name: Tianyang Zhong, name: Yi Pan, name: Shaochen Xu, name: Zihao Wu, name: Zhengliang Liu, name: Xin Zhang, name: Shu Zhang, name: Xintao Hu, name: Tuo Zhang, name: Ning Qiang, name: Tianming Liu, name: Bao Ge
Published: 2024-01-04 02:43:57

The introduction of ChatGPT has led to a significant increase in the
utilization of Large Language Models (LLMs) for addressing downstream tasks.
There's an increasing focus on cost-efficient training and deployment within
this context. Low-cost training and deployment of LLMs represent the future
development trend. This paper reviews the evolution of large language model
training techniques and inference deployment technologies aligned with this
emerging trend. The discussion on training includes various aspects, including
data preprocessing, training architecture, pre-training tasks, parallel
training, and relevant content related to model fine-tuning. On the inference
side, the paper covers topics such as model compression, parallel computation,
memory scheduling, and structural optimization. It also explores LLMs'
utilization and provides insights into their future development.
Efficient Hybrid Zoom using Camera Fusion on Mobile Phones
Authors: name: Xiaotong Wu, name: Wei-Sheng Lai, name: YiChang Shih, name: Charles Herrmann, name: Michael Krainin, name: Deqing Sun, name: Chia-Kai Liang
Published: 2024-01-02 23:28:20

DSLR cameras can achieve multiple zoom levels via shifting lens distances or
swapping lens types. However, these techniques are not possible on smartphone
devices due to space constraints. Most smartphone manufacturers adopt a hybrid
zoom system: commonly a Wide (W) camera at a low zoom level and a Telephoto (T)
camera at a high zoom level. To simulate zoom levels between W and T, these
systems crop and digitally upsample images from W, leading to significant
detail loss. In this paper, we propose an efficient system for hybrid zoom
super-resolution on mobile devices, which captures a synchronous pair of W and
T shots and leverages machine learning models to align and transfer details
from T to W. We further develop an adaptive blending method that accounts for
depth-of-field mismatches, scene occlusion, flow uncertainty, and alignment
errors. To minimize the domain gap, we design a dual-phone camera rig to
capture real-world inputs and ground-truths for supervised training. Our method
generates a 12-megapixel image in 500ms on a mobile platform and compares
favorably against state-of-the-art methods under extensive evaluation on
real-world scenarios.
Incremental FastPitch: Chunk-based High Quality Text to Speech
Authors: name: Muyang Du, name: Chuan Liu, name: Junjie Lai
Published: 2024-01-03 14:17:35

Parallel text-to-speech models have been widely applied for real-time speech
synthesis, and they offer more controllability and a much faster synthesis
process compared with conventional auto-regressive models. Although parallel
models have benefits in many aspects, they become naturally unfit for
incremental synthesis due to their fully parallel architecture such as
transformer. In this work, we propose Incremental FastPitch, a novel FastPitch
variant capable of incrementally producing high-quality Mel chunks by improving
the architecture with chunk-based FFT blocks, training with receptive-field
constrained chunk attention masks, and inference with fixed size past model
states. Experimental results show that our proposal can produce speech quality
comparable to the parallel FastPitch, with a significant lower latency that
allows even lower response time for real-time speech applications.