Orionnnn/hub-stats huggingface dataset:
Gradient Boosting Reinforcement Learning
Authors: name: Benjamin Fuhrer, name: Chen Tessler, name: Gal Dalal
Published: 2024-07-11 07:52:33

Neural networks (NN) achieve remarkable results in various tasks, but lack
key characteristics: interpretability, support for categorical features, and
lightweight implementations suitable for edge devices. While ongoing efforts
aim to address these challenges, Gradient Boosting Trees (GBT) inherently meet
these requirements. As a result, GBTs have become the go-to method for
supervised learning tasks in many real-world applications and competitions.
However, their application in online learning scenarios, notably in
reinforcement learning (RL), has been limited. In this work, we bridge this gap
by introducing Gradient-Boosting RL (GBRL), a framework that extends the
advantages of GBT to the RL domain. Using the GBRL framework, we implement
various actor-critic algorithms and compare their performance with their NN
counterparts. Inspired by shared backbones in NN we introduce a tree-sharing
approach for policy and value functions with distinct learning rates, enhancing
learning efficiency over millions of interactions. GBRL achieves competitive
performance across a diverse array of tasks, excelling in domains with
structured or categorical features. Additionally, we present a
high-performance, GPU-accelerated implementation that integrates seamlessly
with widely-used RL libraries (available at https://github.com/NVlabs/gbrl).
GBRL expands the toolkit for RL practitioners, demonstrating the viability and
promise of GBT within the RL paradigm, particularly in domains characterized by
structured or categorical features.
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
Authors: name: Ali Hatamizadeh, name: Jan Kautz
Published: 2024-07-10 23:02:45

We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision,
which is specifically tailored for vision applications. Our core contribution
includes redesigning the Mamba formulation to enhance its capability for
efficient modeling of visual features. In addition, we conduct a comprehensive
ablation study on the feasibility of integrating Vision Transformers (ViT) with
Mamba. Our results demonstrate that equipping the Mamba architecture with
several self-attention blocks at the final layers greatly improves the modeling
capacity to capture long-range spatial dependencies. Based on our findings, we
introduce a family of MambaVision models with a hierarchical architecture to
meet various design criteria. For Image classification on ImageNet-1K dataset,
MambaVision model variants achieve a new State-of-the-Art (SOTA) performance in
terms of Top-1 accuracy and image throughput. In downstream tasks such as
object detection, instance segmentation and semantic segmentation on MS COCO
and ADE20K datasets, MambaVision outperforms comparably-sized backbones and
demonstrates more favorable performance. Code:
https://github.com/NVlabs/MambaVision.
Still-Moving: Customized Video Generation without Customized Video Data
Authors: name: Hila Chefer, name: Shiran Zada, name: Roni Paiss, name: Ariel Ephrat, name: Omer Tov, name: Michael Rubinstein, name: Lior Wolf, name: Tali Dekel, name: Tomer Michaeli, name: Inbar Mosseri
Published: 2024-07-11 17:06:53

Customizing text-to-image (T2I) models has seen tremendous progress recently,
particularly in areas such as personalization, stylization, and conditional
generation. However, expanding this progress to video generation is still in
its infancy, primarily due to the lack of customized video data. In this work,
we introduce Still-Moving, a novel generic framework for customizing a
text-to-video (T2V) model, without requiring any customized video data. The
framework applies to the prominent T2V design where the video model is built
over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
customized version of the T2I model, trained only on still image data (e.g.,
using DreamBooth or StyleDrop). Naively plugging in the weights of the
customized T2I model into the T2V model often leads to significant artifacts or
insufficient adherence to the customization data. To overcome this issue, we
train lightweight Spatial Adapters that adjust the features produced
by the injected T2I layers. Importantly, our adapters are trained on
"frozen videos" (i.e., repeated images), constructed from image
samples generated by the customized T2I model. This training is facilitated by
a novel Motion Adapter module, which allows us to train on such
static videos while preserving the motion prior of the video model. At test
time, we remove the Motion Adapter modules and leave in only the trained
Spatial Adapters. This restores the motion prior of the T2V model while
adhering to the spatial prior of the customized T2I model. We demonstrate the
effectiveness of our approach on diverse tasks including personalized,
stylized, and conditional generation. In all evaluated scenarios, our method
seamlessly integrates the spatial prior of the customized T2I model with a
motion prior supplied by the T2V model.
CrowdMoGen: Zero-Shot Text-Driven Collective Motion Generation
Authors: name: Xinying Guo, name: Mingyuan Zhang, name: Haozhe Xie, name: Chenyang Gu, name: Ziwei Liu
Published: 2024-07-08 17:59:36

Crowd Motion Generation is essential in entertainment industries such as
animation and games as well as in strategic fields like urban simulation and
planning. This new task requires an intricate integration of control and
generation to realistically synthesize crowd dynamics under specific spatial
and semantic constraints, whose challenges are yet to be fully explored. On the
one hand, existing human motion generation models typically focus on individual
behaviors, neglecting the complexities of collective behaviors. On the other
hand, recent methods for multi-person motion generation depend heavily on
pre-defined scenarios and are limited to a fixed, small number of inter-person
interactions, thus hampering their practicality. To overcome these challenges,
we introduce CrowdMoGen, a zero-shot text-driven framework that harnesses the
power of Large Language Model (LLM) to incorporate the collective intelligence
into the motion generation framework as guidance, thereby enabling
generalizable planning and generation of crowd motions without paired training
data. Our framework consists of two key components: 1) Crowd Scene Planner that
learns to coordinate motions and dynamics according to specific scene contexts
or introduced perturbations, and 2) Collective Motion Generator that
efficiently synthesizes the required collective motions based on the holistic
plans. Extensive quantitative and qualitative experiments have validated the
effectiveness of our framework, which not only fills a critical gap by
providing scalable and generalizable solutions for Crowd Motion Generation task
but also achieves high levels of realism and flexibility.
Do Vision and Language Models Share Concepts? A Vector Space Alignment
  Study
Authors: name: Jiaang Li, name: Yova Kementchedjhieva, name: Constanza Fierro, name: Anders Søgaard
Published: 2023-02-13 17:55:54

Large-scale pretrained language models (LMs) are said to ``lack the ability
to connect utterances to the world'' (Bender and Koller, 2020), because they do
not have ``mental models of the world' '(Mitchell and Krakauer, 2023). If so,
one would expect LM representations to be unrelated to representations induced
by vision models. We present an empirical evaluation across four families of
LMs (BERT, GPT-2, OPT and LLaMA-2) and three vision model architectures
(ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge
towards representations isomorphic to those of vision models, subject to
dispersion, polysemy and frequency. This has important implications for both
multi-modal processing and the LM understanding debate (Mitchell and Krakauer,
2023).
This&That: Language-Gesture Controlled Video Generation for Robot
  Planning
Authors: name: Boyang Wang, name: Nikhil Sridhar, name: Chao Feng, name: Mark Van der Merwe, name: Adam Fishman, name: Nima Fazeli, name: Jeong Joon Park
Published: 2024-07-08 00:28:41

We propose a robot learning method for communicating, planning, and executing
a wide range of tasks, dubbed This&That. We achieve robot planning for general
tasks by leveraging the power of video generative models trained on
internet-scale data containing rich physical and semantic context. In this
work, we tackle three fundamental challenges in video-based planning: 1)
unambiguous task communication with simple human instructions, 2) controllable
video generation that respects user intents, and 3) translating visual planning
into robot actions. We propose language-gesture conditioning to generate
videos, which is both simpler and clearer than existing language-only methods,
especially in complex and uncertain environments. We then suggest a behavioral
cloning design that seamlessly incorporates the video plans. This&That
demonstrates state-of-the-art effectiveness in addressing the above three
challenges, and justifies the use of video generation as an intermediate
representation for generalizable task planning and execution. Project website:
https://cfeng16.github.io/this-and-that/.
On Leakage of Code Generation Evaluation Datasets
Authors: name: Alexandre Matton, name: Tom Sherborne, name: Dennis Aumiller, name: Elena Tommasone, name: Milad Alizadeh, name: Jingyi He, name: Raymond Ma, name: Maxime Voisin, name: Ellen Gilsenan-McMahon, name: Matthias Gallé
Published: 2024-07-10 11:50:20

In this paper we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model selection.
  Key to our findings is a new dataset of 161 prompts with their associated
python solutions, dataset which is released at
https://huggingface.co/datasets/CohereForAI/lbpp .
CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging
Authors: name: Raza Imam, name: Mohammed Talha Alam, name: Umaima Rahman, name: Mohsen Guizani, name: Fakhri Karray
Published: 2024-07-10 02:24:43

Existing vision-text contrastive learning models enhance representation
transferability and support zero-shot prediction by matching paired image and
caption embeddings while pushing unrelated pairs apart. However, astronomical
image-label datasets are significantly smaller compared to general image and
label datasets available from the internet. We introduce CosmoCLIP, an
astronomical image-text contrastive learning framework precisely fine-tuned on
the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet,
attained via FLARE, constitutes ~13k optimally distributed images, while BLIP
acts as a rich knowledge extractor. The rich semantics derived from this
SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to
achieve superior generalization across various in-domain and out-of-domain
tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful
framework, significantly outperforming CLIP in zero-shot classification and
image-text retrieval tasks.
An accurate detection is not all you need to combat label noise in
  web-noisy datasets
Authors: name: Paul Albert, name: Jack Valmadre, name: Eric Arazo, name: Tarun Krishna, name: Noel E. O'Connor, name: Kevin McGuinness
Published: 2024-07-08 00:21:42

Training a classifier on web-crawled data demands learning algorithms that
are robust to annotation errors and irrelevant examples. This paper builds upon
the recent empirical observation that applying unsupervised contrastive
learning to noisy, web-crawled datasets yields a feature representation under
which the in-distribution (ID) and out-of-distribution (OOD) samples are
linearly separable. We show that direct estimation of the separating hyperplane
can indeed offer an accurate detection of OOD samples, and yet, surprisingly,
this detection does not translate into gains in classification accuracy.
Digging deeper into this phenomenon, we discover that the near-perfect
detection misses a type of clean examples that are valuable for supervised
learning. These examples often represent visually simple images, which are
relatively easy to identify as clean examples using standard loss- or
distance-based methods despite being poorly separated from the OOD distribution
using unsupervised learning. Because we further observe a low correlation with
SOTA metrics, this urges us to propose a hybrid solution that alternates
between noise detection using linear separation and a state-of-the-art (SOTA)
small-loss approach. When combined with the SOTA algorithm PLS, we
substantially improve SOTA results for real-world image classification in the
presence of web noise github.com/PaulAlbert31/LSA
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
  Multimodal Models
Authors: name: Feng Li, name: Renrui Zhang, name: Hao Zhang, name: Yuanhan Zhang, name: Bo Li, name: Wei Li, name: Zejun Ma, name: Chunyuan Li
Published: 2024-07-10 17:59:43

Visual instruction tuning has made considerable strides in enhancing the
capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
largely focus on single-image tasks, their applications to multi-image
scenarios remains less explored. Additionally, prior LMM research separately
tackles different scenarios, leaving it impossible to generalize cross
scenarios with new emerging capabilities. To this end, we introduce
LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
(video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
enable these capabilities, we regard the interleaved data format as a general
template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
primary domains with 14 tasks and 41 datasets. We also curate the
LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
results in multi-image, video, and 3D benchmarks, while maintaining the
performance of single-image tasks. Besides, our model also exhibits several
emerging capabilities, e.g., transferring tasks across different settings and
modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT
Controlling Space and Time with Diffusion Models
Authors: name: Daniel Watson, name: Saurabh Saxena, name: Lala Li, name: Andrea Tagliasacchi, name: David J. Fleet
Published: 2024-07-10 17:23:33

We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
(NVS), conditioned on one or more images of a general scene, and a set of
camera poses and timestamps. To overcome challenges due to limited availability
of 4D training data, we advocate joint training on 3D (with camera pose), 4D
(pose+time) and video (time but no pose) data and propose a new architecture
that enables the same. We further advocate the calibration of SfM posed data
using monocular metric depth estimators for metric scale camera control. For
model evaluation, we introduce new metrics to enrich and overcome shortcomings
of current evaluation schemes, demonstrating state-of-the-art results in both
fidelity and pose control compared to existing diffusion models for 3D NVS,
while at the same time adding the ability to handle temporal dynamics. 4DiM is
also used for improved panorama stitching, pose-conditioned video to video
translation, and several other tasks. For an overview see
https://4d-diffusion.github.io
BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark
Authors: name: Nikita Chernyadev, name: Nicholas Backshall, name: Xiao Ma, name: Yunfan Lu, name: Younggyo Seo, name: Stephen James
Published: 2024-07-10 16:04:18

We introduce BiGym, a new benchmark and learning environment for mobile
bi-manual demo-driven robotic manipulation. BiGym features 40 diverse tasks set
in home environments, ranging from simple target reaching to complex kitchen
cleaning. To capture the real-world performance accurately, we provide
human-collected demonstrations for each task, reflecting the diverse modalities
found in real-world robot trajectories. BiGym supports a variety of
observations, including proprioceptive data and visual inputs such as RGB, and
depth from 3 camera views. To validate the usability of BiGym, we thoroughly
benchmark the state-of-the-art imitation learning algorithms and demo-driven
reinforcement learning algorithms within the environment and discuss the future
opportunities.
PaliGemma: A versatile 3B VLM for transfer
Authors: name: Lucas Beyer, name: Andreas Steiner, name: André Susano Pinto, name: Alexander Kolesnikov, name: Xiao Wang, name: Daniel Salz, name: Maxim Neumann, name: Ibrahim Alabdulmohsin, name: Michael Tschannen, name: Emanuele Bugliarello, name: Thomas Unterthiner, name: Daniel Keysers, name: Skanda Koppula, name: Fangyu Liu, name: Adam Grycner, name: Alexey Gritsenko, name: Neil Houlsby, name: Manoj Kumar, name: Keran Rong, name: Julian Eisenschlos, name: Rishabh Kabra, name: Matthias Bauer, name: Matko Bošnjak, name: Xi Chen, name: Matthias Minderer, name: Paul Voigtlaender, name: Ioana Bica, name: Ivana Balazevic, name: Joan Puigcerver, name: Pinelopi Papalampidi, name: Olivier Henaff, name: Xi Xiong, name: Radu Soricut, name: Jeremiah Harmsen, name: Xiaohua Zhai
Published: 2024-07-10 14:57:46

PaliGemma is an open Vision-Language Model (VLM) that is based on the
SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to
be a versatile and broadly knowledgeable base model that is effective to
transfer. It achieves strong performance on a wide variety of open-world tasks.
We evaluate PaliGemma on almost 40 diverse tasks including standard VLM
benchmarks, but also more specialized tasks such as remote-sensing and
segmentation.
Inference Performance Optimization for Large Language Models on CPUs
Authors: name: Pujiang He, name: Shan Zhou, name: Wenhuan Huang, name: Changqing Li, name: Duyi Wang, name: Bin Guo, name: Chen Meng, name: Sheng Gui, name: Weifei Yu, name: Yi Xie
Published: 2024-07-10 01:53:49

Large language models (LLMs) have shown exceptional performance and vast
potential across diverse tasks. However, the deployment of LLMs with high
performance in low-resource environments has garnered significant attention in
the industry. When GPU hardware resources are limited, we can explore
alternative options on CPUs. To mitigate the financial burden and alleviate
constraints imposed by hardware resources, optimizing inference performance is
necessary. In this paper, we introduce an easily deployable inference
performance optimization solution aimed at accelerating LLMs on CPUs. In this
solution, we implement an effective way to reduce the KV cache size while
ensuring precision. We propose a distributed inference optimization approach
and implement it based on oneAPI Collective Communications Library.
Furthermore, we propose optimization approaches for LLMs on CPU, and conduct
tailored optimizations for the most commonly used models. The code is
open-sourced at https://github.com/intel/xFasterTransformer.
VEnhancer: Generative Space-Time Enhancement for Video Generation
Authors: name: Jingwen He, name: Tianfan Xue, name: Dongyang Liu, name: Xinqi Lin, name: Peng Gao, name: Dahua Lin, name: Yu Qiao, name: Wanli Ouyang, name: Ziwei Liu
Published: 2024-07-10 13:46:08

We present VEnhancer, a generative space-time enhancement framework that
improves the existing text-to-video results by adding more details in spatial
domain and synthetic detailed motion in temporal domain. Given a generated
low-quality video, our approach can increase its spatial and temporal
resolution simultaneously with arbitrary up-sampling space and time scales
through a unified video diffusion model. Furthermore, VEnhancer effectively
removes generated spatial artifacts and temporal flickering of generated
videos. To achieve this, basing on a pretrained video diffusion model, we train
a video ControlNet and inject it to the diffusion model as a condition on low
frame-rate and low-resolution videos. To effectively train this video
ControlNet, we design space-time data augmentation as well as video-aware
conditioning. Benefiting from the above designs, VEnhancer yields to be stable
during training and shares an elegant end-to-end training manner. Extensive
experiments show that VEnhancer surpasses existing state-of-the-art video
super-resolution and space-time super-resolution methods in enhancing
AI-generated videos. Moreover, with VEnhancer, exisiting open-source
state-of-the-art text-to-video method, VideoCrafter-2, reaches the top one in
video generation benchmark -- VBench.
Video-to-Audio Generation with Hidden Alignment
Authors: name: Manjie Xu, name: Chenxing Li, name: Yong Ren, name: Rilin Chen, name: Yu Gu, name: Wei Liang, name: Dong Yu
Published: 2024-07-10 08:40:39

Generating semantically and temporally aligned audio content in accordance
with video input has become a focal point for researchers, particularly
following the remarkable breakthrough in text-to-video generation. In this
work, we aim to offer insights into the video-to-audio generation paradigm,
focusing on three crucial aspects: vision encoders, auxiliary embeddings, and
data augmentation techniques. Beginning with a foundational model VTA-LDM built
on a simple yet surprisingly effective intuition, we explore various vision
encoders and auxiliary embeddings through ablation studies. Employing a
comprehensive evaluation pipeline that emphasizes generation quality and
video-audio synchronization alignment, we demonstrate that our model exhibits
state-of-the-art video-to-audio generation capabilities. Furthermore, we
provide critical insights into the impact of different data augmentation
methods on enhancing the generation framework's overall capacity. We showcase
possibilities to advance the challenge of generating synchronized audio from
semantic and temporal perspectives. We hope these insights will serve as a
stepping stone toward developing more realistic and accurate audio-visual
generation models.
LETS-C: Leveraging Language Embedding for Time Series Classification
Authors: name: Rachneet Kaur, name: Zhen Zeng, name: Tucker Balch, name: Manuela Veloso
Published: 2024-07-09 04:07:57

Recent advancements in language modeling have shown promising results when
applied to time series data. In particular, fine-tuning pre-trained large
language models (LLMs) for time series classification tasks has achieved
state-of-the-art (SOTA) performance on standard benchmarks. However, these
LLM-based models have a significant drawback due to the large model size, with
the number of trainable parameters in the millions. In this paper, we propose
an alternative approach to leveraging the success of language modeling in the
time series domain. Instead of fine-tuning LLMs, we utilize a language
embedding model to embed time series and then pair the embeddings with a simple
classification head composed of convolutional neural networks (CNN) and
multilayer perceptron (MLP). We conducted extensive experiments on
well-established time series classification benchmark datasets. We demonstrated
LETS-C not only outperforms the current SOTA in classification accuracy but
also offers a lightweight solution, using only 14.5% of the trainable
parameters on average compared to the SOTA model. Our findings suggest that
leveraging language encoders to embed time series data, combined with a simple
yet effective classification head, offers a promising direction for achieving
high-performance time series classification while maintaining a lightweight
model architecture.