Orionnnn/hub-stats huggingface dataset:
EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video
  Generation
Authors: name: Xiaofeng Wang, name: Kang Zhao, name: Feng Liu, name: Jiayu Wang, name: Guosheng Zhao, name: Xiaoyi Bao, name: Zheng Zhu, name: Yingya Zhang, name: Xingang Wang
Published: 2024-11-13 07:05:40

Video generation has emerged as a promising tool for world simulation,
leveraging visual data to replicate real-world environments. Within this
context, egocentric video generation, which centers on the human perspective,
holds significant potential for enhancing applications in virtual reality,
augmented reality, and gaming. However, the generation of egocentric videos
presents substantial challenges due to the dynamic nature of egocentric
viewpoints, the intricate diversity of actions, and the complex variety of
scenes encountered. Existing datasets are inadequate for addressing these
challenges effectively. To bridge this gap, we present EgoVid-5M, the first
high-quality dataset specifically curated for egocentric video generation.
EgoVid-5M encompasses 5 million egocentric video clips and is enriched with
detailed action annotations, including fine-grained kinematic control and
high-level textual descriptions. To ensure the integrity and usability of the
dataset, we implement a sophisticated data cleaning pipeline designed to
maintain frame consistency, action coherence, and motion smoothness under
egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable
of generating egocentric videos driven simultaneously by action descriptions
and kinematic control signals. The EgoVid-5M dataset, associated action
annotations, and all data cleansing metadata will be released for the
advancement of research in egocentric video generation.
Direct Preference Optimization Using Sparse Feature-Level Constraints
Authors: name: Qingyu Yin, name: Chak Tou Leong, name: Hongbo Zhang, name: Minjun Zhu, name: Hanqi Yan, name: Qiang Zhang, name: Yulan He, name: Wenjie Li, name: Jun Wang, name: Yue Zhang, name: Linyi Yang
Published: 2024-11-12 07:54:13

The alignment of large language models (LLMs) with human preferences remains
a key challenge. While post-training techniques like Reinforcement Learning
from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have
achieved notable success, they often introduce computational inefficiencies and
training instability. In this paper, we propose Feature-level constrained
Preference Optimization (FPO), a novel method designed to simplify the
alignment process while ensuring stability. FPO leverages pre-trained Sparse
Autoencoders (SAEs) and introduces feature-level constraints, allowing for
efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using
sparse features activated in a well-trained sparse autoencoder and the quality
of sequential KL divergence by using the feature-level offline reference.
Experimental results on benchmark datasets demonstrate that FPO achieves a
5.08% absolute improvement in win rate with much lower computational cost
compared to state-of-the-art baselines, making it a promising solution for
efficient and controllable LLM alignments.
Scaling Properties of Diffusion Models for Perceptual Tasks
Authors: name: Rahul Ravishankar, name: Zeeshan Patel, name: Jathushan Rajasegaran, name: Jitendra Malik
Published: 2024-11-12 18:59:35

In this paper, we argue that iterative computation with diffusion models
offers a powerful paradigm for not only generation but also visual perception
tasks. We unify tasks such as depth estimation, optical flow, and segmentation
under image-to-image translation, and show how diffusion models benefit from
scaling training and test-time compute for these perception tasks. Through a
careful analysis of these scaling behaviors, we present various techniques to
efficiently train diffusion models for visual perception tasks. Our models
achieve improved or comparable performance to state-of-the-art methods using
significantly less data and compute. To use our code and models, see
https://scaling-diffusion-perception.github.io .
Acoustic Volume Rendering for Neural Impulse Response Fields
Authors: name: Zitong Lan, name: Chenhao Zheng, name: Zhiwei Zheng, name: Mingmin Zhao
Published: 2024-11-09 23:15:51

Realistic audio synthesis that captures accurate acoustic phenomena is
essential for creating immersive experiences in virtual and augmented reality.
Synthesizing the sound received at any position relies on the estimation of
impulse response (IR), which characterizes how sound propagates in one scene
along different paths before arriving at the listener's position. In this
paper, we present Acoustic Volume Rendering (AVR), a novel approach that adapts
volume rendering techniques to model acoustic impulse responses. While volume
rendering has been successful in modeling radiance fields for images and neural
scene representations, IRs present unique challenges as time-series signals. To
address these challenges, we introduce frequency-domain volume rendering and
use spherical integration to fit the IR measurements. Our method constructs an
impulse response field that inherently encodes wave propagation principles and
achieves state-of-the-art performance in synthesizing impulse responses for
novel poses. Experiments show that AVR surpasses current leading methods by a
substantial margin. Additionally, we develop an acoustic simulation platform,
AcoustiX, which provides more accurate and realistic IR simulations than
existing simulators. Code for AVR and AcoustiX are available at
https://zitonglan.github.io/avr.
BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions
Authors: name: Anas Awadalla, name: Le Xue, name: Manli Shu, name: An Yan, name: Jun Wang, name: Senthil Purushwalkam, name: Sheng Shen, name: Hannah Lee, name: Oscar Lo, name: Jae Sung Park, name: Etash Guha, name: Silvio Savarese, name: Ludwig Schmidt, name: Yejin Choi, name: Caiming Xiong, name: Ran Xu
Published: 2024-11-12 00:52:52

We introduce BLIP3-KALE, a dataset of 218 million image-text pairs that
bridges the gap between descriptive synthetic captions and factual web-scale
alt-text. KALE augments synthetic dense image captions with web-scale alt-text
to generate factually grounded image captions. Our two-stage approach leverages
large vision-language models and language models to create knowledge-augmented
captions, which are then used to train a specialized VLM for scaling up the
dataset. We train vision-language models on KALE and demonstrate improvements
on vision-language tasks. Our experiments show the utility of KALE for training
more capable and knowledgeable multimodal models. We release the KALE dataset
at https://huggingface.co/datasets/Salesforce/blip3-kale
Hardware and Software Platform Inference
Authors: name: Cheng Zhang, name: Hanna Foerster, name: Robert D. Mullins, name: Yiren Zhao, name: Ilia Shumailov
Published: 2024-11-07 21:44:04

It is now a common business practice to buy access to large language model
(LLM) inference rather than self-host, because of significant upfront hardware
infrastructure and energy costs. However, as a buyer, there is no mechanism to
verify the authenticity of the advertised service including the serving
hardware platform, e.g. that it is actually being served using an NVIDIA H100.
Furthermore, there are reports suggesting that model providers may deliver
models that differ slightly from the advertised ones, often to make them run on
less expensive hardware. That way, a client pays premium for a capable model
access on more expensive hardware, yet ends up being served by a (potentially
less capable) cheaper model on cheaper hardware. In this paper we introduce
\textbf{hardware and software platform inference (HSPI)} -- a method
for identifying the underlying  architecture and software stack of a
(black-box) machine learning model solely based on its input-output behavior.
Our method leverages the inherent differences of various  architectures
and compilers to distinguish between different  types and software
stacks. By analyzing the numerical patterns in the model's outputs, we propose
a classification framework capable of accurately identifying the  used
for model inference as well as the underlying software configuration. Our
findings demonstrate the feasibility of inferring  type from black-box
models. We evaluate HSPI against models served on different real hardware and
find that in a white-box setting we can distinguish between different s
with between 83.9% and 100% accuracy. Even in a black-box setting we are
able to achieve results that are up to three times higher than random guess
accuracy.
Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model
  with Compact Wavelet Encodings
Authors: name: Aditya Sanghi, name: Aliasghar Khani, name: Pradyumna Reddy, name: Arianna Rampini, name: Derek Cheung, name: Kamal Rahimi Malekshan, name: Kanika Madan, name: Hooman Shayani
Published: 2024-11-12 18:49:06

Large-scale 3D generative models require substantial computational resources
yet often fall short in capturing fine details and complex geometries at high
resolutions. We attribute this limitation to the inefficiency of current
representations, which lack the compactness required to model the generative
models effectively. To address this, we introduce a novel approach called
Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based,
compact latent encodings. Specifically, we compress a 256^3 signed distance
field into a 12^3 times 4 latent grid, achieving an impressive 2427x
compression ratio with minimal loss of detail. This high level of compression
allows our method to efficiently train large-scale generative networks without
increasing the inference time. Our models, both conditional and unconditional,
contain approximately one billion parameters and successfully generate
high-quality 3D shapes at 256^3 resolution. Moreover, WaLa offers rapid
inference, producing shapes within two to four seconds depending on the
condition, despite the model's scale. We demonstrate state-of-the-art
performance across multiple datasets, with significant improvements in
generation quality, diversity, and computational efficiency. We open-source our
code and, to the best of our knowledge, release the largest pretrained 3D
generative models across different modalities.
Stronger Models are NOT Stronger Teachers for Instruction Tuning
Authors: name: Zhangchen Xu, name: Fengqing Jiang, name: Luyao Niu, name: Bill Yuchen Lin, name: Radha Poovendran
Published: 2024-11-11 17:06:48

Instruction tuning has been widely adopted to ensure large language models
(LLMs) follow user instructions effectively. The resulting
instruction-following capabilities of LLMs heavily rely on the instruction
datasets used for tuning. Recently, synthetic instruction datasets have emerged
as an economically viable solution to provide LLMs diverse and high-quality
instructions. However, existing approaches typically assume that larger or
stronger models are stronger teachers for instruction tuning, and hence simply
adopt these models as response generators to the synthetic instructions. In
this paper, we challenge this commonly-adopted assumption. Our extensive
experiments across five base models and twenty response generators reveal that
larger and stronger models are not necessarily stronger teachers of smaller
models. We refer to this phenomenon as the Larger Models' Paradox. We observe
that existing metrics cannot precisely predict the effectiveness of response
generators since they ignore the compatibility between teachers and base models
being fine-tuned. We thus develop a novel metric, named as
Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response
generators. Our experiments across five base models demonstrate that CAR
outperforms almost all baselines.
JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified
  Multimodal Understanding and Generation
Authors: name: Yiyang Ma, name: Xingchao Liu, name: Xiaokang Chen, name: Wen Liu, name: Chengyue Wu, name: Zhiyu Wu, name: Zizheng Pan, name: Zhenda Xie, name: Haowei Zhang, name: Xingkai yu, name: Liang Zhao, name: Yisong Wang, name: Jiaying Liu, name: Chong Ruan
Published: 2024-11-12 17:55:10

We present JanusFlow, a powerful framework that unifies image understanding
and generation in a single model. JanusFlow introduces a minimalist
architecture that integrates autoregressive language models with rectified
flow, a state-of-the-art method in generative modeling. Our key finding
demonstrates that rectified flow can be straightforwardly trained within the
large language model framework, eliminating the need for complex architectural
modifications. To further improve the performance of our unified model, we
adopt two key strategies: (i) decoupling the understanding and generation
encoders, and (ii) aligning their representations during unified training.
Extensive experiments show that JanusFlow achieves comparable or superior
performance to specialized models in their respective domains, while
significantly outperforming existing unified approaches across standard
benchmarks. This work represents a step toward more efficient and versatile
vision-language models.
SAMPart3D: Segment Any Part in 3D Objects
Authors: name: Yunhan Yang, name: Yukun Huang, name: Yuan-Chen Guo, name: Liangjun Lu, name: Xiaoyang Wu, name: Edmund Y. Lam, name: Yan-Pei Cao, name: Xihui Liu
Published: 2024-11-11 17:59:10

3D part segmentation is a crucial and challenging task in 3D perception,
playing a vital role in applications such as robotics, 3D generation, and 3D
editing. Recent methods harness the powerful Vision Language Models (VLMs) for
2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation.
However, these methods are limited by their reliance on text prompts, which
restricts the scalability to large-scale unlabeled datasets and the flexibility
in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable
zero-shot 3D part segmentation framework that segments any 3D object into
semantic parts at multiple granularities, without requiring predefined part
label sets as text prompts. For scalability, we use text-agnostic vision
foundation models to distill a 3D feature extraction backbone, allowing scaling
to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we
distill scale-conditioned part-aware 3D features for 3D part segmentation at
multiple granularities. Once the segmented parts are obtained from the
scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels
to each part based on the multi-view renderings. Compared to previous methods,
our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse
and handle complex, non-ordinary objects. Additionally, we contribute a new 3D
part segmentation benchmark to address the lack of diversity and complexity of
objects and parts in existing benchmarks. Experiments show that our SAMPart3D
significantly outperforms existing zero-shot 3D part segmentation methods, and
can facilitate various applications such as part-level editing and interactive
segmentation.
Energy Efficient Protein Language Models: Leveraging Small Language
  Models with LoRA for Controllable Protein Generation
Authors: name: Aayush Shah, name: Shankar Jayaratnam
Published: 2024-11-08 20:52:06

Large language models (LLMs) have demonstrated significant success in natural
language processing (NLP) tasks and have shown promising results in other
domains such as protein sequence generation. However, there remain salient
differences between LLMs used for NLP, which effectively handle multiple tasks
and are available in small sizes, and protein language models that are often
specialized for specific tasks and only exist in larger sizes. In this work, we
introduce two small protein language models, based on Llama-3-8B and
Phi-3-mini, that are capable of both uncontrollable and controllable protein
generation. For the uncontrollable generation task, our best model achieves an
average pLDDT score of 69.75, demonstrating robust performance in generating
viable protein structures. For the controllable generation task, in which the
model generates proteins according to properties specified in the prompt, we
achieve a remarkable average TM-Score of 0.84, indicating high structural
similarity to target proteins. We chose 10 properties, including six classes of
enzymes, to extend the capabilities of prior protein language models. Our
approach utilizes the Low-Rank Adaptor (LoRA) technique, reducing trainable
parameters to just 4% of the original model size, lowering computational
requirements. By using a subset of the UniRef50 dataset and small models, we
reduced the overall training time by 70% without compromising performance.
Notably, Phi-3-mini reduced trainable parameters by 60%, decreasing training
cost by 30% compared to Llama 3. Consequently, Phi-3 achieved a comparable
TM-Score of 0.81, demonstrating that smaller models can match the performance
of larger ones, like Llama 3. We also demonstrate the deployment of our models
on the energy efficient ET-SoC-1 chip, significantly improving the TPS/W by a
factor of 3.
Autoregressive Models in Vision: A Survey
Authors: name: Jing Xiong, name: Gongye Liu, name: Lun Huang, name: Chengyue Wu, name: Taiqiang Wu, name: Yao Mu, name: Yuan Yao, name: Hui Shen, name: Zhongwei Wan, name: Jinfa Huang, name: Chaofan Tao, name: Shen Yan, name: Huaxiu Yao, name: Lingpeng Kong, name: Hongxia Yang, name: Mi Zhang, name: Guillermo Sapiro, name: Jiebo Luo, name: Ping Luo, name: Ngai Wong
Published: 2024-11-08 17:15:12

Autoregressive modeling has been a huge success in the field of natural
language processing (NLP). Recently, autoregressive models have emerged as a
significant area of focus in computer vision, where they excel in producing
high-quality visual content. Autoregressive models in NLP typically operate on
subword tokens. However, the representation strategy in computer vision can
vary in different levels, i.e., pixel-level, token-level, or
scale-level, reflecting the diverse and hierarchical nature of visual data
compared to the sequential structure of language. This survey comprehensively
examines the literature on autoregressive models applied to vision. To improve
readability for researchers from diverse research backgrounds, we start with
preliminary sequence representation and modeling in vision. Next, we divide the
fundamental frameworks of visual autoregressive models into three general
sub-categories, including pixel-based, token-based, and scale-based models
based on the strategy of representation. We then explore the interconnections
between autoregressive models and other generative models. Furthermore, we
present a multi-faceted categorization of autoregressive models in computer
vision, including image generation, video generation, 3D generation, and
multi-modal generation. We also elaborate on their applications in diverse
domains, including emerging domains such as embodied AI and 3D medical AI, with
about 250 related references. Finally, we highlight the current challenges to
autoregressive models in vision with suggestions about potential research
directions. We have also set up a Github repository to organize the papers
included in this survey at:
https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.
GitChameleon: Unmasking the Version-Switching Capabilities of Code
  Generation Models
Authors: name: Nizar Islah, name: Justine Gehring, name: Diganta Misra, name: Eilif Muller, name: Irina Rish, name: Terry Yue Zhuo, name: Massimo Caccia
Published: 2024-11-05 23:34:06

The rapid evolution of software libraries presents a significant challenge
for code generation models, which must adapt to frequent version updates while
maintaining compatibility with previous versions. Existing code completion
benchmarks often overlook this dynamic aspect, and the one that does consider
it relies on static code prediction tasks without execution-based evaluation,
offering a limited perspective on a model's practical usability. To address
this gap, we introduce \GitChameleon{}, a novel, manually curated
dataset comprising 116 Python code completion problems, each conditioned on
specific library versions and accompanied by executable unit tests.
 is designed to rigorously assess the ability of modern large
language models (LLMs) to generate version-specific code that is not only
syntactically correct but also functionally accurate upon execution. Our
comprehensive evaluations reveal that state-of-the-art LLMs struggle with this
task; for instance, GPT-4o achieves a pass@10 of only 39.9\% (43.7\%
when provided with error feedback), highlighting the complexity of the problem
and the limitations of current models. By providing an execution-based
benchmark that emphasizes the dynamic nature of code libraries, 
serves as a critical tool to advance the development of more adaptable and
reliable code generation models. For facilitation for further exploration of
version-conditioned code generation, we make our code repository publicly
accessible at https://github.com/NizarIslah/GitChameleon.
NeKo: Toward Post Recognition Generative Correction Large Language
  Models with Task-Oriented Experts
Authors: name: Yen-Ting Lin, name: Chao-Han Huck Yang, name: Zhehuai Chen, name: Piotr Zelasko, name: Xuesong Yang, name: Zih-Ching Chen, name: Krishna C Puvvada, name: Szu-Wei Fu, name: Ke Hu, name: Jun Wei Chiu, name: Jagadeesh Balam, name: Boris Ginsburg, name: Yu-Chiang Frank Wang
Published: 2024-11-08 20:11:24

Construction of a general-purpose post-recognition error corrector poses a
crucial question: how can we most effectively train a model on a large mixture
of domain datasets? The answer would lie in learning dataset-specific features
and digesting their knowledge in a single model. Previous methods achieve this
by having separate correction language models, resulting in a significant
increase in parameters. In this work, we present Mixture-of-Experts as a
solution, highlighting that MoEs are much more than a scalability tool. We
propose a Multi-Task Correction MoE, where we train the experts to become an
``expert'' of speech-to-text, language-to-text and vision-to-text datasets by
learning to route each dataset's tokens to its mapped expert. Experiments on
the Open ASR Leaderboard show that we explore a new state-of-the-art
performance by achieving an average relative 5.0% WER reduction and
substantial improvements in BLEU scores for speech and translation tasks. On
zero-shot evaluation, NeKo outperforms GPT-3.5 and Claude-Opus with 15.5% to
27.6% relative WER reduction in the Hyporadise benchmark. NeKo performs
competitively on grammar and post-OCR correction as a multi-task model.
Watermark Anything with Localized Messages
Authors: name: Tom Sander, name: Pierre Fernandez, name: Alain Durmus, name: Teddy Furon, name: Matthijs Douze
Published: 2024-11-11 18:49:58

Image watermarking methods are not tailored to handle small watermarked
areas. This restricts applications in real-world scenarios where parts of the
image may come from different sources or have been edited. We introduce a
deep-learning model for localized image watermarking, dubbed the Watermark
Anything Model (WAM). The WAM embedder imperceptibly modifies the input image,
while the extractor segments the received image into watermarked and
non-watermarked areas and recovers one or several hidden messages from the
areas found to be watermarked. The models are jointly trained at low resolution
and without perceptual constraints, then post-trained for imperceptibility and
multiple watermarks. Experiments show that WAM is competitive with state-of-the
art methods in terms of imperceptibility and robustness, especially against
inpainting and splicing, even on high-resolution images. Moreover, it offers
new capabilities: WAM can locate watermarked areas in spliced images and
extract distinct 32-bit messages with less than 1 bit error from multiple small
regions - no larger than 10% of the image surface - even for small 256times
256 images.