Orionnnn/hub-stats huggingface dataset:
SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion
  Models
Authors: name: Muyang Li, name: Yujun Lin, name: Zhekai Zhang, name: Tianle Cai, name: Xiuyu Li, name: Junxian Guo, name: Enze Xie, name: Chenlin Meng, name: Jun-Yan Zhu, name: Song Han
Published: 2024-11-07 18:59:58

Diffusion models have been proven highly effective at generating high-quality
images. However, as these models grow larger, they require significantly more
memory and suffer from higher latency, posing substantial challenges for
deployment. In this work, we aim to accelerate diffusion models by quantizing
their weights and activations to 4 bits. At such an aggressive level, both
weights and activations are highly sensitive, where conventional post-training
quantization methods for large language models like smoothing become
insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit
quantization paradigm. Different from smoothing which redistributes outliers
between weights and activations, our approach absorbs these outliers using a
low-rank branch. We first consolidate the outliers by shifting them from
activations to weights, then employ a high-precision low-rank branch to take in
the weight outliers with Singular Value Decomposition (SVD). This process eases
the quantization on both sides. However, na\"{\i}vely running the low-rank
branch independently incurs significant overhead due to extra data movement of
activations, negating the quantization speedup. To address this, we co-design
an inference engine Nunchaku that fuses the kernels of the low-rank branch into
those of the low-bit branch to cut off redundant memory access. It can also
seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for
re-quantization. Extensive experiments on SDXL, PixArt-Sigma, and FLUX.1
validate the effectiveness of SVDQuant in preserving image quality. We reduce
the memory usage for the 12B FLUX.1 models by 3.5times, achieving
3.0times speedup over the 4-bit weight-only quantized baseline on the 16GB
laptop 4090 GPU, paving the way for more interactive applications on PCs. Our
quantization library and inference engine are open-sourced.
GazeGen: Gaze-Driven User Interaction for Visual Content Generation
Authors: name: He-Yen Hsieh, name: Ziyun Li, name: Sai Qian Zhang, name: Wei-Te Mark Ting, name: Kao-Den Chang, name: Barbara De Salvo, name: Chiao Liu, name: H. T. Kung
Published: 2024-11-07 00:22:38

We present GazeGen, a user interaction system that generates visual content
(images and videos) for locations indicated by the user's eye gaze. GazeGen
allows intuitive manipulation of visual content by targeting regions of
interest with gaze. Using advanced techniques in object detection and
generative AI, GazeGen performs gaze-controlled image adding/deleting,
repositioning, and surface material changes of image objects, and converts
static images into videos. Central to GazeGen is the DFT Gaze (Distilled and
Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters,
performing accurate real-time gaze predictions tailored to individual users'
eyes on small edge devices. GazeGen is the first system to combine visual
content generation with real-time gaze estimation, made possible exclusively by
DFT Gaze. This real-time gaze estimation enables various visual content
generation tasks, all controlled by the user's gaze. The input for DFT Gaze is
the user's eye images, while the inputs for visual content generation are the
user's view and the predicted gaze point from DFT Gaze. To achieve efficient
gaze predictions, we derive the small model from a large model (10x larger) via
novel knowledge distillation and personal adaptation techniques. We integrate
knowledge distillation with a masked autoencoder, developing a compact yet
powerful gaze estimation model. This model is further fine-tuned with Adapters,
enabling highly accurate and personalized gaze predictions with minimal user
input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a
wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA
and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low
latency on the edge device (Raspberry Pi 4). Furthermore, we describe
applications of GazeGen, illustrating its versatility and effectiveness in
various usage scenarios.
Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large
  Language Model
Authors: name: Young-Jun Lee, name: Dokyong Lee, name: Junyoung Youn, name: Kyeongjin Oh, name: Ho-Jin Choi
Published: 2024-11-07 07:46:06

To increase social bonding with interlocutors, humans naturally acquire the
ability to respond appropriately in a given situation by considering which
conversational skill is most suitable for the response - a process we call
skill-of-mind. For large language model (LLM)-based conversational agents,
planning appropriate conversational skills, as humans do, is challenging due to
the complexity of social dialogue, especially in interactive scenarios. To
address this, we propose a skill-of-mind-annotated conversation dataset, named
Multifaceted Skill-of-Mind, which includes multi-turn and multifaceted
conversational skills across various interactive scenarios (e.g., long-term,
counseling, task-oriented), grounded in diverse social contexts (e.g.,
demographics, persona, rules of thumb). This dataset consists of roughly 100K
conversations. Using this dataset, we introduce a new family of
skill-of-mind-infused LLMs, named Thanos, with model sizes of 1B, 3B, and 8B
parameters. With extensive experiments, these models successfully demonstrate
the skill-of-mind process and exhibit strong generalizability in inferring
multifaceted skills across a variety of domains. Moreover, we show that Thanos
significantly enhances the quality of responses generated by LLM-based
conversational agents and promotes prosocial behavior in human evaluations.
RetrieveGPT: Merging Prompts and Mathematical Models for Enhanced
  Code-Mixed Information Retrieval
Authors: name: Aniket Deroy, name: Subhankar Maity
Published: 2024-11-07 14:41:01

Code-mixing, the integration of lexical and grammatical elements from
multiple languages within a single sentence, is a widespread linguistic
phenomenon, particularly prevalent in multilingual societies. In India, social
media users frequently engage in code-mixed conversations using the Roman
script, especially among migrant communities who form online groups to share
relevant local information. This paper focuses on the challenges of extracting
relevant information from code-mixed conversations, specifically within Roman
transliterated Bengali mixed with English. This study presents a novel approach
to address these challenges by developing a mechanism to automatically identify
the most relevant answers from code-mixed conversations. We have experimented
with a dataset comprising of queries and documents from Facebook, and Query
Relevance files (QRels) to aid in this task. Our results demonstrate the
effectiveness of our approach in extracting pertinent information from complex,
code-mixed digital conversations, contributing to the broader field of natural
language processing in multilingual and informal text environments. We use
GPT-3.5 Turbo via prompting alongwith using the sequential nature of relevant
documents to frame a mathematical model which helps to detect relevant
documents corresponding to a query.
DimensionX: Create Any 3D and 4D Scenes from a Single Image with
  Controllable Video Diffusion
Authors: name: Wenqiang Sun, name: Shuo Chen, name: Fangfu Liu, name: Zilong Chen, name: Yueqi Duan, name: Jun Zhang, name: Yikai Wang
Published: 2024-11-07 18:07:31

In this paper, we introduce DimensionX, a framework designed to
generate photorealistic 3D and 4D scenes from just a single image with video
diffusion. Our approach begins with the insight that both the spatial structure
of a 3D scene and the temporal evolution of a 4D scene can be effectively
represented through sequences of video frames. While recent video diffusion
models have shown remarkable success in producing vivid visuals, they face
limitations in directly recovering 3D/4D scenes due to limited spatial and
temporal controllability during generation. To overcome this, we propose
ST-Director, which decouples spatial and temporal factors in video diffusion by
learning dimension-aware LoRAs from dimension-variant data. This controllable
video diffusion approach enables precise manipulation of spatial structure and
temporal dynamics, allowing us to reconstruct both 3D and 4D representations
from sequential frames with the combination of spatial and temporal dimensions.
Additionally, to bridge the gap between generated videos and real-world scenes,
we introduce a trajectory-aware mechanism for 3D generation and an
identity-preserving denoising strategy for 4D generation. Extensive experiments
on various real-world and synthetic datasets demonstrate that DimensionX
achieves superior results in controllable video generation, as well as in 3D
and 4D scene generation, compared with previous methods.
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Authors: name: Siming Huang, name: Tianhao Cheng, name: Jason Klein Liu, name: Jiaran Hao, name: Liuyihan Song, name: Yang Xu, name: J. Yang, name: J. H. Liu, name: Chenchen Zhang, name: Linzheng Chai, name: Ruifeng Yuan, name: Zhaoxiang Zhang, name: Jie Fu, name: Qian Liu, name: Ge Zhang, name: Zili Wang, name: Yuan Qi, name: Yinghui Xu, name: Wei Chu
Published: 2024-11-07 17:47:25

Large language models (LLMs) for code have become indispensable in various
domains, including code generation, reasoning tasks and agent systems.While
open-access code LLMs are increasingly approaching the performance levels of
proprietary models, high-quality code LLMs suitable for rigorous scientific
investigation, particularly those with reproducible data processing pipelines
and transparent training protocols, remain limited. The scarcity is due to
various challenges, including resource constraints, ethical considerations, and
the competitive advantages of keeping models advanced. To address the gap, we
introduce OpenCoder, a top-tier code LLM that not only achieves performance
comparable to leading models but also serves as an ``open cookbook'' for the
research community. Unlike most prior efforts, we release not only model
weights and inference code, but also the reproducible training data, complete
data processing pipeline, rigorous experimental ablation results, and detailed
training protocols for open scientific research. Through this comprehensive
release, we identify the key ingredients for building a top-tier code LLM: (1)
code optimized heuristic rules for data cleaning and methods for data
deduplication, (2) recall of text corpus related to code and (3) high-quality
synthetic data in both annealing and supervised fine-tuning stages. By offering
this level of openness, we aim to broaden access to all aspects of a top-tier
code LLM, with OpenCoder serving as both a powerful model and an open
foundation to accelerate research, and enable reproducible advancements in code
AI.
Mixture-of-Transformers: A Sparse and Scalable Architecture for
  Multi-Modal Foundation Models
Authors: name: Weixin Liang, name: Lili Yu, name: Liang Luo, name: Srinivasan Iyer, name: Ning Dong, name: Chunting Zhou, name: Gargi Ghosh, name: Mike Lewis, name: Wen-tau Yih, name: Luke Zettlemoyer, name: Xi Victoria Lin
Published: 2024-11-07 18:59:06

The development of large language models (LLMs) has expanded to multi-modal
systems capable of processing text, images, and speech within a unified
framework. Training these models demands significantly larger datasets and
computational resources compared to text-only LLMs. To address the scaling
challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal
transformer architecture that significantly reduces pretraining computational
costs. MoT decouples non-embedding parameters of the model by modality --
including feed-forward networks, attention matrices, and layer normalization --
enabling modality-specific processing with global self-attention over the full
input sequence. We evaluate MoT across multiple settings and model scales. In
the Chameleon 7B setting (autoregressive text-and-image generation), MoT
matches the dense baseline's performance using only 55.8\% of the FLOPs. When
extended to include speech, MoT reaches speech performance comparable to the
dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where
text and image are trained with different objectives, a 7B MoT model matches
the image modality performance of the dense baseline with one third of the
FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image
generation metrics. System profiling further highlights MoT's practical
benefits, achieving dense baseline image quality in 47.2\% of the wall-clock
time and text quality in 75.6\% of the wall-clock time (measured on AWS
p4de.24xlarge instances with NVIDIA A100 GPUs).
SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Authors: name: Koichi Namekata, name: Sherwin Bahmani, name: Ziyi Wu, name: Yash Kant, name: Igor Gilitschenski, name: David B. Lindell
Published: 2024-11-07 18:56:11

Methods for image-to-video generation have achieved impressive,
photo-realistic quality. However, adjusting specific elements in generated
videos, such as object motion or camera movement, is often a tedious process of
trial and error, e.g., involving re-generating videos with different random
seeds. Recent techniques address this issue by fine-tuning a pre-trained model
to follow conditioning signals, such as bounding boxes or point trajectories.
Yet, this fine-tuning procedure can be computationally expensive, and it
requires datasets with annotated object motion, which can be difficult to
procure. In this work, we introduce SG-I2V, a framework for controllable
image-to-video generation that is self-guidedx2013offering
zero-shot control by relying solely on the knowledge present in a pre-trained
image-to-video diffusion model without the need for fine-tuning or external
knowledge. Our zero-shot method outperforms unsupervised baselines while being
competitive with supervised models in terms of visual quality and motion
fidelity.
DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile
  Manipulation
Authors: name: Peiqi Liu, name: Zhanqiu Guo, name: Mohit Warke, name: Soumith Chintala, name: Chris Paxton, name: Nur Muhammad Mahi Shafiullah, name: Lerrel Pinto
Published: 2024-11-07 18:59:27

Significant progress has been made in open-vocabulary mobile manipulation,
where the goal is for a robot to perform tasks in any environment given a
natural language description. However, most current systems assume a static
environment, which limits the system's applicability in real-world scenarios
where environments frequently change due to human intervention or the robot's
own actions. In this work, we present DynaMem, a new approach to open-world
mobile manipulation that uses a dynamic spatio-semantic memory to represent a
robot's environment. DynaMem constructs a 3D data structure to maintain a
dynamic memory of point clouds, and answers open-vocabulary object localization
queries using multimodal LLMs or open-vocabulary features generated by
state-of-the-art vision-language models. Powered by DynaMem, our robots can
explore novel environments, search for objects not found in memory, and
continuously update the memory as objects move, appear, or disappear in the
scene. We run extensive experiments on the Stretch SE3 robots in three real and
nine offline scenes, and achieve an average pick-and-drop success rate of 70%
on non-stationary objects, which is more than a 2x improvement over
state-of-the-art static systems. Our code as well as our experiment and
deployment videos are open sourced and can be found on our project website:
https://dynamem.github.io/
TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for
  Image-to-Video Generation
Authors: name: Wenhao Wang, name: Yi Yang
Published: 2024-11-05 18:52:43

Video generation models are revolutionizing content creation, with
image-to-video models drawing increasing attention due to their enhanced
controllability, visual consistency, and practical applications. However,
despite their popularity, these models rely on user-provided text and image
prompts, and there is currently no dedicated dataset for studying these
prompts. In this paper, we introduce TIP-I2V, the first large-scale dataset of
over 1.70 million unique user-provided Text and Image Prompts specifically for
Image-to-Video generation. Additionally, we provide the corresponding generated
videos from five state-of-the-art image-to-video models. We begin by outlining
the time-consuming and costly process of curating this large-scale dataset.
Next, we compare TIP-I2V to two popular prompt datasets, VidProM
(text-to-video) and DiffusionDB (text-to-image), highlighting differences in
both basic and semantic information. This dataset enables advancements in
image-to-video research. For instance, to develop better models, researchers
can use the prompts in TIP-I2V to analyze user preferences and evaluate the
multi-dimensional performance of their trained models; and to enhance model
safety, they may focus on addressing the misinformation issue caused by
image-to-video models. The new research inspired by TIP-I2V and the differences
with existing datasets emphasize the importance of a specialized image-to-video
prompt dataset. The project is publicly available at https://tip-i2v.github.io.
ReCapture: Generative Video Camera Controls for User-Provided Videos
  using Masked Video Fine-Tuning
Authors: name: David Junhao Zhang, name: Roni Paiss, name: Shiran Zada, name: Nikhil Karnad, name: David E. Jacobs, name: Yael Pritch, name: Inbar Mosseri, name: Mike Zheng Shou, name: Neal Wadhwa, name: Nataniel Ruiz
Published: 2024-11-07 18:59:45

Recently, breakthroughs in video modeling have allowed for controllable
camera trajectories in generated videos. However, these methods cannot be
directly applied to user-provided videos that are not generated by a video
model. In this paper, we present ReCapture, a method for generating new videos
with novel camera trajectories from a single user-provided video. Our method
allows us to re-generate the reference video, with all its existing scene
motion, from vastly different angles and with cinematic camera motion. Notably,
using our method we can also plausibly hallucinate parts of the scene that were
not observable in the reference video. Our method works by (1) generating a
noisy anchor video with a new camera trajectory using multiview diffusion
models or depth-based point cloud rendering and then (2) regenerating the
anchor video into a clean and temporally consistent reangled video using our
proposed masked video fine-tuning technique.
From Medprompt to o1: Exploration of Run-Time Strategies for Medical
  Challenge Problems and Beyond
Authors: name: Harsha Nori, name: Naoto Usuyama, name: Nicholas King, name: Scott Mayer McKinney, name: Xavier Fernandes, name: Sheng Zhang, name: Eric Horvitz
Published: 2024-11-06 01:09:17

Run-time steering strategies like Medprompt are valuable for guiding large
language models (LLMs) to top performance on challenging tasks. Medprompt
demonstrates that a general LLM can be focused to deliver state-of-the-art
performance on specialized domains like medicine by using a prompt to elicit a
run-time strategy involving chain of thought reasoning and ensembling. OpenAI's
o1-preview model represents a new paradigm, where a model is designed to do
run-time reasoning before generating final responses. We seek to understand the
behavior of o1-preview on a diverse set of medical challenge problem
benchmarks. Following on the Medprompt study with GPT-4, we systematically
evaluate the o1-preview model across various medical benchmarks. Notably, even
without prompting techniques, o1-preview largely outperforms the GPT-4 series
with Medprompt. We further systematically study the efficacy of classic prompt
engineering strategies, as represented by Medprompt, within the new paradigm of
reasoning models. We found that few-shot prompting hinders o1's performance,
suggesting that in-context learning may no longer be an effective steering
approach for reasoning-native models. While ensembling remains viable, it is
resource-intensive and requires careful cost-performance optimization. Our cost
and accuracy analysis across run-time strategies reveals a Pareto frontier,
with GPT-4o representing a more affordable option and o1-preview achieving
state-of-the-art performance at higher cost. Although o1-preview offers top
performance, GPT-4o with steering strategies like Medprompt retains value in
specific contexts. Moreover, we note that the o1-preview model has reached
near-saturation on many existing medical benchmarks, underscoring the need for
new, challenging benchmarks. We close with reflections on general directions
for inference-time computation with LLMs.
Polynomial Composition Activations: Unleashing the Dynamics of Large
  Language Models
Authors: name: Zhijian Zhuo, name: Ya Wang, name: Yutao Zeng, name: Xiaoqing Li, name: Xun Zhou, name: Jinwen Ma
Published: 2024-11-06 13:00:34

Transformers have found extensive applications across various domains due to
the powerful fitting capabilities. This success can be partially attributed to
their inherent nonlinearity. Thus, in addition to the ReLU function employed in
the original transformer architecture, researchers have explored alternative
modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment
representational capacity. In this paper, we propose a novel category of
polynomial composition activations (PolyCom), designed to optimize the dynamics
of transformers. Theoretically, we provide a comprehensive mathematical
analysis of PolyCom, highlighting its enhanced expressivity and efficacy
relative to other activation functions. Notably, we demonstrate that networks
incorporating PolyCom achieve the optimal approximation rate,
indicating that PolyCom networks require minimal parameters to approximate
general smooth functions in Sobolev spaces. We conduct empirical experiments on
the pre-training configurations of large language models (LLMs), including both
dense and sparse architectures. By substituting conventional activation
functions with PolyCom, we enable LLMs to capture higher-order interactions
within the data, thus improving performance metrics in terms of accuracy and
convergence rates. Extensive experimental results demonstrate the effectiveness
of our method, showing substantial improvements over other activation
functions. Code is available at https://github.com/BryceZhuo/PolyCom.
Self-Consistency Preference Optimization
Authors: name: Archiki Prasad, name: Weizhe Yuan, name: Richard Yuanzhe Pang, name: Jing Xu, name: Maryam Fazel-Zarandi, name: Mohit Bansal, name: Sainbayar Sukhbaatar, name: Jason Weston, name: Jane Yu
Published: 2024-11-06 18:36:22

Self-alignment, whereby models learn to improve themselves without human
annotation, is a rapidly growing research area. However, existing techniques
often fail to improve complex reasoning tasks due to the difficulty of
assigning correct rewards. An orthogonal approach that is known to improve
correctness is self-consistency, a method applied at inference time based on
multiple sampling in order to find the most consistent answer. In this work, we
extend the self-consistency concept to help train models. We thus introduce
self-consistency preference optimization (ScPO), which iteratively trains
consistent answers to be preferred over inconsistent ones on unsupervised new
problems. We show ScPO leads to large improvements over conventional reward
model training on reasoning tasks such as GSM8K and MATH, closing the gap with
supervised training with gold answers or preferences, and that combining ScPO
with standard supervised learning improves results even further. On ZebraLogic,
ScPO finetunes Llama-3 8B to be superior to Llama-3 70B, Gemma-2 27B, and
Claude-3 Haiku.