Orionnnn/hub-stats huggingface dataset:
Large Language Model Unlearning via Embedding-Corrupted Prompts
Authors: name: Chris Yuhao Liu, name: Yaxuan Wang, name: Jeffrey Flanigan, name: Yang Liu
Published: 2024-06-12 06:56:20

Large language models (LLMs) have advanced to encompass extensive knowledge
across diverse domains. Yet controlling what a large language model should not
know is important for ensuring alignment and thus safe use. However, accurately
and efficiently unlearning knowledge from an LLM remains challenging due to the
potential collateral damage caused by the fuzzy boundary between retention and
forgetting, and the large computational requirements for optimization across
state-of-the-art models with hundreds of billions of parameters. In this work,
we present Embedding-COrrupted (ECO) Prompts, a lightweight unlearning
framework for large language models to address both the challenges of knowledge
entanglement and unlearning efficiency. Instead of relying on the LLM itself to
unlearn, we enforce an unlearned state during inference by employing a prompt
classifier to identify and safeguard prompts to forget. We learn corruptions
added to prompt embeddings via zeroth order optimization toward the unlearning
objective offline and corrupt prompts flagged by the classifier during
inference. We find that these embedding-corrupted prompts not only lead to
desirable outputs that satisfy the unlearning objective but also closely
approximate the output from a model that has never been trained on the data
intended for forgetting. Through extensive experiments on unlearning, we
demonstrate the superiority of our method in achieving promising unlearning at
nearly zero side effects in general domains and domains closely related to the
unlearned ones. Additionally, we highlight the scalability of our method to 100
LLMs, ranging from 0.5B to 236B parameters, incurring no additional cost as the
number of parameters increases.
Are We Done with MMLU?
Authors: name: Aryo Pradipta Gema, name: Joshua Ong Jun Leang, name: Giwon Hong, name: Alessio Devoto, name: Alberto Carlo Maria Mancino, name: Rohit Saxena, name: Xuanli He, name: Yu Zhao, name: Xiaotang Du, name: Mohammad Reza Ghasemi Madani, name: Claire Barale, name: Robert McHardy, name: Joshua Harris, name: Jean Kaddour, name: Emile van Krieken, name: Pasquale Minervini
Published: 2024-06-06 14:49:06

Maybe not. We identify and analyse errors in the popular Massive Multitask
Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
our analysis demonstrates numerous ground truth errors that obscure the true
capabilities of LLMs. For example, we find that 57% of the analysed questions
in the Virology subset contain errors. To address this issue, we introduce a
comprehensive framework for identifying dataset errors using a novel error
taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
demonstrate significant discrepancies with the model performance metrics that
were originally reported. Our results strongly advocate for revising MMLU's
error-ridden questions to enhance its future utility and reliability as a
benchmark. Therefore, we open up MMLU-Redux for additional annotation
https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.
Hibou: A Family of Foundational Vision Transformers for Pathology
Authors: name: Dmitry Nechaev, name: Alexey Pchelnikov, name: Ekaterina Ivanova
Published: 2024-06-07 16:45:53

Pathology, the microscopic examination of diseased tissue, is critical for
diagnosing various medical conditions, particularly cancers. Traditional
methods are labor-intensive and prone to human error. Digital pathology, which
converts glass slides into high-resolution digital images for analysis by
computer algorithms, revolutionizes the field by enhancing diagnostic accuracy,
consistency, and efficiency through automated image analysis and large-scale
data processing. Foundational transformer pretraining is crucial for developing
robust, generalizable models as it enables learning from vast amounts of
unannotated data.
  This paper introduces the Hibou family of foundational vision transformers
for pathology, leveraging the DINOv2 framework to pretrain two model variants,
Hibou-B and Hibou-L, on a proprietary dataset of over 1 million whole slide
images (WSIs) representing diverse tissue types and staining techniques. Our
pretrained models demonstrate superior performance on both patch-level and
slide-level benchmarks, surpassing existing state-of-the-art methods. Notably,
Hibou-L achieves the highest average accuracy across multiple benchmark
datasets. To support further research and application in the field, we have
open-sourced the Hibou-B model, which can be accessed at
https://github.com/HistAI/hibou
Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models
Authors: name: Yi-Fan Zhang, name: Qingsong Wen, name: Chaoyou Fu, name: Xue Wang, name: Zhang Zhang, name: Liang Wang, name: Rong Jin
Published: 2024-06-12 17:59:49

Seeing clearly with high resolution is a foundation of Large Multimodal
Models (LMMs), which has been proven to be vital for visual perception and
reasoning. Existing works usually employ a straightforward resolution upscaling
method, where the image consists of global and local branches, with the latter
being the sliced image patches but resized to the same resolution as the
former. This means that higher resolution requires more local patches,
resulting in exorbitant computational expenses, and meanwhile, the dominance of
local image tokens may diminish the global context. In this paper, we dive into
the problems and propose a new framework as well as an elaborate optimization
strategy. Specifically, we extract contextual information from the global view
using a mixture of adapters, based on the observation that different adapters
excel at different tasks. With regard to local patches, learnable query
embeddings are introduced to reduce image tokens, the most important tokens
accounting for the user question will be further selected by a similarity-based
selector. Our empirical results demonstrate a `less is more' pattern, where
utilizing fewer but more informative local image tokens leads to
improved performance. Besides, a significant challenge lies in the training
strategy, as simultaneous end-to-end training of the global mining block and
local compression block does not yield optimal results. We thus advocate for an
alternating training way, ensuring balanced learning between global and local
aspects. Finally, we also introduce a challenging dataset with high
requirements for image detail, enhancing the training of the local compression
layer. The proposed method, termed LMM with Sophisticated Tasks, Local image
compression, and Mixture of global Experts (SliME), achieves leading
performance across various benchmarks with only 2 million training data.
Physics3D: Learning Physical Properties of 3D Gaussians via Video
  Diffusion
Authors: name: Fangfu Liu, name: Hanyang Wang, name: Shunyu Yao, name: Shengjun Zhang, name: Jie Zhou, name: Yueqi Duan
Published: 2024-06-06 17:59:47

In recent years, there has been rapid development in 3D generation models,
opening up new possibilities for applications such as simulating the dynamic
movements of 3D objects and customizing their behaviors. However, current 3D
generative models tend to focus only on surface features such as color and
shape, neglecting the inherent physical properties that govern the behavior of
objects in the real world. To accurately simulate physics-aligned dynamics, it
is essential to predict the physical properties of materials and incorporate
them into the behavior prediction process. Nonetheless, predicting the diverse
materials of real-world objects is still challenging due to the complex nature
of their physical attributes. In this paper, we propose Physics3D, a
novel method for learning various physical properties of 3D objects through a
video diffusion model. Our approach involves designing a highly generalizable
physical simulation system based on a viscoelastic material model, which
enables us to simulate a wide range of materials with high-fidelity
capabilities. Moreover, we distill the physical priors from a video diffusion
model that contains more understanding of realistic object materials. Extensive
experiments demonstrate the effectiveness of our method with both elastic and
plastic materials. Physics3D shows great potential for bridging the gap between
the physical world and virtual neural space, providing a better integration and
application of realistic physical principles in virtual environments. Project
page: https://liuff19.github.io/Physics3D.
PowerInfer-2: Fast Large Language Model Inference on a Smartphone
Authors: name: Zhenliang Xue, name: Yixin Song, name: Zeyu Mi, name: Le Chen, name: Yubin Xia, name: Haibo Chen
Published: 2024-06-10 14:01:21

This paper introduces PowerInfer-2, a framework designed for high-speed
inference of Large Language Models (LLMs) on smartphones, particularly
effective for models whose sizes exceed the device's memory capacity. The key
insight of PowerInfer-2 is to utilize the heterogeneous computation, memory,
and I/O resources in smartphones by decomposing traditional matrix computations
into fine-grained neuron cluster computations. Specifically, PowerInfer-2
features a polymorphic neuron engine that adapts computational strategies for
various stages of LLM inference. Additionally, it introduces segmented neuron
caching and fine-grained neuron-cluster-level pipelining, which effectively
minimize and conceal the overhead caused by I/O operations. The implementation
and evaluation of PowerInfer-2 demonstrate its capability to support a wide
array of LLM models on two smartphones, achieving up to a 29.2x speed increase
compared with state-of-the-art frameworks. Notably, PowerInfer-2 is the first
system to serve the TurboSparse-Mixtral-47B model with a generation rate of
11.68 tokens per second on a smartphone. For models that fit entirely within
the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory
usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM.
For more details, including a demonstration video, please visit the project
site at www.powerinfer.ai/v2.
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated
  Parameters
Authors: name: Yixin Song, name: Haotong Xie, name: Zhengyan Zhang, name: Bo Wen, name: Li Ma, name: Zeyu Mi, name: Haibo Chen
Published: 2024-06-10 01:21:59

Exploiting activation sparsity is a promising approach to significantly
accelerating the inference process of large language models (LLMs) without
compromising performance. However, activation sparsity is determined by
activation functions, and commonly used ones like SwiGLU and GeGLU exhibit
limited sparsity. Simply replacing these functions with ReLU fails to achieve
sufficient sparsity. Moreover, inadequate training data can further increase
the risk of performance degradation. To address these challenges, we propose a
novel dReLU function, which is designed to improve LLM activation sparsity,
along with a high-quality training data mixture ratio to facilitate effective
sparsification. Additionally, we leverage sparse activation patterns within the
Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to
further boost efficiency. By applying our neuron sparsification method to the
Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are
activated per inference iteration, respectively, while achieving even more
powerful model performance. Evaluation results demonstrate that this sparsity
achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our
TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
Our models are available at https://huggingface.co/PowerInfer
Discovering Preference Optimization Algorithms with and for Large
  Language Models
Authors: name: Chris Lu, name: Samuel Holt, name: Claudio Fanconi, name: Alex J. Chan, name: Jakob Foerster, name: Mihaela van der Schaar, name: Robert Tjarko Lange
Published: 2024-06-12 16:58:41

Offline preference optimization is a key method for enhancing and controlling
the quality of Large Language Model (LLM) outputs. Typically, preference
optimization is approached as an offline supervised learning task using
manually-crafted convex loss functions. While these methods are based on
theoretical insights, they are inherently constrained by human creativity, so
the large search space of possible loss functions remains under explored. We
address this by performing LLM-driven objective discovery to automatically
discover new state-of-the-art preference optimization algorithms without
(expert) human intervention. Specifically, we iteratively prompt an LLM to
propose and implement new preference optimization loss functions based on
previously-evaluated performance metrics. This process leads to the discovery
of previously-unknown and performant preference optimization algorithms. The
best performing of these we call Discovered Preference Optimization (DiscoPOP),
a novel algorithm that adaptively blends logistic and exponential losses.
Experiments demonstrate the state-of-the-art performance of DiscoPOP and its
successful transfer to held-out tasks.
Chimera: Effectively Modeling Multivariate Time Series with
  2-Dimensional State Space Models
Authors: name: Ali Behrouz, name: Michele Santacatterina, name: Ramin Zabih
Published: 2024-06-06 17:58:09

Modeling multivariate time series is a well-established problem with a wide
range of applications from healthcare to financial markets. Traditional State
Space Models (SSMs) are classical approaches for univariate time series
modeling due to their simplicity and expressive power to represent linear
dependencies. They, however, have fundamentally limited expressive power to
capture non-linear dependencies, are slow in practice, and fail to model the
inter-variate information flow. Despite recent attempts to improve the
expressive power of SSMs by using deep structured SSMs, the existing methods
are either limited to univariate time series, fail to model complex patterns
(e.g., seasonal patterns), fail to dynamically model the dependencies of
variate and time dimensions, and/or are input-independent. We present Chimera
that uses two input-dependent 2-D SSM heads with different discretization
processes to learn long-term progression and seasonal patterns. To improve the
efficiency of complex 2D recurrence, we present a fast training using a new
2-dimensional parallel selective scan. We further present and discuss
2-dimensional Mamba and Mamba-2 as the spacial cases of our 2D SSM. Our
experimental evaluation shows the superior performance of Chimera on extensive
and diverse benchmarks, including ECG and speech time series classification,
long-term and short-term time series forecasting, and time series anomaly
detection.
MotionClone: Training-Free Motion Cloning for Controllable Video
  Generation
Authors: name: Pengyang Ling, name: Jiazi Bu, name: Pan Zhang, name: Xiaoyi Dong, name: Yuhang Zang, name: Tong Wu, name: Huaian Chen, name: Jiaqi Wang, name: Yi Jin
Published: 2024-06-08 03:44:25

Motion-based controllable text-to-video generation involves motions to
control the video generation. Previous methods typically require the training
of models to encode motion cues or the fine-tuning of video diffusion models.
However, these approaches often result in suboptimal motion generation when
applied outside the trained domain. In this work, we propose MotionClone, a
training-free framework that enables motion cloning from a reference video to
control text-to-video generation. We employ temporal attention in video
inversion to represent the motions in the reference video and introduce primary
temporal-attention guidance to mitigate the influence of noisy or very subtle
motions within the attention weights. Furthermore, to assist the generation
model in synthesizing reasonable spatial relationships and enhance its
prompt-following capability, we propose a location-aware semantic guidance
mechanism that leverages the coarse location of the foreground from the
reference video and original classifier-free guidance features to guide the
video generation. Extensive experiments demonstrate that MotionClone exhibits
proficiency in both global camera motion and local object motion, with notable
superiority in terms of motion fidelity, textual alignment, and temporal
consistency.
NaRCan: Natural Refined Canonical Image with Integration of Diffusion
  Prior for Video Editing
Authors: name: Ting-Hsuan Chen, name: Jiewen Chan, name: Hau-Shiang Shiu, name: Shih-Han Yen, name: Chang-Han Yeh, name: Yu-Lun Liu
Published: 2024-06-10 17:59:46

We propose a video editing framework, NaRCan, which integrates a hybrid
deformation field and diffusion prior to generate high-quality natural
canonical images to represent the input video. Our approach utilizes homography
to model global motion and employs multi-layer perceptrons (MLPs) to capture
local residual deformations, enhancing the model's ability to handle complex
video dynamics. By introducing a diffusion prior from the early stages of
training, our model ensures that the generated images retain a high-quality
natural appearance, making the produced canonical images suitable for various
downstream tasks in video editing, a capability not achieved by current
canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA)
fine-tuning and introduce a noise and diffusion prior update scheduling
technique that accelerates the training process by 14 times. Extensive
experimental results show that our method outperforms existing approaches in
various video editing tasks and produces coherent and high-quality edited video
sequences. See our project page for video results at
https://koi953215.github.io/NaRCan_page/.
FontStudio: Shape-Adaptive Diffusion Model for Coherent and Consistent
  Font Effect Generation
Authors: name: Xinzhi Mu, name: Li Chen, name: Bohan Chen, name: Shuyang Gu, name: Jianmin Bao, name: Dong Chen, name: Ji Li, name: Yuhui Yuan
Published: 2024-06-12 16:43:47

Recently, the application of modern diffusion-based text-to-image generation
models for creating artistic fonts, traditionally the domain of professional
designers, has garnered significant interest. Diverging from the majority of
existing studies that concentrate on generating artistic typography, our
research aims to tackle a novel and more demanding challenge: the generation of
text effects for multilingual fonts. This task essentially requires generating
coherent and consistent visual content within the confines of a font-shaped
canvas, as opposed to a traditional rectangular canvas. To address this task,
we introduce a novel shape-adaptive diffusion model capable of interpreting the
given shape and strategically planning pixel distributions within the irregular
canvas. To achieve this, we curate a high-quality shape-adaptive image-text
dataset and incorporate the segmentation mask as a visual condition to steer
the image generation process within the irregular-canvas. This approach enables
the traditionally rectangle canvas-based diffusion model to produce the desired
concepts in accordance with the provided geometric shapes. Second, to maintain
consistency across multiple letters, we also present a training-free,
shape-adaptive effect transfer method for transferring textures from a
generated reference letter to others. The key insights are building a font
effect noise prior and propagating the font effect information in a
concatenated latent space. The efficacy of our FontStudio system is confirmed
through user preference studies, which show a marked preference (78% win-rates
on aesthetics) for our system even when compared to the latest unrivaled
commercial product, Adobe Firefly.
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
  Less Hallucination
Authors: name: Jianing Yang, name: Xuweiyi Chen, name: Nikhil Madaan, name: Madhavan Iyengar, name: Shengyi Qian, name: David F. Fouhey, name: Joyce Chai
Published: 2024-06-07 17:59:59

The integration of language and 3D perception is crucial for developing
embodied agents and robots that comprehend and interact with the physical
world. While large language models (LLMs) have demonstrated impressive language
understanding and generation capabilities, their adaptation to 3D environments
(3D-LLMs) remains in its early stages. A primary challenge is the absence of
large-scale datasets that provide dense grounding between language and 3D
scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
comprising 40,087 household scenes paired with 6.2 million densely-grounded
scene-language instructions. Our results show that instruction tuning with
3D-GRAND significantly enhances grounding capabilities and reduces
hallucinations in 3D-LLMs. As part of our contributions, we propose a
comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
3D-LLMs, enabling fair comparisons among future models. Our experiments
highlight a scaling effect between dataset size and 3D-LLM performance,
emphasizing the critical role of large-scale 3D-text datasets in advancing
embodied AI research. Notably, our results demonstrate early signals for
effective sim-to-real transfer, indicating that models trained on large
synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
3D-POPE, we aim to equip the embodied AI community with essential resources and
insights, setting the stage for more reliable and better-grounded 3D-LLMs.
Project website: https://3d-grand.github.io
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio
  Understanding in Video-LLMs
Authors: name: Zesen Cheng, name: Sicong Leng, name: Hang Zhang, name: Yifei Xin, name: Xin Li, name: Guanzheng Chen, name: Yongxin Zhu, name: Wenqi Zhang, name: Ziyang Luo, name: Deli Zhao, name: Lidong Bing
Published: 2024-06-11 17:22:23

In this paper, we present the VideoLLaMA 2, a set of Video Large Language
Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio
understanding in video and audio-oriented tasks. Building upon its predecessor,
VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC)
connector, which effectively captures the intricate spatial and temporal
dynamics of video data. Additionally, we integrate an Audio Branch into the
model through joint training, thereby enriching the multimodal understanding
capabilities of the model by seamlessly incorporating audio cues. Comprehensive
evaluations on multiple-choice video question answering (MC-VQA), open-ended
video question answering (OE-VQA), and video captioning (VC) tasks demonstrate
that VideoLLaMA 2 consistently achieves competitive results among open-source
models and even gets close to some proprietary models on several benchmarks.
Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and
audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models.
These advancements underline VideoLLaMA 2's superior performance in multimodal
comprehension, setting a new standard for intelligent video analysis systems.
All models are public to facilitate further research.