Orionnnn/hub-stats huggingface dataset:
Learning to Refuse: Towards Mitigating Privacy Risks in LLMs
Authors: name: Zhenhua Liu, name: Tong Zhu, name: Chuanyuan Tan, name: Wenliang Chen
Published: 2024-07-14 03:05:53

Large language models (LLMs) exhibit remarkable capabilities in understanding
and generating natural language. However, these models can inadvertently
memorize private information, posing significant privacy risks. This study
addresses the challenge of enabling LLMs to protect specific individuals'
private data without the need for complete retraining. We propose \return, a
Real-world pErsonal daTa UnleaRNing dataset, comprising 2,492 individuals from
Wikipedia with associated QA pairs, to evaluate machine unlearning (MU) methods
for protecting personal data in a realistic scenario. Additionally, we
introduce the Name-Aware Unlearning Framework (NAUF) for Privacy Protection,
which enables the model to learn which individuals' information should be
protected without affecting its ability to answer questions related to other
unrelated individuals. Our extensive experiments demonstrate that NAUF achieves
a state-of-the-art average unlearning score, surpassing the best baseline
method by 5.65 points, effectively protecting target individuals' personal data
while maintaining the model's general capabilities.
GRUtopia: Dream General Robots in a City at Scale
Authors: name: Hanqing Wang, name: Jiahe Chen, name: Wensi Huang, name: Qingwei Ben, name: Tai Wang, name: Boyu Mi, name: Tao Huang, name: Siheng Zhao, name: Yilun Chen, name: Sizhe Yang, name: Peizhou Cao, name: Wenye Yu, name: Zichao Ye, name: Jialun Li, name: Junfeng Long, name: Zirui Wang, name: Huiling Wang, name: Ying Zhao, name: Zhongying Tu, name: Yu Qiao, name: Dahua Lin, name: Jiangmiao Pang
Published: 2024-07-15 17:40:46

Recent works have been exploring the scaling laws in the field of Embodied
AI. Given the prohibitive costs of collecting real-world data, we believe the
Simulation-to-Real (Sim2Real) paradigm is a crucial step for scaling the
learning of embodied models. This paper introduces project GRUtopia, the first
simulated interactive 3D society designed for various robots. It features
several advancements: (a) The scene dataset, GRScenes, includes 100k
interactive, finely annotated scenes, which can be freely combined into
city-scale environments. In contrast to previous works mainly focusing on home,
GRScenes covers 89 diverse scene categories, bridging the gap of
service-oriented environments where general robots would be initially deployed.
(b) GRResidents, a Large Language Model (LLM) driven Non-Player Character (NPC)
system that is responsible for social interaction, task generation, and task
assignment, thus simulating social scenarios for embodied AI applications. (c)
The benchmark, GRBench, supports various robots but focuses on legged robots as
primary agents and poses moderately challenging tasks involving Object
Loco-Navigation, Social Loco-Navigation, and Loco-Manipulation. We hope that
this work can alleviate the scarcity of high-quality data in this field and
provide a more comprehensive assessment of Embodied AI research. The project is
available at https://github.com/OpenRobotLab/GRUtopia.
Masked Generative Video-to-Audio Transformers with Enhanced
  Synchronicity
Authors: name: Santiago Pascual, name: Chunghsin Yeh, name: Ioannis Tsiamas, name: Joan Serrà
Published: 2024-07-15 01:49:59

Video-to-audio (V2A) generation leverages visual-only video features to
render plausible sounds that match the scene. Importantly, the generated sound
onsets should match the visual actions that are aligned with them, otherwise
unnatural synchronization artifacts arise. Recent works have explored the
progression of conditioning sound generators on still images and then video
features, focusing on quality and semantic matching while ignoring
synchronization, or by sacrificing some amount of quality to focus on improving
synchronization only. In this work, we propose a V2A generative model, named
MaskVAT, that interconnects a full-band high-quality general audio codec with a
sequence-to-sequence masked generative model. This combination allows modeling
both high audio quality, semantic matching, and temporal synchronicity at the
same time. Our results show that, by combining a high-quality codec with the
proper pre-trained audio-visual features and a sequence-to-sequence parallel
structure, we are able to yield highly synchronized results on one hand, whilst
being competitive with the state of the art of non-codec generative audio
models. Sample videos and generated audios are available at
https://maskvat.github.io .
LAB-Bench: Measuring Capabilities of Language Models for Biology
  Research
Authors: name: Jon M. Laurent, name: Joseph D. Janizek, name: Michael Ruzo, name: Michaela M. Hinks, name: Michael J. Hammerling, name: Siddharth Narayanan, name: Manvitha Ponnapati, name: Andrew D. White, name: Samuel G. Rodriques
Published: 2024-07-14 23:52:25

There is widespread optimism that frontier Large Language Models (LLMs) and
LLM-augmented systems have the potential to rapidly accelerate scientific
discovery across disciplines. Today, many benchmarks exist to measure LLM
knowledge and reasoning on textbook-style science questions, but few if any
benchmarks are designed to evaluate language model performance on practical
tasks required for scientific research, such as literature search, protocol
planning, and data analysis. As a step toward building such benchmarks, we
introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of
over 2,400 multiple choice questions for evaluating AI systems on a range of
practical biology research capabilities, including recall and reasoning over
literature, interpretation of figures, access and navigation of databases, and
comprehension and manipulation of DNA and protein sequences. Importantly, in
contrast to previous scientific benchmarks, we expect that an AI system that
can achieve consistently high scores on the more difficult LAB-Bench tasks
would serve as a useful assistant for researchers in areas such as literature
search and molecular cloning. As an initial assessment of the emergent
scientific task capabilities of frontier language models, we measure
performance of several against our benchmark and report results compared to
human expert biology researchers. We will continue to update and expand
LAB-Bench over time, and expect it to serve as a useful tool in the development
of automated research systems going forward. A public subset of LAB-Bench is
available for use at the following URL:
https://huggingface.co/datasets/futurehouse/lab-bench
Make-An-Agent: A Generalizable Policy Network Generator with
  Behavior-Prompted Diffusion
Authors: name: Yongyuan Liang, name: Tingqiang Xu, name: Kaizhe Hu, name: Guangqi Jiang, name: Furong Huang, name: Huazhe Xu
Published: 2024-07-15 17:59:57

Can we generate a control policy for an agent using just one demonstration of
desired behaviors as a prompt, as effortlessly as creating an image from a
textual description? In this paper, we present Make-An-Agent, a novel policy
parameter generator that leverages the power of conditional diffusion models
for behavior-to-policy generation. Guided by behavior embeddings that encode
trajectory information, our policy generator synthesizes latent parameter
representations, which can then be decoded into policy networks. Trained on
policy network checkpoints and their corresponding trajectories, our generation
model demonstrates remarkable versatility and scalability on multiple tasks and
has a strong generalization ability on unseen tasks to output well-performed
policies with only few-shot demonstrations as inputs. We showcase its efficacy
and efficiency on various domains and tasks, including varying objectives,
behaviors, and even across different robot manipulators. Beyond simulation, we
directly deploy policies generated by Make-An-Agent onto real-world robots on
locomotion tasks.
Noise Calibration: Plug-and-play Content-Preserving Video Enhancement
  using Pre-trained Video Diffusion Models
Authors: name: Qinyu Yang, name: Haoxin Chen, name: Yong Zhang, name: Menghan Xia, name: Xiaodong Cun, name: Zhixun Su, name: Ying Shan
Published: 2024-07-14 17:59:56

In order to improve the quality of synthesized videos, currently, one
predominant method involves retraining an expert diffusion model and then
implementing a noising-denoising process for refinement. Despite the
significant training costs, maintaining consistency of content between the
original and enhanced videos remains a major challenge. To tackle this
challenge, we propose a novel formulation that considers both visual quality
and consistency of content. Consistency of content is ensured by a proposed
loss function that maintains the structure of the input, while visual quality
is improved by utilizing the denoising process of pretrained diffusion models.
To address the formulated optimization problem, we have developed a
plug-and-play noise optimization strategy, referred to as Noise Calibration. By
refining the initial random noise through a few iterations, the content of
original video can be largely preserved, and the enhancement effect
demonstrates a notable improvement. Extensive experiments have demonstrated the
effectiveness of the proposed method.
SHERL: Synthesizing High Accuracy and Efficient Memory for
  Resource-Limited Transfer Learning
Authors: name: Haiwen Diao, name: Bo Wan, name: Xu Jia, name: Yunzhi Zhuge, name: Ying Zhang, name: Huchuan Lu, name: Long Chen
Published: 2024-07-10 10:22:35

Parameter-efficient transfer learning (PETL) has emerged as a flourishing
research field for adapting large pre-trained models to downstream tasks,
greatly reducing trainable parameters while grappling with memory challenges
during fine-tuning. To address it, memory-efficient series (METL) avoid
backpropagating gradients through the large backbone. However, they compromise
by exclusively relying on frozen intermediate outputs and limiting the
exhaustive exploration of prior knowledge from pre-trained models. Moreover,
the dependency and redundancy between cross-layer features are frequently
overlooked, thereby submerging more discriminative representations and causing
an inherent performance gap (vs. conventional PETL methods). Hence, we propose
an innovative METL strategy called SHERL for resource-limited scenarios to
decouple the entire adaptation into two successive and complementary processes.
In the early route, intermediate outputs are consolidated via an
anti-redundancy operation, enhancing their compatibility for subsequent
interactions; thereby in the late route, utilizing minimal late pre-trained
layers could alleviate the peak demand on memory overhead and regulate these
fairly flexible features into more adaptive and powerful representations for
new domains. Extensive ablations on vision-and-language and language-only tasks
show that SHERL combines the strengths of both parameter and memory-efficient
techniques, performing on-par or better across diverse architectures with lower
memory during fine-tuning. Our code is publicly available at:
https://github.com/Paranioar/SHERL.
Video Occupancy Models
Authors: name: Manan Tomar, name: Philippe Hansen-Estruch, name: Philip Bachman, name: Alex Lamb, name: John Langford, name: Matthew E. Taylor, name: Sergey Levine
Published: 2024-06-25 17:57:38

We introduce a new family of video prediction models designed to support
downstream control tasks. We call these models Video Occupancy models (VOCs).
VOCs operate in a compact latent space, thus avoiding the need to make
predictions about individual pixels. Unlike prior latent-space world models,
VOCs directly predict the discounted distribution of future states in a single
step, thus avoiding the need for multistep roll-outs. We show that both
properties are beneficial when building predictive models of video for use in
downstream control. Code is available at
https://github.com/manantomar/video-occupancy-models{github.com/manantomar/video-occupancy-models}.
Qwen2 Technical Report
Authors: name: An Yang, name: Baosong Yang, name: Binyuan Hui, name: Bo Zheng, name: Bowen Yu, name: Chang Zhou, name: Chengpeng Li, name: Chengyuan Li, name: Dayiheng Liu, name: Fei Huang, name: Guanting Dong, name: Haoran Wei, name: Huan Lin, name: Jialong Tang, name: Jialin Wang, name: Jian Yang, name: Jianhong Tu, name: Jianwei Zhang, name: Jianxin Ma, name: Jin Xu, name: Jingren Zhou, name: Jinze Bai, name: Jinzheng He, name: Junyang Lin, name: Kai Dang, name: Keming Lu, name: Keqin Chen, name: Kexin Yang, name: Mei Li, name: Mingfeng Xue, name: Na Ni, name: Pei Zhang, name: Peng Wang, name: Ru Peng, name: Rui Men, name: Ruize Gao, name: Runji Lin, name: Shijie Wang, name: Shuai Bai, name: Sinan Tan, name: Tianhang Zhu, name: Tianhao Li, name: Tianyu Liu, name: Wenbin Ge, name: Xiaodong Deng, name: Xiaohuan Zhou, name: Xingzhang Ren, name: Xinyu Zhang, name: Xipin Wei, name: Xuancheng Ren, name: Yang Fan, name: Yang Yao, name: Yichang Zhang, name: Yu Wan, name: Yunfei Chu, name: Zeyu Cui, name: Zhenru Zhang, name: Zhihao Fan
Published: 2024-07-15 12:35:42

This report introduces the Qwen2 series, the latest addition to our large
language models and large multimodal models. We release a comprehensive suite
of foundational and instruction-tuned language models, encompassing a parameter
range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts
model. Qwen2 surpasses most prior open-weight models, including its predecessor
Qwen1.5, and exhibits competitive performance relative to proprietary models
across diverse benchmarks on language understanding, generation, multilingual
proficiency, coding, mathematics, and reasoning.
  The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on
MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base
language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1
on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover, Qwen2
demonstrates robust multilingual capabilities, proficient in approximately 30
languages, spanning English, Chinese, Spanish, French, German, Arabic, Russian,
Korean, Japanese, Thai, Vietnamese, and more, underscoring its versatility and
global reach.
  To foster community innovation and accessibility, we have made the Qwen2
model weights openly available on Hugging Face1 and ModelScope2, and the
supplementary materials including example code on GitHub3. These platforms also
include resources for quantization, fine-tuning, and deployment, facilitating a
wide range of applications and research endeavors.
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore
  Non-Determinism
Authors: name: Yifan Song, name: Guoyin Wang, name: Sujian Li, name: Bill Yuchen Lin
Published: 2024-07-15 06:12:17

Current evaluations of large language models (LLMs) often overlook
non-determinism, typically focusing on a single output per example. This limits
our understanding of LLM performance variability in real-world applications.
Our study addresses this issue by exploring key questions about the performance
differences between greedy decoding and sampling, identifying benchmarks'
consistency regarding non-determinism, and examining unique model behaviors.
Through extensive experiments, we observe that greedy decoding generally
outperforms sampling methods for most evaluated tasks. We also observe
consistent performance across different LLM sizes and alignment methods, noting
that alignment can reduce sampling variance. Moreover, our best-of-N sampling
approach demonstrates that smaller LLMs can match or surpass larger models such
as GPT-4-Turbo, highlighting the untapped potential of smaller LLMs. This
research shows the importance of considering non-determinism in LLM evaluations
and provides insights for future LLM development and evaluation.
Speech Slytherin: Examining the Performance and Efficiency of Mamba for
  Speech Separation, Recognition, and Synthesis
Authors: name: Xilin Jiang, name: Yinghao Aaron Li, name: Adrian Nicolas Florea, name: Cong Han, name: Nima Mesgarani
Published: 2024-07-13 00:35:21

It is too early to conclude that Mamba is a better alternative to
transformers for speech before comparing Mamba with transformers in terms of
both performance and efficiency in multiple speech-related tasks. To reach this
conclusion, we propose and evaluate three models for three tasks: Mamba-TasNet
for speech separation, ConMamba for speech recognition, and VALL-M for speech
synthesis. We compare them with transformers of similar sizes in performance,
memory, and speed. Our Mamba or Mamba-transformer hybrid models show comparable
or higher performance than their transformer counterparts: Sepformer,
Conformer, and VALL-E. They are more efficient than transformers in memory and
speed for speech longer than a threshold duration, inversely related to the
resolution of a speech token. Mamba for separation is the most efficient, and
Mamba for recognition is the least. Further, we show that Mamba is not more
efficient than transformer for speech shorter than the threshold duration and
performs worse in models that require joint modeling of text and speech, such
as cross or masked attention of two inputs. Therefore, we argue that the
superiority of Mamba or transformer depends on particular problems and models.
Code available at https://github.com/xi-j/Mamba-TasNet and
https://github.com/xi-j/Mamba-ASR.
Toto: Time Series Optimized Transformer for Observability
Authors: name: Ben Cohen, name: Emaad Khwaja, name: Kan Wang, name: Charles Masson, name: Elise Ramé, name: Youssef Doubli, name: Othmane Abou-Amal
Published: 2024-07-10 17:40:30

This technical report describes the Time Series Optimized Transformer for
Observability (Toto), a new state of the art foundation model for time series
forecasting developed by Datadog. In addition to advancing the state of the art
on generalized time series benchmarks in domains such as electricity and
weather, this model is the first general-purpose time series forecasting
foundation model to be specifically tuned for observability metrics.
  Toto was trained on a dataset of one trillion time series data points, the
largest among all currently published time series foundation models. Alongside
publicly available time series datasets, 75% of the data used to train Toto
consists of fully anonymous numerical metric data points from the Datadog
platform.
  In our experiments, Toto outperforms existing time series foundation models
on observability data. It does this while also excelling at general-purpose
forecasting tasks, achieving state-of-the-art zero-shot performance on multiple
open benchmark datasets.
RRM: Relightable assets using Radiance guided Material extraction
Authors: name: Diego Gomez, name: Julien Philip, name: Adrien Kaiser, name: Élie Michel
Published: 2024-07-08 21:10:31

Synthesizing NeRFs under arbitrary lighting has become a seminal problem in
the last few years. Recent efforts tackle the problem via the extraction of
physically-based parameters that can then be rendered under arbitrary lighting,
but they are limited in the range of scenes they can handle, usually
mishandling glossy scenes. We propose RRM, a method that can extract the
materials, geometry, and environment lighting of a scene even in the presence
of highly reflective objects. Our method consists of a physically-aware
radiance field representation that informs physically-based parameters, and an
expressive environment light structure based on a Laplacian Pyramid. We
demonstrate that our contributions outperform the state-of-the-art on parameter
retrieval tasks, leading to high-fidelity relighting and novel view synthesis
on surfacic scenes.
Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled
  Refusal Training
Authors: name: Youliang Yuan, name: Wenxiang Jiao, name: Wenxuan Wang, name: Jen-tse Huang, name: Jiahao Xu, name: Tian Liang, name: Pinjia He, name: Zhaopeng Tu
Published: 2024-07-12 09:36:33

This study addresses a critical gap in safety tuning practices for Large
Language Models (LLMs) by identifying and tackling a refusal position bias
within safety tuning data, which compromises the models' ability to
appropriately refuse generating unsafe content. We introduce a novel approach,
Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse
compliance to harmful prompts at any response position, significantly enhancing
their safety capabilities. DeRTa incorporates two novel components: (1) Maximum
Likelihood Estimation (MLE) with Harmful Response Prefix, which trains models
to recognize and avoid unsafe content by appending a segment of harmful
response to the beginning of a safe response, and (2) Reinforced Transition
Optimization (RTO), which equips models with the ability to transition from
potential harm to safety refusal consistently throughout the harmful response
sequence. Our empirical evaluation, conducted using LLaMA3 and Mistral model
families across six attack scenarios, demonstrates that our method not only
improves model safety without compromising performance but also surpasses
well-known models such as GPT-4 in defending against attacks. Importantly, our
approach successfully defends recent advanced attack methods (e.g., CodeAttack)
that have jailbroken GPT-4 and LLaMA3-70B-Instruct. Our code and data can be
found at https://github.com/RobustNLP/DeRTa.
Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
Authors: name: Huanqian Wang, name: Yang Yue, name: Rui Lu, name: Jingxin Shi, name: Andrew Zhao, name: Shenzhi Wang, name: Shiji Song, name: Gao Huang
Published: 2024-07-11 17:52:03

Large Language Models (LLMs) have demonstrated great potential as generalist
assistants, showcasing powerful task understanding and problem-solving
capabilities. To deploy LLMs as AI assistants, it is crucial that these models
exhibit desirable behavioral traits, such as non-toxicity and resilience
against jailbreak attempts. Current methods for detoxification or preventing
jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
Learning from Human Feedback (RLHF), which requires finetuning billions of
parameters through gradient descent with substantial computation cost.
Furthermore, models modified through SFT and RLHF may deviate from the
pretrained models, potentially leading to a degradation in foundational LLM
capabilities. In this paper, we observe that surprisingly, directly editing a
small subset of parameters can effectively modulate specific behaviors of LLMs,
such as detoxification and resistance to jailbreaking. Specifically, for a
behavior that we aim to avoid, we employ a linear classifier, which we term the
behavior probe, to classify binary behavior labels within the hidden state
space of the LLM. Using this probe, we introduce an algorithm to identify a
critical subset of LLM parameters that significantly influence this targeted
behavior. Then we directly edit these selected parameters by shifting them
towards the behavior probe. Such a direct parameter editing method necessitates
only inference-level computational resources. Experiments demonstrate that in
the representative detoxification task, our approach achieves reductions of up
to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
while maintaining the LLM's general capabilities in areas such as common sense,
question answering, and mathematics. Our code is available at
https://github.com/lucywang720/model-surgery.