Orionnnn/hub-stats huggingface dataset:
The Instruction Hierarchy: Training LLMs to Prioritize Privileged
  Instructions
Authors: name: Eric Wallace, name: Kai Xiao, name: Reimar Leike, name: Lilian Weng, name: Johannes Heidecke, name: Alex Beutel
Published: 2024-04-19 22:55:23

Today's LLMs are susceptible to prompt injections, jailbreaks, and other
attacks that allow adversaries to overwrite a model's original instructions
with their own malicious prompts. In this work, we argue that one of the
primary vulnerabilities underlying these attacks is that LLMs often consider
system prompts (e.g., text from an application developer) to be the same
priority as text from untrusted users and third parties. To address this, we
propose an instruction hierarchy that explicitly defines how models should
behave when instructions of different priorities conflict. We then propose a
data generation method to demonstrate this hierarchical instruction following
behavior, which teaches LLMs to selectively ignore lower-privileged
instructions. We apply this method to GPT-3.5, showing that it drastically
increases robustness -- even for attack types not seen during training -- while
imposing minimal degradations on standard capabilities.
How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
Authors: name: Wei Huang, name: Xudong Ma, name: Haotong Qin, name: Xingyu Zheng, name: Chengtao Lv, name: Hong Chen, name: Jie Luo, name: Xiaojuan Qi, name: Xianglong Liu, name: Michele Magno
Published: 2024-04-22 10:03:03

Meta's LLaMA family has become one of the most powerful open-source Large
Language Model (LLM) series. Notably, LLaMA3 models have recently been released
and achieve impressive performance across various with super-large scale
pre-training on over 15T tokens of data. Given the wide application of low-bit
quantization for LLMs in resource-limited scenarios, we explore LLaMA3's
capabilities when quantized to low bit-width. This exploration holds the
potential to unveil new insights and challenges for low-bit quantization of
LLaMA3 and other forthcoming LLMs, especially in addressing performance
degradation problems that suffer in LLM compression. Specifically, we evaluate
the 10 existing post-training quantization and LoRA-finetuning methods of
LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's
low-bit quantization performance. Our experiment results indicate that LLaMA3
still suffers non-negligent degradation in these scenarios, especially in
ultra-low bit-width. This highlights the significant performance gap under low
bit-width that needs to be bridged in future developments. We expect that this
empirical study will prove valuable in advancing future models, pushing the
LLMs to lower bit-width with higher accuracy for being practical. Our project
is released on https://github.com/Macaronlin/LLaMA3-Quantization and quantized
LLaMA3 models are released in https://huggingface.co/LLMQ.
FlowMind: Automatic Workflow Generation with LLMs
Authors: name: Zhen Zeng, name: William Watson, name: Nicole Cho, name: Saba Rahimi, name: Shayleen Reynolds, name: Tucker Balch, name: Manuela Veloso
Published: 2024-03-17 00:36:37

The rapidly evolving field of Robotic Process Automation (RPA) has made
significant strides in automating repetitive processes, yet its effectiveness
diminishes in scenarios requiring spontaneous or unpredictable tasks demanded
by users. This paper introduces a novel approach, FlowMind, leveraging the
capabilities of Large Language Models (LLMs) such as Generative Pretrained
Transformer (GPT), to address this limitation and create an automatic workflow
generation system. In FlowMind, we propose a generic prompt recipe for a
lecture that helps ground LLM reasoning with reliable Application Programming
Interfaces (APIs). With this, FlowMind not only mitigates the common issue of
hallucinations in LLMs, but also eliminates direct interaction between LLMs and
proprietary data or code, thus ensuring the integrity and confidentiality of
information - a cornerstone in financial services. FlowMind further simplifies
user interaction by presenting high-level descriptions of auto-generated
workflows, enabling users to inspect and provide feedback effectively. We also
introduce NCEN-QA, a new dataset in finance for benchmarking question-answering
tasks from N-CEN reports on funds. We used NCEN-QA to evaluate the performance
of workflows generated by FlowMind against baseline and ablation variants of
FlowMind. We demonstrate the success of FlowMind, the importance of each
component in the proposed lecture recipe, and the effectiveness of user
interaction and feedback in FlowMind.
Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image
  Synthesis
Authors: name: Yuxi Ren, name: Xin Xia, name: Yanzuo Lu, name: Jiacheng Zhang, name: Jie Wu, name: Pan Xie, name: Xing Wang, name: Xuefeng Xiao
Published: 2024-04-21 15:16:05

Recently, a series of diffusion-aware distillation algorithms have emerged to
alleviate the computational overhead associated with the multi-step inference
process of Diffusion Models (DMs). Current distillation techniques often
dichotomize into two distinct aspects: i) ODE Trajectory Preservation; and ii)
ODE Trajectory Reformulation. However, these approaches suffer from severe
performance degradation or domain shifts. To address these limitations, we
propose Hyper-SD, a novel framework that synergistically amalgamates the
advantages of ODE Trajectory Preservation and Reformulation, while maintaining
near-lossless performance during step compression. Firstly, we introduce
Trajectory Segmented Consistency Distillation to progressively perform
consistent distillation within pre-defined time-step segments, which
facilitates the preservation of the original ODE trajectory from a higher-order
perspective. Secondly, we incorporate human feedback learning to boost the
performance of the model in a low-step regime and mitigate the performance loss
incurred by the distillation process. Thirdly, we integrate score distillation
to further improve the low-step generation capability of the model and offer
the first attempt to leverage a unified LoRA to support the inference process
at all steps. Extensive experiments and user studies demonstrate that Hyper-SD
achieves SOTA performance from 1 to 8 inference steps for both SDXL and SD1.5.
For example, Hyper-SDXL surpasses SDXL-Lightning by +0.68 in CLIP Score and
+0.51 in Aes Score in the 1-step inference.
A Multimodal Automated Interpretability Agent
Authors: name: Tamar Rott Shaham, name: Sarah Schwettmann, name: Franklin Wang, name: Achyuta Rajaram, name: Evan Hernandez, name: Jacob Andreas, name: Antonio Torralba
Published: 2024-04-22 17:55:11

This paper describes MAIA, a Multimodal Automated Interpretability Agent.
MAIA is a system that uses neural models to automate neural model understanding
tasks like feature interpretation and failure mode discovery. It equips a
pre-trained vision-language model with a set of tools that support iterative
experimentation on subcomponents of other models to explain their behavior.
These include tools commonly used by human interpretability researchers: for
synthesizing and editing inputs, computing maximally activating exemplars from
real-world datasets, and summarizing and describing experimental results.
Interpretability experiments proposed by MAIA compose these tools to describe
and explain system behavior. We evaluate applications of MAIA to computer
vision models. We first characterize MAIA's ability to describe (neuron-level)
features in learned representations of images. Across several trained models
and a novel dataset of synthetic vision neurons with paired ground-truth
descriptions, MAIA produces descriptions comparable to those generated by
expert human experimenters. We then show that MAIA can aid in two additional
interpretability tasks: reducing sensitivity to spurious features, and
automatically identifying inputs likely to be mis-classified.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your
  Phone
Authors: name: Marah Abdin, name: Sam Ade Jacobs, name: Ammar Ahmad Awan, name: Jyoti Aneja, name: Ahmed Awadallah, name: Hany Awadalla, name: Nguyen Bach, name: Amit Bahree, name: Arash Bakhtiari, name: Harkirat Behl, name: Alon Benhaim, name: Misha Bilenko, name: Johan Bjorck, name: Sébastien Bubeck, name: Martin Cai, name: Caio César Teodoro Mendes, name: Weizhu Chen, name: Vishrav Chaudhary, name: Parul Chopra, name: Allie Del Giorno, name: Gustavo de Rosa, name: Matthew Dixon, name: Ronen Eldan, name: Dan Iter, name: Abhishek Goswami, name: Suriya Gunasekar, name: Emman Haider, name: Junheng Hao, name: Russell J. Hewett, name: Jamie Huynh, name: Mojan Javaheripi, name: Xin Jin, name: Piero Kauffmann, name: Nikos Karampatziakis, name: Dongwoo Kim, name: Mahoud Khademi, name: Lev Kurilenko, name: James R. Lee, name: Yin Tat Lee, name: Yuanzhi Li, name: Chen Liang, name: Weishung Liu, name: Eric Lin, name: Zeqi Lin, name: Piyush Madan, name: Arindam Mitra, name: Hardik Modi, name: Anh Nguyen, name: Brandon Norick, name: Barun Patra, name: Daniel Perez-Becker, name: Thomas Portet, name: Reid Pryzant, name: Heyang Qin, name: Marko Radmilac, name: Corby Rosset, name: Sambudha Roy, name: Olli Saarikivi, name: Amin Saied, name: Adil Salim, name: Michael Santacroce, name: Shital Shah, name: Ning Shang, name: Hiteshi Sharma, name: Xia Song, name: Olatunji Ruwase, name: Xin Wang, name: Rachel Ward, name: Guanhua Wang, name: Philipp Witte, name: Michael Wyatt, name: Can Xu, name: Jiahang Xu, name: Sonali Yadav, name: Fan Yang, name: Ziyi Yang, name: Donghan Yu, name: Chengruidong Zhang, name: Cyril Zhang, name: Jianwen Zhang, name: Li Lyna Zhang, name: Yi Zhang, name: Yunan Zhang, name: Xiren Zhou
Published: 2024-04-22 14:32:33

We introduce phi-3-mini, a 3.8 billion parameter language model trained on
3.3 trillion tokens, whose overall performance, as measured by both academic
benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and
GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite
being small enough to be deployed on a phone. The innovation lies entirely in
our dataset for training, a scaled-up version of the one used for phi-2,
composed of heavily filtered web data and synthetic data. The model is also
further aligned for robustness, safety, and chat format. We also provide some
initial parameter-scaling results with a 7B and 14B models trained for 4.8T
tokens, called phi-3-small and phi-3-medium, both significantly more capable
than phi-3-mini (e.g., respectively 75% and 78% on MMLU, and 8.7 and 8.9 on
MT-bench).
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension
  and Generation
Authors: name: Yuying Ge, name: Sijie Zhao, name: Jinguo Zhu, name: Yixiao Ge, name: Kun Yi, name: Lin Song, name: Chen Li, name: Xiaohan Ding, name: Ying Shan
Published: 2024-04-22 17:56:09

The rapid evolution of multimodal foundation model has demonstrated
significant progresses in vision-language understanding and generation, e.g.,
our previous work SEED-LLaMA. However, there remains a gap between its
capability and the real-world applicability, primarily due to the model's
limited capacity to effectively respond to various user instructions and
interact with diverse visual data. In this work, we focus on bridging this gap
through integrating two enhanced features: (1) comprehending images of
arbitrary sizes and ratios, and (2) enabling multi-granularity image
generation. We present a unified and versatile foundation model, namely,
SEED-X, which is able to model multi-granularity visual semantics for
comprehension and generation tasks. Besides the competitive results on public
benchmarks, SEED-X demonstrates its effectiveness in handling real-world
applications across various domains after instruction tuning. We hope that our
work will inspire future research into what can be achieved by versatile
multimodal foundation models in real-world applications. The models, codes, and
datasets will be released in https://github.com/AILab-CVC/SEED-X.
MultiBooth: Towards Generating All Your Concepts in an Image from Text
Authors: name: Chenyang Zhu, name: Kai Li, name: Yue Ma, name: Chunming He, name: Li Xiu
Published: 2024-04-22 14:47:54

This paper introduces MultiBooth, a novel and efficient technique for
multi-concept customization in image generation from text. Despite the
significant advancements in customized generation methods, particularly with
the success of diffusion models, existing methods often struggle with
multi-concept scenarios due to low concept fidelity and high inference cost.
MultiBooth addresses these issues by dividing the multi-concept generation
process into two phases: a single-concept learning phase and a multi-concept
integration phase. During the single-concept learning phase, we employ a
multi-modal image encoder and an efficient concept encoding technique to learn
a concise and discriminative representation for each concept. In the
multi-concept integration phase, we use bounding boxes to define the generation
area for each concept within the cross-attention map. This method enables the
creation of individual concepts within their specified regions, thereby
facilitating the formation of multi-concept images. This strategy not only
improves concept fidelity but also reduces additional inference cost.
MultiBooth surpasses various baselines in both qualitative and quantitative
evaluations, showcasing its superior performance and computational efficiency.
Project Page: https://multibooth.github.io/
Music Consistency Models
Authors: name: Zhengcong Fei, name: Mingyuan Fan, name: Junshi Huang
Published: 2024-04-20 11:52:30

Consistency models have exhibited remarkable capabilities in facilitating
efficient image/video generation, enabling synthesis with minimal sampling
steps. It has proven to be advantageous in mitigating the computational burdens
associated with diffusion models. Nevertheless, the application of consistency
models in music generation remains largely unexplored. To address this gap, we
present Music Consistency Models (MusicCM), which leverages the
concept of consistency models to efficiently synthesize mel-spectrogram for
music clips, maintaining high quality while minimizing the number of sampling
steps. Building upon existing text-to-music diffusion models, the
MusicCM model incorporates consistency distillation and adversarial
discriminator training. Moreover, we find it beneficial to generate extended
coherent music by incorporating multiple diffusion processes with shared
constraints. Experimental results reveal the effectiveness of our model in
terms of computational efficiency, fidelity, and naturalness. Notable,
MusicCM achieves seamless music synthesis with a mere four sampling
steps, e.g., only one second per minute of the music clip, showcasing the
potential for real-time application.
How Far Can We Go with Practical Function-Level Program Repair?
Authors: name: Jiahong Xiang, name: Xiaoyang Xu, name: Fanchu Kong, name: Mingyuan Wu, name: Haotian Zhang, name: Yuqun Zhang
Published: 2024-04-19 12:14:09

Recently, multiple Automated Program Repair (APR) techniques based on Large
Language Models (LLMs) have been proposed to enhance the repair performance.
While these techniques mainly focus on the single-line or hunk-level repair,
they face significant challenges in real-world application due to the limited
repair task scope and costly statement-level fault localization. However, the
more practical function-level APR, which broadens the scope of APR task to fix
entire buggy functions and requires only cost-efficient function-level fault
localization, remains underexplored. In this paper, we conduct the first
comprehensive study of LLM-based function-level APR including investigating the
effect of the few-shot learning mechanism and the auxiliary repair-relevant
information. Specifically, we adopt six widely-studied LLMs and construct a
benchmark in both the Defects4J 1.2 and 2.0 datasets. Our study demonstrates
that LLMs with zero-shot learning are already powerful function-level APR
techniques, while applying the few-shot learning mechanism leads to disparate
repair performance. Moreover, we find that directly applying the auxiliary
repair-relevant information to LLMs significantly increases function-level
repair performance. Inspired by our findings, we propose an LLM-based
function-level APR technique, namely SRepair, which adopts a dual-LLM framework
to leverage the power of the auxiliary repair-relevant information for
advancing the repair performance. The evaluation results demonstrate that
SRepair can correctly fix 300 single-function bugs in the Defects4J dataset,
largely surpassing all previous APR techniques by at least 85%, without the
need for the costly statement-level fault location information. Furthermore,
SRepair successfully fixes 32 multi-function bugs in the Defects4J dataset,
which is the first time achieved by any APR technique ever to our best
knowledge.
AutoCrawler: A Progressive Understanding Web Agent for Web Crawler
  Generation
Authors: name: Wenhao Huang, name: Chenghao Peng, name: Zhixu Li, name: Jiaqing Liang, name: Yanghua Xiao, name: Liqian Wen, name: Zulong Chen
Published: 2024-04-19 09:59:44

Web automation is a significant technique that accomplishes complicated web
tasks by automating common web actions, enhancing operational efficiency, and
reducing the need for manual intervention. Traditional methods, such as
wrappers, suffer from limited adaptability and scalability when faced with a
new website. On the other hand, generative agents empowered by large language
models (LLMs) exhibit poor performance and reusability in open-world scenarios.
In this work, we introduce a crawler generation task for vertical information
web pages and the paradigm of combining LLMs with crawlers, which helps
crawlers handle diverse and changing web environments more efficiently. We
propose AutoCrawler, a two-stage framework that leverages the hierarchical
structure of HTML for progressive understanding. Through top-down and step-back
operations, AutoCrawler can learn from erroneous actions and continuously prune
HTML for better action generation. We conduct comprehensive experiments with
multiple LLMs and demonstrate the effectiveness of our framework. Resources of
this paper can be found at https://github.com/EZ-hwh/AutoCrawler
LLM-R2: A Large Language Model Enhanced Rule-based Rewrite System for
  Boosting Query Efficiency
Authors: name: Zhaodonghui Li, name: Haitao Yuan, name: Huiming Wang, name: Gao Cong, name: Lidong Bing
Published: 2024-04-19 13:17:07

Query rewrite, which aims to generate more efficient queries by altering a
SQL query's structure without changing the query result, has been an important
research problem. In order to maintain equivalence between the rewritten query
and the original one during rewriting, traditional query rewrite methods always
rewrite the queries following certain rewrite rules. However, some problems
still remain. Firstly, existing methods of finding the optimal choice or
sequence of rewrite rules are still limited and the process always costs a lot
of resources. Methods involving discovering new rewrite rules typically require
complicated proofs of structural logic or extensive user interactions.
Secondly, current query rewrite methods usually rely highly on DBMS cost
estimators which are often not accurate. In this paper, we address these
problems by proposing a novel method of query rewrite named LLM-R2, adopting a
large language model (LLM) to propose possible rewrite rules for a database
rewrite system. To further improve the inference ability of LLM in recommending
rewrite rules, we train a contrastive model by curriculum to learn query
representations and select effective query demonstrations for the LLM.
Experimental results have shown that our method can significantly improve the
query execution efficiency and outperform the baseline methods. In addition,
our method enjoys high robustness across different datasets.
TextSquare: Scaling up Text-Centric Visual Instruction Tuning
Authors: name: Jingqun Tang, name: Chunhui Lin, name: Zhen Zhao, name: Shu Wei, name: Binghong Wu, name: Qi Liu, name: Hao Feng, name: Yang Li, name: Siqi Wang, name: Lei Liao, name: Wei Shi, name: Yuliang Liu, name: Hao Liu, name: Yuan Xie, name: Xiang Bai, name: Can Huang
Published: 2024-04-19 11:38:08

Text-centric visual question answering (VQA) has made great strides with the
development of Multimodal Large Language Models (MLLMs), yet open-source models
still fall short of leading models like GPT4V and Gemini, partly due to a lack
of extensive, high-quality instruction tuning data. To this end, we introduce a
new approach for creating a massive, high-quality instruction-tuning dataset,
Square-10M, which is generated using closed-source MLLMs. The data construction
process, termed Square, consists of four steps: Self-Questioning, Answering,
Reasoning, and Evaluation. Our experiments with Square-10M led to three key
findings: 1) Our model, TextSquare, considerably surpasses open-source previous
state-of-the-art Text-centric MLLMs and sets a new standard on OCRBench(62.2%).
It even outperforms top-tier models like GPT4V and Gemini in 6 of 10
text-centric benchmarks. 2) Additionally, we demonstrate the critical role of
VQA reasoning data in offering comprehensive contextual insights for specific
questions. This not only improves accuracy but also significantly mitigates
hallucinations. Specifically, TextSquare scores an average of 75.1% across four
general VQA and hallucination evaluation datasets, outperforming previous
state-of-the-art models. 3) Notably, the phenomenon observed in scaling
text-centric VQA datasets reveals a vivid pattern: the exponential increase of
instruction tuning data volume is directly proportional to the improvement in
model performance, thereby validating the necessity of the dataset scale and
the high quality of Square-10M.
Groma: Localized Visual Tokenization for Grounding Multimodal Large
  Language Models
Authors: name: Chuofan Ma, name: Yi Jiang, name: Jiannan Wu, name: Zehuan Yuan, name: Xiaojuan Qi
Published: 2024-04-19 17:22:51

We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded
and fine-grained visual perception ability. Beyond holistic image
understanding, Groma is adept at region-level tasks such as region captioning
and visual grounding. Such capabilities are built upon a localized visual
tokenization mechanism, where an image input is decomposed into regions of
interest and subsequently encoded into region tokens. By integrating region
tokens into user instructions and model responses, we seamlessly enable Groma
to understand user-specified region inputs and ground its textual output to
images. Besides, to enhance the grounded chat ability of Groma, we curate a
visually grounded instruction dataset by leveraging the powerful GPT-4V and
visual prompting techniques. Compared with MLLMs that rely on the language
model or external module for localization, Groma consistently demonstrates
superior performances in standard referring and grounding benchmarks,
highlighting the advantages of embedding localization into image tokenization.
Project page: https://groma-mllm.github.io/.