Orionnnn/hub-stats huggingface dataset:
ArzEn-LLM: Code-Switched Egyptian Arabic-English Translation and Speech
  Recognition Using LLMs
Authors: name: Ahmed Heakl, name: Youssef Zaghloul, name: Mennatullah Ali, name: Rania Hossam, name: Walid Gomaa
Published: 2024-06-26 07:19:51

Motivated by the widespread increase in the phenomenon of code-switching
between Egyptian Arabic and English in recent times, this paper explores the
intricacies of machine translation (MT) and automatic speech recognition (ASR)
systems, focusing on translating code-switched Egyptian Arabic-English to
either English or Egyptian Arabic. Our goal is to present the methodologies
employed in developing these systems, utilizing large language models such as
LLama and Gemma. In the field of ASR, we explore the utilization of the Whisper
model for code-switched Egyptian Arabic recognition, detailing our experimental
procedures including data preprocessing and training techniques. Through the
implementation of a consecutive speech-to-text translation system that
integrates ASR with MT, we aim to overcome challenges posed by limited
resources and the unique characteristics of the Egyptian Arabic dialect.
Evaluation against established metrics showcases promising results, with our
methodologies yielding a significant improvement of 56% in English
translation over the state-of-the-art and 9.3% in Arabic translation. Since
code-switching is deeply inherent in spoken languages, it is crucial that ASR
systems can effectively handle this phenomenon. This capability is crucial for
enabling seamless interaction in various domains, including business
negotiations, cultural exchanges, and academic discourse. Our models and code
are available as open-source resources. Code:
http://github.com/ahmedheakl/arazn-llm}, Models:
http://huggingface.co/collections/ahmedheakl/arazn-llm-662ceaf12777656607b9524e.
MoA: Mixture of Sparse Attention for Automatic Large Language Model
  Compression
Authors: name: Tianyu Fu, name: Haofeng Huang, name: Xuefei Ning, name: Genghan Zhang, name: Boju Chen, name: Tianqi Wu, name: Hongyi Wang, name: Zixiao Huang, name: Shiyao Li, name: Shengen Yan, name: Guohao Dai, name: Huazhong Yang, name: Yu Wang
Published: 2024-06-21 06:58:37

Sparse attention can effectively mitigate the significant memory and
throughput demands of Large Language Models (LLMs) in long contexts. Existing
methods typically employ a uniform sparse attention mask, applying the same
sparse pattern across different attention heads and input lengths. However,
this uniform approach fails to capture the diverse attention patterns inherent
in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this
challenge, we propose the Mixture of Attention (MoA), which automatically
tailors distinct sparse attention configurations to different heads and layers.
MoA constructs and navigates a search space of various attention patterns and
their scaling rules relative to input sequence lengths. It profiles the model,
evaluates potential configurations, and pinpoints the optimal sparse attention
compression plan. MoA adapts to varying input sizes, revealing that some
attention heads expand their focus to accommodate longer sequences, while other
heads consistently concentrate on fixed-length local contexts. Experiments show
that MoA increases the effective context length by 3.9times with the same
average attention span, boosting retrieval accuracy by 1.5-7.1times over the
uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.
Moreover, MoA narrows the capability gaps between sparse and dense models,
reducing the maximum relative performance drop from 9%-36% to within 5%
across two long-context understanding benchmarks. MoA achieves a
1.2-1.4times GPU memory reduction and boosts decode throughput by 5.5-6.7
times for 7B and 13B dense models on a single GPU, with minimal impact on
performance.
Is Programming by Example solved by LLMs?
Authors: name: Wen-Ding Li, name: Kevin Ellis
Published: 2024-06-12 15:16:40

Programming-by-Examples (PBE) aims to generate an algorithm from input-output
examples. Such systems are practically and theoretically important: from an
end-user perspective, they are deployed to millions of people, and from an AI
perspective, PBE corresponds to a very general form of few-shot inductive
inference. Given the success of Large Language Models (LLMs) in code-generation
tasks, we investigate here the extent to which LLMs can be said to have
`solved' PBE. We experiment on classic domains such as lists and strings, and
an uncommon graphics programming domain not well represented in typical
pretraining data. We find that pretrained models are not effective at PBE, but
that they can be fine-tuned for much higher performance, provided the test
problems are in-distribution. We analyze empirically what causes these models
to succeed and fail, and take steps toward understanding how to achieve better
out-of-distribution generalization. Collectively these results suggest that
LLMs make strong progress toward solving the typical suite of PBE tasks,
potentially increasing the flexibility and applicability of PBE systems, while
also identifying ways in which LLMs still fall short.
Can LLMs Learn by Teaching? A Preliminary Study
Authors: name: Xuefei Ning, name: Zifu Wang, name: Shiyao Li, name: Zinan Lin, name: Peiran Yao, name: Tianyu Fu, name: Matthew B. Blaschko, name: Guohao Dai, name: Huazhong Yang, name: Yu Wang
Published: 2024-06-20 18:00:17

Teaching to improve student models (e.g., knowledge distillation) is an
extensively studied methodology in LLMs. However, for humans, teaching not only
improves students but also improves teachers. We ask: Can LLMs also learn by
teaching (LbT)? If yes, we can potentially unlock the possibility of
continuously advancing the models without solely relying on human-produced data
or stronger models. In this paper, we provide a preliminary exploration of this
ambitious agenda. We show that LbT ideas can be incorporated into existing LLM
training/prompting pipelines and provide noticeable improvements. Specifically,
we design three methods, each mimicking one of the three levels of LbT in
humans: observing students' feedback, learning from the feedback, and learning
iteratively, with the goals of improving answer accuracy without training and
improving models' inherent capability with fine-tuning. The findings are
encouraging. For example, similar to LbT in human, we see that: (1) LbT can
induce weak-to-strong generalization: strong models can improve themselves by
teaching other weak models; (2) Diversity in students might help: teaching
multiple students could be better than teaching one student or the teacher
itself. We hope that this early promise can inspire future research on LbT and
more broadly adopting the advanced techniques in education to improve LLMs. The
code is available at https://github.com/imagination-research/lbt.
Simulating Classroom Education with LLM-Empowered Agents
Authors: name: Zheyuan Zhang, name: Daniel Zhang-Li, name: Jifan Yu, name: Linlu Gong, name: Jinchang Zhou, name: Zhiyuan Liu, name: Lei Hou, name: Juanzi Li
Published: 2024-06-27 14:51:07

Large language models (LLMs) have been employed in various intelligent
educational tasks to assist teaching. While preliminary explorations have
focused on independent LLM-empowered agents for specific educational tasks, the
potential for LLMs within a multi-agent collaborative framework to simulate a
classroom with real user participation remains unexplored. In this work, we
propose SimClass, a multi-agent classroom simulation framework involving user
participation. We recognize representative class roles and introduce a novel
class control mechanism for automatic classroom teaching, and conduct user
experiments in two real-world courses. Utilizing the Flanders Interactive
Analysis System and Community of Inquiry theoretical frame works from
educational analysis, we demonstrate that LLMs can simulate traditional
classroom interaction patterns effectively while enhancing user's experience.
We also observe emergent group behaviors among agents in SimClass, where agents
collaborate to create enlivening interactions in classrooms to improve user
learning process. We hope this work pioneers the application of LLM-empowered
multi-agent systems in virtual classroom teaching.
OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and
  Understanding
Authors: name: Tao Zhang, name: Xiangtai Li, name: Hao Fei, name: Haobo Yuan, name: Shengqiong Wu, name: Shunping Ji, name: Chen Change Loy, name: Shuicheng Yan
Published: 2024-06-27 17:59:01

Current universal segmentation methods demonstrate strong capabilities in
pixel-level image and video understanding. However, they lack reasoning
abilities and cannot be controlled via text instructions. In contrast, large
vision-language multimodal models exhibit powerful vision-based conversation
and reasoning capabilities but lack pixel-level understanding and have
difficulty accepting visual prompts for flexible user interaction. This paper
proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
vision understanding with reasoning abilities. It can accept various visual and
text prompts for flexible user interaction. Specifically, we use a universal
segmentation method as the visual encoder, integrating image information,
perception priors, and visual prompts into visual tokens provided to the LLM.
The LLM is responsible for understanding the user's text instructions and
providing text responses and pixel-level segmentation results based on the
visual information. We propose perception prior embedding to better integrate
perception priors with image features. OMG-LLaVA achieves image-level,
object-level, and pixel-level reasoning and understanding in a single model,
matching or surpassing the performance of specialized methods on multiple
benchmarks. Rather than using LLM to connect each specialist, our work aims at
end-to-end training on one encoder, one decoder, and one LLM. The code and
model have been released for further research.
Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens
  Grounding
Authors: name: Yue Fan, name: Lei Ding, name: Ching-Chen Kuo, name: Shan Jiang, name: Yang Zhao, name: Xinze Guan, name: Jie Yang, name: Yi Zhang, name: Xin Eric Wang
Published: 2024-06-27 15:34:16

Graphical User Interfaces (GUIs) are central to our interaction with digital
devices. Recently, growing efforts have been made to build models for various
GUI understanding tasks. However, these efforts largely overlook an important
GUI-referring task: screen reading based on user-indicated points, which we
name the Screen Point-and-Read (SPR) task. This task is predominantly handled
by rigid accessible screen reading tools, in great need of new models driven by
advancements in Multimodal Large Language Models (MLLMs). In this paper, we
propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism,
to address the SPR task. Based on the input point coordinate and the
corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout
Tree. Based on the tree, our ToL agent not only comprehends the content of the
indicated area but also articulates the layout and spatial relationships
between elements. Such layout information is crucial for accurately
interpreting information on the screen, distinguishing our ToL agent from other
screen reading tools. We also thoroughly evaluate the ToL agent against other
baselines on a newly proposed SPR benchmark, which includes GUIs from mobile,
web, and operating systems. Last but not least, we test the ToL agent on mobile
GUI navigation tasks, demonstrating its utility in identifying incorrect
actions along the path of agent execution trajectories. Code and data:
screen-point-and-read.github.io
SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented
  Generation
Authors: name: Zijun Yao, name: Weijian Qi, name: Liangming Pan, name: Shulin Cao, name: Linmei Hu, name: Weichuan Liu, name: Lei Hou, name: Juanzi Li
Published: 2024-06-27 14:38:33

This paper introduces Self-aware Knowledge Retrieval (SeaKR), a novel
adaptive RAG model that extracts self-aware uncertainty of LLMs from their
internal states. SeaKR activates retrieval when the LLMs present high
self-aware uncertainty for generation. To effectively integrate retrieved
knowledge snippets, SeaKR re-ranks them based on LLM's self-aware uncertainty
to preserve the snippet that reduces their uncertainty to the utmost. To
facilitate solving complex tasks that require multiple retrievals, SeaKR
utilizes their self-aware uncertainty to choose among different reasoning
strategies. Our experiments on both complex and simple Question Answering
datasets show that SeaKR outperforms existing adaptive RAG methods. We release
our code at https://github.com/THU-KEG/SeaKR.
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of
  LLMs
Authors: name: Xin Lai, name: Zhuotao Tian, name: Yukang Chen, name: Senqiao Yang, name: Xiangru Peng, name: Jiaya Jia
Published: 2024-06-26 17:43:06

Mathematical reasoning presents a significant challenge for Large Language
Models (LLMs) due to the extensive and precise chain of reasoning required for
accuracy. Ensuring the correctness of each reasoning step is critical. To
address this, we aim to enhance the robustness and factuality of LLMs by
learning from human feedback. However, Direct Preference Optimization (DPO) has
shown limited benefits for long-chain mathematical reasoning, as models
employing DPO struggle to identify detailed errors in incorrect answers. This
limitation stems from a lack of fine-grained process supervision. We propose a
simple, effective, and data-efficient method called Step-DPO, which treats
individual reasoning steps as units for preference optimization rather than
evaluating answers holistically. Additionally, we have developed a data
construction pipeline for Step-DPO, enabling the creation of a high-quality
dataset containing 10K step-wise preference pairs. We also observe that in DPO,
self-generated data is more effective than data generated by humans or GPT-4,
due to the latter's out-of-distribution nature. Our findings demonstrate that
as few as 10K preference data pairs and fewer than 500 Step-DPO training steps
can yield a nearly 3% gain in accuracy on MATH for models with over 70B
parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves
scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively,
surpassing a series of closed-source models, including GPT-4-1106,
Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at
https://github.com/dvlab-research/Step-DPO.
Aligning Teacher with Student Preferences for Tailored Training Data
  Generation
Authors: name: Yantao Liu, name: Zhao Zhang, name: Zijun Yao, name: Shulin Cao, name: Lei Hou, name: Juanzi Li
Published: 2024-06-27 14:51:17

Large Language Models (LLMs) have shown significant promise as copilots in
various tasks. Local deployment of LLMs on edge devices is necessary when
handling privacy-sensitive data or latency-sensitive tasks. The computational
constraints of such devices make direct deployment of powerful large-scale LLMs
impractical, necessitating the Knowledge Distillation from large-scale models
to lightweight models. Lots of work has been done to elicit diversity and
quality training examples from LLMs, but little attention has been paid to
aligning teacher instructional content based on student preferences, akin to
"responsive teaching" in pedagogy. Thus, we propose ARTE, dubbed Aligning
TeacheR with StudenT PreferencEs, a framework that aligns the teacher model
with student preferences to generate tailored training examples for Knowledge
Distillation. Specifically, we elicit draft questions and rationales from the
teacher model, then collect student preferences on these questions and
rationales using students' performance with in-context learning as a proxy, and
finally align the teacher model with student preferences. In the end, we repeat
the first step with the aligned teacher model to elicit tailored training
examples for the student model on the target task. Extensive experiments on
academic benchmarks demonstrate the superiority of ARTE over existing
instruction-tuning datasets distilled from powerful LLMs. Moreover, we
thoroughly investigate the generalization of ARTE, including the generalization
of fine-tuned student models in reasoning ability and the generalization of
aligned teacher models to generate tailored training data across tasks and
students. In summary, our contributions lie in proposing a novel framework for
tailored training example generation, demonstrating its efficacy in
experiments, and investigating the generalization of both student & aligned
teacher models in ARTE.
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Authors: name: Brandon Huang, name: Chancharik Mitra, name: Assaf Arbelle, name: Leonid Karlinsky, name: Trevor Darrell, name: Roei Herzig
Published: 2024-06-21 17:50:02

The recent success of interleaved Large Multimodal Models (LMMs) in few-shot
learning suggests that in-context learning (ICL) with many examples can be
promising for learning new tasks. However, this many-shot multimodal ICL
setting has one crucial problem: it is fundamentally limited by the model's
context length set at pretraining. The problem is especially prominent in the
multimodal domain, which processes both text and images, requiring additional
tokens. This motivates the need for a multimodal method to compress many shots
into fewer tokens without finetuning. In this work, we enable LMMs to perform
multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors
(MTV)--compact implicit representations of in-context examples compressed in
the model's attention heads. Specifically, we first demonstrate the existence
of such MTV in LMMs and then leverage these extracted MTV to enable many-shot
in-context learning for various vision-and-language tasks. Our experiments
suggest that MTV can scale in performance with the number of compressed shots
and generalize to similar out-of-domain tasks without additional context length
for inference.
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
  Safer Language Models
Authors: name: Liwei Jiang, name: Kavel Rao, name: Seungju Han, name: Allyson Ettinger, name: Faeze Brahman, name: Sachin Kumar, name: Niloofar Mireshghallah, name: Ximing Lu, name: Maarten Sap, name: Yejin Choi, name: Nouha Dziri
Published: 2024-06-26 17:31:22

We introduce WildTeaming, an automatic LLM safety red-teaming framework that
mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
novel jailbreak tactics, and then composes multiple tactics for systematic
exploration of novel jailbreaks. Compared to prior work that performed
red-teaming via recruited human workers, gradient-based optimization, or
iterative revision with LLMs, our work investigates jailbreaks from chatbot
users who were not specifically instructed to break the system. WildTeaming
reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
up to 4.6x more diverse and successful adversarial attacks compared to
state-of-the-art jailbreak methods.
  While many datasets exist for jailbreak evaluation, very few open-source
datasets exist for jailbreak training, as safety training data has been closed
even when model weights are open. With WildTeaming we create WildJailbreak, a
large-scale open-source synthetic safety dataset with 262K vanilla (direct
request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
exaggerated safety behaviors, WildJailbreak provides two contrastive types of
queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
resemble harmful queries in form but contain no harm. As WildJailbreak
considerably upgrades the quality and scale of existing safety resources, it
uniquely enables us to examine the scaling effects of data and the interplay of
data properties and model capabilities during safety training. Through
extensive experiments, we identify the training properties that enable an ideal
balance of safety behaviors: appropriate safeguarding without over-refusal,
effective handling of vanilla and adversarial queries, and minimal, if any,
decrease in general capabilities. All components of WildJailbeak contribute to
achieving balanced safety behaviors of models.
Symbolic Learning Enables Self-Evolving Agents
Authors: name: Wangchunshu Zhou, name: Yixin Ou, name: Shengwei Ding, name: Long Li, name: Jialong Wu, name: Tiannan Wang, name: Jiamin Chen, name: Shuai Wang, name: Xiaohua Xu, name: Ningyu Zhang, name: Huajun Chen, name: Yuchen Eleanor Jiang
Published: 2024-06-26 17:59:18

The AI community has been exploring a pathway to artificial general
intelligence (AGI) by developing "language agents", which are complex large
language models (LLMs) pipelines involving both prompting techniques and tool
usage methods. While language agents have demonstrated impressive capabilities
for many real-world tasks, a fundamental limitation of current language agents
research is that they are model-centric, or engineering-centric. That's to say,
the progress on prompts, tools, and pipelines of language agents requires
substantial manual engineering efforts from human experts rather than
automatically learning from data. We believe the transition from model-centric,
or engineering-centric, to data-centric, i.e., the ability of language agents
to autonomously learn and evolve in environments, is the key for them to
possibly achieve AGI.
  In this work, we introduce agent symbolic learning, a systematic framework
that enables language agents to optimize themselves on their own in a
data-centric way using symbolic optimizers. Specifically, we consider agents as
symbolic networks where learnable weights are defined by prompts, tools, and
the way they are stacked together. Agent symbolic learning is designed to
optimize the symbolic network within language agents by mimicking two
fundamental algorithms in connectionist learning: back-propagation and gradient
descent. Instead of dealing with numeric weights, agent symbolic learning works
with natural language simulacrums of weights, loss, and gradients. We conduct
proof-of-concept experiments on both standard benchmarks and complex real-world
tasks and show that agent symbolic learning enables language agents to update
themselves after being created and deployed in the wild, resulting in
"self-evolving agents".