Orionnnn/hub-stats huggingface dataset:
The FIGNEWS Shared Task on News Media Narratives
Authors: name: Wajdi Zaghouani, name: Mustafa Jarrar, name: Nizar Habash, name: Houda Bouamor, name: Imed Zitouni, name: Mona Diab, name: Samhaa R. El-Beltagy, name: Muhammed AbuOdeh
Published: 2024-07-25 15:58:19

We present an overview of the FIGNEWS shared task, organized as part of the
ArabicNLP 2024 conference co-located with ACL 2024. The shared task addresses
bias and propaganda annotation in multilingual news posts. We focus on the
early days of the Israel War on Gaza as a case study. The task aims to foster
collaboration in developing annotation guidelines for subjective tasks by
creating frameworks for analyzing diverse narratives highlighting potential
bias and propaganda. In a spirit of fostering and encouraging diversity, we
address the problem from a multilingual perspective, namely within five
languages: English, French, Arabic, Hebrew, and Hindi. A total of 17 teams
participated in two annotation subtasks: bias (16 teams) and propaganda (6
teams). The teams competed in four evaluation tracks: guidelines development,
annotation quality, annotation quantity, and consistency. Collectively, the
teams produced 129,800 data points. Key findings and implications for the field
are discussed.
Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic
Authors: name: Fakhraddin Alwajih, name: Gagan Bhatia, name: Muhammad Abdul-Mageed
Published: 2024-07-25 15:36:48

Recent advancements have significantly enhanced the capabilities of
Multimodal Large Language Models (MLLMs) in generating and understanding
image-to-text content. Despite these successes, progress is predominantly
limited to English due to the scarcity of high quality multimodal resources in
other languages. This limitation impedes the development of competitive models
in languages such as Arabic. To alleviate this situation, we introduce an
efficient Arabic multimodal assistant, dubbed Dallah, that utilizes an advanced
language model based on LLaMA-2 to facilitate multimodal interactions. Dallah
demonstrates state-of-the-art performance in Arabic MLLMs. Through fine-tuning
six Arabic dialects, Dallah showcases its capability to handle complex
dialectal interactions incorporating both textual and visual elements. The
model excels in two benchmark tests: one evaluating its performance on Modern
Standard Arabic (MSA) and another specifically designed to assess dialectal
responses. Beyond its robust performance in multimodal interaction tasks,
Dallah has the potential to pave the way for further development of
dialect-aware Arabic MLLMs.
Efficient Inference of Vision Instruction-Following Models with Elastic
  Cache
Authors: name: Zuyan Liu, name: Benlin Liu, name: Jiahui Wang, name: Yuhao Dong, name: Guangyi Chen, name: Yongming Rao, name: Ranjay Krishna, name: Jiwen Lu
Published: 2024-07-25 15:29:05

In the field of instruction-following large vision-language models (LVLMs),
the efficient deployment of these models faces challenges, notably due to the
high memory demands of their key-value (KV) caches. Conventional cache
management strategies for LLMs focus on cache eviction, which often fails to
address the specific needs of multimodal instruction-following models.
Recognizing this gap, in this paper, we introduce Elastic Cache, a novel
approach that benefits from applying distinct acceleration methods for
instruction encoding and output generation stages. We investigate the metrics
of importance in different stages and propose an importance-driven cache
merging strategy to prune redundancy caches. Instead of discarding less
important caches, our strategy identifies important key/value vectors as anchor
points. Surrounding less important caches are then merged with these anchors,
enhancing the preservation of contextual information in the KV caches while
yielding an arbitrary acceleration ratio. For instruction encoding, we utilize
the frequency to evaluate the importance of caches. Regarding output
generation, we prioritize tokens based on their distance with an offset, by
which both the initial and most recent tokens are retained. Results on a range
of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also
notably outperforms existing pruning methods in language generation across
various tasks. Code is available at https://github.com/liuzuyan/ElasticCache
Data Mixture Inference: What do BPE Tokenizers Reveal about their
  Training Data?
Authors: name: Jonathan Hayase, name: Alisa Liu, name: Yejin Choi, name: Sewoong Oh, name: Noah A. Smith
Published: 2024-07-23 16:13:22

The pretraining data of today's strongest language models is opaque. In
particular, little is known about the proportions of various domains or
languages represented. In this work, we tackle a task which we call data
mixture inference, which aims to uncover the distributional make-up of training
data. We introduce a novel attack based on a previously overlooked source of
information -- byte-pair encoding (BPE) tokenizers, used by the vast majority
of modern language models. Our key insight is that the ordered list of merge
rules learned by a BPE tokenizer naturally reveals information about the token
frequencies in its training data: the first merge is the most common byte pair,
the second is the most common pair after merging the first token, and so on.
Given a tokenizer's merge list along with data samples for each category of
interest, we formulate a linear program that solves for the proportion of each
category in the tokenizer's training set. Importantly, to the extent to which
tokenizer training data is representative of the pretraining data, we
indirectly learn about the pretraining data. In controlled experiments, we show
that our attack recovers mixture ratios with high precision for tokenizers
trained on known mixtures of natural languages, programming languages, and data
sources. We then apply our approach to off-the-shelf tokenizers released with
recent LMs. We confirm much publicly disclosed information about these models,
and also make several new inferences: GPT-4o's tokenizer is much more
multilingual than its predecessors, training on 39% non-English data; Llama3
extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and
Claude's tokenizers are trained on predominantly code (~60%). We hope our work
sheds light on current design practices for pretraining data, and inspires
continued research into data mixture inference for LMs.
Course-Correction: Safety Alignment Using Synthetic Preferences
Authors: name: Rongwu Xu, name: Yishuo Cai, name: Zhenhong Zhou, name: Renjie Gu, name: Haiqin Weng, name: Yan Liu, name: Tianwei Zhang, name: Wei Xu, name: Han Qiu
Published: 2024-07-23 16:54:28

The risk of harmful content generated by large language models (LLMs) becomes
a critical concern. This paper presents a systematic study on assessing and
improving LLMs' capability to perform the task of course-correction,
\ie, the model can steer away from generating harmful content autonomously. To
start with, we introduce the C^2-Eval benchmark for quantitative
assessment and analyze 10 popular LLMs, revealing varying proficiency of
current safety-tuned LLMs in course-correction. To improve, we propose
fine-tuning LLMs with preference learning, emphasizing the preference for
timely course-correction. Using an automated pipeline, we create
C^2-Syn, a synthetic dataset with 750K pairwise preferences, to
teach models the concept of timely course-correction through data-driven
preference learning. Experiments on 2 LLMs, Llama2-Chat 7B and
Qwen2 7B, show that our method effectively enhances course-correction
skills without affecting general performance. Additionally, it effectively
improves LLMs' safety, particularly in resisting jailbreak attacks.
BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular
  Depth Estimation
Authors: name: Xiang Zhang, name: Bingxin Ke, name: Hayko Riemenschneider, name: Nando Metzger, name: Anton Obukhov, name: Markus Gross, name: Konrad Schindler, name: Christopher Schroers
Published: 2024-07-25 11:16:37

By training over large-scale datasets, zero-shot monocular depth estimation
(MDE) methods show robust performance in the wild but often suffer from
insufficiently precise details. Although recent diffusion-based MDE approaches
exhibit appealing detail extraction ability, they still struggle in
geometrically challenging scenes due to the difficulty of gaining robust
geometric priors from diverse datasets. To leverage the complementary merits of
both worlds, we propose BetterDepth to efficiently achieve geometrically
correct affine-invariant MDE performance while capturing fine-grained details.
Specifically, BetterDepth is a conditional diffusion-based refiner that takes
the prediction from pre-trained MDE models as depth conditioning, in which the
global depth context is well-captured, and iteratively refines details based on
the input image. For the training of such a refiner, we propose global
pre-alignment and local patch masking methods to ensure the faithfulness of
BetterDepth to depth conditioning while learning to capture fine-grained scene
details. By efficient training on small-scale synthetic datasets, BetterDepth
achieves state-of-the-art zero-shot MDE performance on diverse public datasets
and in-the-wild scenes. Moreover, BetterDepth can improve the performance of
other MDE models in a plug-and-play manner without additional re-training.
Very Large-Scale Multi-Agent Simulation in AgentScope
Authors: name: Xuchen Pan, name: Dawei Gao, name: Yuexiang Xie, name: Zhewei Wei, name: Yaliang Li, name: Bolin Ding, name: Ji-Rong Wen, name: Jingren Zhou
Published: 2024-07-25 05:50:46

Recent advances in large language models (LLMs) have opened new avenues for
applying multi-agent systems in very large-scale simulations. However, there
remain several challenges when conducting multi-agent simulations with existing
platforms, such as limited scalability and low efficiency, unsatisfied agent
diversity, and effort-intensive management processes. To address these
challenges, we develop several new features and components for AgentScope, a
user-friendly multi-agent platform, enhancing its convenience and flexibility
for supporting very large-scale multi-agent simulations. Specifically, we
propose an actor-based distributed mechanism as the underlying technological
infrastructure towards great scalability and high efficiency, and provide
flexible environment support for simulating various real-world scenarios, which
enables parallel execution of multiple agents, centralized workflow
orchestration, and both inter-agent and agent-environment interactions among
agents. Moreover, we integrate an easy-to-use configurable tool and an
automatic background generation pipeline in AgentScope, simplifying the process
of creating agents with diverse yet detailed background settings. Last but not
least, we provide a web-based interface for conveniently monitoring and
managing a large number of agents that might deploy across multiple devices. We
conduct a comprehensive simulation to demonstrate the effectiveness of the
proposed enhancements in AgentScope, and provide detailed observations and
discussions to highlight the great potential of applying multi-agent systems in
large-scale simulations. The source code is released on GitHub at
https://github.com/modelscope/agentscope to inspire further research and
development in large-scale multi-agent simulations.
LAMBDA: A Large Model Based Data Agent
Authors: name: Maojun Sun, name: Ruijian Han, name: Binyan Jiang, name: Houduo Qi, name: Defeng Sun, name: Yancheng Yuan, name: Jian Huang
Published: 2024-07-24 06:26:36

We introduce ``LAMBDA," a novel open-source, code-free multi-agent data
analysis system that that harnesses the power of large models. LAMBDA is
designed to address data analysis challenges in complex data-driven
applications through the use of innovatively designed data agents that operate
iteratively and generatively using natural language. At the core of LAMBDA are
two key agent roles: the programmer and the inspector, which are engineered to
work together seamlessly. Specifically, the programmer generates code based on
the user's instructions and domain-specific knowledge, enhanced by advanced
models. Meanwhile, the inspector debugs the code when necessary. To ensure
robustness and handle adverse scenarios, LAMBDA features a user interface that
allows direct user intervention in the operational loop. Additionally, LAMBDA
can flexibly integrate external models and algorithms through our knowledge
integration mechanism, catering to the needs of customized data analysis.
LAMBDA has demonstrated strong performance on various machine learning
datasets. It has the potential to enhance data science practice and analysis
paradigm by seamlessly integrating human and artificial intelligence, making it
more accessible, effective, and efficient for individuals from diverse
backgrounds. The strong performance of LAMBDA in solving data science problems
is demonstrated in several case studies, which are presented at
https://www.polyu.edu.hk/ama/cmfai/lambda.html.
Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model
Authors: name: Lirui Zhao, name: Tianshuo Yang, name: Wenqi Shao, name: Yuxin Zhang, name: Yu Qiao, name: Ping Luo, name: Kaipeng Zhang, name: Rongrong Ji
Published: 2024-07-24 03:58:58

This paper addresses an important problem of object addition for images with
only text guidance. It is challenging because the new object must be integrated
seamlessly into the image with consistent visual context, such as lighting,
texture, and spatial location. While existing text-guided image inpainting
methods can add objects, they either fail to preserve the background
consistency or involve cumbersome human intervention in specifying bounding
boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree,
a Text-to-Image (T2I) model that facilitates text-guided object addition with
only text control. To this end, we curate OABench, an exquisite synthetic
dataset by removing objects with advanced image inpainting techniques. OABench
comprises 74K real-world tuples of an original image, an inpainted image with
the object removed, an object mask, and object descriptions. Trained on OABench
using the Stable Diffusion model with an additional mask prediction module,
Diffree uniquely predicts the position of the new object and achieves object
addition with guidance from only text. Extensive experiments demonstrate that
Diffree excels in adding new objects with a high success rate while maintaining
background consistency, spatial appropriateness, and object relevance and
quality.
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Authors: name: Yuxiang Chai, name: Siyuan Huang, name: Yazhe Niu, name: Han Xiao, name: Liang Liu, name: Dingyu Zhang, name: Peng Gao, name: Shuai Ren, name: Hongsheng Li
Published: 2024-07-03 17:59:58

AI agents have drawn increasing attention mostly on their ability to perceive
environments, understand tasks, and autonomously achieve goals. To advance
research on AI agents in mobile scenarios, we introduce the Android
Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for
generalist mobile GUI-control agents. Their capabilities of completing complex
tasks by directly interacting with the graphical user interface (GUI) on mobile
devices are trained and evaluated with the proposed dataset. AMEX comprises
over 104K high-resolution screenshots from 110 popular mobile applications,
which are annotated at multiple levels. Unlike existing mobile device-control
datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations:
GUI interactive element grounding, GUI screen and element functionality
descriptions, and complex natural language instructions, each averaging 13
steps with stepwise GUI-action chains. We develop this dataset from a more
instructive and detailed perspective, complementing the general settings of
existing datasets. Additionally, we develop a baseline model SPHINX Agent and
compare its performance across state-of-the-art agents trained on other
datasets. To facilitate further research, we open-source our dataset, models,
and relevant evaluation tools. The project is available at
https://yuxiangchai.github.io/AMEX/
DistilDIRE: A Small, Fast, Cheap and Lightweight Diffusion Synthesized
  Deepfake Detection
Authors: name: Yewon Lim, name: Changyeon Lee, name: Aerin Kim, name: Oren Etzioni
Published: 2024-06-02 20:22:38

A dramatic influx of diffusion-generated images has marked recent years,
posing unique challenges to current detection technologies. While the task of
identifying these images falls under binary classification, a seemingly
straightforward category, the computational load is significant when employing
the "reconstruction then compare" technique. This approach, known as DIRE
(Diffusion Reconstruction Error), not only identifies diffusion-generated
images but also detects those produced by GANs, highlighting the technique's
broad applicability. To address the computational challenges and improve
efficiency, we propose distilling the knowledge embedded in diffusion models to
develop rapid deepfake detection models. Our approach, aimed at creating a
small, fast, cheap, and lightweight diffusion synthesized deepfake detector,
maintains robust performance while significantly reducing operational demands.
Maintaining performance, our experimental results indicate an inference speed
3.2 times faster than the existing DIRE framework. This advance not only
enhances the practicality of deploying these systems in real-world settings but
also paves the way for future research endeavors that seek to leverage
diffusion model knowledge.
Longhorn: State Space Models are Amortized Online Learners
Authors: name: Bo Liu, name: Rui Wang, name: Lemeng Wu, name: Yihao Feng, name: Peter Stone, name: Qiang Liu
Published: 2024-07-19 11:12:08

The most fundamental capability of modern AI methods such as Large Language
Models (LLMs) is the ability to predict the next token in a long sequence of
tokens, known as ``sequence modeling." Although the Transformers model is the
current dominant approach to sequence modeling, its quadratic computational
cost with respect to sequence length is a significant drawback. State-space
models (SSMs) offer a promising alternative due to their linear decoding
efficiency and high parallelizability during training. However, existing SSMs
often rely on seemingly ad hoc linear recurrence designs. In this work, we
explore SSM design through the lens of online learning, conceptualizing SSMs as
meta-modules for specific online learning problems. This approach links SSM
design to formulating precise online learning objectives, with state transition
rules derived from optimizing these objectives. Based on this insight, we
introduce a novel deep SSM architecture based on the implicit update for
optimizing an online regression objective. Our experimental results show that
our models outperform state-of-the-art SSMs, including the Mamba model, on
standard sequence modeling benchmarks and language modeling tasks.
MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent
  Reinforcement Learning
Authors: name: Florian Felten, name: Umut Ucak, name: Hicham Azmani, name: Gao Peng, name: Willem Röpke, name: Hendrik Baier, name: Patrick Mannion, name: Diederik M. Roijers, name: Jordan K. Terry, name: El-Ghazali Talbi, name: Grégoire Danoy, name: Ann Nowé, name: Roxana Rădulescu
Published: 2024-07-23 09:05:06

Many challenging tasks such as managing traffic systems, electricity grids,
or supply chains involve complex decision-making processes that must balance
multiple conflicting objectives and coordinate the actions of various
independent decision-makers (DMs). One perspective for formalising and
addressing such tasks is multi-objective multi-agent reinforcement learning
(MOMARL). MOMARL broadens reinforcement learning (RL) to problems with multiple
agents each needing to consider multiple objectives in their learning process.
In reinforcement learning research, benchmarks are crucial in facilitating
progress, evaluation, and reproducibility. The significance of benchmarks is
underscored by the existence of numerous benchmark frameworks developed for
various RL paradigms, including single-agent RL (e.g., Gymnasium), multi-agent
RL (e.g., PettingZoo), and single-agent multi-objective RL (e.g.,
MO-Gymnasium). To support the advancement of the MOMARL field, we introduce
MOMAland, the first collection of standardised environments for multi-objective
multi-agent reinforcement learning. MOMAland addresses the need for
comprehensive benchmarking in this emerging field, offering over 10 diverse
environments that vary in the number of agents, state representations, reward
structures, and utility considerations. To provide strong baselines for future
research, MOMAland also includes algorithms capable of learning policies in
such settings.
Learning to Manipulate Anywhere: A Visual Generalizable Framework For
  Reinforcement Learning
Authors: name: Zhecheng Yuan, name: Tianming Wei, name: Shuiqi Cheng, name: Gu Zhang, name: Yuanpei Chen, name: Huazhe Xu
Published: 2024-07-22 17:29:02

Can we endow visuomotor robots with generalization capabilities to operate in
diverse open-world scenarios? In this paper, we propose Maniwhere, a
generalizable framework tailored for visual reinforcement learning, enabling
the trained robot policies to generalize across a combination of multiple
visual disturbance types. Specifically, we introduce a multi-view
representation learning approach fused with Spatial Transformer Network (STN)
module to capture shared semantic information and correspondences among
different viewpoints. In addition, we employ a curriculum-based randomization
and augmentation approach to stabilize the RL training process and strengthen
the visual generalization ability. To exhibit the effectiveness of Maniwhere,
we meticulously design 8 tasks encompassing articulate objects, bi-manual, and
dexterous hand manipulation tasks, demonstrating Maniwhere's strong visual
generalization and sim2real transfer abilities across 3 hardware platforms. Our
experiments show that Maniwhere significantly outperforms existing
state-of-the-art methods. Videos are provided at
https://gemcollector.github.io/maniwhere/.
DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car
  Reconstruction
Authors: name: Xiaobiao Du, name: Haiyang Sun, name: Ming Lu, name: Tianqing Zhu, name: Xin Yu
Published: 2024-07-24 04:13:43

Self-driving industries usually employ professional artists to build
exquisite 3D cars. However, it is expensive to craft large-scale digital
assets. Since there are already numerous datasets available that contain a vast
number of images of cars, we focus on reconstructing high-quality 3D car models
from these datasets. However, these datasets only contain one side of cars in
the forward-moving scene. We try to use the existing generative models to
provide more supervision information, but they struggle to generalize well in
cars since they are trained on synthetic datasets not car-specific. In
addition, The reconstructed 3D car texture misaligns due to a large error in
camera pose estimation when dealing with in-the-wild images. These restrictions
make it challenging for previous methods to reconstruct complete 3D cars. To
address these problems, we propose a novel method, named DreamCar, which can
reconstruct high-quality 3D cars given a few images even a single image. To
generalize the generative model, we collect a car dataset, named Car360, with
over 5,600 vehicles. With this dataset, we make the generative model more
robust to cars. We use this generative prior specific to the car to guide its
reconstruction via Score Distillation Sampling. To further complement the
supervision information, we utilize the geometric and appearance symmetry of
cars. Finally, we propose a pose optimization method that rectifies poses to
tackle texture misalignment. Extensive experiments demonstrate that our method
significantly outperforms existing methods in reconstructing high-quality 3D
cars. https://xiaobiaodu.github.io/dreamcar-project/{Our code is
available.}