Orionnnn/hub-stats huggingface dataset:
GLiNER multi-task: Generalist Lightweight Model for Various Information
  Extraction Tasks
Authors: name: Ihor Stepanov, name: Mykhailo Shtopko
Published: 2024-06-14 13:54:29

Information extraction tasks require both accurate, efficient, and
generalisable models. Classical supervised deep learning approaches can achieve
the required performance, but they need large datasets and are limited in their
ability to adapt to different tasks. On the other hand, large language models
(LLMs) demonstrate good generalization, meaning that they can adapt to many
different tasks based on user requests. However, LLMs are computationally
expensive and tend to fail to generate structured outputs. In this article, we
will introduce a new kind of GLiNER model that can be used for various
information extraction tasks while being a small encoder model. Our model
achieved SoTA performance on zero-shot NER benchmarks and leading performance
on question-answering, summarization and relation extraction tasks.
Additionally, in this article, we will cover experimental results on
self-learning approaches for named entity recognition using GLiNER models.
Iterative Length-Regularized Direct Preference Optimization: A Case
  Study on Improving 7B Language Models to GPT-4 Level
Authors: name: Jie Liu, name: Zhanhui Zhou, name: Jiaheng Liu, name: Xingyuan Bu, name: Chao Yang, name: Han-Sen Zhong, name: Wanli Ouyang
Published: 2024-06-17 17:55:38

Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a 50.5% length-controlled win
rate against GPT-4 Preview on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.
Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities
Authors: name: Sachit Menon, name: Richard Zemel, name: Carl Vondrick
Published: 2024-06-20 17:59:45

When presented with questions involving visual thinking, humans naturally
switch reasoning modalities, often forming mental images or drawing visual
aids. Large language models have shown promising results in arithmetic and
symbolic reasoning by expressing intermediate reasoning in text as a chain of
thought, yet struggle to extend this capability to answer text queries that are
easily solved by visual reasoning, even with extensive multimodal pretraining.
We introduce a simple method, whiteboard-of-thought prompting, to unlock the
visual reasoning capabilities of multimodal large language models across
modalities. Whiteboard-of-thought prompting provides multimodal large language
models with a metaphorical `whiteboard' to draw out reasoning steps as images,
then returns these images back to the model for further processing. We find
this can be accomplished with no demonstrations or specialized modules, instead
leveraging models' existing ability to write code with libraries such as
Matplotlib and Turtle. This simple approach shows state-of-the-art results on
four difficult natural language tasks that involve visual and spatial
reasoning. We identify multiple settings where GPT-4o using chain-of-thought
fails dramatically, including more than one where it achieves 0% accuracy,
while whiteboard-of-thought enables up to 92% accuracy in these same
settings. We present a detailed exploration of where the technique succeeds as
well as its sources of error.
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video
  Understanding
Authors: name: Xinyu Fang, name: Kangrui Mao, name: Haodong Duan, name: Xiangyu Zhao, name: Yining Li, name: Dahua Lin, name: Kai Chen
Published: 2024-06-20 17:26:01

The advent of large vision-language models (LVLMs) has spurred research into
their applications in multi-modal contexts, particularly in video
understanding. Traditional VideoQA benchmarks, despite providing quantitative
metrics, often fail to encompass the full spectrum of video content and
inadequately assess models' temporal comprehension. To address these
limitations, we introduce MMBench-Video, a quantitative benchmark designed to
rigorously evaluate LVLMs' proficiency in video understanding. MMBench-Video
incorporates lengthy videos from YouTube and employs free-form questions,
mirroring practical use cases. The benchmark is meticulously crafted to probe
the models' temporal reasoning skills, with all questions human-annotated
according to a carefully constructed ability taxonomy. We employ GPT-4 for
automated assessment, demonstrating superior accuracy and robustness over
earlier LLM-based evaluations. Utilizing MMBench-Video, we have conducted
comprehensive evaluations that include both proprietary and open-source LVLMs
for images and videos. MMBench-Video stands as a valuable resource for the
research community, facilitating improved evaluation of LVLMs and catalyzing
progress in the field of video understanding. The evalutation code of
MMBench-Video will be integrated into VLMEvalKit:
https://github.com/open-compass/VLMEvalKit.
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Authors: name: Yuxuan Qiao, name: Haodong Duan, name: Xinyu Fang, name: Junming Yang, name: Lin Chen, name: Songyang Zhang, name: Jiaqi Wang, name: Dahua Lin, name: Kai Chen
Published: 2024-06-20 17:54:03

Vision Language Models (VLMs) demonstrate remarkable proficiency in
addressing a wide array of visual questions, which requires strong perception
and reasoning faculties. Assessing these two competencies independently is
crucial for model refinement, despite the inherent difficulty due to the
intertwined nature of seeing and reasoning in existing VLMs. To tackle this
issue, we present Prism, an innovative framework designed to disentangle the
perception and reasoning processes involved in visual question solving. Prism
comprises two distinct stages: a perception stage that utilizes a VLM to
extract and articulate visual information in textual form, and a reasoning
stage that formulates responses based on the extracted visual information using
a Large Language Model (LLM). This modular design enables the systematic
comparison and assessment of both proprietary and open-source VLM for their
perception and reasoning strengths. Our analytical framework provides several
valuable insights, underscoring Prism's potential as a cost-effective solution
for vision-language tasks. By combining a streamlined VLM focused on perception
with a powerful LLM tailored for reasoning, Prism achieves superior results in
general vision-language tasks while substantially cutting down on training and
operational expenses. Quantitative evaluations show that Prism, when configured
with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
The project is released at: https://github.com/SparksJoe/Prism.
PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
  Documents
Authors: name: Junjie Wang, name: Yin Zhang, name: Yatai Ji, name: Yuxiang Zhang, name: Chunyang Jiang, name: Yubo Wang, name: Kang Zhu, name: Zekun Wang, name: Tiezhen Wang, name: Wenhao Huang, name: Jie Fu, name: Bei Chen, name: Qunshu Lin, name: Minghao Liu, name: Ge Zhang, name: Wenhu Chen
Published: 2024-06-20 01:43:08

Recent advancements in Large Multimodal Models (LMMs) have leveraged
extensive multimodal datasets to enhance capabilities in complex
knowledge-driven tasks. However, persistent challenges in perceptual and
reasoning errors limit their efficacy, particularly in interpreting intricate
visual data and deducing multimodal relationships. Addressing these issues, we
introduce a novel dataset format, PIN (Paired and INterleaved multimodal
documents), designed to significantly improve both the depth and breadth of
multimodal training. The PIN format is built on three foundational principles:
knowledge intensity, scalability, and support for diverse training modalities.
This innovative format combines markdown files and comprehensive images to
enrich training data with a dense knowledge structure and versatile training
strategies. We present PIN-14M, an open-source dataset comprising 14 million
samples derived from a diverse range of Chinese and English sources, tailored
to include complex web and scientific content. This dataset is constructed
meticulously to ensure data quality and ethical integrity, aiming to facilitate
advanced training strategies and improve model robustness against common
multimodal training pitfalls. Our initial results, forming the basis of this
technical report, suggest significant potential for the PIN format in refining
LMM performance, with plans for future expansions and detailed evaluations of
its impact on model capabilities.
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
  Reinforcement Learning
Authors: name: Hao Bai, name: Yifei Zhou, name: Mert Cemri, name: Jiayi Pan, name: Alane Suhr, name: Sergey Levine, name: Aviral Kumar
Published: 2024-06-14 17:49:55

Training corpuses for vision language models (VLMs) typically lack sufficient
amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
for decision-making tasks such as in-the-wild device control through graphical
user interfaces (GUIs). While training with static demonstrations has shown
some promise, we show that such methods fall short for controlling real GUIs
due to their failure to deal with real-world stochasticity and non-stationarity
not captured in static observational data. This paper introduces a novel
autonomous RL approach, called DigiRL, for training in-the-wild device control
agents through fine-tuning a pre-trained VLM in two stages: offline RL to
initialize the model, followed by offline-to-online RL. To do this, we build a
scalable and parallelizable Android learning environment equipped with a
VLM-based evaluator and develop a simple yet effective RL approach for learning
in this domain. Our approach runs advantage-weighted RL with advantage
estimators enhanced to account for stochasticity along with an automatic
curriculum for deriving maximal learning signal. We demonstrate the
effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
67.2% success rate -- over supervised fine-tuning with static human
demonstration data. These results significantly surpass not only the prior best
agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
trained with AitW data (38.5%), but also the prior best autonomous RL approach
based on filtered behavior cloning (57.8%), thereby establishing a new
state-of-the-art for digital agents for in-the-wild device control.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World
  Domains
Authors: name: Shunyu Yao, name: Noah Shinn, name: Pedram Razavi, name: Karthik Narasimhan
Published: 2024-06-17 19:33:08

Existing benchmarks do not test language agents on their interaction with
human users or ability to follow domain-specific rules, both of which are vital
for deploying them in real world applications. We propose tau-bench, a
benchmark emulating dynamic conversations between a user (simulated by language
models) and a language agent provided with domain-specific API tools and policy
guidelines. We employ an efficient and faithful evaluation process that
compares the database state at the end of a conversation with the annotated
goal state. We also propose a new metric (pass^k) to evaluate the reliability
of agent behavior over multiple trials. Our experiments show that even
state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the
tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point
to the need for methods that can improve the ability of agents to act
consistently and follow rules reliably.
REPOEXEC: Evaluate Code Generation with a Repository-Level Executable
  Benchmark
Authors: name: Nam Le Hai, name: Dung Manh Nguyen, name: Nghi D. Q. Bui
Published: 2024-06-17 10:45:22

The ability of CodeLLMs to generate executable and functionally correct code
at the repository-level scale remains largely unexplored. We introduce
RepoExec, a novel benchmark for evaluating code generation at the
repository-level scale. RepoExec focuses on three main aspects: executability,
functional correctness through automated test case generation with high
coverage rate, and carefully crafted cross-file contexts to accurately generate
code. Our work explores a controlled scenario where developers specify
necessary code dependencies, challenging the model to integrate these
accurately. Experiments show that while pretrained LLMs outperform
instruction-tuned models in correctness, the latter excel in utilizing provided
dependencies and demonstrating debugging capabilities. We also introduce a new
instruction-tuned dataset that focuses on code dependencies and demonstrate
that CodeLLMs fine-tuned on our dataset have a better capability to leverage
these dependencies effectively. RepoExec aims to provide a comprehensive
evaluation of code functionality and alignment with developer intent, paving
the way for more reliable and applicable CodeLLMs in real-world scenarios. The
dataset and source code can be found
at~https://github.com/FSoft-AI4Code/RepoExec.
ExVideo: Extending Video Diffusion Models via Parameter-Efficient
  Post-Tuning
Authors: name: Zhongjie Duan, name: Wenmeng Zhou, name: Cen Chen, name: Yaliang Li, name: Weining Qian
Published: 2024-06-20 09:18:54

Recently, advancements in video synthesis have attracted significant
attention. Video synthesis models such as AnimateDiff and Stable Video
Diffusion have demonstrated the practical applicability of diffusion models in
creating dynamic visual content. The emergence of SORA has further spotlighted
the potential of video generation technologies. Nonetheless, the extension of
video lengths has been constrained by the limitations in computational
resources. Most existing video synthesis models can only generate short video
clips. In this paper, we propose a novel post-tuning methodology for video
synthesis models, called ExVideo. This approach is designed to enhance the
capability of current video synthesis models, allowing them to produce content
over extended temporal durations while incurring lower training expenditures.
In particular, we design extension strategies across common temporal model
architectures respectively, including 3D convolution, temporal attention, and
positional embedding. To evaluate the efficacy of our proposed post-tuning
approach, we conduct extension training on the Stable Video Diffusion model.
Our approach augments the model's capacity to generate up to 5times its
original number of frames, requiring only 1.5k GPU hours of training on a
dataset comprising 40k videos. Importantly, the substantial increase in video
length doesn't compromise the model's innate generalization capabilities, and
the model showcases its advantages in generating videos of diverse styles and
resolutions. We will release the source code and the enhanced model publicly.
Instruction Pre-Training: Language Models are Supervised Multitask
  Learners
Authors: name: Daixuan Cheng, name: Yuxian Gu, name: Shaohan Huang, name: Junyu Bi, name: Minlie Huang, name: Furu Wei
Published: 2024-06-20 16:55:33

Unsupervised multitask pre-training has been the critical method behind the
recent success of language models (LMs). However, supervised multitask learning
still holds significant promise, as scaling it in the post-training stage
trends towards better generalization. In this paper, we explore supervised
multitask pre-training by proposing Instruction Pre-Training, a framework that
scalably augments massive raw corpora with instruction-response pairs to
pre-train LMs. The instruction-response pairs are generated by an efficient
instruction synthesizer built on open-source models. In our experiments, we
synthesize 200M instruction-response pairs covering 40+ task categories to
verify the effectiveness of Instruction Pre-Training. In pre-training from
scratch, Instruction Pre-Training not only consistently enhances pre-trained
base models but also benefits more from further instruction tuning. In
continual pre-training, Instruction Pre-Training enables Llama3-8B to be
comparable to or even outperform Llama3-70B. Our model, code, and data are
available at https://github.com/microsoft/LMOps.
Measuring memorization in RLHF for code completion
Authors: name: Aneesh Pappu, name: Billy Porter, name: Ilia Shumailov, name: Jamie Hayes
Published: 2024-06-17 16:33:35

Reinforcement learning with human feedback (RLHF) has become the dominant
method to align large models to user preferences. Unlike fine-tuning, for which
there are many studies regarding training data memorization, it is not clear
how memorization is affected by or introduced in the RLHF alignment process.
Understanding this relationship is important as real user data may be collected
and used to align large models; if user data is memorized during RLHF and later
regurgitated, this could raise privacy concerns. In this work, we analyze how
training data memorization can surface and propagate through each phase of
RLHF. We focus our study on code completion models, as code completion is one
of the most popular use cases for large language models. We find that RLHF
significantly decreases the chance that data used for reward modeling and
reinforcement learning is memorized, in comparison to aligning via directly
fine-tuning on this data, but that examples already memorized during the
fine-tuning stage of RLHF, will, in the majority of cases, remain memorized
after RLHF.
Intrinsic Evaluation of Unlearning Using Parametric Knowledge Traces
Authors: name: Yihuai Hong, name: Lei Yu, name: Shauli Ravfogel, name: Haiqin Yang, name: Mor Geva
Published: 2024-06-17 15:00:35

The task of "unlearning" certain concepts in large language models (LLMs) has
attracted immense attention recently, due to its importance for mitigating
undesirable model behaviours, such as the generation of harmful, private, or
incorrect information. Current protocols to evaluate unlearning methods largely
rely on behavioral tests, without monitoring the presence of unlearned
knowledge within the model's parameters. This residual knowledge can be
adversarially exploited to recover the erased information post-unlearning. We
argue that unlearning should also be evaluated internally, by considering
changes in the parametric knowledge traces of the unlearned concepts. To this
end, we propose a general methodology for eliciting directions in the parameter
space (termed "concept vectors") that encode concrete concepts, and construct
ConceptVectors, a benchmark dataset containing hundreds of common concepts and
their parametric knowledge traces within two open-source LLMs. Evaluation on
ConceptVectors shows that existing unlearning methods minimally impact concept
vectors, while directly ablating these vectors demonstrably removes the
associated knowledge from the LLMs and significantly reduces their
susceptibility to adversarial manipulation. Our results highlight limitations
in behavioral-based unlearning evaluations and call for future work to include
parametric-based evaluations. To support this, we release our code and
benchmark at https://github.com/yihuaihong/ConceptVectors.
Interface Design for Self-Supervised Speech Models
Authors: name: Yi-Jen Shih, name: David Harwath
Published: 2024-06-18 02:13:48

Self-supervised speech (SSL) models have recently become widely adopted for
many downstream speech processing tasks. The general usage pattern is to employ
SSL models as feature extractors, and then train a downstream prediction head
to solve a specific task. However, different layers of SSL models have been
shown to capture different types of information, and the methods of combining
them are not well studied. To this end, we extend the general framework for SSL
model utilization by proposing the interface that connects the upstream and
downstream. Under this view, the dominant technique of combining features via a
layerwise weighted sum can be regarded as a specific interface. We propose
several alternative interface designs and demonstrate that the weighted sum
interface is suboptimal for many tasks. In particular, we show that a
convolutional interface whose depth scales logarithmically with the depth of
the upstream model consistently outperforms many other interface designs.
Breaking Boundaries: Investigating the Effects of Model Editing on
  Cross-linguistic Performance
Authors: name: Somnath Banerjee, name: Avik Halder, name: Rajarshi Mandal, name: Sayan Layek, name: Ian Soboroff, name: Rima Hazra, name: Animesh Mukherjee
Published: 2024-06-17 01:54:27

The integration of pretrained language models (PLMs) like BERT and GPT has
revolutionized NLP, particularly for English, but it has also created
linguistic imbalances. This paper strategically identifies the need for
linguistic equity by examining several knowledge editing techniques in
multilingual contexts. We evaluate the performance of models such as Mistral,
TowerInstruct, OpenHathi, Tamil-Llama, and Kan-Llama across languages including
English, German, French, Italian, Spanish, Hindi, Tamil, and Kannada. Our
research identifies significant discrepancies in normal and merged models
concerning cross-lingual consistency. We employ strategies like 'each language
for itself' (ELFI) and 'each language for others' (ELFO) to stress-test these
models. Our findings demonstrate the potential for LLMs to overcome linguistic
barriers, laying the groundwork for future research in achieving linguistic
inclusivity in AI technologies.
Self-MoE: Towards Compositional Large Language Models with
  Self-Specialized Experts
Authors: name: Junmo Kang, name: Leonid Karlinsky, name: Hongyin Luo, name: Zhen Wang, name: Jacob Hansen, name: James Glass, name: David Cox, name: Rameswar Panda, name: Rogerio Feris, name: Alan Ritter
Published: 2024-06-17 19:06:54

We present Self-MoE, an approach that transforms a monolithic LLM into a
compositional, modular system of self-specialized experts, named MiXSE (MiXture
of Self-specialized Experts). Our approach leverages self-specialization, which
constructs expert modules using self-generated synthetic data, each equipped
with a shared base LLM and incorporating self-optimized routing. This allows
for dynamic and capability-specific handling of various target tasks, enhancing
overall capabilities, without extensive human-labeled data and added
parameters. Our empirical results reveal that specializing LLMs may exhibit
potential trade-offs in performances on non-specialized tasks. On the other
hand, our Self-MoE demonstrates substantial improvements over the base LLM
across diverse benchmarks such as knowledge, reasoning, math, and coding. It
also consistently outperforms other methods, including instance merging and
weight merging, while offering better flexibility and interpretability by
design with semantic experts and routing. Our findings highlight the critical
role of modularity and the potential of self-improvement in achieving
efficient, scalable, and adaptable systems.