Orionnnn/hub-stats huggingface dataset:
UCFE: A User-Centric Financial Expertise Benchmark for Large Language
  Models
Authors: name: Yuzhe Yang, name: Yifei Zhang, name: Yan Hu, name: Yilin Guo, name: Ruoli Gan, name: Yueru He, name: Mingcong Lei, name: Xiao Zhang, name: Haining Wang, name: Qianqian Xie, name: Jimin Huang, name: Honghai Yu, name: Benyou Wang
Published: 2024-10-17 22:03:52

This paper introduces the UCFE: User-Centric Financial Expertise benchmark,
an innovative framework designed to evaluate the ability of large language
models (LLMs) to handle complex real-world financial tasks. UCFE benchmark
adopts a hybrid approach that combines human expert evaluations with dynamic,
task-specific interactions to simulate the complexities of evolving financial
scenarios. Firstly, we conducted a user study involving 804 participants,
collecting their feedback on financial tasks. Secondly, based on this feedback,
we created our dataset that encompasses a wide range of user intents and
interactions. This dataset serves as the foundation for benchmarking 12 LLM
services using the LLM-as-Judge methodology. Our results show a significant
alignment between benchmark scores and human preferences, with a Pearson
correlation coefficient of 0.78, confirming the effectiveness of the UCFE
dataset and our evaluation approach. UCFE benchmark not only reveals the
potential of LLMs in the financial sector but also provides a robust framework
for assessing their performance and user satisfaction.The benchmark dataset and
evaluation code are available.
DPLM-2: A Multimodal Diffusion Protein Language Model
Authors: name: Xinyou Wang, name: Zaixiang Zheng, name: Fei Ye, name: Dongyu Xue, name: Shujian Huang, name: Quanquan Gu
Published: 2024-10-17 17:20:24

Proteins are essential macromolecules defined by their amino acid sequences,
which determine their three-dimensional structures and, consequently, their
functions in all living organisms. Therefore, generative protein modeling
necessitates a multimodal approach to simultaneously model, understand, and
generate both sequences and structures. However, existing methods typically use
separate models for each modality, limiting their ability to capture the
intricate relationships between sequence and structure. This results in
suboptimal performance in tasks that requires joint understanding and
generation of both modalities. In this paper, we introduce DPLM-2, a multimodal
protein foundation model that extends discrete diffusion protein language model
(DPLM) to accommodate both sequences and structures. To enable structural
learning with the language model, 3D coordinates are converted to discrete
tokens using a lookup-free quantization-based tokenizer. By training on both
experimental and high-quality synthetic structures, DPLM-2 learns the joint
distribution of sequence and structure, as well as their marginals and
conditionals. We also implement an efficient warm-up strategy to exploit the
connection between large-scale evolutionary data and structural inductive
biases from pre-trained sequence-based protein language models. Empirical
evaluation shows that DPLM-2 can simultaneously generate highly compatible
amino acid sequences and their corresponding 3D structures eliminating the need
for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive
performance in various conditional generation tasks, including folding, inverse
folding, and scaffolding with multimodal motif inputs, as well as providing
structure-aware representations for predictive tasks.
Are AI Detectors Good Enough? A Survey on Quality of Datasets With
  Machine-Generated Texts
Authors: name: German Gritsai, name: Anastasia Voznyuk, name: Andrey Grabovoy, name: Yury Chekhovich
Published: 2024-10-18 17:59:57

The rapid development of autoregressive Large Language Models (LLMs) has
significantly improved the quality of generated texts, necessitating reliable
machine-generated text detectors. A huge number of detectors and collections
with AI fragments have emerged, and several detection methods even showed
recognition quality up to 99.9% according to the target metrics in such
collections. However, the quality of such detectors tends to drop dramatically
in the wild, posing a question: Are detectors actually highly trustworthy or do
their high benchmark scores come from the poor quality of evaluation datasets?
In this paper, we emphasise the need for robust and qualitative methods for
evaluating generated data to be secure against bias and low generalising
ability of future model. We present a systematic review of datasets from
competitions dedicated to AI-generated content detection and propose methods
for evaluating the quality of datasets containing AI-generated fragments. In
addition, we discuss the possibility of using high-quality generated data to
achieve two goals: improving the training of detection models and improving the
training datasets themselves. Our contribution aims to facilitate a better
understanding of the dynamics between human and machine text, which will
ultimately support the integrity of information in an increasingly automated
world.
SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
Authors: name: Yizhao Gao, name: Zhichen Zeng, name: Dayou Du, name: Shijie Cao, name: Hayden Kwok-Hay So, name: Ting Cao, name: Fan Yang, name: Mao Yang
Published: 2024-10-17 07:07:09

Attention is the cornerstone of modern Large Language Models (LLMs). Yet its
quadratic complexity limits the efficiency and scalability of LLMs, especially
for those with a long-context window. A promising approach addressing this
limitation is to leverage the sparsity in attention. However, existing
sparsity-based solutions predominantly rely on predefined patterns or
heuristics to approximate sparsity. This practice falls short to fully capture
the dynamic nature of attention sparsity in language-based tasks. This paper
argues that attention sparsity should be learned rather than predefined. To
this end, we design SeerAttention, a new Attention mechanism that augments the
conventional attention with a learnable gate that adaptively selects
significant blocks in an attention map and deems the rest blocks sparse. Such
block-level sparsity effectively balances accuracy and speedup. To enable
efficient learning of the gating network, we develop a customized
FlashAttention implementation that extracts the block-level ground truth of
attention map with minimum overhead. SeerAttention not only applies to
post-training, but also excels in long-context fine-tuning. Our results show
that at post-training stages, SeerAttention significantly outperforms
state-of-the-art static or heuristic-based sparse attention methods, while also
being more versatile and flexible to adapt to varying context lengths and
sparsity ratios. When applied to long-context fine-tuning with YaRN,
SeerAttention can achieve a remarkable 90% sparsity ratio at a 32k context
length with minimal perplexity loss, offering a 5.67x speedup over
FlashAttention-2.
Context is Key(NMF): Modelling Topical Information Dynamics in Chinese
  Diaspora Media
Authors: name: Ross Deans Kristensen-McLachlan, name: Rebecca M. M. Hicke, name: Márton Kardos, name: Mette Thunø
Published: 2024-10-16 17:59:52

Does the People's Republic of China (PRC) interfere with European elections
through ethnic Chinese diaspora media? This question forms the basis of an
ongoing research project exploring how PRC narratives about European elections
are represented in Chinese diaspora media, and thus the objectives of PRC news
media manipulation. In order to study diaspora media efficiently and at scale,
it is necessary to use techniques derived from quantitative text analysis, such
as topic modelling. In this paper, we present a pipeline for studying
information dynamics in Chinese media. Firstly, we present KeyNMF, a new
approach to static and dynamic topic modelling using transformer-based
contextual embedding models. We provide benchmark evaluations to demonstrate
that our approach is competitive on a number of Chinese datasets and metrics.
Secondly, we integrate KeyNMF with existing methods for describing information
dynamics in complex systems. We apply this pipeline to data from five news
sites, focusing on the period of time leading up to the 2024 European
parliamentary elections. Our methods and results demonstrate the effectiveness
of KeyNMF for studying information dynamics in Chinese media and lay groundwork
for further work addressing the broader research questions.
Web Agents with World Models: Learning and Leveraging Environment
  Dynamics in Web Navigation
Authors: name: Hyungjoo Chae, name: Namyoung Kim, name: Kai Tzu-iunn Ong, name: Minju Gwak, name: Gwanwoo Song, name: Jihoon Kim, name: Sunghwan Kim, name: Dongha Lee, name: Jinyoung Yeo
Published: 2024-10-17 05:37:00

Large language models (LLMs) have recently gained much attention in building
autonomous agents. However, the performance of current LLM-based web agents in
long-horizon tasks is far from optimal, often yielding errors such as
repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid
such an irreversible mistake, as we have an awareness of the potential outcomes
(e.g., losing money) of our actions, also known as the "world model". Motivated
by this, our study first starts with preliminary analyses, confirming the
absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet,
etc.). Then, we present a World-model-augmented (WMA) web agent, which
simulates the outcomes of its actions for better decision-making. To overcome
the challenges in training LLMs as world models predicting next observations,
such as repeated elements across observations and long HTML inputs, we propose
a transition-focused observation abstraction, where the prediction objectives
are free-form natural language descriptions exclusively highlighting important
state differences between time steps. Experiments on WebArena and Mind2Web show
that our world models improve agents' policy selection without training and
demonstrate our agents' cost- and time-efficiency compared to recent
tree-search-based agents.
Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning
  via Image-Guided Diffusion
Authors: name: Yijun Liang, name: Shweta Bhardwaj, name: Tianyi Zhou
Published: 2024-10-17 15:33:35

Low-quality or scarce data has posed significant challenges for training deep
neural networks in practice. While classical data augmentation cannot
contribute very different new data, diffusion models opens up a new door to
build self-evolving AI by generating high-quality and diverse synthetic data
through text-guided prompts. However, text-only guidance cannot control
synthetic images' proximity to the original images, resulting in
out-of-distribution data detrimental to the model performance. To overcome the
limitation, we study image guidance to achieve a spectrum of interpolations
between synthetic and real images. With stronger image guidance, the generated
images are similar to the training data but hard to learn. While with weaker
image guidance, the synthetic images will be easier for model but contribute to
a larger distribution gap with the original data. The generated full spectrum
of data enables us to build a novel "Diffusion Curriculum (DisCL)". DisCL
adjusts the image guidance level of image synthesis for each training stage: It
identifies and focuses on hard samples for the model and assesses the most
effective guidance level of synthetic images to improve hard data learning. We
apply DisCL to two challenging tasks: long-tail (LT) classification and
learning from low-quality data. It focuses on lower-guidance images of
high-quality to learn prototypical features as a warm-up of learning
higher-guidance images that might be weak on diversity or quality. Extensive
experiments showcase a gain of 2.7% and 2.1% in OOD and ID macro-accuracy when
applying DisCL to iWildCam dataset. On ImageNet-LT, DisCL improves the base
model's tail-class accuracy from 4.4% to 23.64% and leads to a 4.02%
improvement in all-class accuracy.
MagicTailor: Component-Controllable Personalization in Text-to-Image
  Diffusion Models
Authors: name: Donghao Zhou, name: Jiancheng Huang, name: Jinbin Bai, name: Jiaze Wang, name: Hao Chen, name: Guangyong Chen, name: Xiaowei Hu, name: Pheng-Ann Heng
Published: 2024-10-17 09:22:53

Recent advancements in text-to-image (T2I) diffusion models have enabled the
creation of high-quality images from text prompts, but they still struggle to
generate images with precise control over specific visual concepts. Existing
approaches can replicate a given concept by learning from reference images, yet
they lack the flexibility for fine-grained customization of the individual
component within the concept. In this paper, we introduce
component-controllable personalization, a novel task that pushes the boundaries
of T2I models by allowing users to reconfigure specific components when
personalizing visual concepts. This task is particularly challenging due to two
primary obstacles: semantic pollution, where unwanted visual elements corrupt
the personalized concept, and semantic imbalance, which causes disproportionate
learning of the concept and component. To overcome these challenges, we design
MagicTailor, an innovative framework that leverages Dynamic Masked Degradation
(DM-Deg) to dynamically perturb undesired visual semantics and Dual-Stream
Balancing (DS-Bal) to establish a balanced learning paradigm for desired visual
semantics. Extensive comparisons, ablations, and analyses demonstrate that
MagicTailor not only excels in this challenging task but also holds significant
promise for practical applications, paving the way for more nuanced and
creative image generation.
A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
Authors: name: Siwei Wu, name: Zhongyuan Peng, name: Xinrun Du, name: Tuney Zheng, name: Minghao Liu, name: Jialong Wu, name: Jiachen Ma, name: Yizhi Li, name: Jian Yang, name: Wangchunshu Zhou, name: Qunshu Lin, name: Junbo Zhao, name: Zhaoxiang Zhang, name: Wenhao Huang, name: Ge Zhang, name: Chenghua Lin, name: J. H. Liu
Published: 2024-10-17 15:09:03

Enabling Large Language Models (LLMs) to handle a wider range of complex
tasks (e.g., coding, math) has drawn great attention from many researchers. As
LLMs continue to evolve, merely increasing the number of model parameters
yields diminishing performance improvements and heavy computational costs.
Recently, OpenAI's o1 model has shown that inference strategies (i.e.,
Test-time Compute methods) can also significantly enhance the reasoning
capabilities of LLMs. However, the mechanisms behind these methods are still
unexplored. In our work, to investigate the reasoning patterns of o1, we
compare o1 with existing Test-time Compute methods (BoN, Step-wise BoN, Agent
Workflow, and Self-Refine) by using OpenAI's GPT-4o as a backbone on general
reasoning benchmarks in three domains (i.e., math, coding, commonsense
reasoning). Specifically, first, our experiments show that the o1 model has
achieved the best performance on most datasets. Second, as for the methods of
searching diverse responses (e.g., BoN), we find the reward models' capability
and the search space both limit the upper boundary of these methods. Third, as
for the methods that break the problem into many sub-problems, the Agent
Workflow has achieved better performance than Step-wise BoN due to the
domain-specific system prompt for planning better reasoning processes. Fourth,
it is worth mentioning that we have summarized six reasoning patterns of o1,
and provided a detailed analysis on several reasoning benchmarks.
Fluid: Scaling Autoregressive Text-to-image Generative Models with
  Continuous Tokens
Authors: name: Lijie Fan, name: Tianhong Li, name: Siyang Qin, name: Yuanzhen Li, name: Chen Sun, name: Michael Rubinstein, name: Deqing Sun, name: Kaiming He, name: Yonglong Tian
Published: 2024-10-17 17:59:59

Scaling up autoregressive models in vision has not proven as beneficial as in
large language models. In this work, we investigate this scaling problem in the
context of text-to-image generation, focusing on two critical factors: whether
models use discrete or continuous tokens, and whether tokens are generated in a
random or fixed raster order using BERT- or GPT-like transformer architectures.
Our empirical results show that, while all models scale effectively in terms of
validation loss, their evaluation performance -- measured by FID, GenEval
score, and visual quality -- follows different trends. Models based on
continuous tokens achieve significantly better visual quality than those using
discrete tokens. Furthermore, the generation order and attention mechanisms
significantly affect the GenEval score: random-order models achieve notably
better GenEval scores compared to raster-order models. Inspired by these
findings, we train Fluid, a random-order autoregressive model on continuous
tokens. Fluid 10.5B model achieves a new state-of-the-art zero-shot FID of 6.16
on MS-COCO 30K, and 0.69 overall score on the GenEval benchmark. We hope our
findings and results will encourage future efforts to further bridge the
scaling gap between vision and language models.
JudgeBench: A Benchmark for Evaluating LLM-based Judges
Authors: name: Sijun Tan, name: Siyuan Zhuang, name: Kyle Montgomery, name: William Y. Tang, name: Alejandro Cuadron, name: Chenguang Wang, name: Raluca Ada Popa, name: Ion Stoica
Published: 2024-10-16 17:58:19

LLM-based judges have emerged as a scalable alternative to human evaluation
and are increasingly used to assess, compare, and improve models. However, the
reliability of LLM-based judges themselves is rarely scrutinized. As LLMs
become more advanced, their responses grow more sophisticated, requiring
stronger judges to evaluate them. Existing benchmarks primarily focus on a
judge's alignment with human preferences, but often fail to account for more
challenging tasks where crowdsourced human preference is a poor indicator of
factual and logical correctness. To address this, we propose a novel evaluation
framework to objectively evaluate LLM-based judges. Based on this framework, we
propose JudgeBench, a benchmark for evaluating LLM-based judges on challenging
response pairs spanning knowledge, reasoning, math, and coding. JudgeBench
leverages a novel pipeline for converting existing difficult datasets into
challenging response pairs with preference labels reflecting objective
correctness. Our comprehensive evaluation on a collection of prompted judges,
fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench
poses a significantly greater challenge than previous benchmarks, with many
strong models (e.g., GPT-4o) performing just slightly better than random
guessing. Overall, JudgeBench offers a reliable platform for assessing
increasingly advanced LLM-based judges. Data and code are available at
https://github.com/ScalerLab/JudgeBench .
WorldCuisines: A Massive-Scale Benchmark for Multilingual and
  Multicultural Visual Question Answering on Global Cuisines
Authors: name: Genta Indra Winata, name: Frederikus Hudi, name: Patrick Amadeus Irawan, name: David Anugraha, name: Rifki Afina Putri, name: Yutong Wang, name: Adam Nohejl, name: Ubaidillah Ariq Prathama, name: Nedjma Ousidhoum, name: Afifa Amriani, name: Anar Rzayev, name: Anirban Das, name: Ashmari Pramodya, name: Aulia Adila, name: Bryan Wilie, name: Candy Olivia Mawalim, name: Ching Lam Cheng, name: Daud Abolade, name: Emmanuele Chersoni, name: Enrico Santus, name: Fariz Ikhwantri, name: Garry Kuwanto, name: Hanyang Zhao, name: Haryo Akbarianto Wibowo, name: Holy Lovenia, name: Jan Christian Blaise Cruz, name: Jan Wira Gotama Putra, name: Junho Myung, name: Lucky Susanto, name: Maria Angelica Riera Machin, name: Marina Zhukova, name: Michael Anugraha, name: Muhammad Farid Adilazuarda, name: Natasha Santosa, name: Peerat Limkonchotiwat, name: Raj Dabre, name: Rio Alexander Audino, name: Samuel Cahyawijaya, name: Shi-Xiong Zhang, name: Stephanie Yulia Salim, name: Yi Zhou, name: Yinxuan Gui, name: David Ifeoluwa Adelani, name: En-Shiun Annie Lee, name: Shogo Okada, name: Ayu Purwarianti, name: Alham Fikri Aji, name: Taro Watanabe, name: Derry Tanti Wijaya, name: Alice Oh, name: Chong-Wah Ngo
Published: 2024-10-16 16:11:49

Vision Language Models (VLMs) often struggle with culture-specific knowledge,
particularly in languages other than English and in underrepresented cultural
contexts. To evaluate their understanding of such knowledge, we introduce
WorldCuisines, a massive-scale benchmark for multilingual and multicultural,
visually grounded language understanding. This benchmark includes a visual
question answering (VQA) dataset with text-image pairs across 30 languages and
dialects, spanning 9 language families and featuring over 1 million data
points, making it the largest multicultural VQA benchmark to date. It includes
tasks for identifying dish names and their origins. We provide evaluation
datasets in two sizes (12k and 60k instances) alongside a training dataset (1
million instances). Our findings show that while VLMs perform better with
correct location context, they struggle with adversarial contexts and
predicting specific regional cuisines and languages. To support future
research, we release a knowledge base with annotated food entries and images
along with the VQA data.
Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models
Authors: name: Luis Barroso-Luque, name: Muhammed Shuaibi, name: Xiang Fu, name: Brandon M. Wood, name: Misko Dzamba, name: Meng Gao, name: Ammar Rizvi, name: C. Lawrence Zitnick, name: Zachary W. Ulissi
Published: 2024-10-16 17:48:34

The ability to discover new materials with desirable properties is critical
for numerous applications from helping mitigate climate change to advances in
next generation computing hardware. AI has the potential to accelerate
materials discovery and design by more effectively exploring the chemical space
compared to other computational methods or by trial-and-error. While
substantial progress has been made on AI for materials data, benchmarks, and
models, a barrier that has emerged is the lack of publicly available training
data and open pre-trained models. To address this, we present a Meta FAIR
release of the Open Materials 2024 (OMat24) large-scale open dataset and an
accompanying set of pre-trained models. OMat24 contains over 110 million
density functional theory (DFT) calculations focused on structural and
compositional diversity. Our EquiformerV2 models achieve state-of-the-art
performance on the Matbench Discovery leaderboard and are capable of predicting
ground-state stability and formation energies to an F1 score above 0.9 and an
accuracy of 20 meV/atom, respectively. We explore the impact of model size,
auxiliary denoising objectives, and fine-tuning on performance across a range
of datasets including OMat24, MPtraj, and Alexandria. The open release of the
OMat24 dataset and models enables the research community to build upon our
efforts and drive further advancements in AI-assisted materials science.