Orionnnn/hub-stats huggingface dataset:
Retrieval-Enhanced Machine Learning: Synthesis and Opportunities
Authors: name: To Eun Kim, name: Alireza Salemi, name: Andrew Drozdov, name: Fernando Diaz, name: Hamed Zamani
Published: 2024-07-17 20:01:21

In the field of language modeling, models augmented with retrieval components
have emerged as a promising solution to address several challenges faced in the
natural language processing (NLP) field, including knowledge grounding,
interpretability, and scalability. Despite the primary focus on NLP, we posit
that the paradigm of retrieval-enhancement can be extended to a broader
spectrum of machine learning (ML) such as computer vision, time series
prediction, and computational biology. Therefore, this work introduces a formal
framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by
synthesizing the literature in various domains in ML with consistent notations
which is missing from the current literature. Also, we found that while a
number of studies employ retrieval components to augment their models, there is
a lack of integration with foundational Information Retrieval (IR) research. We
bridge this gap between the seminal IR research and contemporary REML studies
by investigating each component that comprises the REML framework. Ultimately,
the goal of this work is to equip researchers across various disciplines with a
comprehensive, formally structured framework of retrieval-enhanced models,
thereby fostering interdisciplinary future research.
Attention Overflow: Language Model Input Blur during Long-Context
  Missing Items Recommendation
Authors: name: Damien Sileo
Published: 2024-07-18 13:00:30

Large language models (LLMs) can suggest missing elements from items listed
in a prompt, which can be used for list completion or recommendations based on
users' history. However, their performance degrades when presented with too
many items, as they start to suggest items already included in the input list.
This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this
phenomenon on both synthetic problems (e.g., finding missing numbers in a given
range of shuffled integers) and realistic movie recommendation scenarios. We
refer to this issue as attention overflow, as preventing repetition
requires attending to all items simultaneously. Although iterative loops can
mitigate this problem, their costs increase with the repetition rate, affecting
the language models' ability to derive novelty from lengthy inputs.
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive
  Retrieval
Authors: name: Hongjin Su, name: Howard Yen, name: Mengzhou Xia, name: Weijia Shi, name: Niklas Muennighoff, name: Han-yu Wang, name: Haisu Liu, name: Quan Shi, name: Zachary S. Siegel, name: Michael Tang, name: Ruoxi Sun, name: Jinsung Yoon, name: Sercan O. Arik, name: Danqi Chen, name: Tao Yu
Published: 2024-07-16 17:58:27

Existing retrieval benchmarks primarily consist of information-seeking
queries (e.g., aggregated questions from search engines) where keyword or
semantic-based retrieval is usually sufficient. However, many complex
real-world queries require in-depth reasoning to identify relevant documents
that go beyond surface form matching. For example, finding documentation for a
coding question requires understanding the logic and syntax of the functions
involved. To better benchmark retrieval on such challenging queries, we
introduce BRIGHT, the first text retrieval benchmark that requires intensive
reasoning to retrieve relevant documents. BRIGHT is constructed from the 1,398
real-world queries collected from diverse domains (such as economics,
psychology, robotics, software engineering, earth sciences, etc.), sourced from
naturally occurring or carefully curated human data. Extensive evaluation
reveals that even state-of-the-art retrieval models perform poorly on BRIGHT.
The leading model on the MTEB leaderboard [38 ], which achieves a score of 59.0
nDCG@10,2 produces a score of nDCG@10 of 18.0 on BRIGHT. We further demonstrate
that augmenting queries with Chain-of-Thought reasoning generated by large
language models (LLMs) improves performance by up to 12.2 points. Moreover,
BRIGHT is robust against data leakage during pretraining of the benchmarked
models as we validate by showing similar performance even when documents from
the benchmark are included in the training data. We believe that BRIGHT paves
the way for future research on retrieval systems in more realistic and
challenging settings. Our code and data are available at
https://brightbenchmark.github.io.
Benchmarking Trustworthiness of Multimodal Large Language Models: A
  Comprehensive Study
Authors: name: Yichi Zhang, name: Yao Huang, name: Yitong Sun, name: Chang Liu, name: Zhe Zhao, name: Zhengwei Fang, name: Yifan Wang, name: Huanran Chen, name: Xiao Yang, name: Xingxing Wei, name: Hang Su, name: Yinpeng Dong, name: Jun Zhu
Published: 2024-06-11 08:38:13

Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
across diverse tasks, they still face significant trustworthiness challenges.
Yet, current literature on the assessment of trustworthy MLLMs remains limited,
lacking a holistic evaluation to offer thorough insights into future
improvements. In this work, we establish MultiTrust, the first comprehensive
and unified benchmark on the trustworthiness of MLLMs across five primary
aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
employs a rigorous evaluation strategy that addresses both multimodal risks and
cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
Extensive experiments with 21 modern MLLMs reveal some previously unexplored
trustworthiness issues and risks, highlighting the complexities introduced by
the multimodality and underscoring the necessity for advanced methodologies to
enhance their reliability. For instance, typical proprietary models still
struggle with the perception of visually confusing images and are vulnerable to
multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
disclose privacy in text and reveal ideological and cultural biases even when
paired with irrelevant images in inference, indicating that the multimodality
amplifies the internal risks from base LLMs. Additionally, we release a
scalable toolbox for standardized trustworthiness research, aiming to
facilitate future advancements in this important field. Code and resources are
publicly available at: https://multi-trust.github.io/.
PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining
  Tasks
Authors: name: Alessandro Berti, name: Humam Kourani, name: Wil M. P. van der Aalst
Published: 2024-07-18 07:57:31

Large Language Models (LLMs) have the potential to semi-automate some process
mining (PM) analyses. While commercial models are already adequate for many
analytics tasks, the competitive level of open-source LLMs in PM tasks is
unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
benchmark for PM focusing on domain knowledge (process-mining-specific and
process-specific) and on different implementation strategies. We focus also on
the challenges in creating such a benchmark, related to the public availability
of the data and on evaluation biases by the LLMs. Overall, we observe that most
of the considered LLMs can perform some process mining tasks at a satisfactory
level, but tiny models that would run on edge devices are still inadequate. We
also conclude that while the proposed benchmark is useful for identifying LLMs
that are adequate for process mining tasks, further research is needed to
overcome the evaluation biases and perform a more thorough ranking of the
competitive LLMs.
A Comparative Study on Automatic Coding of Medical Letters with
  Explainability
Authors: name: Jamie Glen, name: Lifeng Han, name: Paul Rayson, name: Goran Nenadic
Published: 2024-07-18 16:12:47

This study aims to explore the implementation of Natural Language Processing
(NLP) and machine learning (ML) techniques to automate the coding of medical
letters with visualised explainability and light-weighted local computer
settings. Currently in clinical settings, coding is a manual process that
involves assigning codes to each condition, procedure, and medication in a
patient's paperwork (e.g., 56265001 heart disease using SNOMED CT code). There
are preliminary research on automatic coding in this field using
state-of-the-art ML models; however, due to the complexity and size of the
models, the real-world deployment is not achieved. To further facilitate the
possibility of automatic coding practice, we explore some solutions in a local
computer setting; in addition, we explore the function of explainability for
transparency of AI models. We used the publicly available MIMIC-III database
and the HAN/HLAN network models for ICD code prediction purposes. We also
experimented with the mapping between ICD and SNOMED CT knowledge bases. In our
experiments, the models provided useful information for 97.98\% of codes. The
result of this investigation can shed some light on implementing automatic
clinical coding in practice, such as in hospital settings, on the local
computers used by clinicians , project page
https://github.com/Glenj01/Medical-Coding.
Benchmark Agreement Testing Done Right: A Guide for LLM Benchmark
  Evaluation
Authors: name: Yotam Perlitz, name: Ariel Gera, name: Ofir Arviv, name: Asaf Yehudai, name: Elron Bandel, name: Eyal Shnarch, name: Michal Shmueli-Scheuer, name: Leshem Choshen
Published: 2024-07-18 17:00:23

Recent advancements in Language Models (LMs) have catalyzed the creation of
multiple benchmarks, designed to assess these models' general capabilities. A
crucial task, however, is assessing the validity of the benchmarks themselves.
This is most commonly done via Benchmark Agreement Testing (BAT), where new
benchmarks are validated against established ones using some agreement metric
(e.g., rank correlation). Despite the crucial role of BAT for benchmark
builders and consumers, there are no standardized procedures for such agreement
testing. This deficiency can lead to invalid conclusions, fostering mistrust in
benchmarks and upending the ability to properly choose the appropriate
benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how
some overlooked methodological choices can significantly influence BAT results,
potentially undermining the validity of conclusions. To address these
inconsistencies, we propose a set of best practices for BAT and demonstrate how
utilizing these methodologies greatly improves BAT robustness and validity. To
foster adoption and facilitate future research,, we introduce BenchBench, a
python package for BAT, and release the BenchBench-leaderboard, a
meta-benchmark designed to evaluate benchmarks using their peers. Our findings
underscore the necessity for standardized BAT, ensuring the robustness and
validity of benchmark evaluations in the evolving landscape of language model
research.
  BenchBench Package: https://github.com/IBM/BenchBench
  Leaderboard: https://huggingface.co/spaces/per/BenchBench
Understanding Reference Policies in Direct Preference Optimization
Authors: name: Yixin Liu, name: Pengfei Liu, name: Arman Cohan
Published: 2024-07-18 17:08:10

Direct Preference Optimization (DPO) has become a widely used training method
for the instruction fine-tuning of large language models (LLMs). In this work,
we explore an under-investigated aspect of DPO - its dependency on the
reference model or policy. Such reference policies, typically instantiated as
the model to be further fine-tuned, are important since they can impose an
upper limit on DPO's effectiveness. Therefore, we address three related
research questions in this work. First, we explore the optimal strength of the
KL-divergence constraint in DPO, which penalizes deviations from the reference
policy, and find that DPO is sensitive to this strength. Next, we examine the
necessity of reference policies for instruction fine-tuning by providing both
theoretical and empirical comparisons between DPO and related learning
objectives, demonstrating DPO's superiority. Additionally, we investigate
whether DPO benefits from stronger reference policies, finding that a stronger
reference policy can lead to improved performance, but only when it is similar
to the model being fine-tuned. Our findings highlight the confounding role of
reference policies in DPO and offer insights for best practices, while also
identifying open research questions for future studies.
Scaling Granite Code Models to 128K Context
Authors: name: Matt Stallone, name: Vaibhav Saxena, name: Leonid Karlinsky, name: Bridget McGinn, name: Tim Bula, name: Mayank Mishra, name: Adriana Meza Soria, name: Gaoyuan Zhang, name: Aditya Prasad, name: Yikang Shen, name: Saptha Surendran, name: Shanmukha Guttula, name: Hima Patel, name: Parameswaran Selvam, name: Xuan-Hong Dang, name: Yan Koyfman, name: Atin Sood, name: Rogerio Feris, name: Nirmit Desai, name: David D. Cox, name: Ruchir Puri, name: Rameswar Panda
Published: 2024-07-18 17:46:02

This paper introduces long-context Granite code models that support effective
context windows of up to 128K tokens. Our solution for scaling context length
of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight
continual pretraining by gradually increasing its RoPE base frequency with
repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context
support which are derived by further finetuning the long context base models on
a mix of permissively licensed short and long-context instruction-response
pairs. While comparing to the original short-context Granite code models, our
long-context models achieve significant improvements on long-context tasks
without any noticeable performance degradation on regular code completion
benchmarks (e.g., HumanEval). We release all our long-context Granite code
models under an Apache 2.0 license for both research and commercial use.
Shape of Motion: 4D Reconstruction from a Single Video
Authors: name: Qianqian Wang, name: Vickie Ye, name: Hang Gao, name: Jake Austin, name: Zhengqi Li, name: Angjoo Kanazawa
Published: 2024-07-18 17:59:08

Monocular dynamic reconstruction is a challenging and long-standing vision
problem due to the highly ill-posed nature of the task. Existing approaches are
limited in that they either depend on templates, are effective only in
quasi-static scenes, or fail to model 3D motion explicitly. In this work, we
introduce a method capable of reconstructing generic dynamic scenes, featuring
explicit, full-sequence-long 3D motion, from casually captured monocular
videos. We tackle the under-constrained nature of the problem with two key
insights: First, we exploit the low-dimensional structure of 3D motion by
representing scene motion with a compact set of SE3 motion bases. Each point's
motion is expressed as a linear combination of these bases, facilitating soft
decomposition of the scene into multiple rigidly-moving groups. Second, we
utilize a comprehensive set of data-driven priors, including monocular depth
maps and long-range 2D tracks, and devise a method to effectively consolidate
these noisy supervisory signals, resulting in a globally consistent
representation of the dynamic scene. Experiments show that our method achieves
state-of-the-art performance for both long-range 3D/2D motion estimation and
novel view synthesis on dynamic scenes. Project Page:
https://shape-of-motion.github.io/
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
Authors: name: Rulin Shao, name: Jacqueline He, name: Akari Asai, name: Weijia Shi, name: Tim Dettmers, name: Sewon Min, name: Luke Zettlemoyer, name: Pang Wei Koh
Published: 2024-07-09 08:27:27

Scaling laws with respect to the amount of training data and the number of
parameters allow us to predict the cost-benefit trade-offs of pretraining
language models (LMs) in different configurations. In this paper, we consider
another dimension of scaling: the amount of data available at inference time.
Specifically, we find that increasing the size of the datastore used by a
retrieval-based LM monotonically improves language modeling and several
downstream tasks without obvious saturation, such that a smaller model
augmented with a large datastore outperforms a larger LM-only model on
knowledge-intensive tasks. By plotting compute-optimal scaling curves with
varied datastore, model, and pretraining data sizes, we show that using larger
datastores can significantly improve model performance for the same training
compute budget. We carry out our study by constructing a 1.4 trillion-token
datastore named MassiveDS, which is the largest and the most diverse
open-sourced datastore for retrieval-based LMs to date, and designing an
efficient pipeline for studying datastore scaling in a computationally
accessible manner. Finally, we analyze the effect of improving the retriever,
datastore quality filtering, and other design choices on our observed scaling
trends. Overall, our results show that datastore size should be considered as
an integral part of LM efficiency and performance trade-offs. To facilitate
future research, we open-source our datastore and code at
https://github.com/RulinShao/retrieval-scaling.
Streetscapes: Large-scale Consistent Street View Generation Using
  Autoregressive Video Diffusion
Authors: name: Boyang Deng, name: Richard Tucker, name: Zhengqi Li, name: Leonidas Guibas, name: Noah Snavely, name: Gordon Wetzstein
Published: 2024-07-18 17:56:30

We present a method for generating Streetscapes-long sequences of views
through an on-the-fly synthesized city-scale scene. Our generation is
conditioned by language input (e.g., city name, weather), as well as an
underlying map/layout hosting the desired trajectory. Compared to recent models
for video generation or 3D view synthesis, our method can scale to much
longer-range camera trajectories, spanning several city blocks, while
maintaining visual quality and consistency. To achieve this goal, we build on
recent work on video diffusion, used within an autoregressive framework that
can easily scale to long sequences. In particular, we introduce a new temporal
imputation method that prevents our autoregressive approach from drifting from
the distribution of realistic city imagery. We train our Streetscapes system on
a compelling source of data-posed imagery from Google Street View, along with
contextual map data-which allows users to generate city views conditioned on
any desired city layout, with controllable camera poses. Please see more
results at our project page at https://boyangdeng.com/streetscapes.
Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies
Authors: name: Chaofan Tao, name: Qian Liu, name: Longxu Dou, name: Niklas Muennighoff, name: Zhongwei Wan, name: Ping Luo, name: Min Lin, name: Ngai Wong
Published: 2024-07-18 15:58:54

Research on scaling large language models (LLMs) has primarily focused on
model parameters and training data size, overlooking the role of vocabulary
size. % Intuitively, larger vocabularies enable more efficient tokenization by
representing sentences with fewer tokens, but they also increase the risk of
under-fitting representations for rare tokens. We investigate how vocabulary
size impacts LLM scaling laws by training models ranging from 33M to 3B
parameters on up to 500B characters with various vocabulary configurations. We
propose three complementary approaches for predicting the compute-optimal
vocabulary size: IsoFLOPs analysis, derivative estimation, and parametric fit
of the loss function. Our approaches converge on the same result that the
optimal vocabulary size depends on the available compute budget and that larger
models deserve larger vocabularies. However, most LLMs use too small vocabulary
sizes. For example, we predict that the optimal vocabulary size of Llama2-70B
should have been at least 216K, 7 times larger than its vocabulary of 32K. We
validate our predictions empirically by training models with 3B parameters
across different FLOPs budgets. Adopting our predicted optimal vocabulary size
consistently improves downstream performance over commonly used vocabulary
sizes. By increasing the vocabulary size from the conventional 32K to 43K, we
improve performance on ARC-Challenge from 29.1 to 32.0 with the same 2.3e21
FLOPs. Our work emphasizes the necessity of jointly considering model
parameters and vocabulary size for efficient scaling.
CodeV: Empowering LLMs for Verilog Generation through Multi-Level
  Summarization
Authors: name: Yang Zhao, name: Di Huang, name: Chongxiao Li, name: Pengwei Jin, name: Ziyuan Nan, name: Tianyun Ma, name: Lei Qi, name: Yansong Pan, name: Zhenxing Zhang, name: Rui Zhang, name: Xishan Zhang, name: Zidong Du, name: Qi Guo, name: Xing Hu, name: Yunji Chen
Published: 2024-07-15 03:57:20

The increasing complexity and high costs associated with modern processor
design have led to a surge in demand for processor design automation.
Instruction-tuned large language models (LLMs) have demonstrated remarkable
performance in automatically generating code for general-purpose programming
languages like Python. However, these methods fail on hardware description
languages (HDLs) like Verilog due to the scarcity of high-quality instruction
tuning data, as even advanced LLMs like GPT-3.5 exhibit limited performance on
Verilog generation. Regarding this issue, we observe that (1) Verilog code
collected from the real world has higher quality than those generated by LLMs.
(2) LLMs like GPT-3.5 excel in summarizing Verilog code rather than generating
it. Based on these observations, this paper introduces CodeV, a series of
open-source instruction-tuned Verilog generation LLMs. Instead of generating
descriptions first and then getting the corresponding code from advanced LLMs,
we prompt the LLM with Verilog code and let the LLM generate the corresponding
natural language description by multi-level summarization. Experimental results
show that CodeV relatively surpasses the previous open-source SOTA by 14.4%
(BetterV in VerilogEval) and 11.3% (RTLCoder in RTLLM) respectively, and also
relatively outperforms previous commercial SOTA GPT-4 by 22.1% in VerilogEval.
Towards Understanding Unsafe Video Generation
Authors: name: Yan Pang, name: Aiping Xiong, name: Yang Zhang, name: Tianhao Wang
Published: 2024-07-17 14:07:22

Video generation models (VGMs) have demonstrated the capability to synthesize
high-quality output. It is important to understand their potential to produce
unsafe content, such as violent or terrifying videos. In this work, we provide
a comprehensive understanding of unsafe video generation.
  First, to confirm the possibility that these models could indeed generate
unsafe videos, we choose unsafe content generation prompts collected from 4chan
and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After
filtering out duplicates and poorly generated content, we created an initial
set of 2112 unsafe videos from an original pool of 5607 videos. Through
clustering and thematic coding analysis of these generated videos, we identify
5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic,
Violent/Bloody, and Political. With IRB approval, we then recruit online
participants to help label the generated videos. Based on the annotations
submitted by 403 participants, we identified 937 unsafe videos from the initial
video set. With the labeled information and the corresponding prompts, we
created the first dataset of unsafe videos generated by VGMs.
  We then study possible defense mechanisms to prevent the generation of unsafe
videos. Existing defense methods in image generation focus on filtering either
input prompt or output results. We propose a new approach called Latent
Variable Defense (LVD), which works within the model's internal sampling
process. LVD can achieve 0.90 defense accuracy while reducing time and
computing resources by 10x when sampling a large number of unsafe prompts.