Orionnnn/hub-stats huggingface dataset:
UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
  Models in Multi-View Urban Scenarios
Authors: name: Baichuan Zhou, name: Haote Yang, name: Dairong Chen, name: Junyan Ye, name: Tianyi Bai, name: Jinhua Yu, name: Songyang Zhang, name: Dahua Lin, name: Conghui He, name: Weijia Li
Published: 2024-08-30 13:13:35

Recent evaluations of Large Multimodal Models (LMMs) have explored their
capabilities in various domains, with only few benchmarks specifically focusing
on urban environments. Moreover, existing urban benchmarks have been limited to
evaluating LMMs with basic region-level urban tasks under singular views,
leading to incomplete evaluations of LMMs' abilities in urban environments. To
address these issues, we present UrBench, a comprehensive benchmark designed
for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
11.6K meticulously curated questions at both region-level and role-level that
cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
Understanding, and Object Understanding, totaling 14 task types. In
constructing UrBench, we utilize data from existing datasets and additionally
collect data from 11 cities, creating new annotations using a cross-view
detection-matching method. With these images and annotations, we then integrate
LMM-based, rule-based, and human-based methods to construct large-scale
high-quality questions. Our evaluations on 21 LMMs show that current LMMs
struggle in the urban environments in several aspects. Even the best performing
GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
counting to complex tasks such as orientation, localization and object
attribute recognition, with an average performance gap of 17.4%. Our benchmark
also reveals that LMMs exhibit inconsistent behaviors with different urban
views, especially with respect to understanding cross-view relations. UrBench
datasets and benchmark results will be publicly available at
https://opendatalab.github.io/UrBench/.
InkubaLM: A small language model for low-resource African languages
Authors: name: Atnafu Lambebo Tonja, name: Bonaventure F. P. Dossou, name: Jessica Ojo, name: Jenalea Rajab, name: Fadel Thior, name: Eric Peter Wairagala, name: Aremu Anuoluwapo, name: Pelonomi Moiloa, name: Jade Abbott, name: Vukosi Marivate, name: Benjamin Rosman
Published: 2024-08-30 05:42:31

High-resource language models often fall short in the African context, where
there is a critical need for models that are efficient, accessible, and locally
relevant, even amidst significant computing and data constraints. This paper
introduces InkubaLM, a small language model with 0.4 billion parameters, which
achieves performance comparable to models with significantly larger parameter
counts and more extensive training data on tasks such as machine translation,
question-answering, AfriMMLU, and the AfriXnli task. Notably, InkubaLM
outperforms many larger models in sentiment analysis and demonstrates
remarkable consistency across multiple languages. This work represents a
pivotal advancement in challenging the conventional paradigm that effective
language models must rely on substantial resources. Our model and datasets are
publicly available \url{https://huggingface.co/lelapa} to encourage
research and development on low-resource languages.
Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction
  Retriever
Authors: name: Rohan Jha, name: Bo Wang, name: Michael Günther, name: Saba Sturua, name: Mohammad Kalim Akram, name: Han Xiao
Published: 2024-08-29 16:21:00

Multi-vector dense models, such as ColBERT, have proven highly effective in
information retrieval. ColBERT's late interaction scoring approximates the
joint query-document attention seen in cross-encoders while maintaining
inference efficiency closer to traditional dense retrieval models, thanks to
its bi-encoder architecture and recent optimizations in indexing and search. In
this paper, we introduce several improvements to the ColBERT model architecture
and training pipeline, leveraging techniques successful in the more established
single-vector embedding model paradigm, particularly those suited for
heterogeneous multilingual data. Our new model, Jina-ColBERT-v2, demonstrates
strong performance across a range of English and multilingual retrieval tasks,
while also cutting storage requirements by up to 50% compared to previous
models.
SurveySum: A Dataset for Summarizing Multiple Scientific Articles into a
  Survey Section
Authors: name: Leandro Carísio Fernandes, name: Gustavo Bartz Guedes, name: Thiago Soares Laitz, name: Thales Sales Almeida, name: Rodrigo Nogueira, name: Roberto Lotufo, name: Jayr Pereira
Published: 2024-08-29 11:13:23

Document summarization is a task to shorten texts into concise and
informative summaries. This paper introduces a novel dataset designed for
summarizing multiple scientific articles into a section of a survey. Our
contributions are: (1) SurveySum, a new dataset addressing the gap in
domain-specific summarization tools; (2) two specific pipelines to summarize
scientific articles into a section of a survey; and (3) the evaluation of these
pipelines using multiple metrics to compare their performance. Our results
highlight the importance of high-quality retrieval stages and the impact of
different configurations on the quality of generated summaries.
Large-Scale Multi-omic Biosequence Transformers for Modeling
  Peptide-Nucleotide Interactions
Authors: name: Sully F. Chen, name: Robert J. Steele, name: Beakal Lemeneh, name: Shivanand P. Lad, name: Eric Oermann
Published: 2024-08-29 03:56:40

The transformer architecture has revolutionized bioinformatics and driven
progress in the understanding and prediction of the properties of biomolecules.
Almost all research on large-scale biosequence transformers has focused on one
domain at a time (single-omic), usually nucleotides or peptides. These models
have seen incredible success in downstream tasks in each domain and have
achieved particularly noteworthy breakthroughs in sequences of peptides and
structural modeling. However, these single-omic models are naturally incapable
of modeling multi-omic tasks, one of the most biologically critical being
nucleotide-peptide interactions.
  We present our work training the first multi-omic nucleotide-peptide
foundation models. We show that these multi-omic models (MOMs) can learn joint
representations between various single-omic distributions that are emergently
consistent with the Central Dogma of molecular biology, despite only being
trained on unlabeled biosequences. We further demonstrate that MOMs can be
fine-tuned to achieve state-of-the-art results on peptide-nucleotide
interaction tasks, namely predicting the change in Gibbs free energy
({\Delta}G) of the binding interaction between a given oligonucleotide and
peptide, as well as the effect on this binding interaction due to mutations in
the oligonucleotide sequence ({\Delta}{\Delta}G).
  Remarkably, we show that multi-omic biosequence transformers emergently learn
useful structural information without any prior structural training, allowing
us to predict which peptide residues are most involved in the
peptide-nucleotide binding interaction. Lastly, we provide evidence that
multi-omic biosequence models are non-inferior to foundation models trained on
single-omics distributions, suggesting a more generalized or foundational
approach to building these models.
Automatic Differential Diagnosis using Transformer-Based Multi-Label
  Sequence Classification
Authors: name: Abu Adnan Sadi, name: Mohammad Ashrafuzzaman Khan, name: Lubaba Binte Saber
Published: 2024-08-28 14:40:15

As the field of artificial intelligence progresses, assistive technologies
are becoming more widely used across all industries. The healthcare industry is
no different, with numerous studies being done to develop assistive tools for
healthcare professionals. Automatic diagnostic systems are one such beneficial
tool that can assist with a variety of tasks, including collecting patient
information, analyzing test results, and diagnosing patients. However, the idea
of developing systems that can provide a differential diagnosis has been
largely overlooked in most of these research studies. In this study, we propose
a transformer-based approach for providing differential diagnoses based on a
patient's age, sex, medical history, and symptoms. We use the DDXPlus dataset,
which provides differential diagnosis information for patients based on 49
disease types. Firstly, we propose a method to process the tabular patient data
from the dataset and engineer them into patient reports to make them suitable
for our research. In addition, we introduce two data modification modules to
diversify the training data and consequently improve the robustness of the
models. We approach the task as a multi-label classification problem and
conduct extensive experiments using four transformer models. All the models
displayed promising results by achieving over 97% F1 score on the held-out test
set. Moreover, we design additional behavioral tests to get a broader
understanding of the models. In particular, for one of our test cases, we
prepared a custom test set of 100 samples with the assistance of a doctor. The
results on the custom set showed that our proposed data modification modules
improved the model's generalization capabilities. We hope our findings will
provide future researchers with valuable insights and inspire them to develop
reliable systems for automatic differential diagnosis.
VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language
  Models for Trait Discovery from Biological Images
Authors: name: M. Maruf, name: Arka Daw, name: Kazi Sajeed Mehrab, name: Harish Babu Manogaran, name: Abhilash Neog, name: Medha Sawhney, name: Mridul Khurana, name: James P. Balhoff, name: Yasin Bakis, name: Bahadir Altintas, name: Matthew J. Thompson, name: Elizabeth G. Campolongo, name: Josef C. Uyeda, name: Hilmar Lapp, name: Henry L. Bart, name: Paula M. Mabee, name: Yu Su, name: Wei-Lun Chao, name: Charles Stewart, name: Tanya Berger-Wolf, name: Wasila Dahdul, name: Anuj Karpatne
Published: 2024-08-28 23:53:57

Images are increasingly becoming the currency for documenting biodiversity on
the planet, providing novel opportunities for accelerating scientific
discoveries in the field of organismal biology, especially with the advent of
large vision-language models (VLMs). We ask if pre-trained VLMs can aid
scientists in answering a range of biologically relevant questions without any
additional fine-tuning. In this paper, we evaluate the effectiveness of 12
state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel
dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images
from three groups of organisms: fishes, birds, and butterflies, covering five
biologically relevant tasks. We also explore the effects of applying prompting
techniques and tests for reasoning hallucination on the performance of VLMs,
shedding new light on the capabilities of current SOTA VLMs in answering
biologically relevant questions using images. The code and datasets for running
all the analyses reported in this paper can be found at
https://github.com/sammarfy/VLM4Bio.
SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding
Authors: name: Sihang Li, name: Jin Huang, name: Jiaxi Zhuang, name: Yaorui Shi, name: Xiaochen Cai, name: Mingjun Xu, name: Xiang Wang, name: Linfeng Zhang, name: Guolin Ke, name: Hengxing Cai
Published: 2024-08-28 05:41:52

Scientific literature understanding is crucial for extracting targeted
information and garnering insights, thereby significantly advancing scientific
discovery. Despite the remarkable success of Large Language Models (LLMs), they
face challenges in scientific literature understanding, primarily due to (1) a
lack of scientific knowledge and (2) unfamiliarity with specialized scientific
tasks.
  To develop an LLM specialized in scientific literature understanding, we
propose a hybrid strategy that integrates continual pre-training (CPT) and
supervised fine-tuning (SFT), to simultaneously infuse scientific domain
knowledge and enhance instruction-following capabilities for domain-specific
tasks.cIn this process, we identify two key challenges: (1) constructing
high-quality CPT corpora, and (2) generating diverse SFT instructions. We
address these challenges through a meticulous pipeline, including PDF text
extraction, parsing content error correction, quality filtering, and synthetic
instruction creation. Applying this strategy, we present a suite of LLMs:
SciLitLLM, specialized in scientific literature understanding. These models
demonstrate promising performance on scientific literature understanding
benchmarks.
  Our contributions are threefold: (1) We present an effective framework that
integrates CPT and SFT to adapt LLMs to scientific literature understanding,
which can also be easily adapted to other domains. (2) We propose an LLM-based
synthesis method to generate diverse and high-quality scientific instructions,
resulting in a new instruction set -- SciLitIns -- for supervised fine-tuning
in less-represented scientific domains. (3) SciLitLLM achieves promising
performance improvements on scientific literature understanding benchmarks.
ClimDetect: A Benchmark Dataset for Climate Change Detection and
  Attribution
Authors: name: Sungduk Yu, name: Brian L. White, name: Anahita Bhiwandiwalla, name: Musashi Hinck, name: Matthew Lyle Olson, name: Tung Nguyen, name: Vasudev Lal
Published: 2024-08-28 17:58:53

Detecting and attributing temperature increases due to climate change is
crucial for understanding global warming and guiding adaptation strategies. The
complexity of distinguishing human-induced climate signals from natural
variability has challenged traditional detection and attribution (D&A)
approaches, which seek to identify specific "fingerprints" in climate response
variables. Deep learning offers potential for discerning these complex patterns
in expansive spatial datasets. However, lack of standard protocols has hindered
consistent comparisons across studies. We introduce ClimDetect, a standardized
dataset of over 816k daily climate snapshots, designed to enhance model
accuracy in identifying climate change signals. ClimDetect integrates various
input and target variables used in past research, ensuring comparability and
consistency. We also explore the application of vision transformers (ViT) to
climate data, a novel and modernizing approach in this context. Our open-access
data and code serve as a benchmark for advancing climate science through
improved model evaluations. ClimDetect is publicly accessible via Huggingface
dataet respository at: https://huggingface.co/datasets/ClimDetect/ClimDetect.
CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
  Personalization
Authors: name: Feize Wu, name: Yun Pang, name: Junyi Zhang, name: Lianyu Pang, name: Jian Yin, name: Baoquan Zhao, name: Qing Li, name: Xudong Mao
Published: 2024-08-28 16:27:58

Recent advances in text-to-image personalization have enabled high-quality
and controllable image synthesis for user-provided concepts. However, existing
methods still struggle to balance identity preservation with text alignment.
Our approach is based on the fact that generating prompt-aligned images
requires a precise semantic understanding of the prompt, which involves
accurately processing the interactions between the new concept and its
surrounding context tokens within the CLIP text encoder. To address this, we
aim to embed the new concept properly into the input embedding space of the
text encoder, allowing for seamless integration with existing tokens. We
introduce Context Regularization (CoRe), which enhances the learning of the new
concept's text embedding by regularizing its context tokens in the prompt. This
is based on the insight that appropriate output vectors of the text encoder for
the context tokens can only be achieved if the new concept's text embedding is
correctly learned. CoRe can be applied to arbitrary prompts without requiring
the generation of corresponding images, thus improving the generalization of
the learned text embedding. Additionally, CoRe can serve as a test-time
optimization technique to further enhance the generations for specific prompts.
Comprehensive experiments demonstrate that our method outperforms several
baseline methods in both identity preservation and text alignment. Code will be
made publicly available.
The VoxCeleb Speaker Recognition Challenge: A Retrospective
Authors: name: Jaesung Huh, name: Joon Son Chung, name: Arsha Nagrani, name: Andrew Brown, name: Jee-weon Jung, name: Daniel Garcia-Romero, name: Andrew Zisserman
Published: 2024-08-27 08:57:31

The VoxCeleb Speaker Recognition Challenges (VoxSRC) were a series of
challenges and workshops that ran annually from 2019 to 2023. The challenges
primarily evaluated the tasks of speaker recognition and diarisation under
various settings including: closed and open training data; as well as
supervised, self-supervised, and semi-supervised training for domain
adaptation. The challenges also provided publicly available training and
evaluation datasets for each task and setting, with new test sets released each
year. In this paper, we provide a review of these challenges that covers: what
they explored; the methods developed by the challenge participants and how
these evolved; and also the current state of the field for speaker verification
and diarisation. We chart the progress in performance over the five
installments of the challenge on a common evaluation dataset and provide a
detailed analysis of how each year's special focus affected participants'
performance. This paper is aimed both at researchers who want an overview of
the speaker recognition and diarisation field, and also at challenge organisers
who want to benefit from the successes and avoid the mistakes of the VoxSRC
challenges. We end with a discussion of the current strengths of the field and
open challenges. Project page :
https://mm.kaist.ac.kr/datasets/voxceleb/voxsrc/workshop.html
CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting
  Mitigation
Authors: name: Muhammad Fawi
Published: 2024-08-26 18:42:59

This paper introduces CURLoRA, a novel approach to fine-tuning large language
models (LLMs) that leverages CUR matrix decomposition in the context of
Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM
fine-tuning: mitigating catastrophic forgetting during continual learning and
reducing the number of trainable parameters. We propose a unique modification
to the CUR decomposition process, utilizing inverted probabilities for column
and row selection which acts as an implicit regularization, and initializing
the U matrix as a zero matrix, and only fine-tuning it. We demonstrate
through experiments on multiple datasets that CURLoRA outperforms standard LoRA
in mitigating catastrophic forgetting. It maintains model stability and
performance across tasks while significantly reducing the number of trainable
parameters. Our results show that CURLoRA achieves very good and stable task
accuracy while maintaining base model's perplexity scores fixed compared to
LoRA upon continual fine-tuning, particularly in scenarios with limited data.
Physics of Language Models: Part 2.2, How to Learn From Mistakes on
  Grade-School Math Problems
Authors: name: Tian Ye, name: Zicheng Xu, name: Yuanzhi Li, name: Zeyuan Allen-Zhu
Published: 2024-08-29 06:49:20

Language models have demonstrated remarkable performance in solving reasoning
tasks; however, even the strongest models still occasionally make reasoning
mistakes. Recently, there has been active research aimed at improving reasoning
accuracy, particularly by using pretrained language models to "self-correct"
their mistakes via multi-round prompting. In this paper, we follow this line of
work but focus on understanding the usefulness of incorporating
"error-correction" data directly into the pretraining stage. This data consists
of erroneous solution steps immediately followed by their corrections. Using a
synthetic math dataset, we show promising results: this type of pretrain data
can help language models achieve higher reasoning accuracy directly (i.e.,
through simple auto-regression, without multi-round prompting) compared to
pretraining on the same amount of error-free data. We also delve into many
details, such as (1) how this approach differs from beam search, (2) how such
data can be prepared, (3) whether masking is needed on the erroneous tokens,
(4) the amount of error required, (5) whether such data can be deferred to the
fine-tuning stage, and many others.
Meta Flow Matching: Integrating Vector Fields on the Wasserstein
  Manifold
Authors: name: Lazar Atanackovic, name: Xi Zhang, name: Brandon Amos, name: Mathieu Blanchette, name: Leo J. Lee, name: Yoshua Bengio, name: Alexander Tong, name: Kirill Neklyudov
Published: 2024-08-26 20:05:31

Numerous biological and physical processes can be modeled as systems of
interacting entities evolving continuously over time, e.g. the dynamics of
communicating cells or physical particles. Learning the dynamics of such
systems is essential for predicting the temporal evolution of populations
across novel samples and unseen environments. Flow-based models allow for
learning these dynamics at the population level - they model the evolution of
the entire distribution of samples. However, current flow-based models are
limited to a single initial population and a set of predefined conditions which
describe different dynamics. We argue that multiple processes in natural
sciences have to be represented as vector fields on the Wasserstein manifold of
probability densities. That is, the change of the population at any moment in
time depends on the population itself due to the interactions between samples.
In particular, this is crucial for personalized medicine where the development
of diseases and their respective treatment response depends on the
microenvironment of cells specific to each patient. We propose Meta Flow
Matching (MFM), a practical approach to integrating along these vector fields
on the Wasserstein manifold by amortizing the flow model over the initial
populations. Namely, we embed the population of samples using a Graph Neural
Network (GNN) and use these embeddings to train a Flow Matching model. This
gives MFM the ability to generalize over the initial distributions unlike
previously proposed methods. We demonstrate the ability of MFM to improve
prediction of individual treatment responses on a large scale multi-patient
single-cell drug screen dataset.
StyleRemix: Interpretable Authorship Obfuscation via Distillation and
  Perturbation of Style Elements
Authors: name: Jillian Fisher, name: Skyler Hallinan, name: Ximing Lu, name: Mitchell Gordon, name: Zaid Harchaoui, name: Yejin Choi
Published: 2024-08-28 09:35:15

Authorship obfuscation, rewriting a text to intentionally obscure the
identity of the author, is an important but challenging task. Current methods
using large language models (LLMs) lack interpretability and controllability,
often ignoring author-specific stylistic features, resulting in less robust
performance overall.
  To address this, we develop StyleRemix, an adaptive and interpretable
obfuscation method that perturbs specific, fine-grained style elements of the
original input text. StyleRemix uses pre-trained Low Rank Adaptation (LoRA)
modules to rewrite an input specifically along various stylistic axes (e.g.,
formality and length) while maintaining low computational cost. StyleRemix
outperforms state-of-the-art baselines and much larger LLMs in a variety of
domains as assessed by both automatic and human evaluation.
  Additionally, we release AuthorMix, a large set of 30K high-quality,
long-form texts from a diverse set of 14 authors and 4 domains, and DiSC, a
parallel corpus of 1,500 texts spanning seven style axes in 16 unique
directions