Orionnnn/hub-stats huggingface dataset:
SambaMixer: State of Health Prediction of Li-ion Batteries using Mamba
  State Space Models
Authors: name: José Ignacio Olalde-Verano, name: Sascha Kirch, name: Clara Pérez-Molina, name: Sergio Martin
Published: 2024-10-31 22:12:27

The state of health (SOH) of a Li-ion battery is a critical parameter that
determines the remaining capacity and the remaining lifetime of the battery. In
this paper, we propose SambaMixer a novel structured state space model (SSM)
for predicting the state of health of Li-ion batteries. The proposed SSM is
based on the MambaMixer architecture, which is designed to handle multi-variate
time signals. We evaluate our model on the NASA battery discharge dataset and
show that our model outperforms the state-of-the-art on this dataset. We
further introduce a novel anchor-based resampling method which ensures time
signals are of the expected length while also serving as augmentation
technique. Finally, we condition prediction on the sample time and the cycle
time difference using positional encodings to improve the performance of our
model and to learn recuperation effects. Our results proof that our model is
able to predict the SOH of Li-ion batteries with high accuracy and robustness.
Survey of User Interface Design and Interaction Techniques in Generative
  AI Applications
Authors: name: Reuben Luera, name: Ryan A. Rossi, name: Alexa Siu, name: Franck Dernoncourt, name: Tong Yu, name: Sungchul Kim, name: Ruiyi Zhang, name: Xiang Chen, name: Hanieh Salehy, name: Jian Zhao, name: Samyadeep Basu, name: Puneet Mathur, name: Nedim Lipka
Published: 2024-10-28 23:10:06

The applications of generative AI have become extremely impressive, and the
interplay between users and AI is even more so. Current human-AI interaction
literature has taken a broad look at how humans interact with generative AI,
but it lacks specificity regarding the user interface designs and patterns used
to create these applications. Therefore, we present a survey that
comprehensively presents taxonomies of how a human interacts with AI and the
user interaction patterns designed to meet the needs of a variety of relevant
use cases. We focus primarily on user-guided interactions, surveying
interactions that are initiated by the user and do not include any implicit
signals given by the user. With this survey, we aim to create a compendium of
different user-interaction patterns that can be used as a reference for
designers and developers alike. In doing so, we also strive to lower the entry
barrier for those attempting to learn more about the design of generative AI
applications.
GRS-QA -- Graph Reasoning-Structured Question Answering Dataset
Authors: name: Anish Pahilajani, name: Devasha Trivedi, name: Jincen Shuai, name: Khin S. Yone, name: Samyak Rajesh Jain, name: Namyong Park, name: Ryan A. Rossi, name: Nesreen K. Ahmed, name: Franck Dernoncourt, name: Yu Wang
Published: 2024-11-01 05:14:03

Large Language Models (LLMs) have excelled in multi-hop question-answering
(M-QA) due to their advanced reasoning abilities. However, the impact of the
inherent reasoning structures on LLM M-QA performance remains unclear, largely
due to the absence of QA datasets that provide fine-grained reasoning
structures. To address this gap, we introduce the Graph Reasoning-Structured
Question Answering Dataset (GRS-QA), which includes both semantic contexts and
reasoning structures for QA pairs. Unlike existing M-QA datasets, where
different reasoning structures are entangled together, GRS-QA explicitly
captures intricate reasoning pathways by constructing reasoning graphs, where
nodes represent textual contexts and edges denote logical flows. These
reasoning graphs of different structures enable a fine-grained evaluation of
LLM reasoning capabilities across various reasoning structures. Our empirical
analysis reveals that LLMs perform differently when handling questions with
varying reasoning structures. This finding facilitates the exploration of
textual structures as compared with semantics.
Personalization of Large Language Models: A Survey
Authors: name: Zhehao Zhang, name: Ryan A. Rossi, name: Branislav Kveton, name: Yijia Shao, name: Diyi Yang, name: Hamed Zamani, name: Franck Dernoncourt, name: Joe Barrow, name: Tong Yu, name: Sungchul Kim, name: Ruiyi Zhang, name: Jiuxiang Gu, name: Tyler Derr, name: Hongjie Chen, name: Junda Wu, name: Xiang Chen, name: Zichao Wang, name: Subrata Mitra, name: Nedim Lipka, name: Nesreen Ahmed, name: Yu Wang
Published: 2024-10-29 04:01:11

Personalization of Large Language Models (LLMs) has recently become
increasingly important with a wide range of applications. Despite the
importance and recent progress, most existing works on personalized LLMs have
focused either entirely on (a) personalized text generation or (b) leveraging
LLMs for personalization-related downstream applications, such as
recommendation systems. In this work, we bridge the gap between these two
separate main directions for the first time by introducing a taxonomy for
personalized LLM usage and summarizing the key differences and challenges. We
provide a formalization of the foundations of personalized LLMs that
consolidates and expands notions of personalization of LLMs, defining and
discussing novel facets of personalization, usage, and desiderata of
personalized LLMs. We then unify the literature across these diverse fields and
usage scenarios by proposing systematic taxonomies for the granularity of
personalization, personalization techniques, datasets, evaluation methods, and
applications of personalized LLMs. Finally, we highlight challenges and
important open problems that remain to be addressed. By unifying and surveying
recent research using the proposed taxonomies, we aim to provide a clear guide
to the existing literature and different facets of personalization in LLMs,
empowering both researchers and practitioners.
HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level
  and Fidelity-Rich Conditions in Diffusion Models
Authors: name: Shengkai Zhang, name: Nianhong Jiao, name: Tian Li, name: Chaojie Yang, name: Chenhui Xue, name: Boya Niu, name: Jun Gao
Published: 2024-10-30 11:00:51

We propose an effective method for inserting adapters into text-to-image
foundation models, which enables the execution of complex downstream tasks
while preserving the generalization ability of the base model. The core idea of
this method is to optimize the attention mechanism related to 2D feature maps,
which enhances the performance of the adapter. This approach was validated on
the task of meme video generation and achieved significant results. We hope
this work can provide insights for post-training tasks of large text-to-image
models. Additionally, as this method demonstrates good compatibility with SD1.5
derivative models, it holds certain value for the open-source community.
Therefore, we will release the related code
(https://songkey.github.io/hellomeme).
M2rc-Eval: Massively Multilingual Repository-level Code Completion
  Evaluation
Authors: name: Jiaheng Liu, name: Ken Deng, name: Congnan Liu, name: Jian Yang, name: Shukai Liu, name: He Zhu, name: Peng Zhao, name: Linzheng Chai, name: Yanan Wu, name: Ke Jin, name: Ge Zhang, name: Zekun Wang, name: Guoan Zhang, name: Bangyu Xiang, name: Wenbo Su, name: Bo Zheng
Published: 2024-10-28 15:58:41

Repository-level code completion has drawn great attention in software
engineering, and several benchmark datasets have been introduced. However,
existing repository-level code completion benchmarks usually focus on a limited
number of languages (<5), which cannot evaluate the general code intelligence
abilities across different languages for existing code Large Language Models
(LLMs). Besides, the existing benchmarks usually report overall average scores
of different languages, where the fine-grained abilities in different
completion scenarios are ignored. Therefore, to facilitate the research of code
LLMs in multilingual scenarios, we propose a massively multilingual
repository-level code completion benchmark covering 18 programming languages
(called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
bucket-level and semantic-level) on different completion scenarios are
provided, where we obtain these annotations based on the parsed abstract syntax
tree. Moreover, we also curate a massively multilingual instruction corpora
M2RC- INSTRUCT dataset to improve the repository-level code completion
abilities of existing code LLMs. Comprehensive experimental results demonstrate
the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
Randomized Autoregressive Visual Generation
Authors: name: Qihang Yu, name: Ju He, name: Xueqing Deng, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2024-11-01 17:59:58

This paper presents Randomized AutoRegressive modeling (RAR) for visual
generation, which sets a new state-of-the-art performance on the image
generation task while maintaining full compatibility with language modeling
frameworks. The proposed RAR is simple: during a standard autoregressive
training process with a next-token prediction objective, the input
sequence-typically ordered in raster form-is randomly permuted into different
factorization orders with a probability r, where r starts at 1 and linearly
decays to 0 over the course of training. This annealing training strategy
enables the model to learn to maximize the expected likelihood over all
factorization orders and thus effectively improve the model's capability of
modeling bidirectional contexts. Importantly, RAR preserves the integrity of
the autoregressive modeling framework, ensuring full compatibility with
language modeling while significantly improving performance in image
generation. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48,
not only surpassing prior state-of-the-art autoregressive image generators but
also outperforming leading diffusion-based and masked transformer-based
methods. Code and models will be made available at
https://github.com/bytedance/1d-tokenizer
In-Context LoRA for Diffusion Transformers
Authors: name: Lianghua Huang, name: Wei Wang, name: Zhi-Fan Wu, name: Yupeng Shi, name: Huanzhang Dou, name: Chen Liang, name: Yutong Feng, name: Yu Liu, name: Jingren Zhou
Published: 2024-10-31 09:45:00

Recent research arXiv:2410.15027 has explored the use of diffusion
transformers (DiTs) for task-agnostic image generation by simply concatenating
attention tokens across images. However, despite substantial computational
resources, the fidelity of the generated images remains suboptimal. In this
study, we reevaluate and streamline this framework by hypothesizing that
text-to-image DiTs inherently possess in-context generation capabilities,
requiring only minimal tuning to activate them. Through diverse task
experiments, we qualitatively demonstrate that existing text-to-image DiTs can
effectively perform in-context generation without any tuning. Building on this
insight, we propose a remarkably simple pipeline to leverage the in-context
abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint
captioning of multiple images, and (3) apply task-specific LoRA tuning using
small datasets (e.g., 20sim 100 samples) instead of full-parameter tuning
with large datasets. We name our models In-Context LoRA (IC-LoRA). This
approach requires no modifications to the original DiT models, only changes to
the training data. Remarkably, our pipeline generates high-fidelity image sets
that better adhere to prompts. While task-specific in terms of tuning data, our
framework remains task-agnostic in architecture and pipeline, offering a
powerful tool for the community and providing valuable insights for further
research on product-level task-agnostic generation systems. We release our
code, data, and models at https://github.com/ali-vilab/In-Context-LoRA
Adapting While Learning: Grounding LLMs for Scientific Problems with
  Intelligent Tool Usage Adaptation
Authors: name: Bohan Lyu, name: Yadi Cao, name: Duncan Watson-Parris, name: Leon Bergen, name: Taylor Berg-Kirkpatrick, name: Rose Yu
Published: 2024-11-01 07:18:31

Large Language Models (LLMs) demonstrate promising capabilities in solving
simple scientific problems but often produce hallucinations for complex ones.
While integrating LLMs with tools can increase reliability, this approach
typically results in over-reliance on tools, diminishing the model's ability to
solve simple problems through basic reasoning. In contrast, human experts first
assess problem complexity using domain knowledge before choosing an appropriate
solution approach. Inspired by this human problem-solving process, we propose a
novel two-component fine-tuning method. In the first component World Knowledge
Distillation (WKD), LLMs learn directly from solutions generated using tool's
information to internalize domain knowledge. In the second component Tool Usage
Adaptation (TUA), we partition problems into easy and hard categories based on
the model's direct answering accuracy. While maintaining the same alignment
target for easy problems as in WKD, we train the model to intelligently switch
to tool usage for more challenging problems. We validate our method on six
scientific benchmark datasets, spanning mathematics, climate science and
epidemiology. On average, our models demonstrate a 28.18% improvement in answer
accuracy and a 13.89% increase in tool usage precision across all datasets,
surpassing state-of-the-art models including GPT-4o and Claude-3.5.
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression
  of Neural Networks
Authors: name: Yongchang Hao, name: Yanshuai Cao, name: Lili Mou
Published: 2024-10-28 01:12:20

The performance of neural networks improves when more parameters are used.
However, the model sizes are constrained by the available on-device memory
during training and inference. Although applying techniques like quantization
can alleviate the constraint, they suffer from performance degradation. In this
work, we introduce NeuZip, a new weight compression scheme based on the entropy
of floating-point numbers in neural networks. With NeuZip, we are able to
achieve memory-efficient training and inference without sacrificing
performance. Notably, we significantly reduce the memory footprint of training
a Llama-3 8B model from 31GB to less than 16GB, while keeping the training
dynamics fully unchanged. In inference, our method can reduce memory usage by
more than half while maintaining near-lossless performance. Our code is
publicly available.
AAAR-1.0: Assessing AI's Potential to Assist Research
Authors: name: Renze Lou, name: Hanzi Xu, name: Sijia Wang, name: Jiangshu Du, name: Ryo Kamoi, name: Xiaoxin Lu, name: Jian Xie, name: Yuxuan Sun, name: Yusen Zhang, name: Jihyun Janice Ahn, name: Hongchao Fang, name: Zhuoyang Zou, name: Wenchao Ma, name: Xi Li, name: Kai Zhang, name: Congying Xia, name: Lifu Huang, name: Wenpeng Yin
Published: 2024-10-29 17:58:29

Numerous studies have assessed the proficiency of AI systems, particularly
large language models (LLMs), in facilitating everyday tasks such as email
writing, question answering, and creative content generation. However,
researchers face unique challenges and opportunities in leveraging LLMs for
their own work, such as brainstorming research ideas, designing experiments,
and writing or reviewing papers. In this study, we introduce AAAR-1.0, a
benchmark dataset designed to evaluate LLM performance in three fundamental,
expertise-intensive research tasks: (i) EquationInference, assessing the
correctness of equations based on the contextual information in paper
submissions; (ii) ExperimentDesign, designing experiments to validate research
ideas and solutions; (iii) PaperWeakness, identifying weaknesses in paper
submissions; and (iv) REVIEWCRITIQUE, identifying each segment in human reviews
is deficient or not. AAAR-1.0 differs from prior benchmarks in two key ways:
first, it is explicitly research-oriented, with tasks requiring deep domain
expertise; second, it is researcher-oriented, mirroring the primary activities
that researchers engage in on a daily basis. An evaluation of both open-source
and proprietary LLMs reveals their potential as well as limitations in
conducting sophisticated research tasks. We will keep iterating AAAR-1.0 to new
versions.
Minimum Entropy Coupling with Bottleneck
Authors: name: M. Reza Ebrahimi, name: Jun Chen, name: Ashish Khisti
Published: 2024-10-29 02:19:07

This paper investigates a novel lossy compression framework operating under
logarithmic loss, designed to handle situations where the reconstruction
distribution diverges from the source distribution. This framework is
especially relevant for applications that require joint compression and
retrieval, and in scenarios involving distributional shifts due to processing.
We show that the proposed formulation extends the classical minimum entropy
coupling framework by integrating a bottleneck, allowing for a controlled
degree of stochasticity in the coupling. We explore the decomposition of the
Minimum Entropy Coupling with Bottleneck (MEC-B) into two distinct optimization
problems: Entropy-Bounded Information Maximization (EBIM) for the encoder, and
Minimum Entropy Coupling (MEC) for the decoder. Through extensive analysis, we
provide a greedy algorithm for EBIM with guaranteed performance, and
characterize the optimal solution near functional mappings, yielding
significant theoretical insights into the structural complexity of this
problem. Furthermore, we illustrate the practical application of MEC-B through
experiments in Markov Coding Games (MCGs) under rate limits. These games
simulate a communication scenario within a Markov Decision Process, where an
agent must transmit a compressed message from a sender to a receiver through
its actions. Our experiments highlight the trade-offs between MDP rewards and
receiver accuracy across various compression rates, showcasing the efficacy of
our method compared to conventional compression baseline.
GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for
  Minority Languages
Authors: name: Amir Hossein Kargaran, name: François Yvon, name: Hinrich Schütze
Published: 2024-10-31 11:14:12

The need for large text corpora has increased with the advent of pretrained
language models and, in particular, the discovery of scaling laws for these
models. Most available corpora have sufficient data only for languages with
large dominant communities. However, there is no corpus available that (i)
covers a wide range of minority languages; (ii) is generated by an open-source
reproducible pipeline; and (iii) is rigorously cleaned from noise, making it
trustworthy to use. We present GlotCC, a clean, document-level, 2TB general
domain corpus derived from CommonCrawl, covering more than 1000 languages. We
make GlotCC and the system used to generate it - including the pipeline,
language identification model, and filters - available to the research
community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1,
Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.
Teaching Embodied Reinforcement Learning Agents: Informativeness and
  Diversity of Language Use
Authors: name: Jiajun Xi, name: Yinong He, name: Jianing Yang, name: Yinpei Dai, name: Joyce Chai
Published: 2024-10-31 17:59:52

In real-world scenarios, it is desirable for embodied agents to have the
ability to leverage human language to gain explicit or implicit knowledge for
learning tasks. Despite recent progress, most previous approaches adopt simple
low-level instructions as language inputs, which may not reflect natural human
communication. It's not clear how to incorporate rich language use to
facilitate task learning. To address this question, this paper studies
different types of language inputs in facilitating reinforcement learning (RL)
embodied agents. More specifically, we examine how different levels of language
informativeness (i.e., feedback on past behaviors and future guidance) and
diversity (i.e., variation of language expressions) impact agent learning and
inference. Our empirical results based on four RL benchmarks demonstrate that
agents trained with diverse and informative language feedback can achieve
enhanced generalization and fast adaptation to new tasks. These findings
highlight the pivotal role of language use in teaching embodied agents new
tasks in an open world. Project website:
https://github.com/sled-group/Teachable_RL
SelfCodeAlign: Self-Alignment for Code Generation
Authors: name: Yuxiang Wei, name: Federico Cassano, name: Jiawei Liu, name: Yifeng Ding, name: Naman Jain, name: Zachary Mueller, name: Harm de Vries, name: Leandro von Werra, name: Arjun Guha, name: Lingming Zhang
Published: 2024-10-31 17:55:13

Instruction tuning is a supervised fine-tuning approach that significantly
improves the ability of large language models (LLMs) to follow human
instructions. We propose SelfCodeAlign, the first fully transparent and
permissive pipeline for self-aligning code LLMs without extensive human
annotations or distillation. SelfCodeAlign employs the same base model for
inference throughout the data generation process. It first extracts diverse
coding concepts from high-quality seed snippets to generate new tasks. It then
samples multiple responses per task, pairs each with test cases, and validates
them in a sandbox environment. Finally, passing examples are selected for
instruction tuning. In our primary experiments, we use SelfCodeAlign with
CodeQwen1.5-7B to generate a dataset of 74k instruction-response pairs.
Finetuning on this dataset leads to a model that achieves a 67.1 pass@1 on
HumanEval+, surpassing CodeLlama-70B-Instruct despite being ten times smaller.
Across all benchmarks, this finetuned model consistently outperforms the
original version trained with OctoPack, the previous state-of-the-art method
for instruction tuning without human annotations or distillation. Additionally,
we show that SelfCodeAlign is effective across LLMs of various sizes, from 3B
to 33B, and that the base models can benefit more from alignment with their own
data distribution. We further validate each component's effectiveness in our
pipeline, showing that SelfCodeAlign outperforms both direct distillation from
GPT-4o and leading GPT-3.5-based distillation methods, such as OSS-Instruct and
Evol-Instruct. SelfCodeAlign has also led to the creation of
StarCoder2-Instruct, the first fully transparent, permissively licensed, and
self-aligned code LLM that achieves state-of-the-art coding performance.