Orionnnn/hub-stats huggingface dataset:
The Impact of Copyrighted Material on Large Language Models: A Norwegian
  Perspective
Authors: name: Javier de la Rosa, name: Vladislav Mikhailov, name: Lemei Zhang, name: Freddy Wetjen, name: David Samuel, name: Peng Liu, name: Rolv-Arild Braaten, name: Petter Mæhlum, name: Magnus Breder Birkenes, name: Andrey Kutuzov, name: Tita Enstad, name: Svein Arne Brygfjeld, name: Jon Atle Gulla, name: Stephan Oepen, name: Erik Velldal, name: Wilfred Østgulen, name: Liljia Øvrelid, name: Aslak Sira Myhre
Published: 2024-12-12 17:11:22

The use of copyrighted materials in training generative language models
raises critical legal and ethical questions. This paper presents a framework
for and the results of empirically assessing the impact of copyrighted
materials on the performance of large language models (LLMs) for Norwegian. We
found that both books and newspapers contribute positively when the models are
evaluated on a diverse set of Norwegian benchmarks, while fiction works
possibly lead to decreased performance. Our experiments could inform the
creation of a compensation scheme for authors whose works contribute to AI
development.
JuStRank: Benchmarking LLM Judges for System Ranking
Authors: name: Ariel Gera, name: Odellia Boni, name: Yotam Perlitz, name: Roy Bar-Haim, name: Lilach Eden, name: Asaf Yehudai
Published: 2024-12-12 18:51:13

Given the rapid progress of generative AI, there is a pressing need to
systematically compare and choose between the numerous models and
configurations available. The scale and versatility of such evaluations make
the use of LLM-based judges a compelling solution for this challenge.
Crucially, this approach requires first to validate the quality of the LLM
judge itself. Previous work has focused on instance-based assessment of LLM
judges, where a judge is evaluated over a set of responses, or response pairs,
while being agnostic to their source systems. We argue that this setting
overlooks critical factors affecting system-level ranking, such as a judge's
positive or negative bias towards certain systems. To address this gap, we
conduct the first large-scale study of LLM judges as system rankers. System
scores are generated by aggregating judgment scores over multiple system
outputs, and the judge's quality is assessed by comparing the resulting system
ranking to a human-based ranking. Beyond overall judge assessment, our analysis
provides a fine-grained characterization of judge behavior, including their
decisiveness and bias.
DisPose: Disentangling Pose Guidance for Controllable Human Image
  Animation
Authors: name: Hongxiang Li, name: Yaowei Li, name: Yuhang Yang, name: Junjie Cao, name: Zhihong Zhu, name: Xuxin Cheng, name: Long Chen
Published: 2024-12-12 15:15:59

Controllable human image animation aims to generate videos from reference
images using driving videos. Due to the limited control signals provided by
sparse guidance (e.g., skeleton pose), recent works have attempted to introduce
additional dense conditions (e.g., depth map) to ensure motion alignment.
However, such strict dense guidance impairs the quality of the generated video
when the body shape of the reference character differs significantly from that
of the driving video. In this paper, we present DisPose to mine more
generalizable and effective control signals without additional dense input,
which disentangles the sparse skeleton pose in human image animation into
motion field guidance and keypoint correspondence. Specifically, we generate a
dense motion field from a sparse motion field and the reference image, which
provides region-level dense guidance while maintaining the generalization of
the sparse pose control. We also extract diffusion features corresponding to
pose keypoints from the reference image, and then these point features are
transferred to the target pose to provide distinct identity information. To
seamlessly integrate into existing models, we propose a plug-and-play hybrid
ControlNet that improves the quality and consistency of generated videos while
freezing the existing model parameters. Extensive qualitative and quantitative
experiments demonstrate the superiority of DisPose compared to current methods.
Code:
https://github.com/lihxxx/DisPose{https://github.com/lihxxx/DisPose}.
Arbitrary-steps Image Super-resolution via Diffusion Inversion
Authors: name: Zongsheng Yue, name: Kang Liao, name: Chen Change Loy
Published: 2024-12-12 07:24:13

This study presents a new image super-resolution (SR) technique based on
diffusion inversion, aiming at harnessing the rich image priors encapsulated in
large pre-trained diffusion models to improve SR performance. We design a
Partial noise Prediction strategy to construct an intermediate state of the
diffusion model, which serves as the starting sampling point. Central to our
approach is a deep noise predictor to estimate the optimal noise maps for the
forward diffusion process. Once trained, this noise predictor can be used to
initialize the sampling process partially along the diffusion trajectory,
generating the desirable high-resolution result. Compared to existing
approaches, our method offers a flexible and efficient sampling mechanism that
supports an arbitrary number of sampling steps, ranging from one to five. Even
with a single sampling step, our method demonstrates superior or comparable
performance to recent state-of-the-art approaches. The code and model are
publicly available at https://github.com/zsyOAOA/InvSR.
LoRACLR: Contrastive Adaptation for Customization of Diffusion Models
Authors: name: Enis Simsar, name: Thomas Hofmann, name: Federico Tombari, name: Pinar Yanardag
Published: 2024-12-12 18:59:55

Recent advances in text-to-image customization have enabled high-fidelity,
context-rich generation of personalized images, allowing specific concepts to
appear in a variety of scenarios. However, current methods struggle with
combining multiple personalized models, often leading to attribute entanglement
or requiring separate training to preserve concept distinctiveness. We present
LoRACLR, a novel approach for multi-concept image generation that merges
multiple LoRA models, each fine-tuned for a distinct concept, into a single,
unified model without additional individual fine-tuning. LoRACLR uses a
contrastive objective to align and merge the weight spaces of these models,
ensuring compatibility while minimizing interference. By enforcing distinct yet
cohesive representations for each concept, LoRACLR enables efficient, scalable
model composition for high-quality, multi-concept image synthesis. Our results
highlight the effectiveness of LoRACLR in accurately merging multiple concepts,
advancing the capabilities of personalized image generation.
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web
  Tutorials
Authors: name: Yiheng Xu, name: Dunjie Lu, name: Zhennan Shen, name: Junli Wang, name: Zekun Wang, name: Yuchen Mao, name: Caiming Xiong, name: Tao Yu
Published: 2024-12-12 18:59:27

Graphical User Interface (GUI) agents hold great potential for automating
complex tasks across diverse digital environments, from web applications to
desktop software. However, the development of such agents is hindered by the
lack of high-quality, multi-step trajectory data required for effective
training. Existing approaches rely on expensive and labor-intensive human
annotation, making them unsustainable at scale. To address this challenge, we
propose AgentTrek, a scalable data synthesis pipeline that generates
high-quality GUI agent trajectories by leveraging web tutorials. Our method
automatically gathers tutorial-like texts from the internet, transforms them
into task goals with step-by-step instructions, and employs a visual-language
model agent to simulate their execution in a real digital environment. A
VLM-based evaluator ensures the correctness of the generated trajectories. We
demonstrate that training GUI agents with these synthesized trajectories
significantly improves their grounding and planning performance over the
current models. Moreover, our approach is more cost-efficient compared to
traditional human annotation methods. This work underscores the potential of
guided replay with web tutorials as a viable strategy for large-scale GUI agent
training, paving the way for more capable and autonomous digital agents.
SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices
  with Efficient Architectures and Training
Authors: name: Dongting Hu, name: Jierun Chen, name: Xijie Huang, name: Huseyin Coskun, name: Arpit Sahni, name: Aarush Gupta, name: Anujraaj Goyal, name: Dishani Lahiri, name: Rajesh Singh, name: Yerlan Idelbayev, name: Junli Cao, name: Yanyu Li, name: Kwang-Ting Cheng, name: S. -H. Gary Chan, name: Mingming Gong, name: Sergey Tulyakov, name: Anil Kag, name: Yanwu Xu, name: Jian Ren
Published: 2024-12-12 18:59:53

Existing text-to-image (T2I) diffusion models face several limitations,
including large model sizes, slow runtime, and low-quality generation on mobile
devices. This paper aims to address all of these challenges by developing an
extremely small and fast T2I model that generates high-resolution and
high-quality images on mobile platforms. We propose several techniques to
achieve this goal. First, we systematically examine the design choices of the
network architecture to reduce model parameters and latency, while ensuring
high-quality generation. Second, to further improve generation quality, we
employ cross-architecture knowledge distillation from a much larger model,
using a multi-level approach to guide the training of our model from scratch.
Third, we enable a few-step generation by integrating adversarial guidance with
knowledge distillation. For the first time, our model SnapGen, demonstrates the
generation of 1024x1024 px images on a mobile device around 1.4 seconds. On
ImageNet-1K, our model, with only 372M parameters, achieves an FID of 2.06 for
256x256 px generation. On T2I benchmarks (i.e., GenEval and DPG-Bench), our
model with merely 379M parameters, surpasses large-scale models with billions
of parameters at a significantly smaller size (e.g., 7x smaller than SDXL, 14x
smaller than IF-XL).
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
Authors: name: Zhisheng Zhong, name: Chengyao Wang, name: Yuqi Liu, name: Senqiao Yang, name: Longxiang Tang, name: Yuechen Zhang, name: Jingyao Li, name: Tianyuan Qu, name: Yanwei Li, name: Yukang Chen, name: Shaozuo Yu, name: Sitong Wu, name: Eric Lo, name: Shu Liu, name: Jiaya Jia
Published: 2024-12-12 17:50:39

As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond
single-domain capabilities is essential to meet the demands for more versatile
and efficient AI. However, previous omni-models have insufficiently explored
speech, neglecting its integration with multi-modality. We introduce Lyra, an
efficient MLLM that enhances multimodal abilities, including advanced
long-speech comprehension, sound understanding, cross-modality efficiency, and
seamless speech interaction. To achieve efficiency and speech-centric
capabilities, Lyra employs three strategies: (1) leveraging existing
open-source large models and a proposed multi-modality LoRA to reduce training
costs and data requirements; (2) using a latent multi-modality regularizer and
extractor to strengthen the relationship between speech and other modalities,
thereby enhancing model performance; and (3) constructing a high-quality,
extensive dataset that includes 1.5M multi-modal (language, vision, audio) data
samples and 12K long speech samples, enabling Lyra to handle complex long
speech inputs and achieve more robust omni-cognition. Compared to other
omni-methods, Lyra achieves state-of-the-art performance on various
vision-language, vision-speech, and speech-language benchmarks, while also
using fewer computational resources and less training data.
PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh
  Representations
Authors: name: Namgyu Kang, name: Jaemin Oh, name: Youngjoon Hong, name: Eunbyung Park
Published: 2024-12-08 16:58:29

The approximation of Partial Differential Equations (PDEs) using neural
networks has seen significant advancements through Physics-Informed Neural
Networks (PINNs). Despite their straightforward optimization framework and
flexibility in implementing various PDEs, PINNs often suffer from limited
accuracy due to the spectral bias of Multi-Layer Perceptrons (MLPs), which
struggle to effectively learn high-frequency and non-linear components.
Recently, parametric mesh representations in combination with neural networks
have been investigated as a promising approach to eliminate the inductive
biases of neural networks. However, they usually require very high-resolution
grids and a large number of collocation points to achieve high accuracy while
avoiding overfitting issues. In addition, the fixed positions of the mesh
parameters restrict their flexibility, making it challenging to accurately
approximate complex PDEs. To overcome these limitations, we propose
Physics-Informed Gaussians (PIGs), which combine feature embeddings using
Gaussian functions with a lightweight neural network. Our approach uses
trainable parameters for the mean and variance of each Gaussian, allowing for
dynamic adjustment of their positions and shapes during training. This
adaptability enables our model to optimally approximate PDE solutions, unlike
models with fixed parameter positions. Furthermore, the proposed approach
maintains the same optimization framework used in PINNs, allowing us to benefit
from their excellent properties. Experimental results show the competitive
performance of our model across various PDEs, demonstrating its potential as a
robust tool for solving complex PDEs. Our project page is available at
https://namgyukang.github.io/Physics-Informed-Gaussians/
RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World
  Scenarios
Authors: name: Ruiwen Zhou, name: Wenyue Hua, name: Liangming Pan, name: Sitao Cheng, name: Xiaobao Wu, name: En Yu, name: William Yang Wang
Published: 2024-12-12 06:08:46

This paper introduces RuleArena, a novel and challenging benchmark designed
to evaluate the ability of large language models (LLMs) to follow complex,
real-world rules in reasoning. Covering three practical domains -- airline
baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
proficiency in handling intricate natural language instructions that demand
long-context understanding, logical reasoning, and accurate mathematical
computation. Two key attributes distinguish RuleArena from traditional
rule-based reasoning benchmarks: (1) it extends beyond standard first-order
logic representations, and (2) it is grounded in authentic, practical
scenarios, providing insights into the suitability and reliability of LLMs for
real-world applications. Our findings reveal several notable limitations in
LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
becoming confused by similar but distinct regulations, (2) they cannot
consistently perform accurate mathematical computations, even when they
correctly identify the relevant rules, and (3) in general, they perform poorly
in the benchmark. These results highlight significant challenges in advancing
LLMs' rule-guided reasoning capabilities in real-life applications.
Word Sense Linking: Disambiguating Outside the Sandbox
Authors: name: Andrei Stefan Bejgu, name: Edoardo Barba, name: Luigi Procopio, name: Alberte Fernández-Castro, name: Roberto Navigli
Published: 2024-12-12 15:38:34

Word Sense Disambiguation (WSD) is the task of associating a word in a given
context with its most suitable meaning among a set of possible candidates.
While the task has recently witnessed renewed interest, with systems achieving
performances above the estimated inter-annotator agreement, at the time of
writing it still struggles to find downstream applications. We argue that one
of the reasons behind this is the difficulty of applying WSD to plain text.
Indeed, in the standard formulation, models work under the assumptions that a)
all the spans to disambiguate have already been identified, and b) all the
possible candidate senses of each span are provided, both of which are
requirements that are far from trivial. In this work, we present a new task
called Word Sense Linking (WSL) where, given an input text and a reference
sense inventory, systems have to both identify which spans to disambiguate and
then link them to their most suitable meaning.We put forward a
transformer-based architecture for the task and thoroughly evaluate both its
performance and those of state-of-the-art WSD systems scaled to WSL,
iteratively relaxing the assumptions of WSD. We hope that our work will foster
easier integration of lexical semantics into downstream applications.
Phi-4 Technical Report
Authors: name: Marah Abdin, name: Jyoti Aneja, name: Harkirat Behl, name: Sébastien Bubeck, name: Ronen Eldan, name: Suriya Gunasekar, name: Michael Harrison, name: Russell J. Hewett, name: Mojan Javaheripi, name: Piero Kauffmann, name: James R. Lee, name: Yin Tat Lee, name: Yuanzhi Li, name: Weishung Liu, name: Caio C. T. Mendes, name: Anh Nguyen, name: Eric Price, name: Gustavo de Rosa, name: Olli Saarikivi, name: Adil Salim, name: Shital Shah, name: Xin Wang, name: Rachel Ward, name: Yue Wu, name: Dingli Yu, name: Cyril Zhang, name: Yi Zhang
Published: 2024-12-12 03:37:41

We present phi-4, a 14-billion parameter language model developed with a
training recipe that is centrally focused on data quality. Unlike most language
models, where pre-training is based primarily on organic data sources such as
web content or code, phi-4 strategically incorporates synthetic data throughout
the training process. While previous models in the Phi family largely distill
the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
surpasses its teacher model on STEM-focused QA capabilities, giving evidence
that our data-generation and post-training techniques go beyond distillation.
Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
performance relative to its size -- especially on reasoning-focused benchmarks
-- due to improved data, training curriculum, and innovations in the
post-training scheme.
Learned Compression for Compressed Learning
Authors: name: Dan Jacobellis, name: Neeraja J. Yadwadkar
Published: 2024-12-12 16:09:57

Modern sensors produce increasingly rich streams of high-resolution data. Due
to resource constraints, machine learning systems discard the vast majority of
this information via resolution reduction. Compressed-domain learning allows
models to operate on compact latent representations, allowing higher effective
resolution for the same budget. However, existing compression systems are not
ideal for compressed learning. Linear transform coding and end-to-end learned
compression systems reduce bitrate, but do not uniformly reduce dimensionality;
thus, they do not meaningfully increase efficiency. Generative autoencoders
reduce dimensionality, but their adversarial or perceptual objectives lead to
significant information loss. To address these limitations, we introduce WaLLoC
(Wavelet Learned Lossy Compression), a neural codec architecture that combines
linear transform coding with nonlinear dimensionality-reducing autoencoders.
WaLLoC sandwiches a shallow, asymmetric autoencoder and entropy bottleneck
between an invertible wavelet packet transform. Across several key metrics,
WaLLoC outperforms the autoencoders used in state-of-the-art latent diffusion
models. WaLLoC does not require perceptual or adversarial losses to represent
high-frequency detail, providing compatibility with modalities beyond RGB
images and stereo audio. WaLLoC's encoder consists almost entirely of linear
operations, making it exceptionally efficient and suitable for mobile
computing, remote sensing, and learning directly from compressed data. We
demonstrate WaLLoC's capability for compressed-domain learning across several
tasks, including image classification, colorization, document understanding,
and music source separation. Our code, experiments, and pre-trained audio and
image codecs are available at https://ut-sysml.org/walloc
EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via
  Multimodal LLM
Authors: name: Zhuofan Zong, name: Dongzhi Jiang, name: Bingqi Ma, name: Guanglu Song, name: Hao Shao, name: Dazhong Shen, name: Yu Liu, name: Hongsheng Li
Published: 2024-12-12 18:59:48

Significant achievements in personalization of diffusion models have been
witnessed. Conventional tuning-free methods mostly encode multiple reference
images by averaging their image embeddings as the injection condition, but such
an image-independent operation cannot perform interaction among images to
capture consistent visual elements within multiple references. Although the
tuning-based Low-Rank Adaptation (LoRA) can effectively extract consistent
elements within multiple images through the training process, it necessitates
specific finetuning for each distinct image group. This paper introduces
EasyRef, a novel plug-and-play adaptation method that enables diffusion models
to be conditioned on multiple reference images and the text prompt. To
effectively exploit consistent visual elements within multiple images, we
leverage the multi-image comprehension and instruction-following capabilities
of the multimodal large language model (MLLM), prompting it to capture
consistent visual elements based on the instruction. Besides, injecting the
MLLM's representations into the diffusion process through adapters can easily
generalize to unseen domains, mining the consistent visual elements within
unseen data. To mitigate computational costs and enhance fine-grained detail
preservation, we introduce an efficient reference aggregation strategy and a
progressive training scheme. Finally, we introduce MRBench, a new
multi-reference image generation benchmark. Experimental results demonstrate
EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based
methods like LoRA, achieving superior aesthetic quality and robust zero-shot
generalization across diverse domains.
Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity
  Visual Descriptions
Authors: name: Jiarui Zhang, name: Ollie Liu, name: Tianyu Yu, name: Jinyi Hu, name: Willie Neiswanger
Published: 2024-12-11 19:12:13

Multimodal large language models (MLLMs) have made rapid progress in recent
years, yet continue to struggle with low-level visual perception (LLVP) --
particularly the ability to accurately describe the geometric details of an
image. This capability is crucial for applications in areas such as robotics,
medical image analysis, and manufacturing. In this paper, we first introduce
Geoperception, a benchmark designed to evaluate an MLLM's ability to accurately
transcribe 2D geometric information from an image. Using this benchmark, we
demonstrate the limitations of leading MLLMs, and then conduct a comprehensive
empirical study to explore strategies for improving their performance on
geometric tasks. Our findings highlight the benefits of certain model
architectures, training techniques, and data strategies, including the use of
high-fidelity synthetic data and multi-stage training with a data curriculum.
Notably, we find that a data curriculum enables models to learn challenging
geometry understanding tasks which they fail to learn from scratch. Leveraging
these insights, we develop Euclid, a family of models specifically optimized
for strong low-level geometric perception. Although purely trained on synthetic
multimodal data, Euclid shows strong generalization ability to novel geometry
shapes. For instance, Euclid outperforms the best closed-source model,
Gemini-1.5-Pro, by up to 58.56% on certain Geoperception benchmark tasks and
10.65% on average across all tasks.