Orionnnn/hub-stats huggingface dataset:
In-situ graph reasoning and knowledge expansion using Graph-PReFLexOR
Authors: name: Markus J. Buehler
Published: 2025-01-14 13:52:41

The pursuit of automated scientific discovery has fueled progress from
symbolic logic to modern AI, forging new frontiers in reasoning and pattern
recognition. Transformers function as potential systems, where every possible
relationship remains latent potentiality until tasks impose constraints, akin
to measurement. Yet, refining their sampling requires more than probabilistic
selection: solutions must conform to specific structures or rules, ensuring
consistency and the invocation of general principles. We present
Graph-PReFLexOR (Graph-based Preference-based Recursive Language Modeling for
Exploratory Optimization of Reasoning), a framework that combines graph
reasoning with symbolic abstraction to dynamically expand domain knowledge.
Inspired by reinforcement learning, Graph-PReFLexOR defines reasoning as a
structured mapping, where tasks yield knowledge graphs, abstract patterns, and
ultimately, final answers. Inspired by category theory, it encodes concepts as
nodes and their relationships as edges, supporting hierarchical inference and
adaptive learning through isomorphic representations. Demonstrations include
hypothesis generation, materials design, and creative reasoning, such as
discovering relationships between mythological concepts like 'thin places' with
materials science. We propose a 'knowledge garden growth' strategy that
integrates insights across domains, promoting interdisciplinary connections.
Results with a 3-billion-parameter Graph-PReFLexOR model show superior
reasoning depth and adaptability, underscoring the potential for transparent,
multidisciplinary AI-driven discovery. It lays the groundwork for general
autonomous reasoning solutions.
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token
  Marks
Authors: name: Miran Heo, name: Min-Hung Chen, name: De-An Huang, name: Sifei Liu, name: Subhashree Radhakrishnan, name: Seon Joo Kim, name: Yu-Chiang Frank Wang, name: Ryo Hachiuma
Published: 2025-01-14 18:58:04

We present Omni-RGPT, a multimodal large language model designed to
facilitate region-level comprehension for both images and videos. To achieve
consistent region representation across spatio-temporal dimensions, we
introduce Token Mark, a set of tokens highlighting the target regions within
the visual feature space. These tokens are directly embedded into spatial
regions using region prompts (e.g., boxes or masks) and simultaneously
incorporated into the text prompt to specify the target, establishing a direct
connection between visual and text tokens. To further support robust video
understanding without requiring tracklets, we introduce an auxiliary task that
guides Token Mark by leveraging the consistency of the tokens, enabling stable
region interpretation across the video. Additionally, we introduce a
large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT
achieves state-of-the-art results on image and video-based commonsense
reasoning benchmarks while showing strong performance in captioning and
referring expression comprehension tasks.
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models
Authors: name: Michael Toker, name: Ido Galil, name: Hadas Orgad, name: Rinon Gal, name: Yoad Tewel, name: Gal Chechik, name: Yonatan Belinkov
Published: 2025-01-12 08:36:38

Text-to-image (T2I) diffusion models rely on encoded prompts to guide the
image generation process. Typically, these prompts are extended to a fixed
length by adding padding tokens before text encoding. Despite being a default
practice, the influence of padding tokens on the image generation process has
not been investigated. In this work, we conduct the first in-depth analysis of
the role padding tokens play in T2I models. We develop two causal techniques to
analyze how information is encoded in the representation of tokens across
different components of the T2I pipeline. Using these techniques, we
investigate when and how padding tokens impact the image generation process.
Our findings reveal three distinct scenarios: padding tokens may affect the
model's output during text encoding, during the diffusion process, or be
effectively ignored. Moreover, we identify key relationships between these
scenarios and the model's architecture (cross or self-attention) and its
training process (frozen or trained text encoder). These insights contribute to
a deeper understanding of the mechanisms of padding tokens, potentially
informing future model design and training practices in T2I systems.
Enhancing Automated Interpretability with Output-Centric Feature
  Descriptions
Authors: name: Yoav Gur-Arieh, name: Roy Mayan, name: Chen Agassy, name: Atticus Geiger, name: Mor Geva
Published: 2025-01-14 18:53:00

Automated interpretability pipelines generate natural language descriptions
for the concepts represented by features in large language models (LLMs), such
as plants or the first word in a sentence. These descriptions are derived using
inputs that activate the feature, which may be a dimension or a direction in
the model's representation space. However, identifying activating inputs is
costly, and the mechanistic role of a feature in model behavior is determined
both by how inputs cause a feature to activate and by how feature activation
affects outputs. Using steering evaluations, we reveal that current pipelines
provide descriptions that fail to capture the causal effect of the feature on
outputs. To fix this, we propose efficient, output-centric methods for
automatically generating feature descriptions. These methods use the tokens
weighted higher after feature stimulation or the highest weight tokens after
applying the vocabulary "unembedding" head directly to the feature. Our
output-centric descriptions better capture the causal effect of a feature on
model outputs than input-centric descriptions, but combining the two leads to
the best performance on both input and output evaluations. Lastly, we show that
output-centric descriptions can be used to find inputs that activate features
previously thought to be "dead".
AfriHate: A Multilingual Collection of Hate Speech and Abusive Language
  Datasets for African Languages
Authors: name: Shamsuddeen Hassan Muhammad, name: Idris Abdulmumin, name: Abinew Ali Ayele, name: David Ifeoluwa Adelani, name: Ibrahim Said Ahmad, name: Saminu Mohammad Aliyu, name: Nelson Odhiambo Onyango, name: Lilian D. A. Wanzare, name: Samuel Rutunda, name: Lukman Jibril Aliyu, name: Esubalew Alemneh, name: Oumaima Hourrane, name: Hagos Tesfahun Gebremichael, name: Elyas Abdi Ismail, name: Meriem Beloucif, name: Ebrahim Chekol Jibril, name: Andiswa Bukula, name: Rooweither Mabuya, name: Salomey Osei, name: Abigail Oppong, name: Tadesse Destaw Belay, name: Tadesse Kebede Guge, name: Tesfa Tegegne Asfaw, name: Chiamaka Ijeoma Chukwuneke, name: Paul Röttger, name: Seid Muhie Yimam, name: Nedjma Ousidhoum
Published: 2025-01-14 18:00:07

Hate speech and abusive language are global phenomena that need
socio-cultural background knowledge to be understood, identified, and
moderated. However, in many regions of the Global South, there have been
several documented occurrences of (1) absence of moderation and (2) censorship
due to the reliance on keyword spotting out of context. Further, high-profile
individuals have frequently been at the center of the moderation process, while
large and targeted hate speech campaigns against minorities have been
overlooked. These limitations are mainly due to the lack of high-quality data
in the local languages and the failure to include local communities in the
collection, annotation, and moderation processes. To address this issue, we
present AfriHate: a multilingual collection of hate speech and abusive language
datasets in 15 African languages. Each instance in AfriHate is annotated by
native speakers familiar with the local culture. We report the challenges
related to the construction of the datasets and present various classification
baseline results with and without using LLMs. The datasets, individual
annotations, and hate speech and offensive language lexicons are available on
https://github.com/AfriHate/AfriHate
OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for
  LLM Training
Authors: name: Yijiong Yu, name: Ziyun Dai, name: Zekun Wang, name: Wei Wang, name: Ran Chen, name: Ji Pei
Published: 2025-01-14 15:22:47

Large language models (LLMs) have demonstrated remarkable capabilities, but
their success heavily relies on the quality of pretraining corpora. For Chinese
LLMs, the scarcity of high-quality Chinese datasets presents a significant
challenge, often limiting their performance. To address this issue, we propose
the OpenCSG Chinese Corpus, a series of high-quality datasets specifically
designed for LLM pretraining, post-training, and fine-tuning. This corpus
includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and
Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets
focus on filtered, high-quality content derived from diverse Chinese web
sources; Cosmopedia-chinese provides synthetic, textbook-style data for
knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and
diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its
high-quality text, diverse coverage across domains, and scalable, reproducible
data curation processes. Additionally, we conducted extensive experimental
analyses, including evaluations on smaller parameter models, which demonstrated
significant performance improvements in tasks such as C-Eval, showcasing the
effectiveness of the corpus for training Chinese LLMs.
Potential and Perils of Large Language Models as Judges of Unstructured
  Textual Data
Authors: name: Rewina Bedemariam, name: Natalie Perez, name: Sreyoshi Bhaduri, name: Satya Kapoor, name: Alex Gil, name: Elizabeth Conjar, name: Ikkei Itoku, name: David Theil, name: Aman Chadha, name: Naumaan Nayyar
Published: 2025-01-14 14:49:14

Rapid advancements in large language models have unlocked remarkable
capabilities when it comes to processing and summarizing unstructured text
data. This has implications for the analysis of rich, open-ended datasets, such
as survey responses, where LLMs hold the promise of efficiently distilling key
themes and sentiments. However, as organizations increasingly turn to these
powerful AI systems to make sense of textual feedback, a critical question
arises, can we trust LLMs to accurately represent the perspectives contained
within these text based datasets? While LLMs excel at generating human-like
summaries, there is a risk that their outputs may inadvertently diverge from
the true substance of the original responses. Discrepancies between the
LLM-generated outputs and the actual themes present in the data could lead to
flawed decision-making, with far-reaching consequences for organizations. This
research investigates the effectiveness of LLMs as judge models to evaluate the
thematic alignment of summaries generated by other LLMs. We utilized an
Anthropic Claude model to generate thematic summaries from open-ended survey
responses, with Amazon's Titan Express, Nova Pro, and Meta's Llama serving as
LLM judges. The LLM-as-judge approach was compared to human evaluations using
Cohen's kappa, Spearman's rho, and Krippendorff's alpha, validating a scalable
alternative to traditional human centric evaluation methods. Our findings
reveal that while LLMs as judges offer a scalable solution comparable to human
raters, humans may still excel at detecting subtle, context-specific nuances.
This research contributes to the growing body of knowledge on AI assisted text
analysis. We discuss limitations and provide recommendations for future
research, emphasizing the need for careful consideration when generalizing LLM
judge models across various contexts and use cases.
Tarsier2: Advancing Large Vision-Language Models from Detailed Video
  Description to Comprehensive Video Understanding
Authors: name: Liping Yuan, name: Jiawei Wang, name: Haomiao Sun, name: Yuchen Zhang, name: Yuan Lin
Published: 2025-01-14 06:54:39

We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM)
designed for generating detailed and accurate video descriptions, while also
exhibiting superior general video understanding capabilities. Tarsier2 achieves
significant advancements through three key upgrades: (1) Scaling pre-training
data from 11M to 40M video-text pairs, enriching both volume and diversity; (2)
Performing fine-grained temporal alignment during supervised fine-tuning; (3)
Using model-based sampling to automatically construct preference data and
applying DPO training for optimization. Extensive experiments show that
Tarsier2-7B consistently outperforms leading proprietary models, including
GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K
benchmark, Tarsier2-7B improves F1 by 2.8\% over GPT-4o and 5.8\% over
Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6\%
performance advantage over GPT-4o and +24.9\% over Gemini-1.5-Pro. Tarsier2-7B
also sets new state-of-the-art results across 15 public benchmarks, spanning
tasks such as video question-answering, video grounding, hallucination test,
and embodied question-answering, demonstrating its versatility as a robust
generalist vision-language model.
PokerBench: Training Large Language Models to become Professional Poker
  Players
Authors: name: Richard Zhuang, name: Akshat Gupta, name: Richard Yang, name: Aniket Rahane, name: Zhengyu Li, name: Gopala Anumanchipalli
Published: 2025-01-14 18:59:03

We introduce PokerBench - a benchmark for evaluating the poker-playing
abilities of large language models (LLMs). As LLMs excel in traditional NLP
tasks, their application to complex, strategic games like poker poses a new
challenge. Poker, an incomplete information game, demands a multitude of skills
such as mathematics, reasoning, planning, strategy, and a deep understanding of
game theory and human psychology. This makes Poker the ideal next frontier for
large language models. PokerBench consists of a comprehensive compilation of
11,000 most important scenarios, split between pre-flop and post-flop play,
developed in collaboration with trained poker players. We evaluate prominent
models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models,
finding that all state-of-the-art LLMs underperform in playing optimal poker.
However, after fine-tuning, these models show marked improvements. We validate
PokerBench by having models with different scores compete with each other,
demonstrating that higher scores on PokerBench lead to higher win rates in
actual poker games. Through gameplay between our fine-tuned model and GPT-4, we
also identify limitations of simple supervised fine-tuning for learning optimal
playing strategy, suggesting the need for more advanced methodologies for
effectively training language models to excel in games. PokerBench thus
presents a unique benchmark for a quick and reliable evaluation of the
poker-playing ability of LLMs as well as a comprehensive benchmark to study the
progress of LLMs in complex game-playing scenarios. The dataset and code will
be made available at: https://github.com/pokerllm/pokerbench.
Democratizing Text-to-Image Masked Generative Models with Compact
  Text-Aware One-Dimensional Tokens
Authors: name: Dongwon Kim, name: Ju He, name: Qihang Yu, name: Chenglin Yang, name: Xiaohui Shen, name: Suha Kwak, name: Liang-Chieh Chen
Published: 2025-01-13 22:37:17

Image tokenizers form the foundation of modern text-to-image generative
models but are notoriously difficult to train. Furthermore, most existing
text-to-image models rely on large-scale, high-quality private datasets, making
them challenging to replicate. In this work, we introduce Text-Aware
Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful
image tokenizer that can utilize either discrete or continuous 1-dimensional
tokens. TA-TiTok uniquely integrates textual information during the tokenizer
decoding stage (i.e., de-tokenization), accelerating convergence and enhancing
performance. TA-TiTok also benefits from a simplified, yet effective, one-stage
training process, eliminating the need for the complex two-stage distillation
used in previous 1-dimensional tokenizers. This design allows for seamless
scalability to large datasets. Building on this, we introduce a family of
text-to-image Masked Generative Models (MaskGen), trained exclusively on open
data while achieving comparable performance to models trained on private data.
We aim to release both the efficient, strong TA-TiTok tokenizers and the
open-data, open-weight MaskGen models to promote broader access and democratize
the field of text-to-image masked generative models.
HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
Authors: name: Abhilasha Ravichander, name: Shrusti Ghela, name: David Wadden, name: Yejin Choi
Published: 2025-01-14 18:13:08

Despite their impressive ability to generate high-quality and fluent text,
generative large language models (LLMs) also produce hallucinations: statements
that are misaligned with established world knowledge or provided input context.
However, measuring hallucination can be challenging, as having humans verify
model generations on-the-fly is both expensive and time-consuming. In this
work, we release HALoGEN, a comprehensive hallucination benchmark consisting
of: (1) 10,923 prompts for generative models spanning nine domains including
programming, scientific attribution, and summarization, and (2) automatic
high-precision verifiers for each use case that decompose LLM generations into
atomic units, and verify each unit against a high-quality knowledge source. We
use this framework to evaluate ~150,000 generations from 14 language models,
finding that even the best-performing models are riddled with hallucinations
(sometimes up to 86% of generated atomic facts depending on the domain). We
further define a novel error classification for LLM hallucinations based on
whether they likely stem from incorrect recollection of training data (Type A
errors), or incorrect knowledge in training data (Type B errors), or are
fabrication (Type C errors). We hope our framework provides a foundation to
enable the principled study of why generative models hallucinate, and advances
the development of trustworthy large language models.
FramePainter: Endowing Interactive Image Editing with Video Diffusion
  Priors
Authors: name: Yabo Zhang, name: Xinpeng Zhou, name: Yihan Zeng, name: Hang Xu, name: Hui Li, name: Wangmeng Zuo
Published: 2025-01-14 16:09:16

Interactive image editing allows users to modify images through visual
interaction operations such as drawing, clicking, and dragging. Existing
methods construct such supervision signals from videos, as they capture how
objects change with various physical interactions. However, these models are
usually built upon text-to-image diffusion models, so necessitate (i) massive
training samples and (ii) an additional reference encoder to learn real-world
dynamics and visual consistency. In this paper, we reformulate this task as an
image-to-video generation problem, so that inherit powerful video diffusion
priors to reduce training costs and ensure temporal consistency. Specifically,
we introduce FramePainter as an efficient instantiation of this formulation.
Initialized with Stable Video Diffusion, it only uses a lightweight sparse
control encoder to inject editing signals. Considering the limitations of
temporal attention in handling large motion between two frames, we further
propose matching attention to enlarge the receptive field while encouraging
dense correspondence between edited and source image tokens. We highlight the
effectiveness and efficiency of FramePainter across various of editing signals:
it domainantly outperforms previous state-of-the-art methods with far less
training data, achieving highly seamless and coherent editing of images, \eg,
automatically adjust the reflection of the cup. Moreover, FramePainter also
exhibits exceptional generalization in scenarios not present in real-world
videos, \eg, transform the clownfish into shark-like shape. Our code will be
available at https://github.com/YBYBZhang/FramePainter.
MangaNinja: Line Art Colorization with Precise Reference Following
Authors: name: Zhiheng Liu, name: Ka Leong Cheng, name: Xi Chen, name: Jie Xiao, name: Hao Ouyang, name: Kai Zhu, name: Yu Liu, name: Yujun Shen, name: Qifeng Chen, name: Ping Luo
Published: 2025-01-14 18:59:55

Derived from diffusion models, MangaNinjia specializes in the task of
reference-guided line art colorization. We incorporate two thoughtful designs
to ensure precise character detail transcription, including a patch shuffling
module to facilitate correspondence learning between the reference color image
and the target line art, and a point-driven control scheme to enable
fine-grained color matching. Experiments on a self-collected benchmark
demonstrate the superiority of our model over current solutions in terms of
precise colorization. We further showcase the potential of the proposed
interactive point control in handling challenging cases, cross-character
colorization, multi-reference harmonization, beyond the reach of existing
algorithms.
Diffusion Adversarial Post-Training for One-Step Video Generation
Authors: name: Shanchuan Lin, name: Xin Xia, name: Yuxi Ren, name: Ceyuan Yang, name: Xuefeng Xiao, name: Lu Jiang
Published: 2025-01-14 18:51:48

The diffusion models are widely used for image and video generation, but
their iterative generation process is slow and expansive. While existing
distillation approaches have demonstrated the potential for one-step generation
in the image domain, they still suffer from significant quality degradation. In
this work, we propose Adversarial Post-Training (APT) against real data
following diffusion pre-training for one-step video generation. To improve the
training stability and quality, we introduce several improvements to the model
architecture and training procedures, along with an approximated R1
regularization objective. Empirically, our experiments show that our
adversarial post-trained model, Seaweed-APT, can generate 2-second, 1280x720,
24fps videos in real time using a single forward evaluation step. Additionally,
our model is capable of generating 1024px images in a single step, achieving
quality comparable to state-of-the-art methods.