Orionnnn/hub-stats huggingface dataset:
Qwen2.5-1M Technical Report
Authors: name: An Yang, name: Bowen Yu, name: Chengyuan Li, name: Dayiheng Liu, name: Fei Huang, name: Haoyan Huang, name: Jiandong Jiang, name: Jianhong Tu, name: Jianwei Zhang, name: Jingren Zhou, name: Junyang Lin, name: Kai Dang, name: Kexin Yang, name: Le Yu, name: Mei Li, name: Minmin Sun, name: Qin Zhu, name: Rui Men, name: Tao He, name: Weijia Xu, name: Wenbiao Yin, name: Wenyuan Yu, name: Xiafei Qiu, name: Xingzhang Ren, name: Xinlong Yang, name: Yong Li, name: Zhiying Xu, name: Zipeng Zhang
Published: 2025-01-26 03:47:25

We introduce Qwen2.5-1M, a series of models that extend the context length to
1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series
have significantly enhanced long-context capabilities through long-context
pre-training and post-training. Key techniques such as long data synthesis,
progressive pre-training, and multi-stage supervised fine-tuning are employed
to effectively enhance long-context performance while reducing training costs.
  To promote the use of long-context models among a broader user base, we
present and open-source our inference framework. This framework includes a
length extrapolation method that can expand the model context lengths by at
least four times, or even more, without additional training. To reduce
inference costs, we implement a sparse attention method along with chunked
prefill optimization for deployment scenarios and a sparsity refinement method
to improve precision. Additionally, we detail our optimizations in the
inference engine, including kernel optimization, pipeline parallelism, and
scheduling optimization, which significantly enhance overall inference
performance. By leveraging our inference framework, the Qwen2.5-1M models
achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million
tokens of context. This framework provides an efficient and powerful solution
for developing applications that require long-context processing using
open-source models.
  The Qwen2.5-1M series currently includes the open-source models
Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed
model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly
improved in long-context tasks without compromising performance in
short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model
significantly outperforms GPT-4o-mini in long-context tasks and supports
contexts eight times longer.
Baichuan-Omni-1.5 Technical Report
Authors: name: Yadong Li, name: Jun Liu, name: Tao Zhang, name: Tao Zhang, name: Song Chen, name: Tianpeng Li, name: Zehuan Li, name: Lijun Liu, name: Lingfeng Ming, name: Guosheng Dong, name: Da Pan, name: Chong Li, name: Yuanbo Fang, name: Dongdong Kuang, name: Mingrui Wang, name: Chenglin Zhu, name: Youwei Zhang, name: Hongyu Guo, name: Fengyu Zhang, name: Yuran Wang, name: Bowen Ding, name: Wei Song, name: Xu Li, name: Yuqi Huo, name: Zheng Liang, name: Shusen Zhang, name: Xin Wu, name: Shuai Zhao, name: Linchu Xiong, name: Yozhen Wu, name: Jiahui Ye, name: Wenhao Lu, name: Bowen Li, name: Yan Zhang, name: Yaqi Zhou, name: Xin Chen, name: Lei Su, name: Hongda Zhang, name: Fuzhong Chen, name: Xuezhen Dong, name: Na Nie, name: Zhiying Wu, name: Bin Xiao, name: Ting Li, name: Shunya Dang, name: Ping Zhang, name: Yijia Sun, name: Jincheng Wu, name: Jinjie Yang, name: Xionghai Lin, name: Zhi Ma, name: Kegeng Wu, name: Jia li, name: Aiyuan Yang, name: Hui Liu, name: Jianqiang Zhang, name: Xiaoxi Chen, name: Guangwei Ai, name: Wentao Zhang, name: Yicong Chen, name: Xiaoqin Huang, name: Kun Li, name: Wenjing Luo, name: Yifei Duan, name: Lingling Zhu, name: Ran Xiao, name: Zhe Su, name: Jiani Pu, name: Dian Wang, name: Xu Jia, name: Tianyu Zhang, name: Mengyu Ai, name: Mang Wang, name: Yujing Qiao, name: Lei Zhang, name: Yanjun Shen, name: Fan Yang, name: Miao Zhen, name: Yijie Zhou, name: Mingyang Chen, name: Fei Li, name: Chenzheng Zhu, name: Keer Lu, name: Yaqi Zhao, name: Hao Liang, name: Youquan Li, name: Yanzhao Qin, name: Linzhuang Sun, name: Jianhua Xu, name: Haoze Sun, name: Mingan Lin, name: Zenan Zhou, name: Weipeng Chen
Published: 2025-01-26 02:19:03

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
omni-modal understanding capabilities but also provides end-to-end audio
generation capabilities. To achieve fluent and high-quality interaction across
modalities without compromising the capabilities of any modality, we
prioritized optimizing three key aspects. First, we establish a comprehensive
data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
high-quality data (text, audio, and vision). Second, an audio-tokenizer
(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
acoustic information from audio, enabling seamless integration and enhanced
compatibility with MLLM. Lastly, we designed a multi-stage training strategy
that progressively integrates multimodal alignment and multitask fine-tuning,
ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
comprehensive omni-modal capabilities. Notably, it achieves results comparable
to leading models such as Qwen2-VL-72B across various multimodal medical
benchmarks.
CatV2TON: Taming Diffusion Transformers for Vision-Based Virtual Try-On
  with Temporal Concatenation
Authors: name: Zheng Chong, name: Wenqing Zhang, name: Shiyue Zhang, name: Jun Zheng, name: Xiao Dong, name: Haoxiang Li, name: Yiling Wu, name: Dongmei Jiang, name: Xiaodan Liang
Published: 2025-01-20 08:09:36

Virtual try-on (VTON) technology has gained attention due to its potential to
transform online retail by enabling realistic clothing visualization of images
and videos. However, most existing methods struggle to achieve high-quality
results across image and video try-on tasks, especially in long video
scenarios. In this work, we introduce CatV2TON, a simple and effective
vision-based virtual try-on (V2TON) method that supports both image and video
try-on tasks with a single diffusion transformer model. By temporally
concatenating garment and person inputs and training on a mix of image and
video datasets, CatV2TON achieves robust try-on performance across static and
dynamic settings. For efficient long-video generation, we propose an
overlapping clip-based inference strategy that uses sequential frame guidance
and Adaptive Clip Normalization (AdaCN) to maintain temporal consistency with
reduced resource demands. We also present ViViD-S, a refined video try-on
dataset, achieved by filtering back-facing frames and applying 3D mask
smoothing for enhanced temporal consistency. Comprehensive experiments
demonstrate that CatV2TON outperforms existing methods in both image and video
try-on tasks, offering a versatile and reliable solution for realistic virtual
try-ons across diverse scenarios.
Question Answering on Patient Medical Records with Private Fine-Tuned
  LLMs
Authors: name: Sara Kothari, name: Ayush Gupta
Published: 2025-01-23 14:13:56

Healthcare systems continuously generate vast amounts of electronic health
records (EHRs), commonly stored in the Fast Healthcare Interoperability
Resources (FHIR) standard. Despite the wealth of information in these records,
their complexity and volume make it difficult for users to retrieve and
interpret crucial health insights. Recent advances in Large Language Models
(LLMs) offer a solution, enabling semantic question answering (QA) over medical
data, allowing users to interact with their health records more effectively.
However, ensuring privacy and compliance requires edge and private deployments
of LLMs.
  This paper proposes a novel approach to semantic QA over EHRs by first
identifying the most relevant FHIR resources for a user query (Task1) and
subsequently answering the query based on these resources (Task2). We explore
the performance of privately hosted, fine-tuned LLMs, evaluating them against
benchmark models such as GPT-4 and GPT-4o. Our results demonstrate that
fine-tuned LLMs, while 250x smaller in size, outperform GPT-4 family models by
0.55% in F1 score on Task1 and 42% on Meteor Task in Task2. Additionally, we
examine advanced aspects of LLM usage, including sequential fine-tuning, model
self-evaluation (narcissistic evaluation), and the impact of training data size
on performance. The models and datasets are available here:
https://huggingface.co/genloop
RL + Transformer = A General-Purpose Problem Solver
Authors: name: Micah Rentschler, name: Jesse Roberts
Published: 2025-01-24 01:55:20

What if artificial intelligence could not only solve problems for which it
was trained but also learn to teach itself to solve new problems (i.e.,
meta-learn)? In this study, we demonstrate that a pre-trained transformer
fine-tuned with reinforcement learning over multiple episodes develops the
ability to solve problems that it has never encountered before - an emergent
ability called In-Context Reinforcement Learning (ICRL). This powerful
meta-learner not only excels in solving unseen in-distribution environments
with remarkable sample efficiency, but also shows strong performance in
out-of-distribution environments. In addition, we show that it exhibits
robustness to the quality of its training data, seamlessly stitches together
behaviors from its context, and adapts to non-stationary environments. These
behaviors demonstrate that an RL-trained transformer can iteratively improve
upon its own solutions, making it an excellent general-purpose problem solver.
AdaIR: Adaptive All-in-One Image Restoration via Frequency Mining and
  Modulation
Authors: name: Yuning Cui, name: Syed Waqas Zamir, name: Salman Khan, name: Alois Knoll, name: Mubarak Shah, name: Fahad Shahbaz Khan
Published: 2024-03-21 17:58:14

In the image acquisition process, various forms of degradation, including
noise, haze, and rain, are frequently introduced. These degradations typically
arise from the inherent limitations of cameras or unfavorable ambient
conditions. To recover clean images from degraded versions, numerous
specialized restoration methods have been developed, each targeting a specific
type of degradation. Recently, all-in-one algorithms have garnered significant
attention by addressing different types of degradations within a single model
without requiring prior information of the input degradation type. However,
these methods purely operate in the spatial domain and do not delve into the
distinct frequency variations inherent to different degradation types. To
address this gap, we propose an adaptive all-in-one image restoration network
based on frequency mining and modulation. Our approach is motivated by the
observation that different degradation types impact the image content on
different frequency subbands, thereby requiring different treatments for each
restoration task. Specifically, we first mine low- and high-frequency
information from the input features, guided by the adaptively decoupled spectra
of the degraded image. The extracted features are then modulated by a
bidirectional operator to facilitate interactions between different frequency
components. Finally, the modulated features are merged into the original input
for a progressively guided restoration. With this approach, the model achieves
adaptive reconstruction by accentuating the informative frequency subbands
according to different input degradations. Extensive experiments demonstrate
that the proposed method achieves state-of-the-art performance on different
image restoration tasks, including denoising, dehazing, deraining, motion
deblurring, and low-light image enhancement. Our code is available at
https://github.com/c-yn/AdaIR.
Denoising as Adaptation: Noise-Space Domain Adaptation for Image
  Restoration
Authors: name: Kang Liao, name: Zongsheng Yue, name: Zhouxia Wang, name: Chen Change Loy
Published: 2024-06-26 17:40:30

Although learning-based image restoration methods have made significant
progress, they still struggle with limited generalization to real-world
scenarios due to the substantial domain gap caused by training on synthetic
data. Existing methods address this issue by improving data synthesis
pipelines, estimating degradation kernels, employing deep internal learning,
and performing domain adaptation and regularization. Previous domain adaptation
methods have sought to bridge the domain gap by learning domain-invariant
knowledge in either feature or pixel space. However, these techniques often
struggle to extend to low-level vision tasks within a stable and compact
framework. In this paper, we show that it is possible to perform domain
adaptation via the noise space using diffusion models. In particular, by
leveraging the unique property of how auxiliary conditional inputs influence
the multi-step denoising process, we derive a meaningful diffusion loss that
guides the restoration model in progressively aligning both restored synthetic
and real-world outputs with a target clean distribution. We refer to this
method as denoising as adaptation. To prevent shortcuts during joint training,
we present crucial strategies such as channel-shuffling layer and
residual-swapping contrastive learning in the diffusion model. They implicitly
blur the boundaries between conditioned synthetic and real data and prevent the
reliance of the model on easily distinguishable features. Experimental results
on three classical image restoration tasks, namely denoising, deblurring, and
deraining, demonstrate the effectiveness of the proposed method.
Multiview Equivariance Improves 3D Correspondence Understanding with
  Minimal Feature Finetuning
Authors: name: Yang You, name: Yixin Li, name: Congyue Deng, name: Yue Wang, name: Leonidas Guibas
Published: 2024-11-29 04:02:11

Vision foundation models, particularly the ViT family, have revolutionized
image understanding by providing rich semantic features. However, despite their
success in 2D comprehension, their abilities on grasping 3D spatial
relationships are still unclear. In this work, we evaluate and enhance the 3D
awareness of ViT-based models. We begin by systematically assessing their
ability to learn 3D equivariant features, specifically examining the
consistency of semantic embeddings across different viewpoints. Our findings
indicate that improved 3D equivariance leads to better performance on various
downstream tasks, including pose estimation, tracking, and semantic transfer.
Building on this insight, we propose a simple yet effective finetuning strategy
based on 3D correspondences, which significantly enhances the 3D correspondence
understanding of existing vision models. Remarkably, even finetuning on a
single object for just one iteration results in substantial performance gains.
All code and resources will be made publicly available to support further
advancements in 3D-aware vision models. Our code is available at
https://github.com/qq456cvb/3DCorrEnhance.
GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Authors: name: Akashah Shabbir, name: Mohammed Zumri, name: Mohammed Bennamoun, name: Fahad S. Khan, name: Salman Khan
Published: 2025-01-23 18:59:30

Recent advances in large multimodal models (LMMs) have recognized
fine-grained grounding as an imperative factor of visual understanding and
dialogue. However, the benefits of such representation in LMMs are limited to
the natural image domain, and these models perform poorly for remote sensing
(RS). The distinct overhead viewpoint, scale variation, and presence of small
objects in high-resolution RS imagery present a unique challenge in
region-level comprehension. Moreover, the development of the grounding
conversation capability of LMMs within RS is hindered by the lack of granular,
RS domain-specific grounded data. Addressing these limitations, we propose
GeoPixel - the first end-to-end high resolution RS-LMM that supports
pixel-level grounding. This capability allows fine-grained visual perception by
generating interleaved masks in conversation. GeoPixel supports up to 4K HD
resolution in any aspect ratio, ideal for high-precision RS image analysis. To
support the grounded conversation generation (GCG) in RS imagery, we curate a
visually grounded dataset GeoPixelD through a semi-automated pipeline that
utilizes set-of-marks prompting and spatial priors tailored for RS data to
methodically control the data generation process. GeoPixel demonstrates
superior performance in pixel-level comprehension, surpassing existing LMMs in
both single-target and multi-target segmentation tasks. Our methodological
ablation studies validate the effectiveness of each component in the overall
architecture. Our code and data will be publicly released.
Redundancy Principles for MLLMs Benchmarks
Authors: name: Zicheng Zhang, name: Xiangyu Zhao, name: Xinyu Fang, name: Chunyi Li, name: Xiaohong Liu, name: Xiongkuo Min, name: Haodong Duan, name: Kai Chen, name: Guangtao Zhai
Published: 2025-01-20 08:09:42

With the rapid iteration of Multi-modality Large Language Models (MLLMs) and
the evolving demands of the field, the number of benchmarks produced annually
has surged into the hundreds. The rapid growth has inevitably led to
significant redundancy among benchmarks. Therefore, it is crucial to take a
step back and critically assess the current state of redundancy and propose
targeted principles for constructing effective MLLM benchmarks. In this paper,
we focus on redundancy from three key perspectives: 1) Redundancy of benchmark
capability dimensions, 2) Redundancy in the number of test questions, and 3)
Cross-benchmark redundancy within specific domains. Through the comprehensive
analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we
aim to quantitatively measure the level of redundancy lies in existing MLLM
evaluations, provide valuable insights to guide the future development of MLLM
benchmarks, and offer strategies to refine and address redundancy issues
effectively.
Relightable Full-Body Gaussian Codec Avatars
Authors: name: Shaofei Wang, name: Tomas Simon, name: Igor Santesteban, name: Timur Bagautdinov, name: Junxuan Li, name: Vasu Agrawal, name: Fabian Prada, name: Shoou-I Yu, name: Pace Nalbone, name: Matt Gramlich, name: Roman Lubachersky, name: Chenglei Wu, name: Javier Romero, name: Jason Saragih, name: Michael Zollhoefer, name: Andreas Geiger, name: Siyu Tang, name: Shunsuke Saito
Published: 2025-01-24 18:59:15

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for
modeling relightable full-body avatars with fine-grained details including face
and hands. The unique challenge for relighting full-body avatars lies in the
large deformations caused by body articulation and the resulting impact on
appearance caused by light transport. Changes in body pose can dramatically
change the orientation of body surfaces with respect to lights, resulting in
both local appearance changes due to changes in local light transport
functions, as well as non-local changes due to occlusion between body parts. To
address this, we decompose the light transport into local and non-local
effects. Local appearance changes are modeled using learnable zonal harmonics
for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are
highly efficient to rotate under articulation. This allows us to learn diffuse
radiance transfer in a local coordinate frame, which disentangles the local
radiance transfer from the articulation of the body. To account for non-local
appearance changes, we introduce a shadow network that predicts shadows given
precomputed incoming irradiance on a base mesh. This facilitates the learning
of non-local shadowing between the body parts. Finally, we use a deferred
shading approach to model specular radiance transfer and better capture
reflections and highlights such as eye glints. We demonstrate that our approach
successfully models both the local and non-local light transport required for
relightable full-body avatars, with a superior generalization ability under
novel illumination conditions and unseen poses.
RealCritic: Towards Effectiveness-Driven Evaluation of Language Model
  Critiques
Authors: name: Zhengyang Tang, name: Ziniu Li, name: Zhenyang Xiao, name: Tian Ding, name: Ruoyu Sun, name: Benyou Wang, name: Dayiheng Liu, name: Fei Huang, name: Tianyu Liu, name: Bowen Yu, name: Junyang Lin
Published: 2025-01-24 13:48:10

Critiques are important for enhancing the performance of Large Language
Models (LLMs), enabling both self-improvement and constructive feedback for
others by identifying flaws and suggesting improvements. However, evaluating
the critique capabilities of LLMs presents a significant challenge due to the
open-ended nature of the task. In this work, we introduce a new benchmark
designed to assess the critique capabilities of LLMs. Unlike existing
benchmarks, which typically function in an open-loop fashion, our approach
employs a closed-loop methodology that evaluates the quality of corrections
generated from critiques. Moreover, the benchmark incorporates features such as
self-critique, cross-critique, and iterative critique, which are crucial for
distinguishing the abilities of advanced reasoning models from more classical
ones. We implement this benchmark using eight challenging reasoning tasks. We
have several interesting findings. First, despite demonstrating comparable
performance in direct chain-of-thought generation, classical LLMs significantly
lag behind the advanced reasoning-based model o1-mini across all critique
scenarios. Second, in self-critique and iterative critique settings, classical
LLMs may even underperform relative to their baseline capabilities. We hope
that this benchmark will serve as a valuable resource to guide future
advancements. The code and data are available at
https://github.com/tangzhy/RealCritic.
Chain-of-Retrieval Augmented Generation
Authors: name: Liang Wang, name: Haonan Chen, name: Nan Yang, name: Xiaolong Huang, name: Zhicheng Dou, name: Furu Wei
Published: 2025-01-24 09:12:52

This paper introduces an approach for training o1-like RAG models that
retrieve and reason over relevant information step by step before generating
the final answer. Conventional RAG methods usually perform a single retrieval
step before the generation process, which limits their effectiveness in
addressing complex queries due to imperfect retrieval results. In contrast, our
proposed method, CoRAG (Chain-of-Retrieval Augmented Generation), allows the
model to dynamically reformulate the query based on the evolving state. To
train CoRAG effectively, we utilize rejection sampling to automatically
generate intermediate retrieval chains, thereby augmenting existing RAG
datasets that only provide the correct final answer. At test time, we propose
various decoding strategies to scale the model's test-time compute by
controlling the length and number of sampled retrieval chains. Experimental
results across multiple benchmarks validate the efficacy of CoRAG, particularly
in multi-hop question answering tasks, where we observe more than 10 points
improvement in EM score compared to strong baselines. On the KILT benchmark,
CoRAG establishes a new state-of-the-art performance across a diverse range of
knowledge-intensive tasks. Furthermore, we offer comprehensive analyses to
understand the scaling behavior of CoRAG, laying the groundwork for future
research aimed at developing factual and grounded foundation models.