Orionnnn/hub-stats huggingface dataset:
Video-Panda: Parameter-efficient Alignment for Encoder-free
  Video-Language Models
Authors: name: Jinhui Yi, name: Syed Talal Wasim, name: Yanan Luo, name: Muzammal Naseer, name: Juergen Gall
Published: 2024-12-24 18:59:56

We present an efficient encoder-free approach for video-language
understanding that achieves competitive performance while significantly
reducing computational overhead. Current video-language models typically rely
on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
parameters), creating a substantial computational burden when processing
multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
Block (STAB) that directly processes video inputs without requiring pre-trained
encoders while using only 45M parameters for visual processing - at least a
6.5times reduction compared to traditional approaches. The STAB architecture
combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
efficient spatial downsampling through learned attention and separate
mechanisms for modeling frame-level and video-level relationships. Our model
achieves comparable or superior performance to encoder-based approaches for
open-ended video question answering on standard benchmarks. The fine-grained
video question-answering evaluation demonstrates our model's effectiveness,
outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
aspects like correctness and temporal understanding. Extensive ablation studies
validate our architectural choices and demonstrate the effectiveness of our
spatio-temporal modeling approach while achieving 3-4times faster processing
speeds than previous methods. Code is available at
https://github.com/jh-yi/Video-Panda.
Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via
  Collective Monte Carlo Tree Search
Authors: name: Huanjin Yao, name: Jiaxing Huang, name: Wenhao Wu, name: Jingyi Zhang, name: Yibo Wang, name: Shunyu Liu, name: Yingjie Wang, name: Yuxin Song, name: Haocheng Feng, name: Li Shen, name: Dacheng Tao
Published: 2024-12-24 10:07:51

In this work, we aim to develop an MLLM that understands and solves questions
by learning to create each intermediate step of the reasoning involved till the
final answer. To this end, we propose Collective Monte Carlo Tree Search
(CoMCTS), a new learning-to-reason method for MLLMs, which introduces the
concept of collective learning into ``tree search'' for effective and efficient
reasoning-path searching and learning. The core idea of CoMCTS is to leverage
collective knowledge from multiple models to collaboratively conjecture, search
and identify effective reasoning paths toward correct answers via four
iterative operations including Expansion, Simulation and Error Positioning,
Backpropagation, and Selection. Using CoMCTS, we construct Mulberry-260k, a
multimodal dataset with a tree of rich, explicit and well-defined reasoning
nodes for each question. With Mulberry-260k, we perform collective SFT to train
our model, Mulberry, a series of MLLMs with o1-like step-by-step Reasoning and
Reflection capabilities. Extensive experiments demonstrate the superiority of
our proposed methods on various benchmarks. Code will be available at
https://github.com/HJYao00/Mulberry
PepTune: De Novo Generation of Therapeutic Peptides with
  Multi-Objective-Guided Discrete Diffusion
Authors: name: Sophia Tang, name: Yinuo Zhang, name: Pranam Chatterjee
Published: 2024-12-23 18:38:49

Peptide therapeutics, a major class of medicines, have achieved remarkable
success across diseases such as diabetes and cancer, with landmark examples
such as GLP-1 receptor agonists revolutionizing the treatment of type-2
diabetes and obesity. Despite their success, designing peptides that satisfy
multiple conflicting objectives, such as target binding affinity, solubility,
and membrane permeability, remains a major challenge. Classical drug
development and structure-based design are ineffective for such tasks, as they
fail to optimize global functional properties critical for therapeutic
efficacy. Existing generative frameworks are largely limited to continuous
spaces, unconditioned outputs, or single-objective guidance, making them
unsuitable for discrete sequence optimization across multiple properties. To
address this, we present PepTune, a multi-objective discrete diffusion model
for the simultaneous generation and optimization of therapeutic peptide SMILES.
Built on the Masked Discrete Language Model (MDLM) framework, PepTune ensures
valid peptide structures with state-dependent masking schedules and
penalty-based objectives. To guide the diffusion process, we propose a Monte
Carlo Tree Search (MCTS)-based strategy that balances exploration and
exploitation to iteratively refine Pareto-optimal sequences. MCTS integrates
classifier-based rewards with search-tree expansion, overcoming gradient
estimation challenges and data sparsity inherent to discrete spaces. Using
PepTune, we generate diverse, chemically-modified peptides optimized for
multiple therapeutic properties, including target binding affinity, membrane
permeability, solubility, hemolysis, and non-fouling characteristics on various
disease-relevant targets. In total, our results demonstrate that MCTS-guided
discrete diffusion is a powerful and modular approach for multi-objective
sequence design in discrete state spaces.
Token-Budget-Aware LLM Reasoning
Authors: name: Tingxu Han, name: Chunrong Fang, name: Shiyu Zhao, name: Shiqing Ma, name: Zhenyu Chen, name: Zhenting Wang
Published: 2024-12-24 16:55:45

Reasoning is critical for large language models (LLMs) to excel in a wide
range of tasks. While methods like Chain-of-Thought (CoT) reasoning enhance LLM
performance by decomposing problems into intermediate steps, they also incur
significant overhead in token usage, leading to increased costs. We find that
the reasoning process of current LLMs is unnecessarily lengthy and it can be
compressed by including a reasonable token budget in the prompt, but the choice
of token budget plays a crucial role in the actual compression effectiveness.
We then propose a token-budget-aware LLM reasoning framework, which dynamically
estimates token budgets for different problems based on reasoning complexity
and uses the estimated token budgets to guide the reasoning process.
Experiments show that our method effectively reduces token costs in CoT
reasoning with only a slight performance reduction, offering a practical
solution to balance efficiency and accuracy in LLM reasoning. Code:
https://github.com/GeniusHTX/TALE.
Bridging the Data Provenance Gap Across Text, Speech and Video
Authors: name: Shayne Longpre, name: Nikhil Singh, name: Manuel Cherep, name: Kushagra Tiwary, name: Joanna Materzynska, name: William Brannon, name: Robert Mahari, name: Manan Dey, name: Mohammed Hamdy, name: Nayan Saxena, name: Ahmad Mustafa Anis, name: Emad A. Alghamdi, name: Vu Minh Chien, name: Naana Obeng-Marnu, name: Da Yin, name: Kun Qian, name: Yizhi Li, name: Minnie Liang, name: An Dinh, name: Shrestha Mohanty, name: Deividas Mataciunas, name: Tobin South, name: Jianguo Zhang, name: Ariel N. Lee, name: Campbell S. Lund, name: Christopher Klamm, name: Damien Sileo, name: Diganta Misra, name: Enrico Shippole, name: Kevin Klyman, name: Lester JV Miranda, name: Niklas Muennighoff, name: Seonghyeon Ye, name: Seungone Kim, name: Vipul Gupta, name: Vivek Sharma, name: Xuhui Zhou, name: Caiming Xiong, name: Luis Villa, name: Stella Biderman, name: Alex Pentland, name: Sara Hooker, name: Jad Kabbara
Published: 2024-12-19 01:30:19

Progress in AI is driven largely by the scale and quality of training data.
Despite this, there is a deficit of empirical analysis examining the attributes
of well-established datasets beyond text. In this work we conduct the largest
and first-of-its-kind longitudinal audit across modalities--popular text,
speech, and video datasets--from their detailed sourcing trends and use
restrictions to their geographical and linguistic representation. Our manual
analysis covers nearly 4000 public datasets between 1990-2024, spanning 608
languages, 798 sources, 659 organizations, and 67 countries. We find that
multimodal machine learning applications have overwhelmingly turned to
web-crawled, synthetic, and social media platforms, such as YouTube, for their
training sets, eclipsing all other sources since 2019. Secondly, tracing the
chain of dataset derivations we find that while less than 33% of datasets are
restrictively licensed, over 80% of the source content in widely-used text,
speech, and video datasets, carry non-commercial restrictions. Finally, counter
to the rising number of languages and geographies represented in public AI
training datasets, our audit demonstrates measures of relative geographical and
multilingual representation have failed to significantly improve their coverage
since 2013. We believe the breadth of our audit enables us to empirically
examine trends in data sourcing, restrictions, and Western-centricity at an
ecosystem-level, and that visibility into these questions are essential to
progress in responsible AI. As a contribution to ongoing improvements in
dataset transparency and responsible use, we release our entire multimodal
audit, allowing practitioners to trace data provenance across text, speech, and
video.
MotiF: Making Text Count in Image Animation with Motion Focal Loss
Authors: name: Shijie Wang, name: Samaneh Azadi, name: Rohit Girdhar, name: Saketh Rambhatla, name: Chen Sun, name: Xi Yin
Published: 2024-12-20 18:57:06

Text-Image-to-Video (TI2V) generation aims to generate a video from an image
following a text description, which is also referred to as text-guided image
animation. Most existing methods struggle to generate videos that align well
with the text prompts, particularly when motion is specified. To overcome this
limitation, we introduce MotiF, a simple yet effective approach that directs
the model's learning to the regions with more motion, thereby improving the
text alignment and motion generation. We use optical flow to generate a motion
heatmap and weight the loss according to the intensity of the motion. This
modified objective leads to noticeable improvements and complements existing
methods that utilize motion priors as model inputs. Additionally, due to the
lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V
Bench, a dataset consists of 320 image-text pairs for robust evaluation. We
present a human evaluation protocol that asks the annotators to select an
overall preference between two videos followed by their justifications. Through
a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced
models, achieving an average preference of 72%. The TI2V Bench is released in
https://wang-sj16.github.io/motif/.
Ensembling Large Language Models with Process Reward-Guided Tree Search
  for Better Complex Reasoning
Authors: name: Sungjin Park, name: Xiao Liu, name: Yeyun Gong, name: Edward Choi
Published: 2024-12-20 11:14:29

Despite recent advances in large language models, open-source models often
struggle to consistently perform well on complex reasoning tasks. Existing
ensemble methods, whether applied at the token or output levels, fail to
address these challenges. In response, we present Language model Ensemble with
Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level
ensembling of language models. LE-MCTS formulates step-by-step reasoning with
an ensemble of language models as a Markov decision process. In this framework,
states represent intermediate reasoning paths, while actions consist of
generating the next reasoning step using one of the language models selected
from a predefined pool. Guided by a process-based reward model, LE-MCTS
performs a tree search over the reasoning steps generated by different language
models, identifying the most accurate reasoning chain. Experimental results on
five mathematical reasoning benchmarks demonstrate that our approach
outperforms both single language model decoding algorithms and language model
ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the
MATH and MQA datasets, respectively, highlighting its effectiveness in solving
complex reasoning problems.
In Case You Missed It: ARC 'Challenge' Is Not That Challenging
Authors: name: Łukasz Borchmann
Published: 2024-12-23 18:14:36

ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily
due to an evaluation setup that prevents direct comparison of answer choices
rather than inherent complexity. Although some researchers have quietly shifted
to a more appropriate scheme over the last year, the implications of this
change have yet to be widely acknowledged. We highlight this overlooked shift,
show how similar evaluation practices falsely imply reasoning deficits in other
benchmarks, and demonstrate that fairer methods dramatically reduce performance
gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing
so, we reveal how evaluation shapes perceived difficulty and offer guidelines
to ensure that multiple-choice evaluations accurately reflect actual model
capabilities.
3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
  Scene Understanding
Authors: name: Tatiana Zemskova, name: Dmitry Yudin
Published: 2024-12-24 14:21:58

A 3D scene graph represents a compact scene model, storing information about
the objects and the semantic relationships between them, making its use
promising for robotic tasks. When interacting with a user, an embodied
intelligent agent should be capable of responding to various queries about the
scene formulated in natural language. Large Language Models (LLMs) are
beneficial solutions for user-robot interaction due to their natural language
understanding and reasoning abilities. Recent methods for creating learnable
representations of 3D scenes have demonstrated the potential to improve the
quality of LLMs responses by adapting to the 3D world. However, the existing
methods do not explicitly utilize information about the semantic relationships
between objects, limiting themselves to information about their coordinates. In
this work, we propose a method 3DGraphLLM for constructing a learnable
representation of a 3D scene graph. The learnable representation is used as
input for LLMs to perform 3D vision-language tasks. In our experiments on
popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
datasets, we demonstrate the advantage of this approach over baseline methods
that do not use information about the semantic relationships between objects.
The code is publicly available at
https://github.com/CognitiveAISystems/3DGraphLLM.
PartGen: Part-level 3D Generation and Reconstruction with Multi-View
  Diffusion Models
Authors: name: Minghao Chen, name: Roman Shapovalov, name: Iro Laina, name: Tom Monnier, name: Jianyuan Wang, name: David Novotny, name: Andrea Vedaldi
Published: 2024-12-24 18:59:43

Text- or image-to-3D generators and 3D scanners can now produce 3D assets
with high-quality shapes and textures. These assets typically consist of a
single, fused representation, like an implicit neural field, a Gaussian
mixture, or a mesh, without any useful structure. However, most applications
and creative workflows require assets to be made of several meaningful parts
that can be manipulated independently. To address this gap, we introduce
PartGen, a novel approach that generates 3D objects composed of meaningful
parts starting from text, an image, or an unstructured 3D object. First, given
multiple views of a 3D object, generated or rendered, a multi-view diffusion
model extracts a set of plausible and view-consistent part segmentations,
dividing the object into parts. Then, a second multi-view diffusion model takes
each part separately, fills in the occlusions, and uses those completed views
for 3D reconstruction by feeding them to a 3D reconstruction network. This
completion process considers the context of the entire object to ensure that
the parts integrate cohesively. The generative completion model can make up for
the information missing due to occlusions; in extreme cases, it can hallucinate
entirely invisible parts based on the input 3D asset. We evaluate our method on
generated and real 3D assets and show that it outperforms segmentation and
part-extraction baselines by a large margin. We also showcase downstream
applications such as 3D part editing.
DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion
  Transformer for Tuning-Free Multi-Prompt Longer Video Generation
Authors: name: Minghong Cai, name: Xiaodong Cun, name: Xiaoyu Li, name: Wenze Liu, name: Zhaoyang Zhang, name: Yong Zhang, name: Ying Shan, name: Xiangyu Yue
Published: 2024-12-24 18:51:19

Sora-like video generation models have achieved remarkable progress with a
Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current
video generation models predominantly focus on single-prompt, struggling to
generate coherent scenes with multiple sequential prompts that better reflect
real-world dynamic scenarios. While some pioneering works have explored
multi-prompt video generation, they face significant challenges including
strict training data requirements, weak prompt following, and unnatural
transitions. To address these problems, we propose DiTCtrl, a training-free
multi-prompt video generation method under MM-DiT architectures for the first
time. Our key idea is to take the multi-prompt video generation task as
temporal video editing with smooth transitions. To achieve this goal, we first
analyze MM-DiT's attention mechanism, finding that the 3D full attention
behaves similarly to that of the cross/self-attention blocks in the UNet-like
diffusion models, enabling mask-guided precise semantic control across
different prompts with attention sharing for multi-prompt video generation.
Based on our careful design, the video generated by DiTCtrl achieves smooth
transitions and consistent object motion given multiple sequential prompts
without additional training. Besides, we also present MPVBench, a new benchmark
specially designed for multi-prompt video generation to evaluate the
performance of multi-prompt generation. Extensive experiments demonstrate that
our method achieves state-of-the-art performance without additional training.
Fourier Position Embedding: Enhancing Attention's Periodic Extension for
  Length Generalization
Authors: name: Ermo Hua, name: Che Jiang, name: Xingtai Lv, name: Kaiyan Zhang, name: Ning Ding, name: Youbang Sun, name: Biqing Qi, name: Yuchen Fan, name: Xue Kai Zhu, name: Bowen Zhou
Published: 2024-12-23 17:44:01

Extending the context length of Language Models (LMs) by improving Rotary
Position Embedding (RoPE) has become a trend. While existing works mainly
address RoPE's limitations within attention mechanism, this paper provides an
analysis across nearly all parts of LMs, uncovering their adverse effects on
length generalization for RoPE-based attention. Using Discrete Signal
Processing theory, we show that RoPE enables periodic attention by implicitly
achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
undermined by the spectral damage caused by: 1) linear layers and activation
functions outside of attention; 2) insufficiently trained frequency components
brought by time-domain truncation. Building on our observations, we propose
Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
properties to improve both its periodic extension and length generalization.
FoPE constructs Fourier Series and zero-outs the destructive frequency
components, increasing model robustness against the spectrum damage.
Experiments across various model scales show that, within varying context
windows, FoPE can maintain a more stable perplexity and a more consistent
accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
analyses and ablations bring further support to our method and theoretical
modeling.
ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
Authors: name: Ziteng Wang, name: Jianfei Chen, name: Jun Zhu
Published: 2024-12-19 10:21:20

Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to
scale up model capacity without increasing the computation budget. However,
vanilla TopK routers are trained in a discontinuous, non-differentiable way,
limiting their performance and scalability. To address this issue, we propose
ReMoE, a fully differentiable MoE architecture that offers a simple yet
effective drop-in replacement for the conventional TopK+Softmax routing,
utilizing ReLU as the router instead. We further propose methods to regulate
the router's sparsity while balancing the load among experts. ReMoE's
continuous nature enables efficient dynamic allocation of computation across
tokens and layers, while also exhibiting domain specialization. Our experiments
demonstrate that ReMoE consistently outperforms vanilla TopK-routed MoE across
various model sizes, expert counts, and levels of granularity. Furthermore,
ReMoE exhibits superior scalability with respect to the number of experts,
surpassing traditional MoE architectures. The implementation based on
Megatron-LM is available at https://github.com/thu-ml/ReMoE.
DepthLab: From Partial to Complete
Authors: name: Zhiheng Liu, name: Ka Leong Cheng, name: Qiuyu Wang, name: Shuzhe Wang, name: Hao Ouyang, name: Bin Tan, name: Kai Zhu, name: Yujun Shen, name: Qifeng Chen, name: Ping Luo
Published: 2024-12-24 04:16:38

Missing values remain a common challenge for depth data across its wide range
of applications, stemming from various causes like incomplete data acquisition
and perspective alteration. This work bridges this gap with DepthLab, a
foundation depth inpainting model powered by image diffusion priors. Our model
features two notable strengths: (1) it demonstrates resilience to
depth-deficient regions, providing reliable completion for both continuous
areas and isolated points, and (2) it faithfully preserves scale consistency
with the conditioned known depth when filling in missing values. Drawing on
these advantages, our approach proves its worth in various downstream tasks,
including 3D scene inpainting, text-to-3D scene generation, sparse-view
reconstruction with DUST3R, and LiDAR depth completion, exceeding current
solutions in both numerical performance and visual quality. Our project page
with source code is available at https://johanan528.github.io/depthlab_web/.
SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic
  Retrieval
Authors: name: Aakash Mahalingam, name: Vinesh Kumar Gande, name: Aman Chadha, name: Vinija Jain, name: Divya Chaudhary
Published: 2024-12-19 22:51:56

Retrieval-Augmented Generation (RAG) systems have become pivotal in
leveraging vast corpora to generate informed and contextually relevant
responses, notably reducing hallucinations in Large Language Models. Despite
significant advancements, these systems struggle to efficiently process and
retrieve information from large datasets while maintaining a comprehensive
understanding of the context. This paper introduces SKETCH, a novel methodology
that enhances the RAG retrieval process by integrating semantic text retrieval
with knowledge graphs, thereby merging structured and unstructured data for a
more holistic comprehension. SKETCH, demonstrates substantial improvements in
retrieval performance and maintains superior context integrity compared to
traditional methods. Evaluated across four diverse datasets: QuALITY, QASPER,
NarrativeQA, and Italian Cuisine-SKETCH consistently outperforms baseline
approaches on key RAGAS metrics such as answer_relevancy, faithfulness,
context_precision and context_recall. Notably, on the Italian Cuisine dataset,
SKETCH achieved an answer relevancy of 0.94 and a context precision of 0.99,
representing the highest performance across all evaluated metrics. These
results highlight SKETCH's capability in delivering more accurate and
contextually relevant responses, setting new benchmarks for future retrieval
systems.