Orionnnn/hub-stats huggingface dataset:
Shiksha: A Technical Domain focused Translation Dataset and Model for
  Indian Languages
Authors: name: Advait Joglekar, name: Srinivasan Umesh
Published: 2024-12-12 07:40:55

Neural Machine Translation (NMT) models are typically trained on datasets
with limited exposure to Scientific, Technical and Educational domains.
Translation models thus, in general, struggle with tasks that involve
scientific understanding or technical jargon. Their performance is found to be
even worse for low-resource Indian languages. Finding a translation dataset
that tends to these domains in particular, poses a difficult challenge. In this
paper, we address this by creating a multilingual parallel corpus containing
more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
translation pairs across 8 Indian languages. We achieve this by bitext mining
human-translated transcriptions of NPTEL video lectures. We also finetune and
evaluate NMT models using this corpus and surpass all other publicly available
models at in-domain tasks. We also demonstrate the potential for generalizing
to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
average for these Indian languages on the Flores+ benchmark. We are pleased to
release our model and dataset via this link: https://huggingface.co/SPRINGLab.
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
Authors: name: Fiona Ryan, name: Ajay Bati, name: Sangmin Lee, name: Daniel Bolya, name: Judy Hoffman, name: James M. Rehg
Published: 2024-12-12 18:55:30

We address the problem of gaze target estimation, which aims to predict where
a person is looking in a scene. Predicting a person's gaze target requires
reasoning both about the person's appearance and the contents of the scene.
Prior works have developed increasingly complex, hand-crafted pipelines for
gaze target estimation that carefully fuse features from separate scene
encoders, head encoders, and auxiliary models for signals like depth and pose.
Motivated by the success of general-purpose feature extractors on a variety of
visual tasks, we propose Gaze-LLE, a novel transformer framework that
streamlines gaze target estimation by leveraging features from a frozen DINOv2
encoder. We extract a single feature representation for the scene, and apply a
person-specific positional prompt to decode gaze with a lightweight module. We
demonstrate state-of-the-art performance across several gaze benchmarks and
provide extensive analysis to validate our design choices. Our code is
available at: http://github.com/fkryan/gazelle .
Multimodal Latent Language Modeling with Next-Token Diffusion
Authors: name: Yutao Sun, name: Hangbo Bao, name: Wenhui Wang, name: Zhiliang Peng, name: Li Dong, name: Shaohan Huang, name: Jianyong Wang, name: Furu Wei
Published: 2024-12-11 18:57:32

Multimodal generative models require a unified approach to handle both
discrete data (e.g., text and code) and continuous data (e.g., image, audio,
video). In this work, we propose Latent Language Modeling (LatentLM), which
seamlessly integrates continuous and discrete data using causal Transformers.
Specifically, we employ a variational autoencoder (VAE) to represent continuous
data as latent vectors and introduce next-token diffusion for autoregressive
generation of these vectors. Additionally, we develop sigma-VAE to address
the challenges of variance collapse, which is crucial for autoregressive
modeling. Extensive experiments demonstrate the effectiveness of LatentLM
across various modalities. In image generation, LatentLM surpasses Diffusion
Transformers in both performance and scalability. When integrated into
multimodal large language models, LatentLM provides a general-purpose interface
that unifies multimodal generation and understanding. Experimental results show
that LatentLM achieves favorable performance compared to Transfusion and vector
quantized models in the setting of scaling up training tokens. In
text-to-speech synthesis, LatentLM outperforms the state-of-the-art VALL-E 2
model in speaker similarity and robustness, while requiring 10x fewer decoding
steps. The results establish LatentLM as a highly effective and scalable
approach to advance large multimodal models.
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for
  Long-term Streaming Video and Audio Interactions
Authors: name: Pan Zhang, name: Xiaoyi Dong, name: Yuhang Cao, name: Yuhang Zang, name: Rui Qian, name: Xilin Wei, name: Lin Chen, name: Yifei Li, name: Junbo Niu, name: Shuangrui Ding, name: Qipeng Guo, name: Haodong Duan, name: Xin Chen, name: Han Lv, name: Zheng Nie, name: Min Zhang, name: Bin Wang, name: Wenwei Zhang, name: Xinyue Zhang, name: Jiaye Ge, name: Wei Li, name: Jingwen Li, name: Zhongying Tu, name: Conghui He, name: Xingcheng Zhang, name: Kai Chen, name: Yu Qiao, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-12-12 18:58:30

Creating AI systems that can interact with environments over long periods,
similar to human cognition, has been a longstanding research goal. Recent
advancements in multimodal large language models (MLLMs) have made significant
strides in open-world understanding. However, the challenge of continuous and
simultaneous streaming perception, memory, and reasoning remains largely
unexplored. Current MLLMs are constrained by their sequence-to-sequence
architecture, which limits their ability to process inputs and generate
responses simultaneously, akin to being unable to think while perceiving.
Furthermore, relying on long contexts to store historical data is impractical
for long-term interactions, as retaining all information becomes costly and
inefficient. Therefore, rather than relying on a single foundation model to
perform all functions, this project draws inspiration from the concept of the
Specialized Generalist AI and introduces disentangled streaming perception,
reasoning, and memory mechanisms, enabling real-time interaction with streaming
video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
(IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
Processes multimodal information in real-time, storing key details in memory
and triggering reasoning in response to user queries. (2) Multi-modal Long
Memory Module: Integrates short-term and long-term memory, compressing
short-term memories into long-term ones for efficient retrieval and improved
accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
tasks, coordinating with the perception and memory modules. This project
simulates human-like cognition, enabling multimodal large language models to
provide continuous and adaptive service over time.
OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary
  Embedding Distillation
Authors: name: Jitesh Jain, name: Zhengyuan Yang, name: Humphrey Shi, name: Jianfeng Gao, name: Jianwei Yang
Published: 2024-12-12 18:55:18

The standard practice for developing contemporary MLLMs is to feed features
from vision encoder(s) into the LLM and train with natural language
supervision. In this work, we posit an overlooked opportunity to optimize the
intermediate LLM representations through a vision perspective (objective),
i.e., solely natural language supervision is sub-optimal for the MLLM's visual
understanding ability. To that end, we propose OLA-VLM, the first approach
distilling knowledge into the LLM's hidden representations from a set of target
visual representations. Firstly, we formulate the objective during the
pretraining stage in MLLMs as a coupled optimization of predictive visual
embedding and next text-token prediction. Secondly, we investigate MLLMs
trained solely with natural language supervision and identify a positive
correlation between the quality of visual representations within these models
and their downstream performance. Moreover, upon probing our OLA-VLM, we
observe improved representation quality owing to the embedding optimization.
Thirdly, we demonstrate that our OLA-VLM outperforms the single and
multi-encoder baselines, proving our approach's superiority over explicitly
feeding the corresponding features to the LLM. Particularly, OLA-VLM boosts
performance by an average margin of up to 2.5% on various benchmarks, with a
notable improvement of 8.7% on the Depth task in CV-Bench. Our code is
open-sourced at https://github.com/SHI-Labs/OLA-VLM .
FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D
  Reconstruction
Authors: name: Jiale Xu, name: Shenghua Gao, name: Ying Shan
Published: 2024-12-12 18:52:53

Existing sparse-view reconstruction models heavily rely on accurate known
camera poses. However, deriving camera extrinsics and intrinsics from
sparse-view images presents significant challenges. In this work, we present
FreeSplatter, a highly scalable, feed-forward reconstruction framework capable
of generating high-quality 3D Gaussians from uncalibrated sparse-view images
and recovering their camera parameters in mere seconds. FreeSplatter is built
upon a streamlined transformer architecture, comprising sequential
self-attention blocks that facilitate information exchange among multi-view
image tokens and decode them into pixel-wise 3D Gaussian primitives. The
predicted Gaussian primitives are situated in a unified reference frame,
allowing for high-fidelity 3D modeling and instant camera parameter estimation
using off-the-shelf solvers. To cater to both object-centric and scene-level
reconstruction, we train two model variants of FreeSplatter on extensive
datasets. In both scenarios, FreeSplatter outperforms state-of-the-art
baselines in terms of reconstruction quality and pose estimation accuracy.
Furthermore, we showcase FreeSplatter's potential in enhancing the productivity
of downstream applications, such as text/image-to-3D content creation.
Neural LightRig: Unlocking Accurate Object Normal and Material
  Estimation with Multi-Light Diffusion
Authors: name: Zexin He, name: Tengfei Wang, name: Xin Huang, name: Xingang Pan, name: Ziwei Liu
Published: 2024-12-12 18:58:09

Recovering the geometry and materials of objects from a single image is
challenging due to its under-constrained nature. In this paper, we present
Neural LightRig, a novel framework that boosts intrinsic estimation by
leveraging auxiliary multi-lighting conditions from 2D diffusion priors.
Specifically, 1) we first leverage illumination priors from large-scale
diffusion models to build our multi-light diffusion model on a synthetic
relighting dataset with dedicated designs. This diffusion model generates
multiple consistent images, each illuminated by point light sources in
different directions. 2) By using these varied lighting images to reduce
estimation uncertainty, we train a large G-buffer model with a U-Net backbone
to accurately predict surface normals and materials. Extensive experiments
validate that our approach significantly outperforms state-of-the-art methods,
enabling accurate surface normal and PBR material estimation with vivid
relighting effects. Code and dataset are available on our project page at
https://projects.zxhezexin.com/neural-lightrig.
SAME: Learning Generic Language-Guided Visual Navigation with
  State-Adaptive Mixture of Experts
Authors: name: Gengze Zhou, name: Yicong Hong, name: Zun Wang, name: Chongyang Zhao, name: Mohit Bansal, name: Qi Wu
Published: 2024-12-07 06:12:53

The academic field of learning instruction-guided visual navigation can be
generally categorized into high-level category-specific search and low-level
language-guided navigation, depending on the granularity of language
instruction, in which the former emphasizes the exploration process, while the
latter concentrates on following detailed textual commands. Despite the
differing focuses of these tasks, the underlying requirements of interpreting
instructions, comprehending the surroundings, and inferring action decisions
remain consistent. This paper consolidates diverse navigation tasks into a
unified and generic framework -- we investigate the core difficulties of
sharing general knowledge and exploiting task-specific capabilities in learning
navigation and propose a novel State-Adaptive Mixture of Experts (SAME) model
that effectively enables an agent to infer decisions based on
different-granularity language and dynamic observations. Powered by SAME, we
present a versatile agent capable of addressing seven navigation tasks
simultaneously that outperforms or achieves highly comparable performance to
task-specific agents.
Video Creation by Demonstration
Authors: name: Yihong Sun, name: Hao Zhou, name: Liangzhe Yuan, name: Jennifer J. Sun, name: Yandong Li, name: Xuhui Jia, name: Hartwig Adam, name: Bharath Hariharan, name: Long Zhao, name: Ting Liu
Published: 2024-12-12 18:41:20

We explore a novel video creation experience, namely Video Creation by
Demonstration. Given a demonstration video and a context image from a different
scene, we generate a physically plausible video that continues naturally from
the context image and carries out the action concepts from the demonstration.
To enable this capability, we present delta-Diffusion, a self-supervised
training approach that learns from unlabeled videos by conditional future frame
prediction. Unlike most existing video generation controls that are based on
explicit signals, we adopts the form of implicit latent control for maximal
flexibility and expressiveness required by general videos. By leveraging a
video foundation model with an appearance bottleneck design on top, we extract
action latents from demonstration videos for conditioning the generation
process with minimal appearance leakage. Empirically, delta-Diffusion
outperforms related baselines in terms of both human preference and large-scale
machine evaluations, and demonstrates potentials towards interactive world
simulation. Sampled video generation results are available at
https://delta-diffusion.github.io/.
Track4Gen: Teaching Video Diffusion Models to Track Points Improves
  Video Generation
Authors: name: Hyeonho Jeong, name: Chun-Hao Paul Huang, name: Jong Chul Ye, name: Niloy Mitra, name: Duygu Ceylan
Published: 2024-12-08 18:21:00

While recent foundational video generators produce visually rich output, they
still struggle with appearance drift, where objects gradually degrade or change
inconsistently across frames, breaking visual coherence. We hypothesize that
this is because there is no explicit supervision in terms of spatial tracking
at the feature level. We propose Track4Gen, a spatially aware video generator
that combines video diffusion loss with point tracking across frames, providing
enhanced spatial supervision on the diffusion features. Track4Gen merges the
video generation and point tracking tasks into a single network by making
minimal changes to existing video generation architectures. Using Stable Video
Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
video generation and point tracking, which are typically handled as separate
tasks. Our extensive evaluations show that Track4Gen effectively reduces
appearance drift, resulting in temporally stable and visually coherent video
generation. Project page: hyeonho99.github.io/track4gen
The BrowserGym Ecosystem for Web Agent Research
Authors: name: Thibault Le Sellier De Chezelles, name: Maxime Gasse, name: Alexandre Drouin, name: Massimo Caccia, name: Léo Boisvert, name: Megh Thakkar, name: Tom Marty, name: Rim Assouel, name: Sahar Omidi Shayegan, name: Lawrence Keunho Jang, name: Xing Han Lù, name: Ori Yoran, name: Dehan Kong, name: Frank F. Xu, name: Siva Reddy, name: Quentin Cappart, name: Graham Neubig, name: Ruslan Salakhutdinov, name: Nicolas Chapados, name: Alexandre Lacoste
Published: 2024-12-06 23:43:59

The BrowserGym ecosystem addresses the growing need for efficient evaluation
and benchmarking of web agents, particularly those leveraging automation and
Large Language Models (LLMs) for web interaction tasks. Many existing
benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
making it challenging to achieve reliable comparisons and reproducible results.
BrowserGym aims to solve this by providing a unified, gym-like environment with
well-defined observation and action spaces, facilitating standardized
evaluation across diverse benchmarks. Combined with AgentLab, a complementary
framework that aids in agent creation, testing, and analysis, BrowserGym offers
flexibility for integrating new benchmarks while ensuring consistent evaluation
and comprehensive experiment management. This standardized approach seeks to
reduce the time and complexity of developing web agents, supporting more
reliable comparisons and facilitating in-depth analysis of agent behaviors, and
could result in more adaptable, capable agents, ultimately accelerating
innovation in LLM-driven automation. As a supporting evidence, we conduct the
first large-scale, multi-benchmark web agent experiment and compare the
performance of 6 state-of-the-art LLMs across all benchmarks currently
available in BrowserGym. Among other findings, our results highlight a large
discrepancy between OpenAI and Anthropic's latests models, with
Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
vision-related tasks where GPT-4o is superior. Despite these advancements, our
results emphasize that building robust and efficient web agents remains a
significant challenge, due to the inherent complexity of real-world web
environments and the limitations of current models.
I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token
Authors: name: Roi Cohen, name: Konstantin Dobler, name: Eden Biran, name: Gerard de Melo
Published: 2024-12-09 17:13:20

Large Language Models are known to capture real-world knowledge, allowing
them to excel in many downstream tasks. Despite recent advances, these models
are still prone to what are commonly known as hallucinations, causing them to
emit unwanted and factually incorrect text. In this work, we propose a novel
calibration method that can be used to combat hallucinations. We add a special
[IDK] ("I don't know") token to the model's vocabulary and introduce an
objective function that shifts probability mass to the [IDK] token for
incorrect predictions. This approach allows the model to express uncertainty in
its output explicitly. We evaluate our proposed method across multiple model
architectures and factual downstream tasks. We find that models trained with
our method are able to express uncertainty in places where they would
previously make mistakes while suffering only a small loss of encoded
knowledge. We further perform extensive ablation studies of multiple variations
of our approach and provide a detailed analysis of the precision-recall
tradeoff of our method.
Bootstrapping Language-Guided Navigation Learning with Self-Refining
  Data Flywheel
Authors: name: Zun Wang, name: Jialu Li, name: Yicong Hong, name: Songze Li, name: Kunchang Li, name: Shoubin Yu, name: Yi Wang, name: Yu Qiao, name: Yali Wang, name: Mohit Bansal, name: Limin Wang
Published: 2024-12-11 15:32:24

Creating high-quality data for training robust language-instructed agents is
a long-lasting challenge in embodied AI. In this paper, we introduce a
Self-Refining Data Flywheel (SRDF) that generates high-quality and large-scale
navigational instruction-trajectory pairs by iteratively refining the data pool
through the collaboration between two models, the instruction generator and the
navigator, without any human-in-the-loop annotation. Specifically, SRDF starts
with using a base generator to create an initial data pool for training a base
navigator, followed by applying the trained navigator to filter the data pool.
This leads to higher-fidelity data to train a better generator, which can, in
turn, produce higher-quality data for training the next-round navigator. Such a
flywheel establishes a data self-refining process, yielding a continuously
improved and highly effective dataset for large-scale language-guided
navigation learning. Our experiments demonstrate that after several flywheel
rounds, the navigator elevates the performance boundary from 70% to 78% SPL on
the classic R2R test set, surpassing human performance (76%) for the first
time. Meanwhile, this process results in a superior generator, evidenced by a
SPICE increase from 23.5 to 26.2, better than all previous VLN instruction
generation methods. Finally, we demonstrate the scalability of our method
through increasing environment and instruction diversity, and the
generalization ability of our pre-trained navigator across various downstream
navigation tasks, surpassing state-of-the-art methods by a large margin in all
cases.
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Authors: name: Fan Wang, name: Juyong Jiang, name: Chansung Park, name: Sunghun Kim, name: Jing Tang
Published: 2024-12-08 21:26:22

The increasing sizes of large language models (LLMs) result in significant
computational overhead and memory usage when adapting these models to specific
tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have
been devised to mitigate these challenges by training a small set of parameters
for the task-specific updates of the model weights. Among PEFT methods, LoRA
stands out for its simplicity and efficiency, inspiring the development of a
series of variants. However, LoRA and its successors disregard the knowledge
that is noisy or irrelevant to the targeted task, detrimentally impacting model
performance and leading to suboptimality. To address this limitation, we
introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that
leverages singular value decomposition (SVD) with knowledge-aware singular
values to dynamically activate knowledge based on its relevance to the task at
hand. We conduct extensive experiments across a range of LLMs on tasks spanning
natural language understanding (NLU), generation (NLG), instruction following,
and commonsense reasoning. The experimental results demonstrate that KaSA
consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks
and 4 synthetic datasets, underscoring our method's efficacy and adaptability.
The source code of our method is available at
https://github.com/juyongjiang/KaSA.
StyleStudio: Text-Driven Style Transfer with Selective Control of Style
  Elements
Authors: name: Mingkun Lei, name: Xue Song, name: Beier Zhu, name: Hao Wang, name: Chi Zhang
Published: 2024-12-11 16:13:23

Text-driven style transfer aims to merge the style of a reference image with
content described by a text prompt. Recent advancements in text-to-image models
have improved the nuance of style transformations, yet significant challenges
remain, particularly with overfitting to reference styles, limiting stylistic
control, and misaligning with textual content. In this paper, we propose three
complementary strategies to address these issues. First, we introduce a
cross-modal Adaptive Instance Normalization (AdaIN) mechanism for better
integration of style and text features, enhancing alignment. Second, we develop
a Style-based Classifier-Free Guidance (SCFG) approach that enables selective
control over stylistic elements, reducing irrelevant influences. Finally, we
incorporate a teacher model during early generation stages to stabilize spatial
layouts and mitigate artifacts. Our extensive evaluations demonstrate
significant improvements in style transfer quality and alignment with textual
prompts. Furthermore, our approach can be integrated into existing style
transfer frameworks without fine-tuning.