Orionnnn/hub-stats huggingface dataset:
Neural Fields in Robotics: A Survey
Authors: name: Muhammad Zubair Irshad, name: Mauro Comi, name: Yen-Chen Lin, name: Nick Heppert, name: Abhinav Valada, name: Rares Ambrus, name: Zsolt Kira, name: Jonathan Tremblay
Published: 2024-10-26 16:26:41

Neural Fields have emerged as a transformative approach for 3D scene
representation in computer vision and robotics, enabling accurate inference of
geometry, 3D semantics, and dynamics from posed 2D data. Leveraging
differentiable rendering, Neural Fields encompass both continuous implicit and
explicit neural representations enabling high-fidelity 3D reconstruction,
integration of multi-modal sensor data, and generation of novel viewpoints.
This survey explores their applications in robotics, emphasizing their
potential to enhance perception, planning, and control. Their compactness,
memory efficiency, and differentiability, along with seamless integration with
foundation and generative models, make them ideal for real-time applications,
improving robot adaptability and decision-making. This paper provides a
thorough review of Neural Fields in robotics, categorizing applications across
various domains and evaluating their strengths and limitations, based on over
200 papers. First, we present four key Neural Fields frameworks: Occupancy
Networks, Signed Distance Fields, Neural Radiance Fields, and Gaussian
Splatting. Second, we detail Neural Fields' applications in five major robotics
domains: pose estimation, manipulation, navigation, physics, and autonomous
driving, highlighting key works and discussing takeaways and open challenges.
Finally, we outline the current limitations of Neural Fields in robotics and
propose promising directions for future research. Project page:
https://robonerf.github.io
GrounDiT: Grounding Diffusion Transformers via Noisy Patch
  Transplantation
Authors: name: Phillip Y. Lee, name: Taehoon Yoon, name: Minhyuk Sung
Published: 2024-10-27 15:30:45

We introduce a novel training-free spatial grounding technique for
text-to-image generation using Diffusion Transformers (DiT). Spatial grounding
with bounding boxes has gained attention for its simplicity and versatility,
allowing for enhanced user control in image generation. However, prior
training-free approaches often rely on updating the noisy image during the
reverse diffusion process via backpropagation from custom loss functions, which
frequently struggle to provide precise control over individual bounding boxes.
In this work, we leverage the flexibility of the Transformer architecture,
demonstrating that DiT can generate noisy patches corresponding to each
bounding box, fully encoding the target object and allowing for fine-grained
control over each region. Our approach builds on an intriguing property of DiT,
which we refer to as semantic sharing. Due to semantic sharing, when a smaller
patch is jointly denoised alongside a generatable-size image, the two become
"semantic clones". Each patch is denoised in its own branch of the generation
process and then transplanted into the corresponding region of the original
noisy image at each timestep, resulting in robust spatial grounding for each
bounding box. In our experiments on the HRS and DrawBench benchmarks, we
achieve state-of-the-art performance compared to previous training-free spatial
grounding approaches.
COAT: Compressing Optimizer states and Activation for Memory-Efficient
  FP8 Training
Authors: name: Haocheng Xi, name: Han Cai, name: Ligeng Zhu, name: Yao Lu, name: Kurt Keutzer, name: Jianfei Chen, name: Song Han
Published: 2024-10-25 05:59:30

FP8 training has emerged as a promising method for improving training
efficiency. Existing frameworks accelerate training by applying FP8 computation
to linear layers while leaving optimizer states and activations in higher
precision, which fails to fully optimize memory usage. This paper introduces
COAT (Compressing Optimizer States and Activations for FP8 Training), a novel
FP8 training framework designed to significantly reduce memory footprint when
training large models. COAT addresses current limitations through two key
innovations: (1) Dynamic Range Expansion, which aligns optimizer state
distributions more closely with the FP8 representation range, thereby reducing
quantization error, and (2) Mixed-Granularity Activation Quantization, which
optimizes activation memory using a combination of per-tensor and per-group
quantization strategies. Experiments demonstrate that COAT effectively reduces
end-to-end training memory footprint by 1.54x compared to BF16 while achieving
nearly lossless performance across various tasks, such as Large Language Model
pretraining and fine-tuning and Vision Language Model training. COAT also
achieves a 1.43x end-to-end training speedup compared to BF16, performing on
par with or surpassing TransformerEngine's speedup. COAT enables efficient
full-parameter training of large models on fewer GPUs, and facilitates doubling
the batch size in distributed training settings, providing a practical solution
for scaling large-scale model training. The code is available at
https://github.com/NVlabs/COAT.
Language Models And A Second Opinion Use Case: The Pocket Professional
Authors: name: David Noever
Published: 2024-10-27 23:48:47

This research tests the role of Large Language Models (LLMs) as formal second
opinion tools in professional decision-making, particularly focusing on complex
medical cases where even experienced physicians seek peer consultation. The
work analyzed 183 challenging medical cases from Medscape over a 20-month
period, testing multiple LLMs' performance against crowd-sourced physician
responses. A key finding was the high overall score possible in the latest
foundational models (>80% accuracy compared to consensus opinion), which
exceeds most human metrics reported on the same clinical cases (450 pages of
patient profiles, test results). The study rates the LLMs' performance
disparity between straightforward cases (>81% accuracy) and complex scenarios
(43% accuracy), particularly in these cases generating substantial debate among
human physicians. The research demonstrates that LLMs may be valuable as
generators of comprehensive differential diagnoses rather than as primary
diagnostic tools, potentially helping to counter cognitive biases in clinical
decision-making, reduce cognitive loads, and thus remove some sources of
medical error. The inclusion of a second comparative legal dataset (Supreme
Court cases, N=21) provides added empirical context to the AI use to foster
second opinions, though these legal challenges proved considerably easier for
LLMs to analyze. In addition to the original contributions of empirical
evidence for LLM accuracy, the research aggregated a novel benchmark for others
to score highly contested question and answer reliability between both LLMs and
disagreeing human practitioners. These results suggest that the optimal
deployment of LLMs in professional settings may differ substantially from
current approaches that emphasize automation of routine tasks.
MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
Authors: name: Haozhe Liu, name: Shikun Liu, name: Zijian Zhou, name: Mengmeng Xu, name: Yanping Xie, name: Xiao Han, name: Juan C. Pérez, name: Ding Liu, name: Kumara Kahatapitiya, name: Menglin Jia, name: Jui-Chieh Wu, name: Sen He, name: Tao Xiang, name: Jürgen Schmidhuber, name: Juan-Manuel Pérez-Rúa
Published: 2024-10-26 21:12:32

We introduce MarDini, a new family of video diffusion models that integrate
the advantages of masked auto-regression (MAR) into a unified diffusion model
(DM) framework. Here, MAR handles temporal planning, while DM focuses on
spatial generation in an asymmetric network design: i) a MAR-based planning
model containing most of the parameters generates planning signals for each
masked frame using low-resolution input; ii) a lightweight generation model
uses these signals to produce high-resolution frames via diffusion de-noising.
MarDini's MAR enables video generation conditioned on any number of masked
frames at any frame positions: a single model can handle video interpolation
(e.g., masking middle frames), image-to-video generation (e.g., masking from
the second frame onward), and video expansion (e.g., masking half the frames).
The efficient design allocates most of the computational resources to the
low-resolution planning model, making computationally expensive but important
spatio-temporal attention feasible at scale. MarDini sets a new
state-of-the-art for video interpolation; meanwhile, within few inference
steps, it efficiently generates videos on par with those of much more expensive
advanced image-to-video models.
DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe
  Dataset Curation
Authors: name: Yuang Ai, name: Xiaoqiang Zhou, name: Huaibo Huang, name: Xiaotian Han, name: Zhengyu Chen, name: Quanzeng You, name: Hongxia Yang
Published: 2024-10-24 11:57:20

Image restoration (IR) in real-world scenarios presents significant
challenges due to the lack of high-capacity models and comprehensive datasets.
To tackle these issues, we present a dual strategy: GenIR, an innovative data
curation pipeline, and DreamClear, a cutting-edge Diffusion Transformer
(DiT)-based image restoration model. GenIR, our pioneering contribution, is a
dual-prompt learning pipeline that overcomes the limitations of existing
datasets, which typically comprise only a few thousand images and thus offer
limited generalizability for larger models. GenIR streamlines the process into
three stages: image-text pair construction, dual-prompt based fine-tuning, and
data generation & filtering. This approach circumvents the laborious data
crawling process, ensuring copyright compliance and providing a cost-effective,
privacy-safe solution for IR dataset construction. The result is a large-scale
dataset of one million high-quality images. Our second contribution,
DreamClear, is a DiT-based image restoration model. It utilizes the generative
priors of text-to-image (T2I) diffusion models and the robust perceptual
capabilities of multi-modal large language models (MLLMs) to achieve
photorealistic restoration. To boost the model's adaptability to diverse
real-world degradations, we introduce the Mixture of Adaptive Modulator (MoAM).
It employs token-wise degradation priors to dynamically integrate various
restoration experts, thereby expanding the range of degradations the model can
address. Our exhaustive experiments confirm DreamClear's superior performance,
underlining the efficacy of our dual strategy for real-world image restoration.
Code and pre-trained models will be available at:
https://github.com/shallowdream204/DreamClear.
Vision Search Assistant: Empower Vision-Language Models as Multimodal
  Search Engines
Authors: name: Zhixin Zhang, name: Yiyuan Zhang, name: Xiaohan Ding, name: Xiangyu Yue
Published: 2024-10-28 17:04:18

Search engines enable the retrieval of unknown information with texts.
However, traditional methods fall short when it comes to understanding
unfamiliar visual content, such as identifying an object that the model has
never seen before. This challenge is particularly pronounced for large
vision-language models (VLMs): if the model has not been exposed to the object
depicted in an image, it struggles to generate reliable answers to the user's
question regarding that image. Moreover, as new objects and events continuously
emerge, frequently updating VLMs is impractical due to heavy computational
burdens. To address this limitation, we propose Vision Search Assistant, a
novel framework that facilitates collaboration between VLMs and web agents.
This approach leverages VLMs' visual understanding capabilities and web agents'
real-time information access to perform open-world Retrieval-Augmented
Generation via the web. By integrating visual and textual representations
through this collaboration, the model can provide informed responses even when
the image is novel to the system. Extensive experiments conducted on both
open-set and closed-set QA benchmarks demonstrate that the Vision Search
Assistant significantly outperforms the other models and can be widely applied
to existing VLMs.
LongReward: Improving Long-context Large Language Models with AI
  Feedback
Authors: name: Jiajie Zhang, name: Zhongni Hou, name: Xin Lv, name: Shulin Cao, name: Zhenyu Hou, name: Yilin Niu, name: Lei Hou, name: Yuxiao Dong, name: Ling Feng, name: Juanzi Li
Published: 2024-10-28 17:50:42

Though significant advancements have been achieved in developing long-context
large language models (LLMs), the compromised quality of LLM-synthesized data
for supervised fine-tuning (SFT) often affects the long-context performance of
SFT models and leads to inherent limitations. In principle, reinforcement
learning (RL) with appropriate reward signals can further enhance models'
capacities. However, how to obtain reliable rewards in long-context scenarios
remains unexplored. To this end, we propose LongReward, a novel method that
utilizes an off-the-shelf LLM to provide rewards for long-context model
responses from four human-valued dimensions: helpfulness, logicality,
faithfulness, and completeness, each with a carefully designed assessment
pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
effectively improve long-context SFT models. Our experiments indicate that
LongReward not only significantly improves models' long-context performance but
also enhances their ability to follow short instructions. We also find that
long-context DPO with LongReward and conventional short-context DPO can be used
together without hurting either one's performance.
A Survey of Small Language Models
Authors: name: Chien Van Nguyen, name: Xuan Shen, name: Ryan Aponte, name: Yu Xia, name: Samyadeep Basu, name: Zhengmian Hu, name: Jian Chen, name: Mihir Parmar, name: Sasidhar Kunapuli, name: Joe Barrow, name: Junda Wu, name: Ashish Singh, name: Yu Wang, name: Jiuxiang Gu, name: Franck Dernoncourt, name: Nesreen K. Ahmed, name: Nedim Lipka, name: Ruiyi Zhang, name: Xiang Chen, name: Tong Yu, name: Sungchul Kim, name: Hanieh Deilamsalehy, name: Namyong Park, name: Mike Rimer, name: Zhehao Zhang, name: Huanrui Yang, name: Ryan A. Rossi, name: Thien Huu Nguyen
Published: 2024-10-25 23:52:28

Small Language Models (SLMs) have become increasingly important due to their
efficiency and performance to perform various language tasks with minimal
computational resources, making them ideal for various settings including
on-device, mobile, edge devices, among many others. In this article, we present
a comprehensive survey on SLMs, focusing on their architectures, training
techniques, and model compression techniques. We propose a novel taxonomy for
categorizing the methods used to optimize SLMs, including model compression,
pruning, and quantization techniques. We summarize the benchmark datasets that
are useful for benchmarking SLMs along with the evaluation metrics commonly
used. Additionally, we highlight key open challenges that remain to be
addressed. Our survey aims to serve as a valuable resource for researchers and
practitioners interested in developing and deploying small yet efficient
language models.
LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior
Authors: name: Hanyu Wang, name: Saksham Suri, name: Yixuan Ren, name: Hao Chen, name: Abhinav Shrivastava
Published: 2024-10-28 17:57:07

We present LARP, a novel video tokenizer designed to overcome limitations in
current video tokenization methods for autoregressive (AR) generative models.
Unlike traditional patchwise tokenizers that directly encode local visual
patches into discrete tokens, LARP introduces a holistic tokenization scheme
that gathers information from the visual content using a set of learned
holistic queries. This design allows LARP to capture more global and semantic
representations, rather than being limited to local patch-level information.
Furthermore, it offers flexibility by supporting an arbitrary number of
discrete tokens, enabling adaptive and efficient tokenization based on the
specific requirements of the task. To align the discrete token space with
downstream AR generation tasks, LARP integrates a lightweight AR transformer as
a training-time prior model that predicts the next token on its discrete latent
space. By incorporating the prior model during training, LARP learns a latent
space that is not only optimized for video reconstruction but is also
structured in a way that is more conducive to autoregressive generation.
Moreover, this process defines a sequential order for the discrete tokens,
progressively pushing them toward an optimal configuration during training,
ensuring smoother and more accurate AR generation at inference time.
Comprehensive experiments demonstrate LARP's strong performance, achieving
state-of-the-art FVD on the UCF101 class-conditional video generation
benchmark. LARP enhances the compatibility of AR models with videos and opens
up the potential to build unified high-fidelity multimodal large language
models (MLLMs).
Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with
  System Co-Design
Authors: name: Ruisi Cai, name: Yeonju Ro, name: Geon-Woo Kim, name: Peihao Wang, name: Babak Ehteshami Bejnordi, name: Aditya Akella, name: Zhangyang Wang
Published: 2024-10-24 19:48:51

The proliferation of large language models (LLMs) has led to the adoption of
Mixture-of-Experts (MoE) architectures that dynamically leverage specialized
subnetworks for improved efficiency and performance. Despite their benefits,
MoE models face significant challenges during inference, including inefficient
memory management and suboptimal batching, due to misaligned design choices
between the model architecture and the system policies. Furthermore, the
conventional approach of training MoEs from scratch is increasingly prohibitive
in terms of cost. In this paper, we propose a novel framework Read-ME that
transforms pre-trained dense LLMs into smaller MoE models (in contrast to
"upcycling" generalist MoEs), avoiding the high costs of ground-up training.
Our approach employs activation sparsity to extract experts. To compose
experts, we examine the widely-adopted layer-wise router design and show its
redundancy, and thus we introduce the pre-gating router decoupled from the MoE
backbone that facilitates system-friendly pre-computing and lookahead
scheduling, enhancing expert-aware batching and caching. Our codesign therefore
addresses critical gaps on both the algorithmic and system fronts, establishing
a scalable and efficient alternative for LLM inference in resource-constrained
settings. Read-ME outperforms other popular open-source dense models of similar
scales, achieving improvements of up to 10.1% on MMLU, and improving mean
end-to-end latency up to 6.1%. Codes are available at:
https://github.com/VITA-Group/READ-ME.
Analysing the Residual Stream of Language Models Under Knowledge
  Conflicts
Authors: name: Yu Zhao, name: Xiaotang Du, name: Giwon Hong, name: Aryo Pradipta Gema, name: Alessio Devoto, name: Hongru Wang, name: Xuanli He, name: Kam-Fai Wong, name: Pasquale Minervini
Published: 2024-10-21 15:12:51

Large language models (LLMs) can store a significant amount of factual
knowledge in their parameters. However, their parametric knowledge may conflict
with the information provided in the context. Such conflicts can lead to
undesirable model behaviour, such as reliance on outdated or incorrect
information. In this work, we investigate whether LLMs can identify knowledge
conflicts and whether it is possible to know which source of knowledge the
model will rely on by analysing the residual stream of the LLM. Through probing
tasks, we find that LLMs can internally register the signal of knowledge
conflict in the residual stream, which can be accurately detected by probing
the intermediate model activations. This allows us to detect conflicts within
the residual stream before generating the answers without modifying the input
or model parameters. Moreover, we find that the residual stream shows
significantly different patterns when the model relies on contextual knowledge
versus parametric knowledge to resolve conflicts. This pattern can be employed
to estimate the behaviour of LLMs when conflict happens and prevent unexpected
answers before producing the answers. Our analysis offers insights into how
LLMs internally manage knowledge conflicts and provides a foundation for
developing methods to control the knowledge selection processes.
Mapping the Media Landscape: Predicting Factual Reporting and Political
  Bias Through Web Interactions
Authors: name: Dairazalia Sánchez-Cortés, name: Sergio Burdisso, name: Esaú Villatoro-Tello, name: Petr Motlicek
Published: 2024-10-23 08:18:26

Bias assessment of news sources is paramount for professionals,
organizations, and researchers who rely on truthful evidence for information
gathering and reporting. While certain bias indicators are discernible from
content analysis, descriptors like political bias and fake news pose greater
challenges. In this paper, we propose an extension to a recently presented news
media reliability estimation method that focuses on modeling outlets and their
longitudinal web interactions. Concretely, we assess the classification
performance of four reinforcement learning strategies on a large news media
hyperlink graph. Our experiments, targeting two challenging bias descriptors,
factual reporting and political bias, showed a significant performance
improvement at the source media level. Additionally, we validate our methods on
the CLEF 2023 CheckThat! Lab challenge, outperforming the reported results in
both, F1-score and the official MAE metric. Furthermore, we contribute by
releasing the largest annotated dataset of news source media, categorized with
factual reporting and political bias labels. Our findings suggest that
profiling news media sources based on their hyperlink interactions over time is
feasible, offering a bird's-eye view of evolving media landscapes.
Are LLMs Better than Reported? Detecting Label Errors and Mitigating
  Their Effect on Model Performance
Authors: name: Omer Nahum, name: Nitay Calderon, name: Orgad Keller, name: Idan Szpektor, name: Roi Reichart
Published: 2024-10-24 16:27:03

NLP benchmarks rely on standardized datasets for training and evaluating
models and are crucial for advancing the field. Traditionally, expert
annotations ensure high-quality labels; however, the cost of expert annotation
does not scale well with the growing demand for larger datasets required by
modern models. While crowd-sourcing provides a more scalable solution, it often
comes at the expense of annotation precision and consistency. Recent
advancements in large language models (LLMs) offer new opportunities to enhance
the annotation process, particularly for detecting label errors in existing
datasets. In this work, we consider the recent approach of LLM-as-a-judge,
leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through
a case study of four datasets from the TRUE benchmark, covering different tasks
and domains, we empirically analyze the labeling quality of existing datasets,
and compare expert, crowd-sourced, and our LLM-based annotations in terms of
agreement, label quality, and efficiency, demonstrating the strengths and
limitations of each annotation method. Our findings reveal a substantial number
of label errors, which, when corrected, induce a significant upward shift in
reported model performance. This suggests that many of the LLMs so-called
mistakes are due to label errors rather than genuine model failures.
Additionally, we discuss the implications of mislabeled data and propose
methods to mitigate them in training to improve model performance.