Orionnnn/hub-stats huggingface dataset:
SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction
Authors: name: Marko Mihajlovic, name: Sergey Prokudin, name: Siyu Tang, name: Robert Maier, name: Federica Bogo, name: Tony Tung, name: Edmond Boyer
Published: 2024-09-17 14:04:20

Digitizing 3D static scenes and 4D dynamic events from multi-view images has
long been a challenge in computer vision and graphics. Recently, 3D Gaussian
Splatting (3DGS) has emerged as a practical and scalable reconstruction method,
gaining popularity due to its impressive reconstruction quality, real-time
rendering capabilities, and compatibility with widely used visualization tools.
However, the method requires a substantial number of input views to achieve
high-quality scene reconstruction, introducing a significant practical
bottleneck. This challenge is especially severe in capturing dynamic scenes,
where deploying an extensive camera array can be prohibitively costly. In this
work, we identify the lack of spatial autocorrelation of splat features as one
of the factors contributing to the suboptimal performance of the 3DGS technique
in sparse reconstruction settings. To address the issue, we propose an
optimization strategy that effectively regularizes splat features by modeling
them as the outputs of a corresponding implicit neural field. This results in a
consistent enhancement of reconstruction quality across various scenarios. Our
approach effectively handles static and dynamic cases, as demonstrated by
extensive testing across different setups and scene complexities.
Agile Continuous Jumping in Discontinuous Terrains
Authors: name: Yuxiang Yang, name: Guanya Shi, name: Changyi Lin, name: Xiangyun Meng, name: Rosario Scalise, name: Mateo Guaman Castro, name: Wenhao Yu, name: Tingnan Zhang, name: Ding Zhao, name: Jie Tan, name: Byron Boots
Published: 2024-09-17 06:42:50

We focus on agile, continuous, and terrain-adaptive jumping of quadrupedal
robots in discontinuous terrains such as stairs and stepping stones. Unlike
single-step jumping, continuous jumping requires accurately executing highly
dynamic motions over long horizons, which is challenging for existing
approaches. To accomplish this task, we design a hierarchical learning and
control framework, which consists of a learned heightmap predictor for robust
terrain perception, a reinforcement-learning-based centroidal-level motion
policy for versatile and terrain-adaptive planning, and a low-level model-based
leg controller for accurate motion tracking. In addition, we minimize the
sim-to-real gap by accurately modeling the hardware characteristics. Our
framework enables a Unitree Go1 robot to perform agile and continuous jumps on
human-sized stairs and sparse stepping stones, for the first time to the best
of our knowledge. In particular, the robot can cross two stair steps in each
jump and completes a 3.5m long, 2.8m high, 14-step staircase in 4.5 seconds.
Moreover, the same policy outperforms baselines in various other parkour tasks,
such as jumping over single horizontal or vertical discontinuities. Experiment
videos can be found at https://yxyang.github.io/jumping\_cod/.
On the limits of agency in agent-based models
Authors: name: Ayush Chopra, name: Shashank Kumar, name: Nurullah Giray-Kuru, name: Ramesh Raskar, name: Arnau Quera-Bofarull
Published: 2024-09-14 04:17:24

Agent-based modeling (ABM) seeks to understand the behavior of complex
systems by simulating a collection of agents that act and interact within an
environment. Their practical utility requires capturing realistic environment
dynamics and adaptive agent behavior while efficiently simulating million-size
populations. Recent advancements in large language models (LLMs) present an
opportunity to enhance ABMs by using LLMs as agents with further potential to
capture adaptive behavior. However, the computational infeasibility of using
LLMs for large populations has hindered their widespread adoption. In this
paper, we introduce AgentTorch -- a framework that scales ABMs to millions of
agents while capturing high-resolution agent behavior using LLMs. We benchmark
the utility of LLMs as ABM agents, exploring the trade-off between simulation
scale and individual agency. Using the COVID-19 pandemic as a case study, we
demonstrate how AgentTorch can simulate 8.4 million agents representing New
York City, capturing the impact of isolation and employment behavior on health
and economic outcomes. We compare the performance of different agent
architectures based on heuristic and LLM agents in predicting disease waves and
unemployment rates. Furthermore, we showcase AgentTorch's capabilities for
retrospective, counterfactual, and prospective analyses, highlighting how
adaptive agent behavior can help overcome the limitations of historical data in
policy design. AgentTorch is an open-source project actively being used for
policy-making and scientific discovery around the world. The framework is
available here: github.com/AgentTorch/AgentTorch.
OSV: One Step is Enough for High-Quality Image to Video Generation
Authors: name: Xiaofeng Mao, name: Zhengkai Jiang, name: Fu-Yun Wang, name: Wenbing Zhu, name: Jiangning Zhang, name: Hao Chen, name: Mingmin Chi, name: Yabiao Wang
Published: 2024-09-17 17:16:37

Video diffusion models have shown great potential in generating high-quality
videos, making them an increasingly popular focus. However, their inherent
iterative nature leads to substantial computational and time costs. While
efforts have been made to accelerate video diffusion by reducing inference
steps (through techniques like consistency distillation) and GAN training
(these approaches often fall short in either performance or training
stability). In this work, we introduce a two-stage training framework that
effectively combines consistency distillation with GAN training to address
these challenges. Additionally, we propose a novel video discriminator design,
which eliminates the need for decoding the video latents and improves the final
performance. Our model is capable of producing high-quality videos in merely
one-step, with the flexibility to perform multi-step refinement for further
performance enhancement. Our quantitative evaluation on the OpenWebVid-1M
benchmark shows that our model significantly outperforms existing methods.
Notably, our 1-step performance(FVD 171.15) exceeds the 8-step performance of
the consistency distillation based method, AnimateLCM (FVD 184.79), and
approaches the 25-step performance of advanced Stable Video Diffusion (FVD
156.94).
Kolmogorov-Arnold Transformer
Authors: name: Xingyi Yang, name: Xinchao Wang
Published: 2024-09-16 17:54:51

Transformers stand as the cornerstone of mordern deep learning.
Traditionally, these models rely on multi-layer perceptron (MLP) layers to mix
the information between channels. In this paper, we introduce the
Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces MLP
layers with Kolmogorov-Arnold Network (KAN) layers to enhance the
expressiveness and performance of the model. Integrating KANs into
transformers, however, is no easy feat, especially when scaled up.
Specifically, we identify three key challenges: (C1) Base function. The
standard B-spline function used in KANs is not optimized for parallel computing
on modern hardware, resulting in slower inference speeds. (C2) Parameter and
Computation Inefficiency. KAN requires a unique function for each input-output
pair, making the computation extremely large. (C3) Weight initialization. The
initialization of weights in KANs is particularly challenging due to their
learnable activation functions, which are critical for achieving convergence in
deep neural networks. To overcome the aforementioned challenges, we propose
three key solutions: (S1) Rational basis. We replace B-spline functions with
rational functions to improve compatibility with modern GPUs. By implementing
this in CUDA, we achieve faster computations. (S2) Group KAN. We share the
activation weights through a group of neurons, to reduce the computational load
without sacrificing performance. (S3) Variance-preserving initialization. We
carefully initialize the activation weights to make sure that the activation
variance is maintained across layers. With these designs, KAT scales
effectively and readily outperforms traditional MLP-based transformers.
LLM-Powered Grapheme-to-Phoneme Conversion: Benchmark and Case Study
Authors: name: Mahta Fetrat Qharabagh, name: Zahra Dehghanian, name: Hamid R. Rabiee
Published: 2024-09-13 06:13:55

Grapheme-to-phoneme (G2P) conversion is critical in speech processing,
particularly for applications like speech synthesis. G2P systems must possess
linguistic understanding and contextual awareness of languages with polyphone
words and context-dependent phonemes. Large language models (LLMs) have
recently demonstrated significant potential in various language tasks,
suggesting that their phonetic knowledge could be leveraged for G2P. In this
paper, we evaluate the performance of LLMs in G2P conversion and introduce
prompting and post-processing methods that enhance LLM outputs without
additional training or labeled data. We also present a benchmarking dataset
designed to assess G2P performance on sentence-level phonetic challenges of the
Persian language. Our results show that by applying the proposed methods, LLMs
can outperform traditional G2P tools, even in an underrepresented language like
Persian, highlighting the potential of developing LLM-aided G2P systems.
jina-embeddings-v3: Multilingual Embeddings With Task LoRA
Authors: name: Saba Sturua, name: Isabelle Mohr, name: Mohammad Kalim Akram, name: Michael Günther, name: Bo Wang, name: Markus Krimmel, name: Feng Wang, name: Georgios Mastrapas, name: Andreas Koukounas, name: Andreas Koukounas, name: Nan Wang, name: Han Xiao
Published: 2024-09-16 11:10:29

We introduce jina-embeddings-v3, a novel text embedding model with 570
million parameters, achieves state-of-the-art performance on multilingual data
and long-context retrieval tasks, supporting context lengths of up to 8192
tokens. The model includes a set of task-specific Low-Rank Adaptation (LoRA)
adapters to generate high-quality embeddings for query-document retrieval,
clustering, classification, and text matching. Additionally, Matryoshka
Representation Learning is integrated into the training process, allowing
flexible truncation of embedding dimensions without compromising performance.
Evaluation on the MTEB benchmark shows that jina-embeddings-v3 outperforms the
latest proprietary embeddings from OpenAI and Cohere on English tasks, while
achieving superior performance compared to multilingual-e5-large-instruct
across all multilingual tasks.
Breaking reCAPTCHAv2
Authors: name: Andreas Plesner, name: Tobias Vontobel, name: Roger Wattenhofer
Published: 2024-09-13 13:47:12

Our work examines the efficacy of employing advanced machine learning methods
to solve captchas from Google's reCAPTCHAv2 system. We evaluate the
effectiveness of automated systems in solving captchas by utilizing advanced
YOLO models for image segmentation and classification. Our main result is that
we can solve 100% of the captchas, while previous work only solved 68-71%.
Furthermore, our findings suggest that there is no significant difference in
the number of challenges humans and bots must solve to pass the captchas in
reCAPTCHAv2. This implies that current AI technologies can exploit advanced
image-based captchas. We also look under the hood of reCAPTCHAv2, and find
evidence that reCAPTCHAv2 is heavily based on cookie and browser history data
when evaluating whether a user is human or not. The code is provided alongside
this paper.
Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
Authors: name: Wei Shen, name: Chuheng Zhang
Published: 2024-09-11 02:40:38

Reinforcement learning from human feedback (RLHF) is one of the key
techniques that helps large language models (LLMs) to follow instructions and
provide helpful and harmless responses. While direct policy optimization
methods exist, state-of-the-art LLMs adopt RL-based methods (usually PPO) in
RLHF to train the policy to generate good responses guided by a reward model
learned from preference data. The main challenge of these methods is the
inaccuracy of the intermediate reward model, especially in code generation
tasks that require long and complex reasoning to score a response. We find that
the reliability of the reward model varies across responses assigned with
different rewards. This motivates us to filter the samples whose rewards may be
unreliable to improve signal-to-noise ratio during policy learning, resulting
in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a
proper policy filtration strategy for a given reward model, the coefficient of
determination (R^2) between rewards and actual scores on filtered samples
serves as a good metrics and helps us find several promising strategies. We
provide extensive experiments to validate the effectiveness of PF-PPO in code
generation tasks, and find that some variants of PF-PPO are highly effective
and achieve new state-of-the-art performance across 7-billion-parameter models
on HumanEval, MBPP, and a new and more challenging LeetCode Contest benchmark.
Ferret: Federated Full-Parameter Tuning at Scale for Large Language
  Models
Authors: name: Yao Shu, name: Wenyang Hu, name: See-Kiong Ng, name: Bryan Kian Hsiang Low, name: Fei Richard Yu
Published: 2024-09-10 07:28:13

Large Language Models (LLMs) have become indispensable in numerous real-world
applications. Unfortunately, fine-tuning these models at scale, especially in
federated settings where data privacy and communication efficiency are
critical, presents significant challenges. Existing methods often resort to
parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but
this typically comes at the cost of model accuracy. To address these
limitations, we propose federated full-parameter tuning at scale for LLMs
(Ferret), the first first-order method with shared randomness to enable
scalable full-parameter tuning of LLMs across decentralized data sources while
maintaining competitive model accuracy. Ferret accomplishes this through three
aspects: (1) it employs widely applied first-order methods for efficient local
updates; (2) it projects these updates into a low-dimensional space to
considerably reduce communication overhead; and (3) it reconstructs local
updates from this low-dimensional space with shared randomness to facilitate
effective full-parameter global aggregation, ensuring fast convergence and
competitive final performance. Our rigorous theoretical analyses and insights
along with extensive experiments, show that Ferret significantly enhances the
scalability of existing federated full-parameter tuning approaches by achieving
high computational efficiency, reduced communication overhead, and fast
convergence, all while maintaining competitive model accuracy. Our
implementation is available at https://github.com/allen4747/Ferret.
beeFormer: Bridging the Gap Between Semantic and Interaction Similarity
  in Recommender Systems
Authors: name: Vojtěch Vančura, name: Pavel Kordík, name: Milan Straka
Published: 2024-09-16 14:15:42

Recommender systems often use text-side information to improve their
predictions, especially in cold-start or zero-shot recommendation scenarios,
where traditional collaborative filtering approaches cannot be used. Many
approaches to text-mining side information for recommender systems have been
proposed over recent years, with sentence Transformers being the most prominent
one. However, these models are trained to predict semantic similarity without
utilizing interaction data with hidden patterns specific to recommender
systems. In this paper, we propose beeFormer, a framework for training sentence
Transformer models with interaction data. We demonstrate that our models
trained with beeFormer can transfer knowledge between datasets while
outperforming not only semantic similarity sentence Transformers but also
traditional collaborative filtering methods. We also show that training on
multiple datasets from different domains accumulates knowledge in a single
model, unlocking the possibility of training universal, domain-agnostic
sentence Transformer models to mine text representations for recommender
systems. We release the source code, trained models, and additional details
allowing replication of our experiments at
https://github.com/recombee/beeformer.
Towards Predicting Temporal Changes in a Patient's Chest X-ray Images
  based on Electronic Health Records
Authors: name: Daeun Kyung, name: Junu Kim, name: Tackeun Kim, name: Edward Choi
Published: 2024-09-11 04:49:44

Chest X-ray imaging (CXR) is an important diagnostic tool used in hospitals
to assess patient conditions and monitor changes over time. Generative models,
specifically diffusion-based models, have shown promise in generating realistic
synthetic X-rays. However, these models mainly focus on conditional generation
using single-time-point data, i.e., typically CXRs taken at a specific time
with their corresponding reports, limiting their clinical utility, particularly
for capturing temporal changes. To address this limitation, we propose a novel
framework, EHRXDiff, which predicts future CXR images by integrating previous
CXRs with subsequent medical events, e.g., prescriptions, lab measures, etc.
Our framework dynamically tracks and predicts disease progression based on a
latent diffusion model, conditioned on the previous CXR image and a history of
medical events. We comprehensively evaluate the performance of our framework
across three key aspects, including clinical consistency, demographic
consistency, and visual realism. We demonstrate that our framework generates
high-quality, realistic future images that capture potential temporal changes,
suggesting its potential for further development as a clinical simulation tool.
This could offer valuable insights for patient monitoring and treatment
planning in the medical field.
AudioBERT: Audio Knowledge Augmented Language Model
Authors: name: Hyunjong Ok, name: Suho Yoo, name: Jaeho Lee
Published: 2024-09-12 16:36:39

Recent studies have identified that language models, pretrained on text-only
datasets, often lack elementary visual knowledge, e.g., colors of
everyday objects. Motivated by this observation, we ask whether a similar
shortcoming exists in terms of the auditory knowledge. To answer this
question, we construct a new dataset called AuditoryBench, which consists of
two novel tasks for evaluating auditory knowledge. Based on our analysis using
the benchmark, we find that language models also suffer from a severe lack of
auditory knowledge. To address this limitation, we propose AudioBERT, a novel
method to augment the auditory knowledge of BERT through a retrieval-based
approach. First, we detect auditory knowledge spans in prompts to query our
retrieval model efficiently. Then, we inject audio knowledge into BERT and
switch on low-rank adaptation for effective adaptation when audio knowledge is
required. Our experiments demonstrate that AudioBERT is quite effective,
achieving superior performance on the AuditoryBench. The dataset and code are
available at https://github.com/HJ-Ok/AudioBERT.
One missing piece in Vision and Language: A Survey on Comics
  Understanding
Authors: name: Emanuele Vivoli, name: Andrey Barsky, name: Mohamed Ali Souibgui, name: Artemis LLabres, name: Marco Bertini, name: Dimosthenis Karatzas
Published: 2024-09-14 18:26:26

Vision-language models have recently evolved into versatile systems capable
of high performance across a range of tasks, such as document understanding,
visual question answering, and grounding, often in zero-shot settings. Comics
Understanding, a complex and multifaceted field, stands to greatly benefit from
these advances. Comics, as a medium, combine rich visual and textual
narratives, challenging AI models with tasks that span image classification,
object detection, instance segmentation, and deeper narrative comprehension
through sequential panels. However, the unique structure of comics --
characterized by creative variations in style, reading order, and non-linear
storytelling -- presents a set of challenges distinct from those in other
visual-language domains. In this survey, we present a comprehensive review of
Comics Understanding from both dataset and task perspectives. Our contributions
are fivefold: (1) We analyze the structure of the comics medium, detailing its
distinctive compositional elements; (2) We survey the widely used datasets and
tasks in comics research, emphasizing their role in advancing the field; (3) We
introduce the Layer of Comics Understanding (LoCU) framework, a novel taxonomy
that redefines vision-language tasks within comics and lays the foundation for
future work; (4) We provide a detailed review and categorization of existing
methods following the LoCU framework; (5) Finally, we highlight current
research challenges and propose directions for future exploration, particularly
in the context of vision-language models applied to comics. This survey is the
first to propose a task-oriented framework for comics intelligence and aims to
guide future research by addressing critical gaps in data availability and task
definition. A project associated with this survey is available at
https://github.com/emanuelevivoli/awesome-comics-understanding.
On the Diagram of Thought
Authors: name: Yifan Zhang, name: Yang Yuan, name: Andrew Chi-Chih Yao
Published: 2024-09-16 07:01:41

We introduce Diagram of Thought (DoT), a framework that models iterative
reasoning in large language models (LLMs) as the construction of a directed
acyclic graph (DAG) within a single model. Unlike traditional approaches that
represent reasoning as linear chains or trees, DoT organizes propositions,
critiques, refinements, and verifications into a cohesive DAG structure,
allowing the model to explore complex reasoning pathways while maintaining
logical consistency. Each node in the diagram corresponds to a proposition that
has been proposed, critiqued, refined, or verified, enabling the LLM to
iteratively improve its reasoning through natural language feedback. By
leveraging auto-regressive next-token prediction with role-specific tokens, DoT
facilitates seamless transitions between proposing ideas and critically
evaluating them, providing richer feedback than binary signals. Furthermore, we
formalize the DoT framework using Topos Theory, providing a mathematical
foundation that ensures logical consistency and soundness in the reasoning
process. This approach enhances both the training and inference processes
within a single LLM, eliminating the need for multiple models or external
control mechanisms. DoT offers a conceptual framework for designing
next-generation reasoning-specialized models, emphasizing training efficiency,
robust reasoning capabilities, and theoretical grounding. The code is available
at https://github.com/diagram-of-thought/diagram-of-thought.