Orionnnn/hub-stats huggingface dataset:
Self-Supervised Audio-Visual Soundscape Stylization
Authors: name: Tingle Li, name: Renhao Wang, name: Po-Yao Huang, name: Andrew Owens, name: Gopala Anumanchipalli
Published: 2024-09-22 06:57:33

Speech sounds convey a great deal of information about the scenes, resulting
in a variety of effects ranging from reverberation to additional ambient
sounds. In this paper, we manipulate input speech to sound as though it was
recorded within a different scene, given an audio-visual conditional example
recorded from that scene. Our model learns through self-supervision, taking
advantage of the fact that natural video contains recurring sound events and
textures. We extract an audio clip from a video and apply speech enhancement.
We then train a latent diffusion model to recover the original speech, using
another audio-visual clip taken from elsewhere in the video as a conditional
hint. Through this process, the model learns to transfer the conditional
example's sound properties to the input speech. We show that our model can be
successfully trained using unlabeled, in-the-wild videos, and that an
additional visual signal can improve its sound prediction abilities. Please see
our project webpage for video results:
https://tinglok.netlify.app/files/avsoundscape/
A Case Study of Web App Coding with OpenAI Reasoning Models
Authors: name: Yi Cui
Published: 2024-09-19 06:58:02

This paper presents a case study of coding tasks by the latest reasoning
models of OpenAI, i.e. o1-preview and o1-mini, in comparison with other
frontier models. The o1 models deliver SOTA results for WebApp1K, a single-task
benchmark. To this end, we introduce WebApp1K-Duo, a harder benchmark doubling
number of tasks and test cases. The new benchmark causes the o1 model
performances to decline significantly, falling behind Claude 3.5. Moreover,
they consistently fail when confronted with atypical yet correct test cases, a
trap non-reasoning models occasionally avoid. We hypothesize that the
performance variability is due to instruction comprehension. Specifically, the
reasoning mechanism boosts performance when all expectations are captured,
meanwhile exacerbates errors when key expectations are missed, potentially
impacted by input lengths. As such, we argue that the coding success of
reasoning models hinges on the top-notch base model and SFT to ensure
meticulous adherence to instructions.
A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
Authors: name: Yunfei Xie, name: Juncheng Wu, name: Haoqin Tu, name: Siwei Yang, name: Bingchen Zhao, name: Yongshuo Zong, name: Qiao Jin, name: Cihang Xie, name: Yuyin Zhou
Published: 2024-09-23 17:59:43

Large language models (LLMs) have exhibited remarkable capabilities across
various domains and tasks, pushing the boundaries of our knowledge in learning
and cognition. The latest model, OpenAI's o1, stands out as the first LLM with
an internalized chain-of-thought technique using reinforcement learning
strategies. While it has demonstrated surprisingly strong capabilities on
various general language tasks, its performance in specialized fields such as
medicine remains unknown. To this end, this report provides a comprehensive
exploration of o1 on different medical scenarios, examining 3 key aspects:
understanding, reasoning, and multilinguality. Specifically, our evaluation
encompasses 6 tasks using data from 37 medical datasets, including two newly
constructed and more challenging question-answering (QA) tasks based on
professional medical quizzes from the New England Journal of Medicine (NEJM)
and The Lancet. These datasets offer greater clinical relevance compared to
standard medical QA benchmarks such as MedQA, translating more effectively into
real-world clinical utility. Our analysis of o1 suggests that the enhanced
reasoning ability of LLMs may (significantly) benefit their capability to
understand various medical instructions and reason through complex clinical
scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average
of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios.
But meanwhile, we identify several weaknesses in both the model capability and
the existing evaluation protocols, including hallucination, inconsistent
multilingual ability, and discrepant metrics for evaluation. We release our raw
data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future
research.
An adapted large language model facilitates multiple medical tasks in
  diabetes care
Authors: name: Lai Wei, name: Zhen Ying, name: Muyang He, name: Yutong Chen, name: Qian Yang, name: Yanzhe Hong, name: Jiaping Lu, name: Xiaoying Li, name: Weiran Huang, name: Ying Chen
Published: 2024-09-20 03:47:54

Diabetes is a chronic disease that poses a significant global health burden,
and optimizing diabetes management requires multi-stakeholder collaboration.
Large language models (LLMs) have shown promise in various healthcare
scenarios, but their effectiveness across a diverse range of diabetes tasks
remains unproven. In this study, we introduced a framework to train and
validate diabetes-specific LLMs. We first developed a comprehensive data
processing pipeline that includes data collection, filtering, augmentation and
refinement. This approach contributes to creating a high-quality,
diabetes-specific dataset, and several evaluation benchmarks entirely from
scratch. Utilizing the collected training dataset, we fine-tuned a
diabetes-specific LLM family that demonstrated state-of-the-art proficiency in
understanding and processing various diabetes tasks compared to other LLMs.
Furthermore, clinical studies showed the potential applications of our models
in diabetes care, including providing personalized healthcare, assisting
medical education, and streamlining clinical tasks. In conclusion, our study
introduced a framework to develop and evaluate a diabetes-specific LLM family,
and highlighted its potential to enhance clinical practice and provide
personalized, data-driven support for diabetes support when facing different
end users. The code is provided via GitHub at
https://github.com/waltonfuture/Diabetica.
LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless
  Integration of Multi Active/Passive Core-Agents
Authors: name: Amine B. Hassouna, name: Hana Chaari, name: Ines Belhaj
Published: 2024-09-17 17:54:17

The integration of tools in LLM-based agents overcame the difficulties of
standalone LLMs and traditional agents' limited capabilities. However, the
conjunction of these technologies and the proposed enhancements in several
state-of-the-art works followed a non-unified software architecture resulting
in a lack of modularity. Indeed, they focused mainly on functionalities and
overlooked the definition of the component's boundaries within the agent. This
caused terminological and architectural ambiguities between researchers which
we addressed in this paper by proposing a unified framework that establishes a
clear foundation for LLM-based agents' development from both functional and
software architectural perspectives.
  Our framework, LLM-Agent-UMF (LLM-based Agent Unified Modeling Framework),
clearly distinguishes between the different components of an agent, setting
LLMs, and tools apart from a newly introduced element: the core-agent, playing
the role of the central coordinator of the agent which comprises five modules:
planning, memory, profile, action, and security, the latter often neglected in
previous works. Differences in the internal structure of core-agents led us to
classify them into a taxonomy of passive and active types. Based on this, we
proposed different multi-core agent architectures combining unique
characteristics of various individual agents.
  For evaluation purposes, we applied this framework to a selection of
state-of-the-art agents, thereby demonstrating its alignment with their
functionalities and clarifying the overlooked architectural aspects. Moreover,
we thoroughly assessed four of our proposed architectures by integrating
distinctive agents into hybrid active/passive core-agents' systems. This
analysis provided clear insights into potential improvements and highlighted
the challenges involved in the combination of specific agents.
YesBut: A High-Quality Annotated Multimodal Dataset for evaluating
  Satire Comprehension capability of Vision-Language Models
Authors: name: Abhilash Nandy, name: Yash Agarwal, name: Ashish Patwa, name: Millon Madhur Das, name: Aman Bansal, name: Ankit Raj, name: Pawan Goyal, name: Niloy Ganguly
Published: 2024-09-20 15:45:29

Understanding satire and humor is a challenging task for even current
Vision-Language models. In this paper, we propose the challenging tasks of
Satirical Image Detection (detecting whether an image is satirical),
Understanding (generating the reason behind the image being satirical), and
Completion (given one half of the image, selecting the other half from 2 given
options, such that the complete image is satirical) and release a high-quality
dataset YesBut, consisting of 2547 images, 1084 satirical and 1463
non-satirical, containing different artistic styles, to evaluate those tasks.
Each satirical image in the dataset depicts a normal scenario, along with a
conflicting scenario which is funny or ironic. Despite the success of current
Vision-Language Models on multimodal tasks such as Visual QA and Image
Captioning, our benchmarking experiments show that such models perform poorly
on the proposed tasks on the YesBut Dataset in Zero-Shot Settings w.r.t both
automated as well as human evaluation. Additionally, we release a dataset of
119 real, satirical photographs for further research. The dataset and code are
available at https://github.com/abhi1nandy2/yesbut_dataset.
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented
  Generation
Authors: name: Satyapriya Krishna, name: Kalpesh Krishna, name: Anhad Mohananey, name: Steven Schwarcz, name: Adam Stambler, name: Shyam Upadhyay, name: Manaal Faruqui
Published: 2024-09-19 17:52:07

Large Language Models (LLMs) have demonstrated significant performance
improvements across various cognitive tasks. An emerging application is using
LLMs to enhance retrieval-augmented generation (RAG) capabilities. These
systems require LLMs to understand user queries, retrieve relevant information,
and synthesize coherent and accurate responses. Given the increasing real-world
deployment of such systems, comprehensive evaluation becomes crucial. To this
end, we propose FRAMES (Factuality, Retrieval, And reasoning MEasurement Set),
a high-quality evaluation dataset designed to test LLMs' ability to provide
factual responses, assess retrieval capabilities, and evaluate the reasoning
required to generate final answers. While previous work has provided datasets
and benchmarks to evaluate these abilities in isolation, FRAMES offers a
unified framework that provides a clearer picture of LLM performance in
end-to-end RAG scenarios. Our dataset comprises challenging multi-hop questions
that require the integration of information from multiple sources. We present
baseline results demonstrating that even state-of-the-art LLMs struggle with
this task, achieving 0.40 accuracy with no retrieval. The accuracy is
significantly improved with our proposed multi-step retrieval pipeline,
achieving an accuracy of 0.66 (>50% improvement). We hope our work will help
bridge evaluation gaps and assist in developing more robust and capable RAG
systems.
Hackphyr: A Local Fine-Tuned LLM Agent for Network Security Environments
Authors: name: Maria Rigaki, name: Carlos Catania, name: Sebastian Garcia
Published: 2024-09-17 15:28:25

Large Language Models (LLMs) have shown remarkable potential across various
domains, including cybersecurity. Using commercial cloud-based LLMs may be
undesirable due to privacy concerns, costs, and network connectivity
constraints. In this paper, we present Hackphyr, a locally fine-tuned LLM to be
used as a red-team agent within network security environments. Our fine-tuned 7
billion parameter model can run on a single GPU card and achieves performance
comparable with much larger and more powerful commercial models such as GPT-4.
Hackphyr clearly outperforms other models, including GPT-3.5-turbo, and
baselines, such as Q-learning agents in complex, previously unseen scenarios.
To achieve this performance, we generated a new task-specific cybersecurity
dataset to enhance the base model's capabilities. Finally, we conducted a
comprehensive analysis of the agents' behaviors that provides insights into the
planning abilities and potential shortcomings of such agents, contributing to
the broader understanding of LLM-based agents in cybersecurity contexts
Minstrel: Structural Prompt Generation with Multi-Agents Coordination
  for Non-AI Experts
Authors: name: Ming Wang, name: Yuanzhong Liu, name: Xiaoyu Liang, name: Yijie Huang, name: Daling Wang, name: Xiaocui Yang, name: Sijia Shen, name: Shi Feng, name: Xiaoming Zhang, name: Chaofeng Guan, name: Yifei Zhang
Published: 2024-09-20 12:30:03

LLMs have demonstrated commendable performance across diverse domains.
Nevertheless, formulating high-quality prompts to assist them in their work
poses a challenge for non-AI experts. Existing research in prompt engineering
suggests somewhat scattered optimization principles and designs empirically
dependent prompt optimizers. Unfortunately, these endeavors lack a structural
design, incurring high learning costs and it is not conducive to the iterative
updating of prompts, especially for non-AI experts. Inspired by structured
reusable programming languages, we propose LangGPT, a structural prompt design
framework. Furthermore, we introduce Minstrel, a multi-generative agent system
with reflection to automate the generation of structural prompts. Experiments
and the case study illustrate that structural prompts generated by Minstrel or
written manually significantly enhance the performance of LLMs. Furthermore, we
analyze the ease of use of structural prompts through a user survey in our
online community.
Temporally Aligned Audio for Video with Autoregression
Authors: name: Ilpo Viertola, name: Vladimir Iashin, name: Esa Rahtu
Published: 2024-09-20 17:59:01

We introduce V-AURA, the first autoregressive model to achieve high temporal
alignment and relevance in video-to-audio generation. V-AURA uses a
high-framerate visual feature extractor and a cross-modal audio-visual feature
fusion strategy to capture fine-grained visual motion events and ensure precise
temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
with high audio-visual relevance. VisualSound is based on VGGSound, a video
dataset consisting of in-the-wild samples extracted from YouTube. During the
curation, we remove samples where auditory events are not aligned with the
visual ones. V-AURA outperforms current state-of-the-art models in temporal
alignment and semantic relevance while maintaining comparable audio quality.
Code, samples, VisualSound and models are available at
https://v-aura.notion.site
Prithvi WxC: Foundation Model for Weather and Climate
Authors: name: Johannes Schmude, name: Sujit Roy, name: Will Trojak, name: Johannes Jakubik, name: Daniel Salles Civitarese, name: Shraddha Singh, name: Julian Kuehnert, name: Kumar Ankur, name: Aman Gupta, name: Christopher E Phillips, name: Romeo Kienzler, name: Daniela Szwarcman, name: Vishal Gaur, name: Rajat Shinde, name: Rohit Lal, name: Arlindo Da Silva, name: Jorge Luis Guevara Diaz, name: Anne Jones, name: Simon Pfreundschuh, name: Amy Lin, name: Aditi Sheshadri, name: Udaysankar Nair, name: Valentine Anantharaj, name: Hendrik Hamann, name: Campbell Watson, name: Manil Maskey, name: Tsengdar J Lee, name: Juan Bernabe Moreno, name: Rahul Ramachandran
Published: 2024-09-20 15:53:17

Triggered by the realization that AI emulators can rival the performance of
traditional numerical weather prediction models running on HPC systems, there
is now an increasing number of large AI models that address use cases such as
forecasting, downscaling, or nowcasting. While the parallel developments in the
AI literature focus on foundation models -- models that can be effectively
tuned to address multiple, different use cases -- the developments on the
weather and climate side largely focus on single-use cases with particular
emphasis on mid-range forecasting. We close this gap by introducing Prithvi
WxC, a 2.3 billion parameter foundation model developed using 160 variables
from the Modern-Era Retrospective Analysis for Research and Applications,
Version 2 (MERRA-2). Prithvi WxC employs an encoder-decoder-based architecture,
incorporating concepts from various recent transformer models to effectively
capture both regional and global dependencies in the input data. The model has
been designed to accommodate large token counts to model weather phenomena in
different topologies at fine resolutions. Furthermore, it is trained with a
mixed objective that combines the paradigms of masked reconstruction with
forecasting. We test the model on a set of challenging downstream tasks namely:
Autoregressive rollout forecasting, Downscaling, Gravity wave flux
parameterization, and Extreme events estimation. The pretrained model with 2.3
billion parameters, along with the associated fine-tuning workflows, has been
publicly released as an open-source contribution via Hugging Face.
Imagine yourself: Tuning-Free Personalized Image Generation
Authors: name: Zecheng He, name: Bo Sun, name: Felix Juefei-Xu, name: Haoyu Ma, name: Ankit Ramchandani, name: Vincent Cheung, name: Siddharth Shah, name: Anmol Kalia, name: Harihar Subramanyam, name: Alireza Zareian, name: Li Chen, name: Ankit Jain, name: Ning Zhang, name: Peizhao Zhang, name: Roshan Sumbaly, name: Peter Vajda, name: Animesh Sinha
Published: 2024-09-20 09:21:49

Diffusion models have demonstrated remarkable efficacy across various
image-to-image tasks. In this research, we introduce Imagine yourself, a
state-of-the-art model designed for personalized image generation. Unlike
conventional tuning-based personalization techniques, Imagine yourself operates
as a tuning-free model, enabling all users to leverage a shared framework
without individualized adjustments. Moreover, previous work met challenges
balancing identity preservation, following complex prompts and preserving good
visual quality, resulting in models having strong copy-paste effect of the
reference images. Thus, they can hardly generate images following prompts that
require significant changes to the reference image, \eg, changing facial
expression, head and body poses, and the diversity of the generated images is
low. To address these limitations, our proposed method introduces 1) a new
synthetic paired data generation mechanism to encourage image diversity, 2) a
fully parallel attention architecture with three text encoders and a fully
trainable vision encoder to improve the text faithfulness, and 3) a novel
coarse-to-fine multi-stage finetuning methodology that gradually pushes the
boundary of visual quality. Our study demonstrates that Imagine yourself
surpasses the state-of-the-art personalization model, exhibiting superior
capabilities in identity preservation, visual quality, and text alignment. This
model establishes a robust foundation for various personalization applications.
Human evaluation results validate the model's SOTA superiority across all
aspects (identity preservation, text faithfulness, and visual appeal) compared
to the previous personalization models.
Colorful Diffuse Intrinsic Image Decomposition in the Wild
Authors: name: Chris Careaga, name: Yağız Aksoy
Published: 2024-09-20 17:59:40

Intrinsic image decomposition aims to separate the surface reflectance and
the effects from the illumination given a single photograph. Due to the
complexity of the problem, most prior works assume a single-color illumination
and a Lambertian world, which limits their use in illumination-aware image
editing applications. In this work, we separate an input image into its diffuse
albedo, colorful diffuse shading, and specular residual components. We arrive
at our result by gradually removing first the single-color illumination and
then the Lambertian-world assumptions. We show that by dividing the problem
into easier sub-problems, in-the-wild colorful diffuse shading estimation can
be achieved despite the limited ground-truth datasets. Our extended intrinsic
model enables illumination-aware analysis of photographs and can be used for
image editing applications such as specularity removal and per-pixel white
balancing.
V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic
  Gaussians
Authors: name: Penghao Wang, name: Zhirui Zhang, name: Liao Wang, name: Kaixin Yao, name: Siyuan Xie, name: Jingyi Yu, name: Minye Wu, name: Lan Xu
Published: 2024-09-20 16:54:27

Experiencing high-fidelity volumetric video as seamlessly as 2D videos is a
long-held dream. However, current dynamic 3DGS methods, despite their high
rendering quality, face challenges in streaming on mobile devices due to
computational and bandwidth constraints. In this paper, we introduce
V3(Viewing Volumetric Videos), a novel approach that enables
high-quality mobile rendering through the streaming of dynamic Gaussians. Our
key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of
hardware video codecs. Additionally, we propose a two-stage training strategy
to reduce storage requirements with rapid training speed. The first stage
employs hash encoding and shallow MLP to learn motion, then reduces the number
of Gaussians through pruning to meet the streaming requirements, while the
second stage fine tunes other Gaussian attributes using residual entropy loss
and temporal loss to improve temporal continuity. This strategy, which
disentangles motion and appearance, maintains high rendering quality with
compact storage requirements. Meanwhile, we designed a multi-platform player to
decode and render 2D Gaussian videos. Extensive experiments demonstrate the
effectiveness of V3, outperforming other methods by enabling
high-quality rendering and streaming on common devices, which is unseen before.
As the first to stream dynamic Gaussians on mobile devices, our companion
player offers users an unprecedented volumetric video experience, including
smooth scrolling and instant sharing. Our project page with source code is
available at https://authoritywang.github.io/v3/.
Portrait Video Editing Empowered by Multimodal Generative Priors
Authors: name: Xuan Gao, name: Haiyao Xiao, name: Chenglai Zhong, name: Shimin Hu, name: Yudong Guo, name: Juyong Zhang
Published: 2024-09-20 15:45:13

We introduce PortraitGen, a powerful portrait video editing method that
achieves consistent and expressive stylization with multimodal prompts.
Traditional portrait video editing methods often struggle with 3D and temporal
consistency, and typically lack in rendering quality and efficiency. To address
these issues, we lift the portrait video frames to a unified dynamic 3D
Gaussian field, which ensures structural and temporal coherence across frames.
Furthermore, we design a novel Neural Gaussian Texture mechanism that not only
enables sophisticated style editing but also achieves rendering speed over
100FPS. Our approach incorporates multimodal inputs through knowledge distilled
from large-scale 2D generative models. Our system also incorporates expression
similarity guidance and a face-aware portrait editing module, effectively
mitigating degradation issues associated with iterative dataset updates.
Extensive experiments demonstrate the temporal consistency, editing efficiency,
and superior rendering quality of our method. The broad applicability of the
proposed approach is demonstrated through various applications, including
text-driven editing, image-driven editing, and relighting, highlighting its
great potential to advance the field of video editing. Demo videos and released
code are provided in our project page: https://ustc3dv.github.io/PortraitGen/