Orionnnn/hub-stats huggingface dataset:
Can Large Language Models Understand Symbolic Graphics Programs?
Authors: name: Zeju Qiu, name: Weiyang Liu, name: Haiwen Feng, name: Zhen Liu, name: Tim Z. Xiao, name: Katherine M. Collins, name: Joshua B. Tenenbaum, name: Adrian Weller, name: Michael J. Black, name: Bernhard Schölkopf
Published: 2024-08-15 17:59:57

Assessing the capabilities of large language models (LLMs) is often
challenging, in part, because it is hard to find tasks to which they have not
been exposed during training. We take one step to address this challenge by
turning to a new task: focusing on symbolic graphics programs, which are a
popular representation for graphics content that procedurally generates visual
data. LLMs have shown exciting promise towards program synthesis, but do they
understand symbolic graphics programs? Unlike conventional programs, symbolic
graphics programs can be translated to graphics content. Here, we characterize
an LLM's understanding of symbolic programs in terms of their ability to answer
questions related to the graphics content. This task is challenging as the
questions are difficult to answer from the symbolic programs alone -- yet, they
would be easy to answer from the corresponding graphics content as we verify
through a human experiment. To understand symbolic programs, LLMs may need to
possess the ability to imagine how the corresponding graphics content would
look without directly accessing the rendered visual content. We use this task
to evaluate LLMs by creating a large benchmark for the semantic understanding
of symbolic graphics programs. This benchmark is built via program-graphics
correspondence, hence requiring minimal human efforts. We evaluate current LLMs
on our benchmark to elucidate a preliminary assessment of their ability to
reason about visual scenes from programs. We find that this task distinguishes
existing LLMs and models considered good at reasoning perform better. Lastly,
we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
Specifically, we query GPT4-o with questions and images generated by symbolic
programs. Such data are then used to finetune an LLM. We also find that SIT
data can improve the general instruction following ability of LLMs.
FancyVideo: Towards Dynamic and Consistent Video Generation via
  Cross-frame Textual Guidance
Authors: name: Jiasong Feng, name: Ao Ma, name: Jing Wang, name: Bo Cheng, name: Xiaodan Liang, name: Dawei Leng, name: Yuhui Yin
Published: 2024-08-15 14:47:44

Synthesizing motion-rich and temporally consistent videos remains a challenge
in artificial intelligence, especially when dealing with extended durations.
Existing text-to-video (T2V) models commonly employ spatial cross-attention for
text control, equivalently guiding different frame generations without
frame-specific textual guidance. Thus, the model's capacity to comprehend the
temporal logic conveyed in prompts and generate videos with coherent motion is
restricted. To tackle this limitation, we introduce FancyVideo, an innovative
video generator that improves the existing text-control mechanism with the
well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM
incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner
(TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of
cross-attention, respectively, to achieve frame-specific textual guidance.
Firstly, TII injects frame-specific information from latent features into text
conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines
the correlation matrix between cross-frame textual conditions and latent
features along the time dimension. Lastly, TFB boosts the temporal consistency
of latent features. Extensive experiments comprising both quantitative and
qualitative evaluations demonstrate the effectiveness of FancyVideo. Our
approach achieves state-of-the-art T2V generation results on the EvalCrafter
benchmark and facilitates the synthesis of dynamic and consistent videos. The
video show results can be available at https://fancyvideo.github.io/, and we
will make our code and model weights publicly available.
Seeing and Understanding: Bridging Vision with Chemical Knowledge Via
  ChemVLM
Authors: name: Junxian Li, name: Di Zhang, name: Xunzhi Wang, name: Zeying Hao, name: Jingdi Lei, name: Qian Tan, name: Cai Zhou, name: Wei Liu, name: Weiyun Wang, name: Zhe Chen, name: Wenhai Wang, name: Wei Li, name: Shufei Zhang, name: Mao Su, name: Wanli Ouyang, name: Yuqiang Li, name: Dongzhan Zhou
Published: 2024-08-14 01:16:40

In this technical report, we propose ChemVLM, the first open-source
multimodal large language model dedicated to the fields of chemistry, designed
to address the incompatibility between chemical image understanding and text
analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as
the foundational large model, endowing our model with robust capabilities in
understanding and utilizing chemical text knowledge. Additionally, we employ
InternVIT-6B as a powerful image encoder. We have curated high-quality data
from the chemical domain, including molecules, reaction formulas, and chemistry
examination data, and compiled these into a bilingual multimodal
question-answering dataset. We test the performance of our model on multiple
open-source benchmarks and three custom evaluation sets. Experimental results
demonstrate that our model achieves excellent performance, securing
state-of-the-art results in five out of six involved tasks. Our model can be
found at https://huggingface.co/AI4Chem/ChemVLM-26B.
DeepSpeak Dataset v1.0
Authors: name: Sarah Barrington, name: Matyas Bohacek, name: Hany Farid
Published: 2024-08-09 22:29:43

We describe a large-scale dataset--{\em DeepSpeak}--of real and deepfake
footage of people talking and gesturing in front of their webcams. The real
videos in this first version of the dataset consist of 9 hours of footage
from 220 diverse individuals. Constituting more than 25 hours of footage, the
fake videos consist of a range of different state-of-the-art face-swap and
lip-sync deepfakes with natural and AI-generated voices. We expect to release
future versions of this dataset with different and updated deepfake
technologies. This dataset is made freely available for research and
non-commercial uses; requests for commercial use will be considered.
3D Gaussian Editing with A Single Image
Authors: name: Guan Luo, name: Tian-Xing Xu, name: Ying-Tian Liu, name: Xiao-Xiong Fan, name: Fang-Lue Zhang, name: Song-Hai Zhang
Published: 2024-08-14 13:17:42

The modeling and manipulation of 3D scenes captured from the real world are
pivotal in various applications, attracting growing research interest. While
previous works on editing have achieved interesting results through
manipulating 3D meshes, they often require accurately reconstructed meshes to
perform editing, which limits their application in 3D content generation. To
address this gap, we introduce a novel single-image-driven 3D scene editing
approach based on 3D Gaussian Splatting, enabling intuitive manipulation via
directly editing the content on a 2D image plane. Our method learns to optimize
the 3D Gaussians to align with an edited version of the image rendered from a
user-specified viewpoint of the original scene. To capture long-range object
deformation, we introduce positional loss into the optimization process of 3D
Gaussian Splatting and enable gradient propagation through reparameterization.
To handle occluded 3D Gaussians when rendering from the specified viewpoint, we
build an anchor-based structure and employ a coarse-to-fine optimization
strategy capable of handling long-range deformation while maintaining
structural stability. Furthermore, we design a novel masking strategy to
adaptively identify non-rigid deformation regions for fine-scale modeling.
Extensive experiments show the effectiveness of our method in handling
geometric details, long-range, and non-rigid deformation, demonstrating
superior editing flexibility and quality compared to previous approaches.
PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform
  Generation
Authors: name: Sang-Hoon Lee, name: Ha-Yeong Choi, name: Seong-Whan Lee
Published: 2024-08-14 13:36:17

Recently, universal waveform generation tasks have been investigated
conditioned on various out-of-distribution scenarios. Although GAN-based
methods have shown their strength in fast waveform generation, they are
vulnerable to train-inference mismatch scenarios such as two-stage
text-to-speech. Meanwhile, diffusion-based models have shown their powerful
generative performance in other domains; however, they stay out of the
limelight due to slow inference speed in waveform generation tasks. Above all,
there is no generator architecture that can explicitly disentangle the natural
periodic features of high-resolution waveform signals. In this paper, we
propose PeriodWave, a novel universal waveform generation model. First, we
introduce a period-aware flow matching estimator that can capture the periodic
features of the waveform signal when estimating the vector fields.
Additionally, we utilize a multi-period estimator that avoids overlaps to
capture different periodic features of waveform signals. Although increasing
the number of periods can improve the performance significantly, this requires
more computational costs. To reduce this issue, we also propose a single
period-conditional universal estimator that can feed-forward parallel by
period-wise batch inference. Additionally, we utilize discrete wavelet
transform to losslessly disentangle the frequency information of waveform
signals for high-frequency modeling, and introduce FreeU to reduce the
high-frequency noise for waveform generation. The experimental results
demonstrated that our model outperforms the previous models both in
Mel-spectrogram reconstruction and text-to-speech tasks. All source code will
be available at https://github.com/sh-lee-prml/PeriodWave.
Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space
Authors: name: Hyunjee Lee, name: Youngsik Yun, name: Jeongmin Bae, name: Seoha Kim, name: Youngjung Uh
Published: 2024-08-14 09:50:02

Understanding the 3D semantics of a scene is a fundamental problem for
various scenarios such as embodied agents. While NeRFs and 3DGS excel at
novel-view synthesis, previous methods for understanding their semantics have
been limited to incomplete 3D understanding: their segmentation results are 2D
masks and their supervision is anchored at 2D pixels. This paper revisits the
problem set to pursue a better 3D understanding of a scene modeled by NeRFs and
3DGS as follows. 1) We directly supervise the 3D points to train the language
embedding field. It achieves state-of-the-art accuracy without relying on
multi-scale language embeddings. 2) We transfer the pre-trained language field
to 3DGS, achieving the first real-time rendering speed without sacrificing
training time or accuracy. 3) We introduce a 3D querying and evaluation
protocol for assessing the reconstructed geometry and semantics together. Code,
checkpoints, and annotations will be available online. Project page:
https://hyunji12.github.io/Open3DRF
Aquila2 Technical Report
Authors: name: Bo-Wen Zhang, name: Liangdong Wang, name: Jijie Li, name: Shuhao Gu, name: Xinya Wu, name: Zhengduo Zhang, name: Boyan Gao, name: Yulong Ao, name: Guang Liu
Published: 2024-08-14 09:34:19

This paper introduces the Aquila2 series, which comprises a wide range of
bilingual models with parameter sizes of 7, 34, and 70 billion. These models
are trained based on an innovative framework named HeuriMentor (HM), which
offers real-time insights into model convergence and enhances the training
process and data management. The HM System, comprising the Adaptive Training
Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
allows for precise monitoring of the model's training progress and enables
efficient optimization of data distribution, thereby enhancing training
effectiveness. Extensive evaluations show that the Aquila2 model series
performs comparably well on both English and Chinese benchmarks. Specifically,
Aquila2-34B demonstrates only a slight decrease in performance when quantized
to Int4. Furthermore, we have made our training code
(https://github.com/FlagOpen/FlagScale) and model weights
(https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
research and the development of applications.
Generative Photomontage
Authors: name: Sean J. Liu, name: Nupur Kumari, name: Ariel Shamir, name: Jun-Yan Zhu
Published: 2024-08-13 17:59:51

Text-to-image models are powerful tools for image creation. However, the
generation process is akin to a dice roll and makes it difficult to achieve a
single image that captures everything a user wants. In this paper, we propose a
framework for creating the desired image by compositing it from various parts
of generated images, in essence forming a Generative Photomontage. Given a
stack of images generated by ControlNet using the same input condition and
different seeds, we let users select desired parts from the generated results
using a brush stroke interface. We introduce a novel technique that takes in
the user's brush strokes, segments the generated images using a graph-based
optimization in diffusion feature space, and then composites the segmented
regions via a new feature-space blending method. Our method faithfully
preserves the user-selected regions while compositing them harmoniously. We
demonstrate that our flexible framework can be used for many applications,
including generating new appearance combinations, fixing incorrect shapes and
artifacts, and improving prompt alignment. We show compelling results for each
application and demonstrate that our method outperforms existing image blending
methods and various baselines.
InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic
  Mathematical Reasoning
Authors: name: Bo-Wen Zhang, name: Yan Yan, name: Lin Li, name: Guang Liu
Published: 2024-08-09 08:18:20

Recent advancements in Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT)
methods have greatly enhanced language models' mathematical reasoning
capabilities, facilitating their integration into instruction tuning datasets
with LLMs. However, existing methods for large-scale dataset creation require
substantial seed data and high computational costs for data synthesis, posing
significant challenges for scalability. We introduce InfinityMATH, a scalable
instruction tuning dataset for programmatic mathematical reasoning. The
construction pipeline emphasizes decoupling numbers from mathematical problems
to synthesize number-independent programs, enabling efficient and flexible
scaling while minimizing dependency on specific numerical values. Fine-tuning
experiments with open-source language and code models, such as Llama2 and
CodeLlama, demonstrate the practical benefits of InfinityMATH. These fine-tuned
models, showed significant relative improvements on both in-domain and
out-of-domain benchmarks, ranging from 184.7% to 514.3% on average.
Additionally, these models exhibited high robustness on the GSM8K+ and MATH+
benchmarks, which are enhanced version of test sets with simply the number
variations. InfinityMATH ensures that models are more versatile and effective
across a broader range of mathematical problems. The data is available at
https://huggingface.co/datasets/flagopen/InfinityMATH.
Diversity Empowers Intelligence: Integrating Expertise of Software
  Engineering Agents
Authors: name: Kexun Zhang, name: Weiran Yao, name: Zuxin Liu, name: Yihao Feng, name: Zhiwei Liu, name: Rithesh Murthy, name: Tian Lan, name: Lei Li, name: Renze Lou, name: Jiacheng Xu, name: Bo Pang, name: Yingbo Zhou, name: Shelby Heinecke, name: Silvio Savarese, name: Huan Wang, name: Caiming Xiong
Published: 2024-08-13 17:50:28

Large language model (LLM) agents have shown great potential in solving
real-world software engineering (SWE) problems. The most advanced open-source
SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite.
However, these sophisticated agent frameworks exhibit varying strengths,
excelling in certain tasks while underperforming in others. To fully harness
the diversity of these agents, we propose DEI (Diversity Empowered
Intelligence), a framework that leverages their unique expertise. DEI functions
as a meta-module atop existing SWE agent frameworks, managing agent collectives
for enhanced problem-solving. Experimental results show that a DEI-guided
committee of agents is able to surpass the best individual agent's performance
by a large margin. For instance, a group of open-source SWE agents, with a
maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3%
resolve rate with DEI, making a 25% improvement and beating most closed-source
solutions. Our best-performing group excels with a 55% resolve rate, securing
the highest ranking on SWE-Bench Lite. Our findings contribute to the growing
body of research on collaborative AI systems and their potential to solve
complex software engineering challenges.
ZePo: Zero-Shot Portrait Stylization with Faster Sampling
Authors: name: Jin Liu, name: Huaibo Huang, name: Jie Cao, name: Ran He
Published: 2024-08-10 08:53:41

Diffusion-based text-to-image generation models have significantly advanced
the field of art content synthesis. However, current portrait stylization
methods generally require either model fine-tuning based on examples or the
employment of DDIM Inversion to revert images to noise space, both of which
substantially decelerate the image generation process. To overcome these
limitations, this paper presents an inversion-free portrait stylization
framework based on diffusion models that accomplishes content and style feature
fusion in merely four sampling steps. We observed that Latent Consistency
Models employing consistency distillation can effectively extract
representative Consistency Features from noisy images. To blend the Consistency
Features extracted from both content and style images, we introduce a Style
Enhancement Attention Control technique that meticulously merges content and
style features within the attention space of the target image. Moreover, we
propose a feature merging strategy to amalgamate redundant features in
Consistency Features, thereby reducing the computational load of attention
control. Extensive experiments have validated the effectiveness of our proposed
framework in enhancing stylization efficiency and fidelity. The code is
available at https://github.com/liujin112/ZePo.
Adapting General Disentanglement-Based Speaker Anonymization for
  Enhanced Emotion Preservation
Authors: name: Xiaoxiao Miao, name: Yuxiang Zhang, name: Xin Wang, name: Natalia Tomashenko, name: Donny Cheng Lock Soh, name: Ian Mcloughlin
Published: 2024-08-12 05:40:21

A general disentanglement-based speaker anonymization system typically
separates speech into content, speaker, and prosody features using individual
encoders. This paper explores how to adapt such a system when a new speech
attribute, for example, emotion, needs to be preserved to a greater extent.
While existing systems are good at anonymizing speaker embeddings, they are not
designed to preserve emotion. Two strategies for this are examined. First, we
show that integrating emotion embeddings from a pre-trained emotion encoder can
help preserve emotional cues, even though this approach slightly compromises
privacy protection. Alternatively, we propose an emotion compensation strategy
as a post-processing step applied to anonymized speaker embeddings. This
conceals the original speaker's identity and reintroduces the emotional traits
lost during speaker embedding anonymization. Specifically, we model the emotion
attribute using support vector machines to learn separate boundaries for each
emotion. During inference, the original speaker embedding is processed in two
ways: one, by an emotion indicator to predict emotion and select the
emotion-matched SVM accurately; and two, by a speaker anonymizer to conceal
speaker characteristics. The anonymized speaker embedding is then modified
along the corresponding SVM boundary towards an enhanced emotional direction to
save the emotional cues. The proposed strategies are also expected to be useful
for adapting a general disentanglement-based speaker anonymization system to
preserve other target paralinguistic attributes, with potential for a range of
downstream tasks.
Design Proteins Using Large Language Models: Enhancements and
  Comparative Analyses
Authors: name: Kamyar Zeinalipour, name: Neda Jamshidi, name: Monica Bianchini, name: Marco Maggini, name: Marco Gori
Published: 2024-08-12 08:17:27

Pre-trained LLMs have demonstrated substantial capabilities across a range of
conventional natural language processing (NLP) tasks, such as summarization and
entity recognition. In this paper, we explore the application of LLMs in the
generation of high-quality protein sequences. Specifically, we adopt a suite of
pre-trained LLMs, including Mistral-7B1, Llama-2-7B2, Llama-3-8B3, and
gemma-7B4, to produce valid protein sequences. All of these models are publicly
available.5 Unlike previous work in this field, our approach utilizes a
relatively small dataset comprising 42,000 distinct human protein sequences. We
retrain these models to process protein-related data, ensuring the generation
of biologically feasible protein structures. Our findings demonstrate that even
with limited data, the adapted models exhibit efficiency comparable to
established protein-focused models such as ProGen varieties, ProtGPT2, and
ProLLaMA, which were trained on millions of protein sequences. To validate and
quantify the performance of our models, we conduct comparative analyses
employing standard metrics such as pLDDT, RMSD, TM-score, and REU. Furthermore,
we commit to making the trained versions of all four models publicly available,
fostering greater transparency and collaboration in the field of computational
biology.
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced
  Data
Authors: name: Haoran Sun, name: Renren Jin, name: Shaoyang Xu, name: Leiyu Pan, name: Supryadi, name: Menglong Cui, name: Jiangcun Du, name: Yikun Lei, name: Lei Yang, name: Ling Shi, name: Juesi Xiao, name: Shaolin Zhu, name: Deyi Xiong
Published: 2024-08-12 16:34:56

Large language models (LLMs) have demonstrated prowess in a wide range of
tasks. However, many LLMs exhibit significant performance discrepancies between
high- and low-resource languages. To mitigate this challenge, we present
FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the
need of the research community for balanced and high-performing multilingual
capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is
trained from scratch on a meticulously balanced multilingual data repository
that contains 600 billion tokens covering 43 natural languages and 16
programming languages. In addition to the base model, we also develop two
instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse
multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined
with DPO on a preference dataset for enhanced alignment ability. Extensive
experiments on a wide range of multilingual benchmarks demonstrate the
competitive performance of FuxiTranyu against existing multilingual LLMs, e.g.,
BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability
analyses at both the neuron and representation level suggest that FuxiTranyu is
able to learn consistent multilingual representations across different
languages. To promote further research into multilingual LLMs and their working
mechanisms, we release both the base and instruction-tuned FuxiTranyu models
together with 58 pretraining checkpoints at HuggingFace and Github.