Orionnnn/hub-stats huggingface dataset:
Learning to Predict Program Execution by Modeling Dynamic Dependency on
  Code Graphs
Authors: name: Cuong Chi Le, name: Hoang Nhat Phan, name: Huy Nhat Phan, name: Tien N. Nguyen, name: Nghi D. Q. Bui
Published: 2024-08-05 20:32:00

Predicting program behavior without execution is an essential and challenging
task in software engineering. Traditional models often struggle to capture
dynamic dependencies and interactions within code. This paper introduces a
novel machine learning-based framework called CodeFlowrepresents, which
predicts code coverage and detects runtime errors through Dynamic Dependencies
Learning. Utilizing control flow graphs (CFGs), CodeFlowrepresents all possible
execution paths and the relationships between different statements, offering a
comprehensive understanding of program behavior. It constructs CFGs to depict
execution paths and learns vector representations for CFG nodes, capturing
static control-flow dependencies. Additionally, it learns dynamic dependencies
through execution traces, which reflect the impacts among statements during
execution. This approach enables accurate prediction of code coverage and
identification of runtime errors. Empirical evaluations show significant
improvements in code coverage prediction accuracy and effective localization of
runtime errors, surpassing current models.
Advancing Molecular Machine (Learned) Representations with
  Stereoelectronics-Infused Molecular Graphs
Authors: name: Daniil A. Boiko, name: Thiago Reschützegger, name: Benjamin Sanchez-Lengeling, name: Samuel M. Blau, name: Gabe Gomes
Published: 2024-08-08 15:21:07

Molecular representation is a foundational element in our understanding of
the physical world. Its importance ranges from the fundamentals of chemical
reactions to the design of new therapies and materials. Previous molecular
machine learning models have employed strings, fingerprints, global features,
and simple molecular graphs that are inherently information-sparse
representations. However, as the complexity of prediction tasks increases, the
molecular representation needs to encode higher fidelity information. This work
introduces a novel approach to infusing quantum-chemical-rich information into
molecular graphs via stereoelectronic effects. We show that the explicit
addition of stereoelectronic interactions significantly improves the
performance of molecular machine learning models. Furthermore,
stereoelectronics-infused representations can be learned and deployed with a
tailored double graph neural network workflow, enabling its application to any
downstream molecular machine learning task. Finally, we show that the learned
representations allow for facile stereoelectronic evaluation of previously
intractable systems, such as entire proteins, opening new avenues of molecular
design.
Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language
  Adaptation of LLMs for Low-Resource NLP
Authors: name: François Remy, name: Pieter Delobelle, name: Hayastan Avetisyan, name: Alfiya Khabibullina, name: Miryam de Lhoneux, name: Thomas Demeester
Published: 2024-08-08 08:37:28

The development of monolingual language models for low and mid-resource
languages continues to be hindered by the difficulty in sourcing high-quality
training data. In this study, we present a novel cross-lingual vocabulary
transfer strategy, trans-tokenization, designed to tackle this challenge and
enable more efficient language adaptation. Our approach focuses on adapting a
high-resource monolingual LLM to an unseen target language by initializing the
token embeddings of the target language using a weighted average of
semantically similar token embeddings from the source language. For this, we
leverage a translation resource covering both the source and target languages.
We validate our method with the Tweeties, a series of trans-tokenized LLMs, and
demonstrate their competitive performance on various downstream tasks across a
small but diverse set of languages. Additionally, we introduce Hydra LLMs,
models with multiple swappable language modeling heads and embedding tables,
which further extend the capabilities of our trans-tokenization strategy. By
designing a Hydra LLM based on the multilingual model TowerInstruct, we
developed a state-of-the-art machine translation model for Tatar, in a
zero-shot manner, completely bypassing the need for high-quality parallel data.
This breakthrough is particularly significant for low-resource languages like
Tatar, where high-quality parallel data is hard to come by. By lowering the
data and time requirements for training high-quality models, our
trans-tokenization strategy allows for the development of LLMs for a wider
range of languages, especially those with limited resources. We hope that our
work will inspire further research and collaboration in the field of
cross-lingual vocabulary transfer and contribute to the empowerment of
languages on a global scale.
Transformer Explainer: Interactive Learning of Text-Generative Models
Authors: name: Aeree Cho, name: Grace C. Kim, name: Alexander Karpekov, name: Alec Helbling, name: Zijie J. Wang, name: Seongmin Lee, name: Benjamin Hoover, name: Duen Horng Chau
Published: 2024-08-08 17:49:07

Transformers have revolutionized machine learning, yet their inner workings
remain opaque to many. We present Transformer Explainer, an interactive
visualization tool designed for non-experts to learn about Transformers through
the GPT-2 model. Our tool helps users understand complex Transformer concepts
by integrating a model overview and enabling smooth transitions across
abstraction levels of mathematical operations and model structures. It runs a
live GPT-2 instance locally in the user's browser, empowering users to
experiment with their own input and observe in real-time how the internal
components and parameters of the Transformer work together to predict the next
tokens. Our tool requires no installation or special hardware, broadening the
public's education access to modern generative AI techniques. Our open-sourced
tool is available at https://poloclub.github.io/transformer-explainer/. A video
demo is available at https://youtu.be/ECR4oAwocjs.
Img-Diff: Contrastive Data Synthesis for Multimodal Large Language
  Models
Authors: name: Qirui Jiao, name: Daoyuan Chen, name: Yilun Huang, name: Yaliang Li, name: Ying Shen
Published: 2024-08-08 17:10:16

High-performance Multimodal Large Language Models (MLLMs) rely heavily on
data quality. This study introduces a novel dataset named Img-Diff, designed to
enhance fine-grained image recognition in MLLMs by leveraging insights from
contrastive learning and image difference captioning. By analyzing object
differences between similar images, we challenge models to identify both
matching and distinct components. We utilize the Stable-Diffusion-XL model and
advanced image editing techniques to create pairs of similar images that
highlight object replacements. Our methodology includes a Difference Area
Generator for object differences identifying, followed by a Difference Captions
Generator for detailed difference descriptions. The result is a relatively
small but high-quality dataset of "object replacement" samples. We use the the
proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B,
yielding comprehensive improvements of performance scores over SOTA models that
trained with larger-scale datasets, in numerous image difference and Visual
Question Answering tasks. For instance, our trained models notably surpass the
SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate
alternative methods for generating image difference data through "object
removal" and conduct thorough evaluation to confirm the dataset's diversity,
quality, and robustness, presenting several insights on synthesis of such
contrastive dataset. To encourage further research and advance the field of
multimodal data synthesis and enhancement of MLLMs' fundamental capabilities
for image understanding, we release our codes and dataset at
https://github.com/modelscope/data-juicer/tree/ImgDiff.
LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection
Authors: name: Mervat Abassy, name: Kareem Elozeiri, name: Alexander Aziz, name: Minh Ngoc Ta, name: Raj Vardhan Tomar, name: Bimarsha Adhikari, name: Saad El Dine Ahmed, name: Yuxia Wang, name: Osama Mohammed Afzal, name: Zhuohan Xie, name: Jonibek Mansurov, name: Ekaterina Artemova, name: Vladislav Mikhailov, name: Rui Xing, name: Jiahui Geng, name: Hasan Iqbal, name: Zain Muhammad Mujahid, name: Tarek Mahmoud, name: Akim Tsvigun, name: Alham Fikri Aji, name: Artem Shelmanov, name: Nizar Habash, name: Iryna Gurevych, name: Preslav Nakov
Published: 2024-08-08 07:43:17

The widespread accessibility of large language models (LLMs) to the general
public has significantly amplified the dissemination of machine-generated texts
(MGTs). Advancements in prompt manipulation have exacerbated the difficulty in
discerning the origin of a text (human-authored vs machinegenerated). This
raises concerns regarding the potential misuse of MGTs, particularly within
educational and academic domains. In this paper, we present
LLM-DetectAIve -- a system designed for fine-grained MGT detection.
It is able to classify texts into four categories: human-written,
machine-generated, machine-written machine-humanized, and human-written
machine-polished. Contrary to previous MGT detectors that perform binary
classification, introducing two additional categories in LLM-DetectiAIve offers
insights into the varying degrees of LLM intervention during the text creation.
This might be useful in some domains like education, where any LLM intervention
is usually prohibited. Experiments show that LLM-DetectAIve can effectively
identify the authorship of textual content, proving its usefulness in enhancing
integrity in education, academia, and other domains. LLM-DetectAIve is publicly
accessible at https://huggingface.co/spaces/raj-tomar001/MGT-New. The video
describing our system is available at https://youtu.be/E8eT_bE7k8c.
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from
  User's Casual Sketches
Authors: name: Yongzhi Xu, name: Yonhon Ng, name: Yifu Wang, name: Inkyu Sa, name: Yunfei Duan, name: Yang Li, name: Pan Ji, name: Hongdong Li
Published: 2024-08-08 16:27:37

3D Content Generation is at the heart of many computer graphics applications,
including video gaming, film-making, virtual and augmented reality, etc. This
paper proposes a novel deep-learning based approach for automatically
generating interactive and playable 3D game scenes, all from the user's casual
prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and
convenient way to convey the user's design intention in the content creation
process. To circumvent the data-deficient challenge in learning (i.e. the lack
of large training data of 3D scenes), our method leverages a pre-trained 2D
denoising diffusion model to generate a 2D image of the scene as the conceptual
guidance. In this process, we adopt the isometric projection mode to factor out
unknown camera poses while obtaining the scene layout. From the generated
isometric image, we use a pre-trained image understanding method to segment the
image into meaningful parts, such as off-ground objects, trees, and buildings,
and extract the 2D scene layout. These segments and layouts are subsequently
fed into a procedural content generation (PCG) engine, such as a 3D video game
engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can
be seamlessly integrated into a game development environment and is readily
playable. Extensive tests demonstrate that our method can efficiently generate
high-quality and interactive 3D game scenes with layouts that closely follow
the user's intention.
Better Alignment with Instruction Back-and-Forth Translation
Authors: name: Thao Nguyen, name: Jeffrey Li, name: Sewoong Oh, name: Ludwig Schmidt, name: Jason Weston, name: Luke Zettlemoyer, name: Xian Li
Published: 2024-08-08 17:42:32

We propose a new method, instruction back-and-forth translation, to construct
high-quality synthetic data grounded in world knowledge for aligning large
language models (LLMs). Given documents from a web corpus, we generate and
curate synthetic instructions using the backtranslation approach proposed by Li
et al.(2023a), and rewrite the responses to improve their quality further based
on the initial documents. Fine-tuning with the resulting (backtranslated
instruction, rewritten response) pairs yields higher win rates on AlpacaEval
than using other common instruction datasets such as Humpback, ShareGPT, Open
Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the
responses with an LLM outperforms direct distillation, and the two generated
text distributions exhibit significant distinction in embedding space. Further
analysis shows that our backtranslated instructions are of higher quality than
other sources of synthetic instructions, while our responses are more diverse
and complex than those obtained from distillation. Overall we find that
instruction back-and-forth translation combines the best of both worlds --
making use of the information diversity and quantity found on the web, while
ensuring the quality of the responses which is necessary for effective
alignment.
GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards
  General Medical AI
Authors: name: Pengcheng Chen, name: Jin Ye, name: Guoan Wang, name: Yanjun Li, name: Zhongying Deng, name: Wei Li, name: Tianbin Li, name: Haodong Duan, name: Ziyan Huang, name: Yanzhou Su, name: Benyou Wang, name: Shaoting Zhang, name: Bin Fu, name: Jianfei Cai, name: Bohan Zhuang, name: Eric J Seibel, name: Junjun He, name: Yu Qiao
Published: 2024-08-06 17:59:21

Large Vision-Language Models (LVLMs) are capable of handling diverse data
types such as imaging, text, and physiological signals, and can be applied in
various fields. In the medical field, LVLMs have a high potential to offer
substantial assistance for diagnosis and treatment. Before that, it is crucial
to develop benchmarks to evaluate LVLMs' effectiveness in various medical
applications. Current benchmarks are often built upon specific academic
literature, mainly focusing on a single domain, and lacking varying perceptual
granularities. Thus, they face specific challenges, including limited clinical
relevance, incomplete evaluations, and insufficient guidance for interactive
LVLMs. To address these limitations, we developed the GMAI-MMBench, the most
comprehensive general medical AI benchmark with well-categorized data structure
and multi-perceptual granularity to date. It is constructed from 285 datasets
across 39 medical image modalities, 18 clinical-related tasks, 18 departments,
and 4 perceptual granularities in a Visual Question Answering (VQA) format.
Additionally, we implemented a lexical tree structure that allows users to
customize evaluation tasks, accommodating various assessment needs and
substantially supporting medical AI research and applications. We evaluated 50
LVLMs, and the results show that even the advanced GPT-4o only achieves an
accuracy of 52%, indicating significant room for improvement. Moreover, we
identified five key insufficiencies in current cutting-edge LVLMs that need to
be addressed to advance the development of better medical applications. We
believe that GMAI-MMBench will stimulate the community to build the next
generation of LVLMs toward GMAI.
  Project Page: https://uni-medical.github.io/GMAI-MMBench.github.io/
Task-oriented Sequential Grounding in 3D Scenes
Authors: name: Zhuofan Zhang, name: Ziyu Zhu, name: Pengxiang Li, name: Tengyu Liu, name: Xiaojian Ma, name: Yixin Chen, name: Baoxiong Jia, name: Siyuan Huang, name: Qing Li
Published: 2024-08-07 18:30:18

Grounding natural language in physical 3D environments is essential for the
advancement of embodied artificial intelligence. Current datasets and models
for 3D visual grounding predominantly focus on identifying and localizing
objects from static, object-centric descriptions. These approaches do not
adequately address the dynamic and sequential nature of task-oriented grounding
necessary for practical applications. In this work, we propose a new task:
Task-oriented Sequential Grounding in 3D scenes, wherein an agent must follow
detailed step-by-step instructions to complete daily activities by locating a
sequence of target objects in indoor scenes. To facilitate this task, we
introduce SG3D, a large-scale dataset containing 22,346 tasks with 112,236
steps across 4,895 real-world 3D scenes. The dataset is constructed using a
combination of RGB-D scans from various 3D scene datasets and an automated task
generation pipeline, followed by human verification for quality assurance. We
adapted three state-of-the-art 3D visual grounding models to the sequential
grounding task and evaluated their performance on SG3D. Our results reveal that
while these models perform well on traditional benchmarks, they face
significant challenges with task-oriented sequential grounding, underscoring
the need for further research in this area.
Puppet-Master: Scaling Interactive Video Generation as a Motion Prior
  for Part-Level Dynamics
Authors: name: Ruining Li, name: Chuanxia Zheng, name: Christian Rupprecht, name: Andrea Vedaldi
Published: 2024-08-08 17:59:38

We present Puppet-Master, an interactive video generative model that can
serve as a motion prior for part-level dynamics. At test time, given a single
image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can
synthesize a video depicting realistic part-level motion faithful to the given
drag interactions. This is achieved by fine-tuning a large-scale pre-trained
video diffusion model, for which we propose a new conditioning architecture to
inject the dragging control effectively. More importantly, we introduce the
all-to-first attention mechanism, a drop-in replacement for the widely adopted
spatial attention modules, which significantly improves generation quality by
addressing the appearance and background issues in existing models. Unlike
other motion-conditioned video generators that are trained on in-the-wild
videos and mostly move an entire object, Puppet-Master is learned from
Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We
propose a strategy to automatically filter out sub-optimal animations and
augment the synthetic renderings with meaningful motion trajectories.
Puppet-Master generalizes well to real images across various categories and
outperforms existing methods in a zero-shot manner on a real-world benchmark.
See our project page for more results: vgg-puppetmaster.github.io.
WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language
  Models
Authors: name: Prannaya Gupta, name: Le Qi Yau, name: Hao Han Low, name: I-Shiang Lee, name: Hugo Maximus Lim, name: Yu Xin Teoh, name: Jia Hng Koh, name: Dar Win Liew, name: Rishabh Bhardwaj, name: Rajat Bhardwaj, name: Soujanya Poria
Published: 2024-08-07 15:22:44

WalledEval is a comprehensive AI safety testing toolkit designed to evaluate
large language models (LLMs). It accommodates a diverse range of models,
including both open-weight and API-based ones, and features over 35 safety
benchmarks covering areas such as multilingual safety, exaggerated safety, and
prompt injections. The framework supports both LLM and judge benchmarking, and
incorporates custom mutators to test safety against various text-style
mutations such as future tense and paraphrasing. Additionally, WalledEval
introduces WalledGuard, a new, small and performant content moderation tool,
and SGXSTest, a benchmark for assessing exaggerated safety in cultural
contexts. We make WalledEval publicly available at
https://github.com/walledai/walledevalA.
Facing the Music: Tackling Singing Voice Separation in Cinematic Audio
  Source Separation
Authors: name: Karn N. Watcharasupat, name: Chih-Wei Wu, name: Iroro Orife
Published: 2024-08-07 07:04:29

Cinematic audio source separation (CASS) is a fairly new subtask of audio
source separation. A typical setup of CASS is a three-stem problem, with the
aim of separating the mixture into the dialogue stem (DX), music stem (MX), and
effects stem (FX). In practice, however, several edge cases exist as some sound
sources do not fit neatly in either of these three stems, necessitating the use
of additional auxiliary stems in production. One very common edge case is the
singing voice in film audio, which may belong in either the DX or MX, depending
heavily on the cinematic context. In this work, we demonstrate a very
straightforward extension of the dedicated-decoder Bandit and query-based
single-decoder Banquet models to a four-stem problem, treating non-musical
dialogue, instrumental music, singing voice, and effects as separate stems.
Interestingly, the query-based Banquet model outperformed the dedicated-decoder
Bandit model. We hypothesized that this is due to a better feature alignment at
the bottleneck as enforced by the band-agnostic FiLM layer. Dataset and model
implementation will be made available at
https://github.com/kwatcharasupat/source-separation-landing.
Fast Sprite Decomposition from Animated Graphics
Authors: name: Tomoyuki Suzuki, name: Kotaro Kikuchi, name: Kota Yamaguchi
Published: 2024-08-07 17:30:59

This paper presents an approach to decomposing animated graphics into
sprites, a set of basic elements or layers. Our approach builds on the
optimization of sprite parameters to fit the raster video. For efficiency, we
assume static textures for sprites to reduce the search space while preventing
artifacts using a texture prior model. To further speed up the optimization, we
introduce the initialization of the sprite parameters utilizing a pre-trained
video object segmentation model and user input of single frame annotations. For
our study, we construct the Crello Animation dataset from an online design
service and define quantitative metrics to measure the quality of the extracted
sprites. Experiments show that our method significantly outperforms baselines
for similar decomposition tasks in terms of the quality/efficiency tradeoff.
CodexGraph: Bridging Large Language Models and Code Repositories via
  Code Graph Databases
Authors: name: Xiangyan Liu, name: Bo Lan, name: Zhiyuan Hu, name: Yang Liu, name: Zhicheng Zhang, name: Wenmeng Zhou, name: Fei Wang, name: Michael Shieh
Published: 2024-08-07 17:13:59

Large Language Models (LLMs) excel in stand-alone code tasks like HumanEval
and MBPP, but struggle with handling entire code repositories. This challenge
has prompted research on enhancing LLM-codebase interaction at a repository
scale. Current solutions rely on similarity-based retrieval or manual tools and
APIs, each with notable drawbacks. Similarity-based retrieval often has low
recall in complex tasks, while manual tools and APIs are typically
task-specific and require expert knowledge, reducing their generalizability
across diverse code tasks and real-world applications. To mitigate these
limitations, we introduce \framework, a system that integrates LLM agents with
graph database interfaces extracted from code repositories. By leveraging the
structural properties of graph databases and the flexibility of the graph query
language, \framework enables the LLM agent to construct and execute queries,
allowing for precise, code structure-aware context retrieval and code
navigation. We assess \framework using three benchmarks: CrossCodeEval,
SWE-bench, and EvoCodeBench. Additionally, we develop five real-world coding
applications. With a unified graph database schema, \framework demonstrates
competitive performance and potential in both academic and real-world
environments, showcasing its versatility and efficacy in software engineering.
Our application demo:
https://github.com/modelscope/modelscope-agent/tree/master/apps/codexgraph_agent.