bluuebunny/crossref_metadata_2025 huggingface dataset:
Long-term Detection and Monitory of Chinese Urban Village Using Satellite Imagery
Authors: Lin Yuming , Zhang Xin , Liu Yu , Han Zhenyu , Liao Qingmin , Li Yong
Published: July 2024

Urban villages are areas filled with rural-like improvised structures in Chinese cities, usually housing the most vulnerable groups.

Under the guidance of the Sustainable Development Goals (SDGs), the Chinese government initiated renewal and redevelopment projects, underscoring the meticulous mapping and segmentation of urban villages.

Satellite imagery is advanced and efficient in identifying urban villages and monitoring changes, but traditional methods neglect the morphological diversity in season, shape, size, spacing, and layout of urban villages, which is not satisfying for long-term wide-range data.

Here, we design a targeted approach based on Tobler’s First Law of Geography, using curriculum labeling to solve morphological diversity and semi-automatically generate segmentation for urban village boundaries.

Specifically, we use manually labeled data as seeds for pre-trained SegFormer models and incrementally fine-tune the model based on geographical proximity.

The rigorous experimentation across five diverse cities substantiates the commendable efficacy of our methodology. IoU metric demonstrates a noteworthy improvement of over 119% to baseline.

Our final results cover 265,050 urban villages across 433 cities in China over the past 10 years, and the analysis reveals the uneven redevelopment by geography and city scale. We further examine the within-city distribution and verify the urban scaling law associated with several socio-economic factors.

Our method can be used nationwide to decide redevelopment priority and resource tilt, contributing to SDG 11.1 on affordable housing and upgrading slums. The code and dataset are available at https://github.com/tsinghua-fib-lab/LtCUV.
Benchmarking Fish Dataset and Evaluation Metric in Keypoint Detection - Towards Precise Fish Morphological Assessment in Aquaculture Breeding
Authors: Liu Weizhen , Tan Jiayu , Lan Guangyu , Li Ao , Li Dongye , Zhao Le , Yuan Xiaohui , Dong Nanqing
Published: July 2024

Accurate phenotypic analysis in aquaculture breeding necessitates the quantification of subtle morphological phenotypes. Existing datasets suffer from limitations such as small scale, limited species coverage, and inadequate annotation of keypoints for measuring refined and complex morphological phenotypes of fish body parts. To address this gap, we introduce FishPhenoKey, a comprehensive dataset comprising 23,331 high-resolution images spanning six fish species. Notably, FishPhenoKey includes 22 phenotype-oriented annotations, enabling the capture of intricate morphological phenotypes. Motivated by the nuanced evaluation of these subtle morphologies, we also propose a new evaluation metric, Percentage of Measured Phenotypes (PMP). It is designed to assess the accuracy of individual keypoint positions and is highly sensitive to the phenotype measured using the corresponding keypoints. To enhance keypoint detection accuracy, we further propose a novel loss, Anatomically-Calibrated Regularization (ACR), that can be integrated into keypoint detection models, leveraging biological insights to refine keypoint localization. Our contributions set a new benchmark in fish phenotype analysis, addressing the challenges of precise morphological quantification and opening new avenues for research in sustainable aquaculture and genetic studies. Our dataset and code are available at https://github.com/WeizhenLiuBioinform/FishPhenotype-Detect.
RisQNet: Rescuing SMEs from Financial Shocks with a Novel Networked-Loan Risk Assessment
Authors: Lu Zhaoyuan , Li Taijun , Zhang Jingzhen , Liu Moyang , Li Xiang , Cui Linyi , Chen Junqi , Niu Zhibin
Published: July 2024

In the face of economic downturns, Small and Medium-sized Enterprises (SMEs) within interconnected networked-loans are vulnerable to cascading debt crises, exacerbated by factors like social media-induced financial shocks. Traditional risk assessment models, which mainly rely on financial data, inadequately predict such crises, as evidenced by the collapse of Silicon Valley Bank in 2023. To address this issue, we developed RisQNet, a model that uses temporal graph networks to incorporate diverse risks, including real-time media influences. This approach not only advances risk prediction through news feature extraction and large language models but also enhances risk management strategies with intuitive visualization tools. Validated on a dataset with a total loan volume of USD 3 trillion, RisQNet outperforms the state-of-the-art baseline and achieves 87.1% of AUC. Our collaborative effort with financial regulators and the SME community underpins the model's development, aligning with the UN SDG 8. RisQNet represents a significant step forward in leveraging AI for financial stability, offering a promising approach to combat the propagation of debt crises in financial networks.
Understanding Public Perception Towards Weather Disasters Through the Lens of Metaphor
Authors: Mao Rui , Lin Qika , Liu Qiawen , Mengaldo Gianmarco , Cambria Erik
Published: July 2024

Extreme weather can lead to weather-induced disasters. These have a profound impact on communities worldwide, causing loss of life, damage to properties and infrastructure, and disruption of daily activities. In alignment with the United Nations Sustainable Development Goals, addressing the increasing frequency and severity of these events, exacerbated by climate change, is imperative. Exploring public perception and responses to weather disasters becomes crucial for policymakers to formulate effective strategies that not only mitigate the impacts but also contribute to the goal of ensuring sustainable and resilient communities. Social media, as a pervasive and real-time communication platform, has gathered a large amount of public opinion. In this work, we analyze public perception towards weather disasters based on tweets and metaphors. Metaphor, as a linguistic device, plays a pivotal role in unraveling cognitive processes and understanding how individuals perceive and make sense of concepts. We focus on tweets related to four distinct types of weather disasters i.e., floods, hurricanes, tornadoes, and wildfires, aiming to extract nuanced insights regarding public perceptions, concerns, and attitudes towards these specific events. We also deliver constructive recommendations, based on the insights.
Ensuring Fairness Stability for Disentangling Social Inequality in Access to Education: the FAiRDAS General Method
Authors: Misino Eleonora , Calegari Roberta , Lombardi Michele , Milano Michela
Published: July 2024

Recent advancements in Artificial Intelligence in Education (AIEd) have revolutionized educational practices using machine learning to extract insights from students' activities and behaviours. Performance prediction, a key domain within AIEd, aims to enhance student achievement levels and address sustainable development goals related to education, health, gender equality, and economic growth. However, the potential of AIEd to contribute to these goals is hindered by the lack of attention to fairness in prediction algorithms, leading to educational inequality. To address this gap, we introduce FAiRDAS a general framework that models long-term fairness as an abstract dynamic system. Our approach, illustrated through a case study in AIEd with real data, offers a customizable solution to promote long-term fairness while promoting the stability of mitigation actions over time.
Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation
Authors: Rafiq Riyad Bin , Shi Weishi , Albert Mark V.
Published: July 2024

Hand gestures can provide a natural means of human-computer interaction and enable people who cannot speak to communicate efficiently. Existing hand gesture recognition methods heavily depend on pre-defined gestures, however, motor-impaired individuals require new gestures tailored to each individual's gesture motion and style. Gesture samples collected from different persons have distribution shifts due to their health conditions, the severity of the disability, motion patterns of the arms, etc. In this paper, we introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning (FSCL) framework that significantly improves the performance of fine-tuning a model for out-of-distribution data. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge, along with intra-gesture divergence derived from two additional embeddings. Thus, the model can capture latent statistical structure in highly variable gestures with limited samples. We conduct an experimental evaluation using the SmartWatch Gesture and the Motion Gesture datasets. The proposed method results in an average test accuracy of 57.0%, 64.6%, and 69.3% by using one, three, and five samples for six different gestures. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied in human-computer interaction and social communication. Code is available at: https://github.com/riyadRafiq/wearable-latent-embedding-exploitation.
From Pink and Blue to a Rainbow Hue! Defying Gender Bias through Gender Neutralizing Text Transformations
Authors: Singh Gopendra Vikram , Ghosh Soumitra , Dcruze Neil , Ekbal Asif
Published: July 2024

In an era where language biases contribute to societal inequalities, this research focuses on gender bias in textual data, with profound implications for promoting inclusivity and equity, aligning with United Nations Sustainable Development Goals (SDGs) and upholding the principle of Leave No One Behind (LNOB). Leveraging advances in artificial intelligence, the study introduces the GEnder-NEutralizing Text Transformation (GENETT) framework, addressing gender bias in text through auto-encoders, vector quantization, and Neutrality-Infused Stylization. Furthermore, we present the first-of-its-kind corpus of GEnder Neutralized REvisions (GENRE) crafted from gender-stereotyped versions. This corpus serves a multifaceted utility, offering a resource for diverse downstream tasks in gender-bias analysis. Extensive experimentation on GENRE highlights the superiority of the proposed model over established baselines and state-of-the-art methods. Access the code and dataset at 1. https://www.iitp.ac.in/~ai-nlp-ml/resources.html#GNR, 2. https://github.com/Soumitra816/GNR.

Note: Our research focuses on understanding cyber harassment conversations, especially in under-researched areas, with the exclusion of non-binary cases due to existing dataset limitations, not lack of sensitivity. We strive for inclusivity and plan to address this in future research with suitable datasets.
Fuel-Saving Route Planning with Data-Driven and Learning-Based Approaches – A Systematic Solution for Harbor Tugs
Authors: Wang Shengming , Zhang Xiaocai , Li Jing , Wei Xiaoyang , Lau Hoong Chuin , Dai Bing Tian , Huang Binbin , Xiao Zhe , Fu Xiuju , Qin Zheng
Published: July 2024

In recent years, there are trends toward cleaner port environments through enforcement by imposed legislation. Transit optimisation of fuel-based port service boats like harbour tugs has emerged as a critical task to reduce fuel consumption and carbon emission. In this paper, an innovative learning-based method, comprising a Reinforcement Learning (RL) model together with a fuel consumption prediction model, was proposed to formulate fuel-saving transit routes. Firstly, an ensemble model is established by combining a Long Short-Term Memory (LSTM) model with a Multilayer Perceptron (MLP) model, predicting fuel use based on tugboat movement and environment factors. Subsequently, an innovative RL based on Deep Deterministic Policy Gradient (DDPG) framework is developed considering the characteristics and obstructions of waterway in Singapore as well as the environmental factors to learn the optimal transit strategy that minimizes fuel consumption. We also demonstrate the efficacy of the solution to generate routes from origin to destination terminals, exhibiting significantly reduced fuel consumption in comparison to real-world transit scenarios.
Safeguarding Fraud Detection from Attacks: A Robust Graph Learning Approach
Authors: Wu Jiasheng , Liu Xin , Cheng Dawei , Ouyang Yi , Wu Xian , Zheng Yefeng
Published: July 2024

Financial fraud is one of the most significant social issues and has caused tremendous property losses. Graph neural networks (GNNs) have been applied to anti-fraud practices and achieved decent results. However, recent researches have discovered flaws in the robustness of fraud-detection models based on GNNs, enabling fraudsters to mislead them through attacks like data poisoning. In addition, most existing attack-defense models tend to study on ideal settings and lose information during truncation or filtering, which lowers their performances in complicated financial fraud cases. Therefore, in this paper, we propose a novel robust anti-fraud GNN model. In particular, we first design an attack algorithm tampering with both features and structures of graph data to simulate fraudsters' attacking behaviors in real-life complex fraud scenarios. Then we apply singular value decomposition to the graph and learn the decomposed matrices in a GNN model with specifically designed joint losses. This enables our model to learn the graph patterns in low-rank subspaces without losing too much detailed information and fit the graph structure to characteristics including class-homophily and sparsity to guarantee robustness. The proposed approach is experimented on real-world fraud datasets, which demonstrates its advantages in fraud detection and robustness compared with the state-of-the-art baselines.
CGAP: Urban Region Representation Learning with Coarsened Graph Attention Pooling
Authors: Xu Zhuo , Zhou Xiao
Published: July 2024

The explosion of massive urban data recently has provided us with a valuable opportunity to gain deeper insights into urban regions and the daily lives of residents. Urban region representation learning emerges as a crucial realm for fulfilling this task. Among deep learning approaches, graph neural networks (GNNs) have shown promise, given that city elements can be naturally represented as nodes with various connections between them as edges. However, many existing GNN approaches encounter challenges such as over-smoothing and limitations in capturing information from nodes in other regions, resulting in the loss of crucial urban information and a decline in region representation performance. To address these challenges, we leverage urban graph structure information and introduce a hierarchical graph pooling process called Coarsened Graph Attention Pooling (CGAP). CGAP features local attention units to create coarsened intermediate graphs and global features. Additionally, by incorporating urban region graphs and global features into a global attention layer, we harness relational information to enhance representation effectiveness. Furthermore, CGAP integrates region attributes such as Points of Interest (POIs) and inter-regional contexts like human mobility, enabling the exploitation of multi-modal urban data for more comprehensive representation learning. Experiments on three downstream tasks related to the UN Sustainable Development Goals validate the effectiveness of region representations learned by our approach. Experimental results and analyses demonstrate that CGAP excels in various socioeconomic prediction tasks compared to competitive baselines.
LEEC for Judicial Fairness: A Legal Element Extraction Dataset with Extensive Extra-Legal Labels
Authors: Xue Zongyue , Liu Huanghai , Hu Yiran , Qian Yuliang , Wang Yajing , Kong Kangle , Wang Chenlu , Liu Yun , Shen Weixing
Published: July 2024

An extensive label system is pivotal to facilitate judicial fairness and social justice. Prior empirical research and our interview with legal professionals underscore the importance of extra-legal factors in criminal trials. To help identify sentencing biases and facilitate downstream applications, we introduce the Legal Element ExtraCtion (LEEC) dataset comprising 15,919 judicial documents and 155 labels. This dataset was constructed through two main steps: First, designing the label system by legal experts based on prior empirical research which identified critical factors driving and processes generating sentencing outcomes in criminal cases; Second, employing legal knowledge to annotate judicial documents according to the label system and annotation guideline. LEEC represents the most extensive and domain-specific legal element extraction dataset for the Chinese legal system. Our experiments reveal that despite certain capabilities, both Document Event Extraction (DEE) models and Large Language Models(LLMs) face significant restrictions in legal element extraction tasks. Finally, our empirical analysis based on LEEC provides evidence for judicial unfairness in Chinese criminal sentencing and confirms the applicability of LEEC for future empirical research and other downstream applications. LEEC and related resources are available on https://github.com/THUlawtech/LEEC.
Predicting Carpark Availability in Singapore with Cross-Domain Data: A New Dataset and A Data-Driven Approach
Authors: Zhang Huaiwu , Xia Yutong , Zhong Siru , Wang Kun , Tong Zekun , Wen Qingsong , Zimmermann Roger , Liang Yuxuan
Published: July 2024

The increasing number of vehicles highlights the need for efficient parking space management. Predicting real-time Parking Availability (PA) can help mitigate traffic congestion and the corresponding social problems, which is a pressing issue in densely populated cities like Singapore. In this study, we aim to collectively predict future PA across Singapore with complex factors from various domains. The contributions in this paper are listed as follows: (1) A New Dataset: We introduce the SINPA dataset, containing a year's worth of PA data from 1,687 parking lots in Singapore, enriched with various spatial and temporal factors. (2) A Data-Driven Approach: We present DeepPA, a novel deep-learning framework, to collectively and efficiently predict future PA across thousands of parking lots. (3) Extensive Experiments and Deployment: DeepPA demonstrates a 9.2% reduction in prediction error for up to 3-hour forecasts compared to existing advanced models. Furthermore, we implement DeepPA in a practical web-based platform to provide real-time PA predictions to aid drivers and inform urban planning for the governors in Singapore. We release the dataset and source code at https://github.com/yoshall/SINPA.
Safeguarding Sustainable Cities: Unsupervised Video Anomaly Detection through Diffusion-based Latent Pattern Learning
Authors: Zhang Menghao , Wang Jingyu , Qi Qi , Ren Pengfei , Sun Haifeng , Zhuang Zirui , Zhang Lei , Liao Jianxin
Published: July 2024

Sustainable cities requires high-quality community management and surveillance analytics, which are supported by video anomaly detection techniques. However, mainstream video anomaly detection techniques still require manually labeled data and do not apply to real-world massive videos. Without labeling, unsupervised video anomaly detection (UVAD) is challenged by the problem of pseudo-labeled noise and the openness of anomaly detection. In response, a diffusion-based latent pattern learning UVAD framework is proposed, called DiffVAD. The method learns potential patterns by generating different patterns of the same event through diffusion models. The detection of anomalies is realized by evaluating the pattern distribution. The different patterns of normal events are diverse but correlated, while the different patterns of abnormal events are more diffuse. This manner of detection is equally effective for unseen normal events in the training set. In addition, we design a refinement strategy for pseudo-labels to mitigate the effects of the noise problem. Extensive experiments on six benchmark datasets demonstrate the design’s promising generalization ability and high efficiency. Specifically, DiffVAD obtains an AUC score of 81.9% on the ShanghaiTech dataset.
Integrating View Conditions for Image Synthesis
Authors: Bai Jinbin , Dong Zhen , Feng Aosong , Zhang Xiao , Ye Tian , Zhou Kaicheng
Published: July 2024

In the field of image processing, applying intricate semantic modifications within existing images remains an enduring challenge. This paper introduces a pioneering framework that integrates viewpoint information to enhance the control of image editing tasks, especially for interior design scenes. By surveying existing object editing methodologies, we distill three essential criteria --- consistency, controllability, and harmony --- that should be met for an image editing method. In contrast to previous approaches, our framework takes the lead in satisfying all three requirements for addressing the challenge of image synthesis. Through comprehensive experiments, encompassing both quantitative assessments and qualitative comparisons with contemporary state-of-the-art methods, we present compelling evidence of our framework's superior performance across multiple dimensions. This work establishes a promising avenue for advancing image synthesis techniques and empowering precise object modifications while preserving the visual coherence of the entire composition.
Re:Draw - Context Aware Translation as a Controllable Method for Artistic Production
Authors: Cardoso João Libório , Banterle Francesco , Cignoni Paolo , Wimmer Michael
Published: July 2024

We introduce context-aware translation, a novel method that combines the benefits of inpainting and image-to-image translation, respecting simultaneously the original input and contextual relevance -- where existing methods fall short. By doing so, our method opens new avenues for the controllable use of AI within artistic creation, from animation to digital art.

As an use case, we apply our method to redraw any hand-drawn animated character eyes based on any design specifications -- eyes serve as a focal point that captures viewer attention and conveys a range of emotions; however, the labor-intensive nature of traditional animation often leads to compromises in the complexity and consistency of eye design. Furthermore, we remove the need for production data for training and introduce a new character recognition method that surpasses existing work by not requiring fine-tuning to specific productions. This proposed use case could help maintain consistency throughout production and unlock bolder and more detailed design choices without the production cost drawbacks. A user study shows context-aware translation is preferred over existing work 95.16% of the time.
FastSAG: Towards Fast Non-Autoregressive Singing Accompaniment Generation
Authors: Chen Jianyi , Xue Wei , Tan Xu , Ye Zhen , Liu Qifeng , Guo Yike
Published: July 2024

Singing Accompaniment Generation (SAG), which generates instrumental music to accompany input vocals, is crucial to developing human-AI symbiotic art creation systems. The state-of-the-art method, SingSong, utilizes a multi-stage autoregressive (AR) model for SAG, however, this method is extremely slow as it generates semantic and acoustic tokens recursively, and this makes it impossible for real-time applications. In this paper, we aim to develop a Fast SAG method that can create high-quality and coherent accompaniments. A non-AR diffusion-based framework is developed, which by carefully designing the conditions inferred from the vocal signals, generates the Mel spectrogram of the target accompaniment directly. With diffusion and Mel spectrogram modeling, the proposed method significantly simplifies the AR token-based SingSong framework, and largely accelerates the generation. We also design semantic projection, prior projection blocks as well as a set of loss functions, to ensure the generated accompaniment has semantic and rhythm coherence with the vocal signal. By intensive experimental studies, we demonstrate that the proposed method can generate better samples than SingSong, and accelerate the generation by at least 30 times. Audio samples and code are available at this link.
GEM: Generating Engaging Multimodal Content
Authors: Gao Chongyang , Jian Yiren , Denisenko Natalia , Vosoughi Soroush , Subrahmanian V. S.
Published: July 2024

Generating engaging multimodal content is a key objective in numerous applications, such as the creation of online advertisements that captivate user attention through a synergy of images and text. In this paper, we introduce GEM, a novel framework engineered for the generation of engaging multimodal image-text posts. The GEM framework operates in two primary phases. Initially, GEM integrates a pre-trained engagement discriminator with a technique for deriving an effective continuous prompt tailored for the stable diffusion model. Subsequently, GEM unveils an iterative algorithm dedicated to producing coherent and compelling image-sentence pairs centered around a specified topic of interest. Through a combination of experimental analysis and human evaluations, we establish that the image-sentence pairs generated by GEM not only surpass several established baselines in terms of engagement but also in achieving superior alignment.
KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph
Authors: Jiang Yanbei , Ehinger Krista A. , Lau Jey Han
Published: July 2024

Exploring the narratives conveyed by fine-art paintings is a challenge in image captioning, where the goal is to generate descriptions that not only precisely represent the visual content but also offer a in-depth interpretation of the artwork's meaning. The task is particularly complex for artwork images due to their diverse interpretations and varied aesthetic principles across different artistic schools and styles. In response to this, we present KALE (Knowledge-Augmented vision-Language model for artwork Elaborations), a novel approach that enhances existing vision-language models by integrating artwork metadata as additional knowledge. KALE incorporates the metadata in two ways: firstly as direct textual input, and secondly through a multimodal heterogeneous knowledge graph. To optimize the learning of graph representations, we introduce a new cross-modal alignment loss that maximizes the similarity between the image and its corresponding metadata. Experimental results demonstrate that KALE achieves strong performance (when evaluated with CIDEr, in particular) over existing state-of-the-art work across several artwork datasets. Source code of the project is available at https://github.com/Yanbei-Jiang/Artwork-Interpretation.
Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories
Authors: Jin Xin , Qiao Qianqian , Lu Yi , Wang Huaye , Gao Shan , Huang Heng , Li Guangdong
Published: July 2024

Image aesthetic evaluation is a highly prominent research domain in the field of computer vision. In recent years, there has been a proliferation of datasets and corresponding evaluation methodologies for assessing the aesthetic quality of photographic works, leading to the establishment of a relatively mature research environment. However, in contrast to the extensive research in photographic aesthetics, the field of aesthetic evaluation for paintings and drawings has seen limited attention until the introduction of the BAID dataset in March 2023. This dataset solely comprises overall scores for high-quality artistic images.

Our research marks the pioneering introduction of a multi-attribute, multi-category dataset specifically tailored to the field of painting: Aesthetics of Paintings and Drawings Dataset (APDD).

The construction of APDD received active participation from 28 professional artists worldwide, along with dozens of students specializing in the field of art. This dataset encompasses 24 distinct artistic categories and 10 different aesthetic attributes. Each image in APDD has been evaluated by six professionally trained experts in the field of art, including assessments for both total aesthetic scores and aesthetic attribute scores. The final APDD dataset comprises a total of 4985 images, with an annotation count exceeding 31100 entries.

Concurrently, we propose an innovative approach: Art Assessment Network for Specific Painting Styles (AANSPS), designed for the assessment of aesthetic attributes in mixed-attribute art datasets. Through this research, our goal is to catalyze advancements in the field of aesthetic evaluation for paintings and drawings, while enriching the available resources and methodologies for its further development and application. Dataset is available at https://github.com/BestiVictory/APDD.git
Re-creation of Creations: A New Paradigm for Lyric-to-Melody Generation
Authors: Lv Ang , Tan Xu , Qin Tao , Liu Tie-Yan , Yan Rui
Published: July 2024

Current lyric-to-melody generation methods struggle with the lack of paired lyric-melody data to train, and the lack of adherence to composition guidelines, resulting in melodies that do not sound human-composed. To address these issues, we propose a novel paradigm called Re-creation of Creations (ROC) that combines the strengths of both rule-based and neural-based methods. ROC consists of a two-stage generation-retrieval pipeline: the creation and re-creation stages. In the creation stage, we train a melody language model using melody data to generate high-quality music fragments, which are stored in a database indexed by key features. In the re-creation stage, users provide lyrics and a preferred chord progression, and ROC infers melody features for each lyric sentence. By querying the database, we obtain relevant melody fragments that satisfy composition guidelines, and these candidates are filtered, re-ranked, and concatenated based on the guidelines and the melody language model scores. ROC offers two main advantages: it does not require paired lyric-melody data, and it incorporates commonly used composition guidelines, resulting in music that sounds more human-composed with better controllability. Both objective and subjective evaluation results on English and Chinese lyrics show the effectiveness of ROC.
Musical Phrase Segmentation via Grammatical Induction
Authors: Perkins Reed , Ventura Dan
Published: July 2024

We outline a solution to the challenge of musical phrase segmentation that uses grammatical induction algorithms, a class of algorithms which infer a context-free grammar from an input sequence. We analyze the performance of five grammatical induction algorithms on three datasets using various musical viewpoint combinations. Our experiments show that the LONGESTFIRST algorithm achieves the best F1 scores across all three datasets and that input encodings that include the duration viewpoint result in the best performance.
A Conflict-Embedded Narrative Generation Using Commonsense Reasoning
Authors: Song Youngrok , Cho Gunhee , Kim HyunJu , Kim Youngjune , Bae Byung-Chull , Cheong Yun-Gyung
Published: July 2024

Conflict is a critical element in the narrative, inciting dramatic tension. This paper introduces CNGCI (Conflict-driven Narrative Generation through Commonsense Inference), a neuro-symbolic framework designed to generate coherent stories embedded with conflict using commonsense inference. Our framework defines narrative conflict by leveraging the concept of a soft causal threat, where conflict serves as an obstacle that reduces the likelihood of achieving the protagonist's goal by weakening the causal link between context and goal through defeasible inference. Comparative studies against multiple story generation baselines utilizing commonsense reasoning show that our framework outperforms the baselines in creating narratives that distinctly embody conflict while maintaining coherency.
Expressing Musical Ideas with Constraint Programming Using a Model of Tonal Harmony
Authors: Sprockeels Damien , Van Roy Peter
Published: July 2024

The realm of music composition with artificial intelligence stands as a pertinent and evolving field, attracting increasing interest and exploration in contemporary research and practice. This paper presents a constraint-programming based approach to generating four-voice diatonic chord progressions according to established rules of tonal harmony. It uses the strength of constraint programming as a formal logic to rigorously model musical rules and to offer complete control over the set of rules that are enforced. This allows composers to iteratively interact with the model, adding and removing constraints, allowing them to shape the solutions according to their preferences. We define a constraint model of basic tonal harmony, called Diatony. We show that our implementation using the Gecode solver finds optimal solutions in reasonable time and we show how it can be used by a composer to aid in their composition process.
MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Music
Authors: Wang Zihao , Li Shuyu , Zhang Tao , Wang Qi , Yu Pengfei , Luo Jinyang , Liu Yan , Xi Ming , Zhang Kejun
Published: July 2024

The rapidly evolving multimodal Large Language Models (LLMs) urgently require new benchmarks to uniformly evaluate their performance on understanding and textually describing music. However, due to semantic gaps between Music Information Retrieval (MIR) algorithms and human understanding, discrepancies between professionals and the public, and low precision of annotations, existing music description datasets cannot serve as benchmarks. To this end, we present MuChin, the first open-source music description benchmark in Chinese colloquial language, designed to evaluate the performance of multimodal LLMs in understanding and describing music. We established the Caichong Music Annotation Platform (CaiMAP) that employs an innovative multi-person, multi-stage assurance method, and recruited both amateurs and professionals to ensure the precision of annotations and alignment with popular semantics. Utilizing this method, we built a large-scale, private dataset with multi-dimensional, high-precision music annotations, the Caichong Music Dataset (CaiMD), and carefully selected 1,000 high-quality entries to serve as the test set for MuChin. Based on MuChin, we analyzed the discrepancies between professionals and amateurs in terms of music description, and empirically demonstrated the effectiveness of CaiMD for fine-tuning LLMs. Ultimately, we employed MuChin to evaluate existing music understanding models on their ability to provide colloquial descriptions of music.
GladCoder: Stylized QR Code Generation with Grayscale-Aware Denoising Process
Authors: Xie Yuqiu , Jiang Bolin , Li Jiawei , Li Naiqi , Chen Bin , Dai Tao , Peng Yuang , Xia Shu-Tao
Published: July 2024

Traditional QR codes consist of a grid of black-and-white square modules, which lack aesthetic appeal and meaning for human perception. This has motivated recent research to beautify the visual appearance of QR codes. However, there exists a trade-off between the visual quality and scanning-robustness of the image, causing outputs of previous works are simple and of low quality to ensure scanning-robustness. In this paper, we introduce a novel approach GladCoder to generate stylized QR codes that are personalized, natural, and text-driven. Its pipeline includes a Depth-guided Aesthetic QR code Generator (DAG) to improve quality of image foreground, and a GrayscaLe-Aware Denoising (GLAD) process to enhance scanning-robustness. The overall pipeline is based on diffusion models, which allow users to create stylized QR images from a textual prompt to describe the image and a textual input to be encoded. Experiments demonstrate that our method can generate stylized QR code with appealing perception details, while maintaining robust scanning reliability under real world applications.
End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding
Authors: Zeng Wei , He Xian , Wang Ye
Published: July 2024

Piano audio-to-score transcription (A2S) is an important yet underexplored task with extensive applications for music composition, practice, and analysis. However, existing end-to-end piano A2S systems faced difficulties in retrieving bar-level information such as key and time signatures, and have been trained and evaluated with only synthetic data. To address these limitations, we propose a sequence-to-sequence (Seq2Seq) model with a hierarchical decoder that aligns with the hierarchical structure of musical scores, enabling the transcription of score information at both the bar and note levels by multi-task learning. To bridge the gap between synthetic data and recordings of human performance, we propose a two-stage training scheme, which involves pre-training the model using an expressive performance rendering (EPR) system on synthetic audio, followed by fine-tuning the model using recordings of human performance. To preserve the voicing structure for score reconstruction, we propose a pre-processing method for **Kern scores in scenarios with an unconstrained number of voices. Experimental results support the effectiveness of our proposed approaches, in terms of both transcription performance on synthetic audio data in comparison to the current state-of-the-art, and the first experiment on human recordings.
Towards Proactive Interactions for In-Vehicle Conversational Assistants Utilizing Large Language Models
Authors: Du Huifang , Feng Xuejing , Ma Jun , Wang Meng , Tao Shiyu , Zhong Yijie , Li Yuan-Fang , Wang Haofen
Published: July 2024

Research demonstrates that the proactivity of in-vehicle conversational assistants (IVCAs) can help to reduce distractions and enhance driving safety, better meeting users' cognitive needs. However, existing IVCAs struggle with user intent recognition and context awareness, which leads to suboptimal proactive interactions. Large language models (LLMs) have shown potential for generalizing to various tasks with prompts, but their application in IVCAs and exploration of proactive interaction remain under-explored. These raise questions about how LLMs improve proactive interactions for IVCAs and influence user perception. To investigate these questions systematically, we establish a framework with five proactivity levels across two dimensions—assumption and autonomy—for IVCAs. According to the framework, we propose a ``Rewrite + ReAct + Reflect'' strategy, aiming to empower LLMs to fulfill the specific demands of each proactivity level when interacting with users. Both feasibility and subjective experiments are conducted. The LLM outperforms the state-of-the-art model in success rate and achieves satisfactory results for each proactivity level. Subjective experiments with 40 participants validate the effectiveness of our framework and show the proactive level with strong assumptions and user confirmation is most appropriate.
ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments
Authors: Schleibaum Sören , Feng Lu , Kraus Sarit , Müller Jörg P.
Published: July 2024

In the evolving landscape of human-centered AI, fostering a synergistic relationship between humans and AI agents in decision-making processes stands as a paramount challenge. This work considers a problem setup where an intelligent agent comprising a neural network-based prediction component and a deep reinforcement learning component provides advice to a human decision-maker in complex repeated decision-making environments. Whether the human decision-maker would follow the agent's advice depends on their beliefs and trust in the agent and on their understanding of the advice itself. To this end, we developed an approach named ADESSE to generate explanations about the adviser agent to improve human trust and decision-making. Computational experiments on a range of environments with varying model sizes demonstrate the applicability and scalability of ADESSE. Furthermore, an interactive game-based user study shows that participants were significantly more satisfied, achieved a higher reward in the game, and took less time to select an action when presented with explanations generated by ADESSE. These findings illuminate the critical role of tailored, human-centered explanations in AI-assisted decision-making.
Medical Neural Architecture Search: Survey and Taxonomy
Authors: Benmeziane Hadjer , Hamzaoui Imane , Cherif Zayneb , El Maghraoui Kaoutar
Published: July 2024

This paper presents a comprehensive survey of Medical Neural Architecture Search (MedNAS), a burgeoning field at the confluence of deep learning and medical imaging. With the increasing prevalence of FDA-approved medical deep learning models, MedNAS emerges as a key area in leveraging computational innovations for healthcare advancements. Our survey examines the paradigm shift introduced by Neural Architecture Search (NAS), which automates neural network design, replacing traditional, manual designs. We explore the unique search spaces tailored for medical tasks on different types of data from images to EEG, the methodologies of MedNAS, and their impact on medical applications.
Guide to Numerical Experiments on Elections in Computational Social Choice
Authors: Boehmer Niclas , Faliszewski Piotr , Janeczko Łukasz , Kaczmarczyk Andrzej , Lisowski Grzegorz , Pierczyński Grzegorz , Rey Simon , Stolicki Dariusz , Szufa Stanisław , Wąs Tomasz
Published: July 2024

We analyze how numerical experiments regarding elections were conducted within computational social choice literature (focusing on papers published in the IJCAI, AAAI, and AAMAS conferences). We analyze the sizes of the studied elections and the methods of generating preference data, thereby making previously hidden standards and practices explicit. In particular, we survey a number of statistical cultures for generating elections and their commonly used parameters.
Social Learning through Interactions with Other Agents: A Survey
Authors: Hillier Dylan , Tan Cheston , Jiang Jing
Published: July 2024

Social learning plays an important role in the development of human intelligence. As children, we imitate our parent's speech patterns until we are able to produce sounds; we learn from them praising us and scolding us, and as adults, we learn by working with others. In this work, we survey the degree to which this developmental paradigm -- social learning -- has been mirrored in machine learning. In particular, since learning socially requires interacting with others, we are interested in how embodied agents can and have utilised these techniques. This is especially in light of the degree to which recent advances in natural language processing (NLP) enable us to perform new forms of social learning. We look at how behaviour cloning and next-token prediction mirror human imitation, how learning from human feedback mirrors human education, and how we can go further to enable fully communicative agents that learn from each other.

We find that while individual social learning techniques have been used successfully, there has been little unifying work showing how to bring them together into socially embodied agents.
A Survey of Graph Meets Large Language Model: Progress and Future Directions
Authors: Li Yuhan , Li Zhixun , Wang Peisong , Li Jia , Sun Xiangguo , Cheng Hong , Yu Jeffrey Xu
Published: July 2024

Graph plays a significant role in representing and analyzing complex relationships in real-world applications such as citation networks, social networks, and biological data. Recently, Large Language Models (LLMs), which have achieved tremendous success in various domains, have also been leveraged in graph-related tasks to surpass traditional Graph Neural Networks (GNNs) based methods and yield state-of-the-art performance. In this survey, we first present a comprehensive review and analysis of existing methods that integrate LLMs with graphs. First of all, we propose a new taxonomy, which organizes existing methods into three categories based on the role (i.e., enhancer, predictor, and alignment component) played by LLMs in graph-related tasks. Then we systematically survey the representative methods along the three categories of the taxonomy. Finally, we discuss the remaining limitations of existing studies and highlight promising avenues for future research. The relevant papers are summarized and will be consistently updated at: https://github.com/yhLeeee/Awesome-LLMs-in-Graph-tasks.
UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation
Authors: He Qingdong , Peng Jinlong , Jiang Zhengkai , Wu Kai , Ji Xiaozhong , Zhang Jiangning , Wang Yabiao , Wang Chengjie , Chen Mingang , Wu Yunsheng
Published: July 2024

3D open-vocabulary scene understanding aims to recognize arbitrary novel categories beyond the base label space. However, existing works not only fail to fully utilize all the available modal information in the 3D domain but also lack sufficient granularity in representing the features of each modality. In this paper, we propose a unified multimodal 3D open-vocabulary scene understanding network, namely UniM-OV3D, aligning point clouds with image, language and depth. To better integrate global and local features of the point clouds, we design a hierarchical point cloud feature extraction module that learns fine-grained feature representations. Further, to facilitate the learning of coarse-to-fine point-semantic representations from captions, we propose the utilization of hierarchical 3D caption pairs, capitalizing on geometric constraints across various viewpoints of 3D scenes. Extensive experimental results have demonstrated the effectiveness and superiority of our method in open-vocabulary semantic and instance segmentation, which achieves state-of-the-art performance on both indoor and outdoor benchmarks such as ScanNet, ScanNet200, S3IDS and nuScenes. Code is available at https://github.com/hithqd/UniM-OV3D.
A Lightweight U-like Network Utilizing Neural Memory Ordinary Differential Equations for Slimming the Decoder
Authors: He Quansong , Yao Xiaojun , Wu Jun , Yi Zhang , He Tao
Published: July 2024

In recent years, advanced U-like networks have demonstrated remarkable performance in medical image segmentation tasks. However, their drawbacks, including excessive parameters, high computational complexity, and slow inference speed, pose challenges for practical implementation in scenarios with limited computational resources. Existing lightweight U-like networks have alleviated some problems, but they often have pre-designed structures and consist of non-detachable modules, limiting their application scenarios. In this paper, we propose three plug-and-play decoders by employing different discretization methods of the neural memory Ordinary Differential Equation (nmODE). These decoders integrate features at various levels of abstraction by processing information from skip connections and performing numerical operations on upward paths. Through experiments on the PH2, ISIC2017, and ISIC2018 datasets, we embed these decoders into different U-like networks, demonstrating their effectiveness in significantly reducing the number of parameters and computation while maintaining performance. In summary, the proposed discretized nmODE decoder is capable of reducing the number of parameters by about 20% ~ 50% and computation by up to 74%, while being adaptive to all U-like networks. Our code is available at https://github.com/nayutayuki/Lightweight-nmODE-Decoders-For-U-like-networks.