cometadata/datacite-titles-descriptions-related-identifiers huggingface dataset:
DOI: 10.7274/25582434.v1
title: Motif Mining: Towards Multimodal Retrieval for Discovering Disinformation Trends in Social Media

description: As the world has become more and more connected online, so too has the amount of data that has been produced. Alongside this increase in data production the amount of disinformation has drastically increased. Coming to the fore in recent years and playing outsized roles in the politics of a myriad of countries, disinformation has become a hot topic. The scale of the data, in particular images and videos, has grown beyond the human capacity for processing in any detail. To help aid reviewers of online content we propose extending the idea of image provenance to include multimodal motif mining. This formalization and implementation of the problem of visual motif mining is designed to help humans process the vast amounts of visual data more quickly and to aid in the discovery of emerging disinformation trends in the online space. Early efforts in this space focused on monomodal capabilities, considering images and texts separately. Efforts to combine the two via fusion techniques left something to be desired. With the release of contrastive language-image pre-training techniques a new door has been opened for multimodal shared embedding spaces. In this work I share some of the earlier monomodal motif mining strategies, focusing on images as their subject. This is followed by studies in using CLIP-based strategies for projecting images and texts from social media into a shared latent space, allowing for crossmodal retrieval and understanding, and thus allowing for earlier motif mining pipelines to be expanded beyond one modality. Finally, an extension of contrastive losses to more than two dimensions is formalized and studied, with trimodal and quadmodal models developed. These allow the pipelines to progress even further beyond bimodal models and given proper training data could allow them to encompass every type of artifact that could appear on social media.
DOI: 10.7274/25596291
title: Context-Aware Models for Automatic Source Code Summarization

description: Source Code Summarization is a program comprehension task that consists of writing natural language descriptions of source code. These summaries are important because they are an essential part of software documentation, such as the descriptions in APIs. They are also necessary for maintenance of legacy soft- ware systems. The state-of-the-art for automatic source code summarization, when I started my work were neural networks developed for machine translation. They were designed to accept a snippet of source code, usually a subroutine, as a sequence of tokens and generate an English language description. These techniques were based on sequence-to-sequence learning , i.e., the summary sequence was built one word at a time, using an attention mechanism and code sequence. However, often some of the information required to summarize the subroutine descriptively is not inside the subroutine. The necessary information lives in the ”context” around the code, such as other subroutines, files, and build files, as well as the pre-learnt human knowledge. In this dissertation, I will present my research on modeling various types of contextual information for better automatic source code summarization.
DOI: 10.7274/25596291.v1
title: Context-Aware Models for Automatic Source Code Summarization

description: Source Code Summarization is a program comprehension task that consists of writing natural language descriptions of source code. These summaries are important because they are an essential part of software documentation, such as the descriptions in APIs. They are also necessary for maintenance of legacy soft- ware systems. The state-of-the-art for automatic source code summarization, when I started my work were neural networks developed for machine translation. They were designed to accept a snippet of source code, usually a subroutine, as a sequence of tokens and generate an English language description. These techniques were based on sequence-to-sequence learning , i.e., the summary sequence was built one word at a time, using an attention mechanism and code sequence. However, often some of the information required to summarize the subroutine descriptively is not inside the subroutine. The necessary information lives in the ”context” around the code, such as other subroutines, files, and build files, as well as the pre-learnt human knowledge. In this dissertation, I will present my research on modeling various types of contextual information for better automatic source code summarization.
DOI: 10.7274/25604631
title: Trust, AI, and Synthetic Biometrics

description: Artificial Intelligence-based image generation has recently seen remarkable advancements, largely driven by deep learning techniques, such as Generative Adversarial Networks (GANs). With the influx and development of generative models, so too have biometric re-identification models and presentation attack detection models seen a surge in discriminative performance. However, despite the impressive photo-realism of generated samples and the additive value to the data augmentation pipeline, the role and usage of machine learning models has received intense scrutiny and criticism, especially in the context of biometrics, often being labeled as untrustworthy. Problems that have garnered attention in modern machine learning include: humans' and machines' shared inability to verify the authenticity of (biometric) data, the inadvertent leaking of private biometric data through the image synthesis process, and racial bias in facial recognition algorithms. Given the arrival of these unwanted side effects, public trust has been shaken in the blind use and ubiquity of machine learning. 

However, in tandem with the advancement of generative AI, there are research efforts to re-establish trust in generative and discriminative machine learning models. Explainability methods based on aggregate model salience maps can elucidate the inner workings of a detection model, establishing trust in a post hoc manner. The CYBORG training strategy, originally proposed by Boyd, attempts to actively build trust into discriminative models by incorporating human salience into the training process. 

In doing so, CYBORG-trained machine learning models behave more similar to human annotators and generalize well to unseen types of synthetic data. 
Work in this dissertation also attempts to renew trust in generative models by training generative models on synthetic data in order to avoid identity leakage in models trained on authentic data. In this way, the privacy of individuals whose biometric data was seen during training is not compromised through the image synthesis procedure. Future development of privacy-aware image generation techniques will hopefully achieve the same degree of biometric utility in generative models with added guarantees of trustworthiness.
DOI: 10.7274/25604631.v1
title: Trust, AI, and Synthetic Biometrics

description: Artificial Intelligence-based image generation has recently seen remarkable advancements, largely driven by deep learning techniques, such as Generative Adversarial Networks (GANs). With the influx and development of generative models, so too have biometric re-identification models and presentation attack detection models seen a surge in discriminative performance. However, despite the impressive photo-realism of generated samples and the additive value to the data augmentation pipeline, the role and usage of machine learning models has received intense scrutiny and criticism, especially in the context of biometrics, often being labeled as untrustworthy. Problems that have garnered attention in modern machine learning include: humans' and machines' shared inability to verify the authenticity of (biometric) data, the inadvertent leaking of private biometric data through the image synthesis process, and racial bias in facial recognition algorithms. Given the arrival of these unwanted side effects, public trust has been shaken in the blind use and ubiquity of machine learning. 

However, in tandem with the advancement of generative AI, there are research efforts to re-establish trust in generative and discriminative machine learning models. Explainability methods based on aggregate model salience maps can elucidate the inner workings of a detection model, establishing trust in a post hoc manner. The CYBORG training strategy, originally proposed by Boyd, attempts to actively build trust into discriminative models by incorporating human salience into the training process. 

In doing so, CYBORG-trained machine learning models behave more similar to human annotators and generalize well to unseen types of synthetic data. 
Work in this dissertation also attempts to renew trust in generative models by training generative models on synthetic data in order to avoid identity leakage in models trained on authentic data. In this way, the privacy of individuals whose biometric data was seen during training is not compromised through the image synthesis procedure. Future development of privacy-aware image generation techniques will hopefully achieve the same degree of biometric utility in generative models with added guarantees of trustworthiness.
DOI: 10.7274/25604643
title: Solutions to Limited Annotation Problems of Deep Learning in Medical Image Segmentation

description: Image segmentation holds broad applications in medical image analysis, providing crucial support to doctors in both automatic diagnosis and computer-assisted interventions. The heterogeneity observed across various medical image datasets necessitates the training of task-specific segmentation models. However, effectively supervising the training of deep learning segmentation models typically demands dense label masks, a requirement that becomes challenging due to the constraints posed by privacy and cost issues in collecting large-scale medical datasets. These challenges collectively give rise to the limited annotations problems in medical image segmentation.

In this dissertation, we address the challenges posed by annotation deficiencies through a comprehensive exploration of various strategies. Firstly, we employ self-supervised learning to extract information from unlabeled data, presenting a tailored self-supervised method designed specifically for convolutional neural networks and 3D Vision Transformers. Secondly, our attention shifts to domain adaptation problems, leveraging images with similar content but in different modalities. We introduce the use of contrastive loss as a shape constraint in our image translation framework, resulting in both improved performance and enhanced training robustness. Thirdly, we incorporate diffusion models for data augmentation, expanding datasets with generated image-label pairs. Lastly, we explore to extract segmentation masks from image-level annotations alone.
We propose a multi-task training framework for ECG abnormal beats localization and a conditional diffusion-based algorithm for tumor detection.
DOI: 10.7274/25604643.v1
title: Solutions to Limited Annotation Problems of Deep Learning in Medical Image Segmentation

description: Image segmentation holds broad applications in medical image analysis, providing crucial support to doctors in both automatic diagnosis and computer-assisted interventions. The heterogeneity observed across various medical image datasets necessitates the training of task-specific segmentation models. However, effectively supervising the training of deep learning segmentation models typically demands dense label masks, a requirement that becomes challenging due to the constraints posed by privacy and cost issues in collecting large-scale medical datasets. These challenges collectively give rise to the limited annotations problems in medical image segmentation.

In this dissertation, we address the challenges posed by annotation deficiencies through a comprehensive exploration of various strategies. Firstly, we employ self-supervised learning to extract information from unlabeled data, presenting a tailored self-supervised method designed specifically for convolutional neural networks and 3D Vision Transformers. Secondly, our attention shifts to domain adaptation problems, leveraging images with similar content but in different modalities. We introduce the use of contrastive loss as a shape constraint in our image translation framework, resulting in both improved performance and enhanced training robustness. Thirdly, we incorporate diffusion models for data augmentation, expanding datasets with generated image-label pairs. Lastly, we explore to extract segmentation masks from image-level annotations alone.
We propose a multi-task training framework for ECG abnormal beats localization and a conditional diffusion-based algorithm for tumor detection.
DOI: 10.7274/25607781
title: Trading Fairness and Sharing Accountability in Decentralized Secure Data Sharing

description: In the modern era of big data, guaranteeing data sharing fairness and accountability of the shared data are important for ensuring user privilege and marketplace supervision. On one hand, due to the Internet’s nature, where data sharing usually occurs, every user might be potentially malicious and cheat others. As a result, there is limited trust between data-sharing participants, and ensuring fairness among them without a centralized trusted third party is non-trivial. On the other hand, with the increasing awareness of the data owners’ control over their data and the demands of market-wise regulation, guaranteeing accountability, traceability, and trackability of the sharing behaviors is also necessary. Although different protocols have been proposed to address such concerns, they can be further improved in many aspects.
In this dissertation, I present several lines of research around ensuring trading fairness and sharing accountability during the whole decentralized data-sharing life cycle. I detail research on ensuring traceability and trackability of the sharing behaviors, guaranteeing secure and efficient fairness between untrusted trading participants, and further proposing various designs with the consideration of different trading scenarios. These improvements and innovations promote better guarantees for fairness and accountability of online data sharing.
DOI: 10.7274/25607781.v1
title: Trading Fairness and Sharing Accountability in Decentralized Secure Data Sharing

description: In the modern era of big data, guaranteeing data sharing fairness and accountability of the shared data are important for ensuring user privilege and marketplace supervision. On one hand, due to the Internet’s nature, where data sharing usually occurs, every user might be potentially malicious and cheat others. As a result, there is limited trust between data-sharing participants, and ensuring fairness among them without a centralized trusted third party is non-trivial. On the other hand, with the increasing awareness of the data owners’ control over their data and the demands of market-wise regulation, guaranteeing accountability, traceability, and trackability of the sharing behaviors is also necessary. Although different protocols have been proposed to address such concerns, they can be further improved in many aspects.
In this dissertation, I present several lines of research around ensuring trading fairness and sharing accountability during the whole decentralized data-sharing life cycle. I detail research on ensuring traceability and trackability of the sharing behaviors, guaranteeing secure and efficient fairness between untrusted trading participants, and further proposing various designs with the consideration of different trading scenarios. These improvements and innovations promote better guarantees for fairness and accountability of online data sharing.
DOI: 10.7274/25607826
title: Knowledge-centric Machine Learning on Graphs

description: Graph Machine Learning (GML) has gained considerable attention in modeling complex graph-structured data, but many of them focus on collecting high-quality data (i.e., data-centric) and developing complex model architectures (i.e., model-centric). However, these two paradigms come with inherent limitations and challenges: data-centric approaches often demand intensive labor for tasks like data annotation and cleaning, while model-centric approaches usually require specialized expertise for model refinements. There remains a significant reservoir of unexplored potential in harnessing useful information that already exists in the data and learned by models, i.e., knowledge, as a directive force for learning. 

In this dissertation, I introduce a new paradigm of machine learning on graphs: knowledge-centric. This paradigm seeks to leverage all available knowledge, which may come from data, models, or external sources, to facilitate an effective learning process. My research focuses on three different facets to obtain and leverage knowledge in GML, including learning knowledge from data, distilling knowledge from models, and encoding knowledge from external sources. By anchoring on the knowledge, there is a reduced reliance on massive data and intricate model architectures. In addition, knowledge can enhance GML models' performance, trustworthiness, and efficiency.
DOI: 10.7274/25607826.v1
title: Knowledge-centric Machine Learning on Graphs

description: Graph Machine Learning (GML) has gained considerable attention in modeling complex graph-structured data, but many of them focus on collecting high-quality data (i.e., data-centric) and developing complex model architectures (i.e., model-centric). However, these two paradigms come with inherent limitations and challenges: data-centric approaches often demand intensive labor for tasks like data annotation and cleaning, while model-centric approaches usually require specialized expertise for model refinements. There remains a significant reservoir of unexplored potential in harnessing useful information that already exists in the data and learned by models, i.e., knowledge, as a directive force for learning. 

In this dissertation, I introduce a new paradigm of machine learning on graphs: knowledge-centric. This paradigm seeks to leverage all available knowledge, which may come from data, models, or external sources, to facilitate an effective learning process. My research focuses on three different facets to obtain and leverage knowledge in GML, including learning knowledge from data, distilling knowledge from models, and encoding knowledge from external sources. By anchoring on the knowledge, there is a reduced reliance on massive data and intricate model architectures. In addition, knowledge can enhance GML models' performance, trustworthiness, and efficiency.
DOI: 10.7274/25608504
title: Empowering Graph Neural Networks for Real-World Tasks

description: Numerous types of real-world data can be naturally represented as graphs, such as social networks, trading networks, and biological molecules. This highlights the need for effective graph representations to support various tasks. In recent years, graph neural networks (GNNs) have demonstrated remarkable success in extracting information from graphs and enabling graph-related tasks. However, they still face a series of challenges in solving real-world problems, including scarcity of labeled data, scalability issues, potential bias, etc. These challenges stem from both domain-specific issues and inherent limitations of GNNs. This thesis introduces various strategies to tackle these challenges and empower GNNs on real-world tasks.

For the domain-specific challenges, in this thesis, we especially focus on challenges in the chemistry domain, which plays a pivotal role in the drug discovery process. Considering the significant resources needed for labeling through wet lab experiments, the AI for chemistry domain struggles with the scarcity of labeled datasets. To address this, we present a comprehensive set of strategies that span model-based and data-based strategies alongside a hybrid method. These methods ingeniously utilize the diversity of data, models, and molecular representations to compensate for the lack of labels in individual datasets. For the inherent challenges, this thesis introduces strategies to overcome two main challenges: scalability and degree-based issues, especially in the context of link prediction tasks. Both of these two challenges originate from the mechanism of GNNs, which involves the iterative aggregation of neighboring nodes' information to update each central node. For the scalability issue, our work not only preserves GNNs' prediction performance but also significantly boosts inference speed. Regarding degree bias, our work highly improves the effectiveness of GNNs for underrepresented nodes with very light additional computational costs. These contributions not only address critical gaps in applying GNNs to specific domains but also lay the groundwork for future exploration in the broader field of graph-based real-world tasks.
DOI: 10.7274/25608504.v1
title: Empowering Graph Neural Networks for Real-World Tasks

description: Numerous types of real-world data can be naturally represented as graphs, such as social networks, trading networks, and biological molecules. This highlights the need for effective graph representations to support various tasks. In recent years, graph neural networks (GNNs) have demonstrated remarkable success in extracting information from graphs and enabling graph-related tasks. However, they still face a series of challenges in solving real-world problems, including scarcity of labeled data, scalability issues, potential bias, etc. These challenges stem from both domain-specific issues and inherent limitations of GNNs. This thesis introduces various strategies to tackle these challenges and empower GNNs on real-world tasks.

For the domain-specific challenges, in this thesis, we especially focus on challenges in the chemistry domain, which plays a pivotal role in the drug discovery process. Considering the significant resources needed for labeling through wet lab experiments, the AI for chemistry domain struggles with the scarcity of labeled datasets. To address this, we present a comprehensive set of strategies that span model-based and data-based strategies alongside a hybrid method. These methods ingeniously utilize the diversity of data, models, and molecular representations to compensate for the lack of labels in individual datasets. For the inherent challenges, this thesis introduces strategies to overcome two main challenges: scalability and degree-based issues, especially in the context of link prediction tasks. Both of these two challenges originate from the mechanism of GNNs, which involves the iterative aggregation of neighboring nodes' information to update each central node. For the scalability issue, our work not only preserves GNNs' prediction performance but also significantly boosts inference speed. Regarding degree bias, our work highly improves the effectiveness of GNNs for underrepresented nodes with very light additional computational costs. These contributions not only address critical gaps in applying GNNs to specific domains but also lay the groundwork for future exploration in the broader field of graph-based real-world tasks.