OpenRaiser/Intern-Atlas huggingface dataset:
BodyShapeGPT: SMPL Body Shape Manipulation with LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Generative AI models provide a wide range of tools capable of performing complex tasks in a fraction of the time it would take a human. Among these, Large Language Models (LLMs) stand out for their ability to generate diverse texts, from literary narratives to specialized responses in different fields of knowledge. This paper explores the use of fine-tuned LLMs to identify physical descriptions of people, and subsequently create accurate representations of avatars using the SMPL-X model by inferring shape parameters. We demonstrate that LLMs can be trained to understand and manipulate the shape space of SMPL, allowing the control of 3D human shapes through natural language. This approach promises to improve human-machine interaction and opens new avenues for customization and simulation in virtual environments.
Training Over a Distribution of Hyperparameters for Enhanced Performance and Adaptability on Imbalanced Classification
Venue: arXiv cs.LG 2024
Year: 2024

Although binary classification is a well-studied problem, training reliable classifiers under severe class imbalance remains a challenge. Recent techniques mitigate the ill effects of imbalance on training by modifying the loss functions or optimization methods. We observe that different hyperparameter values on these loss functions perform better at different recall values. We propose to exploit this fact by training one model over a distribution of hyperparameter values--instead of a single value--via Loss Conditional Training (LCT). Experiments show that training over a distribution of hyperparameters not only approximates the performance of several models but actually improves the overall performance of models on both CIFAR and real medical imaging applications, such as melanoma and diabetic retinopathy detection. Furthermore, training models with LCT is more efficient because some hyperparameter tuning can be conducted after training to meet individual needs without needing to retrain from scratch.
Subexponential Algorithms for Clique Cover on Unit Disk and Unit Ball Graphs
Venue: arXiv cs.DS 2024
Year: 2024

In Clique Cover, given a graph $G$ and an integer $k$, the task is to partition the vertices of $G$ into $k$ cliques. Clique Cover on unit ball graphs has a natural interpretation as a clustering problem, where the objective function is the maximum diameter of a cluster. Many classical NP-hard problems are known to admit $2^{O(n^{(1 - 1/d)})}$-time algorithms on unit ball graphs in $\mathbb{R}^d$ [de Berg et al., SIAM J. Comp 2018]. A notable exception is the Maximum Clique problem, which admits a polynomial-time algorithm on unit disk graphs and a subexponential algorithm on unit ball graphs in $\mathbb{R}^3$, but no subexponential algorithm on unit ball graphs in dimensions 4 or larger, assuming the ETH [Bonamy et al., JACM 2021]. In this work, we show that Clique Cover also suffers from a "curse of dimensionality", albeit in a significantly different way compared to Maximum Clique. We present a $2^{O(\sqrt{n})}$-time algorithm for unit disk graphs and argue that it is tight under the ETH. On the other hand, we show that Clique Cover does not admit a $2^{o(n)}$-time algorithm on unit ball graphs in dimension $5$, unless the ETH fails.
A Global Medical Data Security and Privacy Preserving Standards Identification Framework for Electronic Healthcare Consumers
Venue: arXiv cs.LG 2024
Year: 2024

Electronic Health Records (EHR) are crucial for the success of digital healthcare, with a focus on putting consumers at the center of this transformation. However, the digitalization of healthcare records brings along security and privacy risks for personal data. The major concern is that different countries have varying standards for the security and privacy of medical data. This paper proposed a novel and comprehensive framework to standardize these rules globally, bringing them together on a common platform. To support this proposal, the study reviews existing literature to understand the research interest in this issue. It also examines six key laws and standards related to security and privacy, identifying twenty concepts. The proposed framework utilized K-means clustering to categorize these concepts and identify five key factors. Finally, an Ordinal Priority Approach is applied to determine the preferred implementation of these factors in the context of EHRs. The proposed study provides a descriptive then prescriptive framework for the implementation of privacy and security in the context of electronic health records. Therefore, the findings of the proposed framework are useful for professionals and policymakers in improving the security and privacy associated with EHRs.
Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models
Venue: arXiv cs.LG 2024
Year: 2024

Membership inference attacks (MIAs) on diffusion models have emerged as potential evidence of unauthorized data usage in training pre-trained diffusion models. These attacks aim to detect the presence of specific images in training datasets of diffusion models. Our study delves into the evaluation of state-of-the-art MIAs on diffusion models and reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation. We introduce CopyMark, a more realistic MIA benchmark that distinguishes itself through the support for pre-trained diffusion models, unbiased datasets, and fair evaluation pipelines. Through extensive experiments, we demonstrate that the effectiveness of current MIA methods significantly degrades under these more practical conditions. Based on our results, we alert that MIA, in its current state, is not a reliable approach for identifying unauthorized data usage in pre-trained diffusion models. To the best of our knowledge, we are the first to discover the performance overestimation of MIAs on diffusion models and present a unified benchmark for more realistic evaluation. Our code is available on GitHub: \url{https://github.com/caradryanl/CopyMark}.
Making Data: The Work Behind Artificial Intelligence
Venue: arXiv cs.HC 2024
Year: 2024

AI generates both enthusiasm and disillusionment, with promises that often go unfulfilled. It is therefore not surprising that human labor, which is its fundamental component, is also subject to these same deceptions. The development of "smart technologies" depends, at different stages, on a multitude of precarious, underpaid and invisible workers, who, dispersed globally, carry out repetitive, fragmented activities, paid per task and completed in a few seconds. These are workers who label data to train algorithms, through tasks that require the intuitive, creative and cognitive abilities of human beings, such as categorizing images, classifying advertisements, transcribing audio and video, evaluating advertisements, moderating content on social media, labeling human anatomical points of interest, digitizing documents, etc. This form of work is often referred to as "microwork". Our contribution, which documents the conditions of microwork in Brazil and offers portraits of the workers, is a step in the wider effort to overcome the current state of invisibilization. It opens up avenues for future research, with the aim of better characterizing this new form of work, tracing its changes over time in relation to the dynamics of globalization and, ideally, identifying levers for action and transitions.
Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development
Venue: arXiv cs.CL 2024
Year: 2024

This paper aims to answer one central question: to what extent can open-source generative text models be used in a workflow to approximate thematic analysis in social science research? To answer this question, we present the Generative AI-enabled Theme Organization and Structuring (GATOS) workflow, which uses open-source machine learning techniques, natural language processing tools, and generative text models to facilitate thematic analysis. To establish validity of the method, we present three case studies applying the GATOS workflow, leveraging these models and techniques to inductively create codebooks similar to traditional procedures using thematic analysis. Specifically, we investigate the extent to which a workflow comprising open-source models and tools can inductively produce codebooks that approach the known space of themes and sub-themes. To address the challenge of gleaning insights from these texts, we combine open-source generative text models, retrieval-augmented generation, and prompt engineering to identify codes and themes in large volumes of text, i.e., generate a qualitative codebook. The process mimics an inductive coding process that researchers might use in traditional thematic analysis by reading text one unit of analysis at a time, considering existing codes already in the codebook, and then deciding whether or not to generate a new code based on whether the extant codebook provides adequate thematic coverage. We demonstrate this workflow using three synthetic datasets from hypothetical organizational research settings: a study of teammate feedback in teamwork settings, a study of organizational cultures of ethical behavior, and a study of employee perspectives about returning to their offices after the pandemic. We show that the GATOS workflow is able to identify themes in the text that were used to generate the original synthetic datasets.
CliMB: An AI-enabled Partner for Clinical Predictive Modeling
Venue: arXiv cs.HC 2024
Year: 2024

Despite its significant promise and continuous technical advances, real-world applications of artificial intelligence (AI) remain limited. We attribute this to the "domain expert-AI-conundrum": while domain experts, such as clinician scientists, should be able to build predictive models such as risk scores, they face substantial barriers in accessing state-of-the-art (SOTA) tools. While automated machine learning (AutoML) has been proposed as a partner in clinical predictive modeling, many additional requirements need to be fulfilled to make machine learning accessible for clinician scientists. To address this gap, we introduce CliMB, a no-code AI-enabled partner designed to empower clinician scientists to create predictive models using natural language. CliMB guides clinician scientists through the entire medical data science pipeline, thus empowering them to create predictive models from real-world data in just one conversation. CliMB also creates structured reports and interpretable visuals. In evaluations involving clinician scientists and systematic comparisons against a baseline GPT-4, CliMB consistently demonstrated superior performance in key areas such as planning, error prevention, code execution, and model performance. Moreover, in blinded assessments involving 45 clinicians from diverse specialties and career stages, more than 80% preferred CliMB over GPT-4. Overall, by providing a no-code interface with clear guidance and access to SOTA methods in the fields of data-centric AI, AutoML, and interpretable ML, CliMB empowers clinician scientists to build robust predictive models. The proof-of-concept version of CliMB is available as open-source software on GitHub: https://github.com/vanderschaarlab/climb.
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
Venue: arXiv cs.CL 2024
Year: 2024

Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to significant training imbalances, potentially resulting in performance degradation. Consequently, we propose to mitigate this imbalance by merging SFT models fine-tuned with different data orders, thereby enhancing the overall effectiveness of SFT. Additionally, we introduce a novel technique, "parameter-selection merging," which outperforms traditional weighted-average methods on five datasets. Further, through analysis and ablation studies, we validate the effectiveness of our method and identify the sources of performance improvements.
A Two-Stage Proactive Dialogue Generator for Efficient Clinical Information Collection Using Large Language Model
Venue: arXiv cs.CL 2024
Year: 2024

Efficient patient-doctor interaction is among the key factors for a successful disease diagnosis. During the conversation, the doctor could query complementary diagnostic information, such as the patient's symptoms, previous surgery, and other related information that goes beyond medical evidence data (test results) to enhance disease diagnosis. However, this procedure is usually time-consuming and less-efficient, which can be potentially optimized through computer-assisted systems. As such, we propose a diagnostic dialogue system to automate the patient information collection procedure. By exploiting medical history and conversation logic, our conversation agents, particularly the doctor agent, can pose multi-round clinical queries to effectively collect the most relevant disease diagnostic information. Moreover, benefiting from our two-stage recommendation structure, carefully designed ranking criteria, and interactive patient agent, our model is able to overcome the under-exploration and non-flexible challenges in dialogue generation. Our experimental results on a real-world medical conversation dataset show that our model can generate clinical queries that mimic the conversation style of real doctors, with efficient fluency, professionalism, and safety, while effectively collecting relevant disease diagnostic information.
Reward-RAG: Enhancing RAG with Reward Driven Supervision
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we introduce Reward-RAG, a novel approach designed to enhance the Retrieval-Augmented Generation (RAG) model through Reward-Driven Supervision. Unlike previous RAG methodologies, which focus on training language models (LMs) to utilize external knowledge retrieved from external sources, our method adapts retrieval information to specific domains by employing CriticGPT to train a dedicated reward model. This reward model generates synthesized datasets for fine-tuning the RAG encoder, aligning its outputs more closely with human preferences. The versatility of our approach allows it to be effectively applied across various domains through domain-specific fine-tuning. We evaluate Reward-RAG on publicly available benchmarks from multiple domains, comparing it to state-of-the-art methods. Our experimental results demonstrate significant improvements in performance, highlighting the effectiveness of Reward-RAG in improving the relevance and quality of generated responses. These findings underscore the potential of integrating reward models with RAG to achieve superior outcomes in natural language generation tasks.
CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely solely on image or text inputs and lack fine-grained control, our system leverages both images to guide generations at fine-grained levels and natural language texts to describe the features of generations. CalliffusionV2 excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training. Comprehensive tests confirm that our system produces calligraphy that is both stylistically accurate and recognizable by neural network classifiers and human evaluators.
A novel TLS-based Fingerprinting approach that combines feature expansion and similarity mapping
Venue: arXiv cs.CR 2024
Year: 2024

Malicious domains are part of the landscape of the internet but are becoming more prevalent and more dangerous to both companies and individuals. They can be hosted on variety of technologies and serve an array of content, ranging from Malware, command and control, and complex Phishing sites that are designed to deceive and expose. Tracking, blocking and detecting such domains is complex, and very often involves complex allow or deny list management or SIEM integration with open-source TLS fingerprinting techniques. Many fingerprint techniques such as JARM and JA3 are used by threat hunters to determine domain classification, but with the increase in TLS similarity, particularly in CDNs, they are becoming less useful. The aim of this paper is to adapt and evolve open-source TLS fingerprinting techniques with increased features to enhance granularity, and to produce a similarity mapping system that enables the tracking and detection of previously unknown malicious domains. This is done by enriching TLS fingerprints with HTTP header data and producing a fine grain similarity visualisation that represented high dimensional data using MinHash and local sensitivity hashing. Influence was taken from the Chemistry domain, where the problem of high dimensional similarity in chemical fingerprints is often encountered. An enriched fingerprint was produced which was then visualised across three separate datasets. The results were analysed and evaluated, with 67 previously unknown malicious domains being detected based on their similarity to known malicious domains and nothing else. The similarity mapping technique produced demonstrates definite promise in the arena of early detection of Malware and Phishing domains.
Rethinking Selection in Generational Genetic Algorithms to Solve Combinatorial Optimization Problems: An Upper Bound-based Parent Selection Strategy for Recombination
Venue: arXiv cs.NE 2024
Year: 2024

Existing stochastic selection strategies for parent selection in generational GA help build genetic diversity and sustain exploration; however, it ignores the possibility of exploiting knowledge gained by the process to make informed decisions for parent selection, which can often lead to an inefficient search for large, challenging optimization problems. This work proposes a deterministic parent selection strategy for recombination in a generational GA setting called Upper Bound-based Parent Selection (UBS) to solve NP-hard combinatorial optimization problems. Specifically, as part of the UBS strategy, we formulate the parent selection problem using the MAB framework and a modified UCB1 algorithm to manage exploration and exploitation. Further, we provided a unique similarity-based approach for transferring knowledge of the search progress between generations to accelerate the search. To demonstrate the effectiveness of the proposed UBS strategy in comparison to traditional stochastic selection strategies, we conduct experimental studies on two NP-hard combinatorial optimization problems: team orienteering and quadratic assignment. Specifically, we first perform a characterization study to determine the potential of UBS and the best configuration for all the selection strategies involved. Next, we run experiments using these best configurations as part of the comparison study. The results from the characterization studies reveal that UBS, in most cases, favors larger variations among the population between generations. Next, the comparison studies reveal that UBS can effectively search for high-quality solutions faster than traditional stochastic selection strategies on challenging NP-hard combinatorial optimization problems under given experimental conditions.
From Pixels to Personas: Investigating and Modeling Self-Anthropomorphism in Human-Robot Dialogues
Venue: arXiv cs.CL 2024
Year: 2024

Self-anthropomorphism in robots manifests itself through their display of human-like characteristics in dialogue, such as expressing preferences and emotions. Our study systematically analyzes self-anthropomorphic expression within various dialogue datasets, outlining the contrasts between self-anthropomorphic and non-self-anthropomorphic responses in dialogue systems. We show significant differences in these two types of responses and propose transitioning from one type to the other. We also introduce Pix2Persona, a novel dataset aimed at developing ethical and engaging AI systems in various embodiments. This dataset preserves the original dialogues from existing corpora and enhances them with paired responses: self-anthropomorphic and non-self-anthropomorphic for each original bot response. Our work not only uncovers a new category of bot responses that were previously under-explored but also lays the groundwork for future studies about dynamically adjusting self-anthropomorphism levels in AI systems to align with ethical standards and user expectations.
Towards Cost Sensitive Decision Making
Venue: arXiv cs.LG 2024
Year: 2024

Many real-world situations allow for the acquisition of additional relevant information when making decisions with limited or uncertain data. However, traditional RL approaches either require all features to be acquired beforehand (e.g. in a MDP) or regard part of them as missing data that cannot be acquired (e.g. in a POMDP). In this work, we consider RL models that may actively acquire features from the environment to improve the decision quality and certainty, while automatically balancing the cost of feature acquisition process and the reward of task decision process. We propose the Active-Acquisition POMDP and identify two types of the acquisition process for different application domains. In order to assist the agent in the actively-acquired partially-observed environment and alleviate the exploration-exploitation dilemma, we develop a model-based approach, where a deep generative model is utilized to capture the dependencies of the features and impute the unobserved features. The imputations essentially represent the beliefs of the agent. Equipped with the dynamics model, we develop hierarchical RL algorithms to resolve both types of the AA-POMDPs. Empirical results demonstrate that our approach achieves considerably better performance than existing POMDP-RL solutions.
Demystifying Technology for Policymaking: Exploring the Rideshare Context and Data Initiative Opportunities to Advance Tech Policymaking Efforts
Venue: arXiv cs.HC 2024
Year: 2024

In the face of rapidly advancing technologies, evidence of harms they can exacerbate, and insufficient policy to ensure accountability from tech companies, what are HCI opportunities for advancing policymaking of technology? In this paper, we explore challenges and opportunities for tech policymaking through a case study of app-based rideshare driving. We begin with background on rideshare platforms and how they operate. Next, we review literature on algorithmic management about how rideshare drivers actually experience platform features -- often to the detriment of their well-being -- and ways they respond. In light of this, researchers and advocates have called for increased worker protections, thus we turn to rideshare policy and regulation efforts in the U.S. Here, we differentiate the political strategies of platforms with those of drivers to illustrate the conflicting narratives policymakers face when trying to oversee gig work platforms. We reflect that past methods surfacing drivers' experiences may be insufficient for policymaker needs when developing oversight. To address this gap and our original inquiry -- what are HCI opportunities for advancing tech policymaking -- we briefly explore two paths forward for holding tech companies accountable in the rideshare context: (1) data transparency initiatives to enable collective auditing by workers and (2) legal frameworks for holding platforms accountable.