OpenRaiser/Intern-Atlas huggingface dataset:
Advancing human-centric AI for robust X-ray analysis through holistic self-supervised learning
Venue: arXiv cs.CV 2024
Year: 2024

AI Foundation models are gaining traction in various applications, including medical fields like radiology. However, medical foundation models are often tested on limited tasks, leaving their generalisability and biases unexplored. We present RayDINO, a large visual encoder trained by self-supervision on 873k chest X-rays. We compare RayDINO to previous state-of-the-art models across nine radiology tasks, from classification and dense segmentation to text generation, and provide an in depth analysis of population, age and sex biases of our model. Our findings suggest that self-supervision allows patient-centric AI proving useful in clinical workflows and interpreting X-rays holistically. With RayDINO and small task-specific adapters, we reach state-of-the-art results and improve generalization to unseen populations while mitigating bias, illustrating the true promise of foundation models: versatility and robustness.
"Sometimes You Just Gotta Risk It for the Biscuit": A Portrait of Student Risk-Taking
Venue: arXiv cs.SE 2024
Year: 2024

Understanding how individuals make decisions involving risk is a fundamental aspect of behavioral research. Despite the ubiquity of risk in various aspects of life, limited empirical work has explored student risk-taking behavior in computing education. This study aims to partially replicate prior research on risk-taking behavior in software engineers while focusing on students, shedding light on the factors that affect their risk-taking choices. In our work, students were presented with a hypothetical scenario related to meeting a course project deadline, where they had to choose between a risky option and a safer alternative. We examined several factors that might influence these choices, including the framing of the decision (as a potential gain or loss), students' enjoyment of programming, perceived difficulty of programming, and their performance in the course. Our findings reveal intriguing insights into student risk-taking behavior. Similar to software engineers in prior work, the framing of the decision significantly impacted the choices students made, with the loss framing leading to a higher likelihood for risky choices. Surprisingly, students displayed a greater inclination towards risk-taking compared to their professional counterparts in prior research. Furthermore, we observed that students' performance in the course and their enjoyment of programming had a subtle correlation with their risk-taking tendencies, with better-performing students and those who enjoyed programming being marginally more prone to taking risks. Notably, we did not find statistically significant correlations between perceived difficulty of programming and risk-taking behavior among students.
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Venue: arXiv cs.CL 2024
Year: 2024

Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to a thousand GPUs for training the largest open-source LLMs such as Nemotron 4 340B and Llama 3.1 405B. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner
Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models
Venue: arXiv cs.HC 2024
Year: 2024

Knowledge workers often need to extract and analyze information from a collection of documents to solve complex information tasks in the workplace, e.g., hiring managers reviewing resumes or analysts assessing risk in contracts. However, foraging for relevant information can become tedious and repetitive over many documents and criteria of interest. We introduce Marco, a mixed-initiative workspace supporting sensemaking over diverse business document collections. Through collection-centric assistance, Marco reduces the cognitive costs of extracting and structuring information, allowing users to prioritize comparative synthesis and decision making processes. Users interactively communicate their information needs to an AI assistant using natural language and compose schemas that provide an overview of a document collection. Findings from a usability study (n=16) demonstrate that when using Marco, users complete sensemaking tasks 16% more quickly, with less effort, and without diminishing accuracy. A design probe with seven domain experts identifies how Marco can benefit various real-world workflows.
Model-based Deep Learning for Wireless Resource Allocation in RSMA Communications Systems
Venue: arXiv cs.IT 2024
Year: 2024

Rate-splitting multiple access (RSMA) has been proven as an effective communication scheme for 5G and beyond. However, current approaches to RSMA resource management require complicated iterative algorithms, which cannot meet the stringent latency requirement by users with limited resources. Recently, data-driven methods are explored to alleviate this issue. However, they suffer from poor generalizability and scarce training data to achieve satisfactory performance. In this paper, we propose a fractional programming (FP) based deep unfolding (DU) approach to address resource allocation problem for a weighted sum rate optimization in RSMA. By carefully designing the penalty function, we couple the variable update with projected gradient descent algorithm (PGD). Following the structure of PGD, we embed a few learnable parameters in each layer of the DU network. Through extensive simulation, we have shown that the proposed model-based neural networks can yield similar results compared to the traditional optimization algorithm for RSMA resource management but with much lower computational complexity, less training data, and higher resilience to out-of-distribution (OOD) data.
A separability-based approach to quantifying generalization: which layer is best?
Venue: arXiv cs.LG 2024
Year: 2024

Generalization to unseen data remains poorly understood for deep learning classification and foundation models, especially in the open set scenario. How can one assess the ability of networks to adapt to new or extended versions of their input space in the spirit of few-shot learning, out-of-distribution generalization, domain adaptation, and category discovery? Which layers of a network are likely to generalize best? We provide a new method for evaluating the capacity of networks to represent a sampled domain, regardless of whether the network has been trained on all classes in that domain. Our approach is the following: after fine-tuning state-of-the-art pre-trained models for visual classification on a particular domain, we assess their performance on data from related but distinct variations in that domain. Generalization power is quantified as a function of the latent embeddings of unseen data from intermediate layers for both unsupervised and supervised settings. Working throughout all stages of the network, we find that (i) high classification accuracy does not imply high generalizability; and (ii) deeper layers in a model do not always generalize the best, which has implications for pruning. Since the trends observed across datasets are largely consistent, we conclude that our approach reveals (a function of) the intrinsic capacity of the different layers of a model to generalize. Our code is available at https://github.com/dyballa/generalization
Customizing Text-to-Image Models with a Single Image Pair
Venue: arXiv cs.CV 2024
Year: 2024

Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We ask if such an image pair can be used to customize a generative model to capture the demonstrated stylistic difference. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process. Unlike existing methods that learn to mimic a single concept from a collection of images, our method captures the stylistic difference between paired images. This allows us to apply a stylistic change without overfitting to the specific image content in the examples. To address this new task, we employ a joint optimization method that explicitly separates the style and content into distinct LoRA weight spaces. We optimize these style and content weights to reproduce the style and content images while encouraging their orthogonality. During inference, we modify the diffusion process via a new style guidance based on our learned weights. Both qualitative and quantitative experiments show that our method can effectively learn style while avoiding overfitting to image content, highlighting the potential of modeling such stylistic differences from a single image pair.
Early-stage detection of cognitive impairment by hybrid quantum-classical algorithm using resting-state functional MRI time-series
Venue: arXiv cs.LG 2024
Year: 2024

Following the recent development of quantum machine learning techniques, the literature has reported several quantum machine learning algorithms for disease detection. This study explores the application of a hybrid quantum-classical algorithm for classifying region-of-interest time-series data obtained from resting-state functional magnetic resonance imaging in patients with early-stage cognitive impairment based on the importance of cognitive decline for dementia or aging. Classical one-dimensional convolutional layers are used together with quantum convolutional neural networks in our hybrid algorithm. In the classical simulation, the proposed hybrid algorithms showed higher balanced accuracies than classical convolutional neural networks under the similar training conditions. Moreover, a total of nine brain regions (left precentral gyrus, right superior temporal gyrus, left rolandic operculum, right rolandic operculum, left parahippocampus, right hippocampus, left medial frontal gyrus, right cerebellum crus, and cerebellar vermis) among 116 brain regions were found to be relatively effective brain regions for the classification based on the model performances. The associations of the selected nine regions with cognitive decline, as found in previous studies, were additionally validated through seed-based functional connectivity analysis. We confirmed both the improvement of model performance with the quantum convolutional neural network and neuroscientific validities of brain regions from our hybrid quantum-classical model.
Digital Twin-Empowered Task Assignment in Aerial MEC Network: A Resource Coalition Cooperation Approach with Generative Model
Venue: arXiv cs.NI 2024
Year: 2024

To meet the demands for ubiquitous communication and temporary edge computing in 6G networks, aerial mobile edge computing (MEC) networks have been envisioned as a new paradigm. However, dynamic user requests pose challenges for task assignment strategies. Most of the existing research assumes that the strategy is deployed on ground-based stations or UAVs, which will be ineffective in an environment lacking infrastructure and continuous energy supply. Moreover, the resource mutual exclusion problem of dynamic task assignment has not been effectively solved. Toward this end, we introduce the digital twin (DT) into the aerial MEC network to study the resource coalition cooperation approach with the generative model (GM), which provides a preliminary coalition structure for the coalition game. Specifically, we propose a novel network framework that is composed of an application plane, a physical plane, and a virtual plane. After that, the task assignment problem is simplified to convex optimization programming with linear constraints. And then, we also propose a resource coalition cooperation approach that is based on a transferable utility (TU) coalition game to obtain an approximate optimal solution. Numerical results confirm the effectiveness of our proposed approach in terms of energy consumption and utilization of resources.
Copyright related risks in the creation and use of ML/AI systems
Venue: arXiv cs.SE 2024
Year: 2024

This paper summarizes the current copyright related risks that Machine Learning (ML) and Artificial Intelligence (AI) systems (including Large Language Models --LLMs) incur. These risks affect different stakeholders: owners of the copyright of the training data, the users of ML/AI systems, the creators of trained models, and the operators of AI systems. This paper also provides an overview of ongoing legal cases in the United States related to these risks.
Discrete Event Simulation: It's Easy with SimPy!
Venue: arXiv cs.MS 2024
Year: 2024

This paper introduces the practicalities and benefits of using SimPy, a discrete event simulation (DES) module written in Python, for modeling and simulating complex systems. Through a step-by-step exploration of the classical Dining Philosophers Problem, we demonstrate how SimPy enables the efficient construction of discrete event models, emphasizing system states, transitions, and event handling. We extend the scenario to introduce resources, such as chopsticks, to model contention and deadlock conditions, and showcase SimPy's capabilities in managing these scenarios. Furthermore, we explore the integration of SimPy with other Python libraries for statistical analysis, showcasing how simulation results inform system design and optimization. The versatility of SimPy is further highlighted through additional modeling scenarios, including resource constraints and customer service interactions, providing insights into the process of building, debugging, simulating, and optimizing models for a wide range of applications. This paper aims to make DES accessible to practitioners and researchers alike, emphasizing the ease with which complex simulations can be constructed, analyzed, and visualized using SimPy and the broader Python ecosystem.
Mitigating LLM Hallucinations via Conformal Abstention
Venue: arXiv cs.LG 2024
Year: 2024

We develop a principled procedure for determining when a large language model (LLM) should abstain from responding (e.g., by saying "I don't know") in a general domain, instead of resorting to possibly "hallucinating" a non-sensical or incorrect answer. Building on earlier approaches that use self-consistency as a more reliable measure of model confidence, we propose using the LLM itself to self-evaluate the similarity between each of its sampled responses for a given query. We then further leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate). Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets, while also maintaining a significantly less conservative abstention rate on a dataset with long responses (Temporal Sequences) compared to baselines using log-probability scores to quantify uncertainty, while achieveing comparable performance on a dataset with short answers (TriviaQA). To evaluate the experiments automatically, one needs to determine if two responses are equivalent given a question. Following standard practice, we use a thresholded similarity function to determine if two responses match, but also provide a method for calibrating the threshold based on conformal prediction, with theoretical guarantees on the accuracy of the match prediction, which might be of independent interest.
2HCDL: Holistic Human-Centered Development Lifecycle
Venue: arXiv cs.SE 2024
Year: 2024

The recent events affecting global society continuously highlight the need to change the development lifecycle of complex systems by promoting human-centered solutions that increase awareness and ensure critical properties such as security, safety, trust, transparency, and privacy. This fast abstract introduces the Holistic Human-Centered Development Lifecycle (2HCDL) methodology focused on: (i) the enforcement of human values and properties and (ii) the mitigation and prevention of critical issues for more secure, safe, trustworthy, transparent, and private development processes.
A Systematic Literature Review on Reasons and Approaches for Accurate Effort Estimations in Agile
Venue: arXiv cs.SE 2024
Year: 2024

Background: Accurate effort estimation is crucial for planning in Agile iterative development. Agile estimation generally relies on consensus-based methods like planning poker, which require less time and information than other formal methods (e.g., COSMIC) but are prone to inaccuracies. Understanding the common reasons for inaccurate estimations and how proposed approaches can assist practitioners is essential. However, prior systematic literature reviews (SLR) only focus on the estimation practices (e.g., [26, 127]) and the effort estimation approaches (e.g., [6]). Aim: We aim to identify themes of reasons for inaccurate estimations and classify approaches to improve effort estimation. Method: We conducted an SLR and identified the key themes and a taxonomy. Results: The reasons for inaccurate estimation are related to information quality, team, estimation practice, project management, and business influences. The effort estimation approaches were the most investigated in the literature, while only a few aim to support the effort estimation process. Yet, few automated approaches are at risk of data leakage and indirect validation scenarios. Recommendations: Practitioners should enhance the quality of information for effort estimation, potentially by adopting an automated approach. Future research should aim to improve the information quality, while avoiding data leakage and indirect validation scenarios.
Text and Audio Simplification: Human vs. ChatGPT
Venue: arXiv cs.CL 2024
Year: 2024

Text and audio simplification to increase information comprehension are important in healthcare. With the introduction of ChatGPT, an evaluation of its simplification performance is needed. We provide a systematic comparison of human and ChatGPT simplified texts using fourteen metrics indicative of text difficulty. We briefly introduce our online editor where these simplification tools, including ChatGPT, are available. We scored twelve corpora using our metrics: six text, one audio, and five ChatGPT simplified corpora. We then compare these corpora with texts simplified and verified in a prior user study. Finally, a medical domain expert evaluated these texts and five, new ChatGPT simplified versions. We found that simple corpora show higher similarity with the human simplified texts. ChatGPT simplification moves metrics in the right direction. The medical domain expert evaluation showed a preference for the ChatGPT style, but the text itself was rated lower for content retention.
Unifying and extending Precision Recall metrics for assessing generative models
Venue: arXiv cs.LG 2024
Year: 2024

With the recent success of generative models in image and text, the evaluation of generative models has gained a lot of attention. Whereas most generative models are compared in terms of scalar values such as Frechet Inception Distance (FID) or Inception Score (IS), in the last years (Sajjadi et al., 2018) proposed a definition of precision-recall curve to characterize the closeness of two distributions. Since then, various approaches to precision and recall have seen the light (Kynkaanniemi et al., 2019; Naeem et al., 2020; Park & Kim, 2023). They center their attention on the extreme values of precision and recall, but apart from this fact, their ties are elusive. In this paper, we unify most of these approaches under the same umbrella, relying on the work of (Simon et al., 2019). Doing so, we were able not only to recover entire curves, but also to expose the sources of the accounted pitfalls of the concerned metrics. We also provide consistency results that go well beyond the ones presented in the corresponding literature. Last, we study the different behaviors of the curves obtained experimentally.
Explaining models relating objects and privacy
Venue: arXiv cs.CV 2024
Year: 2024

Accurately predicting whether an image is private before sharing it online is difficult due to the vast variety of content and the subjective nature of privacy itself. In this paper, we evaluate privacy models that use objects extracted from an image to determine why the image is predicted as private. To explain the decision of these models, we use feature-attribution to identify and quantify which objects (and which of their features) are more relevant to privacy classification with respect to a reference input (i.e., no objects localised in an image) predicted as public. We show that the presence of the person category and its cardinality is the main factor for the privacy decision. Therefore, these models mostly fail to identify private images depicting documents with sensitive data, vehicle ownership, and internet activity, or public images with people (e.g., an outdoor concert or people walking in a public space next to a famous landmark). As baselines for future benchmarks, we also devise two strategies that are based on the person presence and cardinality and achieve comparable classification performance of the privacy models.
S4: Self-Supervised Sensing Across the Spectrum
Venue: arXiv cs.CV 2024
Year: 2024

Satellite image time series (SITS) segmentation is crucial for many applications like environmental monitoring, land cover mapping and agricultural crop type classification. However, training models for SITS segmentation remains a challenging task due to the lack of abundant training data, which requires fine grained annotation. We propose S4 a new self-supervised pre-training approach that significantly reduces the requirement for labeled training data by utilizing two new insights: (a) Satellites capture images in different parts of the spectrum such as radio frequencies, and visible frequencies. (b) Satellite imagery is geo-registered allowing for fine-grained spatial alignment. We use these insights to formulate pre-training tasks in S4. We also curate m2s2-SITS, a large-scale dataset of unlabeled, spatially-aligned, multi-modal and geographic specific SITS that serves as representative pre-training data for S4. Finally, we evaluate S4 on multiple SITS segmentation datasets and demonstrate its efficacy against competing baselines while using limited labeled data.