mstyslavity/prisma_v3 huggingface dataset:
In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly on small open-sourced LLMs. In this work, we propose LeDex, a training framework that significantly improves the self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating a number of explanations and refinement trajectories from the LLM itself or a larger teacher model and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass@1 by up to 15.92% and pass@10 by 9.30% over four benchmarks. RL training brings additional up to 3.54% improvement on pass@1 and 2.55% improvement on pass@10. The trained LLMs show iterative refinement ability and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs in source code.
The growing interest in eXplainable Artificial Intelligence (XAI) has prompted research into models with built-in interpretability, the most prominent of which are part-prototype models. Part-Prototype Models (PPMs) make decisions by comparing an input image to a set of learned prototypes, providing human-understandable explanations in the form of ``this looks like that''. Despite their inherent interpretability, PPMS are not yet considered a valuable alternative to post-hoc models. In this survey, we investigate the reasons for this and provide directions for future research. We analyze papers from 2019 to 2024, and derive a taxonomy of the challenges that current PPMS face. Our analysis shows that the open challenges are quite diverse. The main concern is the quality and quantity of prototypes. Other concerns are the lack of generalization to a variety of tasks and contexts, and general methodological issues, including non-standardized evaluation. We provide ideas for future research in five broad directions: improving predictive performance, developing novel architectures grounded in theory, establishing frameworks for human-AI collaboration, aligning models with humans, and establishing metrics and benchmarks for evaluation. We hope that this survey will stimulate research and promote intrinsically interpretable models for application domains. Our list of surveyed papers is available at https://github.com/aix-group/ppm-survey.
The ability to generate artificial human movement patterns while meeting location and time constraints is an important problem in the security community, particularly as it enables the study of the analog problem of detecting such patterns while maintaining privacy. We frame this problem as an instance of abduction guided by a novel parsimony function represented as an aggregate truth value over an annotated logic program. This approach has the added benefit of affording explainability to an analyst user. By showing that any subset of such a program can provide a lower bound on this parsimony requirement, we are able to abduce movement trajectories efficiently through an informed (i.e., A*) search. We describe how our implementation was enhanced with the application of multiple techniques in order to be scaled and integrated with a cloud-based software stack that included bottom-up rule learning, geolocated knowledge graph retrieval/management, and interfaces with government systems for independently conducted government-run tests for which we provide results. We also report on our own experiments showing that we not only provide exact results but also scale to very large scenarios and provide realistic agent trajectories that can go undetected by machine learning anomaly detectors.
We study rate-induced phase-tipping (RP-tipping) between two stable limit cycles of a birhythmic oscillator. We say that such an oscillator RP-tips when a time variation of an input parameter preserves the bistability of the limit cycles but induces transitions from one stable limit cycle to the other, causing abrupt changes in the amplitude and frequency of the oscillations. Crucially, these transitions occur when: the rate of change of the input is in a certain interval bounded by critical rate(s), and the system is in certain phases of the cycle. We focus on two illustrative examples: the birhythmic van der Pol oscillator and the birhythmic Decroly-Goldbeter glycolysis model, each subjected to monotone and non-monotone shifts in their input parameters. We explain RP-tipping in terms of properties of the autonomous frozen system, including the phase of a cycle and partial basin instability along the parameter path traced by the changing input. We show that RP-tipping can occur as an irreversible one-way transition or as a series of transitions between the stable limit cycles. Finally, we present RP-tipping diagrams showing combinations of the rate and magnitude of parameter shifts and the phase of the oscillation that give rise to this genuine non-autonomous instability.
Concept-based explanations translate the internal representations of deep learning models into a language that humans are familiar with: concepts. One popular method for finding concepts is Concept Activation Vectors (CAVs), which are learnt using a probe dataset of concept exemplars. In this work, we investigate three properties of CAVs: (1) inconsistency across layers, (2) entanglement with other concepts, and (3) spatial dependency. Each property provides both challenges and opportunities in interpreting models. We introduce tools designed to detect the presence of these properties, provide insight into how each property can lead to misleading explanations, and provide recommendations to mitigate their impact. To demonstrate practical applications, we apply our recommendations to a melanoma classification task, showing how entanglement can lead to uninterpretable results and that the choice of negative probe set can have a substantial impact on the meaning of a CAV. Further, we show that understanding these properties can be used to our advantage. For example, we introduce spatially dependent CAVs to test if a model is translation invariant with respect to a specific concept and class. Our experiments are performed on natural images (ImageNet), skin lesions (ISIC 2019), and a new synthetic dataset, Elements. Elements is designed to capture a known ground truth relationship between concepts and classes. We release this dataset to facilitate further research in understanding and evaluating interpretability methods.
The pervasiveness of large language models and generative AI in online media has amplified the need for effective automated fact-checking to assist fact-checkers in tackling the increasing volume and sophistication of misinformation. The complex nature of fact-checking demands that automated fact-checking systems provide explanations that enable fact-checkers to scrutinise their outputs. However, it is unclear how these explanations should align with the decision-making and reasoning processes of fact-checkers to be effectively integrated into their workflows. Through semi-structured interviews with fact-checking professionals, we bridge this gap by: (i) providing an account of how fact-checkers assess evidence, make decisions, and explain their processes; (ii) examining how fact-checkers use automated tools in practice; and (iii) identifying fact-checker explanation requirements for automated fact-checking tools. The findings show unmet explanation needs and identify important criteria for replicable fact-checking explanations that trace the model's reasoning path, reference specific evidence, and highlight uncertainty and information gaps.
Explainability is a highly demanded requirement for applications in high-risk areas such as medicine. Vision Transformers have mainly been limited to attention extraction to provide insight into the model's reasoning. Our approach combines the high performance of Vision Transformers with the introduction of new explainability capabilities. We present HierViT, a Vision Transformer that is inherently interpretable and adapts its reasoning to that of humans. A hierarchical structure is used to process domain-specific features for prediction. It is interpretable by design, as it derives the target output with human-defined features that are visualized by exemplary images (prototypes). By incorporating domain knowledge about these decisive features, the reasoning is semantically similar to human reasoning and therefore intuitive. Moreover, attention heatmaps visualize the crucial regions for identifying each feature, thereby providing HierViT with a versatile tool for validating predictions. Evaluated on two medical benchmark datasets, LIDC-IDRI for lung nodule assessment and derm7pt for skin lesion classification, HierViT achieves superior and comparable prediction accuracy, respectively, while offering explanations that align with human reasoning.
In recent years, the rapid advancement of large language models (LLMs) in natural language processing has sparked significant interest among researchers to understand their mechanisms and functional characteristics. Although existing studies have attempted to explain LLM functionalities by identifying and interpreting specific neurons, these efforts mostly focus on individual neuron contributions, neglecting the fact that human brain functions are realized through intricate interaction networks. Inspired by cognitive neuroscience research on functional brain networks (FBNs), this study introduces a novel approach to investigate whether similar functional networks exist within LLMs. We use methods similar to those in the field of functional neuroimaging analysis to locate and identify functional networks in LLM. Experimental results show that, similar to the human brain, LLMs contain functional networks that frequently recur during operation. Further analysis shows that these functional networks are crucial for LLM performance. Masking key functional networks significantly impairs the model's performance, while retaining just a subset of these networks is adequate to maintain effective operation. This research provides novel insights into the interpretation of LLMs and the lightweighting of LLMs for certain downstream tasks. Code is available at https://github.com/WhatAboutMyStar/LLM_ACTIVATION.
With the continuous advancement of vision language models (VLMs) technology, remarkable research achievements have emerged in the dermatology field, the fourth most prevalent human disease category. However, despite these advancements, VLM still faces explainable problems to user in diagnosis due to the inherent complexity of dermatological conditions, existing tools offer relatively limited support for user comprehension. We propose SkinGEN, a diagnosis-to-generation framework that leverages the stable diffusion(SD) model to generate reference demonstrations from diagnosis results provided by VLM, thereby enhancing the visual explainability for users. Through extensive experiments with Low-Rank Adaptation (LoRA), we identify optimal strategies for skin condition image generation. We conduct a user study with 32 participants evaluating both the system performance and explainability. Results demonstrate that SkinGEN significantly improves users' comprehension of VLM predictions and fosters increased trust in the diagnostic process. This work paves the way for more transparent and user-centric VLM applications in dermatology and beyond.
Hadfield-Menell et al. (2017) propose the Off-Switch Game, a model of Human-AI cooperation in which AI agents always defer to humans because they are uncertain about our preferences. I explain two reasons why AI agents might not defer. First, AI agents might not value learning. Second, even if AI agents value learning, they might not be certain to learn our actual preferences.
In galaxies, the flattening of the spectrum at low radio frequencies below 300 MHz has been the subject of some debate. A turnover at low frequencies could be caused by multiple physical processes, which can yield new insights into the properties of the ionised gas in the interstellar medium. We investigate the existence and nature of the low-frequency turnover in the HII regions of M 101. We study the nearby galaxy M 101 using the LOw Frequency ARray (LOFAR) at frequencies of 54 and 144 MHz, Apertif at 1370 MHz, and published combined map from the Very Large Array (VLA) and Effelesberg telescope at 4850 MHz. The spectral index between 54 and 144 MHz is inverted at the centres of HII regions. We find a significant low-frequency flattening at the centres of five out of six HII regions that we selected for this study. The low frequency flattening in HII regions of M 101 can be explained with two different free-free absorption models. The flattening is localised in a region smaller than 1.5 kpc and can only be detected with high resolution (better than 45''). The detection of low frequency flattening has important consequences for using radio continuum observations below 100 MHz to measure extinction-free star-formation rates.
Graph Neural Networks (GNNs) are a powerful technique for machine learning on graph-structured data, yet they pose challenges in interpretability. Existing GNN explanation methods usually yield technical outputs, such as subgraphs and feature importance scores, that are difficult for non-data scientists to understand and thereby violate the purpose of explanations. Motivated by recent Explainable AI (XAI) research, we propose GraphXAIN, a method that generates natural language narratives explaining GNN predictions. GraphXAIN is a model- and explainer-agnostic method that uses Large Language Models (LLMs) to translate explanatory subgraphs and feature importance scores into coherent, story-like explanations of GNN decision-making processes. Evaluations on real-world datasets demonstrate GraphXAIN's ability to improve graph explanations. A survey of machine learning researchers and practitioners reveals that GraphXAIN enhances four explainability dimensions: understandability, satisfaction, convincingness, and suitability for communicating model predictions. When combined with another graph explainer method, GraphXAIN further improves trustworthiness, insightfulness, confidence, and usability. Notably, 95% of participants found GraphXAIN to be a valuable addition to the GNN explanation method. By incorporating natural language narratives, our approach serves both graph practitioners and non-expert users by providing clearer and more effective explanations.
Extreme Ultraviolet (EUV) driven atmospheric escape is a key process in the atmospheric evolution of close-in exoplanets. In many evolutionary models, the energy-limited mass-loss rate with a constant efficiency (typically $\sim10\%$) is assumed for calculating the mass-loss rate. However, hydrodynamic simulations have demonstrated that this efficiency depends on various stellar and planetary parameters. Comprehending the underlying physics of the efficiency is essential for understanding planetary atmospheric evolution and recent observations of the upper atmosphere of close-in exoplanets. We introduce relevant temperatures and timescales derived from physical principles to elucidate the mass-loss process. Our analytical mass-loss model is based on phenomenology and consistent across a range of planetary parameters. We compare our mass-loss efficiency and the radiation hydrodynamic simulations. The model can predict efficiency in both energy-limited and recombination-limited regimes. We further apply our model to exoplanets observed with hydrogen absorption (Ly$\alpha$ and H$\alpha$). Our findings suggest that Ly$\alpha$ absorption is detectable in planets subjected to intermediate EUV flux; under these conditions, the escaping outflow is insufficient in low-EUV environments, while the photoionization timescale remains short in high-EUV ranges. Conversely, H$\alpha$ absorption is detectable under high EUV flux conditions, facilitated by the intense Ly$\alpha$ flux exciting hydrogen atoms. According to our model, the non-detection of neutral hydrogen can be explained by a low mass-loss rate and is not necessarily due to stellar wind confinement or the absence of a hydrogen-dominated atmosphere in many cases. This model assists in identifying future observational targets and explicates the unusual absorption detection/non-detection patterns observed in recent studies.
We study a one-dimensional lattice system of free fermions subjected to a generalized measurement process: the system exchanges particles with its environment, but each fermion leaving or entering the system is counted. In contrast to the freezing of dynamics due to frequent measurements of lattice site occupation numbers, a high rate of fermion counts induces fast fluctuations in the state of the system. Still, through numerical simulations of quantum trajectories and an analytical approach based on replica Keldysh field theory, we find that instantaneous correlations and entanglement properties of free fermions subjected to fermion counting and local occupation measurements are strikingly similar. We explain this similarity through a generalized Zeno effect induced by fermion counting and a universal long-wavelength description in terms of a nonlinear sigma model. The physical requirements underlying this universal emergent behavior are conservation of the total number of particles in the system and its environment, and conservation of the purity of the state of the system by keeping a full record of all measurement outcomes. For both types of measurement processes, we present strong evidence against the existence of a critical phase with logarithmic entanglement and conformal invariance. Instead, we identify a finite critical range of length scales on which signatures of conformal invariance are observable. While area-law entanglement is established beyond a scale that is exponentially large in the measurement rate, the upper boundary of the critical range is only algebraically large and thus numerically accessible. Our finding that these properties do not rely on particle number conservation has far reaching implications for measurement-induced phenomena beyond noninteracting fermions, such as charge sharpening in random quantum circuits or generic interacting systems.
This paper reviews Trustworthy Artificial Intelligence (TAI) and its various definitions. Considering the principles respected in any society, TAI is often characterized by a few attributes, some of which have led to confusion in regulatory or engineering contexts. We argue against using terms such as Responsible or Ethical AI as substitutes for TAI. And to help clarify any confusion, we suggest leaving them behind. Given the subjectivity and complexity inherent in TAI, developing a universal framework is deemed infeasible. Instead, we advocate for approaches centered on addressing key attributes and properties such as fairness, bias, risk, security, explainability, and reliability. We examine the ongoing regulatory landscape, with a focus on initiatives in the EU, China, and the USA. We recognize that differences in AI regulations based on geopolitical and geographical reasons pose an additional challenge for multinational companies. We identify risk as a core factor in AI regulation and TAI. For example, as outlined in the EU-AI Act, organizations must gauge the risk level of their AI products to act accordingly (or risk hefty fines). We compare modalities of TAI implementation and how multiple cross-functional teams are engaged in the overall process. Thus, a brute force approach for enacting TAI renders its efficiency and agility, moot. To address this, we introduce our framework Set-Formalize-Measure-Act (SFMA). Our solution highlights the importance of transforming TAI-aware metrics, drivers of TAI, stakeholders, and business/legal requirements into actual benchmarks or tests. Finally, over-regulation driven by panic of powerful AI models can, in fact, harm TAI too. Based on GitHub user-activity data, in 2023, AI open-source projects rose to top projects by contributor account. Enabling innovation in TAI hinges on the independent contributions of the open-source community.
This paper describes MAIA, a Multimodal Automated Interpretability Agent. MAIA is a system that uses neural models to automate neural model understanding tasks like feature interpretation and failure mode discovery. It equips a pre-trained vision-language model with a set of tools that support iterative experimentation on subcomponents of other models to explain their behavior. These include tools commonly used by human interpretability researchers: for synthesizing and editing inputs, computing maximally activating exemplars from real-world datasets, and summarizing and describing experimental results. Interpretability experiments proposed by MAIA compose these tools to describe and explain system behavior. We evaluate applications of MAIA to computer vision models. We first characterize MAIA's ability to describe (neuron-level) features in learned representations of images. Across several trained models and a novel dataset of synthetic vision neurons with paired ground-truth descriptions, MAIA produces descriptions comparable to those generated by expert human experimenters. We then show that MAIA can aid in two additional interpretability tasks: reducing sensitivity to spurious features, and automatically identifying inputs likely to be mis-classified.
Social media is profoundly changing our society with its unprecedented spreading power. Due to the complexity of human behaviors and the diversity of massive messages, the information spreading dynamics are complicated, and the reported mechanisms are different and even controversial. Based on data from mainstream social media platforms, including WeChat, Weibo, and Twitter, cumulatively encompassing a total of 7.45 billion users, we uncover a ubiquitous mechanism that the information spreading dynamics are basically driven by the interplay of social reinforcement and social weakening effects. Accordingly, we propose a concise equation, which, surprisingly, can well describe all the empirical large-scale spreading trajectories. Our theory resolves a number of controversial claims and satisfactorily explains many phenomena previously observed. It also reveals that the highly clustered nature of social networks can lead to rapid and high-frequency information bursts with relatively small coverage per burst. This vital feature enables social media to have a high capacity and diversity for information dissemination, beneficial for its ecological development.