OpenRaiser/Intern-Atlas huggingface dataset:
Learning optimal controllers: a dynamical motion primitive approach
Venue: arXiv cs.RO 2023
Year: 2023

Real-time computation of optimal control is a challenging problem and, to solve this difficulty, many frameworks proposed to use learning techniques to learn (possibly sub-optimal) controllers and enable their usage in an online fashion. Among these techniques, the optimal motion framework is a simple, yet powerful technique, that obtained success in many complex real-world applications. The main idea of this approach is to take advantage of dynamic motion primitives, a widely used tool in robotics to learn trajectories from demonstrations. While usually these demonstrations come from humans, the optimal motion framework is based on demonstrations coming from optimal solutions, such as the ones obtained by numeric solvers. As usual in many learning techniques, a drawback of this approach is that it is hard to estimate the suboptimality of learned solutions, since finding easily computable and non-trivial upper bounds to the error between an optimal solution and a learned solution is, in general, unfeasible. However, we show in this paper that it is possible to estimate this error for a broad class of problems. Furthermore, we apply this estimation technique to achieve a novel and more efficient sampling scheme to be used within the optimal motion framework, enabling the usage of this framework in some scenarios where the computational resources are limited.
Finding Hamiltonian cycles with graph neural networks
Venue: arXiv cs.LG 2023
Year: 2023

We train a small message-passing graph neural network to predict Hamiltonian cycles on Erdős-Rényi random graphs in a critical regime. It outperforms existing hand-crafted heuristics after about 2.5 hours of training on a single GPU. Our findings encourage an alternative approach to solving computationally demanding (NP-hard) problems arising in practice. Instead of devising a heuristic by hand, one can train it end-to-end using a neural network. This has several advantages. Firstly, it is relatively quick and requires little problem-specific knowledge. Secondly, the network can adjust to the distribution of training samples, improving the performance on the most relevant problem instances. The model is trained using supervised learning on artificially created problem instances; this training procedure does not use an existing solver to produce the supervised signal. Finally, the model generalizes well to larger graph sizes and retains reasonable performance even on graphs eight times the original size.
PerfDetectiveAI -- Performance Gap Analysis and Recommendation in Software Applications
Venue: arXiv cs.SE 2023
Year: 2023

PerfDetectiveAI, a conceptual framework for performance gap analysis and suggestion in software applications is introduced in this research. For software developers, retaining a competitive edge and providing exceptional user experiences depend on maximizing application speed. But investigating cutting-edge approaches is necessary due to the complexity involved in determining performance gaps and creating efficient improvement tactics. Modern machine learning (ML) and artificial intelligence (AI) techniques are used in PerfDetectiveAI to monitor performance measurements and identify areas of underperformance in software applications. With the help of the framework, software developers and performance engineers should be able to enhance application performance and raise system productivity. It does this by utilizing sophisticated algorithms and utilizing sophisticated data analysis methodologies. Drawing on theoretical foundations from the fields of AI, ML and software engineering, PerfDetectiveAI envisions a sophisticated system capable of uncovering subtle performance discrepancies and identifying potential bottlenecks. PerfDetectiveAI aims to provide practitioners with data-driven recommendations to guide their decision-making processes by integrating advanced algorithms, statistical modelling, and predictive analytics. While PerfDetectiveAI is currently at the conceptual stage, this paper outlines the framework's fundamental principles, underlying methodologies and envisioned workflow. We want to encourage more research and development in the area of AI-driven performance optimization by introducing this conceptual framework, setting the foundation for the next developments in the quest for software excellence.
Ghosting the Machine: Judicial Resistance to a Recidivism Risk Assessment Instrument
Venue: arXiv cs.CY 2023
Year: 2023

Recidivism risk assessment instruments are presented as an 'evidence-based' strategy for criminal justice reform - a way of increasing consistency in sentencing, replacing cash bail, and reducing mass incarceration. In practice, however, AI-centric reforms can simply add another layer to the sluggish, labyrinthine machinery of bureaucratic systems and are met with internal resistance. Through a community-informed interview-based study of 23 criminal judges and other criminal legal bureaucrats in Pennsylvania, I find that judges overwhelmingly ignore a recently-implemented sentence risk assessment instrument, which they disparage as "useless," "worthless," "boring," "a waste of time," "a non-thing," and simply "not helpful." I argue that this algorithm aversion cannot be accounted for by individuals' distrust of the tools or automation anxieties, per the explanations given by existing scholarship. Rather, the instrument's non-use is the result of an interplay between three organizational factors: county-level norms about pre-sentence investigation reports; alterations made to the instrument by the Pennsylvania Sentencing Commission in response to years of public and internal resistance; and problems with how information is disseminated to judges. These findings shed new light on the important role of organizational influences on professional resistance to algorithms, which helps explain why algorithm-centric reforms can fail to have their desired effect. This study also contributes to an empirically-informed argument against the use of risk assessment instruments: they are resource-intensive and have not demonstrated positive on-the-ground impacts.
RoBERTweet: A BERT Language Model for Romanian Tweets
Venue: arXiv cs.CL 2023
Year: 2023

Developing natural language processing (NLP) systems for social media analysis remains an important topic in artificial intelligence research. This article introduces RoBERTweet, the first Transformer architecture trained on Romanian tweets. Our RoBERTweet comes in two versions, following the base and large architectures of BERT. The corpus used for pre-training the models represents a novelty for the Romanian NLP community and consists of all tweets collected from 2008 to 2022. Experiments show that RoBERTweet models outperform the previous general-domain Romanian and multilingual language models on three NLP tasks with tweet inputs: emotion detection, sexist language identification, and named entity recognition. We make our models and the newly created corpus of Romanian tweets freely available.
Parameter-free version of Adaptive Gradient Methods for Strongly-Convex Functions
Venue: arXiv cs.LG 2023
Year: 2023

The optimal learning rate for adaptive gradient methods applied to λ-strongly convex functions relies on the parameters λ and learning rate η. In this paper, we adapt a universal algorithm along the lines of Metagrad, to get rid of this dependence on λ and η. The main idea is to concurrently run multiple experts and combine their predictions to a master algorithm. This master enjoys O(d log T) regret bounds.
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning
Venue: arXiv cs.CV 2023
Year: 2023

Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources. Most existing methods explore different weighting strategies to obtain a powerful ensemble teacher, while ignoring the student with poor learning ability may not benefit from such specialized integrated knowledge. To address this problem, we propose Adaptive Multi-teacher Knowledge Distillation with Meta-Learning (MMKD) to supervise student with appropriate knowledge from a tailored ensemble teacher. With the help of a meta-weight network, the diverse yet compatible teacher knowledge in the output layer and intermediate layers is jointly leveraged to enhance the student performance. Extensive experiments on multiple benchmark datasets validate the effectiveness and flexibility of our methods. Code is available: https://github.com/Rorozhl/MMKD.
Well-Calibrated Probabilistic Predictive Maintenance using Venn-Abers
Venue: arXiv cs.LG 2023
Year: 2023

When using machine learning for fault detection, a common problem is the fact that most data sets are very unbalanced, with the minority class (a fault) being the interesting one. In this paper, we investigate the usage of Venn-Abers predictors, looking specifically at the effect on the minority class predictions. A key property of Venn-Abers predictors is that they output well-calibrated probability intervals. In the experiments, we apply Venn-Abers calibration to decision trees, random forests and XGBoost models, showing how both overconfident and underconfident models are corrected. In addition, the benefit of using the valid probability intervals produced by Venn-Abers for decision support is demonstrated. When using techniques producing opaque underlying models, e.g., random forest and XGBoost, each prediction will consist of not only the label, but also a valid probability interval, where the width is an indication of the confidence in the estimate. Adding Venn-Abers on top of a decision tree allows inspection and analysis of the model, to understand both the underlying relationship, and finding out in which parts of feature space that the model is accurate and/or confident.
On Minimizing the Impact of Dataset Shifts on Actionable Explanations
Venue: arXiv cs.LG 2023
Year: 2023

The Right to Explanation is an important regulatory principle that allows individuals to request actionable explanations for algorithmic decisions. However, several technical challenges arise when providing such actionable explanations in practice. For instance, models are periodically retrained to handle dataset shifts. This process may invalidate some of the previously prescribed explanations, thus rendering them unactionable. But, it is unclear if and when such invalidations occur, and what factors determine explanation stability i.e., if an explanation remains unchanged amidst model retraining due to dataset shifts. In this paper, we address the aforementioned gaps and provide one of the first theoretical and empirical characterizations of the factors influencing explanation stability. To this end, we conduct rigorous theoretical analysis to demonstrate that model curvature, weight decay parameters while training, and the magnitude of the dataset shift are key factors that determine the extent of explanation (in)stability. Extensive experimentation with real-world datasets not only validates our theoretical results, but also demonstrates that the aforementioned factors dramatically impact the stability of explanations produced by various state-of-the-art methods.
Proteinoid microspheres as proto-neural networks
Venue: arXiv cs.ET 2023
Year: 2023

Proteinoids, also known as thermal proteins, possess a fascinating ability to generate microspheres that exhibit electrical spikes resembling the action potentials of neurons. These spiking microspheres, referred to as protoneurons, hold the potential to assemble into proto-nano-brains. In our study, we investigate the feasibility of utilizing a promising electrochemical technique called differential pulse voltammetry (DPV) to interface with proteinoid nano-brains. We evaluate DPV's suitability by examining critical parameters such as selectivity, sensitivity, and linearity of the electrochemical responses. The research systematically explores the influence of various operational factors, including pulse width, pulse amplitude, scan rate, and scan time. Encouragingly, our findings indicate that DPV exhibits significant potential as an efficient electrochemical interface for proteinoid nano-brains. This technology opens up new avenues for developing artificial neural networks with broad applications across diverse fields of research.
An information-Theoretic Approach to Semi-supervised Transfer Learning
Venue: arXiv cs.LG 2023
Year: 2023

Transfer learning is a valuable tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest novel information-theoretic approaches for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during network training on the source dataset. Our theory suggests that one may improve the transferability of a deep neural network by incorporating regularization terms on the target data based on information-theoretic quantities, namely the Mutual Information and the Lautum Information. We demonstrate the effectiveness of the proposed approaches in various semi-supervised transfer learning experiments.
Implementing AI Ethics: Making Sense of the Ethical Requirements
Venue: arXiv cs.CY 2023
Year: 2023

Society's increasing dependence on Artificial Intelligence (AI) and AI-enabled systems require a more practical approach from software engineering (SE) executives in middle and higher-level management to improve their involvement in implementing AI ethics by making ethical requirements part of their management practices. However, research indicates that most work on implementing ethical requirements in SE management primarily focuses on technical development, with scarce findings for middle and higher-level management. We investigate this by interviewing ten Finnish SE executives in middle and higher-level management to examine how they consider and implement ethical requirements. We use ethical requirements from the European Union (EU) Trustworthy Ethics guidelines for Trustworthy AI as our reference for ethical requirements and an Agile portfolio management framework to analyze implementation. Our findings reveal a general consideration of privacy and data governance ethical requirements as legal requirements with no other consideration for ethical requirements identified. The findings also show practicable consideration of ethical requirements as technical robustness and safety for implementation as risk requirements and societal and environmental well-being for implementation as sustainability requirements. We examine a practical approach to implementing ethical requirements using the ethical risk requirements stack employing the Agile portfolio management framework.
IoT-AD: A Framework To Detect Anomalies Among Interconnected IoT Devices
Venue: arXiv cs.NI 2023
Year: 2023

In an Internet of Things (IoT) environment (e.g., smart home), several IoT devices may be available that are interconnected with each other. In such interconnected environments, a faulty or compromised IoT device could impact the operation of other IoT devices. In other words, anomalous behavior exhibited by an IoT device could propagate to other devices in an IoT environment. In this paper, we argue that mitigating the propagation of the anomalous behavior exhibited by a device to other devices is equally important to detecting this behavior in the first place. In line with this observation, we present a framework, called IoT Anomaly Detector (IoT-AD), that can not only detect the anomalous behavior of IoT devices, but also limit and recover from anomalous behavior that might have affected other devices. We implemented a prototype of IoT-AD, which we evaluated based on open-source IoT device datasets as well as through real-world deployment on a small-scale IoT testbed we have built. We have further evaluated IoT-AD in comparison to prior relevant approaches. Our evaluation results show that IoT-AD can identify anomalous behavior of IoT devices in less than 2.12 milliseconds and with up to 98% of accuracy.
On The Reliability Function of Discrete Memoryless Multiple-Access Channel with Feedback
Venue: arXiv cs.IT 2023
Year: 2023

The reliability function of a channel is the maximum achievable exponential rate of decay of the error probability as a function of the transmission rate. In this work, we derive bounds on the reliability function of discrete memoryless multiple-access channels (MAC) with noiseless feedback. We show that our bounds are tight for a variety of MACs, such as $m$-ary additive and two independent point-to-point channels. The bounds are expressed in terms of a new information measure called ``variable-length directed information". The upper bound is proved by analyzing stochastic processes defined based on the entropy of the message, given the past channel's outputs. Our method relies on tools from the theory of martingales, variable-length information measures, and a new technique called time pruning. We further propose a variable-length achievable scheme consisting of three phases: (i) data transmission, (ii) hybrid data-confirmation, and (iii) full confirmation. We show that two-phase-type schemes are strictly suboptimal in achieving the MAC's reliability function. Moreover, we study the shape of the lower-bound and show that it increases linearly with respect to a specific Euclidean distance measure defined between the transmission rate pair and the capacity boundary. As side results, we derive an upper bound on the capacity of MAC with noiseless feedback and study a new problem involving a hybrid of hypothesis testing and data transmission.
Kepler: Robust Learning for Faster Parametric Query Optimization
Venue: arXiv cs.DB 2023
Year: 2023

Most existing parametric query optimization (PQO) techniques rely on traditional query optimizer cost models, which are often inaccurate and result in suboptimal query performance. We propose Kepler, an end-to-end learning-based approach to PQO that demonstrates significant speedups in query latency over a traditional query optimizer. Central to our method is Row Count Evolution (RCE), a novel plan generation algorithm based on perturbations in the sub-plan cardinality space. While previous approaches require accurate cost models, we bypass this requirement by evaluating candidate plans via actual execution data and training an ML model to predict the fastest plan given parameter binding values. Our models leverage recent advances in neural network uncertainty in order to robustly predict faster plans while avoiding regressions in query performance. Experimentally, we show that Kepler achieves significant improvements in query runtime on multiple datasets on PostgreSQL.
CARNA: Characterizing Advanced heart failure Risk and hemodyNAmic phenotypes using learned multi-valued decision diagrams
Venue: arXiv cs.LG 2023
Year: 2023

Early identification of high risk heart failure (HF) patients is key to timely allocation of life-saving therapies. Hemodynamic assessments can facilitate risk stratification and enhance understanding of HF trajectories. However, risk assessment for HF is a complex, multi-faceted decision-making process that can be challenging. Previous risk models for HF do not integrate invasive hemodynamics or support missing data, and use statistical methods prone to bias or machine learning methods that are not interpretable. To address these limitations, this paper presents CARNA, a hemodynamic risk stratification and phenotyping framework for advanced HF that takes advantage of the explainability and expressivity of machine learned Multi-Valued Decision Diagrams (MVDDs). This interpretable framework learns risk scores that predict the probability of patient outcomes, and outputs descriptive patient phenotypes (sets of features and thresholds) that characterize each predicted risk score. CARNA incorporates invasive hemodynamics and can make predictions on missing data. The CARNA models were trained and validated using a total of five advanced HF patient cohorts collected from previous trials, and compared with six established HF risk scores and three traditional ML risk models. CARNA provides robust risk stratification, outperforming all previous benchmarks. Although focused on advanced HF, the CARNA framework is general purpose and can be used to learn risk stratifications for other diseases and medical applications.
Sticker820K: Empowering Interactive Retrieval with Stickers
Venue: arXiv cs.CV 2023
Year: 2023

Stickers have become a ubiquitous part of modern-day communication, conveying complex emotions through visual imagery. To facilitate the development of more powerful algorithms for analyzing stickers, we propose a large-scale Chinese sticker dataset, namely Sticker820K, which consists of 820k image-text pairs. Each sticker has rich and high-quality textual annotations, including descriptions, optical characters, emotional labels, and style classifications. Although vision-language tasks in the domain of natural images have been well studied, directly applying the those models, such as CLIP, to sticker data is not an optimal solution due to the discrepant nature between natural and emotive image data. Therefore, we propose StickerCLIP as a benchmark model on the Sticker820K dataset. For the text-to-image retrieval task, our StickerCLIP demonstrates strong superiority over the CLIP, which achieves an absolute gain of 66.0\% in mean recall on the Sticker820K test set. Additionally, we endeavor to extend the recently popularized LLM by means of prompt tuning, integrating its ability for sticker retrieval and allowing users to retrieve stickers through instructions. We validate the feasibility of this method, demonstrating the immense potential of prompt tuning in expanding LLM abilities while not affecting the quality of upstream tasks.
Boosting Breast Ultrasound Video Classification by the Guidance of Keyframe Feature Centers
Venue: arXiv cs.CV 2023
Year: 2023

Breast ultrasound videos contain richer information than ultrasound images, therefore it is more meaningful to develop video models for this diagnosis task. However, the collection of ultrasound video datasets is much harder. In this paper, we explore the feasibility of enhancing the performance of ultrasound video classification using the static image dataset. To this end, we propose KGA-Net and coherence loss. The KGA-Net adopts both video clips and static images to train the network. The coherence loss uses the feature centers generated by the static images to guide the frame attention in the video model. Our KGA-Net boosts the performance on the public BUSV dataset by a large margin. The visualization results of frame attention prove the explainability of our method. The codes and model weights of our method will be made publicly available.
Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization
Venue: arXiv cs.CL 2023
Year: 2023

Multilingual sentence representations are the foundation for similarity-based bitext mining, which is crucial for scaling multilingual neural machine translation (NMT) system to more languages. In this paper, we introduce MuSR: a one-for-all Multilingual Sentence Representation model that supports more than 220 languages. Leveraging billions of English-centric parallel corpora, we train a multilingual Transformer encoder, coupled with an auxiliary Transformer decoder, by adopting a multilingual NMT framework with CrossConST, a cross-lingual consistency regularization technique proposed in Gao et al. (2023). Experimental results on multilingual similarity search and bitext mining tasks show the effectiveness of our approach. Specifically, MuSR achieves superior performance over LASER3 (Heffernan et al., 2022) which consists of 148 independent multilingual sentence encoders.