OpenRaiser/Intern-Atlas huggingface dataset:
Mesostructures: Beyond Spectrogram Loss in Differentiable Time-Frequency Analysis
Venue: arXiv cs.SD 2023
Year: 2023

Computer musicians refer to mesostructures as the intermediate levels of articulation between the microstructure of waveshapes and the macrostructure of musical forms. Examples of mesostructures include melody, arpeggios, syncopation, polyphonic grouping, and textural contrast. Despite their central role in musical expression, they have received limited attention in deep learning. Currently, autoencoders and neural audio synthesizers are only trained and evaluated at the scale of microstructure: i.e., local amplitude variations up to 100 milliseconds or so. In this paper, we formulate and address the problem of mesostructural audio modeling via a composition of a differentiable arpeggiator and time-frequency scattering. We empirically demonstrate that time--frequency scattering serves as a differentiable model of similarity between synthesis parameters that govern mesostructure. By exposing the sensitivity of short-time spectral distances to time alignment, we motivate the need for a time-invariant and multiscale differentiable time--frequency model of similarity at the level of both local spectra and spectrotemporal modulations.
A Practitioner's Guide to MDP Model Checking Algorithms
Venue: arXiv cs.LO 2023
Year: 2023

Model checking undiscounted reachability and expected-reward properties on Markov decision processes (MDPs) is key for the verification of systems that act under uncertainty. Popular algorithms are policy iteration and variants of value iteration; in tool competitions, most participants rely on the latter. These algorithms generally need worst-case exponential time. However the problem can equally be formulated as a linear program, solvable in polynomial time. In this paper, we give a detailed overview of today's state-of-the-art algorithms for MDP model checking with a focus on performance and correctness. We highlight their fundamental differences, and describe various optimisations and implementation variants. We experimentally compare floating-point and exact-arithmetic implementations of all algorithms on three benchmark sets using two probabilistic model checkers. Our results show that (optimistic) value iteration is a sensible default, but other algorithms are preferable in specific settings. This paper thereby provides a guide for MDP verification practitioners -- tool builders and users alike.
Digital Twins for Ports: Derived from Smart City and Supply Chain Twinning Experience
Venue: arXiv cs.CY 2023
Year: 2023

Ports are striving for innovative technological solutions to cope with the ever-increasing growth of transport, while at the same time improving their environmental footprint. An emerging technology that has the potential to substantially increase the efficiency of the multifaceted and interconnected port processes is the digital twin. Although digital twins have been successfully integrated in many industries, there is still a lack of cross-domain understanding of what constitutes a digital twin. Furthermore, the implementation of the digital twin in complex systems such as the port is still in its infancy. This paper attempts to fill this research gap by conducting an extensive cross-domain literature review of what constitutes a digital twin, keeping in mind the extent to which the respective findings can be applied to the port. It turns out that the digital twin of the port is most comparable to complex systems such as smart cities and supply chains, both in terms of its functional relevance as well as in terms of its requirements and characteristics. The conducted literature review, considering the different port processes and port characteristics, results in the identification of three core requirements of a digital port twin, which are described in detail. These include situational awareness, comprehensive data analytics capabilities for intelligent decision making, and the provision of an interface to promote multi-stakeholder governance and collaboration. Finally, specific operational scenarios are proposed on how the port's digital twin can contribute to energy savings by improving the use of port resources, facilities and operations.
Engaging with Researchers and Raising Awareness of FAIR and Open Science through the FAIR+ Implementation Survey Tool (FAIRIST)
Venue: arXiv cs.CY 2023
Year: 2023

Six years after the seminal paper on FAIR was published, researchers still struggle to understand how to implement FAIR. For many researchers FAIR promises long-term benefits for near-term effort, requires skills not yet acquired, and is one more thing in a long list of unfunded mandates and onerous requirements on scientists. Even for those required to or who are convinced they must make time for FAIR research practices, the preference is for just-in-time advice properly sized to the scientific artifacts and process. Because of the generality of most FAIR implementation guidance, it is difficult for a researcher to adjust the advice to their situation. Technological advances, especially in the area of artificial intelligence (AI) and machine learning (ML), complicate FAIR adoption as researchers and data stewards ponder how to make software, workflows, and models FAIR and reproducible. The FAIR+ Implementation Survey Tool (FAIRIST) mitigates the problem by integrating research requirements with research proposals in a systematic way. FAIRIST factors in new scholarly outputs such as nanopublications and notebooks, and the various research artifacts related to AI research (data, models, workflows, and benchmarks). Researchers step through a self-serve survey process and receive a table ready for use in their DMP and/or work plan while gaining awareness of the FAIR Principles and Open Science concepts. FAIRIST is a model that uses part of the proposal process as a way to do outreach, raise awareness of FAIR dimensions and considerations, while providing just-in-time assistance for competitive proposals.
Multilingual Multiaccented Multispeaker TTS with RADTTS
Venue: arXiv cs.SD 2023
Year: 2023

We work to create a multilingual speech synthesis system which can generate speech with the proper accent while retaining the characteristics of an individual voice. This is challenging to do because it is expensive to obtain bilingual training data in multiple languages, and the lack of such data results in strong correlations that entangle speaker, language, and accent, resulting in poor transfer capabilities. To overcome this, we present a multilingual, multiaccented, multispeaker speech synthesis model based on RADTTS with explicit control over accent, language, speaker and fine-grained $F_0$ and energy features. Our proposed model does not rely on bilingual training data. We demonstrate an ability to control synthesized accent for any speaker in an open-source dataset comprising of 7 accents. Human subjective evaluation demonstrates that our model can better retain a speaker's voice and accent quality than controlled baselines while synthesizing fluent speech in all target languages and accents in our dataset.
Online variable-weight scheduling with preempting on jobs with linear and exponential penalties
Venue: arXiv cs.DS 2023
Year: 2023

We analyze the problem of job scheduling with preempting on weighted jobs that can have either linear or exponential penalties. We review relevant literature on the problem and create and describe a few online algorithms that perform competitively with the optimal scheduler. We first describe a na{\" i}ve algorithm, which yields a high competitive ratio ($Ω(\frac{M}{s_{\min}})$) with the optimal, then provide an algorithm that yields a lower competitive ratio ($4\sqrt{\frac{M}{s_{\min}}} + n\log{\frac{Mn}{s_{\min}}}$). Finally, we make a minor modification to our algorithm to yield an algorithm that has an even better competitive ratio ($n\log{\frac{Mn}{s_{\min}}}$).
Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning
Venue: arXiv cs.LG 2023
Year: 2023

Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively in a data privacy-preserving manner. However, the data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model severely biased to the head classes with the majority of the training samples. To alleviate this issue, decoupled training has recently been introduced to FL, considering it has achieved promising results in centralized long-tailed learning by re-balancing the biased classifier after the instance-balanced training. However, the current study restricts the capacity of decoupled training in federated long-tailed learning with a sub-optimal classifier re-trained on a set of pseudo features, due to the unavailability of a global balanced dataset in FL. In this work, in order to re-balance the classifier more effectively, we integrate the local real data with the global gradient prototypes to form the local balanced datasets, and thus re-balance the classifier during the local training. Furthermore, we introduce an extra classifier in the training phase to help model the global data distribution, which addresses the problem of contradictory optimization goals caused by performing classifier re-balancing locally. Extensive experiments show that our method consistently outperforms the existing state-of-the-art methods in various settings.
BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testing
Venue: arXiv cs.CL 2023
Year: 2023

Deep neural networks (DNNs) and natural language processing (NLP) systems have developed rapidly and have been widely used in various real-world fields. However, they have been shown to be vulnerable to backdoor attacks. Specifically, the adversary injects a backdoor into the model during the training phase, so that input samples with backdoor triggers are classified as the target class. Some attacks have achieved high attack success rates on the pre-trained language models (LMs), but there have yet to be effective defense methods. In this work, we propose a defense method based on deep model mutation testing. Our main justification is that backdoor samples are much more robust than clean samples if we impose random mutations on the LMs and that backdoors are generalizable. We first confirm the effectiveness of model mutation testing in detecting backdoor samples and select the most appropriate mutation operators. We then systematically defend against three extensively studied backdoor attack levels (i.e., char-level, word-level, and sentence-level) by detecting backdoor samples. We also make the first attempt to defend against the latest style-level backdoor attacks. We evaluate our approach on three benchmark datasets (i.e., IMDB, Yelp, and AG news) and three style transfer datasets (i.e., SST-2, Hate-speech, and AG news). The extensive experimental results demonstrate that our approach can detect backdoor samples more efficiently and accurately than the three state-of-the-art defense approaches.
Flow-guided Semi-supervised Video Object Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

We propose an optical flow-guided approach for semi-supervised video object segmentation. Optical flow is usually exploited as additional guidance information in unsupervised video object segmentation. However, its relevance in semi-supervised video object segmentation has not been fully explored. In this work, we follow an encoder-decoder approach to address the segmentation task. A model to extract the combined information from optical flow and the image is proposed, which is then used as input to the target model and the decoder network. Unlike previous methods where concatenation is used to integrate information from image data and optical flow, a simple yet effective attention mechanism is exploited in our work. Experiments on DAVIS 2017 and YouTube-VOS 2019 show that by integrating the information extracted from optical flow into the original image branch results in a strong performance gain and our method achieves state-of-the-art performance.
AI Tool for Exploring How Economic Activities Impact Local Ecosystems
Venue: arXiv cs.MA 2023
Year: 2023

We present an AI-based ecosystem simulator that uses three-dimensional models of the terrain and animal models controlled by deep reinforcement learning. The simulations take place in a game engine environment, which enables continuous visual observation of the ecosystem model. The terrain models are generated from geographic data with altitudes and land cover type. The animal models combine three-dimensional conformation models with animation schemes and decision-making mechanisms trained with deep reinforcement learning in increasingly complex environments (curriculum learning). We show how AI tools of this kind can be used for modeling the development of specific ecosystems with and without different forms of economic activities. In particular, we show how they might be used for modeling local biodiversity effects of land cover change, exploitation of natural resources, pollution, invasive species, and climate change.
What are the Machine Learning best practices reported by practitioners on Stack Exchange?
Venue: arXiv cs.SE 2023
Year: 2023

Machine Learning (ML) is being used in multiple disciplines due to its powerful capability to infer relationships within data. In particular, Software Engineering (SE) is one of those disciplines in which ML has been used for multiple tasks, like software categorization, bugs prediction, and testing. In addition to the multiple ML applications, some studies have been conducted to detect and understand possible pitfalls and issues when using ML. However, to the best of our knowledge, only a few studies have focused on presenting ML best practices or guidelines for the application of ML in different domains. In addition, the practices and literature presented in previous literature (i) are domain-specific (e.g., concrete practices in biomechanics), (ii) describe few practices, or (iii) the practices lack rigorous validation and are presented in gray literature. In this paper, we present a study listing 127 ML best practices systematically mining 242 posts of 14 different Stack Exchange (STE) websites and validated by four independent ML experts. The list of practices is presented in a set of categories related to different stages of the implementation process of an ML-enabled system; for each practice, we include explanations and examples. In all the practices, the provided examples focus on SE tasks. We expect this list of practices could help practitioners to understand better the practices and use ML in a more informed way, in particular newcomers to this new area that sits at the intersection of software engineering and machine learning.
Variation-Aware Semantic Image Synthesis
Venue: arXiv cs.CV 2023
Year: 2023

Semantic image synthesis (SIS) aims to produce photorealistic images aligning to given conditional semantic layout and has witnessed a significant improvement in recent years. Although the diversity in image-level has been discussed heavily, class-level mode collapse widely exists in current algorithms. Therefore, we declare a new requirement for SIS to achieve more photorealistic images, variation-aware, which consists of inter- and intra-class variation. The inter-class variation is the diversity between different semantic classes while the intra-class variation stresses the diversity inside one class. Through analysis, we find that current algorithms elusively embrace the inter-class variation but the intra-class variation is still not enough. Further, we introduce two simple methods to achieve variation-aware semantic image synthesis (VASIS) with a higher intra-class variation, semantic noise and position code. We combine our method with several state-of-the-art algorithms and the experimental result shows that our models generate more natural images and achieves slightly better FIDs and/or mIoUs than the counterparts. Our codes and models will be publicly available.
An Approximate Algorithm for Maximum Inner Product Search over Streaming Sparse Vectors
Venue: arXiv cs.IR 2023
Year: 2023

Maximum Inner Product Search or top-k retrieval on sparse vectors is well-understood in information retrieval, with a number of mature algorithms that solve it exactly. However, all existing algorithms are tailored to text and frequency-based similarity measures. To achieve optimal memory footprint and query latency, they rely on the near stationarity of documents and on laws governing natural languages. We consider, instead, a setup in which collections are streaming -- necessitating dynamic indexing -- and where indexing and retrieval must work with arbitrarily distributed real-valued vectors. As we show, existing algorithms are no longer competitive in this setup, even against naive solutions. We investigate this gap and present a novel approximate solution, called Sinnamon, that can efficiently retrieve the top-k results for sparse real valued vectors drawn from arbitrary distributions. Notably, Sinnamon offers levers to trade-off memory consumption, latency, and accuracy, making the algorithm suitable for constrained applications and systems. We give theoretical results on the error introduced by the approximate nature of the algorithm, and present an empirical evaluation of its performance on two hardware platforms and synthetic and real-valued datasets. We conclude by laying out concrete directions for future research on this general top-k retrieval problem over sparse vectors.
Finite alphabet phase retrieval
Venue: arXiv cs.IT 2023
Year: 2023

We consider the finite alphabet phase retrieval problem: recovering a signal whose entries lie in a small alphabet of possible values from its Fourier magnitudes. This problem arises in the celebrated technology of X-ray crystallography to determine the atomic structure of biological molecules. Our main result states that for generic values of the alphabet, two signals have the same Fourier magnitudes if and only if several partitions have the same difference sets. Thus, the finite alphabet phase retrieval problem reduces to the combinatorial problem of determining a signal from those difference sets. Notably, this result holds true when one of the letters of the alphabet is zero, namely, for sparse signals with finite alphabet, which is the situation in X-ray crystallography.
Implicit Shape Model Trees: Recognition of 3-D Indoor Scenes and Prediction of Object Poses for Mobile Robots
Venue: arXiv cs.RO 2023
Year: 2023

We present an approach for mobile robots to recognize scenes in object arrangements distributed across cluttered environments. Recognition is enabled by intertwining the robot's search for objects and the assignment of found objects to scenes. Our scene model called "Implicit Shape Model (ISM) trees" allows these two tasks to be solved jointly. This article presents novel algorithms for ISM trees to recognize scenes and predict poses of searched objects. We define scenes as object sets in which some objects are connected via 3-D spatial relations. In previous work, we recognized scenes with single ISMs. However, single ISMs are prone to false positives. As a remedy, we have developed ISM trees, a hierarchical model consisting of multiple ISMs. This article contributes a recognition algorithm that now enables the use of ISM trees for scene recognition. ISM trees should be ideally generated from human demonstrations of object arrangements. As a suitable algorithm was not available, we introduce such a generation algorithm. In line with the active vision paradigm, we combined scene recognition and object search in previous work. However, an efficient algorithm was lacking to make this combination effective. Physical experiments show that this is now overcome with a new algorithm achieving efficient combination through predicted object poses.
An Overview on Cloud Distributed Databases for Business Environments
Venue: arXiv cs.DB 2023
Year: 2023

Cloud-based distributed databases are a popular choice for many current applications, especially those that run over the Internet. By incorporating distributed database systems within cloud environments, it has enabled businesses to scale operations to a global level, all while achieving desired standards of system reliability, availability, and responsiveness. Cloud providers offer infrastructure and management tools for distributed databases as Database-as-a-Service (DBaaS), re-purposing the investment by businesses towards database services. This paper reviews the functionality of these services, by highlighting Amazon Relational Data Service (RDS), suited for handling relational distributed databases.
A Majority Logic Synthesis Framework For Single Flux Quantum Circuits
Venue: arXiv cs.LO 2023
Year: 2023

Exascale computing and its associated applications have required increasing degrees of efficiency. Semiconductor-Transistor-based Circuits (STbCs) have struggled with increasing the GHz frequency while dealing with power dissipation issues. Emerging as an alternative to STbC, single flux quantum (SFQ) logic in the superconducting electrons (SCE) technology promises higher-speed clock frequencies at ultra-low power consumption. However, its quantized pulse-based operation and high environmental requirements, process variations and other SFQ-specific non-idealities are the significant causes of logic error for SFQ circuits. A suitable method of minimizing the impact of the afore-mentioned error sources is to minimize the number of Josephson Junctions (JJs) in the circuits, hence an essential part of the design flow of large SFQ circuits. This paper presents a novel SFQ logic synthesis framework that given a netlist, offers an automated mapping solution including majority (MAJ) logic with the goal of minimizing the number of JJs, while catering to the unique characteristics and requirements of the design. Our experiments confirm that our synthesis framework significantly outperforms the state-of-the-art academic SFQ technology mapper, namely reducing the number of JJs on average by 35.0%.
Set-Theoretic and Type-Theoretic Ordinals Coincide
Venue: arXiv cs.LO 2023
Year: 2023

In constructive set theory, an ordinal is a hereditarily transitive set. In homotopy type theory (HoTT), an ordinal is a type with a transitive, wellfounded, and extensional binary relation. We show that the two definitions are equivalent if we use (the HoTT refinement of) Aczel's interpretation of constructive set theory into type theory. Following this, we generalize the notion of a type-theoretic ordinal to capture all sets in Aczel's interpretation rather than only the ordinals. This leads to a natural class of ordered structures which contains the type-theoretic ordinals and realizes the higher inductive interpretation of set theory. All our results are formalized in Agda.
Reflective Artificial Intelligence
Venue: arXiv cs.AI 2023
Year: 2023

Artificial Intelligence (AI) is about making computers that do the sorts of things that minds can do, and as we progress towards this goal, we tend to increasingly delegate human tasks to machines. However, AI systems usually do these tasks with an unusual imbalance of insight and understanding: new, deeper insights are present, yet many important qualities that a human mind would have previously brought to the activity are utterly absent. Therefore, it is crucial to ask which features of minds have we replicated, which are missing, and if that matters. One core feature that humans bring to tasks, when dealing with the ambiguity, emergent knowledge, and social context presented by the world, is reflection. Yet this capability is utterly missing from current mainstream AI. In this paper we ask what reflective AI might look like. Then, drawing on notions of reflection in complex systems, cognitive science, and agents, we sketch an architecture for reflective AI agents, and highlight ways forward.