Miracle-dz/UniFAR-FTUs huggingface dataset:
Parameterizing Activation Functions for Adversarial Robustness

Deep neural networks are known to be vulnerable to adversarially perturbed inputs. A commonly used defense is adversarial training, whose performance is influenced by model architecture. While previous works have studied the impact of varying model width and depth on robustness, the impact of using learnable parametric activation functions (PAFs) has not been studied. We study how using learnable PAFs can improve robustness in conjunction with adversarial training. We first ask the question: Can changing activation function shape improve robustness? To address this, we choose a set of PAFs with parameters that allow us to independently control behavior on negative inputs, inputs near zero, and positive inputs. Using these PAFs, we train models using adversarial training with fixed PAF shape parameter values. We find that all regions of PAF shape influence the robustness of obtained models, however only variation in certain regions (inputs near zero, positive inputs) can improve robustness over ReLU. We then combine learnable PAFs with adversarial training and analyze robust performance. We find that choice of activation function can significantly impact the robustness of the trained model. We find that only certain PAFs, such as smooth PAFs, are able to improve robustness significantly over ReLU. Overall, our work puts into context the importance of activation functions in adversarially trained models.
On Wave-Based Majority Gates with Cellular Automata

We demonstrate a discrete implementation of a wave-based majority gate in a chaotic Life-like cellular automaton. The gate functions via controlling of patterns' propagation into stationary channels. The gate presented is realisable in many living and non-living substrates that show wave-like activity of its space-time dynamics or pattern propagation. In the gate a symmetric pattern represents a binary value 0 while a non-symmetric pattern represents a binary value 1. Origination of the patterns and their symmetry type are encoded by the particle reactions at the beginning of computation. The patterns propagate in channels of the gate and compete for the space at the intersection of the channels. We implement 3-inputs majority gates using a W topology showing additional implementations of 5-inputs majority gates and one tree (cascade) majority gate.
Embedded-model flows: Combining the inductive biases of model-free deep learning and explicit probabilistic modeling

Normalizing flows have shown great success as general-purpose density estimators. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts of the models that fail to capture the statistics of the data. We demonstrate that EMFs can be used to induce desirable properties such as multimodality, hierarchical coupling and continuity. Furthermore, we show that EMFs enable a high performance form of variational inference where the structure of the prior model is embedded in the variational architecture. In our experiments, we show that this approach outperforms state-of-the-art methods in common structured inference problems.
Spatial data mining of public transport incidents reported in social media

Public transport agencies use social media as an essential tool for communicating mobility incidents to passengers. However, while the short term, day-to-day information about transport phenomena is usually posted in social media with low latency, its availability is short term as the content is rarely made an aggregated form. Social media communication of transport phenomena usually lacks GIS annotations as most social media platforms do not allow attaching non-POI GPS coordinates to posts. As a result, the analysis of transport phenomena information is minimal. We collected three years of social media posts of a polish public transport company with user comments. Through exploration, we infer a six-class transport information typology. We successfully build an information type classifier for social media posts, detect stop names in posts, and relate them to GPS coordinates, obtaining a spatial understanding of long-term aggregated phenomena. We show that our approach enables citizen science and use it to analyze the impact of three years of infrastructure incidents on passenger mobility, and the sentiment and reaction scale towards each of the events. All these results are achieved for Polish, an under-resourced language when it comes to spatial language understanding, especially in social media contexts. To improve the situation, we released two of our annotated data sets: social media posts with incident type labels and matched stop names and social media comments with the annotated sentiment. We also opensource the experimental codebase.
Temporal Abstraction in Reinforcement Learning with the Successor Representation

Reasoning at multiple levels of temporal abstraction is one of the key attributes of intelligence. In reinforcement learning, this is often modeled through temporally extended courses of actions called options. Options allow agents to make predictions and to operate at different levels of abstraction within an environment. Nevertheless, approaches based on the options framework often start with the assumption that a reasonable set of options is known beforehand. When this is not the case, there are no definitive answers for which options one should consider. In this paper, we argue that the successor representation (SR), which encodes states based on the pattern of state visitation that follows them, can be seen as a natural substrate for the discovery and use of temporal abstractions. To support our claim, we take a big picture view of recent results, showing how the SR can be used to discover options that facilitate either temporally-extended exploration or planning. We cast these results as instantiations of a general framework for option discovery in which the agent's representation is used to identify useful options, which are then used to further improve its representation. This results in a virtuous, never-ending, cycle in which both the representation and the options are constantly refined based on each other. Beyond option discovery itself, we also discuss how the SR allows us to augment a set of options into a combinatorially large counterpart without additional learning. This is achieved through the combination of previously learned options. Our empirical evaluation focuses on options discovered for exploration and on the use of the SR to combine them. The results of our experiments shed light on important design decisions involved in the definition of options and demonstrate the synergy of different methods based on the SR, such as eigenoptions and the option keyboard.
Sharing FANCI Features: A Privacy Analysis of Feature Extraction for DGA Detection

© Copyright held by the owner/author(s) 2021. This is the author’s version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in The Sixth International Conference on Cyber-Technologies and Cyber-Systems (CYBER 2021), https://www.thinkmind. org/index.php? view=article&articleid=cyber 2021 1 160 80095 Abstract—The goal of Domain Generation Algorithm (DGA) detection is to recognize infections with bot malware and is often done with help of Machine Learning approaches that classify non-resolving Domain Name System (DNS) traffic and are trained on possibly sensitive data. In parallel, the rise of privacy research in the Machine Learning world leads to privacy-preserving measures that are tightly coupled with a deep learning model’s architecture or training routine, while non deep learning approaches are commonly better suited for the application of privacy-enhancing methods outside the actual classification module. In this work, we aim to measure the privacy capability of the feature extractor of feature-based DGA detector FANCI (Feature-based Automated Nxdomain Classification and Intelligence). Our goal is to assess whether a data-rich adversary can learn an inverse mapping of FANCI’s feature extractor and thereby reconstruct domain names from feature vectors. Attack success would pose a privacy threat to sharing FANCI’s feature representation, while the opposite would enable this representation to be shared without privacy concerns. Using three real-world data sets, we train a recurrent Machine Learning model on the reconstruction task. Our approaches result in poor reconstruction performance and we attempt to back our findings with a mathematical review of the feature extraction process. We thus reckon that sharing FANCI’s feature representation does not constitute a considerable privacy leakage. Keywords—Data privacy; Intrusion detection; Machine learning.
Compact Object Modeling in the Globular Cluster 47 Tucanae

The globular cluster 47 Tucanae (47 Tuc) is one of the most massive star clusters in the Milky Way and is exceptionally rich in exotic stellar populations. For several decades it has been a favorite target of observers, and yet it is computationally very challenging to model because of its large number of stars (N ≳ 106) and high density. Here we present detailed and self-consistent 47 Tuc models computed with the Cluster Monte Carlo code (CMC). The models include all relevant dynamical interactions coupled to stellar and binary evolution, and reproduce various observations, including the surface brightness and velocity dispersion profiles, pulsar accelerations, and numbers of compact objects. We show that the present properties of 47 Tuc are best reproduced by adopting an initial stellar mass function that is both bottom-heavy and top-light relative to standard assumptions (as in, e.g., Kroupa 2001), and an initial Elson profile (Elson et al. 1987) that is overfilling the cluster’s tidal radius. We include new prescriptions in CMC for the formation of binaries through giant star collisions and tidal captures, and we show that these mechanisms play a crucial role in the formation of neutron star binaries and millisecond pulsars in 47 Tuc; our best-fit model contains ∼50 millisecond pulsars, 70% of which are formed through giant collisions and tidal captures. Our models also suggest that 47 Tuc presently contains up to ∼200 stellar-mass black holes, ∼5 binary black holes, ∼15 low-mass X-ray binaries, and ∼300 cataclysmic variables.
M EANINGFULLY EXPLAINING MODEL MISTAKES USING CONCEPTUAL COUNTERFACTUALS

Understanding and explaining the mistakes made by trained models is critical to many machine learning objectives, such as improving robustness, addressing concept drift, and mitigating biases. However, this is often an ad hoc process that involves manually looking at the model’s mistakes on many test samples and guessing at the underlying reasons for those incorrect predictions. In this paper, we propose a systematic approach, conceptual counterfactual explanations (CCE), that explains why a classiﬁer makes a mistake on a particular test sample(s) in terms of human-understandable concepts (e.g. this zebra is misclassiﬁed as a dog because of faint stripes ). We base CCE on two prior ideas: counterfactual explanations and concept activation vectors, and validate our approach on well-known pretrained models, showing that it explains the models’ mistakes meaningfully. In addition, for new models trained on data with spurious correlations, CCE accurately identiﬁes the spurious correlation as the cause of model mistakes from a single misclassiﬁed test sample. On two challenging medical applications, CCE generated useful insights, conﬁrmed by clinicians, into biases and mistakes the model makes in real-world settings. The code for CCE is publicly available and can easily be applied to explain mistakes in new models.
Privacy-Preserving Phishing Email Detection Based on Federated Learning and LSTM

Phishing emails that appear legitimate lure people into clicking on the attached malicious links or documents. Increasingly more sophisticated phishing campaigns in recent years necessitate a more adaptive detection system other than traditional signature-based methods. In this regard, natural language processing (NLP) with deep neural networks (DNNs) is adopted for knowledge acquisition from a large number of emails. However, such sensitive daily communications containing personal information are difficult to collect on a server for centralized learning in real life due to escalating privacy concerns. To this end, we propose a decentralized phishing email detection method called the Federated Phish Bowl (FPB) leveraging federated learning and long short-term memory (LSTM). FPB allows common knowledge representation and sharing among different clients through the aggregation of trained models to safeguard the email security and privacy. A recent phishing email dataset was collected from an intergovernmental organization to train the model. Moreover, we evaluated the model performance based on various assumptions regarding the total client number and the level of data heterogeneity. The comprehensive experimental results suggest that FPB is robust to a continually increasing client number and various data heterogeneity levels, retaining a detection accuracy of 0.83 and protecting the privacy of sensitive email communications.
An Activity Recognition Framework for Continuous Monitoring of Non-Steady-State Locomotion of Individuals with Parkinson's Disease

Fundamental knowledge in activity recognition of individuals with motor disorders such as Parkinson’s disease (PD) has been primarily limited to detection of steady-state/static tasks (e.g., sitting, standing, walking). To date, identification of non-steady-state locomotion on uneven terrains (stairs, ramps) has not received much attention. Furthermore, previous research has mainly relied on data from a large number of body locations which could adversely affect user convenience and system performance. Here, individuals with mild stages of PD and healthy subjects performed non-steady-state circuit trials comprising stairs, ramp, and changes of direction. An offline analysis using a linear discriminant analysis (LDA) classifier and a Long-Short Term Memory (LSTM) neural network was performed for task recognition. The performance of accelerographic and gyroscopic information from varied lower/upper-body segments were tested across a set of user-independent and user-dependent training paradigms. Comparing the F1 score of a given signal across classifiers showed improved performance using LSTM compared to LDA. Using LSTM, even a subset of information (e.g., feet data) in subject-independent training appeared to provide F1 score > 0.8. However, employing LDA was shown to be at the expense of being limited to using a subject-dependent training and/or biomechanical data from multiple body locations. The findings could inform a number of applications in the field of healthcare monitoring and developing advanced lower-limb assistive devices by providing insights into classification schemes capable of handling non-steady-state and unstructured locomotion in individuals with mild Parkinson’s disease.
Improved Pillar with Fine-grained Feature for 3D Object Detection

3D object detection with LiDAR point clouds plays an important role in autonomous driving perception module that requires high speed, stability and accuracy. However, the existing point-based methods are challenging to reach the speed requirements because of too many raw points, and the voxel-based methods are unable to ensure stable speed because of the 3D sparse convolution. In contrast, the 2D grid-based methods, such as PointPillar, can easily achieve a stable and efficient speed based on simple 2D convolution, but it is hard to get the competitive accuracy limited by the coarse-grained point clouds representation. So we propose an improved pillar with fine-grained feature based on PointPillar that can significantly improve detection accuracy. It consists of two modules, including height-aware sub-pillar and sparsity-based tiny-pillar, which get fine-grained representation respectively in the vertical and horizontal direction of 3D space. For height-aware sub-pillar, we introduce a height position encoding to keep height information of each sub-pillar during projecting to a 2D pseudo image. For sparsity-based tiny-pillar, we introduce sparsity-based CNN backbone stacked by dense feature and sparse attention module to extract feature with larger receptive field efficiently. Experimental results show that our proposed method significantly outperforms previous state-of-the-art 3D detection methods on the Waymo Open Dataset. The related code will be released to facilitate the academic and industrial study.
A Taxonomy and Archetypes of Business Analytics in Smart Manufacturing

Fueled by increasing data availability and the rise of technological advances for data processing and communication, business analytics is a key driver for smart manufacturing. However, due to the multitude of different local advances as well as its multidisciplinary complexity, both researchers and practitioners struggle to keep track of the progress and acquire new knowledge within the field, as there is a lack of a holistic conceptualization. To address this issue, we performed an extensive structured literature review, yielding 904 relevant hits, to develop a quadripartite taxonomy as well as to derive archetypes of business analytics in smart manufacturing. The taxonomy comprises the following meta-characteristics: application domain, orientation as the objective of the analysis, data origins, and analysis techniques. Collectively, they comprise eight dimensions with a total of 52 distinct characteristics. Using a cluster analysis, we found six archetypes that represent a synthesis of existing knowledge on planning, maintenance (reactive, offline, and online predictive), monitoring, and quality management. A temporal analysis highlights the push beyond predictive approaches and confirms that deep learning already dominates novel applications. Our results constitute an entry point to the field but can also serve as a reference work and a guide with which to assess the adequacy of one's own instruments.
DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer

Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.
A Deployment Model to Extend Ethically Aligned AI Implementation Method ECCOLA

There is a struggle in Artificial intelligence (AI) ethics to gain ground in actionable methods and models to be utilized by practitioners while developing and implementing ethically sound AI systems. AI ethics is a vague concept without a consensus of definition or theoretical grounding and bearing little connection to practice. Practice involving primarily technical tasks like software development is not aptly equipped to process and decide upon ethical considerations. Efforts to create tools and guidelines to help people working with AI development have been concentrating almost solely on the technical aspects of AI. A few exceptions do apply, such as the ECCOIA method for creating ethically aligned AI -systems. ECCOIA has proven results in terms of increased ethical considerations in AI systems development. Yet, it is a novel innovation, and room for development still exists. This study aims to extend ECCOIA with a deployment model to drive the adoption of ECCOIA, as any method - no matter how good -is of no value without adoption and use. The model includes simple metrics to facilitate the communication of ethical gaps or outcomes of ethical AI development. It offers the opportunity to assess any AI system at any given life-cycle phase, e.g., opening possibilities like analyzing the ethicality of an AI system under acquisition.
Event-Triggered Adaptive Control of a Parabolic PDE–ODE Cascade With Piecewise-Constant Inputs and Identification

We present an adaptive event-triggered boundary control scheme for a parabolic partial differential equation–ordinary differential equation (PDE–ODE) system, where the reaction coefficient of the parabolic PDE and the system parameter of a scalar ODE, are unknown. In the proposed controller, the parameter estimates, which are built by batch least-square identification, are recomputed and the plant states are resampled simultaneously. As a result, both the parameter estimates and the control input employ piecewise-constant values. In the closed-loop system, the following results are proved: 1) the absence of a Zeno phenomenon; 2) finite-time exact identification of the unknown parameters under most initial conditions of the plant (all initial conditions except a set of measure zero); and 3) exponential regulation of the plant states to zero. A simulation example is presented to validate the theoretical result.
Dynamics of Spin Helices in the 1D XX Model

Motivated by cold atom experiments and a desire to understand far-from-equilibrium quantum transport, we analytically study the dynamics of spin helices in the 1D XX model. We use a Jordan-Wigner transformation to map the spin chain onto a non-interacting Fermi gas with simple equations of motion. The resulting dynamics are non-trivial, however, as the spin-helix initial condition corresponds to a highly non-equilibrium distribution of the fermions. We find a separation of timescales between the in-plane and out-of-plane spin dynamics. We gain insights from analyzing the case of a uniform spin chain and from a semiclassical model. One of our key findings is that the spin correlation functions decay as t−1/2 at long time, in contrast to the experimentally-observed exponential decay.