OpenRaiser/Intern-Atlas huggingface dataset:
Over-the-Air Ensemble Inference with Model Privacy
Venue: arXiv cs.LG 2022
Year: 2022

We consider distributed inference at the wireless edge, where multiple clients with an ensemble of models, each trained independently on a local dataset, are queried in parallel to make an accurate decision on a new sample. In addition to maximizing inference accuracy, we also want to maximize the privacy of local models. We exploit the superposition property of the air to implement bandwidth-efficient ensemble inference methods. We introduce different over-the-air ensemble methods and show that these schemes perform significantly better than their orthogonal counterparts, while using less resources and providing privacy guarantees. We also provide experimental results verifying the benefits of the proposed over-the-air inference approach, whose source code is shared publicly on Github.
Existence and perception as the basis of AGI (Artificial General Intelligence)
Venue: arXiv cs.AI 2022
Year: 2022

As is known, AGI (Artificial General Intelligence), unlike AI, should operate with meanings. And that's what distinguishes it from AI. Any successful AI implementations (playing chess, unmanned driving, face recognition etc.) do not operate with the meanings of the processed objects in any way and do not recognize the meaning. And they don't need to. But for AGI, which emulates human thinking, this ability is crucial. Numerous attempts to define the concept of "meaning" have one very significant drawback - all such definitions are not strict and formalized, so they cannot be programmed. The meaning search procedure should use a formalized description of its existence and possible forms of its perception. For the practical implementation of AGI, it is necessary to develop such "ready-to-code" descriptions in the context of their use for processing the related cognitive concepts of "meaning" and "knowledge". An attempt to formalize the definition of such concepts is made in this article.
Efficient Autoprecoder-based deep learning for massive MU-MIMO Downlink under PA Non-Linearities
Venue: arXiv cs.NI 2022
Year: 2022

This paper introduces a new efficient autoprecoder (AP) based deep learning approach for massive multiple-input multiple-output (mMIMO) downlink systems in which the base station is equipped with a large number of antennas with energy-efficient power amplifiers (PAs) and serves multiple user terminals. We present AP-mMIMO, a new method that jointly eliminates the multiuser interference and compensates the severe nonlinear (NL) PA distortions. Unlike previous works, AP-mMIMO has a low computational complexity, making it suitable for a global energy-efficient system. Specifically, we aim to design the PA-aware precoder and the receive decoder by leveraging the concept of autoprecoder, whereas the end-to-end massive multiuser (MU)-MIMO downlink is designed using a deep neural network (NN). Most importantly, the proposed AP-mMIMO is suited for the varying block fading channel scenario. To deal with such scenarios, we consider a two-stage precoding scheme: 1) a NN-precoder is used to address the PA non-linearities and 2) a linear precoder is used to suppress the multiuser interference. The NN-precoder and the receive decoder are trained off-line and when the channel varies, only the linear precoder changes on-line. This latter is designed by using the widely used zero-forcing precoding scheme or its lowcomplexity version based on matrix polynomials. Numerical simulations show that the proposed AP-mMIMO approach achieves competitive performance with a significantly lower complexity compared to existing literature. Index Terms-multiuser (MU) precoding, massive multipleinput multiple-output (MIMO), energy-efficiency, hardware impairment, power amplifier (PA) nonlinearities, autoprecoder, deep learning, neural network (NN)
A Conditional Perspective on the Logic of Iterated Belief Contraction
Venue: arXiv cs.AI 2022
Year: 2022

In this article, we consider iteration principles for contraction, with the goal of identifying properties for contractions that respect conditional beliefs. Therefore, we investigate and evaluate four groups of iteration principles for contraction which consider the dynamics of conditional beliefs. For all these principles, we provide semantic characterization theorems and provide formulations by postulates which highlight how the change of beliefs and of conditional beliefs is constrained, whenever that is possible. The first group is similar to the syntactic Darwiche-Pearl postulates. As a second group, we consider semantic postulates for iteration of contraction by Chopra, Ghose, Meyer and Wong, and by Konieczny and Pino Pérez, respectively, and we provide novel syntactic counterparts. Third, we propose a contraction analogue of the independence condition by Jin and Thielscher. For the fourth group, we consider natural and moderate contraction by Nayak. Methodically, we make use of conditionals for contractions, so-called contractionals and furthermore, we propose and employ the novel notion of $ α$-equivalence for formulating some of the new postulates.
Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery
Venue: arXiv cs.LG 2022
Year: 2022

We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual modalities are embedded into a shared latent space and fused through a product of experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture of expert decoder imposing inductive biases encoding prior scientific knowledge and imparting structured disentanglement of the latent space. This approach enables discovery of fingerprints which may be detected in high-dimensional heterogeneous datasets, avoiding traditional bottlenecks related to high-fidelity measurement and characterization. Motivated by accelerated co-design and optimization of materials manufacturing processes, a dataset of lattice metamaterials from metal additive manufacturing demonstrates accurate cross modal inference between images of mesoscale topology and mechanical stress-strain response.
Online Deep Neural Network for Optimization in Wireless Communications
Venue: arXiv cs.IT 2022
Year: 2022

Recently, deep neural network (DNN) has been widely adopted in the design of intelligent communication systems thanks to its strong learning ability and low testing complexity. However, most current offline DNN-based methods still suffer from unsatisfactory performance, limited generalization ability, and poor interpretability. In this article, we propose an online DNN-based approach to solve general optimization problems in wireless communications, where a dedicated DNN is trained for each data sample. By treating the optimization variables and the objective function as network parameters and loss function, respectively, the optimization problem can be solved equivalently through network training. Thanks to the online optimization nature and meaningful network parameters, the proposed approach owns strong generalization ability and interpretability, while its superior performance is demonstrated through a practical example of joint beamforming in intelligent reflecting surface (IRS)-aided multi-user multiple-input multiple-output (MIMO) systems. Simulation results show that the proposed online DNN outperforms conventional offline DNN and state-of-the-art iterative optimization algorithm, but with low complexity.
Confidence Guided Depth Completion Network
Venue: arXiv cs.CV 2022
Year: 2022

The paper proposes an image-guided depth completion method to estimate accurate dense depth maps with fast computation time. The proposed network has two-stage structure. The first stage predicts a first depth map. Then, the second stage further refines the first depth map using the confidence maps. The second stage consists of two layers, each of which focuses on different regions and generates a refined depth map and a confidence map. The final depth map is obtained by combining two depth maps from the second stage using the corresponding confidence maps. Compared with the top-ranked models on the KITTI depth completion online leaderboard, the proposed model shows much faster computation time and competitive performance.
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration
Venue: arXiv cs.NE 2022
Year: 2022

It has long been observed that the performance of evolutionary algorithms and other randomized search heuristics can benefit from a non-static choice of the parameters that steer their optimization behavior. Mechanisms that identify suitable configurations on the fly ("parameter control") or via a dedicated training process ("dynamic algorithm configuration") are therefore an important component of modern evolutionary computation frameworks. Several approaches to address the dynamic parameter setting problem exist, but we barely understand which ones to prefer for which applications. As in classical benchmarking, problem collections with a known ground truth can offer very meaningful insights in this context. Unfortunately, settings with well-understood control policies are very rare. One of the few exceptions for which we know which parameter settings minimize the expected runtime is the LeadingOnes problem. We extend this benchmark by analyzing optimal control policies that can select the parameters only from a given portfolio of possible values. This also allows us to compute optimal parameter portfolios of a given size. We demonstrate the usefulness of our benchmarks by analyzing the behavior of the DDQN reinforcement learning approach for dynamic algorithm configuration.
Ransomware: Analysing the Impact on Windows Active Directory Domain Services
Venue: arXiv cs.CR 2022
Year: 2022

Ransomware has become an increasingly popular type of malware across the past decade and continues to rise in popularity due to its high profitability. Organisations and enterprises have become prime targets for ransomware as they are more likely to succumb to ransom demands as part of operating expenses to counter the cost incurred from downtime. Despite the prevalence of ransomware as a threat towards organisations, there is very little information outlining how ransomware affects Windows Server environments, and particularly its proprietary domain services such as Active Directory. Hence, we aim to increase the cyber situational awareness of organisations and corporations that utilise these environments. Dynamic analysis was performed using three ransomware variants to uncover how crypto-ransomware affects Windows Server-specific services and processes. Our work outlines the practical investigation undertaken as WannaCry, TeslaCrypt, and Jigsaw were acquired and tested against several domain services. The findings showed that none of the three variants stopped the processes and decidedly left all domain services untouched. However, although the services remained operational, they became uniquely dysfunctional as ransomware encrypted the files pertaining to those services
Approximation error of single hidden layer neural networks with fixed weights
Venue: arXiv cs.LG 2022
Year: 2022

This paper provides an explicit formula for the approximation error of single hidden layer neural networks with two fixed weights.
Composable and Modular Code Generation in MLIR: A Structured and Retargetable Approach to Tensor Compiler Construction
Venue: arXiv cs.PL 2022
Year: 2022

Despite significant investment in software infrastructure, machine learning systems, runtimes and compilers do not compose properly. We propose a new design aiming at providing unprecedented degrees of modularity, composability and genericity. This paper discusses a structured approach to the construction of domain-specific code generators for tensor compilers, with the stated goal of improving the productivity of both compiler engineers and end-users. The approach leverages the natural structure of tensor algebra. It has been the main driver for the design of progressive lowering paths in \MLIR. The proposed abstractions and transformations span data structures and control flow with both functional (SSA form) and imperative (side-effecting) semantics. We discuss the implications of this infrastructure on compiler construction and present preliminary experimental results.
Differential Privacy for Symbolic Systems with Application to Markov Chains
Venue: arXiv cs.CR 2022
Year: 2022

Data-driven systems are gathering increasing amounts of data from users, and sensitive user data requires privacy protections. In some cases, the data gathered is non-numerical or symbolic, and conventional approaches to privacy, e.g., adding noise, do not apply, though such systems still require privacy protections. Accordingly, we present a novel differential privacy framework for protecting trajectories generated by symbolic systems. These trajectories can be represented as words or strings over a finite alphabet. We develop new differential privacy mechanisms that approximate a sensitive word using a random word that is likely to be near it. An offline mechanism is implemented efficiently using a Modified Hamming Distance Automaton to generate whole privatized output words over a finite time horizon. Then, an online mechanism is implemented by taking in a sensitive symbol and generating a randomized output symbol at each timestep. This work is extended to Markov chains to generate differentially private state sequences that a given Markov chain could have produced. Statistical accuracy bounds are developed to quantify the accuracy of these mechanisms, and numerical results validate the accuracy of these techniques for strings of English words.
Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery
Venue: arXiv cs.LG 2022
Year: 2022

Countless signal processing applications include the reconstruction of signals from few indirect linear measurements. The design of effective measurement operators is typically constrained by the underlying hardware and physics, posing a challenging and often even discrete optimization task. While the potential of gradient-based learning via the unrolling of iterative recovery algorithms has been demonstrated, it has remained unclear how to leverage this technique when the set of admissible measurement operators is structured and discrete. We tackle this problem by combining unrolled optimization with Gumbel reparametrizations, which enable the computation of low-variance gradient estimates of categorical random variables. Our approach is formalized by GLODISMO (Gradient-based Learning of DIscrete Structured Measurement Operators). This novel method is easy-to-implement, computationally efficient, and extendable due to its compatibility with automatic differentiation. We empirically demonstrate the performance and flexibility of GLODISMO in several prototypical signal recovery applications, verifying that the learned measurement matrices outperform conventional designs based on randomization as well as discrete optimization baselines.
Link Prediction of Artificial Intelligence Concepts using Low Computational Power
Venue: arXiv cs.SI 2022
Year: 2022

This paper presents an approach proposed for the Science4cast 2021 competition, organized by the Institute of Advanced Research in Artificial Intelligence, whose main goal was to predict the likelihood of future associations between machine learning concepts in a semantic network. The developed methodology corresponds to a solution for a scenario of availability of low computational power only, exploiting the extraction of low order topological features and its incorporation in an optimized classifier to estimate the degree of future connections between the nodes. The reasons that motivated the developed methodologies will be discussed, as well as some results, limitations and suggestions of improvements.
Universal Spam Detection using Transfer Learning of BERT Model
Venue: arXiv cs.CL 2022
Year: 2022

Deep learning transformer models become important by training on text data based on self-attention mechanisms. This manuscript demonstrated a novel universal spam detection model using pre-trained Google's Bidirectional Encoder Representations from Transformers (BERT) base uncased models with four datasets by efficiently classifying ham or spam emails in real-time scenarios. Different methods for Enron, Spamassain, Lingspam, and Spamtext message classification datasets, were used to train models individually in which a single model was obtained with acceptable performance on four datasets. The Universal Spam Detection Model (USDM) was trained with four datasets and leveraged hyperparameters from each model. The combined model was finetuned with the same hyperparameters from these four models separately. When each model using its corresponding dataset, an F1-score is at and above 0.9 in individual models. An overall accuracy reached 97%, with an F1 score of 0.96. Research results and implications were discussed.
Optimizing Warfarin Dosing using Deep Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Warfarin is a widely used anticoagulant, and has a narrow therapeutic range. Dosing of warfarin should be individualized, since slight overdosing or underdosing can have catastrophic or even fatal consequences. Despite much research on warfarin dosing, current dosing protocols do not live up to expectations, especially for patients sensitive to warfarin. We propose a deep reinforcement learning-based dosing model for warfarin. To overcome the issue of relatively small sample sizes in dosing trials, we use a Pharmacokinetic/ Pharmacodynamic (PK/PD) model of warfarin to simulate dose-responses of virtual patients. Applying the proposed algorithm on virtual test patients shows that this model outperforms a set of clinically accepted dosing protocols by a wide margin. We tested the robustness of our dosing protocol on a second PK/PD model and showed that its performance is comparable to the set of baseline protocols.
Four Geometry Problems to Introduce Automated Deduction in Secondary Schools
Venue: arXiv cs.AI 2022
Year: 2022

The introduction of automated deduction systems in secondary schools face several bottlenecks, the absence of the subject of rigorous mathematical demonstrations in the curricula, the lack of knowledge by the teachers about the subject and the difficulty of tackling the task by automatic means. Despite those difficulties we claim that the subject of automated deduction in geometry can be introduced, by addressing it in particular cases: simple to manipulate by students and teachers and reasonably easy to be dealt by automatic deduction tools. The subject is discussed by addressing four secondary schools geometry problems: their rigorous proofs, visual proofs, numeric proofs, algebraic formal proofs, synthetic formal proofs, or the lack of them. For these problems we discuss a lesson plan to address them with the help of Information and Communications Technology, more specifically, automated deduction tools.
Local Explanations for Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

Many works in explainable AI have focused on explaining black-box classification models. Explaining deep reinforcement learning (RL) policies in a manner that could be understood by domain users has received much less attention. In this paper, we propose a novel perspective to understanding RL policies based on identifying important states from automatically learned meta-states. The key conceptual difference between our approach and many previous ones is that we form meta-states based on locality governed by the expert policy dynamics rather than based on similarity of actions, and that we do not assume any particular knowledge of the underlying topology of the state space. Theoretically, we show that our algorithm to find meta-states converges and the objective that selects important states from each meta-state is submodular leading to efficient high quality greedy selection. Experiments on four domains (four rooms, door-key, minipacman, and pong) and a carefully conducted user study illustrate that our perspective leads to better understanding of the policy. We conjecture that this is a result of our meta-states being more intuitive in that the corresponding important states are strong indicators of tractable intermediate goals that are easier for humans to interpret and follow.
Codebook Design for Composite Beamforming in Next-generation mmWave Systems
Venue: arXiv cs.IT 2022
Year: 2022

In pursuance of the unused spectrum in higher frequencies, millimeter wave (mmWave) bands have a pivotal role. However, the high path-loss and poor scattering associated with mmWave communications highlight the necessity of employing effective beamforming techniques. In order to efficiently search for the beam to serve a user and to jointly serve multiple users it is often required to use a composite beam which consists of multiple disjoint lobes. A composite beam covers multiple desired angular coverage intervals (ACIs) and ideally has maximum and uniform gain (smoothness) within each desired ACI, negligible gain (leakage) outside the desired ACIs, and sharp edges. We propose an algorithm for designing such ideal composite codebook by providing an analytical closed-form solution with low computational complexity. There is a fundamental trade-off between the gain, leakage and smoothness of the beams. Our design allows to achieve different values in such trade-off based on changing the design parameters. We highlight the shortcomings of the uniform linear arrays (ULAs) in building arbitrary composite beams. Consequently, we use a recently introduced twin-ULA (TULA) antenna structure to effectively resolve these inefficiencies. Numerical results are used to validate the theoretical findings.
HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis
Venue: arXiv cs.CL 2022
Year: 2022

Contextualized word embeddings have demonstrated state-of-the-art performance in various natural language processing tasks including those that concern historical semantic change. However, language models such as BERT was trained primarily on contemporary corpus data. To investigate whether training on historical corpus data improves diachronic semantic analysis, we present a pre-trained BERT-based language model, HistBERT, trained on the balanced Corpus of Historical American English. We examine the effectiveness of our approach by comparing the performance of the original BERT and that of HistBERT, and we report promising results in word similarity and semantic shift analysis. Our work suggests that the effectiveness of contextual embeddings in diachronic semantic analysis is dependent on the temporal profile of the input text and care should be taken in applying this methodology to study historical semantic change.