OpenRaiser/Intern-Atlas huggingface dataset:
Unintended Memorization and Timing Attacks in Named Entity Recognition Models
Venue: arXiv cs.CR 2022
Year: 2022

Named entity recognition models (NER), are widely used for identifying named entities (e.g., individuals, locations, and other information) in text documents. Machine learning based NER models are increasingly being applied in privacy-sensitive applications that need automatic and scalable identification of sensitive information to redact text for data sharing. In this paper, we study the setting when NER models are available as a black-box service for identifying sensitive information in user documents and show that these models are vulnerable to membership inference on their training datasets. With updated pre-trained NER models from spaCy, we demonstrate two distinct membership attacks on these models. Our first attack capitalizes on unintended memorization in the NER's underlying neural network, a phenomenon NNs are known to be vulnerable to. Our second attack leverages a timing side-channel to target NER models that maintain vocabularies constructed from the training data. We show that different functional paths of words within the training dataset in contrast to words not previously seen have measurable differences in execution time. Revealing membership status of training samples has clear privacy implications, e.g., in text redaction, sensitive words or phrases to be found and removed, are at risk of being detected in the training dataset. Our experimental evaluation includes the redaction of both password and health data, presenting both security risks and privacy/regulatory issues. This is exacerbated by results that show memorization with only a single phrase. We achieved 70% AUC in our first attack on a text redaction use-case. We also show overwhelming success in the timing attack with 99.23% AUC. Finally we discuss potential mitigation approaches to realize the safe use of NER models in light of the privacy and security implications of membership inference attacks.
How Does Adaptive Optimization Impact Local Neural Network Geometry?
Venue: arXiv cs.LG 2022
Year: 2022

Adaptive optimization methods are well known to achieve superior convergence relative to vanilla gradient methods. The traditional viewpoint in optimization, particularly in convex optimization, explains this improved performance by arguing that, unlike vanilla gradient schemes, adaptive algorithms mimic the behavior of a second-order method by adapting to the global geometry of the loss function. We argue that in the context of neural network optimization, this traditional viewpoint is insufficient. Instead, we advocate for a local trajectory analysis. For iterate trajectories produced by running a generic optimization algorithm OPT, we introduce $R^{\text{OPT}}_{\text{med}}$, a statistic that is analogous to the condition number of the loss Hessian evaluated at the iterates. Through extensive experiments, we show that adaptive methods such as Adam bias the trajectories towards regions where $R^{\text{Adam}}_{\text{med}}$ is small, where one might expect faster convergence. By contrast, vanilla gradient methods like SGD bias the trajectories towards regions where $R^{\text{SGD}}_{\text{med}}$ is comparatively large. We complement these empirical observations with a theoretical result that provably demonstrates this phenomenon in the simplified setting of a two-layer linear network. We view our findings as evidence for the need of a new explanation of the success of adaptive methods, one that is different than the conventional wisdom.
Spectral Regularization: an Inductive Bias for Sequence Modeling
Venue: arXiv cs.LG 2022
Year: 2022

Various forms of regularization in learning tasks strive for different notions of simplicity. This paper presents a spectral regularization technique, which attaches a unique inductive bias to sequence modeling based on an intuitive concept of simplicity defined in the Chomsky hierarchy. From fundamental connections between Hankel matrices and regular grammars, we propose to use the trace norm of the Hankel matrix, the tightest convex relaxation of its rank, as the spectral regularizer. To cope with the fact that the Hankel matrix is bi-infinite, we propose an unbiased stochastic estimator for its trace norm. Ultimately, we demonstrate experimental results on Tomita grammars, which exhibit the potential benefits of spectral regularization and validate the proposed stochastic estimator.
An Efficient FPGA-based Accelerator for Deep Forest
Venue: arXiv cs.LG 2022
Year: 2022

Deep Forest is a prominent machine learning algorithm known for its high accuracy in forecasting. Compared with deep neural networks, Deep Forest has almost no multiplication operations and has better performance on small datasets. However, due to the deep structure and large forest quantity, it suffers from large amounts of calculation and memory consumption. In this paper, an efficient hardware accelerator is proposed for deep forest models, which is also the first work to implement Deep Forest on FPGA. Firstly, a delicate node computing unit (NCU) is designed to improve inference speed. Secondly, based on NCU, an efficient architecture and an adaptive dataflow are proposed, in order to alleviate the problem of node computing imbalance in the classification process. Moreover, an optimized storage scheme in this design also improves hardware utilization and power efficiency. The proposed design is implemented on an FPGA board, Intel Stratix V, and it is evaluated by two typical datasets, ADULT and Face Mask Detection. The experimental results show that the proposed design can achieve around 40x speedup compared to that on a 40 cores high performance x86 CPU.
Experiment of Multi-UAV Full-Duplex System Equipped with Directional Antennas
Venue: arXiv cs.IT 2022
Year: 2022

One of the key enablers for the realization of a variety of unmanned aerial vehicle (UAV)-based systems is the high-performance communication system linking many UAVs and ground station. We have proposed a spectrum-efficient full-duplex directional-antennas-equipped multi-UAV communication system with low hardware complexity to address the issues of low spectrum efficiency caused by co-channel interference in areal channels. In this paper, by using the prototype system including UAVs and ground station, field experiments are carried out to confirm the feasibility and effectiveness of the proposed system's key feature, i.e., co-channel interference cancellation among UAVs by directional antennas and UAV relative position control, instead of energy-consuming dedicated self-interference cancellers on UAVs in traditional full-duplex systems. Both uplink and downlink performance are tested. Specially, in downlink experiment, channel power of interference between a pair of two UAVs is measured when UAVs are in different positional relationships. The experiment results agree well with the designs and confirm that the proposed system can greatly improve the system performance.
Better Call Saltzer \& Schroeder: A Retrospective Security Analysis of SolarWinds \& Log4j
Venue: arXiv cs.SE 2022
Year: 2022

Saltzer \& Schroeder's principles aim to bring security to the design of computer systems. We investigate SolarWinds Orion update and Log4j to unpack the intersections where observance of these principles could have mitigated the embedded vulnerabilities. The common principles that were not observed include \emph{fail safe defaults}, \emph{economy of mechanism}, \emph{complete mediation} and \emph{least privilege}. Then we explore the literature on secure software development interventions for developers to identify usable analysis tools and frameworks that can contribute towards improved observance of these principles. We focus on a system wide view of access of codes, checking access paths and aiding application developers with safe libraries along with an appropriate security task list for functionalities.
A General Purpose Neural Architecture for Geospatial Systems
Venue: arXiv cs.LG 2022
Year: 2022

Geospatial Information Systems are used by researchers and Humanitarian Assistance and Disaster Response (HADR) practitioners to support a wide variety of important applications. However, collaboration between these actors is difficult due to the heterogeneous nature of geospatial data modalities (e.g., multi-spectral images of various resolutions, timeseries, weather data) and diversity of tasks (e.g., regression of human activity indicators or detecting forest fires). In this work, we present a roadmap towards the construction of a general-purpose neural architecture (GPNA) with a geospatial inductive bias, pre-trained on large amounts of unlabelled earth observation data in a self-supervised manner. We envision how such a model may facilitate cooperation between members of the community. We show preliminary results on the first step of the roadmap, where we instantiate an architecture that can process a wide variety of geospatial data modalities and demonstrate that it can achieve competitive performance with domain-specific architectures on tasks relating to the U.N.'s Sustainable Development Goals.
Did your child get disturbed by an inappropriate advertisement on YouTube?
Venue: arXiv cs.CY 2022
Year: 2022

YouTube is a popular video platform for sharing creative content and ideas, targeting different demographics. Adults, older children, and young children are all avid viewers of YouTube videos. Meanwhile, countless young-kid-oriented channels have produced numerous instructional and age appropriate videos for young children. However, inappropriate content for young children, such as violent or sexually suggestive content, still exists. And children lack the ability to decide whether a video is appropriate for them or not, which then causes a huge risk to children's mental health. Prior works have focused on identifying YouTube videos that are inappropriate for children. However, these works ignore that not only the actual video content influences children, but also the advertisements that are shown with those videos. In this paper, we quantify the influence of inappropriate advertisements on YouTube videos that are appropriate for young children to watch. We analyze the advertising patterns of 24.6 K diverse YouTube videos appropriate for young children. We find that 9.9% of the 4.6 K unique advertisements shown on these 24.6 K videos contain inappropriate content for young children. Moreover, we observe that 26.9% of all the 24.6 K appropriate videos include at least one ad that is inappropriate for young children. Additionally, we publicly release our datasets and provide recommendations about how to address this issue.
A Jigsaw Puzzle Solver-based Attack on Block-wise Image Encryption for Privacy-preserving DNNs
Venue: arXiv cs.CR 2022
Year: 2022

Privacy-preserving deep neural networks (DNNs) have been proposed for protecting data privacy in the cloud server. Although several encryption schemes for visually protection have been proposed for privacy-preserving DNNs, several attacks enable to restore visual information from encrypted images. On the other hand, it has been confirmed that the block-wise image encryption scheme which utilizes block and pixel shuffling is robust against several attacks. In this paper, we propose a jigsaw puzzle solver-based attack to restore visual information from encrypted images including block and pixel shuffling. In experiments, images encrypted by using the block-wise image encryption are mostly restored by using the proposed attack.
Path Planning Using Wassertein Distributionally Robust Deep Q-learning
Venue: arXiv cs.RO 2022
Year: 2022

We investigate the problem of risk averse robot path planning using the deep reinforcement learning and distributionally robust optimization perspectives. Our problem formulation involves modelling the robot as a stochastic linear dynamical system, assuming that a collection of process noise samples is available. We cast the risk averse motion planning problem as a Markov decision process and propose a continuous reward function design that explicitly takes into account the risk of collision with obstacles while encouraging the robot's motion towards the goal. We learn the risk-averse robot control actions through Lipschitz approximated Wasserstein distributionally robust deep Q-learning to hedge against the noise uncertainty. The learned control actions result in a safe and risk averse trajectory from the source to the goal, avoiding all the obstacles. Various supporting numerical simulations are presented to demonstrate our proposed approach.
Explainable Information Retrieval: A Survey
Venue: arXiv cs.IR 2022
Year: 2022

Explainable information retrieval is an emerging research area aiming to make transparent and trustworthy information retrieval systems. Given the increasing use of complex machine learning models in search systems, explainability is essential in building and auditing responsible information retrieval models. This survey fills a vital gap in the otherwise topically diverse literature of explainable information retrieval. It categorizes and discusses recent explainability methods developed for different application domains in information retrieval, providing a common framework and unifying perspectives. In addition, it reflects on the common concern of evaluating explanations and highlights open challenges and opportunities.
Dealing with Abbreviations in the Slovenian Biographical Lexicon
Venue: arXiv cs.CL 2022
Year: 2022

Abbreviations present a significant challenge for NLP systems because they cause tokenization and out-of-vocabulary errors. They can also make the text less readable, especially in reference printed books, where they are extensively used. Abbreviations are especially problematic in low-resource settings, where systems are less robust to begin with. In this paper, we propose a new method for addressing the problems caused by a high density of domain-specific abbreviations in a text. We apply this method to the case of a Slovenian biographical lexicon and evaluate it on a newly developed gold-standard dataset of 51 Slovenian biographies. Our abbreviation identification method performs significantly better than commonly used ad-hoc solutions, especially at identifying unseen abbreviations. We also propose and present the results of a method for expanding the identified abbreviations in context.
Advanced Automatic Code Generation for Multiple Relaxation-Time Lattice Boltzmann Methods
Venue: arXiv cs.MS 2022
Year: 2022

The scientific code generation package lbmpy supports the automated design and the efficient implementation of lattice Boltzmann methods (LBMs) through metaprogramming. It is based on a new, concise calculus for describing multiple relaxation-time LBMs, including techniques that enable the numerically advantageous subtraction of the constant background component from the populations. These techniques are generalized to a wide range of collision spaces and equilibrium distributions. The article contains an overview of lbmpy's front-end and its code generation pipeline, which implements the new LBM calculus by means of symbolic formula manipulation tools and object-oriented programming. The generated codes have only a minimal number of arithmetic operations. Their automatic derivation rests on two novel Chimera transforms that have been specifically developed for efficiently computing raw and central moments. Information contained in the symbolic representation of the methods is further exploited in a customized sequence of algebraic simplifications, further reducing computational cost. When combined, these algebraic transformations lead to concise and compact numerical kernels. Specifically, with these optimizations, the advanced central moment- and cumulant-based methods can be realized with only little additional cost as when compared with the simple BGK method. The effectiveness and flexibility of the new lbmpy code generation system is demonstrated in simulating Taylor-Green vortex decay and the automatic derivation of an LBM algorithm to solve the shallow water equations.
Network Aware Compute and Memory Allocation in Optically Composable Data Centres with Deep Reinforcement Learning and Graph Neural Networks
Venue: arXiv cs.NI 2022
Year: 2022

Resource-disaggregated data centre architectures promise a means of pooling resources remotely within data centres, allowing for both more flexibility and resource efficiency underlying the increasingly important infrastructure-as-a-service business. This can be accomplished by means of using an optically circuit switched backbone in the data centre network (DCN); providing the required bandwidth and latency guarantees to ensure reliable performance when applications are run across non-local resource pools. However, resource allocation in this scenario requires both server-level \emph{and} network-level resource to be co-allocated to requests. The online nature and underlying combinatorial complexity of this problem, alongside the typical scale of DCN topologies, makes exact solutions impossible and heuristic based solutions sub-optimal or non-intuitive to design. We demonstrate that \emph{deep reinforcement learning}, where the policy is modelled by a \emph{graph neural network} can be used to learn effective \emph{network-aware} and \emph{topologically-scalable} allocation policies end-to-end. Compared to state-of-the-art heuristics for network-aware resource allocation, the method achieves up to $20\%$ higher acceptance ratio; can achieve the same acceptance ratio as the best performing heuristic with $3\times$ less networking resources available and can maintain all-around performance when directly applied (with no further training) to DCN topologies with $10^2\times$ more servers than the topologies seen during training.
Open Multi-Access Network Platform with Dynamic Task Offloading and Intelligent Resource Monitoring
Venue: arXiv cs.NI 2022
Year: 2022

We constructed an open multi-access network platform using open-source hardware and software. The open multi-access network platform is characterized by the flexible utilization of network functions, integral management and control of wired and wireless access networks, zero-touch provisioning, intelligent resource monitoring, and dynamic task offloading. We also propose an application-driven dynamic task offloading that utilizes intelligent resource monitoring to ensure effective task processing in edge and cloud servers. For this purpose, we developed a mobile application and server applications for the open multi-access network platform. To investigate the feasibility and availability of our developed platform, we experimentally and analytically evaluated the effectiveness of application-driven dynamic task offloading and intelligent resource monitoring. The experimental results demonstrated that application-driven dynamic task offloading could reduce real-time task response time and traffic over metro and core networks.
Biased Self-supervised learning for ASR
Venue: arXiv cs.CL 2022
Year: 2022

Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.
Generation of Chinese classical poetry based on pre-trained model
Venue: arXiv cs.CL 2022
Year: 2022

In order to test whether artificial intelligence can create qualified classical poetry like humans, the author proposes a study of Chinese classical poetry generation based on a pre-trained model. This paper mainly tries to use BART and other pre training models, proposes FS2TEXT and RR2TEXT to generate metrical poetry text and even specific style poetry text, and solves the problem that the user's writing intention gradually reduces the relevance of the generated poetry text. In order to test the model's results, the authors selected ancient poets, by combining it with BART's poetic model work, developed a set of AI poetry Turing problems, it was reviewed by a group of poets and poetry writing researchers. There were more than 600 participants, and the final results showed that, high-level poetry lovers can't distinguish between AI activity and human activity, this indicates that the author's working methods are not significantly different from human activities. The model of poetry generation studied by the author generalizes works that cannot be distinguished from those of advanced scholars. The number of modern Chinese poets has reached 5 million. However, many modern Chinese poets lack language ability and skills as a result of their childhood learning. However, many modern poets have no creative inspiration, and the author's model can help them. They can look at this model when they choose words and phrases and they can write works based on the poems they already have, and they can write their own poems. The importance of poetry lies in the author's thoughts and reflections. It doesn't matter how good AI poetry is. The only thing that matters is for people to see and inspire them.
A Data-Driven Slip Estimation Approach for Effective Braking Control under Varying Road Conditions
Venue: arXiv cs.RO 2022
Year: 2022

The performances of braking control systems for robotic platforms, e.g., assisted and autonomous vehicles, airplanes and drones, are deeply influenced by the road-tire friction experienced during the maneuver. Therefore, the availability of accurate estimation algorithms is of major importance in the development of advanced control schemes. The focus of this paper is on the estimation problem. In particular, a novel estimation algorithm is proposed, based on a multi-layer neural network. The training is based on a synthetic data set, derived from a widely used friction model. The open loop performances of the proposed algorithm are evaluated in a number of simulated scenarios. Moreover, different control schemes are used to test the closed loop scenario, where the estimated optimal slip is used as the set-point. The experimental results and the comparison with a model based baseline show that the proposed approach can provide an effective best slip estimation.