OpenRaiser/Intern-Atlas huggingface dataset:
Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning
Venue: arXiv cs.LG 2021
Year: 2021

A commonly cited inefficiency of neural network training using back-propagation is the update locking problem: each layer must wait for the signal to propagate through the full network before updating. Several alternatives that can alleviate this issue have been proposed. In this context, we consider a simple alternative based on minimal feedback, which we call Decoupled Greedy Learning (DGL). It is based on a classic greedy relaxation of the joint training objective, recently shown to be effective in the context of Convolutional Neural Networks (CNNs) on large-scale image classification. We consider an optimization of this objective that permits us to decouple the layer training, allowing for layers or modules in networks to be trained with a potentially linear parallelization. With the use of a replay buffer we show that this approach can be extended to asynchronous settings, where modules can operate and continue to update with possibly large communication delays. To address bandwidth and memory issues we propose an approach based on online vector quantization. This allows to drastically reduce the communication bandwidth between modules and required memory for replay buffers. We show theoretically and empirically that this approach converges and compare it to the sequential solvers. We demonstrate the effectiveness of DGL against alternative approaches on the CIFAR-10 dataset and on the large-scale ImageNet dataset.
FPGA-Based Near-Memory Acceleration of Modern Data-Intensive Applications
Venue: arXiv cs.AR 2021
Year: 2021

Modern data-intensive applications demand high computation capabilities with strict power constraints. Unfortunately, such applications suffer from a significant waste of both execution cycles and energy in current computing systems due to the costly data movement between the computation units and the memory units. Genome analysis and weather prediction are two examples of such applications. Recent FPGAs couple a reconfigurable fabric with high-bandwidth memory (HBM) to enable more efficient data movement and improve overall performance and energy efficiency. This trend is an example of a paradigm shift to near-memory computing. We leverage such an FPGA with high-bandwidth memory (HBM) for improving the pre-alignment filtering step of genome analysis and representative kernels from a weather prediction model. Our evaluation demonstrates large speedups and energy savings over a high-end IBM POWER9 system and a conventional FPGA board with DDR4 memory. We conclude that FPGA-based near-memory computing has the potential to alleviate the data movement bottleneck for modern data-intensive applications.
Property-Preserving Hash Functions from Standard Assumptions
Venue: arXiv cs.CR 2021
Year: 2021

Property-preserving hash functions allow for compressing long inputs $x_0$ and $x_1$ into short hashes $h(x_0)$ and $h(x_1)$ in a manner that allows for computing a predicate $P(x_0, x_1)$ given only the two hash values without having access to the original data. Such hash functions are said to be adversarially robust if an adversary that gets to pick $x_0$ and $x_1$ after the hash function has been sampled, cannot find inputs for which the predicate evaluated on the hash values outputs the incorrect result. In this work we construct robust property-preserving hash functions for the hamming-distance predicate which distinguishes inputs with a hamming distance at least some threshold $t$ from those with distance less than $t$. The security of the construction is based on standard lattice hardness assumptions. Our construction has several advantages over the best known previous construction by Fleischhacker and Simkin. Our construction relies on a single well-studied hardness assumption from lattice cryptography whereas the previous work relied on a newly introduced family of computational hardness assumptions. In terms of computational effort, our construction only requires a small number of modular additions per input bit, whereas previously several exponentiations per bit as well as the interpolation and evaluation of high-degree polynomials over large fields were required. An additional benefit of our construction is that the description of the hash function can be compressed to $λ$ bits assuming a random oracle. Previous work has descriptions of length $\mathcal{O}(\ell λ)$ bits for input bit-length $\ell$, which has a secret structure and thus cannot be compressed. We prove a lower bound on the output size of any property-preserving hash function for the hamming distance predicate. The bound shows that the size of our hash value is not far from optimal.
Semi-Supervised and Unsupervised Sense Annotation via Translations
Venue: arXiv cs.CL 2021
Year: 2021

Acquisition of multilingual training data continues to be a challenge in word sense disambiguation (WSD). To address this problem, unsupervised approaches have been proposed to automatically generate sense annotations for training supervised WSD systems. We present three new methods for creating sense-annotated corpora which leverage translations, parallel bitexts, lexical resources, as well as contextual and synset embeddings. Our semi-supervised method applies machine translation to transfer existing sense annotations to other languages. Our two unsupervised methods refine sense annotations produced by a knowledge-based WSD system via lexical translations in a parallel corpus. We obtain state-of-the-art results on standard WSD benchmarks.
Data-Driven Multiscale Design of Cellular Composites with Multiclass Microstructures for Natural Frequency Maximization
Venue: arXiv cs.CE 2021
Year: 2021

For natural frequency optimization of engineering structures, cellular composites have been shown to possess an edge over solid. However, existing multiscale design methods for cellular composites are either computationally exhaustive or confined to a single class of microstructures. In this paper, we propose a data-driven topology optimization (TO) approach to enable the multiscale design of cellular structures with various choices of microstructure classes. The key component is a newly proposed latent-variable Gaussian process (LVGP) model through which different classes of microstructures are mapped into a low-dimensional continuous latent space. It provides an interpretable distance metric between classes and captures their effects on the homogenized stiffness tensors. By introducing latent vectors as design variables, a differentiable transition of stiffness matrix between classes can be easily achieved with an analytical gradient. After integrating LVGP with the density-based TO, an efficient data-driven cellular composite optimization process is developed to enable concurrent exploration of microstructure concepts and the associated volume fractions for natural frequency optimization. Examples reveal that the proposed cellular designs with multiclass microstructures achieve higher natural frequencies than both single-scale and single-class designs. This framework can be easily extended to other multi-scale TO problems, such as thermal compliance and dynamic response optimization.
Shallow Optical Flow Three-Stream CNN for Macro- and Micro-Expression Spotting from Long Videos
Venue: arXiv cs.CV 2021
Year: 2021

Facial expressions vary from the visible to the subtle. In recent years, the analysis of micro-expressions $-$ a natural occurrence resulting from the suppression of one's true emotions, has drawn the attention of researchers with a broad range of potential applications. However, spotting microexpressions in long videos becomes increasingly challenging when intertwined with normal or macro-expressions. In this paper, we propose a shallow optical flow three-stream CNN (SOFTNet) model to predict a score that captures the likelihood of a frame being in an expression interval. By fashioning the spotting task as a regression problem, we introduce pseudo-labeling to facilitate the learning process. We demonstrate the efficacy and efficiency of the proposed approach on the recent MEGC 2020 benchmark, where state-of-the-art performance is achieved on CAS(ME)$^{2}$ with equally promising results on SAMM Long Videos.
An adaptive cognitive sensor node for ECG monitoring in the Internet of Medical Things
Venue: arXiv cs.LG 2021
Year: 2021

The Internet of Medical Things (IoMT) paradigm is becoming mainstream in multiple clinical trials and healthcare procedures. Cardiovascular diseases monitoring, usually involving electrocardiogram (ECG) traces analysis, is one of the most promising and high-impact applications. Nevertheless, to fully exploit the potential of IoMT in this domain, some steps forward are needed. First, the edge-computing paradigm must be added to the picture. A certain level of near-sensor processing has to be enabled, to improve the scalability, portability, reliability, responsiveness of the IoMT nodes. Second, novel, increasingly accurate, data analysis algorithms, such as those based on artificial intelligence and Deep Learning, must be exploited. To reach these objectives, designers and programmers of IoMT nodes, have to face challenging optimization tasks, in order to execute fairly complex computing tasks on low-power wearable and portable processing systems, with tight power and battery lifetime budgets. In this work, we explore the implementation of a cognitive data analysis algorithm, based on a convolutional neural network trained to classify ECG waveforms, on a resource-constrained microcontroller-based computing platform. To minimize power consumption, we add an adaptivity layer that dynamically manages the hardware and software configuration of the device to adapt it at runtime to the required operating mode. Our experimental results show that adapting the node setup to the workload at runtime can save up to 50% power consumption. Our optimized and quantized neural network reaches an accuracy value higher than 97% for arrhythmia disorders detection on MIT-BIH Arrhythmia dataset.
Unsupervised Neural Hidden Markov Models with a Continuous latent state space
Venue: arXiv cs.LG 2021
Year: 2021

We introduce a new procedure to neuralize unsupervised Hidden Markov Models in the continuous case. This provides higher flexibility to solve problems with underlying latent variables. This approach is evaluated on both synthetic and real data. On top of generating likely model parameters with comparable performances to off-the-shelf neural architecture (LSTMs, GRUs,..), the obtained results are easily interpretable.
Bucket-brigade inspired power line network protocol for sensed quantity profile acquisition with smart sensors deployed as a queue in harsh environment
Venue: arXiv cs.NI 2021
Year: 2021

Pressure and temperature profile are key data for safe production in oil and gas wells. In this paper, a bucket-brigade inspired sensor network protocol is proposed which can be used to extract sensed data profile from the nanoscale up to kilometer long structures. The PHY/MAC layers are discussed. This protocol is best suited for low data rate exchanges in small fixed-size packets, named buckets, transmitted as time-domain bursts among high-precision smart sensors deployed as a queue. There is only one coordinator, which is not directly accessible by most of the sensor nodes. The coordinator is responsible for collecting the measurement profile and send it to a supervisory node. There is no need for complex routing mechanism, as the network topology is determined during deployment. There are many applications which require sensors to be deployed as a long queue and sensed data could be transmitted at low data rates. Examples of such monitoring applications are: neural connected artificial skin, oil/gas/water pipeline integrity, power transmission line tower integrity, (rail)road/highway lighting and integrity, individualized monitoring in vineyard or re-foresting or plantation, underwater telecommunications cable integrity, oil/gas riser integrity, oil/gas well temperature and pressure profile, among others. For robustness and reduced electromagnetic interference, wired network is preferred. Besides in some harsh environment wireless is not feasible. To reduce wiring, communications can be carried out in the same cable used to supply electrical power.
Relaxing Local Robustness
Venue: arXiv cs.LG 2021
Year: 2021

Certifiable local robustness, which rigorously precludes small-norm adversarial examples, has received significant attention as a means of addressing security concerns in deep learning. However, for some classification problems, local robustness is not a natural objective, even in the presence of adversaries; for example, if an image contains two classes of subjects, the correct label for the image may be considered arbitrary between the two, and thus enforcing strict separation between them is unnecessary. In this work, we introduce two relaxed safety properties for classifiers that address this observation: (1) relaxed top-k robustness, which serves as the analogue of top-k accuracy; and (2) affinity robustness, which specifies which sets of labels must be separated by a robustness margin, and which can be $ε$-close in $\ell_p$ space. We show how to construct models that can be efficiently certified against each relaxed robustness property, and trained with very little overhead relative to standard gradient descent. Finally, we demonstrate experimentally that these relaxed variants of robustness are well-suited to several significant classification problems, leading to lower rejection rates and higher certified accuracies than can be obtained when certifying "standard" local robustness.
Provable Adaptation across Multiway Domains via Representation Learning
Venue: arXiv cs.LG 2021
Year: 2021

This paper studies zero-shot domain adaptation where each domain is indexed on a multi-dimensional array, and we only have data from a small subset of domains. Our goal is to produce predictors that perform well on \emph{unseen} domains. We propose a model which consists of a domain-invariant latent representation layer and a domain-specific linear prediction layer with a low-rank tensor structure. Theoretically, we present explicit sample complexity bounds to characterize the prediction error on unseen domains in terms of the number of domains with training data and the number of data per domain. To our knowledge, this is the first finite-sample guarantee for zero-shot domain adaptation. In addition, we provide experiments on two-way MNIST and four-way fiber sensing datasets to demonstrate the effectiveness of our proposed model.
Neural Combinatory Constituency Parsing
Venue: arXiv cs.CL 2021
Year: 2021

We propose two fast neural combinatory models for constituency parsing: binary and multi-branching. Our models decompose the bottom-up parsing process into 1) classification of tags, labels, and binary orientations or chunks and 2) vector composition based on the computed orientations or chunks. These models have theoretical sub-quadratic complexity and empirical linear complexity. The binary model achieves an F1 score of 92.54 on Penn Treebank, speeding at 1327.2 sents/sec. Both the models with XLNet provide near state-of-the-art accuracies for English. Syntactic branching tendency and headedness of a language are observed during the training and inference processes for Penn Treebank, Chinese Treebank, and Keyaki Treebank (Japanese).
Towards a Privacy-preserving Deep Learning-based Network Intrusion Detection in Data Distribution Services
Venue: arXiv cs.LG 2021
Year: 2021

Data Distribution Service (DDS) is an innovative approach towards communication in ICS/IoT infrastructure and robotics. Being based on the cross-platform and cross-language API to be applicable in any computerised device, it offers the benefits of modern programming languages and the opportunities to develop more complex and advanced systems. However, the DDS complexity equally increases its vulnerability, while the existing security measures are limited to plug-ins and static rules, with the rest of the security provided by third-party applications and operating system. Specifically, traditional intrusion detection systems (IDS) do not detect any anomalies in the publish/subscribe method. With the exponentially growing global communication exchange, securing DDS is of the utmost importance to futureproofing industrial, public, and even personal devices and systems. This report presents an experimental work on the simulation of several specific attacks against DDS, and the application of Deep Learning for their detection. The findings show that even though Deep Learning allows to detect all simulated attacks using only metadata analysis, their detection level varies, with some of the advanced attacks being harder to detect. The limitations imposed by the attempts to preserve privacy significantly decrease the detection rate. The report also reviews the drawbacks and limitations of the Deep Learning approach and proposes a set of selected solutions and configurations, that can further improve the DDS security.
Joint Client Scheduling and Resource Allocation under Channel Uncertainty in Federated Learning
Venue: arXiv cs.LG 2021
Year: 2021

The performance of federated learning (FL) over wireless networks depend on the reliability of the client-server connectivity and clients' local computation capabilities. In this article we investigate the problem of client scheduling and resource block (RB) allocation to enhance the performance of model training using FL, over a pre-defined training duration under imperfect channel state information (CSI) and limited local computing resources. First, we analytically derive the gap between the training losses of FL with clients scheduling and a centralized training method for a given training duration. Then, we formulate the gap of the training loss minimization over client scheduling and RB allocation as a stochastic optimization problem and solve it using Lyapunov optimization. A Gaussian process regression-based channel prediction method is leveraged to learn and track the wireless channel, in which, the clients' CSI predictions and computing power are incorporated into the scheduling decision. Using an extensive set of simulations, we validate the robustness of the proposed method under both perfect and imperfect CSI over an array of diverse data distributions. Results show that the proposed method reduces the gap of the training accuracy loss by up to 40.7% compared to state-of-theart client scheduling and RB allocation methods.
Quantifying the Conceptual Error in Dimensionality Reduction
Venue: arXiv cs.LG 2021
Year: 2021

Dimension reduction of data sets is a standard problem in the realm of machine learning and knowledge reasoning. They affect patterns in and dependencies on data dimensions and ultimately influence any decision-making processes. Therefore, a wide variety of reduction procedures are in use, each pursuing different objectives. A so far not considered criterion is the conceptual continuity of the reduction mapping, i.e., the preservation of the conceptual structure with respect to the original data set. Based on the notion scale-measure from formal concept analysis we present in this work a) the theoretical foundations to detect and quantify conceptual errors in data scalings; b) an experimental investigation of our approach on eleven data sets that were respectively treated with a variant of non-negative matrix factorization.
A Low-Compexity Deep Learning Framework For Acoustic Scene Classification
Venue: arXiv cs.SD 2021
Year: 2021

In this paper, we presents a low-complexity deep learning frameworks for acoustic scene classification (ASC). The proposed framework can be separated into three main steps: Front-end spectrogram extraction, back-end classification, and late fusion of predicted probabilities. First, we use Mel filter, Gammatone filter and Constant Q Transfrom (CQT) to transform raw audio signal into spectrograms, where both frequency and temporal features are presented. Three spectrograms are then fed into three individual back-end convolutional neural networks (CNNs), classifying into ten urban scenes. Finally, a late fusion of three predicted probabilities obtained from three CNNs is conducted to achieve the final classification result. To reduce the complexity of our proposed CNN network, we apply two model compression techniques: model restriction and decomposed convolution. Our extensive experiments, which are conducted on DCASE 2021 (IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events) Task 1A development dataset, achieve a low-complexity CNN based framework with 128 KB trainable parameters and the best classification accuracy of 66.7%, improving DCASE baseline by 19.0%
Deep Learning Frameworks Applied For Audio-Visual Scene Classification
Venue: arXiv cs.SD 2021
Year: 2021

In this paper, we present deep learning frameworks for audio-visual scene classification (SC) and indicate how individual visual and audio features as well as their combination affect SC performance. Our extensive experiments, which are conducted on DCASE (IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events) Task 1B development dataset, achieve the best classification accuracy of 82.2%, 91.1%, and 93.9% with audio input only, visual input only, and both audio-visual input, respectively. The highest classification accuracy of 93.9%, obtained from an ensemble of audio-based and visual-based frameworks, shows an improvement of 16.5% compared with DCASE baseline.
Short-term forecasting of global solar irradiance with incomplete data
Venue: arXiv cs.LG 2021
Year: 2021

Accurate mechanisms for forecasting solar irradiance and insolation provide important information for the planning of renewable energy and agriculture projects as well as for environmental and socio-economical studies. This research introduces a pipeline for the one-day ahead forecasting of solar irradiance and insolation that only requires solar irradiance historical data for training. Furthermore, our approach is able to deal with missing data since it includes a data imputation state. In the prediction stage, we consider four data-driven approaches: Autoregressive Integrated Moving Average (ARIMA), Single Layer Feed Forward Network (SL-FNN), Multiple Layer Feed Forward Network (FL-FNN), and Long Short-Term Memory (LSTM). The experiments are performed in a real-world dataset collected with 12 Automatic Weather Stations (AWS) located in the Nariño - Colombia. The results show that the neural network-based models outperform ARIMA in most cases. Furthermore, LSTM exhibits better performance in cloudy environments (where more randomness is expected).